Korean, Edit

MinIO Useful Functions Collection

Recommended Post: 【Python】 Python Table of Contents

1. MinIO

2. Collection of Useful Python Functions

3. Collection of Useful Terminal Commands


1. MinIO

⑴ AWS S3-compatible object storage system

⑵ Higher version of Synology NAS

① Reason 1. Can be set to make deletion impossible once uploaded

② Reason 2. Higher level of security

⑶ Usage

① Step 1. Install MinIO Client here: If you’re using Linux Intel


curl https://dl.min.io/client/mc/release/linux-amd64/mc \
  --create-dirs \
  -o $HOME/minio-binaries/mc

chmod +x $HOME/minio-binaries/mc
export PATH=$PATH:$HOME/minio-binaries/

mc --help


② Step 2. Generate MinIO Access Key, Secret Key

③ Step 3. mc alias set [MinIO Name] [MinIO URL] [ACCESSKEY] [SECRETKEY]

④ Step 4. Refer below

⑤ Related questions to ChatGPT result in really well-written code



2. Collection of Useful Python Functions

⑴ Download from MinIO (ver. 1)


!pip install boto3 minio
import boto3
from botocore.client import Config

def download_from_minio(endpoint_url, access_key, secret_key, bucket_name, object_name, file_path):
    s3 = boto3.resource('s3',
                        endpoint_url=endpoint_url,
                        aws_access_key_id=access_key,
                        aws_secret_access_key=secret_key,
                        config=Config(signature_version='s3v4'),
                        region_name='us-east-1')  # You can adjust the region accordingly.

    try:
        s3.Bucket(bucket_name).download_file(object_name, file_path)
        print(f"{object_name} has been downloaded successfully to {file_path}")
    except Exception as e:
        print(f"An error occurred: {e}")

endpoint_url = 'http://your-minio-server.com:port'
access_key = 'YOUR_ACCESS_KEY'
secret_key = 'YOUR_SECRET_KEY'
bucket_name = 'YOUR_BUCKET_NAME'
object_name = 'YOUR_OBJECT_NAME'
file_path = '/path/on/your/local/machine' 

download_from_minio(endpoint_url, access_key, secret_key, bucket_name, object_name, file_path)


① access_key, secret_key

○ Use MinIO > Access Keys > Create access key to generate

○ You can only know the secret_key when it is first generated, so make sure to note it

② object_name

○ Example : “https://s3.AAA.co.kr/browser/bucket_name/folder1/subfolder2/myobject.txt”

○ object_name is “folder1/subfolder2/myobject.txt”

③ file_path

○ Example: “./myobject.txt”

④ Troubleshooting 1. An error occurred: An error occurred (400) when calling the HeadObject operation: Bad Request

○ Cause: The object_name is not appropriate

⑤ Troubleshooting 2. An error occurred: An error occurred (404) when calling the HeadObject operation: Not Found

○ Cause: The object_name is not appropriate

⑥ Troubleshooting 3. An error occurred: [Errno 20] Not a directory:

○ Cause: The file_path is not appropriate

⑵ Download from MinIO (ver. 2)


from minio import Minio
client = Minio(
    "your_minio_server.com:port",
    access_key="",
    secret_key="",
    region="gabia",
)
file = client.get_object('bucket', 'file_path')
with open(f'file_name, 'wb') as f:
    f.write(file.data)


⑶ Upload to MinIO

① Method 1. put_object


! pip install minio
from minio import Minio
from minio.error import S3Error

endpoint_url = 'http://your-minio-server.com:port'
access_key = 'YOUR_ACCESS_KEY'
secret_key = 'YOUR_SECRET_KEY'
bucket_name = 'YOUR_BUCKET_NAME'
object_name = 'YOUR_OBJECT_NAME'
file_path = 'myfile.txt' 

client = Minio(
    endpoint_url,
    access_key=access_key,
    secret_key=secret_key,
    secure=True,
)

with open(file_path, "rb") as file_data:
    file_stat = os.stat(file_path)
    client.put_object(bucket_name, object_name, file_data, file_stat.st_size)


② Method 2. fput_object


! pip install minio
from minio import Minio
from minio.error import S3Error

endpoint_url = 'http://your-minio-server.com:port'
access_key = 'YOUR_ACCESS_KEY'
secret_key = 'YOUR_SECRET_KEY'
bucket_name = 'YOUR_BUCKET_NAME'
object_name = 'YOUR_OBJECT_NAME'
file_path = 'myfile.txt' 

client = Minio(
    endpoint_url,
    access_key=access_key,
    secret_key=secret_key,
    secure=True,
)

client.fput_object(bucket_name, object_name, file_path)


③ Troubleshooting 1. InvalidEndpointError: InvalidEndpointError: message: Hostname cannot have a scheme.

○ Cause: You should use endpoint_url = "my-minio-server.com" instead of endpoint_url = "https://my-minio-server.com"

○ More specifically, use endpoint_url = "my-minio-server.com:port" including the port number

④ Troubleshooting 2. InvalidArgument: InvalidArgument: message: Invalid Argument

○ In my case, the message appeared when I omitted the port number in endpoint_url

⑤ Troubleshooting 3. InvalidArgumentError: InvalidArgumentError: message: Part size

  • max_parts(10000) is lesser than input length.

○ There are quite a few cases where uploading to MinIO with Python does not work depending on the file size

⑷ Print error logs


import boto3
import logging

# Set up logging
logging.basicConfig(level=logging.DEBUG)
boto3.set_stream_logger('botocore', level='DEBUG')


⑸ Disable error log printing


# Get root logger
logger = logging.getLogger()
while logger.hasHandlers():
    logger.removeHandler(logger.handlers[0])

# Also, reset boto3 and botocore loggers
for log_name in ['boto3', 'botocore']:
    logger = logging.getLogger(log_name)
    while logger.hasHandlers():
        logger.removeHandler(logger.handlers[0])


⑹ Troubleshooting

① MaxRetryError: HTTPConnectionPool(host='#.#.#.#', port={port}): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at #>: Failed to establish a new connection: [WinError #] The connection could not be made because the connected party did not respond, or the connection was closed because there was no response from the host'))

○ Solution: Check VPN connection



3. Collection of Useful Terminal Commands

⑴ Installation: Refer to MinIO Client Technical Documentation (Example: 64-bit Intel)


curl https://dl.min.io/client/mc/release/linux-amd64/mc \
  --create-dirs \
  -o $HOME/minio-binaries/mc

chmod +x $HOME/minio-binaries/mc
export PATH=$PATH:$HOME/minio-binaries/

mc --help


⑵ CLI Setup: Concept of getting permission to access MinIO server. Use the mc alias set command

⑶ File Input/Output: Use the mc cp command

① --recursive

② --limit-upload


Input: 2023.08.20 10:33

results matching ""

    No results matching ""