Access to data stored in MinIO

MinIO is an object storage server that is compatible with Amazon S3 cloud storage service. It is used to store data in the form of objects. MinIO is used in the context of the DataHub to store data that is ingested into the platform. This data can be accessed by users who have the necessary permissions to access the data.

The IDOML platform provides a JupyterHub environment, preconfigured with credentials to securely access the MinIO server. Users can interact with the MinIO server in Python using libraries such as pandas, s3fs and boto3 for the access to the stored object. Here are some examples of how to access a csv file stored in MinIO with pandas library:

Simple file access to MinIO with pandasSimple access to MinIO file system with s3fsSimple file download from MinIO bucket with boto3

df = pd.read_csv(
    "s3://dataset/data/path",
)

fs = s3fs.S3FileSystem()
# List files in a folder/bucket
fs.ls('path/to/folder')

with s3.open("dataset/data/path", "rb") as f:
    df = pd.read_csv(f)

s3 = boto3.resource('s3')
bucket = s3.Bucket(bucket_name)
bucket.download_file(file_name, file_name)

As the Jupyterhub environment is preconfigured with the necessary credentials, users can access the MinIO server without having to provide the credentials. However, if users want to access the MinIO server from a different environment, they can provide the necessary credentials to access the MinIO server. Here is an example of how to access the MinIO server with full configuration for accessing external MinIO server:

Full configuration for file access to MinIO with pandasFull configuration for accessing MinIO file system with s3fsFull configuration for downloading a file from MinIO with boto3

df = pd.read_csv(
    "s3://dataset/data/path",
    storage_options={
        "key": AWS_ACCESS_KEY_ID,
        "secret": AWS_SECRET_ACCESS_KEY,
        "token": AWS_SESSION_TOKEN,
        "client_kwargs": {"endpoint_url": AWS_ENDPOINT_URL}
    }
)

fs = s3fs.S3FileSystem(
    key = AWS_ACCESS_KEY_ID,
    secret = AWS_SECRET_ACCESS_KEY,
    token = AWS_SESSION_TOKEN,
    client_kwargs = {"endpoint_url": AWS_ENDPOINT_URL}
)
# List files in a folder/bucket
fs.ls('path/to/folder')

with s3.open("dataset/data/path", "rb") as f:
    df = pd.read_csv(f)

s3 = boto3.resource('s3',
    aws_access_key_id = AWS_ACCESS_KEY_ID,
    aws_secret_access_key = AWS_SECRET_ACCESS_KEY,
    aws_session_token = AWS_SESSION_TOKEN,
    endpoint_url =AWS_ENDPOINT_URL,
)
bucket = s3.Bucket(bucket_name)
bucket.download_file(file_name, file_name)