I was taking a look at Hub—the dataset format for AI—and noticed that hub integrates with GCP and AWS. I was wondering if it also supported integrations with MinIO.

I know that Hub allows you to directly stream datasets from cloud storage to ML workflows but I’m not sure which ML workflows it integrates with.

I would like to use MinIO over S3 since my team has a self-hosted MinIO instance (aka it's free).

CodePudding user response：

Hub allows you to load data from anywhere. Hub works locally, on Google Cloud, MinIO, AWS as well as Activeloop storage (no servers needed!). So, it allows you to load data and directly stream datasets from cloud storage to ML workflows.

You can find more information about storage authentication in the Hub docs.

Then, Hub allows you to stream data to PyTorch or TensorFlow with simple dataset integrations as if the data were local since you can connect Hub datasets to ML frameworks.