I need to use Huawei OBS service. It is alike S3. SDK is very similar to AWS S3. So if s3 has solution OBS should have too. So if you have a solution on S3. It can work too.

I have a bucket that has 20k objects in it. I need to download that data with using python sdk ONLY.

I tried to list the objects and download 1 by 1 but only first 1k data is gathered.

List Objects --> https://support.huaweicloud.com/intl/en-us/sdk-python-devg-obs/obs_22_0805.html

With a for each Download Objects --> https://support.huaweicloud.com/intl/en-us/sdk-python-devg-obs/obs_22_0909.html

CodePudding user response：

The documentation page you linked says "You can use this API to list objects in a bucket. By default, a maximum of 1000 objects are listed." That's why you are only receiving 1000 objects.

AWS has a list_objects_v2() command that can use a ContinuationToken to obtain more objects in a listing, but that might not be available in OBS.

Instead, you probably need to call the function again and pass the last Key of the previous result set in the marker field because the documentation says: "Specifies a marker when listing objects in a bucket. With a marker configured, objects after this marker will be returned in alphabetical order."