How to paginate s3 bucket query using ruby gem for rails?-CodePudding

I have a query that fetches all the items inside an S3 bucket, and it takes a very long time to load if there are a lot of items in that bucket.

bucket.objects(prefix: 'path/to/items').collect(&:key)

Based on the docs, I see it's possible to limit the number of items in that fetch.

bucket.objects(prefix: 'path/to/items').limit(10).collect(&:key)

However, I don't see any offset parameter for this function. Any idea how I could do pagination with this so it doesn't take so long to load the first few results?

CodePudding user response：

AWS provides some examples to how to do that here. It sounds like you should be able to call .each on bucket.objects(prefix: 'path/to/items') to handle each page as it is returned.

Automatic paging:

s3 = Aws::S3::Client.new

s3.list_objects(bucket: 'aws-sdk').each do |response|
  puts response.contents.map(&:key)
end

Manual paging:

s3 = Aws::S3::Client.new

# Get the first page of data
response = s3.list_objects(bucket: 'aws-sdk')

# Get additional pages
while response.next_page? do
  response = response.next_page
  # Use the response data here...
  puts response.contents.map(&:key)
end