Binance made its data public through an s3 endpoint. The website is 'https://data.binance.vision/?prefix=data/'. Their bucket URL is 'https://s3-ap-northeast-1.amazonaws.com/data.binance.vision'. I want to download all the files in their bucket to my own s3 bucket. I can:

1. crawl this website and download the CSV files.
2. make a URL builder that builds all the URLs and downloads the CSV files using those URLs.
3. Since their data is stored on s3. I wonder if there is a cleaner way to sync their bucket to my bucket. Is the third way really doable?

CodePudding user response：

If you want to copy it to your own s3 bucket, you can do:

aws s3 sync s3://data.binance.vision s3://your-bucket-name --no-sign-request

If you want to copy it to your own computer into your current folder (.) you can do:

aws s3 sync s3://data.binance.vision . --no-sign-request