Why won't ADF list the contents of my S3 bucket?-CodePudding

I'm trying to ingest data from my S3 bucket into Azure via ADF. I added a new connection by entering my Access key ID and Secret access key, and 'Test Connection' against my Service URL (

However, when I try to browse the contents of my bucket, nothing shows up:

From within a VM with the AWS CLI installed, I can use these exact same credentials and list the contents of the bucket successfully:

export AWS_ACCESS_KEY_ID=myAccessKeyId
export AWS_SECRET_ACCESS_KEY=mySecretAccessKey
aws s3 ls s3://myS3BucketName/

For the policy associated with this access key, I even granted all S3 permissions (even though I already had GetObject, ListBucket, GetObjectVersion, ListAllMyBuckets, and more).

How can I best debug this issue? My goal is to do a daily ingestion of files or directories that, in most cases, I will know the exact name of. So if there's a good way to attempt copying of files by exact name without necessarily doing a bucket directory listing, that may be acceptable.

CodePudding user response：

Your troubleshooting approach sounds reasonable, if you can list the content of the bucket using the same credentials through aws cli, you should also be able to list the content of the bucket from ADF. To list the content of the bucket from the ADF, you can do that by using a dataset associated with your S3 linked service, in the dataset you can inform the path and browse your bucket.

I see you mentioned Service URL, the instructions above normally work without issues for linked services that don't have a service URL specified. Have you tried to test your linked service without it?