Can't attach statically provisioned EBS volume to EKS node-CodePudding

I've configured my EKS setup to support EBS by following the docs

I've verified that the PV and PVC for the new statically provisioned volume are ok (firehose-mainnet-test-volume points to the vol-0a493db74622155d0 from the screenshot above)

 ❯❯❯ k get pv
k get pvc
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                               STORAGECLASS   REASON   AGE
firehose-mainnet-reader-test-volume        1000Gi     RWO            Retain           Bound    default/firehose-mainnet-reader-test-volume-claim   io2                     103m
mercury-ipfs-ipfs-efs-pv                   20Gi       RWX            Retain           Bound    default/mercury-ipfs-ipfs-efs-pvc                   efs-sc                  362d
pvc-e823a3b9-94be-449e-8248-ef1a023d8cc9   2000Gi     RWO            Delete           Bound    default/firehose-mainnet-reader-ebs-pvc             gp2-ebs-sc              40d
 9:27AM /Users/paymahn/code/goldsky/firehose/go-ethereum tags/geth-v1.10.25-fh2 ✱ ◼
 ❯❯❯ k get pvc
NAME                                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
firehose-mainnet-reader-ebs-pvc             Bound    pvc-e823a3b9-94be-449e-8248-ef1a023d8cc9   2000Gi     RWO            gp2-ebs-sc     40d
firehose-mainnet-reader-test-volume-claim   Bound    firehose-mainnet-reader-test-volume        1000Gi     RWO            io2            81m
mercury-ipfs-ipfs-efs-pvc                   Bound    mercury-ipfs-ipfs-efs-pv                   20Gi       RWX            efs-sc         362d

This volume fails to mount when describing my pod:

  Normal   SuccessfulAttachVolume  2m44s                  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-e823a3b9-94be-449e-8248-ef1a023d8cc9"
  Warning  FailedMount             45s                    kubelet                  Unable to attach or mount volumes: unmounted volumes=[firehose-mainnet-reader-test-volume-claim], unattached volumes=[kube-api-access-smcss jwt firehose-mainnet-reader-ebs-pvc firehose-mainnet-reader-test-volume-claim]: timed out waiting for the condition
  Warning  FailedAttachVolume      20s                    attachdetach-controller  AttachVolume.Attach failed for volume "firehose-mainnet-reader-test-volume" : Attach timeout for volume vol-0a493db74622155d0

When I look at the logs for the ebs-csi-controller I see the following output:

ebs-csi-controller-7485b8586d-jwng9 csi-attacher I1212 12:43:38.294065       1 csi_handler.go:248] Attaching "csi-f61a123c75ba67f92a9482f96ca3fa133d9f4f6af9bc924c545f08a05da4a85e"
ebs-csi-controller-7485b8586d-jwng9 ebs-plugin E1212 12:43:38.651776       1 driver.go:120] GRPC error: rpc error: code = Internal desc = Could not attach volume "vol-0a493db74622155d0" to node "i-0f994790a01e365d2": could not attach volume "vol-0a493db74622155d0" to node "i-0f994790a01e365d2": UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: 86wH2-qNGBPhlKSt0fyV9CfjiZCvdgoQmTX7EmWlB9Dwmj8VjAWo7VaLEobbh4lNuLM0bmCnWOPgntBaRP2jwqap6koIsjikjt4Takcv49FZZ0SmO4GLhqpzBx88k1YAUE3n0DJp_ip6gW-oVvb2FD16tOfU5iJn6dyCUUzGlguhPn20WbzZuYitvpfXlK4m2RSiQDZPDHOesmsDWREN5d-p086kAQAUdLauoNaTd-qDWf497Yh5pwR4WXYM3hWi8SxSSs7y6sQ8idTjj8GHUjJOrsZC_hRCMO-NhOMy-agBwT4gYziOzaZ-AVxlhv7YW27Yd-azNrumAuA4JykN8YtE76h1RFJ16SuIFBrhx4mEsMR-pjuLLnuTbk6hdkXqeoPE9xoIx6pL12HwiblTDOQo8nM2utNK_p0ZYefb1IlJuIjm8CrgbmxyXu5wZWcsZKOnufdTh6G2Hj46kT0OHDI7-NJbyu4d8NkQ5LvQvX_wHVc2JAonNsRj5VuQZZr6G5C1FJrNiu44-LnhWB92LMR9ho7maHegKKrQ-DLy9UHuVS9fq5xNdyEgYAaNFGn8MkJxsSplFUQm25HByt39hCbeuwoYkUHtpite9ufqoMvTElYJBDFkasVec2RxhnYcPuyQ7pPj26IcUquW5wKD
ebs-csi-controller-7485b8586d-jwng9 csi-attacher I1212 12:43:38.660565       1 csi_handler.go:255] Failed to save attach error to "csi-f61a123c75ba67f92a9482f96ca3fa133d9f4f6af9bc924c545f08a05da4a85e": VolumeAttachment.storage.k8s.io "csi-f61a123c75ba67f92a9482f96ca3fa133d9f4f6af9bc924c545f08a05da4a85e" is invalid: status.attachError.message: Too long: must have at most 262144 bytes
ebs-csi-controller-7485b8586d-jwng9 ebs-plugin  status code: 403, request id: c1eeb49c-955f-4969-9fcd-1a83933de338
ebs-csi-controller-7485b8586d-jwng9 csi-attacher I1212 12:43:38.660604       1 csi_handler.go:231] Error processing "csi-f61a123c75ba67f92a9482f96ca3fa133d9f4f6af9bc924c545f08a05da4a85e": failed to attach: rpc error: code = Internal desc = Could not attach volume "vol-0a493db74622155d0" to node "i-0f994790a01e365d2": could not attach volume "vol-0a493db74622155d0" to node "i-0f994790a01e365d2": UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message: 86wH2-qNGBPhlKSt0fyV9CfjiZCvdgoQmTX7EmWlB9Dwmj8VjAWo7VaLEobbh4lNuLM0bmCnWOPgntBaRP2jwqap6koIsjikjt4Takcv49FZZ0SmO4GLhqpzBx88k1YAUE3n0DJp_ip6gW-oVvb2FD16tOfU5iJn6dyCUUzGlguhPn20WbzZuYitvpfXlK4m2RSiQDZPDHOesmsDWREN5d-p086kAQAUdLauoNaTd-qDWf497Yh5pwR4WXYM3hWi8SxSSs7y6sQ8idTjj8GHUjJOrsZC_hRCMO-NhOMy-agBwT4gYziOzaZ-AVxlhv7YW27Yd-azNrumAuA4JykN8YtE76h1RFJ16SuIFBrhx4mEsMR-pjuLLnuTbk6hdkXqeoPE9xoIx6pL12HwiblTDOQo8nM2utNK_p0ZYefb1IlJuIjm8CrgbmxyXu5wZWcsZKOnufdTh6G2Hj46kT0OHDI7-NJbyu4d8NkQ5LvQvX_wHVc2JAonNsRj5VuQZZr6G5C1FJrNiu44-LnhWB92LMR9ho7maHegKKrQ-DLy9UHuVS9fq5xNdyEgYAaNFGn8MkJxsSplFUQm25HByt39hCbeuwoYkUHtpite9ufqoMvTElYJBDFkasVec2RxhnYcPuyQ7pPj26IcUquW5wKD
ebs-csi-controller-7485b8586d-jwng9 csi-attacher        status code: 403, request id: c1eeb49c-955f-4969-9fcd-1a83933de338

How can I attach a statically generated ebs volume to my node? I've verified that the node and volume are both in the same region, us-west-2c. Is there anything else I need to check to ensure that the volume can be mounted to the node?

CodePudding user response：

Turns out the service account for my ebs csi controller didn't have an assumeRole annotation. Adding that fixed the issue.

kubectl annotate serviceaccount ebs-csi-controller-sa \
    -n kube-system \
    eks.amazonaws.com/role-arn=arn:aws:iam::111122223333:role/AmazonEKS_EBS_CSI_DriverRole