system:node fails to get secrets from apiserver via curl-CodePudding

I'm doing some POC for security research, trying to access namespace secrets directly from a worker node. I have a cluster on GKE running Kubernetes 1.20

I'm running the following command from a worker (none-master) node:

curl -v $APISERVER/api/v1/namespaces/default/pods/ \
  --cacert /etc/srv/kubernetes/pki/ca-certificates.crt \
  --cert /var/lib/kubelet/pki/kubelet-client.crt \
  --key /var/lib/kubelet/pki/kubelet-client.key

And it works fine.

However, trying to get secrets fails:

curl -v $APISERVER/api/v1/namespaces/default/secrets/ \
  --cacert /etc/srv/kubernetes/pki/ca-certificates.crt \
  --cert /var/lib/kubelet/pki/kubelet-client.crt \
  --key /var/lib/kubelet/pki/kubelet-client.key

{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "secrets is forbidden: User \"system:node:gke-XXX--YYY\" cannot list resource \"secrets\" in API group \"\" in the namespace \"pencil\": No Object name found",
"reason": "Forbidden",
"details": {
  "kind": "secrets"
},
"code": 403

Looking at the documentation, I see that kubelet running on node should be able to access secrets: https://kubernetes.io/docs/reference/access-authn-authz/node/

And from my understanding, the authorization is backed by the ClusterRole system:node. Looking at it I do see it has the role to get secrets:

% kubectl get clusterrole system:node -o json
{
    "apiVersion": "rbac.authorization.k8s.io/v1",
    "kind": "ClusterRole",
...
        {
            "apiGroups": [
                ""
            ],
            "resources": [
                "configmaps",
                "secrets"
            ],
            "verbs": [
                "get",
                "list",
                "watch"
            ]
        },
...
    ]
}

And some more relevant documentation for communication between kubelet and kube-apiserver: https://kubernetes.io/docs/concepts/architecture/control-plane-node-communication/#node-to-control-plane

CodePudding user response：

I think the certificates location you are giving is incorrect. I have tried the same on my plain kubernetes cluster with following certificates and it worked fine.

curl --cacert /etc/kubernetes/pki/ca.crt --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key $APISERVER/api/v1/namespaces/default/secrets

CodePudding user response：

This is expected.

Because the access to secrets is permitted(normally) via service account. You need to find the service account token mounted on a pod running on the node. For this you may try to dig into "magical" /proc folder on the node. If you can access the service account token mounted on the pod and that service account has the permission to access secrets then only the secrets can be accessed from the nodes.