Allow mutating webhooks to work with tls-enabled istio-CodePudding

I have the following MutatingWebhookConfiguration

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: example-webhook
webhooks:
  - name: example-webhook.default.svc.cluster.local
    admissionReviewVersions:
      - "v1beta1"
    sideEffects: "None"
    timeoutSeconds: 30
    objectSelector:
      matchLabels:
        example-webhook-enabled: "true"
    clientConfig:
      service:
        name: example-webhook
        namespace: default
        path: "/mutate"
      caBundle: "LS0tLS1CR..."
    rules:
      - operations: [ "CREATE" ]
        apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]

I want to inject the webhook pod in an istio enabled namespace with istio having strict TLS mode on.

Therefore, (I thought) TLS should not be needed in my example-webhook service so it is crafted as follows:

apiVersion: v1
kind: Service
metadata:
  name: example-webhook
  namespace: default
spec:
  selector:
    app: example-webhook
  ports:
    - port: 80
      targetPort: webhook
      name: webhook

However when creating a Pod (that does indeed trigger the webhook) I get the following error:

▶ k create -f demo-pod.yaml
Error from server (InternalError): error when creating "demo-pod.yaml": Internal error occurred: failed calling webhook "example-webhook.default.svc.cluster.local": Post "https://example-webhook.default.svc:443/mutate?timeout=30s": no service port 443 found for service "example-webhook"

Can't I configure the webhook not to be called on 443 but rather on 80? Either way TLS termination is done by the istio sidecar.

Is there a way around this using VirtualService / DestinationRule?

edit: on top of that, why is it trying to reach the service in the example-webhook.default.svc endpoint? (while it should be doing so in example-webhook.default.svc.cluster.local) ?

Update 1

I have tried to use https as follows:

I have created a certificate and private key, using istio's CA.

I can verify that my DNS names in the cert are valid as follows (from another pod)

echo | openssl s_client -showcerts -servername example-webhook.default.svc -connect example-webhook.default.svc:443 2>/dev/null | openssl x509 -inform pem -noout -text

...
 Subject: C = GR, ST = Attica, L = Athens, O = Engineering, OU = FOO, CN = *.cluster.local, emailAddress = [email protected]
...
        X509v3 extensions:
            X509v3 Subject Alternative Name:
                DNS:*.default.svc.cluster.local, DNS:example-webhook, DNS:example-webhook.default.svc
...

but now pod creation fails as follows:

▶ k create -f demo-pod.yaml
Error from server (InternalError): error when creating "demo-pod.yaml": Internal error occurred: failed calling webhook "example-webhook.default.svc.cluster.local": Post "https://example-webhook.default.svc:443/mutate?timeout=30s": x509: certificate is not valid for any names, but wanted to match example-webhook.default.svc

Update 2

The fact that the certs the webhook pod are running with were appropriately created using the istio CA cert, is also validated.

curl --cacert istio_cert https://example-webhook.default.svc
Test

where istio_cert is the file containing istio's CA certificate

What is going on?

CodePudding user response：

Not sure if you can use webhook on port 80...

Perhaps some of this will be useful to you, I used the following script to generate certificates, you can change it to suit your needs:

#!/bin/bash

set -e

service=webhook-svc
namespace=default
secret=webhook-certs
csrName=${service}.${namespace}

cat <<EOF >> csr.conf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = ${service}
DNS.2 = ${service}.${namespace}
DNS.3 = ${service}.${namespace}.svc
EOF

openssl genrsa -out server-key.pem 2048
openssl req -new -key server-key.pem -subj "/CN=${service}.${namespace}.svc" -out server.csr -config csr.conf

kubectl delete csr ${csrName} 2>/dev/null || true

cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
  name: ${csrName}
spec:
  groups:
  - system:authenticated
  request: $(< server.csr base64 | tr -d '\n')
  usages:
  - digital signature
  - key encipherment
  - server auth
EOF

sleep 5

kubectl certificate approve ${csrName}

for i in {1 .. 10}
do
    serverCert=$(kubectl get csr ${csrName} -o jsonpath='{.status.certificate}')
    if [[ ${serverCert} != '' ]]; then
        break
    fi
    sleep 1
done
if [[ ${serverCert} == '' ]]; then
    echo "ERROR: After approving csr ${csrName}, the signed certificate did not appear on the resource. Giving up after 10 attempts." >&2
    exit 1
fi


echo "${serverCert}" | openssl base64 -d -A -out server-cert.pem


# create the secret with CA cert and server cert/key
kubectl create secret generic ${secret} \
        --from-file=key.pem=server-key.pem \
        --from-file=cert.pem=server-cert.pem \
        --dry-run -o yaml |
    kubectl -n ${namespace} apply -f -

The script creates a secret, which I then mounted into the webhook, deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webhook-deployment
  namespace: default
  labels:
    app: webhook
  annotations:
    sidecar.istio.io/inject: "false"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: webhook
  template:
    metadata:
      labels:
        app: webhook
      annotations:
        sidecar.istio.io/inject: "false"
    spec:
      containers:
        - name: webhook
          image: webhook:v1
          imagePullPolicy: IfNotPresent
          volumeMounts:
          - name: webhook-certs
            mountPath: /certs
            readOnly: true
      volumes:
      - name: webhook-certs
        secret:
          secretName: webhook-certs

service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: webhook-svc
  namespace: default
  labels:
    app: webhook
spec:
  ports:
  - port: 443
    targetPort: 8443
  selector:
    app: webhook

CodePudding user response：

Did you try adding the port attribute in your MutatingWebhookConfiguration

clientConfig:
      service:
        name: example-webhook
        namespace: default
        path: "/mutate"
        port: 80