Istio Pods Not Coming Up-CodePudding

I have Installed Istio-1.8.3 via Rancher UI long back and Istio Pods and

Ingress Gateway Pods are Up and Running and My Application is getting served by Istio.

Now, Recently We have Upgraded the K8's Cluster version from 1.21 to 1.22 and then 1.22 to 1.23.

Once We restart the kubelet, Istio Pods used to come up with No Issues.

Now, Because of few Issues We have rebooted the Node and Istio Pods got restarted, They are in Running State but Readiness Probe is getting failed.

The Error I was able to find is

failed to list CRDs: the server could not find the requested resource

Below are the full logs of Istio Pod.

stream logs failed container "discovery" in pod "istiod-5fbc9568cd-qgqkk" is waiting to start: ContainerCreating for istio-system/istiod-5fbc9568cd-qgqkk (discovery)
2022-06-27T05:35:32.772949Z    info    FLAG: --log_rotate_max_age="30"
2022-06-27T05:35:32.772952Z    info    FLAG: --log_rotate_max_backups="1000"
2022-06-27T05:35:32.772955Z    info    FLAG: --log_rotate_max_size="104857600"
2022-06-27T05:35:32.772958Z    info    FLAG: --log_stacktrace_level="default:none"
2022-06-27T05:35:32.772963Z    info    FLAG: --log_target="[stdout]"
2022-06-27T05:35:32.772971Z    info    FLAG: --mcpInitialConnWindowSize="1048576"
2022-06-27T05:35:32.772974Z    info    FLAG: --mcpInitialWindowSize="1048576"
2022-06-27T05:35:32.772977Z    info    FLAG: --mcpMaxMsgSize="4194304"
2022-06-27T05:35:32.772980Z    info    FLAG: --meshConfig="./etc/istio/config/mesh"
2022-06-27T05:35:32.772982Z    info    FLAG: --monitoringAddr=":15014"
2022-06-27T05:35:32.772985Z    info    FLAG: --namespace="istio-system"
2022-06-27T05:35:32.772988Z    info    FLAG: --networksConfig="/etc/istio/config/meshNetworks"
2022-06-27T05:35:32.772999Z    info    FLAG: --plugins="[authn,authz,health]"
2022-06-27T05:35:32.773002Z    info    FLAG: --profile="true"
2022-06-27T05:35:32.773005Z    info    FLAG: --registries="[Kubernetes]"
2022-06-27T05:35:32.773008Z    info    FLAG: --resync="1m0s"
2022-06-27T05:35:32.773011Z    info    FLAG: --secureGRPCAddr=":15012"
2022-06-27T05:35:32.773013Z    info    FLAG: --tlsCertFile=""
2022-06-27T05:35:32.773016Z    info    FLAG: --tlsKeyFile=""
2022-06-27T05:35:32.773018Z    info    FLAG: --trust-domain=""
2022-06-27T05:35:32.801976Z    info    klog    Config not found: /var/run/secrets/remote/config[]
2022-06-27T05:35:32.803516Z    info    initializing mesh configuration ./etc/istio/config/mesh
2022-06-27T05:35:32.804499Z    info    mesh configuration: {
    "proxyListenPort": 15001,
    "connectTimeout": "10s",
    "protocolDetectionTimeout": "0s",
    "ingressClass": "istio",
    "ingressService": "istio-ingressgateway",
    "ingressControllerMode": "STRICT",
    "enableTracing": true,
    "defaultConfig": {
        "configPath": "./etc/istio/proxy",
        "binaryPath": "/usr/local/bin/envoy",
        "serviceCluster": "istio-proxy",
        "drainDuration": "45s",
        "parentShutdownDuration": "60s",
        "discoveryAddress": "istiod.istio-system.svc:15012",
        "proxyAdminPort": 15000,
        "controlPlaneAuthPolicy": "MUTUAL_TLS",
        "statNameLength": 189,
        "concurrency": 2,
        "tracing": {
            "zipkin": {
                "address": "zipkin.istio-system:9411"
            }
        },
        "envoyAccessLogService": {

        },
        "envoyMetricsService": {

        },
        "proxyMetadata": {
            "DNS_AGENT": ""
        },
        "statusPort": 15020,
        "terminationDrainDuration": "5s"
    },
    "outboundTrafficPolicy": {
        "mode": "ALLOW_ANY"
    },
    "enableAutoMtls": true,
    "trustDomain": "cluster.local",
    "trustDomainAliases": [
    ],
    "defaultServiceExportTo": [
        "*"
    ],
    "defaultVirtualServiceExportTo": [
        "*"
    ],
    "defaultDestinationRuleExportTo": [
        "*"
    ],
    "rootNamespace": "istio-system",
    "localityLbSetting": {
        "enabled": true
    },
    "dnsRefreshRate": "5s",
    "certificates": [
    ],
    "thriftConfig": {

    },
    "serviceSettings": [
    ],
    "enablePrometheusMerge": true
}
2022-06-27T05:35:32.804516Z    info    version: 1.8.3-e282a1f927086cc046b967f0171840e238a9aa8c-Clean
2022-06-27T05:35:32.804699Z    info    flags: 
2022-06-27T05:35:32.804706Z    info    initializing mesh networks
2022-06-27T05:35:32.804877Z    info    mesh networks configuration: {
   "networks": {
   }
}
2022-06-27T05:35:32.804938Z    info    initializing mesh handlers
2022-06-27T05:35:32.804949Z    info    initializing controllers
2022-06-27T05:35:32.804952Z    info    No certificates specified, skipping K8S DNS certificate controller
2022-06-27T05:35:32.814002Z    error    kube    failed to list CRDs: the server could not find the requested resource
2022-06-27T05:35:33.816596Z    error    kube    failed to list CRDs: the server could not find the requested resource
2022-06-27T05:35:35.819157Z    error    kube    failed to list CRDs: the server could not find the requested resource
2022-06-27T05:35:39.821510Z    error    kube    failed to list CRDs: the server could not find the requested resource
2022-06-27T05:35:47.823675Z    error    kube    failed to list CRDs: the server could not find the requested resource
2022-06-27T05:36:03.827023Z    error    kube    failed to list CRDs: the server could not find the requested resource
2022-06-27T05:36:35.829441Z    error    kube    failed to list CRDs: the server could not find the requested resource
2022-06-27T05:37:35.831758Z    error    kube    failed to list CRDs: the server could not find the requested resource

CodePudding user response：

Upgrading Istio Pilot and Istio Ingress Gateway from 1.8.3 to 1.10.2 will work.

https://github.com/istio/istio/issues/34665

CodePudding user response：

Istio version 1.8.x is too old version for kubernetes 1.23. You can refer istio documentation for k8s and istio combinations support and upgrade istio