Kubernetes Namespace stuck in Terminating-CodePudding

I want to delete namespace and all resources under it.

So i ran

% kubectl delete namespace observability
namespace "observability" deleted

but this command was stuck, when I checked the namespace status it was showing Terminating

$ kubectl get ns
NAME                STATUS        AGE
observability       Terminating   14h
odigos-system       Terminating   14h

Then I tried to delete it by steps mentioned here

This command didn't returned any result kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n observability so kill it.

Then I tried Manually delete a terminating namespace

But following command return

% curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/observability/finalize
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "the object provided is unrecognized (must be of type Namespace): couldn't get version/kind; json parse error: invalid character 'a' looking for beginning of value (61706956657273696f6e3a2076310a6b696e643a204e616d657370616365 ...)",
  "reason": "BadRequest",
  "code": 400
}%

Here is the namespace definition

kubectl get ns observability -o yaml
apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: "2022-09-20T02:25:56Z"
  deletionTimestamp: "2022-09-20T15:04:26Z"
  labels:
    kubernetes.io/metadata.name: observability
    name: observability
  name: observability
  resourceVersion: "360388862"
  uid: 8cef9b90-af83-4584-b26e-8aa89212b80c
spec:
  finalizers:
  - kubernetes
status:
  conditions:
  - lastTransitionTime: "2022-09-20T15:04:31Z"
    message: 'Discovery failed for some groups, 1 failing: unable to retrieve the
      complete list of server APIs: metrics.k8s.io/v1beta1: an error on the server
      ("Internal Server Error: \"/apis/metrics.k8s.io/v1beta1?timeout=32s\": the server
      could not find the requested resource") has prevented the request from succeeding'
    reason: DiscoveryFailed
    status: "True"
    type: NamespaceDeletionDiscoveryFailure
  - lastTransitionTime: "2022-09-20T15:04:32Z"
    message: All legacy kube types successfully parsed
    reason: ParsedGroupVersions
    status: "False"
    type: NamespaceDeletionGroupVersionParsingFailure
  - lastTransitionTime: "2022-09-20T15:05:43Z"
    message: All content successfully deleted, may be waiting on finalization
    reason: ContentDeleted
    status: "False"
    type: NamespaceDeletionContentFailure
  - lastTransitionTime: "2022-09-20T15:05:43Z"
    message: All content successfully removed
    reason: ContentRemoved
    status: "False"
    type: NamespaceContentRemaining
  - lastTransitionTime: "2022-09-20T15:04:32Z"
    message: All content-preserving finalizers finished
    reason: ContentHasNoFinalizers
    status: "False"
    type: NamespaceFinalizersRemaining
  phase: Terminating

I crated these Namespaces following this https://medium.com/@edeNFed/how-to-build-an-end-to-end-open-source-observability-solution-on-kubernetes-c8725c016dd5

I am on following version

% kubectl version --short
Client Version: v1.23.6
Server Version: v1.22.12-eks-6d3986b

How can i fix this ?

CodePudding user response：

Looks like there is something wrong with your metrics-server. Check to see if its up and try to resolve that root cause kubectl -n kube-system get pods | grep metrics-server. See https://github.com/kubernetes-sigs/metrics-server for more info

CodePudding user response：

This should answer your question.