How to automate uploading AKS logs to Azure storage account-CodePudding

I have a task to automate the uploading of AKS logs (control plane and workload) to the Azure storage account so that they can be viewed later or may be an alert notification to the email/teams channel in case of any failure. It would have been an easy task if the log analytics workspace would have been used however, to save the cost we have kept it disabled.

I have tried using the below cronjob which would upload the pod logs to storage account on a regular basis, but it is throwing me the below errors[1]

apiVersion: batch/v1
kind: CronJob
metadata:
  name: log-uploader
spec:
  schedule: "0 0 * * *" # Run every day at midnight
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: log-uploader
              image: mcr.microsoft.com/azure-cli:latest
              command:
                - bash
                - "-c"
                - |
                  az aks install-cli
                  # Set environment variables for Azure Storage Account and Container
                  export AZURE_STORAGE_ACCOUNT=test-101
                  export AZURE_STORAGE_CONTAINER=logs-101
                  # Iterate over all pods in the cluster and upload their logs to Azure Blob Storage
                  for pod in $(kubectl get pods --all-namespaces -o jsonpath='{range .items[*]}{.metadata.name} {.metadata.namespace}{"\n"}{end}'); do
                    namespace=$(echo $pod | awk '{print $2}')
                    pod_name=$(echo $pod | awk '{print $1}')
                    # Use the Kubernetes logs API to retrieve the logs for the pod
                    logs=$(kubectl logs -n $namespace $pod_name)
                    # Use the Azure CLI to upload the logs to Azure Blob Storage
                    echo $logs | az storage blob upload --file - --account-name $AZURE_STORAGE_ACCOUNT --container-name $AZURE_STORAGE_CONTAINER --name "$namespace/$pod_name_`date`.log"
                  done
          restartPolicy: OnFailure

Errors[1]

error: expected 'logs [-f] [-p] (POD | TYPE/NAME) [-c CONTAINER]'.
POD or TYPE/NAME is a required argument for the logs command
See 'kubectl logs -h' for help and examples

The same commands are running fine outside the container.

Any thoughts/suggestions would be highly appreciated.

Regards,

Piyush

CodePudding user response：

A better approach for achieving this would be deploying a fluentd daemonset in your cluster and use the azure storage plugin to upload logs to a storage account.

This tool was built for this specific purpose and will probably serve you better for this purpose.