How to copy s3 objects into EKS pods directly-CodePudding

I am working on Kubernetes pods and I have some pods running on an application. Now, I want to copy some files inside my pods but the situation here is my files present in my s3 bucket. So, I want to know an automated way that can directly copy my s3 files into my pod's folder directory. I don't know how to do this.

If anyone knows it then please reply.

Thanks

CodePudding user response：

There are multiple ways to achieve this, few of them are:

Using Kubectl (very basic not automated method): first need to download s3 files on local and copy them into application pods can use kubectl command line to perform copy operation from host fs to pod fs.

kubectl cp <local-file> <pod-name>:<file-path>

Using Init-Container (Automated): Write a small script to download files from s3/cloud provider and place into shared volume between init-container and main container. In that way whenever your pod is spin up, the init-container will prepare volume before running actual container.

Sample Pod yaml:

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    app: myapp
spec:
  containers:
 - name: myapp-container
    image: busybox:1.28
    command: ['sh', '-c', 'echo The app is running! && sleep 3600']
    volumeMounts:
    - mountPath: /tmp
      name: data
  initContainers:
 - name: init-downloader
    image: busybox:1.28
    command: ['sh', '-c', "aws s3 cp s3://xyz/ /tmp"]
    volumeMounts:
    - mountPath: /tmp
      name: data
  volumes:
 - emptyDir: {}
    name: data

Using CrobJob (Periodically Update): If the data needs to be updated in volume on periodic basis and not depends on pod restarts, then additional utility can be utilised to perform this action.

Either a script to download on local and perform kubectl copy operation on all pods
Use other opensource project to fullfill that, for example
- https://github.com/maorfr/skbn, Skbn is a tool for copying files and directories between Kubernetes and cloud storage providers

Sample Job Yaml

apiVersion: batch/v1
kind: Job
metadata:
  labels:
    app: skbn
  name: skbn
spec:
  template:
    metadata:
      labels:
        app: skbn
    annotations:
      iam.amazonaws.com/role: skbn 
    spec:
      serviceAccountName: skbn
      containers:
      - name: skbn
        image: maorfr/skbn
        command: ["skbn"]
        args:
        - cp
        - --src
        - k8s://namespace/pod/container/path/to/copy/from
        - --dst
        - s3://bucket/path/to/copy/to
        env:
        - name: AWS_REGION
          value: us-east-1

There can be multiple ways to implement this requirement but mostly it will be using above mentioned methods only.

Thanks

CodePudding user response：

You can use the init container & cronjobs with AWS CLI to copy the files to S3 bucket.

To add further you can also use the Volume mount with datshim it's nice option :

apiVersion: com.ie.ibm.hpsys/v1alpha1
kind: Dataset
metadata:
  name: example-dataset
spec:
  local:
    type: "COS"
    accessKeyID: "{AWS_ACCESS_KEY_ID}"
    secretAccessKey: "{AWS_SECRET_ACCESS_KEY}"
    endpoint: "{S3_SERVICE_URL}"
    bucket: "{BUCKET_NAME}"
    readonly: "true" #OPTIONAL, default is false  
    region: "" #OPTIONAL

https://github.com/datashim-io/datashim

If you are looking for something with FUSE you should also checkout the https://github.com/s3fs-fuse/s3fs-fuse