Home > Blockchain >  How to run multiple files in one container image and use the output as input in another container in
How to run multiple files in one container image and use the output as input in another container in

Time:03-25

I am trying to run 2 python files in one container , save the output as parameters:

- name: training
    serviceAccountName: argo-service
    outputs:
      parameters:
        - name: output-param-1
          valueFrom:
            path: ./tmp/output.txt
        - name: output-param-2
          valueFrom:
            path: ./tmp/output2.txt
    container:
      image: (image name)
      command: [python]
      args: ["/data/argo/model_build.py","/data/argo/model_build_2.py"]

Use that output as input in another container :

- name: evaluate
    serviceAccountName: argo-service
    inputs:
        parameters:
          - name: value-1
          - name: value-2
    container:
      image: (image name)
      command: ["python", "/data/argo/evaluate.py"]
      args: 
          - '{{inputs.parameters.value-1}}'
          - '{{inputs.parameters.value-2}}'

and have defined the chain as :

 - name: dag-chain
    dag:
      tasks:
      - name: src
        template: clone-repo
      - name: prep
        template: data-prep
        dependencies: [src]
      - name: train
        template: training
        dependencies: [prep]
      - name: eval
        template: evaluate
        dependencies: [train]
        args:
          parameters:
            - name: value-1
              value: '{{dag-chain.tasks.train.outputs.parameters.output-param-1}}'
            - name: value-2
              value: '{{dag-chain.tasks.train.outputs.parameters.output-param-2}}'

But with these steps I'm getting the error :

" Internal Server Error: templates.dag-chain.tasks.eval templates.evaluate inputs.parameters.value-1 was not supplied: "

Please help me identify the mistakes I'm making.

I have tried the steps mentioned above but it's not working.

CodePudding user response:

I don't have Argo accessible just now to test, but a couple of things to try:

  • args in DAG tasks should be arguments (see field reference for a DAG task here).
  • Try removing dag-chain from the parameters (see example here).
 - name: dag-chain
    dag:
      tasks:
      - name: src
        template: clone-repo
      - name: prep
        template: data-prep
        dependencies: [src]
      - name: train
        template: training
        dependencies: [prep]
      - name: eval
        template: evaluate
        dependencies: [train]
        arguments:
          parameters:
            - name: value-1
              value: '{{tasks.train.outputs.parameters.output-param-1}}'
            - name: value-2
              value: '{{tasks.train.outputs.parameters.output-param-2}}'

If that doesn't work I'll try some more steps with Argo.

  • Related