Cloud Composer(Airflow) to run dependent python scripts-CodePudding

Python project is running multiple py scripts. Is it possible call other python scripts not from DAG but from a single python script i.e. part-of/mentioned in DAG via PythonOperator or BashOperator in Airflow.

For example I want to call other python scripts using test.py

curl = BashOperator(
task_id='multiplescripts',
bash_command="test.py",
dag=dag)

CodePudding user response：

PythonOperator is used to execute a python callable. If you want to execute a python script than just use the the proper bash command:

python test.py

so the operator would be:

curl = BashOperator(
    task_id='multiplescripts',
    bash_command="python test.py",
    dag=dag
)

CodePudding user response：

You can call the PythonOperator. Like that

def testfunction(text):
    return

test_task = PythonOperator(
                task_id="test_function",
                python_callable=testfunction,
                op_args=[ 'we are calling the function' ],
            )

You can see the documentation