Dynamically passing parameters to an airflow operator through a dictionary-CodePudding

For example with the BashOperator:

bash_task = BashOperator(
     task_id="bash_task",
    bash_command="echo \"here is the message: '$message'\"",
    env={"message": '{{ dag_run.conf["message"] if dag_run else "" }}'},
)

I would not want to "hardocode" the arguments of the operator, i.e. 'task_id', 'bash_command' etc.

But instead pass a dictionary, for example:

arguments = {
    "task_id": "Bash_task",
    "bash_command": "echo \"here is the message: '$message'\"",
}

bash_task = BashOperator(
    arguments=arguments
)

In my case I'm working with more complicated operators and need to dynamically set arguments based on some conditions. My understanding was that you could do this with the 'params' argument but it did not work how I thought. I'm able to dynamically pass argument values through for example a yaml file but not dynamically select which arguments to set, and what their values would be through for example a dictionary.

Is it possible or what is the best approach for this?

CodePudding user response：

This is standard Python feature (** operator).

Use **arguments to expand dictionary to keyword arguments Converting Python dict to kwargs?