In trying to create the DockerOperator, I got this error:
Invalid arguments were passed to DockerOperator (task_id: t2). Invalid arguments were:
**kwargs: {'retrieve_output': True, 'retrieve_output_path': '/tmp/script.out'}
Here is my code:
from airflow.decorators import task, dag
from airflow.providers.docker.operators.docker import DockerOperator
from datetime import datetime
@dag(start_date=datetime(2023, 1, 1), schedule="@daily", catchup=False)
def docker_dag():
@task()
def t1():
pass
t2 = DockerOperator(
task_id='t2',
container_name="task_t2",
image='stock_image:v1.0.2',
command='python3 stock_data.py',
docker_url="tcp://docker-proxy:2375", # I have to use this on MacOS or I'll get a Permission Denied error
network_mode='bridge',
xcom_all=True,
retrieve_output=True,
retrieve_output_path="/tmp/script.out",
auto_remove=True,
mount_tmp_dir=False
)
t1() >> t2
dag = docker_dag()
Note: Here is the link to the documentation which shows my arguments do exist in the documentation. So why would I be getting an InvalidArgument error for just these 2 specific arguments?
CodePudding user response:
retrieve_output
and retrieve_output_path
were added to the DockerOperator in the Docker provider version 2.2.0. :)