I am using Airflow2.0 with KubernetesPodOperator
want to run a command that use as a parameter a file from inside the image ran by the Operator. This is what I used:
KubernetesPodOperator(
namespace=commons.kubernetes_namespace,
labels=commons.labels,
image=f"myregistry.io/myimage:{config['IMAGE_TAG']}",
arguments=[
"python",
"run_module.py ",
"-i",
f'args/{config["INPUT_DIR"]}/{task_id}.json'
],
name=dag_name task_id,
task_id=task_id,
secrets=[secret_volume]
)
But this gives me the error:
raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: args/airflow2test/processing-pipeline.json
The image does not use any macros.
Anyone has any clue? What do I do wrong?
CodePudding user response:
This was a bug that started with PR released in version 2.0.0
of apache-airflow-providers-cncf-kubernetes
.
The goal of the change was to allow templating of .json
files. There was a GitHub issue about the problems it created. The bug was eventually resolved by PR which was released in version 2.0.2 of the provider.
Solution:
- Upgrade to the latest
apache-airflow-providers-cncf-kubernetes
(currently 2.0.2) - If upgrade is not an option use custom
KubernetesPodOperator
There are two ways to workaround that problem one is to change template_fields
the other is to change template_ext
:
1st option: As posted on issue by raphaelauv is not to allow rendering of arguments
field:
class MyKubernetesPodOperator(KubernetesPodOperator):
template_fields = tuple(x for x in KubernetesPodOperator.template_fields if x != "arguments")
2st option: if you prefer not to render .json
files:
class MyKubernetesPodOperator(KubernetesPodOperator):
template_ext = ('.yaml', '.yml',)