I'm writing a DAG in airflow, and have a dictionary similar to this one:
dict1 = {'a':['1','2','3'],'b':['1','2'],'c':['1']}
Each key represents a SQL file, and each value represents a table. I need to execute the SQL files on each value in their specific key value pair. I've already written the task necessary to execute the SQL, but I cannot seem to figure out how to run the loop properly so that it runs each key's SQL on each assigned value (table).
I've attempted:
dag_id = 'dag1'
@dag(
general dag parameters here
)
def dag():
for i in dict1:
for table in i:
operator = SnowflakeOperator(
task_id=f'{dag_id}_{table}_{i}',
OTHER PARAMS BELOW
(
but I get a duplicate task ID error on run. I understand that I need to pull the actual values out of the key value pairs, but I am unsure how to do so.
Any help would be appreciated!
Edit: Needed to include the key:value pair and then just iterate through the values. Spent far too long trying to figure it out and talked myself into an even worse answer.
CodePudding user response:
dict_1 = {'a':['1','2','3'],'b':['1','2'],'c':['1']}
Loop through both keys and values, by using the items() method:
for key, value in dict_1.items():
print(key)
for i in value:
print(i)
you can loop only through keys using .keys() instead of .items() and only through values using .values()