I am trying to write to the glue catalog from my glue script. However, I would like to overwrite the data and not append. I have this piece of ode

def data_export(df, target_db, target_table):
additional_options = {
    "enableUpdateCatalog": True,
    "updateBehavior": "UPDATE_IN_DATABASE"
}
try:
    dynamic_frame_out = DynamicFrame.fromDF(df, glue_context, "mydb")
    glue_context.write_dynamic_frame.from_catalog(
        frame=dynamic_frame_out,
        database=target_db,
        table_name=target_table,
        additional_options=additional_options
    )
except Exception:
    raise ("Error")

I am not able to figure out what additional options would allow me to overwrite the data every time I run my script.

CodePudding user response：

Dynamic Frame unfortunately does not support overwrite mode. So what you are trying is not really possible, you would need to switch to native Spark instead.