I am attempting to carry out two actions in the event of a failure scenario.

1. Write the results of a table to a single file in parquet.
2. Exit a the notebook.

Take action based on the results

if validation_result["success"]:
    print("Successful")
else:
    (
        spark.sql("Select * from notify_error")
        .coalesce(1)
        .write.format("delta")
        .mode("overwrite")
        .save(lakePath)
    )

As you can see from the above, I'm trying to create a single table using coalesce(1). To exit a notebook in Synapse I believe the code is:

mssparkutils.notebook.exit(returnValue)

But I'm not where to place the code in my code above.

Any thoughts?

CodePudding user response：

Can you just do raise with ValueError, eg

%%pyspark

notebookName = mssparkutils.runtime.context.get('notebookname')
errorString = f"error in notebook '{notebookName}'"

raiseError = True

if raiseError:
  raise ValueError(errorString)