I am using spark with scala in which I am getting streaming datas from eventhubs and then storing them in delta table. In order to apply drools rule on them ,i need to pass them through variables...i am stuck where i have to get the data from delta table to variable.

CodePudding user response：

It really depends what data you need to pass to that drools rules, and what you need to return. You can either use:

• User defined function - you define a function that will receive one or more parameters (column values of specific rows). (more examples)
• Use map function of Dataset / Dataframe class to process the whole Row (doc, and examples)

CodePudding user response：

Delta Tables can be read into DataFrames. A variable can be assigned to point to the DataFrame.

df = spark.read.format("delta").load("some/delta/path")

Once the Delta Table is read, you can apply your custom transformations:

transformed_df = df.transform(first_transform).transform(second_transform)

Hope this helps point you in the right direction.