Home > Enterprise >  Add the header x-ms-app To Kusto Spark Connector
Add the header x-ms-app To Kusto Spark Connector

Time:06-25

I'm tasked to identify running queries in our ADX cluster and according to MS docs we should use the x-ms-app header to identify it.

I basically need to fill this info when checking the queries in our cluster

CodePudding user response:

At this point there is no support for populating the Application field in .show queries (might change in the future).
An alternative approach would be to send client request properties.

Here is a demo (minus the credentials):

Spark

Install com.microsoft.azure.kusto:kusto-spark_3.0_2.12:3.0.0 from Maven

kustoCluster = "https://***.westeurope.kusto.windows.net/"
kustoDatabase = "mydb"
kustoQuery = "AnotherStormEvents | take 10"
kustoAadAppId = "***"
kustoAadAppSecret = "***"
kustoAadAuthorityID = "***"

crp = sc._jvm.com.microsoft.azure.kusto.data.ClientRequestProperties()
crp.setOption("query_take_max_records", 3)        # this actually impacts the returned result set
crp.setOption("MyApplicationName"     ,"ThisIsMyApp")
crp.setOption("MyProperty"            ,"SomeValue")
crp.setOption("MyOtherProperty"       ,123.45)
 
df =   (spark.read
        .format("com.microsoft.kusto.spark.datasource")
        .option("kustoCluster", kustoCluster)
        .option("kustoDatabase", kustoDatabase)
        .option("kustoQuery", kustoQuery)
        .option("kustoAadAppId", kustoAadAppId)
        .option("kustoAadAppSecret", kustoAadAppSecret)
        .option("kustoAadAuthorityID", kustoAadAuthorityID)
        .option("clientRequestPropertiesJson", crp.toString())
        .load()
        )

df.display()

ADX

.show queries 
| top 1 by StartedOn
| project kv = ClientRequestProperties.Options
| mv-expand kind=array kv
| project k = kv[0], v = kv[1]
k v
api_version v2
request_readonly true
servertimeout 2400000000
servertimeoutorigin Gateway
query_datascope 1
query_fanout_nodes_percent 100
query_fanout_threads_percent 100
maxmemoryconsumptionperiterator 5368709120
max_memory_consumption_per_query_per_node 8589699072
truncationmaxsize 67108864
truncationmaxrecords 500000
MyProperty SomeValue
query_take_max_records 3
MyOtherProperty 123.45
MyApplicationName ThisIsMyApp
  • Related