Using ipywidgets to load a Databricks Unity Catalog table throws "Missing Credential Scope&quot-CodePudding

We are trying to create a data manipulation Notebook together with ipywidgets to create a simple UI for predefined function calls. During these functions, we rely on loading data from the Metastore. Here is an example of what we are doing.

def loaddf():
    df = spark.read.table('sandbox.mysamples.sampledata')
    return df

def clickButton(b):
    output.clear_output()
    with output:
        df = loaddf()
        return 
...
search_btnOne = Button(
    ....
)
search_btnOne.on_click(clickButton)

A function loaddf loads a dataframe. A function clickButton calls that function to use the dataframe downstream. An ipywidget with a button is defined, which calls the clickButton function.

Now, calling the function loaddf() directly, i.e. simply as a code cell, works fine. The data is loaded and can be displayed using .display(). But clicking the button gives the error message below.

Does anyone know what might cause this problem?

File <command-2062771599631884>:8, in clickButton(b)
      6 output.clear_output()
      7 with output:
----> 8     df = loaddf()
      9     return

File <command-2062771599631884>:2, in loaddf()
      1 def loaddf():
----> 2     df = spark.read.table('sandbox.mysamples.sampledata')
      3     return df

File /databricks/spark/python/pyspark/instrumentation_utils.py:48, in _wrap_function.<locals>.wrapper(*args, **kwargs)
     46 start = time.perf_counter()
     47 try:
---> 48     res = func(*args, **kwargs)
     49     logger.log_success(
     50         module_name, class_name, function_name, time.perf_counter() - start, signature
     51     )
     52     return res

File /databricks/spark/python/pyspark/sql/readwriter.py:320, in DataFrameReader.table(self, tableName)
    303 def table(self, tableName: str) -> "DataFrame":
    304     """Returns the specified table as a :class:`DataFrame`.
    305 
    306     .. versionadded:: 1.4.0
   (...)
    318     [('name', 'string'), ('year', 'int'), ('month', 'int'), ('day', 'int')]
    319     """
--> 320     return self._df(self._jreader.table(tableName))

File /databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py:1321, in JavaMember.__call__(self, *args)
   1315 command = proto.CALL_COMMAND_NAME  \
   1316     self.command_header  \
   1317     args_command  \
   1318     proto.END_COMMAND_PART
   1320 answer = self.gateway_client.send_command(command)
-> 1321 return_value = get_return_value(
   1322     answer, self.gateway_client, self.target_id, self.name)
   1324 for temp_arg in temp_args:
   1325     temp_arg._detach()

File /databricks/spark/python/pyspark/sql/utils.py:196, in capture_sql_exception.<locals>.deco(*a, **kw)
    194 def deco(*a: Any, **kw: Any) -> Any:
    195     try:
--> 196         return f(*a, **kw)
    197     except Py4JJavaError as e:
    198         converted = convert_exception(e.java_exception)

File /databricks/spark/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py:326, in get_return_value(answer, gateway_client, target_id, name)
    324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
    325 if answer[1] == REFERENCE_TYPE:
--> 326     raise Py4JJavaError(
    327         "An error occurred while calling {0}{1}{2}.\n".
    328         format(target_id, ".", name), value)
    329 else:
    330     raise Py4JError(
    331         "An error occurred while calling {0}{1}{2}. Trace:\n{3}\n".
    332         format(target_id, ".", name, value))

Py4JJavaError: An error occurred while calling o10344.table.
: org.apache.spark.SparkException: Missing Credential Scope. 
    at com.databricks.unity.UCSDriver$Manager.$anonfun$scope$1(UCSDriver.scala:103)
    at scala.Option.getOrElse(Option.scala:189)
    at com.databricks.unity.UCSDriver$Manager.scope(UCSDriver.scala:103)
    at com.databricks.unity.UCSDriver$Manager.currentScope(UCSDriver.scala:97)
    at com.databricks.unity.UnityCredentialScope$.currentScope(UnityCredentialScope.scala:100)
    at com.databricks.unity.UnityCredentialScope$.getCredentialManager(UnityCredentialScope.scala:128)
    at com.databricks.unity.CredentialManager$.getUnityApiTokenOpt(CredentialManager.scala:456)
    at com.databricks.unity.UnityCatalogClientHelper$.getToken(UnityCatalogClientHelper.scala:35)
    at com.databricks.managedcatalog.ManagedCatalogClientImpl.$anonfun$bulkGetMetadata$1(ManagedCatalogClientImpl.scala:2889)
    at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80)
    at com.databricks.managedcatalog.ManagedCatalogClientImpl.$anonfun$recordAndWrapException$1(ManagedCatalogClientImpl.scala:2953)
    at com.databricks.managedcatalog.ErrorDetailsHandler.wrapServiceException(ErrorDetailsHandler.scala:25)
    at com.databricks.managedcatalog.ErrorDetailsHandler.wrapServiceException$(ErrorDetailsHandler.scala:23)
    at com.databricks.managedcatalog.ManagedCatalogClientImpl.wrapServiceException(ManagedCatalogClientImpl.scala:79)
    at com.databricks.managedcatalog.ManagedCatalogClientImpl.recordAndWrapException(ManagedCatalogClientImpl.scala:2952)
    at com.databricks.managedcatalog.ManagedCatalogClientImpl.bulkGetMetadata(ManagedCatalogClientImpl.scala:2882)
    at com.databricks.sql.managedcatalog.NonPermissionEnforcingManagedCatalog.updateCache(NonPermissionEnforcingManagedCatalog.scala:49)
    at com.databricks.sql.managedcatalog.PermissionEnforcingManagedCatalog.getTablesByName(PermissionEnforcingManagedCatalog.scala:244)
    at com.databricks.sql.managedcatalog.ManagedCatalogSessionCatalog.fastGetTablesByName(ManagedCatalogSessionCatalog.scala:1061)
    at com.databricks.sql.transaction.tahoe.catalog.DeltaCatalog.fetchFromCatalog(DeltaCatalog.scala:417)
    at com.databricks.sql.transaction.tahoe.catalog.DeltaCatalog.$anonfun$loadTables$1(DeltaCatalog.scala:362)
    at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80)
    at com.databricks.sql.transaction.tahoe.metering.DeltaLogging.recordFrameProfile(DeltaLogging.scala:248)
    at com.databricks.sql.transaction.tahoe.metering.DeltaLogging.recordFrameProfile$(DeltaLogging.scala:246)
    at com.databricks.sql.transaction.tahoe.catalog.DeltaCatalog.recordFrameProfile(DeltaCatalog.scala:80)
    at com.databricks.sql.transaction.tahoe.catalog.DeltaCatalog.loadTables(DeltaCatalog.scala:359)
    at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anon$3.$anonfun$submit$1(Analyzer.scala:1819)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80)
    at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveRelations$$record(Analyzer.scala:1878)
    at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anon$3.submit(Analyzer.scala:1801)
    at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:1430)
    at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:1370)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$4(RuleExecutor.scala:218)
    at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$3(RuleExecutor.scala:218)
    at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
    at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
    at scala.collection.immutable.List.foldLeft(List.scala:91)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:215)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeBatch$1(RuleExecutor.scala:207)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$8(RuleExecutor.scala:277)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$8$adapted(RuleExecutor.scala:277)
    at scala.collection.immutable.List.foreach(List.scala:431)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:277)
    at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:194)
    at org.apache.spark.sql.catalyst.analysis.Analyzer.executeSameContext(Analyzer.scala:353)
    at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$execute$1(Analyzer.scala:346)
    at org.apache.spark.sql.catalyst.analysis.AnalysisContext$.withNewAnalysisContext(Analyzer.scala:253)
    at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:346)
    at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:274)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:186)
    at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:153)
    at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:186)
    at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:326)
    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:331)
    at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:325)
    at org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:163)
    at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80)
    at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:319)
    at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$3(QueryExecution.scala:353)
    at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:789)
    at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:353)
    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:1003)
    at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:350)
    at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:144)
    at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:144)
    at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:136)
    at org.apache.spark.sql.Dataset$.$anonfun$ofRows$1(Dataset.scala:98)
    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:1003)
    at org.apache.spark.sql.SparkSession.$anonfun$withActiveAndFrameProfiler$1(SparkSession.scala:1010)
    at com.databricks.spark.util.FrameProfiler$.record(FrameProfiler.scala:80)
    at org.apache.spark.sql.SparkSession.withActiveAndFrameProfiler(SparkSession.scala:1010)
    at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96)
    at org.apache.spark.sql.DataFrameReader.table(DataFrameReader.scala:802)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
    at py4j.Gateway.invoke(Gateway.java:306)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:195)
    at py4j.ClientServerConnection.run(ClientServerConnection.java:115)
    at java.lang.Thread.run(Thread.java:750)

The problem does appear on shared and single-user clusters.
The problem appears for MANAGED and EXTERNAL tables.
The problem did NOT appear, before UC was enabled and the data was living in the hive-metastore.
The user has full access to the storage-credential and external location (for the EXTERNAL case) and the used catalog.

CodePudding user response：

it could be due to ipywidgets(the frontend lib), the databricks can not identify which user trigger the function(loaddf()), so fail to auth in such case. But this is is my guess

CodePudding user response：

we are working on adding ipywidgets support for UC data. Please checkout Databricks ipywidgets user guide for updates.