Save dict as json using python in databricks-CodePudding

I am trying to save a dict as json in azure data lake/databricks however I am getting a File not found error. Any clue what I am doing wrong?

import json

test_config = {
  "expectations": [
    {
      "kwargs": {
        "column": "role",
        "value_set": [
          "BBV",
          "GEM"
        ]
      },
      "expectation_type": "expect_column_distinct_values_to_be_in_set",
      "meta": {}
    }]
    }

path = '/mnt/lake/enriched/checks/'
json_file = 'test_idm_expectations.json'

with open(path json_file, "w") as fp:
    json.dump(test_config , fp)

And the error I am getting is:

FileNotFoundError: [Errno 2] No such file or directory: "/mnt/lake/enriched/checks/test_idm_expectations.json"

Variations of the path with /dbfs/mnt/lake/enriched/checks/ or dbfs:mnt/lake/enriched/checks/ also do not work.

Any help would be super appreciated. Thanks!

CodePudding user response：

Ensure your python environment sees the mountpoint. You can use os.path.ismount for that. Also, check if the folder tree structure exists. json.dumps will create your file, but only if the folder exists.

Also, tip: to keep indentation, use indent=2 or whatever number of spaces you want in your json, to be pretty printed.

CodePudding user response：

I have reproduced the above and got same error.

enter image description here

The above error occurs if we give wrong folder name in the path. you can see as there is no sourcefolder1 in my container, it generated that error. It will not create new folder for us. It only creates that particular file(mysample1.json in this case).

You can see if I gave correct folder(which is created before) sourcefolder the file is generated successfully.

enter image description here

Give path like /dbfs/mnt/folder/ only as I got same error when I gave path like /mnt/folder/.(open() not identifying paths in the mount might be the reason for it).

File Generated:

enter image description here

CodePudding user response：

Thanks @Remzinho & @Rakesh, the folder structure was indeed not created. adding this snippet to my code before saving the data, solved the issue.

if not os.path.exists(path):
    os.mkdir(path)

with open(path json_file, "w") as fp:
    json.dump(test_config , fp)