How to install python package from private git repo using init bash script on databricks?-CodePudding

I'm trying to pip install a python package from a private github repo using a init.sh script that I uploaded to my s3 bucket.

This is my init.sh file

#!/bin/bash
TOKEN={{secrets/private-repo/github}}
pip install git https://${TOKEN}@github.com/<path-to-repo>

When I try to create my cluster I get the following error messsage. Init script failure: Cluster scoped init script s3://<s3_bucket>/init.sh failed: Script exit status is non-zero

I create a secret through the API with scope and key as private-repo and github. I tested this using a notebook and it worked fine.

Documentation Used: https://docs.databricks.com/security/secrets/secrets.html#reference-a-secret-in-an-environment-variable

CodePudding user response：

The problem is that you're trying to refer to the secret using the {{secrets/private-repo/github}} syntax, but it doesn't work from the inside of the init script.

You need to define an environment variable on the cluster level and use that secret syntax there, and then it will be available inside your init script. See documentation on that topic.