how to delete subfolder in ADLS gen2 via azure databricks-CodePudding

I need to delete ADLS Gen2 subfolder with specific name using databricks dbutils but not able to perform wild card recursion.

e.g. 
adlsstorageaccount/container1/folder2/folder3/folderA/**abc**/.parquet
adlsstorageaccount/container1/folder2/folder3/folderB/**abc**/.parquet
adlsstorageaccount/container1/folder2/folder3/folderC/**abc**/.parquet

Need to delete subfolder name "abc" & its contents only, entire path is dynamic.

CodePudding user response：

We have reproduced the same Folders in our environment and here are the commands that worked.

$FileSystemName="<Your Container Name>"
$dirname="folder2/folder3/folderA/abc/"
$ctx=New-AzStorageContext -StorageAccountName '<Your Storage Account>' -StorageAccountKey '<Your Access Key>'
Remove-AzDataLakeGen2Item -Context $ctx -FileSystem $filesystemName -Path $dirname

and then confirm it with Y - 'yes'

Here are the screenshots for your reference

Before using the command execution

After the command execution

REFERENCES: https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-powershell#delete-a-directory

CodePudding user response：

you need just use magic command %sh in databricks and remove it with wilcard using standard linux commands, something like:

%sh
rm /dbfs/mnt/adlsstorageaccount/container1/folder2/folder3/folderA/*abc*/*.parquet