Python REGEX: remove $ and Curly Brackets from string elements if string element is prefixed by $-CodePudding

In a Python notebook I have a string that I would like to parse in a particular manner and I just can't figure out the necessary regex. This is not important, but the string was priorly a complex nested dictionary derived from transforming an Oozie workflow xml into a Python dictionary via the json.dump() method.

'{"workflow-app": {"@xmlns": "uri:oozie:workflow:0.4", "@name": "simple-Workflow", "start": {"@to": "Create_External_Table"}, "action": [{"@name": "Create_External_Table", "hive": {"@xmlns": "uri:oozie:hive-action:0.4", "job-tracker": "${xyz.com:8088}", "name-node": "${hdfs://rootname}", "script": "${hdfs_path_of_script/external.hive}"}, "ok": {"@to": "Create_orc_Table"}, "error": {"@to": "kill_job"}}, {"@name": "Create_orc_Table", "hive": {"@xmlns": "uri:oozie:hive-action:0.4", "job-tracker": "${xyz.com:8088}", "name-node": "${hdfs://rootname}", "script": "${hdfs_path_of_script/orc.hive}"}, "ok": {"@to": "Insert_into_Table"}, "error": {"@to": "kill_job"}}, {"@name": "Insert_into_Table", "hive": {"@xmlns": "uri:oozie:hive-action:0.4", "job-tracker": "${xyz.com:8088}", "name-node": "${hdfs://rootname}", "script": "${hdfs_path_of_script/Copydata.hive}", "param": "${database_name}"}, "ok": {"@to": "end"}, "error": {"@to": "kill_job"}}], "kill": {"@name": "kill_job", "message": "Job failed"}, "end": {"@name": "end"}}}'

In either case, you'll notice that some of the elements in the string are prefixed by a Dollar Sign. For example "${xyz.com:8088}", "${hdfs_path_of_script/external.hive}" and a couple more.

Other elements are wrapped by curly braces as well, but for those and only those elements that are prefixed by a Dollar Sign, I want to remove the Dollar Sign prefix and the curly braces that immediately wrap around it.

In the above two examples, I would like to obtain "xyz.com:8088" and "hdfs_path_of_script/external.hive". This is what the string would ultimately look like.

'{"workflow-app": {"@xmlns": "uri:oozie:workflow:0.4", "@name": "simple-Workflow", "start": {"@to": "Create_External_Table"}, "action": [{"@name": "Create_External_Table", "hive": {"@xmlns": "uri:oozie:hive-action:0.4", "job-tracker": "xyz.com:8088", "name-node": "hdfs://rootname", "script": "hdfs_path_of_script/external.hive"}, "ok": {"@to": "Create_orc_Table"}, "error": {"@to": "kill_job"}}, {"@name": "Create_orc_Table", "hive": {"@xmlns": "uri:oozie:hive-action:0.4", "job-tracker": "xyz.com:8088", "name-node": "hdfs://rootname", "script": "hdfs_path_of_script/orc.hive"}, "ok": {"@to": "Insert_into_Table"}, "error": {"@to": "kill_job"}}, {"@name": "Insert_into_Table", "hive": {"@xmlns": "uri:oozie:hive-action:0.4", "job-tracker": "xyz.com:8088", "name-node": "hdfs://rootname", "script": "hdfs_path_of_script/Copydata.hive", "param": "database_name"}, "ok": {"@to": "end"}, "error": {"@to": "kill_job"}}], "kill": {"@name": "kill_job", "message": "Job failed"}, "end": {"@name": "end"}}}'

Would someone please help me parse this thing? I am using Python 3.7 if it matters.

CodePudding user response：

You can use recursion to traverse the dictionary and change the appropriate values:

import re
import json


pat = re.compile(r"\$\{(.*)\}")


def transform(d):
    if isinstance(d, dict):
        for k, v in d.items():
            if isinstance(v, str):
                d[k] = pat.sub(r"\1", v)
            else:
                transform(v)
    elif isinstance(d, list):
        for v in d:
            transform(v)


s = '{"workflow-app": {"@xmlns": "uri:oozie:workflow:0.4", "@name": "simple-Workflow", "start": {"@to": "Create_External_Table"}, "action": [{"@name": "Create_External_Table", "hive": {"@xmlns": "uri:oozie:hive-action:0.4", "job-tracker": "${xyz.com:8088}", "name-node": "${hdfs://rootname}", "script": "${hdfs_path_of_script/external.hive}"}, "ok": {"@to": "Create_orc_Table"}, "error": {"@to": "kill_job"}}, {"@name": "Create_orc_Table", "hive": {"@xmlns": "uri:oozie:hive-action:0.4", "job-tracker": "${xyz.com:8088}", "name-node": "${hdfs://rootname}", "script": "${hdfs_path_of_script/orc.hive}"}, "ok": {"@to": "Insert_into_Table"}, "error": {"@to": "kill_job"}}, {"@name": "Insert_into_Table", "hive": {"@xmlns": "uri:oozie:hive-action:0.4", "job-tracker": "${xyz.com:8088}", "name-node": "${hdfs://rootname}", "script": "${hdfs_path_of_script/Copydata.hive}", "param": "${database_name}"}, "ok": {"@to": "end"}, "error": {"@to": "kill_job"}}], "kill": {"@name": "kill_job", "message": "Job failed"}, "end": {"@name": "end"}}}'
data = json.loads(s)
transform(data)
print(json.dumps(data, indent=4))

Prints:

{
    "workflow-app": {
        "@xmlns": "uri:oozie:workflow:0.4",
        "@name": "simple-Workflow",
        "start": {
            "@to": "Create_External_Table"
        },
        "action": [
            {
                "@name": "Create_External_Table",
                "hive": {
                    "@xmlns": "uri:oozie:hive-action:0.4",
                    "job-tracker": "xyz.com:8088",
                    "name-node": "hdfs://rootname",
                    "script": "hdfs_path_of_script/external.hive"
                },
                "ok": {
                    "@to": "Create_orc_Table"
                },
                "error": {
                    "@to": "kill_job"
                }
            },
            {
                "@name": "Create_orc_Table",
                "hive": {
                    "@xmlns": "uri:oozie:hive-action:0.4",
                    "job-tracker": "xyz.com:8088",
                    "name-node": "hdfs://rootname",
                    "script": "hdfs_path_of_script/orc.hive"
                },
                "ok": {
                    "@to": "Insert_into_Table"
                },
                "error": {
                    "@to": "kill_job"
                }
            },
            {
                "@name": "Insert_into_Table",
                "hive": {
                    "@xmlns": "uri:oozie:hive-action:0.4",
                    "job-tracker": "xyz.com:8088",
                    "name-node": "hdfs://rootname",
                    "script": "hdfs_path_of_script/Copydata.hive",
                    "param": "database_name"
                },
                "ok": {
                    "@to": "end"
                },
                "error": {
                    "@to": "kill_job"
                }
            }
        ],
        "kill": {
            "@name": "kill_job",
            "message": "Job failed"
        },
        "end": {
            "@name": "end"
        }
    }
}

CodePudding user response：

I'd probably load with json and process the data, but this regex does what you want:

import re

# your original JSON
ins = '{"workflow-app": {"@xmlns": "uri:oozie:workflow:0.4", "@name": "simple-Workflow", "start": {"@to": "Create_External_Table"}, "action": [{"@name": "Create_External_Table", "hive": {"@xmlns": "uri:oozie:hive-action:0.4", "job-tracker": "${xyz.com:8088}", "name-node": "${hdfs://rootname}", "script": "${hdfs_path_of_script/external.hive}"}, "ok": {"@to": "Create_orc_Table"}, "error": {"@to": "kill_job"}}, {"@name": "Create_orc_Table", "hive": {"@xmlns": "uri:oozie:hive-action:0.4", "job-tracker": "${xyz.com:8088}", "name-node": "${hdfs://rootname}", "script": "${hdfs_path_of_script/orc.hive}"}, "ok": {"@to": "Insert_into_Table"}, "error": {"@to": "kill_job"}}, {"@name": "Insert_into_Table", "hive": {"@xmlns": "uri:oozie:hive-action:0.4", "job-tracker": "${xyz.com:8088}", "name-node": "${hdfs://rootname}", "script": "${hdfs_path_of_script/Copydata.hive}", "param": "${database_name}"}, "ok": {"@to": "end"}, "error": {"@to": "kill_job"}}], "kill": {"@name": "kill_job", "message": "Job failed"}, "end": {"@name": "end"}}}'

# this is your expected output string
outs = '{"workflow-app": {"@xmlns": "uri:oozie:workflow:0.4", "@name": "simple-Workflow", "start": {"@to": "Create_External_Table"}, "action": [{"@name": "Create_External_Table", "hive": {"@xmlns": "uri:oozie:hive-action:0.4", "job-tracker": "xyz.com:8088", "name-node": "hdfs://rootname", "script": "hdfs_path_of_script/external.hive"}, "ok": {"@to": "Create_orc_Table"}, "error": {"@to": "kill_job"}}, {"@name": "Create_orc_Table", "hive": {"@xmlns": "uri:oozie:hive-action:0.4", "job-tracker": "xyz.com:8088", "name-node": "hdfs://rootname", "script": "hdfs_path_of_script/orc.hive"}, "ok": {"@to": "Insert_into_Table"}, "error": {"@to": "kill_job"}}, {"@name": "Insert_into_Table", "hive": {"@xmlns": "uri:oozie:hive-action:0.4", "job-tracker": "xyz.com:8088", "name-node": "hdfs://rootname", "script": "hdfs_path_of_script/Copydata.hive", "param": "database_name"}, "ok": {"@to": "end"}, "error": {"@to": "kill_job"}}], "kill": {"@name": "kill_job", "message": "Job failed"}, "end": {"@name": "end"}}}'

# replace strings that...
# * start with a "
# * then has '${'
# * capture non-greedy arbitrary number of characters with (.*?) 
# * then has '}'
# * then ends with "
# Replace it with the capture in \1 and surround with quotes
subbed = re.sub(r'"\${(.*?)}"', r'"\1"', ins)


print(subbed == outs)
# this output True