Converting a Dictionary to DataFrame in Python-CodePudding

I have a dictionary of a static structure:

Key: Key: Value`

I will need to record data a few extra keys deep to the same depth, so somewhat uniform.

Example Dictionary:

{
    "Emissions": {
        "305-1": [
            "2014_249989",
            "2015_339998",
            "2016_617957",
            "2017_827230"
        ],
        "305-2": [
            "2014_33163",
            "2015_64280",
            "2016_502748",
            "2017_675091"
        ],
    },
    "Effluents and Waste": {
        "306-1": [
            "2014_143.29",
            "2015_277.86",
            "2016_385.67",
            "2017_460.6"
        ],
        "306-2": "blah blah blah",
    }
}

I want a DataFrame of this structure:

Parent Key | Child Key | Child Value
Parent Key | Child Key | Child Value
Parent Key | Child Key | Child Value
Parent Key | Child Key | Child Value

Example Desired DataFrame:

Emissions | 305-1 | ["2014_249989", "2015_339998", "2016_617957", "2017_827230"]
Emissions | 305-2 | ["2014_33163", "2015_64280", "2016_502748", "2017_675091"]
Effluents and Waste| 306-1 | ["2014_249989", "2015_339998", "2016_617957", "2017_827230"]
Effluents and Waste | 306-2 | blah blah blah

Where all Child Values are either a list object of strings or a string object.

From researching I found pandas.DataFrame.from_dict(). However neither orient values help in my case. As it is intended for flat dictionaries.

I genuinely haven't a clue on how to approach this. What a simple libraries may be needed etc.

Please let me know if there are further details/ nuances I could clarify.

CodePudding user response：

Use:

import pandas as pd

data = {
    "Emissions": {
        "305-1": ["2014_249989", "2015_339998", "2016_617957", "2017_827230"],
        "305-2": ["2014_33163", "2015_64280", "2016_502748", "2017_675091"],
    },
    "Effluents and Waste": {
        "306-1": ["2014_143.29", "2015_277.86", "2016_385.67", "2017_460.6"],
        "306-2": "blah blah blah",
    }
}

data = [[key, ikey, value] for key, values in data.items() for ikey, value in values.items()]
res = pd.DataFrame(data)
print(res)

Output

                     0  ...                                                  2
0            Emissions  ...  [2014_249989, 2015_339998, 2016_617957, 2017_8...
1            Emissions  ...  [2014_33163, 2015_64280, 2016_502748, 2017_675...
2  Effluents and Waste  ...  [2014_143.29, 2015_277.86, 2016_385.67, 2017_4...
3  Effluents and Waste  ...                                     blah blah blah

CodePudding user response：

A simple way to do this would just be to "flatten" your dictionary so that you get the "parent, child key, child value" structure you are hoping for and then construct a DataFrame from that.

Example:

example_dictionary = {
    "Emissions": {
        "305-1": [
            "2014_249989",
            "2015_339998",
            "2016_617957",
            "2017_827230"
        ],
        "305-2": [
            "2014_33163",
            "2015_64280",
            "2016_502748",
            "2017_675091"
        ],
    },
    "Effluents and Waste": {
        "306-1": [
            "2014_143.29",
            "2015_277.86",
            "2016_385.67",
            "2017_460.6"
        ],
        "306-2": "blah blah blah",
    }
}

def flatten(d):
    return [[key, subkey, d[key][subkey]] for key in d for subkey in d[key]]

pd.DataFrame(flatten(example_dictionary))

The result looks like:

0   1   2
0   Emissions   305-1   [2014_249989, 2015_339998, 2016_617957, 2017_8...
1   Emissions   305-2   [2014_33163, 2015_64280, 2016_502748, 2017_675...
2   Effluents and Waste     306-1   [2014_143.29, 2015_277.86, 2016_385.67, 2017_4...
3   Effluents and Waste     306-2   blah blah blah