Basically; I messed up. I have pickled some data, a pretty massive dictionary, and while my computer was able to create and pickle that dictionary in the first place, it crashes from running out of memory when I try to unpickle it. I need to unpickle it somehow, to get the data back, and then I can write each entry of the dictionary to a separate file that can actually fit in memory. My best guess for how to do that is to unpickle the dictionary entry by entry and then pickle each entry into it's own file, or failing that to unpickle it but somehow leave it as an on-disk object. I can't seem to find any information on how pickled data is actually stored to start writing a program to recover the data.

CodePudding user response：

pickle is a serialization format unique to Python, and there is no user-level documentation for it.

However, there are extensive comments in a standard distribution's "Lib/pickletools.py" module, and with enough effort you should be able to use that module's dis() function to produce output you can parse yourself, or modify the source itself. dis() does not execute a pickle (meaning it doesn't build any Python objects from the pickle). Instead it reads the pickle file a few bytes at a time, and prints (to stdout, by default) a more human-readable form of what those bytes "mean".