load arrays from a text file-CodePudding

I want to import many big arrays written in a text file to work with them using python and numpy. For instance, my text file looks like:

array_001 = [1.2 5.5 1.0 4.7E-5
             0.9 8.6 2.1 5.8E-4]
.
.
.
array_100 = [1.9 4.6 1.4 8.7E-5
             0.9 4.5 2.5 9.7E-5]

The arrays are in ascending order starting from 001. Any help is appreciated. Thanks!

CodePudding user response：

I can think about something like:

import numpy

with open("lists.txt") as f:
  for line in f:
    nparr = numpy.array([float(x) for x in line.split("=")[1].replace("[", "").replace("]", "").strip().split()])
    print(type(nparr), nparr, nparr.sum())
    # <class 'numpy.ndarray'> [1.2e 00 5.5e 00 1.0e 00 4.7e-05 9.0e-01 8.6e 00 2.1e 00 5.8e-04] 19.300627
    # ...

Demo

CodePudding user response：

If someone delivers data to you in a format like this, the best course of action would be to ask them to export the data in a more standard text-based format, like .csv or .json.

However, sometimes you have to deal with what you have, just don't make the mistake of creating bad formats like this yourself.

Consider whether the data you need to process fits in memory, or whether you need something more efficient. If it does fit in memory, a simple regex-based solution may be best:

import re
from csv import writer

with open('file.txt') as f:
    content = f.read()

with open('better_file.csv', 'w', newline='') as f:
    csv_writer = writer(f)
    for rec in re.findall('array_(...) = \[((?:[^\s\[] \s*) )\]', content):
       rec = [rec[0]]   [float(x) for x in rec[1].split()]
       csv_writer.writerow(rec)

The resulting .csv file can be easily read by any number of Python libraries.

If it's actually a far larger file (but that doesn't appear to be the case, from your example), then you may need to go through it line by line and add some logic.

In that case, please give it a try and come back with a code example - in general, please ask questions on StackOverflow about code you wrote, instead of expecting someone else to do the work for you.