Why does my list shape change in a concurrent futures loop?-CodePudding

Can someone tell me why the second time I loop around I can format my numpy list? This is the error I get.

(venv) C:\Users\ENG-DESKTOP-4\PycharmProjects\Delay Tracker>python -m flight_weather_1

Shape of list is : (500, 1, 10)

shape of unformatted numpy is (500, 1, 10)

shape of formatted numpy is (500, 10)

Shape of list is : (500,)

shape of unformatted numpy is (500,)

Traceback (most recent call last): File "C:\Program Files\Python38\lib\runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Program Files\Python38\lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "C:\Users\ENG-DESKTOP-4\PycharmProjects\Delay Tracker\flight_weather_1.py", line 121, in results_formatted = results_thou.reshape(500, 10)

ValueError: cannot reshape array of size 500 into shape (500,10)


def index_weather(index):
    Iata_ = (first_mil.iloc[index]['ORIGIN'])
    flight_time_ = (first_mil.iloc[index]['FL_DATE'])
    weather_data = weather_from_IATA(Iata_, flight_time_)
    return weather_data


for i in range(2):
    index_ = list(range(1   (i * 500), (500   1)   (i * 500)))

    with concurrent.futures.ThreadPoolExecutor() as executor:
        secs = index_
        results = executor.map(index_weather, secs)
        results_list = list(results)
        print("Shape of list is : "   str(np.shape(results_list)))
        results_thou = np.array(results_list)

        results_list.clear()

        print("shape of unformatted numpy is", np.shape(results_thou))
        results_formatted = results_thou.reshape(500, 10)
        print("shape of formatted numpy is", np.shape(results_formatted))
        results_pd = pd.DataFrame(results_formatted)
        pd.concat((all2, results_pd), axis=0)

CodePudding user response：

I think this may already be answered here.

From Kellem Negasi's answer:

"when you reshape a numpy array the total number elements shouldn't change. e.g. a =[2,3,4,5,1,7] if you want to reshape this to a 2Darray then the dimensions multiplied should be equal to the total number elements in the original array a."

From your output what I can guess you have tried to assign the unformatted numpy From your error output:

  ValueError: cannot reshape array of size 500 into shape (500,10),

leading all the way back to your index_ variable which has a length of 500 elements Hope this is helpful

CodePudding user response：

Even without the concurrent stuff:

In [32]: for i in range(2):
    ...:     index_ = list(range(1   (i * 500), (500   1)   (i * 500)))
    ...: 
    ...:     results = index_.copy()
    ...:     results_list = list(results)
    ...:     print("Shape of list is : "   str(np.shape(results_list)))
    ...:     results_thou = np.array(results_list)
    ...:     print("shape of unformatted numpy is", np.shape(results_thou))
    ...:     results_formatted = results_thou.reshape(500, 10)
    ...:     print("shape of formatted numpy is", np.shape(results_formatted))
    ...: 
Shape of list is : (500,)
shape of unformatted numpy is (500,)
Traceback (most recent call last):
  Input In [32] in <cell line: 1>
    results_formatted = results_thou.reshape(500, 10)
ValueError: cannot reshape array of size 500 into shape (500,10)

For i=0

In [36]: np.shape(index_)
Out[36]: (500,)

In [41]: i=0; range(1   (i * 500), (500   1)   (i * 500))
Out[41]: range(1, 501)

I assumed

results = executor.map(index_weather, secs)

produce a result that matches secs shape. But your (500, 1, 10) first result suggests that for that case index_weather returned (1,10) arrays. Figuring out when that's the case, and when it just returns a scalar, isn't something I can do.