There is a list (price_list) of 10 million numbers as a price in range (0-100). I would like to do the following operations consequently:

1. Filter the list with this condition (list_item < 30)
2. Multiply each list_item with a fixed number (mapping)
3. compute the sum of all elements (reduce)

Here is the sample code:

import time
import numpy as np
from functools import reduce

with open('price_list.txt') as f:
    price_list = f.read().split('\n')
    
price_list = np.array(price_list).astype(int)  # convert string to int

I tried following options:

• Using NumPy module (0.1069948673248291 seconds): total = np.sum(price_list[price_list < 30] * 1.05)

• Using brute force loop (9.485718965530396 seconds)

• Using filter, map, and reduce functions (10.078220844268799 seconds): total = reduce(lambda x,y: x y,map(lambda x: x * 1.05, filter(lambda x: x < 30, price_list)) )

• Using list comprehension (8.609008073806763 seconds): total= sum([x * 1.05 for x in price_list if x <30])

• Using generator (8.780538320541382 seconds): total= sum((x * 1.05 for x in price_list if x <30))

It is obvious that the NumPy module is significantly fast in these operations. Is there any alternative solution for running this code faster using Python built-in functions and capabilities?

CodePudding user response：

Benchmark with three new solutions (starting from a list of strings as the one you got from your file):

 4.1 s  original_numpy
11.6 s  original_listcomp
 4.8 s  better_listcomp
 1.7 s  even_better_listcomp
 0.6 s  counter

better_listcomp's only difference to yours is that I added .tolist(), so that price_list is actually a list and Python gets to work with Python ints instead NumPy ints, which is much faster. (Calling an array a list was really mislabeling).

even_better_listcomp goes a step further, never even going through NumPy at all. (And it multiplies with 1.05 just once, after adding.)

counter goes even further, delaying the conversion from string to int until after counting the price strings.

Code (Try it online!):

def better_listcomp(price_list):
    price_list = np.array(price_list).astype(int).tolist()
    return sum([x * 1.05 for x in price_list if x < 30])

def even_better_listcomp(price_list):
    prices = map(int, price_list)
    return sum([x for x in prices if x < 30]) * 1.05

def counter(price_list):
    return sum(int(k) * v for k, v in Counter(price_list).items() if int(k) < 30) * 1.05

def original_numpy(price_list):
    price_list = np.array(price_list).astype(int)
    return np.sum(price_list[price_list < 30] * 1.05)

def original_listcomp(price_list):
    price_list = np.array(price_list).astype(int)
    return sum([x * 1.05 for x in price_list if x < 30])

funcs = original_numpy, original_listcomp, better_listcomp, even_better_listcomp, counter

from time import time
import numpy as np
from collections import Counter

price_list = list(map(str, range(1, 100))) * 101010

for _ in range(2):
    for f in funcs:
        t = time()
        f(price_list)
        print(f'{time() - t :4.1f} s ', f.__name__)
    print()

CodePudding user response：

Numpy is written in C with efficient routines specialized for vectorial operations on numeric data.

Another particularity is that the data is stored in specialized containers (arrays) that optimize memory access.

In your case you could use:

import numpy as np

# load the data directly as numpy array
a = np.loadtxt('price_list.txt', dtype=np.int64)

# filter, sum, multiply
out = (a[a<30]).sum()*1.05

CodePudding user response：

Short answer: no.

Numpy uses vectorised version of math operations wherever it can. It means that if you have a nd.array of values, it can fit them into the SIMD registers of procesors. This means, that it can multiply tuples of numbers simultaneously (4, 8 or 16 at the same time depending on the SIMD version your processor supports.

Since Python is an interpreted language, you don't really have any alternatives. You would have to manually code procedures in a language that has access to the low level SIMD/AVX instructions of your processor and wrap it in the python API... which is exactly what NumPy is.