How to avoid `for` loops for indexing for huge data-CodePudding

This is my code. I would like to avoid for loop for huge number of iterations.

iterations = 10000
max_number = 9700
W = is a matrix(300,9700)

def grad_function(x,y):
    grad = np.sum(W*(x - y))
    return grad

x = np.arange(300)
y = np.arange(9700)

for i in range (iterations):
    for j in range(max_number):
        result = grad_function(x,y)
print(result)

I tried using the map() function instead but it does not give me a result including 9700 elements; instead, it gives me a list of 300 elements which is not correct.

CodePudding user response：

As far as I understand a map() does essentially the same as your nested for-loops and if you need to check every value in the matrix, this will be one of the fastest options anyway.

CodePudding user response：

I think you're trying to do something like this:

W = np.random.random((300, 9700))
x = np.arange(300)
y = np.arange(9700)

result = (W.T @ x) - y

Then result.shape is (9700,). Alternatively, you can make W have shape (9700, 300) and then you won't need to transpose it.

In general, you never need to loop with NumPy. Just do the maths on the arrays directly. The only trick I've used here is @ instead of np.matmul.

If you really want to use a function for the maths, be sure to pass in everything it needs:

def grad(x, y, W):
    return (W.T @ x) - y