Returning every possible combination of elements from an unknown number of lists-CodePudding

I'm trying to create my own model fitting function (I know SciPy exists, but I want to do it myself) and I'd like it to be able to take any arbitrary number of model parameters. However, the problem I'm having is that I don't know how to generalize for an unknown number of model parameters.

Say I have two parameters (for example, fitting a simple y=mx c straight line). For each parameter, I have a list containing a range of values. I then want to try every possible combination of elements from these two lists, and use least-squares fitting to find the best combination of parameters. Example code:

import numpy as np

#define parameter lists 
p_list_1 = np.linspace(0, 10, 11)
p_list_2 = np.linspace(0, 10, 11)

# get every combination of parameter values
for param_1 in p_list_1:
    for param_2 in p_list_2:

        p_combo = [param_1, param_2]

        # and then I would use this parameter combination to fit a model, compute the residuals, etc.

However, this approach only works if I know the number of parameters I'm trying to fit and becomes incredibly cumbersome if I want to add support for variable numbers of parameters. For example, trying to fit up to, say, 5 parameters would mean I would need to do something like:

import numpy as np

# define parameter lists
p_list_1 = np.linspace(0, 10, 11)
p_list_2 = np.linspace(0, 10, 11)
p_list_3 = np.linspace(0, 10, 11)
p_list_4 = np.linspace(0, 10, 11)
p_list_5 = np.linspace(0, 10, 11)

# define number of parameters
n_params = 5

### Skip lines for 1, 2, 3, 4 parameters ###

if n_params == 5:
    for param_1 in p_list_1:
        for param_2 in p_list_2:
            for param_3 in p_list_3:
                for param_4 in p_list_4:
                    for param_5 in p_list_5:
                        
                        p_combo = [param_1, param_2, param_3, param_4, param_5]
                        
                        # and then I would use this parameter combination to fit a model, compute the residauls, etc.

This approach is obviously not the best way to do it, and requires me to manually add support for specific numbers of parameters.

So, my question is, given some arbitrary number of lists, is there a concise way of getting every possible combination of elements from said lists?

Edit:

I found an answer here. Below is some code for reference:

import itertools
import numpy as np

# define parameter ranges
p_list_1 = np.linspace(0, 10, 11)
p_list_2 = np.linspace(0, 10, 11)
p_list_3 = np.linspace(0, 10, 11)

# combine parameter ranges into single list
long_list = [p_list_1, p_list_2, p_list_3]

# generate Cartesian product/all combinations/all permutations
for combo in itertools.product(*long_list): # note the asterisk
    print(combo)

This generates every possible combination of elements from an arbitrary number of lists, provided those lists are within a master list. This is exactly what I was looking for.

CodePudding user response：

I would suggest using the function product of itertools for that

import itertools

import numpy as np

p_list_1 = np.linspace(0, 10, 11)
p_list_2 = np.linspace(0, 10, 11)
p_list_3 = np.linspace(0, 10, 11)


for combo in itertools.product(p_list_1,p_list_2,p_list_3):
    print(combo)

print (len(list(itertools.product(p_list_1,p_list_2,p_list_3)))) #prins 1331

CodePudding user response：

Assuming you want to use the same parameter list for each parameter (e.g., each parameter should go from 0 to 10 in steps of 1), for r parameters, then there's no reason to maintain separate parameter lists:

def param_list_builder(r, a=0, b=10, n=11):
    return list(itertools.product(np.linspace(a, b, n), repeat=r))

Likely, though, you won't want to store all combinations in memory at the same time. If you are training models one at a time, then there's no reason why you'd want more than one set of parameters at a time. In which case, you should use a generator:

def param_generator(r, a=0, b=10, n=11):
    return itertools.product(np.linspace(a, b, n), repeat=r)

Then, while testing your models:

for params in param_generator(3, 0, 10, 11):
    test_model(params)

Or, if you want to grab some params whenever you want:

In [10]: p_gen = param_generator(2, 0, 3, 4)

In [11]: next(p_gen)
Out[11]: (0.0, 0.0)

In [12]: next(p_gen)
Out[12]: (0.0, 1.0)

In [13]: next(p_gen)
Out[13]: (0.0, 2.0)

In [14]: next(p_gen)
Out[14]: (0.0, 3.0)

In [15]: next(p_gen)
Out[15]: (1.0, 0.0)

In [16]: next(p_gen)
Out[16]: (1.0, 1.0)

.
.
.