Nelder mead isn't converging (scipy). Why is there only one initial point?-CodePudding

Here is a minimal reproducible example. If I set x_0 to 0, it seems that the optimiser is not able to move from it. If I set x_0 to 10, it converges to the global minimum. Any idea of what that is? Is it linked to how the Nelder-Mead algorithm work?

I get that it's probably stuck and terminates before converging and that optimisers are sometimes sensitive to initial guess but I find this hard to check because we only give one initial point and the Nelder-Mead algorithm starts with (number of dimensions 1) points so here we should be starting with two points?? Evidently, we only give one initial guess to SciPy, so my question is: what second point does SciPy take and why does that lead to non-convergence if I use x0 = 0?

With x_0 = 0

import numpy as np
from scipy.optimize import minimize, Bounds


def func(x):
    return ((x - 1000) ** 2   (x - 10000) ** 2)   440 ** (3 / 2) * np.sqrt(x)


x0 = np.array([0])
x_bounds = Bounds(0, 25000)
res = minimize(func, x0, method='Nelder-Mead', tol=1e-6, bounds=x_bounds)
print("Best:", res.x)
print("Loss:", func(res.x[0]))

Output:

Best: [0.]
Loss: 101000000.0

With x0 = 10:

import numpy as np
from scipy.optimize import minimize, Bounds


def func(x):
    return ((x - 1000) ** 2   (x - 10000) ** 2)   440 ** (3 / 2) * np.sqrt(x)


x0 = np.array([10])
x_bounds = Bounds(0, 25000)
res = minimize(func, x0, method='Nelder-Mead', tol=1e-6, bounds=x_bounds)
print("Best:", res.x)
print("Loss:", func(res.x[0]))

Output:

Best: [5484.42156982]
Loss: 41183994.677765995

I couldn't find much on the scipy documentation of Nelder-Mead. Neither could I on the scipy documentation of minimize.

CodePudding user response：

https://machinelearningmastery.com/how-to-use-nelder-mead-optimization-in-python/

"A starting point must be provided to the algorithm, which may be the endpoint of another global optimization algorithm or a random point drawn from the domain.

Given that the algorithm may get stuck, it may benefit from multiple restarts with different starting points.

The algorithm works by using a shape structure (called a simplex) composed of n 1 points (vertices), where n is the number of input dimensions to the function."

The answer for why you are providing only one initial point is here: https://github.com/scipy/scipy/blob/b5d8bab88af61d61de09641243848df63380a67f/scipy/optimize/_optimize.py#L743

So simplex is built from n 1 points, and each of them are your starting point (replicated). So the number of dimensions is sth other than the number of the points.