Python: Gridsearch Without Machine Learning?

Question

I want to optimize an algorithm that has several variable parameters as input.

For machine learning tasks, Sklearn offers the optimization of hyperparameters with the gridsearch functionality.

Is there a standardized way / library in Python that allows the optimization of hyperparameters that is not limited to machine learning topics?

Answer 1

You can create a custom pipeline/estimator ( see link http://scikit-learn.org/dev/developers/contributing.html#rolling-your-own-estimator) with a score method to compare the results.

The ParameterGrid might help you too. It will automatically populated all the hyper-parameters settings.

Answer 2

You might consider scipy's optimize.brute, which essentially is the same, although not that constrained in terms of API-usage. You will just have to define a function, which returns a scalar.

Minimize a function over a given range by brute force.

Uses the “brute force” method, i.e. computes the function’s value at each point of a multidimensional grid of points, to find the global minimum of the function.

Shameless example-copy from the docs:

Code

import numpy as np
from scipy import optimize


params = (2, 3, 7, 8, 9, 10, 44, -1, 2, 26, 1, -2, 0.5)
def f1(z, *params):
    x, y = z
    a, b, c, d, e, f, g, h, i, j, k, l, scale = params
    return (a * x**2 + b * x * y + c * y**2 + d*x + e*y + f)

def f2(z, *params):
    x, y = z
    a, b, c, d, e, f, g, h, i, j, k, l, scale = params
    return (-g*np.exp(-((x-h)**2 + (y-i)**2) / scale))


def f3(z, *params):
    x, y = z
    a, b, c, d, e, f, g, h, i, j, k, l, scale = params
    return (-j*np.exp(-((x-k)**2 + (y-l)**2) / scale))


def f(z, *params):
    return f1(z, *params) + f2(z, *params) + f3(z, *params)

rranges = (slice(-4, 4, 0.25), slice(-4, 4, 0.25))
resbrute = optimize.brute(f, rranges, args=params, full_output=True,
                          finish=optimize.fmin)
print(resbrute[:2])  # x0, feval

Out

(array([-1.05665192,  1.80834843]), -3.4085818767996527)

Brute-force functions are not much black-magic and often one might consider an own implementation. The scipy-example above has one more interesting feature:

finish : callable, optional

An optimization function that is called with the result of brute force minimization as initial guess. finish should take func and the initial guess as positional arguments, and take args as keyword arguments. It may additionally take full_output and/or disp as keyword arguments. Use None if no “polishing” function is to be used. See Notes for more details.

which i would recommend for most use-cases (in continuous-space). But be sure to get some minimal understanding what this is doing to understand there are use-cases where you don't want to do this (discrete-space results needed; slow function-evaluation).

If you are using sklearn, you already have scipy installed (it's a dependency).

Edit: here some small plot i created (code) to show what finish is doing (local-opt) with an 1d-example (not the best example, but easier to plot):

Answer 3

You can have a look also at Bayesian Optimization. In this github repository you can find the easy implementation.

The difference is that Bayesian Optimization doesn't look into specific values that you input range, but it looks for values within the range.

The example below is taken from their repository, so that you see how easy is the implementation!

def black_box_function(x, y):
    """Function with unknown internals we wish to maximize.

    This is just serving as an example, for all intents and
    purposes think of the internals of this function, i.e.: the process
    which generates its output values, as unknown.
    """
    return -x ** 2 - (y - 1) ** 2 + 1

from bayes_opt import BayesianOptimization

# Bounded region of parameter space
pbounds = {'x': (2, 4), 'y': (-3, 3)}

optimizer = BayesianOptimization(
    f=black_box_function,
    pbounds=pbounds,
    random_state=1,
)

optimizer.maximize(
    init_points=2,
    n_iter=3,
)

print(optimizer.max)
>>> {'target': -4.441293113411222, 'params': {'y': -0.005822117636089974, 'x': 2.104665051994087}}

Answer 4

Sklearn can also be used independent of machine learning topics, hence and for the sake of completeness,

I propose:

from sklearn.model_selection import ParameterGrid
param_grid = {'value_1': [1, 2, 3], 'value_2': [0, 1, 2, 3, 5]}
for params in ParameterGrid(param_grid):
    function(params['value_1'], params['value_2'])

Find detailed documentation here.