# Defining the Parameter Configuration Space¶

This pages explains in detail how the types, allowed and default values of all parameters are defined. It also covers how dependencies and constraints between them can be modeled.

## Parameter Definitions¶

The definition of the parameters is stored in a dictionary, where the name of the parameter as the key, and a tuple containing all necessary information constitutes the corresponding value. SMAC, the configuration system underlying pySMAC, discriminates four different parameter types:

**real**– take any value in a specified range.**integer**– take any integral value in a specified range.**categorical**– take one of a finite set of values.**ordinal**– are essentially categorical parameters with a natural ordering between the elements.

We shall look at a simple example for all four types to show how they are defined:

```
parameter_definitions=dict(\
a_float_parameter = ("real" , [-3.5, 2.48] , 1.1 ),
an_integer_parameter = ("integer" , [1, 1000] , 2 ,"log"),
a_categorical_parameter = ("categorical", ["yes", "no", "maybe"] , "yes" ),
a_ordinal_parameter = ("ordinal" , ["cold", "cool", "medium", "warm", "hot"], "medium" )
)
```

The definition of each parameter follows the same pattern: first its type, followed by a list defining the allowed values, and finally the default value.

For **real** and **integer** type, the allowed values are defined by a range,
represented by a list with exactly two elements. The default
value has to be inside this range to yield a legal definition. Both the
range and the default of an **integer** have to be Python **int**s

There exists an optional flag “log” that can be given additionally as the last element of a tuple for these two types. If given, the parameter is varied on a logarithmic scale, meaning that the logarithm of the value is uniformly distributed between the logarithm of the bounds. Therefore, this option can only be given if the parameter is strictly positive!

For **categorical** and **ordinal** the list of allowed values can contain
any number (>0) of elements. Every element constitutes a valid value for
this parameter. The default value has to be among them. The ordering of
an **ordinal** parameter is established by the order in the list.

The Python type for **categorical** and **ordinal** parameters can be a numeric
type of a string. The only restriction is, that all allowed values for one
parameter are all of the same time. For example, the following definition is
not valid:

```
parameter_definitions=dict(\
a = ("integer" , [1, 1000] , 2.0 ), # default value is not of type int
b = ("categorical", ["True", "False",1], 1), # 2 str and one int value
)
```

Note

Defining the parameter configuration space can be quite challenging, e.g., if the number of parameters is large. So typos and/or inconsistencies can happen. SMAC itself checks the definitions in great detail, and pySMAC provides also some sanity checks with (hopefully) helpful error messages to assist in this tedious task.

## Conditional Parameter Clauses¶

In many cases, certain parameters only have any meaning if another one takes a certain value. For example, one parameter might activate or deactivate a subroutine which has parameters itself. Naturally, the latter are only relevant when the subroutine is actually used. These dependencies can be expressed in pySMAC to accelerate the configuration process (by reducing the number of active parameters).

To illustrate this, let’s focus on the example in `sklearn_model_selection.py`

.
The script demonstrates a use-case from machine learning. Given a data set, there
are numerous models that can be used to *learn* from it and apply this *knowledge*
to unseen data points. In this simple example, we generate a random data set and
try to find the model (with the best parameter settings) among only three very basic
ones, namely: k-nearest-neighbors, random forests, and extremely randomized trees.
In order to run that example, you need the scikit-learn package, but the
source code below should be illustrative enough to show how to use conditionals.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 | ```
from __future__ import print_function, division
import pysmac
import sklearn.ensemble
import sklearn.neighbors
import sklearn.datasets
import sklearn.cross_validation
# We use a random classification data set generated by sklearn
# As commonly done, we use a train-test split to avoid overfitting.
X,Y = sklearn.datasets.make_classification(1000, 20)
X_train, X_test, Y_train, Y_test = \
sklearn.cross_validation.train_test_split(X,Y, test_size=0.33, random_state=1)
# Here, SMAC can choose between to different models at each evaluation. To
# make the search more efficient, it is important to tell SMAC that some
# parameters are associated with certain classifiers
def choose_classifier(classifier, # which classifier to use
# parameters for the tree based classifiers
trees_n_estimators = None, trees_criterion = None,
trees_max_features = None, trees_max_depth = None,
# the ones for k-nearest-neighbors
knn_n_neighbors=None, knn_weights=None):
#note that possibly inactive variables have to be optional
#as pysmac does not assign a value for inactive variables
#during the minimization phase
if classifier == 'random_forest':
predictor = sklearn.ensemble.RandomForestClassifier(
trees_n_estimators, trees_criterion,
trees_max_features, trees_max_depth)
elif classifier == 'extra_trees':
predictor = sklearn.ensemble.ExtraTreesClassifier(
trees_n_estimators, trees_criterion,
trees_max_features, trees_max_depth)
elif classifier == 'k_nearest_neighbors':
predictor = sklearn.neighbors.KNeighborsClassifier(
knn_n_neighbors, knn_weights)
predictor.fit(X_train, Y_train)
return -predictor.score(X_test, Y_test)
# defining all the parameters with respective defaults.
parameter_definition=dict(\
trees_max_depth = ("integer", [1,10], 4),
trees_max_features=("integer", [1,20], 10),
trees_n_estimators=("integer", [1,100],10 ,'log'),
trees_criterion =("categorical", ['gini', 'entropy'], 'entropy'),
knn_n_neighbors = ("integer", [1,100], 10, 'log'),
knn_weights = ("categorical", ['uniform', 'distance'], 'uniform'),
classifier = ("ordinal", ['random_forest','extra_trees' ,'k_nearest_neighbors'], 'random_forest'),
# Usually you would make this a categorical, but to showcase all
# conditional clauses, let's pretend it's an ordinal parameter,
# so we can use > and <.
)
# here we define the dependencies between the parameters. the notation is
# <child> | <parent> in { <parent value>, ... }
# and means that the child parameter is only active if the parent parameter
# takes one of the value in the listed set. The notation follows the SMAC
# manual one to one. Note there is no checking for correctness beyond
# what SMAC does. I.e., when you have a typo in here, you don't get any
# meaningful output, unless you set debug = True below!
conditionals = [ 'trees_max_depth | classifier in {random_forest, extra_trees}',
'trees_max_features | classifier in {random_forest} || classifier == extra_trees',
'trees_n_estimators | classifier != k_nearest_neighbors',
'trees_criterion | classifier < k_nearest_neighbors',
'knn_n_neighbors | classifier > extra_trees',
'knn_weights | classifier == k_nearest_neighbors && classifier != extra_trees && classifier != random_forest'
]
# creation of the SMAC_optimizer object. Notice the optional debug flag
opt = pysmac.SMAC_optimizer( debug = 0,
working_directory = '/tmp/pysmac_test/', persistent_files=True, )
# first we try the sklearn default, so we can see if SMAC can improve the performance
predictor = sklearn.ensemble.RandomForestClassifier()
predictor.fit(X_train, Y_train)
print('The default accuracy of the random forest is %f'%predictor.score(X_test, Y_test))
predictor = sklearn.ensemble.ExtraTreesClassifier()
predictor.fit(X_train, Y_train)
print('The default accuracy of the extremely randomized trees is %f'%predictor.score(X_test, Y_test))
predictor = sklearn.neighbors.KNeighborsClassifier()
predictor.fit(X_train, Y_train)
print('The default accuracy of k-nearest-neighbors is %f'%predictor.score(X_test, Y_test))
# The minimize method also has optional arguments (more on that in the section on advanced configuration).
value, parameters = opt.minimize(choose_classifier,
500 , parameter_definition,
conditional_clauses = conditionals)
print('The highest accuracy found: %f'%(-value))
print('Parameter setting %s'%parameters)
``` |

The output looks like that (note the random data set leads to slightly different numbers for every run):

```
The default accuracy of the random forest is 0.909091
The default accuracy of the extremely randomized trees is 0.903030
The default accuracy of k-nearest-neighbors is 0.863636
The highest accuracy found: 0.936364
Parameter setting {'knn_weights': 'distance', 'trees_n_estimators': '8', 'knn_n_neighbors': '1', 'classifier': 'random_forest', 'trees_max_features': '10', 'trees_max_depth': '2', 'trees_criterion': 'gini'}
```

The script shows how pySMAC can be used for model selection and simultaneous
optimization. The function to be minimized (*choose_classifier*, line 19)
returns the negative accuracy of training one out of three machine learning
models (a random forest, extremely randomized trees, and k-nearest-neighbors).
So effectively, SMAC is asked to maximize the accuracy choosing either of
these models and its respective parameters.

The parameter definitions are stated between line 44 and 55. Naturally, the ones for K-nearest-neighbors and the two tree based classifiers are independent. Therefore, the parameters of the former affect the accuracy only if this classifier is actually chosen.

The variable *conditionals*, defined in line 64, shows some examples for
how these dependencies between parameters are expressed. Generally they
follow the template:

```
child_name | condition1 (&& or ||) condition2 (&& or ||) ...
```

The *child* variable is only considered active if the logic expression following
the “|” is true.

Note

- From the SMAC manual
- Parameters not listed as a child in any conditional parameter clause are always active.
- A child’s name can appear only once.
- There is no support for parenthesis with conditionals. The && connective has higher precedence than ||, so a||b&& c||d is the same as a||(b&&c)||d.

The conditions can take different forms:

```
parent_name in {value1, value2, ... }
parent_name == value parent_name != value
parent_name < value parent_name > value
```

The first one is true if the parent takes any of the values listed. The
other expressions have the regular meaning. The operators in the last line
are only legal for **real**, **integer** or **ordinal**, while the
others can be used with any type.

## Forbidden Parameter Clauses¶

In some use-cases, certain parameter configurations might be illegal, or lead to undefined behavior. For example, some algorithms might be able to employ different data structures, and different subroutines, controlled by two parameters:

```
parameter_definition = dict(\
DS = ("categorical", [DataStructure1, DataStructure2, DataStructure3],DataStructure1 ),
SR = ("categorical", [SubRoutine1, SubRoutine2, SubRoutine3], SubRoutine1)
)
```

Let’s assume that DataStructure2 is incompatible with SubRoutine3, i.e. evaluating this combination does not yield a meaningful result, or might even cause a system crash. That means one out of nine possible choices for these two parameters is forbidden.

One can certainly change the parameters and their definitions such that they exclude this case explicitly. One could, i.e., combine the two parameters and list all eight allowed values:

```
parameter_definition = dict(\
DS_SR = ("categorical", [DS1_SR1, DS1_SR2, DS1_SR2, DS2_SR1, DS2_SR2, DS3_SR1, DS3_SR2, DS3_SR3], [DS1_SR1])
)
```

This is not only unpractical, but it forbids SMAC to learn about the data
structures and subroutines independently. It is much more efficient to
specifically exclude this one combination by defining a forbidden parameter
clause. The **classic syntax** is as follows:

```
"{parameter_name1 = value1, ..., parameter_nameN = ValueN}"
```

It allows to specify combinations of values that are forbidden. For our example above, the appropriate forbidden clause would be

```
forbidden_confs = ["{DS = DataStructure2, SR = SubRoutine3}"]
```

Note

The pair of curly braces {} around the expression is mandatory. The pySMAC notation here is a direct copy of the SMAC one. These strings are merely handed over to SMAC without any processing. That way, statements from the SMAC manual are applicable to pySMAC as well.

A list of all forbidden clauses is than passed to the minimize method with the forbidden_clauses keyword. So the corresponding call to the minimize method would look like this:

```
opt = pysmac.SMAC_optimizer()
value, config = opt.minimize(function_to_minimize, num_function_calls,
parameter_definition,
forbidden_clasuses = forbidden_confs)
```

Of course, conditionals and forbidden clauses are not mutual exclusive, but you can define both and use them while minimizing the function.

Introduced in SMAC 2.10, there is an **advanced syntax** that allows more
complex situations to be handled. It allows to compare parameter values to
each other, and apply a limit set of functions:

Arithmetic Operations +,-,*,/,^, Unary +/-, and % Functions abs, (a)cos, (a)sin, (a)tan, exp, sinh, cosh, ceil, floor, log, log2, log10, sqrt, cbrt Logical Operators >=, <=, >, <, ==, !=, ||, &&

Some examples (without the appropriate parameter definitions) to illustrate this notation:

```
forbidden_confs = ["{activity == 'swimming' && water_temperature < 15}",
"{x^2 + y^2 < 1}",
"{sqrt(a) + log10(b) >= - exp(2)}"]
```

Note

The SMAC manual has an extensive section on important tips and caveats related to forbidden parameters. Here are some major points

SMAC generates random configurations without honoring the forbidden clauses, but rejects those that violate at least one. So constraining the space too much will slow down the process.

Even meaningless forbidden clauses (those that are always false) still take time to evaluate slowing SMAC down.

Applying arithmetic operations/function to non-numerical values for categoricals/ordinals leads to undefined behavior! For

categoricaltypes only == and != should be used. You may use >=, <=, <, and > forordinalparameters.Don’t use == and != for

realtypes, as they are almost always false, or true respectively.names and values of parameters must not contain anything besides alphanumeric characters and underscores.

When defining

ordinalparameters, you have to keep the values consistent. E.g.parameter_definition = dict(\ a = ("ordinal", ["warm", "medium", "cold"], "medium"), b = ("ordinal", ["cold", "medium", "warm"], "medium"), c = ("ordinal", [100, 10, 1], 100))The problem here is that “warm < cold” for a, but “warm > cold” for b. For numerical values the definition of c implies “100<10”, which is not true.

For more details, please refer to the SMAC manual.

### Combining Parameter Configuration Spaces¶

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 | ```
import sys
sys.path.append("../../pynisher")
sys.path.append('..')
import pysmac
import sklearn.ensemble
import sklearn.neighbors
import sklearn.datasets
import sklearn.cross_validation
# We use a random classification data set generated by sklearn
# As commonly done, we use a train-test split to avoid overfitting.
X,Y = sklearn.datasets.make_classification(1000, 20)
X_train, X_test, Y_train, Y_test = \
sklearn.cross_validation.train_test_split(X,Y, test_size=0.33, random_state=1)
# training a random forest
def random_forest (n_trees=None, criterion=None, max_features=None, max_depth=None):
predictor = sklearn.ensemble.RandomForestClassifier(n_trees, criterion, max_features, max_depth)
predictor.fit(X_train, Y_train)
return (-predictor.score(X_test, Y_test))
# and defining some of its parameters
parameters_trees = dict(\
max_depth = ("integer", [1,10], 4),
max_features=("integer", [1,20], 10),
n_trees=("integer", [1,100],10 ,'log'),
criterion =("categorical", ['gini', 'entropy'], 'entropy'),
)
# training a k-nearest neighbor classifier
def knn (n_neighbors=None, weights=None):
predictor = sklearn.neighbors.KNeighborsClassifier(n_neighbors, weights)
predictor.fit(X_train, Y_train)
return (-predictor.score(X_test, Y_test))
# and defining some of its paramers
parameters_knn = dict(\
n_neighbors = ("integer", [1,100], 10, 'log'),
weights = ("categorical", ['uniform', 'distance'], 'uniform'),
)
# convenience function
from pysmac.utils.pcs_merge import merge_configuration_spaces
# returns a parameter config space, the conditionals, forbiddens and two wrapper functions
p,c,f,wrapper_str = merge_configuration_spaces(\
(random_forest, parameters_trees, [], []),
(knn, parameters_knn, [],[]))
# workaround to make the generated functions pickable (needed for pySMAC internals):
# they are generated as strings, and instantiated by executing this string
exec(wrapper_str)
# create optimizer object
opt = pysmac.SMAC_optimizer( debug = 0,
working_directory = '/tmp/pySMAC_test/',
persistent_files=True, )
# perform actual optimization
value, parameters = opt.minimize(pysmac_merged_pcs_wrapper,
#wrapper function generated by merge_configuration_spaces
50, # number of function evaluations
p, # parameter defintion
conditional_clauses = c,
forbidden_clauses = f)
print('The highest accuracy found: %f'%(-value))
print('Parameter setting %s'%parameters)
# For very complex configuration spaces, looking at the final values might be hard.
# You can use a function defined in 'wrapper_str' that finds the right callable and
# the corresponding arguments (with their original name). That way, it is more
# human readable.
func, kwargs = pysmac_merged_pcs_reduce_args(**parameters)
print("The best result was found calling '{}' with the arguments {}".format(func.__name__, kwargs))
``` |