Consider the following problem of searching for functional dependency which we call the model of learning from examples or the supervised learning. In this model we are given a set of pairs called the training set. The elements are coming from some space which we call the input set. In what follows we will always assume that are i.i.d. and generated according to some fixed but unknown probability distribution . The elements are the elements of the set which we call the output set.
There is some dependence between the elements of and which is given by the unknown probability distribution on the product space .
The elements of the training set are obtained as follows. Given the input the operator generates an output response according to the conditional distribution function and we observe the sample of these responses: – the training set (this includes the case when the operator uses a function to generate a response). The learning machine observes this output and creates an approximation of an unknown operator. In the context of statistical machine learning by approximation we usually mean “selection of the best function from the given class of functions”.
Each time the problem of selecting a function from a given class of functions arises we will use the following model to find the best one. From the totality of all the possible functions we select the one which maximizes some quality criteria. More formally, given the set of functions we are looking for such that will minimize the value of the risk functional . In the case when the family of functions and the functional are explicitly given this is a problem of calculus of variations.
We will be mostly interested in the functionals of the special form: . The function is a so-called loss function. This functional has a very simple interpretation, the value is an amount of “loss” resulting from the realization of vector .
Our first important observation is that the integral is taken with respect to an unknown distribution which is in fact the one we are trying to estimate. Let us, instead of minimizing the functional minimize the functional .
As a result, we have the following:
Definition: The problem of statistical machine learning consists of
- The input space and the output space with an unknown probability density function ,
- Observed sample called the training set,
- The class of functions called a learning machine or a model,
- The risk functional and an empirical risk functional .
Our task is to find the value of such that empirical risk functional is minimized.
There are two important problems with this definition which we will address later in the course.
- (Model Selection) How do we select the class of functions ?
- (Generalization/Overfitting) How do we guarantee that the value of that minimizes the empirical risk functional would also minimize the risk functional?
To answer the first question there is a host of models that were brought to machine learning in the last 50 years: linear models, neural networks, support vector machines, Hopfield networks, etc are all the examples of the classes of functions that are usually being considered.
In general, criteria for the selection of the model is a combination of flexibility and good generalization ability. It is usually desirable that the function class is flexible enough to model any function on one hand but does not overfit to training data.
Take neural networks as an example. As follows from the theorem of Cybenko any function on a hypercube can be approximated by some neural network (with one hidden layer) thus making neural networks good at approximating unknown functions. On the other hand neural networks with fixed architecture have small VC dimension making them good at generalization.
While we can infer about the quality of the model using some sophisticated methods like VC-dimension, in practice, we do something very different. Instead, we divide our sample into three parts:
- Training set. Used to find the value of that minimizes the value of for a given model.
- Validation set. Used to tune the hyperparameters of the model. For example, the architecture of the model.
- Test set. Used to assess the accuracy of the model. Here, we assume that computed on the test set is a good estimate of .
We use training set to find the parameters of the model, validation set to fine-tune the parameters and architecture and also to help with the stopping rule and testing set as a final benchmark.
Let us now return back to our toy data set and specify the our classification problem as a problem of statistical machine learning, pick some simple model and see how it works out.
We will start with the definition of the loss function for the Higgs boson challenge. The background process is a decay of Z-bozons into two tau mesons and the signal process is the decay of a Higgs boson into two taus. Let us formulate the statistical hypothesis about our data:
– “background hypothesis”. All data comes from the decay of Z-bosons.
– “signal hypothsis”. Oserved data is a combination of signal and background processes.
To understand the result of the search one quantifies the agreement of an observed data with a null hypothesis by computing the p-value , i.e. the probability of observing the data given there is only background process. If this probability is very small we reject the hypothesis. The measure of incompatibility is based for example on the number of certain events found in the designated region of the search space.
p-values are usually converted into the significance levels where is standard Gaussian. We will reject a “background hypothesis” if the level of significance which corresponds to .
Let be the training set where , , and . Denote by and by the indices of the signal and background signals and let and be the numbers of simulated signal and background events. Note that since the dataset from the Higgs challenge is a simulated set the ratio will not in general be equal to the ratio . This is actually very good for us since .
The last thing that needs an explanation in the training set are the weights. We will see later that objective function depends on the unnormalized weights. In order to make it independent on the number of simulated signal and background events the sum of the weights is kept fixed, i.e. and where and are for the expected number of background and signal events in 2012.
Let be any classifier. Selection region is a set and denote by the indices of points in selected by .
We see that the number is an unbiased estimator of the expected number of signal events selected by
Similarly, the number is an unbiased estimate of the expected number of background events
We are now in the position to define the (empirical) risk functional:
called absolute median significance. The term is a regularization term used to reduce the variance of the AMS. We will discuss regulatization in the next lectures.
Finally, we are in the position to formally define the HiggsML challenge as a machine learning problem. Given the training set find an optimal value of parameters in some class of functions that would minimize the value of the empirical risk functional AMS and when tested on the testing set produce the same significance of the test AMS.
Anyone interested in the derivation of the AMS I refer to the paper [Claire Adam-Bourdarios et. al. “Learning to discover: the Higgs boson machine learning challenge”, 2014]