21 Machine learning
Summary
Machine learning is the term for algorithms that learn from the data how to build a model. Previously, lm and cforest were introduced for making models, but there are hundreds more such functions.
rlm creates a robust linear model that is less sensitive to outliers.
svm creates a support vector machine model.
glm creates generalized linear models. In particular, with parameter setting
family = "binomial", this can do logistic regression.
21.1 What is machine learning?
Machine learning is the area of computer science that deals with algorithms designed to learn from datasets how to accomplish various tasks.
A good machine learning algorithm will improve its results as more data is fed into the system. Often it is said that the algorithms gain experience, or learn from the data. There are still models in machine learning, but they are designed to be much more flexible than the classic linear models. Often machine learning algorithms go beyond just fitting parameters to deciding the more basic question of which factors serve as the best predictor variables in the first place.
There are many machine learning algorithms with different goals. Some of the most important are the following.
Classification Here the response is a categorical variable with a finite set of possibilities. The goal is to know which possible outcome is the best fit for our observation.
Regression (Numerical Prediction) This applies when our response is numerical. Here the goal is usually to minimize some measure of how far away the prediction is from the true answer.
Density estimation Often observations do not have factors spread over all possible values, but instead concentrate on a particular area. The goal here is to understand where the density of input values is highest, how the data is spread, and whether data is heavy or light-tailed.
Dimensionality reduction Observations are often very high dimensional. For instance, a photograph might have 16 million pixels. A 16 million dimensional model is usually too much to handle for classical methods. Instead, we look for lower dimensional behavior within the set of models. This allows us to project the data onto a much lower dimensional dataset.
There are many ways that machine learning algorithms can approach their task. The two most common are supervised and unsupervised learning.
In supervised learning there is a training set that has labeled data. For each set of possible predictors, the output is known in the training set.
This is the type of machine learning used in the Titanic survival case study.
In unsupervised learning there is no labeled dataset. The goal is to learn about the data solely from the data values themselves.
Some common methods for supervised learning include:
Decision Trees and Random Forests
Linear Regression
Logistic Regression
Boosting
Support Vector Machines
Bayesian Classifiers/Bayesian Networks
Neural Networks
Deep learning
Some common methods for unsupervised learning include:
Clustering
Anomaly detection
Topic modeling
Neural Networks
21.2 Supervised learning
Here a few of these methods are considered in more detail.
21.2.1 Decision Trees and Random Forests
In this method, the goal is to split the state space of inputs into two parts. The goal is to better predict each part separately (using a criterion such as the sum of the squared residuals) than the dataset as a whole.
In the data above, there is a clear break in the \(y\) values for \(x < 5\) and \(x \geq 5\). On either side of this split, the \(y\) values are much closer to one another.
Each node of a decision tree breaks the input space into two (or more) pieces. Each piece is then modeled independently of the others recursively.
In the toy example above, the decision tree idea works very well. Unfortunately, in real data this method can be prone to overfitting. To reduce the severity of this problem, a random forest runs the decision tree process multiple times. Each time a tree is created some random choices are made. This can include randomly deciding which predictor variables are used to split the space, or randomly resampling some of the data to build a larger dataset.
Such a randomly drawn set of trees is called a random forest. When a user then inputs a new observation and wants a prediction of the response, each tree is run separately. The final result can then be found by looking at the level most commonly seen for a categorical response, or the average of the predictions from each tree for a numerical response.
A random forest is a collection of decision trees where at each step in their formation, some random choices were made.
Some of the trees will end up overweighting certain classes and some will end up underweighting. The hope is that by including more random trees, these will effectively cancel each other out. This also goes by the name of variance reduction.
21.2.2 Linear regression
The nice thing about random forests is that we need to know very little about the structure of the data in order to make accurate predictions. If more is known about the structure of the data, then a linear regression model might be in order.
For instance, consider the following randomly generated dataset.
x <- seq(from = 0, to = 10, by = 0.1)
y2 <- 3 + x + rnorm(length(x), 0, 0.5)
tibble(x, y2) |>
ggplot(aes(x, y2)) +
geom_point()
A decision tree would have to break this down into many pieces to get an accurate read, while a simple linear model does better with only a single slope parameter and a \(y\)-intercept parameter.
Suppose we have \(n\) observations and \(p\) predictor variables. Call the column vector of the response variable values \(Y\). (This is also an \(n\) by 1 matrix.) Then form a model matrix whose \(i,(j+1)\)th entry is the value of the \(j\)th predictor variable in the \(i\)th observation, and whose \(i, 1\) entry is always 1. The model \[ Y = X\beta + \epsilon, \] where \(Y\) is an \(n\) by 1 matrix, \(X\) is an \(n\) by \(p + 1\) matrix, \(\beta\) is a \(p + 1\) by \(1\) matrix, and \(\epsilon\) is an \(n\) by 1 matrix is called a linear model of the data. The \(\beta\) values are called the parameters of the model.
One reason that linear models are widely used is that if our goal is to minimize the sum of the squares of the \(\epsilon\) values, it is possible to compute exactly \(\beta\) values for a given \(Y\) and \(X\) that accomplish this goal. (With extra conditions, this solution will be unique.) If there are many observations and predictors, these computations can still take a long time, however, it is still possible to approximate the values of \(\beta\) that give the best fit.
21.2.3 Logistic regression
Is there a way to use linear models for categorical data? Consider the best least squares fit for the following data where the response is either 0 or 1.
First consider generating some random 0-1 data. The data has a 10% chance of being 1 if x is at most 5, and a 90% chance of being 1 if x is greater than 5.
A regular linear model tries to put a straight line through this cloud of points! First create the linear model
Next, the modelr package will be used.
Now create the predictions.
df3 <- tibble(x, y3)
df3_pred <- df3 |>
add_predictions(mod1)
ggplot() +
geom_point(data = df3, aes(x, y3)) +
geom_line(data = df3_pred, aes(x, pred), color = "red", lwd = 1)
Because the data is more likely to be higher as x is larger, the best fit linear line has positive slope. However, it does not really capture the behavior of the data.
Instead of directly modeling the data using a linear model, let \(p\) be the probability that the data is 1, and \(1 - p\) be the probability that it is 0. Then the log-odds of \(p\) can be modeled using a linear function.
This function works on odds, which are another way of describing probabilities of outcomes. If something has a 30% chance of occurring, then it has a 70% chance of not occurring. Say that there are \(30\) to \(70\) odds of it occurring, or in math notation \(30:70\). Note that these odds \(30:70\) obey the same rules as the fraction \(3:7\). In fact \[ 3:7 = \frac{3}{7} = \frac{30}{70} = 30:70. \] In general, if \(p\) is the probability that an event occurs, then the odds of it occurring are \(p / (1 - p)\).
If the probability of one outcome is \(p\), and the probability of another outcome is \(1 - p\), then the logit function is the logarithm of the odds of the first outcome. That is, \[ \operatorname{logit}(p) = \log\left(\frac{p}{1 - p}\right). \]
With this definition \(\operatorname{logit}(p)\) can be any positive or negative real number. The idea of logistic regression is to model \(\operatorname{logit}(p)\) using a linear function. That is, \[ \operatorname{logit}(p) = \log\left(\frac{p}{1 - p}\right) = \beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p. \]
In R, the glm, or generalized linear models function, can be used to fit this model. The family = "binomial" option tells glm that the response variable is either 0 or 1. (The binomial generalized linear model actually is a bit more general than that, but only this functionality will be needed here.)
Then add predictions.
Take a look:
## # A tibble: 6 × 3
## x y3 pred
## <dbl> <dbl> <dbl>
## 1 0 0 0.00857
## 2 0.1 0 0.00938
## 3 0.2 0 0.0103
## 4 0.3 0 0.0112
## 5 0.4 0 0.0123
## 6 0.5 1 0.0134A bit hard to see from the first few points what the prediction looks like, so graph it.
ggplot(data = df3, aes(x)) +
geom_point(aes(y = y3)) +
geom_line(data = df4_pred,
aes(x, pred),
color = "red",
linewidth = 3)
That S-shaped curve is a special type of logistic curve and gives this regression its name.
A simple prediction method is then: if \(p \geq 0.5\) predict 1, otherwise predict 0. Note that this is very close to what the decision tree would do for this dataset. Note, however, that the true probability of a 1 in the model jumps abruptly at \(x = 5\), while the logistic regression model necessarily still fits a smooth curve.
21.2.4 Boosting
Boosting works by building an ensemble (collection) of weak learners. At each step, it examines how well the current group is working and either adds a new learner or adjusts the weights on existing ones. Common examples include Ada Boost (Adaptive Boosting), Gradient Boosting, and XGBoost (Extreme Gradient Boosting).
21.2.5 Support Vector Machines
A support vector machine classifies by trying to split the data into two groups using a hyperplane. In some cases, this is very easy, and the hyperplane can be used directly.
In other cases, the groups are separated by a curve. In order to deal with data like this, we need to develop a feature, a function of the predictors that gives a new predictor. Then we apply the hyperplane to this new feature. The hyperplane in the higher dimensional space might look like a curve in the lower dimensional space.
A common example of this is data in the \((x, y)\) plane that exhibits radial symmetry. A feature such as \(r = \sqrt{x^2 + y^2}\) makes it possible to separate the data based on (for instance) if \(r < 7\) or \(r \geq 7\). In the original plane, this boundary \(r = 7\) is a circle, but it is a hyperplane in \((x, y, r)\).
A feature is a new predictor whose value is a function of other predictors.
21.2.6 Bayesian Classifiers/Bayesian Networks
Suppose that given which class an observation is in, the probability of certain predictor values appearing is known. Then Bayes’ Rule allows us to reverse the calculation and use predictor values to find the chance that a data point falls in a particular class.
These methods are in some sense the gold standard of classification because they are principled updates of the information about the class of the points given the prior and model. However, applying Bayes’ Rule to complex models is computationally very expensive.
The computations can be simplified by making assumptions about the model. The most basic assumption is that the predictors are conditionally independent given the class of the observation. That is, once we know which class we are in, all the predictors are independent of each other! This is often called the naive Bayes classifier. Although this is a very powerful assumption, it can actually give models that are very useful for making predictions.
21.2.7 Neural Networks
A neural network tries to understand how the input and output variables of a model connect through a graph. In mathematics, a graph consists of nodes (also called vertices) that are connected by edges. Inputs to nodes become outputs to other nodes through the weights on the edges.
This process was inspired by biological neurons, which fire to other neurons when they are excited. Although mathematical neural networks are somewhat different from this process, the name has stuck, which is why they are called neural networks.
By using the data in observations, the weights on the edges are fine-tuned to make the final output of the network close to the observed response.
21.2.8 Deep Learning
Neural networks turned out to be difficult to use in complex situations. Consider a first neural network (called a layer) that attempts to capture simple patterns in the data. Then another layer might refine this to more closely approach the actual data. By adding several layers of neural networks, the resulting deep learning neural network can be very close to the observed output.
This idea has proved especially effective in areas such as computer vision, speech recognition, and natural language processing where the data naturally lends itself to large groups, then smaller more refined groups.
21.3 Unsupervised learning
In unsupervised learning, we are not given any labeled data. It is up to the algorithm to discover structure or patterns within the observed data without any known labels on the response variable.
21.3.1 Clustering
An example of this type of learning is cluster analysis or more simply, clustering. Here the goal is to determine which observations in the sample space are close together to one another.
As with most models, there is no one right clustering. Instead, different clustering algorithms will achieve different results. In the end, the question is whether or not the clustering is useful for the purposes that the person analyzing the data is trying to achieve.
For example, suppose that each observation is a numerical \(n\)-tuple. Then a simple way to cluster is to calculate the distance between each pair of points. Setting a threshold value, connect any pairs of points whose distance is below that threshold. Then points that be reached from each other via these connections are called connected components or clusters.
When the threshold is 0 and the point values are distinct, no points are connected. When the threshold is the maximum distance between any two points, all the nodes are connected. So as the threshold rises from 0 up to the maximum distance, the number of clusters is monotone decreasing.
Another type of clustering is based upon kernel density estimation. Here a normal density is placed on top of each point in the dataset. For a given point, sum all of these densities together. For instance, suppose we have some \(x\) values drawn using a normal distribution centered at 3, and others drawn using a normal distribution centered at 8. This gives bimodal data. Then the kernel density plot might look as follows.
This kernel density plot has two local maxima, which suggests that the data falls into two clusters. More generally this idea is known as density-based clustering.