Coordinate Descent for Linear Regression: User Guide

Introduction to Linear Regression and Coordinate Descent

Linear regression is a fundamental machine learning method where the goal is to model the relationship between the response (or dependent variable) y and one or more explanatory variables (or independent variables) X.

The performance of a linear regression model is typically assessed using the mean squared error (MSE). The Coordinate Descent algorithm is an optimization algorithm that updates one coefficient at a time, holding all others constant, and iterates this process until convergence, utilizing the gradient of the MSE with respect to each single coefficient.

This document guides you through utilizing the coordinate_descent function within the lr_cd package to implement linear regression using the Coordinate Descent algorithm. A more comprehensive tutorial on the coordinate descent algorithm is available in the review paper on Springer.

Setup

Before starting, ensure you have installed the lr_cd package and import the necessary libraries:

import lr_cd
import numpy as np

Synthetic Data Generation

The generate_data_lr function offers a convenient method for generating random data points. Initially, it creates an array of random numbers within the half-open interval [0.0, 1.0) to serve as the x-coordinates. Then, the y-coordinates are calculated using the true coefficients, which the user specifies. A random noise factor is generated from a normal distribution with a mean of zero and a variance specified by the user, which is then added to the y-coordinates.

Users have control over the number of data points, the number of features (regressors), and the true coefficients by providing values for n, n_features, and theta in the function. An array with at least 2 elements is expected for theta, with the first element always being the true intercept.

Optional inputs include noise and random_seed. Specifying a value larger than the default 0.2 increases the variation and dispersion of the data away from the true regression line. It is recommended for users to include a random_seed for reproducibility.

from lr_cd.lr_data_generation import generate_data_lr

# example 1
theta = np.array([4, 3])
X, y = generate_data_lr(n=10, n_features=1, theta=theta)
print('Generated X:')
print(X)
print('Generated y:')
print(y)

Generated X:
[[0.69646919]
 [0.28613933]
 [0.22685145]
 [0.55131477]
 [0.71946897]
 [0.42310646]
 [0.9807642 ]
 [0.68482974]
 [0.4809319 ]
 [0.39211752]]
Generated y:
[[6.34259481]
 [4.68506992]
 [4.54477713]
 [5.63500251]
 [6.45668483]
 [5.14153898]
 [6.8534962 ]
 [5.96761896]
 [5.88398172]
 [5.61370977]]

Coordinate Descent Algorithm

The coordinate_descent function, imported from the lr_cd module, implements the coordinate descent algorithm for the linear regression problem. With proper input, including the feature matrix X and response y, the function provides the estimations for a linear regression.

This function requires two parameters: the numpy arrays X and y, which can be simulated data generated by internal function or sourced externally. Additionally, there are two optional parameters: ϵ, which stops the algorithm if the change in weights is smaller than the specified value (default is 1e-6), and max_iterations, which represents the maximum number of iterations (default is 1000). The result includes the estimated intercept, coefficients, and the number of iterations when the algorithm terminates.

Working with Synthetic Dataset

In the first example, we apply this function to the synthetic data set from example 1. This function determines the best-fit intercept and coefficients. In this example, we used the underscore (_) to disregard any additional outputs that we don’t wish to store.

from lr_cd.lr_cd import coordinate_descent

# Use the synthetic data generated from example 1
intercept_1, coef_1, _ = coordinate_descent(X, y)
print(f"lr_cd Intercept for example 1: {intercept_1}")
print(f"lr_cd Coefficients for example 1: {coef_1}")

lr_cd Intercept for example 1: 4.0240072117306145
lr_cd Coefficients for example 1: [[3.10261496]]

Working with Diabetes Dataset

The diabetes dataset contains ten baseline variables—age, sex, body mass index (BMI), average blood pressure, and six blood serum measurements—taken from 442 patients. The target variable for prediction is a quantitative measure of disease progression one year after baseline. For more details, visit the Diabetes Data page.

Below we use the diabetes dataset for a more realistic example:

from sklearn import datasets, linear_model
diabetes_X, diabetes_y = datasets.load_diabetes(return_X_y=True)

# Isolate one feature for simplicity 
diabetes_X = diabetes_X[:, np.newaxis, 2]

# Split the data into training/testing sets
diabetes_X_train = diabetes_X[:-90]
diabetes_X_test = diabetes_X[-90:]
diabetes_y_train = diabetes_y[:-90]
diabetes_y_test = diabetes_y[-90:]

# Apply the Coordinate Descent algorithm
intercept_diabetes, coef_diabetes, _ = coordinate_descent(diabetes_X_train, diabetes_y_train)
print(f"lr_cd Intercept for diabetes dataset: {intercept_diabetes}")
print(f"lr_cd Coefficients for diabetes dataset: {coef_diabetes}")

lr_cd Intercept for diabetes dataset: 152.07072678879112
lr_cd Coefficients for diabetes dataset: [[957.17579359]]

Validation with Scikit-learn

To ensure the reliability of the lr_cd package’s coordinate descent algorithm, it is compared with sklearn’s established implementation LinearRegression. This comparison validates the accuracy and verifies that both yield consistent results for linear regression tasks. Benchmarking against sklearn not only confirms the algorithm’s correctness but also helps pinpoint any discrepancies or bugs.

# compare sklearn result for example 1
from sklearn.linear_model import LinearRegression
lin_reg = LinearRegression()
lin_reg.fit(X, y)

intercept_s_1, coef_s_1= lin_reg.intercept_, lin_reg.coef_
print(f"Sklearn Intercept for example 1: {intercept_s_1}")
print(f"Sklearn Coefficients for example 1: {coef_s_1}")

Sklearn Intercept for example 1: [4.02400155]
Sklearn Coefficients for example 1: [[3.10262393]]

# compare sklearn result for example 2
# Create linear regression object
regr = linear_model.LinearRegression()

# Train the model using the training sets
regr.fit(diabetes_X_train, diabetes_y_train)

# Make predictions using the testing set
diabetes_y_pred = regr.predict(diabetes_X_test)

# results
intercept_s_diabetes, coef_s_diabetes= regr.intercept_, regr.coef_
print(f"Sklearn Intercept for diabetes dataset: {intercept_s_diabetes}")
print(f"Sklearn Coefficients for diabetes dataset: {coef_s_diabetes}")

Sklearn Intercept for diabetes dataset: 152.07072678879126
Sklearn Coefficients for diabetes dataset: [957.17579359]

Visualization

The plot_lr function provides a graphical representation of linear regression, overlaying the regression line—calculated using the estimated coefficients and intercept—onto the scatter plot of the data points.

This function requires four parameters: the X and y, which can be generated internally by the package’s data generation function or sourced externally, as well as the intercept and coefficient values derived from the lr_cd algorithm. The resulting plot offers visual confirmation of how well the linear regression model fits the data.

In the example provided, the dataset used for plotting is sourced from the package’s own data_generation function (example 1).

# example 1 from data generation function
from lr_cd.lr_plotting import plot_lr
plot_1 = plot_lr(X, y, intercept_1, coef_1)

The plot_lr function is flexible, allowing for the visualization of linear regression models regardless of the algorithm used to estimate the intercept and coefficients. The key requirement is that these values are provided in the correct data type.

In the demonstration below, we employ the plot_lr function to display a linear regression model whose parameters were computed using the sklearn algorithm from example 1.

To confrom with the input specifications of plot_lr, we convert the sklearn-generated intercept from a numpy array to a float. This visualization not only confirms the successful application of the plot_lr function with parameters from an external source but also visually underscores that the lr_cd model is in alignment with the results from sklearn’s linear regression algorithm.

# example 1 from sklearn
plot_1_sk = plot_lr(X, y, intercept_s_1[0], coef_s_1)

In the second example, we took the dataset taken from the real-world scenario.

# diabetes dataset
plot_2 = plot_lr(diabetes_X_test, diabetes_y_test, intercept_s_diabetes,
                  coef_s_diabetes)

Using the matplotlib package, we can cross-validate the output of the plot_lr function by generating a comparable plot. We can confirm the accuracy of our model’s coefficients and intercept. However, our plot_lr function simplifies this process, condensing it into a single line of code for user convenience.

# Plot outputs from sklearn for diabetes dataset
import matplotlib.pyplot as plt
plt.scatter(diabetes_X_test, diabetes_y_test, color="blue")
plt.plot(diabetes_X_test, diabetes_y_pred, color="red", linewidth=2)
plt.title('Linear Regression Model Plotted by Matplotlib')
plt.xlabel('Independent variable (X)')
plt.ylabel('Dependent variable (y)')

plt.show()

Reference

• Efron, B., Hastie, T., Johnstone, I., & Tibshirani, R. (2004). Least angle regression. The Annals of Statistics, 32(2), 407-499.

• Scikit-learn Developers. (n.d.). Linear Regression Example. Scikit-learn. https://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.html

• NCSU. (n.d.). Diabetes Dataset. https://www4.stat.ncsu.edu/~boos/var.select/diabetes.html

• Wright, S. J. (2015). Coordinate descent algorithms. Mathematical Programming, 151(1), 3-34.