Linear Regression: From Theory to Practice

1. Introduction: The Supervised Prediction Problem

In previous lessons, we mastered dataset creation, exploratory data analysis (EDA), data cleaning, feature transformation, train-validation-test splitting, and preprocessing pipelines. With a pristine, leakage-free feature matrix in hand, our next objective is to train algorithms that extract generalizable patterns from data.

The foundational starting point for supervised machine learning is Linear Regression.

Suppose you are developing an intelligent appraisal engine for Byteion Real Estate. Your task is to estimate the market valuation of residential properties based on structural attributes. Consider historical sales data from recent urban transactions:

An immediate trend emerges: as square footage expands, selling price increases in a steady, linear proportion.

By capturing this relationship as a rigorous mathematical equation, our model can infer the fair market price for any unseen home—such as a 1,750 sq ft property—with empirical confidence.

2. Practical Real-World Applications

Wherever systems require continuous numeric forecasting rather than discrete category assignment, Linear Regression serves as a rapid, interpretable baseline:

3. Regression vs. Classification: Fundamental Dichotomy

Supervised learning divides into two core problem types based on the target variable's mathematical domain:

4. Simple Linear Regression: Mathematical Formulation

When estimating a continuous target ($y$) from exactly one scalar predictor ($x$), the framework is termed Simple Linear Regression.

In elementary geometry, straight lines are expressed as $y = mx + c$. In statistical machine learning, standard parameter notation is adopted:

Concrete Parameter Interpretation

Suppose our trained model yields: $\hat{y} = 0.05x + 10$ (where $x$ is square feet and $\hat{y}$ is price in Lakh BDT).

• Slope ($w = 0.05$): Each additional square foot of floor space increases the estimated price by $0.05$ Lakh BDT (BDT 5,000).
• Intercept ($b = 10$): At zero square feet, $\hat{y} = 10$ Lakh BDT. Since a zero-sq-ft home is a physical impossibility, the intercept functions as an anchor baseline to center the linear hyperplane across the observed data range.

For an apartment measuring 1,200 sq ft, the model's prediction is computed directly:

5. Residuals & Prediction Errors

No empirical dataset falls perfectly onto a 1-dimensional line. The vertical deviation between an actual observed target point and the model's predicted line is the Residual ($e_i$):

Where $y_i$ is the ground-truth target recorded in our dataset, and $\hat{y}_i = wx_i + b$ is the model's estimate.

6. Multiple Linear Regression: Multivariable Prediction

In realistic deployments, property valuation relies on multiple simultaneous features: floor area, bedroom count, age, proximity to transit, and neighborhood metrics.

When $n$ feature attributes ($x_1, x_2, \dots, x_n$) are jointly modeled, the architecture becomes Multiple Linear Regression:

Expressed in compact linear algebra notation: $\hat{y} = Xw + b$, where $X \in \mathbb{R}^{m \times n}$ and $w \in \mathbb{R}^{n \times 1}$.

7. How Linear Regression Learns: OLS Optimization

The central optimization task is identifying the unique parameter vector ($w^*, b^*$) that minimizes the total aggregate prediction error across all $n$ training records.

The standard closed-form optimization criterion is Ordinary Least Squares (OLS).

Residual Sum of Squares (RSS)

OLS formulates an explicit loss objective by summing the squared residuals of all training points:

Two Methods of Parameter Optimization

8. Regression Performance Evaluation Metrics

To validate generalization accuracy on unseen test partitions, practitioners utilize four complementary statistical metrics:

9. End-to-End Implementation with Scikit-Learn

The complete executable workflow: synthetic generation with realistic Gaussian variance, train/test partition, model fitting, parameter inspection, test set scoring, and single-sample inference:

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

# 1. Synthesize 100 realistic real estate samples with controlled noise
rng = np.random.default_rng(42)
area = rng.uniform(500, 3000, size=100)
# Ground-truth: Price = 10 + 0.05 * Area + Gaussian Noise
price = 10 + 0.05 * area + rng.normal(0, 5, size=100)

df = pd.DataFrame({"area": area, "price": price})

# 2. Separate 2D Feature Matrix (X) and 1D Target Vector (y)
X = df[["area"]]
y = df["price"]

# 3. Partition 80% Train, 20% Test
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.20, random_state=42
)

# 4. Instantiate and fit Ordinary Least Squares estimator
model = LinearRegression()
model.fit(X_train, y_train)

# 5. Inspect learned parameters
print("Learned Slope (w):", round(model.coef_[0], 4))
print("Learned Intercept (b):", round(model.intercept_, 4))

# 6. Evaluate predictions on holdout test partition
y_pred = model.predict(X_test)

mae = mean_absolute_error(y_test, y_pred)
mse = mean_squared_error(y_test, y_pred)
rmse = np.sqrt(mse)
r2 = r2_score(y_test, y_pred)

print(f"MAE:  {mae:.2f} Lakh BDT")
print(f"RMSE: {rmse:.2f} Lakh BDT")
print(f"R² Score: {r2:.4f}")

# 7. Perform inference on a newly listed 1,500 sq ft property
new_apartment = pd.DataFrame({"area": [1500]})
pred_price = model.predict(new_apartment)
print(f"Predicted Price for 1500 sqft: {pred_price[0]:.2f} Lakh BDT")

10. Five Classical Assumptions of Linear Regression

For OLS coefficient estimates to be Best Linear Unbiased Estimators (Gauss-Markov Theorem), five diagnostic assumptions must hold:

11. Limitations & Production Caveats

12. Summary & Key Takeaways