Random Forest (Regression Algorithm)

Random Forest is a machine learning algorithm that creates many decision trees with various subsets data. Each tree can have a different root node. The method uses these trees to make predictions by randomly looking at different parts of the data. For classification tasks, the final answer is chosen based on what most trees agree on. For regression tasks, the result is the average of all the trees' predictions. This method is called ensemble learning, which helps make predictions more accurate and reduces mistakes. The way the trees are built, including their root and decision points, makes Random Forest powerful and less likely to learn from the training data too closely.

Minimizing the Error: Mean Absolute Error
The mean absolute error calculates the average discrepancy between a dataset's actual and projected values. Without taking direction into account, it displays the deviation between predicted and actual values.


Minimizing the Error: Mean Squared Error (MSE)
A key idea in statistics and machine learning, mean squared error (MSE) is essential for evaluating the precision of predictive models. The model's accuracy can be examined using the MSE value.It calculates the average squared difference between the dataset's actual and projected values. It is computed by averaging the squared residuals, where the residual is the difference between each data point's actual value and its anticipated value.

Machine Learning • Worked Example

Random Forest Regression

A small, hand-calculable house price prediction example using 10 houses, 3 independent variables, and 3 decision trees.

1. The setup

We want to predict a house's price using three independent variables:

10Houses

3Independent variables

3Decision trees

Features: Size, Bedrooms, and Age.

Target: Price, measured in thousands of dollars ($000s).

House	Size (100s sq ft)	Bedrooms	Age (years)	Price ($000s)
A	10	2	30	180
B	12	3	25	220
C	15	3	20	260
D	18	4	15	310
E	20	4	10	350
F	22	5	8	390
G	25	5	5	430
H	28	4	3	470
I	30	5	2	510
J	35	6	1	580

Example: Size = 20 means 2,000 square feet. Price = 350 means $350,000.

2. What does a Random Forest do?

1. Create several different training samples using bootstrap sampling.
2. Build a decision tree on each sample.
3. Let every tree make a prediction.
4. For regression, average the tree predictions.

Forest prediction = (Tree 1 + Tree 2 + Tree 3) / 3

3. The key calculation: Mean Squared Error (MSE)

For regression, a decision tree commonly uses MSE to judge how good a split is.

Suppose a node contains prices:

180, 220, 260

First calculate the mean:

Ȳ = (180 + 220 + 260) / 3 = 220

Then calculate squared deviations:

(180 − 220)² = 1,600 (220 − 220)² = 0 (260 − 220)² = 1,600

Therefore:

MSE = (1,600 + 0 + 1,600) / 3 = 1,066.67

Lower MSE means a more homogeneous node. The tree searches for splits that reduce the weighted MSE.

4. Tree 1 — working through a split

Suppose Tree 1 receives this bootstrap sample:

A, B, C, D, E, E, G, H, I, J

House E appears twice. That is normal in bootstrap sampling.

Candidate split: Size < 20

Left node: A, B, C → prices 180, 220, 260

Mean = 220 MSE = 1,066.67

Right node: D, E, E, G, H, I, J → prices 310, 350, 350, 430, 470, 510, 580

Mean = 428.57 MSE ≈ 7,677.55

Weighted MSE:

MSEsplit = (3/10)(1,066.67) + (7/10)(7,677.55) = 320.00 + 5,374.29 = 5,694.29

Try another split: Size < 25

Left: A, B, C, D, E, E

Mean = 278.33    MSE ≈ 3,647.22

Right: G, H, I, J

Mean = 497.50    MSE = 3,337.50

Weighted MSE:

MSEsplit = (6/10)(3,647.22) + (4/10)(3,337.50) = 2,188.33 + 1,335.00 = 3,523.33

Because 3,523.33 < 5,694.29, this split is better than the first candidate.

5. How a regression tree makes predictions

The tree continues splitting until it reaches terminal leaves. A regression leaf predicts the average target value of the observations inside it.

Size < 25? / \ Yes No / \ Size/other Size < 30? / \ G,H I,J | | $450 $545

If the new house follows the rightmost path, Tree 1 predicts:

Tree 1 prediction = (510 + 580) / 2 = 545 ($000s)

So:

$545,000

Tree 1 prediction

6. Tree 2

Tree 2 gets a different bootstrap sample:

A, A, C, D, F, F, G, H, J, J

Suppose our new house ends in a leaf containing F, F, and G:

Prices = 390, 390, 430 Tree 2 prediction = (390 + 390 + 430) / 3 = 403.33 ($000s)

$403,330

Tree 2 prediction

7. Tree 3

Tree 3 gets yet another bootstrap sample:

B, C, C, D, E, G, H, I, I, J

Suppose our new house ends in a leaf containing H, I, and I:

Prices = 470, 510, 510 Tree 3 prediction = (470 + 510 + 510) / 3 = 496.67 ($000s)

$496,670

Tree 3 prediction

8. The Random Forest prediction

Tree 1

$545,000

Tree 2

$403,330

Tree 3

$496,670

For regression, the Random Forest averages the predictions:

ŷRF = (545 + 403.33 + 496.67) / 3 = 1,445 / 3 = 481.67 ($000s)

🏠 Predicted house price

$481,670

approximately

9. Where do the 3 independent variables matter?

Our model has:

X₁Size

X₂Bedrooms

X₃Age

At each split, the tree can consider candidate splits involving these variables. For example:

Root | ┌─────────┼─────────┐ Size Bedrooms Age ↓ ↓ ↓ candidate candidate candidate splits splits splits ↓ choose split with lowest weighted MSE

If Size produces the largest reduction in MSE, the tree may choose Size for that split. Later splits can use Bedrooms or Age.

10. MSE reduction

Suppose the parent node has:

MSEparent = 8,000

and a candidate split produces:

MSEsplit = 3,500

Then:

MSE reduction = 8,000 − 3,500 = 4,500

A good split has a large reduction in error.

11. Why is it called “Random” Forest?

Randomness #1 — Bootstrap samples

Tree 1: A B C D E E G H I J Tree 2: A A C D F F G H J J Tree 3: B C C D E G H I I J

Randomness #2 — Random feature selection

When considering a split, Random Forest can randomly select a subset of the available features.

One split might consider Size + Age, while another might consider Bedrooms + Age. This helps make the trees less correlated.

12. Complete miniature Random Forest

Stage	Calculation	Result
Tree 1	Leaf mean	$545,000
Tree 2	Leaf mean	$403,330
Tree 3	Leaf mean	$496,670
Forest	(545 + 403.33 + 496.67) / 3	$481,670

$481,670

Final Random Forest house-price prediction

Important note

This is a deliberately small, hand-calculable demonstration. The tree structures and leaf assignments are simplified so that the arithmetic can be followed manually. A real Random Forest implementation would evaluate many candidate thresholds across the three variables, recursively select the best splits, and typically grow many more trees.

USE CASE 1: Use Random Forest with scikit-learn, predict the product price. The Production cost, Advertising spend, and Demand level are the independent variables.

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error, r2_score
# -----------------------------------
# 1. Load data from Excel
# -----------------------------------
data = pd.read_excel("product_data.xlsx")

print("Dataset Preview:")
print(data.head())

# -----------------------------------
# 2. Define features and target
# -----------------------------------
X = data[['Production_Cost', 'Advertising_Spend', 'Demand_Level']]
y = data['Product_Price']

# -----------------------------------
# 3. Split into training and testing
# -----------------------------------
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42
)

## What is random_state?

#train_test_split randomly shuffles the dataset before splitting.
#Without random_state:
#Each run → different split
#Model performance changes slightly
#With random_state=42:
#Same rows go to train/test every time
#Results are reproducible

# -----------------------------------
# 4. Train the Random Forest model
# -----------------------------------
model = RandomForestRegressor(
    n_estimators=200,
    max_depth=6,
    random_state=42
)

model.fit(X_train, y_train)

# -----------------------------------
# 5. Make predictions & evaluate
# -----------------------------------
y_pred = model.predict(X_test)

print("MAE:", mean_absolute_error(y_test, y_pred))
print("R² score:", r2_score(y_test, y_pred))


# -----------------------------------
# 6. Predict price for a new product
# -----------------------------------
# New product: Cost=32, Advertising=160, Demand=62
new_product = np.array([[32, 160, 62]])

predicted_price = model.predict(new_product)
print("Predicted product price:", predicted_price[0])

Why Random Forest works well here

1. It captures non-linear relationships very well.
2. It handles feature interactions automatically.
3. It is more robust than a single decision tree.
4. It reduces overfitting by averaging.

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error, r2_score

# -----------------------------------
# 1. Load data from Excel
# -----------------------------------
#sample data can be exported to 
#excel from the URL
# https://pythonPlaza.com/linear_school_grade_data.html

data = pd.read_excel("student_data.xlsx")

print("Dataset Preview:")
print(data.head())

# -----------------------------------
# 2. Define features and target
# -----------------------------------
X = data[['Hours_Studied', 'Attendance_%', 'Previous_Score']]
y = data['Final_Grade']

# -----------------------------------
# 3. Split into training and testing
# -----------------------------------
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42
)

# -----------------------------------
# 4. Train the Random Forest model
# -----------------------------------
model = RandomForestRegressor(
    n_estimators=200,
    max_depth=6,
    random_state=42
)


## What is random_state?

#train_test_split randomly shuffles the dataset before splitting.
#Without random_state:
#Each run → different split
#Model performance changes slightly
#With random_state=42:
#Same rows go to train/test every time
#Results are reproducible

model.fit(X_train, y_train)

# -----------------------------------
# 5. Make predictions & evaluate
# -----------------------------------
y_pred = model.predict(X_test)

print("MAE:", mean_absolute_error(y_test, y_pred))
print("R² score:", r2_score(y_test, y_pred))

Example: Predict a new student’s grade
# New student: [hours_studied, attendance %, previous_score]
new_student = np.array([[6, 85, 78]])

predicted_grade = model.predict(new_student)
print("Predicted final grade:", predicted_grade[0])

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error, r2_score

# -----------------------------------
# 1. Load data from Excel
# -----------------------------------
#sample data can be exported to 
#excel from the URL

  
  Get the Profit Optimization data in Excel
  

data = pd.read_excel("profit_optimization.xlsx")

print("Dataset Preview:")
print(data.head())

# -----------------------------------
# 2. Define features and target  Price (P)	
# -----------------------------------
X = data[['Price', 'Advertising', 'Units_Sold']]
y = data['Profit']

# -----------------------------------
# 3. Split into training and testing
# -----------------------------------
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42
)

# -----------------------------------
# 4. Train the Random Forest model
# -----------------------------------
model = RandomForestRegressor(
    n_estimators=200,
    max_depth=6,
    random_state=42
)


## What is random_state?

#train_test_split randomly shuffles the dataset before splitting.
#Without random_state:
#Each run → different split
#Model performance changes slightly
#With random_state=42:
#Same rows go to train/test every time
#Results are reproducible

model.fit(X_train, y_train)

# -----------------------------------
# 5. Make predictions & evaluate
# -----------------------------------
y_pred = model.predict(X_test)

print("MAE:", mean_absolute_error(y_test, y_pred))
print("R² score:", r2_score(y_test, y_pred))

#Predict profit for a new business strategy
# Example: Price = 15, Advertising = 165, Units Sold = 460
new_strategy = np.array([[15, 165, 460]])

predicted_profit = model.predict(new_strategy)
print("Predicted profit:", predicted_profit[0])

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error, r2_score

# -----------------------------------
# 1. Load data from Excel
# -----------------------------------
#sample data can be exported to 
#excel from the URL

  
  Get the Patient Response Data in Excel
  

data = pd.read_excel("patient_dosage_response.xlsx")

print("Dataset Preview:")
print(data.head())

# -----------------------------------
# 2. Define features and target  Price (P)	
# -----------------------------------
X = data[['Dosage', 'Age', 'Weight']]
y = data['Patient_Response']

# -----------------------------------
# 3. Split into training and testing
# -----------------------------------
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42
)

# -----------------------------------
# 4. Train the Random Forest model
# -----------------------------------
model = RandomForestRegressor(
    n_estimators=200,
    max_depth=6,
    random_state=42
)


## What is random_state?

#train_test_split randomly shuffles the dataset before splitting.
#Without random_state:
#Each run → different split
#Model performance changes slightly
#With random_state=42:
#Same rows go to train/test every time
#Results are reproducible

model.fit(X_train, y_train)

# -----------------------------------
# 5. Make predictions & evaluate
# -----------------------------------
y_pred = model.predict(X_test)

print("MAE:", mean_absolute_error(y_test, y_pred))
print("R² score:", r2_score(y_test, y_pred))


Predict response for a new patient
# New patient: Dosage=72mg, Age=36yrs, Weight=172lbs
new_patient = np.array([[72, 36, 172]])

predicted_response = model.predict(new_patient)
print("Predicted patient response:", predicted_response[0])