PythonPlaza - Python & AI
< >

Supervised Machine Learning Algorithms

Gradient Boosting


A machine learning method called Gradient Boosting creates an ensemble by combining several weak prediction models. Decision trees, which are sequentially trained to reduce errors and increase accuracy, are commonly used as these weak models. Gradient boosting can efficiently capture intricate correlations between features by combining several decision tree regressors or decision tree classifiers.

Gradient boosting's capacity to iteratively minimize the loss function is one of its main advantages. One loss function used to assess how well a machine learning model matches actual data is Mean Squared Error (MSE). MSE determines the mean of the squared discrepancies between the observed and expected values.


MAE (Mean Absolute Error) quantifies the average magnitude of errors for Gradient Boosting Regression.

Mean Absolute Error
It calculates the average discrepancy between a dataset's actual and forecasted values. Without taking direction into account, it displays the deviation between predicted and actual values.
1. Determined by utilizing absolute differences
2. Easy to calculate and understand
3. Handles every mistake equally
4. Not as susceptible to significant errors as MSE
5. Frequently employed to assess regression models


When the model stops producing new trees, it uses three primary mechanisms:

1. Early Stopping (The Optimal Method)

A different validation dataset that the model does not learn from is given to it.
After each new tree, the model verifies the validation MSE.
The model terminates entirely if the validation MSE stops decreasing for a predetermined number of trees, such as ten consecutive trees.
This saves time and avoids overfitting. [1]

2. The quantity of estimators (n_estimators)

A set limit on the total number of trees is hardcoded (e.g., n_estimators = 100). Regardless of whether the MSE is still declining or has already leveled out, the model will construct precisely 100 trees before stopping.

3. Tolerance

A minimal improvement threshold, such as tol = 0.0001, is established. The model determines that additional progress is insignificant and ceases building trees if a new tree lowers the MSE by less than this minuscule amount.




Gradient Boosting Regression

Let's see how Gradient Boosting works for regression problems. (Complete Home Price Prediction Calculation)

10 observations • 3 independent variables • 3 decision trees

1. Problem Definition

```

We want to predict a home's price using three independent variables.

X₁ = House Size
Measured in square feet
X₂ = Bedrooms
Number of bedrooms
X₃ = House Age
Age in years
Y = Home Price
Measured in $000s
Example: If Y = 360, this means a home price of $360,000.
```

2. Dataset

```
House X₁ Size (sq ft) X₂ Bedrooms X₃ Age Y Price ($000s)
11000230180
21200220210
31500315260
41800310320
5200048360
6220045400
7250043450
8280052510
9300051550
10320050590
```

3. Gradient Boosting Concept

```

Gradient Boosting builds decision trees sequentially. Each new tree tries to correct the errors made by the previous model.

Original Data
↓
Calculate Initial Prediction F₀
↓
Calculate Residuals
Residual = Actual − Prediction
↓
Build Tree 1
↓
Update Prediction
↓
Calculate New Residuals
↓
Build Tree 2
↓
Calculate New Residuals Again
↓
Build Tree 3
↓
Final Prediction
```

4. Step 1 — Calculate F₀

```

For squared-error regression, the initial prediction is the mean of all target values.

F₀ = ΣY / n

Our prices are:

180 + 210 + 260 + 320 + 360 + 400 + 450 + 510 + 550 + 590

Sum = 3830
F₀ = 3830 / 10

Initial Prediction

F₀ = 383

$383,000

Important: F₀ is NOT 0.5.

F₀ = 383 is the initial prediction.
η = 0.5 is the learning rate.
```

5. Initial Predictions

```

Because F₀ = 383, every house initially receives the same prediction.

House Actual Y Initial Prediction F₀
1180383
2210383
3260383
4320383
5360383
6400383
7450383
8510383
9550383
10590383
```

6. Step 2 — Calculate Initial Residuals

```

The residual tells us how far our prediction is from the actual value.

Residual = Actual − Prediction

Example: House 1

Residual₁ = 180 − 383
Residual₁ = −203

The model predicted $203,000 too high.

Example: House 10

Residual₁₀ = 590 − 383
Residual₁₀ = +207

The model predicted $207,000 too low.

House Actual F₀ Residual
1180383−203
2210383−173
3260383−123
4320383−63
5360383−23
6400383+17
7450383+67
8510383+127
9550383+167
10590383+207
```

7. Step 3 — Build Tree 1

```

Tree 1 uses the house-size variable X₁.

X₁ = House Size in square feet

For this teaching example, we use the split:

Tree 1 Split

X₁ ≤ 2100?

This means: Is the house size less than or equal to 2,100 square feet?

House Size X₁
↓
X₁ ≤ 2100?
YES
Houses 1–5
Tree Prediction = −117
NO
Houses 6–10
Tree Prediction = +117
```

8. Tree 1 — Calculate Left Leaf

```

Houses 1–5 have residuals:

−203, −173, −123, −63, −23

A regression tree predicts the average residual in the leaf.

Leaf₁ = (−203 − 173 − 123 − 63 − 23) / 5

Leaf₁ = −585 / 5

Leaf₁ = −117
Tree 1 Left Leaf = −117
```

9. Tree 1 — Calculate Right Leaf

```

Houses 6–10 have residuals:

17, 67, 127, 167, 207
Leaf₂ = (17 + 67 + 127 + 167 + 207) / 5

Leaf₂ = 585 / 5

Leaf₂ = 117
Tree 1 Right Leaf = +117
```

10. Apply Learning Rate to Tree 1

```

We use a learning rate:

η = 0.5

The learning rate controls how much of the tree's correction is added to the model.

F₁ = F₀ + ηT₁

Houses 1–5

F₁ = 383 + (0.5)(−117)
F₁ = 383 − 58.5
F₁ = 324.5

Houses 6–10

F₁ = 383 + (0.5)(117)
F₁ = 383 + 58.5
F₁ = 441.5
```

11. Predictions After Tree 1

```
House Actual F₀ Tree 1 0.5 × Tree 1 F₁
1180383−117−58.5324.5
2210383−117−58.5324.5
3260383−117−58.5324.5
4320383−117−58.5324.5
5360383−117−58.5324.5
640038311758.5441.5
745038311758.5441.5
851038311758.5441.5
955038311758.5441.5
1059038311758.5441.5
```

12. Step 4 — Calculate New Residuals

```

We now calculate errors using the new predictions.

New Residual = Actual − F₁
House Actual F₁ New Residual
1180324.5−144.5
2210324.5−114.5
3260324.5−64.5
4320324.5−4.5
5360324.5+35.5
6400441.5−41.5
7450441.5+8.5
8510441.5+68.5
9550441.5+108.5
10590441.5+148.5
Very important: Tree 2 learns these NEW residuals. It does not go back and use the original residuals.
```

13. Step 5 — Build Tree 2

```

Tree 2 uses:

X₂ = Number of Bedrooms

We use the split:

Tree 2 Split

X₂ ≤ 3?
Bedrooms X₂
↓
X₂ ≤ 3?
YES
Houses 1–4
Prediction = −82
NO
Houses 5–10
Prediction = +54.667
```

14. Tree 2 — Calculate Leaves

```

Left Leaf — Houses 1–4

(−144.5 − 114.5 − 64.5 − 4.5) / 4

= −328 / 4

= −82

Right Leaf — Houses 5–10

(35.5 − 41.5 + 8.5 + 68.5 + 108.5 + 148.5) / 6

= 328 / 6

= 54.6667
```

15. Apply Tree 2

```
F₂ = F₁ + ηT₂

Houses 1–4

F₂ = 324.5 + (0.5)(−82)
F₂ = 324.5 − 41
F₂ = 283.5

Houses 5–10

F₂ = 441.5 + (0.5)(54.6667)
F₂ = 441.5 + 27.33335
F₂ ≈ 468.833
```

16. Calculate New Residuals After Tree 2

```
Residual = Actual − F₂
House Actual F₂ Residual
1180283.5−103.5
2210283.5−73.5
3260283.5−23.5
4320283.5+36.5
5360468.833−108.833
6400468.833−68.833
7450468.833−18.833
8510468.833+41.167
9550468.833+81.167
10590468.833+121.167
```

17. Step 6 — Build Tree 3

```

Tree 3 uses the third independent variable:

X₃ = House Age

We use:

Tree 3 Split

X₃ > 7?
House Age X₃
↓
X₃ > 7?
YES
Houses 1–4
Prediction = −41
NO
Houses 5–10
Prediction = +7.833
```

18. Tree 3 — Calculate Leaves

```

Older Houses — Houses 1–4

(−103.5 − 73.5 − 23.5 + 36.5) / 4

= −164 / 4

= −41

Newer Houses — Houses 5–10

(−108.833 − 68.833 − 18.833 \+ 41.167 + 81.167 + 121.167) / 6

= 47 / 6

= 7.8333
```

19. Apply Tree 3

```
F₃ = F₂ + ηT₃

Houses 1–4

F₃ = 283.5 + (0.5)(−41)
F₃ = 283.5 − 20.5
F₃ = 263

Houses 5–10

F₃ = 468.833 + (0.5)(7.833)
F₃ ≈ 472.75
```

20. Final Predictions

```
House Actual F₀ Tree 1 Tree 2 Tree 3 Final F₃
1180383−117−82−41263.00
2210383−117−82−41263.00
3260383−117−82−41263.00
4320383−117−82−41263.00
536038311754.6677.833472.75
640038311754.6677.833472.75
745038311754.6677.833472.75
851038311754.6677.833472.75
955038311754.6677.833472.75
1059038311754.6677.833472.75
```

21. Complete Prediction Example — House 8

```

House 8 has:

X₁ = 2800 sq ft
X₂ = 5 bedrooms
X₃ = 2 years old
Actual Y = 510 ($000s)

Tree 1

Since:

2800 > 2100

Tree 1 gives:

T₁ = 117
Contribution = 0.5 × 117 = 58.5

Tree 2

5 > 3

Therefore:

T₂ = 54.667
Contribution = 0.5 × 54.667
= 27.3335

Tree 3

2 ≤ 7

Therefore:

T₃ = 7.833
Contribution = 0.5 × 7.833
= 3.9165

Final Prediction

F₃ = F₀ + 0.5T₁ + 0.5T₂ + 0.5T₃

F₃ = 383 + 58.5 + 27.3335 + 3.9165

F₃ = 472.75

House 8 Predicted Price

$472,750

Actual Price = $510,000

Error = $37,250

```

22. Final Errors

```
House Actual Prediction Error
1180263.00−83.00
2210263.00−53.00
3260263.00−3.00
4320263.00+57.00
5360472.75−112.75
6400472.75−72.75
7450472.75−22.75
8510472.75+37.25
9550472.75+77.25
10590472.75+117.25
```

23. Calculate MSE

```

Mean Squared Error measures the average squared prediction error.

MSE = Σ(Y − Ŷ)² / n

Squared Errors

(−83)² = 6889
(−53)² = 2809
(−3)² = 9
(57)² = 3249
(−112.75)² = 12712.5625
(−72.75)² = 5292.5625
(−22.75)² = 517.5625
(37.25)² = 1387.5625
(77.25)² = 5967.5625
(117.25)² = 13747.5625

Sum of squared errors:

SSE = 52,581.375
MSE = 52,581.375 / 10

Mean Squared Error

MSE = 5,258.1375
```

24. Calculate RMSE

```
RMSE = √MSE
RMSE = √5258.1375

Root Mean Squared Error

RMSE ≈ 72.51

Approximately $72,510

Because the target variable was measured in $000s, RMSE = 72.51 means approximately $72,510.
```

25. Complete Gradient Boosting Formula

```
F₃(x) = F₀(x) + ηT₁(x) + ηT₂(x) + ηT₃(x)
With:
F₀ = 383
η = 0.5
T₁ = Tree 1
T₂ = Tree 2
T₃ = Tree 3
Therefore:
F₃(x) = 383 + 0.5T₁(x) + 0.5T₂(x) + 0.5T₃(x)
```

26. The Entire Algorithm in One Diagram

```
HOME PRICE DATA
X₁ = Size
X₂ = Bedrooms
X₃ = Age
Y = Price
↓
Calculate Mean
F₀ = 383
↓
Calculate Residuals
r = Y − F₀
↓
TREE 1
X₁ ≤ 2100?
−117 / +117
↓
Apply Learning Rate
η = 0.5
↓
F₁ = F₀ + 0.5T₁
↓
Calculate NEW Residuals
↓
TREE 2
X₂ ≤ 3?
−82 / +54.667
↓
F₂ = F₁ + 0.5T₂
↓
Calculate NEW Residuals
↓
TREE 3
X₃ > 7?
−41 / +7.833
↓
F₃ = F₂ + 0.5T₃
↓
FINAL PREDICTION
```

27. Most Important Things to Remember

```

1. Initial prediction

F₀ = Mean(Y)

In our example: F₀ = 383

2. Residual

Residual = Actual − Prediction

3. Tree prediction

Tree Leaf = Mean of Residuals in the Leaf

4. Learning rate

η = 0.5

5. Update prediction

Fᵢ = Fᵢ₋₁ + ηTᵢ

6. MSE

MSE = Σ(Y − Ŷ)² / n

7. RMSE

RMSE = √MSE
```

28. The Big Picture

```

Gradient Boosting = Sequential Error Correction

Start with a simple prediction.

↓

Find the errors.

↓

Build a tree to correct those errors.

↓

Apply only part of the correction using the learning rate.

↓

Find the remaining errors.

↓

Build another tree.

↓

Continue until the model has enough trees.

The key idea:

Tree 2 does NOT learn the original errors. It learns the errors remaining after Tree 1.

Tree 3 does NOT learn the original errors either. It learns the errors remaining after Tree 1 and Tree 2.

This sequential correction is the core idea behind Gradient Boosting.



USE CASE 1: Use Gradient Boosting with scikit-learn, predict the product price. The Production cost, Advertising spend, and Demand level are the independent variables. 



import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import mean_absolute_error, r2_score

# -----------------------------------
# 1. Load data from Excel
# -----------------------------------
data = pd.read_excel("product_data.xlsx")

print("Dataset Preview:")
print(data.head())

# -----------------------------------
# 2. Define features and target
# -----------------------------------
X = data[['Production_Cost', 'Advertising_Spend', 'Demand_Level']]
y = data['Product_Price']

# -----------------------------------
# 3. Split into training and testing
# -----------------------------------
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42
)

## What is random_state?

#train_test_split randomly shuffles the dataset before splitting.
#Without random_state:
#Each run → different split
#Model performance changes slightly
#With random_state=42:
#Same rows go to train/test every time
#Results are reproducible


# -----------------------------------
# 4: Train the Gradient Boosting model
# -----------------------------------

model = GradientBoostingRegressor(
    n_estimators=200,
    learning_rate=0.05,
    max_depth=3,
    random_state=42
)

model.fit(X_train, y_train)


# -----------------------------------
# 5: Make predictions & evaluate
# -----------------------------------
y_pred = model.predict(X_test)

print("MAE:", mean_absolute_error(y_test, y_pred))
print("R² score:", r2_score(y_test, y_pred))

# -----------------------------------
# 6. Predict price for a new product
# -----------------------------------
new_product = [[65, 18, 275]]  # Production Cost, Advertising Spend, Demand Level
predicted_price = model.predict(new_product)

print("Predicted Product Price:", predicted_price[0])

USE CASE 2: Use Gradient Boosting with scikit-learn to predict the Student Grade. The 'Hours_Studied, 'Attendance_%', 'Previous_Score' are the independent variables. 







import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import mean_absolute_error, r2_score
# -----------------------------------
# 1. Load data from Excel
# -----------------------------------
#sample data can be exported to 
#excel from the URL
# https://pythonPlaza.com/linear_school_grade_data.html

data = pd.read_excel("student_data.xlsx")

print("Dataset Preview:")
print(data.head())

# -----------------------------------
# 2. Define features and target
# -----------------------------------
X = data[['Hours_Studied', 'Attendance_%', 'Previous_Score']]
y = data['Final_Grade']

# -----------------------------------
# 3. Split into training and testing
# -----------------------------------
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42
)

## What is random_state?

#train_test_split randomly shuffles the dataset before splitting.
#Without random_state:
#Each run → different split
#Model performance changes slightly
#With random_state=42:
#Same rows go to train/test every time
#Results are reproducible


# -----------------------------------
# 4: Train the Gradient Boosting model
# -----------------------------------

model = GradientBoostingRegressor(
    n_estimators=200,
    learning_rate=0.05,
    max_depth=3,
    random_state=42
)

model.fit(X_train, y_train)


# -----------------------------------
# 5: Make predictions & evaluate
# -----------------------------------
y_pred = model.predict(X_test)

print("MAE:", mean_absolute_error(y_test, y_pred))
print("R² score:", r2_score(y_test, y_pred))

Example: Predict a new student’s grade
# New student: [hours_studied, attendance %, previous_score]
new_student = np.array([[6, 85, 78]])

predicted_grade = model.predict(new_student)
print("Predicted final grade:", predicted_grade[0])

USE CASE 3: Use Gradient Boosting with scikit-learn to predict the Profit Optimization. The Price (P), Advertising (A), Units Sold (Q) are the independent variables, and Profit is the dependent variable. 







import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import mean_absolute_error, r2_score

# -----------------------------------
# 1. Load data from Excel
# -----------------------------------
#sample data can be exported to 
#excel from the URL

  
  Get the Profit Optimization data in Excel
  

data = pd.read_excel("profit_optimization.xlsx")

print("Dataset Preview:")
print(data.head())

# -----------------------------------
# 2. Define features and target  Price (P)	
# -----------------------------------
X = data[['Price', 'Advertising', 'Units_Sold']]
y = data['Profit']

# -----------------------------------
# 3. Split into training and testing
# -----------------------------------
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42
)

## What is random_state?

#train_test_split randomly shuffles the dataset before splitting.
#Without random_state:
#Each run → different split
#Model performance changes slightly
#With random_state=42:
#Same rows go to train/test every time
#Results are reproducible


# -----------------------------------
# 4: Train the Gradient Boosting model
# -----------------------------------

model = GradientBoostingRegressor(
    n_estimators=200,
    learning_rate=0.05,
    max_depth=3,
    random_state=42
)

model.fit(X_train, y_train)


# -----------------------------------
# 5: Make predictions & evaluate
# -----------------------------------
y_pred = model.predict(X_test)

print("MAE:", mean_absolute_error(y_test, y_pred))
print("R² score:", r2_score(y_test, y_pred))


#Predict profit for a new business strategy
# Example: Price = 15, Advertising = 165, Units Sold = 460
new_strategy = np.array([[15, 165, 460]])

predicted_profit = model.predict(new_strategy)
print("Predicted profit:", predicted_profit[0])

USE CASE 4: Use Gradient Boosting with scikit-learn to predict the Patient Response. The Dosage (mg), Age (yrs), Weight (lbs) are the independent variables, and Patient Response is the dependent variable. 







import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import mean_absolute_error, r2_score

# -----------------------------------
# 1. Load data from Excel
# -----------------------------------
#sample data can be exported to 
#excel from the URL

  
  Get the Patient Response Data in Excel
  

data = pd.read_excel("patient_dosage_response.xlsx")

print("Dataset Preview:")
print(data.head())

# -----------------------------------
# 2. Define features and target  Price (P)	
# -----------------------------------
X = data[['Dosage', 'Age', 'Weight']]
y = data['Patient_Response']

# -----------------------------------
# 3. Split into training and testing
# -----------------------------------
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42
)

## What is random_state?

#train_test_split randomly shuffles the dataset before splitting.
#Without random_state:
#Each run → different split
#Model performance changes slightly
#With random_state=42:
#Same rows go to train/test every time
#Results are reproducible

# -----------------------------------
# 4: Train the Gradient Boosting model
# -----------------------------------

model = GradientBoostingRegressor(
    n_estimators=200,
    learning_rate=0.05,
    max_depth=3,
    random_state=42
)




model.fit(X_train, y_train)


# -----------------------------------
# 5: Make predictions & evaluate
# -----------------------------------
y_pred = model.predict(X_test)

print("MAE:", mean_absolute_error(y_test, y_pred))
print("R² score:", r2_score(y_test, y_pred))

#Predict response for a new patient
# New patient: Dosage=72mg, Age=36yrs, Weight=172lbs
new_patient = np.array([[72, 36, 172]])

#predicted_response = model.predict(new_patient)
print("Predicted patient response:", predicted_response[0])





About Us  | Contact Us | Sitemap  | Privacy Policy