HeadnotesMany equations and formulas look intimidating. However, when you hunt them down, they are definitely not! Just papertigers!
Now let's hunt the papertiger.
We will use this simple data1 set in all our tutorials. If we use 4th column as label, the 3rd column will be feature, vice versa.
Gradient boost for regression
The loss function of gradient boost defined as
Gradient boost for classification
Caveat To avoid heavy notation, we ignored summation symbol.
For a binary classificaton problem, we can define odds as
and probability as
You might wonder why we define this. In the following developments, you will find this definition will make the result be consistent with regression.
With some simple algebra,
We can define our loss function as cross entropy, such that
We want to find which can minimize the loss, in symbol,
We could directly work on Equation 6 with gradient descent or closed-form solution, such that
However, this will be quite complex.
We can use Taylor series to approximate the loss fucntion, you should convince yourself this will make things simpler, such that
Caveat Caveat Two kinds of derivatives of appeared here, one is w.r.t. and one is w.r.t. .
With Equation 9, and set
can be solved that
With Equation 6, the first order derivative with respect to can be calculated as
with some illustration
The second derivative of with respect to is
- 1.Dana D. Sleep Data Personal Sleep Data from Sleep Cycle iOS App. Kaggle. https://www.kaggle.com/danagerous/sleep-data#
FootnotesThere are many excellent tutorials out there. Some tutorials are too intuitive and it's helpful, but you cannot get it straight on the math details. Some focused on dymestifying math. Some focused on code. I found the best tutorials that give you the conceptual ideas and are possible for implementation without being blind to the math details. Drop a comment if I failed. It would be really appreciable.
Lachlan Chen, "Hunt the papertiger from boosting to XGBoost, intuitively, mathematically, implementably," in EarnFromScratch, 9月 8, 2020, https://www.earnfs.com/zh/html/2180.htm.
title=Hunt the papertiger from boosting to XGBoost, intuitively, mathematically, implementably,
year=9月 8, 2020