QTM 347 HW1 (2599822, 2539356, 2453794, 2396043)
February 25, 2024
[ ]:
import numpy as np
import pandas as pd
from ISLP import load_data
from sklearn.model_selection import train_test_split
[ ]:
import warnings
warnings.filterwarnings('ignore')
Promblem 1
(a)When the sample size n is extremely large and the number of predictors p is small, the perfor-mance of a flexible statistical learning method is expected to be better than an inflexible method. With a large sample size, a flexible method can better capture the underlying patterns between predictors and dependent variables, leading to improved predictive accuracy. However, a inflexible method may have the risk of overfitting.
(b)When the number of predictors p is extremely large and the number of observations n is small, the performance of a inflexible statistical learning method is expected to be better than an flexible method. With a small sample size relative to the number of predictors, flexible methods may largely be influenced by noise in the data, confusing it with true underlying patterns. Inflexible methods, on the other hand, are less susceptible to overfitting and may provide more stable and reliable predictions.
(c) In the case where the relationship between the predictors and response is highly non-linear, we would generally expect the performance of a flexible statistical learning method to be better than an inflexible method. Flexible methods have the ability to capture complex non-linear relationships in the data. They can adapt their complexity to fit the intricacies of the non-linear relationship, resulting in improved predictive accuracy. In contrast, inflexible methods may struggle to capture the nuances of the non-linear relationship and may result in poorer performance.
Promblem 2
a) lstat = 25 - When k = 1, we find the nearest point to be (24,39, 8.3), medv = 8.3 - When k = 5, we find the 5 nearest points to be (23.6, 11.3), (24.16, 14), (24.39, 8.3), (25.68, 9.7), (26.4, 17.2), medv = (11.3+14+8.3+9.7+17.2)/5 = 12.1
(b) lstat = 27
• When k = 1, we find the nearest point to be (26.82, 13.4), medv = 13.4
• When k = 5, we find the 5 nearest points to be (24.39, 8.3), (25.68, 9.7), (26.4, 17.2), (26.82, 13.4), (29.68, 8.1), medv = (8.3+9.7+17.2+13.4+8.1)/5 = 11.34
(c) lstat = 25
• When k = 1, we find the nearest point to be (24.39, 0), medv_cat = 0
• When k = 5, we find the 5 nearest points to be (23.6, 1), (24.16, 2), (24.39, 0), (25.68, 0), (26.4, 2), medv_cat = (1+2+0+0+2)/5 = 1
(d) lstat = 27
• When k = 1, we find the nearest point to be (26.82, 1), medv_cat = 1
• When k = 5, we find the 5 nearest points to be (24.39, 0), (25.68, 0), (26.4, 2), (26.82, 1), (29.68, 0), medv_cat = (0+0+2+1+0)/5 = 0.6
(e)
• In KNN, the model is less felxible with larger k. With a small k, the model relies on a smaller number of neighbors to make predictions, and so it can capture local patterns in the data. However, as k increases, the model becomes less flexible because it considers a larger number of neighbors, leading to a smoother function and a more generalized model.
(f)
• Bias: As k increases, the bias increases. With a larger k, the model averages over more data points to make predictions. This results in a smoother function that may not capture the underlying true relationship between the predictors and the response as accurately, leading to higher bias.
• Variance: As k increases, the variance decreases. With a larger k, the predictions are based on more data points, resulting in a more stable and robust model that is less sensitive to variations in the training data. This leads to lower variance because the predictions are less affected by small changes in the training data.
• Training MSE: As k increases, the training MSE increase. With a larger k, the model becomes less flexible and the predictions tend to be further away from the true values, leading to higher errors in the training data.
• Test MSE: As k increase, the test MSE initially decreases, reaches a minimum, and then starts to increase again. As k increases, the bias increases while the variance decreases. Initially, the decrease in variance dominates, leading to lower test MSE. However, as k continues to increase, the increase in bias starts to dominate, leading to higher test MSE.
Promblem 3
(a)
• The number of observations n is 100, and the number of features is 3 (different powers of x correspond to different features). The model is: y = x^5 - 2x^4 + x^3 (y = ff1x^5 - ff2x^4 + ff3x^3 + ff)
(b)
[ ]:
def f(x):
return x ** 5 - 2 * x ** 4 + x ** 3
def get_sim_data(f, sample_size=100, std=0.01):
x = np.random.uniform(0, 1, sample_size)
y = f(x) + np.random.normal(0, std, sample_size)
df = pd.DataFrame({"x": x, "y": y})
return df
(c)
[ ]:
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
import matplotlib.pyplot as plt
[ ]:
sim_data = get_sim_data(f)
degrees = np.arange(16)
for degree in degrees:
polynomial_features = PolynomialFeatures(degree=degree)
X_poly = polynomial_features.fit_transform(sim_data[['x']])
model = LinearRegression()
model.fit(X_poly, sim_data['y'])
[ ]:
x_fixed = np.array([0.18]).reshape(-1,1)
predictions = pd.DataFrame(columns=[f'Degree {i}' for i in range(16)],␣
↪index=[0])
sim_data = get_sim_data(f, sample_size=100, std=0.01)
X_train = np.array(sim_data['x']).reshape(-1, 1)
y_train = np.array(sim_data['y']).reshape(-1, 1)
for degree in range(16):
polynomial_features = PolynomialFeatures(degree=degree)
X_poly = polynomial_features.fit_transform(X_train)
model = LinearRegression().fit(X_poly, y_train)
x_fixed_poly = polynomial_features.transform(x_fixed)
predictions.loc[0, f'Degree {degree}'] = model.predict(x_fixed_poly).item()
predictions
[ ]:
Degree 0 Degree 1 Degree 2 Degree 3 Degree 4 Degree 5 Degree 6 \
0 0.016151 0.009605 0.006157 0.003526 0.00013 0.000486 0.000333
Degree 7 Degree 8 Degree 9 Degree 10 Degree 11 Degree 12 Degree 13 \
0 -0.000021 -0.000999 -0.001182 -0.000977 -0.002414 -0.001029 -0.001101
Degree 14 Degree 15
0 -0.001405 0.000981
(d)
[ ]:
predictions = pd.DataFrame(columns=[f'Degree {i}' for i in range(16)],␣
↪index=range(250))
for i in range(250):
sim_data = get_sim_data(f, sample_size=100, std=0.01)
X_train = np.array(sim_data['x']).reshape(-1, 1)
y_train = np.array(sim_data['y']).reshape(-1, 1)
for degree in range(16):
polynomial_features = PolynomialFeatures(degree=degree)
X_poly = polynomial_features.fit_transform(X_train)
model = LinearRegression().fit(X_poly, y_train)
x_fixed_poly = polynomial_features.transform(x_fixed)
predictions.loc[i, f'Degree {degree}'] = model.predict(x_fixed_poly).
↪item()
predictions.head()
[ ]:
Degree 0 Degree 1 Degree 2 Degree 3 Degree 4 Degree 5 Degree 6 \
0 0.017343 0.015916 0.010586 0.007302 0.003962 0.004943 0.004888
1 0.015583 0.010303 0.00762 0.005239 0.00154 0.001006 0.000951
2 0.016678 0.014221 0.0091 0.007389 0.003267 0.00375 0.003649
3 0.013869 0.010774 0.008704 0.007213 0.004782 0.005209 0.005071
4 0.011751 0.008241 0.007625 0.00356 0.000319 0.000681 0.000473
Degree 7 Degree 8 Degree 9 Degree 10 Degree 11 Degree 12 Degree 13 \
0 0.004635 0.005485 0.005357 0.004927 0.00464 0.005533 0.005648
1 0.000508 0.000307 0.000626 0.000169 0.00103 -0.001761 -0.001532
2 0.004055 0.003242 0.003486 0.003251 0.003898 0.004619 0.00462
3 0.005546 0.005243 0.00505 0.005669 0.006693 0.006152 0.006053
4 0.002626 0.003598 0.00365 0.00316 0.006308 0.005549 0.005766
Degree 14 Degree 15
0 0.005964 0.005617
1 -0.001252 -0.002004
2 0.004486 0.005213
3 0.005498 0.004036
4 0.006062 0.007607
(e) The Square of Bias
[ ]:
mean_predictions = predictions.mean()
true_value = f(0.18)
square_of_bias = (mean_predictions - true_value) ** 2
print("The Square of Bias")
square_of_bias
The Square of Bias
[ ]:
Degree 0 1.594870e-04
Degree 1 6.967211e-05
Degree 2 2.812878e-05
Degree 3 4.343497e-06
Degree 4 1.049161e-06
Degree 5 2.969633e-08
Degree 6 2.674706e-08
Degree 7 2.533310e-08
Degree 8 1.673926e-08
Degree 9 1.127903e-08
Degree 10 2.466444e-08
Degree 11 2.047126e-08
Degree 12 3.051183e-08
Degree 13 2.633141e-08
Degree 14 2.652020e-08
Degree 15 2.428047e-09
dtype: float64
(f) Variance
[ ]:
variance_of_predictions = predictions.var()
print("Variance")
variance_of_predictions
Variance
[ ]:
Degree 0 0.000003
Degree 1 0.000005
Degree 2 0.000003
Degree 3 0.000003
Degree 4 0.000004
Degree 5 0.000005
Degree 6 0.000006
Degree 7 0.000006
Degree 8 0.000008
Degree 9 0.000008
Degree 10 0.000008
Degree 11 0.000010
Degree 12 0.000012
Degree 13 0.000012
Degree 14 0.000013
Degree 15 0.000015
dtype: float64
(g) Irreducible Error
[ ]:
std = 0.01
irreducible_error = std ** 2
print("Irreducible Error")
irreducible_error
Irreducible Error
[ ]:
0.0001
The irreducible error represents the inherent variability in the data that cannot be reduced by any model. It is specified in the data generating process when we set the std to be 0.01.
(h) MSE
[ ]:
mse = square_of_bias + variance_of_predictions + irreducible_error
print("MSE")
mse
MSE
[ ]:
Degree 0 0.000262
Degree 1 0.000174
Degree 2 0.000131
Degree 3 0.000108
Degree 4 0.000105
Degree 5 0.000105
Degree 6 0.000106
Degree 7 0.000106
Degree 8 0.000108
Degree 9 0.000108
Degree 10 0.000108
Degree 11 0.000110
Degree 12 0.000112
Degree 13 0.000112
Degree 14 0.000113
Degree 15 0.000115
dtype: float64
(i) Plot
[ ]:
import matplotlib.pyplot as plt
degrees = range(16)
plt.figure(figsize=(14, 8))
plt.plot(degrees, square_of_bias, label='Square of Bias', marker='o')
plt.plot(degrees, variance_of_predictions, label='Variance', marker='x')
plt.plot(degrees, [irreducible_error]*16, label='Irreducible Error',␣
↪linestyle='--')
plt.plot(degrees, mse, label='MSE', marker='^')
plt.xlabel('Degree of Polynomial')
plt.ylabel('Error')
plt.title('Bias-Variance-MSE vs Degree')
plt.legend()
plt.show()
• Square of Bias: It decreases significantly as the degree of the polynomial increases from 0 to around 4 or 5, reflecting that higher-degree polynomials fit the data more closely and thus have a lower bias. However, beyond a certain point, the reduction in bias becomes marginal.
• Variance: Initially, the variance is relatively low for simpler models (low-degree polynomials) but starts to increase as the model complexity grows. This increase indicates that more complex models are more sensitive to the training data, leading to higher variability in their predictions across different datasets.
• Irreducible Error: It remains constant across all degrees of polynomial, as expected. This component of the error is not affected by the model choice or complexity because it represents the noise inherent in the data generation process.
• MSE: The Mean Squared Error decreases initially as the degree of the polynomial increases, benefiting from the reduced bias. However, as the model becomes more complex (especially beyond degree 4 or 5), the decrease in MSE slows down, and it eventually stabilizes or slightly increases. This pattern is due to the trade-off between bias and variance; while higher-degree polynomials reduce bias, their increased variance eventually offsets these gains, leading to a stabilization or slight increase in MSE.
• Conclusion: The plot and analysis underscore the importance of balancing model complex-ity to minimize overall error. While increasing the polynomial degree reduces bias, it also increases variance, which can harm the model’s generalization ability. The optimal model complexity (in terms of polynomial degree) appears to be in the range where the decrease in bias is balanced by the increase in variance, minimizing the MSE. This balance point is crucial for developing models that generalize well to unseen data.