feat: Add hyperparameter tuning to theory section

src/_quarto.yml

@@ -72,6 +72,10 @@ website:
               text: "Dying Neurons"
             - href: theory/overfitting.qmd
               text: "Overfitting"
+            - href: theory/underfitting.qmd
+              text: "Underfitting"
+            - href: theory/hyperparameter_tuning.qmd
+              text: "Hyperparameter Tuning"
 
           - href: theory/perplexity_in_ai.qmd
             text: "Perplexity and Quantization"

src/theory/hyperparameter_tuning.qmd

@@ -0,0 +1,79 @@
+# Understanding Hyperparameter Tuning
+
+Hyperparameters are crucial parameters that define a machine learning model's behavior during training. They play an essential role in determining how well a model learns from data and generalizes to unseen examples. In this article, we will explore the concept of hyperparameter tuning, its importance, and various techniques used for optimizing these parameters.
+
+## What are Hyperparameters?
+
+Hyperparameters are settings or configurations that control the learning process in a machine learning model. They differ from model parameters (weights) as they cannot be learned directly from data during training. Instead, hyperparameters must be set beforehand and remain constant throughout the training process. Some common examples of hyperparameters include:
+
+- Learning rate
+- Number of hidden layers and neurons in a neural network
+- Kernel type and regularization parameters for Support Vector Machines (SVM)
+- Tree depth or number of trees in ensemble methods like Random Forest or Gradient Boosting
+
+## Why is Hyperparameter Tuning Important?
+
+Hyperparameters significantly impact the performance of machine learning models. Properly tuned hyperparameters can lead to better model accuracy, faster convergence during training, and improved generalization on unseen data. On the other hand, poorly chosen hyperparameters may result in underfitting or overfitting issues, leading to suboptimal predictions.
+
+## Hyperparameter Tuning Techniques
+
+There are several techniques available for optimizing hyperparameters:
+
+1. Grid Search
+2. Random Search
+3. Bayesian Optimization
+4. Gradient-based optimization
+5. Evolutionary Algorithms
+6. Population Based Training (PBT)
+
+### 1. Grid Search
+
+Grid search is a brute force approach that exhaustively searches through all possible combinations of hyperparameters within predefined ranges or values. It evaluates the model's performance for each combination and selects the best one based on a chosen metric, such as accuracy or loss.
+
+```python
+from sklearn.model_selection import GridSearchCV
+# Define parameter grid
+param_grid = {
+    'learning_rate': [0.1, 0.01, 0.001],
+    'max_depth': [3, 5, 7]
+}
+# Create a model instance and perform GridSearchCV
+model = SomeModel()
+grid_search = GridSearchCV(estimator=model, param_grid=param_grid)
+```
+
+### 2. Random Search
+
+Random search is an alternative to grid search that randomly samples hyperparameter combinations from a predefined distribution or range. It can be more efficient than grid search when the number of hyperparameters and their possible values are large.
+
+```python
+from sklearn.model_selection import RandomizedSearchCV
+# Define parameter distributions
+param_distributions = {
+    'learning_rate': [0.1, 0.01, 0.001],
+    'max_depth': [3, 5, 7]
+}
+# Create a model instance and perform RandomizedSearchCV
+model = SomeModel()
+random_search = RandomizedSearchCV(estimator=model, param_distributions=param_distributions)
+```
+
+### 3. Bayesian Optimization
+
+Bayesian optimization is an approach that uses a probabilistic model to estimate the performance of hyperparameter combinations and select new ones based on this estimation. It can be more efficient than grid or random search, especially when evaluating each combination's cost (e.g., time) is high.
+
+```python
+from skopt import BayesSearchCV
+# Define parameter space
+param_space = [
+    Real(0.1, 0.3, name='learning_rate'),
+    Integer(2, 8, name='max_depth')
+]
+# Create a model instance and perform BayesSearchCV
+model = SomeModel()
+bayes_search = BayesSearchCV(estimator=model, search_spaces=param_space)
+```
+
+### Conclusion
+
+Hyperparameter tuning is an essential step in building effective machine learning models. By using techniques like grid search, random search, or Bayesian optimization, we can find the best hyperparameters for our model and improve its performance on unseen data.

src/theory/underfitting.qmd

@@ -0,0 +1,105 @@
+# Understanding Underfitting: Detection Using Training Metrics & Visualizations
+
+## Introduction
+
+In machine learning, underfitting occurs when a model is too simple to capture the underlying patterns of the data it's trying to learn from. This results in poor performance on both training and test datasets. Detecting underfitting early can help improve your models by guiding you towards more complex architectures or better feature engineering techniques. In this article, we will explore how to detect underfitting using training metrics and visualizations with Python code examples.
+
+## Training Metrics for Underfitting Detection
+
+To identify if a model is underfitting, it's essential to monitor its performance on the training data over time. Two key metrics can help in this process: accuracy (for classification problems) or mean squared error (MSE) and R-squared (R²) for regression tasks.
+
+### Accurcuacy (Classification Problems)
+
+For a binary classification problem, the model's accuracy is calculated as follows:
+
+```python
+accuracy = (true_positives + true_negatives) / total_samples
+```
+
+A low training accuracy indicates that the model may be underfitting.
+
+### Mean Squared Error & R-squared (Regression Problems)
+
+For regression problems, we use MSE and R² to evaluate performance:
+
+#### Mean Squared Error (MSE):
+
+```python
+mse = np.mean((y_true - y_pred)**2)
+```
+
+A high MSE indicates that the model's predictions are far from the true values, suggesting underfitting.
+
+#### R-squared:
+
+R² measures how well the regression line approximates the real data points. An R² close to 0 suggests a poor fit and potential underfitting.
+
+```python
+r_squared = 1 - (np.sum((y_true - y_pred)**2) / np.sum((y_true - np.mean(y_true))**2))
+```
+
+## Visualizing Underfitting with Python Code Examples
+
+To better understand underfitting, let's visualize it using a simple linear regression example in Python. We will use the `sklearn` library to create an overly simplistic model and plot its performance metrics.
+
+First, install necessary libraries:
+
+```bash
+pip install numpy matplotlib scikit-learn seaborn
+```
+
+Now, let's generate some synthetic data for our regression problem:
+
+```{python}
+import numpy as np
+import matplotlib.pyplot as plt
+from sklearn.linear_model import LinearRegression
+from sklearn.metrics import mean_squared_error, r2_score
+import seaborn as sns
+
+# Generate synthetic data for regression problem
+np.random.seed(42)
+X = np.random.rand(100, 1) * 10 # Features (inputs)
+y = 3*X.ravel() + np.random.randn(100)*3 # Target variable with some noise
+```
+
+Next, we'll fit a simple linear regression model and calculate its MSE and R²:
+
+```{python}
+# Fit the model
+model = LinearRegression().fit(X, y)
+y_pred = model.predict(X)
+
+# Calculate metrics
+mse = mean_squared_error(y, y_pred)
+r2 = r2_score(y, y_pred)
+print("MSE:", mse)
+print("R²:", r2)
+```
+
+Finally, let's visualize the data and model performance using seaborn:
+
+```{python}
+import seaborn as sns
+# Plotting synthetic data points
+plt.figure(figsize=(10, 6))
+sns.scatterplot(x=X.ravel(), y=y, color='blue', label="Data Points")
+
+# Plotting the regression line
+sns.lineplot(x=X.ravel(), y=model.predict(X), color='red', label="Regression Line")
+plt.title("Underfitting Example: Linear Regression on Synthetic Data")
+plt.legend()
+plt
+```
+
+
+
+In this example, the model's MSE and R² values indicate that it is underfitting the data:
+
+- The regression line does not capture the underlying pattern of the synthetic dataset well.
+- The high MSE value suggests a poor fit to the true target variable.
+- The low R² value indicates that our model explains only a small portion of the variance in the target variable.
+
+## Conclusion
+
+Detecting underfitting is crucial for improving machine learning models' performance. By monitoring training metrics such as accuracy, MSE, and R², we can identify when a model may be too simple to capture the underlying patterns in our data. Visualizations using Python code examples help us better understand these concepts and guide us towards more complex architectures or feature engineering techniques that could improve our models' performance.