feat: Add slideshow on optimizers in neural networks

src/theory/activations.qmd

@@ -1,7 +1,28 @@
+---
+title: "Activation functions"
+notebook-links: false
+crossref:
+  lof-title: "List of Figures"
+number-sections: false
+---
 
-## **1. Sigmoid (Logistic)**
+When choosing an activation function, consider the following:
+
+-   **Non-saturation:** Avoid activations that saturate (e.g., sigmoid, tanh) to prevent vanishing gradients.
+
+-   **Computational efficiency:** Choose activations that are computationally efficient (e.g., ReLU, Swish) for large models or real-time applications.
+
+-   **Smoothness:** Smooth activations (e.g., GELU, Mish) can help with optimization and convergence.
+
+-   **Domain knowledge:** Select activations based on the problem domain and desired output (e.g., softmax for multi-class classification).
+
+-   **Experimentation:** Try different activations and evaluate their performance on your specific task.
+
+[Slideshow](activations_slideshow.qmd)
+
+{{< embed ActivationFunctions.ipynb#fig-overview >}}
 
-**Formula:** σ(x) = 1 / (1 + exp(-x))
+## Sigmoid {#sec-sigmoid}
 
 **Strengths:** Maps any real-valued number to a value between 0 and 1, making it suitable for binary classification problems.
 
@@ -9,9 +30,24 @@
 
 **Usage:** Binary classification, logistic regression.
 
-## **2. Hyperbolic Tangent (Tanh)**
+::: columns
+::: {.column width="50%"}
+$$ 
+\sigma(x) = \frac{1}{1 + e^{-x}}
+$$
 
-**Formula:** tanh(x) = 2 / (1 + exp(-2x)) - 1
+``` python
+def sigmoid(x):
+    return 1 / (1 + np.exp(-x))
+```
+:::
+
+::: {.column width="50%"}
+{{< embed ActivationFunctions.ipynb#fig-sigmoid >}}
+:::
+:::
+
+## Hyperbolic Tangent (Tanh) {#sec-tanh}
 
 **Strengths:** Similar to sigmoid, but maps to (-1, 1), which can be beneficial for some models.
 
@@ -19,9 +55,24 @@
 
 **Usage:** Similar to sigmoid, but with a larger output range.
 
-## **3. Rectified Linear Unit (ReLU)**
+::: columns
+::: {.column width="50%"}
+$$
+\tanh(x) = \frac{e^{x} - e^{-x}}{e^{x} + e^{-x}}
+$$
+
+``` python
+def tanh(x):
+    return np.tanh(x)
+```
+:::
+
+::: {.column width="50%"}
+{{< embed ActivationFunctions.ipynb#fig-tanh >}}
+:::
+:::
 
-**Formula:** f(x) = max(0, x)
+## Rectified Linear Unit (ReLU)
 
 **Strengths:** Computationally efficient, non-saturating, and easy to compute.
 
@@ -29,9 +80,24 @@
 
 **Usage:** Default activation function in many deep learning frameworks, suitable for most neural networks.
 
-## **4. Leaky ReLU**
+::: columns
+::: {.column width="50%"}
+$$
+\text{ReLU}(x) = \max(0, x)
+$$
 
-**Formula:** f(x) = max(αx, x), where α is a small constant (e.g., 0.01)
+``` python
+def relu(x):
+    return np.maximum(0, x)
+```
+:::
+
+::: {.column width="50%"}
+{{< embed ActivationFunctions.ipynb#fig-relu >}}
+:::
+:::
+
+## Leaky ReLU
 
 **Strengths:** Similar to ReLU, but allows a small fraction of the input to pass through, helping with dying neurons.
 
@@ -39,9 +105,33 @@
 
 **Usage:** Alternative to ReLU, especially when dealing with dying neurons.
 
-## **5. Swish**
+::: columns
+::: {.column width="50%"}
+$$
+\text{Leaky ReLU}(x) = 
+\begin{cases} 
+x & \text{if } x > 0 \\
+\alpha x & \text{if } x \leq 0 
+\end{cases}
+$$
+
+``` python
+def leaky_relu(x, alpha=0.01):
+    # where α is a small constant (e.g., 0.01)
+    return np.where(x > 0, x, x * alpha)
+```
+:::
 
-**Formula:** f(x) = x \* g(x), where g(x) is a learned function (e.g., sigmoid or ReLU)
+::: {.column width="50%"}
+{{< embed ActivationFunctions.ipynb#fig-leaky_relu >}}
+:::
+:::
+
+## Swish
+
+**Formula:**
+
+where g(x) is a learned function (e.g., sigmoid or ReLU)
 
 **Strengths:** Self-gated, adaptive, and non-saturating.
 
@@ -49,9 +139,53 @@
 
 **Usage:** Can be used in place of ReLU or other activations, but may not always outperform them.
 
-## **6. Softmax**
+::: columns
+::: {.column width="50%"}
+$$
+\text{Swish}(x) = x \cdot \sigma(x)
+$$
+
+``` python
+def swish(x):
+    return x * sigmoid(x)
+```
+
+See also: [sigmoid](#sec-sigmoid)
+:::
+
+::: {.column width="50%"}
+{{< embed ActivationFunctions.ipynb#fig-swish >}}
+:::
+:::
+
+## Mish
+
+**Strengths:** Non-saturating, smooth, and computationally efficient.
+
+**Weaknesses:** Not as well-studied as ReLU or other activations.
+
+**Usage:** Alternative to ReLU, especially in computer vision tasks.
+
+::: columns
+::: {.column width="50%"}
+$$
+\text{Mish}(x) = x \cdot \tanh(\text{Softplus}(x))
+$$
 
-**Formula:** softmax(x) = exp(x) / Σ exp(x)
+``` python
+def mish(x):
+    return x * np.tanh(softplus(x))
+```
+:::
+
+::: {.column width="50%"}
+{{< embed ActivationFunctions.ipynb#fig-mish >}}
+:::
+:::
+
+See also: [softplus](#softplus) [tanh](#sec-tanh)
+
+## Softmax
 
 **Strengths:** Normalizes output to ensure probabilities sum to 1, making it suitable for multi-class classification.
 
@@ -59,9 +193,25 @@
 
 **Usage:** Output layer activation for multi-class classification problems.
 
-## **7. Softsign**
+::: columns
+::: {.column width="50%"}
+$$
+\text{Softmax}(x_i) = \frac{e^{x_i}}{\sum_{k=1}^{K} e^{x_k}}
+$$
+
+``` python
+def softmax(x):
+    e_x = np.exp(x - np.max(x))
+    return e_x / e_x.sum()
+```
+:::
 
-**Formula:** f(x) = x / (1 + |x|)
+::: {.column width="50%"}
+{{< embed ActivationFunctions.ipynb#fig-softmax >}}
+:::
+:::
+
+## Softsign
 
 **Strengths:** Similar to sigmoid, but with a more gradual slope.
 
@@ -69,29 +219,74 @@
 
 **Usage:** Alternative to sigmoid or tanh in certain situations.
 
-## **8. ArcTan**
+::: columns
+::: {.column width="50%"}
+$$
+\text{Softsign}(x) = \frac{x}{1 + |x|}
+$$
 
-**Formula:** f(x) = arctan(x)
+``` python
+def softsign(x):
+    return x / (1 + np.abs(x))
+```
+:::
 
-**Strengths:** Non-saturating, smooth, and continuous.
+::: {.column width="50%"}
+{{< embed ActivationFunctions.ipynb#fig-softsign >}}
+:::
+:::
+
+## SoftPlus {#softplus}
+
+**Strengths:** Smooth, continuous, and non-saturating.
 
 **Weaknesses:** Not commonly used, may not outperform other activations.
 
 **Usage:** Experimental or niche applications.
 
-## **9. SoftPlus**
+::: columns
+::: {.column width="50%"}
+$$
+\text{Softplus}(x) = \log(1 + e^x)
+$$
 
-**Formula:** f(x) = log(1 + exp(x))
+``` python
+def softplus(x):
+    return np.log1p(np.exp(x))
+```
+:::
 
-**Strengths:** Smooth, continuous, and non-saturating.
+::: {.column width="50%"}
+{{< embed ActivationFunctions.ipynb#fig-softplus >}}
+:::
+:::
+
+## ArcTan
+
+**Strengths:** Non-saturating, smooth, and continuous.
 
 **Weaknesses:** Not commonly used, may not outperform other activations.
 
 **Usage:** Experimental or niche applications.
 
-## **10. Gaussian Error Linear Unit (GELU)**
+::: columns
+::: {.column width="50%"}
+$$
+arctan(x) = arctan(x)
+$$
+
+``` python
+def arctan(x):
+    return np.arctan(x)
+```
+:::
 
-**Formula:** f(x) = x \* Φ(x), where Φ is the cumulative distribution function of the standard normal distribution
+::: {.column width="50%"}
+{{< embed ActivationFunctions.ipynb#fig-arctan >}}
+:::
+:::
+
+## Gaussian Error Linear Unit (GELU)
 
 **Strengths:** Non-saturating, smooth, and computationally efficient.
 
@@ -99,9 +294,28 @@
 
 **Usage:** Alternative to ReLU, especially in Bayesian neural networks.
 
-## **11. Mish**
+::: columns
+::: {.column width="50%"}
+$$
+\text{GELU}(x) = x \cdot \Phi(x)
+$$
+
+``` python
+def gelu(x):
+    return 0.5 * x 
+        * (1 + np.tanh(np.sqrt(2 / np.pi) 
+        * (x + 0.044715 * np.power(x, 3))))
+```
+:::
+
+::: {.column width="50%"}
+{{< embed ActivationFunctions.ipynb#fig-gelu >}}
+:::
+:::
+
+See also: [tanh](#sec-tanh)
 
-**Formula:** f(x) = x \* tanh(softplus(x))
+## Silu (SiLU)
 
 **Strengths:** Non-saturating, smooth, and computationally efficient.
 
@@ -109,19 +323,29 @@
 
 **Usage:** Alternative to ReLU, especially in computer vision tasks.
 
-## **12. Silu (SiLU)**
+::: columns
+::: {.column width="50%"}
+$$
+silu(x) = x * sigmoid(x)
+$$
 
-**Formula:** f(x) = x \* sigmoid(x)
 
-**Strengths:** Non-saturating, smooth, and computationally efficient.
-
-**Weaknesses:** Not as well-studied as ReLU or other activations.
+``` python
+def silu(x):
+    return x / (1 + np.exp(-x))
+```
+:::
 
-**Usage:** Alternative to ReLU, especially in computer vision tasks.
+::: {.column width="50%"}
+{{< embed ActivationFunctions.ipynb#fig-silu >}}
+:::
+:::
 
-## **13. GELU Approximation (GELU Approx.)**
+## GELU Approximation (GELU Approx.)
 
-**Formula:** f(x) ≈ 0.5 \* x \* (1 + tanh(√(2/π) \* (x + 0.044715 \* x^3)))
+$$
+f(x) ≈ 0.5 * x * (1 + tanh(√(2/π) * (x + 0.044715 * x^3)))
+$$
 
 **Strengths:** Fast, non-saturating, and smooth.
 
@@ -129,9 +353,15 @@
 
 **Usage:** Alternative to GELU, especially when computational efficiency is crucial.
 
-## **14. SELU (Scaled Exponential Linear Unit)**
+## SELU (Scaled Exponential Linear Unit)
 
-**Formula:** f(x) = λ { x if x > 0, α(e^x - 1) if x ≤ 0 }
+$$
+f(x) = \lambda 
+\begin{cases} 
+x & x > 0 \\
+\alpha e^x - \alpha & x \leq 0 
+\end{cases}
+$$
 
 **Strengths:** Self-normalizing, non-saturating, and computationally efficient.
 
@@ -139,14 +369,3 @@
 
 **Usage:** Alternative to ReLU, especially in deep neural networks.
 
-When choosing an activation function, consider the following:
-
-* **Non-saturation:** Avoid activations that saturate (e.g., sigmoid, tanh) to prevent vanishing gradients.
-
-* **Computational efficiency:** Choose activations that are computationally efficient (e.g., ReLU, Swish) for large models or real-time applications.
-
-* **Smoothness:** Smooth activations (e.g., GELU, Mish) can help with optimization and convergence.
-
-* **Domain knowledge:** Select activations based on the problem domain and desired output (e.g., softmax for multi-class classification).
-
-* **Experimentation:** Try different activations and evaluate their performance on your specific task.

src/theory/activations_slideshow.qmd

@@ -0,0 +1,355 @@
+---
+title: "Activation functions in Neural Networks"
+author: "Sébastien De Greef"
+format:
+  revealjs: 
+    theme: solarized
+    navigation-mode: grid 
+    controls-layout: bottom-right
+    controls-tutorial: true
+---
+
+# Activation functions
+
+When choosing an activation function, consider the following:
+
+-   **Non-saturation:** Avoid activations that saturate (e.g., sigmoid, tanh) to prevent vanishing gradients.
+
+-   **Computational efficiency:** Choose activations that are computationally efficient (e.g., ReLU, Swish) for large models or real-time applications.
+
+-   **Smoothness:** Smooth activations (e.g., GELU, Mish) can help with optimization and convergence.
+
+-   **Domain knowledge:** Select activations based on the problem domain and desired output (e.g., softmax for multi-class classification).
+
+-   **Experimentation:** Try different activations and evaluate their performance on your specific task.
+
+## Sigmoid {#sec-sigmoid}
+
+**Strengths:** Maps any real-valued number to a value between 0 and 1, making it suitable for binary classification problems.
+
+**Weaknesses:** Saturates (i.e., output values approach 0 or 1) for large inputs, leading to vanishing gradients during backpropagation.
+
+**Usage:** Binary classification, logistic regression.
+
+::: columns
+::: {.column width="50%"}
+$$ 
+\sigma(x) = \frac{1}{1 + e^{-x}}
+$$
+
+``` python
+def sigmoid(x):
+    return 1 / (1 + np.exp(-x))
+```
+:::
+
+::: {.column width="50%"}
+{{< embed ActivationFunctions.ipynb#fig-sigmoid >}}
+:::
+:::
+
+## Hyperbolic Tangent (Tanh) {#sec-tanh}
+
+**Strengths:** Similar to sigmoid, but maps to (-1, 1), which can be beneficial for some models.
+
+**Weaknesses:** Also saturates, leading to vanishing gradients.
+
+**Usage:** Similar to sigmoid, but with a larger output range.
+
+::: columns
+::: {.column width="50%"}
+$$
+\tanh(x) = \frac{e^{x} - e^{-x}}{e^{x} + e^{-x}}
+$$
+
+``` python
+def tanh(x):
+    return np.tanh(x)
+```
+:::
+
+::: {.column width="50%"}
+{{< embed ActivationFunctions.ipynb#fig-tanh >}}
+:::
+:::
+
+## Rectified Linear Unit (ReLU)
+
+**Strengths:** Computationally efficient, non-saturating, and easy to compute.
+
+**Weaknesses:** Not differentiable at x=0, which can cause issues during optimization.
+
+**Usage:** Default activation function in many deep learning frameworks, suitable for most neural networks.
+
+::: columns
+::: {.column width="50%"}
+$$
+\text{ReLU}(x) = \max(0, x)
+$$
+
+``` python
+def relu(x):
+    return np.maximum(0, x)
+```
+:::
+
+::: {.column width="50%"}
+{{< embed ActivationFunctions.ipynb#fig-relu >}}
+:::
+:::
+
+## Leaky ReLU
+
+**Strengths:** Similar to ReLU, but allows a small fraction of the input to pass through, helping with dying neurons.
+
+**Weaknesses:** Still non-differentiable at x=0.
+
+**Usage:** Alternative to ReLU, especially when dealing with dying neurons.
+
+::: columns
+::: {.column width="50%"}
+$$
+\text{Leaky ReLU}(x) = 
+\begin{cases} 
+x & \text{if } x > 0 \\
+\alpha x & \text{if } x \leq 0 
+\end{cases}
+$$
+
+``` python
+def leaky_relu(x, alpha=0.01):
+    # where α is a small constant (e.g., 0.01)
+    return np.where(x > 0, x, x * alpha)
+```
+:::
+
+::: {.column width="50%"}
+{{< embed ActivationFunctions.ipynb#fig-leaky_relu >}}
+:::
+:::
+
+## Swish
+
+**Strengths:** Self-gated, adaptive, and non-saturating.
+
+**Weaknesses:** Computationally expensive, requires additional learnable parameters.
+
+**Usage:** Can be used in place of ReLU or other activations, but may not always outperform them.
+
+::: columns
+::: {.column width="50%"}
+$$
+\text{Swish}(x) = x \cdot \sigma(x)
+$$
+
+``` python
+def swish(x):
+    return x * sigmoid(x)
+```
+
+See also: [sigmoid](#sec-sigmoid)
+:::
+
+::: {.column width="50%"}
+{{< embed ActivationFunctions.ipynb#fig-swish >}}
+:::
+:::
+
+## Mish
+
+**Strengths:** Non-saturating, smooth, and computationally efficient.
+
+**Weaknesses:** Not as well-studied as ReLU or other activations.
+
+**Usage:** Alternative to ReLU, especially in computer vision tasks.
+
+::: columns
+::: {.column width="50%"}
+$$
+\text{Mish}(x) = x \cdot \tanh(\text{Softplus}(x))
+$$
+
+``` python
+def mish(x):
+    return x * np.tanh(softplus(x))
+```
+:::
+
+::: {.column width="50%"}
+{{< embed ActivationFunctions.ipynb#fig-mish >}}
+:::
+:::
+
+See also: [softplus](#softplus) [tanh](#sec-tanh)
+
+## Softmax
+
+**Strengths:** Normalizes output to ensure probabilities sum to 1, making it suitable for multi-class classification.
+
+**Weaknesses:** Only suitable for output layers with multiple classes.
+
+**Usage:** Output layer activation for multi-class classification problems.
+
+::: columns
+::: {.column width="50%"}
+$$
+\text{Softmax}(x_i) = \frac{e^{x_i}}{\sum_{k=1}^{K} e^{x_k}}
+$$
+
+``` python
+def softmax(x):
+    e_x = np.exp(x - np.max(x))
+    return e_x / e_x.sum()
+```
+:::
+
+::: {.column width="50%"}
+{{< embed ActivationFunctions.ipynb#fig-softmax >}}
+:::
+:::
+
+## Softsign
+
+**Strengths:** Similar to sigmoid, but with a more gradual slope.
+
+**Weaknesses:** Not commonly used, may not provide significant benefits over sigmoid or tanh.
+
+**Usage:** Alternative to sigmoid or tanh in certain situations.
+
+::: columns
+::: {.column width="50%"}
+$$
+\text{Softsign}(x) = \frac{x}{1 + |x|}
+$$
+
+``` python
+def softsign(x):
+    return x / (1 + np.abs(x))
+```
+:::
+
+::: {.column width="50%"}
+{{< embed ActivationFunctions.ipynb#fig-softsign >}}
+:::
+:::
+
+## SoftPlus {#softplus}
+
+**Strengths:** Smooth, continuous, and non-saturating.
+
+**Weaknesses:** Not commonly used, may not outperform other activations.
+
+**Usage:** Experimental or niche applications.
+
+::: columns
+::: {.column width="50%"}
+$$
+\text{Softplus}(x) = \log(1 + e^x)
+$$
+
+``` python
+def softplus(x):
+    return np.log1p(np.exp(x))
+```
+:::
+
+::: {.column width="50%"}
+{{< embed ActivationFunctions.ipynb#fig-softplus >}}
+:::
+:::
+
+## ArcTan
+
+**Strengths:** Non-saturating, smooth, and continuous.
+
+**Weaknesses:** Not commonly used, may not outperform other activations.
+
+**Usage:** Experimental or niche applications.
+
+::: columns
+::: {.column width="50%"}
+$$
+arctan(x) = arctan(x)
+$$
+
+``` python
+def arctan(x):
+    return np.arctan(x)
+```
+:::
+
+::: {.column width="50%"}
+{{< embed ActivationFunctions.ipynb#fig-arctan >}}
+:::
+:::
+
+## Gaussian Error Linear Unit (GELU)
+
+**Strengths:** Non-saturating, smooth, and computationally efficient.
+
+**Weaknesses:** Not as well-studied as ReLU or other activations.
+
+**Usage:** Alternative to ReLU, especially in Bayesian neural networks.
+
+::: columns
+::: {.column width="50%"}
+$$
+\text{GELU}(x) = x \cdot \Phi(x)
+$$
+
+``` python
+def gelu(x):
+    return 0.5 * x 
+        * (1 + np.tanh(np.sqrt(2 / np.pi) 
+        * (x + 0.044715 * np.power(x, 3))))
+```
+:::
+
+::: {.column width="50%"}
+{{< embed ActivationFunctions.ipynb#fig-gelu >}}
+:::
+:::
+
+See also: [tanh](#sec-tanh)
+
+## Silu (SiLU)
+
+$$
+silu(x) = x * sigmoid(x)
+$$
+
+**Strengths:** Non-saturating, smooth, and computationally efficient.
+
+**Weaknesses:** Not as well-studied as ReLU or other activations.
+
+**Usage:** Alternative to ReLU, especially in computer vision tasks.
+
+## GELU Approximation (GELU Approx.)
+
+$$
+f(x) ≈ 0.5 * x * (1 + tanh(√(2/π) * (x + 0.044715 * x^3)))
+$$
+
+**Strengths:** Fast, non-saturating, and smooth.
+
+**Weaknesses:** Approximation, not exactly equal to GELU.
+
+**Usage:** Alternative to GELU, especially when computational efficiency is crucial.
+
+## SELU (Scaled Exponential Linear Unit)
+
+$$
+f(x) = \lambda 
+\begin{cases} 
+x & x > 0 \\
+\alpha e^x - \alpha & x \leq 0 
+\end{cases}
+$$
+
+**Strengths:** Self-normalizing, non-saturating, and computationally efficient.
+
+**Weaknesses:** Requires careful initialization and α tuning.
+
+**Usage:** Alternative to ReLU, especially in deep neural networks.
+
+\listoffigures

src/theory/architectures.qmd

@@ -1,104 +1,763 @@
-## **1. Feedforward Neural Networks (FNNs)**
+---
+title: Network Architectures
+---
 
-* Usage: Image classification, regression, function approximation
-* Description: A basic neural network architecture where data flows only in one direction, from input layer to output layer, without any feedback loops.
-* Strengths: Simple to implement, computationally efficient
-* Caveats: Limited capacity to model complex relationships, prone to overfitting
+Neural network architectures are foundational frameworks designed to tackle diverse problems in artificial intelligence and machine learning. Each architecture is structured to optimize learning and performance for specific types of data and tasks, ranging from simple classification problems to complex sequence generation challenges. This guide explores the various architectures employed in neural networks, providing insights into how they are constructed, their applications, and why certain architectures are preferred for particular tasks.
 
-## **2. Convolutional Neural Networks (CNNs)**
+The architecture of a neural network dictates how information flows and is processed. It determines the arrangement and connectivity of layers, the type of data processing that occurs, and how input data is ultimately transformed into outputs. The choice of a suitable architecture is crucial because it impacts the efficiency, accuracy, and feasibility of training models on given datasets.
 
-* Usage: Image classification, object detection, image segmentation
-* Description: A neural network architecture that uses convolutional and pooling layers to extract features from images.
-* Strengths: Excellent performance on image-related tasks, robust to image transformations
-* Caveats: Computationally expensive, require large datasets
+## Feedforward Neural Networks (FNNs)
 
-## **3. Recurrent Neural Networks (RNNs)**
+A basic neural network architecture where data flows only in one direction, from input layer to output layer, without any feedback loops. Feedforward Neural Networks are the simplest type of neural network architecture where connections between the nodes do not form a cycle. This is ideal for problems where the output is directly mapped from the input.
 
-* Usage: Natural Language Processing (NLP), sequence prediction, time series forecasting
-* Description: A neural network architecture that uses feedback connections to model sequential data.
-* Strengths: Excellent performance on sequential data, can model long-term dependencies
-* Caveats: Suffer from vanishing gradients, difficult to train
+* **Usage**: Image classification, regression, function approximation
+* **Strengths**: Simple to implement, computationally efficient
+* **Caveats**: Limited capacity to model complex relationships, prone to overfitting
 
-## **4. Long Short-Term Memory (LSTM) Networks**
+```{dot}
+digraph FNN {
+    // Graph properties
+    node [shape=record];
 
-* Usage: NLP, sequence prediction, time series forecasting
-* Description: A type of RNN that uses memory cells to learn long-term dependencies.
-* Strengths: Excellent performance on sequential data, can model long-term dependencies
-* Caveats: Computationally expensive, require large datasets
+    // Nodes definitions
+    input_layer [label="Input Layer" shape=ellipse];
+    hidden_layer1 [label="Hidden Layer 1" shape=box];
+    hidden_layer2 [label="Hidden Layer 2" shape=box];
+    output_layer [label="Output Layer" shape=ellipse];
 
-## **5. Transformers**
+    // Edges definitions
+    input_layer -> hidden_layer1;
+    hidden_layer1 -> hidden_layer2;
+    hidden_layer2 -> output_layer;
+}
+```
 
-* Usage: NLP, machine translation, language modeling
-* Description: A neural network architecture that uses self-attention mechanisms to model relationships between input sequences.
-* Strengths: Excellent performance on sequential data, parallelizable, can handle long-range dependencies
-* Caveats: Computationally expensive, require large datasets
+**Input Layer**: This layer represents the initial data that is fed into the network. Each node in this layer typically corresponds to a feature in the input dataset.
 
-## **6. Autoencoders**
+**Hidden Layers**: These are intermediary layers between the input and output layers. Hidden layers allow the network to learn complex patterns in the data. They are called "hidden" because they are not directly exposed to the input or output.
 
-* Usage: Dimensionality reduction, anomaly detection, generative modeling
-* Description: A neural network architecture that learns to compress and reconstruct input data.
-* Strengths: Excellent performance on dimensionality reduction, can learn robust representations
-* Caveats: May not perform well on complex data distributions
+**Output Layer**: The final layer that produces the network’s predictions. The function of this layer can vary depending on the specific application — for example, it might use a softmax activation function for classification tasks or a linear activation for regression tasks.
 
-## **7. Generative Adversarial Networks (GANs)**
+**Edges**: Represent the connections between neurons in consecutive layers. In feedforward networks, every neuron in one layer connects to every neuron in the next layer. These connections are weighted, and these weights are adjusted during training to minimize error.
 
-* Usage: Generative modeling, data augmentation, style transfer
-* Description: A neural network architecture that consists of a generator and discriminator, which compete to generate realistic data.
-* Strengths: Excellent performance on generative tasks, can generate realistic data
-* Caveats: Training can be unstable, require careful tuning of hyperparameters
+## Convolutional Neural Networks (CNNs)
 
-## **8. Residual Networks (ResNets)**
+A neural network architecture that uses convolutional and pooling layers to extract features from images. CNNs are highly effective at processing data that has a grid-like topology, such as images, due to their ability to exploit spatial hierarchies and structures within the data.
 
-* Usage: Image classification, object detection
-* Description: A neural network architecture that uses residual connections to ease training.
-* Strengths: Excellent performance on image-related tasks, ease of training
-* Caveats: May not perform well on sequential data
+* **Usage**: Image classification, object detection, image segmentation
+* **Strengths**: Excellent performance on image-related tasks, robust to image transformations
+* **Caveats**: Computationally expensive, require large datasets
 
-## **9. U-Net**
 
-* Usage: Image segmentation, object detection
-* Description: A neural network architecture that uses a encoder-decoder structure with skip connections.
-* Strengths: Excellent performance on image segmentation tasks, fast training
-* Caveats: May not perform well on sequential data
+```{dot}
+digraph CNN {
+    // Graph properties
+    node [shape=record];
 
-## **10. Attention-based Models**
+    // Nodes definitions
+    input_image [label="Input Image" shape=ellipse];
+    conv1 [label="Convolution Layer 1\nReLU" shape=box];
+    pool1 [label="Pooling Layer 1" shape=box];
+    conv2 [label="Convolution Layer 2\nReLU" shape=box];
+    pool2 [label="Pooling Layer 2" shape=box];
+    fully_connected [label="Fully Connected Layer" shape=box];
+    output [label="Output\n(Classification)" shape=ellipse];
 
-* Usage: NLP, machine translation, question answering
-* Description: A neural network architecture that uses attention mechanisms to focus on relevant input regions.
-* Strengths: Excellent performance on sequential data, can model long-range dependencies
-* Caveats: Require careful tuning of hyperparameters
+    // Edges definitions
+    input_image -> conv1;
+    conv1 -> pool1;
+    pool1 -> conv2;
+    conv2 -> pool2;
+    pool2 -> fully_connected;
+    fully_connected -> output;
+}
+```
+**Input Image**: The initial input where images are fed into the network.
 
-## **11. Graph Neural Networks (GNNs)**
+**Convolution Layer 1 and 2**: These layers apply a set of filters to the input image to create feature maps. These filters are designed to detect spatial hierarchies such as edges, colors, gradients, and more complex patterns as the network deepens. Each convolution layer is typically followed by a non-linear activation function like ReLU (Rectified Linear Unit).
 
-* Usage: Graph-based data, social network analysis, recommendation systems
-* Description: A neural network architecture that uses graph structures to model relationships between nodes.
-* Strengths: Excellent performance on graph-based data, can model complex relationships
-* Caveats: Computationally expensive, require large datasets
+**Pooling Layer 1 and 2**: These layers reduce the spatial size of the feature maps to decrease the amount of computation and weights in the network. Pooling (often max pooling) helps make the detection of features invariant to scale and orientation changes.
 
-## **12. Reinforcement Learning (RL) Architectures**
+**Fully Connected Layer**: This layer takes the flattened output of the last pooling layer and performs classification based on the features extracted by the convolutional and pooling layers.
 
-* Usage: Game playing, robotics, autonomous systems
-* Description: A neural network architecture that uses reinforcement learning to learn from interactions with an environment.
-* Strengths: Excellent performance on sequential decision-making tasks, can learn complex policies
-* Caveats: Require large datasets, can be slow to train
+**Output**: The final output layer, which classifies the input image into categories based on the training dataset.
 
-## **13. Evolutionary Neural Networks**
+## Recurrent Neural Networks (RNNs)
 
-* Usage: Neuroevolution, optimization problems
-* Description: A neural network architecture that uses evolutionary principles to evolve neural networks.
-* Strengths: Excellent performance on optimization problems, can learn complex policies
-* Caveats: Computationally expensive, require large datasets
+A neural network architecture that uses feedback connections to model sequential data. RNNs are capable of processing sequences of data by maintaining a state that acts as a memory. They are particularly useful for applications where the context or sequence of data points is important.
 
-## **14. Spiking Neural Networks (SNNs)**
+* **Usage**: Natural Language Processing (NLP), sequence prediction, time series forecasting
+* **Strengths**: Excellent performance on sequential data, can model long-term dependencies
+* **Caveats**: Suffer from vanishing gradients, difficult to train
+```{dot}
+digraph RNN {
+    // Graph properties
+    node [shape=record];
 
-* Usage: Neuromorphic computing, edge AI
-* Description: A neural network architecture that uses spiking neurons to process data.
-* Strengths: Excellent performance on edge AI applications, energy-efficient
-* Caveats: Limited software support, require specialized hardware
+    // Nodes definitions
+    input_seq [label="Input Sequence" shape=ellipse];
+    rnn_cell [label="RNN Cell" shape=box];
+    hidden_state [label="Hidden State" shape=box];
+    output_seq [label="Output Sequence" shape=ellipse];
 
-## **15. Conditional Random Fields (CRFs)**
+    // Edges definitions
+    input_seq -> rnn_cell;
+    rnn_cell -> hidden_state [label="Next"];
+    hidden_state -> rnn_cell [label="Recurrence", dir=back];
+    rnn_cell -> output_seq;
 
-* Usage: NLP, sequence labeling, information extraction
-* Description: A probabilistic model that uses graphical models to model sequential data.
-* Strengths: Excellent performance on sequential data, can model complex relationships
-* Caveats: Computationally expensive, require large datasets
+    // Additional details for clarity
+    edge [style=dashed];
+    rnn_cell -> output_seq [label="Each timestep", style=dashed];
+}
+```
+
+**Input Sequence**: Represents the sequence of data being fed into the RNN, such as a sentence or time-series data.
+
+**RNN Cell**: This is the core of an RNN, where the computation happens. It takes input from the current element of the sequence and combines it with the hidden state from the previous element of the sequence.
+
+**Hidden State**: This node represents the memory of the network, carrying information from one element of the sequence to the next. The hidden state is updated continuously as the sequence is processed.
+
+**Output Sequence**: The RNN can produce an output at each timestep, depending on the task. For example, in sequence labeling, there might be an output corresponding to each input timestep.
+
+
+## Long Short-Term Memory (LSTM) Networks
+
+A type of RNN that uses memory cells to learn long-term dependencies. LSTM networks are designed to avoid the long-term dependency problem, making them effective at tasks where the context can extend over longer sequences.
+
+* **Usage**: NLP, sequence prediction, time series forecasting
+* **Strengths**: Excellent performance on sequential data, can model long-term dependencies
+* **Caveats**: Computationally expensive, require large datasets
+
+```{dot}
+digraph LSTM {
+    // Graph properties
+    node [shape=record];
+
+    // Nodes definitions
+    input_seq [label="Input Sequence" shape=ellipse];
+    lstm_cell [label="LSTM Cell" shape=box];
+    cell_state [label="Cell State" shape=box];
+    hidden_state [label="Hidden State" shape=box];
+    output_seq [label="Output Sequence" shape=ellipse];
+
+    // Edges definitions
+    input_seq -> lstm_cell;
+    cell_state -> lstm_cell [label="Recurrence", dir=back];
+    hidden_state -> lstm_cell [label="Recurrence", dir=back];
+    lstm_cell -> cell_state [label="Update"];
+    lstm_cell -> hidden_state [label="Update"];
+    lstm_cell -> output_seq;
+
+    // Additional explanations
+    edge [style=dashed];
+    lstm_cell -> output_seq [label="Each timestep", style=dashed];
+}
+```
+
+**Input Sequence**: Represents the sequential data input, such as a series of words or time-series data points.
+
+**LSTM Cell**: The core unit in an LSTM network that processes input data one element at a time. It interacts intricately with both the cell state and the hidden state to manage and preserve information over long periods.
+
+**Cell State**: A "long-term" memory component of the LSTM cell. It carries relevant information throughout the processing of the sequence, with the ability to add or remove information via gates (not explicitly shown here).
+
+**Hidden State**: A "short-term" memory component that also transfers information to the next time step but is more sensitive and responsive to recent inputs than the cell state.
+
+**Output Sequence**: Depending on the task, LSTMs can output at each timestep (for tasks like sequence labeling) or after processing the entire sequence (like sentiment analysis).
+
+## Transformers
+
+A neural network architecture that uses self-attention mechanisms to model relationships between input sequences. Transformers are particularly effective in NLP tasks due to their ability to handle sequences in parallel and consider all parts of the input at once.
+
+* **Usage**: NLP, machine translation, language modeling
+* **Strengths**: Excellent performance on sequential data, parallelizable, can handle long-range dependencies
+* **Caveats**: Computationally expensive, require large datasets
+```{dot}
+digraph Transformer {
+    // Graph properties
+    node [shape=record];
+
+    // Nodes definitions
+    input_tokens [label="Input Tokens" shape=ellipse];
+    embedding_layer [label="Embedding Layer" shape=box];
+    positional_encoding [label="Add Positional Encoding" shape=box];
+    encoder [label="Encoder Stack" shape=box];
+    decoder [label="Decoder Stack" shape=box];
+    output_tokens [label="Output Tokens" shape=ellipse];
+
+    // Edges definitions
+    input_tokens -> embedding_layer;
+    embedding_layer -> positional_encoding;
+    positional_encoding -> encoder;
+    encoder -> decoder;
+    decoder -> output_tokens;
+
+    // Additional components for clarity (not actual flow)
+    encoder_output [label="Encoder Output" shape=note];
+    decoder_input [label="Decoder Input" shape=note];
+    encoder -> encoder_output [style=dashed];
+    decoder_input -> decoder [style=dashed];
+
+    // Descriptions for self-attention and cross-attention mechanisms
+    self_attention [label="Self-Attention" shape=plaintext];
+    cross_attention [label="Cross-Attention" shape=plaintext];
+    encoder -> self_attention [style=dotted];
+    decoder -> self_attention [style=dotted];
+    decoder -> cross_attention [style=dotted];
+    cross_attention -> encoder_output [style=dotted, dir=none];
+}
+```
+
+**Input Tokens**: Represents the initial sequence of tokens (e.g., words in a sentence) that are fed into the Transformer.
+
+**Embedding Layer**: Converts tokens into vectors that the model can process. Each token is mapped to a unique vector.
+
+**Positional Encoding**: Adds information about the position of each token in the sequence to the embeddings, which is crucial as Transformers do not inherently process sequential data.
+
+**Encoder Stack**: A series of encoder layers that process the input. Each layer uses self-attention mechanisms to consider all parts of the input simultaneously.
+
+**Decoder Stack**: A series of decoder layers that generate the output sequence step by step. Each layer uses both self-attention mechanisms to attend to its own output so far, and cross-attention mechanisms to focus on the output from the encoder.
+
+**Output Tokens**: The final output sequence generated by the Transformer, such as a translated sentence or the continuation of an input text.
+
+**Encoder Output and Decoder Input**: Not actual data flow, but illustrate how information is transferred from the encoder to the decoder.
+
+**Self-Attention and Cross-Attention**: These mechanisms are core features of Transformer models. Self-attention allows layers to consider other parts of the input or output at each step, while cross-attention allows the decoder to focus on relevant parts of the input sequence.
+
+## Autoencoders
+
+A neural network architecture that learns to compress and reconstruct input data. Autoencoders are typically used for dimensionality reduction tasks, as they learn to encode the essential aspects of the data in a smaller representation.
+
+* **Usage**: Dimensionality reduction, anomaly detection, generative modeling
+* **Strengths**: Excellent performance on dimensionality reduction, can learn robust representations
+* **Caveats**: May not perform well on complex data distributions
+```{dot}
+digraph Autoencoder {
+    // Graph properties
+    node [shape=record];
+
+    // Nodes definitions
+    input_data [label="Input Data" shape=ellipse];
+    encoder [label="Encoder" shape=box];
+    latent_space [label="Latent Space" shape=box];
+    decoder [label="Decoder" shape=box];
+    reconstructed_output [label="Reconstructed Output" shape=ellipse];
+
+    // Edges definitions
+    input_data -> encoder;
+    encoder -> latent_space;
+    latent_space -> decoder;
+    decoder -> reconstructed_output;
+}
+```
+
+**Input Data**: Represents the data that is fed into the Autoencoder. This could be any kind of data, such as images, text, or sound.
+
+**Encoder**: The first part of the Autoencoder that processes the input data and compresses it into a smaller, dense representation. This part typically consists of several layers that gradually reduce the dimensionality of the input.
+
+**Latent Space**: Also known as the "encoded" state or "bottleneck". This is a lower-dimensional representation of the input data and serves as the compressed "code" that the decoder will use to reconstruct the input.
+
+**Decoder**: Mirrors the structure of the encoder but in reverse. It takes the encoded data from the latent space and reconstructs the original data as closely as possible. This part typically consists of layers that gradually increase in dimensionality to match the original input size.
+
+**Reconstructed Output**: The final output of the Autoencoder. This is the reconstruction of the original input data based on the compressed code stored in the latent space. The quality of this reconstruction is often a measure of the Autoencoder’s performance.
+
+## Generative Adversarial Networks (GANs)
+
+A neural network architecture that consists of a generator and discriminator, which compete to generate realistic data. GANs are highly effective at generating new data that mimics the input data, often used in image generation and editing.
+
+* **Usage**: Generative modeling, data augmentation, style transfer
+* **Strengths**: Excellent performance on generative tasks, can generate realistic data
+* **Caveats**: Training can be unstable, require careful tuning of hyperparameters
+```{dot}
+digraph GAN {
+    // Graph properties
+    node [shape=record];
+
+    // Nodes definitions
+    noise [label="Noise vector (z)" shape=ellipse];
+    generator [label="Generator (G)" shape=box];
+    generated_image [label="Generated image (G(z))" shape=cds];
+    real_image [label="Real image (x)" shape=cds];
+    discriminator [label="Discriminator (D)" shape=box];
+    D_output_fake [label="D(G(z))" shape=ellipse];
+    D_output_real [label="D(x)" shape=ellipse];
+
+    // Edges definitions
+    noise -> generator;
+    generator -> generated_image;
+    generated_image -> discriminator [label="Fake"];
+    real_image -> discriminator [label="Real"];
+    discriminator -> D_output_fake [label="Output for fake"];
+    discriminator -> D_output_real [label="Output for real"];
+}
+
+```
+
+
+**Noise vector (z)**: Represents the random noise input to the generator.
+
+**Generator (G)**: The model that learns to generate new data with the same statistics as the training set from the noise vector.
+
+**Generated image (G(z))**: The fake data produced by the generator.
+
+**Real image (x)**: Actual data samples from the training dataset.
+
+**Discriminator (D)**: The model that learns to distinguish between real data and synthetic data generated by the Generator.
+
+**D(G(z)) and D(x)**: Outputs of the Discriminator when evaluating fake data and real data, respectively.
+
+
+The Noise vector feeds into the Generator.
+
+The Generator outputs a Generated image, which is input to the Discriminator labeled as "Fake".
+
+The Real image also feeds into the Discriminator but is labeled as "Real".
+
+The Discriminator outputs evaluations for both fake and real inputs.
+
+## Residual Networks (ResNets)
+
+A neural network architecture that uses residual connections to ease training. ResNets are particularly effective for very deep networks, as they allow for training deeper networks by providing pathways for gradients to flow through.
+
+* **Usage**: Image classification, object detection
+* **Strengths**: Excellent performance on image-related tasks, ease of training
+* **Caveats**: May not perform well on sequential data
+```{dot}
+digraph ResNet {
+    // Graph properties
+    node [shape=record];
+
+    // Nodes definitions
+    input [label="Input Image" shape=ellipse];
+    conv1 [label="Initial Conv + BN + ReLU" shape=box];
+    resblock1 [label="<f0> ResBlock | <f1> + | <f2> ReLU" shape=Mrecord];
+    resblock2 [label="<f0> ResBlock | <f1> + | <f2> ReLU" shape=Mrecord];
+    resblock3 [label="<f0> ResBlock | <f1> + | <f2> ReLU" shape=Mrecord];
+    avgpool [label="Average Pooling" shape=box];
+    fc [label="Fully Connected Layer" shape=box];
+    output [label="Output" shape=ellipse];
+
+    // Edges definitions
+    input -> conv1;
+    conv1 -> resblock1:f0;
+    resblock1:f2 -> resblock2:f0;
+    resblock2:f2 -> resblock3:f0;
+    resblock3:f2 -> avgpool;
+    avgpool -> fc;
+    fc -> output;
+
+    // Adding skip connections
+    edge [style=dashed];
+    conv1 -> resblock1:f1;
+    resblock1:f1 -> resblock2:f1;
+    resblock2:f1 -> resblock3:f1;
+}
+```
+**Input Image**: The initial input layer where images are fed into the network.
+
+**Initial Conv + BN + ReLU**: Represents an initial convolutional layer followed by batch normalization and a ReLU activation function to prepare the data for residual blocks.
+
+**ResBlock**: These are the residual blocks that define the ResNet architecture. Each block contains two parts: a sequence of convolutional layers and a skip connection that adds the input of the block to its output.
+
+**Average Pooling**: This layer averages the feature maps spatially to reduce their dimensions before passing to a fully connected layer.
+
+**Fully Connected Layer**: This layer maps the feature representations to the final output classes.
+
+**Output**: The final prediction of the network.
+
+
+## U-Net
+
+A neural network architecture that uses an encoder-decoder structure with skip connections. U-Net is designed primarily for biomedical image segmentation, where it is crucial to localize objects precisely within an image.
+
+* **Usage**: Image segmentation, object detection
+* **Strengths**: Excellent performance on image segmentation tasks, fast training
+* **Caveats**: May not perform well on sequential data
+```{dot}
+digraph UNet {
+    // Graph properties
+    node [shape=record];
+
+    // Nodes definitions
+    input [label="Input Image" shape=ellipse];
+    conv1 [label="Conv + ReLU\nDownsampling" shape=box];
+    conv2 [label="Conv + ReLU\nDownsampling" shape=box];
+    bottom [label="Conv + ReLU" shape=box];
+    upconv1 [label="UpConv + ReLU\nUpsampling" shape=box];
+    concat1 [label="Concatenate" shape=circle];
+    upconv2 [label="UpConv + ReLU\nUpsampling" shape=box];
+    concat2 [label="Concatenate" shape=circle];
+    finalconv [label="Conv + ReLU\n1x1 Conv" shape=box];
+    output [label="Output\nSegmentation Map" shape=ellipse];
+
+    // Edges definitions
+    input -> conv1;
+    conv1 -> conv2;
+    conv2 -> bottom;
+    bottom -> upconv1;
+    upconv1 -> concat1;
+    concat1 -> upconv2;
+    upconv2 -> concat2;
+    concat2 -> finalconv;
+    finalconv -> output;
+
+    // Skip connections
+    edge [style=dashed];
+    conv1 -> concat1 [label="Copy\ncrop"];
+    conv2 -> concat2 [label="Copy\ncrop"];
+}
+```
+
+**Input Image**: The initial input layer where images are fed into the network.
+
+**Conv + ReLU / Downsampling**: These blocks represent convolutional operations followed by a ReLU activation function. The "Downsampling" indicates that each block reduces the spatial dimensions of the input.
+
+**Bottom**: This is the lowest part of the U, consisting of convolutional layers without downsampling, positioned before the upsampling starts.
+
+**UpConv + ReLU / Upsampling**: These blocks perform transposed convolutions (or up-convolutions) that increase the resolution of the feature maps.
+
+**Concatenate**: These layers concatenate feature maps from the downsampling pathway with the upsampled feature maps to preserve high-resolution features for precise localization.
+
+**Final Conv**: This typically includes a 1x1 convolution to map the deep feature representations to the desired number of classes for segmentation.
+
+**Output / Segmentation Map**: The final output layer which produces the segmented image.
+
+## Attention-based Models
+
+A neural network architecture that uses attention mechanisms to focus on relevant input regions. Attention-based models are particularly effective for tasks that require understanding of complex relationships within the data, such as interpreting a document or translating a sentence.
+
+* **Usage**: NLP, machine translation, question answering
+* **Strengths**: Excellent performance on sequential data, can model long-range dependencies
+* **Caveats**: Require careful tuning of hyperparameters
+```{dot}
+digraph AttentionBasedModels {
+    // Graph properties
+    node [shape=record];
+
+    // Nodes definitions
+    input [label="Input Sequence" shape=ellipse];
+    embedding [label="Embedding Layer" shape=box];
+    positional [label="Add Positional Encoding" shape=box];
+    multihead [label="Multi-Head Attention" shape=box];
+    addnorm1 [label="Add & Norm" shape=box];
+    feedforward [label="Feedforward Network" shape=box];
+    addnorm2 [label="Add & Norm" shape=box];
+    output [label="Output Sequence" shape=ellipse];
+
+    // Edges definitions
+    input -> embedding;
+    embedding -> positional;
+    positional -> multihead;
+    multihead -> addnorm1;
+    addnorm1 -> feedforward;
+    feedforward -> addnorm2;
+    addnorm2 -> output;
+
+    // Skip connections
+    edge [style=dashed];
+    positional -> addnorm1 [label="Skip Connection"];
+    addnorm1 -> addnorm2 [label="Skip Connection"];
+}
+
+```
+**Input Sequence**: Initial data input, typically a sequence of tokens.
+
+**Embedding Layer**: Converts tokens into vectors that the model can process.
+
+**Add Positional Encoding**: Incorporates information about the position of tokens in the sequence into their embeddings, which is crucial since attention mechanisms do not inherently process sequential data.
+
+**Multi-Head Attention**: Allows the model to focus on different parts of the sequence for different representations, facilitating better understanding and processing of the input.
+
+**Add & Norm**: A layer that combines residuals (from skip connections) with the output of the attention or feedforward layers, followed by layer normalization.
+
+**Feedforward Network**: A dense neural network that processes the sequence after attention has been applied.
+
+**Output Sequence**: The final processed sequence output by the model, often used for tasks like translation, text generation, or classification.
+
+**Skip Connections**: Dashed lines represent skip connections that help to alleviate the vanishing gradient problem by allowing gradients to flow through the network directly. They also help the model to learn an identity function which ensures that the model does not lose information throughout the layers.
+
+## Graph Neural Networks (GNNs)
+
+A neural network architecture that uses graph structures to model relationships between nodes. GNNs are effective for data that can be represented as graphs, such as social networks or molecules, as they capture the relationships between entities.
+
+* **Usage**: Graph-based data, social network analysis, recommendation systems
+* **Strengths**: Excellent performance on graph-based data, can model complex relationships
+* **Caveats**: Computationally expensive, require large datasets
+```{dot}
+digraph GNN {
+    // Graph properties
+    node [shape=record];
+
+    // Nodes definitions
+    input_graph [label="Input Graph" shape=ellipse];
+    node_features [label="Node Features" shape=box];
+    edge_features [label="Edge Features" shape=box];
+    gnn_layers [label="GNN Layers" shape=box];
+    aggregate [label="Aggregate Messages" shape=box];
+    update [label="Update States" shape=box];
+    readout [label="Graph-level Readout" shape=box];
+    output [label="Output" shape=ellipse];
+
+    // Edges definitions
+    input_graph -> node_features;
+    input_graph -> edge_features;
+    node_features -> gnn_layers;
+    edge_features -> gnn_layers;
+    gnn_layers -> aggregate;
+    aggregate -> update;
+    update -> readout;
+    readout -> output;
+}
+```
+
+**Input Graph**: The initial graph input containing nodes and edges.
+
+**Node Features**: Processes the features associated with each node. These can include node labels, attributes, or other data.
+
+**Edge Features**: Processes features associated with edges in the graph, which might include types of relationships, weights, or other characteristics.
+
+**GNN Layers**: A series of graph neural network layers that apply convolution-like operations over the graph. These layers can involve message passing between nodes, where a node's new state is determined based on its neighbors.
+
+**Aggregate Messages**: Combines the information (messages) received from neighboring nodes into a single unified message. Aggregation functions can include sums, averages, or max operations.
+
+**Update States**: Updates the states of the nodes based on aggregated messages, typically using some form of neural network or transformation.
+
+**Graph-level Readout**: Aggregates node states into a graph-level representation, which can be used for tasks that require a holistic view of the graph (e.g., determining the properties of a molecule).
+
+**Output**: The final output, which can vary depending on the specific application (node classification, link prediction, graph classification, etc.).
+
+## Reinforcement Learning (RL) Architectures
+
+A neural network architecture that uses reinforcement learning to learn from interactions with an environment. RL architectures are highly effective for sequential decision-making tasks, such as playing games or navigating environments.
+
+* **Usage**: Game playing, robotics, autonomous systems
+* **Strengths**: Excellent performance on sequential decision-making tasks, can learn complex policies
+* **Caveats**: Require large datasets, can be slow to train
+```{dot}
+digraph RL {
+    // Graph properties
+    node [shape=record];
+
+    // Nodes definitions
+    environment [label="Environment" shape=ellipse];
+    state [label="State" shape=ellipse];
+    agent [label="Agent" shape=box];
+    action [label="Action" shape=ellipse];
+    reward [label="Reward" shape=ellipse];
+    updated_state [label="Updated State" shape=ellipse];
+
+    // Edges definitions
+    environment -> state;
+    state -> agent;
+    agent -> action;
+    action -> environment;
+    environment -> reward;
+    reward -> agent;
+    environment -> updated_state [label="Feedback Loop"];
+    updated_state -> state [label="New State"];
+}
+```
+
+**Environment**: This is where the agent operates. It defines the dynamics of the system including how the states transition and how rewards are assigned for actions.
+
+**State**: Represents the current situation or condition in which the agent finds itself. It is the information that the environment provides to the agent, which then bases its decisions on this data.
+
+**Agent**: This is the decision-maker. It uses a strategy, which may involve a neural network or another function approximator, to decide what actions to take based on the state it perceives.
+
+**Action**: The decision taken by the agent, which will affect the environment.
+
+**Reward**: After taking an action, the agent receives a reward (or penalty) from the environment. This reward is an indication of how good the action was in terms of achieving the goal.
+
+**Updated State**: After an action is taken, the environment transitions to a new state. This new state and the reward feedback are then used by the agent to learn and refine its strategy.
+
+## Evolutionary Neural Networks (ENNs)
+
+A neural network architecture that uses evolutionary principles to evolve neural networks. Evolutionary Neural Networks are particularly effective for optimization problems, where they can evolve solutions over generations.
+
+* **Usage**: Neuroevolution, optimization problems
+* **Strengths**: Excellent performance on optimization problems, can learn complex policies
+* **Caveats**: Computationally expensive, require large datasets
+```{dot}
+digraph ENN {
+    // Graph properties
+    node [shape=record];
+
+    // Nodes definitions
+    population [label="Initial Population\n(Neural Networks)" shape=ellipse];
+    selection [label="Selection" shape=box];
+    crossover [label="Crossover" shape=box];
+    mutation [label="Mutation" shape=box];
+    fitness [label="Fitness Evaluation" shape=box];
+    new_population [label="New Generation" shape=ellipse];
+    best_network [label="Best Performing Network" shape=ellipse, fillcolor=lightblue];
+
+    // Edges definitions
+    population -> selection;
+    selection -> crossover;
+    crossover -> mutation;
+    mutation -> fitness;
+    fitness -> new_population;
+    new_population -> selection [label="Next Generation"];
+    fitness -> best_network [label="If Optimal", style=dashed];
+
+    // Additional explanatory nodes
+    edge [style=dashed];
+    best_network -> new_population [label="Update Population", style=dotted];
+}
+```
+
+**Initial Population**: This represents the initial set of neural networks. These networks might differ in architecture, weights, or hyperparameters.
+
+**Selection**: Part of the evolutionary process where individual networks are selected based on their performance, often using a fitness function.
+
+**Crossover**: A genetic operation used to combine features from two or more parent neural networks to create offspring. This simulates sexual reproduction.
+
+**Mutation**: Introduces random variations to the offspring, potentially leading to new neural network configurations. This step enhances diversity within the population.
+
+**Fitness Evaluation**: Each network in the population is evaluated based on how well it performs the given task. The fitness often determines which networks survive and reproduce.
+
+**New Generation**: After selection, crossover, mutation, and evaluation, a new generation of neural networks is formed. This generation forms the new population for further evolution.
+
+**Best Performing Network**: Out of all generations, the network that performs best on the task.
+
+**Feedback Loops**:
+  
+  - **Next Generation**: The cycle from selection to fitness evaluation and then back to selection with the new generation is a loop that continues until a satisfactory solution (network) is found.
+  
+  - **If Optimal**: If during any fitness evaluation a network meets the predefined criteria or optimality, it may be selected as the final model.
+
+## Spiking Neural Networks (SNNs)
+
+A neural network architecture that uses spiking neurons to process data. SNNs are particularly effective for neuromorphic computing applications, where they can operate in energy-efficient ways.
+
+* **Usage**: Neuromorphic computing, edge AI
+* **Strengths**: Excellent performance on edge AI applications, energy-efficient
+* **Caveats**: Limited software support, require specialized hardware
+```{dot}
+digraph SNN {
+    // Graph properties
+    node [shape=record];
+
+    // Nodes definitions
+    input_neurons [label="Input Neurons" shape=ellipse];
+    synaptic_layers [label="Synaptic Layers\n(Weighted Connections)" shape=box];
+    spiking_neurons [label="Spiking Neurons" shape=box];
+    output_neurons [label="Output Neurons" shape=ellipse];
+    threshold_mechanism [label="Threshold Mechanism" shape=box];
+    spike_train [label="Spike Train Output" shape=ellipse];
+
+    // Edges definitions
+    input_neurons -> synaptic_layers;
+    synaptic_layers -> spiking_neurons;
+    spiking_neurons -> threshold_mechanism;
+    threshold_mechanism -> output_neurons;
+    output_neurons -> spike_train;
+
+    // Additional explanatory nodes
+    edge [style=dashed];
+    synaptic_layers -> threshold_mechanism [label="Dynamic Weights", style=dashed];
+}
+```
+
+**Input Neurons**: These neurons receive the initial input signals, which could be any time-varying signal or a pattern encoded in the timing of spikes.
+
+**Synaptic Layers**: Represents the connections between neurons. In SNNs, these connections are often dynamic, changing over time based on the activity of the network (Hebbian learning principles).
+
+**Spiking Neurons**: Neurons that operate using spikes, which are brief and discrete events typically caused by reaching a certain threshold in the neuron’s membrane potential.
+
+**Threshold Mechanism**: A critical component in SNNs that determines when a neuron should fire based on its membrane potential. This mechanism can adapt based on the history of spikes and neuronal activity.
+
+**Output Neurons**: Neurons that produce the final output of the network. These may also operate using spikes, especially in SNNs designed for specific tasks like motor control or sensory processing.
+
+**Spike Train Output**: The output from the network is often in the form of a spike train, representing the timing and sequence of spikes from the output neurons.
+
+**Dynamic Weights**: Indicates that the synaptic weights are not static and can change based on the spike timing differences between pre- and post-synaptic neurons (STDP - Spike-Timing-Dependent Plasticity).
+
+## Conditional Random Fields (CRFs)
+
+A probabilistic model that uses graphical models to model sequential data. CRFs are particularly effective for sequence labeling tasks, where they can model complex relationships between labels in a sequence.
+
+* **Usage**: NLP, sequence labeling, information extraction
+* **Strengths**: Excellent performance on sequential data, can model complex relationships
+* **Caveats**: Computationally expensive, require large datasets
+
+```{dot}
+digraph CRF {
+    // Graph properties
+    node [shape=record];
+
+    // Nodes definitions
+    input_sequence [label="Input Sequence" shape=ellipse];
+    feature_extraction [label="Feature Extraction" shape=box];
+    crf_layer [label="CRF Layer" shape=box];
+    output_labels [label="Output Labels" shape=ellipse];
+
+    // Edges definitions
+    input_sequence -> feature_extraction;
+    feature_extraction -> crf_layer;
+    crf_layer -> output_labels;
+
+    // Additional nodes for clarity
+    state_transition [label="State Transition Features" shape=plaintext];
+    feature_extraction -> state_transition [style=dotted];
+    state_transition -> crf_layer [style=dotted];
+}
+```
+
+**Input Sequence**: Represents the raw data input, such as sentences in text or other sequential data.
+
+**Feature Extraction**: Processes the input data to extract features that are relevant for making predictions. This could include lexical features, part-of-speech tags, or contextual information in a natural language processing application.
+
+**CRF Layer**: The core of the CRF model where the actual conditional random field is applied. This layer models the dependencies between labels in the sequence, considering both the input features and the labels of neighboring items in the sequence.
+
+**Output Labels**: The final output of the CRF, which provides a label for each element in the input sequence. In the context of NLP, these might be tags for named entity recognition, part-of-speech tags, etc.
+
+**State Transition Features**: This represents how CRFs utilize state transition features to model the relationships and dependencies between different labels in the sequence. These are not actual data flow but indicate the type of information that influences the CRF layer's decisions.
+
+## Mixture of Experts (MoE)
+
+A neural network architecture that consists of multiple expert networks (submodels), each specialized in different parts of the data or tasks. A gating network determines which expert(s) are most relevant for a given input. MoE is particularly effective for large-scale machine learning models, where it can dynamically route tasks to the most appropriate experts.
+
+* **Usage**: Large-scale machine learning models, task-specific adaptations, dynamic routing of tasks
+* **Strengths**: Highly scalable, capable of handling diverse tasks simultaneously, efficient use of resources by activating only relevant experts for each input.
+* **Caveats**: Complex to implement and train, requires careful tuning to balance the load across experts and avoid overfitting in individual experts.
+
+```{dot}
+digraph MoE {
+    // Graph properties
+    node [shape=record];
+
+    // Nodes definitions
+    input_data [label="Input Data" shape=ellipse];
+    gating_network [label="Gating Network" shape=box];
+    expert1 [label="Expert 1" shape=box];
+    expert2 [label="Expert 2" shape=box];
+    expert3 [label="Expert 3" shape=box];
+    combined_output [label="Combined Output" shape=ellipse];
+
+    // Edges definitions
+    input_data -> gating_network;
+    gating_network -> expert1 [label="Weight"];
+    gating_network -> expert2 [label="Weight"];
+    gating_network -> expert3 [label="Weight"];
+    expert1 -> combined_output [label="Output 1"];
+    expert2 -> combined_output [label="Output 2"];
+    expert3 -> combined_output [label="Output 3"];
+
+    // Additional explanatory nodes
+    edge [style=dashed];
+    gating_network -> combined_output [label="Decision Weights", style=dotted];
+}
+```
+
+**Input Data**: Represents the data being fed into the model. This could be anything from images, text, to structured data.
+
+**Gating Network**: A crucial component that dynamically determines which expert model should handle the given input. It evaluates the input data and allocates weights to different experts based on their relevance to the current data point.
+
+**Experts**: These are specialized models (expert1, expert2, expert3) that are trained on subsets of the data or specific types of tasks. Each expert processes the input independently.
+
+**Combined Output**: The final output of the MoE model, which typically involves aggregating the outputs of the experts weighted by the gating network’s decisions.
+
+**Weights**: These edges show how the gating network influences the contribution of each expert to the final decision. The weights are not fixed but are determined dynamically based on each input.
+
+**Output 1, 2, 3**: These labels on the edges from experts to the combined output represent the contribution of each expert to the final model output. Each expert contributes its processed output, which is then combined based on the weights provided by the gating network.

src/theory/layers.qmd

@@ -1,103 +1,118 @@
+---
+title: Layer Types
+format:
+  html:
+    mermaid:
+      theme: default
+---
 
-## **1. Input Layers**
+Neural networks are complex architectures made up of various types of layers, each performing distinct functions that contribute to the network's ability to learn from data. Understanding the different types of layers and their specific roles is essential for designing effective neural network models. This knowledge not only helps in building tailored architectures for different tasks but also aids in optimizing performance and efficiency.
+
+Each layer in a neural network processes the input data in a unique way, and the choice of layers depends on the problem at hand. For instance, convolutional layers are primarily used in image processing tasks due to their ability to capture spatial hierarchies, while recurrent layers are favored in tasks involving sequential data like natural language processing or time series analysis due to their ability to maintain a memory of previous inputs.
+
+The structure of a neural network can be seen as a stack of layers where each layer feeds into the next, transforming the input step-by-step into a more abstract and ultimately useful form. The output of each layer becomes the input for the next until a final output is produced. This modular approach allows for the construction of deep learning models that can handle a wide range of complex tasks, from speech recognition and image classification to generating coherent text and beyond.
+
+In the sections that follow, we will explore various types of layers commonly used in neural networks, discussing their usage, descriptions, strengths, and weaknesses. This will include foundational layers like input and dense layers, as well as more specialized ones like convolutional, recurrent, and attention layers. We'll also look at layers designed for specific functions such as normalization, regularization, and activation, each vital for enhancing the network's learning capability and stability. This comprehensive overview will provide a clearer understanding of how each layer works and how they can be combined to create powerful neural network models.
+
+## Input Layers
 
 * Usage: Receive input data, propagate it to subsequent layers
 * Description: The first layer in a neural network that receives input data
 * Strengths: Essential for processing input data, easy to implement
 * Weaknesses: Limited functionality, no learning occurs in this layer
 
-## **2. Dense Layers (Fully Connected Layers)**
+## Dense Layers (Fully Connected Layers)
 
 * Usage: Feature extraction, classification, regression
 * Description: A layer where every input is connected to every output, using a weighted sum
 * Strengths: Excellent for feature extraction, easy to implement, fast computation
 * Weaknesses: Can be prone to overfitting, computationally expensive for large inputs
 
-## **3. Convolutional Layers (Conv Layers)**
+## Convolutional Layers (Conv Layers)
 
 * Usage: Image classification, object detection, image segmentation
 * Description: A layer that applies filters to small regions of the input data, scanning the input data horizontally and vertically
 * Strengths: Excellent for image processing, reduces spatial dimensions, retains spatial hierarchy
 * Weaknesses: Computationally expensive, require large datasets
 
-## **4. Pooling Layers (Downsampling Layers)**
+## Pooling Layers (Downsampling Layers)
 
 * Usage: Image classification, object detection, image segmentation
 * Description: A layer that reduces spatial dimensions by taking the maximum or average value across a region
 * Strengths: Reduces spatial dimensions, reduces number of parameters, retains important features
 * Weaknesses: Loses some information, can be sensitive to hyperparameters
 
-## **5. Recurrent Layers (RNNs)**
+## Recurrent Layers (RNNs)
 
 * Usage: Natural Language Processing (NLP), sequence prediction, time series forecasting
 * Description: A layer that processes sequential data, using hidden state to capture temporal dependencies
 * Strengths: Excellent for sequential data, can model long-term dependencies
 * Weaknesses: Suffers from vanishing gradients, difficult to train, computationally expensive
 
-## **6. Long Short-Term Memory (LSTM) Layers**
+## Long Short-Term Memory (LSTM) Layers
 
 * Usage: NLP, sequence prediction, time series forecasting
 * Description: A type of RNN that uses memory cells to learn long-term dependencies
 * Strengths: Excellent for sequential data, can model long-term dependencies, mitigates vanishing gradients
 * Weaknesses: Computationally expensive, require large datasets
 
-## **7. Gated Recurrent Unit (GRU) Layers**
+## Gated Recurrent Unit (GRU) Layers
 
 * Usage: NLP, sequence prediction, time series forecasting
 * Description: A simpler alternative to LSTM, using gates to control the flow of information
 * Strengths: Faster computation, simpler than LSTM, easier to train
 * Weaknesses: May not perform as well as LSTM, limited capacity to model long-term dependencies
 
-## **8. Batch Normalization Layers**
+## Batch Normalization Layers
 
 * Usage: Normalizing inputs, stabilizing training, improving performance
 * Description: A layer that normalizes inputs, reducing internal covariate shift
 * Strengths: Improves training stability, accelerates training, improves performance
 * Weaknesses: Requires careful tuning of hyperparameters, can be computationally expensive
 
-## **9. Dropout Layers**
+## Dropout Layers
 
 * Usage: Regularization, preventing overfitting
 * Description: A layer that randomly drops out neurons during training, reducing overfitting
 * Strengths: Effective regularization technique, reduces overfitting, improves generalization
 * Weaknesses: Can slow down training, requires careful tuning of hyperparameters
 
-## **10. Flatten Layers**
+## Flatten Layers
 
 * Usage: Reshaping data, preparing data for dense layers
 * Description: A layer that flattens input data into a one-dimensional array
 * Strengths: Essential for preparing data for dense layers, easy to implement
 * Weaknesses: Limited functionality, no learning occurs in this layer
 
-## **11. Embedding Layers**
+## Embedding Layers
 
 * Usage: NLP, word embeddings, language modeling
 * Description: A layer that converts categorical data into dense vectors
 * Strengths: Excellent for NLP tasks, reduces dimensionality, captures semantic relationships
 * Weaknesses: Require large datasets, can be computationally expensive
 
-## **12. Attention Layers**
+## Attention Layers
 
 * Usage: NLP, machine translation, question answering
 * Description: A layer that computes weighted sums of input data, focusing on relevant regions
 * Strengths: Excellent for sequential data, can model long-range dependencies, improves performance
 * Weaknesses: Computationally expensive, require careful tuning of hyperparameters
 
-## **13. Upsampling Layers**
+## Upsampling Layers
 
 * Usage: Image segmentation, object detection, image generation
 * Description: A layer that increases spatial dimensions, using interpolation or learned upsampling filters
 * Strengths: Excellent for image processing, improves spatial resolution, enables image generation
 * Weaknesses: Computationally expensive, require careful tuning of hyperparameters
 
-## **14. Normalization Layers**
+## Normalization Layers
 
 * Usage: Normalizing inputs, stabilizing training, improving performance
 * Description: A layer that normalizes inputs, reducing internal covariate shift
 * Strengths: Improves training stability, accelerates training, improves performance
 * Weaknesses: Requires careful tuning of hyperparameters, can be computationally expensive
 
-## **15. Activation Functions**
+## Activation Functions
 
 * Usage: Introducing non-linearity, enhancing model capacity
 * Description: A function that introduces non-linearity into the model, enabling complex representations

src/theory/metrics.qmd

@@ -1,150 +1,574 @@
-# **Metrics for Model Performance Monitoring and Validation**
+---
+title: Metrics
+---
 
 In machine learning, it's essential to evaluate the performance of a model to ensure it's accurate, reliable, and effective. There are various metrics to measure model performance, each with its strengths and limitations. Here's an overview of popular metrics, their pros and cons, and examples of tasks that apply to each.
 
-## **1. Mean Squared Error (MSE)**
+For a quick [Overview](#overview) of metrics by use cases.
+
+
+## Accuracy
+
+Accuracy measures the ratio of correctly predicted observation to the total observations.
+
+::: columns
+::: {.column width="50%"}
+Pros:
+
+* Simplest and most intuitive metric.
+::: 
+::: {.column width="50%"}
+Cons:
+
+* Can be misleading in the presence of imbalanced classes.
+::: 
+::: 
+Example tasks:
+
+* General classification tasks where classes are balanced.
+* Entry-level benchmarking
+
+
+
+## Mean Squared Error (MSE)
 
 MSE measures the average squared difference between predicted and actual values.
 
+
+::: columns
+::: {.column width="50%"}
+
 Pros:
 
 * Easy to calculate
 * Sensitive to outliers
-
+:::
+::: {.column width="50%"}
 Cons:
 
 * Can be heavily influenced by extreme values
-
+:::
+:::
 Example tasks:
 
 * Regression tasks, such as predicting house prices or stock prices
 * Time series forecasting
 
-## **2. Mean Absolute Error (MAE)**
+## Mean Absolute Error (MAE)
 
 MAE measures the average absolute difference between predicted and actual values.
 
+::: columns
+::: {.column width="50%"}
 Pros:
 
 * Robust to outliers
 * Easy to interpret
-
+:::
+::: {.column width="50%"}
 Cons:
 
 * Can be sensitive to skewness in the data
-
+:::
+:::
 Example tasks:
 
 * Regression tasks, such as predicting house prices or stock prices
 * Time series forecasting
 
-## **3. Mean Absolute Percentage Error (MAPE)**
+## Mean Absolute Percentage Error (MAPE)
 
 MAPE measures the average absolute percentage difference between predicted and actual values.
 
+::: columns
+::: {.column width="50%"}
 Pros:
 
 * Easy to interpret
 * Sensitive to relative errors
-
+:::
+::: {.column width="50%"}
 Cons:
 
 * Can be sensitive to outliers
-
+:::
+:::
 Example tasks:
 
 * Regression tasks, such as predicting house prices or stock prices
 * Time series forecasting
 
-## **4. R-Squared (R²)**
+
+## Binary Crossentropy
+
+Binary Crossentropy measures the error rate between the true label and the predicted probability for binary classification tasks.
+
+::: columns
+::: {.column width="50%"}
+Pros:
+
+* Effective for measuring performance on binary classification problems.
+* Directly optimizes the classification threshold.
+
+::: 
+::: {.column width="50%"}
+Cons:
+
+* Not suitable for multi-class classification tasks.
+* Sensitive to the balance of classes in the dataset.
+::: 
+:::
+Example tasks:
+
+* Predicting customer churn (yes or no).
+* Medical diagnostics with binary outcomes.
+
+## Categorical Crossentropy
+
+Categorical Crossentropy is used to measure the error rate between the true label and the predicted probabilities for multi-class classification tasks where each class is mutually exclusive.
+
+::: columns
+::: {.column width="50%"}
+Pros:
+
+* Well-suited for multi-class classification problems.
+* Directly optimizes probability distribution across all classes.
+
+::: 
+::: {.column width="50%"}
+Cons:
+
+* Requires one-hot encoding of labels.
+* Can be computationally intensive with many class labels.
+::: 
+:::
+Example tasks:
+
+* Image classification with multiple classes.
+* Text classification into various categories.
+
+## Sparse Categorical Crossentropy
+
+Sparse Categorical Crossentropy is similar to Categorical Crossentropy but used for multi-class classification tasks where the classes are encoded as integers, not one-hot vectors.
+
+::: columns
+::: {.column width="50%"}
+Pros:
+
+* More memory efficient than Categorical Crossentropy when dealing with many classes.
+* Eliminates the need for one-hot encoding of labels.
+
+::: 
+::: {.column width="50%"}
+Cons:
+
+* Not suitable for problems where multiple categories may be applicable to a single observation.
+* Requires careful handling of class indices to ensure they are mapped correctly.
+::: 
+:::
+Example tasks:
+
+* Classifying news articles into a predefined set of topics.
+* Recognizing the type of product from its description in a single-label system.
+
+
+## R-Squared (R²)
 
 R² measures the proportion of variance in the dependent variable that's explained by the independent variables.
 
+::: columns
+::: {.column width="50%"}
 Pros:
 
 * Easy to interpret
 * Sensitive to the strength of the relationship
 
+::: 
+::: {.column width="50%"}
 Cons:
 
 * Can be sensitive to outliers
 * Can be misleading for non-linear relationships
+::: 
+::: 
 
 Example tasks:
 
 * Regression tasks, such as predicting house prices or stock prices
 * Feature selection
 
-## **5. Brier Score**
+## Brier Score
 
 The Brier Score measures the average squared difference between predicted and actual probabilities.
 
+::: columns
+::: {.column width="50%"}
 Pros:
 
 * Sensitive to the quality of the predictions
 * Can handle multi-class classification tasks
-
+::: 
+::: {.column width="50%"}
 Cons:
 
 * Can be sensitive to the choice of threshold
+:::
+::: 
 
 Example tasks:
 
 * Multi-class classification tasks, such as image classification
 * Multi-label classification tasks
 
-## **6. F1 Score**
+## F1 Score
 
 The F1 Score measures the harmonic mean of precision and recall.
 
+::: columns
+::: {.column width="50%"}
 Pros:
 
 * Sensitive to the balance between precision and recall
 * Can handle imbalanced datasets
 
+::: 
+::: {.column width="50%"}
 Cons:
 
 * Can be sensitive to the choice of threshold
 
+::: 
+::: 
+
 Example tasks:
 
 * Binary classification tasks, such as spam detection
 * Multi-class classification tasks
 
-## **7. Matthews Correlation Coefficient (MCC)**
+## Matthews Correlation Coefficient (MCC)
 
 MCC measures the correlation between predicted and actual labels.
 
+::: columns
+::: {.column width="50%"}
 Pros:
 
 * Sensitive to the quality of the predictions
 * Can handle imbalanced datasets
-
+::: 
+::: {.column width="50%"}
 Cons:
 
 * Can be sensitive to the choice of threshold
+::: 
+::: 
 
 Example tasks:
 
 * Binary classification tasks, such as spam detection
 * Multi-class classification tasks
 
-## **8. Log Loss**
+## Log Loss
 
 Log Loss measures the average log loss between predicted and actual probabilities.
 
+::: columns
+::: {.column width="50%"}
 Pros:
 
 * Sensitive to the quality of the predictions
 * Can handle multi-class classification tasks
-
+::: 
+::: {.column width="50%"}
 Cons:
 
 * Can be sensitive to the choice of threshold
+::: 
+::: 
 
 Example tasks:
 
 * Multi-class classification tasks, such as image classification
 * Multi-label classification tasks
 
-When choosing a metric, consider the specific task, data characteristics, and desired outcome. It's essential to understand the strengths and limitations of each metric to ensure accurate model evaluation.
+## Area Under the Receiver Operating Characteristic Curve (AUC-ROC)
+
+The AUC-ROC curve is a performance measurement for classification problems at various threshold settings.
+
+::: columns
+::: {.column width="50%"}
+Pros:
+
+* Measures the ability of the model to discriminate between classes.
+* Useful for binary classification problems.
+::: 
+::: {.column width="50%"}
+Cons:
+
+* Less effective when dealing with highly imbalanced datasets.
+* Does not differentiate between types of errors.
+::: 
+::: 
+
+Example tasks:
+
+* Medical diagnosis classification.
+* Spam detection in emails.
+
+## Area Under the Precision-Recall Curve (AUC-PR)
+
+The AUC-PR summarizes the precision-recall curve as a single number, which is the weighted average of precisions achieved at each threshold.
+
+::: columns
+::: {.column width="50%"}
+Pros:
+
+* Focuses on the positive class more than AUC-ROC, useful in imbalanced datasets.
+::: 
+::: {.column width="50%"}
+Cons:
+
+* More complex to interpret and explain than AUC-ROC.
+:::
+::: 
+
+Example tasks:
+
+* Information retrieval.
+* Ranking tasks where positive class prevalence is low.
+
+
+## Precision and Recall
+
+Precision measures the ratio of correctly predicted positive observations to the total predicted positives. Recall measures the ratio of correctly predicted positive observations to all observations in actual class.
+
+::: columns
+::: {.column width="50%"}
+Pros:
+
+* Provide a more detailed understanding of model performance than accuracy, especially in imbalanced datasets.
+
+::: 
+::: {.column width="50%"}
+Cons:
+
+* High precision can sometimes be achieved at the expense of recall, and vice versa (precision-recall trade-off).
+:::
+:::
+
+Example tasks:
+
+* Document classification.
+* Customer churn prediction.
+
+
+## Cohen’s Kappa
+
+Cohen’s Kappa measures the agreement between two raters who each classify N items into C mutually exclusive categories.
+
+::: columns
+::: {.column width="50%"}
+Pros:
+
+* Accounts for the possibility of the agreement occurring by chance.
+* More robust than simple accuracy.
+::: 
+::: {.column width="50%"}
+Cons:
+
+* Can be difficult to interpret, especially with multiple classes.
+::: 
+::: 
+
+Example tasks:
+
+* Multi-rater reliability test.
+* Medical image classification where multiple doctors provide diagnoses.
+
+Here's the explanation of the additional metrics (BLEU, ROUGE, METEOR, MCC, Precision@k, Recall@k) in the format used previously in your document.
+
+---
+
+## BLEU (Bilingual Evaluation Understudy)
+
+BLEU is a metric used to evaluate the quality of text which has been machine-translated from one natural language to another.
+
+::: columns
+::: {.column width="50%"}
+Pros:
+
+* Provides a quantitative measure for the quality of machine translation.
+* Widely used and easy to compute.
+
+:::
+::: {.column width="50%"}
+Cons:
+
+* May not capture the fluency and grammaticality of the translation.
+* Focuses more on precision than recall.
+
+:::
+:::
+Example tasks:
+
+* Machine translation evaluation.
+* Automated text generation assessment.
+
+## ROUGE (Recall-Oriented Understudy for Gisting Evaluation)
+
+ROUGE is used to evaluate automatic summarization of texts as well as machine translation.
+
+::: columns
+::: {.column width="50%"}
+Pros:
+
+* Measures both the quality and quantity of match between reference and generated summaries.
+* Supports evaluation of multiple types of summaries (extractive, abstractive).
+
+:::
+::: {.column width="50%"}
+Cons:
+
+* Can be biased towards extractive summarization techniques.
+* Depends heavily on the quality of reference summaries.
+
+:::
+:::
+Example tasks:
+
+* Text summarization.
+* Machine translation evaluation.
+
+## METEOR (Metric for Evaluation of Translation with Explicit ORdering)
+
+METEOR is a metric for evaluating machine translation that extends beyond simple overlap of vocabulary.
+
+::: columns
+::: {.column width="50%"}
+Pros:
+
+* Incorporates synonyms, stemming, and paraphrase matching.
+* Aligns closely with human judgment compared to BLEU.
+
+:::
+::: {.column width="50%"}
+Cons:
+
+* More complex to compute than BLEU.
+* Can require extensive linguistic resources.
+
+:::
+:::
+Example tasks:
+
+* Machine translation.
+* Natural language generation.
+
+## Matthews Correlation Coefficient (MCC)
+
+MCC is a measure of the quality of binary classifications.
+
+::: columns
+::: {.column width="50%"}
+Pros:
+
+* Provides a balanced measure even if classes are of very different sizes.
+* Considers true and false positives and negatives.
+
+:::
+::: {.column width="50%"}
+Cons:
+
+* Not applicable to multi-class classifications.
+* Can be difficult to interpret in non-binary classification contexts.
+
+:::
+:::
+Example tasks:
+
+* Binary classification tasks such as spam detection.
+* Medical imaging classification.
+
+## Precision-k
+
+Precision-k measures the proportion of relevant items found in the top-k recommendations of a system.
+
+::: columns
+::: {.column width="50%"}
+Pros:
+
+* Useful for evaluating ranking systems where the order of items is important.
+* Easy to understand and implement.
+
+:::
+::: {.column width="50%"}
+Cons:
+
+* Does not consider the actual ranking of items beyond the cutoff of k.
+* Sensitive to the choice of k.
+
+:::
+:::
+Example tasks:
+
+* Recommender systems.
+* Search engine result ranking.
+
+## Recall-k
+
+Recall-k measures the proportion of relevant items that appear in the top-k recommendations out of all relevant items.
+
+::: columns
+::: {.column width="50%"}
+Pros:
+
+* Indicates the ability of a system to retrieve highly relevant items.
+* Useful for systems where retrieving most relevant items is critical.
+
+:::
+::: {.column width="50%"}
+Cons:
+
+* Like precision-k, it is sensitive to the choice of k.
+* Does not account for the relevance of items outside the top k.
+
+:::
+:::
+Example tasks:
+
+* Information retrieval.
+* Content-based filtering in recommender systems.
+
+
+## Overview
+Below is a table outlining typical example cases in machine learning and the corresponding metrics that could be effectively used for evaluating model performance in each case. This table helps in quickly identifying which metrics are most applicable for a given type of task.
+
+| Use Case                                 | Relevant Metrics                                                                      |
+|------------------------------------------|---------------------------------------------------------------------------------------|
+| **Predicting House Prices**              | MSE, MAE, R²                                                                          |
+| **Stock Price Forecasting**              | MSE, MAE, R²                                                                          |
+| **Medical Diagnosis Classification**     | Accuracy, F1 Score, AUC-ROC, Cohen’s Kappa                                            |
+| **Spam Detection**                       | Precision, Recall, F1 Score, MCC, AUC-ROC                                             |
+| **Image Classification**                 | Accuracy, F1 Score, Brier Score, Log Loss, AUC-ROC                                    |
+| **Multi-label Classification**           | F1 Score, Brier Score, Log Loss, AUC-PR                                               |
+| **Customer Churn Prediction**            | Accuracy, Precision, Recall, F1 Score                                                 |
+| **Document Classification**              | Accuracy, Precision, Recall, F1 Score                                                 |
+| **Ranking Tasks**                        | AUC-PR, Log Loss                                                                      |
+| **Multi-rater Reliability Tests**        | Cohen’s Kappa                                                                         |
+| **Multi-class Text Classification**      | Categorical Crossentropy, Sparse Categorical Crossentropy, Accuracy, F1 Score         |
+| **Binary Classification of Customer Data** | Binary Crossentropy, Accuracy, Precision, Recall, F1 Score                          |
+| **Sentiment Analysis**                   | Accuracy, Precision, Recall, F1 Score, MAE, Log Loss                                  |
+| **Time Series Forecasting**              | MSE, MAE, R², MAPE                                                                    |
+| **Anomaly Detection**                    | F1 Score, Precision, Recall, MCC                                                      |
+| **Search Engine Ranking**                | Precision-k, Recall-k, AUC-ROC, AUC-PR                                                |
+| **Natural Language Generation**          | BLEU, ROUGE, METEOR                                                                   |
+| **Recommendation Systems**               | Precision, Recall, F1 Score, AUC-ROC                                                  |
+| **Real-time Bidding in Advertising**     | Log Loss, ROC AUC                                                                     |
+| **User Engagement Prediction**           | MSE, MAE, Log Loss, Brier Score                                                       |
+| **Protein Structure Prediction**         | Accuracy, MCC, F1 Score                                                               |
+| **Drug Response Modeling**               | MSE, MAE, R², Log Loss                                                                |
+| **Credit Scoring**                       | AUC-ROC, Precision, Recall, F1 Score, Log Loss                                        |
+| **Fraud Detection**                      | Accuracy, Precision, Recall, F1 Score, AUC-ROC                                        |
+
+Each task is paired with metrics that are most appropriate based on the nature of the data and the specific requirements of the task.
+This approach helps ensure that the evaluation of a model is not only accurate but also relevant to the goals of the analysis.

src/theory/optimizers.qmd

@@ -0,0 +1,89 @@
+---
+title: Optimizers
+---
+
+Optimizers play a critical role in training neural networks by updating the network's weights based on the loss gradient. The choice of an optimizer can significantly impact the speed and quality of training, making it a fundamental component of deep learning. This page explores various types of optimizers, their mechanisms, and their applications, providing insights into how they work and why certain optimizers are preferred for specific tasks.
+
+[Slideshow](optimizers_slideshow.qmd)
+
+## The Role of Optimizers
+
+The primary function of an optimizer is to minimize (or maximize) a loss function or objective function that measures how well the model performs on a given task. This is achieved by iteratively adjusting the weights of the network. Optimizers not only help in converging to a solution more quickly but also affect the stability and quality of the model. They navigate the complex, high-dimensional landscape formed by a model's weights and aim to find a combination that results in the best possible predictions.
+
+## Types of Optimizers
+
+### Gradient Descent
+
+The simplest form of an optimizer, which updates the weights by moving in the direction of the negative gradient of the objective function with respect to the network's weights.
+
+* **Usage**: Basic learning tasks, small datasets
+* **Strengths**: Simple, easy to understand and implement
+* **Caveats**: Slow convergence, sensitive to the choice of learning rate, can get stuck in local minima
+
+### Stochastic Gradient Descent (SGD)
+
+An extension of the gradient descent algorithm that updates the model's weights using only a single sample or a mini-batch of samples, which makes the training process much faster.
+
+* **Usage**: General machine learning and deep learning tasks
+* **Strengths**: Faster convergence than standard gradient descent, less memory intensive
+* **Caveats**: Variability in the training updates can lead to unstable convergence
+
+### Momentum
+
+SGD with momentum considers the past gradients to smooth out the update. It helps accelerate SGD in the relevant direction and dampens oscillations.
+
+* **Usage**: Deep networks, training with high variability or sparse gradients
+* **Strengths**: Faster convergence than SGD, reduces oscillations in updates
+* **Caveats**: Additional hyperparameter to tune (momentum coefficient)
+
+### Nesterov Accelerated Gradient (NAG)
+
+A variant of the momentum method that helps to speed up training. NAG first makes a big jump in the direction of the previous accumulated gradient, then measures the gradient where it ends up and makes a correction.
+
+* **Usage**: Convolutional neural networks, large-scale neural networks
+* **Strengths**: Often converges faster than momentum
+* **Caveats**: Can overshoot in settings with noisy data
+
+### Adagrad
+
+An algorithm that adapts the learning rate to the parameters, performing larger updates for infrequent parameters and smaller updates for frequent parameters. Useful for sparse data.
+
+* **Usage**: Sparse datasets, NLP and image recognition
+* **Strengths**: Removes the need to manually tune the learning rate
+* **Caveats**: The accumulated squared gradients in the denominator can cause the learning rate to shrink and become extremely small
+
+### RMSprop
+
+Addresses the radically diminishing learning rates of Adagrad by using a moving average of squared gradients to normalize the gradient. This ensures that the learning rate does not decrease too quickly.
+
+* **Usage**: Non-stationary objectives, training RNNs
+* **Strengths**: Balances the step size decrease, making it more robust
+* **Caveats**: Still requires setting a learning rate
+
+### Adam (Adaptive Moment Estimation)
+
+Combines the advantages of Adagrad and RMSprop and calculates an exponential moving average of the gradients and the squared gradients. It can handle non-stationary objectives and problems with very noisy and/or sparse gradients.
+
+* **Usage**: Broad range of applications from general machine learning to deep learning
+* **Strengths**: Computationally efficient, little memory requirement, well suited for problems with lots of data and/or parameters
+* **Caveats**: Can sometimes lead to suboptimal solutions for some problems
+
+### AdamW
+
+AdamW is a variant of the Adam optimizer that incorporates weight decay directly into the optimization process. By decoupling the weight decay from the optimization steps, AdamW tends to outperform the standard Adam, especially in settings where regularizing and preventing overfitting are crucial.
+
+* **Usage**: Training deep neural networks across a wide range of tasks including classification and regression where regularization is key.
+* **Strengths**: Addresses some of the issues found in Adam related to poor generalization performance. It provides a more effective way to use L2 regularization, avoiding common pitfalls of Adam related to the scale of the updates.
+* **Caveats**: Like Adam, it requires tuning of hyperparameters such as the learning rate and weight decay coefficients. It may still suffer from some of the convergence issues inherent to adaptive gradient methods but to a lesser extent.
+
+### AdaMax, Nadam
+
+Variations of Adam with modifications for better convergence in specific scenarios.
+
+* **Usage**: Specific optimizations where Adam shows suboptimal behavior
+* **Strengths**: Provides alternative ways to scale the learning rates
+* **Caveats**: Can be more sensitive to hyperparameter settings
+
+## Conclusion
+
+Choosing the right optimizer is crucial as it directly influences the efficiency and outcome of training neural networks. While some optimizers are better suited for large datasets and models, others might be designed to handle specific types of data or learning tasks more effectively. Understanding the strengths and limitations of each optimizer helps in selecting the most appropriate one for a given problem, leading to better performance and more robust models.

src/theory/optimizers_slideshow.qmd

@@ -0,0 +1,100 @@
+---
+title: "Optimizers in Neural Networks"
+author: "Sébastien De Greef"
+format:
+  revealjs: 
+    theme: solarized
+    navigation-mode: grid 
+    controls-layout: bottom-right
+    controls-tutorial: true
+notebook-links: false
+crossref:
+  lof-title: "List of Figures"
+number-sections: false
+---
+
+## Introduction to Optimizers
+
+Optimizers are crucial for training neural networks by updating the network's weights based on the loss gradient. They impact the training speed, quality, and the model's final performance.
+
+---
+
+## Role of Optimizers
+
+- **Function**: Minimize the loss function
+- **Mechanism**: Iteratively adjust the weights
+- **Impact**: Affect efficiency, accuracy, and model feasibility
+
+---
+
+## Gradient Descent
+
+- **Usage**: Basic learning tasks, small datasets
+- **Strengths**: Simple, easy to understand
+- **Caveats**: Slow convergence, sensitive to learning rate settings
+
+---
+
+## Stochastic Gradient Descent (SGD)
+
+- **Usage**: General learning tasks
+- **Strengths**: Faster than batch gradient descent
+- **Caveats**: Higher variance in updates
+
+---
+
+## Momentum
+
+- **Usage**: Training deep networks
+- **Strengths**: Accelerates SGD, dampens oscillations
+- **Caveats**: Additional hyperparameter (momentum)
+
+---
+
+## Nesterov Accelerated Gradient (NAG)
+
+- **Usage**: Large-scale neural networks
+- **Strengths**: Faster convergence than Momentum
+- **Caveats**: Can overshoot in noisy settings
+
+---
+
+## Adagrad
+
+- **Usage**: Sparse data problems like NLP and image recognition
+- **Strengths**: Adapts the learning rate to the parameters
+- **Caveats**: Shrinking learning rate over time
+
+---
+
+## RMSprop
+
+- **Usage**: Non-stationary objectives, training RNNs
+- **Strengths**: Balances decreasing learning rates
+- **Caveats**: Still requires learning rate setting
+
+---
+
+## Adam (Adaptive Moment Estimation)
+
+- **Usage**: Broad range of deep learning tasks
+- **Strengths**: Efficient, handles noisy/sparse gradients well
+- **Caveats**: Complex hyperparameter tuning
+
+---
+
+## AdamW
+
+- **Usage**: Regularization heavy tasks
+- **Strengths**: Better generalization than Adam
+- **Caveats**: Requires careful tuning of decay terms
+
+---
+
+## Conclusion
+
+Choosing the right optimizer is crucial for training efficiency and model performance. 
+
+Each optimizer has its strengths and is suited for specific types of tasks.
+
+

src/theory/training.qmd

@@ -0,0 +1,191 @@
+---
+title: Training
+---
+
+Training is a fundamental aspect of developing artificial intelligence (AI) systems. It involves teaching AI models to make predictions or decisions based on data. This process is crucial for AI models to learn from experiences and improve over time. This exhaustive article delves into the various facets of training in AI, including its principles, methods, types of training data, challenges, and best practices.
+
+[The spelled-out intro to neural networks and backpropagation from Andrej Karpathy](https://www.youtube.com/watch?v=VMj-3S1tku0&t=1734s)
+
+## Principles of AI Training
+
+**1. Learning from Data**: At the core of AI training is the principle that models learn from data. The quality, quantity, and relevance of the data directly influence how well an AI model can perform its tasks.
+
+**2. Generalization**: The ultimate goal of training an AI model is for it to generalize from its training data to new, unseen situations. Generalization ensures that the model performs well not just on the data it was trained on but also on new data.
+
+**3. Overfitting and Underfitting**: Two common challenges in training AI models are overfitting and underfitting. Overfitting occurs when a model learns the training data too well, including the noise and errors, and performs poorly on new data. Underfitting happens when a model is too simple to learn the underlying pattern of the data.
+
+## Methods of AI Training
+
+**1. Supervised Learning**: This is the most common training method where the model learns from a labeled dataset. It tries to learn a function that, given a set of inputs, produces the correct output.
+
+**2. Unsupervised Learning**: In unsupervised learning, the model learns from data without labels. The goal is to identify patterns and relationships in the data.
+
+**3. Semi-Supervised Learning**: This method combines a small amount of labeled data with a large amount of unlabeled data during training. It is useful when labeling data is expensive or time-consuming.
+
+**4. Reinforcement Learning**: Here, an agent learns to make decisions by performing actions in an environment to maximize some notion of cumulative reward.
+
+**5. Transfer Learning**: In transfer learning, a model developed for a specific task is reused as the starting point for a model on a second task. It is an effective strategy when you have a small amount of data for the second task.
+
+## Types of Training Data
+
+**1. Labeled Data**: Data that has been tagged with one or more labels identifying certain properties or classifications used in supervised learning.
+
+**2. Unlabeled Data**: Data that does not contain labels, used mainly in unsupervised learning setups.
+
+**3. Synthetic Data**: Artificially created data that mimics real-world data, useful when training data is insufficient or hard to collect.
+
+**4. Augmented Data**: Real data that has been modified or expanded through techniques such as rotation, scaling, or cropping to improve the robustness of the model.
+
+## Challenges in AI Training
+
+**1. Data Quality**: Poor quality data can lead the AI to make incorrect predictions. Ensuring data is clean, representative, and well-prepared is crucial.
+
+**2. Scalability**: As models and data grow, it becomes challenging to scale training processes efficiently.
+
+**3. Bias**: AI systems can inadvertently learn and perpetuate biases present in the training data.
+
+**4. Computational Resources**: Training state-of-the-art AI models often requires significant computational resources, which can be expensive and energy-intensive.
+
+## Best Practices in AI Training
+
+**1. Data Preprocessing**: Clean and preprocess data to improve quality and training efficiency.
+
+**2. Model Selection**: Choose the right model based on the complexity of the task and the nature of the data.
+
+**3. Regularization Techniques**: Use techniques such as dropout, L1 and L2 regularization to prevent overfitting.
+
+**4. Cross-validation**: Use cross-validation techniques to ensure the model's performance is consistent across different subsets of the data.
+
+**5. Continuous Monitoring and Updating**: Regularly update the model with new data to adapt to changes in the underlying data distribution.
+
+**6. Ethical Considerations**: Address ethical considerations, ensuring the AI system does not harm and works fairly.
+
+Training is a dynamic and ongoing process in AI development. Understanding its intricacies helps in designing models that are not only accurate but also robust and fair, capable of performing well across a wide range of scenarios. This understanding forms the backbone of successful AI implementations, paving the way for innovative applications that can truly leverage the power of artificial intelligence.
+
+## Diagnosing the Training Process in AI
+
+Diagnosing the training process is crucial for developing effective AI models. It involves monitoring the model during training to identify and resolve issues that can negatively impact its performance. Here, we cover how to recognize and troubleshoot common problems like vanishing and exploding gradients, as well as unusual observations in training metrics such as loss and validation loss.
+
+### Common Training Issues
+
+**1. Vanishing Gradients**: This occurs when gradients, used in the training process to update weights, become very small, effectively preventing weights from changing their values. As a result, the training process stalls.
+
+**Causes**: 
+
+- Deep networks with many layers using non-linear activation functions that squish input values into a small range, like the sigmoid or tanh functions.
+
+- Improperly initialized weights.
+
+**Diagnosis**: 
+
+- Monitor the gradients during training. If the gradients are consistently near zero, vanishing gradients may be occurring.
+
+- Use histogram summaries in TensorBoard or similar tools to visualize layer outputs and weights during training.
+
+**Solutions**:
+
+- Use ReLU or variants of ReLU which are less likely to cause vanishing gradients because they do not squish large input values.
+
+- Implement better weight initialization strategies, like He or Glorot initialization.
+
+- Use Batch Normalization to maintain healthy gradients throughout the network.
+
+**2. Exploding Gradients**: This problem occurs when gradients grow exponentially through the layers during backpropagation, leading to very large updates to weights and, consequently, an unstable network.
+
+**Causes**:
+
+- Deep networks with accumulative multiplication of gradients can lead to exponentially increasing gradients.
+
+- High learning rates.
+
+**Diagnosis**:
+
+- Monitor the gradients. If the gradient values are increasing dramatically over epochs, it’s likely an issue.
+
+- Watch for NaN values in gradients or weights.
+
+**Solutions**:
+
+- Apply gradient clipping to limit the maximum value of gradients during backpropagation.
+
+- Adjust the learning rate.
+
+- Use weight regularization techniques, like L2 regularization, to penalize large weights.
+
+### Unusual Metrics Observations
+
+**1. High Training Loss or Not Decreasing Loss**: If the loss does not decrease, or decreases very slowly, it indicates the model is not learning effectively.
+
+**Causes**:
+
+- Inappropriate model architecture.
+
+- Inadequate learning rate (too high or too low).
+
+- Poor quality or insufficient training data.
+
+**Diagnosis**:
+
+- Plot the loss over training epochs. A flat line or a line that does not trend downwards indicates a problem.
+
+- Evaluate learning rate and data quality.
+
+**Solutions**:
+
+- Adjust the learning rate.
+
+- Check and preprocess the training data correctly.
+
+- Consider changing the model architecture.
+
+**2. High Variance Between Training and Validation Loss (Overfitting)**: If the training loss decreases but the validation loss does not decrease or increases, the model may be overfitting.
+
+**Causes**:
+
+- Model is too complex with too many parameters.
+
+- Insufficient or non-representative training data.
+
+**Diagnosis**:
+
+- Monitor both training and validation loss. A diverging pattern suggests overfitting.
+
+**Solutions**:
+
+- Simplify the model by reducing the number of layers or parameters.
+
+- Use dropout or regularization techniques.
+
+- Increase training data, or use data augmentation.
+
+**3. High Bias Between Training and Validation Loss (Underfitting)**: If both training and validation losses are high or the model performs poorly even on training data, the model may be underfitting.
+
+**Causes**:
+
+- Overly simple model unable to capture underlying patterns.
+
+- Inadequate training epochs.
+
+**Diagnosis**:
+
+- If both losses are high, consider evaluating the complexity of the model.
+
+**Solutions**:
+
+- Increase model complexity by adding more layers or parameters.
+
+- Train for more epochs.
+
+- Experiment with different model architectures.
+
+### Advanced Diagnostics Tools
+
+**1. Learning Rate Schedulers**: Implement learning rate schedulers to adjust the learning rate during training, which can help in stabilizing the training process.
+
+**2. Early Stopping**: Use early stopping to terminate training when validation metrics stop improving, preventing overfitting and saving computational resources.
+
+**3. Regularization Techniques**: Techniques like L1 and L2 regularization penalize large weights, helping control overfitting.
+
+**4. Hyperparameter Tuning**: Use grid search or random search to optimize hyperparameters like the number of layers, number of neurons, learning rate, etc.
+
+Proper diagnosis and resolution of training issues are vital for building robust AI systems. By systematically monitoring and adjusting the training process, one can significantly enhance model performance and reliability.