src/theory/architectures.qmd

---
title: Network Architectures
---

Neural network architectures are foundational frameworks designed to tackle diverse problems in artificial intelligence and machine learning. Each architecture is structured to optimize learning and performance for specific types of data and tasks, ranging from simple classification problems to complex sequence generation challenges. This guide explores the various architectures employed in neural networks, providing insights into how they are constructed, their applications, and why certain architectures are preferred for particular tasks.

The architecture of a neural network dictates how information flows and is processed. It determines the arrangement and connectivity of layers, the type of data processing that occurs, and how input data is ultimately transformed into outputs. The choice of a suitable architecture is crucial because it impacts the efficiency, accuracy, and feasibility of training models on given datasets.

## Feedforward Neural Networks (FNNs)

A basic neural network architecture where data flows only in one direction, from input layer to output layer, without any feedback loops. Feedforward Neural Networks are the simplest type of neural network architecture where connections between the nodes do not form a cycle. This is ideal for problems where the output is directly mapped from the input.

* **Usage**: Image classification, regression, function approximation
* **Strengths**: Simple to implement, computationally efficient
* **Caveats**: Limited capacity to model complex relationships, prone to overfitting

```{dot}
digraph FNN {
    // Graph properties
    node [shape=record];

    // Nodes definitions
    input_layer [label="Input Layer" shape=ellipse];
    hidden_layer1 [label="Hidden Layer 1" shape=box];
    hidden_layer2 [label="Hidden Layer 2" shape=box];
    output_layer [label="Output Layer" shape=ellipse];

    // Edges definitions
    input_layer -> hidden_layer1;
    hidden_layer1 -> hidden_layer2;
    hidden_layer2 -> output_layer;
}
```

**Input Layer**: This layer represents the initial data that is fed into the network. Each node in this layer typically corresponds to a feature in the input dataset.

**Hidden Layers**: These are intermediary layers between the input and output layers. Hidden layers allow the network to learn complex patterns in the data. They are called "hidden" because they are not directly exposed to the input or output.

**Output Layer**: The final layer that produces the network’s predictions. The function of this layer can vary depending on the specific application — for example, it might use a softmax activation function for classification tasks or a linear activation for regression tasks.

**Edges**: Represent the connections between neurons in consecutive layers. In feedforward networks, every neuron in one layer connects to every neuron in the next layer. These connections are weighted, and these weights are adjusted during training to minimize error.

## Convolutional Neural Networks (CNNs)

A neural network architecture that uses convolutional and pooling layers to extract features from images. CNNs are highly effective at processing data that has a grid-like topology, such as images, due to their ability to exploit spatial hierarchies and structures within the data.

* **Usage**: Image classification, object detection, image segmentation
* **Strengths**: Excellent performance on image-related tasks, robust to image transformations
* **Caveats**: Computationally expensive, require large datasets


```{dot}
digraph CNN {
    // Graph properties
    node [shape=record];

    // Nodes definitions
    input_image [label="Input Image" shape=ellipse];
    conv1 [label="Convolution Layer 1\nReLU" shape=box];
    pool1 [label="Pooling Layer 1" shape=box];
    conv2 [label="Convolution Layer 2\nReLU" shape=box];
    pool2 [label="Pooling Layer 2" shape=box];
    fully_connected [label="Fully Connected Layer" shape=box];
    output [label="Output\n(Classification)" shape=ellipse];

    // Edges definitions
    input_image -> conv1;
    conv1 -> pool1;
    pool1 -> conv2;
    conv2 -> pool2;
    pool2 -> fully_connected;
    fully_connected -> output;
}
```
**Input Image**: The initial input where images are fed into the network.

**Convolution Layer 1 and 2**: These layers apply a set of filters to the input image to create feature maps. These filters are designed to detect spatial hierarchies such as edges, colors, gradients, and more complex patterns as the network deepens. Each convolution layer is typically followed by a non-linear activation function like ReLU (Rectified Linear Unit).

**Pooling Layer 1 and 2**: These layers reduce the spatial size of the feature maps to decrease the amount of computation and weights in the network. Pooling (often max pooling) helps make the detection of features invariant to scale and orientation changes.

**Fully Connected Layer**: This layer takes the flattened output of the last pooling layer and performs classification based on the features extracted by the convolutional and pooling layers.

**Output**: The final output layer, which classifies the input image into categories based on the training dataset.

## Recurrent Neural Networks (RNNs)

A neural network architecture that uses feedback connections to model sequential data. RNNs are capable of processing sequences of data by maintaining a state that acts as a memory. They are particularly useful for applications where the context or sequence of data points is important.

* **Usage**: Natural Language Processing (NLP), sequence prediction, time series forecasting
* **Strengths**: Excellent performance on sequential data, can model long-term dependencies
* **Caveats**: Suffer from vanishing gradients, difficult to train
```{dot}
digraph RNN {
    // Graph properties
    node [shape=record];

    // Nodes definitions
    input_seq [label="Input Sequence" shape=ellipse];
    rnn_cell [label="RNN Cell" shape=box];
    hidden_state [label="Hidden State" shape=box];
    output_seq [label="Output Sequence" shape=ellipse];

    // Edges definitions
    input_seq -> rnn_cell;
    rnn_cell -> hidden_state [label="Next"];
    hidden_state -> rnn_cell [label="Recurrence", dir=back];
    rnn_cell -> output_seq;

    // Additional details for clarity
    edge [style=dashed];
    rnn_cell -> output_seq [label="Each timestep", style=dashed];
}
```

**Input Sequence**: Represents the sequence of data being fed into the RNN, such as a sentence or time-series data.

**RNN Cell**: This is the core of an RNN, where the computation happens. It takes input from the current element of the sequence and combines it with the hidden state from the previous element of the sequence.

**Hidden State**: This node represents the memory of the network, carrying information from one element of the sequence to the next. The hidden state is updated continuously as the sequence is processed.

**Output Sequence**: The RNN can produce an output at each timestep, depending on the task. For example, in sequence labeling, there might be an output corresponding to each input timestep.


## Long Short-Term Memory (LSTM) Networks

A type of RNN that uses memory cells to learn long-term dependencies. LSTM networks are designed to avoid the long-term dependency problem, making them effective at tasks where the context can extend over longer sequences.

* **Usage**: NLP, sequence prediction, time series forecasting
* **Strengths**: Excellent performance on sequential data, can model long-term dependencies
* **Caveats**: Computationally expensive, require large datasets

```{dot}
digraph LSTM {
    // Graph properties
    node [shape=record];

    // Nodes definitions
    input_seq [label="Input Sequence" shape=ellipse];
    lstm_cell [label="LSTM Cell" shape=box];
    cell_state [label="Cell State" shape=box];
    hidden_state [label="Hidden State" shape=box];
    output_seq [label="Output Sequence" shape=ellipse];

    // Edges definitions
    input_seq -> lstm_cell;
    cell_state -> lstm_cell [label="Recurrence", dir=back];
    hidden_state -> lstm_cell [label="Recurrence", dir=back];
    lstm_cell -> cell_state [label="Update"];
    lstm_cell -> hidden_state [label="Update"];
    lstm_cell -> output_seq;

    // Additional explanations
    edge [style=dashed];
    lstm_cell -> output_seq [label="Each timestep", style=dashed];
}
```

**Input Sequence**: Represents the sequential data input, such as a series of words or time-series data points.

**LSTM Cell**: The core unit in an LSTM network that processes input data one element at a time. It interacts intricately with both the cell state and the hidden state to manage and preserve information over long periods.

**Cell State**: A "long-term" memory component of the LSTM cell. It carries relevant information throughout the processing of the sequence, with the ability to add or remove information via gates (not explicitly shown here).

**Hidden State**: A "short-term" memory component that also transfers information to the next time step but is more sensitive and responsive to recent inputs than the cell state.

**Output Sequence**: Depending on the task, LSTMs can output at each timestep (for tasks like sequence labeling) or after processing the entire sequence (like sentiment analysis).

## Transformers

A neural network architecture that uses self-attention mechanisms to model relationships between input sequences. Transformers are particularly effective in NLP tasks due to their ability to handle sequences in parallel and consider all parts of the input at once.

* **Usage**: NLP, machine translation, language modeling
* **Strengths**: Excellent performance on sequential data, parallelizable, can handle long-range dependencies
* **Caveats**: Computationally expensive, require large datasets
```{dot}
digraph Transformer {
    // Graph properties
    node [shape=record];

    // Nodes definitions
    input_tokens [label="Input Tokens" shape=ellipse];
    embedding_layer [label="Embedding Layer" shape=box];
    positional_encoding [label="Add Positional Encoding" shape=box];
    encoder [label="Encoder Stack" shape=box];
    decoder [label="Decoder Stack" shape=box];
    output_tokens [label="Output Tokens" shape=ellipse];

    // Edges definitions
    input_tokens -> embedding_layer;
    embedding_layer -> positional_encoding;
    positional_encoding -> encoder;
    encoder -> decoder;
    decoder -> output_tokens;

    // Additional components for clarity (not actual flow)
    encoder_output [label="Encoder Output" shape=note];
    decoder_input [label="Decoder Input" shape=note];
    encoder -> encoder_output [style=dashed];
    decoder_input -> decoder [style=dashed];

    // Descriptions for self-attention and cross-attention mechanisms
    self_attention [label="Self-Attention" shape=plaintext];
    cross_attention [label="Cross-Attention" shape=plaintext];
    encoder -> self_attention [style=dotted];
    decoder -> self_attention [style=dotted];
    decoder -> cross_attention [style=dotted];
    cross_attention -> encoder_output [style=dotted, dir=none];
}
```

**Input Tokens**: Represents the initial sequence of tokens (e.g., words in a sentence) that are fed into the Transformer.

**Embedding Layer**: Converts tokens into vectors that the model can process. Each token is mapped to a unique vector.

**Positional Encoding**: Adds information about the position of each token in the sequence to the embeddings, which is crucial as Transformers do not inherently process sequential data.

**Encoder Stack**: A series of encoder layers that process the input. Each layer uses self-attention mechanisms to consider all parts of the input simultaneously.

**Decoder Stack**: A series of decoder layers that generate the output sequence step by step. Each layer uses both self-attention mechanisms to attend to its own output so far, and cross-attention mechanisms to focus on the output from the encoder.

**Output Tokens**: The final output sequence generated by the Transformer, such as a translated sentence or the continuation of an input text.

**Encoder Output and Decoder Input**: Not actual data flow, but illustrate how information is transferred from the encoder to the decoder.

**Self-Attention and Cross-Attention**: These mechanisms are core features of Transformer models. Self-attention allows layers to consider other parts of the input or output at each step, while cross-attention allows the decoder to focus on relevant parts of the input sequence.

## Autoencoders

A neural network architecture that learns to compress and reconstruct input data. Autoencoders are typically used for dimensionality reduction tasks, as they learn to encode the essential aspects of the data in a smaller representation.

* **Usage**: Dimensionality reduction, anomaly detection, generative modeling
* **Strengths**: Excellent performance on dimensionality reduction, can learn robust representations
* **Caveats**: May not perform well on complex data distributions
```{dot}
digraph Autoencoder {
    // Graph properties
    node [shape=record];

    // Nodes definitions
    input_data [label="Input Data" shape=ellipse];
    encoder [label="Encoder" shape=box];
    latent_space [label="Latent Space" shape=box];
    decoder [label="Decoder" shape=box];
    reconstructed_output [label="Reconstructed Output" shape=ellipse];

    // Edges definitions
    input_data -> encoder;
    encoder -> latent_space;
    latent_space -> decoder;
    decoder -> reconstructed_output;
}
```

**Input Data**: Represents the data that is fed into the Autoencoder. This could be any kind of data, such as images, text, or sound.

**Encoder**: The first part of the Autoencoder that processes the input data and compresses it into a smaller, dense representation. This part typically consists of several layers that gradually reduce the dimensionality of the input.

**Latent Space**: Also known as the "encoded" state or "bottleneck". This is a lower-dimensional representation of the input data and serves as the compressed "code" that the decoder will use to reconstruct the input.

**Decoder**: Mirrors the structure of the encoder but in reverse. It takes the encoded data from the latent space and reconstructs the original data as closely as possible. This part typically consists of layers that gradually increase in dimensionality to match the original input size.

**Reconstructed Output**: The final output of the Autoencoder. This is the reconstruction of the original input data based on the compressed code stored in the latent space. The quality of this reconstruction is often a measure of the Autoencoder’s performance.

## Generative Adversarial Networks (GANs)

A neural network architecture that consists of a generator and discriminator, which compete to generate realistic data. GANs are highly effective at generating new data that mimics the input data, often used in image generation and editing.

* **Usage**: Generative modeling, data augmentation, style transfer
* **Strengths**: Excellent performance on generative tasks, can generate realistic data
* **Caveats**: Training can be unstable, require careful tuning of hyperparameters
```{dot}
digraph GAN {
    // Graph properties
    node [shape=record];

    // Nodes definitions
    noise [label="Noise vector (z)" shape=ellipse];
    generator [label="Generator (G)" shape=box];
    generated_image [label="Generated image (G(z))" shape=cds];
    real_image [label="Real image (x)" shape=cds];
    discriminator [label="Discriminator (D)" shape=box];
    D_output_fake [label="D(G(z))" shape=ellipse];
    D_output_real [label="D(x)" shape=ellipse];

    // Edges definitions
    noise -> generator;
    generator -> generated_image;
    generated_image -> discriminator [label="Fake"];
    real_image -> discriminator [label="Real"];
    discriminator -> D_output_fake [label="Output for fake"];
    discriminator -> D_output_real [label="Output for real"];
}

```


**Noise vector (z)**: Represents the random noise input to the generator.

**Generator (G)**: The model that learns to generate new data with the same statistics as the training set from the noise vector.

**Generated image (G(z))**: The fake data produced by the generator.

**Real image (x)**: Actual data samples from the training dataset.

**Discriminator (D)**: The model that learns to distinguish between real data and synthetic data generated by the Generator.

**D(G(z)) and D(x)**: Outputs of the Discriminator when evaluating fake data and real data, respectively.


The Noise vector feeds into the Generator.

The Generator outputs a Generated image, which is input to the Discriminator labeled as "Fake".

The Real image also feeds into the Discriminator but is labeled as "Real".

The Discriminator outputs evaluations for both fake and real inputs.

## Residual Networks (ResNets)

A neural network architecture that uses residual connections to ease training. ResNets are particularly effective for very deep networks, as they allow for training deeper networks by providing pathways for gradients to flow through.

* **Usage**: Image classification, object detection
* **Strengths**: Excellent performance on image-related tasks, ease of training
* **Caveats**: May not perform well on sequential data
```{dot}
digraph ResNet {
    // Graph properties
    node [shape=record];

    // Nodes definitions
    input [label="Input Image" shape=ellipse];
    conv1 [label="Initial Conv + BN + ReLU" shape=box];
    resblock1 [label="<f0> ResBlock | <f1> + | <f2> ReLU" shape=Mrecord];
    resblock2 [label="<f0> ResBlock | <f1> + | <f2> ReLU" shape=Mrecord];
    resblock3 [label="<f0> ResBlock | <f1> + | <f2> ReLU" shape=Mrecord];
    avgpool [label="Average Pooling" shape=box];
    fc [label="Fully Connected Layer" shape=box];
    output [label="Output" shape=ellipse];

    // Edges definitions
    input -> conv1;
    conv1 -> resblock1:f0;
    resblock1:f2 -> resblock2:f0;
    resblock2:f2 -> resblock3:f0;
    resblock3:f2 -> avgpool;
    avgpool -> fc;
    fc -> output;

    // Adding skip connections
    edge [style=dashed];
    conv1 -> resblock1:f1;
    resblock1:f1 -> resblock2:f1;
    resblock2:f1 -> resblock3:f1;
}
```
**Input Image**: The initial input layer where images are fed into the network.

**Initial Conv + BN + ReLU**: Represents an initial convolutional layer followed by batch normalization and a ReLU activation function to prepare the data for residual blocks.

**ResBlock**: These are the residual blocks that define the ResNet architecture. Each block contains two parts: a sequence of convolutional layers and a skip connection that adds the input of the block to its output.

**Average Pooling**: This layer averages the feature maps spatially to reduce their dimensions before passing to a fully connected layer.

**Fully Connected Layer**: This layer maps the feature representations to the final output classes.

**Output**: The final prediction of the network.


## U-Net

A neural network architecture that uses an encoder-decoder structure with skip connections. U-Net is designed primarily for biomedical image segmentation, where it is crucial to localize objects precisely within an image.

* **Usage**: Image segmentation, object detection
* **Strengths**: Excellent performance on image segmentation tasks, fast training
* **Caveats**: May not perform well on sequential data
```{dot}
digraph UNet {
    // Graph properties
    node [shape=record];

    // Nodes definitions
    input [label="Input Image" shape=ellipse];
    conv1 [label="Conv + ReLU\nDownsampling" shape=box];
    conv2 [label="Conv + ReLU\nDownsampling" shape=box];
    bottom [label="Conv + ReLU" shape=box];
    upconv1 [label="UpConv + ReLU\nUpsampling" shape=box];
    concat1 [label="Concatenate" shape=circle];
    upconv2 [label="UpConv + ReLU\nUpsampling" shape=box];
    concat2 [label="Concatenate" shape=circle];
    finalconv [label="Conv + ReLU\n1x1 Conv" shape=box];
    output [label="Output\nSegmentation Map" shape=ellipse];

    // Edges definitions
    input -> conv1;
    conv1 -> conv2;
    conv2 -> bottom;
    bottom -> upconv1;
    upconv1 -> concat1;
    concat1 -> upconv2;
    upconv2 -> concat2;
    concat2 -> finalconv;
    finalconv -> output;

    // Skip connections
    edge [style=dashed];
    conv1 -> concat1 [label="Copy\ncrop"];
    conv2 -> concat2 [label="Copy\ncrop"];
}
```

**Input Image**: The initial input layer where images are fed into the network.

**Conv + ReLU / Downsampling**: These blocks represent convolutional operations followed by a ReLU activation function. The "Downsampling" indicates that each block reduces the spatial dimensions of the input.

**Bottom**: This is the lowest part of the U, consisting of convolutional layers without downsampling, positioned before the upsampling starts.

**UpConv + ReLU / Upsampling**: These blocks perform transposed convolutions (or up-convolutions) that increase the resolution of the feature maps.

**Concatenate**: These layers concatenate feature maps from the downsampling pathway with the upsampled feature maps to preserve high-resolution features for precise localization.

**Final Conv**: This typically includes a 1x1 convolution to map the deep feature representations to the desired number of classes for segmentation.

**Output / Segmentation Map**: The final output layer which produces the segmented image.

## Attention-based Models

A neural network architecture that uses attention mechanisms to focus on relevant input regions. Attention-based models are particularly effective for tasks that require understanding of complex relationships within the data, such as interpreting a document or translating a sentence.

* **Usage**: NLP, machine translation, question answering
* **Strengths**: Excellent performance on sequential data, can model long-range dependencies
* **Caveats**: Require careful tuning of hyperparameters
```{dot}
digraph AttentionBasedModels {
    // Graph properties
    node [shape=record];

    // Nodes definitions
    input [label="Input Sequence" shape=ellipse];
    embedding [label="Embedding Layer" shape=box];
    positional [label="Add Positional Encoding" shape=box];
    multihead [label="Multi-Head Attention" shape=box];
    addnorm1 [label="Add & Norm" shape=box];
    feedforward [label="Feedforward Network" shape=box];
    addnorm2 [label="Add & Norm" shape=box];
    output [label="Output Sequence" shape=ellipse];

    // Edges definitions
    input -> embedding;
    embedding -> positional;
    positional -> multihead;
    multihead -> addnorm1;
    addnorm1 -> feedforward;
    feedforward -> addnorm2;
    addnorm2 -> output;

    // Skip connections
    edge [style=dashed];
    positional -> addnorm1 [label="Skip Connection"];
    addnorm1 -> addnorm2 [label="Skip Connection"];
}

```
**Input Sequence**: Initial data input, typically a sequence of tokens.

**Embedding Layer**: Converts tokens into vectors that the model can process.

**Add Positional Encoding**: Incorporates information about the position of tokens in the sequence into their embeddings, which is crucial since attention mechanisms do not inherently process sequential data.

**Multi-Head Attention**: Allows the model to focus on different parts of the sequence for different representations, facilitating better understanding and processing of the input.

**Add & Norm**: A layer that combines residuals (from skip connections) with the output of the attention or feedforward layers, followed by layer normalization.

**Feedforward Network**: A dense neural network that processes the sequence after attention has been applied.

**Output Sequence**: The final processed sequence output by the model, often used for tasks like translation, text generation, or classification.

**Skip Connections**: Dashed lines represent skip connections that help to alleviate the vanishing gradient problem by allowing gradients to flow through the network directly. They also help the model to learn an identity function which ensures that the model does not lose information throughout the layers.

## Graph Neural Networks (GNNs)

A neural network architecture that uses graph structures to model relationships between nodes. GNNs are effective for data that can be represented as graphs, such as social networks or molecules, as they capture the relationships between entities.

* **Usage**: Graph-based data, social network analysis, recommendation systems
* **Strengths**: Excellent performance on graph-based data, can model complex relationships
* **Caveats**: Computationally expensive, require large datasets
```{dot}
digraph GNN {
    // Graph properties
    node [shape=record];

    // Nodes definitions
    input_graph [label="Input Graph" shape=ellipse];
    node_features [label="Node Features" shape=box];
    edge_features [label="Edge Features" shape=box];
    gnn_layers [label="GNN Layers" shape=box];
    aggregate [label="Aggregate Messages" shape=box];
    update [label="Update States" shape=box];
    readout [label="Graph-level Readout" shape=box];
    output [label="Output" shape=ellipse];

    // Edges definitions
    input_graph -> node_features;
    input_graph -> edge_features;
    node_features -> gnn_layers;
    edge_features -> gnn_layers;
    gnn_layers -> aggregate;
    aggregate -> update;
    update -> readout;
    readout -> output;
}
```

**Input Graph**: The initial graph input containing nodes and edges.

**Node Features**: Processes the features associated with each node. These can include node labels, attributes, or other data.

**Edge Features**: Processes features associated with edges in the graph, which might include types of relationships, weights, or other characteristics.

**GNN Layers**: A series of graph neural network layers that apply convolution-like operations over the graph. These layers can involve message passing between nodes, where a node's new state is determined based on its neighbors.



**Aggregate Messages**: Combines the information (messages) received from neighboring nodes into a single unified message. Aggregation functions can include sums, averages, or max operations.



**Update States**: Updates the states of the nodes based on aggregated messages, typically using some form of neural network or transformation.



**Graph-level Readout**: Aggregates node states into a graph-level representation, which can be used for tasks that require a holistic view of the graph (e.g., determining the properties of a molecule).



**Output**: The final output, which can vary depending on the specific application (node classification, link prediction, graph classification, etc.).



## Reinforcement Learning (RL) Architectures



A neural network architecture that uses reinforcement learning to learn from interactions with an environment. RL architectures are highly effective for sequential decision-making tasks, such as playing games or navigating environments.



* **Usage**: Game playing, robotics, autonomous systems

* **Strengths**: Excellent performance on sequential decision-making tasks, can learn complex policies

* **Caveats**: Require large datasets, can be slow to train

```{dot}

digraph RL {

    // Graph properties

    node [shape=record];



    // Nodes definitions

    environment [label="Environment" shape=ellipse];

    state [label="State" shape=ellipse];

    agent [label="Agent" shape=box];

    action [label="Action" shape=ellipse];

    reward [label="Reward" shape=ellipse];

    updated_state [label="Updated State" shape=ellipse];



    // Edges definitions

    environment -> state;

    state -> agent;

    agent -> action;

    action -> environment;

    environment -> reward;

    reward -> agent;

    environment -> updated_state [label="Feedback Loop"];

    updated_state -> state [label="New State"];

}

```



**Environment**: This is where the agent operates. It defines the dynamics of the system including how the states transition and how rewards are assigned for actions.



**State**: Represents the current situation or condition in which the agent finds itself. It is the information that the environment provides to the agent, which then bases its decisions on this data.



**Agent**: This is the decision-maker. It uses a strategy, which may involve a neural network or another function approximator, to decide what actions to take based on the state it perceives.



**Action**: The decision taken by the agent, which will affect the environment.



**Reward**: After taking an action, the agent receives a reward (or penalty) from the environment. This reward is an indication of how good the action was in terms of achieving the goal.



**Updated State**: After an action is taken, the environment transitions to a new state. This new state and the reward feedback are then used by the agent to learn and refine its strategy.



## Evolutionary Neural Networks (ENNs)



A neural network architecture that uses evolutionary principles to evolve neural networks. Evolutionary Neural Networks are particularly effective for optimization problems, where they can evolve solutions over generations.



* **Usage**: Neuroevolution, optimization problems

* **Strengths**: Excellent performance on optimization problems, can learn complex policies

* **Caveats**: Computationally expensive, require large datasets

```{dot}

digraph ENN {

    // Graph properties

    node [shape=record];



    // Nodes definitions

    population [label="Initial Population\n(Neural Networks)" shape=ellipse];

    selection [label="Selection" shape=box];

    crossover [label="Crossover" shape=box];

    mutation [label="Mutation" shape=box];

    fitness [label="Fitness Evaluation" shape=box];

    new_population [label="New Generation" shape=ellipse];

    best_network [label="Best Performing Network" shape=ellipse, fillcolor=lightblue];



    // Edges definitions

    population -> selection;

    selection -> crossover;

    crossover -> mutation;

    mutation -> fitness;

    fitness -> new_population;

    new_population -> selection [label="Next Generation"];

    fitness -> best_network [label="If Optimal", style=dashed];



    // Additional explanatory nodes

    edge [style=dashed];

    best_network -> new_population [label="Update Population", style=dotted];

}

```



**Initial Population**: This represents the initial set of neural networks. These networks might differ in architecture, weights, or hyperparameters.



**Selection**: Part of the evolutionary process where individual networks are selected based on their performance, often using a fitness function.



**Crossover**: A genetic operation used to combine features from two or more parent neural networks to create offspring. This simulates sexual reproduction.



**Mutation**: Introduces random variations to the offspring, potentially leading to new neural network configurations. This step enhances diversity within the population.



**Fitness Evaluation**: Each network in the population is evaluated based on how well it performs the given task. The fitness often determines which networks survive and reproduce.



**New Generation**: After selection, crossover, mutation, and evaluation, a new generation of neural networks is formed. This generation forms the new population for further evolution.



**Best Performing Network**: Out of all generations, the network that performs best on the task.



**Feedback Loops**:

  

  - **Next Generation**: The cycle from selection to fitness evaluation and then back to selection with the new generation is a loop that continues until a satisfactory solution (network) is found.

  

  - **If Optimal**: If during any fitness evaluation a network meets the predefined criteria or optimality, it may be selected as the final model.



## Spiking Neural Networks (SNNs)



A neural network architecture that uses spiking neurons to process data. SNNs are particularly effective for neuromorphic computing applications, where they can operate in energy-efficient ways.



* **Usage**: Neuromorphic computing, edge AI

* **Strengths**: Excellent performance on edge AI applications, energy-efficient

* **Caveats**: Limited software support, require specialized hardware

```{dot}

digraph SNN {

    // Graph properties

    node [shape=record];



    // Nodes definitions

    input_neurons [label="Input Neurons" shape=ellipse];

    synaptic_layers [label="Synaptic Layers\n(Weighted Connections)" shape=box];

    spiking_neurons [label="Spiking Neurons" shape=box];

    output_neurons [label="Output Neurons" shape=ellipse];

    threshold_mechanism [label="Threshold Mechanism" shape=box];

    spike_train [label="Spike Train Output" shape=ellipse];



    // Edges definitions

    input_neurons -> synaptic_layers;

    synaptic_layers -> spiking_neurons;

    spiking_neurons -> threshold_mechanism;

    threshold_mechanism -> output_neurons;

    output_neurons -> spike_train;



    // Additional explanatory nodes

    edge [style=dashed];

    synaptic_layers -> threshold_mechanism [label="Dynamic Weights", style=dashed];

}

```



**Input Neurons**: These neurons receive the initial input signals, which could be any time-varying signal or a pattern encoded in the timing of spikes.



**Synaptic Layers**: Represents the connections between neurons. In SNNs, these connections are often dynamic, changing over time based on the activity of the network (Hebbian learning principles).



**Spiking Neurons**: Neurons that operate using spikes, which are brief and discrete events typically caused by reaching a certain threshold in the neuron’s membrane potential.



**Threshold Mechanism**: A critical component in SNNs that determines when a neuron should fire based on its membrane potential. This mechanism can adapt based on the history of spikes and neuronal activity.



**Output Neurons**: Neurons that produce the final output of the network. These may also operate using spikes, especially in SNNs designed for specific tasks like motor control or sensory processing.



**Spike Train Output**: The output from the network is often in the form of a spike train, representing the timing and sequence of spikes from the output neurons.



**Dynamic Weights**: Indicates that the synaptic weights are not static and can change based on the spike timing differences between pre- and post-synaptic neurons (STDP - Spike-Timing-Dependent Plasticity).



## Conditional Random Fields (CRFs)



A probabilistic model that uses graphical models to model sequential data. CRFs are particularly effective for sequence labeling tasks, where they can model complex relationships between labels in a sequence.



* **Usage**: NLP, sequence labeling, information extraction

* **Strengths**: Excellent performance on sequential data, can model complex relationships

* **Caveats**: Computationally expensive, require large datasets



```{dot}

digraph CRF {

    // Graph properties

    node [shape=record];



    // Nodes definitions

    input_sequence [label="Input Sequence" shape=ellipse];

    feature_extraction [label="Feature Extraction" shape=box];

    crf_layer [label="CRF Layer" shape=box];

    output_labels [label="Output Labels" shape=ellipse];



    // Edges definitions

    input_sequence -> feature_extraction;

    feature_extraction -> crf_layer;

    crf_layer -> output_labels;



    // Additional nodes for clarity

    state_transition [label="State Transition Features" shape=plaintext];

    feature_extraction -> state_transition [style=dotted];

    state_transition -> crf_layer [style=dotted];

}

```



**Input Sequence**: Represents the raw data input, such as sentences in text or other sequential data.



**Feature Extraction**: Processes the input data to extract features that are relevant for making predictions. This could include lexical features, part-of-speech tags, or contextual information in a natural language processing application.



**CRF Layer**: The core of the CRF model where the actual conditional random field is applied. This layer models the dependencies between labels in the sequence, considering both the input features and the labels of neighboring items in the sequence.



**Output Labels**: The final output of the CRF, which provides a label for each element in the input sequence. In the context of NLP, these might be tags for named entity recognition, part-of-speech tags, etc.



**State Transition Features**: This represents how CRFs utilize state transition features to model the relationships and dependencies between different labels in the sequence. These are not actual data flow but indicate the type of information that influences the CRF layer's decisions.

## Mixture of Experts (MoE)

A neural network architecture that consists of multiple expert networks (submodels), each specialized in different parts of the data or tasks. A gating network determines which expert(s) are most relevant for a given input. MoE is particularly effective for large-scale machine learning models, where it can dynamically route tasks to the most appropriate experts.

* **Usage**: Large-scale machine learning models, task-specific adaptations, dynamic routing of tasks
* **Strengths**: Highly scalable, capable of handling diverse tasks simultaneously, efficient use of resources by activating only relevant experts for each input.
* **Caveats**: Complex to implement and train, requires careful tuning to balance the load across experts and avoid overfitting in individual experts.

```{dot}
digraph MoE {
    // Graph properties
    node [shape=record];

    // Nodes definitions
    input_data [label="Input Data" shape=ellipse];
    gating_network [label="Gating Network" shape=box];
    expert1 [label="Expert 1" shape=box];
    expert2 [label="Expert 2" shape=box];
    expert3 [label="Expert 3" shape=box];
    combined_output [label="Combined Output" shape=ellipse];

    // Edges definitions
    input_data -> gating_network;
    gating_network -> expert1 [label="Weight"];
    gating_network -> expert2 [label="Weight"];
    gating_network -> expert3 [label="Weight"];
    expert1 -> combined_output [label="Output 1"];
    expert2 -> combined_output [label="Output 2"];
    expert3 -> combined_output [label="Output 3"];

    // Additional explanatory nodes
    edge [style=dashed];
    gating_network -> combined_output [label="Decision Weights", style=dotted];
}
```

**Input Data**: Represents the data being fed into the model. This could be anything from images, text, to structured data.

**Gating Network**: A crucial component that dynamically determines which expert model should handle the given input. It evaluates the input data and allocates weights to different experts based on their relevance to the current data point.

**Experts**: These are specialized models (expert1, expert2, expert3) that are trained on subsets of the data or specific types of tasks. Each expert processes the input independently.

**Combined Output**: The final output of the MoE model, which typically involves aggregating the outputs of the experts weighted by the gating network’s decisions.

**Weights**: These edges show how the gating network influences the contribution of each expert to the final decision. The weights are not fixed but are determined dynamically based on each input.

**Output 1, 2, 3**: These labels on the edges from experts to the combined output represent the contribution of each expert to the final model output. Each expert contributes its processed output, which is then combined based on the weights provided by the gating network.