Spaces:

sebdg
/

ai-cookbook

Running

Sébastien De Greef commited on May 8

Commit

c9e7f26

•

1 Parent(s): 255d69d

feat: Add HuggingFace Embeddings Leaderboard link to embeddings.qmd

Files changed (2) hide show

src/llms/embeddings.qmd CHANGED Viewed

@@ -1,6 +1,11 @@
-# Embeddings
 Embeddings in Large Language Models (LLMs) are a foundational component in the field of natural language processing (NLP). These embeddings transform words, phrases, or even longer texts into a vector space, capturing the semantic meaning that enables LLMs to perform a variety of language-based tasks with remarkable proficiency. This article focuses on the role of embeddings in LLMs, how they are generated, and their impact on the performance of these models.
 ## What are Embeddings in LLMs?
 In the context of LLMs, embeddings are dense vector representations of text. Each vector aims to encapsulate aspects of linguistic meaning such as syntax, semantics, and context. Unlike simpler models that might use one-hot encoding, LLM embeddings map words or tokens to vectors in a way that reflects their semantic and contextual relationships.

+---
+title: Embeddings
+---
 Embeddings in Large Language Models (LLMs) are a foundational component in the field of natural language processing (NLP). These embeddings transform words, phrases, or even longer texts into a vector space, capturing the semantic meaning that enables LLMs to perform a variety of language-based tasks with remarkable proficiency. This article focuses on the role of embeddings in LLMs, how they are generated, and their impact on the performance of these models.
+[HuggingFace Embeddings Leaderboard](https://huggingface.co/spaces/mteb/leaderboard)
 ## What are Embeddings in LLMs?
 In the context of LLMs, embeddings are dense vector representations of text. Each vector aims to encapsulate aspects of linguistic meaning such as syntax, semantics, and context. Unlike simpler models that might use one-hot encoding, LLM embeddings map words or tokens to vectors in a way that reflects their semantic and contextual relationships.

src/llms/tokenizers.qmd CHANGED Viewed

@@ -4,8 +4,8 @@ title: Tokenizers
 Tokenization is a fundamental step in natural language processing (NLP) that involves breaking down text into smaller components, such as words, phrases, or symbols. These smaller components are called tokens. Tokenizers, the tools that perform tokenization, play a crucial role in preparing text for various NLP tasks like machine translation, sentiment analysis, and text summarization. This article provides an exhaustive overview of tokenizers, exploring their types, how they function, their importance, and the challenges they present.
-[Excellent video of Andrej Karpathy about Tokenizers](https://www.youtube.com/watch?v=zduSFxRajkE)
-[Online Tokenizer Playground](https://gpt-tokenizer.dev/)
 ## What is Tokenization?

 Tokenization is a fundamental step in natural language processing (NLP) that involves breaking down text into smaller components, such as words, phrases, or symbols. These smaller components are called tokens. Tokenizers, the tools that perform tokenization, play a crucial role in preparing text for various NLP tasks like machine translation, sentiment analysis, and text summarization. This article provides an exhaustive overview of tokenizers, exploring their types, how they function, their importance, and the challenges they present.
+* [Excellent video of Andrej Karpathy about Tokenizers](https://www.youtube.com/watch?v=zduSFxRajkE)
+* [Online Tokenizer Playground](https://gpt-tokenizer.dev/)
 ## What is Tokenization?