LLM (Basics)

This post shall give a very basic view on LLMs and therefore can be a little bit off the actual behavior.

Creation

  1. Data crawling: The crawler searches the internet and extracts the actual text from the web pages.
  2. Tokenization: The tokenizer translates the text into tokens, represented by unique token IDs. 1
  3. Network pre-training: The training happens in a loop over the tokenized training data. 2, 3
    1. The first/next token (sequence) from the training data gets fed into the network.
    2. The network returns a probability distribution over the vocabulary, indicating how likely each possible next token is.
    3. The network’s weights are adjusted based on the difference between its prediction and the actual next token.
  4. Network post-training: Trains the model with specific user-defined datasets to improve its behavior.

1 You want to try it out? Here is a tool showing you tokenizers: https://tiktokenizer.com/.
2 The ANN in an LLM is a Transformer, which contains of more than just FFNNs/MLPs.
3 The LLM will actually be trained, processing multiple token sequences in parallel.

Usage

  1. Prompting: You are entering text for the LLM in the provided input field via the GUI. 4
  2. Tokenization: The tokenizer processes the prompt in the same way as the training data.
  3. Inference: It’s like the network training process, but without the actual training.
    1. The model receives the prompt and the tokens already generated for the response.
    2. The model calculates the probability distribution and returns the next token based on it.

4 The actual input for the LLM is called context, which contains more than the prompt from the user.

Summary

An LLM is a huge ANN
… which has been trained on a lot of tokenized text data
… that responses by repeatedly predicting the next token.