I was learning the basics of how an LLM works, even though I've been using it for a while. So, I went through the general cycle of how an input goes through tokenization, embeddings, the transformer, probability, sampling, and how this loop continues for every token that is output.

Usually, we would expect that models would know what the output would be, but the way it generally works, or from what I understand, it's more like a word-guessing game, where the immediate next token is selected based on probabilities, and the future output is not always predictable. It can be different for every output, even though the input prompt might be the same.

So, how it works is: you have an input, which is the query that we provide. Then you have a tokenizer, which converts the text into tokens. Usually, common words like "the," "and," and "was," or even punctuation like a dot or comma, can be represented as individual tokens. But longer or less common words can be split into multiple tokens.

Now, I've attached an image as an example of what I have understood. The text is tokenized, and after it's tokenized, it is converted into token IDs. These token IDs are numerical representations of the tokens and can be mapped back to the original tokens. The token IDs are then used to look up their corresponding embedding vectors.

From my understanding, embeddings are not tokens. Embeddings are vectors that represent tokens as numerical values in a high-dimensional space. These vectors are then passed into the transformer along with other information, such as positional information, so that the transformer can process the sequence.

The transformer has various components, and one of the main things is attention. The attention mechanism, from my understanding, helps the model determine which other tokens are more relevant to a particular token when processing the context. For example, "The flower is beautiful, and the sun makes it radiant." When we see "it," we can infer that it refers to the flower. Attention helps the model consider the relationship between these tokens, although it's not simply identifying the "primary subject" of the sentence. There can be multiple relevant relationships happening at the same time.

The transformer then produces scores for possible next tokens. These are converted into probabilities, so there are many possible tokens that could come next, each with a different probability. What determines which one actually gets picked is the sampling step.

Two concepts that can come into play here are temperature and top-p. Temperature determines how concentrated or spread out the probability distribution is. We usually associate temperature with creativity, but from what I've seen or learned today, it's more accurate to think of it as controlling how strongly the model favors higher-probability tokens. A lower temperature makes the output more predictable, while a higher temperature makes lower-probability choices more likely.

For example, if you're continuing the statement, "The flower is beautiful and the sun makes it radiant," a predictable continuation might be, "Yes, you're absolutely right." A higher-temperature output could be something more unexpected, such as, "Absolutely, the flower is very vibrant-looking today." The point isn't necessarily that one output is more creative, but that the probability distribution is being sampled differently.

Next, we have top-p. Top-p is based on the cumulative probability of the possible next tokens. So, if we set top-p to 0.7, the model considers the smallest set of highest-probability tokens whose cumulative probability reaches approximately 0.7, and removes the rest from consideration for that sampling step. It then samples from that remaining set.

So, for each token that the model generates, this general process happens again: the current context is processed, the model produces probabilities for the next token, sampling determines which token is selected, and that new token is added to the context. The cycle then continues until the model reaches the end of the response.

That is also why, for local LLMs or SLMs, you often see a "tokens per second" parameter. It represents how quickly the model can process and generate tokens, which depends on things like the model, hardware, context length, and the particular workload.

visual learning: https://drive.google.com/file/d/1MrrtY5PcqNISDzwwShVGDmE_TBhJAFtZ/view?usp=drive_link (note: some things might be inconsistent, please cross check 😅)