Large Language Model

I. Massive parameters and emergent intelligence — overview of LLM

    %%{init: { 'theme': 'base', 'themeVariables': { 'edgeLabelBackground': '#fff' }}}%%
flowchart LR
    A1["Simple sentence-completion model"] -- "Massive parameters and large-scale pre-training" --> B1["Emergent problem-solving and general intelligence"]
    style A1 fill:#f9f9f9,stroke:#333,stroke-width:1px
    style B1 fill:#e1f5fe,stroke:#01579b,stroke-width:1px
  

Definition: an artificial intelligence model trained on massive datasets using a huge neural network with hundreds of billions or more parameters ( Parameters ), maximizing natural language understanding and generation capability

Characteristics: ( Emergence ) the phenomenon of Emergent Abilities, in which specific capabilities suddenly appear once the model exceeds a certain scale ( Generality ) capable of performing a wide range of tasks from prompts ( Prompt ) alone, without separate fine-tuning ( Knowledge Compression ) compresses the vast body of text humanity has accumulated into the form of model weights

II. Core architecture and training process of LLM

A. The LLM lifecycle: from pre-training to alignment

    graph TD
    A2["Pre-training\n(Self-supervised)"] --> B2["SFT\n(Instruction Tuning)"]
    B2 --> C2["RLHF\n(Human Alignment)"]
    C2 --> D2["Inference\n(Prompt/RAG)"]
  

B. Core technical elements

C. From text to numbers — subword tokenization and embeddings

A model only ever sees numbers, so text has to be cut into units and each unit mapped to a vector. The choice of unit is a trade-off between sequence length and vocabulary coverage:

    graph LR
    A["Raw text"] --> B["Subword tokenizer\n(BPE)"]
    B --> C["Token IDs\n[1023, 47, 8891, …]"]
    C --> D["Embedding matrix\nlookup"]
    D --> E["Sequence of real-valued\nvectors → Transformer"]
  

Tokenization is why cost and context limits are quoted in tokens rather than words, and why byte-level fallbacks matter: with a subword vocabulary built mostly from English text, the same sentence in another script can cost several times as many tokens. The embedding matrix is then simply a lookup table with one learned row per vocabulary entry — turning the ID sequence into the sequence of vectors the Transformer consumes.

D. Next-token prediction — the chain-rule factorization

Assigning a probability directly to every possible sequence is hopeless, since the number of sequences grows exponentially in length. The chain rule of probability sidesteps this by factorizing the joint distribution into a product of conditionals:

P(x₁, x₂, …, x_T) = Π P(x_t | x₁, …, x_{t-1})

Each factor is one forward pass through the network, and the model only ever has to answer a single question: given everything so far, what comes next?

Because causal masking keeps each position blind to its own future, training scores all T positions in a single parallel pass. Generation cannot be parallelized the same way: the model samples one token, appends it to the input, and runs again — the autoregressive loop.

E. Decoding — the temperature knob and its neighbours

Sampling from the next-token distribution is a separate choice from the model itself, and the same weights behave very differently depending on how the distribution is shaped. Temperature τ divides the logits before the softmax, controlling how sharp the distribution is:

III. Limitations of LLM and key mitigation strategies

Technology trends: LLMs are now expanding beyond text into multimodal ( Multimodal ) models that simultaneously process images and audio, while the market for small, domain-specialized large language models ( sLLM ) is also growing rapidly