Large Language Models (LLMs)

Beginner

An LLM is a very large neural network trained to predict the next token in text. Because it learns broad language patterns from enormous datasets, it can answer questions, summarize, write code, classify text, and more.

• Pretraining teaches general language behavior.
• Instruction tuning and alignment improve usefulness for user tasks.
• Prompting, retrieval, and tools extend what the model can do in practice.

Data collection -> pretraining -> alignment / instruction tuning -> deployment -> prompting / tools / retrieval

llm_capabilities = ["summarization", "classification", "reasoning", "code generation", "tool use"]
print(llm_capabilities)

Real-world example: a company assistant can summarize meetings, search internal policy documents, draft emails, and help engineers inspect logs, all using variations of the same base model.

Batch Size in LLM Training and Inference

Batch size appears at two stages of the LLM lifecycle — pretraining and inference serving — with different trade-offs in each.

During pretraining

• Modern LLMs use massive effective batch sizes — often millions of tokens per update step — achieved via gradient accumulation across many GPUs.
• Batch size ramp-up: training often starts with a smaller batch size and gradually increases it. Early in training, the loss landscape is steep and smaller batches (with noisier gradients) help explore broadly. Later, larger batches provide stability and efficiency.
• Learning rate coupling: batch size and learning rate must be tuned together. The linear scaling rule (double batch → double LR) with warmup is a common starting point.

During inference (serving)

• Continuous batching: production LLM servers group multiple user requests into a single batch to maximise GPU utilisation. Larger inference batches increase throughput (tokens per second) but also increase per-request latency.
• KV cache memory scales with batch size: each request in the batch maintains its own key-value cache. Doubling the batch size roughly doubles KV cache memory, which is often the binding constraint on how many requests can be served simultaneously.
• Trade-off: smaller inference batches → lower latency per request; larger inference batches → higher overall throughput but each request waits longer.

Advanced

At engineering level, an LLM is not just a model checkpoint. It is part of a full system involving tokenization, inference infrastructure, retrieval, prompt management, observability, evaluation, privacy controls, and product constraints. Capability emerges from the interaction between the model and these system components.

Key engineering realities

• LLMs are powerful generalists but unreliable without scaffolding.
• Deployment quality depends on prompts, retrieval, validation, and user-interface design.
• Fine-tuning, quantization, and routing are business decisions as much as model decisions.
• Evaluation must cover correctness, latency, cost, safety, and user trust.

system_layers = {
    "model": "decoder-only transformer",
    "retrieval": "optional external knowledge",
    "validation": "schema and policy checks",
    "observability": "latency, cost, and quality traces"
}
print(system_layers)

The strongest practical mental model is that LLMs are probabilistic reasoning-and-generation engines that need surrounding structure to become reliable software components.