Knowledge / Generative AI

Large Language Models

Token prediction, scaling, pretraining, contextual representation, decoding, and emergent task capabilities.

By Siddhant Krishna · Published 2026-10-06 · Updated 2026-10-06

Language Modeling

Autoregressive language models optimize the probability of the next token conditioned on previous context.

L(θ) = -Σ_t log P_θ(x_t | x_{<t})

Scaling

GPT-3 provided an influential empirical demonstration that increasing language-model scale could substantially improve few-shot performance. Modern systems continue to explore scaling across parameters, data, training computation, inference-time computation, and context.

References

Vaswani et al. (2017), Advances in Neural Information Processing Systems.
https://arxiv.org/abs/1706.03762
Brown et al. (2020), Advances in Neural Information Processing Systems.
https://arxiv.org/abs/2005.14165

Pretraining and Post-TrainingThe lifecycle from raw data to useful generative-model behavior.Alignment and RLHFInstruction following, preference learning, reward modeling, and behavioral alignment.

Language Modeling

Scaling

Get in Touch