￦'s blog

Language Models

2 min read

강의_재생목록

Remember it’s all about efficiency:

  • Resources: data + hardware (compute, memory, communication bandwidth)
  • How do you train the best model given a fixed set of resources?

Today, we are compute-constrained, so design decisions will reflect squeezing the most out of given hardware.

  • Systems: clearly about efficiency
  • Tokenization: working with raw bytes is elegant, but compute-inefficient with today’s model architectures
  • Model architecture: many changes motivated by reducing memory or FLOPs (e.g., sharing KV caches, sliding window attention)
  • Data filtering: avoid wasting precious compute updating on bad / irrelevant data
  • Scaling laws: use less compute on smaller models to do hyperparameter tuning

Tokenization

a tokenizer converts between raw inputs (bytes) and sequences of integers (tokens)

Popular tokenizer: Byte-Pair Encoding (BPE)  

Efficiency lens

  • Reduce context length (1000 bytes → ~250 tokens)
  • Adaptive computation (more modeling capacity on interesting parts of input) The dream: tokenizer-free model architectures, which operate directly on bytes  
  • [Xue+ 2021]
  • [Yu+ 2023]
  • [Pagnoni+ 2024]
  • [Deiseroth+ 2024]
  • [Hwang+ 2025] These are promising, but have not yet been scaled up to the frontier.

Model architecture

Starting point: original Transformer  

Refinements:

1 item under this folder.