Editor's Synopsis
Andrej Karpathy proposes that the current LLM development landscape is missing a major learning paradigm. He tentatively calls it "system prompt learning" and frames it alongside the two existing paradigms: pretraining (which imparts knowledge) and finetuning via supervised learning or reinforcement learning (which shapes habitual behavior). Both of these involve changing model parameters, but Karpathy suggests there's a distinct category of learning that doesn't fit neatly into either.
The post is truncated, but the core insight points to a gap in how LLMs acquire and adapt behavior — potentially referring to how models learn from context, instructions, or system-level prompts at inference time without parameter updates. This idea resonates with ongoing discussions in the AI community about in-context learning, prompt engineering, and the boundary between training-time and inference-time adaptation. Coming from one of the most influential voices in deep learning, this observation is likely to spark significant discussion about future directions in LLM architecture and training methodology.