- Core Question
- How did outcomes after actions enter value learning, deep representations, preference post-training, and agent systems?
- Curatorial Boundary
- This is a reviewed teaching path, not the unique, complete, or proven causal history. It highlights how different systems reuse feedback signals while every unselected event remains part of the full history. The Samuel step represents an early combination of search, evaluation, and experience updates; it does not retroactively label the 1959 work as the complete reinforcement-learning paradigm later formalized.
Step 1 of 6
Samuel Checkers: Experience Changes Later Choices
- What It Inherited
- It inherited game-tree search and hand-designed board representations.
- What It Solved
- Playing experience updated an evaluation function and changed later search tendencies.
- What Was Still Missing
- It did not yet provide a unified later reinforcement-learning formulation, deep representations, or a general convergence result.
Step 2 of 6
Q-learning: Outcome Feedback Updates Action Values
- What It Inherited
- It inherited sequential decisions described by states, actions, and cumulative outcomes.
- What It Solved
- It supplied an incremental action-value update and a convergence result under the paper's conditions without target actions at every step.
- What Was Still Missing
- Tabular or discrete representations did not directly handle high-dimensional pixels, and the theorem did not automatically cover finite data with function approximation.
Step 3 of 6
DQN: Deep Representations Enter Value Learning
- What It Inherited
- It inherited the Q-learning action-value objective and CNN visual representations.
- What It Solved
- A deep network and experience replay learned control policies for multiple Atari games from pixel input.
- What Was Still Missing
- Results on game benchmarks did not establish reliable real-world control or remove exploration, stability, and generalization problems.
Step 4 of 6
AlphaGo: Learning Recombines With Search
- What It Inherited
- It inherited supervised learning, reinforcement learning, policy and value networks, and classical tree search.
- What It Solved
- Learned evaluations worked with Monte Carlo tree search in Go's enormous action space.
- What Was Still Missing
- The result cannot be reduced to reinforcement learning replacing search, nor can Go performance establish general intelligence.
Step 5 of 6
InstructGPT: Preferences Enter Language-Model Post-Training
- What It Inherited
- It inherited pretrained language models, supervised demonstrations, candidate rankings, and policy optimization.
- What It Solved
- A reward model and PPO made outputs more preferred under the study's tasks, labeling protocol, and evaluation distributions.
- What Was Still Missing
- Preferences are not universal truth, being preferred does not guarantee factuality or complete safety, and preference optimization has other routes.
Step 6 of 6
Agent / Safety: Runtime Feedback Is Not A Training Update
- What It Inherited
- It inherited language models, tool calls, and plan revision driven by observations.
- What It Solved
- External responses can change the next action in a current task, with permissions, stop conditions, and evaluation constraining action.
- What Was Still Missing
- A retry during one run does not prove online weight updates. Reliability still needs an explicit training pipeline and ongoing safety evaluation.
- Core Question
- When search, rules, uncertainty modeling, and hand-crafted features each met their limits, how did AI gradually shift toward data-driven and learnable representations?
- Curatorial Boundary
- This curated path shows changing and recombined problem representations, not a single line in which probability replaced rules, classical machine learning replaced probability, and deep learning replaced everything before it. The routes continued to overlap. The ImageNet and AlexNet steps only highlight how shared data, evaluation, and compute combined with existing convolutional mechanisms. Every unselected event remains part of the full history.
Step 1 of 7
A*: Costs and Heuristics Organize Search
- What It Inherited
- It inherited the representation of path problems as graphs, states, and comparable costs.
- What It Solved
- Under explicit conditions, path cost and a domain heuristic could jointly determine node expansion order.
- What Was Still Missing
- The heuristic still came from designers. The method did not learn features from data or automatically represent uncertainty.
Step 2 of 7
MYCIN: Domain Knowledge Becomes Explainable Rules
- What It Inherited
- It inherited explicit knowledge representation, rule matching, and constrained problem solving.
- What It Solved
- Rules, certainty factors, and explanation mechanisms supported a constrained infectious-disease consultation problem.
- What Was Still Missing
- Strong performance depended on narrow domain knowledge, while knowledge acquisition, exception maintenance, and accountability remained bottlenecks.
Step 3 of 7
Bayesian Networks: Uncertain Dependencies Become Graphs
- What It Inherited
- They inherited the problems of structured knowledge representation and updating judgments from evidence.
- What It Solved
- Directed graphs represented conditional dependence and put evidence propagation and probabilistic reasoning into one framework.
- What Was Still Missing
- The graph structure and variables still required modeling, and the framework did not directly learn task features from high-dimensional raw input.
Step 4 of 7
SVMs: Samples Determine a Maximum-Margin Boundary
- What It Inherited
- They inherited a classification problem made from labeled samples, feature representations, and an optimization objective.
- What It Solved
- Inputs were mapped into a high-dimensional feature space, where support vectors determined a maximum-margin boundary.
- What Was Still Missing
- Learning a classification boundary did not automatically produce a multilayer representation for every task or a complete perception system.
Step 5 of 7
LeNet: Convolutional Representations Enter Document Pipelines
- What It Inherited
- It inherited multilayer network training while reusing local connectivity and shared parameters.
- What It Solved
- Convolutional networks and global training entered real handwriting and cheque-reading pipelines.
- What Was Still Missing
- Success on a constrained document task did not establish that the same data and compute scale covered broad visual problems.
Step 6 of 7
ImageNet: Shared Data Makes Vision Progress Comparable
- What It Inherited
- It inherited object recognition's need for labeled examples, organized categories, and a common evaluation task.
- What It Solved
- A large image collection organized by the WordNet hierarchy provided a shared base for training and comparable evaluation.
- What Was Still Missing
- A dataset did not select or train a model by itself, and its label hierarchy bounded the questions it could measure.
Step 7 of 7
AlexNet: Algorithm, Data, and GPU Scale Combine
- What It Inherited
- It inherited deep convolutional networks, the ImageNet task, and parallel-compute conditions.
- What It Solved
- An efficient GPU implementation trained a deep CNN on about 1.2 million images and sharply reduced ImageNet error.
- What Was Still Missing
- A vision-benchmark breakthrough did not make rules, probabilistic reasoning, or classical search obsolete, nor did it solve general vision.
- Core Question
- When scaled language models enter concrete applications, why might they still need external knowledge, post-training, action loops, and ongoing risk evaluation?
- Curatorial Boundary
- This is a reviewed system-boundary path, not a pipeline that every application must follow or a proven linear causal history. The steps address different problems and may be combined, replaced, or omitted. RAG does not thereby update model weights, preference post-training does not guarantee truth or safety, and a runtime observation is not a training update. Every unselected event remains part of the full history.
Step 1 of 7
The Transformer: Parallel Attention Provides a Scalable Backbone
- What It Inherited
- It inherited neural sequence transduction and the problem of learning token representations from data.
- What It Solved
- Attention replaced the recurrent and convolutional backbone, directly connected tokens, and improved training parallelism.
- What Was Still Missing
- An architecture alone did not determine training scale or automatically provide current knowledge, reliable action, or risk governance.
Step 2 of 7
Scaling Laws: Scale Becomes a Measurable Direction
- What It Inherited
- They inherited language-model prediction loss and engineering choices across model, data, and compute scale.
- What It Solved
- Within the study's scope, power-law relationships connected cross-entropy loss with model size, dataset size, and training compute.
- What Was Still Missing
- Lower training loss alone did not establish reliability, safety, or alignment with a concrete user's intent.
Step 3 of 7
GPT-3: Task Adaptation Enters the Text Context
- What It Inherited
- It inherited large-scale pretrained language models and a natural-language generation interface.
- What It Solved
- Instructions and a few prompt examples supported many tasks without updating model weights.
- What Was Still Missing
- In-context adaptation did not guarantee current knowledge or verifiable answers, and it did not itself grant external action capabilities.
Step 4 of 7
RAG: Retrievable Knowledge Enters Generation Context
- What It Inherited
- It inherited a pretrained generator, a neural retriever, and an external document index.
- What It Solved
- Parametric memory, updateable external knowledge, and generation became separable system components.
- What Was Still Missing
- Retrieved passages entered only the current context. They did not update model weights or guarantee a correct final answer.
Step 5 of 7
InstructGPT: Demonstrations and Preferences Shape Assistant Behavior
- What It Inherited
- It inherited pretrained models, supervised demonstrations, candidate rankings, and policy optimization.
- What It Solved
- Outputs became more preferred within the study's tasks, labeling protocol, and evaluation distributions.
- What Was Still Missing
- Preferences are not universal truth, being preferred does not guarantee factuality or complete safety, and post-training has other routes.
Step 6 of 7
ReAct: Observations Change the Next Action
- What It Inherited
- It inherited language-model reasoning, calls to external sources, and plan revision.
- What It Solved
- Reasoning traces, environment actions, and observations alternated so responses could update the current plan.
- What Was Still Missing
- A runtime observation was not a weight update. Tool errors, permissions, and stopping conditions still required system constraints.
Step 7 of 7
The NIST GenAI Profile: Risk Evidence Becomes an Ongoing Process
- What It Inherited
- It inherited the AI RMF lifecycle framework for managing risk.
- What It Solved
- Red teaming, incident records, ongoing evaluation, and risk treatment were mapped onto generative-AI risk management.
- What Was Still Missing
- Governance guidance is not a model capability and cannot prove that an organization implemented controls or permanently removed risk.