Left: immediate small reward
Take the currently more common left action and finish in one step.
- Actions
- Pending
- Rewards
- Pending
- Return
- Pending
Demo 06
Reinforcement Learning And Feedback Loops: How do outcomes change a future policy?
When a system cannot receive one uniquely correct answer for every step, the outcome after an action can become a learning signal. Two fixed episodes show how an immediate small reward and a delayed higher return can influence the next policy, while separating a training update from an agent revising its plan after an observation during one run.
Teaching interaction
Feedback Learning Loop Lab
When do rewards, preference comparisons, and runtime observations change the system?
Read the state and initial illustrative policy
The starting state has left and right actions. The initial policy is fixed at 70% left and 30% right; these values are not live samples.
Mechanism boundary Before training
A policy describes action tendencies. It does not tell the environment that one action is the uniquely correct answer at every step.
Two fixed episodes
These paths are scripted for comparison; no action is randomly sampled.
Right: delayed larger reward
Explore right, receive 0 first, then receive +3 on the second step.
- Actions
- Pending
- Rewards
- Pending
- Return
- Pending
Initial illustrative policy
70% left and 30% right; fixed values with no random sampling.
Outcome feedback enters an explicit update stage
- Model weights
- Training boundary: parameters may update
- Next action
- The updated policy affects later episodes
What kind of feedback is this?
The same word, feedback, can name signals that act at different stages.
| Signal | Input | When it acts | What it changes | Boundary |
|---|---|---|---|---|
| Target label | A target output paired with an input | Training | Directly describes the expected answer for a parameter update | Not the same as a long-term outcome in an environment |
| Reward / return | Outcomes after actions and cumulative sequence return | Training | Changes a value estimate or policy tendency | Does not guarantee one correct action at each step |
| Preference comparison | Candidate comparisons or rankings under a specified labeling protocol | Training | Can feed a reward model plus RL or another direct preference objective | Not universal truth and not one mandatory RLHF pipeline |
| Runtime observation | A current tool response or new environment state | Runtime | Changes the next action in the current task | Does not automatically become training data or a weight update |
Learning goals
- Distinguish target labels, reward and return, preference comparisons, and runtime observations.
- Understand that policy changes only inside an explicit training boundary and that ordinary agent runs do not update model weights automatically.
- See why delayed reward requires comparing complete episode outcomes rather than looking for one correct action at every step.
Simplification note
This demo uses two fixed paths and two scripted training episodes. It performs no random sampling or real optimization. Probabilities and rewards only explain how outcome feedback can change later policy; they do not model a real robot, game, or general decision problem. Reinforcement learning is a family of problem settings and methods, not one algorithm. RLHF is not all post-training or complete alignment.
Historical Position And Evidence Boundary
Feedback can shape a future policy, but not every feedback loop updates weights
Reinforcement learning is a family of problem settings and methods, not one algorithm. Q-learning gives an update rule that learns optimal action values under the paper's conditions; DQN combines deep networks, experience replay, and Q-learning for Atari; and AlphaGo combines supervised learning, reinforcement learning, policy and value networks, and tree search. Language-model preference comparisons can enter a reward-model-plus-PPO pipeline, as in InstructGPT, or a direct preference objective such as DPO. Human preferences remain bounded by the task, annotation protocol, and sample distribution. A runtime observation can change the next action in the current task without proving that model weights were updated online.
References
Concept Check
Test The Core Intuition With One Question
This check supports reflection and never blocks the next chapter. Records stay on this device.