1:10 AM | Oct 23, 2026

// Coupled Learning and Planning from Limited Experience

How much competence can develop from limited experience when learning and planning support each other?

Limited experience can give little guidance about good decisions. Observations may be incomplete, and feedback may arrive only after an attempt ends. The learner can still retain useful values or routines.

Learning and planning are coupled through the state that the learner retains. Learning changes the values used to assess continuations or the routines used to propose actions. Planning assessments can contribute to later training targets. Selected trials can produce new experience. Further learning changes what the next planning step can use.

The benefit can accumulate across attempts. A retained improvement can make the same planning budget more useful. An acquired skill can also produce commands from a new observation without another search.

In Adapt-1, these changes occur within the substrate that retains evidence, organizes relationships and trains models. The experiments below examine this interaction through value learning and action acquisition.

Learning across attempts

An attempt can change what the learner uses on later attempts. These changes persist within Adapt-1's substrate. The substrate admits evidence, organizes relationships and trains models. A revised value can change an assessment. A retained execution can change a proposal. Learning can therefore change the basis of the next comparison.

Each mechanism sets conditions for the evidence it accepts and the changes it permits. New evidence can establish an association, support a more specific rule, or revise an existing rule. Repeated retrieval of records together can strengthen their association. Unused associations can weaken and disappear. Retained structures keep their supporting evidence and its source (Adapt-1 Preview).

Figure 1. Evolving field

Retained influence shapes later assessments, proposals and actions.

Conceptual view of retained learning. Experience changes the state used for later assessments, proposals and actions. Assessments can contribute to new training targets. Selected executions can supply new evidence. Further learning can change the next search.

Learning changes the information available to the next comparison. A retained value model can assess continuations when search stops before a final outcome. An acquired routine can supply commands for new proposals. A simulator can execute these proposals in full and measure their outcomes. The experiments below use both forms of comparison.

An assessment can contribute to a later training target. The observed outcome stays separate and retains a contribution to that target. The substrate fits and tests a candidate model from admitted evidence. A separate adoption step changes the deployed values. Those values then change later assessments.

For action acquisition, search selects a routine for separate execution. The executed commands and measured outcome enter learning through admission. A later proposal can preserve part of this execution and revise other commands. Selection therefore affects what the learner can reuse on later attempts.

This effect can extend across many generations of proposals. A selected execution can supply the basis for later trials, which can supply further revisions. A graded outcome can identify partial progress before an attempt meets every success condition. Learning can carry this progress forward while search continues to compare a small number of candidates.

Feedback can also change how the learner proposes its next revision. If confirmation rejects a correction, the current correction stays in use. The feedback can still reduce the range of later proposals.

Retained learning can also support direct action. A frozen skill keeps its acquired routine and correction function fixed. The current observation determines the commands. The system can use this skill without comparing a new set of candidates.

Value learning

Value learning changes the estimates used to assess possible outcomes. Planning, fitting, validation and adoption are substrate operations. The application supplies observations, possible transitions and measured outcomes. The substrate fits values from admitted observations and their targets. A retained estimate can influence a decision and contribute to a later fitting target.

Failure modes

A value function trained on a weak policy's returns estimates that policy's expected outcomes. Here, the learner assesses alternative actions and uses those assessments in later training. It must also use new outcomes and preserve its fit to earlier evidence. Different states can have the same observation. A value function that uses only this observation gives those states the same prediction.

Figure 2 shows four conditions that can cause errors in value learning. Existing methods address the first three through different uses of recorded evidence. Conservative offline methods penalize or exclude actions that the data does not support (Fujimoto et al., 2019; Kumar et al., 2020).

Under the conditions studied by Gerstgrasser et al. (2024), retaining real data with generated data can prevent model collapse. Rehearsal keeps earlier data in each update (Robins, 1995). Elastic weight consolidation adds a penalty computed from earlier data (Kirkpatrick et al., 2017). These methods preserve recorded evidence or a penalty derived from it.

Extrapolation

A learner can select an overestimate when it maximizes values for actions that its data does not support (Fujimoto et al., 2019). In this path, an assessment alone cannot replace the deployed value model. The substrate fits a candidate and tests it on held-out data. It then compares the candidate's decisions with those of the deployed model. Adoption is a separate step.

Self-training

Training on its own outputs can reinforce a model's prediction errors (Arazo et al., 2020). When generated data replaces real data, the distribution can narrow. Its tails can disappear first (Shumailov et al., 2024). Adapt-1 records each assessment with its provenance. The observed outcome stays in a separate field. Each target in the chess continuation retains a contribution from that outcome.

Interference

A fit that uses only new data can increase error on earlier data (McCloskey and Cohen, 1989). The substrate fits candidate parameters on a snapshot that includes earlier observations. The deployed model stays in use during the fit.

Aliasing

Different states with the same input receive the same predicted value. Differences in their true values remain unresolved. Lookahead can produce different targets for these states. To learn separate predictions, a later fit needs richer inputs that express the difference. The application can supply these inputs from the same stored states.

Figure 2. Failure modes and relevant mechanisms. These schematics do not show measured data. Each panel shows a possible failure in value learning and the relevant Adapt-1 mechanism.

Substrate planning uses retained values to assess possible continuations, including alternatives absent from recorded experience. An assessment can guide a choice and contribute to later learning. The record stores the observed outcome and assessment in separate fields. It also stores the assessment's provenance: the episode, step and parameter version that produced it.

A fit reads a fixed snapshot of observations and produces candidate value parameters. The deployed value model still serves decisions during candidate evaluation. A held-out test evaluates predictions. A fixed comparison evaluates decisions. Adoption installs the candidate in a separate step. Figure 3 shows this lifecycle within the substrate.

Figure 3. Model lifecycle

The deployed model serves decisions while the substrate fits and tests a candidate. Adoption is a separate step.

Planning and fitting are substrate operations. Records preserve the observed outcome, assessment and provenance. A continuation fit starts from deployed parameters. A held-out test and a fixed comparison come before adoption.

A target can combine a planning assessment with an observed outcome (Figure 4). The observed outcome retains a separate contribution to the target. Lookahead examines the states produced by transitions that the application supplies. The values assigned to these states are estimates.

Recorded return

One-ply lookahead

Target

y = λu + (1 − λ) $\bar{z}$

Figure 4. Evidence and estimates in one target. The backup diagrams use the notation of Sutton and Barto (2018). Solid lines show recorded transitions or transitions supplied by the application. Diamonds represent estimated values. The arc marks the action with the highest estimated value. The record stores the observed outcome z in a separate field.

The substrate uses the same crude experience for several learning operations. A first fit learns from observed outcomes. Planning then examines alternatives from each recorded state using transitions supplied by the application. Retained values assess these alternatives. A later fit uses these assessments in its targets.

The application can also supply richer inputs for the same states. Targets may differ for states that the initial inputs cannot distinguish. Richer inputs let the next fit learn those differences.

A separate object-graph path keeps each relation attached to the objects it connects. A projected representation converts objects and relations into counts. The application supplies the objects and relations in both paths.

In the graph path, the continuation fit also learns differences between alternatives. The substrate keeps related alternatives in comparison groups. Each group uses the same viewpoint for its value targets. The fit uses the individual targets and the differences between targets within each group.

For example, two alternatives with targets of 0.70 and 0.50 give a target difference of 0.20. The training objective includes both values and their difference. Differences between alternatives thus become an explicit part of the fit.

An assessment of an untried continuation is an estimate. A new interaction produces another observation. The two have different roles in further learning. Retained judgment can guide a later decision. The observed outcome supplies evidence for revision.

Chess

Chess tests whether Adapt-1 can learn useful values from limited experience in a strategic, adversarial environment.

The application supplies the board, legal moves, and the positions those moves produce. Adapt-1 begins with no learned chess values. Its initial records contain only twelve piece counts and each game's observed outcome. They contain no predefined piece values, position evaluations, or labels identifying better moves.

The opponent chooses its own replies and can actively reduce the learner's chances of winning. The rules determine which positions each move produces, but do not indicate which positions are strategically favorable. Because bounded search often stops before a game ends, Adapt-1 must use its retained value estimates to assess positions beyond its search horizon.

Chess therefore tests whether Adapt-1 can develop tactical and strategic competence from limited experience against an opponent. The goal is not chess mastery.

We use Stockfish 19 at its lowest skill setting (Level 0, 1,000 search nodes per move). This reduces its playing strength, but it retains its trained chess evaluation, search methods, and ability to identify tactical and strategic opportunities.

Coarse inputs

Twelve counts omit the positions of the pieces. For king and rook against king and knight with White to move, 10,780,728 legal placements share one count vector. Positions A and B in Figure 5 are two of them.

φ₀(s_A) = φ₀(s_B)

Both positions have the same twelve counts. A model fitted to these inputs therefore gives both positions the same prediction.

After Rxd5 in B, the Black knight count decreases from 1 to 0.

Figure 5. Two positions, one input. A and B differ only in the square of the White rook. Counts use K, Q, R, B, N, P order for each side.

Initial records

The initial collection contains 133,584 records from 761 completed games (Figure 6). In 640 of the 800 game attempts, a random player faced base Stockfish (beginner level). In the other 160, both sides played random moves. The records contain no engine move labels or evaluations.

Base Stockfish (beginner level) is a chess engine with its own chess knowledge, tactics and strategies.

These counts and observed outcomes can be accurate and still give little guidance about good decisions. The records do not show which alternative moves would have worked better.

Figure 6. Initial records

761 completed games · 133,584 perspective records

103,056 records (77.1%) follow random moves.

Record contents

Each record contains 12 piece counts and one eventual outcome. It contains no expert move label, engine evaluation or supplied strategy.

Repeated observations

Count

For 42.9% of records, the same input also appears with a different outcome.

Missed immediate wins

390 positions across 85 games offered mate in one, but the player eventually drew or lost. Bars count games with at least k such positions.

A later analysis checks which positions offered mate in one. The learning records contain piece counts and game outcomes.

A low target can follow a position with an immediate win available. In 390 positions across 85 games, the player had a mate in one but chose another move. Each game then ended in a draw or a loss for that player.

Reconsideration

Lookahead distinguishes A from B. In B, Rxd5 captures the knight and changes the counts. In A, no legal move changes the counts. After the first fit, substrate planning reconsidered the stored positions with one ply of lookahead. Targets for A and B can therefore differ, even though their inputs are identical.

The next fit needs an input that can express this difference. The application supplies a richer observation φ₁ with attack relations and turn context. In φ₁, the rook attacks the knight in B and not in A. The application defines these chess inputs.

A later run used a processed snapshot of 11,940 observations. The snapshot already contained counts of objects and attack relations. At prediction time, the substrate projects new objects and relations into the same form. These inputs include turn context and castling availability, but exclude piece coordinates. A fresh value model learned from the stored inputs and refined targets.

The separate graph path retains individual objects, their positions and the relations between them. Its continuation fit also uses the comparison groups described above.

Training targets

In the continuation, substrate planning used fitted values to play new games. Each game produced new records. Each target combines the search assessment with the observed outcome. Discounting moves the outcome toward ½ according to the number of steps before the game ends. In this run, λ = 0.75 and the discount is 0.99 for each step.

\bar{z}

= ½ + (z − ½) · 0.99^{T − k}y = λu + (1 − λ)

\bar{z}

u search assessmentz observed outcome: 0, ½ or 1k step of the recordT last step of the game

Target construction

The game ended in a loss. Lookahead supplied an assessment of an earlier position. Both signals contribute to the next fitting target.

Observed outcomeLoss at step 119

0.000000

Discounted outcome54 steps earlier

0.209417

Search assessmentBacked-up root value

0.522911

Target for the next fitAssessment + outcome

0.444538

The target combines the search assessment with the discounted outcome. The next fit adjusts Adapt-1's parameters to predict this target from the inputs.

Target calculation

The game ends in a loss at step 119. The displayed observation is from step 65. The target uses 75% search assessment and 25% discounted outcome.

For search budget B, the target combines assessment u_i and discounted outcome $\bar{z}$ _i:

u_i = Search_B(s_i; V_θ)y_i = 0.75u_i + 0.25

\bar{z}

Outcome contribution

\bar{z}

= ½ + (0 − ½) × 0.99⁵⁴

Discount toward the neutral value, 0.5.

Target construction

y = 0.75 × 0.522911 + 0.25 × 0.209417

The display rounds the components. The record retains full precision.

Recorded continuation target

Figure 7. One target from the continuation. The values come from a recorded target in the later continuation.

Continued learning

Later decisions help determine which new observations the learner receives. A continuation fit starts the candidate from the deployed parameters. Its data includes earlier and new observations. Shared parameters let an update for one input change predictions for other inputs. The earlier targets remain in the objective and limit those changes.

θ_candidate⁽⁰⁾ = θ_deployed𝒟_fit = 𝒟_earlier,fit ∪ 𝒟_new,fit

Model update

Figure 8. The continuation fit. Select a label to highlight its connections. Select it again to show the whole comparison.

Validation uses whole source games that the fit excludes. It tests how well the candidate predicts their targets. A paired comparison tests decisions with each parameter set, using fixed openings and search settings. Both parameter sets stay fixed during the comparison. The candidate stays staged until a separate adoption step.

Evaluation

The new records came from 25 completed games against the same Stockfish engine at a higher setting. Comparison openings do not occur in those records. No comparison game entered a fit.

Each parameter set played 50 games against each opponent setting. Against base Stockfish (beginner level), the candidate scored approximately 84% of the points and the deployed parameters approximately 56%. At the higher setting, the candidate scored approximately 48% and the deployed parameters approximately 30% on the same openings (Figure 9).

Figure 9. Results

Base Stockfish (beginner level)

50 games each · paired openings

Stockfish · higher setting

50 games each · paired openings

Bars show the approximate share of available points. Outcome columns show approximate percentages. Each parameter set played 50 games against each opponent setting, using paired openings and colors. Parameters stayed fixed, and planning used the same settings for both sets. A game with no outcome after 300 further plies stops and scores 0.

Figure 9 compares decisions made with revised values under the same planning settings. The comparison tests what changed estimates make possible within the same search. The retained value model connects earlier experience to later behavior.

Machina

Machina examines how Adapt-1 acquires procedures for action. Earlier work on this docking task produced no successful completions. In the articulated-trailer experiment, a routine has eight commands and two continuous controls per command. These commands must coordinate the approach, steering and braking. Feedback measures the outcome of the whole attempt. It does not supply the correct control value for each command (Machina overview).

Retained experience supplies the basis for later attempts. The learner records the commands that ran and their measured consequences. A new proposal can use an earlier routine as its parent, preserve an initial span and revise selected controls or the continuation. These revisions use the recorded execution, which can differ from the proposal because of control limits or early termination (Acquisition guide).

Four Machina trajectory panels. Parent A and Refined A form the first pair. Parent B and Refined B form the second pair. Height represents elapsed time. — Parent routines and refined routines in two docking cases. Height represents elapsed time.

Acquisition

Acquisition starts without a supplied successful routine or a pretrained task policy. At each choice, the learner proposes four complete routines. Both runs use the same native learner, configuration and requested seeds. Each run produces 2,048 proposals and admits 512 separately executed routines.

The search-assisted run uses native search for 384 choices and reserves 128 for unfiltered exploration. The control makes all 512 choices without search. Search compares terminal outcomes supplied by the simulator. The selected routine then runs separately, with no mid-sequence replanning. Its executed commands and measured outcome enter learning.

The simulator measures the quality of each complete candidate. This graded score can distinguish incomplete attempts before a routine meets every success condition. Search has depth one because one choice covers the full eight-command routine.

Some robot planners compare large sets of candidates and refine them over repeated rounds before execution (planning examples; sequence optimization). A short horizon can therefore still involve many candidate evaluations. Here, each acquisition choice compares a small set of complete routines. The selected routine runs separately, and its outcome enters learning. This experience can change the routines proposed next. The comparison stays small while the state that supplies its candidates changes across attempts.

In chess, bounded lookahead often ends while the game remains undecided and the opponent can still reply. Retained values assess those unfinished continuations. Here, acquired procedures supply the candidates, and the simulator measures the result of each complete execution.

Figure 11. Four candidates, changing state

Retained experience

Executed commands and observed outcomes

Four complete routines

Eight commands per routine · two controls

A search-assisted acquisition cycle. The selected routine runs separately. Admission changes the learned state, which supplies later proposals. Those proposals can reuse and revise the admitted commands. The surface and traces are schematic, with no measured values.

Figure 12. The next four proposals

Before execution 5: both arms recall record 0 and receive the same four candidates.

Unfiltered acquisition

Execution 5Proposal 1 0.061

Executed quality

After admissionRecord 0

Earlier parent remains in use.

Execution 6Four variants of record 0

Proposals use the earlier parent.

Search-assisted acquisition

Execution 5Proposal 2 0.413

Executed quality

After admissionRecord 4

The new execution becomes the parent.

Execution 6Four variants of record 4

All four matrices differ from the unfiltered arm.

The requested seed is the same at execution 6. The retained experience and proposed commands differ.

Search selects a routine for separate execution. In this transition, admission changes the parent used for the next proposals. Qualities are rounded from 0.060926 and 0.413023. Execution numbers start at one; record IDs start at zero.

Figure 12 follows the first change in the proposed action sets. The better execution becomes the parent for the next batch. With the same requested seed, all four command matrices differ between the runs, as do the edited coordinates. Across both runs, every proposal after initial exploration refers to an admitted routine. The search width stays at four while experience changes the routines proposed.

The benefit can accumulate across generations of proposals. A selected execution can become the parent of later trials. Those trials can supply further retained revisions. Each search choice compares four candidates; acquisition repeats across hundreds of rounds. Both runs propose 2,048 routines; the selection rule differs for 384 choices. Admission carries that difference into the experience available for later proposals.

The graded score can identify a better attempt while the approach, alignment and braking still need revision. A routine can reach the bay before it combines these parts into a complete maneuver. Further proposals can preserve useful changes until an attempt succeeds. The first strict success in this run occurs at acquisition execution 351.

The best routine from unfiltered acquisition reached the bay but still moved backward at 1.29 m/s. The routine from search-assisted acquisition arrived accurately and nearly stopped. These routines gave later refinement different starting behavior.

Contextual refinement learns bounded corrections to an acquired routine. The routine acquired with search already combined accurate arrival with braking. Refinement could adjust this behavior for different starts. The unfiltered routine still lacked the stop needed to complete the maneuver.

Search selects which routine will run. Its outcome enters learning through admission. In the recorded acquisition states, simulation and planning leave the exposed learned signature unchanged. Admission of the separately executed routine changes it.

After acquisition, selection compares retained routines across starts. Deletion tests whether commands or spans can be removed. Ordering tests when control changes occur. Each stage compares candidate executions and checks them on confirmation starts before adoption. The confirmed routine becomes the base for the next stage (Structural stages).

One acquired routine can work differently from different starts. Contextual refinement learns how the starting observation should change its commands. For a new start, the learned function produces an offset for each control at each command position. The application combines these offsets with the retained base using fixed scaling and control bounds (Contextual refinement).

A frozen skill keeps the base and correction function fixed. Its output can still change with the starting observation. Direct execution supplies this observation, computes the correction and runs the resulting commands. It needs no batch of candidate executions at that point (Retained use).

Figure 13. A retained skill

Current observationc_A

Acquired baseB

Command	Control 1	Control 2
1	b₁₁	b₁₂
2	b₂₁	b₂₂
3	b₃₁	b₃₂
4	b₄₁	b₄₂
5	b₅₁	b₅₂
6	b₆₁	b₆₂
7	b₇₁	b₇₂
8	b₈₁	b₈₂

Context correctionΔ(c_A)

Command	Control 1	Control 2
1	δ₁₁	δ₁₂
2	δ₂₁	δ₂₂
3	δ₃₁	δ₃₂
4	δ₄₁	δ₄₂
5	δ₅₁	δ₅₂
6	δ₆₁	δ₆₂
7	δ₇₁	δ₇₂
8	δ₈₁	δ₈₂

Fixed
bounds

Executable routineU(c_A)

Command	Control 1	Control 2
1	u₁₁	u₁₂
2	u₂₁	u₂₂
3	u₃₁	u₃₂
4	u₄₁	u₄₂
5	u₅₁	u₅₂
6	u₆₁	u₆₂
7	u₇₁	u₇₂
8	u₈₁	u₈₂

Base stays fixedSame correction ruleCommands for this start

Acquisition and structural stages supply the base. Contextual training changes the correction rule. Frozen use holds both fixed and computes corrections for the current observation without comparing alternatives. Composition uses the task's fixed scaling and bounds. Symbols and colors are schematic.

Both runs start with zero offsets and learn corrections from two values that describe the starting context. The configuration specifies normalization and a quadratic basis, so the correction can vary nonlinearly with context. It also specifies guidance from measured execution responses and a trust region that limits proposed changes (Configuration).

Contextual refinement uses sixteen updates. Each update compares four versions of the correction function, including the current function, on the same twelve training starts. The simulator measures all 48 outcomes, giving 768 evaluations per run in addition to acquisition and structural comparisons. Six starts nominate a candidate and six confirm it; the two reused groups alternate roles. An accepted candidate becomes the retained correction for subsequent use.

The search-assisted run accepts eight of sixteen updates. Its retained mean quality on the twelve training starts rises from 0.450 to 0.826. At each boundary between updates, all 192 predicted offsets on those starts match the outputs selected at the preceding update. The selected correction persists into later use.

Feedback can also change how the learner proposes its next correction. On the final update, confirmation rejects the nominated correction. The current function stays in use, while the reported search radius halves from about 0.146 to 0.073. The learner admits the feedback and limits the variation in later proposals.

Retained behavior

The skill acquired with search was frozen and evaluated on 64 new starts. Direct execution computes commands from each start using the retained skill. A separate evaluation condition compares 27 complete candidate routines with search. The earlier unfiltered control was evaluated on 32 starts.

Figure 14. Frozen skills on new starts

Unfiltered acquisition · earlier 32-start panel

Search-assisted acquisition · updated 64-start panel

Direct execution uses the retained contextual skill. Evaluation search assesses 27 complete candidate routines. Both skills remain frozen. The updated skill produces 18 more successes with evaluation search. The earlier control and updated evaluation use different panel sizes.

The skill acquired with search succeeds on 34 of 64 starts through direct execution, or 53.1%. This tests the acquired routine and its learned correction together. Evaluation search increases the result to 52 of 64 starts, or 81.25%. The earlier unfiltered skill completed none of its 32 test starts under either condition. The docking task now produces useful behavior from a retained skill, including direct execution without further search.

These results follow 512 admitted acquisition executions, structural comparisons and 768 contextual evaluations per run. They cover one pair of training histories. The runs have equal proposal and admission counts, but their total simulation and compute costs have not been established as equal.

YAM placement success also rises from 60.42% to 76.3%. AIM's reported hit rate stayed flat. Read earlier study for further context.

Visit the Adapt-1 documentation for API details and examples.