End-to-End Experiment Troubleshooting

Use this guide when a complete experiment stops, duplicates work, produces an incomplete result, or reaches the wrong user-facing state.

ASCEND
  -> Composer
  -> PACE
  -> LabFlow
  -> LabBridge
  -> PACE result delivery
  -> PRISM analysis
  -> ASCEND next-round decision

The fastest reliable method is to start a new experiment from the real user entry point, record its identifiers, and inspect what each module received and handed to the next owner.

1. Define the Expected Result First

Write the expected outcome as directly observable states before running the test. For the four-round CoN4Cl XPS benchmark, that means:

one Experiment
├── Round 1: completed with an independent Analysis Result
├── Round 2: completed with an independent Analysis Result
├── Round 3: completed with an independent Analysis Result
└── Round 4: completed as a saved design, no equipment bound or execution started

“It runs” is not an acceptance criterion. State:

• how many rounds should exist;
• which rounds should complete automatically;
• what data and analysis each round should produce;
• where execution should pause and what the user should be asked to supply;
• which physical actions must never start automatically.

2. Start from the Real User Entry Point

Use the action a researcher would actually use, such as running a preset from ASCEND Home. Do not bypass ASCEND and manually create Composer or PACE records during the first test.

This checks the complete product path: preset discovery, Experiment creation, automatic advancement, cross-module handoff, and the final Completed, Problem, or Needs you presentation.

3. Use a New Experiment Every Time

After a fix, create a new Experiment instead of continuing from the failed one. An old record may retain a previous model response, duplicate Composer work, incomplete PACE delivery, a stale error, or an already-dispatched callback.

Track one consistent set of identifiers for the new run:

experiment / coordination id
iteration ids
composition job ids
PACE loop-run ids
LabFlow run ids
LabBridge operation and artifact ids
PRISM analysis-run ids

4. Find the First Incorrect Handoff

Inspect the chain in this order:

1. Did ASCEND create the expected Iteration?
2. Did Composer create exactly one plan/package attempt?
3. Did PACE create exactly one execution attempt?
4. Did LabFlow complete the required steps?
5. Did LabBridge commit the expected result or artifact?
6. Did PACE include that result in the delivery archive?
7. Did PRISM receive and validate the archive?
8. Did ASCEND use this Iteration's committed analysis for its decision?

At every boundary, answer only:

1. What did this module actually receive?
2. What did it actually hand to the next owner?

Stop at the first mismatch. Later failures are usually consequences. For example:

LabBridge committed an artifact
  -> PACE did not retain its artifact id
  -> PACE could not retrieve the bytes
  -> PRISM received no measurement
  -> ASCEND could not design the next round from evidence

Fix the PACE artifact handoff first; changing PRISM or ASCEND would only hide the original defect.

5. Prefer Durable Records over Transient Logs

Logs help locate an exception, but persisted records show the outcome the system will recover from. Inspect:

• Experiment and Iteration state;
• candidate and plan lineage;
• Composer job count and package hash;
• PACE run state and delivery archive;
• LabFlow step state;
• LabBridge Operation result, verification, artifact id, and checksum;
• PRISM committed Analysis Result;
• ASCEND decision and evidence references;
• the exact task and allowed actions shown in Needs you.

Do not diagnose from one UI message alone. It normally summarizes the terminal state, not the earliest failing module.

6. When a Model Response Is Wrong

Capture and compare three things before editing a prompt:

1. the bounded experiment evidence the model actually received;
2. the requested output meaning and schema;
3. the raw structured response returned by the model.

Common causes include missing analysis context, similar field names with different meanings, reuse of an old conversation, ambiguous ordering between “design the next round” and “ask the user,” or valid prose without the required structured result.

Prompt changes do not replace deterministic validation. Code must still reject out-of-bounds, duplicate, incomplete, or unsupported model output.

7. Fix One Owning Boundary at a Time

find the first mismatch
  -> preserve its input and output
  -> change the module that owns that state
  -> add a focused regression test
  -> run the focused test
  -> continue to the next boundary

Examples of ownership:

• Composer owns duplicate composition prevention.
• PACE and LabBridge own the terminal-operation handoff.
• PACE owns delivery completeness; PRISM owns input validation.
• ASCEND owns presenting missing physical conditions as Needs you.
• ASCEND deterministic validation rejects illegal or duplicate next-round plans.

Never hard-code the expected answer for one benchmark sample. The fix must apply to other experiments using the same contract.

8. Test from Small to Large

Use this order after each fix:

focused regression test
  -> owning module suite
  -> a new full Experiment
  -> broader repository qualification

For an artifact-delivery defect, first test artifact-id resolution, byte and checksum retrieval, delivery archive contents, and PRISM extraction. Then run a new end-to-end Experiment before the larger PACE/PRISM suites.

9. Classify the Failure

Data or provenance

Symptoms: missing, malformed, corrupt, or incorrectly attributed files.

Check artifact id, byte size, SHA-256, sample identity, source lineage, and archive contents.

Ordering or concurrency

Symptoms: intermittent failure, duplicate work, or an empty result.

Check for multiple advancers, a downstream step starting before terminal commitment, and device availability or heartbeat behavior.

Durable state

Symptoms: submitted information does not resume work, or a human task is shown as a generic failure.

Check persisted phase, blocking reason, backend-provided actions, decision application, and the state transition after recovery.

Model context or output

Symptoms: a stale result is cited, the wrong field is used, or advice is returned without a valid next-round plan.

Check the context snapshot, session/AgentRun id, prompt/output contract, raw response, and deterministic validation result.

10. Recheck Safety Boundaries

A passing happy path is not enough. Confirm that:

• missing physical SOP or setup facts stop execution;
• Needs you states exactly what the user must provide;
• no physical execution is created before qualification and approval;
• no fabricated measurement is committed;
• XPS composition is not presented as proof of coordination structure or mechanism;
• Pause, Stop, safe reset, and Take over remain effective.

Short Checklist

[ ] Define the expected final states
[ ] Confirm required services are healthy
[ ] Start a new Experiment from the real user entry point
[ ] Record the Experiment and downstream identifiers
[ ] Locate the first stopped or incorrect handoff
[ ] Compare that owner's actual input and output
[ ] Fix only the first owning defect
[ ] Add and run a focused regression test
[ ] Continue through the remaining boundaries
[ ] Run a new full Experiment
[ ] Recheck Needs you and physical safety boundaries
[ ] Run the relevant module and qualification suites
[ ] Record verified and unverified outcomes in Project Progress

Lessons from the CoN4Cl Benchmark

The benchmark exposed several independent boundary failures: concurrent advancers created duplicate Composer work; the virtual XPS device did not keep reporting availability; PACE read an Operation before terminal commitment and omitted LabBridge artifacts from the PRISM archive; ASCEND reused stale analysis context; missing fourth-round fields were ambiguous; the system asked the user before creating the proposed round; missing SOP data appeared as a generic failure; and submitted information did not restore the correct state.

The lesson is broader than this benchmark: module-local tests are necessary but not sufficient. Regular qualification must start from the real user entry point, use real evidence and a new Experiment identity, and complete the whole chain.