Self-Driving Lab Architecture Plan

1. Vision and Core Requirements

The goal is to build a self-driving lab for continuous materials discovery with minimal human intervention.

Core principles:

• Exploratory experiment workflows must be reviewed and approved by humans before execution.
• Safety-critical operations must include human review and, where necessary, direct human involvement.
• Low-level device actions should be simple, deterministic, observable, and stoppable.
• High-level reasoning, planning, orchestration, and state aggregation should stay in the gateway, workflow, and agent layers instead of being embedded in ESP32 firmware.

2. Architecture Diagrams

The current technical route is captured as PlantUML source files. They are kept as .puml files so they can be previewed directly in VS Code:

diagrams/sdl-overall-architecture.puml
diagrams/lab-architecture-detailed-roadmap.puml
diagrams/labflow-labbridge-implemented.puml
diagrams/labflow-labbridge-quick-view.puml

The SDL overall diagram is the smallest technical view of agents, loop execution, gateway, devices, and data domains. The detailed roadmap diagram shows the broader technical route, including communication planes, robotics, edge integration, and data feedback. The implemented labflow/labbridge diagram is a code-level current-state map for AI onboarding. The labflow/labbridge quick-view diagram is the compact human-facing view of the current runtime boundary.

3. Overall Technical Route

The system is organized into five major layers:

• Planning and approval: the Agent / LLM creates structured experiment plans, while the workflow engine coordinates approval-aware execution.
• LabBridge: the gateway exposes only approved device capabilities to the agent and enforces policy checks before commands or operations are issued.
• Communication planes: different traffic types use different control paths instead of forcing everything through one protocol.
• Lab edge and devices: device controllers, ESPHome nodes, serial adapters, mobile robots, robotic arms, cameras, and physical I/O devices execute concrete actions.
• Data and feedback: experiment knowledge, relationships, telemetry, media artifacts, state snapshots, logs, and multimodal reviews feed the next planning cycle.

The preferred control boundary is:

Agent / LLM or Workflow Engine
  -> LabBridge / Gateway Service
  -> Capability and Operation Registry
  -> MQTT control plane for lightweight devices
     or Device Controllers for complex devices
  -> robot / media / instrument / simulation control paths
  -> ESPHome nodes / robots / cameras / instruments
  -> sensors / actuators / physical controls / artifacts

4. Hardware Layer and Hybrid Communication

As multimodal perception and robotics become part of the lab workflow, a single communication protocol is not enough. The architecture therefore uses several communication planes with clear responsibilities.

ESPHome-Based Lightweight Device Access

ESPHome nodes are used for simple physical I/O:

• Temperature and humidity sensors
• Relays
• Displays
• Control knobs
• Buttons
• Simple servo or GPIO-based actions

ESP32 nodes should not perform complex reasoning or long-running orchestration. Their role is to expose reliable local I/O actions and report state.

Local Hardware Safety Fallback

For dangerous operations, local protection must not depend on remote instructions. Examples include overheating, abnormal pressure, or unsafe actuator duration.

The ESP32 or nearby hardware controller should be able to execute a local fallback rule such as:

unsafe threshold detected -> cut relay / stop actuator / hold safe state

This fallback is separate from agent approval. Approval reduces operational risk, while local hardware fallback protects against network, software, or orchestration failures.

Legacy Instrument Adaptation

Traditional serial instruments should be connected to an edge gateway such as a Raspberry Pi or NUC. The edge gateway can run containerized adapters that normalize instrument commands, state, and telemetry into the lab control network.

This approach keeps existing serial-device integration close to the hardware, reduces deployment complexity, and prevents the agent from depending directly on raw serial protocols.

Communication Planes

• MQTT control plane: low-rate telemetry, retained manifests, device state, and simple commands for ESPHome nodes and instrument adapters.
• Robot control plane: real-time motion coordination, navigation, arm manipulation, and collision-aware robot execution.
• Media and model plane: camera streams, snapshots, large experiment payloads, and bench-specific robot policy artifacts.

The key design point is that each plane serves a different operational need. Simple device I/O should remain lightweight and observable, while robot motion and media/model transfer use channels better suited to their latency and payload requirements.

The workflow engine and agent should not bypass the gateway to talk directly to robot, media, serial, camera, or instrument SDKs. Complex device traffic is mediated by registered device controllers, and the gateway sees those actions as capabilities and operations.

5. Embodied Robotics and Multimodal Execution

Robotics and vision extend the lab beyond fixed workstations and make it possible to physically connect multiple process steps.

Visual State Confirmation

Camera nodes provide snapshots or video for scene checks. A vision service can verify whether the expected physical state is present before an action continues.

Example:

Before the robot arm starts a pick operation, the system checks whether the target vial is inside the marked bench region.

Visual confirmation should be treated as a workflow signal, not as a direct low-level actuator command.

Mobile Robot and Flexible Bench Connection

An AMR can transport samples, containers, fixtures, or tools between workstations. The robotic arm can then perform loading, unloading, sample handling, or manipulation tasks at each bench.

This makes the lab layout more flexible because process steps do not need to be hardwired into a single fixed station.

Bench-Specific Robot Policies

A robotic arm mounted on an AMR may need different control policies or calibration profiles at different benches. For example:

• A weighing bench may require precise powder scooping or dosing.
• A reaction bench may require cap handling, tube insertion, or vial placement.
• A camera inspection bench may require repeatable positioning for imaging.

The architecture therefore includes a model manager that can load bench-specific robot policies or model artifacts based on the current location and task context.

6. Data and Feedback Layer

The architecture uses several data stores, named by purpose rather than by concrete database product.

• Knowledge store: SOPs, prior experiments, operating notes, literature-derived process knowledge, and RAG context.
• Relationship store: links between materials, formulations, process conditions, outcomes, and experimental hypotheses.
• Telemetry store: time-series history for temperature, humidity, speed, pressure, actuator state, device availability, and other sensor readings.
• Object store: MinIO/S3-compatible storage for images, videos, snapshots, experiment artifacts, generated reports, robot policy files, model outputs, and logs.
• State cache: latest device snapshot for fast gateway and agent reads.
• Observability store: logs, metrics, traces, errors, workflow events, and audit records.

This separation keeps the architecture flexible. Product selection can be made later according to deployment size, query patterns, reliability requirements, and operational constraints.

7. Closed-Loop Workflow

The workflow is organized around an LLM - Action - Feedback - HITL loop.

Planning Stage

The Agent / LLM generates a structured experiment protocol, such as JSON or YAML. The protocol should describe the intended steps, required devices, expected parameters, safety constraints, and required approvals.

Approval Stage

The agent submits the workflow for review. The workflow engine pauses execution when approval is required.

Human approval can be performed through a web UI, a mobile interface, or a dedicated lab control device. The approval decision is recorded for audit.

Execution Stage

After approval, the workflow engine dispatches actions through the gateway. The gateway then routes each action through the correct device boundary:

• Simple device commands are sent through the MQTT control plane.
• Robot, arm, camera, mixer, instrument, and simulated-device tasks are created as gateway operations and executed by device controllers.
• Device controllers may use the robot control plane, media and model plane, serial adapters, local SDKs, or simulation APIs internally.
• Vision checks, image capture, large payloads, and model artifact transfers return explicit results or artifacts to the gateway/workflow instead of being treated as hidden side effects.

Safety-critical steps should pause automatically until the required approval or confirmation is available.

Feedback Stage

After execution, the system collects telemetry, images, logs, and experiment artifacts.

The feedback process can:

• Extract important telemetry segments, peaks, plateaus, outliers, and state changes.
• Render time-series curves for review.
• Combine images, curves, and structured records into a multimodal result summary.
• Feed the summarized result back to the Agent / LLM for the next iteration.

8. Current LabBridge Prototype

The current labbridge/ implementation is a working prototype gateway for two classes of lab devices:

• MQTT-native lightweight devices, especially ESPHome nodes and simple instrument adapters that publish retained manifests, state, telemetry, and command topics.
• Controller-backed devices, such as cameras, AMRs, robot arms, mixers, instruments, and simulated devices whose SDK calls, planning logic, vision logic, recipes, or private protocols should stay outside the gateway process.

It does not yet handle full experiment planning. labflow/ owns workflow validation, run state, approval, and safe-stop orchestration. labbridge/ owns device/capability metadata, policy checks, operation creation, status tracking, audit logs, Admin Web visibility, and MCP exposure.

Core boundaries:

• ESP32: simple I/O, state reporting, retained manifest, and local hardware fallback.
• MQTT control plane: device state, telemetry, manifest, and command topics for lightweight devices.
• Device controllers: local or edge processes that own hardware SDKs, navigation, obstacle avoidance, arm motion, camera capture, instrument protocols, simulation logic, and artifact storage for complex devices.
• Gateway Service: synchronizes MQTT state, maintains the registry, creates operations, tracks operation status, and performs permission checks.
• Admin Web: assigns lab semantics and controls which capabilities are exposed.
• FastMCP Server: exposes only approved tools and resources to the Agent / LLM.
• Workflow Engine: calls the gateway for approved hardware actions and never talks directly to MQTT, GPIO, serial, robot SDKs, or camera SDKs.

Current Data Flow

MQTT-native lightweight path:

Agent / LLM
  -> MCP Streamable HTTP or WebSocket
  -> labbridge FastMCP Server
  -> Gateway Service
  -> Capability Registry
  -> MQTT control plane
  -> ESP32 ESPHome nodes
  -> sensors / relay / display / simple controls

Controller-backed complex device path:

Agent / LLM or labflow
  -> MCP or HTTP gateway API
  -> Gateway Service
  -> Operation Registry
  -> Device Controller
  -> robot / camera / instrument / simulation SDK
  -> physical or simulated device

The controller reports operation status and terminal results back to the gateway. If the capability has wait_for_result enabled, call_capability can block until a terminal result or timeout. Otherwise it returns an operation_id and callers should use get_operation(operation_id) for precise status checks.

Gateway Processes

run.sh currently starts four processes:

• listener: subscribes to device manifests, availability, state, telemetry, and actuator state topics.
• admin: provides the management UI for device descriptions, workspaces, benches, capability exposure, and control permissions.
• mcp-ws: serves FastMCP WebSocket and Xiaozhi MCP endpoints, with the default MCP endpoint at ws://localhost:8090/mcp.
• mcp-http: serves FastMCP Streamable HTTP, with the default endpoint at http://localhost:8091/mcp.

Registry Model

The current registry tracks:

• devices: device id, friendly name, workspace, bench, online state, and manifest.
• capabilities: sensors, states, actuators, displays, and controller-backed operations; direction; unit; payload schema; MCP exposure; agent control permission; wait-result behavior; and controller metadata.
• states: latest retained or live state cache.
• workspaces and benches: lab-level semantic locations that are not written into firmware.
• operations: operation id, capability id, payload, status, result, error, cancelability, and timestamps.
• audit_logs: control actions issued by the agent or admin interface.

The prototype uses a local registry database for both capability metadata and latest state snapshots. A separate state cache can be added later if lower-latency reads or higher write volume become necessary.

Capability Exposure Rules

Default import policy:

• sensor and state: expose_to_mcp=true, allow_agent_control=false.
• actuator and display: expose_to_mcp=false, allow_agent_control=false.

The agent can control a capability only when all of the following are true:

• The capability is explicitly marked expose_to_mcp=true.
• The capability is explicitly marked allow_agent_control=true.
• The capability is a permitted write target, such as actuator, display, or a controller-backed operation capability.
• The device is online, unless offline control is explicitly enabled.
• For MQTT-native devices, the write topic comes from a valid device manifest and is not inferred from a state topic or subscription pattern.
• For controller-backed devices, the controller registration defines the command schema, operation behavior, and result shape.

For long or asynchronous work, the gateway returns an operation_id. Clients should use get_operation(operation_id) when they already know the operation id instead of repeatedly calling list_operations(). list_operations() is for overview and troubleshooting, not tight polling.

Current MCP Tools

• list_devices(): list compact agent-facing devices exposed to MCP, without internal routing fields such as source, controller_id, or display-only friendly_name.
• get_device_capabilities(device_id): inspect exposed capabilities for a device.
• get_device_readings(device_id): inspect readable capability cache for a device.
• read_capability(device_id, capability_id): inspect one current MCP-exposed readable capability value.
• list_workspaces(): list workspaces.
• list_benches(workspace?): list benches.
• get_bench_state(bench_id): inspect current MCP-exposed state for all devices assigned to one bench.
• call_capability(device_id, capability_id, input, workflow_token?): call an exposed capability, including controller-backed operation capabilities. Normal MCP clients omit workflow_token; the signed token is reserved for internal labflow wrappers.
• list_operations(status?, device_id?, controller_id?, limit?): list recent controller-backed operations for overview, recovery, and troubleshooting.
• get_operation(operation_id): inspect one operation by id.
• cancel_operation(operation_id, workflow_token?): request cancellation when the operation is cancelable.
• get_operation_trace(operation_id, view?): inspect a compressed operation trace.
• get_workflow_trace(workflow_token, view?): inspect a compressed signed labflow run trace.
• get_device_usage(device_id, limit?, time_from?, time_to?): inspect recent usage for one device.

Admin HTTP audit logs can be filtered by tool, device_id, capability_id, workflow_id, and time range. For labflow runs, the workflow runner signs a workflow_token; Gateway verifies that token and records the signed workflow_id in structured audit context. Ordinary MCP clients do not pass workflow_id, actor, source, request_id, or approval fields as tool arguments and are recorded as workflow_id=external.

Current MCP Resources

• lab://lab-overview
• lab://safety-rules
• lab://mqtt-conventions
• lab://node-hardware-map
• lab://sop/read-capabilities
• lab://sop/control-actuator

Current Controller Examples

The example root is:

labbridge/examples/

Current examples:

• camera0_controller: registers a local camera0 service, exposes camera0.capture_snapshot, saves timestamped images under its local images/ directory, and includes an end-to-end smoke test through the gateway.
• vision_agent0_controller: registers a virtual vision device with local camera capture, exposes visual boolean inspection and image description actions, and trims MCP input/result surfaces with controller metadata.
• amr0_controller: registers a mock AMR, reports moving/arrived/idle/blocked-style state through state_values, and exposes a cancelable move_to operation.

The controller integration contract is documented in:

labbridge/docs/controller-protocol.md

9. Planned MCP Tool Extensions

Future extensions should prefer device capabilities over new top-level MCP tools when the action belongs to a specific device. For example, AMR movement should normally be represented as:

call_capability("amr0", "move_to", {"target": "bench_01_a", "speed": 0.4})

The same pattern applies to mixer speed/direction, robot-arm transfer tasks, camera capture, and simulated smart devices. This keeps the MCP tool surface small while allowing many device types to be registered through gateway metadata.

Possible future top-level tools or semantic services:

• search_knowledge_base(query): retrieve prior processes, SOPs, and experiment notes.
• get_live_status(workspace_id): read current device and bench state from the gateway state cache.
• visual_confirm_status(camera_id, prompt): higher-level convenience service that may call camera and vision controller capabilities internally.
• load_robot_policy(robot_id, bench_id, policy_id): semantic wrapper for loading a bench-specific robot policy or calibration profile when this is not just a normal robot capability.
• propose_experiment(workflow_json): submit an experiment plan for HITL approval.
• emergency_stop(bench_id): stop execution and trigger the appropriate software and hardware safety path.

10. Abbreviations

Abbreviation	Meaning
Agent / LLM	Language-model planning agent
AMR	Autonomous mobile robot
ESPHome	Firmware framework for ESP32 nodes
GPIO	General-purpose input/output
HITL	Human in the loop
I/O	Input and output
I2C	Inter-Integrated Circuit bus
MCP	Model Context Protocol
MQTT	Message Queuing Telemetry Transport
NUC	Small edge computer
PWM	Pulse-width modulation
RAG	Retrieval-augmented generation
SOP	Standard operating procedure