Deployment and Services

This page covers the operational entry points for the RIGOR development stack. Runtime settings live in the centralized ignored configuration described in Centralized Configuration; credentials and local databases must not be committed.

Start the Stack

From the repository root:

cd /home/flare/sdlx
./run.sh start
./run.sh status

Root startup first launches the independent RIGOR Supervisor and required Docs Web service, then starts LabBridge, Execution, Analysis Worker, and finally Control. Shutdown uses the reverse service dependency direction so LabBridge remains available until the other domain services have stopped. The standalone control panel is:

local:  http://127.0.0.1:8079
remote: http://<server-ip-or-hostname>:8079

It can also be managed separately:

supervisor/run.sh start
supervisor/run.sh status
supervisor/run.sh logs

Bootstrap and Controller Recovery

Use ./run.sh start after a machine restart, during deployment, or after changing the centralized configuration. It validates and renders the runtime projections, starts the independent Supervisor and Docs service, and then starts the domain stack. The repository does not currently install an operating-system startup service for Supervisor, so the Controller cannot be reached until this bootstrap command, or supervisor/run.sh start, has run.

Once Supervisor is online, operators do not need to start domain modules individually. Controller's Start all, Stop all, and Restart all actions cover the fixed domain topology below. Start follows dependency order; stop uses the reverse order. The LabFlow controller adapter runs inside the LabFlow API process and does not require a separate operator action.

Controller lifecycle coverage	Services
Control	Composer, ASCEND
Execution	LabFlow and its controller adapter, PACE
LabBridge	LabBridge and its bundled gateway workers
Analysis Worker	PRISM API, callback dispatcher, sandbox Agent, and worker

Controller deliberately remains online when Stop all completes, so the same page can recover the stack. It cannot start itself. It also does not manage Docs, managed S3 or the optional local MinIO fixture, the optional EMQX broker, or optional physical-device controllers such as camera0, vision0, and the standalone amr0. Manage those infrastructure or device processes through their own deployment entry points.

The resulting operating rule is:

machine or deployment bootstrap
  -> ./run.sh start
  -> Supervisor, Docs, and all domain units become available

normal research use
  -> no manual service lifecycle actions

maintenance or failed automatic recovery
  -> Controller manages the domain stack
  -> no individual module startup is required

The Supervisor bind address comes from infrastructure.supervisor in config/sdlx.yaml and accepts only fixed service, unit, and whole-stack actions. Port 8079 therefore must be restricted to a trusted management network or protected with the deployment authentication and TLS boundary. Set its configured host to 127.0.0.1 for local-only access. Set infrastructure.supervisor.public_url to the browser-facing Controller URL; ASCEND uses that value for both its header Settings shortcut and the Engineering Console recovery link instead of exposing the Supervisor's loopback bind URL.

Controller is an exception-first recovery surface rather than the normal research workflow. Its overview opens modules through ASCEND, and reveals per-service or whole-stack lifecycle controls only from recovery panels. The configuration editor is likewise collapsed until requested. The interface supports English, Simplified Chinese, and Spanish and shares the selected language with ASCEND through the sdlx_locale preference.

The Supervisor also applies the repository log-rotation policy once per minute. Each *.log under the ASCEND, Composer, documentation, LabBridge, LabFlow, PACE, PRISM, and Supervisor log directories is archived after it exceeds 100 MiB. Archives use UTC date-time suffixes such as admin.log-20260801T061500Z.gz; run scripts open active logs in append mode so standard copytruncate rotation can retain the active filename without restarting services. The latest 14 archives per log are retained. Runtime policy and state are written to supervisor/run/logrotate.conf and logrotate.state and remain local deployment data.

Supervisor also owns automatic Agent sandbox maintenance. Configure the shared policy once under infrastructure.agent_sandbox in config/sdlx.yaml:

The rationale for the shared tool baseline, package groups, and deliberate security exclusions is documented in Agent Sandbox Image.

agent_sandbox:
  enabled: true
  cleanup_interval_seconds: 300
  stale_container_seconds: 86400
  min_free_bytes: 10737418240
  min_free_percent: 10

ASCEND, Composer, and PRISM containers carry bounded RIGOR owner, session/run, and policy labels. The maintenance worker removes only labeled containers that have remained in a terminal Podman state beyond the retention interval; it never selects a running container and never deletes the mounted workspace, domain record, audit, or evidence. When either free-space threshold is crossed, new sandbox creation fails closed with an explicit storage-reserve error while existing work continues. Controller shows the condition as an operational exception in English, Simplified Chinese, or Spanish. /health and /api/runtime expose the redaction-safe maintenance status; material cleanup and failures are appended to the ignored supervisor/run/sandbox-maintenance.jsonl audit.

The shared Agent shell emits compatibility telemetry containing only the bounded first executable name, outcome, and exit code. Command arguments, working directories, environment values, and secret material are not included. Use aggregated 127/ENOENT results to decide whether a later image revision needs another tool; do not treat this telemetry as experiment evidence.

ASCEND Engineering Console windows use fixed paths under /api/console-proxy/{service}/. ASCEND forwards those requests to the server-side integration addresses such as 127.0.0.1:8088; a remote browser must not be given those loopback URLs directly because browser 127.0.0.1 means the researcher's own computer. The proxy allowlist is limited to Composer, PACE, LabFlow, LabBridge, and PRISM and does not accept a user-supplied host.

The default stack is four deployable responsibility boundaries:

Unit	Co-deployed domains
Control	ASCEND and Composer planning
Execution	PACE, LabFlow, and the LabFlow controller adapter
LabBridge	LabBridge device/controller boundary
Analysis Worker	PRISM API, callback dispatcher, sandbox Agent and worker

Composer and ASCEND still require their configured model credentials. The unit wrappers retain separate processes, domain APIs, data stores, and safety authority. Docs starts by default and remains infrastructure outside the four domain units; docs-site/run.sh remains its explicit engineering adapter. Set infrastructure.docs.public_url to its browser-facing URL. ASCEND exposes that address as the Docs item in its primary navigation. Object storage is deployment infrastructure rather than a fifth RIGOR domain unit; configure managed S3 or start the optional MinIO fixture described in Object Storage. The optional local EMQX broker is likewise rendered from infrastructure.emqx and managed with scripts/emqx.sh.

Service Directory

Service	Local port	Responsibility
LabBridge Admin	8088	Devices, Human Tasks, spatial registry, operations, artifacts
LabBridge MCP HTTP	8091	Agent and scheduler gateway boundary
LabBridge MCP WebSocket	8090	WebSocket and Xiaozhi MCP clients
LabFlow	8102	Workflow editing and deterministic execution
PACE	8103	Loop packages, experiment runs, Result Bundles
ASCEND	8110	Research coordination, chat, iterations, and decisions
RIGOR Docs	8112	Searchable architecture and operations documentation
Composer	8120	Scientific composition and reviewed package generation
PRISM	8114	Isolated analysis runs, results, callbacks and worker status

Configuration

Use RIGOR Control's Centralized Configuration page or the scripts/sdlx-config.py CLI. Root startup validates and renders all service projections. Module CLIs accept an explicit --config only for isolated tests; normal deployment does not select configuration through process environment.

Object Storage

PACE and PRISM retain local file:// archives only when their object-store backend is disabled. Production enables their scoped S3/MinIO configuration and LabBridge's existing artifact store configuration. Service users must be different, and PACE/PRISM runtime identities receive no delete permission.

See Object Storage for bucket initialization, centralized settings, prefixes, retention, and recovery checks.

ASCEND

ASCEND and Composer retain separate module sections and separate keys inside the central bundle. Their generated runtime inputs remain independent.

Configure modules.ascend.ascend.llm.provider, model, base_url, and the api_key secret reference. ASCEND renders its private nanobot config into ascend/run/; it does not read Composer's key, sessions, or workspace.

Documentation Site

The documentation site reads Markdown directly from docs/:

docs-site/run.sh build
docs-site/run.sh start
docs-site/run.sh status

Configure infrastructure.docs.host and port to change the binding.

The documentation runtime exposes GET /api/status. It queries LabBridge, LabFlow, PACE, ASCEND, Composer, and PRISM in parallel and returns a read-only operational summary for the live dashboard on the home page. The browser refreshes the snapshot every 15 seconds; device counts come directly from LabBridge rather than from generated documentation. The documentation service itself is already proven reachable when this page and status endpoint load.

The home-page laboratory view reads GET /api/showcase, a short-lived proxy of ASCEND's sdlx.lab-showcase/v1 projection. ASCEND selects the newest eligible committed execution or rehearsal and supplies an explicit evidence-authority label; when neither exists it returns the current LabBridge layout. Docs does not infer truth, issue actions, or reconstruct missing historical paths. The narrow /api/showcase/assets/* route carries only allowlisted DeviceModel files through ASCEND.

Docs service probes are derived from the six module ports. Configure its status and showcase timeouts plus public ASCEND URL under infrastructure.docs; the generated Docs JSON contains the resulting complete service directory.

Reverse Proxy

The minimum public surface is:

ascend.sdl.digauto.org -> 8110
docs.sdl.digauto.org   -> 8112

ASCEND serves the normal product, the allowlisted Engineering Console proxy, and the governed module MCP gateway from the same origin:

https://ascend.sdl.digauto.org/mcp/labbridge
wss://ascend.sdl.digauto.org/xiaozhi/mcp/labbridge?token=<scoped-token>

The HTTP route maps to LabBridge Streamable HTTP at 127.0.0.1:8091/mcp. The Xiaozhi route maps to the LabBridge compatibility WebSocket at 127.0.0.1:8090/mcp_endpoint/mcp/. ASCEND consumes the client credential and uses a separately configured upstream service identity; it never forwards the public bearer or query token to a module. Configure either trusted-header auth or a per-service access_token secret reference under modules.ascend.ascend.mcp_gateway.services. Store the referenced value only in ignored config/secrets.yaml; standard clients send it as a Bearer token. Without one of those identity boundaries, MCP requests fail closed.

The outer proxy for ascend.sdl.digauto.org must preserve streaming responses and WebSocket upgrades. Typical WebSocket settings are:

proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 3600s;
proxy_buffering off;

The two hostnames above are the complete normal public surface. Composer, PACE, LabFlow, LabBridge, PRISM, Controller, and module MCP ports remain private; their consoles are available through ASCEND where supported.

LabBridge artifact.public_base_url now uses https://ascend.sdl.digauto.org/artifacts/labbridge. Live LabBridge API, trace, Human Task, and MCP projections derive the current URL from immutable artifact_id values. Do not provision a separate LabBridge or MCP hostname for current traffic. Static historical exports with retired absolute hosts are historical data, not a reason to expose current module services directly.

Terminate TLS and enforce authentication at the proxy. Do not directly expose module Admin ports, MCP ports 8090/8091, MQTT, EMQX management, MinIO internals, SQLite files, or controller polling endpoints.

Recovery Verification

The contract suite checks state-machine behavior in-process. The live recovery suite starts isolated real services, writes persistent records, force-stops every process, restarts them, and verifies records through HTTP:

scripts/test-golden-loop-recovery.sh
scripts/test-live-process-recovery.sh

The PostgreSQL qualification path starts an isolated PostgreSQL instance, runs the supported ASCEND lineage migrations and immutability checks, takes a native pg_dump, restores it into a newly created database, and compares the complete lineage digest and migration version:

scripts/test-postgres-backup-restore.sh /path/to/postgres-restore-evidence.json

The output JSON is an operational qualification artifact and should be stored outside the repository. This isolated test proves the implemented migration and restore mechanics; the production gate still fails closed until the selected deployment actually uses and restores its provisioned PostgreSQL service.

Set SDLX_RECOVERY_PORT_BASE when the default isolated port range beginning at 18288 is in use.