Operations

Operations

Observability and operational interfaces for the self-hosted engine.


This page documents Community Edition operational behavior. CE supports Prometheus metrics as an optional emitter and keeps live clients, tunnels, and streams in process memory.

The Engine API exposes live inventory and signaling endpoints under /api. GET /api/clients and GET /api/tunnels return point-in-time state. /api/sse and /api/websocket stream an initial snapshot followed by lifecycle events.

Runtime Checks

Use container or service-manager status first:

docker compose ps
docker compose logs --tail=100 engine

The Engine OpenAPI document is a simple TLS check for the base hostname:

curl -fsS https://edge.example.com/api/openapi.json | jq -r '.info.title'

Inventory uses the base engine host and a valid engine JWT:

curl -fsS \
  -H "Authorization: Bearer $RSTREAM_AGENT_TOKEN" \
  https://edge.example.com/api/tunnels | jq

If you validate against a local self-signed certificate, use --cacert and --resolve rather than disabling TLS verification:

curl -fsS \
  --resolve edge.localhost.rstream.test:18443:127.0.0.1 \
  --cacert tls/ca.crt \
  -H "Authorization: Bearer $RSTREAM_AGENT_TOKEN" \
  https://edge.localhost.rstream.test:18443/api/tunnels | jq

Prometheus Metrics

When Prometheus metrics are enabled, the engine exposes a metrics endpoint. The default path is /metrics, and the path is normalized if it is configured without a leading slash.

Prometheus metrics require a listen address to be configured. When the emitter is disabled, no metrics server is started.

When the metrics listener binds to a non-loopback address, the engine requires metrics.prometheus.bearer_token. Scrape it with:

curl -fsS \
  -H "Authorization: Bearer $RSTREAM_METRICS_TOKEN" \
  http://127.0.0.1:9090/metrics

Keep the metrics listener private when possible. If it must be reachable from another host or container, use a long random bearer token and restrict network access at the infrastructure layer.

Certificate Rotation

CE uses the static certificate provider. The provider reloads the certificate pair automatically on a later TLS handshake when cert_file or key_file changes.

Use atomic replacement from a certificate sidecar or deploy hook:

install -m 0644 fullchain.pem /etc/rstream/tls/cert.pem.new
install -m 0600 privkey.pem /etc/rstream/tls/key.pem.new
chown "$RSTREAM_RUNTIME_UID:$RSTREAM_RUNTIME_GID" /etc/rstream/tls/cert.pem.new /etc/rstream/tls/key.pem.new
mv /etc/rstream/tls/cert.pem.new /etc/rstream/tls/cert.pem
mv /etc/rstream/tls/key.pem.new /etc/rstream/tls/key.pem

Then verify the served certificate without restarting the engine:

echo | openssl s_client \
  -connect edge.example.com:443 \
  -servername edge.example.com \
  -verify_return_error 2>/dev/null \
  | openssl x509 -noout -subject -fingerprint -sha256

If the new files are invalid, the engine keeps serving the previously loaded certificate and tries again on later handshakes after the files change.

Upgrades

Pin image tags for production deployments. Upgrade with the same Compose lifecycle used for other infrastructure services:

docker compose pull engine
docker compose up -d engine
docker compose logs --tail=100 engine

Live tunnels are in memory and are tied to agent control channels. Restarting the engine disconnects agents; agents should run with retry enabled so they reconnect and recreate their tunnels.

CE Integration Boundary

CE exposes Prometheus metrics and live Engine API state. It does not provide a persisted event store in the public runtime, so operational dashboards should scrape Prometheus and query or watch the live Engine API.

CLI Event Stream

The CLI can subscribe to the engine event stream using rstream events and can optionally forward each event as JSON. Add --webhook when the local receiver should see the signed webhook request shape.

See Signaling for transport behavior and examples.

Troubleshooting

SymptomCheck
Engine exits on startup with a certificate errorConfirm certs.static.cert_file and certs.static.key_file exist, are readable by the container user, and contain a matching certificate/key pair.
TLS works for edge.example.com but agents cannot connectAgents must use the base engine host, for example edge.example.com:443, and DNS must resolve that name to the engine.
Published tunnel hostname does not resolveAdd or fix *.t.<engine.host> DNS. A wildcard under engine.host does not cover the .t. tunnel namespace.
Prometheus exporter refuses to startA non-loopback metrics bind address requires metrics.prometheus.bearer_token.
Local self-signed validation failsTrust the local CA with --cacert, SSL_CERT_FILE, or the OS trust store. Do not use the local CA pattern for production.
A tunnel option is rejected as unavailableConfirm that the requested option is part of the CE scope documented on Self-Hosted. Hosted and private-runtime controls are rejected by the public CE engine.