Run Edge Vision Inference on Your Own Machines with rstream

A vision workload often spans two locations: cameras run near the scene, while inference runs on a workstation, a lab server, or another machine with suitable compute. Connecting those locations usually requires a hosted inference API or a private network plus additional service discovery.

Here, devices and workers each run a Python process and use rstream for transport and discovery. Workers join a pool by creating a labeled private tunnel. Devices watch the tunnel registry, dial a worker, and select another when the current worker disappears. Neither role needs a public listener.

The sample includes the mechanisms needed to evaluate this architecture: frame identifiers and monotonic timestamps, an adaptive display buffer, object tracking across frames and worker changes, and an explicit encoding policy between devices and workers. The WebRTC and GStreamer guides cover direct video delivery instead.

Clone the reference implementation:

git clone https://github.com/rstreamlabs/rstream-examples.git
cd rstream-examples/python-vision-inference

Architecture

Two roles, connected only by rstream. The worker loads a YOLO model and creates a private tunnel labeled role=inference. Each session it accepts opens with a hello advertising the model, its input size, and the codecs it decodes, then answers framed images with JSON detections. The device captures video, negotiates its encoding policy against that hello, keeps up to two frames in flight, and serves the annotated result to browsers through a published, optionally token-protected tunnel.

Browser
        |
        | HTTPS, published tunnel (optional --token-auth)
        v
rstream engine
        |
        | routes the viewer to the device
        v
Device (camera + annotated viewer)
        |
        | dials a worker by label, role=inference, over rstream
        v
rstream engine
        |
        | tunnel inventory is the registry; watch tracks the pool
        v
Inference worker (GPU or CPU, advertised by labels)

A frame's path runs the engine twice: the browser reaches the device over the published viewer tunnel, and the device reaches a worker over a private dial, detections returning the same way. Neither the device nor the workers expose a public IP or open a port; each holds one outbound tunnel to the engine, which mediates every leg. Across that one connection the device publishes its viewer, dials workers, and watches the pool; each worker registers simply by creating its labeled tunnel.

The split between published and private follows the client protocol. The viewer endpoint serves standard browsers, so it is published and protected at the edge. The inference channel is machine-to-machine between two SDK processes, so it stays unpublished and is reached by an rstream dial, as laid out in Private Tunnels.

What makes the pool elastic is that there is no registry service to deploy. The tunnel inventory itself is the registry: workers exist in it exactly as long as they run, labels make them discoverable, and the engine's real-time watch pushes every arrival and departure to the devices.

Prepare the machines

Every machine involved needs an rstream CLI configuration resolving to the same project, following Installation and CLI Workflow. For a first run, rstream login and rstream project use <project-endpoint> on each machine is enough; the security section below tightens this for real deployments.

Both roles need Python 3.10+, but they keep separate environments. The worker installs the YOLO inference stack from worker/requirements.txt; the device installs the capture, viewer, registry, and tracking stack from device/requirements.txt.

make build

The model is yolov8n, small enough to run usefully on CPU, so the walkthrough works without a GPU and gets faster when one is present.

Run a worker

The worker is the simpler half, deliberately. It loads the model, registers itself by creating a labeled private tunnel, and serves sessions as a pure frame-to-detections function.

tunnel = await control.create_tunnel(
    name=args.name,
    publish=False,
    labels={"role": "inference", "model": args.model,
            "device": device, "accelerator": accelerator},
)
async for stream in tunnel:
    task = asyncio.create_task(serve_session(stream, model, ...))

The labels are the worker's registration, and they do double duty as signaling. Beyond role, the worker advertises the model, the device it runs on (cpu, mps, cuda:0), and the accelerator name, which it resolves itself: ultralytics defaults to the CPU even when a GPU is present, so the worker selects CUDA, then Apple MPS, then CPU, and publishes the result. A viewer reads "Apple M1 Max (mps)" or "NVIDIA RTX 4090 (cuda:0)" straight from the registry, with no inference-specific endpoint to query, which is the point: the tunnel inventory is already a typed, real-time directory of who is running what, on what hardware. Keeping workers stateless is the other design decision the rest of the guide collects: any pool member can serve any device, and everything that has memory, the tracker, the latency estimator, the display buffer, lives at the edge.

make run-worker

Start a second worker on another machine, or the same one, and the pool has two members.

Run the device

The device owns the system's state, composed as concurrent loops on one event loop: capture, registry, inference, and rendering. The registry loop is where rstream acts as the signaling layer, seeding the pool from the inventory and keeping it current from the real-time watch.

WORKER_FILTERS = rstream.TunnelFilters(labels={"role": "inference"})
 
for tunnel in await client.list_tunnels(filters=WORKER_FILTERS):
    if tunnel.properties.name and tunnel.status == "online":
        state.workers[tunnel.properties.name] = tunnel.status
async with client.watch(tunnels=WORKER_FILTERS) as events:
    async for event in events:
        ...  # add on tunnel.created, drop on tunnel.deleted

The inference loop picks a worker, dials it by name, and keeps at most two frames in flight so transfer overlaps inference. Since the device always sends its most recent frame rather than a backlog, backpressure is automatic: a CPU worker lowers the detection rate, a GPU worker raises it, and no queue builds anywhere. Failure handling is the same loop. A dial error, a response timeout, or a watch event removing the current worker all put the worker on a short cooldown and pick another.

Two more decisions complete the loop, both solved without deploying anything. Load distribution uses rendezvous hashing to give each device its own preference order over the pool, then the power of two choices on top: the device probes its two best candidates, reads the session count each worker reports in its hello, and keeps the lighter one. Dials are cheap, so balancing costs exactly one extra hello. And when the pool empties entirely, the device suspends detection and resumes it by itself when capacity returns, while an explicit user action always overrides the automatism.

Automatic selection is the default, not a cage. Clicking a worker in the viewer pins the session to it, which is how you steer a specific camera onto the GPU box and watch the inference time drop in real time. The pin is advisory: if the chosen worker disappears the device falls back to automatic selection and snaps back the moment it returns, so a deliberate routing choice never becomes a single point of failure. The control reuses the same last-intent-wins reconciliation as the detection toggle, so clicking through workers never strands the UI in a transitional state.

Start the device where the camera is. The default source downloads a short highway-traffic clip on first run, dense enough that the model has fifteen or more vehicles per frame to find, so the walkthrough is reproducible without hardware.

make run-device ARGS="--source 0"  # first local camera
make run-device                    # downloaded highway clip

The device prints its viewer address. The page shows the annotated stream and the full telemetry, source and detection FPS, inference time, network time, display buffer, and uplink bandwidth, with a button that disables and re-enables detection while the video keeps playing. The viewer is published, so anyone with the URL can also press that button; --token-auth puts the whole page, controls included, behind edge token authentication.

Timestamps without trusted clocks

Every frame leaves the device with a frame_id and a timestamp from the device's own monotonic clock, and every result echoes the frame_id. What never happens is a comparison between clocks on different machines: a field device and a lab workstation share no time reference worth trusting. The worker therefore reports its inference time as a duration, the device measures the full round trip on its own clock, and the difference is the network share. This is the same discipline RTCP applies to round-trip measurement, and it is why the Inference and Network values on the page are correct rather than approximately plausible.

A jitter buffer for the display

Overlaying the latest detections on the latest frame is what naive demos do, and it shows: with 60 ms of round trip, boxes trail behind every moving object. While detection runs, the sample holds the display behind a small adaptive delay instead, pairing each frame with the detections computed for that exact frame by frame_id; with detection off there is nothing to align, so frames show immediately. The delay target comes from the smoothed round trip and its variance, the same estimator TCP uses for retransmission timeouts (RFC 6298), clamped to a sane range, and the current value is the Buffer figure on the page.

If this sounds familiar from the GStreamer guide, it should: it is an application-layer rtpjitterbuffer, trading a fixed, visible slice of latency for correctness on arrival jitter. The viewer applies the same idea to the last leg: frames reach the browser as timestamped JPEGs, and the page plays them through its own small adaptive buffer, so jitter on the path to the operator's screen never turns into stutter. Between detection results, a ByteTrack tracker running on the device carries box identities forward, so labels like car #12 persist at source frame rate, and when detections stop being fresh, after a worker loss or a stopped detection, the overlay disappears while the video keeps playing.

An encoding policy for inference traffic

What the device sends to workers is a policy, not an accident. The single biggest saving costs nothing: frames are resized to the model's input size, negotiated in the session hello, before encoding, because sending pixels the worker would immediately throw away is pure waste. The default codec is JPEG at quality 80, which is where the detection literature and practice agree accuracy is unaffected, and the Uplink figure on the page makes the result concrete, a few hundred kbps per device at street-scene rates.

Both knobs are explicit. --quality trades bandwidth against accuracy with a cliff below roughly 50, and --codec png switches to lossless for the domains that genuinely need it, medical or industrial inspection traffic where fidelity outranks bandwidth, at ten to twenty times the bytes on natural scenes. When bandwidth is the binding constraint, the next step beyond this sample is delta-encoded video toward the worker, at the price of decode state that breaks the stateless-worker property; the trade is worth knowing and deliberately not taken here.

In production, the worker side of this protocol often already exists as a model server. Because rstream tunnels carry TCP, a Triton or KServe endpoint can stand behind the same labeled tunnel and replace worker/worker.py without changing the device's discovery or failover logic.

Kill a worker and watch

This is the demonstration the architecture exists for. With two workers in the pool and the viewer open, stop the worker currently serving the device.

The video keeps playing because display is decoupled from inference. The pool loses the stopped worker when the engine emits tunnel.deleted, the active session reconnects, and detections resume from another worker. The tracker remains on the device, so object identities can survive the worker change. Starting the stopped worker again adds it back to the pool without restarting the device.

Nothing in the sample special-cases any of this. Discovery, failover, and elasticity fall out of two primitives, labeled tunnels and the watch stream, and display continuity falls out of keeping every stateful component at the edge.

Security postures

Two postures fit this architecture, and the code is identical in both; only the credentials on each machine change.

For a trusted setup, a lab, a homelab, or a single-team deployment, every machine uses a project-scoped credential from the remote device path in CLI Workflow. All members of the project can create and dial tunnels within it, which matches how the machines are actually trusted.

For a hardened deployment, mint per-role tokens with the CLI and put boundaries on them. A device has no reason to dial other devices, and a worker has no reason to dial anything at all.

# Worker machines: create inference tunnels, nothing else to dial
rstream token create -p tunnels.tunnels.create-delete --resources-json '<boundary>'
 
# Device machines: dial worker tunnels, create the viewer tunnel
rstream token create -p tunnels.streams.create-delete -p tunnels.tunnels.create-delete --resources-json '<boundary>'

The resources.tunnels boundary JSON scopes which tunnels each credential can create or connect to, by name pattern or labels, and is documented with complete examples in Fine-grained tokens. Start with the trusted posture, and tighten when machines stop being interchangeable.

Run it in production

Both roles are long-running services and deploy the same way as the rest of the guide family. Put each command under systemd with Restart=always, give each machine its project-scoped token context, and name things for the fleet: worker-gpu-lab-1, cam-gate-3. Two details keep a URL working once it leaves your hands. Both roles recreate their tunnels with a capped backoff when the engine connection drops, and the device derives a stable viewer domain once at startup, the same way the Go and C++ SDKs do, reusing it on every reconnect so a network blip costs seconds and the open tab survives on the same URL rather than needing an operator. That generated address is per run; the --host flag pins a fixed name that also survives full restarts, documented in Stable Domains. The pool view in any device's UI, rstream tunnel ls filtered by label, and rstream events all read the same registry, so operations see exactly what the devices see.

The latency breakdown in each viewer separates inference time from network time. When inference dominates the round trip, another worker can be added to the pool without changing the devices.

Where to go next

The SDK fundamentals used here, in-process ASGI serving, private dials, and edge token policy, are introduced on a single app in Serve a FastAPI App through rstream Tunnels. The inventory and watch surface is documented in Python SDK. For delivering the camera itself rather than detections, the media path guides start at Stream Video with GStreamer and rstream, and browser-scale delivery is the WebRTC series starting at Build Device-to-Browser Video Streaming with WebRTC and rstream.