Smart Glasses Field Assistant

A SwiftUI iOS application captures camera and microphone from glasses-mounted hardware and streams it to a Python backend over WebSockets, with a media server handling the video transport. A companion desktop client handles system audio, and a desk-mounted camera provides a second fixed viewpoint.

Every one of those inputs feeds a single conversational session with shared memory. The assistant does not care which camera saw the thing you are asking about.

A wearable assistant that takes five seconds to answer is a wearable assistant nobody wears. The budget runs capture → relay → model → speech, and every hop had to be defended.

Frames are encoded and relayed rather than round-tripped through disk. The model runs as a live streaming session instead of discrete request/response calls, so audio and video flow continuously and the reply begins before the question is fully processed. The system prompt caps answers at one or two sentences unless more is explicitly asked for, because a spoken paragraph is a latency cost the wearer pays in full.

Always-on cameras and microphones are a trust problem before they are a technical one. Activation runs through an on-device wake-word model, so the stream opens on intent rather than running continuously into the cloud.

Face recognition is enrolled from a local directory of known faces and matched on the machine. The assistant can tell the wearer who walked in without anyone’s face being uploaded to a third-party identity service.

The backend serves a small web console for monitoring live feeds, checking session state, and configuring which inputs are active. Prototypes get abandoned when nobody can see what they are doing; this one is observable from a browser tab.