The brief

A client needed a telemedicine platform where patient privacy was non-negotiable. Video processing had to stay on the device — no frames could leave the browser except through the encrypted WebRTC peer connection. That ruled out cloud-based background blur and noise suppression.

Background segmentation

We used TensorFlow.js with the BodyPix model to segment the speaker from the background in real time. Each video frame is drawn to an offscreen canvas, run through the model, and composited with a replacement background before being fed back into the MediaStream track. On a mid-range laptop the pipeline sustains 25 frames per second at 720p.

Audio processing

For noise suppression we leveraged the Web Audio API's AudioWorklet with a small RNNoise WASM module. The worklet runs in a dedicated thread so it doesn't block the main thread or the video pipeline. We wrapped it in a custom MediaStream track processor that plugs transparently into the WebRTC connection.

Performance tuning

The biggest challenge was keeping GPU memory stable over long calls. We implemented a frame-skip strategy that reduces model inference frequency when the speaker is stationary, and an adaptive resolution scaler that drops to 480p if the device thermal state crosses a threshold. Battery-powered devices saw a forty percent improvement in call duration after these optimisations.

Lessons

On-device ML in the browser is viable for real-time video, but you have to treat the GPU as a shared resource. Profile early, budget your frame time, and always have a graceful fallback for low-end hardware.