Mac computer use with local perception (X)

Play the video in the post below

Milind S built a Mac computer-use demo that never sends a screenshot to a language model and does not read DOM elements. A local CoreML model segments every button and UI element on screen, and on-device OCR reads their labels. That text is the only thing Jev sees.

Jev returns a probability across those elements and names the best one to click. The system clicks it, re-runs detection on the new screen and asks Jev again, looping until the goal is done. No pixels leave the machine, and no general-purpose language model is in the loop.

Milind reports about 90 milliseconds per decision, faster than any LLM-based computer use Milind had tried.

Open on X