milind-soni/tiptour-macos on GitHub
- Stars
- 662
- Forks
- 103
- Language
- Swift
- Last push
- 19 Sep 2026
Post on X
- Views
- 61,095
- Likes
- 509
- Reposts
- 29
Read from GitHub and X on . Counts change daily.
Reported by the author
- Action limit
- 12 actions
The loop stops when JEV chooses none or reports completion, on an execution pause/failure, or at its 12-action limit.
github.com
TipTour is a macOS menu bar app with two ways to drive your desktop. Press Ctrl+Option and talk, and Gemini handles clicking, typing, shortcuts, opening apps or URLs, scrolling, and highlighted-text edits, one action per turn. Press Ctrl+K instead and type a task, and Jev takes over: a local OmniParser model running on CoreML detects the controls on screen, and Jev chooses which one to click, double-click, or right-click.
Jev’s role is narrower than Gemini’s. It works from on-screen labels and locations, not images, and it cannot generate replacement text, so the README says to use Gemini for writing or keyboard actions. It acts on its top-ranked target each turn with no confidence cutoff. The loop stops when Jev chooses none or reports completion, on an execution pause or failure, or at a 12-action limit. TipTour executes and validates each action.
You bring your own API keys, stored in macOS Keychain, with no shared keys or hosted proxies. In the earlier demo post, Soni reports about 90ms per Jev decision. The open-source post says voice mode is next if people want it.
How it uses Jev
private let metrics = JevCallMetrics(milliseconds: 1, inputTokens: 100, model: "jev-latest")
private func candidate(_ id: String = "save", label: String = "Save") -> JevCandidate {
JevCandidate(id: id, label: label, source: "ocr", confidence: 1, centre: .zero)
}
private func answers(choice: String = "save", absent: Double = 0.1,
probability: Double = 0.9, kind: String = "click") throws -> [String: JevAnswer] {
let data = try JSONSerialization.data(withJSONObject: [
"pick": ["choice": choice, "probabilities": [choice: probability]],
"done": ["noul": 0.1], "absent": ["noul": absent], "kind": ["choice": kind]
])
return try JSONDecoder().decode([String: JevAnswer].self, from: data)
}View in TipTourTests/JevTests.swift
