# SDK reference (/extensions/sdk) `lumi-extension-api` is the crate your extension depends on. Import it under a short name and everything below is one path away: ```rust use lumi_extension_api as lumi; ``` ## What each call costs [#what-each-call-costs] Every function below exists whether or not your manifest declares what it needs. An undeclared one is refused **when it is called**, with an error that names the capability — so the person who sees it is you, with the manifest open. | Costs | Functions | | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Nothing at all | `about`, `license` | | One call from the run's budget | `alert`, `settings`, `setting`, `set_setting`, `profiles`, `transforms::all`, `transforms::apply`, everything in `lumi::translation`, everything in `lumi::dictionary`, `open_window`, `open_window_on`, `open_window_at`, `close_window`, `window_state`, `preload_enabled`, `set_preload_enabled`, `set_tab_label`, `hide_window`, `show_window`, `show_window_on`, `show_window_at`, `post`, `set_material`, `set_position`, `set_anchor`, `set_theme`, `set_pinned`, `raise_window`, `set_size`, `set_titlebar_height`, `hyper_key_enabled`, everything in `lumi::storage`, everything in `lumi::folders` (for the folders your manifest declares), everything in `lumi::files`, everything in `lumi::owner`, `config::extension_shortcuts` and `config::set_extension_shortcut`, `ocr::languages`, and `ocr::read` of a picture in your storage. Some answer only a press — a command, a menu bar row or one of your windows: `open_settings`, `open_permissions`, `switch_off`, `set_setting`, `reveal`, `set_pinned`, `files::open` and `files::save`, `owner::authenticate`, `translation::prepare`, `dictionary::show`, and `folders::choose`, `write` and `reveal` | | `accessibility` | `selection` (the ⌘C fallback needs `clipboard` too), `selection_editable` | | `accessibility` and `input` | `replace_selection` and `replace_selection_and_select` — and only from a command or one of your windows; a replacement too long to type is pasted only with `clipboard` too | | `clipboard` | `clipboard_text`, `set_clipboard_text`, `clipboard_image`, `read_clipboard`, `write_clipboard`, and `drag` and `drag_named` — those two only while the mouse button is down in one of your windows | | `clipboard`, `input` and `accessibility` | `paste` and `copy_selection` — and only from a command or one of your windows | | `clipboard-history` | hearing `on_event("clipboard", …)` and `on_event("clipboard-ocr", …)`, with `events = [...]` in the manifest | | `selection-watch` | `selection_watching`, `set_selection_watching`, `selection_watchers`, and hearing `on_event("selection", …)` and `on_event("selection-watching", …)`, with `events = [...]` in the manifest | | `applications` | `open_url` from `on_event`, a lifecycle hook or a flow node — nobody pressed anything (a web address needs `network` too). From a press it costs no capability | | `network` | `fetch` | | `config` | everything in `lumi::config` | | `screen` | everything in `lumi::screen`, `ocr::read` of the screen, `lumi::permissions` for Screen Recording (`ask`, `request` and `open_system_settings` only from a command or one of your windows), and hearing `on_event("screen-capture", …)` with `events = [...]` in the manifest | | `snippets` | everything in `lumi::snippets` (an expansion that reads the clipboard needs `clipboard` too) | | `menu` | everything in `lumi::menu` — your rows in Lumi's menu bar menu | | `menu-bar-item` | everything in `lumi::menu_bar_item` — an item of your own in the menu bar, beside Lumi's — and hearing `on_event("menu-bar-item-shown", …)` with `events = [...]` in the manifest (Lumi 1.61.0) | | `ai` | everything in `lumi::ai` — `complete`, `start` and `prepare` only from a command, a menu bar row or one of your windows (Lumi 1.52.0; `prepare` 1.64.0) | | `speech` | everything in `lumi::transcribe` — `prepare`, `prepare_for`, `complete`, `start`, `start_after` and `switch` only from a command, a menu bar row or one of your windows; hearing the microphone live costs `microphone` too, what the Mac plays `system-audio`, and a call both (Lumi 1.64.0) | | `audio-levels` | a `transcribe::Request` with `levels`, or a `recordings::Request` with `levels` (Lumi 1.65.0): how loud what is heard is, for drawing it; declared beside `speech` or `keep-audio` (Lumi 1.64.0) | | `voices` | everything in `lumi::voices` — `remember` and `rename` only from a command, a menu bar row or one of your windows — and `Speakers::Named` in `lumi::transcribe`; declared beside `speech` (Lumi 1.64.0) | | `keep-audio` | everything in `lumi::recordings` but `longest_minutes` and `set_longest_minutes`, which need only `microphone` or `system-audio` — `start` and `keep` only from a command, a menu bar row or one of your windows; `start` costs its source's word too. Declared beside `microphone` or `system-audio`, with no `speech` needed (Lumi 1.65.0) | | `timers` | everything in `lumi::timers` (Lumi 1.59.0) and `lumi::schedules` (Lumi 1.60.0), and hearing `on_event("timer", …)` and `on_event("schedule", …)` with `events = [...]` in the manifest | | `time-zone` | everything in `lumi::time_zone`, and hearing `on_event("time-zone", …)` with `events = [...]` in the manifest (Lumi 1.59.0) | | `microphone-use` | everything in `lumi::microphone_use` but `sites`, and hearing `on_event("microphone-use", …)` with `events = [...]` in the manifest (Lumi 1.65.0) | | `microphone-use` and `microphone-sites` | `microphone_use::sites` — the site an app using a microphone shows, by host (Lumi 1.65.0) | | `offer` | everything in `lumi::offer` — from any run, an event's or a timer's included (Lumi 1.65.0) | | `sound` | everything in `lumi::sound` — from any run, an event's or a timer's included — and hearing `on_event("sound-ended", …)` with `events = [...]` in the manifest (Lumi 1.74.0) | Every call into Lumi except `about` and `license` spends from one allowance of 200 per run, so a loop of alerts runs dry long before it can flood the screen. The call is spent before its capability is checked, so a refused call counts too. Past the allowance every call answers an error — and `settings()`, which has no error to give, answers an empty object, so read your settings early in a run rather than after a long loop. ## Your extension [#your-extension] ### `Guest` and `register!` [#guest-and-register] The trait you implement and the macro that wires it up. Call `register!` once, at the crate root, on the type that implements `Guest`. ```rust impl lumi::Guest for MyExtension { fn run_command(name: String, params: String) -> Result; fn run_node(name: String, params: String, items: String) -> Result; fn run_ui(window: String, request: String) -> Result; fn on_lifecycle(event: lumi::Lifecycle) -> Result<(), String>; fn on_event(name: String, payload: String) -> Result<(), String>; } lumi::register!(MyExtension); ``` | Method | Called for | Receives | Answers | | -------------- | ------------------------------------------------------ | ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | `run_command` | A bound `[[command]]` pressed | The command's `name`; its params as a JSON object, variables already rendered | Any string, which is not shown — call `alert` to tell the person something — or an error Lumi reports like a failed action | | `run_node` | A `[[node]]` reached in a flow | The node's `name`, params as a JSON object, and the incoming items as a JSON array | The outgoing items as a JSON array — empty ends the branch | | `run_ui` | A request from one of your windows | The window's `name`, and whatever the page posted | Whatever the page should get back — see [Windows](/extensions/windows#the-bridge) | | `on_lifecycle` | Your extension being installed, updated or uninstalled | A [`Lifecycle`](#on_lifecycleevent) | `Ok(())`, or an error that is only written to Lumi's log | | `on_event` | Something outside your extension that it asked to hear | The event's `name`, and a JSON payload | `Ok(())`, or an error that is only written to Lumi's log | Answer the names you do not know with an error rather than a panic. A panic is reported to the person as the extension having crashed on an error of its own, with no more detail than that; the backtrace goes to Lumi's log. An error's text is cut to 64 KiB. ### `on_lifecycle(event)` [#on_lifecycleevent] ```rust pub enum Lifecycle { Installed, Updated(String), // the version that was replaced Uninstalling, } ``` Three moments in your extension's life, each told to it once: | Event | When | Typical use | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------- | | `Installed` | Right after a first install — after your [installer](/extensions/windows#an-installer-of-your-own), if you ship one, has saved its answers | Open a Welcome window | | `Updated(from)` | Right after a new version replaces an installed one | Say what is new since `from` | | `Uninstalling` | Just before your files are removed | Open a feedback page in the browser | Nothing to do is the ordinary answer: ```rust fn on_lifecycle(_event: lumi::Lifecycle) -> Result<(), String> { Ok(()) } ``` The rules, all of which follow from nobody having pressed anything: * **Only while switched on.** A switched-off extension is not told it was updated or uninstalled — off means none of its code runs. * **Nobody sees the answer.** An error is written to Lumi's log, never shown. * **It cannot stop anything.** An `Uninstalling` that fails or runs out of time is logged, and the uninstall goes ahead. * **`Installed` and `Updated` run after the install has finished**, on their own, with the same allowance and deadline as a command. * **It takes one of your four runs at once.** A hook that arrives while four runs of yours are going is skipped and logged — `Uninstalling` included. * **`Uninstalling` is short and has no windows.** The person is waiting on the Uninstall button, so it has five seconds, and your `ui/` files are about to be deleted, so `open_window` is refused. Open a web page with `open_url` instead — which needs `applications`, plus `network` for a web address. ```rust fn on_lifecycle(event: lumi::Lifecycle) -> Result<(), String> { match event { lumi::Lifecycle::Installed => lumi::open_window("welcome"), lumi::Lifecycle::Updated(from) => lumi::alert(&format!("Updated from {from}")), lumi::Lifecycle::Uninstalling => lumi::open_url("https://example.dev/why-uninstall"), } } ``` A Welcome window is an ordinary [`[[window]]`](/extensions/manifest#window): declare it, and open it from `Installed`. ### `on_event(name, payload)` [#on_eventname-payload] Something that happened outside your extension, which it asked to hear. The payload is a JSON object with a `"v"` field for its own version; a new kind of event is a new `name`, so this method is the last one the trait gains for it. An extension that asks to hear nothing is never called, and answers `Ok(())`: ```rust fn on_event(_name: String, _payload: String) -> Result<(), String> { Ok(()) } ``` The events Lumi sends today: | Name | Payload | When | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `clipboard` | `{"v":1, "at", "hash", "source":{"bundleId","name"}, "origin", "items":[[{"uti", "text" or "blob", "bytes", "path"?, "fileSize"?, "fileToken"?}]]}` | Each copy, less what password managers mark private. Images and large text arrive as blobs already in your [`storage`](#storage); a copied file's `public.file-url` carries `path`, where it is — Finder copies the unreadable `file:///.file/id=…` form — and `fileSize`, its size in bytes (Lumi 1.26+; a folder has no size, a file already gone neither); a PDF, picture, text or code file, sound or film also carries `fileToken`, which a page gives to [`GET /__lumi__/file/`](/extensions/windows) to play or show it; `origin` is your id when the board was your own `write_clipboard` or `paste`. | | `clipboard-ocr` | `{"v":1, "hash", "text"}` | A moment after a copy holding an image, with the text read in it; `hash` names that copy. Never empty. | | `screen-capture` | `{"v":1, "phase":"start" or "end", "region"}` | Lumi taking a picture of the screen for itself — Copy Text from Screen, a flow reading the screen — once as it starts and once as it ends, however it ends. `region` is `"area"` or `"window"` when the person chooses on an overlay, which leaves time to step a window aside with [`hide_window`](#hide_windowname-and-show_windowname); `"frontWindow"` or `"display"` is taken the instant it is asked, which leaves none — a window that must never be in the picture says [`capturable = false`](/extensions/manifest). Lumi never waits on your answer. Needs the `screen` capability and Lumi 1.36.0. | | `window-closed` | `{"v":1, "window", "at"}` | One of your own windows leaving the screen for good: a [panel](/extensions/windows) put away — Esc, a click away, your `close_window`, a `paste` taking it down — or a declared window closed, by you or by the person. `window` is its manifest name, `at` the moment it closed in milliseconds since the Unix epoch — not when you hear it, which can be later. Once per close: a panel already away says nothing. Not for `hide_window`, whose window comes back with `show_window`, and not for a pinned panel losing the keyboard, which is still up; nor for the windows that close because your extension was switched off, uninstalled or updated. Lumi never waits on your answer. Needs no capability — it is only about your own window — and Lumi 1.37.0. | | `preload-enabled` | `{"v":1, "enabled", "at"}` | The person turning loading your panels ahead on or off with the **Preload** switch on your Permissions sheet — see [`preload_enabled()`](#preload_enabled-and-set_preload_enabledon). `enabled` is the switch as it now stands, `at` the moment it moved in milliseconds since the Unix epoch. Only when the person moves it there, never for your own `set_preload_enabled`; `preload_enabled()` answers the same question any time. Lumi never waits on your answer. Needs the `preload` capability — declared without it, the manifest is refused — and Lumi 1.45.0. | | `selection` | `{"v":1, "text", "by", "lines", "editable", "app":{"bundleId","name"}, "at", "clickInterval"}` | The person selecting text with the mouse in another app: `by` is `"drag"`, `"double-click"`, `"triple-click"` or `"shift-click"` — never a plain click, never with ⌘ or ⌃ held, never from the keyboard, which is `keyboard-selection`. `text` is read through Accessibility only, never by a ⌘C, and is not sent for a password field, Lumi's own windows, blank text or text over 16 KiB; a page's list numbers and bullets are left out, and since Lumi 1.59.0 a page nobody can edit reads as it shows — see [`selection()`](#selection). A Chromium browser — Chrome, Edge — tells Accessibility nothing about a page's own text until the person turns on **General → Advanced → Ask Chromium browsers for a page's text**, so a selection in a page is told only then; its address bar is told either way. An editor that keeps its selection from Accessibility — Visual Studio Code with its screen reader support off — is never told; [`selection()`](#selection) from a shortcut still reads it there by ⌘C with `clipboard`. `editable` is [`selection_editable()`](#selection_editable)'s answer for it. `clickInterval` is the Mac's double-click interval in milliseconds: a click within it continues the gesture, so a double-click may yet become a triple-click — [`open_window_after`](#open_window_aftername-after_ms) waits it out (Lumi 1.50.0). Nothing says where on the screen. Only while the **Selection** switch on your Permissions sheet is on — see [`selection_watching()`](#selection_watching-set_selection_watchingon-and-selection_watchers). The one event whose run may open a window: one [panel declared `position = "selection"`](/extensions/windows), once, beside that selection — below. A newer selection replaces one still waiting, and one more than a second old is not delivered. Needs the `selection-watch` capability and Lumi 1.47.0. | | `keyboard-selection` | `selection`'s, with `by` `"keyboard"` and `"keys"` | The person selecting text with the keyboard in another app — ⇧ and the arrows, ⇧⌥ by words, ⇧⌘ to the line's end, ⌘A, ⌘G — told once the selection has rested: unchanged for 300 ms and every modifier up for 150 ms, so a selection made to type over is never told. Only a change the person's last key just made, and a key that selects — ⇧ with a key that moves the caret, or anything with ⌘ or ⌃ held: a press or a drag is `selection`'s, what an app selects as the person types (a word it completes) is never told, and what an app or Lumi itself selects is nobody's. Its own name so an extension written for the mouse is never told of the keys; hear both to be told of both. `keys` says what kind of key made it, never which: `"extend"` — ⇧ with a key that moves the caret, ⌥ or ⌘ held or not — `"select-all"` — ⌘A, by the letter it types on any layout — or `"command"` — any other key with ⌘ or ⌃ held, a command that left text selected: Find filling its field, an address bar's ⌘L, Find Next, the ⌃⇧ text bindings. An extension that would answer only the person selecting on purpose answers the first two. Read, gated and granted as `selection` is — its run may open the same one panel beside it — and Lumi measures the place without the pointer, which is wherever the person left it. An app that does not report its selection changing to Accessibility, as Visual Studio Code's editor does not unless its screen reader support is on, is never told. Needs the `selection-watch` capability and Lumi 1.50.0. | | `selection-watching` | `{"v":1, "watching", "at"}` | The person turning the **Selection** switch on your Permissions sheet on or off. Only when they move it, never for your own `set_selection_watching`. Needs the `selection-watch` capability and Lumi 1.47.0. | | `timer` | `{"v":1, "name", "at", "late", "missed"}` | One of your own [timers](#timers) coming due: `name` as you set it, `at` the moment it fired in milliseconds since the Unix epoch — not when you hear it, which can be later. `late` is true when that was more than a minute after it was due: the Mac slept, Lumi was quit or your extension was off. `missed` counts the further repeats that came due meanwhile, folded into this one firing. Told at least once — see [Timers](#timers). `timers::fired` reads it. Needs the `timers` capability and Lumi 1.59.0. | | `schedule` | `{"v":1, "name", "at", "late", "missed"}` | One of your own [schedules](#schedules) coming due — `timer`'s payload, under its own name: `at` the moment it fired, `late` more than a minute after its moment, `missed` the further moments folded into this firing. `schedules::fired` reads it. Needs the `timers` capability and Lumi 1.60.0. | | `time-zone` | `{"v":1, "id", "offsetMinutes", "at"}` | This Mac's time zone changing — see [Time zone](#time-zone). `id` and `offsetMinutes` are the new zone as `time_zone::current()` answers it, `at` the moment it changed in milliseconds since the Unix epoch. Once per change; a change of clocks for summer is not a change of zone and is not told. `time_zone::changed` reads it. Needs the `time-zone` capability and Lumi 1.59.0. | | `microphone-use` | `{"v":1, "apps":[{"app", "name", "since"}], "at"}` | The apps using a microphone changing — see [Microphone in use](#microphone-in-use). The whole set each time, never a change: every app with a microphone open then, by name, `app` `null` for a process without a bundle id. Told once too, as it stands, when your extension starts hearing it. Changes within a quarter of a second are one. `microphone_use::changed` reads it. Needs the `microphone-use` capability, macOS 14 and Lumi 1.65.0. | | `menu-shown` | `{"v":1, "shown", "at"}` | The person showing or hiding your rows in Lumi's menu bar menu with the **Menu bar** switch on your Permissions sheet under Extensions — see [Whether your rows show](#whether-your-rows-show). `shown` is the switch as it now stands, `at` the moment it moved in milliseconds since the Unix epoch. Only when the person moves it there, never for your own `menu::set` or `menu::set_shown`; `menu::shown()` answers the same question any time. Lumi never waits on your answer. Needs the `menu` capability and Lumi 1.44.0. | | `menu-bar-item-shown` | `{"v":1, "shown", "at"}` | The person showing or hiding your own item in the menu bar with the **Show its icon in the menu bar** switch on your Permissions sheet — see [Menu bar item](#menu-bar-item). `menu-shown`'s payload and rules: only when the person moves it there, never for your own `menu_bar_item::set_shown`; `menu_bar_item::shown()` answers the same question any time. Lumi never waits on your answer. Needs the `menu-bar-item` capability and Lumi 1.61.0. | | `sound-ended` | `{"v":1, "sound", "name", "finished", "at"}` | One of your own [sounds](#sounds) stopping: `sound` the number `play` answered, `name` the one it was played under or null, `finished` true when it came to its end and false when it was stopped — by you, by a sound of the same name, or by the person's Stop row — and `at` when. Not told when your extension is switched off or removed. Lumi never waits on your answer. Needs the `sound` capability and Lumi 1.74.0. | It runs under `on_lifecycle`'s rules — only while switched on, its answer only written to Lumi's log, one of your four runs at once — and two more that follow from nobody having pressed anything: `open_window` cannot open a [panel](/extensions/windows) from here, and `paste` is refused. `selection` and `keyboard-selection` are the exception, because the person selecting is the moment: its run may `open_window` one panel declared `position = "selection"`, once, within two seconds of the selection and only while it is still the person's latest. The panel opens beside that selection without the keyboard, whatever its `focus` says, and goes at their next click, key or scroll. The run may set that panel's size, material, theme and anchor and `post` to it; `paste`, `replace_selection`, `replace_selection_and_select`, `set_pinned` and opening Settings are still refused. ```rust fn on_event(name: String, payload: String) -> Result<(), String> { if name == "selection" { let said: serde_json::Value = serde_json::from_str(&payload).map_err(|e| e.to_string())?; lumi::post("bar", &said.to_string())?; return lumi::open_window("bar"); } Ok(()) } ``` ### `settings()` and `setting(name)` [#settings-and-settingname] ```rust pub fn settings() -> serde_json::Value pub fn setting(name: &str) -> Option ``` Your `[[settings]]` values as the person saved them, one key per field, with defaults filled in for anything untouched. A field declared `scope = "profile"` holds the live profile's value. `setting` is the shortcut for one string field. Settings are fixed for the length of a run: a save in Lumi while your code is running reaches the next run. Your own [`set_setting`](#set_settingname-value) is the exception — the rest of the run reads what it saved. ### `set_setting(name, value)` [#set_settingname-value] ```rust pub fn set_setting(name: &str, value: &str) -> Result<(), String> ``` Saves one setting, as a save from your Settings page does — for a menu row that switches it without the person opening the page: ```rust lumi::set_setting("method", "vni")?; ``` The value is text, as every setting is (`"true"` and `"false"` for a `bool`). It is held to the field's own rules — a value they refuse is refused with the sentence the page would show — filed under the live profile for a field declared `scope = "profile"`, and told to your open pages. Only for something the person did: a command, a menu bar row or one of your windows. Needs Lumi 1.61.0. ### `open_window(name)` [#open_windowname] ```rust pub fn open_window(name: &str) -> Result<(), String> ``` Opens — or brings forward — one of the windows your manifest declares. A name the manifest does not declare is refused. A window with [`copies`](/extensions/windows#several-of-one-window) is opened copy by copy, `open_window("card-3")`, in this call and every other that names a window. One already up is focused, never closed, your command's own press included; for a key that also puts it away, see `window_state` below. See [Windows](/extensions/windows). It is refused from `run_node` too: a flow runs in the background, and a dialog out of a flow nobody just pressed is not something a flow may do yet. And it is refused from `Uninstalling`, since your `ui/` files are about to go. ```rust pub fn open_window_on(name: &str, beside: &str) -> Result<(), String> ``` `open_window_on` is `open_window` on the display your window `beside` stands on now. A window that opens where it was left (`remember-frame`) opens at that place carried to this display — as far from the same edges, the nearer of left and right and of top and bottom, held inside — and one that opens at a corner or in the middle opens at that one of this display; one that opens at the pointer or a selection opens there, as ever. One already up is focused where it stands; with `beside` not on screen, or on the same display, it is `open_window`. It is for a smaller window standing in for a bigger one, opened as the bigger one is put away — a browser minimized to a pill — which should appear on the display the person was looking at. Open it before you [`hide_window`](#hide_windowname-and-show_windowname) the bigger one, while that one is still on screen to be measured. Both are names your manifest declares (Lumi 1.65.0). ```rust pub fn open_window_at(name: &str, beside: &str, corner: &str) -> Result<(), String> ``` `open_window_at` is `open_window` at a corner of your window `beside` as it stands now: the panel opens with its `corner` on that window's same corner — `"top-left"`, `"top-right"`, `"bottom-left"` or `"bottom-right"`, or `"center"` for its middle on that one's middle — held inside the display `beside` is on, against its edge where it would run past it. A window `hide_window` took away counts as standing where it was, since that is where `show_window` puts it back. It is for a smaller window standing in for a bigger one, which should appear where that one was — a pill at the foot of the browser minimized to it: ```rust lumi::open_window_at("pill", "browser", "bottom-right")?; lumi::hide_window("browser")?; ``` A panel only — macOS places a window as it opens — and not one Lumi places beside the selection, the menu bar item or another window. One already up is focused where it stands; with `beside` neither on screen nor stepped aside it is `open_window`. Both are names your manifest declares (Lumi 1.75.0). ### `open_window_after(name, after_ms)` [#open_window_aftername-after_ms] ```rust pub fn open_window_after(name: &str, after_ms: u32) -> Result<(), String> ``` `open_window`, `after_ms` milliseconds from now — from the run of a `selection` or `keyboard-selection` event only, as its one opening of the panel beside that selection. It answers at once; the panel opens when the time is up, unless the person has pressed, typed or scrolled since — then it never opens — or the selection is two seconds old. At most 1000; 0 is `open_window`. Refused from any other run. For a panel that waits for the person to finish: a double-click may yet become a triple-click, and the `selection` event's `clickInterval` says for how long. Waited out, the double-click's panel opens; cut short by the third click, it never shows, and the triple-click's own run opens one. ```rust let said: serde_json::Value = serde_json::from_str(&payload).map_err(|e| e.to_string())?; if said["by"] == "double-click" { let wait = said["clickInterval"].as_u64().unwrap_or(500) as u32; return lumi::open_window_after("bar", wait); } lumi::open_window("bar") ``` Needs Lumi 1.50.0. ### `close_window(name)` [#close_windowname] ```rust pub fn close_window(name: &str) -> Result<(), String> ``` Closes one of your declared windows — what a panel's Esc does. A window that is not open is not an error. ### `window_state(name)` [#window_statename] ```rust pub fn window_state(name: &str) -> Result // Hidden | Up | Focused ``` Where one of your declared windows stands: `Hidden` (not on screen, a panel put away included), `Up` (on screen while the keyboard is elsewhere — a pinned panel the person left, even with the pointer over it) or `Focused`. What a command asks before it decides — a key that toggles closes a `Focused` panel and opens the rest. Works from `on_event` too. Needs Lumi 1.31.0. ### `hide_window(name)` and `show_window(name)` [#hide_windowname-and-show_windowname] ```rust pub fn hide_window(name: &str) -> Result pub fn show_window(name: &str) -> Result pub fn show_window_on(name: &str, beside: &str) -> Result pub fn show_window_at(name: &str, beside: &str, corner: &str) -> Result ``` `hide_window` takes one of your declared windows off screen **without closing it**: its page keeps running and keeps everything in it, where `close_window` ends a window's page and blanks a panel's (unless the panel [keeps its page](/extensions/windows#keep-the-page-between-openings)). `show_window` puts it back where it stood among everybody's windows — under whatever was over it, with the keyboard if it had it. A [panel beside the selection](/extensions/windows#a-panel-beside-the-selection) comes back as it opened: one without the keyboard goes again at the next key, press or scroll elsewhere, as it would have had it never stepped aside. Each answers whether it did: `hide_window` is `false` for a window already off screen, closed or never opened, or in full screen — it has a Space of its own, and taking it away would leave the person on an empty one, while nothing is behind it to see — and `show_window` is `false` for any window `hide_window` did not take away — it restores, it never opens, and a window the person opened or closed in between stays as they left it. Lumi remembers what it took away and where, so your code needs to remember nothing between the two. A window showing [web pages](/extensions/windows) keeps them playing: it stays where it is, see-through and letting every click through to what is under it, rather than leaving the screen, because a site whose window leaves is told its page is hidden, and some stop playing for that — YouTube's phone layout does (Lumi 1.65.0). `window_state` says `hidden` for it all the same. `show_window_on` is `show_window` on the display your window `beside` stands on now: the window comes back at the place it had on its own display — as far from the same edges, the nearer of left and right and of top and bottom — held inside that one. On the same display, or with `beside` not on screen, it is `show_window`. It is for a window stepped aside in favour of a smaller one the person can carry to another display — a browser minimized to a pill — which should come back where that one is. Both are names your manifest declares (Lumi 1.65.0). `show_window_at` is `show_window` at a corner of your window `beside` as it stands now, in [`open_window_at`](#open_windowname)'s words: the window comes back with its `corner` on that window's same corner, or its middle on that one's middle, held inside the display `beside` is on. With `beside` not on screen it is `show_window`. It is for a window that should come back from where the smaller one stands, wherever the person dragged it, rather than from where it was left — a browser rising from its pill, `"bottom-right"` both ways, so that minimizing and restoring without a drag in between puts it back exactly where it stood. Refused for a panel Lumi places beside something. Both are names your manifest declares (Lumi 1.75.0). For getting out of the way of something on the screen — an editor while Lumi reads the text under it: ```rust fn on_event(name: String, payload: String) -> Result<(), String> { if name == "screen-capture" { let event: serde_json::Value = serde_json::from_str(&payload).map_err(|e| e.to_string())?; match (event["phase"].as_str(), event["region"].as_str()) { (Some("start"), Some("area" | "window")) => { lumi_extension_api::hide_window("editor")?; } (Some("end"), _) => { lumi_extension_api::show_window("editor")?; } _ => {} } } Ok(()) } ``` Both work from `on_event`. Needs Lumi 1.36.0. ### `post(window, message)` [#postwindow-message] ```rust pub fn post(window: &str, message: &str) -> Result ``` Sends `message`, JSON text, to one of your own pages while it is on screen — a declared window by its manifest `name`, or a page of yours drawn in a tab by the name its `call` arrives under: `:about`, `:settings`, `:shortcuts`, `:changelog` or `:page:`. The page hears it as a [`lumi:message` event](/extensions/windows#when-your-extension-has-news), its `detail` the parsed JSON. It answers `true` when the page was there to hear it and `false` when it was not, so nothing was sent: a post is news for whoever is looking, never queued for later. A page that opens afterwards asks for what it shows with a `call`. A panel put away that [keeps its page](/extensions/windows#keep-the-page-between-openings) is told all the same — that page is the one shown next (Lumi 1.45.0). It is the way to keep an open page current with what just happened — a copy heard in `on_event`, a count that changed — and the one way to reach an embedded page while Lumi's Settings window is behind another app, when the page's own `call` is refused. Data goes to the page and nothing comes back, so it works from anywhere, `on_event` included, and needs no capability. ```rust fn on_event(name: String, _payload: String) -> Result<(), String> { if name == "clipboard" { // … keep the copy under "item." … let copies = lumi::storage::keys("item.")?.len(); lumi::post(":page:dashboard", &serde_json::json!({ "kind": "count", "copies": copies }).to_string())?; } Ok(()) } ``` A window your manifest does not declare is refused, and so is a message over 64 KiB or one that is not JSON. A [sheet](/extensions/windows#a-sheet) cannot be posted to: its page answers the page that opened it. ### `open_settings()` [#open_settings] ```rust pub fn open_settings() -> Result<(), String> ``` Brings up Lumi's Settings window on your extension's own page, on its Settings tab — the "Settings…" row of a panel or a menu. Only for a press: a command, a menu bar row or one of your windows, never `on_event` or a lifecycle hook, since Settings coming forward on its own would be a window jumping out at somebody. No capability: it shows Lumi's own pane, and only yours. ### `switch_off()` [#switch_off] ```rust pub fn switch_off() -> Result<(), String> ``` Switches your extension off, as the switch beside it under **Extensions** in Lumi's Settings does — for a "Quit" row of your own, in a panel or in your [menu bar item's](#menu-bar-item) menu. Your windows close, your menu bar item and rows go, and none of your code runs — commands, shortcuts, events, timers — until the person switches it on again there. You can switch yourself off, never on. Only for a press, as `open_settings` is, and refused while your extension is being uninstalled. No capability: it only takes away. Needs Lumi 1.61.0. ### `open_permissions()` [#open_permissions] ```rust pub fn open_permissions() -> Result<(), String> ``` Brings up Lumi's Settings window on your extension's own page with its Permissions sheet open: what your extension may reach, and the switches Lumi keeps over it — the folders it writes into, whether its rows show in Lumi's menu bar menu, whether its panels are loaded ahead. Lumi draws that sheet itself, whatever your own pages draw. Only for a press, like `open_settings`. No capability. Needs Lumi 1.44.0. ### `preload_enabled()` and `set_preload_enabled(on)` [#preload_enabled-and-set_preload_enabledon] ```rust pub fn preload_enabled() -> Result pub fn set_preload_enabled(on: bool) -> Result<(), String> ``` Whether Lumi loads your panels ahead — those whose `[[window]]` says [`preload = true`](/extensions/manifest#window) — as your extension is loaded: at launch, once installed or updated, when switched on. One switch, kept by Lumi the way your menu bar rows' is — on from install unless your manifest says [`preload-default = false`](/extensions/manifest#extension), then where it was left: your own settings move it with `set_preload_enabled`, and the person moves the same switch with **Start when Lumi opens** on your Permissions sheet. While it is off, such a panel loads at its first opening instead and keeps its page from then on, as any [kept panel](/extensions/windows#keep-the-page-between-openings) does — so the first opening after Lumi starts is the slower one. Turned on, your panels are loaded ahead at once; turned off, a page loaded ahead and not opened yet is let go. ```rust // In run_ui, for a "Start when Lumi opens" in your own settings: "preload" => lumi::preload_enabled().map(|on| format!(r#"{{"on":{on}}}"#)), "set-preload" => lumi::set_preload_enabled(on).map(|()| String::new()), ``` Declare `events = ["preload-enabled"]` to hear the person move it on the sheet, with `{"v":1, "enabled", "at"}` — never for your own `set_preload_enabled`. Both answer from anywhere, `on_event` included, like `window_state`; `set_preload_enabled` not from an uninstall. Both need the `preload` capability, and are refused for an extension none of whose panels says `preload`. Needs Lumi 1.45.0 — set `min-lumi-version`, since a component that calls them does not load on an older Lumi. ### `set_tab_label(tab, label)` [#set_tab_labeltab-label] ```rust pub fn set_tab_label(tab: &str, label: Option<&str>) -> Result<(), String> ``` Puts your own words on one of the tabs of your page under **Settings → Extensions** — for pages shown in the language the person picked, whose tabs should follow. `tab` is named as a call from it arrives: `":about"`, `":settings"`, `":shortcuts"`, `":changelog"` or `":page:"`. `None` puts back the label your [manifest](/extensions/manifest#labels-that-follow-your-language) gives it. ```rust fn on_lifecycle(_event: lumi::Lifecycle) -> Result<(), String> { // Right before anybody opens the page: on install and on every update. label_tabs() } fn label_tabs() -> Result<(), String> { let vietnamese = lumi::setting("language").as_deref() == Some("vi"); lumi::set_tab_label(":settings", Some(if vietnamese { "Cài đặt" } else { "Settings" })) } ``` Call it wherever your language setting changes too — your settings page's `call` is the usual place, or the page sends [`POST /__lumi__/tab-label`](/extensions/windows#tabs-in-the-persons-language) itself. The label is held to a manifest label's rules: up to 24 characters on one line, not Permissions, not the name of another of Lumi's tabs — the tab's own name back is fine — and no two of your tabs alike. A tab your manifest hides or does not declare is refused. Lumi keeps the label across restarts and updates, and the same label again is no change, so calling it on every run costs one call. The review sheet and your Permissions sheet keep the manifest's labels. From any run but an uninstall; no capability. Needs Lumi 1.61.0 — set `min-lumi-version`, since a component that calls it does not load on an older Lumi. ### `storage` [#storage] ```rust pub mod storage { pub fn get(key: &str) -> Result, String>; // Entry { value, rev } pub fn put(key: &str, value: &str, if_rev: Option) -> Result; pub fn delete(key: &str) -> Result<(), String>; pub fn keys(prefix: &str) -> Result, String>; pub fn blob_write(bytes: &[u8]) -> Result; pub fn blob_read(id: &str) -> Result, String>; pub fn blob_delete(id: &str) -> Result<(), String>; pub fn blob_save(id: &str, name: &str) -> Result<(), String>; // Lumi 1.26+ pub fn blob_save_dated(id: &str, name: &str) -> Result<(), String>; // Lumi 1.34+ pub fn blobs() -> Result, String>; pub fn usage() -> Result<(u64, u64), String>; } ``` Your extension's own place to keep things between runs — nothing survives in memory, since every command, node and page request starts a fresh instance. Lumi keeps it on the Mac, encrypted with a key it holds in the Keychain: your component never sees the key, no other extension can read it, and it is not in a Lumi backup. It survives updates and is removed when your extension is uninstalled. `put` writes only if the key is still at the revision you read (`None`: it must not exist yet), and answers `PutError::Conflict` if another run wrote first — read again, reapply, write again. Any other refusal is `PutError::Failed`, with the sentence; `into_message()` turns either into a `String` for `?`: ```rust use lumi::storage::{self, PutError}; loop { let seen = storage::get("count")?; let n = seen.as_ref().map_or(0, |e| e.value.parse().unwrap_or(0)) + 1; match storage::put("count", &n.to_string(), seen.map(|e| e.rev)) { Err(PutError::Conflict) => continue, other => break other.map(drop).map_err(PutError::into_message)?, } } ``` **Key names are not encrypted**; they are file names, so keep secrets in values and use random ids in names. A name is 1–128 lowercase letters, digits, `.`, `-` or `_`; a value is at most 1 MB, a blob written from your component at most 4 MB, and the whole store 512 MB. Lumi writes blobs into your store too — a copied image, for an extension that hears `clipboard` (up to 32 MB), a `screen` capture, or what your page uploads with `PUT /__lumi__/blob` (both up to 160 MB). Reading one that large back into your component fails at the 4 MB boundary; you rarely need to. Hand its id to your page, which shows it through `GET /__lumi__/blob/` ([the bridge](/extensions/windows#the-bridge)), or put it back on the clipboard with `Data::Blob`, which never passes through your component. `blob_save` hands a blob to the person as a file. Your panels close, and Lumi's Save panel comes up with `name` filled in and your extension's name above it; the file goes wherever they choose. It returns once the Save panel is up and does not say where the file went or whether they cancelled — a write that fails is reported to them in an alert. To know whether it was saved, use [`files::save`](#files). `name` is a file name, not a path: 1–255 bytes, no `/`, no control characters, not starting with `.`. Any size of blob works, Lumi's own large ones included, since the bytes never pass through your component. Like `paste`, it is refused unless it answers a press — a command or one of your windows. Needs Lumi 1.26.0; say so with `min-lumi-version`. `blob_save_dated` is `blob_save` with every `{date}` and `{time}` in `name` filled in from the Mac's own clock and time zone — `2026-10-02` and `14.03.11`, the way macOS names a screenshot. Your component's clock is in UTC, so a name it writes itself is hours off for most people. The name filled in is not told back: `blob_save_dated` costs no capability, and Lumi lets a component learn the person's time zone only through one that says so — [`time-zone`](#time-zone), and [`timers`](#timers), whose clock times fire on the person's clock. Needs Lumi 1.34.0. ```rust lumi::storage::blob_save_dated(&shot.blob, "Screenshot {date} at {time}.png")?; ``` Needs Lumi 1.25.0 or later — set `min-lumi-version` if you call it. A component imports the interfaces it calls, so an older Lumi refuses to start one that uses storage; one that never calls it runs as before. ## Talking to the person [#talking-to-the-person] ### `alert(text)` [#alerttext] ```rust pub fn alert(text: &str) -> Result<(), String> ``` Draws a line of text over whatever is in front, in the alert style the person chose in Lumi. It takes no focus and fades on its own — the way an extension answers somebody who pressed a key with no Lumi window open. Text over 64 KiB is refused with an error rather than cut. ## The desk [#the-desk] ### `selection()` [#selection] ```rust pub fn selection() -> Result pub struct Found { pub text: String, pub how: String, // "accessibility", "copy" or "nothing" } ``` The text selected in the frontmost application. Needs `accessibility`. Lumi asks the application through Accessibility first. If you also declared `clipboard`, an application that exposes nothing is asked again with a ⌘C, and the pasteboard is put back afterwards; `how` says which one answered. So is one whose only answer is an empty text field saying nothing in it is selected — the stand-in an editor that draws its own text keeps for the keys, as Visual Studio Code's is with its screen reader support off. Asked from one of your panels while it holds the keys, where the ⌘C would go into the panel, Lumi first gives the keys back to that application, as `replace_selection_and_select` does (Lumi 1.50.0) — and since Lumi 1.55.0 only lends them: once the run is over, the panel holds them again, so the person goes on typing in its field rather than over the selection. A run that goes on to write over the selection or paste leaves them with the application, as before. A panel put away meanwhile — the person pressed or typed elsewhere while the keys were lent — stays put away. Empty text with `how == "nothing"` is the ordinary miss — nothing selected — not an error. Read from a web page through Accessibility, the text leaves out the numbers and bullets of any list it runs through, with a line break where each item began. Since Lumi 1.59.0 a page nobody can edit reads as the page shows it, where a browser's own text would lose it. Chrome and the apps built on it answer a radio button or a checkbox with U+FFFC or with nothing, its label's text gone with it — a quiz's question and its answers read as the question alone — and glue a page's blocks together. So Lumi puts each control's label where it stands, leaves pictures out, starts each paragraph, heading or list item on a line of its own, and keeps a table row's cells apart with tabs. A selection across a whole long page may be left as the browser gives it, but never with U+FFFC in it. Text that can be edited reads as the app gives it — a picture's U+FFFC included, below. ### `copy_selection()` [#copy_selection] ```rust pub fn copy_selection() -> Result ``` Copies the selection in the frontmost application the way that application copies it — its own ⌘C — so the board holds what it puts there: the text as it lays it out, and the formatting, links and pictures it keeps, which no reading of the text carries. For a Copy button: what the person pastes is what the app's own Copy would have given them. `true` once the board has changed; `false` when nothing changed within a moment — nothing selected, or nothing the app will copy. Unlike `selection()`'s ⌘C, the board is not put back: this is the person's copy, and a clipboard history keeps it as one. Asked from one of your panels while it holds the keys, the keys are lent to the app for the copy and come back to the panel once the run is over, as `selection()` lends them. Refused with one of Lumi's own windows in front. Needs `clipboard`, `input` and `accessibility` — it presses a key in somebody else's app and changes the board — and answers only a command or a request from one of your windows; refused from `on_event` and `on_lifecycle`. Needs Lumi 1.59.0. ### `selection_editable()` [#selection_editable] ```rust pub fn selection_editable() -> Result, String> ``` Whether the selection in the frontmost application can be written over, as Accessibility says. `Some(false)` is text nobody can edit there — a web page, a terminal's scrollback — where `replace_selection` is refused; `Some(true)` is a field or a document; `None` is an app that says neither, where `replace_selection` still types or pastes. Ask it before offering to replace, so a button that can do nothing is never shown. Needs `accessibility` and Lumi 1.47.0. ### `replace_selection(expected, replacement)` [#replace_selectionexpected-replacement] ```rust pub fn replace_selection(expected: &str, replacement: &str) -> Result ``` Puts `replacement` where `expected` is selected in the frontmost application. `expected` is the text `selection()` answered: a selection the person has changed since is refused, never written over. The answer says how it went in: `"accessibility"`, `"typed"` or `"pasted"`. Lumi writes it through Accessibility where the app takes it, types it when it is short, and otherwise pastes it through the general pasteboard and puts the board back — that last only when you also declare `clipboard`; without it a long replacement the app will not take through Accessibility is refused. Where `selection_editable()` answers `Some(false)` it is refused with nothing typed: there a keystroke is a key — a page's shortcut, a word at a shell prompt — not a replacement. Your panels close first, so the keys land where the selection is. A picture in the selection — an image, or an emoji an app draws as a picture rather than a character, as Slack does — reads as U+FFFC where it stands in the text `selection()` answers, and says nothing about which picture it was. In text that can be edited it does so in every app since Lumi 1.59.0, Safari's included, which leaves it out on its own; on a page nobody can edit the picture is left out of the text since then, as a browser's own copy leaves it out. Since Lumi 1.59.0 a write whose `expected` holds one is refused before anything closes: no text written over it can put the picture back. Ask `text.contains('\u{FFFC}')` before offering to replace, as you ask `selection_editable()`. Needs `accessibility` and `input`, and answers only a command or a request from one of your windows — refused from `on_event` and `on_lifecycle`, where nobody asked for anything to change. Needs Lumi 1.47.0. ### `replace_selection_and_select(expected, replacement)` [#replace_selection_and_selectexpected-replacement] ```rust pub fn replace_selection_and_select(expected: &str, replacement: &str) -> Result ``` `replace_selection`, every rule of it, then the text written selected again where it was written — whatever its length. "I love you" written over as "tôi yêu em nhiều" is selected as those sixteen characters, so the person sees what changed and your next action works on it. `Replaced::how` is `replace_selection`'s answer; `Replaced::selected` says whether the new text is selected now. Lumi reads where the selection began before it writes, and selects the new text only once it can read it there exactly — typed or pasted text is waited for until it arrives. `selected` is `false` where the app gives no place for a selection through Accessibility, or rewrote the text on its way in; the caret is then where the write left it, as after `replace_selection`. Unlike `replace_selection`, your panels stay up: only a panel holding the keyboard gives it back first — one beside the selection staying up without it, any other closed — so a panel beside the text never blinks out and in around the write. To keep it with the text, open it again when this answers with `selected`: Lumi measures the new selection and moves the panel beside it. ```rust let found = lumi::selection()?; let changed = lumi::transforms::apply("upper", &found.text)?; if lumi::replace_selection_and_select(&found.text, &changed)?.selected { lumi::open_window("peek")?; } ``` Same capabilities and the same callers as `replace_selection`. Needs Lumi 1.48.0. ### `selection_watching()`, `set_selection_watching(on)` and `selection_watchers()` [#selection_watching-set_selection_watchingon-and-selection_watchers] ```rust pub fn selection_watching() -> Result pub fn set_selection_watching(on: bool) -> Result<(), String> pub fn selection_watchers() -> Result, String> ``` Whether you are told of the selections the person makes — the `selection` and `keyboard-selection` events — is one switch, kept by Lumi and shown as **Selection** on your Permissions sheet. It starts on when you are installed, unless your manifest says `selection-watch-default = false`, and stays where it was left: your own settings move it with `set_selection_watching`, and the person moves it on the sheet. Declare `events = ["selection-watching"]` to hear the person move it. `selection_watchers` names the other extensions told of the person's selections now. Lumi does not choose between you — each opens its own panel — so you can say so, or offer to step aside. All three need `selection-watch` and Lumi 1.47.0; `set_selection_watching` not from an uninstall. ### `transforms::all()` and `transforms::apply(id, text)` [#transformsall-and-transformsapplyid-text] ```rust pub mod transforms { pub fn all() -> Result, String>; pub fn apply(id: &str, text: &str) -> Result; } pub struct Transform { pub id: String, // "upper", "sortLines", "jsonPretty", … pub label: String, // "UPPERCASE", "Sort lines", "Pretty-print JSON", … pub group: String, // "case", "lines" or "encode" } ``` Lumi's own text transforms — the ones a **Transform selection** shortcut offers — by name. `all` lists every built-in one in Lumi's order; one the person wrote as a script is not among them, since that is their own code. `apply` runs `text` through the one `id` names, refused for an id `all` does not list and for text over 1 MiB; a transform that cannot do what it says — JSON that does not parse, base64 that is not — answers why rather than an empty string. Pure: nothing is read or written but the text you hand over, so no capability is needed. Needs Lumi 1.47.0. ```rust let found = lumi::selection()?; if lumi::selection_editable()? != Some(false) { let shouted = lumi::transforms::apply("upper", &found.text)?; lumi::replace_selection(&found.text, &shouted)?; } ``` ### `translation` [#translation] ```rust pub mod translation { pub fn languages() -> Result, String>; pub fn preferred() -> Result, String>; pub fn status(source: &str, target: &str) -> Result; pub fn detect(text: &str) -> Result, String>; pub fn translate(request: &Request) -> Result; pub fn prepare(source: &str, target: &str) -> Result; } pub enum Readiness { Installed, Downloadable, Unsupported } pub enum Quality { Fast, Best } pub struct Request { pub text: String, pub source: Option, // None or empty: Lumi's guess, when it is sure pub target: Option, // None or empty: the person's own language pub quality: Quality, } pub struct Translated { pub text: String, pub source: String, // "en", "vi", "zh-TW", … pub target: String, } pub struct Guess { pub language: String, pub confidence: f32, // 0 to 1 } ``` Translation with the translator built into macOS, the one the Translate app uses. The text is translated on this Mac and never leaves it, so no capability is needed. Languages are BCP 47 ids as the system names them — `"en"`, `"vi"`, `"zh-TW"` — and any spelling of one is understood (`"vi-VN"`, `"zh-Hant"`). Needs Lumi 1.51.0 and macOS 26: on an older macOS every function answers that translating needs macOS 26, so check `languages` before you offer to translate. * `languages` lists every language this Mac translates. The list grows with macOS, so read it rather than keeping one. * `preferred` is the person's own languages that the Mac translates: the Mac's preferred languages in their order, then the language of its region. A Mac set to English in Vietnam answers `["en", "vi"]` — so an English text has somewhere to go. * `status` says whether a pair is `Installed`, `Downloadable` — macOS downloads its languages once, for every app — or `Unsupported`. Only an installed pair translates. * `detect` answers up to three guesses at a text's language, most likely first. A sentence is told reliably; a word or two often is not ("OK" reads as Polish), nor is Vietnamese typed without its accents. Weigh the confidence, and let the person correct the language rather than trusting a low guess. * `translate` translates line by line: blank lines, each line's indent, and web and email addresses come back as they were, and a line with no words in it is left alone. It does not join a paragraph that was wrapped at a fixed width into one line first — that is a guess about the text, and yours to make. With no `source`, Lumi takes its top guess when it is at least 0.8 sure, and refuses otherwise; with no `target`, the first of `preferred` that is not the source. Refused over 20,000 characters, for a pair that is not installed, and for a source and target that are the same language — each answers why. The first call takes about a second while macOS loads the language; after that, a sentence is a few dozen milliseconds. `Quality::Best` asks for Apple Intelligence's model, on a Mac that has it with macOS 26.4: more fluent and many times slower. Anywhere else it is `Fast`. * `prepare` puts up macOS's own "Download Languages to Translate" prompt for a pair, and answers how ready the pair is once the person is done with it. Only from a command or one of your windows — a prompt nobody asked for is one somebody answers blind. It stands at the top of the screen the pointer is on, as macOS's own sheets do, takes the keyboard while it is up and hands it back to the app in front when it closes; one at a time across Lumi, and taken down after two minutes unanswered. It answers once the prompt is gone, so what you do next — read the selection, open a panel beside it — finds the app in front with its selection again, and a transient panel of yours beside the selection stays up while the person answers it. A declined prompt is no error: the answer is `Downloadable`, as it stays while macOS downloads in the background, so ask `status` again at the next press rather than waiting on it. Offer it when `status` says `Downloadable`, behind a button the person presses. Apple says it may count which languages an app translates between, but never what was translated. ```rust let found = lumi::selection()?; let guesses = lumi::translation::detect(&found.text)?; let source = guesses.first().filter(|g| g.confidence >= 0.8).map(|g| g.language.clone()); let done = lumi::translation::translate(&lumi::translation::Request { text: found.text, source, target: None, quality: lumi::translation::Quality::Fast, })?; lumi::alert(&format!("{} → {}: {}", done.source, done.target, done.text))?; ``` ### `dictionary` [#dictionary] ```rust pub mod dictionary { pub fn dictionaries() -> Result, String>; pub fn lookup(text: &str, books: &[String]) -> Result, String>; pub fn term(text: &str, at: u32) -> Result, String>; pub fn show(text: &str) -> Result<(), String>; } pub struct Book { pub id: String, // "com.apple.dictionary.NOAD" pub name: String, // "New Oxford American Dictionary" pub short_name: String, // "Dictionary" pub looks_up: Vec, // BCP 47: "en-US"; two for a bilingual one pub explains_in: Vec, } pub struct Entry { pub book: String, // a `Book`'s id pub headword: String, pub syllables: String, // "ser·en·dip·i·ty", or empty pub pronunciations: Vec, pub groups: Vec, pub text: String, // the whole entry as plain text, always there } pub struct Group { pub heading: String, // "noun", "transitive verb", "PHRASAL VERBS" pub pronunciations: Vec, // for this part of speech alone pub senses: Vec, } pub struct Sense { pub phrase: String, // "set off", or empty for the headword's own pub definition: String, // empty in a bilingual dictionary or a thesaurus pub translations: Vec, pub synonyms: Vec, pub antonyms: Vec, pub examples: Vec, // at most three pub labels: Vec, // "informal", "British English" } pub struct Example { pub text: String, pub translation: String } pub struct Pronunciation { pub text: String, pub notation: Notation, pub dialect: String } // "us", "uk" or "" pub enum Notation { Ipa, Respelling, Other } ``` Words looked up in the dictionaries built into macOS — the ones the Dictionary app and Look Up show. Only the dictionaries the person turned on in **Dictionary › Settings**, in their order: one they did not turn on is never opened, so a lookup downloads nothing. The word is looked up on this Mac and nothing leaves it, so no capability is needed. Needs Lumi 1.55.0. * `dictionaries` lists them. Empty when the person turned them all off; offer to look a word up only when it is not. To get another dictionary, such as Vietnamese – English, the person turns it on in the Dictionary app, which downloads it once, for every app. * `lookup` looks `text` up in the books you name, in that order, or in every one when you name none. An inflected form finds its headword ("houses" finds "house"), case does not matter, and phrases work ("look up"). A dictionary may also answer near entries ("set" finds "sett") after the exact ones. Each entry comes as fields read from the dictionary's own markup and, always, as plain text in `text`: a dictionary laid out in a way Lumi does not read still answers its headword and its text, with the other fields empty, so draw what is there. A monolingual dictionary fills `definition`, a bilingual one `translations` (and an example's `translation`), a thesaurus `synonyms` and `antonyms`. Pronunciations are on the entry when the word is said one way, and on its groups when a part of speech is said its own way. At most 8 entries a dictionary, 64 senses an entry, 3 examples a sense and 40 words a list; empty when nothing matches. Refused over 256 characters, and for a book that is not turned on. A lookup takes a few milliseconds. * `term` finds the dictionary term around character `at` of a sentence — in "Please look up the running total", at 19, `(19, 13)`: "running total" — so a caller holding a caret can look up the phrase it sits in, not only the word. * `show` brings the Dictionary app forward at `text`, with every dictionary's whole entry — or, given empty text, as the person left it, for them to turn on another dictionary in its Settings. Only from a command or one of your windows: an app coming forward is something the person does. No dictionary has audio; to say a word, speak its headword with the Web Speech API in your window. What a dictionary says is its publisher's: show its name (a `Book`'s `name`) beside what you draw from it, and keep nothing. ```rust let found = lumi::selection()?; for entry in lumi::dictionary::lookup(found.text.trim(), &[])? { let said = entry.pronunciations.first().map(|p| p.text.clone()).unwrap_or_default(); let first = entry.groups.iter().find_map(|g| g.senses.first()); let meaning = first.map(|s| s.definition.clone()).unwrap_or_default(); lumi::alert(&format!("{} /{said}/ — {meaning}", entry.headword))?; break; } ``` ### `clipboard_text()` and `set_clipboard_text(text)` [#clipboard_text-and-set_clipboard_texttext] ```rust pub fn clipboard_text() -> Result, String> pub fn set_clipboard_text(text: &str) -> Result<(), String> ``` The pasteboard's plain text, or `None` when it holds no text — an image, a file and an empty board are one answer. Writing replaces what was there, and text over 4 MiB is refused. Both need `clipboard`. ### `read_clipboard(types)` [#read_clipboardtypes] ```rust pub fn read_clipboard(types: &[&str]) -> Result>, String> ``` The pasteboard read back the way [`write_clipboard`](#write_clipboarditems-and-pasteitems) writes it: one list per item, of the representations it holds among `types` — uniform type identifiers, in the order you want them — or every type it holds, in its own order, when `types` is empty. An item holding none of them is left out. Words come back as `Data::Text`: a type that is text or a URL in macOS's type tree (`public.utf8-plain-text`, `public.html`, `public.file-url`), whose bytes are UTF-8. Everything else comes back as a `Data::Blob` written into your [storage](#storage), so the bytes never pass through your component — counted against your limits, and yours to delete. At most 160 MB in all; a board holding more is refused whole and nothing is written. Needs `clipboard` and Lumi 1.36.0. ```rust use lumi_extension_api::{read_clipboard, Data}; for item in read_clipboard(&["public.png", "public.utf8-plain-text"])? { for rep in item { match rep.data { Data::Text(text) => { /* the words */ } Data::Blob(id) => { /* the picture, in your storage */ } Data::Bytes(_) => {} } } } ``` ### `clipboard_image()` [#clipboard_image] ```rust pub fn clipboard_image() -> Result, String> pub struct Image { pub blob: String, pub width: u32, pub height: u32 } ``` The picture on the pasteboard, written into your [storage](#storage) as a PNG blob, with its size in pixels — or `None` when the board holds no picture. Whatever the board held it as, it arrives as PNG: a screenshot, a TIFF, a PDF page, a picture copied out of a browser. The bytes never pass through your component, so a page of yours opens the blob at `/__lumi__/blob/` and a blob function hands it on. It counts against your storage limits like any blob, and it is yours to delete. Needs `clipboard` and Lumi 1.36.0. ### `write_clipboard(items)` and `paste(items)` [#write_clipboarditems-and-pasteitems] ```rust pub struct Rep { pub uti: String, pub data: Data } pub enum Data { Text(String), Bytes(Vec), Blob(String) } pub fn write_clipboard(items: &[Vec]) -> Result<(), String> pub fn paste(items: &[Vec]) -> Result<(), String> ``` Put whole items on the pasteboard: each item is the same thing in several types (`public.utf8-plain-text`, `public.html`, `public.png`, …), richest first. `Data::Blob` names a blob in your own [`storage`](#storage), which goes on the board without passing through your extension. At most 160 MB in all. Lumi marks the board with your extension's id under `app.lumi.origin` — a type you cannot write yourself — so when you hear the copy you can tell it was yours. `write_clipboard` needs `clipboard`. `paste` writes the same way and then presses ⌘V into the app in front: what a clipboard history does when a row is picked. Your panels close first, so the keystroke lands where the caret was, and the board keeps what was pasted. It needs `clipboard`, `input` and `accessibility`, and answers only something the person did — a command, or a request from one of your windows. From `on_event` or `on_lifecycle` it is refused. A plain-text paste is the same call with only the text type in the item. ### `drag(items, close_on_drop)` [#dragitems-close_on_drop] ```rust pub fn drag(items: &[Vec], close_on_drop: bool) -> Result<(), String> ``` Drag whole items out of the window the person is pressing in, into whatever app they drop them on — Finder, a browser's upload field, a chat. Items are `write_clipboard`'s, and so is the 160 MB limit; the board the person copies with is not touched. An item with a `public.file-url` carries that file; an image with no file (`public.png` or `public.tiff`) is dragged as a PNG file and as the image — every app that takes a file gets one, Finder, a browser's upload field and an Electron app alike, and an app that takes images gets the image — named the way macOS names a screenshot; anything else goes as its types. Call it from the request your page sends when a press on an item starts to move. It is refused unless one of your windows is up and key and the mouse button is still down in it — an extension cannot start a drag nobody is making — and, like `paste`, from `on_event` and `on_lifecycle`. The call returns as the drag begins. `close_on_drop` takes your panels down after a drop that took the items; a drag given up never closes anything. While the drag is up your panel stays open even as the drop target takes the keyboard. Needs `clipboard` and Lumi 1.26.0; say so with `min-lumi-version`. Before Lumi 1.34 the image was a file only to an app that reads a file promise, such as Finder. ### `drag_named(items, names, close_on_drop)` [#drag_nameditems-names-close_on_drop] ```rust pub fn drag_named(items: &[Vec], names: &[Option], close_on_drop: bool) -> Result<(), String> ``` `drag`, with the files named: `names[i]` is what item `i` is called where it becomes a file, without the extension — the file is a PNG. `{date}` and `{time}` are filled in as `blob_save_dated` fills them, so a screenshot app passes its own file-name setting and a picture dragged out lands under the name it would have been saved with. A `/` or `:` becomes `-`, control characters and a leading dot go, and the name is cut to 120 characters. `None`, an empty name or an item past the end of `names` gets the default name; an item with a file URL keeps its file's own. More names than items is refused. Everything else is `drag`'s. Needs Lumi 1.34.0. ### `open_url(url)` [#open_urlurl] ```rust pub fn open_url(url: &str) -> Result<(), String> ``` Hands a URL to whichever application handles it. Web and mail addresses only: `https://`, `http://` and `mailto:`, in any case. What it costs depends on who is opening it. **From a press** — a command, a menu bar row, a link in one of your windows — the person is opening it, as they would by clicking it anywhere else, so it costs no capability, only the call. **Anywhere else** — `on_event`, `on_lifecycle`, a flow node — it needs `applications`, and a web address needs `network` too. ### `reveal(target)` [#revealtarget] ```rust pub fn reveal(target: &str) -> Result<(), String> ``` Selects one file in Finder and brings Finder forward — "Show in Finder" for a file your extension knows about. `target` is a `file:` URL, Finder's `file:///.file/id=…` form included, which is what a copied file's `public.file-url` carries, or an absolute path. No capability: it reads nothing and changes nothing. Only for a press — a command or one of your windows — since Finder coming forward on its own would be a window jumping out at somebody. ## The network [#the-network] ### `fetch(request)` [#fetchrequest] ```rust pub fn fetch(request: &Request) -> Result pub struct Request { pub method: String, // empty means GET pub url: String, pub headers: Vec<(String, String)>, pub body: Option, } pub struct Response { pub status: u16, pub ok: bool, // status in 200..300 pub url: String, // the final URL, after redirects pub headers: Vec<(String, String)>, pub body: String, } ``` One HTTP request, with a ten-second timeout. Needs `network`. The method is upper-cased, and empty means `GET`. * **Redirects** are followed, five at most. * **The body** of a response is at most 5 MiB; a larger one is an error, not a truncated body. * **Local addresses** — loopback, the local network, link-local — are refused unless the person has switched on **Let flows reach the local network** under Settings → General → Advanced, whatever the request asks for. The address is resolved once and judged, and every redirect is judged again, so a redirect cannot reach what the first address could not. The switch is named for flows because it is the same one: an extension's `fetch` and a flow's script reach the network through the same door. ## AI [#ai] ### `ai` [#ai-1] ```rust pub mod ai { pub fn ready() -> Result; pub fn prepare(language: Option<&str>) -> Result; pub fn models() -> Result, String>; pub fn complete(request: &Request) -> Result; pub fn start(request: &Request) -> Result; pub fn cancel(id: u64) -> Result; } pub struct Model { pub id: String, // "anthropic:claude-haiku-5-5", what Request.model takes pub name: String, // "Claude Haiku 5.5" pub provider: String, // "Anthropic" pub place: Place, // OnDevice or Online pub thinks: bool, pub is_default: bool, } pub struct Request { pub system: String, // your instructions pub input: String, // the text to work on, always sent as data pub model: Option, // None: the person's default pub max_tokens: Option, // fewer than the person allows, never more pub thinking: Thinking, // Off, Low, Medium, High pub answer_language: Option, // "same", "vi", "en-GB", or None pub format: Format, // Text or Json } pub struct Answer { pub text: String, pub model: String, // "provider:model" that answered pub model_name: String, pub place: Place, pub fell_back: Option, // why another model ran, for the person pub thought: bool, pub cut_short: bool, pub usage: Usage, // input, output and thinking tokens, where reported pub ms: u64, } pub struct Failure { pub message: String, // a sentence for the person pub kind: FailureKind, // Declined, Stopped, Warning, Fault } ``` The AI the person set up in Lumi: Apple Intelligence, the models Lumi downloads and runs itself (Lumi Models), the models on their Mac (Ollama, LM Studio), Claude Code on their own Claude plan, and providers on their own keys. The person chooses the models under **Settings → AI** ([AI](/features/ai) says what they see there); you choose among them, and never see a key. Costs `ai`. Needs Lumi 1.52.0. * `models` lists the models your extension may ask for now, the person's default first: the ones they switched on, that their copy of Lumi may use. Offer a model picker for each of your features from it, keep the chosen `id` in your settings, and name none to run on the person's default. Read it again when you offer the picker: the person switches models on and off. * `ready` says whether a model answers now — Lumi asks the engines on this Mac — so a button that would fail can say why before it is pressed. * `prepare` makes one ready, from a press. Where a model answers already it returns `true` at once and asks nothing. Where none does, Lumi offers the person the text model it would download for `language` (`Some("vi")`, or `None` for any) in its own panel at the pointer — the model's name, size and licence — and returns once it is installed, or once they said Not Now or stopped the download: `true` when a model answers then. It never downloads without asking, and never offers a model the person has here and switched off — that refusal names the switch. Needs Lumi 1.64.0. * `complete` asks and waits for the whole answer. Use it from a command, which has no window to stream into. * `start` asks and answers at once with the call's id; the answer streams to your own windows as [`lumi:ai` events](/extensions/windows#when-an-ai-answer-comes). * Both answer a press only — a command, a menu bar row or one of your windows. From `on_event`, a lifecycle hook or a flow node they are refused: nobody asked for that text to go anywhere. * `cancel` stops one of your calls wherever it is. **What the model is sent.** `system` as its instructions and `input` as the text to work on, always as data: Lumi tells the model the text is something to work on, not instructions, so a selection that says "ignore your instructions" is just text. Nothing else of the person's goes with it, and the model is given no tools. `answer_language: Some("same")` has Lumi detect the input's language and name it to the model, which helps every model answer in it. **The person decides where text goes.** Text that would leave the Mac — a provider, or Claude Code — waits on the person: Lumi asks at the pointer, naming your extension and where the text would go, unless they told it not to ask again for your extension. They can send it, have a model on their Mac answer instead (the answer's `place` and `fell_back` say so), or decline: a `Declined` failure, which is theirs and nothing to show. A panel of yours beside the selection stays up while they answer. They can switch your extension's AI off under Settings → AI: then `ready` is false, `models` is empty, and every call is a `Warning`. **Their limits and plan hold.** The longest input, the longest answer — `max_tokens` may ask for fewer, never more — the seconds of silence before a call is given up, and how many calls your extension may have going at once. A model that cannot run now — switched off, gone, or on a provider the person's copy of Lumi does not include — falls back to one that can, and `fell_back` says why in a sentence you can show as it is. Show a `Warning` or `Fault` failure's message; `Declined` and `Stopped` are nothing to show. ```rust let found = lumi::selection()?; let answer = lumi::ai::complete(&lumi::ai::Request { system: "Summarize the text in two sentences.".into(), input: found.text, model: None, max_tokens: None, thinking: lumi::ai::Thinking::Off, answer_language: Some("same".into()), format: lumi::ai::Format::Text, }); match answer { Ok(answer) => lumi::alert(&answer.text)?, Err(failure) if matches!(failure.kind, lumi::ai::FailureKind::Declined | lumi::ai::FailureKind::Stopped) => {} Err(failure) => lumi::alert(&failure.message)?, } ``` ### `transcribe` [#transcribe] ```rust pub mod transcribe { pub fn status(language: &str, speakers: Speakers) -> Result; pub fn models() -> Result, String>; pub fn prepare(language: &str, speakers: Speakers) -> Result; pub fn status_for(languages: &[&str], speakers: Speakers) -> Result; // Lumi 1.65.0 pub fn prepare_for(languages: &[&str], speakers: Speakers) -> Result; // Lumi 1.65.0 pub fn complete(request: &Request) -> Result; pub fn start(request: &Request) -> Result; pub fn start_after(request: &Request, previous: u64) -> Result; // Lumi 1.65.0 pub fn stop(id: u64) -> Result; pub fn cancel(id: u64) -> Result; pub fn switch(id: u64, model: Option<&str>) -> Result; // Lumi 1.65.0 } pub enum Source { Microphone, // costs `microphone` too SystemAudio(Playing), // what this Mac plays; costs `system-audio` too Call(Playing), // both at once, as two streams; costs both Blob(String), // a recording in your storage, by its blob id } pub enum Playing { Everything, OneApp } // OneApp: the person picks, in Lumi pub enum Speakers { Off, Apart, Named } // Apart: each segment says which voice; Named: and who, costs `voices` pub struct Request { pub source: Source, pub languages: Vec, // "vi", "en-GB", most likely first; empty for any; at most 8 pub model: Option, // a Model.id; None: the person's default pub speakers: Speakers, pub hints: Vec, // names and words the speech is likely to hold pub partials: bool, // text while it is said (live sources) pub words: bool, // each word's times, where the model gives them pub max_seconds: Option, // shorter than the person's longest recording, never longer pub levels: Option, // how loud it is, for drawing it; costs `audio-levels` } pub struct Levels { pub bands: u8, // frequency bands with each reading of a live source, 0 to 16; 0: loudness alone pub overview: u32, // a recording from storage: the points its loudness is summed into, 0 to 4096 } pub struct Segment { pub id: u64, pub revision: u32, // said again under its id while partial, then as its final pub text: String, pub start_ms: u64, // from the start of the recording pub end_ms: u64, pub partial: bool, pub heard: Heard, // Microphone, SystemAudio or Recording pub language: Option, pub speaker: Option, // with Speakers::Apart or Named: which voice said it pub words: Vec, pub confidence: Option, } pub struct Transcript { pub id: u64, // the recording's id: what voices::remember names pub text: String, // every final's text, in time order pub segments: Vec, pub model: String, // "provider:model" that heard it pub model_name: String, pub place: Place, // OnDevice or Online pub fell_back: Option, // why another model ran, or that it does not hear the language spoken, for the person pub speakers_model: Option, // "provider:model" that told the voices apart pub duration_ms: u64, pub cut_short: bool, // the longest recording ended it pub notes: Vec, // sentences for the person: a stream that heard nothing at all pub ms: u64, pub overview: Vec, // with Levels: a recording's loudness, each point 0 to 100 } pub struct Failure { pub message: String, // a sentence for the person pub kind: FailureKind, // Declined, Stopped, Warning, Fault, Unprepared, NotPermitted, Busy, Unsupported } ``` Speech to text: what is said into the microphone, what this Mac plays, both at once in a call, or a recording in your storage, heard by the speech model the person chose under **Settings → AI** — on their Mac (Apple Speech on macOS 26, or a model they downloaded in Lumi Models), or online on their own key (OpenAI, Groq). You get text, and never the sound. Costs `speech`; hearing the microphone costs `microphone` too, what the Mac plays `system-audio`, and a call both — declared only beside `speech` or `keep-audio` — and an extension declaring `text-input` may not declare `speech`. To keep the audio itself, as a file, see [`recordings`](#recordings). Needs Lumi 1.64.0. * `status` says how ready the Mac is to hear `language` (`""` for any): `Installed` when a model runs now, `Downloadable` when Lumi can download one, `Unsupported` otherwise. With `Speakers::Apart` or `Named` it is the less ready of the speech model and the model that tells voices apart. An error is a sentence: AI switched off, your extension's AI switched off, or the model the person downloaded switched off in Lumi Models. * `prepare` has Lumi offer the person the model it would pick for `language`, in its own panel at the pointer — a model in Lumi Models with its name, size, source and licence, or Apple Speech's language from macOS when Apple Speech is the person's choice or nothing in Lumi Models hears it — then, with `Speakers::Apart` or `Named`, the model that tells voices apart, and answers once they are installed, or they said not now, with how ready the Mac is then. Only from a press. * `status_for` and `prepare_for` (Lumi 1.65.0) are `status` and `prepare` for a recording heard among several languages — ask with the list your `Request` will carry. Several `languages` are heard among them: each utterance in the one it was said in, where the model tells them apart — Lumi Models' own, and an online model that takes several — while an online model that takes one language is given none and tells for itself. Apple Speech hears one language at a time, so it is passed over for several, and counts only for one in `status_for`. That is why asking `status` once per language is no answer: each could say installed with only Apple Speech here. * `models` lists the speech models your extension may ask for, the person's default first, each with the languages its list says it hears well — none for Apple Speech, whose languages `status` answers one at a time, nor for a model online, which hears most — where it is (`place`), and whether it gives `partials`: a model online without them gives each segment when its pause comes. * `complete` hears a recording in your storage (`Source::Blob`) and waits for the whole transcript. Use it from a command, which has no window to stream into. A live source has no end to wait for, and is refused. * `start` begins hearing any source and answers at once with the recording's id; what is heard streams to your own windows as [`lumi:transcribe` events](/extensions/windows#when-speech-is-heard) — each segment said again as it grows (`partial`) and once more as its final, under one id. A file streams the same way, as it is read. * `start_after` (Lumi 1.65.0) is `start` going on from the voices of `previous`, an earlier recording of yours: a voice heard there keeps its `label`, and a new one is numbered after them, so a meeting you hear as several recordings — after the longest recording, a pause, the Mac asleep — labels its people alike in all of them. Lumi keeps a recording's voices for ten minutes after it ends (stopped, cut at the longest recording, or the Mac gone away; never a cancel), while your pages are open, and only for a live source with speakers told apart by the same speakers model; otherwise it starts as `start` does. Your pages' `started` event says `continues` with `previous`'s id when it went on, which is when to carry your own map of labels over. * All three answer a press only — a command, a menu bar row or one of your windows. * `stop` ends a recording and still hears what was said up to it; `cancel` throws it away. * `switch` (Lumi 1.65.0) goes on hearing a live recording with another model — an id from `models`, or the person's default for `None` — without stopping it, so nothing said is missed: the model hearing now goes on while Lumi loads the new one, and asks the person first where it hears online; then every utterance after is the new model's, and the one being said finishes on the model before. Your windows hear `switched`. It answers the model hearing now (at once for the model it already has); refused, the model before goes on — `Unprepared` for one that cannot hear now (offer `prepare`), `Declined` where the person kept the voice on this Mac. Only from a press, and only for a recording of yours that is live and going. Stopping and starting again instead loses what is said in between. **One live recording at a time** for your extension: a second `start` of a live source while one runs is `Busy`. Several extensions may record at once; Lumi hears the source once and tells each its own segments. **Lumi asks the person, never you.** Where the model is online, Lumi asks once per recording, before any of it is heard, whether what is said may leave the Mac — answered apart from your extension's text, and kept as the person chooses; your pages hear `asking` while it is up. `Declined` is their Don't Send. `Playing::OneApp` has Lumi ask the person which app to hear, in its own panel, the apps playing sound first — and you are never told which they chose. The first time the microphone is used, Lumi says what it is for before macOS asks. A `NotPermitted` failure's message names where in System Settings to allow it. **It ends for every reason.** `stop` and `cancel`; the person's longest recording (`cut_short`); the person pressing Stop beside your extension's name in the menu bar, which shows every recording while it runs; your extension switched off or uninstalled; AI switched off; no window of yours left to hear it — a `keep-alive` panel running unseen still counts; or the Mac going to sleep or its screen locking. A recording ended by `stop`, the longest recording or the Mac going away is still heard to its end, and a note says why; the rest are a failure. **Voices told apart** (`Speakers::Apart`): each segment's `speaker` says which voice said it — `label`, from 0, one voice for the whole recording, numbered as voices are first heard; a call's two sides are never the same voice. A file is told apart once it is heard, so each final comes again with its `speaker` and a higher `revision`; live, each utterance's voice is found when it ends, and a final told before that comes again the same way. While an utterance is still being said its partials carry a guess (Lumi 1.65.0): from two seconds into it, measured again as it goes, the voice what was heard of it so far is — a voice nobody has spoken in yet guessed the `label` it will be numbered, which another voice heard first may take, and with `Speakers::Named` a remembered voice by its name. Only the final says who it was. An utterance too short to tell may have no `speaker`. With `Speakers::Named` (`voices` too), a voice your extension [remembered](#voices) is named where it speaks: `voice` is its id, `name` its name and `confidence` how alike it is — given only where it is clearly that person and nobody else you remember, and once given, kept for the recording; a final named after it was told comes again the same way. The model is the person's, chosen under **Settings → AI**, and always runs on their Mac; `speakers_model` names it. **How loud, for drawing it** (`levels`, costs `audio-levels`, declared beside `speech`). A waveform, a spectrum or a shape that swells with the voice needs how loud the audio is, not the audio: ask with `levels`, and your windows hear a [`level` event](/extensions/windows#when-speech-is-heard) for each stream about every 128 ms — its `level`, 0 for silence to 100 for full scale, and `bands`, as many as you asked for up to 16, lowest pitch first, each 0 to 15. The bands are eased over a quarter of a second, so they move smoothly and a little behind the `level`, which is not. Draw at your own frame rate by easing toward the latest reading. A call's two sides are two streams of readings, told apart by `heard`. A recording from storage is summed into `overview` points instead — each 0 to 100, told as an `overview` event once it is read and kept on the transcript — for a picture of the whole of it; a point is never shorter than a reading, so a short recording has fewer points than asked. Lumi bounds all of it whatever you ask, so it can never be the sound — measured: speech made back from the readings is heard as no words and as nobody's voice — and more than 16 bands or 4096 points is refused by name. **What the model is sent.** The sound, your `hints` and your `languages`, and nothing else of the person's — on the Mac, or to the provider the person set up and agreed to. Show a `Warning`, `Fault`, `NotPermitted` or `Unsupported` failure's message; for `Unprepared`, offer `prepare`; `Declined` and `Stopped` are nothing to show. A stream that heard nothing at all — exact silence, which is also what a refused permission records — is a sentence in `notes`. ```rust use lumi::transcribe::{self, FailureKind, Source, Speakers}; // A recording the person picked, heard whole from a command. let picked = lumi::files::open(&["public.audio"], false, "Choose a recording to turn into text.")?; let Some(file) = picked.into_iter().next() else { return Ok(String::new()) }; match transcribe::complete(&transcribe::Request { source: Source::Blob(file.blob), languages: vec!["vi".into()], model: None, speakers: Speakers::Off, hints: vec![], partials: false, words: false, max_seconds: None, levels: None, }) { Ok(transcript) => lumi::alert(&transcript.text)?, Err(failure) if failure.kind == FailureKind::Unprepared => { transcribe::prepare("vi", Speakers::Off)?; } Err(failure) if matches!(failure.kind, FailureKind::Declined | FailureKind::Stopped) => {} Err(failure) => lumi::alert(&failure.message)?, } ``` ### `voices` [#voices] ```rust pub mod voices { pub fn list() -> Result, String>; pub fn remember(recording: u64, label: u32, name: &str) -> Result; pub fn held(recording: u64) -> Result, String>; // Lumi 1.65.0 pub fn rename(id: &str, name: &str) -> Result; pub fn forget(id: &str) -> Result; } pub struct HeldVoice { pub label: u32, // the speaker's label in the recording pub ms: u32, // how much of the voice Lumi keeps, at most ten seconds pub enough: bool, // enough to remember } pub struct Voice { pub id: String, // what a segment's speaker.voice names pub name: String, pub remembered_at: u64, // milliseconds since 1970 pub recognized_at: Option, // when a recording last named it } pub enum Answer { Remembered(Voice), // the person agreed Already(Voice), // a voice you already keep: nobody was asked Cancelled, // the person said no } ``` The people your recordings name by their voice. Lumi keeps each one — a print of how the person sounds and a few seconds of them speaking, encrypted, on this Mac only — and you get its id and name, never the sound or the print. No other extension recognizes a voice you remember, and the person can rename or forget any of them under **Settings → AI → Voices**, whether your extension is switched on or not; your windows hear [`lumi:voices`](/extensions/windows#when-your-voices-change) when that happens. Costs `voices`, declared beside `speech`. Needs Lumi 1.64.0. * `remember` asks to remember voice `label` of recording `recording` — the id `transcribe::start` answered, or a transcript's `id` — as `name`. Lumi asks the person in its own panel at the pointer, with the voice's seconds to play first, and answers with what they chose. It needs at least five seconds of that voice, from a recording that told speakers apart, while Lumi still keeps what it heard: as it records, and for ten minutes after while your windows are open. A voice you already keep is `Already`, and nobody is asked. An extension keeps at most 64 voices. Only from a press. * `held` (Lumi 1.65.0) says which voices of `recording` `remember` could take now, with how much of each Lumi keeps and whether that is `enough` — ask it before offering to save a voice, so the offer is off where Lumi no longer has the voice (ten minutes after the recording, once your windows closed, after Lumi restarted) or has too little of it yet. Labels and lengths only, never the sound. * `rename` gives a voice another name, at most 64 characters, one line. Only from a press. * `forget` lets go of a voice — its print, its seconds and its name — at any time. Segments already told keep the name they were told with. * `list` is every voice you keep, in the order they were remembered. Name somebody only when they know: Lumi's panel says so to the person too. ```rust use lumi::voices::{self, Answer}; // From a press beside "Speaker 2" of the recording `id` your window is showing. match voices::remember(id, 1, &typed_name)? { Answer::Remembered(voice) | Answer::Already(voice) => lumi::post("transcribe", &named(voice))?, Answer::Cancelled => {} } ``` ### `recordings` [#recordings] ```rust pub mod recordings { pub fn start(request: &Request) -> Result; pub fn stop(id: &str) -> Result; pub fn cancel(id: &str) -> Result; pub fn keep(recording: u64) -> Result; } pub struct Kept { pub audio: String, // every stream mixed into one channel pub sides: Option, // a call's sides apart: the microphone in the first channel, the Mac in the second } pub struct Request { pub source: Source, // Microphone, SystemAudio(Playing) or Call(Playing) pub max_seconds: Option, // None: until it is stopped; never past the person's longest recording pub levels: Option, // bands of how loud, told as it is heard; None: nothing of it } ``` A recording made into a file of your own. Lumi hears a source, compresses it, and when the recording ends writes it among your blobs as an M4A file, under the id it answered when the recording began — so you can keep that id with whatever the recording belongs to before anything can go wrong. From then on it is an ordinary [blob](#storage): read it with `storage::blob_read`, play it in a page from `/__lumi__/blob/` (Lumi answers ranges and says it is `audio/mp4`, so an `