Lumi
Extensions

SDK reference

Every function in lumi-extension-api, what it answers, and what it costs.

lumi-extension-api is the crate your extension depends on. Import it under a short name and everything below is one path away:

use lumi_extension_api as lumi;

What each call costs

Every function below exists whether or not your manifest declares what it needs. An undeclared one is refused when it is called, with an error that names the capability — so the person who sees it is you, with the manifest open.

CostsFunctions
Nothing at allabout, license
One call from the run's budgetalert, settings, setting, set_setting, profiles, transforms::all, transforms::apply, everything in lumi::translation, everything in lumi::dictionary, open_window, open_window_on, open_window_at, close_window, window_state, preload_enabled, set_preload_enabled, set_tab_label, hide_window, show_window, show_window_on, show_window_at, post, set_material, set_position, set_anchor, set_theme, set_pinned, raise_window, set_size, set_titlebar_height, hyper_key_enabled, everything in lumi::storage, everything in lumi::folders (for the folders your manifest declares), everything in lumi::files, everything in lumi::owner, config::extension_shortcuts and config::set_extension_shortcut, ocr::languages, and ocr::read of a picture in your storage. Some answer only a press — a command, a menu bar row or one of your windows: open_settings, open_permissions, switch_off, set_setting, reveal, set_pinned, files::open and files::save, owner::authenticate, translation::prepare, dictionary::show, and folders::choose, write and reveal
accessibilityselection (the ⌘C fallback needs clipboard too), selection_editable
accessibility and inputreplace_selection and replace_selection_and_select — and only from a command or one of your windows; a replacement too long to type is pasted only with clipboard too
clipboardclipboard_text, set_clipboard_text, clipboard_image, read_clipboard, write_clipboard, and drag and drag_named — those two only while the mouse button is down in one of your windows
clipboard, input and accessibilitypaste and copy_selection — and only from a command or one of your windows
clipboard-historyhearing on_event("clipboard", …) and on_event("clipboard-ocr", …), with events = [...] in the manifest
selection-watchselection_watching, set_selection_watching, selection_watchers, and hearing on_event("selection", …) and on_event("selection-watching", …), with events = [...] in the manifest
applicationsopen_url from on_event, a lifecycle hook or a flow node — nobody pressed anything (a web address needs network too). From a press it costs no capability
networkfetch
configeverything in lumi::config
screeneverything in lumi::screen, ocr::read of the screen, lumi::permissions for Screen Recording (ask, request and open_system_settings only from a command or one of your windows), and hearing on_event("screen-capture", …) with events = [...] in the manifest
snippetseverything in lumi::snippets (an expansion that reads the clipboard needs clipboard too)
menueverything in lumi::menu — your rows in Lumi's menu bar menu
menu-bar-itemeverything in lumi::menu_bar_item — an item of your own in the menu bar, beside Lumi's — and hearing on_event("menu-bar-item-shown", …) with events = [...] in the manifest (Lumi 1.61.0)
aieverything in lumi::ai — complete, start and prepare only from a command, a menu bar row or one of your windows (Lumi 1.52.0; prepare 1.64.0)
speecheverything in lumi::transcribe — prepare, prepare_for, complete, start, start_after and switch only from a command, a menu bar row or one of your windows; hearing the microphone live costs microphone too, what the Mac plays system-audio, and a call both (Lumi 1.64.0)
audio-levelsa transcribe::Request with levels, or a recordings::Request with levels (Lumi 1.65.0): how loud what is heard is, for drawing it; declared beside speech or keep-audio (Lumi 1.64.0)
voiceseverything in lumi::voices — remember and rename only from a command, a menu bar row or one of your windows — and Speakers::Named in lumi::transcribe; declared beside speech (Lumi 1.64.0)
keep-audioeverything in lumi::recordings but longest_minutes and set_longest_minutes, which need only microphone or system-audio — start and keep only from a command, a menu bar row or one of your windows; start costs its source's word too. Declared beside microphone or system-audio, with no speech needed (Lumi 1.65.0)
timerseverything in lumi::timers (Lumi 1.59.0) and lumi::schedules (Lumi 1.60.0), and hearing on_event("timer", …) and on_event("schedule", …) with events = [...] in the manifest
time-zoneeverything in lumi::time_zone, and hearing on_event("time-zone", …) with events = [...] in the manifest (Lumi 1.59.0)
microphone-useeverything in lumi::microphone_use but sites, and hearing on_event("microphone-use", …) with events = [...] in the manifest (Lumi 1.65.0)
microphone-use and microphone-sitesmicrophone_use::sites — the site an app using a microphone shows, by host (Lumi 1.65.0)
offereverything in lumi::offer — from any run, an event's or a timer's included (Lumi 1.65.0)
soundeverything in lumi::sound — from any run, an event's or a timer's included — and hearing on_event("sound-ended", …) with events = [...] in the manifest (Lumi 1.74.0)

Every call into Lumi except about and license spends from one allowance of 200 per run, so a loop of alerts runs dry long before it can flood the screen. The call is spent before its capability is checked, so a refused call counts too. Past the allowance every call answers an error — and settings(), which has no error to give, answers an empty object, so read your settings early in a run rather than after a long loop.

Your extension

Guest and register!

The trait you implement and the macro that wires it up. Call register! once, at the crate root, on the type that implements Guest.

impl lumi::Guest for MyExtension {
    fn run_command(name: String, params: String) -> Result<String, String>;
    fn run_node(name: String, params: String, items: String) -> Result<String, String>;
    fn run_ui(window: String, request: String) -> Result<String, String>;
    fn on_lifecycle(event: lumi::Lifecycle) -> Result<(), String>;
    fn on_event(name: String, payload: String) -> Result<(), String>;
}

lumi::register!(MyExtension);
MethodCalled forReceivesAnswers
run_commandA bound [[command]] pressedThe command's name; its params as a JSON object, variables already renderedAny string, which is not shown — call alert to tell the person something — or an error Lumi reports like a failed action
run_nodeA [[node]] reached in a flowThe node's name, params as a JSON object, and the incoming items as a JSON arrayThe outgoing items as a JSON array — empty ends the branch
run_uiA request from one of your windowsThe window's name, and whatever the page postedWhatever the page should get back — see Windows
on_lifecycleYour extension being installed, updated or uninstalledA LifecycleOk(()), or an error that is only written to Lumi's log
on_eventSomething outside your extension that it asked to hearThe event's name, and a JSON payloadOk(()), or an error that is only written to Lumi's log

Answer the names you do not know with an error rather than a panic. A panic is reported to the person as the extension having crashed on an error of its own, with no more detail than that; the backtrace goes to Lumi's log. An error's text is cut to 64 KiB.

on_lifecycle(event)

pub enum Lifecycle {
    Installed,
    Updated(String), // the version that was replaced
    Uninstalling,
}

Three moments in your extension's life, each told to it once:

EventWhenTypical use
InstalledRight after a first install — after your installer, if you ship one, has saved its answersOpen a Welcome window
Updated(from)Right after a new version replaces an installed oneSay what is new since from
UninstallingJust before your files are removedOpen a feedback page in the browser

Nothing to do is the ordinary answer:

fn on_lifecycle(_event: lumi::Lifecycle) -> Result<(), String> {
    Ok(())
}

The rules, all of which follow from nobody having pressed anything:

  • Only while switched on. A switched-off extension is not told it was updated or uninstalled — off means none of its code runs.
  • Nobody sees the answer. An error is written to Lumi's log, never shown.
  • It cannot stop anything. An Uninstalling that fails or runs out of time is logged, and the uninstall goes ahead.
  • Installed and Updated run after the install has finished, on their own, with the same allowance and deadline as a command.
  • It takes one of your four runs at once. A hook that arrives while four runs of yours are going is skipped and logged — Uninstalling included.
  • Uninstalling is short and has no windows. The person is waiting on the Uninstall button, so it has five seconds, and your ui/ files are about to be deleted, so open_window is refused. Open a web page with open_url instead — which needs applications, plus network for a web address.
fn on_lifecycle(event: lumi::Lifecycle) -> Result<(), String> {
    match event {
        lumi::Lifecycle::Installed => lumi::open_window("welcome"),
        lumi::Lifecycle::Updated(from) => lumi::alert(&format!("Updated from {from}")),
        lumi::Lifecycle::Uninstalling => lumi::open_url("https://example.dev/why-uninstall"),
    }
}

A Welcome window is an ordinary [[window]]: declare it, and open it from Installed.

on_event(name, payload)

Something that happened outside your extension, which it asked to hear. The payload is a JSON object with a "v" field for its own version; a new kind of event is a new name, so this method is the last one the trait gains for it. An extension that asks to hear nothing is never called, and answers Ok(()):

fn on_event(_name: String, _payload: String) -> Result<(), String> {
    Ok(())
}

The events Lumi sends today:

NamePayloadWhen
clipboard{"v":1, "at", "hash", "source":{"bundleId","name"}, "origin", "items":[[{"uti", "text" or "blob", "bytes", "path"?, "fileSize"?, "fileToken"?}]]}Each copy, less what password managers mark private. Images and large text arrive as blobs already in your storage; a copied file's public.file-url carries path, where it is — Finder copies the unreadable file:///.file/id=… form — and fileSize, its size in bytes (Lumi 1.26+; a folder has no size, a file already gone neither); a PDF, picture, text or code file, sound or film also carries fileToken, which a page gives to GET /__lumi__/file/<token> to play or show it; origin is your id when the board was your own write_clipboard or paste.
clipboard-ocr{"v":1, "hash", "text"}A moment after a copy holding an image, with the text read in it; hash names that copy. Never empty.
screen-capture{"v":1, "phase":"start" or "end", "region"}Lumi taking a picture of the screen for itself — Copy Text from Screen, a flow reading the screen — once as it starts and once as it ends, however it ends. region is "area" or "window" when the person chooses on an overlay, which leaves time to step a window aside with hide_window; "frontWindow" or "display" is taken the instant it is asked, which leaves none — a window that must never be in the picture says capturable = false. Lumi never waits on your answer. Needs the screen capability and Lumi 1.36.0.
window-closed{"v":1, "window", "at"}One of your own windows leaving the screen for good: a panel put away — Esc, a click away, your close_window, a paste taking it down — or a declared window closed, by you or by the person. window is its manifest name, at the moment it closed in milliseconds since the Unix epoch — not when you hear it, which can be later. Once per close: a panel already away says nothing. Not for hide_window, whose window comes back with show_window, and not for a pinned panel losing the keyboard, which is still up; nor for the windows that close because your extension was switched off, uninstalled or updated. Lumi never waits on your answer. Needs no capability — it is only about your own window — and Lumi 1.37.0.
preload-enabled{"v":1, "enabled", "at"}The person turning loading your panels ahead on or off with the Preload switch on your Permissions sheet — see preload_enabled(). enabled is the switch as it now stands, at the moment it moved in milliseconds since the Unix epoch. Only when the person moves it there, never for your own set_preload_enabled; preload_enabled() answers the same question any time. Lumi never waits on your answer. Needs the preload capability — declared without it, the manifest is refused — and Lumi 1.45.0.
selection{"v":1, "text", "by", "lines", "editable", "app":{"bundleId","name"}, "at", "clickInterval"}The person selecting text with the mouse in another app: by is "drag", "double-click", "triple-click" or "shift-click" — never a plain click, never with ⌘ or ⌃ held, never from the keyboard, which is keyboard-selection. text is read through Accessibility only, never by a ⌘C, and is not sent for a password field, Lumi's own windows, blank text or text over 16 KiB; a page's list numbers and bullets are left out, and since Lumi 1.59.0 a page nobody can edit reads as it shows — see selection(). A Chromium browser — Chrome, Edge — tells Accessibility nothing about a page's own text until the person turns on General → Advanced → Ask Chromium browsers for a page's text, so a selection in a page is told only then; its address bar is told either way. An editor that keeps its selection from Accessibility — Visual Studio Code with its screen reader support off — is never told; selection() from a shortcut still reads it there by ⌘C with clipboard. editable is selection_editable()'s answer for it. clickInterval is the Mac's double-click interval in milliseconds: a click within it continues the gesture, so a double-click may yet become a triple-click — open_window_after waits it out (Lumi 1.50.0). Nothing says where on the screen. Only while the Selection switch on your Permissions sheet is on — see selection_watching(). The one event whose run may open a window: one panel declared position = "selection", once, beside that selection — below. A newer selection replaces one still waiting, and one more than a second old is not delivered. Needs the selection-watch capability and Lumi 1.47.0.
keyboard-selectionselection's, with by "keyboard" and "keys"The person selecting text with the keyboard in another app — ⇧ and the arrows, ⇧⌥ by words, ⇧⌘ to the line's end, ⌘A, ⌘G — told once the selection has rested: unchanged for 300 ms and every modifier up for 150 ms, so a selection made to type over is never told. Only a change the person's last key just made, and a key that selects — ⇧ with a key that moves the caret, or anything with ⌘ or ⌃ held: a press or a drag is selection's, what an app selects as the person types (a word it completes) is never told, and what an app or Lumi itself selects is nobody's. Its own name so an extension written for the mouse is never told of the keys; hear both to be told of both. keys says what kind of key made it, never which: "extend" — ⇧ with a key that moves the caret, ⌥ or ⌘ held or not — "select-all" — ⌘A, by the letter it types on any layout — or "command" — any other key with ⌘ or ⌃ held, a command that left text selected: Find filling its field, an address bar's ⌘L, Find Next, the ⌃⇧ text bindings. An extension that would answer only the person selecting on purpose answers the first two. Read, gated and granted as selection is — its run may open the same one panel beside it — and Lumi measures the place without the pointer, which is wherever the person left it. An app that does not report its selection changing to Accessibility, as Visual Studio Code's editor does not unless its screen reader support is on, is never told. Needs the selection-watch capability and Lumi 1.50.0.
selection-watching{"v":1, "watching", "at"}The person turning the Selection switch on your Permissions sheet on or off. Only when they move it, never for your own set_selection_watching. Needs the selection-watch capability and Lumi 1.47.0.
timer{"v":1, "name", "at", "late", "missed"}One of your own timers coming due: name as you set it, at the moment it fired in milliseconds since the Unix epoch — not when you hear it, which can be later. late is true when that was more than a minute after it was due: the Mac slept, Lumi was quit or your extension was off. missed counts the further repeats that came due meanwhile, folded into this one firing. Told at least once — see Timers. timers::fired reads it. Needs the timers capability and Lumi 1.59.0.
schedule{"v":1, "name", "at", "late", "missed"}One of your own schedules coming due — timer's payload, under its own name: at the moment it fired, late more than a minute after its moment, missed the further moments folded into this firing. schedules::fired reads it. Needs the timers capability and Lumi 1.60.0.
time-zone{"v":1, "id", "offsetMinutes", "at"}This Mac's time zone changing — see Time zone. id and offsetMinutes are the new zone as time_zone::current() answers it, at the moment it changed in milliseconds since the Unix epoch. Once per change; a change of clocks for summer is not a change of zone and is not told. time_zone::changed reads it. Needs the time-zone capability and Lumi 1.59.0.
microphone-use{"v":1, "apps":[{"app", "name", "since"}], "at"}The apps using a microphone changing — see Microphone in use. The whole set each time, never a change: every app with a microphone open then, by name, app null for a process without a bundle id. Told once too, as it stands, when your extension starts hearing it. Changes within a quarter of a second are one. microphone_use::changed reads it. Needs the microphone-use capability, macOS 14 and Lumi 1.65.0.
menu-shown{"v":1, "shown", "at"}The person showing or hiding your rows in Lumi's menu bar menu with the Menu bar switch on your Permissions sheet under Extensions — see Whether your rows show. shown is the switch as it now stands, at the moment it moved in milliseconds since the Unix epoch. Only when the person moves it there, never for your own menu::set or menu::set_shown; menu::shown() answers the same question any time. Lumi never waits on your answer. Needs the menu capability and Lumi 1.44.0.
menu-bar-item-shown{"v":1, "shown", "at"}The person showing or hiding your own item in the menu bar with the Show its icon in the menu bar switch on your Permissions sheet — see Menu bar item. menu-shown's payload and rules: only when the person moves it there, never for your own menu_bar_item::set_shown; menu_bar_item::shown() answers the same question any time. Lumi never waits on your answer. Needs the menu-bar-item capability and Lumi 1.61.0.
sound-ended{"v":1, "sound", "name", "finished", "at"}One of your own sounds stopping: sound the number play answered, name the one it was played under or null, finished true when it came to its end and false when it was stopped — by you, by a sound of the same name, or by the person's Stop row — and at when. Not told when your extension is switched off or removed. Lumi never waits on your answer. Needs the sound capability and Lumi 1.74.0.

It runs under on_lifecycle's rules — only while switched on, its answer only written to Lumi's log, one of your four runs at once — and two more that follow from nobody having pressed anything: open_window cannot open a panel from here, and paste is refused.

selection and keyboard-selection are the exception, because the person selecting is the moment: its run may open_window one panel declared position = "selection", once, within two seconds of the selection and only while it is still the person's latest. The panel opens beside that selection without the keyboard, whatever its focus says, and goes at their next click, key or scroll. The run may set that panel's size, material, theme and anchor and post to it; paste, replace_selection, replace_selection_and_select, set_pinned and opening Settings are still refused.

fn on_event(name: String, payload: String) -> Result<(), String> {
    if name == "selection" {
        let said: serde_json::Value = serde_json::from_str(&payload).map_err(|e| e.to_string())?;
        lumi::post("bar", &said.to_string())?;
        return lumi::open_window("bar");
    }
    Ok(())
}

settings() and setting(name)

pub fn settings() -> serde_json::Value
pub fn setting(name: &str) -> Option<String>

Your [[settings]] values as the person saved them, one key per field, with defaults filled in for anything untouched. A field declared scope = "profile" holds the live profile's value. setting is the shortcut for one string field.

Settings are fixed for the length of a run: a save in Lumi while your code is running reaches the next run. Your own set_setting is the exception — the rest of the run reads what it saved.

set_setting(name, value)

pub fn set_setting(name: &str, value: &str) -> Result<(), String>

Saves one setting, as a save from your Settings page does — for a menu row that switches it without the person opening the page:

lumi::set_setting("method", "vni")?;

The value is text, as every setting is ("true" and "false" for a bool). It is held to the field's own rules — a value they refuse is refused with the sentence the page would show — filed under the live profile for a field declared scope = "profile", and told to your open pages. Only for something the person did: a command, a menu bar row or one of your windows. Needs Lumi 1.61.0.

open_window(name)

pub fn open_window(name: &str) -> Result<(), String>

Opens — or brings forward — one of the windows your manifest declares. A name the manifest does not declare is refused. A window with copies is opened copy by copy, open_window("card-3"), in this call and every other that names a window. One already up is focused, never closed, your command's own press included; for a key that also puts it away, see window_state below. See Windows.

It is refused from run_node too: a flow runs in the background, and a dialog out of a flow nobody just pressed is not something a flow may do yet. And it is refused from Uninstalling, since your ui/ files are about to go.

pub fn open_window_on(name: &str, beside: &str) -> Result<(), String>

open_window_on is open_window on the display your window beside stands on now. A window that opens where it was left (remember-frame) opens at that place carried to this display — as far from the same edges, the nearer of left and right and of top and bottom, held inside — and one that opens at a corner or in the middle opens at that one of this display; one that opens at the pointer or a selection opens there, as ever. One already up is focused where it stands; with beside not on screen, or on the same display, it is open_window. It is for a smaller window standing in for a bigger one, opened as the bigger one is put away — a browser minimized to a pill — which should appear on the display the person was looking at. Open it before you hide_window the bigger one, while that one is still on screen to be measured. Both are names your manifest declares (Lumi 1.65.0).

pub fn open_window_at(name: &str, beside: &str, corner: &str) -> Result<(), String>

open_window_at is open_window at a corner of your window beside as it stands now: the panel opens with its corner on that window's same corner — "top-left", "top-right", "bottom-left" or "bottom-right", or "center" for its middle on that one's middle — held inside the display beside is on, against its edge where it would run past it. A window hide_window took away counts as standing where it was, since that is where show_window puts it back. It is for a smaller window standing in for a bigger one, which should appear where that one was — a pill at the foot of the browser minimized to it:

lumi::open_window_at("pill", "browser", "bottom-right")?;
lumi::hide_window("browser")?;

A panel only — macOS places a window as it opens — and not one Lumi places beside the selection, the menu bar item or another window. One already up is focused where it stands; with beside neither on screen nor stepped aside it is open_window. Both are names your manifest declares (Lumi 1.75.0).

open_window_after(name, after_ms)

pub fn open_window_after(name: &str, after_ms: u32) -> Result<(), String>

open_window, after_ms milliseconds from now — from the run of a selection or keyboard-selection event only, as its one opening of the panel beside that selection. It answers at once; the panel opens when the time is up, unless the person has pressed, typed or scrolled since — then it never opens — or the selection is two seconds old. At most 1000; 0 is open_window. Refused from any other run.

For a panel that waits for the person to finish: a double-click may yet become a triple-click, and the selection event's clickInterval says for how long. Waited out, the double-click's panel opens; cut short by the third click, it never shows, and the triple-click's own run opens one.