Your Mac

While the Sensei desktop app is open and signed in, your Mac offers Sensei a set of tools: screenshots, the accessibility tree of a window, clicks and typing in a window without bringing it forward, shell commands, files, and a headless Google Chrome signed in with your cookies. Sensei reaches them through these routes, from a Sensei Turn. Your own session or API key reads whether your Mac is online and what it offers, and is refused the tools with 403 FORBIDDEN.

Your Mac never opens a port. The desktop app holds one outbound connection to the platform, and the platform passes each call along it. Every call is recorded with its tool, what it touched and how it ended.

• A pinned Mac never falls back to another device. Only that Mac returning resumes its waiting browser work.
• A Mac with no connection answers 409 DEVICE_OFFLINE at once.
• A call cut off because the connection dropped while it ran, usually a Sensei restart, answers 409 DEVICE_CALL_INTERRUPTED. Your Mac reconnects within seconds. The call may have run, so check its effect before retrying it once.
• One Sensei Turn uses your Mac's screen, shell and files at a time. A call from another Turn answers 409 MAC_BUSY until the first one ends. The browser is per thread instead: see A browser per thread.
• An error the Mac reports answers 422 with its own code: PERMISSION_MISSING, NOT_FOUND, REF_STALE, REFUSED_BACKGROUND, TIMEOUT, NO_CHROME, UPDATE_REQUIRED, BROWSER_LIMIT, STALE_TURN, PROFILE_DISCONNECTED, SIGN_IN_TIMEOUT, KEYCHAIN_UNAVAILABLE, BAD_ARGS, TOO_LARGE or INTERNAL.
• A result too large for the connection, such as a screenshot or a big file, comes back as file.url, a link valid for an hour. Without file storage on the server, such a result answers 422 TOO_LARGE instead.
• A tool that needs a macOS permission the app does not hold, Accessibility or Screen Recording, is left out of the list. So are browser tools on a Mac without Google Chrome.

Choose a Chrome profile

Sensei first runs kosuke mac browser profiles list. Discovery lists stable directory IDs, display names, and the default choice, plus the Mac’s ID and name, and on every profile its access: allowed, or off for a profile you turned off. It opens no browser and reads no history.

For a site that may need a sign-in, Sensei adds --site <host>. Discovery then names the site and whether the app watches it, and gives every allowed profile a chromeSignIn: candidate when that profile in your Chrome holds an unexpired cookie for the site (for LinkedIn and X, their sign-in cookie), none when it holds none, and unknown when the profile could not be read. It reads only whether those cookies are there, never their values, and never asks for the Keychain. A profile that is off gets no chromeSignIn, and Google's sites answer BAD_ARGS. A candidate is not a working sign-in: Sensei checks the account by browsing. Desktop apps that predate the site-aware discovery answer without access and chromeSignIn.

Sensei keeps the same Mac and profile for each browsing task, and reads a download back from that Mac with kosuke mac files get --device-id <device-id>.

kosuke mac browser profiles list --device-id <device-id>
kosuke mac browser profiles list --device-id <device-id> --site linkedin.com
kosuke mac browser run --device-id <device-id> --profile 'Profile 5' --cmd 'open https://example.com' --cmd 'snapshot -i'

--profile accepts a directory ID or an exact, unambiguous name, ignoring case. Duplicate names require a directory ID. An unknown or removed profile fails with NOT_FOUND; invalid or ambiguous selections fail with BAD_ARGS.

Each profile has separate cookies and local storage. Your everyday Chrome stays open.

Each thread's browser works on its own copy of the profile, made the first time the thread browses with it. The copy keeps its sign-ins from then on, across Turns and restarts of the desktop app: signing in or out, or switching accounts, in your everyday Chrome does not change it, until a sign-in you tap for or Update sign-ins from Chrome takes from it. The app leaves your Google sign-in out of every copy, so the Mac browser is signed out of Google. X and LinkedIn need desktop app 0.1.23 or later; an older app answers 422 UPDATE_REQUIRED. A thread that browsed before you updated keeps its earlier copy, which is signed out of X and LinkedIn. Successful browser results identify profile.id, profile.name, device.id, and device.name. Discovery marks its default with default and returns defaultProfileId.

In Settings, Devices, Chrome sign-ins lists every Chrome profile of the Mac. A profile is Allowed unless you turned it off. A profile you turned off answers 422 PROFILE_DISCONNECTED, and the app deletes Sensei's copies of it on the Mac; your everyday Chrome stays signed in. Only you change this, in Settings: Sensei and an API key cannot.

Sign in again from Chrome

When a site shows Sensei's browser a sign-in page, turns it away, or shows another account than the expected one, Sensei first picks up that site's sign-in from your Chrome by itself, with no card:

kosuke mac browser login --device-id <device-id> --profile 'Profile 6' --site linkedin.com --reuse

• It never opens your Chrome and never waits for you. It answers chrome: none when that profile in your Chrome holds no sign-in for the site, same when Sensei's browser already holds the same one, and different when it took your Chrome's sign-in into the thread's browser, with taken: true. On none and same nothing changes.
• It runs only in a Turn your own message started, at most once per site and profile in a Turn, on an allowed profile named by its directory ID; a name answers BAD_ARGS. In any other Turn, or a second time, it answers 403 SIGN_IN_NEEDS_TAP, and on a desktop app that predates the pickup, which refuses its mode, 422 UPDATE_REQUIRED.
• Sensei then opens the site again and checks the account before going on.

Only when that cannot give access to a site that asks you to sign in does Sensei pause that work and put a card in its thread. Your tap on Sign in with Chrome allows one call:

kosuke mac browser login --device-id <device-id> --profile 'Profile 6' --site linkedin.com

• The Mac takes the site's sign-in from that profile in your everyday Chrome at once, without opening anything, when Chrome holds one that Sensei's browser lacks and that it did not take before, and answers taken: true and opened: false. Otherwise it opens the site's sign-in page, or for a site other than LinkedIn and X the site or --url, in your everyday Chrome, and watches for up to 8 minutes for you to sign in.
• On LinkedIn and X it waits for their own sign-in cookie, takes it and answers signedIn: true. On any other site it takes the site's cookies once they change and then stay unchanged for 60 seconds, so a sign-in in several steps is taken once its last step is done, and answers taken: true with signedIn: false: a change is not a sign-in, so Sensei opens the site in its own browser and checks the account. On Apple (apple.com, App Store Connect included) it also waits for Apple's own sign-in cookie, which lands only after the password and the code. A sign-in cookie that lasts only while your Chrome runs, as Apple's does, is kept in Sensei's browser for 24 hours, so it survives that browser's next start. When nothing changes, it answers taken: false. A desktop app too old to watch the site opens it and answers at once, with no taken.
• Every answer with taken: true carries waitNewLeft. While the site is still signed out in Sensei's browser and waitNewLeft is above 0, Sensei calls again with --wait-new, and the same --url, with no new tap. That call waits for a newer sign-in, never takes the one taken already, and opens the site in your Chrome only when this Turn has not, so you never get a second tab. A tap allows 6 of them, on the same Mac, within 9 minutes of the tap. After that, a --wait-new answers 422 SIGN_IN_WINDOW_ENDED, and Sensei puts the I'm signed in card instead.
• Your tap on I'm signed in, on any site, allows --take, which takes that site's cookies, never Google's. Sensei then checks the account the same way.
• --forget removes a sign-in the same Turn took, a pickup by itself included, when it is another account than the one expected. It needs no tap.
• The sign-in lands only in the thread's own copy of the profile. Google's sites are refused with BAD_ARGS.
• A call other than a pickup by itself without a tap, a second call on the same tap, or a --wait-new no take allowed, answers 403 SIGN_IN_NEEDS_TAP. Stopping the Turn ends the wait, which answers STALE_TURN; 8 minutes with no new sign-in to LinkedIn or X answer SIGN_IN_TIMEOUT, and a refused Keychain prompt KEYCHAIN_UNAVAILABLE. On SIGN_IN_TIMEOUT, KEYCHAIN_UNAVAILABLE, TIMEOUT, DEVICE_OFFLINE and DEVICE_CALL_INTERRUPTED the card comes back by itself.

A browser per thread

On an updated desktop app, each Sensei Thread has its own browser, so threads browse at once, and an Event Turn gets one of its own. Two calls of the same thread run one after the other in that browser. The browser closes when the Turn ends, so the thread's next Turn starts on a fresh page; the profile's sign-in carries over. The Mac also closes browsers idle for 15 minutes and closes them all when the app quits.

The Mac keeps between 2 and 12 browsers open, depending on its memory and free disk. Past that, it closes the least recently used idle one; with none idle, a call answers 422 BROWSER_LIMIT. A delayed call from an earlier Turn of a thread whose next Turn already has the browser answers 422 STALE_TURN.

Older desktop apps keep one browser per Chrome profile, which one Turn at a time uses: a browser call from another Turn answers 409 MAC_BUSY.

A Turn's browser recordings show in the chat in one place, in the Turn's thread, or in the channel for an Event Turn. While the Turn runs, the place reads "Recording, N steps", counting the browser runs recorded so far. When the Turn ends, all of its recordings, steers included, become one video, cut as QA cuts its videos, and it plays in the same player as a QA Report's recording. It keeps only the actions, each with a moment before and after. A drawn cursor moves to each element, with a ripple on clicks, and a caption bar names each step. A caption never shows text typed into the page. If the cut fails, or a recording comes from an older desktop app, the separate recordings play in that player instead, one after another. The result's recording.url is the run's own lasting link, which opens only for you, for 30 days, and keeps working after the cut. A run whose upload failed has no recording.

Chat Sessions wait in line

A Chat Session uses a device only after an approver allowed it. One session at a time holds a device's screen, from its first call until it lets the device go or its Turn ends.

• A call to a device another session or Sensei holds answers 409 MAC_BUSY, and the session joins the device's line, once. details.position is its place in the line, and details.heldBy says sensei or session.
• POST /api/devices/{deviceId}/lock (kosuke devices locks create) waits in the line until the device is the session's, for up to timeoutSeconds (300 by default, at most 600). It answers granted: true once the device is the session's, or granted: false with the current position and heldBy when the time runs out. Agents wait there instead of calling again.
• The line is first come, first served. When the holder lets the device go, the device is reserved for the first session in line for 2 minutes. Calls from anyone else, Sensei and the sessions further back included, answer MAC_BUSY until then. A session that does not claim it in time leaves the line, and the next one gets the reservation.
• DELETE /api/devices/{deviceId}/lock (kosuke devices locks delete) lets the device go as soon as the session is done with it, after QA for example, and takes the session out of the line. The end of the Turn lets it go too, and so does its lock lapsing, 2 minutes after the session's last call, or longer for a long call.
• A session whose Turn ends, fails or is stopped leaves every line it waits in. A call or wait from that Turn still on its way answers 409 TURN_ENDED and takes nothing. A later call fails sign-in, because the Turn's credential ended with it.

Limits

Only Sensei calls discovery and browser tools. Older desktop apps return UPDATE_REQUIRED for discovery or explicit selection. Update the desktop app before continuing. Calls without a selector may return no profile identity on older apps.

A preparation error stops browsing before page commands. A profile without cookies starts with its own local state. Sensei reports missing sign-in and checks the site’s account before account-specific work. Switching profiles ends the previous page state and element refs.

Method	Path	Description
POST	/api/devices/{deviceId}/lock	From a Chat Session, when a mac command answered MAC_BUSY: wait in the device’s line, first come, first served, until the device is this session’s, or the timeout passes. Answers granted true once it is yours: go on with your mac commands. Answers granted false with your position and heldBy (sensei or session) on timeout: run it again. Never call mac commands in a loop instead. Once done with the device, run devices locks delete.
DELETE	/api/devices/{deviceId}/lock	From a Chat Session: let a device go as soon as you are done with it, after QA say, rather than at the end of your Turn, and leave its line if you wait in it. The next session in line gets it at once. Your access stays: a later mac command or devices locks create takes the device again. Answers released and leftQueue.
POST	/api/devices/{deviceId}/requests	From a Chat Session: ask to use a personal Mac from devices list, with the reason its owner reads on a card in this session's chat. A workspace device needs no request: it answers approved at once with grantId null, and no one is asked. The owner approves a personal Mac for the Turn that asked: asking again in that Turn answers approved at once, and a later Turn asks again, with a new card for the owner. Answers status pending or approved. Check devices list again for the answer; it expires after 30 minutes without an answer, or when this Turn ends. While it is pending, go on with other work; when it is declined or expires, go on without it and say so. Once approved it lasts until this Turn ends: pass --device-id on every mac command; after that, mac commands on it are refused. One session at a time uses the device: on MAC_BUSY, wait with devices locks create, and run devices locks delete once done.
GET	/api/devices/available	From a Chat Session: the devices it may use, the workspace's devices and the session member's own Macs, each with its id, kind, whether it is online, the tools it offers, its macOS grants (grants.systemEvents: menu bar icons open with osascript through System Events), this session's access: none, pending or approved (a workspace device is always approved: it needs no request), and occupied with occupiedBy (session or sensei) when someone else uses it now. Pick an online device with the tools the test needs: a workspace device first, and the member's own Mac only when no workspace device fits, whatever the names or past approvals. Of those, pick a free one, and ask for an occupied one only when no free device fits. Use a workspace device at once with --device-id; request a personal Mac with kosuke devices requests create before any mac command on it. Use a device only when the test truly needs a real Mac. One session at a time uses a device: on MAC_BUSY, wait in its line with kosuke devices locks create rather than give up or call again, and let it go with kosuke devices locks delete once done.
GET	/api/devices/mac	Whether your Mac is online now, with the Sensei desktop app open and signed in, the tools it offers, and its macOS grants: a missing macOS permission or no Google Chrome leaves its tools out, and grants.systemEvents says whether osascript through System Events can open menu bar icons and their menus there. Offline answers online false and nothing else.
POST	/api/devices/mac/apps	List the apps running on the member's Mac, with each one's name, bundle id, pid and whether it is active. Sensei calls it, and a Chat Session does with --device-id for a device approved for it.
POST	/api/devices/mac/ax-tree	Read one window of the member's Mac as its accessibility tree, one line per element with a ref such as e12 for click, type and scroll. Prefer it to a screenshot. Sensei calls it, and a Chat Session does with --device-id for a device approved for it.
POST	/api/devices/mac/browser	Run agent-browser commands in order, in one headless Google Chrome session on the member's Mac signed in with their cookies, for sites that need their sign-in. On updated desktops each Sensei Thread has its own browser, so threads browse at once, and it closes when the Turn ends, so page state does not carry over to the next Turn; past the Mac's cap on open browsers, with none idle, a call answers BROWSER_LIMIT. Select a Chrome profile by directory ID or exact unambiguous name, and pin its Mac with deviceId. Updated desktops return profile ID/name; the platform returns device ID/name. Unknown or removed profiles fail without fallback, and a profile the member turned off in Settings → Devices fails with PROFILE_DISCONNECTED. Older desktops require an update for explicit selection, and share one browser per profile under the Mac lock. Answers each command's output, the recording's lasting link as recording.url (the recording is posted in the chat too), and the paths of files it downloaded to ~/Downloads. Only Sensei calls it.
GET	/api/devices/mac/browser-connections?site=	List the member's LinkedIn and X browser connections, oldest first: each is one account Sensei may act as, with its id, revision, account (null until Sensei fills it in), device (id, name, removed), profile (id, name), the last check (unchecked, connected, signed_out, other_account) with checkedAt, and connectThreadId, the only thread whose Turn may fill the account in. Pass its device id and profile id to every browser call for it. Only Sensei calls it.
POST	/api/devices/mac/browser-connections	Report what one check of the live page saw for a browser connection, with the revision kosuke mac browser connections list answered: connected with the account's name and handle, signed_out, or other_account with the name found. The first connected fills the account in, only from a Turn of its connectThreadId (else 403 NOT_CONNECT_THREAD); afterwards the handle must match it (else 409 ACCOUNT_MISMATCH). 409 STALE_REVISION after a newer Connect or Reconnect: read again. 404 once the member disconnected it. Only Sensei calls it.
POST	/api/devices/mac/browser-profiles	List Chrome profile IDs and display names on the member's Mac, including the default choice and device ID/name, with access allowed or off on every profile (off: turned off in Settings → Devices, never to be used). With --site, a bare host (Google's sites answer BAD_ARGS), it names the site and whether the platform watches it, and every allowed profile carries chromeSignIn: candidate when that profile in their Chrome holds a live cookie for the site (the sign-in cookie for LinkedIn and X), none when it holds none, unknown when it could not be read. An off profile carries none. A candidate is not a sign-in: check the account by browsing. Discovery reads cookie presence only, never a value, never the Keychain, and no history, and opens no browser. An older desktop app answers without access and chromeSignIn: read that as unknown. Only Sensei calls it.
POST	/api/devices/mac/browser-sign-in	Get a site signed in in this Sensei Thread's browser from the member's everyday Chrome: pass the browser call's profile and deviceId, and the site as a bare hostname (twitter.com is x.com; Google's sites are refused). --reuse picks up the sign-in that profile in their Chrome holds by itself, with no tap: once per site and profile in a Turn, with the profile's exact ID. It never opens Chrome or waits for anyone, and answers opened false and signedIn false with chrome none (their Chrome holds no sign-in for the site, so nothing changed), same (the one this thread's browser already holds) or different (taken into this thread's browser, taken true). Then, unless it answered none or same, open the site with browser and check the account. On SIGN_IN_NEEDS_TAP or UPDATE_REQUIRED, put the sign-in card: kosuke sensei cards set with --sign-in naming this site, profile and deviceId. A --reuse spends no tap, so it gives no card back. Every tapped mode (sign-in, take, waitNew) needs the member's tap on the sign-in card in this thread; --forget and --reuse need none. A sign-in takes the sign-in that profile in their Chrome holds when it differs from this thread's, without opening anything, and answers taken true, opened false and signedIn false, since the site has not confirmed it; or else it opens --url, or the site's page, in their Chrome and waits up to 8 minutes for the member to sign in. On LinkedIn and X it answers signedIn true and opened true. On any other site a change of the site's cookies is taken and answered taken true and signedIn false, since a change is not a sign-in, and an end with no change answers taken false; a desktop app too old to watch the site opens it and answers at once, with no taken. Every answer with taken true carries waitNewLeft: while it is above 0 and the site is still signed out, --wait-new (with the same --url), in the same Turn and with no new tap, waits for a newer sign-in, never taking the one taken already, and opens the site only when this Turn has not; once the tap's time has run out it fails with SIGN_IN_WINDOW_ENDED, and no card comes back. After the member's I'm signed in tap, --take takes that site's cookies, never Google's, and answers taken. --forget removes a sign-in this Turn took, a --reuse's included, for an account other than the one expected, and needs no tap. Each tap allows one call: a call without one, or a --wait-new that no take armed, fails with SIGN_IN_NEEDS_TAP. Then open the site with browser and check the account before account-specific work. On DEVICE_OFFLINE, DEVICE_CALL_INTERRUPTED, TIMEOUT, SIGN_IN_TIMEOUT and KEYCHAIN_UNAVAILABLE the platform gives the member's card back by itself. Fails with SIGN_IN_TIMEOUT when Sensei saw no new sign-in to LinkedIn or X within 8 minutes, PROFILE_DISCONNECTED for a profile turned off in Settings → Devices, KEYCHAIN_UNAVAILABLE when macOS did not let Sensei read Chrome's sign-ins, STALE_TURN when the Turn stopped, and UPDATE_REQUIRED on a desktop app too old for it. Only Sensei calls it.
POST	/api/devices/mac/click	Click in a window of the member's Mac, on an element ref from ax-tree or at x and y in window coordinates, without bringing the window forward. Answers the method used. Sensei calls it, and a Chat Session does with --device-id for a device approved for it.
POST	/api/devices/mac/file-delete	Move a file on the member's Mac to the Trash. Sensei calls it, and a Chat Session does with --device-id for a device approved for it.
POST	/api/devices/mac/file-list	List a folder on the member's Mac, with each entry's name, kind, size and modification time. Sensei calls it, and a Chat Session does with --device-id for a device approved for it.
POST	/api/devices/mac/file-move	Move or rename a file on the member's Mac. Sensei calls it, and a Chat Session does with --device-id for a device approved for it.
POST	/api/devices/mac/file-read	Read a file on the member's Mac: its text up to 256 KiB, or file.url for a larger or binary file. ~ is the member's home. Sensei calls it, and a Chat Session does with --device-id for a device approved for it.
POST	/api/devices/mac/file-write	Write a file on the member's Mac, from text or from a URL the Mac downloads. Sensei calls it, and a Chat Session does with --device-id for a device approved for it.
POST	/api/devices/mac/key	Press a key chord, such as cmd+shift+s, in a window of the member's Mac, without bringing the window forward. Sensei calls it, and a Chat Session does with --device-id for a device approved for it.
POST	/api/devices/mac/open	Open an app or a URL on the member's Mac. Sensei calls it, and a Chat Session does with --device-id for a device approved for it.
POST	/api/devices/mac/screenshot	Take a screenshot of the member's Mac: the main display, or one window named by app, pid or window title. Answers a PNG as file.url, with width and height. Sensei calls it, and a Chat Session does with --device-id for a device approved for it.
POST	/api/devices/mac/scroll	Scroll a window of the member's Mac by dx and dy, inside an element ref when given. Sensei calls it, and a Chat Session does with --device-id for a device approved for it.
POST	/api/devices/mac/shell	Run a shell command on the member's Mac with /bin/zsh -lc, in cwd when given, for up to timeoutMs (60000 by default, at most 600000). Answers exitCode, stdout and stderr; output over 256 KiB comes as file.url. Sensei calls it, and a Chat Session does with --device-id for a device approved for it.
POST	/api/devices/mac/type	Type text into a window of the member's Mac, into an element ref when given, without bringing the window forward. Sensei calls it, and a Chat Session does with --device-id for a device approved for it.
POST	/api/devices/mac/windows	List the windows open on the member's Mac, with each one's app, pid, title, frame and whether it is minimized. Sensei calls it, and a Chat Session does with --device-id for a device approved for it.

Next steps

• Desktop and mobile apps
• Sensei