Home

Deep dive 04 · Architecture

One backend, many ways in.

The system keeps capture flexible while preserving one identity, permission, storage, and retrieval model.

AgentUX core

System map

Every client reaches the same source of truth.

Ten edge functions handle focused jobs around a shared Supabase core. New clients reuse the same product model instead of creating a parallel library.

system-architecture.tsx
01

Capture + access

02

Edge layer

03

Shared core

04

Product surfaces

Shared coreSupabase

References, labels, people, permissions, originals, and previews stay in one source of truth.

Clients reuse focused edge functions around one permission, storage, and retrieval model.

Surfaces

Ten ways in, and not all of them are shipped.

Built and shipped are different claims, so the status column says which. The extension works and is used; it has never been published to a store.

  • Web appLive

    The main surface: browse, capture, label, share. Seven routes.

  • Telegram botLive

    Send a screenshot or a link from your phone and it lands in the catalogue without opening the app.

  • Browser extensionBuilt, sideloaded

    Right-click any image to save it while you are still reading the page. Chromium and Safari builds exist; neither is in a store.

  • iOS appTestFlight

    Native client at V0.1, distributed through TestFlight.

  • Folder uploadLive

    Drop a whole directory in. A single dropped folder name pre-fills the flow it belongs to.

  • iOS ShortcutLive, dormant client

    Share-sheet capture. The endpoint stays live for shortcuts already in use; the native app replaced its setup screen.

  • REST APILive

    Read access for products and scripts. Search, or fetch one reference.

  • MCP serverLive

    Eleven tools for agents. Nine read, two write, and both writes land in a human queue.

  • Read-only share linksLive

    Send a focused collection to someone without an account, instead of a folder of screenshots.

  • Prototype hostingLive

    Upload an HTML prototype and it serves on its own subdomain.

Edge layer

Ten functions, each with one job.

Grouped by what they are for. The three metadata functions run after the fact, on first render rather than at capture, so a slow third-party lookup never blocks someone saving a reference.

Getting references in

  • telegram-botsync

    Routes an inbound message: an image becomes a screenshot, an X status becomes a video reference, any other URL becomes a link.

  • inbox-ingestsync

    Accepts one image from an agent and lands it unlabelled in a holding group for a person to triage.

  • shortcut-uploadsync

    The same routing as the bot, for the iOS share sheet.

Labelling

  • label-ingestsync

    The only surface that writes a label. Lists the oldest unlabelled screens, and saves one label. It will not overwrite a label that arrived while it was reading.

Access

  • auth-loginsync

    Validates an email and passcode, issues a session. Repeated failures lock out temporarily.

  • auth-adminsync

    Members, roles, and agent session tokens: mint, rotate, disable, force-logout. Minted credentials are shown once.

Filling in metadata after the fact

  • fetch-tweet-metadatadeferred

    Author, excerpt, poster and like count for a batch of tweets. Runs on first render, not at capture.

  • fetch-youtube-metadatadeferred

    Title, channel and thumbnail through a public endpoint, no API key.

  • fetch-link-metadatadeferred

    Title, description and image for saved links. Results are not persisted, and private hosts are refused.

Housekeeping

  • purge-orphan-storagesync

    Deletes files nothing in the database points at. Operator-invoked, and safe to re-run.

One capture, traced

A screenshot from a group chat to a searchable reference.

The rest of this page describes the shape of the system. This follows one thing through it. Telegram is a capture path because the designers were already dropping screenshots into a chat, so the habit got supported instead of fought.

  1. A designer sends a photo to the bot
  2. Telegram posts a webhook update
  3. Confirm the request is genuinely from Telegram
    Fails closed. An unconfigured check rejects everyone rather than admitting everyone.
  4. Check the sender against the team
    Also fails closed.
  5. Ask Telegram where the file is, then download it
  6. Name it by counting today's captures
  7. Upload the bytes to storage
  8. Insert one row, every label field empty
  9. Reply in the chat with the name
  10. The reference is in the catalogue, unlabelled and searchable by name
  11. Later, separately, a person or an agent labels it
    Never automatic. Nothing runs on insert.
  12. A person clicks Verify
    Only now is it exposed to agents.

Why “verified” can be trusted

Labelling is never automatic. There is no server-side model call anywhere in the project: an AI drafts a label through a prompt a person runs, or an agent submits one over MCP. Either way it arrives as “needs review”. Four independent things stop a machine publishing to the library.

  1. The write endpoint hard-codes the status to needs_review. A submitted "verified" is discarded, not honoured.
  2. Verified is written in exactly one place in the codebase: a person clicking Verify, behind ten validation rules.
  3. A human pasting AI output that claims verified gets a draft instead, with a visible warning.
  4. The read API distrusts it anyway, filtering AI-authored records that are not verified on both search and direct lookup.

What happens when it goes wrong

Sender is not on the team
Ignored silently. No reply at all, so the bot does not confirm it exists.
Unsupported file
A help reply naming what is accepted. Nothing is stored. A video or PDF sent as a document is refused.
Upload fails
Reported in the chat. No row is created, so there is no orphan record.
Insert fails after the upload succeeded
Reported in the chat, and the file is left orphaned in storage. This is the case the manual purge function exists to clean up.
The same URL again
Answered as "already in catalogue" rather than as an error.

Offline capture

A lost connection should not lose the reference.

Mutations wait in a local queue while image blobs remain available. Reconnection flushes the queue in order.

offline-sync.tsx
DesignerDeviceLocal queueNetworkAgentUX core
  1. Capture a referenceDesigner → Device
  2. Store metadata and blob pointerDevice → Local queue
  3. Connection restoredNetwork → Local queue
  4. Upload queued changes in orderLocal queue → AgentUX core
  5. Reconcile canonical stateAgentUX core → Device
  6. Confirm syncDevice → Designer
Local persistence protects the reference; ordered replay protects the final state.

Security model

Humans and machines use different credentials.

The product separates durable human authentication from narrow machine sessions. Each credential follows the risk of its client.

CredentialStorageControl
Human passcodeArgon2id hashAccount lifecycle
Agent sessionSHA-256 hashScoped, expiring, revocable
Telegram webhookAuthenticity checkReject untrusted requests

Platform details

Small infrastructure choices protect the experience.

Prototype proxy

External prototypes load through a controlled hosting layer.

Share target

The service worker intercepts only the share flow it owns.

Email-keyed collaboration

Team identity stays stable across invitations and comments.