Home

Deep dive 01 · AI access

Give agents context without giving away control.

AgentUX exposes the same trusted library through REST and MCP. Human review stays between AI analysis and team-wide reference material.

The knowledge layer

One screenshot, and everything the labeller found in it.

This is a real record from the library, not a sample. A Robinhood options screen went in. The labeller located thirteen interface elements, described the screen, and wrote down when to reuse it. A designer verified the result before it became trusted team knowledge.

Robinhood Simulated Returns, the source screen AgentUX labelled12345678910111213
Robinhood · Simulated Returns · 13 elements located, confidence 0.50 to 0.95
  1. 1Header / navigation0.72
  2. 2Primary content area0.70
  3. 3Add widget
  4. 4Buy to open
  5. 5Sell to open
  6. 6Buying power
  7. 7Cancel
  8. 8Submit
  9. 9Daily layout
  10. 10Stock trading
  11. 11SIMRET0.50
  12. 12Overnight
  13. 13Simulated Returns
Derived structureai draft

Identity

source_app
Robinhood
product_category
finance
page_types
application screen
platform
web
device_type
desktop
screen_state
default

Visual

theme
dark
density
compact

Journey

user_problem
The user needs to understand the current financial state or complete a high-stakes account action without ambiguity.
step_goal
Help the user complete the visible task with clear controls and feedback.
user_action
Interacts with the primary controls visible on the screen.

Style

style_keywords
financial UI, data dense, transactional, workflow, dark
colors
approximate
verified · no missing fields
Reuse guidancedesign_reference

Patterns to steal

  • Keep critical values close to their labels and controls.
  • Use clear state feedback for submitted, warning, modal, or empty conditions.
  • Anchor the primary task in a stable header/content/action structure.

Avoid using when

  • Avoid copying the density for casual consumer onboarding or low-stakes browsing screens.
  • Avoid using it without additional accessibility checks for contrast, tap targets, and numeric readability.

Left unset on this record: flow, next_step, previous_step, step_index, screens_count.

One core, two doors

REST for products. MCP for agents.

Both interfaces use the same retrieval logic and permission model. The transport changes, but the source of truth does not.

agent-access.tsx

Interfaces

  1. GET /api/referencesREST products
  2. POST /api/mcpClaude · Cursor · Codex

Shared search core

  1. Permission scopesession + collection
  2. Filters + rankingone retrieval model

Trusted output

  1. Structured references
  2. Signed media URLs
  3. Verified metadata
The interface changes; retrieval, permissions, and reviewed data stay shared.

Least privilege

Agent access can expire, narrow, and stop.

A connection creates an expiring session token. The database stores a SHA-256 hash, and the user can revoke the session at any time.

agent-session.tsx
PersonAgentUXSession storeAgent
  1. Connect agent with selected scopePerson → AgentUX
  2. Save token hash, scope, and expiryAgentUX → Session store
  3. Return session token onceAgentUX → Agent
  4. Search allowed collectionsAgent → AgentUX
  5. Verify hash, scope, and expiryAgentUX → Session store
  6. Access allowedSession store → AgentUX
  7. Return signed referencesAgentUX → Agent
  8. Revoke connectionPerson → AgentUX
  9. Invalidate session immediatelyAgentUX → Session store
Only a hash reaches storage. Scope and expiry are checked on every request.

Design decision

AI speeds up the draft, not the judgement.

Assist

AI turns a visual into a first pass of structured metadata.

Verify

A designer checks the language, pattern, and product context.

Reuse

People and agents retrieve the same reviewed source later.