[00] Audya AgentPrototype · phase 4

Everybuttonisalsoatool.

Audya is agentic end to end. Anything you can do with a mouse or a controller, an agent can do too: from the built-in dock, from Claude over MCP, or from a script. It digs, preps and plans, and only touches what's on air when you let it.

SuggestAskAutopilot
[01]Architecture

One action registry. Three front ends.

Every action in Audya is declared once, with a schema, a description and a risk class. Controllers, the agent and MCP clients all call the same thing.
Action registry · single source of truth
deck.1.playperform
library.crate.createprep
mixer.ch2.eq.lowlive
library.searchread
Typed schema (zod → JSON Schema), a description and a risk class, declared once.
  • Controller mappings

    MIDI / HID → action

  • Agent tools

    action → LLM tool definition

  • MCP server

    audya-mcp, for Claude and other clients

Every action, from a hand, a controller, the agent or MCP, goes through one command bus with an undo stack and an activity log.

[02]What it does

From crate digging to the last record.

  • 01

    Library curation

    “A 90-minute warm-up crate, 118–124, nothing from my last 3 gigs.” Metadata, analysis and ‘sounds like’ search together.

  • 02

    Set planning

    A harmonic and energy path through a crate: Camelot moves, BPM ramps, an energy arc. Edit it as a timeline.

  • 03

    Auto-prep

    Grid checks, hot cues at intro, drop, breakdown and outro, gain, bad rips and clipping. In batch, over a whole crate.

  • 04

    Live co-pilot

    Suggests the next track, pre-loads it on the idle deck and proposes a transition on a ghost lane you accept with one tap.

  • 05

    Autopilot

    Mixes by itself for breaks, streams and practice. Human override always wins, instantly.

  • 06

    Voice

    Push to talk: “loop 8, swap the bass on the next phrase.”

  • 07

    Explain & teach

    “Why did that clash?” “Show me an echo-out.”

[03]Autonomy

You decide how much it does.

Three levels, switchable mid-set. Each action carries a risk class, and the level decides which classes run, which ask first, and which are off.

Prep runs and is logged. Each perform and live action shows up as a card you approve with a tap or a controller button.

What the agent may do at the Ask level
Risk classAsk
read
Search the library, read deck state, analyze
Allowed
prep
Create crates, set cues and grids, tag, plan a set
Allowed
perform
Load a non-live deck, set a loop, queue a transition
Asks first
live
Touch the deck on air, master FX, stop
Asks first
[04]Real-time safety

Built so it can't drop the beat.

  1. 01

    Audio never waits on the model.

    The agent only schedules actions, through the same lock-free queue the controllers use. The audio thread never blocks on it.

  2. 02

    Quantized to the music.

    Actions land on beats or phrases. The co-pilot pre-computes suggestions while a track plays, so a slow answer is still on time.

  3. 03

    Preview before it plays.

    Live actions show up as ghost automation first, unless Autopilot is on.

  4. 04

    Touch a deck to take back control.

    Any physical touch cancels the agent's pending actions for that deck.

Ghost lane · next 32 beatsQueued · phrase-quantized
Deck A · EQ low
Deck B · EQ low
Deck A · Filter
A physical touch cancels the agent's pending actions for that deck.
[05]The dock

Mid-set, with the co-pilot on.

A dock on the right that pushes the decks aside. Mention a track, deck or crate with @, run a / command, or hold to talk. Tool calls show up as live cards you can approve or undo.

Talk to it. Commands: /build-crate, /plan-set, /prep, /next, /transition.

Illustrative demo. Track names are invented. The dock prototype in the web preview isn't connected to a model yet.

Audya AgentAsk
    Ask, @mention a deck, or type / for commands
    [06]MCP

    Drive Audya from Claude.

    The same registry is exposed as an MCP server, audya-mcp. Claude Desktop, Claude Code and other MCP clients can search your library, build crates and prep tracks in a running Audya. The desktop app serves it locally over stdio or localhost; the web app goes through a relay.

    • +Same risk classes and autonomy level as the dock
    • +Every call lands in the activity log, undoable
    • +Library stays local: tools return small, focused views
    Preview · illustrative
    claude_desktop_config.json
    {
    "mcpServers": {
    "audya": {
    "command": "audya-mcp",
    "args": ["--stdio"],
    "env": { "AUDYA_AUTONOMY": "ask" }
    }
    }
    }
    Not published yet. The package name, flags and relay URL will change before release.
    Agent dock prototype in the web preview
    Meet your co-pilot.