ADR-016: LLM Signal Awareness & Contextual Engagement

Status

Proposed (Pending Review)

Context

Problem Statement

Conductor's Chat panel LLM operates in a configuration-only world. It can query mappings, device bindings, and settings via MCP tools — but it has zero awareness of what is actually happening at runtime. It cannot see events flowing through the system, doesn't know which mappings are firing, can't detect anomalies like feedback loops or sustained unmapped activity, and has no sense of whether the user's setup is working as intended.

This creates a fundamental disconnect: the user is staring at live events scrolling through the Events panel and a Signal Flow diagram showing real-time routing topology, but the LLM they're conversing with is blind to all of it. The user must manually describe runtime behaviour ("CC7 keeps firing but nothing happens") instead of the LLM being able to observe and reason about it directly.

Current LLM Architecture (ADR-007)

The Chat panel's LLM integration is built on three layers:

System prompt: Hardcoded domain knowledge (trigger types, action types, tool specifications). No runtime data.
MCP tools: 20 read/write tools for config queries and modifications. All are request-response — the LLM must actively query; nothing is pushed to it.
Conversation history: Rolling window of 20 messages. No injected context about device state or event activity.

The LLM's tool risk tiers (ReadOnly → ConfigChange → HardwareIO) and Plan/Apply pattern for config changes are well-established. This ADR extends the architecture with a new concern: runtime signal awareness — giving the LLM curated, summarised knowledge of what the system is doing right now.

Why Not Just Add More Tools?

Adding a get_recent_events tool would let the LLM query the event buffer on demand. But this addresses only the reactive case (user asks "what's happening?"). The higher-value scenarios are proactive:

Detecting a feedback loop before it becomes a MIDI storm
Noticing sustained unmapped activity and offering to create a mapping
Observing that a mapping hasn't fired in a session and suggesting a check
Recognising that a device disconnected mid-performance

These require the LLM to have ambient awareness without the user explicitly asking — which means signal data must be pushed into the LLM's context, not pulled.

Design Exploration

The Signal Flow Track Diagram V2 mockups (event-provenance-mockups-v2.html) visualise routing topology including fan-out, cross-track routing, and feedback loop detection. ADR-014 introduces mapping_fired events. This ADR considers how these same signals can inform LLM behaviour.

Decision

D1: Three-Tier Signal Context Model

Signal data is categorised into three tiers based on volatility, token cost, and LLM utility. Each tier has a different injection mechanism and refresh cadence.

Tier	Name	Content	Injection Method	Refresh
T1	Structural Topology	Device count, mapping count, fan-out junctions, cross-track routes, detected loops, unmapped trigger types	Appended to system prompt	On config change
T2	Aggregate Runtime State	Per-device event rate (events/sec), per-channel activity (active/idle), mapping fire counts (last N minutes), error counts	Injected as a `system` message before the user's latest message	Every 30s (configurable)
T3	Significant Events	Loop warnings, device connect/disconnect, sustained unmapped activity, mapping errors, mode changes	Injected as individual `system` messages when they occur	Event-driven

Rationale: T1 is cheap (changes rarely, ~200 tokens) and gives the LLM structural reasoning ability. T2 is moderate (~100–300 tokens per refresh) and gives temporal awareness. T3 is surgical — only fires on significant events that the LLM should know about, keeping token cost proportional to signal value.

D2: T1 — Structural Topology Summary

On config load and on every config mutation, the frontend computes a topology summary and appends it to the system prompt. This replaces the LLM needing to query conductor_list_mappings + conductor_list_device_bindings + mentally reconstruct the signal flow.

## Current Signal Topology

Devices: 2 connected, 1 disconnected
  • Mikro MK3 (mikro-mk3-midi) — enabled, ch.1 + ch.10
  • LaunchControl XL (launch-control-xl) — enabled, ch.2
  • Gamepad (gamepad-001) — disconnected

Mappings: 8 active (Mode: Default)
  • 5 simple (1:1 trigger→action)
  • 2 fan-out (1:N) — CC7 → [Keystroke, SendMidi, OscSend]
  • 1 sequence — CC1 → [MouseClick, Text]

Routing:
  • Cross-track: Mikro → IAC Bus 1 (MidiForward, ch.3)
  • Cross-track: LaunchControl → IAC Bus 1 (SendMidi, ch.1)

⚠ Warnings:
  • Feedback loop detected: Mikro CC7 → SendMidi IAC Bus 1 ch.1 → IAC Bus 1 CC7 mapping (confirmed)
  • Potential loop: LaunchControl CC1 → IAC Bus 1 (port is also input)

Unmapped trigger types observed: CC74 (ch.2), Aftertouch (ch.1)

Token cost: ~150–250 tokens depending on setup complexity. Recomputed only on config change (not per-message).

Implementation: A new buildTopologySummary() function in chat.js (or a dedicated signal-context.js module) that reads from mappingsStore, deviceBindingsStore, and the loop detection logic from Signal Flow. Appended to the system prompt in _buildMessages().

D3: T2 — Aggregate Runtime State (Signal Pulse)

A periodic summary of runtime activity, injected as a system message. This gives the LLM a "pulse check" without overwhelming it with individual events.

{
  "type": "signal_pulse",
  "window_seconds": 30,
  "devices": {
    "mikro-mk3": {
      "events_per_sec": 12.4,
      "channels": { "1": "active", "10": "idle" },
      "top_types": ["CC", "Note On"]
    },
    "launch-control-xl": {
      "events_per_sec": 0.2,
      "channels": { "2": "idle" },
      "top_types": ["CC"]
    }
  },
  "mappings_fired": {
    "total": 47,
    "top_3": [
      { "name": "CC7 → Volume", "count": 31 },
      { "name": "N36 → Copy", "count": 12 },
      { "name": "CC1 → Sequence", "count": 4 }
    ],
    "errors": 0
  },
  "unmapped_events": 23,
  "mode": "Default"
}

Injection: Formatted as a concise system message prepended to the user's latest message: [Signal Pulse — last 30s: Mikro 12.4 ev/s (CC,Note), LCXL idle. 47 mappings fired (CC7→Vol: 31×). 23 unmapped. Mode: Default]

Token cost: ~60–100 tokens per pulse. Only injected when the user sends a message (not continuously). The pulse is computed from a rolling window maintained in a new signalPulseStore.

Cadence: Recomputed every 30 seconds (configurable via settings). Stale pulses older than 60s are dropped. When no events have occurred, pulse is suppressed entirely ("system idle — no recent activity").

D4: T3 — Significant Event Alerts

High-signal events that warrant LLM awareness regardless of whether the user asked. These are injected as system messages into the conversation when they occur.

Event	Trigger Condition	LLM Message	Proactive?
Loop Detected	Static analysis finds confirmed feedback loop	`⚠ Feedback loop detected: [description]. This could cause a MIDI storm.`	Yes — LLM should mention this
Device Disconnected	`DeviceBinding.connected` transitions false	`Device "[alias]" disconnected.`	Passive — mention if user asks
Device Connected	New device appears in bindings	`New device connected: "[alias]" on [port].`	Passive
Sustained Unmapped	>50 unmapped events of same type in 30s window	`Sustained unmapped activity: [type] on [device] ch.[N] (estimated [X] events in last 30s). No mapping exists for this trigger.`	Yes — LLM should offer to help
Mapping Error	`action_error` event from daemon	`Mapping "[name]" failed: [error]. Action: [action_type].`	Yes — LLM should explain
Mode Change	`mode_change` event from daemon	`Mode changed to "[mode]". [N] mappings now active.`	Passive
High Event Rate	>200 events/sec sustained for 5s on any port	`⚠ High event rate on [device]: [N] events/sec. Possible feedback loop or stuck controller.`	Yes — LLM should warn

Proactive vs passive: "Proactive" means the LLM should reference the alert in its next response even if the user didn't ask about it. "Passive" means the LLM has the context but only surfaces it if relevant to the user's question.

Implementation: T3 events are pushed into a signalAlerts array in the chat store. When _buildMessages() runs, any unacknowledged alerts are prepended as system messages. Once the LLM has seen them (included in a request), they're marked as acknowledged. Alerts older than 5 minutes are pruned.

D5: LLM Responsibility Model

The LLM's engagement with signal data follows a graduated responsibility model aligned with ADR-007's risk tiers:

Observe: The LLM can always reference signal data in responses. "I can see your Mikro is sending a lot of CC7 events — that mapping is firing about 31 times in the last 30 seconds."

Suggest: The LLM can proactively suggest actions based on signal data. "You have sustained CC74 activity on channel 2 that isn't mapped to anything. Would you like me to create a mapping for it?" This is bounded by a suggestion cooldown — the LLM won't repeat the same suggestion within a 5-minute window.

Act (with confirmation): The LLM can propose config changes in response to signal observations, but all changes go through the existing Plan/Apply pattern (ADR-007 D4). "I've detected a feedback loop. I can modify the CC7 mapping to exclude IAC Bus 1 as a SendMidi target — here's the proposed change." User must explicitly approve.

Never: The LLM never autonomously modifies config, disconnects devices, or suppresses events based on signal data alone. No "auto-fix" behaviour. The LLM is an advisor, not an autonomous agent.

D6: Performance Mode Flag

A new performanceMode boolean in settings. When enabled:

T3 proactive alerts are suppressed — the LLM still has the data but won't proactively mention it. User must ask.
T2 pulse cadence doubles (60s instead of 30s) to reduce background computation.
Suggestion cooldown increases to 15 minutes.
Toast notifications from ADR-014 are minimised (only errors, not successful fires).

This prevents the LLM from interjecting during a live performance. The user can still ask "is everything working?" and get a full signal-aware answer.

UI: A toggle in the Chat panel header or a /performance chat command. The Signal Flow view shows a subtle "PERFORMANCE MODE" badge when active.

D7: New MCP Tools for Signal Queries

In addition to the pushed context (T1–T3), the LLM gets new tools for on-demand signal queries:

Tool	Risk Tier	Description
`conductor_get_signal_pulse`	ReadOnly	Returns the current T2 signal pulse (same data as injected, but on-demand)
`conductor_get_topology_summary`	ReadOnly	Returns the T1 topology summary as structured data
`conductor_get_recent_events`	ReadOnly	Returns last N events from the event buffer, optionally filtered by device/type/channel
`conductor_get_mapping_stats`	ReadOnly	Returns fire counts, last-fired timestamps, and error counts per mapping for the current session
`conductor_get_loop_analysis`	ReadOnly	Returns all detected feedback loops with severity, path description, and suggested resolution

Rationale: Pushed context (T1–T3) gives the LLM ambient awareness. These tools give it the ability to drill deeper when the user asks specific questions like "which mapping fires most often?" or "show me the last 10 events from my Mikro."

D8: Signal Context Store Architecture

A new signalContext.js store module manages the three tiers:

signalContextStore
├── topology          // T1: recomputed on config change
│   ├── deviceSummary
│   ├── mappingSummary
│   ├── routingSummary
│   └── warnings[]
├── pulse             // T2: recomputed every 30s
│   ├── window_seconds
│   ├── devices{}
│   ├── mappings_fired{}
│   └── unmapped_count
├── alerts[]          // T3: event-driven
│   ├── { type, message, severity, timestamp, acknowledged }
│   └── ...
└── settings
    ├── pulseInterval (default: 30000)
    ├── alertMaxAge (default: 300000)
    ├── performanceMode (default: false)
    └── enabled (default: true)

Derived store: signalContextForLLM — a derived store that formats T1+T2+T3 into the message format expected by _buildMessages(). This keeps the formatting logic out of the chat store.

D9: Integration with Signal Flow View

The Signal Flow workspace view (V2 Track Diagram) and the LLM share the same underlying data:

Topology summary (T1) is the text equivalent of what the track diagram visualises
Loop detection feeds both the return-arc warnings in the diagram and the T3 alerts for the LLM
Mapping fire counts (T2) correspond to the junction ⚡ fired states (ADR-014)
Unmapped events (T2/T3) correspond to the dashed-circle passthrough indicators in V2

This means a single signalAnalysis computation serves both the visual and conversational interfaces. The Signal Flow view renders it spatially; the LLM receives it as structured text.

Interaction synergy: When the user clicks a loop warning badge in Signal Flow, the Chat panel could pre-populate with "Help me resolve this feedback loop" — bridging the visual and conversational interfaces.

Implementation Plan

Phase 1: Structural Topology (T1) — 6–8h

Create signal-context.js store with buildTopologySummary() function
Subscribe to mappingsStore and deviceBindingsStore for reactive recomputation
Implement loop detection logic (port-match + trigger-match analysis)
Modify _buildMessages() in chat.js to append topology summary to system prompt
Add conductor_get_topology_summary MCP tool

Phase 2: Signal Pulse (T2) — 8–10h

Add signalPulseStore with rolling window event aggregation
Subscribe to eventBuffer for per-device, per-channel, per-type counters
Implement mapping fire count tracking (subscribe to mapping_fired Tauri events from ADR-014)
Add pulse formatting for LLM injection (concise single-line format)
Modify _buildMessages() to prepend latest pulse as system message
Add conductor_get_signal_pulse and conductor_get_recent_events MCP tools
Add pulse interval setting to Chat settings

Phase 3: Significant Event Alerts (T3) — 6–8h

Implement alert detection rules (loop, disconnect, unmapped sustained, error, high rate)
Add signalAlerts array with acknowledgement tracking
Inject unacknowledged alerts as system messages in _buildMessages()
Add proactive/passive classification and Performance Mode flag
Add suggestion cooldown logic (per-alert-type deduplication)
Add conductor_get_loop_analysis and conductor_get_mapping_stats MCP tools

Phase 4: Conversational Integration — 4–6h

Prompt engineering: update system prompt with signal awareness guidelines
Add examples of signal-aware responses to skill context
Implement Chat → Signal Flow bridge (clicking loop warning pre-populates chat)
Add /performance command and Performance Mode toggle
Test proactive engagement quality across providers (OpenAI, Anthropic, OpenRouter)

Phase 5: Optimisation & Tuning — 4–6h

Token budget analysis across conversation lengths (T1+T2+T3 combined)
Pulse compression for low-activity periods (suppress when idle)
Alert deduplication for rapid-fire events (e.g., 10 device disconnect/reconnect cycles)
A/B testing of proactive vs passive engagement for different alert types
Documentation and settings UI

Total estimated effort: 28–38 hours across 5 phases.

Consequences

Positive

The LLM becomes a genuine co-pilot rather than a config editor — it can observe, reason about, and respond to live system behaviour
Feedback loop detection surfaces in both visual (Signal Flow) and conversational (Chat) interfaces simultaneously
Users can ask natural-language questions about runtime state: "is my Mikro working?", "why isn't CC7 doing anything?", "which mapping fires most?"
Proactive suggestions for unmapped events could significantly reduce setup time for new users
Performance Mode gives experienced users control over engagement level

Negative

Token cost increases by ~200–400 tokens per message (T1: ~200, T2: ~80, T3: variable)
Proactive LLM engagement risks feeling intrusive if poorly calibrated — Performance Mode and suggestion cooldowns mitigate this
Signal context is frontend-computed — if the GUI is not running, the LLM has no runtime awareness (acceptable for a desktop app)
Multi-provider testing needed — proactive behaviour quality varies significantly between LLM providers

Risks

Token budget pressure: T1+T2+T3 in a long conversation with many alerts could consume 500+ tokens of the context window. Mitigation: alert pruning (5min max age), pulse suppression when idle, topology summary caching.
Stale context: If the pulse hasn't refreshed and the user asks "what's happening right now?", the LLM might reference 30-second-old data. Mitigation: the conductor_get_signal_pulse tool provides on-demand fresh data.
False positive alerts: Amber loop warnings may fire for ports that are technically bidirectional but not actually looping. Mitigation: two-tier severity (amber/red) and user-dismissable alerts.

Dependencies

ADR-014 (Mapping Feedback & Simulate): Provides mapping_fired Tauri events that feed T2 fire counts and T3 mapping error alerts
Signal Flow V2 (Track Diagram): Shares loop detection logic and topology analysis with T1/T3
MIDI Channel Pipeline (docs/midi-channel-pipeline): Channel-aware events would enrich T1 topology and T2 pulse with per-channel granularity. Currently channel-blind on input side — T1/T2 will show channel data only where available (output side, GUI-parsed events)

References

ADR-007: LLM Integration Architecture (tool risk tiers, Plan/Apply pattern)
ADR-014: Mapping Feedback & Simulate (mapping_fired events, action visibility)
event-provenance-mockups-v2.html: Signal Flow Track Diagram V2 with routing topology
conductor-gui/ui/src/lib/stores/chat.js: Current LLM message building (_buildMessages())
conductor-gui/ui/src/lib/stores/events.js: Event buffer and device filter stores

Status​

Context​

Problem Statement​

Current LLM Architecture (ADR-007)​

Why Not Just Add More Tools?​

Design Exploration​

Decision​

D1: Three-Tier Signal Context Model​

D2: T1 — Structural Topology Summary​

D3: T2 — Aggregate Runtime State (Signal Pulse)​

D4: T3 — Significant Event Alerts​

D5: LLM Responsibility Model​

D6: Performance Mode Flag​

D7: New MCP Tools for Signal Queries​

D8: Signal Context Store Architecture​

D9: Integration with Signal Flow View​

Implementation Plan​

Phase 1: Structural Topology (T1) — 6–8h​

Phase 2: Signal Pulse (T2) — 8–10h​

Phase 3: Significant Event Alerts (T3) — 6–8h​

Phase 4: Conversational Integration — 4–6h​

Phase 5: Optimisation & Tuning — 4–6h​

Consequences​

Positive​

Negative​

Risks​

Dependencies​

References​