28 posts tagged with "ai"

This post covers Phase 1 of ADR-017 (Automated Market Discovery and Matching) - establishing the data types and persistence layer for the discovery system.

The Problem​

Manual market mapping is error-prone and doesn't scale. Polymarket and Kalshi list hundreds of markets; finding equivalent pairs requires:

1. Persistent storage - Track discovered markets across restarts
2. Status tracking - Pending → Approved/Rejected workflow
3. Audit trail - Record all approval decisions for compliance
4. Safety gates - Prevent automated trading without human review

Design Decisions​

CandidateStatus State Machine​

The core safety mechanism is a one-way state machine:

Pending ──┬──► Approved
          │
          └──► Rejected

Once a candidate is approved or rejected, the status is immutable. This prevents accidental re-processing or status manipulation:

impl CandidateStatus {
    pub fn can_transition_to(&self, new_status: CandidateStatus) -> bool {
        match (self, new_status) {
            (CandidateStatus::Pending, CandidateStatus::Approved) => true,
            (CandidateStatus::Pending, CandidateStatus::Rejected) => true,
            // Once approved or rejected, status is final
            (CandidateStatus::Approved, _) => false,
            (CandidateStatus::Rejected, _) => false,
            _ => false,
        }
    }
}

Semantic Warnings​

Markets that appear similar may have different settlement criteria. The CandidateMatch struct includes a semantic_warnings field that Phase 2's matcher will populate:

pub struct CandidateMatch {
    pub semantic_warnings: Vec<String>,  // e.g., "Settlement timing differs"
    // ...
}

Approval will require explicit acknowledgment of these warnings (FR-MD-003).

SQLite Storage​

We chose SQLite over PostgreSQL for the discovery cache because:

1. Single-tenant - Discovery runs locally per operator
2. Portable - No external dependencies for development
3. Atomic - Transactions prevent partial state

Schema design separates markets from candidates:

-- Discovered markets (one per platform/id combination)
CREATE TABLE discovered_markets (
    id TEXT PRIMARY KEY,
    platform TEXT NOT NULL,
    platform_id TEXT NOT NULL,
    title TEXT NOT NULL,
    -- ...
    UNIQUE(platform, platform_id)
);

-- Candidate matches (references two markets)
CREATE TABLE candidates (
    id TEXT PRIMARY KEY,
    polymarket_id TEXT NOT NULL,
    kalshi_id TEXT NOT NULL,
    similarity_score REAL NOT NULL,
    status TEXT NOT NULL DEFAULT 'Pending',
    -- ...
);

-- Audit log for compliance
CREATE TABLE audit_log (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    timestamp TEXT NOT NULL,
    action TEXT NOT NULL,
    candidate_id TEXT NOT NULL,
    details TEXT NOT NULL  -- Full JSON context
);

Parameterized Queries​

All SQL uses the params![] macro to prevent injection:

conn.execute(
    "UPDATE candidates SET status = ?1, updated_at = ?2 WHERE id = ?3",
    params![status_str, now, id.to_string()],
)?;

Test Coverage​

Phase 1 includes 12 tests covering:

Key safety test:

#[test]
fn test_candidate_status_transitions() {
    // Once approved, cannot transition to any other status
    assert!(!CandidateStatus::Approved.can_transition_to(CandidateStatus::Pending));
    assert!(!CandidateStatus::Approved.can_transition_to(CandidateStatus::Rejected));
}

What's Next​

Phase 2 will implement the text matching engine:

• TextNormalizer - Lowercase, remove punctuation, tokenize
• SimilarityScorer - Jaccard (0.6 weight) + Levenshtein (0.4 weight)
• Semantic warning detection for settlement differences

Council Review​

Phase 1 passed council verification with confidence 0.88. Key findings:

• ✅ Human-in-the-loop enforced via CandidateStatus state machine
• ✅ Audit logging captures all required fields
• ✅ No SQL injection (all parameterized queries)
• ✅ No unsafe code

Implementation: arbiter-engine/src/discovery/ | Issues: #41, #42 | ADR: 017

This post covers Phase 2 of ADR-017 - implementing the text similarity matching engine that powers automated market discovery between Polymarket and Kalshi.

The Problem​

Phase 1 established the data types and storage layer. Now we need to actually find matching markets across platforms. The challenge:

1. Fuzzy matching - Market titles differ in phrasing ("Will Trump win?" vs "Trump wins 2024?")
2. False positives - Similar titles may have different settlement criteria
3. Scalability - Must compare thousands of markets efficiently

Algorithm Design​

Combined Similarity Scoring​

We use a weighted combination of two complementary algorithms:

score = 0.6 × Jaccard + 0.4 × Levenshtein

Jaccard similarity (0.6 weight) measures token set overlap:

let intersection = set_a.intersection(&set_b).count();
let union = set_a.union(&set_b).count();
jaccard = intersection / union

This captures semantic similarity when words are reordered.

Levenshtein similarity (0.4 weight) measures edit distance:

let distance = levenshtein(&norm_a, &norm_b);
levenshtein_sim = 1.0 - (distance / max_length)

This catches typos and minor variations.

Text Normalization​

Before comparison, titles are normalized:

impl TextNormalizer {
    pub fn normalize(&self, text: &str) -> String {
        // 1. Lowercase
        // 2. Replace punctuation with spaces
        // 3. Collapse whitespace
    }

    pub fn tokenize(&self, text: &str) -> Vec<String> {
        // 4. Split into words
        // 5. Filter stop words (a, an, the, will, be, ...)
    }
}

Example: "Will Bitcoin reach $100k?" → ["bitcoin", "reach", "100k"]

Pre-Filtering​

Before scoring, candidates are filtered to reduce false positives:

Semantic Warning Detection (FR-MD-008)​

Even similar titles may have different settlement criteria. We detect and flag:

Conditional language mismatches:

Polymarket: "Will Fed announce rate cut?"
Kalshi:     "Will Fed cut rates?"
⚠️ Warning: Settlement trigger mismatch - one market references 'announce'

Resolution source differences:

Polymarket resolution: "Associated Press"
Kalshi resolution:     "Official FEC results"
⚠️ Warning: Resolution source differs

Expiration differences:

⚠️ Warning: Expiration differs by 3 day(s)

These warnings flow to the human reviewer (FR-MD-003) for acknowledgment before approval.

Implementation​

SimilarityScorer​

pub struct SimilarityScorer {
    jaccard_weight: f64,      // 0.6
    levenshtein_weight: f64,  // 0.4
    threshold: f64,           // 0.6
    normalizer: TextNormalizer,
    pre_filter: PreFilterConfig,
}

impl SimilarityScorer {
    pub fn find_matches(
        &self,
        market: &DiscoveredMarket,
        candidates: &[DiscoveredMarket],
    ) -> Vec<CandidateMatch> {
        candidates.iter()
            .filter(|c| c.platform != market.platform)  // Cross-platform only
            .filter(|c| self.passes_pre_filter(market, c))
            .filter_map(|c| {
                let score = self.score(&market.title, &c.title);
                if score >= self.threshold {
                    let warnings = self.detect_warnings(market, c);
                    Some(CandidateMatch::new(/*...*/).with_warnings(warnings))
                } else {
                    None
                }
            })
            .collect()
    }
}

Match Reason Classification​

let match_reason = if score >= 0.95 {
    MatchReason::ExactTitle
} else {
    MatchReason::HighTextSimilarity { score: (score * 100.0) as u32 }
};

Test Coverage​

Phase 2 adds 10 tests (22 total for discovery module):

Key test:

#[test]
fn test_semantic_warning_announcement() {
    let scorer = SimilarityScorer::default();

    let poly = create_market(Platform::Polymarket, "Will Fed announce rate cut?");
    let kalshi = create_market(Platform::Kalshi, "Will Fed cut rates?");

    let warnings = scorer.detect_warnings(&poly, &kalshi);
    assert!(warnings.iter().any(|w| w.contains("announce")));
}

What's Next​

Phase 3 will implement the API clients:

• Polymarket Gamma API client (FR-MD-006)
• Kalshi /v2/markets API client (FR-MD-007)
• Rate limiting and pagination

Council Review​

Phase 2 passed council verification with confidence 0.85. Key findings:

• No unsafe code
• Human-in-the-loop preserved (find_matches returns candidates, not verified mappings)
• Semantic warnings properly flag settlement differences
• All tests passing (22 total)

Implementation: arbiter-engine/src/discovery/{normalizer,matcher}.rs | Issue: #43 | ADR: 017

This post covers Phase 3 of ADR-017 - implementing the API clients that fetch market listings from Polymarket and Kalshi for automated discovery.

The Problem​

Phase 1 and 2 established storage and matching. Now we need data sources:

1. Polymarket - Gamma API at gamma-api.polymarket.com
2. Kalshi - Trade API at api.elections.kalshi.com/trade-api/v2

Both APIs have:

• Pagination (different styles)
• Rate limits (different thresholds)
• Different response schemas

Design: DiscoveryClient Trait​

We define a common trait for both platforms:

#[async_trait]
pub trait DiscoveryClient: Send + Sync {
    async fn list_markets(
        &self,
        limit: Option<u32>,
        cursor: Option<&str>,
    ) -> Result<DiscoveryPage, DiscoveryError>;

    fn platform_name(&self) -> &'static str;
}

This allows the scanner (Phase 4) to enumerate markets from either platform interchangeably.

Rate Limiting​

Both APIs have rate limits we must respect:

We implement a token bucket rate limiter:

struct RateLimiter {
    tokens: AtomicU64,
    last_refill: Mutex<Instant>,
    max_tokens: u32,
}

impl RateLimiter {
    async fn acquire(&self) -> Option<Duration> {
        // Refill tokens based on elapsed time
        let elapsed = last.elapsed();
        let refill = (elapsed.as_secs_f64() / 60.0 * max_tokens) as u64;

        // Try to consume a token
        if tokens > 0 {
            tokens -= 1;
            return None; // Success
        }

        // Return wait time
        Some(Duration::from_secs_f64(60.0 / max_tokens))
    }
}

If rate limited, we return DiscoveryError::RateLimited with the retry time.

Pagination Strategies​

Polymarket: Offset-based​

GET /markets?limit=100&offset=0
GET /markets?limit=100&offset=100
...

We use the offset as the cursor, incrementing by page size.

Kalshi: Cursor-based​

GET /markets?limit=100&status=open
→ { markets: [...], cursor: "abc123" }

GET /markets?limit=100&cursor=abc123
→ { markets: [...], cursor: null }

We pass through the cursor directly.

Response Mapping​

Each API returns different schemas that we map to DiscoveredMarket:

Polymarket Gamma API​

struct GammaMarket {
    condition_id: String,    // → platform_id
    question: String,        // → title
    outcomes: String,        // JSON array → outcomes
    end_date: String,        // → expiration
    volume_24hr: f64,        // → volume_24h
    active: bool,            // Filter: skip if false
    closed: bool,            // Filter: skip if true
}

Kalshi Markets API​

struct KalshiMarket {
    ticker: String,          // → platform_id
    title: String,           // → title
    expiration_time: String, // → expiration
    volume_24h: i64,         // Cents → dollars
    status: String,          // Filter: only "open"/"active"
}

Key transformations:

• Kalshi volume is in cents, converted to dollars (/ 100.0)
• Inactive/closed markets are filtered out before returning
• Missing fields use sensible defaults

Error Handling​

pub enum DiscoveryError {
    Http(reqwest::Error),           // Network failures
    Parse(String),                  // JSON parsing
    RateLimited { retry_after_secs: u64 },  // 429 responses
    ApiError { status: u16, message: String },  // Other HTTP errors
}

The scanner (Phase 4) can handle these appropriately - retrying on rate limits, logging API errors.

Test Strategy​

We use wiremock for HTTP mocking:

#[tokio::test]
async fn test_list_markets_success() {
    let mock_server = MockServer::start().await;

    Mock::given(method("GET"))
        .and(path("/markets"))
        .respond_with(ResponseTemplate::new(200)
            .set_body_json(mock_response()))
        .mount(&mock_server)
        .await;

    let client = GammaApiClient::with_base_url(&mock_server.uri());
    let page = client.list_markets(Some(10), None).await.unwrap();

    assert_eq!(page.markets.len(), 2);
}

Test Coverage​

Phase 3 adds 8 tests (30 total for discovery):

What's Next​

Phase 4 will implement the scanner and approval workflow:

• DiscoveryScannerActor for periodic discovery runs
• ApprovalWorkflow for human review (FR-MD-003)
• Integration with MappingManager.verify_mapping()

Council Review​

Phase 3 passed council verification with confidence 0.87. Key findings:

• No unsafe code
• Proper rate limiting prevents API abuse
• 30-second timeout prevents hanging
• No credentials hardcoded
• Closed/inactive markets filtered out

Implementation: arbiter-engine/src/market/discovery_client/ | Issues: #44, #45 | ADR: 017

This post covers Phase 4 of ADR-017 - the scanner actor for periodic discovery and the safety-critical human approval workflow.

The Problem​

Phase 1-3 established storage, matching, and API clients. Now we need:

1. Automated Discovery - Periodic scanning of both platforms
2. Human Approval - Safety gate preventing automated mappings from entering trading

This phase implements FR-MD-003 (human confirmation required) and FR-MD-004 (auto-discover markets).

Safety-First Design​

FR-MD-003 is SAFETY CRITICAL. The approval workflow enforces:

1. Warning Acknowledgment - Cannot approve candidates with semantic warnings without explicit acknowledgment
2. Audit Logging - All decisions logged with full context
3. MappingManager Integration - Approved candidates go through existing safety gate

pub fn approve(&self, id: Uuid, acknowledge_warnings: bool) -> Result<Uuid, ApprovalError> {
    let candidate = self.get_candidate(id)?;

    // SAFETY CHECK: Require warning acknowledgment if warnings exist
    if !candidate.semantic_warnings.is_empty() && !acknowledge_warnings {
        return Err(ApprovalError::WarningsNotAcknowledged);
    }

    // Create verified mapping through the existing safety gate
    let mapping_id = {
        let mut manager = self.mapping_manager.lock().unwrap();
        let id = manager.propose_mapping(/*...*/);
        manager.verify_mapping(id);  // MappingManager safety gate
        id
    };

    // Update status and log decision
    // ...
}

Scanner Actor​

The DiscoveryScannerActor implements the Actor trait for periodic discovery:

pub enum ScannerMsg {
    Scan,           // Trigger a scan
    ForceRefresh,   // Ignore cache
    Stop,           // Graceful shutdown
    GetStatus(tx),  // Query status
}

Deduplication​

The scanner prevents duplicate candidates:

async fn is_duplicate_candidate(&self, candidate: &CandidateMatch) -> Result<bool, ScanError> {
    let storage = self.storage.lock().await;

    // Check pending candidates
    let pending = storage.query_candidates_by_status(CandidateStatus::Pending)?;
    for existing in pending {
        if existing.polymarket.platform_id == candidate.polymarket.platform_id
            && existing.kalshi.platform_id == candidate.kalshi.platform_id
        {
            return Ok(true);
        }
    }

    // Also check approved candidates
    let approved = storage.query_candidates_by_status(CandidateStatus::Approved)?;
    // ...
}

Scan Flow​

1. Fetch markets from Polymarket (with pagination)
2. Fetch markets from Kalshi (with cursor pagination)
3. Store all markets in SQLite
4. Run similarity matching
5. Deduplicate against existing candidates
6. Store new candidates with Pending status

Approval Workflow​

The ApprovalWorkflow provides the human interface:

// List candidates awaiting review
let pending = workflow.list_pending()?;

// Approve (must acknowledge warnings if present)
let mapping_id = workflow.approve(candidate_id, true)?;

// Reject with reason (required)
workflow.reject(candidate_id, "Different settlement criteria")?;

Rejection Requires Reason​

To maintain audit trail quality, rejections require a non-empty reason:

pub fn reject(&self, id: Uuid, reason: &str) -> Result<(), ApprovalError> {
    if reason.trim().is_empty() {
        return Err(ApprovalError::ReasonRequired);
    }
    // ...
}

Audit Trail​

Every decision is logged with full context:

let entry = AuditLogEntry {
    timestamp: Utc::now(),
    action: AuditAction::Approve,  // or Reject
    candidate_id: id,
    polymarket_id: candidate.polymarket.platform_id.clone(),
    kalshi_id: candidate.kalshi.platform_id.clone(),
    similarity_score: candidate.similarity_score,
    semantic_warnings: candidate.semantic_warnings.clone(),
    acknowledged_warnings: acknowledge_warnings,
    reason: None,  // or Some("...") for rejections
    session_id: self.session_id.clone(),
};
storage.append_audit_log(&entry)?;

Test Coverage​

Phase 4 adds 10 tests (40 total for discovery):

Critical Safety Test​

#[test]
fn test_approve_requires_warning_acknowledgment() {
    // Add candidate WITH warnings
    let candidate = setup_candidate(&storage, true);
    let workflow = ApprovalWorkflow::new(storage, mapping_manager);

    // Try to approve WITHOUT acknowledging warnings - MUST FAIL
    let result = workflow.approve(candidate.id, false);
    assert!(result.is_err(), "SAFETY VIOLATION: Should require warning acknowledgment");

    match result {
        Err(ApprovalError::WarningsNotAcknowledged) => {
            // Correct error type
        }
        Ok(_) => panic!("SAFETY VIOLATION: Approved without acknowledging warnings!"),
        // ...
    }
}

What's Next​

Phase 5 will implement CLI integration:

• --discover-markets - Trigger discovery scan
• --list-candidates - List pending/approved/rejected
• --approve-candidates - Approve by ID
• --reject-candidates - Reject with reason

Council Review​

Phase 4 passed council verification with confidence 0.91 (Safety focus). Key findings:

• FR-MD-003 enforcement verified
• Warning acknowledgment required
• Audit logging on all decisions
• Integration with MappingManager.verify_mapping() confirmed
• Deduplication prevents duplicate reviews

Implementation: arbiter-engine/src/discovery/{scanner,approval}.rs | Issues: #46, #47 | ADR: 017

This post covers Phase 5, the final phase of ADR-017 - CLI command integration for the discovery workflow.

The Problem​

Phases 1-4 built the complete discovery infrastructure:

• Storage and data types
• Text similarity matching
• API clients for both platforms
• Scanner actor and approval workflow

But operators had no way to interact with this system. Phase 5 bridges that gap.

CLI Commands​

Four commands enable the human-in-the-loop workflow:

# Trigger discovery scan
cargo run --features discovery -- --discover-markets

# List candidates by status
cargo run --features discovery -- --list-candidates --status pending
cargo run --features discovery -- --list-candidates --status approved
cargo run --features discovery -- --list-candidates --status rejected
cargo run --features discovery -- --list-candidates --status all

# Approve a candidate (with optional warning acknowledgment)
cargo run --features discovery -- --approve-candidate <uuid>
cargo run --features discovery -- --approve-candidate <uuid> --acknowledge-warnings

# Reject with required reason
cargo run --features discovery -- --reject-candidate <uuid> --reason "Different settlement criteria"

Testable Command Handlers​

The CLI handlers are separated from main.rs into src/discovery/cli.rs for testability:

pub struct DiscoveryCli {
    storage: Arc<Mutex<CandidateStorage>>,
    mapping_manager: Arc<Mutex<MappingManager>>,
    config: DiscoveryCliConfig,
}

impl DiscoveryCli {
    pub fn list_candidates(&self, status: Option<CandidateStatus>) -> CliResult { ... }
    pub fn approve_candidate(&self, id: Uuid, acknowledge_warnings: bool) -> CliResult { ... }
    pub fn reject_candidate(&self, id: Uuid, reason: &str) -> CliResult { ... }
}

This separation allows comprehensive unit testing without spawning the full async runtime.

Safety Enforcement at CLI Layer​

The CLI layer preserves FR-MD-003 safety guarantees:

pub fn approve_candidate(&self, id: Uuid, acknowledge_warnings: bool) -> CliResult {
    let workflow = ApprovalWorkflow::new(...);

    match workflow.approve(id, acknowledge_warnings) {
        Ok(mapping_id) => CliResult::Success(format!(
            "Candidate {} approved. Verified mapping ID: {}", id, mapping_id
        )),
        Err(ApprovalError::WarningsNotAcknowledged) => CliResult::Error(
            "Cannot approve: candidate has semantic warnings. \
             Use --acknowledge-warnings to proceed.".to_string()
        ),
        // ... other error handling
    }
}

Error messages guide operators to the correct action.

Feature Gate Error Handling​

When the discovery feature is not enabled, helpful error messages are shown:

#[cfg(not(feature = "discovery"))]
{
    if is_discovery_command {
        eprintln!("Discovery commands require the 'discovery' feature.");
        eprintln!("   Build with: cargo build --features discovery");
        eprintln!("   Run with:   cargo run --features discovery -- --discover-markets");
        return Ok(());
    }
}

Test Coverage​

Phase 5 adds 8 tests (48 total for discovery, 377 overall):

ADR-017 Complete​

With Phase 5, ADR-017 is fully implemented:

Council Review​

Phase 5 passed council verification with confidence 0.95 (Safety focus). Key findings:

• FR-MD-003 enforcement verified at CLI layer
• Warning acknowledgment required for candidates with semantic warnings
• Rejection requires non-empty reason for audit trail
• Clear error messages guide operators
• Feature gate prevents confusion when feature disabled
• No code path bypasses human review

Implementation: arbiter-engine/src/discovery/cli.rs | Issue: #48 | ADR: 017

Completing the remaining implementation gaps across ADRs 004, 005, 007, and 009 with thread-safe nonce management, risk manager actor, compensation executor, and key rotation support.

The Gap Analysis​

After implementing the core architecture, a review revealed several gaps between documented ADRs and actual implementation:

Nonce Management (ADR-004)​

Polymarket orders require monotonically increasing nonces. In a concurrent environment, this needs careful handling.

The Problem​

// WRONG: Race condition
let nonce = self.nonce + 1;
self.nonce = nonce; // Another thread could read same value

The Solution​

pub struct NonceManager {
    nonces: RwLock<HashMap<String, Arc<AtomicU64>>>,
}

impl NonceManager {
    pub async fn next_nonce(&self, address: &str) -> U256 {
        let address_lower = address.to_lowercase();

        // Get or create atomic counter for this address
        let counter = {
            let nonces = self.nonces.read().await;
            if let Some(counter) = nonces.get(&address_lower) {
                counter.clone()
            } else {
                drop(nonces);
                let mut nonces = self.nonces.write().await;
                let counter = Arc::new(AtomicU64::new(
                    Utc::now().timestamp_millis() as u64
                ));
                nonces.insert(address_lower.clone(), counter.clone());
                counter
            }
        };

        // Atomic increment - guaranteed unique
        U256::from(counter.fetch_add(1, Ordering::SeqCst))
    }
}

Key properties:

• Atomic increment: fetch_add is a single CPU instruction
• Case-insensitive: Ethereum addresses normalized to lowercase
• Timestamp initialization: Prevents collisions after restart

Risk Manager Actor (ADR-005)​

The actor model requires all state mutation through message passing. Risk checks are a natural fit.

Message Protocol​

pub enum RiskMessage {
    CheckRisk {
        user_id: UserId,
        opportunity: Opportunity,
        respond_to: oneshot::Sender<Result<(), RiskViolation>>,
    },
    RecordFill {
        user_id: UserId,
        fill: FillDetails,
    },
    // ... other messages
}

Actor Implementation​

impl RiskManagerActor {
    pub async fn run(mut self) {
        while let Some(msg) = self.receiver.recv().await {
            match msg {
                RiskMessage::CheckRisk { user_id, opportunity, respond_to } => {
                    let result = self.check_risk(&user_id, &opportunity);
                    let _ = respond_to.send(result);
                }
                RiskMessage::RecordFill { user_id, fill } => {
                    self.record_fill(&user_id, &fill);
                }
            }
        }
    }
}

Risk checks include:

• Open position limits (per-user, per-market)
• Exposure limits (max capital at risk)
• Daily loss limits with cooldown periods
• Order rate limiting

Compensation Executor (ADR-007)​

The saga pattern requires compensation when Leg 2 fails after Leg 1 succeeds.

Strategy Selection​

pub enum HedgeStrategy {
    Hold(String),        // Hold position, manual intervention
    DumpLeg1,            // Market sell Leg 1 immediately
    RetryLeg2,           // Retry original Leg 2
    LimitChaseLeg2,      // Chase price with limit orders
}

impl HedgeCalculator {
    pub fn select_strategy(
        leg1_fill: &FillDetails,
        leg2_intent: Option<&Leg2Intent>,
        retry_count: u32,
        config: &HedgeConfig,
    ) -> HedgeStrategy {
        match retry_count {
            0 => HedgeStrategy::RetryLeg2,
            1..=2 => HedgeStrategy::LimitChaseLeg2,
            _ if config.allow_market_fallback => HedgeStrategy::DumpLeg1,
            _ => HedgeStrategy::Hold("Max retries exceeded".into()),
        }
    }
}

Execution with Retries​

impl CompensationExecutor {
    pub async fn execute(&self, leg1_fill: &FillDetails, ...) -> CompensationResult {
        let mut retry_count = 0;

        loop {
            let strategy = HedgeCalculator::select_strategy(..., retry_count, ...);
            let hedge_order = HedgeCalculator::calculate(&strategy, leg1_fill);

            match self.execute_hedge_order(&hedge_order).await {
                Ok(fill) => return CompensationResult::Success(fill),
                Err(_) if retry_count < self.config.max_retries => {
                    retry_count += 1;
                    continue;
                }
                Err(e) => return CompensationResult::Failed { reason: e, ... },
            }
        }
    }
}

Key Rotation (ADR-009)​

Zero-downtime key rotation requires careful version management.

Rotation Workflow​

1. Add new key version (v2)
2. Activate v2 for new encryptions
3. Old credentials still decrypt with v1
4. Re-encrypt all credentials to v2
5. Retire v1 (disable for decrypt)
6. Remove v1

Implementation​

pub struct KeyRotationManager {
    stores: RwLock<HashMap<u32, Arc<CredentialStore>>>,
    versions: RwLock<HashMap<u32, KeyVersionInfo>>,
    active_version: RwLock<u32>,
}

impl KeyRotationManager {
    pub fn encrypt(&self, user_id: &str, credential_id: &str, plaintext: &[u8])
        -> Result<VersionedCredential, KeyRotationError>
    {
        let version = *self.active_version.read().unwrap();
        let store = self.stores.read().unwrap()
            .get(&version).cloned()
            .ok_or(KeyRotationError::NoKeysAvailable)?;

        let encrypted = store.encrypt(user_id, plaintext)?;

        Ok(VersionedCredential {
            key_version: version,
            encrypted,
            user_id: user_id.to_string(),
        })
    }

    pub fn decrypt_versioned(&self, versioned: &VersionedCredential)
        -> Result<Vec<u8>, KeyRotationError>
    {
        // Try recorded version first
        if let Some(store) = self.stores.read().unwrap().get(&versioned.key_version) {
            if let Ok(plaintext) = store.decrypt(&versioned.user_id, &versioned.encrypted) {
                return Ok(plaintext);
            }
        }

        // Try other active versions (migration fallback)
        for (&version, info) in self.versions.read().unwrap().iter() {
            if version == versioned.key_version || !info.active_for_decrypt {
                continue;
            }
            // ... try decrypt with other versions
        }

        Err(KeyRotationError::NoKeysAvailable)
    }
}

Security Scan Results​

All new code passed security scanning:

The unwrap() calls on RwLock are acceptable because:

1. They only fail if a thread panicked while holding the lock
2. At that point the system is already in a bad state
3. This is idiomatic Rust for lock acquisition

Test Coverage​

All implementations follow TDD with comprehensive tests:

test market::nonce::tests::test_concurrent_nonce_uniqueness ... ok
test actors::risk::tests::test_risk_check_within_limits ... ok
test execution::compensation::tests::test_compensation_retries ... ok
test security::key_rotation::tests::test_full_rotation_workflow ... ok

test result: ok. 198 passed; 0 failed

Conclusion​

Closing these gaps ensures the architecture matches documentation:

• ADR-004: Thread-safe nonce management prevents order collisions
• ADR-005: Risk actor enforces limits through message passing
• ADR-007: Compensation executor implements full hedge strategy suite
• ADR-009: Key rotation enables zero-downtime credential key changes

All changes tracked via GitHub issues #18-21 and verified by council review.

How we resolved an ADR naming conflict and established bidirectional traceability between requirements, decisions, and implementation.

Context​

Our docs/architecture/index.md contained a document titled "ADR-001: InertialEvent System Architecture" with 8 embedded sub-decisions (ADR-001.1 through ADR-001.8). This created several problems:

1. Naming conflict: docs/adrs/001-connectivity-check.md already existed as the "real" ADR-001
2. No traceability: These architectural decisions weren't tracked in the ledger
3. No spec mapping: Requirements didn't reference these ADRs
4. Discoverability: Decisions buried in a large document are hard to find

Decision​

We extracted the embedded decisions into standalone ADR files with a new numbering scheme:

Each standalone ADR includes:

• Full context and rationale
• Alternatives considered with verdict
• Consequences (positive, negative, neutral)
• Linked requirements (NFR-ARCH-*)
• References to related documentation

Implementation​

ADR Format​

Each extracted ADR follows this structure:

# ADR NNN: Title

## Status
Accepted

## Context
Why this decision was needed...

## Decision
What was decided and how...

## Alternatives Considered
| Approach | Pros | Cons | Verdict |
|----------|------|------|---------|
...

## Consequences
### Positive
### Negative
### Neutral

## References
- Links to related docs
- Linked Requirements (NFR-ARCH-*)

New Requirements​

We added NFR-ARCH-* requirements to the spec, each linking to its governing ADR:

- [ ] NFR-ARCH-001: Core engine in Rust - [ADR-004](https://github.com/amiable-dev/arbiter-bot/blob/cdfd9518694a96f67c7f7ff1599afba42bb25baf/docs/blog/adrs/004-rust-core-engine.md)
- [ ] NFR-ARCH-002: Actor model - [ADR-005](https://github.com/amiable-dev/arbiter-bot/blob/cdfd9518694a96f67c7f7ff1599afba42bb25baf/docs/blog/adrs/005-actor-model.md)
...

Traceability Matrix​

The ledger now tracks both ADR status and requirement implementation:

Architecture Document Update​

The architecture index was streamlined:

Before: 500+ lines with full decision content embedded After: ~150 lines with cross-references to standalone ADRs

Each section now links to its detailed ADR:

### Core Technology ([ADR-004](https://github.com/amiable-dev/arbiter-bot/blob/cdfd9518694a96f67c7f7ff1599afba42bb25baf/docs/blog/adrs/004-rust-core-engine.md))
**Decision:** Implement the trading core in Rust...

Verification​

1. Build passes: mkdocs build --strict
2. Navigation works: All 11 ADRs accessible from ADRs tab
3. Cross-references valid: Links between architecture doc and ADRs work
4. Ledger complete: All ADRs tracked with status
5. Requirements linked: NFR-ARCH-* documented in spec

Lessons Learned​

1. Flat numbering is cleaner - ADR-004 is easier to reference than ADR-001.4
2. Bidirectional links matter - ADRs reference requirements, requirements reference ADRs
3. Ledger as source of truth - Single place to check implementation status against decisions
4. Extract early - Embedded decisions are harder to find and maintain

The full ADR inventory is now available at ADRs Index.

How we built infrastructure-as-code with Terraform for deploying our trading system to AWS, including ECS Fargate, Aurora PostgreSQL, and ElastiCache Redis.

Why us-east-1?​

Both Polymarket and Kalshi have infrastructure in the US East region. Deploying our trading core to us-east-1 minimizes network latency for API calls and WebSocket connections.

Every millisecond matters when detecting and executing arbitrage opportunities.

Architecture Overview​

┌─────────────────────────────────────────────────────────────────┐
│                          us-east-1                               │
├─────────────────────────────────────────────────────────────────┤
│  ┌─────────────────┐                                            │
│  │   CloudFront    │                                            │
│  └────────┬────────┘                                            │
│           │                                                      │
│  ┌────────▼────────┐     ┌──────────────────────────────────┐  │
│  │       ALB       │     │         Private Subnets           │  │
│  │   (public)      │     │  ┌───────────┐  ┌────────────┐   │  │
│  └────────┬────────┘     │  │  Trading  │  │  Telegram  │   │  │
│           │              │  │   Core    │  │    Bot     │   │  │
│           │              │  │ (4 vCPU)  │  │ (0.5 vCPU) │   │  │
│           │              │  └─────┬─────┘  └──────┬─────┘   │  │
│  ┌────────▼────────┐     │        │               │         │  │
│  │    Web API      │     │        │   Service     │         │  │
│  │   (1 vCPU)      │◄────┼────────┤   Discovery   ├─────────│  │
│  │   x2 tasks      │     │        │               │         │  │
│  └─────────────────┘     │  ┌─────▼───────────────▼─────┐   │  │
│                          │  │      Aurora PostgreSQL      │   │  │
│                          │  │     (Serverless v2)         │   │  │
│                          │  └───────────────────────────┘   │  │
│                          │  ┌───────────────────────────┐   │  │
│                          │  │     ElastiCache Redis      │   │  │
│                          │  │       (Multi-AZ)           │   │  │
│                          │  └───────────────────────────┘   │  │
│                          └──────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘

Terraform Module Structure​

We organized infrastructure into reusable modules:

infrastructure/terraform/
├── main.tf           # Root module, wires everything together
├── variables.tf      # Input variables
├── outputs.tf        # Exported values
└── modules/
    ├── vpc/          # VPC, subnets, NAT gateways
    ├── ecs/          # ECS cluster, services, ALB
    ├── rds/          # Aurora PostgreSQL Serverless v2
    ├── elasticache/  # Redis cluster
    └── secrets/      # AWS Secrets Manager + KMS

VPC Module​

Multi-AZ setup with public and private subnets:

module "vpc" {
  source = "./modules/vpc"

  project_name       = var.project_name
  environment        = var.environment
  vpc_cidr           = "10.0.0.0/16"
  availability_zones = ["us-east-1a", "us-east-1b", "us-east-1c"]
}

Private subnets for ECS tasks, public subnets for ALB. NAT gateways enable outbound internet access for exchange APIs.

ECS Module​

Three services with different resource profiles:

Trading Core gets compute-optimized resources because it runs the hot loop:

resource "aws_ecs_task_definition" "trading_core" {
  family                   = "${local.name_prefix}-trading-core"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = 4096  # 4 vCPU
  memory                   = 8192  # 8 GB

  container_definitions = jsonencode([{
    name  = "trading-core"
    image = var.trading_core_image

    secrets = [
      { name = "POLY_PRIVATE_KEY", valueFrom = "..." },
      { name = "KALSHI_PRIVATE_KEY", valueFrom = "..." }
    ]
  }])
}

Secrets Management​

Credentials are stored in AWS Secrets Manager with KMS encryption:

resource "aws_kms_key" "secrets" {
  description             = "KMS key for secrets encryption"
  deletion_window_in_days = 30
  enable_key_rotation     = true
}

resource "aws_secretsmanager_secret" "exchange_credentials" {
  name       = "${local.name_prefix}/exchange-credentials"
  kms_key_id = aws_kms_key.secrets.arn
}

ECS tasks have IAM permissions to read secrets at startup. Secrets never touch disk.

Database: Aurora Serverless v2​

Auto-scaling PostgreSQL for variable workloads:

resource "aws_rds_cluster" "main" {
  cluster_identifier = "${local.name_prefix}-postgres"
  engine             = "aurora-postgresql"
  engine_mode        = "provisioned"
  engine_version     = "15.4"
  database_name      = "arbiter"

  serverlessv2_scaling_configuration {
    min_capacity = 0.5   # Scale to zero when idle
    max_capacity = 16    # Scale up under load
  }
}

Serverless v2 scales automatically based on load, reducing costs during low-activity periods.

GitHub Actions CI/CD​

Two workflows handle CI and deployment:

CI Workflow (ci.yml)​

Runs on every push:

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: cargo fmt --check
      - run: cargo clippy -- -D warnings

  test:
    runs-on: ubuntu-latest
    steps:
      - run: cargo test --all-features

  build:
    runs-on: ubuntu-latest
    steps:
      - run: cargo build --release

  security:
    runs-on: ubuntu-latest
    steps:
      - run: cargo audit

Deploy Workflow (deploy.yml)​

Triggered by version tags:

on:
  push:
    tags: ['v*']

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - name: Build and push images
        run: |
          docker build -t $ECR_REPO:$TAG ./arbiter-engine
          docker push $ECR_REPO:$TAG

      - name: Deploy infrastructure
        run: |
          cd infrastructure/terraform
          terraform init
          terraform apply -auto-approve

      - name: Update ECS services
        run: |
          aws ecs update-service --cluster $CLUSTER --service trading-core --force-new-deployment

Security Considerations​

Defense in depth: even if one layer is compromised, others provide protection.

Cost Optimization​

Production uses dedicated NAT gateways per AZ for high availability.

Verification​

# Validate Terraform configuration
terraform validate

# Plan changes
terraform plan -out=tfplan

# Apply infrastructure
terraform apply tfplan

# Verify services are running
aws ecs describe-services --cluster arbiter-prod-cluster

Lessons Learned​

1. Module everything - Reusable modules simplify multi-environment setups
2. Secrets rotation - Build in rotation from day one
3. Serverless v2 - Aurora's new mode is genuinely useful
4. Service discovery - ECS Cloud Map simplifies internal communication
5. Tag-based deploys - Version tags make rollback straightforward

The infrastructure supports the application's needs while remaining maintainable and cost-effective.

How we built a trading control plane with gRPC for programmatic access and Telegram for mobile-friendly monitoring.

The Interface Problem​

A trading bot needs multiple interaction modes:

No single interface serves all needs well. We implemented two complementary interfaces: gRPC for machines, Telegram for humans.

gRPC Service Layer​

gRPC provides strongly-typed, efficient communication for programmatic access.

Service Design​

We organized services by domain:

service UserService {
  rpc Authenticate(AuthRequest) returns (AuthResponse);
  rpc GetProfile(ProfileRequest) returns (ProfileResponse);
  rpc UpdateSettings(SettingsRequest) returns (SettingsResponse);
}

service TradingService {
  rpc GetPositions(PositionsRequest) returns (PositionsResponse);
  rpc PlaceOrder(OrderRequest) returns (OrderResponse);
  rpc CancelOrder(CancelRequest) returns (CancelResponse);
  rpc StreamPositions(PositionsRequest) returns (stream PositionUpdate);
}

service StrategyService {
  rpc ListStrategies(ListRequest) returns (StrategiesResponse);
  rpc EnableStrategy(StrategyRequest) returns (StrategyResponse);
  rpc DisableStrategy(StrategyRequest) returns (StrategyResponse);
  rpc GetArbOpportunities(ArbRequest) returns (stream ArbOpportunity);
}

Authentication​

JWT-based authentication with tier-aware authorization:

impl AuthInterceptor {
    pub fn verify(&self, request: &Request<()>) -> Result<UserContext, Status> {
        let token = request.metadata()
            .get("authorization")
            .ok_or(Status::unauthenticated("Missing token"))?;

        let claims = self.jwt_manager.verify(token)?;
        let context = self.get_user_context(claims.user_id)?;

        // Check rate limits
        context.check_api_rate().await?;

        Ok(context)
    }
}

Each request validates the JWT, loads the user context with their subscription tier, and checks rate limits before processing.

Streaming​

For real-time updates, gRPC streaming pushes position changes and arbitrage opportunities:

async fn stream_positions(
    &self,
    request: Request<PositionsRequest>,
) -> Result<Response<Self::StreamPositionsStream>, Status> {
    let user_ctx = self.auth.verify(&request)?;

    let (tx, rx) = mpsc::channel(32);

    // Subscribe to position updates for this user
    self.position_tracker.subscribe(user_ctx.user_id, tx);

    Ok(Response::new(ReceiverStream::new(rx)))
}

Telegram Bot​

Telegram provides instant mobile access without building a custom app.

Command Structure​

/start          - Link Telegram account to trading account
/status         - Current positions and P&L
/positions      - Detailed position list
/arb            - Active arbitrage opportunities
/copy <trader>  - Start copy trading
/stop           - Emergency stop all trading
/settings       - View/modify settings

Architecture​

The Telegram bot is a separate Python service that communicates with the Rust core via gRPC:

┌─────────────────┐     gRPC      ┌──────────────────┐
│  Telegram Bot   │◄────────────►│   Trading Core   │
│    (Python)     │               │     (Rust)       │
└─────────────────┘               └──────────────────┘
        │
        │ Telegram API
        ▼
┌─────────────────┐
│   Telegram      │
│   Servers       │
└─────────────────┘

Command Handler Pattern​

Commands follow a consistent pattern:

@bot.command("positions")
async def positions_handler(update: Update, context: Context):
    user_id = await get_linked_user(update.effective_user.id)
    if not user_id:
        return await update.message.reply_text("Link account with /start")

    try:
        positions = await grpc_client.get_positions(user_id)
        message = format_positions(positions)
        await update.message.reply_text(message, parse_mode="Markdown")
    except RateLimitError:
        await update.message.reply_text("Rate limited. Try again shortly.")

Security Considerations​

The /stop command requires explicit confirmation to prevent accidental triggers:

@bot.command("stop")
async def stop_handler(update: Update, context: Context):
    # Require explicit confirmation
    if not context.args or context.args[0] != "CONFIRM":
        return await update.message.reply_text(
            "This will stop ALL trading.\n"
            "Type /stop CONFIRM to proceed."
        )

    await grpc_client.emergency_stop(user_id)
    await update.message.reply_text("Trading stopped.")

Test Coverage​

The Telegram bot uses python-telegram-bot's testing utilities for isolated command handler tests.

Why Two Interfaces?​

A REST API could serve both use cases, but:

1. gRPC streaming - Real-time updates without polling
2. Telegram familiarity - Users already have it installed
3. Push notifications - Telegram handles delivery
4. No app maintenance - Telegram updates their client

The dual-interface approach serves different needs without compromising either.

Lessons Learned​

1. Separate concerns - Bot logic separate from trading core
2. Test command handlers - Telegram bots can be tested
3. Rate limit at the core - Not the interface layer
4. Confirmation for destructive actions - Prevent accidents

The control interface transforms the bot from a black box into a manageable system.

How we implemented AES-256-GCM encryption with HKDF key derivation for secure credential storage, including memory safety with zeroize.

The Threat Model​

Trading bots hold sensitive credentials: exchange API keys, private keys for signing, and secrets. If an attacker gains read access to the system, they shouldn't be able to extract usable credentials.

Our defense-in-depth strategy:

1. Encryption at rest - Credentials encrypted with AES-256-GCM
2. Key separation - Per-user derived keys via HKDF
3. Memory safety - Sensitive data zeroized on drop
4. Tamper detection - GCM authentication tag prevents modification

Key Hierarchy​

Master Key (from AWS Secrets Manager)
    └── User Key (HKDF derived with user_id as info)
            └── Credential (encrypted with user key)

The master key never encrypts data directly. HKDF derives user-specific keys, so compromising one user's credentials doesn't affect others.

Implementation​

Key Derivation​

We use HKDF-SHA256 for key derivation:

fn derive_user_key(&self, user_id: &str) -> Result<DerivedKey, CredentialError> {
    let hk = Hkdf::<Sha256>::new(Some(&self.salt), &self.master_key.0);

    let mut okm = [0u8; KEY_SIZE];
    hk.expand(user_id.as_bytes(), &mut okm)?;

    Ok(DerivedKey(okm))
}

The salt is random per store instance. Combined with user_id in the info parameter, this ensures each user gets a unique encryption key.

Encryption​

AES-256-GCM provides authenticated encryption:

pub fn encrypt(&self, user_id: &str, plaintext: &[u8]) -> Result<EncryptedCredential, CredentialError> {
    let user_key = self.derive_user_key(user_id)?;
    let cipher = Aes256Gcm::new_from_slice(&user_key.0)?;

    // Random nonce per encryption
    let mut nonce_bytes = [0u8; NONCE_SIZE];
    OsRng.fill_bytes(&mut nonce_bytes);
    let nonce = Nonce::from_slice(&nonce_bytes);

    let ciphertext = cipher.encrypt(nonce, plaintext)?;

    Ok(EncryptedCredential { nonce: nonce_bytes, ciphertext })
}

Each encryption uses a fresh random nonce. Even encrypting the same credential twice produces different ciphertext.

Memory Safety​

The zeroize crate ensures sensitive data is wiped when no longer needed:

#[derive(Zeroize, ZeroizeOnDrop)]
struct DerivedKey([u8; KEY_SIZE]);

This prevents secrets from lingering in memory after use, reducing the window for memory-scanning attacks.

Verification​

Our test suite validates security properties:

Example: verifying that tampering fails authentication:

#[test]
fn test_tampered_ciphertext_fails() {
    let store = CredentialStore::with_salt(&test_master_key(), test_salt()).unwrap();
    let mut encrypted = store.encrypt("user1", b"secret").unwrap();

    // Tamper with ciphertext
    encrypted.ciphertext[0] ^= 0xFF;

    // Decryption should fail due to authentication
    let result = store.decrypt("user1", &encrypted);
    assert!(result.is_err());
}