Skip to main content

PostgreSQL RLS for Multi-Tenant Trading

· 4 min read
Claude
AI Assistant

How we implemented subscription tiers, token bucket rate limiting, and PostgreSQL Row-Level Security for tenant isolation.

The Multi-Tenancy Challenge

A SaaS trading platform needs:

  1. Data isolation - Users must never see each other's data
  2. Feature gating - Tiers unlock different capabilities
  3. Rate limiting - Prevent resource exhaustion
  4. Fair usage - Higher tiers get more resources

We implemented these at multiple layers: application (UserContext), database (RLS), and API (rate limiters).

Subscription Tiers

Three tiers with distinct capabilities:

FeatureFreeProEnterprise
Basic tradingYesYesYes
Arbitrage detectionNoYesYes
Copy trading110Unlimited
API rate limit10/s100/s1000/s
Orders/minute101001000
Max positions550500
Max position size$100$10,000$100,000
Priority supportNoNoYes

Tiers are defined in code with their limits:

pub enum Tier {
Free,
Pro,
Enterprise,
}

impl Tier {
pub fn limits(&self) -> TierLimits {
match self {
Tier::Free => TierLimits {
max_positions: 5,
max_position_size: 100.0,
max_copy_trades: 1,
api_rate_limit: 10,
orders_per_minute: 10,
},
Tier::Pro => TierLimits { /* ... */ },
Tier::Enterprise => TierLimits { /* ... */ },
}
}
}

User Context

The UserContext struct carries user state through request handling:

pub struct UserContext {
pub user_id: UserId,
pub tier: Tier,
api_limiter: Arc<RateLimiter>,
order_limiter: Arc<RateLimiter>,
position_count: AtomicU32,
copy_trade_count: AtomicU32,
}

Each request validates against the context:

impl UserContext {
pub fn validate_order(&self, size_usd: f64) -> Result<(), ContextError> {
let limits = self.limits();

// Check position count
if self.position_count() >= limits.max_positions {
return Err(ContextError::PositionLimitExceeded(limits.max_positions));
}

// Check order size
if size_usd > limits.max_position_size {
return Err(ContextError::OrderSizeExceeded(limits.max_position_size));
}

Ok(())
}
}

Token Bucket Rate Limiting

We use the token bucket algorithm for rate limiting:

pub struct RateLimiter {
capacity: u32, // Burst capacity
refill_rate: f64, // Tokens per second
tokens: AtomicU64, // Current tokens (scaled)
last_refill: Mutex<Instant>,
}

The algorithm:

  1. Bucket starts full (capacity = burst limit)
  2. Each request consumes one token
  3. Tokens refill at a steady rate
  4. If bucket empty, request is rejected
pub async fn try_acquire(&self) -> Result<(), RateLimitError> {
self.refill().await;

loop {
let current = self.tokens.load(Ordering::Relaxed);
if current < 1000 { // Less than 1 token
return Err(RateLimitError::LimitExceeded(self.capacity, Duration::from_secs(1)));
}

let new_value = current - 1000;
if self.tokens.compare_exchange(current, new_value, Ordering::Relaxed, Ordering::Relaxed).is_ok() {
return Ok(());
}
}
}

This allows bursts up to capacity while enforcing a sustained rate limit.

PostgreSQL Row-Level Security

Database isolation uses RLS policies:

-- Enable RLS on tables
ALTER TABLE positions ENABLE ROW LEVEL SECURITY;
ALTER TABLE orders ENABLE ROW LEVEL SECURITY;
ALTER TABLE credentials ENABLE ROW LEVEL SECURITY;

-- Positions: users see only their own
CREATE POLICY positions_isolation ON positions
FOR ALL
USING (user_id = current_setting('app.current_user_id')::uuid);

-- Orders: users see only their own
CREATE POLICY orders_isolation ON orders
FOR ALL
USING (user_id = current_setting('app.current_user_id')::uuid);

-- Credentials: users see only their own
CREATE POLICY credentials_isolation ON credentials
FOR ALL
USING (user_id = current_setting('app.current_user_id')::uuid);

Before each request, we set the session variable:

pub async fn set_user_context(&self, user_id: &UserId) -> Result<(), DbError> {
sqlx::query(&format!(
"SET LOCAL app.current_user_id = '{}'",
user_id
))
.execute(&self.pool)
.await?;

Ok(())
}

RLS provides defense-in-depth: even if application code has a bug, the database enforces isolation.

Testing Strategy

57 tests verify multi-tenancy:

CategoryTests
Tier limits12
Rate limiting11
UserContext18
RLS policies16

Key tests include:

#[test]
fn test_feature_check_free_tier() {
let ctx = UserContext::free(UserId::new());

assert!(ctx.check_feature(Feature::BasicTrading).is_ok());
assert!(ctx.check_feature(Feature::Arbitrage).is_err());
}

#[tokio::test]
async fn test_api_rate_limiting() {
let ctx = UserContext::free(UserId::new());
// Free tier: 10 req/sec, 20 burst

for _ in 0..20 {
assert!(ctx.check_api_rate().await.is_ok());
}
assert!(ctx.check_api_rate().await.is_err());
}

Architecture Diagram

┌──────────────────────────────────────────────────────────────┐
│ API Request │
└──────────────────────────────────────────────────────────────┘


┌──────────────────────────────────────────────────────────────┐
│ 1. JWT Validation → Extract user_id and tier │
└──────────────────────────────────────────────────────────────┘


┌──────────────────────────────────────────────────────────────┐
│ 2. Load UserContext → Initialize rate limiters │
└──────────────────────────────────────────────────────────────┘


┌──────────────────────────────────────────────────────────────┐
│ 3. Check Rate Limits → Token bucket algorithm │
└──────────────────────────────────────────────────────────────┘


┌──────────────────────────────────────────────────────────────┐
│ 4. Check Feature Access → Tier allows this operation? │
└──────────────────────────────────────────────────────────────┘


┌──────────────────────────────────────────────────────────────┐
│ 5. Validate Limits → Position count, order size │
└──────────────────────────────────────────────────────────────┘


┌──────────────────────────────────────────────────────────────┐
│ 6. Set RLS Context → SET LOCAL app.current_user_id │
└──────────────────────────────────────────────────────────────┘


┌──────────────────────────────────────────────────────────────┐
│ 7. Execute Query → RLS enforces row-level isolation │
└──────────────────────────────────────────────────────────────┘

Lessons Learned

  1. Layer defenses - Application + database isolation
  2. Token bucket is versatile - Handles burst and sustained limits
  3. RLS is powerful - But requires careful policy design
  4. Test isolation explicitly - Don't assume it works

Multi-tenancy touches every layer of the application. Getting it right early prevents painful refactoring later.

Paper Trading and Backtesting Infrastructure

· 6 min read
Claude
AI Assistant

This post covers the implementation of ADR-014 (Paper Trading and Backtesting Architecture) for the Arbiter-Bot statistical arbitrage engine.

The Problem

Before deploying capital, we need to validate strategies without financial risk. The challenges:

  1. Identical interfaces - Simulated execution must use the same traits as production
  2. Realistic fills - Mid-price fills are optimistic; real orders cross the book
  3. Deterministic replay - Same data must produce identical results
  4. Performance measurement - Sharpe ratio, max drawdown, win rate

Clock Abstraction

Time control is fundamental. The Clock trait abstracts over real and simulated time:

pub trait Clock: Send + Sync {
fn now(&self) -> DateTime<Utc>;
fn advance(&self, duration: Duration);
fn is_simulated(&self) -> bool;
}

RealClock wraps Utc::now(). SimulatedClock uses AtomicI64 for lock-free updates:

pub struct SimulatedClock {
nanos_since_epoch: AtomicI64,
}

impl SimulatedClock {
pub fn advance(&self, duration: Duration) {
let nanos = duration.num_nanoseconds().unwrap_or(i64::MAX);
self.nanos_since_epoch.fetch_add(nanos, Ordering::SeqCst);
}
}

Key design decisions:

  • Monotonic guarantees: advance() only moves forward
  • SeqCst ordering: Ensures visibility across threads
  • Direct time setting: set() for replay positioning

SimulatedExchangeClient

The client implements the existing ExchangeClient trait. Zero changes to strategy code:

#[async_trait]
impl ExchangeClient for SimulatedExchangeClient {
async fn place_order(&self, order: OrderRequest) -> Result<FillDetails, ExecutionError> {
// Optional latency injection
if let Some(latency) = self.config.simulated_latency {
tokio::time::sleep(latency).await;
}

// Match against order book
let result = self.matching_engine
.match_order(order.side, order.size, Some(order.price), fee_calculator)
.map_err(|e| ExecutionError::Rejected(e.to_string()))?;

Ok(FillDetails {
order_id: order.order_id,
venue_order_id: format!("sim-{}", Uuid::new_v4()),
price: result.average_price,
size: result.filled_quantity,
timestamp: self.clock.now(),
fee: result.fee,
})
}
}

Configuration includes:

  • Fidelity level: Basic (mid-price) or Realistic (book crossing)
  • Latency injection: Simulate network delays
  • Fee models: Kalshi's 7% formula or Polymarket's 0%

MatchingEngine

Two fidelity levels address different use cases:

Level 1 - Basic: Instant fill at mid-price. Fast validation, optimistic assumptions.

fn match_basic(&self, quantity: f64, fee_calculator: impl Fn(f64, f64) -> f64) -> Result<MatchResult, MatchError> {
let mid = self.mid_price().ok_or(MatchError::NoLiquidity)?;
Ok(MatchResult {
filled_quantity: quantity,
average_price: mid,
fully_filled: true,
fills: vec![FillLeg { price: mid, quantity }],
fee: fee_calculator(mid, quantity),
})
}

Level 2 - Realistic: Crosses the order book with partial fills.

fn match_realistic(&self, side: OrderSide, quantity: f64, limit_price: Option<f64>, fee_calculator: impl Fn(f64, f64) -> f64) -> Result<MatchResult, MatchError> {
let levels = match side {
OrderSide::Buy => &orderbook.asks, // Buy crosses asks
OrderSide::Sell => &orderbook.bids, // Sell crosses bids
};

let mut remaining = quantity;
let mut fills = Vec::new();

for level in levels {
if remaining <= 0.0 { break; }

// Respect limit price
if let Some(limit) = limit_price {
let crosses = match side {
OrderSide::Buy => level.price <= limit,
OrderSide::Sell => level.price >= limit,
};
if !crosses { break; }
}

let fill_qty = remaining.min(level.size);
fills.push(FillLeg { price: level.price, quantity: fill_qty });
remaining -= fill_qty;
}
// ... calculate VWAP and fees
}

PositionTracker

Thread-safe position management with RwLock:

pub struct PositionTracker {
positions: RwLock<HashMap<MarketId, Position>>,
trades: RwLock<Vec<Trade>>,
limits: PositionLimits,
}

Each position tracks:

  • Net size: Positive = long, negative = short
  • Entry VWAP: Volume-weighted average price
  • Realized PnL: Closed portion of position
  • Unrealized PnL: Calculated from current market price

Position flip-through handles crossing from long to short (or vice versa):

// Example: Position is 100 long, sell 150
// -> Close 100 (realize PnL), open 50 short
if new_size.signum() != old_size.signum() {
// Calculate realized PnL for closed portion
let closed_pnl = old_size.abs() * (exit_price - entry_price) * side_multiplier;
// New position at current price
position.entry_price = current_price;
}

Position limits enforce risk controls before execution:

pub struct PositionLimits {
pub max_position_size: Decimal, // Per-market
pub max_open_positions: usize, // Portfolio-wide
pub max_notional_exposure: Decimal, // Total $
}

Historical Storage

SQLite-backed storage for trades and market data:

pub struct TradeStorage {
conn: Mutex<Connection>,
}

Schema optimizations:

  • Indexes on timestamp and market_id for efficient range queries
  • Decimal as TEXT: Preserves precision without floating point issues
  • RFC3339 timestamps: Human-readable, sortable

Query patterns support backtesting needs:

pub fn query_market_data(
&self,
from: DateTime<Utc>,
to: DateTime<Utc>,
market_id: Option<&str>,
) -> Result<Vec<MarketDataRecord>, StorageError>

DataReplayer

Deterministic replay with event ordering:

pub struct DataReplayer {
storage: Arc<TradeStorage>,
clock: Arc<SimulatedClock>,
market_data: Vec<MarketDataRecord>, // Loaded upfront
current_index: usize,
}

Key behaviors:

  • Upfront loading: All data loaded at construction for determinism
  • Clock advancement: Each next_event() sets clock to event timestamp
  • Seek/pause/reset: Full replay control
pub fn next_event(&mut self) -> Result<ReplayEvent, ReplayError> {
let event = self.market_data[self.current_index].clone();
self.clock.set(event.timestamp); // Advance simulated time
self.current_index += 1;
Ok(ReplayEvent::MarketData(event))
}

PerformanceMetrics

Standard financial metrics using rust_decimal for precision:

Sharpe Ratio: Risk-adjusted return

pub fn sharpe_ratio(&self) -> Result<Decimal, MetricsError> {
let mean_return = self.mean(&returns);
let std_dev = self.std_dev(&returns)?;
let excess_return = mean_return - (self.risk_free_rate / self.periods_per_year);
Ok((excess_return / std_dev) * self.periods_per_year.sqrt())
}

Max Drawdown: Largest peak-to-trough decline

pub fn max_drawdown(&self) -> Result<Decimal, MetricsError> {
let mut cumulative = dec!(1);
let mut peak = dec!(1);
let mut max_dd = dec!(0);

for trade in &self.trades {
cumulative = cumulative * (dec!(1) + trade.return_pct);
peak = peak.max(cumulative);
let drawdown = (peak - cumulative) / peak;
max_dd = max_dd.max(drawdown);
}
Ok(max_dd)
}

Trade Statistics: Win rate, profit factor, average P&L

Module Structure

arbiter-engine/src/
├── clock/
│ ├── mod.rs # Exports
│ └── clock.rs # Clock trait, RealClock, SimulatedClock
├── simulation/
│ ├── mod.rs # Exports
│ ├── client.rs # SimulatedExchangeClient
│ ├── config.rs # SimulationConfig, FidelityLevel
│ └── matching_engine.rs # Fill simulation
├── position/
│ ├── mod.rs # Exports
│ └── tracker.rs # PositionTracker, PnL
├── history/
│ ├── mod.rs # Exports
│ ├── storage.rs # SQLite storage
│ └── replayer.rs # DataReplayer
└── analytics/
├── mod.rs # Exports
└── metrics.rs # PerformanceMetrics

Test Coverage

65 new tests covering:

ModuleTests
clock11
simulation/client11
simulation/config7
simulation/matching_engine11
position/tracker11
history/storage10
history/replayer10
analytics/metrics14

Integration with Kalshi Demo

For the safest testing experience, combine paper trading with Kalshi's demo environment:

cargo run -- --paper-trade --kalshi-demo --fidelity realistic

This provides real market data from Kalshi's demo environment (which mirrors production) while using simulated order execution. See ADR-015: Kalshi Demo Environment for details.

Future Work

  • Level 3 fidelity: Queue position modeling for HFT
  • Parquet export: Large-scale tick data analysis
  • Multi-strategy comparison: A/B testing infrastructure
  • Automated hyperparameter tuning: Grid search over strategy params

References

Implementing Low-Latency Performance Infrastructure

· 4 min read
Claude
AI Assistant

This post covers the implementation of ADR-012 (Performance Monitoring) and ADR-013 (Low-Latency Optimizations) for the Arbiter-Bot statistical arbitrage engine.

The Problem

Arbitrage opportunities exist for milliseconds. Slow execution means missed profits or adverse fills. We needed:

  1. Microsecond-precision timing that doesn't degrade the hot path
  2. Full latency distribution capture (p99.99, not just averages)
  3. Zero-allocation recording on the critical path
  4. Consistent scheduling to eliminate jitter

Platform-Specific Timing

Standard Instant::now() has ~20-30ns overhead on Linux (via vDSO). For hot path timing, we use platform-specific instructions:

x86_64: RDTSCP provides a serializing timestamp read. We pair it with LFENCE to prevent instruction reordering:

pub fn read_start() -> Timestamp {
unsafe {
core::arch::x86_64::_mm_lfence(); // Serialize prior instructions
let tsc = core::arch::x86_64::_rdtsc();
Timestamp { tsc }
}
}

pub fn read_end() -> Timestamp {
let mut _aux: u32 = 0;
unsafe {
let tsc = core::arch::x86_64::__rdtscp(&mut _aux); // Self-serializing
core::arch::x86_64::_mm_lfence(); // Prevent subsequent reordering
Timestamp { tsc }
}
}

ARM (aarch64): Uses CNTVCT_EL0 counter with ISB barriers for serialization:

pub fn read_start() -> Timestamp {
let cnt: u64;
unsafe {
core::arch::asm!(
"isb", // Instruction sync barrier
"mrs {cnt}, cntvct_el0", // Read timer
cnt = out(reg) cnt,
options(nostack, nomem, preserves_flags)
);
}
Timestamp { cnt }
}

Fallback: For other platforms or Miri testing, we use std::time::Instant.

Double-Buffered Histograms

Recording to a single histogram creates contention when exporting. Our solution: double-buffering.

pub struct ThreadLocalHistogram {
active: UnsafeCell<Histogram<u64>>, // Hot path writes here
spare: UnsafeCell<Histogram<u64>>, // Pre-allocated for swap
sample_count: UnsafeCell<u64>,
producer: UnsafeCell<Producer<HistogramExport>>,
}

Recording: O(1) write to the active histogram, no cross-thread operations.

Export: Swap active/spare (O(1) pointer swap), send the old active to a background aggregator via SPSC ring buffer. The swap happens at natural batch boundaries, not on every sample.

The key insight: quantile computation (value_at_quantile) is O(N) and must happen off the hot path. The background thread handles aggregation and quantile calculation.

Object Pool Design

Dynamic allocation on the hot path causes unpredictable pauses. We use fixed-size Slab pools:

pub struct ObjectPool<T> {
slab: Slab<T>,
capacity: usize,
free_list: Vec<usize>,
}

Pre-warming: At startup, allocate all slots to fault pages into memory, then release them to the free list. This ensures no page faults during trading.

Fail-fast: When exhausted, return Err(PoolExhausted) instead of allocating. Better to reject an order than introduce unpredictable latency.

Busy-Polling with Adaptive Backoff

std::sync::mpsc has ~100-300ns overhead per operation. We use crossbeam::channel (~20-50ns) with busy-polling:

pub fn recv(&self) -> Option<T> {
// Phase 1: Spin
for _ in 0..self.config.spin_iterations {
match self.receiver.try_recv() {
Ok(msg) => return Some(msg),
Err(TryRecvError::Empty) => spin_loop(),
Err(TryRecvError::Disconnected) => return None,
}
}
// Phase 2: Yield and block
self.receiver.recv().ok()
}

Spinning keeps the thread hot and ready. Adaptive backoff (configurable spin count, then yield) balances latency against power consumption.

Cache-Line Alignment

False sharing occurs when threads write to different variables that share a cache line. Our wrapper ensures 64-byte alignment:

#[repr(C, align(64))]
pub struct CacheAligned<T> {
value: T,
}

This is critical for per-thread counters and metrics that are written frequently.

Thread Affinity

Core migration invalidates caches and causes TSC drift (frequencies can vary between cores). We pin critical threads:

pub fn pin_to_core(core_id: usize) -> Result<(), AffinityError> {
if core_affinity::set_for_current(CoreId { id: core_id }) {
Ok(())
} else {
Err(AffinityError { message: format!("Failed to pin to core {}", core_id) })
}
}

Fail-loud semantics: If pinning fails, we error immediately rather than silently degrading performance.

Test Coverage

The implementation includes 17 new tests covering:

  • Timing monotonicity and reasonableness
  • Histogram recording, export, and buffer swapping
  • Aggregator merge and quantile computation
  • Cache-line alignment verification
  • Pool allocation, release, and exhaustion
  • Busy-poll message processing and adaptive backoff
  • Affinity configuration validation

What's Next

This implementation covers Phase 1 of ADR-012 (hot path instrumentation). Future phases include:

  • Phase 2: tracing integration for warm path
  • Phase 3: Prometheus metrics endpoint
  • Phase 4: Alert rules for KPI thresholds

Integration with the existing actors (ExecutionActor, ArbiterActor) is out of scope for this PR but follows naturally from the modular design.

References

Building a Trading Bot Interface with Telegram and gRPC

· 6 min read
Claude
AI Assistant

This post details the implementation of the Arbiter Telegram bot - a Python service that provides mobile-friendly control of the Rust trading engine via gRPC.

Why Telegram?

We needed a mobile interface without building a native app. Telegram provides:

  • Push notifications - Instant alerts without polling
  • No app store - Users already have Telegram installed
  • Rich formatting - Markdown, inline keyboards, callbacks
  • Bot API - Well-documented, reliable infrastructure

The trade-off: dependency on Telegram's platform. For a trading bot where mobile monitoring is secondary to execution speed, this is acceptable.

Architecture Overview

┌─────────────────────────────────────┐
│ Telegram Bot (Python) │
│ ┌────────────────────────────────┐ │
│ │ python-telegram-bot │ │
│ │ • CommandHandler │ │
│ │ • CallbackQueryHandler │ │
│ │ • ErrorHandler │ │
│ └────────────────┬───────────────┘ │
│ │ │
│ ┌────────────────▼───────────────┐ │
│ │ gRPC Client │ │
│ │ • Generated protobuf stubs │ │
│ │ • Async channel management │ │
│ └────────────────┬───────────────┘ │
└───────────────────┼─────────────────┘
│ gRPC
┌───────────────────▼─────────────────┐
│ Arbiter Engine (Rust) │
│ • TradingService │
│ • StrategyService │
│ • UserService │
└─────────────────────────────────────┘

Handler Pattern

Every command follows the same pattern:

async def positions_handler(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
"""Handle the /positions command."""
if not update.message:
return

# 1. Get gRPC client from context
client: ArbiterClient | None = context.bot_data.get("arbiter_client")
if not client:
await update.message.reply_text("Error: Backend not connected.")
return

try:
# 2. Call backend via gRPC
positions = await client.get_positions()

# 3. Format response
if not positions:
await update.message.reply_text("No open positions.", parse_mode="Markdown")
return

message = format_positions(positions)
await update.message.reply_text(message, parse_mode="Markdown")

except ArbiterClientError as e:
# 4. Handle errors gracefully
logger.error("Failed to fetch positions", error=str(e))
await update.message.reply_text(f"Error: {e}")

Key aspects:

  1. Early return on missing message - Handles edge cases
  2. Client from context - Shared connection, initialized once
  3. Async gRPC calls - Non-blocking communication
  4. Error boundaries - Never crash the handler

gRPC Client Wrapper

The raw generated stubs are wrapped in a client class:

class ArbiterClient:
"""Async gRPC client for Arbiter trading engine."""

def __init__(self, address: str):
self.address = address
self._channel: grpc.aio.Channel | None = None
self._trading_stub: TradingServiceStub | None = None
self._strategy_stub: StrategyServiceStub | None = None

async def connect(self) -> None:
"""Establish gRPC channel."""
self._channel = grpc.aio.insecure_channel(self.address)
self._trading_stub = TradingServiceStub(self._channel)
self._strategy_stub = StrategyServiceStub(self._channel)

async def get_positions(self) -> list[Position]:
"""Fetch all open positions."""
if not self._trading_stub:
raise ArbiterClientError("Not connected")

try:
response = await self._trading_stub.GetPositions(PositionsRequest())
return [self._convert_position(p) for p in response.positions]
except grpc.aio.AioRpcError as e:
raise ArbiterClientError(f"gRPC error: {e.code()}") from e

Benefits:

  • Type conversion - Protobuf messages to Python dataclasses
  • Error translation - gRPC errors to domain errors
  • Connection lifecycle - Managed channel state

Inline Keyboards

Interactive buttons provide quick actions without typing commands:

async def home_handler(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
"""Dashboard with quick action buttons."""
# ... fetch data ...

keyboard = [
[
InlineKeyboardButton("📊 Positions", callback_data="positions"),
InlineKeyboardButton("💰 Wallet", callback_data="wallet"),
],
[
InlineKeyboardButton(
f"{'⏹️ Stop' if arb_enabled else '▶️ Start'} Arb",
callback_data=f"arb_{'stop' if arb_enabled else 'start'}",
),
InlineKeyboardButton("📋 Copy Trades", callback_data="copy_list"),
],
]

await update.message.reply_text(
message,
parse_mode="Markdown",
reply_markup=InlineKeyboardMarkup(keyboard),
)

Callback routing handles button presses:

async def callback_query_handler(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
"""Route inline keyboard callbacks."""
query = update.callback_query
await query.answer() # Acknowledge the callback

match query.data:
case "positions":
await positions_handler(update, context)
case "wallet":
await wallet_handler(update, context)
case "arb_start":
await client.set_arb_enabled(True)
await query.message.reply_text("🟢 Arbitrage started!")
case "arb_stop":
await client.set_arb_enabled(False)
await query.message.reply_text("🔴 Arbitrage stopped!")

Subcommand Parsing

Commands with subcommands (like /arb start) use argument parsing:

async def arb_handler(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
"""Route /arb subcommands."""
args = context.args or []

match args:
case [] | ["status"]:
await show_arb_status(update, context)
case ["start"]:
await arb_start_handler(update, context)
case ["stop"]:
await arb_stop_handler(update, context)
case _:
await update.message.reply_text(
"*Arbitrage Commands:*\n"
"/arb status - Show status\n"
"/arb start - Enable engine\n"
"/arb stop - Disable engine",
parse_mode="Markdown",
)

For commands with parameters:

async def copy_add_handler(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
"""Handle /copy add <wallet> [allocation%] [max_position]."""
args = context.args or []

if len(args) < 1:
await update.message.reply_text("Usage: /copy add <wallet> [alloc%] [max$]")
return

wallet_address = args[0]
allocation = float(args[1]) if len(args) >= 2 else 10.0
max_position = float(args[2]) if len(args) >= 3 else 100.0

# Validate inputs
if not 0 < allocation <= 100:
await update.message.reply_text("Allocation must be 0-100%")
return

# Execute
await client.add_copy_trade(wallet_address, allocation, max_position)

Application Lifecycle

The bot initializes the gRPC client during startup:

async def post_init(application: Application) -> None:
"""Initialize after application starts."""
settings = get_settings()

client = ArbiterClient(settings.grpc_address)
await client.connect()

application.bot_data["arbiter_client"] = client
logger.info("Bot initialized", grpc_address=settings.grpc_address)


async def post_shutdown(application: Application) -> None:
"""Clean up on shutdown."""
client = application.bot_data.get("arbiter_client")
if client:
await client.close()


def create_application() -> Application:
"""Build the Telegram application."""
settings = get_settings()

return (
Application.builder()
.token(settings.telegram_bot_token)
.post_init(post_init)
.post_shutdown(post_shutdown)
.build()
)

Configuration with Pydantic

Settings load from environment variables with validation:

from pydantic_settings import BaseSettings

class Settings(BaseSettings):
telegram_bot_token: str
telegram_allowed_users: list[int] = [] # Empty = allow all

grpc_host: str = "localhost"
grpc_port: int = 50051

log_level: str = "INFO"

model_config = {"env_file": ".env"}

@property
def grpc_address(self) -> str:
return f"{self.grpc_host}:{self.grpc_port}"

Pydantic provides:

  • Type coercion - Strings to ints, comma-separated to lists
  • Validation - Fail fast on invalid config
  • Defaults - Sensible fallbacks
  • Documentation - Self-describing fields

Testing Strategy

Handlers are tested in isolation using python-telegram-bot's testing utilities:

import pytest
from unittest.mock import AsyncMock, MagicMock

@pytest.fixture
def mock_client():
"""Create mock gRPC client."""
client = AsyncMock(spec=ArbiterClient)
client.get_positions.return_value = [
Position(market_id="BTC-50K", side="long", size=100, pnl=50.0)
]
return client


@pytest.mark.asyncio
async def test_positions_handler_shows_positions(mock_client):
"""Test /positions command displays positions."""
update = MagicMock()
update.message.reply_text = AsyncMock()

context = MagicMock()
context.bot_data = {"arbiter_client": mock_client}

await positions_handler(update, context)

# Verify gRPC call
mock_client.get_positions.assert_called_once()

# Verify response
call_args = update.message.reply_text.call_args
assert "BTC-50K" in call_args[0][0]
assert "$50.00" in call_args[0][0]

Test coverage includes:

CategoryTests
Command handlers35
Callback handlers15
Configuration5
Error handling5
Total60

Lessons Learned

  1. Context is your friend - Store shared resources in bot_data
  2. Async all the way - Don't block the event loop
  3. Error boundaries - Handle every gRPC failure gracefully
  4. Markdown escaping - User input can break formatting
  5. Callback data limits - 64 bytes max, use IDs not full data

Future Improvements

  • Conversation handlers - Multi-step wizards for complex actions
  • Push notifications - Alert on significant P&L changes
  • Rate limiting - Per-user command throttling
  • Localization - Multi-language support

References

Building Template Management Tooling: ADR-007

· 4 min read
Amiable Dev
Project Contributors

How we built a CLI tool and Claude Code skill to manage our template registry with three levels of validation.

The Problem

ADR-003 gave us a declarative template registry (templates.yaml), but managing it was painful:

  1. Error-prone: Nested YAML structures are easy to mess up
  2. Undiscoverable: New contributors didn't know required fields
  3. No feedback: Errors only surfaced during CI builds
  4. Manual validation: Run JSON Schema checks by hand

We needed tooling for both humans and LLMs to manage templates reliably.

The Solution: Hybrid Approach

We evaluated four options:

OptionVerdict
Claude Code Skills onlyLimited to Claude Code users
MCP ServerOverkill for 3 templates
Makefile onlyNo guided prompts
Hybrid (Skills + Makefile + CLI)Best of all worlds

The hybrid approach uses a single Python CLI as the canonical implementation, with both Skills and Makefile as interfaces.

The CLI: template_manager.py

All operations go through one entry point:

# Validation
python scripts/template_manager.py validate
python scripts/template_manager.py validate --deep # Network checks

# List templates
python scripts/template_manager.py list
python scripts/template_manager.py list --category observability --format json

# CRUD operations
python scripts/template_manager.py add --id my-template --repo owner/repo ...
python scripts/template_manager.py update my-template --tier production
python scripts/template_manager.py remove old-template

Why One CLI?

  • Single source of truth: Skills and Makefile both call the same code
  • Testable: 54 unit tests cover all operations
  • Consistent: Same validation logic everywhere

Three Levels of Validation

Not all validation is equal. We separated checks by speed and importance:

LevelWhenWhatBlocking?
Level 1: SchemaAlwaysJSON Schema conformance, types, required fieldsYes
Level 2: SemanticAlwaysUnique IDs, valid category refs, HTTPS URLsYes
Level 3: Network--deep onlyURL reachability, GitHub repo existenceNo (warning)

Level 3 is opt-in because network checks are slow and external services can be flaky:

$ python scripts/template_manager.py validate --deep

Network warnings:
- Template 'litellm-langfuse-starter' links.railway_template not found (404)
Validation passed: templates.yaml

The template is still valid—we just warn about the broken link.

Claude Code Skill Integration

For LLM-assisted workflows, we created a skill at .claude/skills/template-registry/:

template-registry/
├── SKILL.md # Main instructions (safety rules, CLI commands)
├── schema-reference.md # Field documentation
└── examples.md # Common patterns

The skill teaches Claude to use the CLI safely:

## Important Safety Rules
1. ALWAYS run validation before any write operation
2. NEVER commit directly to main - create a branch/PR
3. Treat all LLM outputs as untrusted until validated

Now you can ask Claude: "Add a new template for my-awesome-project" and it will:

  1. Use the CLI with proper arguments
  2. Run validation
  3. Create a branch and PR

Security Hardening

LLM-assisted editing introduces risks. We added multiple protections:

Input Validation

# Reject malformed GitHub owner/repo names
GITHUB_OWNER_PATTERN = re.compile(r"^[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,37}[a-zA-Z0-9])?$")
GITHUB_REPO_PATTERN = re.compile(r"^[a-zA-Z0-9._-]{1,100}$")
# Reject symlinks to prevent LFI attacks
fd = os.open(str(path), os.O_RDONLY | os.O_NOFOLLOW)

YAML Hardening

yaml.allow_duplicate_keys = False  # Catch accidental overwrites

Atomic Writes

# Write to temp file, then atomic rename
fd, temp_path = tempfile.mkstemp(dir=path.parent)
# ... write content ...
os.replace(temp_path, path) # Atomic on POSIX

Makefile Integration

For automation and CI, everything is available via make:

make validate        # Level 1 + 2
make validate-deep # Level 1 + 2 + 3
make templates # List all
make templates-json # JSON output
make help # Show all targets

The build target runs validation first:

build: validate
python scripts/aggregate_templates.py
mkdocs build --strict

Pre-commit Hook

Validation runs automatically before commits:

# .pre-commit-config.yaml
- repo: local
hooks:
- id: template-manager-validate
name: Validate templates.yaml (semantic)
entry: python scripts/template_manager.py validate
files: ^templates\.yaml$

Now invalid templates can't even be committed locally.

What We Learned

  1. One CLI, many interfaces: Skills and Makefile are just wrappers
  2. Tiered validation saves time: Fast checks always, slow checks on demand
  3. LLMs need guardrails: Validation-first prevents hallucinated YAML
  4. Atomic operations matter: Temp file + rename prevents corruption

Implementation Stats

  • 4 phases over 2 days
  • 54 tests with full coverage
  • 1,053 lines of Python
  • 16 GitHub issues tracked and closed

What's Next

  • MCP Server: Reconsider at 20+ templates (current: 3)
  • Template Linting: Check for common misconfigurations
  • Auto-sync: Fetch metadata from Railway API

Links:

Building a Cross-Project ADR Aggregator with TDD

· 7 min read
Chris
Amiable Dev
Claude
AI Assistant

Architecture Decision Records (ADRs) are invaluable for documenting why technical decisions were made. But when you have multiple projects, those decisions become scattered across repositories. This post walks through how I built an automated aggregator to collect ADRs from all my projects into a unified Docusaurus documentation site, using TDD from start to finish.

Building an OSS Foundation: ADR-001 Implementation

· 3 min read
Amiable Dev
Project Contributors

How we established community standards for amiable-templates using Architecture Decision Records (ADRs) and multi-model AI review.

!!! info "What's an ADR?" An Architecture Decision Record documents significant technical decisions with context, options considered, and rationale. It creates a searchable history of why things are the way they are.

The Problem

We're building amiable-templates to aggregate deployment templates for AI infrastructure into a single portal. Before writing any aggregation code, we needed to answer: How do we structure an OSS project that invites contribution?

Starting from scratch means making a lot of decisions:

  • What license?
  • How do contributors know what's expected?
  • How do we handle security reports?
  • What governance model fits a small project?

The Solution: Adopt Proven Patterns

Instead of reinventing the wheel, we borrowed the existing OSS ADR-033, which had already been reviewed with the LLM Council and battle-tested for llm-council.dev.

The Files

FilePurpose
LICENSEMIT - maximum flexibility
CODE_OF_CONDUCT.mdContributor Covenant v2.1
CONTRIBUTING.mdHow to contribute
SECURITY.md48hr target response time
GOVERNANCE.mdDecision-making process
SUPPORT.mdWhere to get help

GitHub Configuration

.github/
├── CODEOWNERS # Auto-assign reviewers
├── dependabot.yml # Keep deps updated
├── ISSUE_TEMPLATE/ # Structured bug reports
└── PULL_REQUEST_TEMPLATE.md

Example: CODEOWNERS

Here's how we route reviews to the right people:

# Default: maintainers review everything
* @amiable-dev/maintainers

# Critical config requires explicit maintainer approval
templates.yaml @amiable-dev/maintainers
mkdocs.yml @amiable-dev/maintainers

# CI/CD changes are sensitive
.github/ @amiable-dev/maintainers

# ADRs need architectural review
docs/adrs/ @amiable-dev/maintainers

This means any PR touching templates.yaml (our template registry) automatically requests review from maintainers. As the project grows, we can split ownership - e.g., docs/ @docs-team.

The Interesting Part: LLM Council Review

We used LLM Council to review our ADR before accepting it. LLM Council is an MCP server that queries multiple AI models in parallel, has them critique each other's responses, and synthesizes a consensus verdict.

Four models (GPT-5.2, Claude Opus 4.5, Gemini 3 Pro, Grok 4.1) reviewed our draft ADR:

What they caught:

FindingOur Response
Missing CI/CD workflowsAdded deploy.yml and security.yml
GOVERNANCE.md premature for solo projectSimplified, will expand at 3+ maintainers
Need template intake policyAdded to CONTRIBUTING.md

The full review is documented in ADR-001.

Tracking It All

We used GitHub Issues to track implementation:

  • Epic: #5 - Complete OSS Foundation
  • Sub-issues: Labels (#6), Branch Protection (#7), Blog (#8), etc.

This gives visibility into what's done and what's remaining.

What's Next

With the foundation in place, we're moving through the remaining ADRs:

  • ADR-002: MkDocs site architecture
  • ADR-003: Template configuration system
  • ADR-004: CI/CD & deployment
  • ADR-005: DevSecOps implementation
  • ADR-006: Cross-project documentation aggregation

Each follows the same process: draft, LLM Council review, implement, document.


Links:

Choosing MkDocs Material: ADR-002 Site Architecture

· 4 min read
Amiable Dev
Project Contributors

Why we chose MkDocs Material over Docusaurus or a custom solution, and how we structured the site.

The Problem

We needed a documentation site that could showcase templates in a scannable, attractive format. The site also needed to aggregate documentation from multiple template repositories, provide excellent search, support dark/light mode for accessibility, and be easy for contributors to work with.

Three options emerged: MkDocs Material, Docusaurus, or a custom Next.js/Astro site.

Why MkDocs Material?

CriteriaMkDocs MaterialDocusaurusCustom
Stack alignmentPython (matches scripts)React/NodeVaries
Setup timeHoursHoursDays/Weeks
MaintenanceLowMediumHigh
SearchBuilt-in (lunr.js)Algolia neededBuild it
Dark modeBuilt-inBuilt-inBuild it

Why not Docusaurus? It's a great framework, and we use it in our own blog amiable.dev but it would introduce React/Node into a Python-focused project. Our aggregation scripts are Python, and having a consistent stack reduces cognitive load.

Why not custom? A Next.js or Astro site would give us full control, but it's overkill for documentation. We'd spend weeks building what MkDocs Material gives us out of the box.

The deciding factor: consistency. Our llm-council docs already use MkDocs Material. Same tooling, same patterns, same contributor experience.

The Architecture

docs/
├── index.md # Hero + featured templates
├── quickstart.md # Prominent, top-level
├── templates/ # Template grid + aggregated docs
├── adrs/ # Architecture decisions
├── blog/ # You're reading it
└── stylesheets/
└── extra.css # Hero + grid styling

We use top-level tabs for main sections:

nav:
- Home: index.md
- Quick Start: quickstart.md
- Templates: templates/index.md
- ADRs: adrs/index.md
- Contributing: contributing.md
- Blog: blog/index.md

Quick Start gets its own tab because that's what most visitors want.

The Template Grid

We wanted a scannable grid of template cards without any JavaScript complexity. Here's how we built it using pure markdown with custom CSS:

<div class="template-grid" markdown="1">

<div class="template-card" markdown="1">

### LiteLLM + Langfuse Starter

Production-ready LLM proxy with observability.

**Features:**
- 100+ LLM providers via LiteLLM
- Request tracing with Langfuse
- Cost tracking and analytics

**Estimated Cost:** ~$29-68/month

[:octicons-rocket-16: Deploy](https://railway.app/template/...)
[:octicons-mark-github-16: Source](https://github.com/amiable-dev/litellm-langfuse-railway)

</div>

<div class="template-card" markdown="1">

### Another Template

Description here...

</div>

</div>

The markdown attribute is key—it tells MkDocs Material to process the markdown inside the HTML divs.

The CSS does the heavy lifting:

.template-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(320px, 1fr));
gap: 1.5rem;
}

.template-card {
border: 1px solid var(--md-default-fg-color--lightest);
border-radius: 0.5rem;
padding: 1.5rem;
}

.template-card:hover {
box-shadow: 0 4px 12px rgba(0, 0, 0, 0.1);
transform: translateY(-2px);
}

No framework needed. Pure CSS grid with markdown content.

Dark Mode

MkDocs Material handles dark mode elegantly with palette configuration. Users get automatic detection based on their system preference, plus a toggle to override:

theme:
name: material
palette:
- media: "(prefers-color-scheme: light)"
scheme: default
primary: deep purple
accent: deep purple
toggle:
icon: material/brightness-7
name: Switch to dark mode
- media: "(prefers-color-scheme: dark)"
scheme: slate
primary: deep purple
accent: deep purple
toggle:
icon: material/brightness-4
name: Switch to light mode

The toggle icon appears in the header. No JavaScript to write, no state to manage—it just works.

What We Learned

  1. Start with constraints: "No JavaScript" forced simpler, more maintainable solutions
  2. Reuse organizational patterns: Same theme as llm-council = less cognitive load
  3. Put Quick Start first: Most visitors want to deploy, not read architecture docs

What's Next

  • ADR-003: Template configuration system (templates.yaml)
  • ADR-006: Cross-project documentation aggregation

Links:

Designing the Template Registry: ADR-003

· 4 min read
Amiable Dev
Project Contributors

How we built a declarative configuration system for Railway templates with JSON Schema validation.

The Problem

We needed a way to register templates without touching code. Every new template shouldn't require modifying Python scripts or HTML—just add an entry to a configuration file and the site rebuilds.

The configuration needed to:

  1. Define which templates appear on the site
  2. Specify where to find documentation in each repo
  3. Provide metadata for the template grid (features, cost, tags)
  4. Catch errors before they reach production

Why YAML?

We considered three formats:

FormatProsCons
YAMLHuman-readable, comments allowed, matches mkdocs.ymlSyntax can be tricky
TOMLMatches Railway's railway.tomlLess common in Python ecosystem
JSONStrict, universalNo comments, verbose

YAML won because:

  1. Consistency: Our mkdocs.yml is already YAML
  2. Comments: We can document inline why certain fields exist
  3. Readability: Non-engineers can understand and edit it

The Schema

Here's what a template entry looks like:

templates:
- id: litellm-langfuse-starter
repo:
owner: "amiable-dev"
name: "litellm-langfuse-railway"
title: "LiteLLM + Langfuse Starter"
description: "Production-ready LLM gateway with observability"
category: observability
tags:
- litellm
- langfuse
directories:
docs:
- path: "starter/README.md"
target: "overview.md"
links:
railway_template: "https://railway.app/template/..."
github: "https://github.com/amiable-dev/litellm-langfuse-railway"
features:
- "100+ LLM Providers"
- "Cost Tracking"
estimated_cost:
min: 29
max: 68
currency: "USD"
period: "month"

Required fields: id, repo, title, description, category, directories.docs

Everything else is optional.

Validation with JSON Schema

YAML is flexible, which means it's easy to make mistakes. We use JSON Schema to catch errors early:

# templates.schema.yaml
properties:
templates:
items:
type: object
additionalProperties: false # Catch typos!
required:
- id
- repo
- title
- description
- category
- directories
properties:
id:
type: string
pattern: "^[a-z][a-z0-9-]*$"
# ...

The additionalProperties: false is important. Here's what happens with a typo:

# Bad config - spot the typo
templates:
- id: my-template
repo:
owner: "amiable-dev"
name: "my-repo"
title: "My Template"
description: "A template"
category: observability
directories:
docs:
- path: "README.md"
target: "overview.md"
featurs: # Typo!
- "Feature 1"

With additionalProperties: false, the schema rejects this:

$.templates[0]: Additional properties are not allowed ('featurs' was unexpected)

Without it, the typo would silently pass—and the aggregation script would just skip the field.

CI Integration

Validation runs on every PR:

# .github/workflows/validate.yml
- name: Validate templates.yaml schema
run: check-jsonschema --schemafile templates.schema.yaml templates.yaml

- name: Validate unique template IDs
run: |
python -c "
import yaml
with open('templates.yaml') as f:
config = yaml.safe_load(f)
ids = [t['id'] for t in config.get('templates', [])]
duplicates = [id for id in ids if ids.count(id) > 1]
if duplicates:
exit(1)
"

Error messages are actionable:

$.templates[0]: 'repo' is a required property
$.templates[0].id: 'INVALID_ID' does not match '^[a-z][a-z0-9-]*$'

The Aggregation Flow

  1. CI validates templates.yaml against the JSON Schema
  2. Python aggregator reads the config (YAML → Python dict)
  3. For each template, fetches docs from directories.docs paths
  4. Transforms content (rewrites links, adds attribution)
  5. Writes to docs/templates/{id}/
  6. MkDocs builds the static site

The script uses .get() for optional fields, so missing features or estimated_cost don't break the build.

What We Learned

  1. Strict schemas catch bugs early: additionalProperties: false is your friend
  2. Validate on PRs, not just deploys: Developers should see errors before merge
  3. JSON Schema can't do everything: We added a Python check for unique IDs

What's Next

  • ADR-006: Cross-project documentation aggregation (the script that reads this config)

Links: