ADR-004: AI Model Selection with Fallback Strategy

Status: REQUIRES IMMEDIATE ATTENTION
Date: 2025-08-25
Author: Architecture Review Team

Context

The project has critical dependency on OpenAI GPT-4 for semantic classification. With projected costs of $50K+/month at scale and single-point-of-failure risk, this needs robust fallback strategies.

Current plan: GPT-4 primary, CodeBERT secondary, custom model tertiary.

Decision

IMPLEMENT IMMEDIATE RISK MITIGATION with diversified model strategy:

Primary Strategy: Cost-Optimized Tiered Approach

class ModelSelector {
  selectModel(repo: Repository): ModelConfig {
    // Tier 1: High-value repos (>1000 stars)
    if (repo.stars > 1000) {
      return { primary: 'gpt-4', fallback: 'gpt-3.5-turbo' }
    }
    
    // Tier 2: Medium repos (100-1000 stars)  
    if (repo.stars > 100) {
      return { primary: 'gpt-3.5-turbo', fallback: 'claude-instant' }
    }
    
    // Tier 3: Low-value repos (<100 stars)
    return { primary: 'similarity-based', fallback: 'rule-based' }
  }
}

Consequences

Critical Issues with Current Plan:

• Cost Explosion: $50K/month makes unit economics impossible
• Single Vendor Risk: OpenAI outage = complete service failure
• Rate Limiting: Could hit API limits during growth spurts
• Quality Variance: Different models produce inconsistent results

Benefits of Tiered Approach:

• 80% cost reduction while maintaining quality for important repos
• Vendor diversification reduces single-point-of-failure
• Graceful degradation during API issues
• Better unit economics for sustainable business

Alternatives Considered

1. OpenAI Only (Original Plan)

   • Pros: Consistent quality, simple implementation
   • Cons: Cost explosion, single point of failure, vendor lock-in

2. Tiered Multi-Model (RECOMMENDED)

   • Pros: Cost optimization, risk distribution, quality preservation
   • Cons: Increased complexity, consistency challenges

3. Open Source Models Only

   • Pros: No API costs, full control
   • Cons: Significant infrastructure costs, quality uncertainty

4. Hybrid Local + Cloud

   • Pros: Cost optimization, reduced latency
   • Cons: Infrastructure complexity, model management overhead

Risk Assessment

CRITICAL RISKS:

• Financial Runway: Current approach could consume 100% of budget on AI costs
• Service Availability: 99.9% dependent on external AI services
• Quality Consistency: Multiple models could produce conflicting classifications
• Rate Limits: Could hit OpenAI limits during viral growth

Mitigation Strategy - IMPLEMENT IMMEDIATELY:

1. Emergency Fallback System:

class EmergencyFallback {
  async classifyWithFallback(repo: Repository): Promise<Classification> {
    try {
      return await this.primaryModel.classify(repo)
    } catch (error) {
      console.warn('Primary model failed, using similarity fallback')
      
      // Find most similar analyzed repository
      const similar = await this.findMostSimilar(repo)
      if (similar.similarity > 0.8) {
        return this.adaptClassification(similar.classification, repo)
      }
      
      // Final fallback: rule-based classification
      return this.ruleBasedClassification(repo)
    }
  }
}

2. Cost Circuit Breakers:

class CostGuard {
  private dailyCost = 0
  private readonly DAILY_LIMIT = 500 // $500/day
  
  async checkBudget(): Promise<boolean> {
    if (this.dailyCost > this.DAILY_LIMIT) {
      // Switch to free tier only
      this.switchToFallbackMode()
      return false
    }
    return true
  }
}

Migration Strategy

IMMEDIATE (Week 1):

• Implement cost tracking and circuit breakers
• Add Anthropic Claude as secondary provider
• Create rule-based fallback for common repository types

Short-term (Month 1):

• Deploy similarity-based classification for low-value repos
• Fine-tune cost optimization based on real usage data
• Implement quality monitoring across different models

Long-term (Month 3):

• Evaluate fine-tuning custom model on accumulated data
• Consider local model deployment for cost optimization
• Implement advanced ensemble voting for quality improvement

Conclusion

The current single-vendor AI strategy represents an existential risk to the business. Immediate implementation of tiered model selection, cost controls, and fallback systems is critical for financial viability and service reliability. The recommended approach reduces costs by 80% while preserving quality for high-value repositories.