System Status

## Incident Report **System:** website **Status:** Down **HTTP Status Code:** 0 **Time:** 2025-11-12T12:37:16.530Z **URL:** https://null.amiable.dev/health **Error:** getaddrinfo ENOTFOUND null.amiable.dev The system is currently experiencing issues and is not responding as expected. --- *This issue was automatically created by the monitoring workflow.*

## Incident Report **System:** website **Status:** Down **HTTP Status Code:** 0 **Time:** 2025-11-05T20:27:58.178Z **URL:** https://null.amiable.dev/health **Error:** getaddrinfo ENOTFOUND null.amiable.dev The system is currently experiencing issues and is not responding as expected. --- *This issue was automatically created by the monitoring workflow.*

## Problem The status page at https://amiable.dev/status/ is not showing updates despite the monitoring workflow (`monitor-systems.yml`) successfully running and committing data to `status-data/`. ## Investigation ### What's Working ✅ 1. **Monitoring workflow** (`monitor-systems.yml`) - Running successfully every 5 minutes - Latest commit: `78e3846` (Nov 4, 2025 22:07 UTC) - Commit message: "Update monitoring data [skip ci]" - Data location: `status-data/current.json` (contains 11 readings) - Both systems being monitored: - `workflow` system: UP (200ms latency) - `website` system: DOWN (expected - `null.amiable.dev` is test domain) 2. **Plugin version** - Using latest `@amiable-dev/docusaurus-plugin-stentorosaur@^0.4.10` - Sequential monitoring working correctly - Zero data loss architecture validated 3. **Status data committed** - `status-data/current.json` exists and is being updated in git ### What's NOT Working ❌ **Deployment workflow is not running after monitoring commits.** ## Root Cause The monitoring workflow commits include `[skip ci]` in the commit message: ```yaml execSync('git commit -m "Update monitoring data [skip ci]"'); ``` This was intentionally added in v0.4.10 to prevent infinite deployment loops, but it means: 1. ✅ Monitoring runs every 5 minutes and commits data 2. ❌ Deployment workflow (`deploy.yml`) is **never triggered** by these commits 3. ❌ Status page never gets rebuilt with new data ## Current Workflow State ### `deploy.yml` Configuration ```yaml on: push: branches: - main workflow_dispatch: # repository_dispatch is COMMENTED OUT (Option B disabled) ``` The deployment workflow only triggers on: - Manual workflow dispatch - Pushes to main (but monitoring commits have `[skip ci]`) ### `deploy-scheduled.yml` Status The repository has a scheduled deployment workflow, but checking if it's active... ## Solutions ### Option 1: Enable Scheduled Deployments (Recommended) Enable `deploy-scheduled.yml` to run daily/hourly: ```yaml on: schedule: - cron: '0 * * * *' # Every hour # OR - cron: '0 0 * * *' # Daily at midnight ``` **Pros:** - Predictable deployment schedule - Efficient (fewer Actions minutes) - Status data accumulates between deployments **Cons:** - Data shown on site could be up to 1 hour/1 day stale ### Option 2: Remove `[skip ci]` from Monitoring Commits Change `monitor-systems.yml`: ```javascript execSync('git commit -m "Update monitoring data"'); // No [skip ci] ``` **Pros:** - Status page updates within minutes of changes - Real-time status updates **Cons:** - 288 deployments per day (every 5 minutes) - High GitHub Actions minutes usage - Potential for deployment queue buildup ### Option 3: Use Repository Dispatch (Hybrid) Uncomment in `deploy.yml`: ```yaml repository_dispatch: types: [status-updated] ``` Add to `monitor-systems.yml` after commit: ```javascript github.rest.repos.createDispatchEvent({ owner: context.repo.owner, repo: context.repo.repo, event_type: 'status-updated' }); ``` **Pros:** - Deployments only when status actually changes - More efficient than removing `[skip ci]` **Cons:** - More complex workflow logic - Still potentially many deployments if status fluctuates ## Recommended Fix **Use Option 1 with hourly scheduled deployments:** 1. Enable `deploy-scheduled.yml` to run hourly 2. Keep `[skip ci]` in monitoring commits 3. Status page updates every hour with latest data This balances freshness (max 1-hour lag) with efficiency (24 deploys/day instead of 288). ## Immediate Workaround Manually trigger deployment: 1. Go to Actions → Deploy to GitHub Pages 2. Click "Run workflow" 3. Select `main` branch 4. Click "Run workflow" This will deploy the latest monitoring data that's already committed. ## Files to Update **For Option 1 (Hourly Scheduled - Recommended):** - `.github/workflows/deploy-scheduled.yml` - Verify cron schedule is active **For Option 2 (Remove skip ci):** - `.github/workflows/monitor-systems.yml` - Line 58: Remove `[skip ci]` **For Option 3 (Repository dispatch):** - `.github/workflows/deploy.yml` - Uncomment `repository_dispatch` section - `.github/workflows/monitor-systems.yml` - Add dispatch event after commit ## Related - Plugin version: v0.4.10 (sequential monitoring) - Issue #32: Race condition fixes that introduced `[skip ci]` - Monitoring data location: `status-data/` (not `build/status-data/`) ## Action Items - [ ] Decide on deployment strategy (Option 1, 2, or 3) - [ ] Update appropriate workflow files - [ ] Test deployment triggers - [ ] Verify status page updates at expected intervals - [ ] Document chosen strategy in README --- **Priority:** Medium (monitoring works, just not visible on site) **Impact:** User-facing - status page shows stale data **Effort:** Low - simple workflow configuration change

## Incident Report **System:** website **Status:** Down **HTTP Status Code:** 0 **Time:** 2025-11-04T09:54:43.956Z **URL:** https://null.amiable.dev/health The system is currently experiencing issues and is not responding as expected. --- *This issue was automatically created by the monitoring workflow.*

### Affected System Website ### Severity Major (Significant degradation) ### Description stuff is happening bro ### Impact Impactful ### Actions Taken _No response_ ### Estimated Resolution Time _No response_

### Affected System API ### Severity Critical (Complete outage) ### Description It's all broken ### Impact Everything is broken. ### Actions Taken We've looked at some stuff ### Estimated Resolution Time _No response_

### Affected System API ### Severity Major (Significant degradation) ### Description ## Status Update Our API is currently experiencing intermittent 500 errors affecting approximately 30% of requests. ### Affected Systems - API Gateway - Authentication Service ### Impact Users may experience: - Failed login attempts - Slow response times - Intermittent request failures ### Actions Taken ### Status We are actively investigating the root cause and working on a fix. ### Timeline - **Started:** 2025-11-01 14:30 UTC - **Last Updated:** 2025-11-01 14:45 UTC ### Updates - 14:30 - Issue detected via automated monitoring - 14:35 - Team notified and investigation started - 14:45 - Identified database connection pool exhaustion as likely cause ### Estimated Resolution Time 2 hours