Add EC2 source-code discovery and reliability triage
Second discovery pass over the Scrivas EC2 estate (716468089330, us-east-2, 7 instances), prompted by the client reporting reliability issues and by their lack of access to source code held under contract by the incumbent vendor. Source code recovery - All 10 application repositories exist as complete git checkouts on Scrivas-owned instances, with full history rather than deployed artifacts: 4 app repos on Scrivas_dev_env, 6 ML repos on ML_dev. - Every remote points at git@git.devteam.space (the contractor's self-hosted GitLab), which Scrivas does not control. The on-instance checkouts are the client's only independent leverage over their own source. - Gap: both /var/www frontends are build output with no .git, so frontend source is not recoverable from EC2. - Time-sensitive: scrivas_backend received a commit on the assessment date. Reliability triage - Production runs 23 containers on a single 15 GiB host, including 3 Postgres instances, Kafka and OpenSearch, at 73% memory at rest with no per-container memory limits and no swap on any of the 7 instances. - Kafka, OpenSearch and search-api carry restart policy `no`, so a host reboot yields a partially-recovered stack. - Recorded as a structural exposure, not an observed root cause: no OOM event is present in retained logs and RestartCount is 0 on every prod container. Confirming the hypothesis needs CloudWatch history the boxes do not retain. Contents - scripts/ec2_code_discovery.py EC2 inventory (describe + user data) - scripts/ec2_code_inspect.py read-only SSM probe set, reviewable in PROBES - findings/ec2_code_discovery_report.md narrative writeup - findings/code_dashboard.html client-facing dashboard - findings/ec2_code_inspect*.json raw probe output - index.html links the new dashboard and evidence All access was read-only: no writes, restarts or config changes on any instance. Probe output was scanned for credentials before commit; git metadata was read as the owning user rather than by writing a safe.directory entry. Claude-Session: https://claude.ai/code/session_01YMxVaHXJsqpqKwncNQ9b1e
This commit is contained in:
Alvaro Del Valle
@@ -0,0 +1,142 @@
|
||||
# Scrivas — EC2 Code Discovery & Reliability Triage
|
||||
|
||||
**Prepared by:** Dasnuve · **Date:** 2026-08-28
|
||||
**Access:** `dasnuve-scrivas-louis-impersonation` → `arn:aws:iam::716468089330:user/louis`
|
||||
**Method:** Read-only EC2 describe + SSM `AWS-RunShellScript` probes (no writes, no restarts, no config changes).
|
||||
**Scripts:** `scripts/ec2_code_discovery.py` (inventory), `scripts/ec2_code_inspect.py` (probes)
|
||||
**Raw:** `findings/ec2_code_inspect.json`, `findings/ec2_code_inspect_prod.json`
|
||||
|
||||
---
|
||||
|
||||
## 1. Headline: the source code is recoverable
|
||||
|
||||
**All 10 application repositories exist as full git checkouts on Scrivas-owned EC2 instances**, with complete
|
||||
history — not just deployed artifacts. The client can recover their codebase from their own account today.
|
||||
|
||||
All repos point to a single external remote:
|
||||
|
||||
> `git@git.devteam.space:scrivas/<repo>.git`
|
||||
|
||||
`git.devteam.space` is the **incumbent contractor's self-hosted GitLab** (DevTeam.Space; commit authors are
|
||||
`@devteamspace.com`). Scrivas does not control this host. The checkouts on EC2 are therefore the client's
|
||||
only leverage over their own source.
|
||||
|
||||
### Application repos — `Scrivas_dev_env` (i-010066e6c9027aa6e), `/home/admin/`
|
||||
|
||||
| Repo | Commits | Branch | Last commit |
|
||||
|---|---:|---|---|
|
||||
| `scrivas_backend` | **1380** | `dev` | 2026-08-28 — Vasilii |
|
||||
| `scrivas_gate` | 183 | `dev` | 2026-08-14 — Vasilii |
|
||||
| `scrivas_search` | 96 | `dev` | 2026-06-24 — Azamat |
|
||||
| `patient` | 59 | `dev` | 2026-08-13 — vturtugeshev@webiomed.ru |
|
||||
|
||||
### ML repos — `ML_dev` (i-095bd68aff22b103b), `/srv/`
|
||||
|
||||
| Repo | Commits | Last commit |
|
||||
|---|---:|---|
|
||||
| `post_processor` | 137 | 2026-06-30 — Yegor Kovalev |
|
||||
| `sai_suggestions` | 93 | 2026-06-16 |
|
||||
| `ml_monitoring` | 62 | 2026-07-20 |
|
||||
| `patient-summary-service` | 49 | 2026-06-12 |
|
||||
| `patient_document_parser` | 41 | 2026-06-16 |
|
||||
| `soniox_transcriber` | 25 | 2026-06-16 |
|
||||
|
||||
**Not in git:** `/var/www/scrivas_frontend` and `/var/www/Scrivas_admin_frontend` on the dev box are
|
||||
**deployed build output only — no `.git`**. Frontend source is *not* recoverable from EC2 and must come
|
||||
from the contractor's GitLab. This is the one genuine gap.
|
||||
|
||||
### Recommended immediate action
|
||||
Mirror all 10 repos (`git clone --mirror`) off the instances to Scrivas-controlled storage **before** any
|
||||
contract conversation changes access posture. `scrivas_backend` received a commit *today* — the contractor
|
||||
is actively developing, so the on-box checkouts are current.
|
||||
|
||||
---
|
||||
|
||||
## 2. Architecture as actually deployed
|
||||
|
||||
Two clearly different engineering standards are in play.
|
||||
|
||||
**Application tier — `scrivas_prod_env` (i-073154fb4fa773bbd), Ubuntu 26.04, m6a.xlarge, 15 GiB**
|
||||
23 containers on a **single host**: `encounter_api`, `gate_api`, `patient_api`, 8 Celery workers,
|
||||
`celery_beat`, **3× Postgres 17.5**, Redis 8.2.1, Kafka 4.1.1, Debezium CDC 2.7.3, OpenSearch, autoheal.
|
||||
Deployed via `docker-compose` from git checkouts. No orchestration, no registry.
|
||||
|
||||
**ML tier — `Ml_prod` / `ML_stage` / `ML_dev`, 7.6 GiB**
|
||||
Deployed from **ECR** (`716468089330.dkr.ecr.us-east-2.amazonaws.com/scrivas/*`) with commit-SHA image
|
||||
tags and **blue/green slots** (`dev-green-*`, `stage-blue-*`). Health checks present and passing.
|
||||
This tier is materially more mature than the application tier.
|
||||
|
||||
**`Netbird_scrivas` (i-02754e7ae419cd5db)** — NetBird VPN mesh (server, dashboard, Traefik v3.6).
|
||||
|
||||
---
|
||||
|
||||
## 3. Reliability findings
|
||||
|
||||
Ranked by likely contribution to the reported instability.
|
||||
|
||||
| # | Finding | Evidence | Risk |
|
||||
|---|---|---|---|
|
||||
| 1 | **No memory limit on any prod container** | `HostConfig.Memory=0` on all 23 | **High** |
|
||||
| 2 | **No swap on any instance** | `swapon --show` empty, all 7 | **High** |
|
||||
| 3 | **Prod at 11 GiB / 15 GiB (73%) steady-state** | `free -h` | **High** |
|
||||
| 4 | **Entire prod stack on one host — incl. 3 databases** | `docker ps` | **High** |
|
||||
| 5 | **Three prod containers have restart policy `no`** | `scrivas-kafka`, `scrivas-opensearch`, `scrivas_search-search-api-1` | **High** |
|
||||
| 6 | **Prod OS differs from dev/stage** | prod Ubuntu 26.04 vs 24.04 elsewhere | Medium |
|
||||
| 7 | **`autoheal` deployed in prod** | `willfarrell/autoheal:1.2.0` | Medium |
|
||||
| 8 | **Uncommitted config drift in prod checkouts** | `docker-compose.yml.bkp`, `.bkpwithlogs`, `src/gate/.env.save` | Medium |
|
||||
| 9 | **47 interactive login sessions on prod** | `uptime` (32 on `Ml_prod`) | Medium |
|
||||
| 10 | `sai-suggestions-redis.service` failed on ML_stage | `systemctl --state=failed` | Low |
|
||||
|
||||
### The primary hypothesis: unbounded memory on a shared host
|
||||
|
||||
Findings 1–4 compound into one failure mode. Prod runs 23 containers — including three Postgres
|
||||
instances, Kafka, and OpenSearch — on a single 15 GiB box that is already **73% consumed at rest**,
|
||||
with **no per-container memory limits** and **no swap**. Any single Celery worker processing an
|
||||
oversized document or audio chunk can exhaust host memory. With no limits, the kernel OOM killer
|
||||
chooses the victim — and it will typically pick the largest RSS process, which is a **database**, not
|
||||
the worker that caused the problem.
|
||||
|
||||
That produces exactly the symptom profile a client describes as "random reliability issues":
|
||||
unpredictable, uncorrelated with the triggering workload, and hard to reproduce.
|
||||
|
||||
Finding 5 makes it worse. Kafka, OpenSearch, and the search API have restart policy `no` — if they
|
||||
die or the host reboots, **they stay down** until someone intervenes manually. Everything else is
|
||||
`unless-stopped`, so a reboot yields a partially-recovered stack.
|
||||
|
||||
The presence of `autoheal` (finding 7) is corroborating: someone deliberately installed a daemon whose
|
||||
only job is restarting containers that fail health checks. That is treatment of a symptom, and it tells
|
||||
us health-check failures were frequent enough to be worth automating around.
|
||||
|
||||
### What the evidence does *not* show
|
||||
|
||||
Stated plainly, because it constrains the conclusion:
|
||||
|
||||
- **No OOM kill was captured in the current logs.** `dmesg` and `journalctl` on prod show no OOM events
|
||||
in the retained window. `State.OOMKilled` is `false` on all 23 containers.
|
||||
- **No crash-looping.** `RestartCount` is **0** on every prod container. The Celery workers showing
|
||||
"Up 6 hours" were **redeployed** at 13:49 today, not restarted by failure.
|
||||
- `last` is not installed on prod, so reboot history could not be read there.
|
||||
|
||||
So findings 1–5 are a demonstrated **structural exposure**, not an observed root cause. Confirming the
|
||||
hypothesis requires history the boxes do not currently retain.
|
||||
|
||||
---
|
||||
|
||||
## 4. Recommended next steps
|
||||
|
||||
**Immediately**
|
||||
1. Mirror all 10 repos to Scrivas-controlled storage. Highest-value, lowest-effort, time-sensitive.
|
||||
2. Set restart policy to `unless-stopped` on Kafka, OpenSearch, and search-api.
|
||||
|
||||
**To confirm the memory hypothesis** — the current boxes cannot answer this from retained logs
|
||||
3. Enable CloudWatch per-container memory metrics (the `EC2-CloudWatchAgent-Role` profile is already attached).
|
||||
4. Add `mem_limit` per service in compose, sized from observed peaks; add swap as an OOM buffer.
|
||||
5. Query CloudWatch for historic `mem_used_percent` on prod to date the onset of instability.
|
||||
|
||||
**Structural**
|
||||
6. Move the three Postgres instances off the shared application host.
|
||||
7. Align prod OS/AMI with stage (26.04 vs 24.04 makes staging an unreliable rehearsal).
|
||||
8. Extend the ML tier's ECR + commit-SHA + blue/green pattern to the application tier — the better
|
||||
pattern already exists in-house.
|
||||
9. Reconcile the uncommitted prod config drift into version control.
|
||||
10. Audit the 47 interactive prod sessions.
|
||||
Reference in New Issue
Block a user