adelvalleandAlvaro Del Valle
c987ed07c5
Second discovery pass over the Scrivas EC2 estate (`716468089330`, `us-east-2`, 7 instances). Prompted by two client signals: they do not hold the source code for their own platform (built under contract by the incumbent vendor), and they are reporting production reliability issues. All access was **read-only** — no writes, restarts or config changes on any instance. ## Source code is recoverable **All 10 application repositories exist as complete git checkouts on Scrivas-owned instances**, with full commit history rather than deployed artifacts. | Tier | Location | Repos | |---|---|---| | App | `Scrivas_dev_env:/home/admin/` | `scrivas_backend` (1380 commits), `scrivas_gate` (183), `scrivas_search` (96), `patient` (59) | | ML | `ML_dev:/srv/` | `post_processor` (137), `sai_suggestions` (93), `ml_monitoring` (62), `patient-summary-service` (49), `patient_document_parser` (41), `soniox_transcriber` (25) | Every remote points at `git@git.devteam.space:scrivas/*` — the contractor's self-hosted GitLab, which Scrivas does **not** control. The on-instance checkouts are the client's only independent leverage over their own source. **Gap:** both `/var/www` frontends are build output with no `.git`. Frontend source is not recoverable from EC2. **Time-sensitive:** `scrivas_backend` received a commit on the assessment date. Development is active on infrastructure the contractor controls. Mirroring the repos to Scrivas-controlled storage is the recommended immediate action and is **not** included in this PR — it needs a scope decision first (it involves the contractor's GitLab credentials held on the dev box). ## Reliability triage Production runs **23 containers on a single 15 GiB host** — including 3 Postgres instances, Kafka and OpenSearch — at **73% memory at rest**, with **no per-container memory limits** and **no swap** on any of the 7 instances. With no limits the OOM killer selects by resident size, so it would typically kill a database rather than the worker that caused the pressure. That matches the "random, unreproducible" symptom profile. Kafka, OpenSearch and search-api additionally carry restart policy `no`, so a host reboot yields a partially-recovered stack that looks healthy from outside. **Recorded as a structural exposure, not an observed root cause.** No OOM event appears in retained logs, `State.OOMKilled` is false on all 23 containers, and `RestartCount` is **0** on every one — the Celery workers showing "Up 6 hours" were redeployed, not crash-restarted. Confirming the hypothesis needs CloudWatch history these boxes do not retain, which is itself a finding and the basis for recommendations 3–5 in the report. One encouraging contrast: the **ML tier is well-built** — ECR images tagged by commit SHA, blue/green slots, passing health checks. The application tier is compose-from-git-checkout with uncommitted `.env.save` and `docker-compose.yml.bkp` files in the prod working tree. The better pattern already exists in-house. ## Contents | File | | |---|---| | `scripts/ec2_code_discovery.py` | EC2 inventory — describe + user data | | `scripts/ec2_code_inspect.py` | read-only SSM probe set; commands reviewable in `PROBES` | | `findings/ec2_code_discovery_report.md` | narrative writeup | | `findings/code_dashboard.html` | client-facing dashboard, matching the existing design system | | `findings/ec2_code_inspect*.json` | raw probe output | | `index.html` | links the new dashboard and evidence | ## Review notes - Probe output was **scanned for credentials before commit** ��� no AWS keys, passwords, tokens or private key blocks. The probe set reads manifests and VCS metadata, never file contents. - Git metadata was read **as the owning user** (`sudo -u`) rather than by writing a `safe.directory` entry into root's gitconfig, to preserve the read-only guarantee. - Several prod probes initially returned empty and were re-run with stderr visible; the empties were artifacts (`last` is not installed on prod), not clean health. Worth knowing when reading the JSON. - The dashboard keeps the "what the evidence does not show" section in the **client-facing** version. If CloudWatch later points elsewhere, overclaiming a root cause would cost more credibility than the softer framing gains. https://claude.ai/code/session_01YMxVaHXJsqpqKwncNQ9b1e Co-authored-by: Alvaro Del Valle <alvaro.delvalle2@gmail.com> Reviewed-on: #2
143 lines
7.4 KiB
Markdown
143 lines
7.4 KiB
Markdown
# Scrivas — EC2 Code Discovery & Reliability Triage
|
||
|
||
**Prepared by:** Dasnuve · **Date:** 2026-08-28
|
||
**Access:** `dasnuve-scrivas-louis-impersonation` → `arn:aws:iam::716468089330:user/louis`
|
||
**Method:** Read-only EC2 describe + SSM `AWS-RunShellScript` probes (no writes, no restarts, no config changes).
|
||
**Scripts:** `scripts/ec2_code_discovery.py` (inventory), `scripts/ec2_code_inspect.py` (probes)
|
||
**Raw:** `findings/ec2_code_inspect.json`, `findings/ec2_code_inspect_prod.json`
|
||
|
||
---
|
||
|
||
## 1. Headline: the source code is recoverable
|
||
|
||
**All 10 application repositories exist as full git checkouts on Scrivas-owned EC2 instances**, with complete
|
||
history — not just deployed artifacts. The client can recover their codebase from their own account today.
|
||
|
||
All repos point to a single external remote:
|
||
|
||
> `git@git.devteam.space:scrivas/<repo>.git`
|
||
|
||
`git.devteam.space` is the **incumbent contractor's self-hosted GitLab** (DevTeam.Space; commit authors are
|
||
`@devteamspace.com`). Scrivas does not control this host. The checkouts on EC2 are therefore the client's
|
||
only leverage over their own source.
|
||
|
||
### Application repos — `Scrivas_dev_env` (i-010066e6c9027aa6e), `/home/admin/`
|
||
|
||
| Repo | Commits | Branch | Last commit |
|
||
|---|---:|---|---|
|
||
| `scrivas_backend` | **1380** | `dev` | 2026-08-28 — Vasilii |
|
||
| `scrivas_gate` | 183 | `dev` | 2026-08-14 — Vasilii |
|
||
| `scrivas_search` | 96 | `dev` | 2026-06-24 — Azamat |
|
||
| `patient` | 59 | `dev` | 2026-08-13 — vturtugeshev@webiomed.ru |
|
||
|
||
### ML repos — `ML_dev` (i-095bd68aff22b103b), `/srv/`
|
||
|
||
| Repo | Commits | Last commit |
|
||
|---|---:|---|
|
||
| `post_processor` | 137 | 2026-06-30 — Yegor Kovalev |
|
||
| `sai_suggestions` | 93 | 2026-06-16 |
|
||
| `ml_monitoring` | 62 | 2026-07-20 |
|
||
| `patient-summary-service` | 49 | 2026-06-12 |
|
||
| `patient_document_parser` | 41 | 2026-06-16 |
|
||
| `soniox_transcriber` | 25 | 2026-06-16 |
|
||
|
||
**Not in git:** `/var/www/scrivas_frontend` and `/var/www/Scrivas_admin_frontend` on the dev box are
|
||
**deployed build output only — no `.git`**. Frontend source is *not* recoverable from EC2 and must come
|
||
from the contractor's GitLab. This is the one genuine gap.
|
||
|
||
### Recommended immediate action
|
||
Mirror all 10 repos (`git clone --mirror`) off the instances to Scrivas-controlled storage **before** any
|
||
contract conversation changes access posture. `scrivas_backend` received a commit *today* — the contractor
|
||
is actively developing, so the on-box checkouts are current.
|
||
|
||
---
|
||
|
||
## 2. Architecture as actually deployed
|
||
|
||
Two clearly different engineering standards are in play.
|
||
|
||
**Application tier — `scrivas_prod_env` (i-073154fb4fa773bbd), Ubuntu 26.04, m6a.xlarge, 15 GiB**
|
||
23 containers on a **single host**: `encounter_api`, `gate_api`, `patient_api`, 8 Celery workers,
|
||
`celery_beat`, **3× Postgres 17.5**, Redis 8.2.1, Kafka 4.1.1, Debezium CDC 2.7.3, OpenSearch, autoheal.
|
||
Deployed via `docker-compose` from git checkouts. No orchestration, no registry.
|
||
|
||
**ML tier — `Ml_prod` / `ML_stage` / `ML_dev`, 7.6 GiB**
|
||
Deployed from **ECR** (`716468089330.dkr.ecr.us-east-2.amazonaws.com/scrivas/*`) with commit-SHA image
|
||
tags and **blue/green slots** (`dev-green-*`, `stage-blue-*`). Health checks present and passing.
|
||
This tier is materially more mature than the application tier.
|
||
|
||
**`Netbird_scrivas` (i-02754e7ae419cd5db)** — NetBird VPN mesh (server, dashboard, Traefik v3.6).
|
||
|
||
---
|
||
|
||
## 3. Reliability findings
|
||
|
||
Ranked by likely contribution to the reported instability.
|
||
|
||
| # | Finding | Evidence | Risk |
|
||
|---|---|---|---|
|
||
| 1 | **No memory limit on any prod container** | `HostConfig.Memory=0` on all 23 | **High** |
|
||
| 2 | **No swap on any instance** | `swapon --show` empty, all 7 | **High** |
|
||
| 3 | **Prod at 11 GiB / 15 GiB (73%) steady-state** | `free -h` | **High** |
|
||
| 4 | **Entire prod stack on one host — incl. 3 databases** | `docker ps` | **High** |
|
||
| 5 | **Three prod containers have restart policy `no`** | `scrivas-kafka`, `scrivas-opensearch`, `scrivas_search-search-api-1` | **High** |
|
||
| 6 | **Prod OS differs from dev/stage** | prod Ubuntu 26.04 vs 24.04 elsewhere | Medium |
|
||
| 7 | **`autoheal` deployed in prod** | `willfarrell/autoheal:1.2.0` | Medium |
|
||
| 8 | **Uncommitted config drift in prod checkouts** | `docker-compose.yml.bkp`, `.bkpwithlogs`, `src/gate/.env.save` | Medium |
|
||
| 9 | **47 interactive login sessions on prod** | `uptime` (32 on `Ml_prod`) | Medium |
|
||
| 10 | `sai-suggestions-redis.service` failed on ML_stage | `systemctl --state=failed` | Low |
|
||
|
||
### The primary hypothesis: unbounded memory on a shared host
|
||
|
||
Findings 1–4 compound into one failure mode. Prod runs 23 containers — including three Postgres
|
||
instances, Kafka, and OpenSearch — on a single 15 GiB box that is already **73% consumed at rest**,
|
||
with **no per-container memory limits** and **no swap**. Any single Celery worker processing an
|
||
oversized document or audio chunk can exhaust host memory. With no limits, the kernel OOM killer
|
||
chooses the victim — and it will typically pick the largest RSS process, which is a **database**, not
|
||
the worker that caused the problem.
|
||
|
||
That produces exactly the symptom profile a client describes as "random reliability issues":
|
||
unpredictable, uncorrelated with the triggering workload, and hard to reproduce.
|
||
|
||
Finding 5 makes it worse. Kafka, OpenSearch, and the search API have restart policy `no` — if they
|
||
die or the host reboots, **they stay down** until someone intervenes manually. Everything else is
|
||
`unless-stopped`, so a reboot yields a partially-recovered stack.
|
||
|
||
The presence of `autoheal` (finding 7) is corroborating: someone deliberately installed a daemon whose
|
||
only job is restarting containers that fail health checks. That is treatment of a symptom, and it tells
|
||
us health-check failures were frequent enough to be worth automating around.
|
||
|
||
### What the evidence does *not* show
|
||
|
||
Stated plainly, because it constrains the conclusion:
|
||
|
||
- **No OOM kill was captured in the current logs.** `dmesg` and `journalctl` on prod show no OOM events
|
||
in the retained window. `State.OOMKilled` is `false` on all 23 containers.
|
||
- **No crash-looping.** `RestartCount` is **0** on every prod container. The Celery workers showing
|
||
"Up 6 hours" were **redeployed** at 13:49 today, not restarted by failure.
|
||
- `last` is not installed on prod, so reboot history could not be read there.
|
||
|
||
So findings 1–5 are a demonstrated **structural exposure**, not an observed root cause. Confirming the
|
||
hypothesis requires history the boxes do not currently retain.
|
||
|
||
---
|
||
|
||
## 4. Recommended next steps
|
||
|
||
**Immediately**
|
||
1. Mirror all 10 repos to Scrivas-controlled storage. Highest-value, lowest-effort, time-sensitive.
|
||
2. Set restart policy to `unless-stopped` on Kafka, OpenSearch, and search-api.
|
||
|
||
**To confirm the memory hypothesis** — the current boxes cannot answer this from retained logs
|
||
3. Enable CloudWatch per-container memory metrics (the `EC2-CloudWatchAgent-Role` profile is already attached).
|
||
4. Add `mem_limit` per service in compose, sized from observed peaks; add swap as an OOM buffer.
|
||
5. Query CloudWatch for historic `mem_used_percent` on prod to date the onset of instability.
|
||
|
||
**Structural**
|
||
6. Move the three Postgres instances off the shared application host.
|
||
7. Align prod OS/AMI with stage (26.04 vs 24.04 makes staging an unreliable rehearsal).
|
||
8. Extend the ML tier's ECR + commit-SHA + blue/green pattern to the application tier — the better
|
||
pattern already exists in-house.
|
||
9. Reconcile the uncommitted prod config drift into version control.
|
||
10. Audit the 47 interactive prod sessions.
|