Files

T

adelvalleandAlvaro Del Valle c987ed07c5 EC2 source-code discovery and reliability triage (#2 )

Second discovery pass over the Scrivas EC2 estate (`716468089330`, `us-east-2`, 7 instances). Prompted by two client signals: they do not hold the source code for their own platform (built under contract by the incumbent vendor), and they are reporting production reliability issues.

All access was **read-only** — no writes, restarts or config changes on any instance.

## Source code is recoverable

**All 10 application repositories exist as complete git checkouts on Scrivas-owned instances**, with full commit history rather than deployed artifacts.

| Tier | Location | Repos |
|---|---|---|
| App | `Scrivas_dev_env:/home/admin/` | `scrivas_backend` (1380 commits), `scrivas_gate` (183), `scrivas_search` (96), `patient` (59) |
| ML | `ML_dev:/srv/` | `post_processor` (137), `sai_suggestions` (93), `ml_monitoring` (62), `patient-summary-service` (49), `patient_document_parser` (41), `soniox_transcriber` (25) |

Every remote points at `git@git.devteam.space:scrivas/*` — the contractor's self-hosted GitLab, which Scrivas does **not** control. The on-instance checkouts are the client's only independent leverage over their own source.

**Gap:** both `/var/www` frontends are build output with no `.git`. Frontend source is not recoverable from EC2.

**Time-sensitive:** `scrivas_backend` received a commit on the assessment date. Development is active on infrastructure the contractor controls. Mirroring the repos to Scrivas-controlled storage is the recommended immediate action and is **not** included in this PR — it needs a scope decision first (it involves the contractor's GitLab credentials held on the dev box).

## Reliability triage

Production runs **23 containers on a single 15 GiB host** — including 3 Postgres instances, Kafka and OpenSearch — at **73% memory at rest**, with **no per-container memory limits** and **no swap** on any of the 7 instances. With no limits the OOM killer selects by resident size, so it would typically kill a database rather than the worker that caused the pressure. That matches the "random, unreproducible" symptom profile.

Kafka, OpenSearch and search-api additionally carry restart policy `no`, so a host reboot yields a partially-recovered stack that looks healthy from outside.

**Recorded as a structural exposure, not an observed root cause.** No OOM event appears in retained logs, `State.OOMKilled` is false on all 23 containers, and `RestartCount` is **0** on every one — the Celery workers showing "Up 6 hours" were redeployed, not crash-restarted. Confirming the hypothesis needs CloudWatch history these boxes do not retain, which is itself a finding and the basis for recommendations 3–5 in the report.

One encouraging contrast: the **ML tier is well-built** — ECR images tagged by commit SHA, blue/green slots, passing health checks. The application tier is compose-from-git-checkout with uncommitted `.env.save` and `docker-compose.yml.bkp` files in the prod working tree. The better pattern already exists in-house.

## Contents

| File | |
|---|---|
| `scripts/ec2_code_discovery.py` | EC2 inventory — describe + user data |
| `scripts/ec2_code_inspect.py` | read-only SSM probe set; commands reviewable in `PROBES` |
| `findings/ec2_code_discovery_report.md` | narrative writeup |
| `findings/code_dashboard.html` | client-facing dashboard, matching the existing design system |
| `findings/ec2_code_inspect*.json` | raw probe output |
| `index.html` | links the new dashboard and evidence |

## Review notes

- Probe output was **scanned for credentials before commit** ��� no AWS keys, passwords, tokens or private key blocks. The probe set reads manifests and VCS metadata, never file contents.
- Git metadata was read **as the owning user** (`sudo -u`) rather than by writing a `safe.directory` entry into root's gitconfig, to preserve the read-only guarantee.
- Several prod probes initially returned empty and were re-run with stderr visible; the empties were artifacts (`last` is not installed on prod), not clean health. Worth knowing when reading the JSON.
- The dashboard keeps the "what the evidence does not show" section in the **client-facing** version. If CloudWatch later points elsewhere, overclaiming a root cause would cost more credibility than the softer framing gains.

https://claude.ai/code/session_01YMxVaHXJsqpqKwncNQ9b1e
Co-authored-by: Alvaro Del Valle <alvaro.delvalle2@gmail.com>
Reviewed-on: #2

2026-09-02 14:36:28 -06:00

7.4 KiB

Raw Permalink Blame History

Scrivas — EC2 Code Discovery & Reliability Triage

Prepared by: Dasnuve · Date: 2026-08-28 Access: dasnuve-scrivas-louis-impersonation → arn:aws:iam::716468089330:user/louis Method: Read-only EC2 describe + SSM AWS-RunShellScript probes (no writes, no restarts, no config changes). Scripts: scripts/ec2_code_discovery.py (inventory), scripts/ec2_code_inspect.py (probes) Raw: findings/ec2_code_inspect.json, findings/ec2_code_inspect_prod.json

1. Headline: the source code is recoverable

All 10 application repositories exist as full git checkouts on Scrivas-owned EC2 instances, with complete history — not just deployed artifacts. The client can recover their codebase from their own account today.

All repos point to a single external remote:

git@git.devteam.space:scrivas/<repo>.git

git.devteam.space is the incumbent contractor's self-hosted GitLab (DevTeam.Space; commit authors are @devteamspace.com). Scrivas does not control this host. The checkouts on EC2 are therefore the client's only leverage over their own source.

Application repos — `Scrivas_dev_env` (i-010066e6c9027aa6e), `/home/admin/`

Repo	Commits	Branch	Last commit
`scrivas_backend`	1380	`dev`	2026-08-28 — Vasilii
`scrivas_gate`	183	`dev`	2026-08-14 — Vasilii
`scrivas_search`	96	`dev`	2026-06-24 — Azamat
`patient`	59	`dev`	2026-08-13 — vturtugeshev@webiomed.ru

ML repos — `ML_dev` (i-095bd68aff22b103b), `/srv/`

Repo	Commits	Last commit
`post_processor`	137	2026-06-30 — Yegor Kovalev
`sai_suggestions`	93	2026-06-16
`ml_monitoring`	62	2026-07-20
`patient-summary-service`	49	2026-06-12
`patient_document_parser`	41	2026-06-16
`soniox_transcriber`	25	2026-06-16

Not in git: /var/www/scrivas_frontend and /var/www/Scrivas_admin_frontend on the dev box are deployed build output only — no .git. Frontend source is not recoverable from EC2 and must come from the contractor's GitLab. This is the one genuine gap.

Recommended immediate action

Mirror all 10 repos (git clone --mirror) off the instances to Scrivas-controlled storage before any contract conversation changes access posture. scrivas_backend received a commit today — the contractor is actively developing, so the on-box checkouts are current.

2. Architecture as actually deployed

Two clearly different engineering standards are in play.

Application tier — scrivas_prod_env (i-073154fb4fa773bbd), Ubuntu 26.04, m6a.xlarge, 15 GiB 23 containers on a single host: encounter_api, gate_api, patient_api, 8 Celery workers, celery_beat, 3× Postgres 17.5, Redis 8.2.1, Kafka 4.1.1, Debezium CDC 2.7.3, OpenSearch, autoheal. Deployed via docker-compose from git checkouts. No orchestration, no registry.

ML tier — Ml_prod / ML_stage / ML_dev, 7.6 GiB Deployed from ECR (716468089330.dkr.ecr.us-east-2.amazonaws.com/scrivas/*) with commit-SHA image tags and blue/green slots (dev-green-*, stage-blue-*). Health checks present and passing. This tier is materially more mature than the application tier.

Netbird_scrivas (i-02754e7ae419cd5db) — NetBird VPN mesh (server, dashboard, Traefik v3.6).

3. Reliability findings

Ranked by likely contribution to the reported instability.

#	Finding	Evidence	Risk
1	No memory limit on any prod container	`HostConfig.Memory=0` on all 23	High
2	No swap on any instance	`swapon --show` empty, all 7	High
3	Prod at 11 GiB / 15 GiB (73%) steady-state	`free -h`	High
4	Entire prod stack on one host — incl. 3 databases	`docker ps`	High
5	Three prod containers have restart policy `no`	`scrivas-kafka`, `scrivas-opensearch`, `scrivas_search-search-api-1`	High
6	Prod OS differs from dev/stage	prod Ubuntu 26.04 vs 24.04 elsewhere	Medium
7	`autoheal` deployed in prod	`willfarrell/autoheal:1.2.0`	Medium
8	Uncommitted config drift in prod checkouts	`docker-compose.yml.bkp`, `.bkpwithlogs`, `src/gate/.env.save`	Medium
9	47 interactive login sessions on prod	`uptime` (32 on `Ml_prod`)	Medium
10	`sai-suggestions-redis.service` failed on ML_stage	`systemctl --state=failed`	Low

The primary hypothesis: unbounded memory on a shared host

Findings 1–4 compound into one failure mode. Prod runs 23 containers — including three Postgres instances, Kafka, and OpenSearch — on a single 15 GiB box that is already 73% consumed at rest, with no per-container memory limits and no swap. Any single Celery worker processing an oversized document or audio chunk can exhaust host memory. With no limits, the kernel OOM killer chooses the victim — and it will typically pick the largest RSS process, which is a database, not the worker that caused the problem.

That produces exactly the symptom profile a client describes as "random reliability issues": unpredictable, uncorrelated with the triggering workload, and hard to reproduce.

Finding 5 makes it worse. Kafka, OpenSearch, and the search API have restart policy no — if they die or the host reboots, they stay down until someone intervenes manually. Everything else is unless-stopped, so a reboot yields a partially-recovered stack.

The presence of autoheal (finding 7) is corroborating: someone deliberately installed a daemon whose only job is restarting containers that fail health checks. That is treatment of a symptom, and it tells us health-check failures were frequent enough to be worth automating around.

What the evidence does not show

Stated plainly, because it constrains the conclusion:

No OOM kill was captured in the current logs. dmesg and journalctl on prod show no OOM events in the retained window. State.OOMKilled is false on all 23 containers.
No crash-looping. RestartCount is 0 on every prod container. The Celery workers showing "Up 6 hours" were redeployed at 13:49 today, not restarted by failure.
last is not installed on prod, so reboot history could not be read there.

So findings 1–5 are a demonstrated structural exposure, not an observed root cause. Confirming the hypothesis requires history the boxes do not currently retain.

4. Recommended next steps

Immediately

Mirror all 10 repos to Scrivas-controlled storage. Highest-value, lowest-effort, time-sensitive.
Set restart policy to unless-stopped on Kafka, OpenSearch, and search-api.

To confirm the memory hypothesis — the current boxes cannot answer this from retained logs 3. Enable CloudWatch per-container memory metrics (the EC2-CloudWatchAgent-Role profile is already attached). 4. Add mem_limit per service in compose, sized from observed peaks; add swap as an OOM buffer. 5. Query CloudWatch for historic mem_used_percent on prod to date the onset of instability.

Structural 6. Move the three Postgres instances off the shared application host. 7. Align prod OS/AMI with stage (26.04 vs 24.04 makes staging an unreliable rehearsal). 8. Extend the ML tier's ECR + commit-SHA + blue/green pattern to the application tier — the better pattern already exists in-house. 9. Reconcile the uncommitted prod config drift into version control. 10. Audit the 47 interactive prod sessions.

7.4 KiB Raw Permalink Blame History Unescape Escape