Alvaro Del Valle
44d4bf6748
Second discovery pass over the Scrivas EC2 estate (716468089330, us-east-2, 7 instances), prompted by the client reporting reliability issues and by their lack of access to source code held under contract by the incumbent vendor. Source code recovery - All 10 application repositories exist as complete git checkouts on Scrivas-owned instances, with full history rather than deployed artifacts: 4 app repos on Scrivas_dev_env, 6 ML repos on ML_dev. - Every remote points at git@git.devteam.space (the contractor's self-hosted GitLab), which Scrivas does not control. The on-instance checkouts are the client's only independent leverage over their own source. - Gap: both /var/www frontends are build output with no .git, so frontend source is not recoverable from EC2. - Time-sensitive: scrivas_backend received a commit on the assessment date. Reliability triage - Production runs 23 containers on a single 15 GiB host, including 3 Postgres instances, Kafka and OpenSearch, at 73% memory at rest with no per-container memory limits and no swap on any of the 7 instances. - Kafka, OpenSearch and search-api carry restart policy `no`, so a host reboot yields a partially-recovered stack. - Recorded as a structural exposure, not an observed root cause: no OOM event is present in retained logs and RestartCount is 0 on every prod container. Confirming the hypothesis needs CloudWatch history the boxes do not retain. Contents - scripts/ec2_code_discovery.py EC2 inventory (describe + user data) - scripts/ec2_code_inspect.py read-only SSM probe set, reviewable in PROBES - findings/ec2_code_discovery_report.md narrative writeup - findings/code_dashboard.html client-facing dashboard - findings/ec2_code_inspect*.json raw probe output - index.html links the new dashboard and evidence All access was read-only: no writes, restarts or config changes on any instance. Probe output was scanned for credentials before commit; git metadata was read as the owning user rather than by writing a safe.directory entry. Claude-Session: https://claude.ai/code/session_01YMxVaHXJsqpqKwncNQ9b1e
7.4 KiB
Scrivas — EC2 Code Discovery & Reliability Triage
Prepared by: Dasnuve · Date: 2026-08-28
Access: dasnuve-scrivas-louis-impersonation → arn:aws:iam::716468089330:user/louis
Method: Read-only EC2 describe + SSM AWS-RunShellScript probes (no writes, no restarts, no config changes).
Scripts: scripts/ec2_code_discovery.py (inventory), scripts/ec2_code_inspect.py (probes)
Raw: findings/ec2_code_inspect.json, findings/ec2_code_inspect_prod.json
1. Headline: the source code is recoverable
All 10 application repositories exist as full git checkouts on Scrivas-owned EC2 instances, with complete history — not just deployed artifacts. The client can recover their codebase from their own account today.
All repos point to a single external remote:
git@git.devteam.space:scrivas/<repo>.git
git.devteam.space is the incumbent contractor's self-hosted GitLab (DevTeam.Space; commit authors are
@devteamspace.com). Scrivas does not control this host. The checkouts on EC2 are therefore the client's
only leverage over their own source.
Application repos — Scrivas_dev_env (i-010066e6c9027aa6e), /home/admin/
| Repo | Commits | Branch | Last commit |
|---|---|---|---|
scrivas_backend |
1380 | dev |
2026-08-28 — Vasilii |
scrivas_gate |
183 | dev |
2026-08-14 — Vasilii |
scrivas_search |
96 | dev |
2026-06-24 — Azamat |
patient |
59 | dev |
2026-08-13 — vturtugeshev@webiomed.ru |
ML repos — ML_dev (i-095bd68aff22b103b), /srv/
| Repo | Commits | Last commit |
|---|---|---|
post_processor |
137 | 2026-06-30 — Yegor Kovalev |
sai_suggestions |
93 | 2026-06-16 |
ml_monitoring |
62 | 2026-07-20 |
patient-summary-service |
49 | 2026-06-12 |
patient_document_parser |
41 | 2026-06-16 |
soniox_transcriber |
25 | 2026-06-16 |
Not in git: /var/www/scrivas_frontend and /var/www/Scrivas_admin_frontend on the dev box are
deployed build output only — no .git. Frontend source is not recoverable from EC2 and must come
from the contractor's GitLab. This is the one genuine gap.
Recommended immediate action
Mirror all 10 repos (git clone --mirror) off the instances to Scrivas-controlled storage before any
contract conversation changes access posture. scrivas_backend received a commit today — the contractor
is actively developing, so the on-box checkouts are current.
2. Architecture as actually deployed
Two clearly different engineering standards are in play.
Application tier — scrivas_prod_env (i-073154fb4fa773bbd), Ubuntu 26.04, m6a.xlarge, 15 GiB
23 containers on a single host: encounter_api, gate_api, patient_api, 8 Celery workers,
celery_beat, 3× Postgres 17.5, Redis 8.2.1, Kafka 4.1.1, Debezium CDC 2.7.3, OpenSearch, autoheal.
Deployed via docker-compose from git checkouts. No orchestration, no registry.
ML tier — Ml_prod / ML_stage / ML_dev, 7.6 GiB
Deployed from ECR (716468089330.dkr.ecr.us-east-2.amazonaws.com/scrivas/*) with commit-SHA image
tags and blue/green slots (dev-green-*, stage-blue-*). Health checks present and passing.
This tier is materially more mature than the application tier.
Netbird_scrivas (i-02754e7ae419cd5db) — NetBird VPN mesh (server, dashboard, Traefik v3.6).
3. Reliability findings
Ranked by likely contribution to the reported instability.
| # | Finding | Evidence | Risk |
|---|---|---|---|
| 1 | No memory limit on any prod container | HostConfig.Memory=0 on all 23 |
High |
| 2 | No swap on any instance | swapon --show empty, all 7 |
High |
| 3 | Prod at 11 GiB / 15 GiB (73%) steady-state | free -h |
High |
| 4 | Entire prod stack on one host — incl. 3 databases | docker ps |
High |
| 5 | Three prod containers have restart policy no |
scrivas-kafka, scrivas-opensearch, scrivas_search-search-api-1 |
High |
| 6 | Prod OS differs from dev/stage | prod Ubuntu 26.04 vs 24.04 elsewhere | Medium |
| 7 | autoheal deployed in prod |
willfarrell/autoheal:1.2.0 |
Medium |
| 8 | Uncommitted config drift in prod checkouts | docker-compose.yml.bkp, .bkpwithlogs, src/gate/.env.save |
Medium |
| 9 | 47 interactive login sessions on prod | uptime (32 on Ml_prod) |
Medium |
| 10 | sai-suggestions-redis.service failed on ML_stage |
systemctl --state=failed |
Low |
The primary hypothesis: unbounded memory on a shared host
Findings 1–4 compound into one failure mode. Prod runs 23 containers — including three Postgres instances, Kafka, and OpenSearch — on a single 15 GiB box that is already 73% consumed at rest, with no per-container memory limits and no swap. Any single Celery worker processing an oversized document or audio chunk can exhaust host memory. With no limits, the kernel OOM killer chooses the victim — and it will typically pick the largest RSS process, which is a database, not the worker that caused the problem.
That produces exactly the symptom profile a client describes as "random reliability issues": unpredictable, uncorrelated with the triggering workload, and hard to reproduce.
Finding 5 makes it worse. Kafka, OpenSearch, and the search API have restart policy no — if they
die or the host reboots, they stay down until someone intervenes manually. Everything else is
unless-stopped, so a reboot yields a partially-recovered stack.
The presence of autoheal (finding 7) is corroborating: someone deliberately installed a daemon whose
only job is restarting containers that fail health checks. That is treatment of a symptom, and it tells
us health-check failures were frequent enough to be worth automating around.
What the evidence does not show
Stated plainly, because it constrains the conclusion:
- No OOM kill was captured in the current logs.
dmesgandjournalctlon prod show no OOM events in the retained window.State.OOMKilledisfalseon all 23 containers. - No crash-looping.
RestartCountis 0 on every prod container. The Celery workers showing "Up 6 hours" were redeployed at 13:49 today, not restarted by failure. lastis not installed on prod, so reboot history could not be read there.
So findings 1–5 are a demonstrated structural exposure, not an observed root cause. Confirming the hypothesis requires history the boxes do not currently retain.
4. Recommended next steps
Immediately
- Mirror all 10 repos to Scrivas-controlled storage. Highest-value, lowest-effort, time-sensitive.
- Set restart policy to
unless-stoppedon Kafka, OpenSearch, and search-api.
To confirm the memory hypothesis — the current boxes cannot answer this from retained logs
3. Enable CloudWatch per-container memory metrics (the EC2-CloudWatchAgent-Role profile is already attached).
4. Add mem_limit per service in compose, sized from observed peaks; add swap as an OOM buffer.
5. Query CloudWatch for historic mem_used_percent on prod to date the onset of instability.
Structural 6. Move the three Postgres instances off the shared application host. 7. Align prod OS/AMI with stage (26.04 vs 24.04 makes staging an unreliable rehearsal). 8. Extend the ML tier's ECR + commit-SHA + blue/green pattern to the application tier — the better pattern already exists in-house. 9. Reconcile the uncommitted prod config drift into version control. 10. Audit the 47 interactive prod sessions.