# Scrivas — EC2 Code Discovery & Reliability Triage

**Prepared by:** Dasnuve · **Date:** 2026-08-28
**Access:** `dasnuve-scrivas-louis-impersonation` → `arn:aws:iam::716468089330:user/louis`
**Method:** Read-only EC2 describe + SSM `AWS-RunShellScript` probes (no writes, no restarts, no config changes).
**Scripts:** `scripts/ec2_code_discovery.py` (inventory), `scripts/ec2_code_inspect.py` (probes)
**Raw:** `findings/ec2_code_inspect.json`, `findings/ec2_code_inspect_prod.json`

---

## 1. Headline: the source code is recoverable

**All 10 application repositories exist as full git checkouts on Scrivas-owned EC2 instances**, with complete
history — not just deployed artifacts. The client can recover their codebase from their own account today.

All repos point to a single external remote:

> `git@git.devteam.space:scrivas/<repo>.git`

`git.devteam.space` is the **incumbent contractor's self-hosted GitLab** (DevTeam.Space; commit authors are
`@devteamspace.com`). Scrivas does not control this host. The checkouts on EC2 are therefore the client's
only leverage over their own source.

### Application repos — `Scrivas_dev_env` (i-010066e6c9027aa6e), `/home/admin/`

| Repo | Commits | Branch | Last commit |
|---|---:|---|---|
| `scrivas_backend` | **1380** | `dev` | 2026-08-28 — Vasilii |
| `scrivas_gate` | 183 | `dev` | 2026-08-14 — Vasilii |
| `scrivas_search` | 96 | `dev` | 2026-06-24 — Azamat |
| `patient` | 59 | `dev` | 2026-08-13 — vturtugeshev@webiomed.ru |

### ML repos — `ML_dev` (i-095bd68aff22b103b), `/srv/`

| Repo | Commits | Last commit |
|---|---:|---|
| `post_processor` | 137 | 2026-06-30 — Yegor Kovalev |
| `sai_suggestions` | 93 | 2026-06-16 |
| `ml_monitoring` | 62 | 2026-07-20 |
| `patient-summary-service` | 49 | 2026-06-12 |
| `patient_document_parser` | 41 | 2026-06-16 |
| `soniox_transcriber` | 25 | 2026-06-16 |

**Not in git:** `/var/www/scrivas_frontend` and `/var/www/Scrivas_admin_frontend` on the dev box are
**deployed build output only — no `.git`**. Frontend source is *not* recoverable from EC2 and must come
from the contractor's GitLab. This is the one genuine gap.

### Recommended immediate action
Mirror all 10 repos (`git clone --mirror`) off the instances to Scrivas-controlled storage **before** any
contract conversation changes access posture. `scrivas_backend` received a commit *today* — the contractor
is actively developing, so the on-box checkouts are current.

---

## 2. Architecture as actually deployed

Two clearly different engineering standards are in play.

**Application tier — `scrivas_prod_env` (i-073154fb4fa773bbd), Ubuntu 26.04, m6a.xlarge, 15 GiB**
23 containers on a **single host**: `encounter_api`, `gate_api`, `patient_api`, 8 Celery workers,
`celery_beat`, **3× Postgres 17.5**, Redis 8.2.1, Kafka 4.1.1, Debezium CDC 2.7.3, OpenSearch, autoheal.
Deployed via `docker-compose` from git checkouts. No orchestration, no registry.

**ML tier — `Ml_prod` / `ML_stage` / `ML_dev`, 7.6 GiB**
Deployed from **ECR** (`716468089330.dkr.ecr.us-east-2.amazonaws.com/scrivas/*`) with commit-SHA image
tags and **blue/green slots** (`dev-green-*`, `stage-blue-*`). Health checks present and passing.
This tier is materially more mature than the application tier.

**`Netbird_scrivas` (i-02754e7ae419cd5db)** — NetBird VPN mesh (server, dashboard, Traefik v3.6).

---

## 3. Reliability findings

Ranked by likely contribution to the reported instability.

| # | Finding | Evidence | Risk |
|---|---|---|---|
| 1 | **No memory limit on any prod container** | `HostConfig.Memory=0` on all 23 | **High** |
| 2 | **No swap on any instance** | `swapon --show` empty, all 7 | **High** |
| 3 | **Prod at 11 GiB / 15 GiB (73%) steady-state** | `free -h` | **High** |
| 4 | **Entire prod stack on one host — incl. 3 databases** | `docker ps` | **High** |
| 5 | **Three prod containers have restart policy `no`** | `scrivas-kafka`, `scrivas-opensearch`, `scrivas_search-search-api-1` | **High** |
| 6 | **Prod OS differs from dev/stage** | prod Ubuntu 26.04 vs 24.04 elsewhere | Medium |
| 7 | **`autoheal` deployed in prod** | `willfarrell/autoheal:1.2.0` | Medium |
| 8 | **Uncommitted config drift in prod checkouts** | `docker-compose.yml.bkp`, `.bkpwithlogs`, `src/gate/.env.save` | Medium |
| 9 | **47 interactive login sessions on prod** | `uptime` (32 on `Ml_prod`) | Medium |
| 10 | `sai-suggestions-redis.service` failed on ML_stage | `systemctl --state=failed` | Low |

### The primary hypothesis: unbounded memory on a shared host

Findings 1–4 compound into one failure mode. Prod runs 23 containers — including three Postgres
instances, Kafka, and OpenSearch — on a single 15 GiB box that is already **73% consumed at rest**,
with **no per-container memory limits** and **no swap**. Any single Celery worker processing an
oversized document or audio chunk can exhaust host memory. With no limits, the kernel OOM killer
chooses the victim — and it will typically pick the largest RSS process, which is a **database**, not
the worker that caused the problem.

That produces exactly the symptom profile a client describes as "random reliability issues":
unpredictable, uncorrelated with the triggering workload, and hard to reproduce.

Finding 5 makes it worse. Kafka, OpenSearch, and the search API have restart policy `no` — if they
die or the host reboots, **they stay down** until someone intervenes manually. Everything else is
`unless-stopped`, so a reboot yields a partially-recovered stack.

The presence of `autoheal` (finding 7) is corroborating: someone deliberately installed a daemon whose
only job is restarting containers that fail health checks. That is treatment of a symptom, and it tells
us health-check failures were frequent enough to be worth automating around.

### What the evidence does *not* show

Stated plainly, because it constrains the conclusion:

- **No OOM kill was captured in the current logs.** `dmesg` and `journalctl` on prod show no OOM events
  in the retained window. `State.OOMKilled` is `false` on all 23 containers.
- **No crash-looping.** `RestartCount` is **0** on every prod container. The Celery workers showing
  "Up 6 hours" were **redeployed** at 13:49 today, not restarted by failure.
- `last` is not installed on prod, so reboot history could not be read there.

So findings 1–5 are a demonstrated **structural exposure**, not an observed root cause. Confirming the
hypothesis requires history the boxes do not currently retain.

---

## 4. Recommended next steps

**Immediately**
1. Mirror all 10 repos to Scrivas-controlled storage. Highest-value, lowest-effort, time-sensitive.
2. Set restart policy to `unless-stopped` on Kafka, OpenSearch, and search-api.

**To confirm the memory hypothesis** — the current boxes cannot answer this from retained logs
3. Enable CloudWatch per-container memory metrics (the `EC2-CloudWatchAgent-Role` profile is already attached).
4. Add `mem_limit` per service in compose, sized from observed peaks; add swap as an OOM buffer.
5. Query CloudWatch for historic `mem_used_percent` on prod to date the onset of instability.

**Structural**
6. Move the three Postgres instances off the shared application host.
7. Align prod OS/AMI with stage (26.04 vs 24.04 makes staging an unreliable rehearsal).
8. Extend the ML tier's ECR + commit-SHA + blue/green pattern to the application tier — the better
   pattern already exists in-house.
9. Reconcile the uncommitted prod config drift into version control.
10. Audit the 47 interactive prod sessions.