adelvalleandAlvaro Del Valle
c987ed07c5
Second discovery pass over the Scrivas EC2 estate (`716468089330`, `us-east-2`, 7 instances). Prompted by two client signals: they do not hold the source code for their own platform (built under contract by the incumbent vendor), and they are reporting production reliability issues. All access was **read-only** — no writes, restarts or config changes on any instance. ## Source code is recoverable **All 10 application repositories exist as complete git checkouts on Scrivas-owned instances**, with full commit history rather than deployed artifacts. | Tier | Location | Repos | |---|---|---| | App | `Scrivas_dev_env:/home/admin/` | `scrivas_backend` (1380 commits), `scrivas_gate` (183), `scrivas_search` (96), `patient` (59) | | ML | `ML_dev:/srv/` | `post_processor` (137), `sai_suggestions` (93), `ml_monitoring` (62), `patient-summary-service` (49), `patient_document_parser` (41), `soniox_transcriber` (25) | Every remote points at `git@git.devteam.space:scrivas/*` — the contractor's self-hosted GitLab, which Scrivas does **not** control. The on-instance checkouts are the client's only independent leverage over their own source. **Gap:** both `/var/www` frontends are build output with no `.git`. Frontend source is not recoverable from EC2. **Time-sensitive:** `scrivas_backend` received a commit on the assessment date. Development is active on infrastructure the contractor controls. Mirroring the repos to Scrivas-controlled storage is the recommended immediate action and is **not** included in this PR — it needs a scope decision first (it involves the contractor's GitLab credentials held on the dev box). ## Reliability triage Production runs **23 containers on a single 15 GiB host** — including 3 Postgres instances, Kafka and OpenSearch — at **73% memory at rest**, with **no per-container memory limits** and **no swap** on any of the 7 instances. With no limits the OOM killer selects by resident size, so it would typically kill a database rather than the worker that caused the pressure. That matches the "random, unreproducible" symptom profile. Kafka, OpenSearch and search-api additionally carry restart policy `no`, so a host reboot yields a partially-recovered stack that looks healthy from outside. **Recorded as a structural exposure, not an observed root cause.** No OOM event appears in retained logs, `State.OOMKilled` is false on all 23 containers, and `RestartCount` is **0** on every one — the Celery workers showing "Up 6 hours" were redeployed, not crash-restarted. Confirming the hypothesis needs CloudWatch history these boxes do not retain, which is itself a finding and the basis for recommendations 3–5 in the report. One encouraging contrast: the **ML tier is well-built** — ECR images tagged by commit SHA, blue/green slots, passing health checks. The application tier is compose-from-git-checkout with uncommitted `.env.save` and `docker-compose.yml.bkp` files in the prod working tree. The better pattern already exists in-house. ## Contents | File | | |---|---| | `scripts/ec2_code_discovery.py` | EC2 inventory — describe + user data | | `scripts/ec2_code_inspect.py` | read-only SSM probe set; commands reviewable in `PROBES` | | `findings/ec2_code_discovery_report.md` | narrative writeup | | `findings/code_dashboard.html` | client-facing dashboard, matching the existing design system | | `findings/ec2_code_inspect*.json` | raw probe output | | `index.html` | links the new dashboard and evidence | ## Review notes - Probe output was **scanned for credentials before commit** ��� no AWS keys, passwords, tokens or private key blocks. The probe set reads manifests and VCS metadata, never file contents. - Git metadata was read **as the owning user** (`sudo -u`) rather than by writing a `safe.directory` entry into root's gitconfig, to preserve the read-only guarantee. - Several prod probes initially returned empty and were re-run with stderr visible; the empties were artifacts (`last` is not installed on prod), not clean health. Worth knowing when reading the JSON. - The dashboard keeps the "what the evidence does not show" section in the **client-facing** version. If CloudWatch later points elsewhere, overclaiming a root cause would cost more credibility than the softer framing gains. https://claude.ai/code/session_01YMxVaHXJsqpqKwncNQ9b1e Co-authored-by: Alvaro Del Valle <alvaro.delvalle2@gmail.com> Reviewed-on: #2
319 lines
20 KiB
HTML
319 lines
20 KiB
HTML
<!DOCTYPE html>
|
||
<html lang="en">
|
||
<head>
|
||
<meta charset="utf-8">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||
<title>Scrivas — Source Code & Reliability</title>
|
||
<style>
|
||
:root {
|
||
--bg:#eef0f3; --panel:#ffffff; --panel-2:#f7f8fa; --ink:#171d27;
|
||
--ink-soft:#4a5563; --ink-faint:#79828f; --line:#dde1e7; --line-soft:#e8ebef;
|
||
--accent:#0d6e78; --accent-ink:#0a565e;
|
||
--good:#2f8657; --warn:#b5832a; --crit:#bd463b;
|
||
--crit-bg:#fbeceb; --good-bg:#eaf4ee; --warn-bg:#f8f1e2;
|
||
--mono: ui-monospace,"SF Mono","SFMono-Regular",Menlo,Consolas,monospace;
|
||
--sans: ui-sans-serif,system-ui,"Segoe UI",Roboto,Helvetica,Arial,sans-serif;
|
||
--shadow:0 1px 2px rgba(20,26,38,.04),0 6px 20px -12px rgba(20,26,38,.18);
|
||
}
|
||
@media (prefers-color-scheme:dark){:root{
|
||
--bg:#0d1117;--panel:#161c26;--panel-2:#1b222e;--ink:#e8ecf1;--ink-soft:#aab3c0;
|
||
--ink-faint:#7c8695;--line:#28303c;--line-soft:#212934;--accent:#3bb4bf;--accent-ink:#8fdfe6;
|
||
--good:#54b681;--warn:#d6a94a;--crit:#e26b60;--crit-bg:#2a1917;--good-bg:#132419;--warn-bg:#251d10;
|
||
--shadow:0 1px 2px rgba(0,0,0,.3),0 8px 24px -14px rgba(0,0,0,.6);}}
|
||
:root[data-theme="dark"]{
|
||
--bg:#0d1117;--panel:#161c26;--panel-2:#1b222e;--ink:#e8ecf1;--ink-soft:#aab3c0;
|
||
--ink-faint:#7c8695;--line:#28303c;--line-soft:#212934;--accent:#3bb4bf;--accent-ink:#8fdfe6;
|
||
--good:#54b681;--warn:#d6a94a;--crit:#e26b60;--crit-bg:#2a1917;--good-bg:#132419;--warn-bg:#251d10;
|
||
--shadow:0 1px 2px rgba(0,0,0,.3),0 8px 24px -14px rgba(0,0,0,.6);}
|
||
:root[data-theme="light"]{
|
||
--bg:#eef0f3;--panel:#ffffff;--panel-2:#f7f8fa;--ink:#171d27;--ink-soft:#4a5563;
|
||
--ink-faint:#79828f;--line:#dde1e7;--line-soft:#e8ebef;--accent:#0d6e78;--accent-ink:#0a565e;
|
||
--good:#2f8657;--warn:#b5832a;--crit:#bd463b;--crit-bg:#fbeceb;--good-bg:#eaf4ee;--warn-bg:#f8f1e2;
|
||
--shadow:0 1px 2px rgba(20,26,38,.04),0 6px 20px -12px rgba(20,26,38,.18);}
|
||
|
||
*{box-sizing:border-box;}
|
||
body{margin:0;background:var(--bg);color:var(--ink);font-family:var(--sans);line-height:1.5;-webkit-font-smoothing:antialiased;}
|
||
.wrap{max-width:1120px;margin:0 auto;padding:32px 24px 64px;}
|
||
header{display:flex;flex-wrap:wrap;align-items:flex-end;justify-content:space-between;gap:16px;padding-bottom:20px;border-bottom:2px solid var(--ink);}
|
||
.eyebrow{font-size:11px;letter-spacing:.16em;text-transform:uppercase;color:var(--accent);font-weight:700;}
|
||
h1{font-size:clamp(24px,4vw,34px);margin:6px 0 4px;letter-spacing:-.02em;text-wrap:balance;font-weight:750;}
|
||
.sub{color:var(--ink-soft);font-size:14px;}
|
||
.idcard{font-family:var(--mono);font-size:12px;color:var(--ink-soft);text-align:right;line-height:1.7;}
|
||
.idcard b{color:var(--ink);}
|
||
|
||
.kpis{display:grid;grid-template-columns:repeat(4,1fr);gap:14px;margin:22px 0 12px;}
|
||
@media (max-width:760px){.kpis{grid-template-columns:repeat(2,1fr);}}
|
||
.kpi{background:var(--panel);border:1px solid var(--line);border-radius:12px;padding:16px;box-shadow:var(--shadow);position:relative;overflow:hidden;}
|
||
.kpi::before{content:"";position:absolute;left:0;top:0;bottom:0;width:3px;background:var(--accent);}
|
||
.kpi.crit::before{background:var(--crit);}.kpi.warn::before{background:var(--warn);}.kpi.good::before{background:var(--good);}
|
||
.kpi .n{font-size:28px;font-weight:750;letter-spacing:-.02em;font-variant-numeric:tabular-nums;}
|
||
.kpi.crit .n{color:var(--crit);}.kpi.warn .n{color:var(--warn);}.kpi.good .n{color:var(--good);}
|
||
.kpi .lbl{font-size:12px;color:var(--ink-faint);margin-top:2px;}
|
||
.kpi .meta{font-size:11px;color:var(--ink-soft);margin-top:8px;}
|
||
|
||
section{background:var(--panel);border:1px solid var(--line);border-radius:14px;padding:22px 24px;margin-bottom:18px;box-shadow:var(--shadow);}
|
||
.sec-head{display:flex;align-items:baseline;gap:10px;margin-bottom:16px;flex-wrap:wrap;}
|
||
.sec-head .tag{font-family:var(--mono);font-size:12px;color:var(--accent);font-weight:700;}
|
||
.sec-head h2{font-size:17px;margin:0;letter-spacing:-.01em;}
|
||
.sec-head .note{margin-left:auto;font-size:12px;color:var(--ink-faint);}
|
||
.grid2{display:grid;grid-template-columns:1fr 1fr;gap:18px;}
|
||
@media (max-width:820px){.grid2{grid-template-columns:1fr;}}
|
||
|
||
table{width:100%;border-collapse:collapse;font-size:13px;}
|
||
th{text-align:left;font-size:11px;text-transform:uppercase;letter-spacing:.07em;color:var(--ink-faint);
|
||
font-weight:700;padding:0 10px 8px 0;border-bottom:1px solid var(--line);}
|
||
td{padding:9px 10px 9px 0;border-bottom:1px solid var(--line-soft);vertical-align:top;color:var(--ink-soft);}
|
||
td b{color:var(--ink);}
|
||
tbody tr:last-child td{border-bottom:none;}
|
||
.num{font-family:var(--mono);font-variant-numeric:tabular-nums;text-align:right;color:var(--ink);}
|
||
code{font-family:var(--mono);font-size:12px;color:var(--ink-soft);}
|
||
.scroll{overflow-x:auto;}
|
||
|
||
.pill{display:inline-flex;align-items:center;gap:6px;font-size:11px;font-weight:700;padding:3px 9px;border-radius:999px;white-space:nowrap;}
|
||
.pill.crit{background:var(--crit-bg);color:var(--crit);}
|
||
.pill.good{background:var(--good-bg);color:var(--good);}
|
||
.pill.warn{background:var(--warn-bg);color:var(--warn);}
|
||
|
||
.callout{border-radius:10px;padding:14px 16px;font-size:13px;color:var(--ink-soft);margin:0 0 16px;
|
||
background:var(--warn-bg);border:1px solid color-mix(in srgb,var(--warn) 30%,transparent);}
|
||
.callout.crit{background:var(--crit-bg);border-color:color-mix(in srgb,var(--crit) 30%,transparent);}
|
||
.callout.good{background:var(--good-bg);border-color:color-mix(in srgb,var(--good) 30%,transparent);}
|
||
.callout b{color:var(--ink);}
|
||
.callout p{margin:8px 0 0;}
|
||
|
||
.memtrack{height:26px;background:var(--panel-2);border:1px solid var(--line);border-radius:7px;overflow:hidden;display:flex;margin:6px 0 4px;}
|
||
.memtrack .used{background:var(--crit);height:100%;}
|
||
.memtrack .cache{background:color-mix(in srgb,var(--warn) 55%,transparent);height:100%;}
|
||
.memlegend{display:flex;gap:16px;font-size:11.5px;color:var(--ink-faint);flex-wrap:wrap;}
|
||
.memlegend i{display:inline-block;width:9px;height:9px;border-radius:2px;margin-right:5px;vertical-align:baseline;}
|
||
|
||
ol.steps{margin:0;padding-left:20px;font-size:13px;color:var(--ink-soft);}
|
||
ol.steps li{margin-bottom:9px;}
|
||
ol.steps b{color:var(--ink);}
|
||
|
||
footer{display:flex;flex-wrap:wrap;gap:18px;margin-top:24px;font-size:12px;color:var(--ink-faint);font-family:var(--mono);}
|
||
</style>
|
||
</head>
|
||
<body>
|
||
<div class="wrap">
|
||
|
||
<header>
|
||
<div>
|
||
<div class="eyebrow">Dasnuve · Cloud Discovery</div>
|
||
<h1>Source Code Recovery & Reliability Triage</h1>
|
||
<div class="sub">What is running on the Scrivas EC2 estate, where the source lives, and why production is fragile.</div>
|
||
</div>
|
||
<div class="idcard">
|
||
Account <b>716468089330</b><br>
|
||
Region <b>us-east-2</b> · 7 instances<br>
|
||
2026-08-28 · <b>CONFIDENTIAL</b>
|
||
</div>
|
||
</header>
|
||
|
||
<div class="kpis">
|
||
<div class="kpi good">
|
||
<div class="n">10</div>
|
||
<div class="lbl">Repos recoverable on EC2</div>
|
||
<div class="meta">Full git history, not just build output</div>
|
||
</div>
|
||
<div class="kpi warn">
|
||
<div class="n">2</div>
|
||
<div class="lbl">Frontends <u>not</u> recoverable</div>
|
||
<div class="meta">Deployed build only — no <code>.git</code></div>
|
||
</div>
|
||
<div class="kpi crit">
|
||
<div class="n">23</div>
|
||
<div class="lbl">Prod containers, one host</div>
|
||
<div class="meta">Incl. 3 × Postgres, Kafka, OpenSearch</div>
|
||
</div>
|
||
<div class="kpi crit">
|
||
<div class="n">0</div>
|
||
<div class="lbl">Memory limits set</div>
|
||
<div class="meta">And zero swap, on all 7 instances</div>
|
||
</div>
|
||
</div>
|
||
|
||
<section>
|
||
<div class="sec-head">
|
||
<span class="tag">01</span><h2>The source code is recoverable — and that is time-sensitive</h2>
|
||
<span class="note">Scrivas_dev_env · ML_dev</span>
|
||
</div>
|
||
|
||
<div class="callout good">
|
||
<b>All 10 application repositories exist as complete git checkouts on Scrivas-owned instances</b>, with full
|
||
commit history — not merely deployed artifacts. Scrivas can recover its codebase from its own AWS account today.
|
||
<p>Every repository points at a single external remote: <code>git@git.devteam.space:scrivas/<repo>.git</code> —
|
||
the incumbent contractor's self-hosted GitLab, which Scrivas does <b>not</b> control. The on-instance checkouts are
|
||
therefore the client's only independent leverage over their own source.</p>
|
||
</div>
|
||
|
||
<div class="grid2">
|
||
<div>
|
||
<div class="sec-head"><span class="tag">APP</span><h2>Scrivas_dev_env <code>/home/admin/</code></h2></div>
|
||
<div class="scroll"><table>
|
||
<thead><tr><th>Repository</th><th class="num">Commits</th><th>Last commit</th></tr></thead>
|
||
<tbody>
|
||
<tr><td><b>scrivas_backend</b></td><td class="num">1380</td><td>2026-08-28 · Vasilii</td></tr>
|
||
<tr><td><b>scrivas_gate</b></td><td class="num">183</td><td>2026-08-14 · Vasilii</td></tr>
|
||
<tr><td><b>scrivas_search</b></td><td class="num">96</td><td>2026-06-24 · Azamat</td></tr>
|
||
<tr><td><b>patient</b></td><td class="num">59</td><td>2026-08-13 · vturtugeshev</td></tr>
|
||
</tbody>
|
||
</table></div>
|
||
</div>
|
||
<div>
|
||
<div class="sec-head"><span class="tag">ML</span><h2>ML_dev <code>/srv/</code></h2></div>
|
||
<div class="scroll"><table>
|
||
<thead><tr><th>Repository</th><th class="num">Commits</th><th>Last commit</th></tr></thead>
|
||
<tbody>
|
||
<tr><td><b>post_processor</b></td><td class="num">137</td><td>2026-06-30</td></tr>
|
||
<tr><td><b>sai_suggestions</b></td><td class="num">93</td><td>2026-06-16</td></tr>
|
||
<tr><td><b>ml_monitoring</b></td><td class="num">62</td><td>2026-07-20</td></tr>
|
||
<tr><td><b>patient-summary-service</b></td><td class="num">49</td><td>2026-06-12</td></tr>
|
||
<tr><td><b>patient_document_parser</b></td><td class="num">41</td><td>2026-06-16</td></tr>
|
||
<tr><td><b>soniox_transcriber</b></td><td class="num">25</td><td>2026-06-16</td></tr>
|
||
</tbody>
|
||
</table></div>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="callout" style="margin:18px 0 0;">
|
||
<b>The gap:</b> <code>/var/www/scrivas_frontend</code> and <code>/var/www/Scrivas_admin_frontend</code> are
|
||
<b>deployed build output with no <code>.git</code></b>. Frontend source cannot be recovered from EC2 and must be
|
||
obtained from the contractor's GitLab.
|
||
<p><b>Why this is urgent:</b> <code>scrivas_backend</code> received a commit <b>on the day of this assessment</b>.
|
||
Development is active on infrastructure the contractor controls. Mirror all 10 repos
|
||
(<code>git clone --mirror</code>) to Scrivas-controlled storage <i>before</i> any contract conversation changes
|
||
access posture.</p>
|
||
</div>
|
||
</section>
|
||
|
||
<section>
|
||
<div class="sec-head">
|
||
<span class="tag">02</span><h2>Two tiers, two engineering standards</h2>
|
||
<span class="note">the better pattern already exists in-house</span>
|
||
</div>
|
||
<div class="grid2">
|
||
<div>
|
||
<div class="sec-head"><span class="tag">APP</span><h2>Application tier <span class="pill crit">fragile</span></h2></div>
|
||
<p class="sub" style="font-size:13px;">23 containers on a <b>single host</b> — <code>encounter_api</code>,
|
||
<code>gate_api</code>, <code>patient_api</code>, 8 Celery workers, <code>celery_beat</code>,
|
||
<b>3 × Postgres 17.5</b>, Redis, Kafka, Debezium CDC, OpenSearch, autoheal.</p>
|
||
<p class="sub" style="font-size:13px;">Deployed by <code>docker-compose</code> directly from git checkouts.
|
||
No orchestration, no image registry, no immutable artifact.</p>
|
||
</div>
|
||
<div>
|
||
<div class="sec-head"><span class="tag">ML</span><h2>ML tier <span class="pill good">mature</span></h2></div>
|
||
<p class="sub" style="font-size:13px;">Deployed from <b>ECR</b>
|
||
(<code>…dkr.ecr.us-east-2…/scrivas/*</code>) with <b>commit-SHA image tags</b> and
|
||
<b>blue/green slots</b> (<code>dev-green-*</code>, <code>stage-blue-*</code>).</p>
|
||
<p class="sub" style="font-size:13px;">Health checks defined and passing across every service.
|
||
This is a materially stronger delivery pipeline — and a template the application tier can adopt.</p>
|
||
</div>
|
||
</div>
|
||
</section>
|
||
|
||
<section>
|
||
<div class="sec-head">
|
||
<span class="tag">03</span><h2>Reliability findings</h2>
|
||
<span class="note">ranked by likely contribution to reported instability</span>
|
||
</div>
|
||
<div class="scroll"><table>
|
||
<thead><tr><th>#</th><th>Finding</th><th>Evidence</th><th>Risk</th></tr></thead>
|
||
<tbody>
|
||
<tr><td class="num">1</td><td><b>No memory limit on any prod container</b></td><td><code>HostConfig.Memory=0</code> on all 23</td><td><span class="pill crit">High</span></td></tr>
|
||
<tr><td class="num">2</td><td><b>No swap on any instance</b></td><td><code>swapon --show</code> empty, all 7</td><td><span class="pill crit">High</span></td></tr>
|
||
<tr><td class="num">3</td><td><b>Prod at 73% memory at rest</b></td><td>11 GiB used of 15 GiB</td><td><span class="pill crit">High</span></td></tr>
|
||
<tr><td class="num">4</td><td><b>Whole stack on one host, incl. 3 databases</b></td><td><code>docker ps</code> — 23 containers</td><td><span class="pill crit">High</span></td></tr>
|
||
<tr><td class="num">5</td><td><b>3 prod containers set to never restart</b></td><td>Kafka, OpenSearch, search-api — policy <code>no</code></td><td><span class="pill crit">High</span></td></tr>
|
||
<tr><td class="num">6</td><td>Prod OS differs from dev/stage</td><td>Ubuntu <b>26.04</b> vs <b>24.04</b> elsewhere</td><td><span class="pill warn">Medium</span></td></tr>
|
||
<tr><td class="num">7</td><td><code>autoheal</code> deployed in production</td><td><code>willfarrell/autoheal:1.2.0</code></td><td><span class="pill warn">Medium</span></td></tr>
|
||
<tr><td class="num">8</td><td>Uncommitted config drift in prod checkouts</td><td><code>docker-compose.yml.bkp</code>, <code>.env.save</code></td><td><span class="pill warn">Medium</span></td></tr>
|
||
<tr><td class="num">9</td><td>47 interactive login sessions on prod</td><td><code>uptime</code> (32 on Ml_prod)</td><td><span class="pill warn">Medium</span></td></tr>
|
||
<tr><td class="num">10</td><td><code>sai-suggestions-redis.service</code> failed</td><td>ML_stage <code>systemctl --state=failed</code></td><td><span class="pill good">Low</span></td></tr>
|
||
</tbody>
|
||
</table></div>
|
||
</section>
|
||
|
||
<section>
|
||
<div class="sec-head">
|
||
<span class="tag">04</span><h2>Primary hypothesis: unbounded memory on a shared host</h2>
|
||
</div>
|
||
|
||
<p class="sub" style="font-size:13px;margin-top:0;">Production memory, at rest — before any workload spike:</p>
|
||
<div class="memtrack" role="img" aria-label="Production memory: 11 GiB used, 3.5 GiB cache, of 15 GiB total">
|
||
<div class="used" style="width:73%"></div><div class="cache" style="width:23%"></div>
|
||
</div>
|
||
<div class="memlegend">
|
||
<span><i style="background:var(--crit)"></i>11 GiB used (73%)</span>
|
||
<span><i style="background:color-mix(in srgb,var(--warn) 55%,transparent)"></i>3.5 GiB cache</span>
|
||
<span><i style="background:var(--panel-2);border:1px solid var(--line)"></i>441 MiB free</span>
|
||
<span><b style="color:var(--crit)">0 B swap</b></span>
|
||
</div>
|
||
|
||
<div class="callout crit" style="margin-top:18px;">
|
||
Findings 1–4 compound into a single failure mode. Production runs 23 containers — including three Postgres
|
||
instances, Kafka and OpenSearch — on one 15 GiB host already <b>73% consumed at rest</b>, with
|
||
<b>no per-container memory limits</b> and <b>no swap</b>. A single Celery worker handling an oversized document
|
||
or audio chunk can exhaust host memory. With no limits, the kernel OOM killer selects by resident size — so it
|
||
will typically kill a <b>database</b>, not the worker that caused the problem.
|
||
<p>That yields precisely the profile a client reports as "random reliability issues": unpredictable, uncorrelated
|
||
with the triggering workload, and difficult to reproduce.</p>
|
||
<p><b>Finding 5 compounds recovery.</b> Kafka, OpenSearch and search-api carry restart policy <code>no</code> — if
|
||
they die or the host reboots, they <b>stay down</b> until someone intervenes. Everything else is
|
||
<code>unless-stopped</code>, so a reboot yields a partially-recovered stack that looks healthy from the outside.</p>
|
||
<p><b>Finding 7 corroborates.</b> Someone deliberately deployed <code>autoheal</code>, a daemon whose only job is
|
||
restarting containers that fail health checks. That treats a symptom — and indicates health-check failures were
|
||
frequent enough to be worth automating around.</p>
|
||
</div>
|
||
|
||
<div class="callout">
|
||
<b>What the evidence does not show — stated plainly, because it bounds the conclusion.</b>
|
||
<p>· <b>No OOM kill was captured.</b> <code>dmesg</code> and <code>journalctl</code> on prod show no OOM events in
|
||
the retained window; <code>State.OOMKilled</code> is <code>false</code> on all 23 containers.</p>
|
||
<p>· <b>No crash-looping.</b> <code>RestartCount</code> is <b>0</b> on every prod container. The Celery workers
|
||
showing "Up 6 hours" were <b>redeployed</b> at 13:49 that day, not restarted by failure.</p>
|
||
<p>· <code>last</code> is not installed on prod, so reboot history could not be read there.</p>
|
||
<p>Findings 1–5 are therefore a demonstrated <b>structural exposure</b>, not an observed root cause. Confirming
|
||
the hypothesis requires history these instances do not currently retain — which is itself a finding.</p>
|
||
</div>
|
||
</section>
|
||
|
||
<section>
|
||
<div class="sec-head"><span class="tag">05</span><h2>Recommended next steps</h2></div>
|
||
<div class="grid2">
|
||
<div>
|
||
<div class="sec-head"><span class="tag">NOW</span><h2>Immediate</h2></div>
|
||
<ol class="steps">
|
||
<li><b>Mirror all 10 repos</b> to Scrivas-controlled storage. Highest value, lowest effort, time-sensitive.</li>
|
||
<li><b>Set <code>unless-stopped</code></b> on Kafka, OpenSearch and search-api — removes the silent partial-recovery failure.</li>
|
||
</ol>
|
||
<div class="sec-head" style="margin-top:18px;"><span class="tag">PROVE</span><h2>Confirm the hypothesis</h2></div>
|
||
<ol class="steps" start="3">
|
||
<li><b>Enable per-container memory metrics</b> in CloudWatch — <code>EC2-CloudWatchAgent-Role</code> is already attached.</li>
|
||
<li><b>Add <code>mem_limit</code> per service</b>, sized from observed peaks; add swap as an OOM buffer.</li>
|
||
<li><b>Query historic <code>mem_used_percent</code></b> to date the onset of instability.</li>
|
||
</ol>
|
||
</div>
|
||
<div>
|
||
<div class="sec-head"><span class="tag">FIX</span><h2>Structural</h2></div>
|
||
<ol class="steps" start="6">
|
||
<li><b>Move the three Postgres instances</b> off the shared application host.</li>
|
||
<li><b>Align prod OS/AMI with stage</b> — 26.04 vs 24.04 makes staging an unreliable rehearsal for production.</li>
|
||
<li><b>Extend the ML tier's pattern</b> — ECR, commit-SHA tags, blue/green — to the application tier. The better pattern already exists in-house.</li>
|
||
<li><b>Reconcile config drift</b> (<code>.env.save</code>, <code>docker-compose.yml.bkp</code>) into version control.</li>
|
||
<li><b>Audit the 47 interactive prod sessions.</b></li>
|
||
</ol>
|
||
</div>
|
||
</div>
|
||
</section>
|
||
|
||
<footer>
|
||
<span>Read-only · SSM probes via <code>dasnuve-scrivas-louis-impersonation</code></span>
|
||
<span>Reproduce: <code>scripts/ec2_code_inspect.py</code></span>
|
||
<span>Raw: <code>findings/ec2_code_inspect.json</code></span>
|
||
<span>Narrative: <code>findings/ec2_code_discovery_report.md</code></span>
|
||
</footer>
|
||
</div>
|
||
</body>
|
||
</html>
|