Add EC2 source-code discovery and reliability triage
Second discovery pass over the Scrivas EC2 estate (716468089330, us-east-2, 7 instances), prompted by the client reporting reliability issues and by their lack of access to source code held under contract by the incumbent vendor. Source code recovery - All 10 application repositories exist as complete git checkouts on Scrivas-owned instances, with full history rather than deployed artifacts: 4 app repos on Scrivas_dev_env, 6 ML repos on ML_dev. - Every remote points at git@git.devteam.space (the contractor's self-hosted GitLab), which Scrivas does not control. The on-instance checkouts are the client's only independent leverage over their own source. - Gap: both /var/www frontends are build output with no .git, so frontend source is not recoverable from EC2. - Time-sensitive: scrivas_backend received a commit on the assessment date. Reliability triage - Production runs 23 containers on a single 15 GiB host, including 3 Postgres instances, Kafka and OpenSearch, at 73% memory at rest with no per-container memory limits and no swap on any of the 7 instances. - Kafka, OpenSearch and search-api carry restart policy `no`, so a host reboot yields a partially-recovered stack. - Recorded as a structural exposure, not an observed root cause: no OOM event is present in retained logs and RestartCount is 0 on every prod container. Confirming the hypothesis needs CloudWatch history the boxes do not retain. Contents - scripts/ec2_code_discovery.py EC2 inventory (describe + user data) - scripts/ec2_code_inspect.py read-only SSM probe set, reviewable in PROBES - findings/ec2_code_discovery_report.md narrative writeup - findings/code_dashboard.html client-facing dashboard - findings/ec2_code_inspect*.json raw probe output - index.html links the new dashboard and evidence All access was read-only: no writes, restarts or config changes on any instance. Probe output was scanned for credentials before commit; git metadata was read as the owning user rather than by writing a safe.directory entry. Claude-Session: https://claude.ai/code/session_01YMxVaHXJsqpqKwncNQ9b1e
This commit is contained in:
Alvaro Del Valle
@@ -0,0 +1,318 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Scrivas — Source Code & Reliability</title>
|
||||
<style>
|
||||
:root {
|
||||
--bg:#eef0f3; --panel:#ffffff; --panel-2:#f7f8fa; --ink:#171d27;
|
||||
--ink-soft:#4a5563; --ink-faint:#79828f; --line:#dde1e7; --line-soft:#e8ebef;
|
||||
--accent:#0d6e78; --accent-ink:#0a565e;
|
||||
--good:#2f8657; --warn:#b5832a; --crit:#bd463b;
|
||||
--crit-bg:#fbeceb; --good-bg:#eaf4ee; --warn-bg:#f8f1e2;
|
||||
--mono: ui-monospace,"SF Mono","SFMono-Regular",Menlo,Consolas,monospace;
|
||||
--sans: ui-sans-serif,system-ui,"Segoe UI",Roboto,Helvetica,Arial,sans-serif;
|
||||
--shadow:0 1px 2px rgba(20,26,38,.04),0 6px 20px -12px rgba(20,26,38,.18);
|
||||
}
|
||||
@media (prefers-color-scheme:dark){:root{
|
||||
--bg:#0d1117;--panel:#161c26;--panel-2:#1b222e;--ink:#e8ecf1;--ink-soft:#aab3c0;
|
||||
--ink-faint:#7c8695;--line:#28303c;--line-soft:#212934;--accent:#3bb4bf;--accent-ink:#8fdfe6;
|
||||
--good:#54b681;--warn:#d6a94a;--crit:#e26b60;--crit-bg:#2a1917;--good-bg:#132419;--warn-bg:#251d10;
|
||||
--shadow:0 1px 2px rgba(0,0,0,.3),0 8px 24px -14px rgba(0,0,0,.6);}}
|
||||
:root[data-theme="dark"]{
|
||||
--bg:#0d1117;--panel:#161c26;--panel-2:#1b222e;--ink:#e8ecf1;--ink-soft:#aab3c0;
|
||||
--ink-faint:#7c8695;--line:#28303c;--line-soft:#212934;--accent:#3bb4bf;--accent-ink:#8fdfe6;
|
||||
--good:#54b681;--warn:#d6a94a;--crit:#e26b60;--crit-bg:#2a1917;--good-bg:#132419;--warn-bg:#251d10;
|
||||
--shadow:0 1px 2px rgba(0,0,0,.3),0 8px 24px -14px rgba(0,0,0,.6);}
|
||||
:root[data-theme="light"]{
|
||||
--bg:#eef0f3;--panel:#ffffff;--panel-2:#f7f8fa;--ink:#171d27;--ink-soft:#4a5563;
|
||||
--ink-faint:#79828f;--line:#dde1e7;--line-soft:#e8ebef;--accent:#0d6e78;--accent-ink:#0a565e;
|
||||
--good:#2f8657;--warn:#b5832a;--crit:#bd463b;--crit-bg:#fbeceb;--good-bg:#eaf4ee;--warn-bg:#f8f1e2;
|
||||
--shadow:0 1px 2px rgba(20,26,38,.04),0 6px 20px -12px rgba(20,26,38,.18);}
|
||||
|
||||
*{box-sizing:border-box;}
|
||||
body{margin:0;background:var(--bg);color:var(--ink);font-family:var(--sans);line-height:1.5;-webkit-font-smoothing:antialiased;}
|
||||
.wrap{max-width:1120px;margin:0 auto;padding:32px 24px 64px;}
|
||||
header{display:flex;flex-wrap:wrap;align-items:flex-end;justify-content:space-between;gap:16px;padding-bottom:20px;border-bottom:2px solid var(--ink);}
|
||||
.eyebrow{font-size:11px;letter-spacing:.16em;text-transform:uppercase;color:var(--accent);font-weight:700;}
|
||||
h1{font-size:clamp(24px,4vw,34px);margin:6px 0 4px;letter-spacing:-.02em;text-wrap:balance;font-weight:750;}
|
||||
.sub{color:var(--ink-soft);font-size:14px;}
|
||||
.idcard{font-family:var(--mono);font-size:12px;color:var(--ink-soft);text-align:right;line-height:1.7;}
|
||||
.idcard b{color:var(--ink);}
|
||||
|
||||
.kpis{display:grid;grid-template-columns:repeat(4,1fr);gap:14px;margin:22px 0 12px;}
|
||||
@media (max-width:760px){.kpis{grid-template-columns:repeat(2,1fr);}}
|
||||
.kpi{background:var(--panel);border:1px solid var(--line);border-radius:12px;padding:16px;box-shadow:var(--shadow);position:relative;overflow:hidden;}
|
||||
.kpi::before{content:"";position:absolute;left:0;top:0;bottom:0;width:3px;background:var(--accent);}
|
||||
.kpi.crit::before{background:var(--crit);}.kpi.warn::before{background:var(--warn);}.kpi.good::before{background:var(--good);}
|
||||
.kpi .n{font-size:28px;font-weight:750;letter-spacing:-.02em;font-variant-numeric:tabular-nums;}
|
||||
.kpi.crit .n{color:var(--crit);}.kpi.warn .n{color:var(--warn);}.kpi.good .n{color:var(--good);}
|
||||
.kpi .lbl{font-size:12px;color:var(--ink-faint);margin-top:2px;}
|
||||
.kpi .meta{font-size:11px;color:var(--ink-soft);margin-top:8px;}
|
||||
|
||||
section{background:var(--panel);border:1px solid var(--line);border-radius:14px;padding:22px 24px;margin-bottom:18px;box-shadow:var(--shadow);}
|
||||
.sec-head{display:flex;align-items:baseline;gap:10px;margin-bottom:16px;flex-wrap:wrap;}
|
||||
.sec-head .tag{font-family:var(--mono);font-size:12px;color:var(--accent);font-weight:700;}
|
||||
.sec-head h2{font-size:17px;margin:0;letter-spacing:-.01em;}
|
||||
.sec-head .note{margin-left:auto;font-size:12px;color:var(--ink-faint);}
|
||||
.grid2{display:grid;grid-template-columns:1fr 1fr;gap:18px;}
|
||||
@media (max-width:820px){.grid2{grid-template-columns:1fr;}}
|
||||
|
||||
table{width:100%;border-collapse:collapse;font-size:13px;}
|
||||
th{text-align:left;font-size:11px;text-transform:uppercase;letter-spacing:.07em;color:var(--ink-faint);
|
||||
font-weight:700;padding:0 10px 8px 0;border-bottom:1px solid var(--line);}
|
||||
td{padding:9px 10px 9px 0;border-bottom:1px solid var(--line-soft);vertical-align:top;color:var(--ink-soft);}
|
||||
td b{color:var(--ink);}
|
||||
tbody tr:last-child td{border-bottom:none;}
|
||||
.num{font-family:var(--mono);font-variant-numeric:tabular-nums;text-align:right;color:var(--ink);}
|
||||
code{font-family:var(--mono);font-size:12px;color:var(--ink-soft);}
|
||||
.scroll{overflow-x:auto;}
|
||||
|
||||
.pill{display:inline-flex;align-items:center;gap:6px;font-size:11px;font-weight:700;padding:3px 9px;border-radius:999px;white-space:nowrap;}
|
||||
.pill.crit{background:var(--crit-bg);color:var(--crit);}
|
||||
.pill.good{background:var(--good-bg);color:var(--good);}
|
||||
.pill.warn{background:var(--warn-bg);color:var(--warn);}
|
||||
|
||||
.callout{border-radius:10px;padding:14px 16px;font-size:13px;color:var(--ink-soft);margin:0 0 16px;
|
||||
background:var(--warn-bg);border:1px solid color-mix(in srgb,var(--warn) 30%,transparent);}
|
||||
.callout.crit{background:var(--crit-bg);border-color:color-mix(in srgb,var(--crit) 30%,transparent);}
|
||||
.callout.good{background:var(--good-bg);border-color:color-mix(in srgb,var(--good) 30%,transparent);}
|
||||
.callout b{color:var(--ink);}
|
||||
.callout p{margin:8px 0 0;}
|
||||
|
||||
.memtrack{height:26px;background:var(--panel-2);border:1px solid var(--line);border-radius:7px;overflow:hidden;display:flex;margin:6px 0 4px;}
|
||||
.memtrack .used{background:var(--crit);height:100%;}
|
||||
.memtrack .cache{background:color-mix(in srgb,var(--warn) 55%,transparent);height:100%;}
|
||||
.memlegend{display:flex;gap:16px;font-size:11.5px;color:var(--ink-faint);flex-wrap:wrap;}
|
||||
.memlegend i{display:inline-block;width:9px;height:9px;border-radius:2px;margin-right:5px;vertical-align:baseline;}
|
||||
|
||||
ol.steps{margin:0;padding-left:20px;font-size:13px;color:var(--ink-soft);}
|
||||
ol.steps li{margin-bottom:9px;}
|
||||
ol.steps b{color:var(--ink);}
|
||||
|
||||
footer{display:flex;flex-wrap:wrap;gap:18px;margin-top:24px;font-size:12px;color:var(--ink-faint);font-family:var(--mono);}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="wrap">
|
||||
|
||||
<header>
|
||||
<div>
|
||||
<div class="eyebrow">Dasnuve · Cloud Discovery</div>
|
||||
<h1>Source Code Recovery & Reliability Triage</h1>
|
||||
<div class="sub">What is running on the Scrivas EC2 estate, where the source lives, and why production is fragile.</div>
|
||||
</div>
|
||||
<div class="idcard">
|
||||
Account <b>716468089330</b><br>
|
||||
Region <b>us-east-2</b> · 7 instances<br>
|
||||
2026-08-28 · <b>CONFIDENTIAL</b>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<div class="kpis">
|
||||
<div class="kpi good">
|
||||
<div class="n">10</div>
|
||||
<div class="lbl">Repos recoverable on EC2</div>
|
||||
<div class="meta">Full git history, not just build output</div>
|
||||
</div>
|
||||
<div class="kpi warn">
|
||||
<div class="n">2</div>
|
||||
<div class="lbl">Frontends <u>not</u> recoverable</div>
|
||||
<div class="meta">Deployed build only — no <code>.git</code></div>
|
||||
</div>
|
||||
<div class="kpi crit">
|
||||
<div class="n">23</div>
|
||||
<div class="lbl">Prod containers, one host</div>
|
||||
<div class="meta">Incl. 3 × Postgres, Kafka, OpenSearch</div>
|
||||
</div>
|
||||
<div class="kpi crit">
|
||||
<div class="n">0</div>
|
||||
<div class="lbl">Memory limits set</div>
|
||||
<div class="meta">And zero swap, on all 7 instances</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<section>
|
||||
<div class="sec-head">
|
||||
<span class="tag">01</span><h2>The source code is recoverable — and that is time-sensitive</h2>
|
||||
<span class="note">Scrivas_dev_env · ML_dev</span>
|
||||
</div>
|
||||
|
||||
<div class="callout good">
|
||||
<b>All 10 application repositories exist as complete git checkouts on Scrivas-owned instances</b>, with full
|
||||
commit history — not merely deployed artifacts. Scrivas can recover its codebase from its own AWS account today.
|
||||
<p>Every repository points at a single external remote: <code>git@git.devteam.space:scrivas/<repo>.git</code> —
|
||||
the incumbent contractor's self-hosted GitLab, which Scrivas does <b>not</b> control. The on-instance checkouts are
|
||||
therefore the client's only independent leverage over their own source.</p>
|
||||
</div>
|
||||
|
||||
<div class="grid2">
|
||||
<div>
|
||||
<div class="sec-head"><span class="tag">APP</span><h2>Scrivas_dev_env <code>/home/admin/</code></h2></div>
|
||||
<div class="scroll"><table>
|
||||
<thead><tr><th>Repository</th><th class="num">Commits</th><th>Last commit</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><b>scrivas_backend</b></td><td class="num">1380</td><td>2026-08-28 · Vasilii</td></tr>
|
||||
<tr><td><b>scrivas_gate</b></td><td class="num">183</td><td>2026-08-14 · Vasilii</td></tr>
|
||||
<tr><td><b>scrivas_search</b></td><td class="num">96</td><td>2026-06-24 · Azamat</td></tr>
|
||||
<tr><td><b>patient</b></td><td class="num">59</td><td>2026-08-13 · vturtugeshev</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</div>
|
||||
<div>
|
||||
<div class="sec-head"><span class="tag">ML</span><h2>ML_dev <code>/srv/</code></h2></div>
|
||||
<div class="scroll"><table>
|
||||
<thead><tr><th>Repository</th><th class="num">Commits</th><th>Last commit</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td><b>post_processor</b></td><td class="num">137</td><td>2026-06-30</td></tr>
|
||||
<tr><td><b>sai_suggestions</b></td><td class="num">93</td><td>2026-06-16</td></tr>
|
||||
<tr><td><b>ml_monitoring</b></td><td class="num">62</td><td>2026-07-20</td></tr>
|
||||
<tr><td><b>patient-summary-service</b></td><td class="num">49</td><td>2026-06-12</td></tr>
|
||||
<tr><td><b>patient_document_parser</b></td><td class="num">41</td><td>2026-06-16</td></tr>
|
||||
<tr><td><b>soniox_transcriber</b></td><td class="num">25</td><td>2026-06-16</td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="callout" style="margin:18px 0 0;">
|
||||
<b>The gap:</b> <code>/var/www/scrivas_frontend</code> and <code>/var/www/Scrivas_admin_frontend</code> are
|
||||
<b>deployed build output with no <code>.git</code></b>. Frontend source cannot be recovered from EC2 and must be
|
||||
obtained from the contractor's GitLab.
|
||||
<p><b>Why this is urgent:</b> <code>scrivas_backend</code> received a commit <b>on the day of this assessment</b>.
|
||||
Development is active on infrastructure the contractor controls. Mirror all 10 repos
|
||||
(<code>git clone --mirror</code>) to Scrivas-controlled storage <i>before</i> any contract conversation changes
|
||||
access posture.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<div class="sec-head">
|
||||
<span class="tag">02</span><h2>Two tiers, two engineering standards</h2>
|
||||
<span class="note">the better pattern already exists in-house</span>
|
||||
</div>
|
||||
<div class="grid2">
|
||||
<div>
|
||||
<div class="sec-head"><span class="tag">APP</span><h2>Application tier <span class="pill crit">fragile</span></h2></div>
|
||||
<p class="sub" style="font-size:13px;">23 containers on a <b>single host</b> — <code>encounter_api</code>,
|
||||
<code>gate_api</code>, <code>patient_api</code>, 8 Celery workers, <code>celery_beat</code>,
|
||||
<b>3 × Postgres 17.5</b>, Redis, Kafka, Debezium CDC, OpenSearch, autoheal.</p>
|
||||
<p class="sub" style="font-size:13px;">Deployed by <code>docker-compose</code> directly from git checkouts.
|
||||
No orchestration, no image registry, no immutable artifact.</p>
|
||||
</div>
|
||||
<div>
|
||||
<div class="sec-head"><span class="tag">ML</span><h2>ML tier <span class="pill good">mature</span></h2></div>
|
||||
<p class="sub" style="font-size:13px;">Deployed from <b>ECR</b>
|
||||
(<code>…dkr.ecr.us-east-2…/scrivas/*</code>) with <b>commit-SHA image tags</b> and
|
||||
<b>blue/green slots</b> (<code>dev-green-*</code>, <code>stage-blue-*</code>).</p>
|
||||
<p class="sub" style="font-size:13px;">Health checks defined and passing across every service.
|
||||
This is a materially stronger delivery pipeline — and a template the application tier can adopt.</p>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<div class="sec-head">
|
||||
<span class="tag">03</span><h2>Reliability findings</h2>
|
||||
<span class="note">ranked by likely contribution to reported instability</span>
|
||||
</div>
|
||||
<div class="scroll"><table>
|
||||
<thead><tr><th>#</th><th>Finding</th><th>Evidence</th><th>Risk</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td class="num">1</td><td><b>No memory limit on any prod container</b></td><td><code>HostConfig.Memory=0</code> on all 23</td><td><span class="pill crit">High</span></td></tr>
|
||||
<tr><td class="num">2</td><td><b>No swap on any instance</b></td><td><code>swapon --show</code> empty, all 7</td><td><span class="pill crit">High</span></td></tr>
|
||||
<tr><td class="num">3</td><td><b>Prod at 73% memory at rest</b></td><td>11 GiB used of 15 GiB</td><td><span class="pill crit">High</span></td></tr>
|
||||
<tr><td class="num">4</td><td><b>Whole stack on one host, incl. 3 databases</b></td><td><code>docker ps</code> — 23 containers</td><td><span class="pill crit">High</span></td></tr>
|
||||
<tr><td class="num">5</td><td><b>3 prod containers set to never restart</b></td><td>Kafka, OpenSearch, search-api — policy <code>no</code></td><td><span class="pill crit">High</span></td></tr>
|
||||
<tr><td class="num">6</td><td>Prod OS differs from dev/stage</td><td>Ubuntu <b>26.04</b> vs <b>24.04</b> elsewhere</td><td><span class="pill warn">Medium</span></td></tr>
|
||||
<tr><td class="num">7</td><td><code>autoheal</code> deployed in production</td><td><code>willfarrell/autoheal:1.2.0</code></td><td><span class="pill warn">Medium</span></td></tr>
|
||||
<tr><td class="num">8</td><td>Uncommitted config drift in prod checkouts</td><td><code>docker-compose.yml.bkp</code>, <code>.env.save</code></td><td><span class="pill warn">Medium</span></td></tr>
|
||||
<tr><td class="num">9</td><td>47 interactive login sessions on prod</td><td><code>uptime</code> (32 on Ml_prod)</td><td><span class="pill warn">Medium</span></td></tr>
|
||||
<tr><td class="num">10</td><td><code>sai-suggestions-redis.service</code> failed</td><td>ML_stage <code>systemctl --state=failed</code></td><td><span class="pill good">Low</span></td></tr>
|
||||
</tbody>
|
||||
</table></div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<div class="sec-head">
|
||||
<span class="tag">04</span><h2>Primary hypothesis: unbounded memory on a shared host</h2>
|
||||
</div>
|
||||
|
||||
<p class="sub" style="font-size:13px;margin-top:0;">Production memory, at rest — before any workload spike:</p>
|
||||
<div class="memtrack" role="img" aria-label="Production memory: 11 GiB used, 3.5 GiB cache, of 15 GiB total">
|
||||
<div class="used" style="width:73%"></div><div class="cache" style="width:23%"></div>
|
||||
</div>
|
||||
<div class="memlegend">
|
||||
<span><i style="background:var(--crit)"></i>11 GiB used (73%)</span>
|
||||
<span><i style="background:color-mix(in srgb,var(--warn) 55%,transparent)"></i>3.5 GiB cache</span>
|
||||
<span><i style="background:var(--panel-2);border:1px solid var(--line)"></i>441 MiB free</span>
|
||||
<span><b style="color:var(--crit)">0 B swap</b></span>
|
||||
</div>
|
||||
|
||||
<div class="callout crit" style="margin-top:18px;">
|
||||
Findings 1–4 compound into a single failure mode. Production runs 23 containers — including three Postgres
|
||||
instances, Kafka and OpenSearch — on one 15 GiB host already <b>73% consumed at rest</b>, with
|
||||
<b>no per-container memory limits</b> and <b>no swap</b>. A single Celery worker handling an oversized document
|
||||
or audio chunk can exhaust host memory. With no limits, the kernel OOM killer selects by resident size — so it
|
||||
will typically kill a <b>database</b>, not the worker that caused the problem.
|
||||
<p>That yields precisely the profile a client reports as "random reliability issues": unpredictable, uncorrelated
|
||||
with the triggering workload, and difficult to reproduce.</p>
|
||||
<p><b>Finding 5 compounds recovery.</b> Kafka, OpenSearch and search-api carry restart policy <code>no</code> — if
|
||||
they die or the host reboots, they <b>stay down</b> until someone intervenes. Everything else is
|
||||
<code>unless-stopped</code>, so a reboot yields a partially-recovered stack that looks healthy from the outside.</p>
|
||||
<p><b>Finding 7 corroborates.</b> Someone deliberately deployed <code>autoheal</code>, a daemon whose only job is
|
||||
restarting containers that fail health checks. That treats a symptom — and indicates health-check failures were
|
||||
frequent enough to be worth automating around.</p>
|
||||
</div>
|
||||
|
||||
<div class="callout">
|
||||
<b>What the evidence does not show — stated plainly, because it bounds the conclusion.</b>
|
||||
<p>· <b>No OOM kill was captured.</b> <code>dmesg</code> and <code>journalctl</code> on prod show no OOM events in
|
||||
the retained window; <code>State.OOMKilled</code> is <code>false</code> on all 23 containers.</p>
|
||||
<p>· <b>No crash-looping.</b> <code>RestartCount</code> is <b>0</b> on every prod container. The Celery workers
|
||||
showing "Up 6 hours" were <b>redeployed</b> at 13:49 that day, not restarted by failure.</p>
|
||||
<p>· <code>last</code> is not installed on prod, so reboot history could not be read there.</p>
|
||||
<p>Findings 1–5 are therefore a demonstrated <b>structural exposure</b>, not an observed root cause. Confirming
|
||||
the hypothesis requires history these instances do not currently retain — which is itself a finding.</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section>
|
||||
<div class="sec-head"><span class="tag">05</span><h2>Recommended next steps</h2></div>
|
||||
<div class="grid2">
|
||||
<div>
|
||||
<div class="sec-head"><span class="tag">NOW</span><h2>Immediate</h2></div>
|
||||
<ol class="steps">
|
||||
<li><b>Mirror all 10 repos</b> to Scrivas-controlled storage. Highest value, lowest effort, time-sensitive.</li>
|
||||
<li><b>Set <code>unless-stopped</code></b> on Kafka, OpenSearch and search-api — removes the silent partial-recovery failure.</li>
|
||||
</ol>
|
||||
<div class="sec-head" style="margin-top:18px;"><span class="tag">PROVE</span><h2>Confirm the hypothesis</h2></div>
|
||||
<ol class="steps" start="3">
|
||||
<li><b>Enable per-container memory metrics</b> in CloudWatch — <code>EC2-CloudWatchAgent-Role</code> is already attached.</li>
|
||||
<li><b>Add <code>mem_limit</code> per service</b>, sized from observed peaks; add swap as an OOM buffer.</li>
|
||||
<li><b>Query historic <code>mem_used_percent</code></b> to date the onset of instability.</li>
|
||||
</ol>
|
||||
</div>
|
||||
<div>
|
||||
<div class="sec-head"><span class="tag">FIX</span><h2>Structural</h2></div>
|
||||
<ol class="steps" start="6">
|
||||
<li><b>Move the three Postgres instances</b> off the shared application host.</li>
|
||||
<li><b>Align prod OS/AMI with stage</b> — 26.04 vs 24.04 makes staging an unreliable rehearsal for production.</li>
|
||||
<li><b>Extend the ML tier's pattern</b> — ECR, commit-SHA tags, blue/green — to the application tier. The better pattern already exists in-house.</li>
|
||||
<li><b>Reconcile config drift</b> (<code>.env.save</code>, <code>docker-compose.yml.bkp</code>) into version control.</li>
|
||||
<li><b>Audit the 47 interactive prod sessions.</b></li>
|
||||
</ol>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<footer>
|
||||
<span>Read-only · SSM probes via <code>dasnuve-scrivas-louis-impersonation</code></span>
|
||||
<span>Reproduce: <code>scripts/ec2_code_inspect.py</code></span>
|
||||
<span>Raw: <code>findings/ec2_code_inspect.json</code></span>
|
||||
<span>Narrative: <code>findings/ec2_code_discovery_report.md</code></span>
|
||||
</footer>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user