Files
scrivas/findings/code_dashboard.html
T
Alvaro Del Valle 44d4bf6748 Add EC2 source-code discovery and reliability triage
Second discovery pass over the Scrivas EC2 estate (716468089330, us-east-2,
7 instances), prompted by the client reporting reliability issues and by their
lack of access to source code held under contract by the incumbent vendor.

Source code recovery
- All 10 application repositories exist as complete git checkouts on
  Scrivas-owned instances, with full history rather than deployed artifacts:
  4 app repos on Scrivas_dev_env, 6 ML repos on ML_dev.
- Every remote points at git@git.devteam.space (the contractor's self-hosted
  GitLab), which Scrivas does not control. The on-instance checkouts are the
  client's only independent leverage over their own source.
- Gap: both /var/www frontends are build output with no .git, so frontend
  source is not recoverable from EC2.
- Time-sensitive: scrivas_backend received a commit on the assessment date.

Reliability triage
- Production runs 23 containers on a single 15 GiB host, including 3 Postgres
  instances, Kafka and OpenSearch, at 73% memory at rest with no per-container
  memory limits and no swap on any of the 7 instances.
- Kafka, OpenSearch and search-api carry restart policy `no`, so a host reboot
  yields a partially-recovered stack.
- Recorded as a structural exposure, not an observed root cause: no OOM event
  is present in retained logs and RestartCount is 0 on every prod container.
  Confirming the hypothesis needs CloudWatch history the boxes do not retain.

Contents
- scripts/ec2_code_discovery.py  EC2 inventory (describe + user data)
- scripts/ec2_code_inspect.py    read-only SSM probe set, reviewable in PROBES
- findings/ec2_code_discovery_report.md   narrative writeup
- findings/code_dashboard.html            client-facing dashboard
- findings/ec2_code_inspect*.json         raw probe output
- index.html                              links the new dashboard and evidence

All access was read-only: no writes, restarts or config changes on any
instance. Probe output was scanned for credentials before commit; git metadata
was read as the owning user rather than by writing a safe.directory entry.

Claude-Session: https://claude.ai/code/session_01YMxVaHXJsqpqKwncNQ9b1e
2026-08-28 16:24:24 -04:00

319 lines
20 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Scrivas — Source Code &amp; Reliability</title>
<style>
:root {
--bg:#eef0f3; --panel:#ffffff; --panel-2:#f7f8fa; --ink:#171d27;
--ink-soft:#4a5563; --ink-faint:#79828f; --line:#dde1e7; --line-soft:#e8ebef;
--accent:#0d6e78; --accent-ink:#0a565e;
--good:#2f8657; --warn:#b5832a; --crit:#bd463b;
--crit-bg:#fbeceb; --good-bg:#eaf4ee; --warn-bg:#f8f1e2;
--mono: ui-monospace,"SF Mono","SFMono-Regular",Menlo,Consolas,monospace;
--sans: ui-sans-serif,system-ui,"Segoe UI",Roboto,Helvetica,Arial,sans-serif;
--shadow:0 1px 2px rgba(20,26,38,.04),0 6px 20px -12px rgba(20,26,38,.18);
}
@media (prefers-color-scheme:dark){:root{
--bg:#0d1117;--panel:#161c26;--panel-2:#1b222e;--ink:#e8ecf1;--ink-soft:#aab3c0;
--ink-faint:#7c8695;--line:#28303c;--line-soft:#212934;--accent:#3bb4bf;--accent-ink:#8fdfe6;
--good:#54b681;--warn:#d6a94a;--crit:#e26b60;--crit-bg:#2a1917;--good-bg:#132419;--warn-bg:#251d10;
--shadow:0 1px 2px rgba(0,0,0,.3),0 8px 24px -14px rgba(0,0,0,.6);}}
:root[data-theme="dark"]{
--bg:#0d1117;--panel:#161c26;--panel-2:#1b222e;--ink:#e8ecf1;--ink-soft:#aab3c0;
--ink-faint:#7c8695;--line:#28303c;--line-soft:#212934;--accent:#3bb4bf;--accent-ink:#8fdfe6;
--good:#54b681;--warn:#d6a94a;--crit:#e26b60;--crit-bg:#2a1917;--good-bg:#132419;--warn-bg:#251d10;
--shadow:0 1px 2px rgba(0,0,0,.3),0 8px 24px -14px rgba(0,0,0,.6);}
:root[data-theme="light"]{
--bg:#eef0f3;--panel:#ffffff;--panel-2:#f7f8fa;--ink:#171d27;--ink-soft:#4a5563;
--ink-faint:#79828f;--line:#dde1e7;--line-soft:#e8ebef;--accent:#0d6e78;--accent-ink:#0a565e;
--good:#2f8657;--warn:#b5832a;--crit:#bd463b;--crit-bg:#fbeceb;--good-bg:#eaf4ee;--warn-bg:#f8f1e2;
--shadow:0 1px 2px rgba(20,26,38,.04),0 6px 20px -12px rgba(20,26,38,.18);}
*{box-sizing:border-box;}
body{margin:0;background:var(--bg);color:var(--ink);font-family:var(--sans);line-height:1.5;-webkit-font-smoothing:antialiased;}
.wrap{max-width:1120px;margin:0 auto;padding:32px 24px 64px;}
header{display:flex;flex-wrap:wrap;align-items:flex-end;justify-content:space-between;gap:16px;padding-bottom:20px;border-bottom:2px solid var(--ink);}
.eyebrow{font-size:11px;letter-spacing:.16em;text-transform:uppercase;color:var(--accent);font-weight:700;}
h1{font-size:clamp(24px,4vw,34px);margin:6px 0 4px;letter-spacing:-.02em;text-wrap:balance;font-weight:750;}
.sub{color:var(--ink-soft);font-size:14px;}
.idcard{font-family:var(--mono);font-size:12px;color:var(--ink-soft);text-align:right;line-height:1.7;}
.idcard b{color:var(--ink);}
.kpis{display:grid;grid-template-columns:repeat(4,1fr);gap:14px;margin:22px 0 12px;}
@media (max-width:760px){.kpis{grid-template-columns:repeat(2,1fr);}}
.kpi{background:var(--panel);border:1px solid var(--line);border-radius:12px;padding:16px;box-shadow:var(--shadow);position:relative;overflow:hidden;}
.kpi::before{content:"";position:absolute;left:0;top:0;bottom:0;width:3px;background:var(--accent);}
.kpi.crit::before{background:var(--crit);}.kpi.warn::before{background:var(--warn);}.kpi.good::before{background:var(--good);}
.kpi .n{font-size:28px;font-weight:750;letter-spacing:-.02em;font-variant-numeric:tabular-nums;}
.kpi.crit .n{color:var(--crit);}.kpi.warn .n{color:var(--warn);}.kpi.good .n{color:var(--good);}
.kpi .lbl{font-size:12px;color:var(--ink-faint);margin-top:2px;}
.kpi .meta{font-size:11px;color:var(--ink-soft);margin-top:8px;}
section{background:var(--panel);border:1px solid var(--line);border-radius:14px;padding:22px 24px;margin-bottom:18px;box-shadow:var(--shadow);}
.sec-head{display:flex;align-items:baseline;gap:10px;margin-bottom:16px;flex-wrap:wrap;}
.sec-head .tag{font-family:var(--mono);font-size:12px;color:var(--accent);font-weight:700;}
.sec-head h2{font-size:17px;margin:0;letter-spacing:-.01em;}
.sec-head .note{margin-left:auto;font-size:12px;color:var(--ink-faint);}
.grid2{display:grid;grid-template-columns:1fr 1fr;gap:18px;}
@media (max-width:820px){.grid2{grid-template-columns:1fr;}}
table{width:100%;border-collapse:collapse;font-size:13px;}
th{text-align:left;font-size:11px;text-transform:uppercase;letter-spacing:.07em;color:var(--ink-faint);
font-weight:700;padding:0 10px 8px 0;border-bottom:1px solid var(--line);}
td{padding:9px 10px 9px 0;border-bottom:1px solid var(--line-soft);vertical-align:top;color:var(--ink-soft);}
td b{color:var(--ink);}
tbody tr:last-child td{border-bottom:none;}
.num{font-family:var(--mono);font-variant-numeric:tabular-nums;text-align:right;color:var(--ink);}
code{font-family:var(--mono);font-size:12px;color:var(--ink-soft);}
.scroll{overflow-x:auto;}
.pill{display:inline-flex;align-items:center;gap:6px;font-size:11px;font-weight:700;padding:3px 9px;border-radius:999px;white-space:nowrap;}
.pill.crit{background:var(--crit-bg);color:var(--crit);}
.pill.good{background:var(--good-bg);color:var(--good);}
.pill.warn{background:var(--warn-bg);color:var(--warn);}
.callout{border-radius:10px;padding:14px 16px;font-size:13px;color:var(--ink-soft);margin:0 0 16px;
background:var(--warn-bg);border:1px solid color-mix(in srgb,var(--warn) 30%,transparent);}
.callout.crit{background:var(--crit-bg);border-color:color-mix(in srgb,var(--crit) 30%,transparent);}
.callout.good{background:var(--good-bg);border-color:color-mix(in srgb,var(--good) 30%,transparent);}
.callout b{color:var(--ink);}
.callout p{margin:8px 0 0;}
.memtrack{height:26px;background:var(--panel-2);border:1px solid var(--line);border-radius:7px;overflow:hidden;display:flex;margin:6px 0 4px;}
.memtrack .used{background:var(--crit);height:100%;}
.memtrack .cache{background:color-mix(in srgb,var(--warn) 55%,transparent);height:100%;}
.memlegend{display:flex;gap:16px;font-size:11.5px;color:var(--ink-faint);flex-wrap:wrap;}
.memlegend i{display:inline-block;width:9px;height:9px;border-radius:2px;margin-right:5px;vertical-align:baseline;}
ol.steps{margin:0;padding-left:20px;font-size:13px;color:var(--ink-soft);}
ol.steps li{margin-bottom:9px;}
ol.steps b{color:var(--ink);}
footer{display:flex;flex-wrap:wrap;gap:18px;margin-top:24px;font-size:12px;color:var(--ink-faint);font-family:var(--mono);}
</style>
</head>
<body>
<div class="wrap">
<header>
<div>
<div class="eyebrow">Dasnuve · Cloud Discovery</div>
<h1>Source Code Recovery &amp; Reliability Triage</h1>
<div class="sub">What is running on the Scrivas EC2 estate, where the source lives, and why production is fragile.</div>
</div>
<div class="idcard">
Account <b>716468089330</b><br>
Region <b>us-east-2</b> · 7 instances<br>
2026-08-28 · <b>CONFIDENTIAL</b>
</div>
</header>
<div class="kpis">
<div class="kpi good">
<div class="n">10</div>
<div class="lbl">Repos recoverable on EC2</div>
<div class="meta">Full git history, not just build output</div>
</div>
<div class="kpi warn">
<div class="n">2</div>
<div class="lbl">Frontends <u>not</u> recoverable</div>
<div class="meta">Deployed build only — no <code>.git</code></div>
</div>
<div class="kpi crit">
<div class="n">23</div>
<div class="lbl">Prod containers, one host</div>
<div class="meta">Incl. 3 × Postgres, Kafka, OpenSearch</div>
</div>
<div class="kpi crit">
<div class="n">0</div>
<div class="lbl">Memory limits set</div>
<div class="meta">And zero swap, on all 7 instances</div>
</div>
</div>
<section>
<div class="sec-head">
<span class="tag">01</span><h2>The source code is recoverable — and that is time-sensitive</h2>
<span class="note">Scrivas_dev_env · ML_dev</span>
</div>
<div class="callout good">
<b>All 10 application repositories exist as complete git checkouts on Scrivas-owned instances</b>, with full
commit history — not merely deployed artifacts. Scrivas can recover its codebase from its own AWS account today.
<p>Every repository points at a single external remote: <code>git@git.devteam.space:scrivas/&lt;repo&gt;.git</code>
the incumbent contractor's self-hosted GitLab, which Scrivas does <b>not</b> control. The on-instance checkouts are
therefore the client's only independent leverage over their own source.</p>
</div>
<div class="grid2">
<div>
<div class="sec-head"><span class="tag">APP</span><h2>Scrivas_dev_env <code>/home/admin/</code></h2></div>
<div class="scroll"><table>
<thead><tr><th>Repository</th><th class="num">Commits</th><th>Last commit</th></tr></thead>
<tbody>
<tr><td><b>scrivas_backend</b></td><td class="num">1380</td><td>2026-08-28 · Vasilii</td></tr>
<tr><td><b>scrivas_gate</b></td><td class="num">183</td><td>2026-08-14 · Vasilii</td></tr>
<tr><td><b>scrivas_search</b></td><td class="num">96</td><td>2026-06-24 · Azamat</td></tr>
<tr><td><b>patient</b></td><td class="num">59</td><td>2026-08-13 · vturtugeshev</td></tr>
</tbody>
</table></div>
</div>
<div>
<div class="sec-head"><span class="tag">ML</span><h2>ML_dev <code>/srv/</code></h2></div>
<div class="scroll"><table>
<thead><tr><th>Repository</th><th class="num">Commits</th><th>Last commit</th></tr></thead>
<tbody>
<tr><td><b>post_processor</b></td><td class="num">137</td><td>2026-06-30</td></tr>
<tr><td><b>sai_suggestions</b></td><td class="num">93</td><td>2026-06-16</td></tr>
<tr><td><b>ml_monitoring</b></td><td class="num">62</td><td>2026-07-20</td></tr>
<tr><td><b>patient-summary-service</b></td><td class="num">49</td><td>2026-06-12</td></tr>
<tr><td><b>patient_document_parser</b></td><td class="num">41</td><td>2026-06-16</td></tr>
<tr><td><b>soniox_transcriber</b></td><td class="num">25</td><td>2026-06-16</td></tr>
</tbody>
</table></div>
</div>
</div>
<div class="callout" style="margin:18px 0 0;">
<b>The gap:</b> <code>/var/www/scrivas_frontend</code> and <code>/var/www/Scrivas_admin_frontend</code> are
<b>deployed build output with no <code>.git</code></b>. Frontend source cannot be recovered from EC2 and must be
obtained from the contractor's GitLab.
<p><b>Why this is urgent:</b> <code>scrivas_backend</code> received a commit <b>on the day of this assessment</b>.
Development is active on infrastructure the contractor controls. Mirror all 10 repos
(<code>git clone --mirror</code>) to Scrivas-controlled storage <i>before</i> any contract conversation changes
access posture.</p>
</div>
</section>
<section>
<div class="sec-head">
<span class="tag">02</span><h2>Two tiers, two engineering standards</h2>
<span class="note">the better pattern already exists in-house</span>
</div>
<div class="grid2">
<div>
<div class="sec-head"><span class="tag">APP</span><h2>Application tier <span class="pill crit">fragile</span></h2></div>
<p class="sub" style="font-size:13px;">23 containers on a <b>single host</b><code>encounter_api</code>,
<code>gate_api</code>, <code>patient_api</code>, 8 Celery workers, <code>celery_beat</code>,
<b>3 × Postgres 17.5</b>, Redis, Kafka, Debezium CDC, OpenSearch, autoheal.</p>
<p class="sub" style="font-size:13px;">Deployed by <code>docker-compose</code> directly from git checkouts.
No orchestration, no image registry, no immutable artifact.</p>
</div>
<div>
<div class="sec-head"><span class="tag">ML</span><h2>ML tier <span class="pill good">mature</span></h2></div>
<p class="sub" style="font-size:13px;">Deployed from <b>ECR</b>
(<code>…dkr.ecr.us-east-2…/scrivas/*</code>) with <b>commit-SHA image tags</b> and
<b>blue/green slots</b> (<code>dev-green-*</code>, <code>stage-blue-*</code>).</p>
<p class="sub" style="font-size:13px;">Health checks defined and passing across every service.
This is a materially stronger delivery pipeline — and a template the application tier can adopt.</p>
</div>
</div>
</section>
<section>
<div class="sec-head">
<span class="tag">03</span><h2>Reliability findings</h2>
<span class="note">ranked by likely contribution to reported instability</span>
</div>
<div class="scroll"><table>
<thead><tr><th>#</th><th>Finding</th><th>Evidence</th><th>Risk</th></tr></thead>
<tbody>
<tr><td class="num">1</td><td><b>No memory limit on any prod container</b></td><td><code>HostConfig.Memory=0</code> on all 23</td><td><span class="pill crit">High</span></td></tr>
<tr><td class="num">2</td><td><b>No swap on any instance</b></td><td><code>swapon --show</code> empty, all 7</td><td><span class="pill crit">High</span></td></tr>
<tr><td class="num">3</td><td><b>Prod at 73% memory at rest</b></td><td>11 GiB used of 15 GiB</td><td><span class="pill crit">High</span></td></tr>
<tr><td class="num">4</td><td><b>Whole stack on one host, incl. 3 databases</b></td><td><code>docker ps</code> — 23 containers</td><td><span class="pill crit">High</span></td></tr>
<tr><td class="num">5</td><td><b>3 prod containers set to never restart</b></td><td>Kafka, OpenSearch, search-api — policy <code>no</code></td><td><span class="pill crit">High</span></td></tr>
<tr><td class="num">6</td><td>Prod OS differs from dev/stage</td><td>Ubuntu <b>26.04</b> vs <b>24.04</b> elsewhere</td><td><span class="pill warn">Medium</span></td></tr>
<tr><td class="num">7</td><td><code>autoheal</code> deployed in production</td><td><code>willfarrell/autoheal:1.2.0</code></td><td><span class="pill warn">Medium</span></td></tr>
<tr><td class="num">8</td><td>Uncommitted config drift in prod checkouts</td><td><code>docker-compose.yml.bkp</code>, <code>.env.save</code></td><td><span class="pill warn">Medium</span></td></tr>
<tr><td class="num">9</td><td>47 interactive login sessions on prod</td><td><code>uptime</code> (32 on Ml_prod)</td><td><span class="pill warn">Medium</span></td></tr>
<tr><td class="num">10</td><td><code>sai-suggestions-redis.service</code> failed</td><td>ML_stage <code>systemctl --state=failed</code></td><td><span class="pill good">Low</span></td></tr>
</tbody>
</table></div>
</section>
<section>
<div class="sec-head">
<span class="tag">04</span><h2>Primary hypothesis: unbounded memory on a shared host</h2>
</div>
<p class="sub" style="font-size:13px;margin-top:0;">Production memory, at rest — before any workload spike:</p>
<div class="memtrack" role="img" aria-label="Production memory: 11 GiB used, 3.5 GiB cache, of 15 GiB total">
<div class="used" style="width:73%"></div><div class="cache" style="width:23%"></div>
</div>
<div class="memlegend">
<span><i style="background:var(--crit)"></i>11 GiB used (73%)</span>
<span><i style="background:color-mix(in srgb,var(--warn) 55%,transparent)"></i>3.5 GiB cache</span>
<span><i style="background:var(--panel-2);border:1px solid var(--line)"></i>441 MiB free</span>
<span><b style="color:var(--crit)">0 B swap</b></span>
</div>
<div class="callout crit" style="margin-top:18px;">
Findings 14 compound into a single failure mode. Production runs 23 containers — including three Postgres
instances, Kafka and OpenSearch — on one 15 GiB host already <b>73% consumed at rest</b>, with
<b>no per-container memory limits</b> and <b>no swap</b>. A single Celery worker handling an oversized document
or audio chunk can exhaust host memory. With no limits, the kernel OOM killer selects by resident size — so it
will typically kill a <b>database</b>, not the worker that caused the problem.
<p>That yields precisely the profile a client reports as "random reliability issues": unpredictable, uncorrelated
with the triggering workload, and difficult to reproduce.</p>
<p><b>Finding 5 compounds recovery.</b> Kafka, OpenSearch and search-api carry restart policy <code>no</code> — if
they die or the host reboots, they <b>stay down</b> until someone intervenes. Everything else is
<code>unless-stopped</code>, so a reboot yields a partially-recovered stack that looks healthy from the outside.</p>
<p><b>Finding 7 corroborates.</b> Someone deliberately deployed <code>autoheal</code>, a daemon whose only job is
restarting containers that fail health checks. That treats a symptom — and indicates health-check failures were
frequent enough to be worth automating around.</p>
</div>
<div class="callout">
<b>What the evidence does not show — stated plainly, because it bounds the conclusion.</b>
<p>· <b>No OOM kill was captured.</b> <code>dmesg</code> and <code>journalctl</code> on prod show no OOM events in
the retained window; <code>State.OOMKilled</code> is <code>false</code> on all 23 containers.</p>
<p>· <b>No crash-looping.</b> <code>RestartCount</code> is <b>0</b> on every prod container. The Celery workers
showing "Up 6 hours" were <b>redeployed</b> at 13:49 that day, not restarted by failure.</p>
<p>· <code>last</code> is not installed on prod, so reboot history could not be read there.</p>
<p>Findings 15 are therefore a demonstrated <b>structural exposure</b>, not an observed root cause. Confirming
the hypothesis requires history these instances do not currently retain — which is itself a finding.</p>
</div>
</section>
<section>
<div class="sec-head"><span class="tag">05</span><h2>Recommended next steps</h2></div>
<div class="grid2">
<div>
<div class="sec-head"><span class="tag">NOW</span><h2>Immediate</h2></div>
<ol class="steps">
<li><b>Mirror all 10 repos</b> to Scrivas-controlled storage. Highest value, lowest effort, time-sensitive.</li>
<li><b>Set <code>unless-stopped</code></b> on Kafka, OpenSearch and search-api — removes the silent partial-recovery failure.</li>
</ol>
<div class="sec-head" style="margin-top:18px;"><span class="tag">PROVE</span><h2>Confirm the hypothesis</h2></div>
<ol class="steps" start="3">
<li><b>Enable per-container memory metrics</b> in CloudWatch — <code>EC2-CloudWatchAgent-Role</code> is already attached.</li>
<li><b>Add <code>mem_limit</code> per service</b>, sized from observed peaks; add swap as an OOM buffer.</li>
<li><b>Query historic <code>mem_used_percent</code></b> to date the onset of instability.</li>
</ol>
</div>
<div>
<div class="sec-head"><span class="tag">FIX</span><h2>Structural</h2></div>
<ol class="steps" start="6">
<li><b>Move the three Postgres instances</b> off the shared application host.</li>
<li><b>Align prod OS/AMI with stage</b> — 26.04 vs 24.04 makes staging an unreliable rehearsal for production.</li>
<li><b>Extend the ML tier's pattern</b> — ECR, commit-SHA tags, blue/green — to the application tier. The better pattern already exists in-house.</li>
<li><b>Reconcile config drift</b> (<code>.env.save</code>, <code>docker-compose.yml.bkp</code>) into version control.</li>
<li><b>Audit the 47 interactive prod sessions.</b></li>
</ol>
</div>
</div>
</section>
<footer>
<span>Read-only · SSM probes via <code>dasnuve-scrivas-louis-impersonation</code></span>
<span>Reproduce: <code>scripts/ec2_code_inspect.py</code></span>
<span>Raw: <code>findings/ec2_code_inspect.json</code></span>
<span>Narrative: <code>findings/ec2_code_discovery_report.md</code></span>
</footer>
</div>
</body>
</html>