Files
adelvalleandAlvaro Del Valle c987ed07c5 EC2 source-code discovery and reliability triage (#2)
Second discovery pass over the Scrivas EC2 estate (`716468089330`, `us-east-2`, 7 instances). Prompted by two client signals: they do not hold the source code for their own platform (built under contract by the incumbent vendor), and they are reporting production reliability issues.

All access was **read-only** — no writes, restarts or config changes on any instance.

## Source code is recoverable

**All 10 application repositories exist as complete git checkouts on Scrivas-owned instances**, with full commit history rather than deployed artifacts.

| Tier | Location | Repos |
|---|---|---|
| App | `Scrivas_dev_env:/home/admin/` | `scrivas_backend` (1380 commits), `scrivas_gate` (183), `scrivas_search` (96), `patient` (59) |
| ML | `ML_dev:/srv/` | `post_processor` (137), `sai_suggestions` (93), `ml_monitoring` (62), `patient-summary-service` (49), `patient_document_parser` (41), `soniox_transcriber` (25) |

Every remote points at `git@git.devteam.space:scrivas/*` — the contractor's self-hosted GitLab, which Scrivas does **not** control. The on-instance checkouts are the client's only independent leverage over their own source.

**Gap:** both `/var/www` frontends are build output with no `.git`. Frontend source is not recoverable from EC2.

**Time-sensitive:** `scrivas_backend` received a commit on the assessment date. Development is active on infrastructure the contractor controls. Mirroring the repos to Scrivas-controlled storage is the recommended immediate action and is **not** included in this PR — it needs a scope decision first (it involves the contractor's GitLab credentials held on the dev box).

## Reliability triage

Production runs **23 containers on a single 15 GiB host** — including 3 Postgres instances, Kafka and OpenSearch — at **73% memory at rest**, with **no per-container memory limits** and **no swap** on any of the 7 instances. With no limits the OOM killer selects by resident size, so it would typically kill a database rather than the worker that caused the pressure. That matches the "random, unreproducible" symptom profile.

Kafka, OpenSearch and search-api additionally carry restart policy `no`, so a host reboot yields a partially-recovered stack that looks healthy from outside.

**Recorded as a structural exposure, not an observed root cause.** No OOM event appears in retained logs, `State.OOMKilled` is false on all 23 containers, and `RestartCount` is **0** on every one — the Celery workers showing "Up 6 hours" were redeployed, not crash-restarted. Confirming the hypothesis needs CloudWatch history these boxes do not retain, which is itself a finding and the basis for recommendations 3–5 in the report.

One encouraging contrast: the **ML tier is well-built** — ECR images tagged by commit SHA, blue/green slots, passing health checks. The application tier is compose-from-git-checkout with uncommitted `.env.save` and `docker-compose.yml.bkp` files in the prod working tree. The better pattern already exists in-house.

## Contents

| File | |
|---|---|
| `scripts/ec2_code_discovery.py` | EC2 inventory — describe + user data |
| `scripts/ec2_code_inspect.py` | read-only SSM probe set; commands reviewable in `PROBES` |
| `findings/ec2_code_discovery_report.md` | narrative writeup |
| `findings/code_dashboard.html` | client-facing dashboard, matching the existing design system |
| `findings/ec2_code_inspect*.json` | raw probe output |
| `index.html` | links the new dashboard and evidence |

## Review notes

- Probe output was **scanned for credentials before commit** ��� no AWS keys, passwords, tokens or private key blocks. The probe set reads manifests and VCS metadata, never file contents.
- Git metadata was read **as the owning user** (`sudo -u`) rather than by writing a `safe.directory` entry into root's gitconfig, to preserve the read-only guarantee.
- Several prod probes initially returned empty and were re-run with stderr visible; the empties were artifacts (`last` is not installed on prod), not clean health. Worth knowing when reading the JSON.
- The dashboard keeps the "what the evidence does not show" section in the **client-facing** version. If CloudWatch later points elsewhere, overclaiming a root cause would cost more credibility than the softer framing gains.

https://claude.ai/code/session_01YMxVaHXJsqpqKwncNQ9b1e
Co-authored-by: Alvaro Del Valle <alvaro.delvalle2@gmail.com>
Reviewed-on: #2
2026-09-02 14:36:28 -06:00

111 lines
6.9 KiB
HTML

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Scrivas — AWS Discovery Reports</title>
<style>
:root{
--bg:#eef0f3;--panel:#ffffff;--panel-2:#f7f8fa;--ink:#171d27;--ink-soft:#4a5563;
--ink-faint:#79828f;--line:#dde1e7;--accent:#0d6e78;--accent-ink:#0a565e;
--good:#2f8657;--warn:#b5832a;--crit:#bd463b;--warn-bg:#f8f1e2;
--mono:ui-monospace,"SF Mono",Menlo,Consolas,monospace;
--sans:ui-sans-serif,system-ui,"Segoe UI",Roboto,Helvetica,Arial,sans-serif;
--shadow:0 1px 2px rgba(20,26,38,.04),0 6px 20px -12px rgba(20,26,38,.18);
}
@media (prefers-color-scheme:dark){:root{
--bg:#0d1117;--panel:#161c26;--panel-2:#1b222e;--ink:#e8ecf1;--ink-soft:#aab3c0;
--ink-faint:#7c8695;--line:#28303c;--accent:#3bb4bf;--accent-ink:#8fdfe6;
--good:#54b681;--warn:#d6a94a;--crit:#e26b60;--warn-bg:#251d10;
--shadow:0 1px 2px rgba(0,0,0,.3),0 8px 24px -14px rgba(0,0,0,.6);}}
*{box-sizing:border-box;}
body{margin:0;background:var(--bg);color:var(--ink);font-family:var(--sans);line-height:1.5;-webkit-font-smoothing:antialiased;}
.wrap{max-width:880px;margin:0 auto;padding:44px 24px 64px;}
.eyebrow{font-size:11px;letter-spacing:.16em;text-transform:uppercase;color:var(--accent);font-weight:700;}
h1{font-size:clamp(26px,4vw,36px);margin:6px 0 6px;letter-spacing:-.02em;font-weight:750;}
.lede{color:var(--ink-soft);font-size:15px;max-width:60ch;}
.meta{font-family:var(--mono);font-size:12px;color:var(--ink-faint);margin-top:10px;}
.cards{display:grid;grid-template-columns:1fr 1fr;gap:16px;margin:30px 0 8px;}
@media (max-width:680px){.cards{grid-template-columns:1fr;}}
a.card{display:block;text-decoration:none;color:inherit;background:var(--panel);border:1px solid var(--line);
border-radius:14px;padding:20px;box-shadow:var(--shadow);position:relative;overflow:hidden;transition:transform .12s,border-color .12s;}
a.card:hover{transform:translateY(-2px);border-color:var(--accent);}
a.card:focus-visible{outline:2px solid var(--accent);outline-offset:2px;}
a.card::before{content:"";position:absolute;left:0;top:0;bottom:0;width:3px;background:var(--accent);}
.card .k{font-family:var(--mono);font-size:11px;color:var(--accent);font-weight:700;letter-spacing:.04em;}
.card h2{font-size:16px;margin:8px 0 6px;letter-spacing:-.01em;}
.card p{font-size:13px;color:var(--ink-soft);margin:0;}
.card .go{font-size:12px;color:var(--accent-ink);font-weight:600;margin-top:12px;display:inline-block;}
.full{grid-column:1/-1;}
.badge{display:inline-flex;align-items:center;gap:6px;font-size:10.5px;font-weight:700;padding:2px 8px;border-radius:999px;background:var(--warn-bg);color:var(--warn);margin-left:8px;vertical-align:middle;}
.files{margin-top:26px;border-top:1px solid var(--line);padding-top:18px;}
.files h3{font-size:12px;text-transform:uppercase;letter-spacing:.08em;color:var(--ink-faint);margin:0 0 10px;}
.files ul{margin:0;padding:0;list-style:none;display:flex;flex-direction:column;gap:6px;}
.files li{font-size:13px;}
.files code{font-family:var(--mono);font-size:12px;color:var(--ink-soft);}
.files a{color:var(--accent-ink);}
.note{margin-top:24px;background:var(--warn-bg);border:1px solid color-mix(in srgb,var(--warn) 30%,transparent);
border-radius:10px;padding:13px 16px;font-size:12.5px;color:var(--ink-soft);}
.note b{color:var(--ink);}
footer{margin-top:28px;font-size:12px;color:var(--ink-faint);font-family:var(--mono);}
</style>
</head>
<body>
<div class="wrap">
<div class="eyebrow">Dasnuve · Cloud Discovery</div>
<h1>Scrivas — AWS Discovery Reports</h1>
<p class="lede">Read-only assessment of the Scrivas AWS Organization (<code>o-qfj0pvhhv7</code>) — footprint,
security posture, cost, and Organizations governance — extended with an EC2 source-code recovery and production reliability triage. Prepared to inform a proposal.</p>
<div class="meta">2 accounts · us-east-2 primary · generated 2026-08-19, EC2 code pass 2026-08-28 · CONFIDENTIAL</div>
<div class="cards">
<a class="card full" href="findings/code_dashboard.html">
<span class="k">CODE + RELIABILITY</span>
<h2>Source Code &amp; Reliability Triage <span class="badge">time-sensitive</span></h2>
<p>All 10 application repos found as full git checkouts on Scrivas-owned EC2 — recoverable despite the contractor holding the GitLab. Plus the reliability triage: 23 prod containers on one host, no memory limits, no swap.</p>
<span class="go">Open dashboard →</span>
</a>
<a class="card" href="findings/discovery_dashboard.html">
<span class="k">FINDINGS</span>
<h2>Discovery Findings</h2>
<p>Resource footprint, security posture, cost baseline, and the CIS/SOC&nbsp;2 benchmark summary.</p>
<span class="go">Open dashboard →</span>
</a>
<a class="card" href="findings/assessment_dashboard.html">
<span class="k">GOVERNANCE</span>
<h2>Organizations Assessment</h2>
<p>Delegated administrators, trust policies, and trusted access services across the org.</p>
<span class="go">Open dashboard →</span>
</a>
<a class="card full" href="findings/prowler/prowler-output-716468089330-20260819124039.html">
<span class="k">BENCHMARK</span>
<h2>Prowler Full Report <span class="badge">needs internet</span></h2>
<p>All 632 checks with per-framework compliance (CIS, SOC&nbsp;2, ISO&nbsp;27001, NIST, PCI). Interactive tables load styling from a CDN, so open this one online.</p>
<span class="go">Open full benchmark →</span>
</a>
</div>
<div class="files">
<h3>Written report &amp; raw evidence</h3>
<ul>
<li><a href="findings/discovery_report.md">discovery_report.md</a> — narrative writeup <code>(renders on GitHub)</code></li>
<li><a href="findings/ec2_code_discovery_report.md">ec2_code_discovery_report.md</a> — code recovery + reliability triage <code>(renders on GitHub)</code></li>
<li><code>findings/ec2_code_inspect.json</code> — SSM probe output, all 7 instances</li>
<li><code>findings/ec2_code_inspect_prod.json</code> — prod/stage probe snapshot</li>
<li><code>findings/org_assessment_report.json</code> — org / trust / delegated-admin raw data</li>
<li><code>findings/fast_discovery.json</code> — footprint, security, cost raw data</li>
<li><code>findings/member_lazka_547868853286.json</code> — member account raw data</li>
</ul>
</div>
<div class="note">
<b>Confidential — private repo only.</b> These reports contain AWS account IDs, role ARNs, external IDs, and live
security findings for a prospective client. The large raw Prowler scans (<code>*.ocsf.json</code>, <code>*.csv</code>)
are intentionally git-ignored; regenerate them with <code>pipenv run prowler aws …</code> if needed.
</div>
<footer>Dasnuve · read-only discovery · reproduce via scripts/ + Pipfile</footer>
</div>
</body>
</html>