Second discovery pass over the Scrivas EC2 estate (716468089330, us-east-2, 7 instances). Prompted by two client signals: they do not hold the source code for their own platform (built under contract by the incumbent vendor), and they are reporting production reliability issues.
All access was read-only — no writes, restarts or config changes on any instance.
Source code is recoverable
All 10 application repositories exist as complete git checkouts on Scrivas-owned instances, with full commit history rather than deployed artifacts.
Every remote points at git@git.devteam.space:scrivas/* — the contractor's self-hosted GitLab, which Scrivas does not control. The on-instance checkouts are the client's only independent leverage over their own source.
Gap: both /var/www frontends are build output with no .git. Frontend source is not recoverable from EC2.
Time-sensitive:scrivas_backend received a commit on the assessment date. Development is active on infrastructure the contractor controls. Mirroring the repos to Scrivas-controlled storage is the recommended immediate action and is not included in this PR — it needs a scope decision first (it involves the contractor's GitLab credentials held on the dev box).
Reliability triage
Production runs 23 containers on a single 15 GiB host — including 3 Postgres instances, Kafka and OpenSearch — at 73% memory at rest, with no per-container memory limits and no swap on any of the 7 instances. With no limits the OOM killer selects by resident size, so it would typically kill a database rather than the worker that caused the pressure. That matches the "random, unreproducible" symptom profile.
Kafka, OpenSearch and search-api additionally carry restart policy no, so a host reboot yields a partially-recovered stack that looks healthy from outside.
Recorded as a structural exposure, not an observed root cause. No OOM event appears in retained logs, State.OOMKilled is false on all 23 containers, and RestartCount is 0 on every one — the Celery workers showing "Up 6 hours" were redeployed, not crash-restarted. Confirming the hypothesis needs CloudWatch history these boxes do not retain, which is itself a finding and the basis for recommendations 3–5 in the report.
One encouraging contrast: the ML tier is well-built — ECR images tagged by commit SHA, blue/green slots, passing health checks. The application tier is compose-from-git-checkout with uncommitted .env.save and docker-compose.yml.bkp files in the prod working tree. The better pattern already exists in-house.
Contents
File
scripts/ec2_code_discovery.py
EC2 inventory — describe + user data
scripts/ec2_code_inspect.py
read-only SSM probe set; commands reviewable in PROBES
findings/ec2_code_discovery_report.md
narrative writeup
findings/code_dashboard.html
client-facing dashboard, matching the existing design system
findings/ec2_code_inspect*.json
raw probe output
index.html
links the new dashboard and evidence
Review notes
Probe output was scanned for credentials before commit ��� no AWS keys, passwords, tokens or private key blocks. The probe set reads manifests and VCS metadata, never file contents.
Git metadata was read as the owning user (sudo -u) rather than by writing a safe.directory entry into root's gitconfig, to preserve the read-only guarantee.
Several prod probes initially returned empty and were re-run with stderr visible; the empties were artifacts (last is not installed on prod), not clean health. Worth knowing when reading the JSON.
The dashboard keeps the "what the evidence does not show" section in the client-facing version. If CloudWatch later points elsewhere, overclaiming a root cause would cost more credibility than the softer framing gains.
Second discovery pass over the Scrivas EC2 estate (`716468089330`, `us-east-2`, 7 instances). Prompted by two client signals: they do not hold the source code for their own platform (built under contract by the incumbent vendor), and they are reporting production reliability issues.
All access was **read-only** — no writes, restarts or config changes on any instance.
## Source code is recoverable
**All 10 application repositories exist as complete git checkouts on Scrivas-owned instances**, with full commit history rather than deployed artifacts.
| Tier | Location | Repos |
|---|---|---|
| App | `Scrivas_dev_env:/home/admin/` | `scrivas_backend` (1380 commits), `scrivas_gate` (183), `scrivas_search` (96), `patient` (59) |
| ML | `ML_dev:/srv/` | `post_processor` (137), `sai_suggestions` (93), `ml_monitoring` (62), `patient-summary-service` (49), `patient_document_parser` (41), `soniox_transcriber` (25) |
Every remote points at `git@git.devteam.space:scrivas/*` — the contractor's self-hosted GitLab, which Scrivas does **not** control. The on-instance checkouts are the client's only independent leverage over their own source.
**Gap:** both `/var/www` frontends are build output with no `.git`. Frontend source is not recoverable from EC2.
**Time-sensitive:** `scrivas_backend` received a commit on the assessment date. Development is active on infrastructure the contractor controls. Mirroring the repos to Scrivas-controlled storage is the recommended immediate action and is **not** included in this PR — it needs a scope decision first (it involves the contractor's GitLab credentials held on the dev box).
## Reliability triage
Production runs **23 containers on a single 15 GiB host** — including 3 Postgres instances, Kafka and OpenSearch — at **73% memory at rest**, with **no per-container memory limits** and **no swap** on any of the 7 instances. With no limits the OOM killer selects by resident size, so it would typically kill a database rather than the worker that caused the pressure. That matches the "random, unreproducible" symptom profile.
Kafka, OpenSearch and search-api additionally carry restart policy `no`, so a host reboot yields a partially-recovered stack that looks healthy from outside.
**Recorded as a structural exposure, not an observed root cause.** No OOM event appears in retained logs, `State.OOMKilled` is false on all 23 containers, and `RestartCount` is **0** on every one — the Celery workers showing "Up 6 hours" were redeployed, not crash-restarted. Confirming the hypothesis needs CloudWatch history these boxes do not retain, which is itself a finding and the basis for recommendations 3–5 in the report.
One encouraging contrast: the **ML tier is well-built** — ECR images tagged by commit SHA, blue/green slots, passing health checks. The application tier is compose-from-git-checkout with uncommitted `.env.save` and `docker-compose.yml.bkp` files in the prod working tree. The better pattern already exists in-house.
## Contents
| File | |
|---|---|
| `scripts/ec2_code_discovery.py` | EC2 inventory — describe + user data |
| `scripts/ec2_code_inspect.py` | read-only SSM probe set; commands reviewable in `PROBES` |
| `findings/ec2_code_discovery_report.md` | narrative writeup |
| `findings/code_dashboard.html` | client-facing dashboard, matching the existing design system |
| `findings/ec2_code_inspect*.json` | raw probe output |
| `index.html` | links the new dashboard and evidence |
## Review notes
- Probe output was **scanned for credentials before commit** ��� no AWS keys, passwords, tokens or private key blocks. The probe set reads manifests and VCS metadata, never file contents.
- Git metadata was read **as the owning user** (`sudo -u`) rather than by writing a `safe.directory` entry into root's gitconfig, to preserve the read-only guarantee.
- Several prod probes initially returned empty and were re-run with stderr visible; the empties were artifacts (`last` is not installed on prod), not clean health. Worth knowing when reading the JSON.
- The dashboard keeps the "what the evidence does not show" section in the **client-facing** version. If CloudWatch later points elsewhere, overclaiming a root cause would cost more credibility than the softer framing gains.
https://claude.ai/code/session_01YMxVaHXJsqpqKwncNQ9b1e
Second discovery pass over the Scrivas EC2 estate (716468089330, us-east-2,
7 instances), prompted by the client reporting reliability issues and by their
lack of access to source code held under contract by the incumbent vendor.
Source code recovery
- All 10 application repositories exist as complete git checkouts on
Scrivas-owned instances, with full history rather than deployed artifacts:
4 app repos on Scrivas_dev_env, 6 ML repos on ML_dev.
- Every remote points at git@git.devteam.space (the contractor's self-hosted
GitLab), which Scrivas does not control. The on-instance checkouts are the
client's only independent leverage over their own source.
- Gap: both /var/www frontends are build output with no .git, so frontend
source is not recoverable from EC2.
- Time-sensitive: scrivas_backend received a commit on the assessment date.
Reliability triage
- Production runs 23 containers on a single 15 GiB host, including 3 Postgres
instances, Kafka and OpenSearch, at 73% memory at rest with no per-container
memory limits and no swap on any of the 7 instances.
- Kafka, OpenSearch and search-api carry restart policy `no`, so a host reboot
yields a partially-recovered stack.
- Recorded as a structural exposure, not an observed root cause: no OOM event
is present in retained logs and RestartCount is 0 on every prod container.
Confirming the hypothesis needs CloudWatch history the boxes do not retain.
Contents
- scripts/ec2_code_discovery.py EC2 inventory (describe + user data)
- scripts/ec2_code_inspect.py read-only SSM probe set, reviewable in PROBES
- findings/ec2_code_discovery_report.md narrative writeup
- findings/code_dashboard.html client-facing dashboard
- findings/ec2_code_inspect*.json raw probe output
- index.html links the new dashboard and evidence
All access was read-only: no writes, restarts or config changes on any
instance. Probe output was scanned for credentials before commit; git metadata
was read as the owning user rather than by writing a safe.directory entry.
Claude-Session: https://claude.ai/code/session_01YMxVaHXJsqpqKwncNQ9b1e
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Second discovery pass over the Scrivas EC2 estate (
716468089330,us-east-2, 7 instances). Prompted by two client signals: they do not hold the source code for their own platform (built under contract by the incumbent vendor), and they are reporting production reliability issues.All access was read-only — no writes, restarts or config changes on any instance.
Source code is recoverable
All 10 application repositories exist as complete git checkouts on Scrivas-owned instances, with full commit history rather than deployed artifacts.
Scrivas_dev_env:/home/admin/scrivas_backend(1380 commits),scrivas_gate(183),scrivas_search(96),patient(59)ML_dev:/srv/post_processor(137),sai_suggestions(93),ml_monitoring(62),patient-summary-service(49),patient_document_parser(41),soniox_transcriber(25)Every remote points at
git@git.devteam.space:scrivas/*— the contractor's self-hosted GitLab, which Scrivas does not control. The on-instance checkouts are the client's only independent leverage over their own source.Gap: both
/var/wwwfrontends are build output with no.git. Frontend source is not recoverable from EC2.Time-sensitive:
scrivas_backendreceived a commit on the assessment date. Development is active on infrastructure the contractor controls. Mirroring the repos to Scrivas-controlled storage is the recommended immediate action and is not included in this PR — it needs a scope decision first (it involves the contractor's GitLab credentials held on the dev box).Reliability triage
Production runs 23 containers on a single 15 GiB host — including 3 Postgres instances, Kafka and OpenSearch — at 73% memory at rest, with no per-container memory limits and no swap on any of the 7 instances. With no limits the OOM killer selects by resident size, so it would typically kill a database rather than the worker that caused the pressure. That matches the "random, unreproducible" symptom profile.
Kafka, OpenSearch and search-api additionally carry restart policy
no, so a host reboot yields a partially-recovered stack that looks healthy from outside.Recorded as a structural exposure, not an observed root cause. No OOM event appears in retained logs,
State.OOMKilledis false on all 23 containers, andRestartCountis 0 on every one — the Celery workers showing "Up 6 hours" were redeployed, not crash-restarted. Confirming the hypothesis needs CloudWatch history these boxes do not retain, which is itself a finding and the basis for recommendations 3–5 in the report.One encouraging contrast: the ML tier is well-built — ECR images tagged by commit SHA, blue/green slots, passing health checks. The application tier is compose-from-git-checkout with uncommitted
.env.saveanddocker-compose.yml.bkpfiles in the prod working tree. The better pattern already exists in-house.Contents
scripts/ec2_code_discovery.pyscripts/ec2_code_inspect.pyPROBESfindings/ec2_code_discovery_report.mdfindings/code_dashboard.htmlfindings/ec2_code_inspect*.jsonindex.htmlReview notes
sudo -u) rather than by writing asafe.directoryentry into root's gitconfig, to preserve the read-only guarantee.lastis not installed on prod), not clean health. Worth knowing when reading the JSON.https://claude.ai/code/session_01YMxVaHXJsqpqKwncNQ9b1e