Add EC2 source-code discovery and reliability triage

Browse Source

Second discovery pass over the Scrivas EC2 estate (716468089330, us-east-2,
7 instances), prompted by the client reporting reliability issues and by their
lack of access to source code held under contract by the incumbent vendor.

Source code recovery
- All 10 application repositories exist as complete git checkouts on
  Scrivas-owned instances, with full history rather than deployed artifacts:
  4 app repos on Scrivas_dev_env, 6 ML repos on ML_dev.
- Every remote points at git@git.devteam.space (the contractor's self-hosted
  GitLab), which Scrivas does not control. The on-instance checkouts are the
  client's only independent leverage over their own source.
- Gap: both /var/www frontends are build output with no .git, so frontend
  source is not recoverable from EC2.
- Time-sensitive: scrivas_backend received a commit on the assessment date.

Reliability triage
- Production runs 23 containers on a single 15 GiB host, including 3 Postgres
  instances, Kafka and OpenSearch, at 73% memory at rest with no per-container
  memory limits and no swap on any of the 7 instances.
- Kafka, OpenSearch and search-api carry restart policy `no`, so a host reboot
  yields a partially-recovered stack.
- Recorded as a structural exposure, not an observed root cause: no OOM event
  is present in retained logs and RestartCount is 0 on every prod container.
  Confirming the hypothesis needs CloudWatch history the boxes do not retain.

Contents
- scripts/ec2_code_discovery.py  EC2 inventory (describe + user data)
- scripts/ec2_code_inspect.py    read-only SSM probe set, reviewable in PROBES
- findings/ec2_code_discovery_report.md   narrative writeup
- findings/code_dashboard.html            client-facing dashboard
- findings/ec2_code_inspect*.json         raw probe output
- index.html                              links the new dashboard and evidence

All access was read-only: no writes, restarts or config changes on any
instance. Probe output was scanned for credentials before commit; git metadata
was read as the owning user rather than by writing a safe.directory entry.

Claude-Session: https://claude.ai/code/session_01YMxVaHXJsqpqKwncNQ9b1e

...

This commit is contained in:

Alvaro Del Valle

2026-08-28 16:24:24 -04:00

parent 9e3d1f58d5

commit 44d4bf6748

7 changed files with 1392 additions and 2 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

findings/ec2_code_inspect_prod.json

+302

File diff suppressed because one or more lines are too long