EC2 source-code discovery and reliability triage (#2)
Second discovery pass over the Scrivas EC2 estate (`716468089330`, `us-east-2`, 7 instances). Prompted by two client signals: they do not hold the source code for their own platform (built under contract by the incumbent vendor), and they are reporting production reliability issues. All access was **read-only** — no writes, restarts or config changes on any instance. ## Source code is recoverable **All 10 application repositories exist as complete git checkouts on Scrivas-owned instances**, with full commit history rather than deployed artifacts. | Tier | Location | Repos | |---|---|---| | App | `Scrivas_dev_env:/home/admin/` | `scrivas_backend` (1380 commits), `scrivas_gate` (183), `scrivas_search` (96), `patient` (59) | | ML | `ML_dev:/srv/` | `post_processor` (137), `sai_suggestions` (93), `ml_monitoring` (62), `patient-summary-service` (49), `patient_document_parser` (41), `soniox_transcriber` (25) | Every remote points at `git@git.devteam.space:scrivas/*` — the contractor's self-hosted GitLab, which Scrivas does **not** control. The on-instance checkouts are the client's only independent leverage over their own source. **Gap:** both `/var/www` frontends are build output with no `.git`. Frontend source is not recoverable from EC2. **Time-sensitive:** `scrivas_backend` received a commit on the assessment date. Development is active on infrastructure the contractor controls. Mirroring the repos to Scrivas-controlled storage is the recommended immediate action and is **not** included in this PR — it needs a scope decision first (it involves the contractor's GitLab credentials held on the dev box). ## Reliability triage Production runs **23 containers on a single 15 GiB host** — including 3 Postgres instances, Kafka and OpenSearch — at **73% memory at rest**, with **no per-container memory limits** and **no swap** on any of the 7 instances. With no limits the OOM killer selects by resident size, so it would typically kill a database rather than the worker that caused the pressure. That matches the "random, unreproducible" symptom profile. Kafka, OpenSearch and search-api additionally carry restart policy `no`, so a host reboot yields a partially-recovered stack that looks healthy from outside. **Recorded as a structural exposure, not an observed root cause.** No OOM event appears in retained logs, `State.OOMKilled` is false on all 23 containers, and `RestartCount` is **0** on every one — the Celery workers showing "Up 6 hours" were redeployed, not crash-restarted. Confirming the hypothesis needs CloudWatch history these boxes do not retain, which is itself a finding and the basis for recommendations 3–5 in the report. One encouraging contrast: the **ML tier is well-built** — ECR images tagged by commit SHA, blue/green slots, passing health checks. The application tier is compose-from-git-checkout with uncommitted `.env.save` and `docker-compose.yml.bkp` files in the prod working tree. The better pattern already exists in-house. ## Contents | File | | |---|---| | `scripts/ec2_code_discovery.py` | EC2 inventory — describe + user data | | `scripts/ec2_code_inspect.py` | read-only SSM probe set; commands reviewable in `PROBES` | | `findings/ec2_code_discovery_report.md` | narrative writeup | | `findings/code_dashboard.html` | client-facing dashboard, matching the existing design system | | `findings/ec2_code_inspect*.json` | raw probe output | | `index.html` | links the new dashboard and evidence | ## Review notes - Probe output was **scanned for credentials before commit** ��� no AWS keys, passwords, tokens or private key blocks. The probe set reads manifests and VCS metadata, never file contents. - Git metadata was read **as the owning user** (`sudo -u`) rather than by writing a `safe.directory` entry into root's gitconfig, to preserve the read-only guarantee. - Several prod probes initially returned empty and were re-run with stderr visible; the empties were artifacts (`last` is not installed on prod), not clean health. Worth knowing when reading the JSON. - The dashboard keeps the "what the evidence does not show" section in the **client-facing** version. If CloudWatch later points elsewhere, overclaiming a root cause would cost more credibility than the softer framing gains. https://claude.ai/code/session_01YMxVaHXJsqpqKwncNQ9b1e Co-authored-by: Alvaro Del Valle <alvaro.delvalle2@gmail.com> Reviewed-on: #2
This commit was merged in pull request #2.
This commit is contained in:
Alvaro Del Valle
@@ -0,0 +1,159 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Scrivas — EC2 source-code discovery (stage 1: inventory)
|
||||
========================================================
|
||||
Read-only. Enumerates EC2 instances in active regions and collects the
|
||||
signals that indicate where application source code lives:
|
||||
|
||||
- instance identity, state, type, AMI, launch time, tags
|
||||
- IAM instance profile (what the box can reach)
|
||||
- SSM managed status (can we run a read-only inspection on it?)
|
||||
- EC2 user data (frequently contains the bootstrap/deploy script,
|
||||
git remote URLs, artifact bucket names, image registries)
|
||||
|
||||
Context: the client does not hold the source code for their own platform;
|
||||
it was built under contract by a third party. This inventories what is
|
||||
running in the client's own account to locate their code.
|
||||
|
||||
Usage: python3 scripts/ec2_code_discovery.py [--profile PROFILE]
|
||||
Output: findings/ec2_code_discovery.json
|
||||
"""
|
||||
import argparse
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
from datetime import datetime, date
|
||||
|
||||
import boto3
|
||||
from botocore.exceptions import ClientError, BotoCoreError
|
||||
|
||||
DEFAULT_PROFILE = "dasnuve-scrivas-louis-impersonation"
|
||||
REGIONS = ["us-east-2", "us-east-1"]
|
||||
FINDINGS = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "findings")
|
||||
|
||||
|
||||
def _default(o):
|
||||
return o.isoformat() if isinstance(o, (datetime, date)) else str(o)
|
||||
|
||||
|
||||
def safe(fn, *a, **k):
|
||||
try:
|
||||
return fn(*a, **k), None
|
||||
except (ClientError, BotoCoreError) as e:
|
||||
return None, str(e)
|
||||
|
||||
|
||||
def tag_map(tags):
|
||||
return {t["Key"]: t["Value"] for t in (tags or [])}
|
||||
|
||||
|
||||
def user_data(ec2, iid):
|
||||
resp, err = safe(ec2.describe_instance_attribute, InstanceId=iid, Attribute="userData")
|
||||
if err:
|
||||
return {"error": err}
|
||||
raw = (resp.get("UserData") or {}).get("Value")
|
||||
if not raw:
|
||||
return None
|
||||
try:
|
||||
return {"text": base64.b64decode(raw).decode("utf-8", "replace")}
|
||||
except Exception as e: # noqa: BLE001
|
||||
return {"error": f"decode failed: {e}"}
|
||||
|
||||
|
||||
def ssm_managed(session, region):
|
||||
"""Instance IDs SSM can reach — these are inspectable without SSH keys."""
|
||||
ssm = session.client("ssm", region_name=region)
|
||||
out = {}
|
||||
try:
|
||||
for page in ssm.get_paginator("describe_instance_information").paginate():
|
||||
for i in page.get("InstanceInformationList", []):
|
||||
out[i.get("InstanceId")] = {
|
||||
"ping_status": i.get("PingStatus"),
|
||||
"platform": f'{i.get("PlatformName")} {i.get("PlatformVersion")}',
|
||||
"agent": i.get("AgentVersion"),
|
||||
"last_ping": i.get("LastPingDateTime"),
|
||||
}
|
||||
except (ClientError, BotoCoreError) as e:
|
||||
return {"_error": str(e)}
|
||||
return out
|
||||
|
||||
|
||||
def scan_region(session, region):
|
||||
ec2 = session.client("ec2", region_name=region)
|
||||
managed = ssm_managed(session, region)
|
||||
instances = []
|
||||
|
||||
pages, err = safe(lambda: list(ec2.get_paginator("describe_instances").paginate()))
|
||||
if err:
|
||||
return {"error": err}
|
||||
|
||||
for page in pages:
|
||||
for res in page.get("Reservations", []):
|
||||
for i in res.get("Instances", []):
|
||||
iid = i["InstanceId"]
|
||||
tags = tag_map(i.get("Tags"))
|
||||
instances.append({
|
||||
"instance_id": iid,
|
||||
"name": tags.get("Name"),
|
||||
"state": i["State"]["Name"],
|
||||
"type": i.get("InstanceType"),
|
||||
"image_id": i.get("ImageId"),
|
||||
"launch_time": i.get("LaunchTime"),
|
||||
"key_name": i.get("KeyName"),
|
||||
"private_ip": i.get("PrivateIpAddress"),
|
||||
"public_ip": i.get("PublicIpAddress"),
|
||||
"iam_instance_profile": (i.get("IamInstanceProfile") or {}).get("Arn"),
|
||||
"security_groups": [g["GroupId"] for g in i.get("SecurityGroups", [])],
|
||||
"volumes": [
|
||||
b["Ebs"]["VolumeId"]
|
||||
for b in i.get("BlockDeviceMappings", []) if b.get("Ebs")
|
||||
],
|
||||
"tags": tags,
|
||||
"ssm": managed.get(iid),
|
||||
"user_data": user_data(ec2, iid),
|
||||
})
|
||||
return {
|
||||
"region": region,
|
||||
"ssm_managed_count": len([k for k in managed if not k.startswith("_")]),
|
||||
"instance_count": len(instances),
|
||||
"instances": instances,
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--profile", default=DEFAULT_PROFILE)
|
||||
ap.add_argument("--regions", nargs="*", default=REGIONS)
|
||||
args = ap.parse_args()
|
||||
|
||||
session = boto3.Session(profile_name=args.profile)
|
||||
ident = session.client("sts").get_caller_identity()
|
||||
|
||||
report = {
|
||||
"generated": datetime.now().astimezone().isoformat(),
|
||||
"account": ident["Account"],
|
||||
"principal": ident["Arn"],
|
||||
"regions": [scan_region(session, r) for r in args.regions],
|
||||
}
|
||||
|
||||
os.makedirs(FINDINGS, exist_ok=True)
|
||||
path = os.path.join(FINDINGS, "ec2_code_discovery.json")
|
||||
with open(path, "w") as fh:
|
||||
json.dump(report, fh, indent=2, default=_default)
|
||||
print(f"wrote {path}")
|
||||
|
||||
for r in report["regions"]:
|
||||
if r.get("error"):
|
||||
print(f'{r}')
|
||||
continue
|
||||
print(f'\n== {r["region"]}: {r["instance_count"]} instances, {r["ssm_managed_count"]} SSM-managed')
|
||||
for i in r["instances"]:
|
||||
ud = i["user_data"]
|
||||
ud_flag = "userdata" if (ud and ud.get("text")) else "-"
|
||||
ssm_flag = i["ssm"]["ping_status"] if i.get("ssm") else "-"
|
||||
print(f' {i["instance_id"]:22} {i["state"]:10} {i["type"]:14} '
|
||||
f'ssm={ssm_flag:8} {ud_flag:9} {i.get("name")}')
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,158 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Scrivas — EC2 code & reliability inspection (stage 2)
|
||||
=====================================================
|
||||
Runs a fixed, read-only command set on SSM-managed instances to locate the
|
||||
deployed application code and collect reliability evidence.
|
||||
|
||||
Context: Scrivas does not hold the source for their own platform (built by a
|
||||
third-party contractor) and is reporting production reliability issues. This
|
||||
inspects the client's own instances, in the client's own account, to (a) find
|
||||
where the code lives and (b) gather triage evidence.
|
||||
|
||||
SCOPE NOTE: ssm:SendCommand executes on the host. The command set below is
|
||||
read-only by construction -- no writes, no restarts, no config changes, and
|
||||
no dumping of file *contents* beyond manifests and VCS metadata. Review
|
||||
PROBES before running. Requires explicit operator approval.
|
||||
|
||||
Usage:
|
||||
python3 scripts/ec2_code_inspect.py --list
|
||||
python3 scripts/ec2_code_inspect.py --instance i-073154fb4fa773bbd
|
||||
python3 scripts/ec2_code_inspect.py --all
|
||||
Output: findings/ec2_code_inspect.json
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from datetime import datetime, date
|
||||
|
||||
import boto3
|
||||
from botocore.exceptions import ClientError, BotoCoreError
|
||||
|
||||
DEFAULT_PROFILE = "dasnuve-scrivas-louis-impersonation"
|
||||
REGION = "us-east-2"
|
||||
FINDINGS = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "findings")
|
||||
|
||||
# Read-only probes. Each is (label, shell). Keep every command non-mutating.
|
||||
PROBES = [
|
||||
("os_release", "cat /etc/os-release; uname -a"),
|
||||
("uptime_load", "uptime; cat /proc/loadavg"),
|
||||
("disk", "df -h; echo '--- inodes'; df -i"),
|
||||
("memory", "free -h; echo '--- swap'; swapon --show"),
|
||||
("code_dirs", "ls -la /opt /srv /var/www /home 2>/dev/null"),
|
||||
("git_checkouts", "find / -maxdepth 6 -name .git -type d "
|
||||
"-not -path '*/node_modules/*' 2>/dev/null | head -40"),
|
||||
("git_remotes", "for g in $(find / -maxdepth 6 -name .git -type d "
|
||||
"-not -path '*/node_modules/*' 2>/dev/null | head -20); do "
|
||||
"r=$(dirname $g); echo \"== $r\"; "
|
||||
"git -C $r remote -v 2>/dev/null; "
|
||||
"git -C $r log -1 --format='%H %ad %an %s' 2>/dev/null; "
|
||||
"git -C $r status -sb 2>/dev/null | head -5; done"),
|
||||
("manifests", "find / -maxdepth 6 \\( -name package.json -o -name requirements.txt "
|
||||
"-o -name pyproject.toml -o -name go.mod -o -name Dockerfile "
|
||||
"-o -name docker-compose.y*ml \\) -not -path '*/node_modules/*' "
|
||||
"2>/dev/null | head -40"),
|
||||
("processes", "ps auxww --sort=-%mem | head -30"),
|
||||
("listening", "ss -tulpnH 2>/dev/null | head -40"),
|
||||
("systemd_units", "systemctl list-units --type=service --state=running --no-pager --no-legend | head -40"),
|
||||
("systemd_failed", "systemctl list-units --state=failed --no-pager --no-legend"),
|
||||
("docker", "docker ps -a --format '{{.Names}}\t{{.Image}}\t{{.Status}}' 2>/dev/null | head -30"),
|
||||
("docker_images", "docker images --format '{{.Repository}}:{{.Tag}}\t{{.CreatedAt}}' 2>/dev/null | head -20"),
|
||||
("oom_kills", "sudo dmesg -T 2>/dev/null | grep -iE 'oom|killed process' | tail -20"),
|
||||
("svc_restarts", "sudo journalctl --since '7 days ago' --no-pager 2>/dev/null "
|
||||
"| grep -iE 'segfault|out of memory|failed with result|start-limit' | tail -40"),
|
||||
("reboots", "last -x reboot 2>/dev/null | head -10"),
|
||||
("cron", "ls -la /etc/cron.d 2>/dev/null; crontab -l 2>/dev/null"),
|
||||
]
|
||||
|
||||
|
||||
def _default(o):
|
||||
return o.isoformat() if isinstance(o, (datetime, date)) else str(o)
|
||||
|
||||
|
||||
def managed(session):
|
||||
ssm = session.client("ssm", region_name=REGION)
|
||||
out = []
|
||||
for page in ssm.get_paginator("describe_instance_information").paginate():
|
||||
out += [i for i in page.get("InstanceInformationList", [])
|
||||
if i.get("PingStatus") == "Online"]
|
||||
return out
|
||||
|
||||
|
||||
def run_probe(ssm, iid, label, shell, timeout=120):
|
||||
try:
|
||||
cmd = ssm.send_command(
|
||||
InstanceIds=[iid],
|
||||
DocumentName="AWS-RunShellScript",
|
||||
Comment=f"dasnuve-discovery:{label}"[:100],
|
||||
Parameters={"commands": [shell], "executionTimeout": [str(timeout)]},
|
||||
)["Command"]["CommandId"]
|
||||
except (ClientError, BotoCoreError) as e:
|
||||
return {"error": str(e)}
|
||||
|
||||
for _ in range(int(timeout / 2)):
|
||||
time.sleep(2)
|
||||
try:
|
||||
r = ssm.get_command_invocation(CommandId=cmd, InstanceId=iid)
|
||||
except ClientError as e:
|
||||
if "InvocationDoesNotExist" in str(e):
|
||||
continue
|
||||
return {"error": str(e)}
|
||||
if r["Status"] in ("Pending", "InProgress", "Delayed"):
|
||||
continue
|
||||
return {
|
||||
"status": r["Status"],
|
||||
"stdout": r.get("StandardOutputContent", "").rstrip(),
|
||||
"stderr": r.get("StandardErrorContent", "").rstrip(),
|
||||
}
|
||||
return {"error": "timed out waiting for invocation"}
|
||||
|
||||
|
||||
def inspect(session, iid):
|
||||
ssm = session.client("ssm", region_name=REGION)
|
||||
print(f"\n=== {iid}")
|
||||
res = {}
|
||||
for label, shell in PROBES:
|
||||
print(f" .. {label}", flush=True)
|
||||
res[label] = run_probe(ssm, iid, label, shell)
|
||||
return res
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--profile", default=DEFAULT_PROFILE)
|
||||
ap.add_argument("--instance", action="append", dest="instances")
|
||||
ap.add_argument("--all", action="store_true")
|
||||
ap.add_argument("--list", action="store_true")
|
||||
args = ap.parse_args()
|
||||
|
||||
session = boto3.Session(profile_name=args.profile)
|
||||
online = managed(session)
|
||||
|
||||
if args.list:
|
||||
for i in online:
|
||||
print(f'{i["InstanceId"]:22} {i["PingStatus"]:8} '
|
||||
f'{i.get("PlatformName")} {i.get("PlatformVersion")}')
|
||||
return
|
||||
|
||||
targets = args.instances or ([i["InstanceId"] for i in online] if args.all else [])
|
||||
if not targets:
|
||||
ap.error("pass --instance ID (repeatable), --all, or --list")
|
||||
|
||||
report = {
|
||||
"generated": datetime.now().astimezone().isoformat(),
|
||||
"region": REGION,
|
||||
"probes": [p[0] for p in PROBES],
|
||||
"results": {iid: inspect(session, iid) for iid in targets},
|
||||
}
|
||||
|
||||
os.makedirs(FINDINGS, exist_ok=True)
|
||||
path = os.path.join(FINDINGS, "ec2_code_inspect.json")
|
||||
with open(path, "w") as fh:
|
||||
json.dump(report, fh, indent=2, default=_default)
|
||||
print(f"\nwrote {path}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user