Talk Around Town: Pinterest Says a Ghost AWS Agent Was Quietly Strangling Its ML Training Jobs

What Folks Are Saying

Well, hold onto your sweet tea, because Pinterest's engineering blog is tellin' a story that'd make any seasoned sysadmin spit out their chaw. According to Pinterest's own post-mortem, their GPU-based machine learning platform — which the company describes as PinCompute — started showin' mysterious performance problems in training jobs. The culprit, Pinterest says, turned out to be an Amazon ECS agent that nobody ever invited to the party, just sittin' there in their AWS Deep Learning AMI base image like a uninvited cousin who won't leave the couch.

InfoQ, a respected independent specialist publication, separately reported on and corroborated the key technical details of the incident in May 2026, lending the Pinterest account more weight than a blog post standin' alone. The Hacker News community also picked up the story and commenced to pokin' at the details, as those folks do. The core narrative — a default cloud provider software package quietly rottin' out performance from the inside — has been circlin' the infrastructure engineering world like a bad smell from a hog pen you just can't locate.

What Pinterest Actually Claims Happened

According to Pinterest's engineering blog, their AWS Deep Learning AMI base image shipped with an Amazon ECS agent enabled by default. The trouble was, Pinterest's machines had no permissions to join an ECS cluster, so the agent just kept crashin' and restartin', over and over, like a hound dog that keeps runnin' into the same electric fence. Pinterest says each one of those restarts leaked a memory cgroup — a kernel-level accounting structure — without ever cleaning up after itself.

Over time, the company says, this accumulation reached nearly 70,000 leaked zombie memory cgroups on machines that had only about 240 cgroups in active use. That is roughly the equivalent of havin' 70,000 empty feed buckets stacked in a barn where only 240 animals actually live. InfoQ confirmed these figures in its independent coverage. Pinterest's blog says their kubelet process — the Kubernetes node agent that normally sips less than 1% of CPU — was observed spikiing to approximately 6.5%, per InfoQ's reporting, burnin' most of that time inside a kernel function called mem_cgroup_nr_lru_pages as it dutifully walked every single one of those ghost cgroups on every stats sync cycle.

Pinterest engineers further determined, according to their blog, that CPU starvation can occur when even one heavily utilized core blocks a network thread scheduled onto that core. On their 96-core GPU machines, the company says, this made the problem nearly invisible in aggregate metrics — the overall CPU utilization dashboard looked fine while one poor core was gettin' absolutely whupped. That is like the whole county lookin' prosperous while one field over the back fence is on fire.

The Fix, Per Pinterest

Pinterest says the resolution was, in hindsight, almost insultingly simple: disable the ECS agent's systemd unit in their base images and then reboot all the machines. The reboot, the company explains, purged the accumulated zombie memcgs. Pinterest also notes that rebooting alone had previously provided only temporary relief, because the ECS agent would just start right back up crashin' and leakin' again — like moppin' the kitchen floor while the dog is still trackin' mud. Disabling the agent first and then rebooting was what actually stuck.

It is worth noticing, and InfoQ's reporting flags this distinction too, that what Pinterest is callin' 'zombie memcgs' ain't quite the same critter as the classic zombie process most Linux documentation describes. Traditional zombie processes, as LinuxBlog.io and other sources explain, consume no CPU or memory — they just occupy a slot in the process table, waitin' for a parent to reap 'em. Pinterest's incident involves leaked kernel memory cgroup structures, which is a related but meaningfully different beast. The 'zombie' label in Pinterest's blog is used loosely, a fact worth keepin' in mind when evaluatin' their narrative.

What Remains Unverified or Disputed

Pinterest's internal performance numbers — the exact degree to which ML training jobs were degraded — have not been independently verified by any third party, per the research available. The blog is the sole first-hand account of those specifics. Additionally, the full scope of how many other organizations are runnin' similar AWS Deep Learning AMIs with the ECS agent just sittin' there crash-loopin' in the background is entirely unknown. That question is hangin' in the air like humidity in August.

A commenter on Hacker News raised a reasonable head-scratcher: why would a single busy core out of 96 be enough to block a network thread and tank training throughput, and wouldn't one core monopolizing kernel work be more immediately visible? Pinterest's post acknowledges this Linux scheduler behavior but doesn't exhaustively walk through the mechanism, which leaves that particular question somewhat underexplained. It is a legitimate technical gap in the public account, not just folks bein' ornery.

The Broader Picture: Zombie-Class Problems in Cloud-Native Infra

SRE practitioners and independent specialist sources have been flagging for a while now that in modern container-orchestrated environments, orphaned containers, pods, or leaked kernel data structures have gotten considerably harder to detect at scale. SRE School's published guidance on zombie processes notes that monitoring dashboards can show what appears to be healthy CPU utilization — say, 22% — and ample free memory, while applications are quietly sufferin'. That is exactly the kind of situation Pinterest describes: aggregates lyin' to ya while somethin' stinky is hidin' in the corner.

A DEV Community deep-dive published in April 2026 notes that troubleshooting zombie-class kernel issues in 2026 now requires tools like bpftrace to track which containers are approachin' PID limits or which processes are skippin' the wait() call that would normally reap their children — well beyond what traditional ps-based inspection can show. That raises the practical bar for diagnosing this class of problem considerably. If Pinterest's incident is a representative sample of what's lurkin' in GPU fleets built on vendor AMIs, it suggests a whole category of quiet performance rot that most teams aren't currently equipped to hunt.

Analysis: What This Might Mean (If the Story Holds Up)

This is analysis, not reporting: if Pinterest's account is accurate and broadly generalizable, it points to a genuinely underappreciated risk in how organizations assemble ML infrastructure. Grabbing a cloud provider's pre-built Deep Learning AMI is the path of least resistance — it is the infrastructure equivalent of buyin' a pre-loaded truck at the dealer lot — but it apparently comes with default services that may be entirely irrelevant to your actual workload and actively harmful if they start crashin' in a loop. The fact that this particular failure mode was invisible to high-level metrics for long enough to become a documented post-mortem is the part that should keep platform engineers up at night.

The incident also, analytically speakin', underscores a harder general lesson about 96-core and higher-core-count machines: aggregate utilization metrics get less informative the more cores you have, because a single-core catastrophe becomes a rounding error in the overall picture. That asymmetry between per-core reality and aggregate dashboard appearance may be a design debt that GPU platform teams will need to address more deliberately as machines keep gettin' wider. Again — that is analysis based on Pinterest's claims, not established fact, and the full picture awaits broader independent scrutiny.