PyPI's August download failures exposed a Fastly canary fault and cache bugs

Why it matters

PyPI sits in automated build paths across the Python community. The incident shows how a fault at one CDN location, combined with registry configuration bugs, can produce persistent CI failures without generating an obvious aggregate signal.

Mike Fiedler, PyPI's safety and security engineer, traced intermittent August package-download failures to two overlapping faults: a Fastly canary problem at one Seattle-area point of presence and older bugs in PyPI's content-delivery configuration. Downloads returned to full operation on August 28, according to the incident report PyPI published on September 8.

Fiedler began contributing to PyPI in early 2021 and became a maintainer in 2022. The Python Software Foundation hired him in 2023 as its first full-time PyPI safety and security engineer. The PSF's Developers in Residence page currently describes him as PyPI's sole Safety and Security Engineer. His usual work spans the Warehouse codebase, malware response and software supply-chain defenses. The August incident applied the same investigative work to a failure inside PyPI's delivery stack.

The first user reports arrived on August 17. Developers described intermittent CDN errors, HTTP 502 responses and broken CI or Databricks jobs in PyPI's support tracker. Additional reports helped PyPI narrow the failures to one Fastly point of presence in the Seattle area. PyPI has not disclosed a complete count of failed downloads, affected builds or organizations.

One symptom, two failures

PyPI stores the durable copy of each uploaded package in Amazon S3 and synchronizes it to Backblaze B2. Fastly sits in front of those stores and normally fetches from B2, where an egress agreement with Fastly reduces delivery costs. S3 provides a fallback origin. Once Fastly caches an immutable package file, it can remain at the edge for extended periods.

According to Fiedler's report, a Fastly canary deployment left routing and caching configurations mismatched at one Seattle-area point of presence. Requests reaching the affected location could receive 502 errors. PyPI and Fastly restored downloads by August 28, but the report does not provide a verified public count of requests affected by that fault.

The investigation also exposed faults under PyPI's control. Bugs in its Fastly configuration affected fallback between the storage origins and the handling of HTTP range requests. In some cases, installer metadata requests received synthetic HTTP 501 responses instead of the requested content. PyPI corrected the configuration as part of the incident response.

The disclosed failures concern availability. The supplied incident materials do not indicate package tampering, unauthorized access or another security breach.

User reports filled a monitoring gap

PyPI's aggregate file-hosting metrics did not immediately isolate the fault. User reports provided request-level evidence from failing builds and helped maintainers distinguish the incident from routine error traffic. The episode gave PyPI two separate debugging targets: the Fastly canary fault and configuration bugs within PyPI's own delivery path.

The lack of a complete impact count limits the retrospective. PyPI has not published the number of failed installations, affected projects, CI jobs or organizations during the incident. The available reports establish that failures reached automated build systems, where an intermittent package-download error can stop an otherwise unrelated deployment.

That exposure matters because PyPI operates at public-infrastructure scale. In an August infrastructure account, PSF engineering director Jacob Coffee said PyPI handles more than 6 billion requests each day, or roughly 13 billion when file downloads are included, with about 10 petabytes of daily package egress. Fastly serves almost all of that traffic from its edge network.

Dependency caches reduce the blast radius

Fastly's edge cache and Backblaze's egress arrangement allow the nonprofit registry to distribute large volumes of public software without a commercial registry's revenue model. That architecture also creates fallback paths and vendor boundaries that the PSF must monitor.

Fiedler's immediate advice is practical: cache dependencies in continuous-integration jobs. GitHub's setup-python action disables dependency caching by default and requires users to select a cache such as pip. Astral's setup-uv documentation says caching is enabled by default on GitHub-hosted runners. A dependency already stored in a CI cache generates no PyPI request on the next run, removing one external failure point from the build.

PyPI began as Richard Jones's project in 2002 and was operational by 2003. More than two decades later, the index sits in the installation path of Python projects ranging from personal scripts to production infrastructure. Fiedler's report shows that users with broken builds remain an important part of its monitoring system, especially when an edge failure is too narrow to stand out in aggregate metrics.