Netflix gives Spark jobs internal identities by having AWS verify their roles

Why it matters

Netflix's design shows how managed-compute workloads can use an organization's internal identity without trusting the workload to identify itself. Its security depends on a correct role-to-project mapping and a tightly controlled signer.

Netflix engineer Dhruv Pratap describes a way for Spark jobs on Amazon EMR to carry a Netflix identity that internal services can trust, even though the managed service starts them with AWS credentials. In a Netflix Technology Blog post published September 25th, Pratap lays out a system that makes AWS attest to the cloud role a process holds while Netflix's control plane vouches for which workload it was authorized to run.

That division of responsibility addresses a practical gap in managed compute. AWS can identify the role attached to a process, but a role by itself does not say which Netflix application or data project should receive access. Netflix needs its internal services to recognize that workload under its own identity system, Metatron, which issues short-lived X.509 certificates used for mutual TLS.

Pratap's design does not ask the Spark process to simply declare who it is. It makes two other parties establish that identity from different evidence.

Two claims, checked against each other

The system starts with a one-to-one mapping between each Data Project identity and a dedicated AWS IAM role. Netflix's Data Project service keeps the mapping authoritative. The control plane, the only service allowed to launch Spark jobs, validates the caller, retrieves the mapped role, signs metadata about the workload, and submits the job with that role.

The signed metadata is specific, but it is not proof on its own. It travels as job configuration through infrastructure Netflix says it does not fully control, where it could be read or replayed. The Spark driver plugin separately uses the role's credentials to sign a request to AWS STS's GetCallerIdentity endpoint. That creates a presigned URL whose response identifies the AWS role that signed it.

Netflix's Identity service checks both artifacts. It verifies the control plane's signature, fetches the URL from STS to confirm the role, and compares that role with the one named in the signed metadata. Only after those checks does it issue Metatron certificates scoped to the workload's environment, stack and detail.

Netflix takes the application identity from the signed metadata and uses AWS's response to corroborate the role. Deriving an application name from a role name, Pratap writes, would depend on fragile string matching; session identifiers assigned by the platform are outside Netflix's control. The tradeoff is an extra network round trip and a requirement to keep the signing key restricted to the control plane.

The project, not the person, gets the identity

The system fits Netflix's broader effort to make data workloads run as durable team-owned identities instead of the individual engineers who created them. In a May 11th article on Data Projects, Netflix engineers described the project as a unit for grouping data assets, managing permissions and providing an identity for asynchronous work. That earlier post said Netflix's data warehouse held millions of tables and supported tens of thousands of scheduled workloads.

For Spark on EMR, the IAM role is the cloud-side counterpart to that Netflix identity. The new post says Netflix expects Data Projects to number in the tens of thousands, so it deterministically distributes their roles across a small pool of dedicated AWS accounts. That both addresses role capacity and places workload roles across an account boundary from the control planes that launch them. The post describes an expected scale, not a count of projects currently using this particular attestation flow.

A managed-compute pattern with a Netflix-specific trust anchor

Pratap says the general pattern applies wherever a managed environment gives a process cloud credentials but no identity recognized by an organization's internal services. Netflix's implementation is tied to Metatron, Data Projects and its Spark platform; the underlying exchange uses a narrower principle: accept neither the workload's own claim nor a single platform's statement as sufficient.

That principle puts the most consequential operational responsibility in the mapping. Cryptographic signatures can establish that a claim came from an authorized signer, and AWS can establish which role signed a request. The system still depends on Netflix's Data Project service keeping the role-to-identity mapping correct and the control plane protecting its signing key. The blog explains the design, but does not report an independent audit, failure rate or current adoption count.

For Netflix, this is infrastructure that lets internal authorization follow a workload onto managed compute without reverting to a human's credentials. Teams facing the same split between cloud IAM and internal identity need to define which service can make each claim and what independent evidence will verify it.