The Non-Human Identity Problem Is Back → RAG Access Control → OIDC and Workload Identity for LLM Pipelines
TL;DR
- OIDC workload identity solved the static-key problem for cloud-native workloads; the same patterns apply directly to LLM pipelines — but most teams building RAG systems aren’t applying them
- A typical LLM pipeline has 4–6 distinct services (embedding, retrieval, generation, tool execution, orchestration, monitoring) — each should have its own bounded identity with short-lived tokens
- Static API keys in environment variables are the single most common credential anti-pattern in AI deployments today; they are long-lived, hard to rotate, and not scoped to a single service
- The OIDC pattern: the inference workload proves its identity to a cloud OIDC provider and exchanges a short-lived identity token for a scoped access token — no static credential ever exists in the environment
- For LLM tool integrations (agents calling external APIs), OAuth 2.0 device authorization and token exchange patterns scope what the agent can do on behalf of a user — the agent should never hold the user’s full credentials
OWASP Mapping: OWASP LLM03 — Supply Chain. Static credentials in LLM pipeline services are supply chain vulnerabilities: they can be exfiltrated via prompt injection, leaked via LLM02 (Sensitive Information Disclosure), or extracted from container images. Workload identity removes the credential from the attack surface entirely.
The Big Picture
OIDC WORKLOAD IDENTITY FOR A RAG PIPELINE
Without OIDC (common today) With OIDC (what it should be)
─────────────────────────────────────────────────────────────────────
┌─────────────────────┐ ┌─────────────────────────────┐
│ K8s Pod │ │ K8s Pod │
│ ┌───────────────┐ │ │ ┌──────────────────────┐ │
│ │ Generation │ │ │ │ Generation Service │ │
│ │ Service │ │ │ │ │ │
│ │ │ │ │ │ OIDC token (auto) │ │
│ │ API_KEY=sk-.. │ │ │ │ → exchange for: │ │
│ │ DB_PASS=xxx │ │ │ │ LLM API: invoke │ │
│ │ VDB_TOKEN=yyy │ │ │ │ (scoped, 1hr TTL) │ │
│ └───────────────┘ │ │ └──────────────────────┘ │
└─────────────────────┘ └─────────────────────────────┘
│ │
Static keys in env vars: No static keys in environment:
- Long-lived (months/years) - OIDC assertion from pod SA
- Not scoped to one service - Exchanged for short-lived token
- Visible in process env - Scoped to this service's actions
- Exfiltrable via prompt injection - Not present if workload is absent
- Shared across environments - Separate identity per environment
OIDC workload identity is the pattern that eliminated static instance credentials from well-run cloud deployments. It works the same way for LLM pipeline services — and most of the infrastructure to support it already exists in every major cloud platform.
Why LLM Pipelines Have a Worse Static Key Problem
Cloud-native workloads standardized on workload identity over the last five years, but the teams building LLM pipelines in 2024–2025 were often moving fast — data scientists, ML engineers, product engineers — not the same people who spent years cleaning up IAM in cloud infrastructure.
The result is a category of deployments that looks modern (Kubernetes, managed LLM APIs, vector databases) but runs on credentials hygiene from 2016:
- OpenAI/Anthropic/Bedrock API key in a Kubernetes secret, synced to an environment variable, unchanged since the pilot
- Pinecone/Weaviate token in the same pattern
- Database password for the metadata store sitting in a ConfigMap
- No credential rotation because the system works and rotation requires downtime planning
This is not a failure of intent. It’s a failure of infrastructure readiness: the workload identity patterns that exist for S3 and DynamoDB don’t have equivalents that are obvious for OpenAI API calls or third-party vector store APIs. The path of least resistance is a static key.
But the attack surface created by static keys in LLM workloads is significantly worse than in traditional cloud workloads, for one reason: prompt injection can exfiltrate credentials from the runtime environment.
If your LLM generation service runs with OPENAI_API_KEY and DATABASE_URL in its environment, and an attacker can inject a prompt that causes the model to execute a tool call that reads environment variables, those credentials are exposed. The static key that took a year to rotate is now in the attacker’s hands in a single request.
The Four Services That Need Separate Identities
A production RAG pipeline typically has these services. Each needs its own identity — not one shared service account.
┌──────────────────────────────────────────────────────────────────┐
│ RAG PIPELINE — SERVICE IDENTITY MAP │
│ │
│ ┌─────────────────┐ identity: embed-sa │
│ │ Embedding │ permissions: │
│ │ Service │ - vector_store: write (own namespace) │
│ │ │ - source_docs: read │
│ └────────┬────────┘ │
│ │ vectors │
│ ▼ │
│ ┌─────────────────┐ identity: vectordb-sa │
│ │ Vector │ permissions: │
│ │ Database │ - internal service, accessed via API │
│ └────────┬────────┘ │
│ │ filtered query │
│ ▼ │
│ ┌─────────────────┐ identity: retrieve-sa │
│ │ Retrieval │ permissions: │
│ │ Service │ - vector_store: read (user-scoped) │
│ │ │ - No LLM API access │
│ └────────┬────────┘ │
│ │ authorized chunks │
│ ▼ │
│ ┌─────────────────┐ identity: generate-sa │
│ │ Generation │ permissions: │
│ │ Service │ - llm_api: invoke │
│ │ │ - No vector store access │
│ └────────┬────────┘ - No source_docs access │
│ │ prompt + context │
│ ▼ │
│ ┌─────────────────┐ identity: tools-sa │
│ │ Tool Execution │ permissions: │
│ │ Layer │ - per-tool, per-action scoping │
│ │ │ - human gate for write operations │
│ └─────────────────┘ │
└──────────────────────────────────────────────────────────────────┘
Why this separation matters:
If the generation service is compromised (prompt injection), the attacker has LLM API invocation rights — they can burn your API budget. They cannot read the vector store, because the generation service has no access to it. They cannot read source documents. They cannot write to the vector database. The blast radius is bounded.
If the retrieval service is compromised, the attacker gets query access to the vector store, scoped to the user context that was being served. They cannot write to it, cannot reach the LLM API, cannot access source documents.
This is the same principle that makes micro-segmentation effective in network security. The breach happens; you contain what the breach can reach.
Implementing OIDC: AWS, GCP, and Kubernetes
AWS: IAM Roles for Service Accounts (IRSA)
For LLM services running on EKS, IRSA is the standard pattern. The pod gets a Kubernetes service account that is annotated with an IAM role ARN. The pod’s credential chain automatically exchanges the OIDC token from the pod’s projected service account volume for a short-lived AWS STS credential.
apiVersion: v1
kind: ServiceAccount
metadata:
name: llm-generate-sa
namespace: llm-prod
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/llm-generate-prod
// IAM role trust policy — only this specific K8s SA can assume it
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::123456789:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/XXXX"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/XXXX:sub": "system:serviceaccount:llm-prod:llm-generate-sa"
}
}
}]
}
// IAM policy — scoped to only what the generation service needs
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["bedrock:InvokeModel"],
"Resource": "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-5-sonnet*"
}]
}
No static key. The pod proves its identity via the OIDC token from the Kubernetes projected volume. The token has a 1-hour TTL and is bound to this specific service account in this specific namespace in this specific cluster.
GCP: Workload Identity Federation
For GCP workloads on GKE:
# K8s service account bound to a GCP service account
apiVersion: v1
kind: ServiceAccount
metadata:
name: llm-retrieve-sa
namespace: llm-prod
annotations:
iam.gke.io/gcp-service-account: [email protected]
# Bind K8s SA to GCP SA
gcloud iam service-accounts add-iam-policy-binding \
[email protected] \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:my-project.svc.id.goog[llm-prod/llm-retrieve-sa]"
# Grant the GCP SA only what the retrieval service needs
gcloud projects add-iam-policy-binding my-project \
--role roles/datastore.viewer \
--member "serviceAccount:[email protected]"
Third-Party APIs: The Gap That Still Needs Static Keys
OIDC works cleanly for cloud provider resources. For third-party LLM APIs (OpenAI, Anthropic) and third-party vector stores (Pinecone, Weaviate), there is currently no OIDC exchange — those providers do not accept cloud-native OIDC tokens.
For these cases, the correct pattern is:
- Store in a secrets manager, not environment variables — AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault
- Inject at runtime via the secrets manager API, not via environment variables
- Scope the IAM permission to read the specific secret to the relevant service account only
- Set a rotation schedule — 90 days maximum, 30 days preferred
- Use separate API keys per service — the generation service and the embedding service should have different API keys with different usage quotas
# Retrieve API key at runtime from secrets manager — not from env vars
import boto3
def get_llm_api_key(secret_name: str, region: str = "us-east-1") -> str:
client = boto3.client("secretsmanager", region_name=region)
# boto3 uses the pod's IRSA role — no static credential needed to call Secrets Manager
response = client.get_secret_value(SecretId=secret_name)
return response["SecretString"]
llm_client = Anthropic(api_key=get_llm_api_key("llm-prod/anthropic-api-key"))
The IAM credential (IRSA) accesses Secrets Manager; Secrets Manager holds the third-party API key. One layer of OIDC-based identity; one layer of secrets management. No static key in the environment.
Agent-Level Identity: When the AI Calls Your APIs
Agents that call tools are a distinct identity problem from services that call LLM APIs. When an agent calls an internal API on behalf of a user, it needs to be clear:
- Which identity is making the call — the agent’s service identity, or the user’s delegated identity?
- What scope the agent has — can it call any API the user can call, or only the APIs the agent was designed to use?
The correct model is delegated authorization, not impersonation. The agent should receive a narrowly-scoped token representing the user’s consent to specific actions, not the user’s full credentials.
WRONG: Agent uses user's session token
User logs in → agent receives user's session cookie
Agent can call any API the user can call
Prompt injection = full user account compromise
RIGHT: Agent uses delegated, scoped token
User authorizes agent for specific actions
Agent receives token with limited scope:
- read:documents (user's own documents only)
- write:calendar (only create events, not delete)
Agent cannot call billing API, admin API, etc.
Prompt injection = limited to authorized scope
OAuth 2.0 token exchange (RFC 8693) formalizes this pattern. The user authenticates and consents to specific scopes; those scopes are encoded in a token issued specifically for the agent. The agent presents this token to downstream services; those services verify the scope before accepting the request.
# OAuth 2.0 token exchange: user token → agent-scoped token
def exchange_for_agent_token(user_token: str, agent_id: str, requested_scopes: list) -> str:
response = requests.post(
"https://auth.internal/oauth/token",
data={
"grant_type": "urn:ietf:params:oauth:grant-type:token-exchange",
"subject_token": user_token,
"subject_token_type": "urn:ietf:params:oauth:token-type:access_token",
"requested_token_type": "urn:ietf:params:oauth:token-type:access_token",
"scope": " ".join(requested_scopes),
"actor_token": agent_id,
}
)
return response.json()["access_token"]
# The agent gets a token scoped only to what it needs
agent_token = exchange_for_agent_token(
user_token=current_user.session_token,
agent_id="doc-summarizer-v2",
requested_scopes=["read:own_documents", "read:shared_documents"]
)
The downstream APIs see a token with explicit scope. They don’t need to know whether the caller is a human or an agent — they check the scope. The agent cannot call APIs outside its declared scope, regardless of what a prompt injection instructs it to do.
⚠ Production Gotchas
IRSA/Workload Identity breaks when pods share a service account
If multiple pods share the same Kubernetes service account, they all get the same IAM role. A compromised embedding service pod now has the retrieval service’s permissions too. One service account per deployment, no exceptions.
Secrets Manager still needs rotation automation
Moving from environment variables to Secrets Manager removes static keys from the container environment — it does not automatically rotate them. Rotation requires: a Lambda function (or Cloud Run job) that calls the third-party API to generate a new key, stores it in Secrets Manager, and invalidates the old one. Most third-party LLM providers now support API key rotation without downtime. Build the rotation automation at the same time you build the Secrets Manager integration, not as a follow-up task.
OIDC token audience must be validated
When you accept OIDC tokens from Kubernetes, validate the aud (audience) claim. A token issued for one service should not be accepted by another. Without audience validation, a compromised service can present its own token to other services and receive their resources.
The agent token scope must match what you’ve tested
If you scope the agent token to read:documents but your integration test used a full admin token, you will find scope failures in production. Test with scoped tokens in staging. The first time you discover a missing scope should not be during a production incident.
Quick Reference: Credential Pattern by Service Type
| Service | Static Key | Secrets Manager | OIDC / Workload Identity |
|---|---|---|---|
| Cloud provider API (S3, GCS, BigQuery) | Never | Not needed | Use OIDC directly |
| Third-party LLM API (OpenAI, Anthropic) | Avoid | Use Secrets Manager + OIDC to access it | Not supported by provider |
| Third-party vector store (Pinecone) | Avoid | Use Secrets Manager + OIDC to access it | Not supported by provider |
| Internal database | Never | Use Secrets Manager + OIDC to access it | DB supports IAM auth (Postgres IAM, Cloud SQL IAM) |
| Internal API | Never | Not needed | OIDC service-to-service tokens |
| Agent calling user-scoped API | Never | Not applicable | OAuth 2.0 token exchange (user-delegated) |
Framework Alignment
| Framework | Reference | Connection |
|---|---|---|
| OWASP LLM03 | Supply Chain | Static credentials are a supply chain risk; workload identity removes them |
| OWASP LLM06 | Excessive Agency | Token exchange scoping limits agent authority to declared actions |
| ISO 27001:2022 | 5.16 Identity management | Non-human identity lifecycle: creation, rotation, revocation |
| ISO 27001:2022 | 8.24 Use of cryptography | Short-lived OIDC tokens preferred over long-lived symmetric keys |
| NIST SP 800-207 | Zero Trust Architecture | No implicit trust from network location; identity-based access for every service |
| SOC 2 | CC6.1 Logical access controls | Workload identity is the technical control that makes service account lifecycle auditable |
Key Takeaways
- LLM pipeline services need separate service accounts the same way Lambda functions and Kubernetes workloads do — the multi-year lesson from cloud-native IAM applies directly to AI pipelines
- OIDC/workload identity eliminates static keys for cloud provider API calls; third-party APIs (OpenAI, Pinecone) still need secrets management — the difference is where the credential lives, not whether one exists
- One Kubernetes service account per deployment; validate OIDC token audience; build rotation automation at the same time as secrets manager integration
- Agents calling user-scoped APIs should use OAuth 2.0 token exchange, not user session tokens — the agent gets a scoped, delegated token, not the user’s full credentials
- The blast radius of prompt injection is bounded by the compromised service’s identity scope; over-provisioned pipeline service accounts turn every injection into a data breach
What’s Next
EP01 and EP02 covered the agent as a credential holder. EP03 covered the pipeline services that surround it. EP04 covers the interaction between prompt injection and IAM — specifically, how a successful injection becomes an IAM attack when the agent’s permissions are broader than its function requires. The attacker doesn’t need to compromise the credential store. They use the agent’s valid credentials as a proxy.
When Prompt Injection Becomes IAM Abuse →
Get EP04 in your inbox when it publishes → subscribe
