Exposed Secrets: How I Uncovered a Kubernetes Service Account Token in Plaintext Logs


⚠️ Disclaimer
This article assumes you're already somewhat familiar with Kubernetes concepts (Pods, ServiceAccounts) and the basics of JSON Web Tokens (JWTs).
It was a Tuesday.Nothing special - just your average day as a platform engineer. My team's notifications were mercifully quiet, and I th...

🔗 https://www.roastdev.com/post/....exposed-secrets-how-

#news #tech #development

Favicon 
www.roastdev.com

Exposed Secrets: How I Uncovered a Kubernetes Service Account Token in Plaintext Logs

⚠️ Disclaimer
This article assumes you're already somewhat familiar with Kubernetes concepts (Pods, ServiceAccounts) and the basics of JSON Web Tokens (JWTs).
It was a Tuesday.Nothing special - just your average day as a platform engineer. My team's notifications were mercifully quiet, and I thought, "Perfect, I can finally clean up that old Helm chart that's been bothering me."I opened the repo of the underlying image written in Go to double-check the config before merging. As I scrolled through the config file, something caught my eye:log.Println("SA Token:", token)Wait. What?A debug statement. Still in production code. Logging an actual Kubernetes ServiceAccount token. Not cool...I paused. My heart rate didn't. Curious but mostly horrified, I grabbed the token and decoded the payload in my shell:
⛶{
"iss": "https://kubernetes.default.svc.cluster.local",
"kubernetes.io/serviceaccount/namespace": "payments",
"kubernetes.io/serviceaccount/secret.name": "payments-token-6gh49",
"kubernetes.io/serviceaccount/service-account.name": "payments-sa",
"kubernetes.io/serviceaccount/service-account.uid": "f9a2c144-11b3-4eb0-9f30-3c2a5063e2e7",
"aud": "https://kubernetes.default.svc.cluster.local",
"sub": "system:serviceaccount:payments:payments-sa",
"exp": 1788201600, // Sat, 01 Aug 2026 00:00:00 GMT
"iat": 1756665600 // Fri, 01 Aug 2025 00:00:00 GMT
}Default audience claim. A 1-year expiry.This "bad boy" wasn't just a dev leftover - it was a high-privilege token with zero constraints floating around in plaintext logs!


What This Article Covers
In this post, I'll guide you through:
The inner workings of Vault authentication with JWT and Kubernetes methods
What Kubernetes ServiceAccounts and their tokens are, and how they’re (mis)used
How projected ServiceAccount tokens fix many of the hidden dangers of older token behavior
Why you should start adopting token projection and Vault integration today
We'll cover real-world use cases, implementation tips, and common pitfalls - so you don't end up like I did, staring at a:
⛶log.Println("SA token:", token)...and wondering how close you just came to a security incident.


Why This Matters
To really understand why that log statement gave me chills, we need to unpack a few core concepts:
What is a JWT?
How do Kubernetes ServiceAccounts and their tokens work?
And what role do these tokens play in authenticating to systems like Vault?
Let's start with the fundamentals.


What Is a JWT?
If you've been around authentication systems long enough, you've probably seen one of these beasts:
⛶eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9...This is a JSON Web Token (short: JWT).It's a compact, URL-safe format for representing claims between two parties. They're used everywhere: web apps, APIs, and yes — inside your Kubernetes cluster.A JWT consists of three parts:

Header – declares the algorithm used to sign the token (e.g. RS256)

Payload – contains the claims (who you are, what you're allowed to do, etc.)

Signature – a cryptographic seal that verifies the payload hasn't been tampered with
Claims are the heart of a JWT — key-value pairs that describe who the token refers to and what it can be used for. They can be:
Standard claims defined by the spec (e.g., iss, sub, exp, aud)
Custom claims added by the issuer for domain-specific needs



Closer Look at aud
The audience (aud) claim tells who the token is meant for. Think of it as the intended recipient.Example: Imagine a Coldplay concert ticket. It says valid for Stadium X on 01-09-2025. You can't take the same ticket and use it at Stadium Y — they'll reject it (...trust me, I tried).A JWT works the same way:
If the token has "aud": "https://kubernetes.default.svc", then only the Kubernetes API server should accept it.
If some other service receives that token, the aud won't match and the token must be rejected.
Without this check, a token could be misused anywhere that trusts the signing key. With aud, it's scoped to the right system.


Kubernetes and ServiceAccounts
Kubernetes is an open-source platform that orchestrates containers at scale. At its heart is the Pod — the smallest deployable unit.But every pod needs an identity. That's where ServiceAccounts come in.


ServiceAccounts 101

Every Pod references a ServiceAccount (default if none is set), but a token is only mounted if enabled
Kubernetes mounts the identity at:

⛶/var/run/secrets/kubernetes.io/serviceaccount/token
That token is a JWT, signed by the Kubernetes control plane
It lets the pod authenticate with the API server — and sometimes even external systems like Vault



The Catch
Until recently, these tokens came with dangerous defaults:
Long-lived (often valid for a year)
Previous to Kubernetes v1.24, there was no default audience set (https://kubernetes.default.svc)
Automatically mounted into every pod, even if unused



Enter Vault: The Gatekeeper of Secrets
HashiCorp Vault is your cluster’s paranoid librarian:
it stores API keys, certs, passwords — and only hands them out when it's sure you should have them.How? Authentication methods.


Vault Authentication Methods

Username password
AppRole
LDAP
Kubernetes
JWT
Let's zoom into the last two.


Kubernetes Auth Method

Pod sends its mounted ServiceAccount token to Vault
Vault validates it against the Kubernetes API
If valid, Vault maps it to a policy
This is simple and works well when Vault runs inside the cluster.


JWT Auth Method

Vault verifies the JWT itself (signature, claims, expiration)
No need for Kubernetes API access
More portable
Rule of thumb:
Use Kubernetes if Vault runs inside your cluster and simplicity matters
Use JWT if you want portability, stronger boundaries, and flexibility



Projected Tokens: Because It's 2025
Old tokens were static and long-lived. Projected tokens fix this mess.Instead of mounting a one-year token into every pod, Kubernetes can now generate short-lived, audience-bound tokens on demand.


What You Get

Short TTL (e.g. 10 minutes)
Audience restrictions (aud: vault)
Automatic rotation by kubelet

No automatic mounting into pods



Example Pod with Projected Token
⛶apiVersion: v1
kind: Pod
metadata:
name: projected-token-test-pod
namespace: demo
spec:
serviceAccountName: projected-auth-sa
containers:
- name: projected-auth-test
image: demo/vault-curl:latest
command: ["sleep", "3600"]
volumeMounts:
- name: token
mountPath: /var/run/secrets/projected
readOnly: true
volumes:
- name: token
projected:
sources:
- serviceAccountToken:
path: token
expirationSeconds: 600
audience: vault


Why Vault Loves This
Vault's JWT auth method is tailor-made for projected tokens:
It parses and verifies the JWT signature (via a configured PEM key or JWKS endpoint)
Validates all claims (aud, sub, exp, iss) locally
Issues secrets only if every check passes
Minimal dependencies. Strong claim validation. Secure, verifiable checks.


Back to the Log
Imagine you stumble upon this in a Go app:
⛶log.Println("Auth Token:", token)

Old world: a one-year, cluster-wide token with no audience. A time bomb.

New world: a 10-minute token, scoped to Vault, rotating automatically.
It's still bad to log tokens — but at least it's not catastrophic.


Try It Yourself: Vault + K8s AuthN Lab
I've built a hands-on demo repo where you can test this locally with KIND (Kubernetes in Docker) and Vault Helm charts:? GitHub: VincentvonBueren/erfa-projected-sa-token


What's Inside

KIND cluster with Vault
Both Kubernetes and JWT auth methods enabled
Vault policies and roles
Four demo pods:


Kubernetes auth method
JWT with static token
JWT with projected token
JWT with wrong audience (failure demo)








Final Drop ?
If your pods still run with default, long-lived tokens:
you’re one debug log away from giving away the keys to your cluster.Projected tokens aren't optional. They're essential.
Adopt them today — and stop shipping security disasters.

Similar Posts

Similar

Craft a Stunning CSS-Only Time Progress Bar for Markdown & GitHub Pages

For our weekly WeAreDevelopers Live Show I wanted to have a way to include a time progress bar into the page we show. The problem there was that these are markdown files using GitHub Pages and whilst I do use some scripting in them, I wanted to make sure that I could have this functionality in pure ...

🔗 https://www.roastdev.com/post/....craft-a-stunning-css

#news #tech #development

Favicon 
www.roastdev.com

Craft a Stunning CSS-Only Time Progress Bar for Markdown & GitHub Pages

For our weekly WeAreDevelopers Live Show I wanted to have a way to include a time progress bar into the page we show. The problem there was that these are markdown files using GitHub Pages and whilst I do use some scripting in them, I wanted to make sure that I could have this functionality in pure CSS so that it can be used on GitHub without having to create an html template. And here we are. You can check out the demo page to see the effect in action with the liquid source code or play with the few lines of CSS in this codepen. Fork this repo to use it in your pages or just copy the _includes folder.


Using the CSS time progress bar
You can use as many bars as you want to in a single page. The syntax to include a bar is the following:
⛶{​% include cssbar.html duration="2s" id="guesttopic" styleblock="yes" %​}
The duration variable defines how long the progress should take
The id variable is necessary to and has to be unique to make the functionality work
If the styleblock is set, the include will add a style with the necessary css rules so you don't have to add them to the main site styles. You only need to do that in one of the includes.



Using the bar in HTML documents
You can of course also use the bar in pure HTML documents, as shown in the codepen. The syntax is:
⛶class="progressbar" style="--duration: 2s;"
type="checkbox" id="progress"
for="progress"startDon't forget to set a unique id both in the checkbox and the label and define the duration in the inline style.


Drawbacks

This is a bit of a hack as it is not accessible to non-visual users and abuses checkboxes to keep it CSS only. It is keyboard accessible though.
In a better world, I'd have used an HTML progress element and styled that one…
Similar

Why I Dived into LeetCode: Unleashing My Problem-Solving Superpowers!




What lead me to Leetcode:
I am Student and Whenever a beginner student in this dev field think about coding or programming they usually always think about making websites, apps, cool UI's, animations etc. and I was one of them, I have been making apps, websites and even games but in some sp...

🔗 https://www.roastdev.com/post/....why-i-dived-into-lee

#news #tech #development

Favicon 
www.roastdev.com

Why I Dived into LeetCode: Unleashing My Problem-Solving Superpowers!

What lead me to Leetcode:
I am Student and Whenever a beginner student in this dev field think about coding or programming they usually always think about making websites, apps, cool UI's, animations etc. and I was one of them, I have been making apps, websites and even games but in some space i knew i am just following the tutorials, trying and making things using existing solutions and I was not solving problems, I was not budling my problem solving skills I was just using the premade solutions. So I came across Leetcode a widely popular platform for solving problems.


What I Think Now:
When I started I didn't think much of it I thought they will be easy to solve and all so I directly jumped onto That day's Daily Problem and it was marked as Hard so first after seeing it I was scared, then I read the title I got even more scared and when I read the full Problem I was EVEN MORE SCARED seeing how brainstorming the problem that was, Even If i wasn't able to solve that problem even after seeing the solution as it so many concepts, algorithms so beautifully so solve the problem I was in love with it, so I decided that I need to become so good at solving problems that i can solve daily problems like that by myself and in the best way possible.


What I am doing Now:
After getting a Reality check I came to the problems tab and started the Easy marked problems and to my surprise even the easy problems were so good some of them really made me sweat and lead me to hours of learning the algorithms and concepts needed in those problems which i enjoyed every moment of. I am now solving Easy problems as the starting learning new things with each new problem.


What I Learned:
After solving some problems, I got to learn that coding and programming aren't just about development, but as essential the development is the main crust of coding and programming is Solving Problems. When you come across a tough problem and then we solve it the joy we get is unmatched. It Improves how we think, how we approach, how we solve.
It even indirectly improve how efficiently we code during development of even UI.


Final Thoughts:
I just wrote my heart out here, It may not be good but I am still open to Improve so I am doing it the best way i can, there are many things I haven't included like how this problems let's us manage, access data and all because I am still in early phase so I can't say much about those real topics I tried to keep it simple. But i Just want to only one thing START DOING LEETCODE, or any other platform to solve problems.
Similar

Unlocking Rust's Power: Mastering the Chain of Responsibility Design Pattern

I remember in my earlier phase as a software engineer, how I mapped the MFC's command routing algorithm to Chain of Responsibility - and when finally it became clear, there was immense joy for me.I always say to my students that we should approach a problem not only from how's point of view but also...

🔗 https://www.roastdev.com/post/....unlocking-rust-s-pow

#news #tech #development

Favicon 
www.roastdev.com

Unlocking Rust's Power: Mastering the Chain of Responsibility Design Pattern

I remember in my earlier phase as a software engineer, how I mapped the MFC's command routing algorithm to Chain of Responsibility - and when finally it became clear, there was immense joy for me.I always say to my students that we should approach a problem not only from how's point of view but also from why's point of views.Curiosity is one of the greatest traits for a software engineer - nurture that, grow that - if you want to extract joy from your day-to-day journey as a software engineer.So here we go.My today's contribution to the learning community - Chain of Responsibility design pattern using Rust.The Chain of Responsibility design pattern is a behavioural design pattern that allows a request to pass through a series of handlers. Each handler can either handle the request or pass it to the next handler in the chain. This design pattern promotes loose coupling between the sender and receiver of a request, allowing more flexibility in handling specific actions.


Key Features of the Chain of Responsibility Pattern:



Decoupling:
The sender of the request doesn't need to know which handler will process the request.


Dynamic Chains:
You can change the chain of handlers at runtime or configure it based on the application's needs.


Multiple Handlers:
A request can be processed by a series of handlers in sequence until one of them handles the request, or it can propagate through all of them.Here's the source code...
⛶```
#[derive(Clone)]
struct Request{
request_type : String,
}
impl Request{
fn new(request_type : String) - Self{
Request{
request_type : request_type,
}
}
}

trait Employee {
fn set_successor(mut self, successor:Box);
fn handle_request(mut self, request: Request) ;
}
struct ProjectManager {
successor: Option,
}

struct TeamLeader {
successor: Option,
}

struct GeneralManager{
successor: Option,
}

struct CTO {
successor: Option,
}

impl Employee for CTO{

fn set_successor(mut self, successor: Box) {
//end of the chain... no successor
}
fn handle_request(mut self, request : Request) {
println!("I am the CTO... I must handle the request as this is the end of the chain");
println!("The request is handled at the CTO level...");
}
}

impl Employee for GeneralManager {
fn set_successor(mut self, successor: Box) {
self.successor = Some(successor);
}

fn handle_request(mut self, request : Request) {
if request.request_type == "generalmanager_level" {
println!("Handling the request at the General Manager level");
}
else if let Some(ref mut successor) = self.successor {
println!("I am the general manager...Forwarding the request to CTO...");
successor.handle_request(request);
}
}
}

impl Employee for ProjectManager {
fn set_successor(mut self, successor: Box) {
self.successor = Some(successor);
}

fn handle_request(mut self, request: Request) {
if request.request_type == "projectmanager_level" {
println!("Handling the request at the Project Manager level");
} else if let Some(ref mut successor) = self.successor {
println!("I am the Project Manager...Forwarding the request to General Manager");
successor.handle_request(request);
}
}
}

impl Employee for TeamLeader {
fn set_successor(mut self, successor: Box) {
self.successor = Some(successor);
}

fn handle_request(mut self, request : Request) {
if request.request_type == "teamleader_level" {
println!("Handling the request at the team_leader level");
}
else if let Some(ref mut successor) = self.successor {
println!("I am the teamleader....Forwarding the request to Project Manager");
successor.handle_request(request);
}
}
}

fn main() {
let request1 = Request::new("teamleader_level".to_string());
let request2 = Request::new("generalmanager_level".to_string());
let request3 = Request::new("cto_level".to_string());
let mut cto = CTO{successor: None};
let mut gm = GeneralManager{successor: None};
let mut manager = ProjectManager {successor: None};
let mut teamLeader = TeamLeader {successor: None};
gm.set_successor(Box::new(cto));
manager.set_successor(Box::new(gm));
teamLeader.set_successor(Box::new(manager));
teamLeader.handle_request(request3);
}
```Enjoy...