A story about investigating AWS cost anomalies, Amazon Q's AI debugging attempts, and discovering the real culprit was expired Reserved Instances all along.
Recently, I started getting those lovely "AWS Cost Anomaly Detected" emails. You know the ones — they sound like the IRS of the cloud:
"We've identified unusual activity in your account."
I open it up and it says: "EC2 and RDS costs increased by about $1 per day."
Now, $1 a day doesn't sound like much… but if you've used AWS long enough, you know that's how it starts. It's never just one dollar. It's the cloud equivalent of hearing a drip in your house — you know that's going to end with a plumber and a $600 bill.
The Setup
The report didn't give much detail. Just that my EC2 instances were somehow racking up 48 hours worth of charges every 24 hours. Cool. Love it when my servers discover time travel.
I only have two EC2 instances in an Auto Scaling group and a single RDS instance (not even Multi-AZ). So the math kind of made sense — two instances, two extra "days" of billing — but the logic didn't.
The only recent change I'd made? A tiny, innocent-looking security group update.
I'd added a self-reference rule to allow my instances to talk to each other. No big deal, right? Security groups are free, and it's a super common pattern. But in AWS land, "no big deal" is usually followed by a surprise on your bill.
Calling In the AI Cavalry (a.k.a. Amazon Q)
Naturally, I turned to Amazon Q, the new AI assistant in the AWS console, and asked it what was happening. Here's the summary of what it told me (paraphrased for sanity):
"You're right — security groups themselves don't cost anything. But maybe you accidentally unblocked traffic that was previously blocked, and now your VPC is generating new data transfer charges."
It even gave me a little cost breakdown, CSI-style:
- Amazon VPC: +$1.01
- EC2 Compute: +$2.07
- RDS: +$1.89
And then came the grand theory:
"The self-reference might have enabled internal communication, database replication, backups, or secret background processes that were previously blocked."
In other words, I accidentally opened a portal and now my instances were gossiping behind my back.
I Tested the Theory
Okay, fair enough. Maybe I unblocked some chatty microservice traffic. So, I rolled back the change — deleted the self-reference rule — and waited for the next cost report.
The charges didn't stop.
Now I'm thinking… great. I've created some sort of VPC feedback loop and my EC2s are stuck sending each other cat memes.
The Real Culprit
After some digging, I discovered the real cause — and this is where it gets hilarious (and slightly sad):
My 3-year Reserved Instances had expired.
That's right. Nothing to do with security groups, data transfer, or phantom network traffic. My discount just ran out.
Those "new" charges? They weren't new at all — they were just the actual on-demand prices I hadn't paid in years.
Lessons Learned (and a Tiny Rant)
So what did we learn?
- AWS Cost Anomaly Detection is great at spotting spikes — but not so great at explaining them.
- Amazon Q, bless its virtual heart, can't (yet) say, "Hey, your Reserved Instances expired. This is literally the entire reason your bill went up."
- Sometimes the scariest anomalies are just… time.
Pro Tip
If you ever get cost anomaly alerts and can't figure out why, check your Reserved Instances and Savings Plans expiration dates first. It's the easiest $3 mystery you'll ever solve.
So, at the end of the day:
- I didn't break my VPC.
- I didn't cause a billing black hole.
- I just forgot that three years ago I did something responsible — and time finally caught up.
Now, excuse me while I go buy another 3-year reservation before my wallet notices.
Happy DevOps/FinOps!