The Point-in-Time Problem
It's 3 AM. A FinOps tool scans your infrastructure. It sees a VM running at 4% CPU. The recommendation: "This resource is idle. Consider downsizing or terminating."
At 10 AM, that same VM handles 2,000 requests per second and runs at 78% CPU. It powers your checkout flow. If you'd followed that 3 AM recommendation, your customers would get 503 errors during peak hours.
This is the fundamental flaw in most cloud optimization tools: they take a point-in-time snapshot and make a recommendation based on that single measurement. It's like measuring someone's heart rate while they sleep and diagnosing them as having dangerously low cardiac output.
Real workloads have patterns. Some spike during business hours and idle at night. Some burst on weekdays and flatline on weekends. Some have monthly peaks around billing cycles or quarterly spikes during reporting. A single measurement tells you nothing about the right optimization action.
The Three Types of Workload Patterns
After analyzing thousands of cloud resources, workload utilization falls into one of five patterns. Each pattern requires a different optimization strategy:
Pattern 1: Diurnal (Day/Night Cycle)
What it looks like: High utilization during business hours (8 AM to 8 PM), low utilization overnight and weekends.
Example: Web application serving users in a single timezone. B2B SaaS product used during working hours. Internal tools accessed by office employees.
Key metric: Peak-to-trough ratio > 2.5x (e.g., 70% peak vs 10% trough)
Correct optimization: AUTOSCALE, not rightsize.
If you rightsize this VM to handle the average load, you'll crash during peak. If you keep it provisioned for peak, you waste money 16 hours/day. The correct answer is autoscaling that adds capacity during peak and removes it during trough.
Pattern 2: Steady State (Flat Line)
What it looks like: Consistent utilization 24/7. Doesn't matter what time of day or which day of week. CPU stays between 30-50% (or 5-15%) all the time.
Example: Background queue processors. Database replicas. Cache servers. API gateways with global traffic.
Key metric: Coefficient of variation < 0.3 (low variance relative to mean)
Correct optimization: RIGHTSIZE.
If a VM consistently runs at 15% CPU with no peaks, it's genuinely over-provisioned. Downsize it. If it consistently runs at 40%, it's well-sized. Leave it alone. The stability of the pattern means a point-in-time measurement actually IS representative.
Pattern 3: Burst/Sporadic (Mostly Idle, Occasional Spikes)
What it looks like: 80%+ of the time near zero. Occasional bursts to 60-90% lasting minutes to hours. Unpredictable timing.
Example: CI/CD build servers. Batch processing jobs triggered by events. Data pipeline workers. ML inference endpoints with variable traffic.
Key metric: P50 < 10%, P99 > 50% (idle most of the time, intense during bursts)
Correct optimization: SPOT INSTANCES or SERVERLESS.
This workload doesn't need a permanently provisioned VM. It needs capacity on demand. Use Azure Spot VMs (60-90% discount, accept 30-second eviction notice), AWS Spot Fleet, or move to serverless (Azure Functions, AWS Lambda, Cloud Run) where you pay only for actual execution time.
Pattern 4: Weekly Cycle
What it looks like: Active on weekdays, dead on weekends. Or the reverse (media/entertainment workloads peak on weekends).
Example: Enterprise B2B applications. Payroll processing. Internal analytics dashboards.
Key metric: Weekday average > 2x weekend average
Correct optimization: SCHEDULE.
Stop the resource on Friday evening, start it Monday morning. You save ~28% of the monthly cost (2 out of 7 days) with zero risk to weekday operations. AWS Instance Scheduler and Azure Auto-shutdown handle this natively.
Pattern 5: Growing (Upward Trend)
What it looks like: Utilization has been climbing steadily over weeks. What was 30% a month ago is now 55% and still rising.
Example: Applications gaining users. Data volumes growing. Feature additions increasing baseline load.
Key metric: Trend slope > 5% increase per week over 14+ days
Correct optimization: AUTOSCALE NOW (preventive).
This workload will hit capacity limits soon. Don't wait for it to crash. Set up autoscaling proactively with the current load as the baseline and 2-3x as the ceiling. This is cheaper than an emergency over-provision after an outage.
The Decision Matrix
| Condition | Recommendation | Why |
|---|---|---|
| Peak/trough ratio > 2.5x, repeating daily | Autoscale | Needs capacity elasticity, not a fixed size |
| Steady utilization < 25%, no peaks above 40% | Rightsize down | Genuinely over-provisioned |
| Steady utilization 25-70%, no concerning peaks | No action | Well-sized for its workload |
| Idle 80%+ of time, rare burst spikes | Spot/Serverless | Paying for idle capacity you rarely use |
| Weekday > 2x weekend utilization | Schedule (stop weekends) | Pay for 5 days instead of 7 |
| Trending upward > 5%/week | Autoscale (proactive) | Will hit limits soon |
| Under 5% CPU, < 1MB network, 14+ days | Actually idle (zombie) | Terminate or investigate |
The critical insight: only the last row is a true "idle" resource. Everything else is a workload pattern that requires a specific, different action.
See how much you're wasting
Get a free 7-day cloud audit. No credit card, no agents, read-only access.
How to Detect Patterns: The 30-Day Signal Method
A single metric check tells you nothing. You need 30 days of hourly data to classify a workload pattern. Here's why:
- 7 days catches daily patterns but misses weekly cycles
- 14 days catches weekly patterns but can't distinguish steady trends from temporary spikes
- 30 days gives you 4 full weekly cycles, reveals monthly patterns, and provides enough data for statistical confidence
What to Measure
CPU alone is not enough. A full picture requires:
| Metric | What It Tells You | Tool Example |
|---|---|---|
| CPU (average + P95) | Compute demand pattern | CloudWatch, Azure Monitor |
| Memory (average + P95) | Working set size; cache vs compute workload | CloudWatch agent, Azure VM Insights |
| Network In/Out | Traffic pattern; often correlates with user demand | VPC Flow Logs, NSG Flow Logs |
| Disk IOPS | Storage-bound vs compute-bound | EBS metrics, Managed Disk metrics |
| Connection count | Active users or service dependencies | Load balancer metrics |
Multi-signal classification:
| CPU | Memory | Network | Classification |
|---|---|---|---|
| Low | Low | Low | Zombie (all signals agree) |
| Low | High | Low | Memory workload (cache/in-memory DB). Do NOT downsize. |
| Low | Low | High | Network proxy/gateway. Do NOT terminate. |
| Diurnal | Diurnal | Diurnal | Application with user traffic. Autoscale. |
| Low average, high P95 | Stable | Bursty | Batch processor. Use spot or serverless. |
How to Set Up Autoscaling Correctly
Most teams set up autoscaling with default thresholds and wonder why it doesn't work. The defaults (70% scale-out, 30% scale-in on AWS) are reasonable starting points, but optimal thresholds depend on your workload's response time requirements.
Computing the Right Thresholds
Scale-out threshold (when to add capacity):
Take your P90 CPU utilization during normal peak hours. That's your "comfortable operating point." Set scale-out at 80% of that value.
Example: Peak hours run at 65% CPU average. Scale-out threshold = 65% × 0.8 = 52%.
Why 80%? Because by the time the autoscaler detects the threshold breach, launches a new instance, and the instance passes health checks (typically 2-5 minutes), your existing instances need headroom to handle the traffic that arrived during that scaling delay.
Scale-in threshold (when to remove capacity):
Set this at the P25 of your trough utilization. You want to be conservative about removing capacity because scaling down too aggressively causes flip-flopping (scale out, scale in, scale out, scale in).
Example: Trough hours run at 15% CPU. Scale-in threshold = 15% (P25 of trough).
Cooldown period:
Minimum 5 minutes between scaling actions. This prevents flip-flopping. AWS uses 300 seconds default; GCP uses 600 seconds (10 minutes). For most workloads, 5 minutes is correct. For slowly-warming applications (JVM, large ML models), use 10-15 minutes.
Min and Max Instance Count
Minimum instances: Must handle your trough load + one instance failure. If trough load needs 1 instance, set min to 2 (fault tolerance). If trough needs 3, set min to 4.
Maximum instances: Must handle your historical peak + 30% buffer for growth. If peak has historically needed 8 instances, set max to 10-12.
Don't set max too high. A runaway autoscaler during a traffic spike or DDoS can scale to 50 instances before anyone notices. Set a realistic maximum and alert if it's reached.
The Rightsizing Methodology (When It IS Appropriate)
For workloads that are genuinely steady-state (Pattern 2), rightsizing is straightforward:
- Collect P95 CPU and memory for 30 days
- Target 60-70% utilization at the new instance size (P95 should be 60-70% of the new instance's capacity)
- Validate with P99 (must stay below 85% of new capacity; otherwise you'll have occasional performance issues)
- Step down one size at a time (don't jump from xlarge to small; go xlarge → large → medium over 2-3 weeks)
- Monitor for 7 days after each step-down before continuing
Instance family selection matters too:
| Your P95 Pattern | Current Instance | Recommended Family |
|---|---|---|
| High CPU, low memory | General purpose (m5) | Compute-optimized (c5) - more CPU per dollar |
| Low CPU, high memory | General purpose (m5) | Memory-optimized (r5) - more RAM per dollar |
| Both high | General purpose (m5) | Stay general purpose, just resize |
| Very low both | Large instance | Burstable (t3) - cheapest for low utilization |
Real-World Example: A B2B SaaS Application
A team runs their API on 3x Standard_D4s_v3 (Azure) at $140/month each ($420/month total).
A basic optimization tool says: "Average CPU is 18%. Rightsize to D2s_v3."
But the 30-day analysis reveals:
- Peak hours (9 AM - 6 PM IST): 62% average CPU, P95 = 78%
- Off hours (6 PM - 9 AM): 4% average CPU
- Weekends: 2% average CPU
- Pattern: Diurnal + Weekly
The correct recommendation:
- Enable Azure Virtual Machine Scale Sets (autoscaling)
- Min instances: 1 (handles off-hours/weekend traffic)
- Max instances: 4 (handles peak with 30% buffer)
- Scale-out at 55% CPU, scale-in at 20% CPU
- Result: runs 1 instance for 14 hours/day + weekends, 3-4 instances for 10 peak hours
Cost before: $420/month (3 instances 24/7) Cost after: ~$185/month (autoscaled) Savings: $235/month (56% reduction) without any performance impact
If they'd followed the "rightsize to D2s_v3" recommendation instead: $210/month but with P95 CPU at 156% during peak (application crashes).
FAQ
How long should I collect data before making an optimization decision?
30 days minimum for any change. 7 days is not enough because you'll miss weekly patterns and monthly spikes. 14 days gives you a partial picture but can be thrown off by holidays or unusual events. 30 days captures 4 full weekly cycles and provides statistical confidence for pattern classification.
Can I use both autoscaling and rightsizing together?
Yes, and you should. First, rightsize the instance type that your autoscaler uses (the launch template). If your autoscaler launches m5.2xlarge instances but each instance only needs m5.large capacity, fix the template. Then the autoscaler adds smaller, cheaper instances during peaks.
What about memory-based autoscaling? My app doesn't use much CPU but needs RAM.
Most cloud autoscalers support custom metrics. On AWS, publish memory utilization to CloudWatch via the CloudWatch agent, then create a target tracking policy on that metric. On Azure, Virtual Machine Scale Sets can scale on custom Application Insights metrics. Memory-based scaling is correct for JVM applications, caches, and in-memory databases.
Is autoscaling worth it if my peak is only 2x my trough?
Yes if the cost difference justifies the setup effort. A 2x ratio means you're paying 50% more than necessary during off-peak. For a $500/month instance, that's $250/month in potential savings. Autoscaling setup takes 2-4 hours. Payback period: less than 1 month.
My application takes 5 minutes to start. Does autoscaling still work?
Yes, but you need to configure the instance warmup period accordingly. Set warmup to 5-7 minutes so the autoscaler doesn't count still-booting instances in its capacity calculation. Also consider: predictive scaling (AWS has this), which pre-warms instances 5-10 minutes before the predicted spike based on historical patterns. For applications with very long startup times (15+ minutes), pre-warming on a cron schedule might be more reliable than reactive autoscaling.