BIPI
BIPI

Your SIEM Bill Doubled Again. Here is How to Stop the Bleed

Cybersecurity

Splunk and Sentinel costs are growing 40-80% YoY for most teams we audit. The fix is not a vendor swap, it is log tiering, source-side filtering, and routing logs to the storage they actually deserve.

By Arjun Raghavan, Security & Systems Lead, BIPI · January 3, 2024 · 7 min read

#siem#cost-optimization#soc#operations

A retail customer called us in panic last quarter. Their Splunk bill went from 480K to 1.1M in eighteen months. Nobody added a major workload. Logs just kept growing, retention stayed at 13 months, and the indexer cluster needed two more nodes. The CFO wanted answers. The CISO wanted a plan that did not say 'rip and replace'.

Where the volume actually comes from

When we ran the breakdown, six log sources accounted for 71% of daily ingest. Windows DNS debug logs were the worst offender at 22%. Cloud audit logs from a single AWS account hit 14% because someone enabled CloudTrail data events for every S3 bucket. Proxy logs were 18%. The detections written against these sources? About 9 active rules total. We were paying full ingestion price to feed search infrastructure that touched the data once a quarter.

The tiering model that works

Stop treating all logs as equal. We split them into three tiers and routed accordingly:

  • Hot tier: 30-90 days in the SIEM, indexed, searchable in seconds. Reserve this for sources you actually run detections against. Authentication, EDR, firewall denies, identity provider, suspicious DNS.
  • Warm tier: 90-365 days in cheap object storage with a query layer like Splunk SmartStore, Sentinel basic logs, or a separate data lake. You can search it during incidents. It just takes minutes instead of seconds.
  • Cold tier: compliance archive. S3 Glacier or equivalent. Pull it out only for audit or major investigation. Costs pennies per GB-month.

For that retail customer, moving DNS debug, proxy, and verbose CloudTrail to warm tier dropped indexed daily volume by 38%. Splunk renewal landed at 720K instead of 1.4M. The detections still worked because the rules already pointed at the hot tier sources.

Drop at the source, not at the SIEM

Cribl, Logstash, or even native filtering at the agent level lets you drop fields and events before they hit the priced indexer. We routinely strip 20-40% of Windows event log volume by dropping noisy event IDs that no detection cares about. 4624 type 3 logons from service accounts. 5156 firewall connection allowed. 4634 logoff events. Keep the schema for forensics by archiving raw to cold tier, but stop paying to index every byte.

Sentinel specific traps

Microsoft Sentinel charges per GB ingested into Log Analytics. The trap is that 'free' connectors like Defender for Endpoint quietly add 5-20GB per thousand devices per day depending on configuration. Auxiliary Logs and Basic Logs tiers exist now and run at roughly 20% of the analytics tier price. Use them for high-volume, low-query sources. Do not put authentication or EDR there because the query language and retention behavior is restricted.

What to negotiate at renewal

  1. Lock in a workload-based or ingest-day pricing model if you are above 200GB/day. Per-GB pricing punishes growth.
  2. Negotiate auto-discount tiers that kick in at volume thresholds. Vendors will give 15-25% off list at 1TB/day if you ask.
  3. Get a written commit credit policy. If you over-purchase capacity, those credits should roll for at least 12 months.
  4. Refuse multi-year ramps without exit ramps. The market is moving fast and lock-in is leverage you give up.
If your SIEM team cannot tell you the cost-per-detection for each log source, you do not have a SIEM strategy. You have a logging habit.

The hard part of this work is not technical. It is convincing the team that dropping logs is safe. Run a six-week pilot, archive the dropped data to cold so you can replay it if a detection needs tuning, and measure detection performance before and after. In every engagement we have run, detection efficacy stayed flat or improved because analysts stopped drowning in noise.

Read more field notes, explore our services, or get in touch at info@bipi.in. Privacy Policy · Terms.