Why Security Teams Are Losing 80% of Their Logs

We started with the documentation. Turned on Palo Alto's S3-based log feed. The files immediately failed to decompress.

Time to break out the hex editor.

What the Documentation Said vs. What I Actually Found

Palo Alto documents their S3 integration format as ".json compressed by Snappy." Sounds straightforward. It's not.

The files started with plain JSON followed by control characters. The format uses an undocumented 0x10 record terminator—the first record is uncompressed JSON metadata, followed by a byte stream of Snappy-compressed JSON data.

This isn't corruption. This is a custom format with a clear disconnect between what engineering built and what the public documentation describes.

The result? We couldn't find a single competitor with support for this solution. Most users can't build and customize interfaces to handle undocumented formats.

Where Customers End Up Instead

When the S3 integration fails because nobody documented it properly, security teams fall back to legacy methods: Syslog or HTTP.

Syslog over internet and cloud networks drops 3-80% of events. Industry data confirms users consistently report 30-40% message loss for syslog over UDP, with drops up to 90% documented in production environments. At 500 messages per second, organizations see about 50% loss. At 1000 msg/sec, closer to 60% packet loss.

HTTP requires over-provisioning and constant high uptime to handle microbursting. The infrastructure costs add up fast.

Data is money. When you're losing 30-80% of your security logs, you're paying for storage twice—once at the source, once in your SIEM—and you're still operating with incomplete visibility.

The Real Cost Nobody Talks About

The technical problems are bad enough. The human cost is worse.

Security teams face a constant time sink. They have to go back to origin systems, hope the data is still there, and piece together what happened. But the bigger issue? Never knowing if something didn't happen or if the logs are just missing.

That uncertainty destroys trust in your entire detection capability.

Teams start to take their jobs less seriously. They quit on investigations without completion. They get frustrated. They miss things.

When you can't trust your data pipeline, you can't trust your security posture. Alert fatigue compounds because analysts lack the situational awareness to know whether hundreds of disparate alerts are linked to the same incident or separate issues.

Why the Industry Ignores the Problem

Here's what's damning: Palo Alto actually built the right solution. S3 is cloud-native, designed for bursty writes, compatible with serverless functions. It's the most reliable and cost-effective approach.

But the poor documentation tells a story about what the industry is ignoring.

Legacy vendors can use the old ways—Syslog, HTTP—without investment. They don't have to build support for properly documented S3 integration because customers don't know it exists or can't make it work.

We built the modern connector in less than a day.

Less than a day's engineering effort to build what nobody else bothered to support. The entire ecosystem has a financial incentive to keep customers on inferior methods that lose data and cost more.

What We're Doing About It

Ziggiz released an alpha version of our cloud-to-cloud Palo Alto Networks SASE connector. We dug in and built the solution on the modern option Palo Alto offered but failed to document properly.

We don't make conversations difficult. We make them easy.

Commoditizing the cyber lakehouse means not accepting half-truths in documentation or legacy because we can avoid work. It means building comprehensive solutions that address what vendors won't talk about.

When customers see it working reliably—built in less than a day, delivering complete log streams without the 30-80% loss—the reaction is anger and relief in equal measure.

Anger that they've been dealing with broken architectures.

Relief that someone finally solved it.

It's up to customers to demand better from their vendors. Or talk to us, because we're here to help.