Defense Information Systems Agency
DISA Thunderdome — Kafka & NiFi on Multi-Cloud
Q2 2023 – Q4 2023
Challenge
DISA required real-time ingestion of massive network security logs across DoD multi-cloud environments (EKS + AKS) with zero tolerance for data loss or latency.
Solution
Designed and deployed a Confluent Kafka + Apache NiFi pipeline architecture on Kubernetes across both AWS EKS and Azure AKS, delivered via full CI/CD automation with GitOps-managed configuration.
Outcomes
- 10TB+ daily security log ingestion
- 99.9% uptime across multi-cloud deployment
- 60% reduction in log processing latency
- Enabled real-time threat detection at DoD scale