Multi-Node Threat Intelligence Pipeline (Honeypot)

Wanted one project touching everything learned: cloud, automation, DevOps, AI integration, data, visualization, security. Not another off-the-shelf Cowrie/T-Pot deployment.

Three architecture iterations over roughly five weeks. V1: single DigitalOcean droplet logging HTTP requests, Gemini-cleaned, cron-pushed to GitHub Pages, which proved bots find an unadvertised IP within hours but overloaded one droplet doing everything. V2: split across three droplets writing to Supabase, with hourly GitHub Actions pulling, cleaning, and pushing. Over 100 commits in one day tuning batch size and timeout triggered GitHub's anomaly detection and broke the Actions schedule, requiring a rebuild in a fresh repo to recover. Also hit Gemini free-tier rate limits and fell about 95% behind on cleaning, so switched to paid Vertex AI Gemini.

47,598 total probes, 5,095 unique IPs, 11,366 critical and 14,501 high severity classified, published openly as a dataset with a Zenodo DOI, and 1,755 high-severity IPs reported to AbuseIPDB. Audited AI severity labels against 100 human-reviewed samples, improving Cohen's kappa from 0.66 to 0.80.