The problem
Threat feeds are inputs, not finished intelligence. They arrive with different field names, timestamps, duplicate indicators, and confidence levels. An hourly detection workflow needs to turn that moving collection into a clean, bounded, and explainable set of domains for scoring.
Feed ingestion
Python collectors retrieved new suspicious-domain observations from approved sources, including DNS-oriented sources, URLScan-derived data, and internal or commercial feeds. Each adapter translated source-specific fields into a common record shape before the data moved downstream.
Feed names, endpoints, and contractual details are intentionally omitted.
Deduplication
Normalization happened before identity checks so superficial differences—case, trailing dots, or source formatting—did not create duplicate work. OpenSearch state and object-storage artifacts provided durable checkpoints across runs.
Deduplication operated at more than one level: within a feed response, across feeds in the same run, and against recently processed indicators.
Whitelist handling
Allowlist logic was explicit, reviewable, and applied after normalization. The design accounted for exact domains and parent/child relationships without assuming that every subdomain inherited the same risk. Changes to whitelist data were treated as controlled configuration rather than hard-coded exceptions.
Time-window logic
Every run defined a clear observation window and retained source timestamps. Late-arriving data, missing dates, and replays were handled deliberately. This made reruns deterministic and helped distinguish a genuinely quiet hour from a failed collector.
Scheduled processing
Cron provided a straightforward hourly cadence. Each run was designed to be idempotent: a retry should not multiply outputs or spend enrichment quota twice. Intermediate checkpoints separated ingestion, validation, feature generation, scoring, and publication.
Model scoring
Validated domains flowed through the same feature transformations used in model training. Model scores were carried alongside provenance and rule-based context so downstream analysts could understand both the prediction and the evidence behind it.
Storage
Amazon S3 held batch inputs, outputs, models, and audit-friendly artifacts. OpenSearch supported fast lookups and stateful deduplication. JSON and CSV remained useful interchange formats at system boundaries where simplicity and inspectability mattered.
Monitoring
Operational checks focused on feed freshness, records received, records rejected, deduplication rate, scoring volume, processing time, and publication success. Empty batches were recorded as first-class outcomes; silence should be explainable.
Reliability considerations
- Bound network calls with timeouts and retries.
- Quarantine malformed records instead of silently dropping them.
- Persist checkpoints so individual stages can be replayed.
- Make output writes atomic where possible.
- Track schema changes and source-specific failure rates.
- Keep secrets and private feeds outside code and public documentation.
Lessons learned
Hourly data systems reward clarity. Normalization rules, time boundaries, and durable state are often more important than elaborate orchestration. Reliability improves when each stage can explain what it received, changed, rejected, and emitted.
Technology stack
Python, DNS data, URLScan, threat feeds, cron, Amazon S3, OpenSearch, data validation, feature engineering, JSON, and CSV.