Keynote: Passive DNS Collection and Analysis—The "dnstap" Approach, presented at Flocon 2014

by Paul A. Vixie,

Summary : DNS is a high volume low latency datagram protocol at the heart of the Internet -- it enables almost all other traffic flows. Any analysis of network traffic for security purposes will necessarily include contemporaneous DNS traffic which might have resulted from or directed that traffic. Netflow by itself can answer the question, "what happened?" but it cannot by itself answer the equally important question, "why?"
Collecting DNS query and response data has always been challenging due to the impedance mismatch between DNS as an asynchronous datagram service and available synchronous persistent storage systems. Success in DNS telemetry has historically come from the PCAP/BPF approach, where the collection agent reassembles packets seen 'on the wire' into DNS transaction records, with complete asynchrony from the DNS server itself. It is literally and always preferable to miss a transaction than to slow down a production DNS server due to passive DNS collection costs.
BPF/PCAP is not a panacea, though, since the complexity of state-keeping means that most passive DNS collectors are blind to TCP transactions, and all are blind to events which don't appear on the wire, such as cache purge or cache expiration events. The Farsight Security team has therefore designed a new open source and open protocol system called 'dnstap' with a transmission/reception paradigm that preserves the necessary lossiness of DNS transaction collection while avoiding the state-keeping of BPF/PCAP based systems.