-
-
-
Scout Monitoring
Free Django app performance insights with Scout Monitoring. Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.
-
shodohflo
Pure Python netflow and DNS correlation, with reusable Frame Streams, DnsTap and Protobuf implementations
To be fair, if that's what you need ProtoBuf isn't the only option. Cap'n Proto[1], JSON Schema[2], or any other well supported message-definition language could probably achieve that as well, each with their own positives and negatives.
[1]: https://capnproto.org/
[2]: https://json-schema.org/
To be fair, if that's what you need ProtoBuf isn't the only option. Cap'n Proto[1], JSON Schema[2], or any other well supported message-definition language could probably achieve that as well, each with their own positives and negatives.
[1]: https://capnproto.org/
[2]: https://json-schema.org/
It was also used for Farsight's tunnelled SIE called NMSG. I wrote a pure python protobuf dissector implementation for use with Scapy (https://scapy.readthedocs.io/en/latest/introduction.html) for dissecting / tasting random protobuf traffic. I packaged it with an NMSG definition (https://github.com/m3047/tahoma_nmsg).
I re-used the dissector for my Dnstap fu, which has since been refactored to a simple composable agent (https://github.com/m3047/shodohflo/tree/master/agents) based on what was originally a demo program (https://github.com/m3047/shodohflo/blob/master/examples/dnst...) because "the people have spoken".
Notice that the demo program (and by extension dnstap_agent) convert protobuf to JSON: the demo program is "dnstap2json". It's puzzlingly shortsighted to me that the BIND implementation is not network aware it only outputs to files or unix sockets.
The moment I start thinking about network traffic / messaging the first question in my mind is "network or application", or "datagram or stream"? DNS data is emblematic of this in the sense that the protocol itself supports both datagrams and streams, recognizing that there are different use cases for distributed key-value store. JSON seems punctuation and metadata-heavy for very large amounts of streaming data, but a lot of use cases for DNS data only need a few fields of the DNS request or response so in practice cherry picking fields to pack into a JSON datagram works for a lot of classes of problems. In my experience protobuf suffers from a lack of "living off the land" options for casual consumption, especially in networked situations.
It was also used for Farsight's tunnelled SIE called NMSG. I wrote a pure python protobuf dissector implementation for use with Scapy (https://scapy.readthedocs.io/en/latest/introduction.html) for dissecting / tasting random protobuf traffic. I packaged it with an NMSG definition (https://github.com/m3047/tahoma_nmsg).
I re-used the dissector for my Dnstap fu, which has since been refactored to a simple composable agent (https://github.com/m3047/shodohflo/tree/master/agents) based on what was originally a demo program (https://github.com/m3047/shodohflo/blob/master/examples/dnst...) because "the people have spoken".
Notice that the demo program (and by extension dnstap_agent) convert protobuf to JSON: the demo program is "dnstap2json". It's puzzlingly shortsighted to me that the BIND implementation is not network aware it only outputs to files or unix sockets.
The moment I start thinking about network traffic / messaging the first question in my mind is "network or application", or "datagram or stream"? DNS data is emblematic of this in the sense that the protocol itself supports both datagrams and streams, recognizing that there are different use cases for distributed key-value store. JSON seems punctuation and metadata-heavy for very large amounts of streaming data, but a lot of use cases for DNS data only need a few fields of the DNS request or response so in practice cherry picking fields to pack into a JSON datagram works for a lot of classes of problems. In my experience protobuf suffers from a lack of "living off the land" options for casual consumption, especially in networked situations.