Rsyslog and ZeroMQ

Posted on May, 30th 2012 in Uncategorized · Write a comment

As we get ready for the first deployments of our big data streaming platform, we’re pleased to announce another open source contribution. As we mentioned in our previous post, we’re big fans of Rsyslog. Well, we also have a lot of affection for a complementary component, ZeroMQ. As many probably know, ZeroMQ is a mature open-source high-speed message library, currently supporting bindings in about 30 languages. We use ZeroMQ as a component of our real-time data streaming solutions.

For any project using ZMQ, the ability to utilize centralized logging via syslog is desirable. We have created input and output plugins for Rsyslog that allow ZMQ messages to be put into Rsyslog, or output from Rsyslog. We built our plugins using the czmq library (https://github.com/zeromq/czmq), which sits atop the ZMQ API. This enabled us to support the latest version of ZMQ (3.1), and the older stable version (2.1). This library was chosen as it hides the differences in the ZMQ 2.1 vs 3.1 API, and allows our plugin to work in both environments. Prior work (https://github.com/aggregateknowledge/rsyslog-zeromq) was written directly on top of the ZMQ API, and doesn’t support the current version of ZMQ. The output plugin supports Rsyslog V6 config format, and there are example config settings here (TBD).

So, what do we allow you to do with the plugins?

  • BIND or CONNECT
  • Specify any number of worker threads
  • Specify thread affinities
  • Specify any zsockopt option
  • Input plugin supports SUB, PULL and XSUB
  • Output plugin supports PUB, PUSH and XPUB

Hammer on it! We want to see these plugins put through their paces. We are doing the same, and plan to continue to improve them over time.

 

Hello world!

Posted on May, 7th 2012 in Introductions · Write a comment

As Platform Architect for Talksum, I’m beyond pleased that the first post I’m writing  for the company is to announce an open source contribution.  I can’t think of a better way to start.  Writing this, I’m looking back to when I first found Yggdrasil Linux in a book store in 1995, and I’m blown away by what’s happened to our industry since then.   This global network that so many of us collaborate over today exists thanks to the efforts of people who believe that we can all benefit from working together.

Today, in the world built on those contributions,  we find ourselves struggling to find new ways to make sense of all of the information we’re producing.  We all know that with better ways to catalog and analyze data, we can gain deeper insights into ourselves and the challenges we face as individuals, companies, and societies.  At Talksum, we’ve assembled an experienced and talented team of individuals who believe that we can understand the impact of our decisions earlier, and we’re building the tools to do it.  As the world moves ever faster, we must understand the consequences of our decisions in less time.

Over the years, as Dale Russell (our CTO) and I have collaborated on the “big data issue” (can we coin a better term for this, as an industry?), we’ve come to deeply respect an open source project that is at the heart of many Linux systems: rsyslog.  As a workhorse for getting systems and applications data to where it needs to go, we feel it is without peer.  Additionally, Rainer Gerhards, author of rsyslog, is an absolute pleasure to collaborate with.  So as a company, with the blessing of Alex Varshavsky (our CEO… thanks, Alex!) we made the decision to contribute to the project.

Our first contribution is an experimental output module to connect Rsyslog to another project we have enormous respect for:  Redis.  It is now checked in on the development branch of rsyslog.  I’ll be smoothing out some rough edges in the code over the next couple of days, but it’s ready to start playing with.  We’re keeping a copy of the code on our github site as well.  Feedback and contributions are more than welcome!

Hello, World.

Brian Knox, Platform Architect, Talksum, Inc.