We use many processes running in parallel on many different servers in HiveSight. At some points, we have over a hundred EC2 instances running. That scale creates a logging problem – it isn’t feasible to ssh into every machine to check its logs. There are open source systems like Facebook’s Scribe that handle data collection from lots of servers, but in some cases you’re looking for something much more simple.
Then Tal came up with the ingeneous solution of logging to Twitter.
Using Twitter API you can simply post a message to your account. For instance, in a bash script you can add the following line:
Next, we decided we want to be able to integrate that in a seamless way into our java code. The solution was to write a log4j appender. Nothing needs change inside your code – it’s just log4j. All you have to do is include a couple of jars, and add a configuration for the twitter appender to your log4j.properties file.
The code is available on http://code.google.com/p/twitter-log4j/.
We also log to twitter from our nagios monitors (using the same ‘curl’ method mentioned above) and from our continuous integration system (we use Hudson) using a plugin that was written by the community.
The main gotcha to keep in mind is that twitter has API rate limits, so you log all too frequently.
One solution to that is to set up a few accounts that you use just for logging, then follow them from your main account. However, that doesn’t scale too well either, and wouldn’t be very nice to do anyway given twitter’s own infamous scalability problem, which brings us to our next solution – logging to AWS SimpleDB. We’ll release that code too very soon, so stay tuned.
HiveSight Technology Blog 