HiveSight Technology Blog

R&D Lab – You pay for your mistakes

Posted by Tal on December 8, 2008

I made a big mistake a few months ago. I bought a few servers for us to use in our office – you know, it’s important to have a few machines at your disposal so you can run intensive tasks on them or run all kind of different tests you couldn’t otherwise do on your single development machine.

Makes sense right? Well, not in term of ROI and here is why:

I bought a fairly good machine with 4GB of RAM and an AMD 6400+ X2 CPU – that cost me about $1000. Let’s assume the machine is utilized for about 8 hours a day (in reality I guess most lab machines are used much less). Assuming 250 work days a year that’s 2000 CPU hours a year. Let’s say the life expectancy of this machine is 3 years so for 6000 hours I paid $1000. That’s about $0.16 a machine hour.

Now let’s compare this to a machine I can rent from Amazon EC2 (prices). Assuming Moore’s law will hold true for the next three years (limits of Moore’s law) today’s EC2 large instance (m1.large) will be equivalent to two of the machines I bought for my office (I’m assuming that Amazon are nice enough to lower prices for a given hardware configuration as it becomes older). Add to my in-house investment the electricity / cooling / management (as in personnel) and it will make my (already) bad investment even worse. Oh, and for my in-house machine I had to put the money upfront and not “pay as I go”.

Now what happens If I need more machines at one time (for instance, to do some tests before a release) and none at another time (like every night). Having these machines hosted on EC2 gives me the flexibility to use as many machines as I need and then don’t pay for the time they sit idle.

The bottom line – If you’re a small startup – don’t buy lab machine. If you’re a big company you could probably save thousands of dollars every month by hosting your lab on EC2.

Posted in AWS | Tagged: , , | Leave a Comment »

Sovling New Version Deployment: Staging to Production with EC2

Posted by Elad Kehat on December 8, 2008

This is the first in a series of posts inspired by meeting Martin Buhr and Simone Brunozzi from Amazon Web Services Europe, where we’ll share some of our experience using AWS.

Summary first (for those of you who don’t feel like reading through my musings): When deploying a new version of our web application, we stage it on a new set of EC2 instances first, leaving he old production intact, then switch the elastic IP and (well tested) staging becomes production. No downtime involved.
I find that the most enjoyable “eureka moments” happen when you didn’t think of a situation as a problem and weren’t even looking for a solution. You just do something in a new way without giving it too much thought, only then to realize that you stumbled into something great. That kind of thing happened to us a short while after we moved our web application to EC2.

The problem was deploying a new version of our web app. We run short development cycles, so that process takes place every 2-4 weeks. Occasionally it runs smoothly – that is to say, most of the time it doesn’t, so there’s downtime involved (not to mention stress).
The best practice is to have a staging environment where you set up the new version first and test it before you load it to the production environment. Ideally, the staging environment mimics the production environment one-to-one. Of course, that seldom happens.
In the bad old days, when you either had your own data center, or (as in our case) paid a monthly fee for hosting, a perfect staging environment would mean paying twice. That just doesn’t feel like cash well spent in a start-up. So our stating environment was a mini-set up in our office, that didn’t mimic production all too well. As you can expect, new version deployment involved lots of down time, blood, sweat and tears.

The first time we set up our version on EC2, a different mechanism came very naturally:
1. The “old” production system is running. Don’t touch it.
2. Reserve all the EC2 instances you need to set up a new system from scratch. They become the staging environment for the new version.
3. Set up the new version on those new instances.
4. Test. Fix. Repeat.
5. Test some more (truth be told, we never actually do that, and neither will ou, but we all really should 😦 )
6. Switch the elastic IP address to the new web server instance.
7. The well-tested staging environment now becomes production. No down time. No unexpected problems.
8. Leave the old system up for a while – just in case.

The cost:
You paid for a replica of the system for just a few hours, or maybe a couple of days: 15 – 100 times less expensive than keeping all the hardware for a staging environment.

The benefits:
No downtime!
No stress – you have all the time you need – the world can keep on using the old version while you’re setting up the new one.
Only set-up once – as the staging environment actually becomes the production after you switch the elastic IP, there’s no need to repeat the process again on the production.
If you missed something, the old production system is still available for a while, so you can easily switch back.

In short, it’s even better even than what I’d call a “perfect” staging environment in the past. And it all happened without even planning – we only realized what we’ve done after the fact. Bliss.

Posted in AWS | Tagged: , , | Leave a Comment »

Twitter as a Logging Tool

Posted by Elad Kehat on December 8, 2008

We use many processes running in parallel on many different servers in HiveSight. At some points, we have over a hundred EC2 instances running. That scale creates a logging problem – it isn’t feasible to ssh into every machine to check its logs. There are open source systems like Facebook’s Scribe that handle data collection from lots of servers, but in some cases you’re looking for something much more simple.
Then Tal came up with the ingeneous solution of logging to Twitter.

Using Twitter API you can simply post a message to your account. For instance, in a bash script you can add the following line:

curl -u “<your account name>:<your password>” -d status=”Your log message here” http://twitter.com/statuses/update.json

Next, we decided we want to be able to integrate that in a seamless way into our java code. The solution was to write a log4j appender. Nothing needs change inside your code – it’s just log4j. All you have to do is include a couple of jars, and add a configuration for the twitter appender to your log4j.properties file.
The code is available on http://code.google.com/p/twitter-log4j/.

We also log to twitter from our  nagios monitors (using the same ‘curl’ method mentioned above) and from our continuous integration system (we use Hudson) using a plugin that was written by the community.

The main gotcha to keep in mind is that twitter has API rate limits, so you log all too frequently.
One solution to that is to set up a few accounts that you use just for logging, then follow them from your main account. However, that doesn’t scale too well either, and wouldn’t be very nice to do anyway given twitter’s own infamous scalability problem, which brings us to our next solution – logging to AWS SimpleDB. We’ll release that code too very soon, so stay tuned.

Posted in Uncategorized | Tagged: , , | Leave a Comment »

Introducing The HiveSight Tech Blog

Posted by hivesight on December 7, 2008

We’ve been meaning to start a technology blog for a while in HiveSight, just never found the time.
We’re building some cool technology, if I may say so myself, and like any decent developers nowadays, we couldn’t do half of it if not for open source, and great blog posts by other developers on cool new technologies, and on how they solved problems.
So it’s high time we started giving back to the community by sharing our own experiences, tips and ideas. We hope you find them useful.

Posted in Uncategorized | Leave a Comment »

 
Create your own website, free on WordPress.com.Create your own website.
Start free