Alright, a while ago kevin write an erlang wrapper for libmemcache. As anyone who has used libmemcache is probably aware of, libmemcache does not fail gracefully. In fact it fails in a way that takes everything down in one blow.

Of course, it also rarely fails, unless the CPU load is very high, or there is a problem with your memcacheD servers.

In any case if the memcache servers failed, or timed out the Jabber server was breaking. This is similar to our problems with cmemcache on the RPC server side, where on occasion under high loads libmemcache will timeout and fail hard, taking the entire RPC server with it.

To mitigate these problems for the time being I replaced the libmemcache wrappers with python interface, http://www.lysator.liu.se/~tab/erlang/py_interface/. Currently python interface is just being used to interface between erlang and the memcache servers, to assure that memcacheD and high loads won’t take the jabber server down.

In the long run I want to look into http://code.google.com/p/cacherl/ and at least benchmark it against memcacheD.

Work It, Make It, Do It, Makes Us

Admittedly it’s been a long time since my last post, but a lot has been happening in the background. We’ve recruited another team member, Eric, whom we will introduce on the blog soon. Roland and George have been plowing along on HablaPlus – and I just pushed the stable multi-server edition of Hab.la.

What does multi-server mean for you?

First of all it means faster response times from your Hab.la widget, and a very simple process for scaling the server going forward.

Second, it means that we have outgrown our first machine, meaning that we have grown a lot in the past year. We processed over 18 million RPC requests in September on a single machine, let’s see what we can do with 2 :-).

Third, it means that we will be able to guarantee a SLA for our paid users, and that the overall quality of service will improve significantly.

It was somewhat rocky rolling multi-server out, a few of you may have noticed intermediate problems over the past 3 days. But you can rest assured that we are live, stable, and good to go :-).

With multi-server working, I can get back to some of the more fun features to implement.

Though, I promise to have a more in depth blog about some of the underlying tools I had to use to make everything work.

A short summary of software I use to scale the RPC side of Hab.la

haproxy lighttpd 1.4.20 memcached memqd (with some custom hacks) python

I know most of our users really like updates to the front end, to the Hab.la chat box, and methods of theming the window. However, everyone should be happy to know that since the memcacheD edition of Hab.la has been deployed we have sped up Hab.la’s average response time, while cutting our CPU usage in half. I.e. we are now handling more traffic than before with about 1/2 the load. Bring on the new users :-) – and we haven’t finished optimizing!

Front end improvements will resume later this week :-)

The new memcacheD based Hab.la RPC server is online, with all the kinks worked out. The kinks were very minor, but many of you might have noticed the hab.la users on your buddy lists flipping on and off a few times today as we solved a few of the last small bugs in the server code.

Please let us know if you notice any other issues.

haha, I wish I could tell you about the exact performance gain in the version of Hab.la that I just pushed to production. However, we don’t really have a great method of benchmarking code outside of production and so we won’t really know the TRUE speed ups until we have a few hours of statistics generated from actual use.

What I can tell you is that I have spent the last week or so refactoring the backend RPC server that handles the connection between Jabber and the JavaScript widget to make the code much cleaner, and the session management far better.

Here’s a brief summary comparing the new setup with the old setup.

Old method of session management: mysql Minimum of 2 SQL calls per RPC call. I.e. we load the session from mySQL perform updates on it, and save it back to SQL one time per hit. We were doing this over 200,000 times a day, generating approximately 400,000 SQL calls from session management alone (200,000 writes).

New method of session management: MemcacheD Unlike most high traffic sites that use MemcacheD we don’t have to perform a MySQL write every time a session is updated. Therefore memcacheD not only reduced mysql reads, it also reduces mysql writes. We now only write conversation buffer changes to Mysql. This means that instead of updating the mysql copy of the Hab.la session each time a visitor hits the RPC server, we only write it to SQL when an important event occurs (i.e. a message is sent), so if a server goes down we might lose our temporary record of what page your visitor is on – but won’t lose the conversation.

We get about 3,000 messages sent on Hab.la a day. So this reduces our number of mysql writes to about 3,000 + the number of visitors we handle a day (currently about 12,000 on average). So 200,000 mysql writes, becomes 15,000 mysql writes, and 200,000 mysql reads, becomes 0 session related mysql reads.

Of course there are other mysql related reads that occur during session setup, and before initiating a chat, but the vast majority of our reads and writes come from session handling, and the new memcacheD option pretty much eliminates the need to read from mysql to get session data.

In the coming weeks I will be phasing out even more of the mysql reads, to continue to make it easy for Hab.la to scale going forward.

As always, please let me know if you are experiencing any problems with the new RPC server, it passes my unit tests, and I haven’t seen anything in the error log, but that doesn’t mean there aren’t conditions we’ve overlooked.