So, I was reading the memcached listserv digests today and realized, hey, I actually knew one of the main contributors. In fact we even hosted a server for him many years ago.

So my friend works at six apart, meaning, he knows a lot about memcache, and scaling using perl, and he told me pretty much flat out, that no one should be using libraries linked against memcache anymore.

He pointed me to http://code.google.com/p/python-libmemcached/

A python wrapper for memcached using libmemcached, a new and improved client library. Apparently there is going to be an effort to make sure module authors are linking their coded against this new library.

In any case, I’ll run some tests tomorrow, and most likely spend the 5 minutes porting from cmemcache to a library that uses the latest C client bindings. As for my python_interface work yesterday, I think we can keep the memcached interface in python for now.

Alright, a while ago kevin write an erlang wrapper for libmemcache. As anyone who has used libmemcache is probably aware of, libmemcache does not fail gracefully. In fact it fails in a way that takes everything down in one blow.

Of course, it also rarely fails, unless the CPU load is very high, or there is a problem with your memcacheD servers.

In any case if the memcache servers failed, or timed out the Jabber server was breaking. This is similar to our problems with cmemcache on the RPC server side, where on occasion under high loads libmemcache will timeout and fail hard, taking the entire RPC server with it.

To mitigate these problems for the time being I replaced the libmemcache wrappers with python interface, http://www.lysator.liu.se/~tab/erlang/py_interface/. Currently python interface is just being used to interface between erlang and the memcache servers, to assure that memcacheD and high loads won’t take the jabber server down.

In the long run I want to look into http://code.google.com/p/cacherl/ and at least benchmark it against memcacheD.

Work It, Make It, Do It, Makes Us

Admittedly it’s been a long time since my last post, but a lot has been happening in the background. We’ve recruited another team member, Eric, whom we will introduce on the blog soon. Roland and George have been plowing along on HablaPlus – and I just pushed the stable multi-server edition of Hab.la.

What does multi-server mean for you?

First of all it means faster response times from your Hab.la widget, and a very simple process for scaling the server going forward.

Second, it means that we have outgrown our first machine, meaning that we have grown a lot in the past year. We processed over 18 million RPC requests in September on a single machine, let’s see what we can do with 2 :-).

Third, it means that we will be able to guarantee a SLA for our paid users, and that the overall quality of service will improve significantly.

It was somewhat rocky rolling multi-server out, a few of you may have noticed intermediate problems over the past 3 days. But you can rest assured that we are live, stable, and good to go :-).

With multi-server working, I can get back to some of the more fun features to implement.

Though, I promise to have a more in depth blog about some of the underlying tools I had to use to make everything work.

A short summary of software I use to scale the RPC side of Hab.la

haproxy lighttpd 1.4.20 memcached memqd (with some custom hacks) python

If I was really slick, I would have written a log file handler that was smart enough to rotate logs when the computer was running out of space. But I think I’ll let logrotate take care of that.

I have posted our new log file handler, which can recover if files disappear during operation (i.e. are rotated or frantically deleted) and will automatically rotate logs daily.

http://pastie.org/290949

Now I can safely let logrotate loose on our Hab.la logs.

Some of our users are in environments where they can’t embed the hab.la javascript directly onto their websites. As a workaround we developed a method of showing an image that displays hab.la status on their websites.

A few days ago, I noticed that the status display was a little slow for my liking. In fact, it was running as a normal CGI ( a big no no), but since not to many people were using it, it’s performance wasn’t that big of a deal.

Today, I increased performance by 1800%, speeding up the status image response time from ~0.474 seconds to ~0.026 seconds.

ben@lyle:/var/www/online# time curl "http://dyn.hab.la/online/image2.cgi/4324-24618-10-8812/image.png"

real    0m0.474s
user    0m0.016s
sys     0m0.004s

Versus:

ben@lyle:/var/www/online# time curl "http://hab.la/status/4324-24618-10-8812/image.png"

real    0m0.026s
user    0m0.012s
sys     0m0.004s

What sped it up so much? First, I replaced the CGI script, with a very lightweight webserver. This made it so that python didn’t have to instantiate from scratch every time image.cgi was loaded. Second, I replaced the old calls to our RPC server, with newer calls to our memcacheD servers. There are still too many MySQL calls in this process, but optimizing those into pure memcache can wait for a while.

To learn more about using Hab.la without Javascript, take a look at My blog post or our tutorial for websites without Javascript.

Another boring update for you hab.la fans. We have made some backend tweaks to the RPC server which should slightly speed up Hab.la (for end users) and reduce the amount of memory and CPU used for each site (which will help us better scale hab.la as traffic increases).

We’ve also fixed a few small bugs with operator availability that should speed up the load time for the initial hab.la window.

Hope everyone is having a great New Year

-Ben

So we fixed a few bugs, we’ve improved how messages are sent when you have more than one hab.la message window open in Firefox, increased the number of concurrent hab.la chats, and added hooks for some new features that are in the works.

If there is something in particular you’d like us to focus on, please post it in the blog.

So I spent way to much time trying to figure out exactly how Adium and Meebo handle updating nicknames in a buddy list (roster). The answer is, while they are both based on libpurple/gaim they both pretty much ignore setting the nickname in presence updates and messages. Let’s just say I spent a lot of time figuring out exactly how adium deals with this: i.e. doing traces through libpurple’s source code: http://www.google.com/codesearch?hl=en&q=+libpurple..,

I tried to find a socks proxy where I could rewrite packets and watch jabber messages, but I couldn’t find anything that quite fit my needs, so I ended up modifying something called PySocks (which hasn’t been updated in a few years, and required a bit of hacking to evne run, but works quite well as a socks proxy for rewriting packets and monitoring jabber to figure out what’s going on. [sourceforge.net/projects/pysocks/]

So google codesearch, grep, and pysocks was the method, but what was the result? And why does this matter to you?

The idea is to have your hab.la buddy list become more personal, so instead of having visitors appear as “webuser1” and “webuser2”, they’ll appear as “John the Spammer”, and “Fred the friendly reader”. I.e. your buddy list will not only tell you you have 2 people browsing your site, but also who those people are [if they’ve chatted with you using hab.la in the past]

Hab.la building relationships – sounds like a good motto to me :-)