A Fix :-)
Alright, a while ago kevin write an erlang wrapper for libmemcache. As anyone who has used libmemcache is probably aware of, libmemcache does not fail gracefully. In fact it fails in a way that takes everything down in one blow.
Of course, it also rarely fails, unless the CPU load is very high, or there is a problem with your memcacheD servers.
In any case if the memcache servers failed, or timed out the Jabber server was breaking. This is similar to our problems with cmemcache on the RPC server side, where on occasion under high loads libmemcache will timeout and fail hard, taking the entire RPC server with it.
To mitigate these problems for the time being I replaced the libmemcache wrappers with python interface, http://www.lysator.liu.se/~tab/erlang/py_interface/. Currently python interface is just being used to interface between erlang and the memcache servers, to assure that memcacheD and high loads won’t take the jabber server down.
In the long run I want to look into http://code.google.com/p/cacherl/ and at least benchmark it against memcacheD.
Comments [0]