I Debug Therefore I Exist

If there is ever a better example as to why Steve Balmer is bad for Microsoft:

It’s always dangerous when a company starts requiring their employees to become ‘yes men’ and employees start fearing their own jobs if they disagree or don’t show the same level of enthusiasm.

I know a lot of you are Google fans. I think they do a lot of neat things and are one of the more interesting companies, in terms of technology and they contribute a lot to the progress in the tech world. But ChromeOS really has me scratching my head. We all know that the browser is going to be the next OS, but this isn’t quite how I imagined it. Maybe it’s a little too soon say goodbye to the desktop. I honestly don’t see a future in ChromeOS and I’m confused about what direction Google is taking this. Maybe there is something I’m missing or that they haven’t revealed yet. The web really needs to undergo another transformation before something like this becomes a practical reality, in my opinion.

Well, that’s all I really had to say about it. Seems Brian and Chris from the Computer Action Show share very much the same sentiments.

As if the world needed a catastrophic event to show us of the pitfalls of cloud computing, Microsoft messed up royally by losing data (contacts, calendars, pictures, etc) for hundreds of thousands of Sidekick users and I’m surprised something of this magnitude didn’t happen sooner.

Ouch, though… ouch…

The effects of this will continue to ripple for a while and it’s a reminder about the importance of keeping your own backups, but, on the other hand, it does pose the question about how easily this can be done with everything in the cloud made more complicated by the fact that it is in the hands of many different companies & services.

Here’s one ‘feature’ that I bumped into and scratched my head over for a while, until it occurred to me what was happening. It appears that, if you are using replication, when you enable replication only on select databases (instead of all databases and excluding what you don’t want to replicate), you are running the risk that some of your queries will not replicate.

This happens when you have selected a database, which is not being replicated, (or you have no databases selected) and you perform a query specifying another database in the query, which does replicate. Normally, if you had selected the replicating database, then executing the query would allow it to replicate, but not if another is selected. This caught me by surprise, because you would expect the parser to be clever enough to know that you have specified a database in your query that does replicate, but it isn’t. I assume this is for performance reasons.

To give you an example in code, assume 2 databases: FRUITS, NUTS and VEGETABLES. NUTS and FRUITS are being replicated (ie. in the configuration: ‘replicate-do-db=FRUITS’), while VEGETABLES is not being replicated.

Secondly, assume we are going to run the query, which should replicate to our second database:

INSERT INTO FRUITS.list VALUES ( ‘apple’, ‘canada’, ‘red’ );

If we ran the query in this order, it would work fine:

USE FRUITS

INSERT INTO FRUITS.list VALUES ( ‘apple’, ‘canada’, ‘red’ );

However, if we ran the query as such, it would NOT replicate:

USE VEGETABLES

INSERT INTO FRUITS.list VALUES ( ‘apple’, ‘canada’, ‘red’ );

Likewise, because NUTS is being replicated, if you ran the following it would work again:

USE NUTS

INSERT INTO FRUITS.list VALUES ( ‘apple’, ‘canada’, ‘red’ );

So, so long as you have a selected database that is being replicated, then it replicates fine. But select a database which is being ignored or if you have no database selected at all, then it doesn’t.

Obviously, this is kind of annoying, because it’s something that seems so trivial, you would expect MySQL to be able to handle it, but apparently not.

So, the obvious solution is to specify in your database configuration that all databases replicate and exclude the ones you don’t want to replicate. Of course, bear in mind, the opposite is true. You can break replication if you select a replicating database and run a query specifying a non-replicating database, because it will still add that query to the replication binary log. Oh brother.

So the best thing to do, really, is to make sure that all of your replicating database servers are identical, turn on full replication and do not have any tables that do not replicate. I wish this sort of thing is more clearly documented, so you don’t run around with the belief that MySQL is doing something that would otherwise seem so trivial and logical to do and then suffer the consequences with a terrifying potential.

Just a quick post:

Here are some beautiful art cards. They are by Corinna Jasmine.

If you like her artwork, you will be able to find her work at her (soon to be opened) Etsy shop.

Just a quick post on an interesting image I stumbled upon today. It really brings into perspective what the slowest part of a computer system is and where your optimizations should be focused on if you want efficiency. Since it’s a large, but thin image, I couldn’t thumbnail it, so you will find it in the link below.

Image: Latency

I would highly recommend you check out the blog post referenced in the image: What Your Computer Does While You Wait

I really like this quote from the article:

Most simple instructions on the Core 2 take one clock cycle to execute, hence a third of a nanosecond at 3.0Ghz. For reference, light only travels ~4 inches (10 cm) in the time taken by a clock cycle.

Hypothetical:

So, your website has been gaining traffic and the code has gotten more and more complex and things are starting to slow down. You’ve exhausted the most obvious tweaks, like optimizing a bit of your code (there’s only so much this can do to help), your database queries (a usual source of bottlenecks) and deployed a distributed cache (ie Memcache, but there are only so many things that you can cache effectively) and it’s alleviated things for a little bit, but it’s not really fixing the problem. You are just a sudden surge in traffic away from a dead web or database server. It’s no fun.

Sometimes, it’s not even the load on your servers that is the main problem. Maybe you already implemented a load balancer that is helping soften the load to each server, but you just aren’t able to effectively cache your pages, images, css, or javascript files (ie. each web server outputting their own etag’s, continuously invalidating the client browsers cache) and you are just wasting bandwidth, causing sluggish response times.

If any of this is familiar, you should look into Varnish. While there are some things that can be solved using PHP’s APC (or whatever alternative there is for other web languages) or even with your own cooked up solution, sometimes you just need to look at your assets from a structural point of view. Inevitably, you will need to consider having a reverse proxy.

I can certainly appreciate a certain apprehension towards using reverse proxies, because you are handing a level of control away from your application (your web servers), which just makes things a bit more complication than it already is, but you need to consider the benefits.

In the case of Varnish, it’s really fast. Really, really fast… One of the neat things about Varnish is what it offers you in terms of scripting. You can really pack a nice and simple, but powerful configuration into one file. If you want load balancing, Varnish will let you define your back-end servers and, if you want to get fancy, you can set up multiple back-end groups, if you want to apply different rules depending on whatever factors your define – url, http host, client IP, request method, etc. This can provide with a powerful way of architecting your services, even if it’s something as simple as blacklisting offending IP’s and directing them to a secondary server, which is low priority and doesn’t matter if it gets foobar’ed.

In terms of caching, you have to consider the two aspects of it:

1. The most ideal situation for caching is having the user/client’s browser cache as much as possible and only expire when need be.
2. Varnish will cache content to memory, if you want it to (Varnish will be conservative by default, though).

What this means is that a page (or item) will start by being requested from your web server, but will be cached in Varnish as well. If that same client requests the same resource, it will be cached in their browser, resulting in the fastest response time for the user and the least amount of bandwidth used by your servers. If another client requests the same resource, they will need to download it, but it will come directly from Varnish’s cache and never hitting your busy web servers. The benefits should be immediately obvious.

You need to work a bit on defining what pages are cached and how. You’ll probably want to put in rules for caching of certain types of files, if you want to take advantage of Varnish’s own cache. It requires a bit of thought and a bit of experimentation, but, on the flip side, it’s not all that complicated, once you understand the basics and there are only so many things to remember. My own configuration is nothing more than 20 or so lines of code.

One other very neat thing about Varnish is that you are able to switch configurations on the fly, by telnet’ing into Varnish’s admin console. Just load in the new configuration and tell Varnish to deploy it. This protects you from having to invalidate Varnish’s cache, by restarting it, and makes the switch seamless to your users.

I will write a follow up post on what configuration I’ve been using, since I have a slightly more complicated setup – I’ve also taken in account a few additional things, like what happens under certain conditions where pages cannot be served or Varnish is restarted and your back-end servers are yet to be recognized as healthy.

PS. Be sure to use as recent a version of Varnish, however. If you run Debian servers, look to use the testing packages.

Since, at my company, we have built and are scaling our architecture, we often have to add and reformat servers. In addition, we use Debian pretty much exclusively. Generally, I am quite happy with the quality of Dell’s PowerEdge series. I won’t speak to their desktop workstations, but their servers are quite powerful, stable (haven’t had a failure as of yet, knock on wood), and come with a lot of nice extra features.

Debian is not an officially supported OS (and I don’t see why not). It’s either Windows, RedHat Linux or SUSE Linux. While RedHat Linux (including CentOS) and Novells SUSE Linux Enterprise Server are respectable server operating systems, we’ve settled on Debian because we’ve generally had a good experience with it and the community support for Debian far outweighs any other Linux distro that I’m aware of. Good community support means faster access to knowledge and fixes. That is just a fact of life and I see that, moving forward, this will make it harder for so-called enterprise products to compete with their limited resources. On the other hand, the downside to this is that you need an active IT staff, willing to commit time to managing quirks and solving problems on their own, without having any corporate support. This works well for us. We like more control.

Nevertheless, this post isn’t about the virtues of running one server OS over another. This is a post about a problem that has plagued us for a little while. Generally, Debian has an astounding support for all sorts of hardware, including server hardware, but occasionally there will be something that doesn’t quite work the way you want it to. In our case, it was using Debian (Etch & Lenny) on Dell PowerEdge 1950/2950 servers with the integrated PERC 5/i raid controller. The specific problem is that during the install process (using the netinstall cd, although I believe it should apply to all other install methods), the installer gets the SCSI drive assignment order all wrong, resulting in an un-bootable operating system once the install is complete and you reboot it.

If you are simply installing it without LVM, this should not cause too much of a headache. You just change the root drive in grub, boot it up, change /etc/fstab & /boot/grub/menu.1st and reboot. But the problem comes when you use LVM, and especially, as in our case, LVM + Luks encryption. Doing so will drop your system to BusyBox, with a system that would not boot and requires some massaging to get working. So, this post describes our solution, so that if anyone else hits this problem with their PERC 5/i raid controller, this should hopefully help them.

First things first, I will assume you have installed Debian Lenny at this point with LVM + encryption (in our case, the installer installed the boot partition into /dev/sdb1 and the encrypted volume group into /dev/sdb2) and you have rebooted only to be dumped to BusyBox with an error saying that such and such volume group could not be found. The next thing you need to do is unlock your encrypted drive.

I shall also assume that /dev/sda1 is your plain ext2 boot partition, while /dev/sda2 is your encrypted partition at this point.

The next thing you need to do is run:

# cryptsetup luksOpen /dev/sda2 sda2_crypt

This will ask you for your encrypted password and will then unlock the partition.

The next thing you need to do is initialize the LVM volumes:

# lvm

lvm> vgchange -a y <VOLUME-GROUP>

<VOLUME-GROUP> is the name of the volume group on your system. If you are unsure what it is, run “vgs” and it should list it. Usually it is something akin to your hostname, so if you hostname is “SRV1″, it would likely also be “SRV1″.

Now, all you have to do is exit the lvm console with the command “exit“.

Everything should be set up correctly now to continue your boot from BusyBox, so type “exit” again. Your system should resume booting. Don’t worry about any errors you get. You might get a warning about not being about to mount your boot partition and to enter your password to enter maintenance mode. This shouldn’t be necessary, though.  Just press CONTROL+D on your keyboard to continue.

Finally, you should hit a normal login prompt. Login as your root user. Now we will be editing a few files to adjust everything.

First, edit “/etc/fstab“:

Change

/dev/sdb1       /boot           ext2    defaults        0       2

to

/dev/sda1       /boot           ext2    defaults        0       2

Next up, you need to edit “/etc/crypttab“:

Change

sdb2_crypt /dev/sdb2 none luks

to

sda2_crypt /dev/sda2 none luks

Finally, we need to edit “/boot/grub/device.map“, but the /boot partition isn’t mounted yet, so type:

# mount -t ext2 /dev/sda1 /boot

Now edit the file “/boot/grub/device.map“:

Change

(hd0)    /dev/sdb

to

(hd0)    /dev/sda

Finally, that’s it for editing things. You just need to run a final command to update your boot image. Run this:

# update-initramfs -u

That’s it, really. Now you can reboot your system and see how it should finally pick everything up automagically, prompt you for your encryption password and boot into Debian without any headaches.

Enjoy!

If anyone wants a more in-depth explanation as to what is going on and why, leave me a comment and I’ll be happy to fill you in on the details.

Please note, you might have to repeat this process again if APT ever upgrades your kernel, so be aware of this.