July 1, 2004

Mono + 1.0 = 85kH/hr = (unh!)

As Miguel noted, the machine we had allocated to handle the Mono team's release and news/blogging needs just couldn't take the strain. 85k hits in the first hour and the server made with the faceplant; although, perhaps most frustrating, it didn't *totally* faceplant. It dove into swap like a coked-up porn star at a couples party, but never...quite...went all the way down (the prude).

Consequently, I have spent much time not only on the phone with our colocation provider's NOC ("Well...yeh...you better reboot it...wait...hold on, it just came up with a prompt...but...no...I can't login...better kick it...") but logged into the machine itself via both network and indirect serial console connections (go Cyclades!). It's running Linux (duh) with Apache 2, and the real limiting factor at this point (after Miguel's team and I have made several hours of uncoordinated and hence likely contradictory tweaks) is memory. Load is now running around a constant 35; at peak, I witnessed it hit 576 before the machine quit responding. At this load, the box is maybe 50MB into swap and is handling a max of 100 simultaneous Apache connections. This sounds low, I know, but the machine is also running a Whole Bunch O' Mono worker threads (at least twenty) to handle the various Mono back-end stuff. These, too, suck up RAM. The machine is at least reliably responsive, if slow, at these loads.

I'm not really sure how this could be improved; I just don't know enough about the guts not only of the Mono module/runtime but of the site they have on there to determine if the content could be easily clustered across multiple boxen, either using dumbshit-but-quick methods like DNS round-robin or more complex-but-satisfyingly-crunchy methods like Linux Virtual Server. Perhaps a mid-range solution whereby the Mono processes are banished to their own machine, communicating with the web server threads over the net? I don't know if that comm link is a bottleneck. Now that Mono is at 1.0, though, I think I better start learning the answers to some of these.

Congratulations to Miguel and all the other Mono Monkeys (yes, I know that's redundant, deal with it). Posted by jbz at July 1, 2004 4:41 AM | TrackBack

Post a comment

Remember personal info?