Sharp Tools - Monkeys

Let Them Eat Spam

We recently performed some badly-needed upgrades to our mail-handling infrastructure at work. This gave me a chance to, amongst the swearing and threats to rewire various servers into toasters unless they started doing their jobs, twiddle some spam-handling and have a look at the results. One of the things I'd been curious about was the degree to which our use of blackhole lists affected the amount of spam hitting us, given the recent brouhaha over the Spamhaus lawsuit and various claims from various folks about their effectiveness.

The following graphs are 12-hr snapshots of Upgrade Day. Note that this system is not our primary MTA, but is one of our front-end mail scanners; in other words, it is the first system to see the mail. Furthermore, this system is purely a mail receiver; the 'Sent' messages are those which are accepted into the company and 'sent on' to internal-facing mail handling. This is why Received and Send track so very well.

The time of change is fairly easy to spot.

There were two changes made to mail handling on this host at the same time, which makes this graph not nearly as clean or useful an indicator. We turned on the use of rbl-xbl.spamhaus.com in the SMTP handshake phase, and we implemented postgrey greylisting at the same time. We had always been doing some fairly aggressive SMTP HELO/EHLO checking. Here's a graph showing what happened to the 'bad' mail over the same time period. Note the scale difference (msgs/min).

Note that 'rejected' takes a sharp, sharp upturn, and bounces (which usually indicate the mail got through the format checking but were addressed to illicit addresses) drop way off. In other words, we start catching that stuff prior to queueing, which is what we wanted - having to do postqueue checks is more expensive by far (I'm assuming, at the moment, that REJECT indicators in mailgraph are prequeue given that that's what our log entries are for - NOQUEUE: reject - and that bounces indicate it was in fact queued before being checked and returned, but I may be wrong). The one thing I need to check is whether the initial greylisting straightarm is caught by these graphs as a REJECT, or if the Defer is handled differently.

So these graphs aren't worth much from a 'make your point scientificially' standpoint. They are enough to make me shake my head at the spamflow in.

Oh, yeah - there's no virus scan done on this particular machine. Also, it's not the only front-end mailscanner/MX, so while any initial dip in traffic could simply be due to the half-hour interrupt causing inbound servers to cycle elsewhere in the MX tree, that should have evened out over time considering that all of them got the same changes at roughly the same time.

Congratulations without measure

To Mike Wolf and his lovely bride. Mike (or maw, as we know him) relocated to Mexico City to broaden his horizons while still working for Ximian/Novell - and boy, did that ever work out. Huzzah.

Shoutback to the French Guy

Do I get it?

"...the man they call Jaaaaaaaaaaayne!"

Heh.

Shiny.

Okay, and I retract my earlier comment about the robot thing. That's a worthy use for it. Grappling hook, boy, grappling hook, how else you gonna capsize those damn sailboats? Heeheehee. Or maybe torpedoes. I could make you some spiffing damn powered torpedoes, you damn betcha.

Baaa, children!

Like all those other monkeys (who unlike me actually do useful stuff around here) I have followed the herd and used the tool. Behold my genius 2D self.

Bottom feeders and chum

Robert Love has ascended in my personal pantheon.

Why? For this, followed by this. Also, of course, this.

Salt and Asbestos

So this other monkey brought in grasshoppers. In a baggie. For snacks. They appear to be smoked. He claimed they were like 'salty raisins.' After trying one, I can state that they are, in fact salty - but are like raisins only in the texture, in that 'there are now strange awful-grtty-between-the-teeth-objects-in-my-mouth' sort of way. Plus, they taste like overworked (read: burnt) brake pads tend to smell.

No sir, Mr. Fox sir, I do not like it. Not at all, sir.

Junkies? JUNKIES!??! Pah.

My boss has entertaining notions involving toys. His latest is worthy, but I have to take exception to his terminology:

With my cameras, binoculars and telescopes I can get a pretty good visual lock on the junkies. But my idea is to get one of those parabolic microphones, so that I can listen to what they're saying.

And then, when they're good and stoned, pipe their own words back to them over loudspeakers. On a sixty second delay. :-)

Um, Nat, 'junkies' take heroin. People who smoke dope are, er, maybe potheads. Max. Or something. :-) :-) :-)

Of such things are projects, messes and demonic unholy creations made.

Hm.

In one hand, I have a Braun Espresso/Cappucino/coffee maker.

In the other, I have a Sony VAIO laptop that hasn't had working charge circuitry for years but otherwise seems OK.

At home, I have a Dremel MotoTool.

Here one can find an RFC for HTCPCP, the HyperText CoffeePot Control Protocol for internet control and monitoring of coffeepots.

Hm.

Old skool, newer me?

Dammit, I took that test a year ago. I just took it again, and now:

I am a Light Cycle. I drive fast, I turn fast, I do everything fast. I even breakfast. I tend to confuse people with my sudden changes of heart. Sometimes I even confuse myself, which tends to cause problems. What Video Game Character Are You?

Must be the damn coffee. Really. That or the amphetamines. Anyway, I don't have a Tron machine. May have to look into that. I wonder if there's a way to be classified as a Tempest spinner. Hm...

Old Skool, me-style

This has cropped up on a buncha monkeyblogs. I've had mine on my monkeypage for a while now:

I am a Thrust-ship.
I am small and tricky - where you think I am, I probably am not. I can work very fast, but I tend to go about things in a round about way, which often leaves me effectively standing still. I hate rocks. Bloody rocks. What Video Game Character Are You?

This result is really interesting, given what I keep on the wall of my office:

Sources of the Red Red Haze

It's not really that I hate my job. This is not the case. I actually like my job quite a bit. This has led several of my friends and family to ask me why, if this is so, have I recently been driven to damage my manual infrastructure against the office's vertical superstructure?

This is a fair question. It deserves an answer that contains a bit more thought than my typical content of flip sarcasm and self-satisfying in-jokes. So here goes.

I came to work at Ximian for several reasons. It wasn't because I needed a job. At the time, I was, in fact employed, and I took a pay cut to make to move. While the dot.com I was at wasn't on the most stable of ground, neither (at the time) was Ximian looking much more so; a few more months, perhaps, barring something breakout happening. I came to Ximian because someone I respected asked me to, because the company was making things I used myself, and because (after coming over and chatting with a few of the monkeys) it appeared that it was populated by people who Gave a Shit.

Many dot.coms Gave a Shit. These guys, however, Gave a Shit about something other than (well, besides, and before) getting fuck-you money. The technical people had almost all already been doing what they were doing for love not money, and Ximian was supporting them as they did it. Some were hired on and Got It. Some others didn't get it but worked hard. A few (but fewer than in most firms) didn't get it, and had to be worked around. The CEO of the company, though, ran Linux on his laptop and bitched about the fact that his kids ran Windows at home.

These guys Got It.

To a sysadmin who had been running from Windows for as long as he could, it was a fricking godsend. The environment alone would have been that - but the entire purpose of this little place was to bring this to everyone trapped inside corporations, as I had been in the past. To bring what we had, there, to those suffering in cube farms everywhere. Hell, we used it every day, why couldn't everyone?

There were (and still are) reasons why some people can't, people with situations ranging from the complex and custom to the very mundane. We try to fix the products, to get rid of those reasons. That's what we do. (Note: I say "we" in a very self-aggrandizing manner, here. I don't write code. I don't do QA. I'm just an Op.) Still, we get to work on problems here (even as Ops) that you don't see every day. We get to try to solve problems that may not have come up before, using tools that are so new the developers haven't finished them. We're figuring out how to deploy stuff that doesn't even work fully yet, making it do stuff it was never designed to do because we have it sitting there to play with and someone else has something they need done. We can walk out the door, because Ximian was small enough and our office remains tight enough even as part of Novell, and grab a developer and say Yo, homeslice, that shizzle no worky. Make fixy or I cut off yer pr0n feed. And they will; not because they believe our crude threats (usually) but because they, too, want the damn thing to work and be worth something. When us Ops come to them and say it doesn't work, they know it's because we're trying to use it, and that counts.

Anyway.

These guys are fairly young (younger than I, at least) and they work hard. They're here late. I'm here late, sometimes, but I'm older now and my job doesn't usually generate the kind of deadlines or late night inspiration chasing theirs does. I try to be here when they need me here to make things work, or when things break of course, but still. While sometimes there's creative slacking, on the job and off, an awful damn lot of oil gets burned around here. This is mostly because they care about this stuff, and did before Novell, and did before Ximian. Novell bought Ximian, and Ximian came to be (at least in part) to harness that culture and energy.

Back to me.

The problem is that in every company, there seem to be a certain percentage of people whose skill set seems to consist mostly of parasitic bureaucratic manipulation. I have come to the conclusion that this is an unavoidable characteristic of any organization based on SOPs, a la James Q. Wilson's theories; but that doesn't make it any less annoying. These people exist solely to manipulate the organization to provide for their own job and security. They survive because it would cost the organization more to get rid of them than it does to simply tolerate them. Classic parasite behavior. They are usually spread out thinly enough that at no point in the organization is it worth rooting them out; if they clump too thickly, at some point it is cost-effective to simply burn down a big chunk of the org and start over (or, more efficiently, tie it off and let it wither).

Which brings me to Novell. Something very interesting is happening at Novell at the moment. A middlin' sized tech company is trying hard to reinvent itself around an entirely new (to it) concept. Not the Internet - it's fairly clear that Novell missed that commuter ferry entirely during the 1990s while getting pissed in the pub on Netware dividends. No, around Linux (which, if you read any form of trade rags, you already know). This is a fascinating process to watch, especially frm the inside, as it involves something new in my experience - a change process mandated from the top but pushed from both the top and the bottom via the acquisition of Ximian and SuSE, and the evangelism of members of those organizations and 'converted' technical personnel at the grunt technical levels and up the engineering tree, which at Novell should be 'those who matter' for the Company's future direction (it being a software company, after all). I'm not going to go into how well that evangelism is going - that's for analysts. We're still here, though, and haven't been standing around...and reading my colleague's blogs will tell you that Novell, true to its word, has in fact been supporting their Open Source efforts. No Ximian code that was Open Source when we were acquired has been closed (afaik), and some product that was proprietary has in fact been released to the community (Ximian Connector for Exchange, e.g.).

However, the parasites are still around. And in a lots of cases, they've managed to hole up in the non-PBU departments. Or perhaps just survive longer there. Why? I don't know; maybe when your department budget isn't based on revenue, it's easier to stay hidebound. Maybe overhead isn't viewed as critical to this new reorg, being viewed as one of those 'old fashioned' attempts at cost-cutting. All I know is this: I get work done when I don't talk to my department, which is one of those overhead departments. I get work done, and things completed, and people helped, when I respond to the needs and requests of the people I've always worked with here in my office.

As soon as I try to interact with Novell, the corporate structure, from my 'slot' within it, everything goes to hell in a God-damned handbasket. Resources? Well, sure...as long as I can pimp the budget from other people in the office. Servers? Same. Infrastructure? No, then it's gotta come from another person who also lives in Provo, which isn't itself a problem, save for the fact that our data center explicitly wasn't put in his cost center, so we're not in his planning cycle, so how? Not sure. Software licenses? Nope. We lost those. Wait three months. Mail client development stalled? Well, maybe two months. We'll call you.

Hardware? What kind? While being visited, a gent from the home office commented snarkily on the fact that my co-worker and I got Macintosh Powerbooks. No, I replied calmly, we don't.

What're those? He asked, pointing at the 12" and 15" Powerbooks in front of us.

Our day to day machines that we bought with our own money, we told him.

What kind of laptop did Novell buy you? he asked.

We had to laugh at him. Laptop? Not likely. My primary workstation was a Dell P3/500 that Ximian had owned when I was hired. I didn't (and don't) have a Novell laptop.

Let me stop and make something perfectly clear. I do not believe it is my right or privilege to have cool fast hardware on my desk. In fact, I have taken a perverse pride since coming to Ximian in being able to do my job on the hardware I have. I have also felt a quiet happiness at finding a job where I cared about the dev team to the point where their hardware was much more important to me than mine. However, I do feel very protective of the people I support, and the fact that (for example) the crappy NFS server appliance we had when we were acquired, and that was #1 on our list to be replaced, still hasn't been despite piecemeal buys of over $15K in hardware due to critical PO reqs sitting ignored in the requisitioning system - or lost, for all I know, either would have the same effect - these start to seriously tick me off.

Being told to seriously consider a $109 cheaper monitor for my colleague despite the fact that the one he was trying to purchase is the one every developer has on their workstation (and, hence, we have to support) while being told there is never any extra in the budget for computers for us - this starts to severely tick me off.

Reading on Provo department personnel blogs about wonderful fun family-included outings to AAA baseball games followed by business-day golf outings with the CEO that same week which we (politely) weren't informed of, because we are in a remote location and can't participate in them, while having to cope with the above-mentioned hardware and software license subscription shortages - well, one begins to feel a tad superfluous in one's assigned slot in the organization.

When the only contact one has with the head office is a completely fucking useless boondoggle of a training session which wastes three days of our time, three days which in fact we had been asked to contribute to an important product which others in our office had been busting their collective asses on for months, this is a problem. When said training involves training people who in no way do what we do in a task which we in no way do for a living nor have we ever really done for Ximian or Novell, well, then, it begins to appear that not only are we superfluous but completely fucking misfiled under a completely incorrect fucking heading in a completely fucking wrong department with many of those People of Special Bureaucratic Skills I mentioned earlier.

Add on top of this the loudly voiced opinion from more than one of those being trained that they cannot understand why we appear so frustrated, because, after all, this is just a job and they're just here for the paycheck and isn't that why we're here, and we're almost there.

Spice the top of the third day of this environment with what no doubt seemed like good natured joshing about my apparent naivete at believing the Democrats could do a better job, from people who by their own admission have no idea who Karl Rove even is, and you may, perhaps, begin to understand why that poor whiteboard had to die.

And this is how you end up with a broken right hand.

Hm. I wonder if this violates the ACLB.

Suicide Shopping

It was a dirty job, but someone had to do it. The cliche held just so, so true. A colleague who had been sturdily resisting the addiction had caved a while back and agreed to shop, and today was the day we all had time. Another coworker from Provo who was in town and had succumbed on his last trip acknowledged that he might need, you know, maybe one more. So off we went, the four of us; myself, the two newly afflicted, and another who I have pulled deeply down into the sickness.

It was a suicide mission. I had to go, to shepherd the newbies through the process, provide some gentle support, ensure they didn't overspend too egregiously. That, however, meant going into the damned place myself.

Sure enough, I'm $286 poorer.

On the other hand, I now have a second dedicated piston-fill pen, this one a daily driver and oh so light and quick. I broke down, joined the other stylonauts with me and bought a one-hander daily use pen - ah, modern technology. Round that off with a new red/black ink and a Moleskine notebook...joy.

Peter bought two vintage pens, a Sheaffer Vacu-fill and an older Sheaffer replaceable nib. We are losers. We revel in it.

On the other hand, my handwriting continues to improve.

The door really never did anything to me.

Was treated at work today to a breathtaking waste of time - a two-and-a-half hour presentation on 'Culture Change at Novell.' Let me start right off by saying that I don't consider the concept or idea of culture change at Novell a waste of time; however, I have never in my life been to a meeting at a large corporation dominated by a PowerPoint slide deck which has ever improved my life. Ever. Especially those which purport to talk so earnestly about improving my work environment.

To be sure, this isn't a fair comparison. I am a lucky, lucky bastard to work where I do, and I know it. This may also be one of the reasons that the presentation in question created such an unpleasant resonance with me. I do know that when it was finally admitted that the slide show in question had been done by a consultant, I refused to be surprised, because I could have told you that hours before. I was put through a hard school of presentations at a company which gives them for a living to an audience which is by some lights the premier consumer of PowerPoint excrement on the planet. Whoever did that slide deck would not have survived to collect their first paycheck.

These are perhaps harsh words. Harsher than deserved. However, I cannot help but wonder what this outside person was paid to come up with this stuff. Some of it, to be sure, was OK; some was common sense (well, an awful lot of it was common sense) and (here's the rub) an awful lot of it was so trite that it seriously insulted my intelligence to be sat down and told it by The Man.

I'm not going to go into specifics. I'm not sure what if any of it can be publicly disseminated. Suffice to say that I came out of one particular section of it feeling like I'd been told to bend over that nice handy table and take it like a man; lubrication was for sissies. Besides, I was expected to make myself nice and pretty for the anal violation that was expected; it wasn't the Company's job to do that for me.

I do realize that probably isn't what was meant. But it sure is what that slide deck communicated. So if anyone up there cares, well, there's a problem, right there.

Mono + 1.0 = 85kH/hr = (unh!)

As Miguel noted, the machine we had allocated to handle the Mono team's release and news/blogging needs just couldn't take the strain. 85k hits in the first hour and the server made with the faceplant; although, perhaps most frustrating, it didn't *totally* faceplant. It dove into swap like a coked-up porn star at a couples party, but never...quite...went all the way down (the prude).

Consequently, I have spent much time not only on the phone with our colocation provider's NOC ("Well...yeh...you better reboot it...wait...hold on, it just came up with a prompt...but...no...I can't login...better kick it...") but logged into the machine itself via both network and indirect serial console connections (go Cyclades!). It's running Linux (duh) with Apache 2, and the real limiting factor at this point (after Miguel's team and I have made several hours of uncoordinated and hence likely contradictory tweaks) is memory. Load is now running around a constant 35; at peak, I witnessed it hit 576 before the machine quit responding. At this load, the box is maybe 50MB into swap and is handling a max of 100 simultaneous Apache connections. This sounds low, I know, but the machine is also running a Whole Bunch O' Mono worker threads (at least twenty) to handle the various Mono back-end stuff. These, too, suck up RAM. The machine is at least reliably responsive, if slow, at these loads.

I'm not really sure how this could be improved; I just don't know enough about the guts not only of the Mono module/runtime but of the site they have on there to determine if the content could be easily clustered across multiple boxen, either using dumbshit-but-quick methods like DNS round-robin or more complex-but-satisfyingly-crunchy methods like Linux Virtual Server. Perhaps a mid-range solution whereby the Mono processes are banished to their own machine, communicating with the web server threads over the net? I don't know if that comm link is a bottleneck. Now that Mono is at 1.0, though, I think I better start learning the answers to some of these.

Congratulations to Miguel and all the other Mono Monkeys (yes, I know that's redundant, deal with it).

"Gimme two faucets, a bench, three grilles, a two-by-eight and one of those freaky slope things."

Yes, I acknowledge, I am a nerd. Hence the dialogue of my Sunday.

"Lemme have two grilles, two benches, a right wing and two one-by-twelve technics, dark grey."

"Um, we need four six-by-sixteens here, not three."

"Does this magnet on, or does it wedge?"

"No, Chris has the faucets and the blue half pins..."

...and so it goes. That is a couple sound bites of a twelve-hour day spent team-assembling the Grand Poo-Bah of Lego - the Everest of Bricks - The Imperial Star Destroyer.

It was actually pretty damn challenging (3,104 pieces!) and not so simple. Imagine a twenty-page section of the instructions involving a few hundred pieces that, at the end, says 'okay, now go back to the beginning and do it again.' That happens frequently. Building it was bad enough; I can't imagine designing this thing. It is composed entirely of angles that are completely unnatural in the Legoverse, requiring all manner of massively cool hacks to work.

The end result, however, makes it alllll worthwhile.

On the Move

We're almost there - the Monkeys are moving offices. This, of course, means that yors utrly is on the hotseat - and at least partially due to reasons of my own making.

See, my forte isn't planning. It isn't meticulous prechecking of zillions of conditions. It's not the joy that (I'm told) can come from an exacting schedule and/or checklist of tasks and sub-tasks and sub-sub-tasks and so on that can make a complex operation such as moving a tech firm a synchronized waltz of precision and efficiency. Nope nope.

I'm a seat-of-the-pants Op. This means I excel at pulling chestnuts from the fire of chaos. I'm good at triage; I'm pretty good at finding a workable answer quickly, if not the best answer. I'm pretty good at intuiting now what my answer to a question I haven't thought about yet should be so as to avoid hosing myself later.

This means, natch, that moves are usually several days of sheer pain and suffering, coupled with adrenalin highs, concomitant lows, bouts of depression, rage, glee, loopiness, drunken flights of fancy and strained back muscles.

So, to any of you monkeys that may wander in here before the move: beware. Your Op is in BOFH mode and will not hesitate to LART if you get in the way. Oh, and the answer to 'When will (x) work' is always "WHEN I DAMN WELL GET TO IT." But, likely, before the 29th.

Place the brownies on the floor, sir.

<keelyn> it's like a mastercard commercial:
<keelyn> 4 cops, 3 firefighters, 1 pan of brownies... priceless!

Well, that needs some explanation.

If the links on this site haven't already told you, I'm a confirmed Everything2 junkie. I spend a fair amount of time on that site. Recently, we had a bake sale (more specifically, the Ninjagirls had a bake sale) to benefit the site, and all of us who donated were sent baked goods from the Ninjagirls' kitchens. Mmm. So I had donated, and was therefore sent a batch of Chocolate Death Brownies from a compadre.

The problem is that when the brownies showed up at my office, they were addressed to (naturally) The Custodian, in the Operations department. Our office manager / den mother, Keelyn (who is always looking out for us) wasn't sure what to do with a package for someone fictional, and called the Post Office. When the Post Office found that there was food in it, they advised her to call 911.

So she did.

I got wind of this because folks on IRC at the office were saying that a strange package had arrived for The Custodian. I jumped up and headed for the front desk, but by that point, the cops had told Keelyn that they were on the way and not to touch the package.

So we all waited. One cop showed up. Then three firemen. Then another couple of cops (the firemen had been lost). After clanking their HAZMAT air tanks to the ground, the firemen examined the packaged and proclaimed that from the texture there was an awful lot of butter in them brownies.

Meanwhile, I'm hoping that there aren't any, um, additions in these brownies - not that I have any reason to expect there to be, but at this point...

So eventually, they all left, and I got to eat brownies. Keelyn declined, as she's dieting.

Oh, the cops did come back and ask Keelyn if they should bring ice cream next time.

Requescat i pacem

A co-worker I knew, albeit not that well, passed away this past weekend. He died doing what he loved more than most things in this world. His fellow hobbyists have posted a picture of him in his element as a form of memorial. More images of them cavorting are available.

I'm not sure how I feel about it; I didn't know Chema all that well, as I said. Worse, I was used to seeing him for a few days at a time in between long, random-length absences as he traveled (he had just moved to Boston, but still considered Mexico 'home' and was in Europe often for work). As a result, it sort of just feels like he's still 'away.' We're going to have a bit of a wake/gathering this week to try to create some closure. I think we'll go for a party atmosphere, and find some of his skydiving videos to play.