December 31, 2004

Ops, Time Management, and the proper use of laze

This is actually about Operations, not Code. I apologize for offkilter categories. I'm trying to resist the temptation to balloon the number of categories out to fit what I feel they should be, as I think that would approach roughly the number of posts in the blog.

Hi ho.

Anyhow, on the meat of it.

I was recently involved in a bit of a disagreement with a cow orker ( moo) about the proper way to go about 'fixing a problem' at work. I am an Op. What does that mean? Well, that's part of the problem. Since this is my blog, I get to pontificate for a moment. When I say I'm an Op, I mean I am responsible for Operations, in the computer and information technology sense, for my firm. I do not mean simply that "I work for IT." That's not specific enough. I do a strange mix of jobs that doesn't fit well in a large corp, and fits much more neatly at a small dotcom - I do whatever is required to keep things running at the corporate level if it involves computers and infrastructure. While this can include fixing the CEO's secretary's laptop, you'll have to convince me that the CEO's secretary's laptop is damn well mission critical, unless I signed on to do end-user support when you hired me...and believe me, you're paying me more if I did.

The problem with being an Op is that it is extremely difficult to explain to those who are not Ops what I do with my time, and how I prioritize it and parcel it out. Please note: this is not an attempt to pretend that I am not, at some level, lazy. Far from it. In fact, I am strongly of the belief that at the core of every truly inspired Op is a tightly held and well-managed streak of pure slothfulness, which when properly channeled can produce genius of time-fu. I shall explain. And no, I don't have it, really; I can only approximate it.

I spend my work time performing three types of task - or rather, in three states of work. I shall call them

  • Routinized
  • Structured
  • Chaotic
Routinized tasks are pre-planned, scheduled tasks. Maintenance, in other words; expected, allocated time, used to perform regular tasks in a (usually) repetitive cycle. Backups, security audits, user account maintenance, inventory checks, system management e.g. OS upgrade installations (as opposed to testing), etc. etc. Anything and everything that you know needs to be done in advance, you've scheduled time for, and you can check a box off your task list for the day/week/month/whatever when you're done. Whack-a-mole type stuff.

These tasks are usually boring, but important. Stuff that cannot get skipped, or things stop working. It is the goal of all good Ops to attempt, on a continuous basis, to minimize the amount of time they spend on these tasks - through automation, through process management, and through careful sanity checking on their task lists. The reasons are manifold, but here are some of the most important. First of all, the more of these tasks you have to do, the more time each routine 'cycle' takes - and the less time you have for other types of tasks, as I'll outline below. Second, the more tasks there are, and the more complex they are for humans to carry out (i.e. the more manual they are) the easier it is for mistakes to be made, steps to be forgotten, or subtasks to be missed - resulting in bad state which then causes Problems. Problems are Not Good, and result in Chaotic Tasks. Third, we're Ops, and we're lazy - we don't like working on boring things. It's not fun and makes us irritable, and you won't like us when we're irritable.

With me so far? Okay, good. This brings us to our next type of task - the Scheduled task, which occupies scheduled time. These are 'one off' tasks which are not routine, but are usually preplanned, known outcome tasks with deadlines. While they can be interrupted, doing so exacts a cost in both 'shutdown' and 'startup' costs for the task, as well as in resources (lab space, spare/swap machines, etc.) needed to maintain potentially fragile 'middle states' of the task. Some examples of this include hacking/scripting to develop new automation, configuring machines to perform new duties, configuring new machines to replace older servers in a preplanned switchover, implementing new services on existing servers, implementing test plans for new services/software, etc. In short, the expected use of time to carry out more demanding work with less tolerance for interruption, but which does not represent a 'regular task list.' Time spent in Routinized tasks, naturally, cannot be used for Scheduled tasks, and vice versa. So this is a primary reason to avoid overburdening Ops with Routinized task loads.

Now, if these were all an Op had to worry about (like, say, if s/he were a coder, or a web developer, perhaps) then things wouldn't be too bad. Nice Gantt Charts could be drawn, timeslices could be set up, and spreadsheets could be done showing managers precisely what was being done when. The problem is that this isn't what Ops do. This brings us to the third type of task and time.

Sometimes, a server gets compromised at 2 am. Sometimes the mailserver eats its disk array at noon on saturday. Sometimes for whatever reason, somebody needs a new public-facing machine up in 5 hours, and even the Op is forced to agree they may have a point. This is a chaotic task. I call it this because even in the case of building a new machine, being forced to do so with no advance warning usually introduces enough uncertainty (what machine are we using? Do we have enough RAM for that? Are the disks OK? Do we have drivers for that? Well, will it fit that rack? Oh, they want SLES, but it's a Dell 2650? Well, we have to run RHEL then, because we don't have RAID drivers for SLES for that box...etcetera, etcetera) that claiming these are preplanned processes is a bit of a joke. In the case of a full recovery, making time predictions may even be criminally stupid.

So there's the rub. You have three different types of demands. You want to make sure that when chaotic events, or tasks, arise, they'll get done - because they are almost always mission-critical for somebody or other. You need to make sure that the routinized tasks get done without too much interruption, or things start breaking. You need to make sure that the scheduled tasks get done in some reasonable timeframe, or you start falling behind in your ability to implement services and become a purely maintenance organization (and a poorly performing one, at that).

This will mean people will come in and want to know why you're reading mail rather than implementing the new mail server. One answer: Because I don't have enough of a scheduled time block to do so at the moment. Another: I'm on call at the moment, and can't get too deep into something that I can't drop.

Whatever the reason, it's true, there are limits. They're different for people and places, and they're hard to know without knowledge of the organization and people in question. But the next time you're told, as an Op, that you can easily fix something with a minor little tweak that requires semi-regular touching by a human, remember - if you say yes, you just added something small but recurring to your routinized workload - another thing that might get forgotten or missed if time runs short, and another thing that steals cycle time on a regular basis. Posted by jbz at 1:37 AM | Comments (0) | TrackBack

December 28, 2004

A spray of darker tone for the delectation of the sighted

The darkness is in matched cones, one forward and one back. Spumes of light and dark. What interrupted the quiet whispers of the pale thin winds then must have been a shock, too solid and too bright - and too soon accepted into the scenery. A night of jarring difference, perhaps, before a dawn of wonderment and awe, replaced by the next which was to bring only the same scene again, bereft of change.

For nearly a year.

Now, however, change has come, and it is curious - sniffing cautiously at the scene of this, its own one-time disturbance.

I have been transported, momentarily - the curiosity and inquisitiveness that made my distant ancestors come down from trees, till the soil, build cities, travel over the next hill - those qualities have suddenly liberated me across the scorched and freezing miles and embodied me in a clicking, whirring form of metal and plastic. I have come to see - and the view is wondrous.

It's unreasonable to be emotionally proud of a small machine, much less one I had no hand in building. But, damn it, I am, of both of them.

Keep rolling, little brothers. Keep seeing.

Posted by jbz at 10:57 PM | Comments (0) | TrackBack

You'd think an alcoholic lived here

Hm. Took stock of the liquor cabinet yesterday and realized that my intake has dropped fairly sharply, and hence, stocks are up. Resolved to get tippling. Currently in stock, through the efforts of myself and other, connoisseur-types -

I feel a tad overwhelmed. Ah well, and now with a box of Macanudos and a box of Fonsecas to go with 'em, I better get to it. Rough job, but somebody's got to do it.

Posted by jbz at 4:49 PM | Comments (0) | TrackBack

December 22, 2004

"Was it a dream where you see yourself..."

I was thoroughly enjoying the story posted to /. about the complete hack (in the MIT sense) that was the development of the Macintosh Graphing Calculator. Then I noticed the name of his company again, and was so tickled I emailed the gent asking him if my suspicions were correct, and he fired back an email within minutes telling me that yep, indeed, the company is named for the school in one of my favorite movies, and shared some additional funny about other company names he's been connected to. Hint: "Welcome to Pacific Tech...!"

He's unquestionably one of my heroes now.

Posted by jbz at 1:43 AM | Comments (0) | TrackBack

Nanny Engineering

Argh. So I'm having scads of fun playing with my new toy. I'm happily cruising fantasy ways to blow money on the beast (other than buying winter shoes, which I have deemed to be my Upgrade Monies for the Year). One of the few things that has been bothering me about driving it has been my slightly poor clutch performance. I have been shamefacedly putting it down to unfamiliarity with the car and transmission; perhaps too long behind the wheel of the shitbox I'd been driving for a year. Maybe I was just rusty. Perhaps I just didn't 'get' the Real Man's (tm) German Shifter.

Then, while researching the mods I could drop on the car, I came across this.

Now, I'm not sure this is true (although it looks pretty plausibly true). I'm not sure what the actual effect of this part is. However, based on their description of it, it would seem to me that it does what they say - it's a fucking nannywidget. It's designed to make the thing easier to handle for the bottom of the bell curve, but by fucking over those of us who'd rather take our lumps and learn to drive it good and proper. I mean, there's no way I'm going to learn how to shift the car properly if not only will it not let me directly control the clutch, but I don't know the thing is there.

This has made it to the top of the 'must unbreak this design decision' list.

Posted by jbz at 12:18 AM | Comments (0) | TrackBack

December 14, 2004

That shit is *WACK*.

You will go purchase

Newman's Own Suicide Mind Eraser

like, pronto. Capice? Now.

It is by the inestimably awesome Don Red and can be purchased for the low low price of six clams here. There are a few tracks available as full mp3 on that website as well.

This genius is not safe for work, bitches.

Posted by jbz at 8:42 PM | Comments (0) | TrackBack

So I'm a crassly materialistic snot.

I acknowledge this about myself. It's one of my many failings.

However, I had been ready to purchase a 2004 Toyota Prius about a year ago, and an absolutely abysmal experience with Herb Chambers Toyota convinced me to never, ever purchase a car from the Toyota retail sales organization ever again. It's a pity, too...while the Prius was a nice geek vehicle, it wasn't a particularly great car - but I do love some of their toys.

However, for less than I was willing to pay for that car, I have acquired a new friend.

His name is Darthwagen.

Every time I feel a pang of guilt over my excessive consumerism, there is a deep hiss from the climate control system as he informs me that I do not yet understand the power of the Dark Side.

I think he will kill me if I show weakness. But together we can rule the highways as Sith Lord and disciple.

Besides, I'm a good Jew. Pay retail? Ha! As if.

Posted by jbz at 1:09 AM | Comments (1) | TrackBack

December 1, 2004

Mea culpa spamorica.

I must offer my apologies to the Novell spam gateway system. As I have been informed by numerous cow orkers (moo) there is, in fact, no need to deal with messages the gateway traps. All messages are default flushed unless 'preserved' by their recipient, so the message digests are in fact means of watching for false *positives.*
Posted by jbz at 1:08 AM | Comments (2) | TrackBack