SysAdmins in academia

SysAdmin to SysAdmin: SysAdmins in academia

By: Brian Jones

The job of a system and network administrator in academia differs wildly from the job in a corporate environment. Over the past few weeks I have contacted other administrators who have worked in both environments in an effort to make a comparison between the two. I’ve received a lot of feedback from admins all over the country. I hope that you find the results as interesting as I did.

After reading the email I received, I found that the basis for comparison revolved around three main areas, which I’ve broken down below.

How decisions are made

A large part of the job of a system administrator in any environment involves either making quick decisions in order to put out a fire, or trying to convince others in your group or department to go in a certain direction when making higher-level, long-term decisions. However, that’s where the similarities end.

In my reading, and in my personal experience, academia has a far greater tendency to be what I’d like to call a “stateless meritocracy” than does a corporation. A stateless meritocracy, as I’m defining it here, follows these basic rules:

* Decisions are based on input from multiple parties or individuals.
* Decisions are based on empirical or demonstrable evidence that a particular tool or direction has the most favorable result.
* The decision is made with little or no regard to earlier mistakes of the person putting forth evidence toward the making of the current decision.

In other words, if you previously made a decision that didn’t work out, that is largely irrelevant in the context of what’s happening now. If your way looks like the right way, that’s probably the way the group will tend to go. Of course, long strings of poor choices should be avoided at all costs. Also remember that this is a tendency in academia, not an absolute truth. There are academic environments that make decisions with almost no input from anyone, but this would appear to be (in my informal survey) more the exception than the rule.

In the corporate world, there tend to be more layers of administration, separate internal groups affected by decisions, separate user communities with (sometimes) separate departmental infrastructures, and even separate policies concerning things like security and quality assurance, such that the job can be like building out a single solution for more than one company!

As a result of this complexity, decisions tend to involve more people in more meetings to discuss every aspect of a solution, from how funds are appropriated for purchase to the system calls it uses to perform a certain function. In academia, these discussions take place, but it’s usually a couple of emails and a 20-minute meeting instead of several lunch hours spent at a large conference table.

Finally, corporate environments tend to be more track-record-oriented than our semi-utopian stateless meritocracy. Earlier bad decisions count against you. In some environments it gets to the point where it seems like decisions are made based solely on the track record of the group or individual putting forth the idea. Maybe this can be called a “stateful meritocracy,” in that the merit of an idea is based on the results of earlier decisions made by the group or individual.

How projects are organized

Project organization and management in corporate and academic environments is the most fun to compare in some ways; in other ways, it’s a little scary.

In a corporate environment, projects are assigned to teams that generally include one or more project managers. Project managers have several duties, not the least of which is coming up with a plan for the completion of the project, and implementing that plan by assigning deliverables to the specialists with well-defined deadlines. However, what many consider the absolutely crucial job of a project manager is managing the expectations of the client. In fact, one company I worked for actually had a position called a “relationship manager” whose job was to collect data from the project teams and then spin that into something digestible for the client.

Specialists are people who will be delivering on their assigned tasks and reporting their status to their project manager on a regular basis. Each individual specialist manages the expectation of the project manager on their own behalf and to their own benefit.

What’s interesting about this is that the flow of data about the real status of the project is bottom-up — from the specialists, to the project managers, to the clients. Meanwhile, the data concerning the goal of the project flows in the exact opposite direction, again using the project manager as the middleman.

As both project manager and specialist, I have always found this to be an ineffective way to manage a project. However, when you have a job to do, you’re likely to do it in the canonical way. That’s what I did — and I always assumed that someday I’d see why it was done that way. I never have. Academia has taught me that I was not a dim-wit after all!

Projects in academia are far simpler by comparison: those who need a technical solution to a problem tell a team of technical specialists (possibly accompanied by a manager) what they want. The specialists begin probing to better understand the problem. They eventually figure out what the end user actually needs, and they figure out the resources and time needed to perform the task. They then split up the work into component parts and build out the solution.

This simplicity and relative informality makes working on projects in academia much more enjoyable, and less stressful — once you get used to it. Coming from a corporate environment, this aspect of academia flat out scared me, because while accountability is well known in this scenario, the relative lack of planning and formalized deliverables and deadlines means that the expectations are not as well defined.

How solutions are implemented

This seems to be the largest difference between the sandals and the suits. If you have worked only in corporate, academia will scare the pants off you, and if you’ve only worked in academia, corporate will, well, scare the pants off you.

In a corporate environment, a project is developed in a testing environment that closely, if not exactly, emulates production. If it works there, that’s great! It doesn’t mean you’re going to stroll into the production datacenter and start logging in as root to roll out your brainchild. What it means, a lot of the time, is that you’ve earned the right to help put together “change management documentation.” Change management documentation is a collection of documents that, very slowly, very deliberately, in a step-by-step manner, with screen shots, tells somebody else how to do the production rollout, and how to respond to any conceivable mishap along the way. You hand this over to some folks in the datacenter, and wait by your pager on the day of the rollout in case something goes wrong that isn’t in your documentation.

In academia, what is more likely to happen is you find a machine suitable for the service you’re implementing. You set up the machine on a private subnet and bang away on it, using as many clients in as many use case scenarios as you and your group can think of. If it works, you can then (if appropriate) invite some of the more risk-averse users to help test this alpha service. Once that works, you can do a beta release of the service, which consists of having one or two clients completely dependent on the service, and others using it on and off. This grows over the course of a couple of weeks until you’re ready for production. “Ready for production” means that the people who set this up are willing to bet their Nerf arsenal that this thing is going to work without a hitch.

Again, coming from corporate, this is scary. Again, accountability and expectations are a concern. Again, it is far less formal. Maybe the biggest difference on a technical level is that “production rollout” in some cases means changing a DNS record, moving the machine to the public service subnet, or removing records from a hosts.deny file or iptables script.

In conclusion

In the end, the core difference between corporate and academia is that, while a Web site becoming unavailable for two hours in one day is inconvenient, undesirable, and stressful for administrators in academia, it is not likely to cost the university millions of dollars. There are many Web sites in corporate America that cost thousands of dollars per second of downtime. Web sites are only one example.

That fact changes the cultures of the two communities as well. In corporate, administrators are generally laden with tech gear that enable them to do their job from more locations, and be reachable in more ways, at any hour of the day. There is less time to call your own, but in exchange, you’re generally paid a good bit more than in academia, where things are a little more relaxed (not that there aren’t emergencies).

I haven’t touched on the user communities, standardization, why Linux has been so big in production in academia but slower to catch on in big business, what “availability of applications” means in the two environments with respect to Linux, and how ROI and TCO are measured, which answers lots of other questions. This will all make great material for another article.