How did I get here? It's 2025 - 3 decades of system administration

In the early days of my professional life, computer systems were mostly static. It was a relatively simple setup: one would install a system of some kind, give users access to said system, and that is that. Most of the demands of business were not (yet) put on computers and most of the business logic and processes had not been digitized. In fact, the Internet was nascent and companies had not fully embraced this new technology; fax machines were the way to send documents from point A to point B. I used to be able to bike to my ISP, knock on their door and ask for help. Later, when I got a job with them I would call up our upstream peer directly, handle DNS, Web and Usenet servers (later, E-Mail, too). I would take customer calls in the morning to help them resolve their issues and directly logged into our servers to manually take care of things. Backups were done via shell script. This textured relationship between the physical and virtual world would start to break with virtualization, where for the first time there wasn't a correspondence between "server" and "service" anymore. Standardization on use cases for business (file server, print server, web server, and so on) meant that the beautiful world of custom hand-woven and intricate relationships between Linux servers was now gone; and replaced with repeatable, cold, virtual machine images design to accelerate (and reduce errors of) IT as a business function. I used to know all the parameters and their impact in each configuration file whose service I was responsible for. In fact, I used to know (and frequently adjust!) all the config flags when compiling the executable as I was not using the 'packaged' stock version of whatever server I was installing: it was hand-made, adjusted and optimized for my system. As was the kernel itself it was running on. Reducing the size of the kernel, carefully selecting and strategizing which driver to compile into the kernel and which to load as a module later; which service would be running and which would not; decide on a partitioning scheme to ensure critical parts of the system would be protected from potential misconfiguration of others (/var vs /usr vs /home); and so on. I would optimize the blocksizes on partitions based on their use (many small files? a few large files?). Shell and Perl scripts would rule the day and become the duct tape our datacenter ran on. Hell, it was the duct tape the entire Internet ran on.

I don't know many of the details anymore. I have forgotten more about Linux system configuration than one would ever care to learn. Dealing with a virtual server reduced the complexity and intricacy of system configuration by a large margin. Coupled with the rapid progresses computer systems made in terms of available memory, processor speeds and disk space, handcrafted Linux systems were not only unnecessary anymore - they quickly became downright antiquated. It wasn't just obsolete, it was ancient. It wasn't even frowned upon, it was ridiculed. This transition effectively ended a sense of pride and purpose of a System Administrator in what I would consider the "classic" sense: someone who'd not only read the man pages of the system they would administer, but also know that they were typeset in troff, and would produce their own man pages for the comprehensive software they'd written to make things easier. This knowledge, and sense of common purpose and general politeness to document the system for others to pick up one's responsibilities should the day come vanished all of a sudden and was replaced with a semi-standard of pre-configured machine images that followed vendor-imposed rules that did not allow for one to "make it their own". The Linux man pages were much less important than the VMware online documentation for one's success; and the solutions this new generation of system administration professionals found, and often times (re-)invented, differed drastically from those of the previous generations, often comically so; in essence, "have you tried turning it off and on again"-style solutions found their way into the mindset of Linux admins.

I did not experience closely what happened the next decade or so as I was busy pursuing other things (academically) and I did not re-enter the IT space until 2011, when I found that the world had changed yet again. In place of Shell and Perl scripts, one would interact with an API to control the management of a fleet of virtualized servers, which booted of golden images that would perform a singular function upon boot and which registered themselves to a state database containing the configuration key value pairs which were of interest to remain centrally unified rather than individually diversified. In fact, heterogenous computing infrastructure was to be avoided at all costs; instead, purchase identical systems to run identical hypervisors and ideally, identical guest operating systems on these hosts across the entire fleet. This, of course, was largely a pipe dream; but that didn't stop the entire industry bemoaning the fact that they still had to deal with "legacy" services that did not support the new way of administration. My impression was that of all the systems deployed in this time frame, only 5% of the code actually mattered or was ever loaded into memory for the purposes of supporting its system to fulfill an actual purpose. The rest was standard fare, not to be judged, questioned or -God forbid- optimized to perform better. I have no better example than syslog: of course, it is important to capture system logging messages, but in a large fleet that is always done in central syslogging infrastructures where these logs are captured together, on synchronized clocks for time-correlated analysis of events. As no one ever logs into these systems (golden images are fire-and-forget type entities that get updated by updating the image and then rebooted), there is no purpose whatsoever to logging to disk. Yet they all did. Again, this did not matter as much since the machines had grown so powerful that nobody cared whether additional disk space or processing time was being wasted logging to ephemeral, virtual disk space that would be lost upon reboot.

And so, a whole generation of system administrators' knowledge base was eradicated from consciousness and lost to the annals of time. As someone who'd gotten their feet wet prior to virtualization, I cannot imagine any lessons worth remembering coming out of the decade of virtualization that would come anywhere close to all the lessons and teachings and experiences we made in the 90s. Maybe I'm wrong. But I don't think so.

In any case, the advent of "cloud" further sharpened the contrast between hand-woven system administration and API driven infrastructure management. Here, I observed another phenomenon: that of the developer getting deeper involved in the so-called "operations" side of the house. Web developers, once at least belittled, but mostly ignored by system administrators, gained more power and influence as they managed to be part of the "business". IT, traditionally a cost-center, had to yield to a revenue-generating, new generation of developers who had very little interest in Shell and Perl, and instead leaned in on a particular protocol that seemed to have triumphed over practically all others: HTTP. As any good sysadmin will tell you, HTTP is simply the protocol spoken by an application that commonly listens on port 80, and is called a "web server". It was designed to transfer hypertextual information, but could also supply binary appendages alongside the text for a program to interpret and render on the screen as a web page. This new kind of web developers was not interested in writing Hypertext (HTML) pages as much as it was interested in calling functions by hijacking HTTP "verbs" (commands) such as GET, POST, PUT and DELETE. They were now not assuming a web browser accessing a web server, but two programs exchanging information and commands, each running on their own virtual server (or, as it was now called, cloud instance). The reason for this development was quite simple. As the scale of inquiries of any service increased dramatically (and often followed its own "seasonal" pattern, like a storefront being busy at Christmas time), the ability to address application bottlenecks dynamically by breaking up services in 'microservices', each running as their own patterned cloud service behind a load balancer, with telemetry measuring leading indicators of performance degradation, and then applying automatic scale out (i.e., adding more instances behind the load balancer to spread said load out more), we now had for each 1 connection of a user, N connections on the backend across a multitude of servers.

Boy, we had come a long way from knowing every configuration parameter of the service we were running. To be fair, there were efforts, especially by key folks in organizations who could afford them, who knew that at the end of the day, performance of individual systems actually matters, whether the application is now spread out or not; and that this would be especially true if the service transcended the textual and multimedia payloads started to dominate the dataset being processed by all these systems (streaming movies or videos "over the Internet" is what mainly comes to mind here). However, by and large that was an exception. At least the experts weren't ridiculed anymore (payday, baby!).

This was, roughly, the state of things until a little over a year ago. In 2023, a disruptive way to interact with computer systems made its way into the landscape: AI. We are - as I write this - in the middle of an explosion of innovation around AI models, and the brightest minds in the world are currently chasing AGI - artificial general intelligence. However, what virtualization did for system administration, it looks like AI is doing for application development. It is no longer necessary to learn a particular language to program a computer; instead, plain English is enough. Sometimes, you do not even need English, a drawing of what the application should do suffices in a lot of cases, and code is being generated. Now, to be fair, most of it is still crappy, but it's getting better and will first accelerate, and then obsolete, the programmer. The pattern seems to be to skew (again!) the ratio of human-computer interaction and computer-computer interaction heavily towards the latter; as now most system interactions are now done my machines to other machines, except now it's machines that told them to do that (and how).

However, my thesis is that there is a transcendence of core knowledge that is both useful and necessary to successfully manage, plan, architect and operate systems across all these years. I still believe that in order for someone to be a good web developer turned prompt engineer, one should know that HTTP is on layer 5 of the OSI layer model; and that high IOWAIT times are an indicator of slow disk activity. However, I don't know why that is important, concretely. I guess I tend to think that knowing at least the fundamentals of how something ought to work would let you ask better questions, whether you ask them a human or an AI. I know very few people with a journey similar to mine, but they are out there. In case any one of them bothers to read all this, I would be incredibly interested in their opinion and feedback on these thoughts. I still think that there is necessity of apprenticeship between the prior generation and the next; and that there is value in the preservation of knowledge of details in human brains, not just artificial ones.