Incident lesson learned: Do not depends on system dependencies!

Last month, my company had an incident that causes a certain part of our system to goes down. We're running everything on-premise since it's regulated by the law of financial institution of Indonesia, so we can't really go with cloud. It was caused by a sudden Virtual Machine failure that causes the VM to be corrupted. There were no backups or snapshot for that VM. The incident itself last for the entire week, and we're struggling to get it working again.

The case of tightly-coupled systems

There are numerous cases and articles on the internet that talks about the danger of tightly-coupled systems, those who doesn't provide an easy way to change external dependencies, those who strictly requires some very specific version of runtime or host operating system, or even those who only works using an old binary that's compiled and installed in 2018. Whatever the case was, every each of the reason of making the system being tightly coupled are not acceptable for me.

Although, we should acknowledge that not every system can be made loosely-coupled, sometimes the requirements demands your system to be tightly-coupled, and the limit to "loosely-coupled" changes overtime, but we should also acknowledge that having a big chunk of tightly-coupled system is not good for maintainability. It will be even worse if you're tightly-coupling yourself with something that only guarantee a support of 5 years maximum. Once that support years have passed, you're left alone. Even if there is some emails and a few phone calls that remind you to do some upgrade, if you don't have it as a priority and you're just ignoring it, that will also means you're still left alone.

What are system dependencies?

System dependencies are dependencies (obviously) that's strictly only comes on one specific system. Usually this is lib*-dev that's installed on a very specific operating system distribution. Some programming language like C, C++, and PHP most of the times depends on system dependencies that you have to install on your operating system. A simple example would be installing libpq, libpq5, or libpq-dev for you to able to use PostgreSQL client. In PHP, the functions are just a bridge to the C libraries. But, say libpq-dev that you installed is in version 7, and you need to upgrade your Ubuntu from Ubuntu 14.04 to Ubuntu 20.04, then how can you guarantee that the libpq-dev version would be the same?

Problem is, you can't.

Every update to the system dependencies would make a version bump to those kind of development dependencies. That would mean you'll have to face with other kinds of upgrade rather than just upgrading the operating system (in the case above, your Ubuntu version). Would that be a easy and breeze upgrade? I bet it wouldn't.

What can we do to improve?

I would suggest to vendor your dependencies. If you're using C or C++, you can leverage CMake and download your dependencies as Git submodules, then compile it from source by using CMake (or other build tool that you prefer, like Bazel). Vendoring dependencies is not something that only works on C or C++, it also will works for Nodejs, Go, or even Rust.

For PHP, I don't think there is any other options. You should rewrite your PHP application to some other language.