ChrisDowning

@ChrisDowning

2,209 words

Guestbook
You'll only receive email when ChrisDowning publishes a new post

Development Environments

I've never really settled on a good way of managing software-related projects - both things I'm putting together myself, and downloaded resources I'm experimenting with. Not being a software developer, it has never become a priority, and so any structure which might be detectable in the contents of my "Projects" directory has thus far been purely coincidental. However, over the last year I've had several halting attempts at trying to change that.

The end goal of my recent experiments is to determine what the structure of my personal laptop OS is likely to be when Ubuntu 20.04 rolls around - I've settled on having a LTS distro on the device itself, and so my "development" workflow is likely to require more attention. With that in mind, here is a summary of the things I've tried while stumbling towards some sort of sane, professional setup.

Arbitrarily labelled directories: A sadly-too-common fall-back; but not very neat or easy to manage after a period of non-activity. Backups become opaque ("Have I archived this one already?"), and nothing ever truly feels finished. The one exception to this, shining like a beacon of sanity, is the directory which contains example code I've put together as part of an effort to learn Rust. I adore Cargo, and the Rust toolchain management experience in general. It's nice to see that some lessons have been learned from the shit-show that is Python packaging.

Git repositories: A significant step up from raw directories - the trick I haven't mastered is figuring out when something is "significant" enough to require version control. The correct answer is probably to use git for everything immediately, shifting the decision back a step - ie to determining what remote origin to use (if any). Git-by-default is the new habit I'm trying to form. Fortunately I love using git - I just wish I worked on more projects which allowed me to experiment with the more advanced bits of functionality.

Python virtualenvs: More of a necessary evil than a deliberate choice, Python virtual environments have probably introduced more hassle than benefit in the small use-cases I have had to date. I am certainly guilty of over-complicating things when using Python - my default position in the past has always been to install the Anaconda distribution, and occasionally Intel Python as well. In reality I hardly ever used Intel Python, or the high-performance functionality of Anaconda. As a result I've concluded that in future I'll stick to the OS system Python, or use Intel Python/Anaconda exclusively inside a VM or container. More generally, I need to do some more tests to decide between simply using pip+venv and using poetry, since these seem to be the best current options.

Docker containers: Using Docker for local development environments has never really appealed to me. Given that my work doesn't involve the development of persistent services, the benefits of matching a development workflow to a production one are mostly lost. Without this benefit, Docker really becomes just a convoluted and opaque alternative to running in a VM - as such I have tended to avoid the hassle.

Other application containers: Working in the HPC space, I have a natural disposition towards containers other than Docker - namely Singularity, Charliecloud, and more recently Sarus. While I toyed with the idea of Singularity-based development environments, the default approach to bind-mounting host directories wasn't really what I wanted. Similarly, immutable container images are what I want when building an executable - but not when doing ad-hoc experimentation with software packages.

Vagrant VMs: For a time, managing VirtualBox VMs with Vagrant was my preferred option for personal projects. However, I never really liked that the VirtualBox application itself had a license structure which effectively precluded professional use, or that the Vagrant templates themselves were Ruby syntax. While I have nothing against the language, it seemed unnecissarily burdensome that I needed to know anything about a particular programming language rather than just using a standard, well-documented config file syntax like YAML or TOML. Using alternative Vargant providers had some appeal, but I soon decided that keeping a bunch of plugins up to date was an unnecessary hassle.

LXD/LXC: In recent months, LXD with ZFS-backed storage has become my go-to solution for isolated environments. Creating a system container is fast, and the set of available images is acceptable (albeit rather biased towards Ubuntu). LXD is about as lightweight an interface as I could ask for without skimping on any features - my one minor complaint is that customization is only really possible via quite "raw" methods like using cloud-init.

Multipass: I only became aware of Multipass very recently (late 2019), but quickly concluded that if LXD wasn't the right answer for me, then this might do the trick instead. Most of the ease-of-use features I want (copying files in/out, mounting directories) are available in Multipass by default, while LXD requires tweaks or hacks to the host OS. These are trivial enough, but knowing that the problem is being handled in an "official" way is nice. The downside is that despite now being in a 1.0 release, there still seem to be significant bugs - I've yet to get a multipass VM to remain accessible across host sessions when suspending or rebooting.

The One True Answer (for now, at least)

Realistically, I'll probably just use either Multipass or LXD as the "outer" layer for projects, with Python virtualenvs internally where necessary. Both have an acceptable approach to bind-mounting directories from outside, though the automation of UID/GID mapping in Multipass probably means that it will win out in the long run if the problems can be resolved. If it turns out using LXD is still part of my workflow in the coming months, it will almost certainly be because of ZFS and the ease of managing storage. I'm currently considering the best way to manage an isolated ZFS pool (based on the same approach as LXD, ie a file-system in a file) which can be shared between Multipass VMs. However, the fact that this is best achieved by just using a directory mounted from the host begs the question what benefit ZFS is actually adding in this instance.

Regardless of the choice of Multipass or LXD for workspace isolation, enforcing git-by-default and a simpler Python workflow will definitely help to make my life easier.

The Desktop App

Recently (that is, for the last 2-3 years), I've resisted using the desktop version of browser-based applications. Either they tend to be incredibly bloated and slow compared to the browser itself, or they have a shoddy half-finished feel about them.

This one seems different. While writing this short blog, I got distracted by testing the creation of AppImage binaries (of which the StandardNotes app is one). When I came back to finish off the post, I forgot which window I had been in a couple of times, and flipped back and forth between my "home" browser tabs and the desktop version. The fact that I couldn't really tell the difference means that everything is working as intended.

Aesthetic

It's now been just under two weeks since signing up for StandardNotes.

Boy - do I owe the creator an apology. "Slightly uglier Simplenotes"? Now I've adjusted to the layout, I can't imagine anything better. Over this two week period I've written more - and been inspired to structure my ideas more - than at any time over the last several years. That impulse has leaked over into my use of other tools, in a positive way.

I'm still on the fence about whether this should be the "permanent home" for a blog. A GitHub/GitLab static site has a certain appeal due being more familiar - but avoiding the need to generate the static pages and push them to a repo is nice. The reduced friction of being able to just type something, hit publish, and "fuhgeddaboudit" is somehow quite freeing.

Link Dump #1 - 2019-12-05

The first of (hopefully) many link dumps - a place for me to offload the interesting articles or pieces or software I come across while trawling the internet.

Python

A consistent frustration for me has been the difficulty of using even moderately complicated Python projects across multiple devices. A lot of my recent reading has involved a search for whatever is the canonical/best approach to bundling dependencies together as a portable development environment, and also create a simple binary distribution.

I haven't come across the ultimate solution yet - but these are some of the tools or guides which will hopefully ease the burden the next time I have to tangle with Python.

Pex: A Python tool for generating executables from Python environments or individual packages.
Voila: A tool for converting Jupyter notebooks to interactive web dashboards, while retaining a functional compute kernel.
Pipx: A tool for consuming already-published Python packages as regular binaries.
Poetry: A popular alternative package/environment manager. Although I haven't used it extensively, my feeling is that this tool is most likely the one which will allow me to consider my frustrations with Python "solved".
PyOxidizer: The most modern in a selection of tools which promises the ability to generate application binaries from Python projects.

Chess

I recently started playing chess (using the Lichess android app) - having progressed from "beaten every time by the most basic AI" to "find the basic AI annoyingly bad in its stupidity" - I'm ready to step up to Stockfish 2! Thinking about how modern chess engines compare to real humans sent me down a rabbit hole of learning about how they are developed.

Stockfish: The home of the most popular chess engine, which is also one of the strongest.
Chessprogramming Wiki: A handy resource for anyone exploring the idea of writing their own chess engine, or those with an interest in the techniques used to make the engine more or less competent.

Rust

Learning some form of low-level programming language has been on my to-do list for far too long. Having never really progressed beyond useful-but-not-professional Python, I recently decided that targeting C/C++/Fortran to do anything numerical was probably a waste of time; anything I need will be far better served via existing open-source libraries than whatever novice version I could put together.

That decision led me towards Go and Rust, both of which have the appeal of being potentially useful for writing CLI applications - an area where I'm far more likely to be able to put my learning to real productive use.

The Rust Programming Language: A free online version of the (definitive?) guide to Rust. I have a copy of the physical book, and so far it is probably the best text I've come across about a particular programming language.

Encryption

A topic I find fascinating, but don't have much scope to experiment with on a day-to-day basis. One of my general concerns has been that gpg is easy enough after a bit of practice or memory-jogging, but not simple enough to be a comfortable default choice when only using it on a very occasional basis.

Toplip: A CLI tool for encryption with steganography and plausible-deniability features.

Raspberry Pi

My current count is 10 Raspberry Pi devices scattered around the house. One is permanently occupied as a bastion host/generic testing device, while the others tend to spend a few weeks in a cluster configuration before I get bored and put them aside, only to come back to them a few months later. My plans for the next round include a k3s cluster and a set of lightweight file-sync nodes. In that vein...

k3sup: A tool to simplify the deployment of k3s. While k3s is already a seemingly much more straightforward version of Kubernetes, impatient people like me can benefit from the "quick wins" a tool like this provides, and use it as a staging point to start learning more.
Resilio Sync on a Raspberry Pi: Many moons ago I was a BTSync user, but moved away from it as I was lacking a use-case. More recently I have decided to try to ditch Dropbox and Google Drive, so a private p2p sync tool seemed like a logical consideration. I tried Syncthing, but wasn't impressed by ther performance or stability. While Resilio is proprietary, the improved user experience seems to more than make up for this so far.

Operating Systems

I've now been a preferential user of Linux for over 10 years. One thing I've consistently overlooked is the world of open-source, or generally non-Windows operating systems outside the Linux space. People who have worked with a modern Unix OS seem to swear by them; so these are the two I intend to try when time allows.

OmniOS: A Unix OS, which might be useful as a means to learn about Illumos-based systems, and also make a nice hypervisor for Linux VMs/containers, with ZFS as the underlying storage.
SmartOS: Another Unix OS, seemingly an even better choice for a pure hypervisor deployment.

Hello World!

I'm experimenting with StandardNotes, after seeing it mentioned on HackerNews and being sucked in by a Black Friday deal.

Initial impressions weren't so great; it felt like I'd just signed up to pay for a slightly uglier version of Simplenote, which I've already used for years.

This feature might change my mind, however...