Brutalist Builds - Keeping Builds Rugged and Simple

Overview

Here are some basic guidelines I use for myself for keeping builds simple, rugged, chainable, and maintainable.

1. All the CI server should do is pass build metadata like git branch and revision as environment variables. Secrets are passed as command line parameters to an entry shell script or makefile.
2. Once invoked by the CI server, minimal scripting should assemble any needed build metadata into an easily parsed format such as YAML, XML, JSON, or TOML. Secrets from the command line can be used to set build container environment variables.
3. The entry script starts a containerized build, using a Dockerfile or something similar. Builds should be kept small, typically not building more than one package.
4. Built assets are pushed to a binary repository like Nexus itself. This would typically occur within the container itself, otherwise if the resulting container image in and of itself is the desired built product, it is pushed to a container registry by the script that invoked the containerized build.

The 4 steps above define rugged simple builds that can be easily chained. Assets pushed to a repository in Nexus can server as input assets for downstream builds.

Governing Guidelines

Here are a few guidelines I stick by. So far, I mention more the what but less of the how. Google is always your friend learning more about the how. Stay tuned for more articles with more focus on the how.

Keep CI Scripts (e.g. Jenkinsfile's) Minimized

This is less of an issue as more and more builds moving to the cloud. Build plans are evolving into simpler more declarative YAML based files. Self hosted build servers such as GoCD and GitLab are also moving in this direction.

Jenkins on the other hand is a different story. Build developers often create piles of Groovy code in Jenkinsfile's, or worse figure out how to cram build logic into in-house Jenkins plugins so they can get Java on their resume. This quickly turns into a mess, often resulting in hard to maintain builds with complex build logic in Groovy or Java.

A Jenkinsfile or any other CI script should just do the following:

1. Pass build metadata as environment variables to build scripts,
2. Pass secrets to the build, typically on the command line.
3. Call a build script that is part of the project, like a makefile, MSBuild script, NodeJS script, etc. providing the information from 1. & 2.

If you have to scroll through the Jenkinsfile in an editor, it's too big.

Ensure Builds are Runnable Without a CI Server

One should be able to run the build without the CI server with only a source checkout. This mandates that CI scripts need to be simple and small. Anyone troubleshooting a build should be able to manually provide build metadata and secrets to the build entry script and run the build.

Consequentially this leads to the following:

• Developers should be able to build locally.
• Components should be buildable and testable in a local development environment.

Simple Shell Entry Scripts

The entry scripts or makefiles invoked by the CI server on the build agent should be simple and crisp. If they are a couple hundred lines or more, they are too long.

Use Containers - No Complex Build Server Setup

Build the project inside of a container. The big problem with not using containers is that you end up with build machines having stale old versions of .NET, Java, Node, Python, you name it, that holds back teams from using current stable framework versions and runtimes.

Windows Containers

For Windows targets, that's Docker running Windows based containers on Windows Servers. You can get Windows Server 2022 Standard for less than $1000 for 16 cores at the time of this writing. This is much cheaper than developer time lost dealing with an old Windows Server 2012 R2 build agent, no containers, having rotting old versions of .NET and other runtimes to build against.

Setting up Windows based Docker containers can be difficult, and I admit I have not worked with them. I have found base images though so my guess is that these are usable for builds. If containers are not feasible then use at least Windows VMs with an automated setup using something like Ansible.

Only build in Windows if you have to. Crosscompiling is often an option, and I have heard you can even build an MSI installer in a Linux based container using the WiX Toolset (worth a good Brutalist Build article if I achieve that in a POC).

Whenever possible, use a Linux build server with Linux based containers. They are much easier to setup and maintain

Linux Containers

Setup for Linux build servers should be simple. Get something to run containers e.g. Podman or Docker, and then keep the packages installed in the host OS at a minimum. This means sticking with Bash or Zsh for a shell, GNU Make, and maybe additional like jq or xmllint. You always have good old sed and awk. All you need is minimal setup to kick off what needs to run in the container.

Complex build logic should run inside the container.

Avoid and Get Rid of In-House Build Plugins & Apps

These are the custom tasks for MSBuild, plugins for Mavin, Gradle, and other builds systems that get created by and maintained by in-house development teams. Build applications that are standalone executables for build logic.

Initially, these seem to be good, but often mushroom into a Swiss Army Knifes then quickly evolving into a poorly maintained Lava Flows. As frameworks age, these quite often also result in Dependency Hell.

The best way to combat the pitfalls of in-house plugins and apps is to know and use what you already have available 'Off the Shelf'.

'Off the Shelf' Berfore DIY

Off the Shelf is the scope of what's available to you in your own build system. Before the build runs in the container this means your shell and available utilities. Without complex code you can do things like

• Generate simple files with basic templating.
• Orchestrate other tasks imperatively and/or declaratively.
• Make HTTP requests and parse responses.
• Send emails.

Once inside your build container, the sticking with Off the Shelf motivates the developer to use the features and tools of their existing project frameworks and toolchains instead of a disjoint polyglot approach for complex tasks.

Future articles will get into this with concrete examples and links to resources.

Tattoo Your Binaries and Packages

You need to have traceability in your binaries form of code checkout revision.

At a minimum you can embed items like the git commit in custom metadata in .NET assemblies, Java jars, Go binaries, and Windows C++ executables. For other languages not supporting that, at worst a command line switch like -v, --version can be added to output this information using code generation to create a code file that gets linked into the executable.

In the case of scripting languages, Python packages, Node packages, and Ruby gems all support adding this type of metadata.

Git, Mercurial, and Subversion all offer ways to get repository revision information for build traceability. If you are using one that does not, here is your encouragement to get off of that tool.