Brutalist Builds - Bash & GNU Make Pearls
For brutalist builds it's best to keep complex imperative logic out of shell scripts and makefiles. In builds, the most important and common things I do in shell scripts and make files repeatedly are:
- Get the directory of the currently executing makefile or script. Given that we can easily locate project files, other scripts, etc. using dynamically generated absolute paths for orchestrating our builds.
- Properly handle an error, returning a non-zero exit code in-case of error. While make does this for you, one needs to make sure errors are propagated out of shell scripts.
- Output build metadata such as git revisions, git branches, CI environment variables, etc. into simple JSON, Yaml, or other format files that are easily consumed by downstream build steps. See the section on Stingy Templating for more on this.
- Parsing delimited strings, such as versions.
- Store the output of a command in a variable. In shell scripting, this is easy, but we can do this fairly easily in a dependent makefile target.
Given the above, use Bash and/or GNU Make for orchestration, but not much else.
Here are some gists for Bash and GNU Make. More or less, these are notes to myself and anyone else looking for some quick copy paste snippets.
Bash
Crash & Die on Error – but say something and cleanup on the way out serving up an error code
trap 'catch $? $LINENO' ERR
catch() {
if [ "$1" != "0" ]; then
echo "Error $1 occurred on $2"
cleanupthecrap --force
exit $1
fi
}
Usually this prints where the error occurred, and you get to cleanup any mess on the way out (substitute whatever for cleanupthecrap and anything more you need to stuff in before the exit $1). Here you'll return an error code to cause Jenkins, Make, etc. to error out.
Bash Heredoc
cat << EOF > index.html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<title>Hello, world!</title>
<meta name="viewport" content="width=device-width,initial-scale=1" />
<meta name="description" content="" />
<link rel="icon" href="favicon.png">
</head>
<body>
<h1>Hello, world!</h1>
</body>
</html>
EOF
and the below also works in the same manner. A boiler plate index.html files created in these 2 examples. Enclosing the EOF in single quotes, e.g. 'EOF' prevents variable expansion. Typically EOF is used as a delimiter string, however you can use something else.
cat > index.html << EOF
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<title>Hello, world!</title>
<meta name="viewport" content="width=device-width,initial-scale=1" />
<meta name="description" content="" />
<link rel="icon" href="favicon.png">
</head>
<body>
<h1>Hello, ${USER}!</h1>
</body>
</html>
EOF
Git Checkout Single File From Different Branch
Here is you you grab a file from another branch without switching over to that branch.
git checkout feature/some-other-branch -- ./some-file-or-dir
Of course, this works in PowerShell with PoshGit just as easily.
Getting the Script Directory
This sets the variable $DIR to the directory of the bash script with a fully qualified path. It will be the directory of the actual script, even if ran from a symbolic link referencing it.
DIR=$(dirname $(readlink -f $0))
Netcat for One Shot File Transfer
Here is a quick and dirty way of copying files between VM's, from VM to host, or host to host on internal network. Understand, this is not a secure way to copy files.
send it (example)
nc -l -p 8000 < cushy-goodies.tgz
The -p 8000 is for port 8000, pick whichever port you want. cushy-goodies.tgz is an example file.
fetch it (example)
nc -w 3 192.168.1.100 > cushy-goodies.tgz
The -w 3 is a 3 second timeout argument. You may not need it, the idea is to keep the client from hanging after the transfer. Use a resolvable hostname or raw ip address of whatever you are trying to copy the file out of.
Netcat for Directory Transfer
send it (example)
tar cf - cushy-dir/. | nc $desthost $port
fetch it (example)
nc -l $port | tar xf -
The -p switch can be used to preserve permissions
Split String
Splits string on a delimiter. Ref bash shell script split array
PythonVersion=3.11.1
arrIN=(${PythonVersion//./ })
MajorVersion=${arrIN[0]}
MinorVersion=${arrIN[1]}
PatchVersion=${arrIN[2]}
This example splits a version string, 3.11.1 into 3, 11, and 1, assigning them to their respective semver component variables.
Stingy Templating
For some stuff we just don't need to go as far as Go with it's build-in templating or a Python virtualenv with Jinja2 installed.
read -r -d '' some_variable <<EOF
${COOLCOMAND}
--switch $BIGVAL
--other-switch cushiness=$COMFYNESS
EOF
So set values for COOLCOMMAND, BIGVAL, and COMFYNESS and they get expanded into the value for some_variable.
Useful Linux Command Line Utilities
These are all common tools available on Linux and OS X. Basic knowledge of a few of these and how to use together will help with creathing clean, crisp and functional bash scripts.
- xmllint – xmllint in Linux | Baeldung on Linux for parsing XML files.
- jq – parse JSON files.
- netcat – Netcat: the TCP/IP swiss army.
- sed – You will come across this tool soon, it typically used for modifying text files. [Getting Started With SED Command [Beginner’s Guide]].(https://linuxhandbook.com/sed-command-basics/)
- awk – Another text processing tool, more powerful than sed, present in many various such as gawk, mawk and nawk. Difference Between awk, nawk, gawk and mawk | Baeldung on Linux
GNUMake
Get Current Makefile Filename and Directory
thismakefile := $(abspath $(lastword $(MAKEFILE_LIST)))
thismakefiledir := $(dir $(thismakefile))
Set Variable In Dependency Target
The demo below we are able to read the trimmed output of a shell script, placing it in a variable for later use. The trick is using eval to set the variable as shown for target second.
SHELL := /bin/bash
thismakefile := $(abspath $(lastword $(MAKEFILE_LIST)))
thismakefiledir := $(dir $(thismakefile))
FOO := "blah"
first:
@echo $(FOO)
second: first
$(eval FOO := $(strip $(shell $(thismakefiledir)/getquid.sh)))
third: second
@echo "---> $(FOO) <---"