Managing site certificates with NGINX and Certbot

And removing a single domain certificate without breaking everything else

Do you operate multiple domains from the same webserver? Do you have a webserver operated by NGINX? Do you have Certbot managing your certificates? This is a set of instructions for creating your certificates correctly and removing a single domain from your configuration, after I found some confusing ones that resulted in me knocking out my server for a little while…

A note before we begin

If you’re rather in a hurry to remove a domain from a messy configuration, STOP. Re-organizing your sites and regenerating your certificates is not only pretty quick and mostly painless — and required, if you want to remove a single domain without making NGINX break down and throw a wobbly — it’s very much the same process.

Organizing your existing NGINX sites

Ensure that you know which domains are configured in which site files, in particular make sure that you do not include servers for multiple domains in the same file.

To do this, look through your enabled site files under /etc/nginx/sites-enabled to find relevant server entries. While you’re there, you might want to note any certificates which are already used by those server entries; those will be the lines starting with ssl_certificate.

If you need to reorganize your site files, remember that their actual location must be in the /etc/nginx/sites-available path. To enable a site /etc/nginx/sites-available/example, create a symlink in the /etc/nginx/sites-enabled path with

> ln -s /etc/nginx/sites-available/example.com /etc/nginx/sites-enabled/

and to disable a site, remove it from the /etc/nginx/sites-enabled path with

> rm /etc/nginx/sites-enabled/example.com

Generating certificates with Certbot

Once your sites are organized in a way that each domain has its own file, generate certificates for each domain and its subdomains with

> sudo certbot --nginx -d example.com -d www.example.com

This will generate a new certificate if needed and update the site file accordingly.

To ensure that everything is as it should be, review the updated site files and then validate them with

> sudo nginx -t

To restart NGINX once you’re ready, run

> sudo service nginx restart

Removing obsolete domains and certificates

Now that your site files and certificates are configured correctly, it’s time to remove any obsolete certificates that are no longer referenced.

Run sudo certbot certificates to list the existing certificates, paying attention to their names as well as their certificate and key paths. These paths will be registered in your NGINX site files so you can review what’s active and required and be certain that the certificate(s) you’re removing are unused.

When you’re confident that a certificate example.com is no longer in use, simply remove it by running

> sudo certbot delete --cert-name example.com 

...

Originally published at https://therightstuff.medium.com.

A VSCode extension to make your code more secure

 I recently installed Red Hat’s YAML VS Code extension to assist me with Bamboo Specs, convinced by the Bald Bearded Builder that this was the linter for me (check out its schema support!). I don’t usually appreciate extensions recommending things to me (and, to be fair, I don’t know that that’s precisely what happened), but this morning a toaster popped up suggesting that I install their Dependency Analytics extension and I am SO glad that I clicked on it!

Red Hat’s “Dependency Analytics” extension is fantastic, it’s powered by Snyk’s vulnerability database and when opening one of my projects’ dependency files* I immediately saw red and was able to click my way clear in a matter of minutes**.

* My current team has projects written in all four of the supported languages, the only thing I’m personally missing is an extension for Visual Studio “proper” for C#…

** Well, okay, one of the dependency suggestions included a breaking change, but the rest of them were trivial upgrades.

Well done, Red Hat, for making safety and security just a little bit easier!

...

Originally published at https://therightstuff.medium.com.

A quick-start guide to setting up a Debian guest on VMWare WorkStation 15/16 Player

I don’t know why everything needs to be subtly non-standard, but over the course of the last twenty or so virtual machine reconstructions I’ve come up with a simple checklist for setting up a Debian guest on VMWare’s WorkStation (Windows) and I thought I’d share it here.

1. Download your Debian .iso here (I recommend the netinst CD image)
2. Create a new virtual machine using the downloaded .iso and check that the default configuration is satisfactory (I tend to need a little more power, usually 2 CPUs does it for me). Note that increasing the size of a hard disk is for more complicated and risky than a regular user would expect, so give yourself a healthy buffer. In my experience, it’s less painful to rebuild a bigger machine than it is to extend the disk size.
3. Install the OS — I find the graphical installer to be just fine for my purposes. The two configuration options that are the most impactful are your choice of desktop environment, I usually choose Xfce but I’m starting to like GNOME again. It’s probably a good idea to install the ssh server as well.
4. Once installation is complete, click the VMWare button at the bottom of the screen to signal that and restart the machine.
5. To grant yourself the ability to use sudo, open the terminal and run either
   > su -
   or
   > su -c ‘su -’
   if the first isn’t allowed (I’m not sure how to use the correct single quotes in Medium, so note that the above apostrophes aren’t correct).
   Then run
   > usermod -aG sudo <username>
   to add yourself to the sudoers group. You will need to log out and back in for this to take effect.
6. Install the following to be able to install VMWare Tools, which enables things like copying and pasting between host and guest machines:
   > sudo apt install -y open-vm-tools open-vm-tools-desktop linux-source

Installing VMWare Tools in WorkStation Player 16

Open the Virtual Machine Settings, select the Options tab, then select VMWare Tools: select “Sychronize guest time with host” and “Update automatically”, then restart the virtual machine.

Installing VMWare Tools in WorkStation Player 15

1. Open the VM menu, select Install VMWare Tools.
2. Mount the VMWare Tools CD:
   > sudo mount /dev/cdrom
3. Extract the installer to your current directory (or maybe create a subdirectory for it) using tab auto-complete in place of the ellipse:
   > tar -xf /media/cdrom/VMWareTools…
4. Install the required build tools:
   > sudo apt-get install -y autoconf automake binutils cpp gcc linux-headers-$(uname -r) make psmisc
5. Try to run the .pl script in the extracted folder, expect it to fail, restart the machine anyway.

At this point you should have your VM up and running and be able to copy / paste / drag files between your machines. Now go grab yourself another cup of coffee, you deserve it! 

...

Originally published at https://therightstuff.medium.com.

An Impatient Developer’s Guide to Debian Maintenance (Installation) Scripts and Package Diverts

Photo by Andrea Piacquadio from Pexels

 The people involved in coming up with the dpkg scheme for installing / upgrading / downgrading / removing packages are very clever. While unintuitive to the uninitiated, the scheme is mostly logical and reasonable, though there are some points where I feel a little more effort and consideration could have made a world of difference.

“The evil that men do lives on and on” — Iron Maiden

In addition to regular installation behaviour, I needed to wrap my head around “package diverts”, which is a very clever system for enabling packages to handle file conflicts. Except that it doesn’t handle what I would consider to be a very basic use case:

1. Install an initial version of our package.
   
   
2. Discover that our package needs to overwrite a file that’s installed by an upstream dependency.
   
   
3. Create a new version of our package that includes the file and configures a “package divert” to safely stow the dependency’s version.
   
   
4. Remove the “package divert” on the file if the newer version of the package is uninstalled or downgraded to the previous version that doesn’t include it.

That last part, in italics? That’s the kicker right there. Read on to understand why.

Debian Installation Script Logic In Plain English

After poring over the Debian maintainer scripts flowcharts, I felt I had a pretty good handle on things but there’re a couple of little “gotcha”s, so I feel like it’s worth providing a brief summary of the general flow in common English.

Debian maintenance scripts are run by apt and dpkg at specific points in the installation process:

https://www.debian.org/doc/debian-policy/ap-flowcharts.html

1. If the package is being removed or upgraded (meaning that a different version is being installed, “upgraded” also means “downgraded” to Debian maintainers), the previously installed version’s prerm script is called.
   
   
2. If the package is being installed or upgraded, the new version’s preinst script is run.
   
   
3. If the package is not being removed, the new version’s package contents are unpacked. This will overwrite existing files if they were installed by the same package, or fail the installation if it attempts to overwrite another package’s files without a package divert configured.
   
   
4. If the package is being removed, its files are deleted.
   
   
5. The previously installed version’s postrm script is called if the package is being removed or upgraded.
   
   
6. If the package is not being removed, any package contents belonging to the previous version that do not also exist in the new one are removed.
   
   
7. If the package is not being removed, the new version’s postinst script is called.

It is important to note that if a maintenance script fails with a non-zero exit code, the package will be in a broken state that can be very difficult (sometimes impossible) to recover. From our experience, it’s best to catch all exceptions, log them, “exit gracefully” with an exit code of 0, and hope for the best.

Also from our experience, it’s a good idea for maintenance scripts to log everything to the stderr stream in order to preserve chronological order.

Debian Package Diverts for the Uninitiated

The principle of Debian package diverts is straightforward enough: when you want to include a file in your package contents that conflicts with another package’s file (i.e. the absolute paths are identical), you create a “divert” on that file so that any other package’s versions of that file is “diverted” to a different file name.

Creating a Package Divert

To create a package divert, your package’s preinst script should run the following command:

dpkg-divert --package my-package-name --add --rename \

    --divert /path/to/original.divert-ext /path/to/original

The preinst script is the place to do this because the divert must be in place before the package contents are unpacked. The dpkg-divert commands are idempotent, so having it called in the preinst of every install is fine.

Removing a Package Divert

When your package is uninstalled, it’s good practice to remove the package divert and rename the diverted files back to their original file names. It’s recommended to remove the package divert in the postrm script, which makes perfect sense when uninstalling a package because the files are deleted before postrm is called.

dpkg-divert --package my-package-name --remove --rename \

    --divert /path/to/original.divert-ext /path/to/original

When removing a package, the package’s files have been deleted already and removing a divert simply renames the diverted file back to its original file name.

When upgrading a package, however, the files are only deleted after postrm has been called. This means that a call to dpkg-divert — remove will fail because it would have to overwrite the upgraded package’s copy of the file that hasn’t yet been removed.

It also means that if you delete your package’s file in the postrm in order to remove the divert, the original package’s file will be deleted after your postrm because it will have been identified as belonging to the upgraded package.

“Insanity is contagious.” ― Joseph Heller, Catch-22

It is for this reason that if we remove the divert in the postrm during a downgrade to a version that does not include the file in its package contents, we will lose the original file. If we do not remove the package divert, we will retain the diverted original file, but it will be renamed and therefore not serve its purpose. In our downgrading scenario, the postinst script that’s run after the file removal phase of an installation belongs to the older version of the package that didn’t know about the file, or package diverts, so that won’t be of any use to us. In short, the only way to downgrade our package is to completely remove it and install the older version, which for us is simply not an option.

Epilogue

Fortunately, my team and I are in the position that the original file also belongs to a package that we maintain, and we are able to overwrite the original file in the postinst script* with confidence and impunity. That means no rolling back without removing and then reinstalling the original package, which in our case happens to be impossible.

* Of course, the file can no longer be included in the package contents with its original path or the installation will fail.

Hope you’ve found this helpful! Please share in the comments if you’ve had similar experiences, or if you know of any other workarounds!

...

Originally published at https://therightstuff.medium.com.

Bamboo YAML Specs Tips and Tricks (For Fun and Profit)

[UPDATED 2021.10.01 WITH ADDITIONAL DEPLOYMENT PLAN LEARNINGS]

Introduction

“In for a penny, in for a pound” — that’s our team’s current approach regarding Atlassian’s products. As part of our efforts to code as much of our infrastructure as possible, we’ve recently begun migrating old build plans to Bamboo Specs and, as non-Java devs who don’t want more headaches than are absolutely necessary, we’ve chosen to work with the Bamboo Specs YAML offering.

YAML specs are pretty great. Migrations are assisted by the fact that manually-created plans will show you their YAML translations, so in most cases a simple copy/paste is all you need. I mean, aside from migrating to linked repositories and ensuring they’re configured correctly…

Having said that, Bamboo’s YAML specs are an incomplete product with undocumented critical features, and fail to provide out-of-the-box support for a surprising number of standard use cases that one would expect software engineers building software for other software engineers to appreciate the value of. 

They’re still pretty great in spite of that, but overcoming the limitations of incomplete specs definitions and deployment plans is not exactly intuitive. This article attempts to cover some of the missing pieces and suggest some workarounds that we’ve found useful.

Configuring a linked repository for Bamboo Specs

Bamboo Specs require the configuration of a linked repository. Head to settings -> linked repositories (this will require admin permissions) to create or update a linked repository configuration. The two most important steps to be taken here are as follows:

1. Determine which branch of your repository will be used by Bamboo to read the specs file. Only one branch can be considered the source of truth, my personal recommendation is to make it the development branch.
   
   
2. Two sets of permissions need to be configured correctly in order for Bamboo Specs to be able to do their job:
   
   a. First, the Bamboo project must give permissions to the linked repository to create and modify the build and deployment plans. Head to the project you want your linked repository to operate in, go to Project Settings, then Bamboo Specs repositories, and add the linked repository.
   
   b. Under the Bamboo Specs tab of the linked repository, enable both Access all projects and Access all repositories. I’m pretty confident that this requirement is not a security best-practice, but in my experience Bamboo Specs won’t work without them.

Testing Bamboo Specs

When experimenting with changes to the specs, I’ve found it useful to do the following:

1. Create a testing branch (a feature / bugfix branch).
   
   
2. Change the build and deployment plans’ names and keys and push them to origin so that your changes won’t overwrite the existing plans.
   
   
3. Set the testing branch to be the linked repository’s branch in the General tab and proceed with your changes.
   
   
4. Once you’re happy with the changes and they’ve been reviewed: set the linked repository’s branch back to the development branch, then revert the plan names and keys in the testing branch to the existing plans’.
   
   
5. Delete the test plans manually.

Unexpected Limitations of Bamboo Specs

Plan dependencies

The most glaring omission in the YAML specs is the inability to set up plan dependencies. While we were initially upset by this, we quickly realized that at the end of the day plan dependencies are a nice hack that we shouldn’t really have been relying on in the first place. Bamboo appears to encourage the download of artifacts from matching branches on other build plans, but this can quickly break down into chaos with dependency versioning and management. I warmly recommend using Bamboo artifacts for build debugging and deployment plans exclusively, and proper package repositories for storing and retrieving versioned build artifacts.

When exporting Bamboo Specs from existing plans, build plans and deployment plans are not considered strongly related so you will need to gather and combine the specs from both into a single file. To do this, copy in the deployment plan specs beneath the build plan specs, retaining the --- as separators.

NOTE: the ordering of the sections is important! The build plan definition must be followed by the build plan permissions, then the deployment plan definition(s), with each deployment plan definition followed by a section for its permissions. See the outline at the end of the article for clarity.

An interesting omission is the inability to include unset plan variables. This actually makes sense, as manually maintained plans need some way to ensure that they’re all using the same variable names, but with Bamboo Specs it’s really on you to be consistent and it’s obviously much* easier to search through a single file than it is to hunt for variables across different plan branches via the Bamboo interface.

* infinitely easier

Deployment Plans — sharing build variables and tooling

The principal idea behind a deployment plan is to separate deployment from the build process. Bamboo implements deployment plans as distinct entities with entirely different environments, with the intention that your only interaction with the related build plan is to download your artifacts from it.

For us, this proved problematic as we require shared environment variables and tooling to deploy our builds. To work around this, we required the following mechanisms:

1. Environment variable injection. Early in our build plan tasks, we prepare an environment variable file in the following format and include values like build versions and git branch names.
   
   version=1.0.2
   git_branch=feature/example
   
   WARNING: variable values MUST NOT be surrounded by quotations, as this leads to unpredictable behaviour.
   
   It is recommended to use the inject namespace for injections. When the scope of the injection is RESULT, the variables will be available to all subsequent build tasks as well as the attached deployment plan. In Bamboo Specs they’ll be available in the form ${bamboo.inject.git_branch} and in inline scripts as $bamboo_inject_git_branch (on Unix agents) or %bamboo_inject_git_branch% (on Windows agents).
   
   One of my favourite uses of this technique is the ability to name releases automatically based on the build version (see the following section for the example).
   
   
2. Deployment plans are not really designed to use git directly, but we have found that we sometimes require non-build folders to be available for deployment, such as documentation. In these cases, we simply zip the desired folders and make them available as artifacts as well.
   
   
3. Running the deployment in a docker container. I find it disconcerting that such an extremely useful feature is undocumented! Deployment plan environments can be configured to run in a docker container just like a build plan, which provides us with all requisite tooling and context.
   
   DevEnvironment:
     docker:
       image: golang:1.16.6-buster
       docker-run-arguments:
       - --net=host
     tasks:

Linking multiple branches to a single deployment plan

Ironically, while deployment plans are supposed to operate independently from build plans, they only really function well when linked to specific build branches. If your intention is to build once, then deploy the build artifacts to multiple stages, you're out of luck!

UPDATE: it turns out we missed an important option! If release-naming is set to an environment variable, it only works for the specified branch (even if that specified branch is the default branch). If you want release-naming to be set to an environment variable for any branch, then it needs to be configured as follows:

release-naming:
  next-version-name: ${bamboo.inject.version}
  applies-to-branches: true

The disadvantage of using a single deployment plan is that the link to the deployment plan will only be available from the default build plan branch, but in my experience this is a very minor price to pay for the simplicity. The alternative - a single deployment plan for each branch of interest - is not only messier, but is also annoying to configure as you have to know the branch keys in advance so the branches cannot be automatically managed (plan branch keys are autoincremented and uneditable).

Regardless of your choice, it's probably a good idea to handle your branch management manually:

branches:
  create: manually
  delete: never

Putting it all together

My recommendation for the general outline of a YAML specs file is as follows:

---

version: 2

# build plan definition

plan:

  project-key: PROJECTKEY

  key: PLANKEY

  name: Product Name

  ...

branches:

  create: manually

  delete: never

---

version: 2

# build plan permissions

plan:

  key: PROJECTKEY-PLANKEY

plan-permissions:

  ...

---

version: 2

# deployment plan definition

deployment:

  # NOTE: deployment plan names must be unique

  name: Product Name

  source-plan: PROJECTKEY-PLANKEY

release-naming:
  next-version-name: ${bamboo.inject.version}
  applies-to-branches: true

...

---

version: 2

# deployment plan permissions

deployment:

  name: Product Name

deployment-permissions:

  ...

These are the tips and tricks that have helped us overcome our biggest migration challenges so far, I hope they can help others as well. If you have any others that come to mind, or improvements over the above, please let me know in the comments!

...

Originally published at https://therightstuff.medium.com.