My website can still be found at industrialcuriosity.com, but I have not been posting on this blog as I've been primarily focused on therightstuff.medium.com - please head over there and take a look!
And removing a single domain certificate without breaking everything else
Do you operate multiple domains from the same webserver? Do you have a webserver operated by NGINX? Do you have Certbot managing your certificates? This is a set of instructions for creating your certificates correctly and removing a single domain from your configuration, after I found some confusing ones that resulted in me knocking out my server for a little while…
A note before we begin
If you’re rather in a hurry to remove a domain from a messy configuration, STOP. Re-organizing your sites and regenerating your certificates is not only pretty quick and mostly painless — and required, if you want to remove a single domain without making NGINX break down and throw a wobbly — it’s very much the same process.
Organizing your existing NGINX sites
Ensure that you know which domains are configured in which site files, in particular make sure that you do not include servers for multiple domains in the same file.
To do this, look through your enabled site files under /etc/nginx/sites-enabled to find relevant server entries. While you’re there, you might want to note any certificates which are already used by those server entries; those will be the lines starting with ssl_certificate.
If you need to reorganize your site files, remember that their actual location must be in the /etc/nginx/sites-available path. To enable a site /etc/nginx/sites-available/example, create a symlink in the /etc/nginx/sites-enabled path with
This will generate a new certificate if needed and update the site file accordingly.
To ensure that everything is as it should be, review the updated site files and then validate them with
> sudo nginx -t
To restart NGINX once you’re ready, run
> sudo service nginx restart
Removing obsolete domains and certificates
Now that your site files and certificates are configured correctly, it’s time to remove any obsolete certificates that are no longer referenced.
Run sudo certbot certificates to list the existing certificates, paying attention to their names as well as their certificate and key paths. These paths will be registered in your NGINX site files so you can review what’s active and required and be certain that the certificate(s) you’re removing are unused.
When you’re confident that a certificate example.com is no longer in use, simply remove it by running
Red Hat’s “Dependency Analytics” extension is fantastic, it’s powered by Snyk’s vulnerability database and when opening one of my projects’ dependency files* I immediately saw red and was able to click my way clear in a matter of minutes**.
* My current team has projects written in all four of the supported languages, the only thing I’m personally missing is an extension for Visual Studio “proper” for C#…
** Well, okay, one of the dependency suggestions included a breaking change, but the rest of them were trivial upgrades.
Well done, Red Hat, for making safety and security just a little bit easier!
On the 30th of September 2021, a heavily-used root certificate — DST Root CA X3 — expired. You can read all about it here.
According to a handful of forum posts and github issues I’ve come across, the change has caused a fair amount of pain to those unfortunates who failed to heed the warnings, but for most of us this really wasn’t a surprise. For our team, the expiration date came and went and we didn’t even notice! Until our primary in-house testing tool began failing its connection tests with the following:
Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
Our gRPC connection tests are written in Python (using the grpcio and grpcio-tools packages), and run on a variety of linux machines and Docker images. Hunting through the forums, it looked like upgrading to the latest versions of the grpcio dependencies should do the trick, but it didn’t.
At least not by itself.
We eventually determined that the problem was that DST Root CA X3 was still registered as a certificate authority, and it took so long to figure out how to remove it on Debian that I realized that I had to post about it:
To see if the DST Root CA X3 certificate is configured as a root authority, list the contents of your /etc/ssl/certs folder:
2. Edit /etc/ca-certificates.conf and insert a ! at the beginning of the name of the DST Root CA X3 certificate to flag it as removed:
> sed -i “s@^mozilla/DST_Root_CA_X3.crt@!mozilla/DST_Root_CA_X3.crt@” “/etc/ca-certificates.conf”
To update the certificates, run the following:
> sudo /usr/sbin/update-ca-certificates -f
Note that it must be fully qualified as the /usr/sbin directory is not in the PATH by default, and it might be necessary to install the ca-certificates package using apt. The “f” of the -f flag apparently stands for “fresh”.
3. Set the GRPC_DEFAULT_SSL_ROOTS_FILE_PATH environment variable, which is required for the above changes to be respected:
If you’re not using a password manager by now, you should be. Ever since reading the xkcd: Password Strength comic many years ago, I’ve become increasingly frustrated by how the software industry has continued to enforce bad password practices, and by how few services and applications apply best practices in securing our credentials.
The main reason for password reuse or using poor passwords in the first place is because it’s way too hard to remember lots of good ones.
By forcing us to remember more and more passwords with outdated rules such as demanding symbols, numbers and a mix of uppercase and lowercase characters, most people have turned to using weak passwords, or reusing the same passwords or patterned recombinations of those passwords and leaving us vulnerable to simple exploits.
I recently learned about ‘; — have i been pwned?, and I was shocked to discover that some of the breaches that included my personal data included passwords that I had no idea were compromised… for years. Then I looked up my wife’s email address, and together we were horrified.
Lots of those compromised credentials were on platforms we didn’t even remember we had accounts on, so asking us what those passwords were and whether we’ve reused them elsewhere is futile.
A developer’s perspective
As an experienced software engineer, I understand just enough about security to be keenly aware of how little most of us know and how important it is to be familiar with security best practices and the latest security news in order to protect my clients.
I will never forget that moment a few years back when, while working for a well-established company with many thousands of users and highly sensitive data, I came across their password hashing solution for the first time: my predecessor had “rolled his own” security by MD5 hashing the password before storing it… a thousand times in a loop. As ignorant as I was myself regarding hashing, a quick search made it clear that this was making the system less secure, not more.
This was a professional who thought he was caring for his customers.
In 2019 I put together an open-sourced javascript package, the simple-free-encryption-tool, for simple but standard javascript encryption that’s compatible with C#, after finding the learning curve for system security to be surprisingly steep for something so critical to the safe operations of the interwebs.
The biggest takeaways from my little ventures into information security are as follows:
Most websites, platforms and services that we trust with our passwords cannot be relied upon to protect our most sensitive information.
Companies should not be relying exclusively on their software developers to protect customer credentials and personal data.
As a consumer, or customer, or client, we need to take responsibility for our passwords and secrets into our own hands.
Trust (almost) no-one.
What’s wrong with writing down my passwords on paper?
It’s so hard to remember and share passwords that lots of people have taken to recording them on sticky notes, or in a notebook, and I cannot stress enough just how dangerous a practice this is.
First, any bad actor who has physical access to your desk or belongings and (in their mind) an excuse to snoop on you or hurt you, will generally be privy to more of your personal data than some online hacker who picks up a couple of your details off an underground website. This means that it will be far easier for them to get into your secrets and do you harm.
Second, and far more likely, if those papers are lost or damaged you’re probably going to find yourself in hot water. For example, I’ve run into trouble with my Google credentials before and locked myself out of my account, and even after providing all the correct answers it was still impossible for me to get back in. There are many faceless services like this, so even a simple accident (or just misplacement) and you could find yourself in a very uncomfortable position.
What is a password manager?
A password manager is an encrypted database that securely stores all of your secrets (credentials or others) and enables you to retrieve them with a single set of credentials and authentication factors. Modern password managers tend to provide the ability to synchronize these databases on multiple devices and even inject your credentials directly where you need them.
Things to consider when picking a password manager
Standalone, cloud-based, or self-hosted
For individuals who aren’t prepared to trust the internet (or even their local networks) with their secrets, there are password managers that are designed to be stored and accessed locally. These are essentially interfaces to encrypted database files that reside on your local hard disk, and you are responsible for backing them up and copying them between devices. A word of caution: if you’re synchronizing these databases by uploading them to a file sharing service like Dropbox, you’re operating in a way that’s likely less secure than using a cloud-based service.
Cloud-based solutions are services provided by an organization that allows you to store your secrets on their platforms and trust in their experts to secure them. While user costs may vary, they don’t require any effort when it comes to maintenance, syncing between devices and backing up and they usually provide great interfaces with integrations for desktops, browsers and mobile phones.
An important aspect to take into consideration when it comes to cloud-based solutions is the provider’s reputation and history of breaches. Nobody’s perfect in the world of security — security is a perpetual arms race between the white hats and the black hats — but what speaks volumes is how an organization comports itself when things go wrong. Do they consistently apply best practices and upgrades? Do they react to breaches quickly, transparently, and in their clients’ best interests?
Self-hosted solutions are where you or your organization are required to install and maintain the service on a web server, preferably on a secure internal network, so that your users (your family or coworkers) can operate as if it’s a cloud-based solution. These are generally cheaper for businesses, but somewhat more difficult to maintain and often less secure than cloud-based solutions (depending on the competence of whoever’s responsible for your network), but from a user’s point of view it amounts to the same thing.
Password sharing for family and teams
Some people need to share credentials more than others. In my family, my wife and I are consistently sharing accounts so it doesn’t make sense for us to have individual duplicate copies of our shared accounts in each of our password accounts, and the same goes for me and my coworkers when it comes to our developer and administrator passwords for some of our products and service accounts. For these uses, it’s a good idea to use a solution that facilitates password sharing, and some of the services make it easy to set up groups and group ownership of credentials.
Mobile, OS and desktop browser support
Many password managers provide varying levels of integration for the wide variety of devices and browsers available — some solutions simply won’t give you any more than the barest essentials. Some people prefer to be able to unlock their passwords using biometrics, some prefer not to use their mobile devices at all, so before looking at the feature comparisons it’s worth giving a minute or two of thought towards how you intend to use it.
The good news is that most of the major solutions allow exporting and importing of your secrets, so if you have any doubts about your decisions you probably won’t have to worry too much about being locked in.
Free vs Paid
While pricing is obviously an important factor, I feel like one should first have an idea of what features one needs before comparing on pricing. Most of the solutions offer similar prices per user, with some exceptions.
This is one of those rare situations where, depending on your requirements, you might actually be better off with a free product!
With the wide variety of needs and options available, each solution listed above has its benefits and its tradeoffs. I hope you’ve found this helpful, if you have any questions, corrections, comments or suggestions I look forward to reading them in the comments below!
I've spent the last few days wrestling with the serverless framework, growing fonder and fonder of AWS' CDK at every turn. I appreciate that serverless has been, until recently, very necessary and very useful, but I feel confident in suggesting that when deployment mechanisms become more complex than the code they're deploying things aren't moving in the right direction.
That said, this post is about overcoming what for us was the final hurdle: getting Docker able to connect to a private PyPI repository for our Python lambdas and lambda layers.
Python is great, kind of, but is marred by two rather severe complications: the split between python and python3, and pip… so it's a great language, with an awful tooling experience. Anyway, our Docker container couldn't communicate with our PyPI repo, it took way too long to figure out why, here's what we learned:
The Solution
If you want to use a private PyPI repository without typing in your credentials at every turn, there are two options:
1.
~/.pip/pip.conf:
2.
~/.pip/pip.conf:
~/.netrc:
It is important that ~/.netrc has the same owner:group as pip.conf, and that its permissions are 0600 (chmod 0600 ~/.netrc).
What's not obvious - or even discussed anywhere - is that special characters are problematic and are handled differently by the two mechanisms.
In pip.conf, the password MUST be URL encoded.
In .netrc, the password MUST NOT be URL encoded.
The Docker Exception
For whatever reason, solution 2 (the combination of pip.conf and .netrc) does NOT work with Docker.
Conclusion
Amazon's CDK is excellent, and unless you have a very specific use-case that it doesn't support it really is worth trying out!
Oh! And that Python is Very Nice, but simply isn't nice enough to justify the cost of its tooling.
after hitting a wall last night with the existing node.js mysql packages, which for some reason have been years behind integrating with mysql 8's authentication, i discovered @mysql/xdevapi. the documentation's a bit heavy and not really comprehensive, but now that i've figured out how to use it, i'm very pleased with how it operates! unfortunately, i'm going to have to roll my own migrations management, but that's a small price to pay for being able to interface with the latest mysql databases in an intelligent way.
in the interests of reducing friction for other adopters, i've rolled some sample queries into my database creation script. enjoy!
UPDATE: i've subsequently learned that table joins have not been implemented, so to perform those you'd have to use the session.sql method and use it in the same way (the same promise chaining) as the CRUD methods. seems like a serious oversight, but whatever.
This makes strong RSA key generation and encryption and AES encryption simple to implement, and practical to use for client-server communications. It also allows me to give someone an encryption package on a stick (just copy the dist folder) that they can open on any machine.
[THIS ARTICLE IS OBSOLETE: you'll find better ones here and here]
if there are devs i really admire, it's everyone behind the software listed in this guide. they're out there making the internet a safer place and saner place, building easy-to-use tools and providing their services and products for free.
today i needed to self-sign certificates, and while there are good guides available they make a lot of assumptions or use complicated tools. here's what i figured out this morning after a long struggle with scripts that windows doesn't like:
1. install openssh for windows, and make sure to remember where the installation directory is. there are a number of options available from the openssl wiki, shining light productions' version is the most official. download the default build (the larger installation file) paying attention to whether your system is 32-bit or 64-bit.
2. install babun (bash and zsh on windows for people who don't want to micromanage their software)
3. using babun, change to the openssl bin directory. run the following command from letsencrypt: openssl req -x509 -out localhost.crt -keyout localhost.key -newkey rsa:2048 -nodes -sha256 -subj '/CN=localhost' -extensions EXT -config <( printf "[dn]\nCN=localhost\n[req]\ndistinguished_name = dn\n[EXT]\nsubjectAltName=DNS:localhost\nkeyUsage=digitalSignature\nextendedKeyUsage=serverAuth")
once installed in your app you'll be able to access with http or https - your browser will warn you that the certificate isn't signed but it's yours, so just accept it and get back to work!
There's a vast amount of documentation available, but all of it assumes a lot of prior knowledge or very particular use cases. Here's a simple and straightforward guide to creating and using a service principal for an Azure Key Vault so that your secrets can be managed programmatically.
Please note: I'm excited because I've finally managed to authenticate using a secret, it's probably more secure to use certificates but I'll get to that another time.
Step 1: Registered App
Under Azure Active Directory in the Azure Portal, select App registrations.
Add a New application registration, the application type being Web app / API and the Sign-on URL anything being any valid URL (just the format, it doesn't have to exist). If the name you enter isn't simple to remember then it would be a good idea to take a note of it for step 2.
Take note of the Application ID as that will be your Client ID for authentication, then select the Keys blade under Settings. Enter a Key description (preferably indicating the user or application that will be using this key), select a duration and Save. Immediately store the resulting value somewhere safe as it will never be displayed again.
I recommend adding yourself as an owner on the Owners blade (also under Settings), whatever else this may be good for it'll let you see the app registration immediately on the App registrations blade without having to select "All apps".
Step 2: Key Vault permissions
Open the Key Vault in the Azure Portal and select the Access policies blade under Settings. Click Add New and click on Select principal - you'll have to enter the full name of the registered app you created in the previous step in the search box before it'll show up, at which point you'll be able to select it.
You can either select an appropriate template from the top dropdown or choose Key, Secret or Certificate permissions manually. Don't worry about Authorized application at this stage.
IMPORTANT: pressing the OK button will add your new policy to the list, but it will not be saved! Be sure to click Save before continuing.
Step 3: Accessing the Key Vault from your Code
There are many different ways to authenticate, most of them obscure and undocumented. This is the simplest method, I've put the credentials in the code for clarity but I have faith that you'll store them somewhere more intelligent. Never store credentials in the codebase. Seriously. Just don't.
Let's Encrypt is brilliant, but why site deletion makes things so complicated is beyond me. I had a bunch of sites running on Apache that Let's Encrypt automagically generated certificates for, which was fine right up until I needed to make one of them unavailable. What follows are the instructions for regenerating the certificates quickly and (relatively) painlessly.
Ensure that Apache site configurations are removed from /etc/apache2/sites-enabled
Move unwanted site configurations from /etc/apache2/sites-available to a backup location if needed
Remove all certificate files from /etc/letsencrypt/archive/ /etc/letsencrypt/live/ /etc/letsencrypt/renewal/
The struggle to uncover the secrets of importing from and export to OpenSSH keys with Microsoft's .NET RSACryptoServiceProvider is real. It's possible but not practical to do this without BouncyCastle, which may or may not be well-documented (navigating their website is far from a joyful experience), but after trawling the web and playing around I've created the following gists that should be of assistance to anyone who needs to do this in a straightforward manner.
And if you want to share RSA keys between JavaScript and .NET platforms, well, you're going to need to do this. Import and export RSA Keys between C# and PEM format using BouncyCastle
And just because the actual encryption and decryption are always annoying: Simple RSA Encryption to and Decryption from Base64 encoded strings in C#
There are many cases where HTTPS / SSL isn't an option, and usually when that happens everything, including sensitive data, is sent plain text. Additionally, most websites still use passwords instead of external authentication and password policies and storage that were outdated years ago.
The risk is entirely unnecessary.
Implementing the most crucial elements of HTTPS with a combination of client and server security is non-trivial but inexpensive, and can protect your users and your organization from most types of attacks. The following protocol is based on existing protocols and algorithms - the first rule of crypto is "don't roll your own" - and as I am a crypto amateur I believe I can explain the steps in terms simple enough for most developers.
I have illustrated the implementation of this protocol in pure Javascript and Node.js here.
1. Session initiation
The only plaintext transmission is the initial transfer of the username / email address.
As a session initiation request, the client transmits the username / email address to the server, which is used to create a server-side session object identified by a token (preferably a GUID).
The server looks up the user's email address and any other authentication factor that can receive messages (ie other email addresses, mobile devices, etc) and sends them a verification code which will be used as a shared secret when authenticating. To this end the verification code must be stored in the session object.
This verification code must never be returned to the client directly.
The server responds to the session initiation request with the session identification token.
2. Session authentication
The first step to properly securing the communication is exchanging RSA public keys which will be used to protect the AES passphrases. Because the public keys must be exchanged before they can be used, the verification code sent to the user in the previous step will be the first AES passphrase.
AES 256 requires a passphrase 32 bytes (256 bits) long. We convert the verification code to 32 bytes by hashing it with the MD5 algorithm.
From this point on, every transmission will be encrypted using AES (specifically AES-256 in CTR mode). Every transmission must include the session token and an encrypted payload object. The payload object must include a timestamp, to ensure that the transmission was received in an acceptable amount of time, and a nonce, which is a randomly generated string used to prevent replay attacks.
The client generates an RSA private / public key pair and adds it to the payload. For browser calls, store the client keys in an HTML Local Storage Object.
At this point it is possible to include the user's password in the payload if one is required for authentication, although it is advisable to rely on other forms of authentication instead such as external applications (Duo, Authy), OAuth providers (Google, Facebook) or whatever else is available and practical with regards the user experience.
The payload is encrypted using the hashed verification code and is sent to the server along with the session token.
The server decrypts the payload using the hashed verification code for the session and validates the payload. Validating the payload includes testing that the timestamp isn't too old and the nonce against a list of nonces that the session has accepted.
If validation is successful the session stores the client's public key and generates its own RSA key pair which will be used until the session expires.
The server generates a new AES passphrase and uses it to encrypt the session's public key. This will be returned to the client along with the AES passphrase encrypted using the client's public key. At this point only the client will be able to decrypt the AES passphrase protecting the session's public key.
3. Closure
The client decrypts the AES passphrase and stores the session's public key. From this point on, every transmission to the server will include the following:
a) the session token
b) a newly generated AES passphrase (or key) encrypted using the session's public key
c) the payload, including timestamp and nonce, encrypted using the generated AES passphrase
And from this point on the response format will be as follows:
a) a newly generated AES passphrase (or key) encrypted using the client's public key
b) the payload encrypted using the generated AES passphrase
Even when HTTPS is available, with the current computing power available the performance cost of this form of encryption is quite small and can add an extra layer of protection.
I have illustrated the implementation of this protocol in pure Javascript and Node.js here.
If you have any questions, suggestions or criticism, please drop me a line in the comments!
