ARCHIVE NOTICE

My website can still be found at industrialcuriosity.com, but I have not been posting on this blog as I've been primarily focused on therightstuff.medium.com - please head over there and take a look!

Monday 5 April 2021

Choosing the right password manager to keep your secrets safe

Photo by George Becker from Pexels

If you’re not using a password manager by now, you should be. Ever since reading the xkcd: Password Strength comic many years ago, I’ve become increasingly frustrated by how the software industry has continued to enforce bad password practices, and by how few services and applications apply best practices in securing our credentials. 

The main reason for password reuse or using poor passwords in the first place is because it’s way too hard to remember lots of good ones.

By forcing us to remember more and more passwords with outdated rules such as demanding symbols, numbers and a mix of uppercase and lowercase characters, most people have turned to using weak passwords, or reusing the same passwords or patterned recombinations of those passwords and leaving us vulnerable to simple exploits.

I recently learned about ‘; — have i been pwned?, and I was shocked to discover that some of the breaches that included my personal data included passwords that I had no idea were compromised… for years. Then I looked up my wife’s email address, and together we were horrified.

Lots of those compromised credentials were on platforms we didn’t even remember we had accounts on, so asking us what those passwords were and whether we’ve reused them elsewhere is futile.

A developer’s perspective

As an experienced software engineer, I understand just enough about security to be keenly aware of how little most of us know and how important it is to be familiar with security best practices and the latest security news in order to protect my clients.

I will never forget that moment a few years back when, while working for a well-established company with many thousands of users and highly sensitive data, I came across their password hashing solution for the first time: my predecessor had “rolled his own” security by MD5 hashing the password before storing it… a thousand times in a loop. As ignorant as I was myself regarding hashing, a quick search made it clear that this was making the system less secure, not more.
This was a professional who thought he was caring for his customers.

In 2019 I put together an open-sourced javascript package, the simple-free-encryption-tool, for simple but standard javascript encryption that’s compatible with C#, after finding the learning curve for system security to be surprisingly steep for something so critical to the safe operations of the interwebs.

The biggest takeaways from my little ventures into information security are as follows:

  1. Most websites, platforms and services that we trust with our passwords cannot be relied upon to protect our most sensitive information.
  2. Companies should not be relying exclusively on their software developers to protect customer credentials and personal data.
  3. As a consumer, or customer, or client, we need to take responsibility for our passwords and secrets into our own hands.
  4. Trust (almost) no-one.

What’s wrong with writing down my passwords on paper?

It’s so hard to remember and share passwords that lots of people have taken to recording them on sticky notes, or in a notebook, and I cannot stress enough just how dangerous a practice this is.

First, any bad actor who has physical access to your desk or belongings and (in their mind) an excuse to snoop on you or hurt you, will generally be privy to more of your personal data than some online hacker who picks up a couple of your details off an underground website. This means that it will be far easier for them to get into your secrets and do you harm.

Second, and far more likely, if those papers are lost or damaged you’re probably going to find yourself in hot water. For example, I’ve run into trouble with my Google credentials before and locked myself out of my account, and even after providing all the correct answers it was still impossible for me to get back in. There are many faceless services like this, so even a simple accident (or just misplacement) and you could find yourself in a very uncomfortable position.

What is a password manager?

A password manager is an encrypted database that securely stores all of your secrets (credentials or others) and enables you to retrieve them with a single set of credentials and authentication factors. Modern password managers tend to provide the ability to synchronize these databases on multiple devices and even inject your credentials directly where you need them.

Things to consider when picking a password manager

Standalone, cloud-based, or self-hosted

For individuals who aren’t prepared to trust the internet (or even their local networks) with their secrets, there are password managers that are designed to be stored and accessed locally. These are essentially interfaces to encrypted database files that reside on your local hard disk, and you are responsible for backing them up and copying them between devices. A word of caution: if you’re synchronizing these databases by uploading them to a file sharing service like Dropbox, you’re operating in a way that’s likely less secure than using a cloud-based service.

Cloud-based solutions are services provided by an organization that allows you to store your secrets on their platforms and trust in their experts to secure them. While user costs may vary, they don’t require any effort when it comes to maintenance, syncing between devices and backing up and they usually provide great interfaces with integrations for desktops, browsers and mobile phones.

An important aspect to take into consideration when it comes to cloud-based solutions is the provider’s reputation and history of breaches. Nobody’s perfect in the world of security — security is a perpetual arms race between the white hats and the black hats — but what speaks volumes is how an organization comports itself when things go wrong. Do they consistently apply best practices and upgrades? Do they react to breaches quickly, transparently, and in their clients’ best interests?

Self-hosted solutions are where you or your organization are required to install and maintain the service on a web server, preferably on a secure internal network, so that your users (your family or coworkers) can operate as if it’s a cloud-based solution. These are generally cheaper for businesses, but somewhat more difficult to maintain and often less secure than cloud-based solutions (depending on the competence of whoever’s responsible for your network), but from a user’s point of view it amounts to the same thing.

Password sharing for family and teams

Some people need to share credentials more than others. In my family, my wife and I are consistently sharing accounts so it doesn’t make sense for us to have individual duplicate copies of our shared accounts in each of our password accounts, and the same goes for me and my coworkers when it comes to our developer and administrator passwords for some of our products and service accounts. For these uses, it’s a good idea to use a solution that facilitates password sharing, and some of the services make it easy to set up groups and group ownership of credentials.

Mobile, OS and desktop browser support

Many password managers provide varying levels of integration for the wide variety of devices and browsers available — some solutions simply won’t give you any more than the barest essentials. Some people prefer to be able to unlock their passwords using biometrics, some prefer not to use their mobile devices at all, so before looking at the feature comparisons it’s worth giving a minute or two of thought towards how you intend to use it.

The good news is that most of the major solutions allow exporting and importing of your secrets, so if you have any doubts about your decisions you probably won’t have to worry too much about being locked in.

Free vs Paid

While pricing is obviously an important factor, I feel like one should first have an idea of what features one needs before comparing on pricing. Most of the solutions offer similar prices per user, with some exceptions.

This is one of those rare situations where, depending on your requirements, you might actually be better off with a free product!

The Feature Comparison

Standalone, cloud-based, or self-hosted

Password sharing for family and teams

Mobile, OS and desktop browser support

Free vs Paid

Summary

With the wide variety of needs and options available, each solution listed above has its benefits and its tradeoffs. I hope you’ve found this helpful, if you have any questions, corrections, comments or suggestions I look forward to reading them in the comments below!

Friday 2 April 2021

Simple safe (atomic) writes in Python3

 In sensitive circumstances, trusting a traditional file write can be a costly mistake - a simple power cut before the write is completed and synced may at best leave you with some corrupt data, but depending on what that file is used for you could be in for some serious trouble.

While there are plenty of interesting, weird, or over-engineered solutions available to ensure safe writing, I struggled to find a solution online that was simple, correct and easy-to-read and that could be run without installing additional modules, so my teammates and i came up with the following solution:

Explanation:

temp_file = tempfile.NamedTemporaryFile(delete=False,
                                        dir=os.path.dirname(target_file_path))

The first thing to do is create a temporary file in the same directory as the file we're trying to create or update. We do this because move operations (which we'll need later) aren't guaranteed to be atomic when they're between different file systems. Additionally, it's import to set delete=False as the standard behaviour of the NamedTemporaryFile is to delete itself as soon as it's not in use.

# preserve file metadata if it already exists
if os.path.exists(target_file_path):
    copyWithMetaData(target_file_path, temp_file.name)

We needed to support both file creation and updates, so in the case that we’re overwriting or appending to an existing file, we initialize the temporary file with the target file’s contents and metadata.

with open(temp_file.name, mode) as f:
    f.write(file_contents)
    f.flush()
    os.fsync(f.fileno())

Here we write or append the given file contents to the temporary file, and we flush and sync to disk manually to prepare for the most critical step:

os.replace(temp_file.name, target_file_path)

This is where the magic happens: os.replace is an atomic operation (when the source and target are on the same file system), so we're now guaranteed that if this fails to complete, no harm will be done.

We use the finally clause to remove the temporary file in case something did go wrong along the way, but now the very worst thing that can happen is that we end up with a temporary file ¯\_(ツ)_/¯