A VSCode extension to make your code more secure

 I recently installed Red Hat’s YAML VS Code extension to assist me with Bamboo Specs, convinced by the Bald Bearded Builder that this was the linter for me (check out its schema support!). I don’t usually appreciate extensions recommending things to me (and, to be fair, I don’t know that that’s precisely what happened), but this morning a toaster popped up suggesting that I install their Dependency Analytics extension and I am SO glad that I clicked on it!

Red Hat’s “Dependency Analytics” extension is fantastic, it’s powered by Snyk’s vulnerability database and when opening one of my projects’ dependency files* I immediately saw red and was able to click my way clear in a matter of minutes**.

* My current team has projects written in all four of the supported languages, the only thing I’m personally missing is an extension for Visual Studio “proper” for C#…

** Well, okay, one of the dependency suggestions included a breaking change, but the rest of them were trivial upgrades.

Well done, Red Hat, for making safety and security just a little bit easier!

...

Originally published at https://therightstuff.medium.com.

An Impatient Developer’s Guide to Debian Maintenance (Installation) Scripts and Package Diverts

Photo by Andrea Piacquadio from Pexels

 The people involved in coming up with the dpkg scheme for installing / upgrading / downgrading / removing packages are very clever. While unintuitive to the uninitiated, the scheme is mostly logical and reasonable, though there are some points where I feel a little more effort and consideration could have made a world of difference.

“The evil that men do lives on and on” — Iron Maiden

In addition to regular installation behaviour, I needed to wrap my head around “package diverts”, which is a very clever system for enabling packages to handle file conflicts. Except that it doesn’t handle what I would consider to be a very basic use case:

1. Install an initial version of our package.
   
   
2. Discover that our package needs to overwrite a file that’s installed by an upstream dependency.
   
   
3. Create a new version of our package that includes the file and configures a “package divert” to safely stow the dependency’s version.
   
   
4. Remove the “package divert” on the file if the newer version of the package is uninstalled or downgraded to the previous version that doesn’t include it.

That last part, in italics? That’s the kicker right there. Read on to understand why.

Debian Installation Script Logic In Plain English

After poring over the Debian maintainer scripts flowcharts, I felt I had a pretty good handle on things but there’re a couple of little “gotcha”s, so I feel like it’s worth providing a brief summary of the general flow in common English.

Debian maintenance scripts are run by apt and dpkg at specific points in the installation process:

https://www.debian.org/doc/debian-policy/ap-flowcharts.html

1. If the package is being removed or upgraded (meaning that a different version is being installed, “upgraded” also means “downgraded” to Debian maintainers), the previously installed version’s prerm script is called.
   
   
2. If the package is being installed or upgraded, the new version’s preinst script is run.
   
   
3. If the package is not being removed, the new version’s package contents are unpacked. This will overwrite existing files if they were installed by the same package, or fail the installation if it attempts to overwrite another package’s files without a package divert configured.
   
   
4. If the package is being removed, its files are deleted.
   
   
5. The previously installed version’s postrm script is called if the package is being removed or upgraded.
   
   
6. If the package is not being removed, any package contents belonging to the previous version that do not also exist in the new one are removed.
   
   
7. If the package is not being removed, the new version’s postinst script is called.

It is important to note that if a maintenance script fails with a non-zero exit code, the package will be in a broken state that can be very difficult (sometimes impossible) to recover. From our experience, it’s best to catch all exceptions, log them, “exit gracefully” with an exit code of 0, and hope for the best.

Also from our experience, it’s a good idea for maintenance scripts to log everything to the stderr stream in order to preserve chronological order.

Debian Package Diverts for the Uninitiated

The principle of Debian package diverts is straightforward enough: when you want to include a file in your package contents that conflicts with another package’s file (i.e. the absolute paths are identical), you create a “divert” on that file so that any other package’s versions of that file is “diverted” to a different file name.

Creating a Package Divert

To create a package divert, your package’s preinst script should run the following command:

dpkg-divert --package my-package-name --add --rename \

    --divert /path/to/original.divert-ext /path/to/original

The preinst script is the place to do this because the divert must be in place before the package contents are unpacked. The dpkg-divert commands are idempotent, so having it called in the preinst of every install is fine.

Removing a Package Divert

When your package is uninstalled, it’s good practice to remove the package divert and rename the diverted files back to their original file names. It’s recommended to remove the package divert in the postrm script, which makes perfect sense when uninstalling a package because the files are deleted before postrm is called.

dpkg-divert --package my-package-name --remove --rename \

    --divert /path/to/original.divert-ext /path/to/original

When removing a package, the package’s files have been deleted already and removing a divert simply renames the diverted file back to its original file name.

When upgrading a package, however, the files are only deleted after postrm has been called. This means that a call to dpkg-divert — remove will fail because it would have to overwrite the upgraded package’s copy of the file that hasn’t yet been removed.

It also means that if you delete your package’s file in the postrm in order to remove the divert, the original package’s file will be deleted after your postrm because it will have been identified as belonging to the upgraded package.

“Insanity is contagious.” ― Joseph Heller, Catch-22

It is for this reason that if we remove the divert in the postrm during a downgrade to a version that does not include the file in its package contents, we will lose the original file. If we do not remove the package divert, we will retain the diverted original file, but it will be renamed and therefore not serve its purpose. In our downgrading scenario, the postinst script that’s run after the file removal phase of an installation belongs to the older version of the package that didn’t know about the file, or package diverts, so that won’t be of any use to us. In short, the only way to downgrade our package is to completely remove it and install the older version, which for us is simply not an option.

Epilogue

Fortunately, my team and I are in the position that the original file also belongs to a package that we maintain, and we are able to overwrite the original file in the postinst script* with confidence and impunity. That means no rolling back without removing and then reinstalling the original package, which in our case happens to be impossible.

* Of course, the file can no longer be included in the package contents with its original path or the installation will fail.

Hope you’ve found this helpful! Please share in the comments if you’ve had similar experiences, or if you know of any other workarounds!

...

Originally published at https://therightstuff.medium.com.

The Day Our Python gRPC Connections Died

Image by OpenIcons from Pixabay

On the 30th of September 2021, a heavily-used root certificate — DST Root CA X3 — expired. You can read all about it here.

According to a handful of forum posts and github issues I’ve come across, the change has caused a fair amount of pain to those unfortunates who failed to heed the warnings, but for most of us this really wasn’t a surprise. For our team, the expiration date came and went and we didn’t even notice! Until our primary in-house testing tool began failing its connection tests with the following:

Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

Our gRPC connection tests are written in Python (using the grpcio and grpcio-tools packages), and run on a variety of linux machines and Docker images. Hunting through the forums, it looked like upgrading to the latest versions of the grpcio dependencies should do the trick, but it didn’t.

At least not by itself.

We eventually determined that the problem was that DST Root CA X3 was still registered as a certificate authority, and it took so long to figure out how to remove it on Debian that I realized that I had to post about it:

To see if the DST Root CA X3 certificate is configured as a root authority, list the contents of your /etc/ssl/certs folder:

> ls -l /etc/ssl/certs | grep dst

lrwxrwxrwx 1 root root 53 Sep 11 2020 DST_Root_CA_X3.pem -> /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt

2. Edit /etc/ca-certificates.conf and insert a ! at the beginning of the name of the DST Root CA X3 certificate to flag it as removed:

> sed -i “s@^mozilla/DST_Root_CA_X3.crt@!mozilla/DST_Root_CA_X3.crt@” “/etc/ca-certificates.conf”

To update the certificates, run the following:

> sudo /usr/sbin/update-ca-certificates -f

Note that it must be fully qualified as the /usr/sbin directory is not in the PATH by default, and it might be necessary to install the ca-certificates package using apt. The “f” of the -f flag apparently stands for “fresh”.

3. Set the GRPC_DEFAULT_SSL_ROOTS_FILE_PATH environment variable, which is required for the above changes to be respected:

> export GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=/etc/ssl/certs/ca-certificates.crt

Once all that’s done, you should be able to connect successfully!

...

Originally published at https://therightstuff.medium.com.

Simple safe (atomic) writes in Python3

 In sensitive circumstances, trusting a traditional file write can be a costly mistake - a simple power cut before the write is completed and synced may at best leave you with some corrupt data, but depending on what that file is used for you could be in for some serious trouble.

While there are plenty of interesting, weird, or over-engineered solutions available to ensure safe writing, I struggled to find a solution online that was simple, correct and easy-to-read and that could be run without installing additional modules, so my teammates and i came up with the following solution:

Explanation:

temp_file = tempfile.NamedTemporaryFile(delete=False,
                                        dir=os.path.dirname(target_file_path))

The first thing to do is create a temporary file in the same directory as the file we're trying to create or update. We do this because move operations (which we'll need later) aren't guaranteed to be atomic when they're between different file systems. Additionally, it's import to set delete=False as the standard behaviour of the NamedTemporaryFile is to delete itself as soon as it's not in use.

# preserve file metadata if it already exists
if os.path.exists(target_file_path):
    copyWithMetaData(target_file_path, temp_file.name)

We needed to support both file creation and updates, so in the case that we’re overwriting or appending to an existing file, we initialize the temporary file with the target file’s contents and metadata.

with open(temp_file.name, mode) as f:
    f.write(file_contents)
    f.flush()
    os.fsync(f.fileno())

Here we write or append the given file contents to the temporary file, and we flush and sync to disk manually to prepare for the most critical step:

os.replace(temp_file.name, target_file_path)

This is where the magic happens: os.replace is an atomic operation (when the source and target are on the same file system), so we're now guaranteed that if this fails to complete, no harm will be done.

We use the finally clause to remove the temporary file in case something did go wrong along the way, but now the very worst thing that can happen is that we end up with a temporary file ¯\_(ツ)_/¯

Approaching Software Engineering as an evolutionary process

A year or two ago I had an opportunity to sit down with AWS's Marcin Kowalski in a cafeteria and discuss the problems of software development at almost-unimaginable scale. I walked away with a new (for me) conception of software engineering that is part engineering, part organic biology, and I've found this perspective has shifted my approach to software development in a powerful and immensely helpful way.

As Computer Scientists and Software Engineers, we've been trained to employ precision in algorithm design, architecture and implementation: Everything must be Perfect. Everything must be Done Right.

For smaller, isolated projects, this engineering approach is critical, sound and practical, but once we begin to creep into the world of integrated solutions and micro-services it rapidly begins to break down.

Towards a Python version of my Javascript AWS CDK guide

After successfully using a Typescript CDK project to deploy a python lambda on Thursday, I decided to spend some time this evening creating a Python CDK guide. It's very limited at the moment (just a simple function and a basic lambda layer), but it's a start!

Keeping Track of your Technical Debt

The impact of technical debt

Over the years the concept of "technical debt" has become a phrase that can generate anxiety and a lack of trust, as well as setting up developers and their managers for failure. The metaphor might not be perfect (Ralf Westphal makes a strong case for treating it like an addiction), but I feel it's pretty apt if you think of the story of The Pied Piper of Hamelin - if you don't pay the piper promptly for keeping you alive, he'll come back to steal your future away.

Maybe I'm being a bit dramatic, but I value my time on this planet and over the course of two decades as a software developer in various sectors of the industry I have (along with countless others) paid with so much of my own precious lifetime (sleep hours, in particular) for others' (and occasionally my own) quick fixes, rushed decisions and hacky workarounds that it pains me to even think about it.

It's almost always worthwhile investing in doing things right the first time, but we rarely have the resources to invest and we are often unable to accurately predict what's right in the first place.

So let's at least find a way to mitigate the harm.

Why TODOs don't help

The traditional method of initiating technical debt is the TODO. You write a well-meaning (and hopefully descriptive) comment starting with TODO, and "Hey, presto!" you have a nice, easy way to find all those little things you meant to fix. Right? Except that's usually not the case. We are rarely able to make time to go looking for more things to do, and even when we can, with modern software practices it's unlikely that searching across all of our different repositories will be effective.

What generally ends up happening, then, is that we only come across TODOs by coincidence, when we happen to be working with the code around it, and the chances are that it'll be written by a different developer from a different time period and be explained in... suboptimal language.

"Why hasn't this been done?", you may well ask.

"Leave that alone! We don't remember why it works," you may well be told.

Track your technical debt with this one simple trick!

No matter the size of your team (even a team of one), you should always be working with some kind of issue tracking software. There's decent, free software out there, so there's really no excuse.

The fix? All you need to do is log a ticket for each TODO. Every time you find yourself writing a TODO, and I mean every single time, create a ticket. In both the ticket and the TODO comment, explain what you're doing, what needs to be done, why it needs to be deferred, and how urgent completing the TODO task is.

Then reference the ticket in the TODO comment.

This way you'll have a ticket - which I like to think of as an IOU - which can be added to the backlog and remembered when grooming and planning. This also provides the developer who encounters the TODO in the code a way to review any details and subsequent conversations in the ticket.

One interesting side effect of this approach? I often find that it's more effort to create the TODO ticket than to do the right thing in the first place, which can be a great incentive for the juniors to avoid the practice.

Another?

Indiana Jones No Ticket GIF from Indianajones GIFs

After establishing with my team that I will not approve a TODO unless there's a ticket attached, I must admit... I do sleep just a little bit better at night.

Reading and writing regular expressions for sane people

Your regular expressions need love. Reviewers and future maintainers of your regular expressions need even more.

No matter how well you've mastered regex, regex is regex and is not designed with human-readability in mind. No matter how clear and obvious you think your regex is, in most cases it will be maintained by a developer who a) is not you and b) lacks context. Many years ago I developed a simple method for sanity checking regex with comments, and I'm constantly finding myself demonstrating its utility to new people.

There are some great guides out there, like this one, but what I'm proposing takes things a step or two further. It may take a minute or two of your time, but it almost invariably saves a lot more than it costs. I'm not even discussing flagrant abuse or performance considerations.

Traditional regex: the do-it-yourself pattern

The condescending regex. Here you're left to your own devices. Thoughts and prayers.

var parse_url = /^(?:([A-Za-z]+):)?(\/{0,3})(0-9.\-A-Za-z]+)(?::(\d+))?(?:\/([^?#]*))?(?:\?([^#]*))?(?:#(.*))?$/;

(example taken from this question)

Kind regex: intention explained

It's the least you can do! A short line explaining what you're matching with an example or two (or three).

// parse a url, only capture the host part

// eg. protocol://host

//     protocol://host:port/path?querystring#anchor

//     host

//     host/path

//     host:port/path

var parse_url = /^(?:([A-Za-z]+):)?(\/{0,3})(0-9.\-A-Za-z]+)(?::(\d+))?(?:\/([^?#]*))?(?:\?([^#]*))?(?:#(.*))?$/;

Careful regex: a human-readable breakdown

Here we ensure that each element of the regex pattern, no matter how simple, is explained in a way that makes it easy to verify that it's doing what we think it's doing and can modify it safely. If you're not an expert with regex, I recommend using one of the many available tools such as regexr.com.

// parse a url, only capture the host part

//     (?:([A-Za-z]+):)?

//         protocol - an optional alphabetic protocol followed by a colon

//     (\/{0,3})(0-9.\-A-Za-z]+)

//         host - 0-3 forward slashes followed by alphanumeric characters

//     (?::(\d+))?

//         port - an optional colon and a sequence of digits

//     (?:\/([^?#]*))?

//         path - an optional forward slash followed by any number of

//         characters not including ? or #

//     (?:\?([^#]*))?

//         query string - an optional ? followed by any number of

//         characters not including #

//     (?:#(.*))?

//         anchor - an optional # followed by any number of characters

var parse_url = /^(?:([A-Za-z]+):)?(\/{0,3})(0-9.\-A-Za-z]+)(?::(\d+))?(?:\/([^?#]*))?(?:\?([^#]*))?(?:#(.*))?$/;

Now that we've taken the time to break this down, we can identify the intention behind the patterns and ask better questions: why is the host the only matched group? Was this tested? (Because (0-9.\-A-Za-z] is clearly an error, and there are almost no restrictions on invalid characters)

Unless you're a sadist (or a masochist), this is definitely a better way to operate: be careful, and if you can't be careful then at least be kind.

Priority pinning in apt-preferences with different versions and architectures

I'm posting this because I've lost too many hours figuring it out myself, the documentation is missing several important notes and I haven't found any forum posts that really relate to this:

Question: How do I prioritize specific package versions for multiple architectures? In particular, I have a number of different packages which I would like to download for multiple architectures, and I would like to prioritize the versions so that if they're not explicitly provided, apt will try to get a version matching my arbitrary requirements (in my case, the current git branch name) and fall back to our develop branch versions.

eg. I would like to download package my-package for both i386 and amd64 architectures and I would like to pull the latest version that includes my-git-branch-name before falling back to the latest that includes develop.

Answer:

The official documentation is here.

1. In order to support multiple architectures, all packages being pinned must have their architecture specified, and there must be an entry for each architecture. A pinning for the package name without the architecture specified will only influence the default (platform) architecture:

Package: my-package

Pin: version /your regex here/

Pin-Priority: 1001

2. The entries are whitespace-sensitive, although no errors will be reported if you have whitespace. The following pinning will be silently disregarded:

Package: my-package:amd64

    Pin: version /your regex here/

    Pin-Priority: 1001

3. apt update must be called after updating the preferences file in order for them to be respected and after adding additional architectures using (for example) dpkg --add-architecture i386

The following excerpt from /etc/apt/preferences solves the stated problem:

Package: my-package:amd64

Pin: version /-my-git-branch-name-/

Pin-Priority: 1001

Package: my-package:i386

Pin: version /-my-git-branch-name-/

Pin-Priority: 1001

Package: my-package:amd64

Pin: version /-develop-/

Pin-Priority: 900

Package: my-package:i386

Pin: version /-develop-/

Pin-Priority: 900 

It may be worthwhile noting that to download or install a package with a specified architecture and version use the command apt download package-name:arch=version

Enabling Programmable Bank Cards To Do All The Things

A little while back, Offerzen and Investec opened up their Programmable Bank Card Beta for South Africans, taking over from Root, and I was excited to be able to sign up!

My long-term goal with programmable banking is to integrate crypto transactions directly with my regular banking - which might be possible quite soon as Investec is starting to roll out its open API offering in addition to the card programme - and as far as I can tell that could be a real game-changer in terms of making crypto trading practical for non-investment purposes.

When I joined, though, what I found was a disparate group of really interesting projects and only a two-second time window to influence whether your card transaction will be approved or not.

I decided to bide my time by building a bridge for everyone else's solutions, one which would provide a central location to determine transaction approval and enable card holders (including myself, of course) to forward their transactions securely to as many services as they like without costing them running time on their card logic.

It was obvious to me that this needed to be a cloud solution, and as someone with zero serverless experience¹ I spent some time evaluating AWS, Google and Microsoft's offerings. My priorities were simple:

• right tool for the job
• low cost
• high performance
• good user experience

In the end AWS was the clear winner with the first two priorities, so much so that I didn't even bother worrying about any potential performance tradeoffs (there probably aren't any) and I was prepared to put up with their generally mixed bag of user experience. I also had the added incentive that my current employer uses AWS in their stack so I would be benefiting my employer at the same time.

Overall, I'm very glad that I chose AWS in spite of the trials and tribulations, which you can follow in my earlier post about building a template for CDK for beginners (like myself). In addition to learning how to use CDK, this project has given me solid experience in adapting to cloud architecture and patterns that are substantially different from anything I've worked with before, and I've already been able to effectively apply my learnings to my contract work.

One of the obligations of the Programmable Bank Card beta programme is sharing your work and presenting it; I was happy to do so, but for months I've been locked down and working in the same space as my wife and four year-old (we don't have room for an office or much privacy in our apartment); with all the distractions of home schooling and having to work while my wife and kid play my favourite video games I've barely been able to keep up my hours with my paid gig, so making time for extra stuff? That hasn't been so easy.

A couple of weeks ago my presentation was due, and I spent a lot of the preceding two weekends in a mad scramble to get my project as production-ready as I could - secure, reliable, usable. Half an hour before "go time" I finally deployed my offering, and I was a little bit nervous doing a live demo... I mean, sure, I'd tested all the end points after each deployment - but as we all know: Things Happen. Usually During Live Demos.

Fortunately, the presentation went very well! I spent the following weekend adding any missing "critical" functionality (such as scheduling lambda warmups and implementing external card usage updates), and I'm hoping that some of my fellow community members get some good use out of it (whether on their own stacks or mine).

The code can be found here on GitLab, and a few days ago OfferZen published the recording of my presentation on their blog along with its transcript² and my slide deck.

Thank you for joining me on my journey!

---

¹ Although I was employed by one of the major cloud providers for a while, I've only worked "on" their cloud rather than "in", and while I do have extensive experience with Azure none of it included serverless functions.

² The transcript has a few minor transcription errors, but they're more amusing than confusing.

Resolving private PyPI configuration issues

The Story

I've spent the last few days wrestling with the serverless framework, growing fonder and fonder of AWS' CDK at every turn. I appreciate that serverless has been, until recently, very necessary and very useful, but I feel confident in suggesting that when deployment mechanisms become more complex than the code they're deploying things aren't moving in the right direction.

That said, this post is about overcoming what for us was the final hurdle: getting Docker able to connect to a private PyPI repository for our Python lambdas and lambda layers.

Python is great, kind of, but is marred by two rather severe complications: the split between python and python3, and pip… so it's a great language, with an awful tooling experience. Anyway, our Docker container couldn't communicate with our PyPI repo, it took way too long to figure out why, here's what we learned:

The Solution

If you want to use a private PyPI repository without typing in your credentials at every turn, there are two options:

1.

~/.pip/pip.conf:

2.

~/.pip/pip.conf:



~/.netrc:



It is important that ~/.netrc has the same owner:group as pip.conf, and that its permissions are 0600 (chmod 0600 ~/.netrc).

What's not obvious - or even discussed anywhere - is that special characters are problematic and are handled differently by the two mechanisms.

In pip.conf, the password MUST be URL encoded.
In .netrc, the password MUST NOT be URL encoded.

The Docker Exception

For whatever reason, solution 2 (the combination of pip.conf and .netrc) does NOT work with Docker.

Conclusion

Amazon's CDK is excellent, and unless you have a very specific use-case that it doesn't support it really is worth trying out!

Oh! And that Python is Very Nice, but simply isn't nice enough to justify the cost of its tooling.

the rights and wrongs of git push with tags

My employer uses tags for versioning builds.

git push && git push --tags

I'm guessing this is standard practice, but we've been running into an issue with our build server - Atlassian's Bamboo - picking up commits without their tags. The reason was obvious: we've been using git push && git push --tags, pushing tags after the commits, and Bamboo doesn't trigger builds on tags, only on commits. This means that every build would be versioned with the previous version's tag!

The solution should have been obvious, too: push tags with the commits. Or before the commits? This week, we played around with an experimental repo and learned the following:

git push --tags && git push

To the uninitiated (like us), the result of running git push --tags can be quite surprising! We created a commit locally, tagged it*, and ran git push --tags. This resulted in the commit being pushed to the server (Bitbucket, in our case) along with its tag, but the commit was rendered invisible. Not even git ls-remote --tags origin would return it, and it was not listed under the commits on its branch, although it showed up with its commit on Bitbucket's feature to search commits by tag quite clearly:

* All tags are annotated, automatically, courtesy of SourceTree. If you're not using SourceTree then you should be, and I promise you I'm not being paid to say that. If you must insist on the barbaric practice of using the command line, just add -a (and -m to specify the message inline if you don't want the editor to pop up, check out the documentation for more details).

Pushing a tag first isn't the end of the world - simply pushing the commit with git push afterwards puts everything in order - unless someone else pushes a different commit first. At that point the original commit will need to be merged, with merge conflicts to be expected. Alternatively - and relevant for us - any scripts that perform commits automatically might fail between pushing the tags and their commits, leading to "lost" commits.

git push --tags origin refs/heads/develop:refs/heads/develop

This ugly-to-type command does what we want it to do: it pushes the commit and its tags together. From the documentation:

When the command line does not specify what to push with <refspec>... arguments or --all, --mirror, --tags options, the command finds the default <refspec> by consulting remote.*.push configuration, and if it is not found, honors push.default configuration to decide what to push (See git-config[1] for the meaning of push.default).

Okay, so that works. But there's no way I'm typing that each time, or asking any of my coworkers to. I want English. I'm funny that way.

git push --follow-tags

Here we go: the real solution to our problems. As long as the tags are annotated - which they should be anyway, see the earlier footnote on tagging - running git push --follow-tags will push any outstanding commits from the current branch along with their annotated tags. Any tags not annotated, or not attached to a commit being pushed, will be left behind.

Resolved!

A templated guide to AWS serverless development with CDK

If all you're looking for is a no-frills, quick, step-by-step guide, just scroll on down to The Guide!

The Story

Or, how I learned to let go of my laptop and embrace The Cloud...

Motivation

A while ago I outlined a project that I intend to build, one that I'm rather enthusiastic about, and after spending a few hours evaluating my options I decided that an AWS solution was the right way to go: their options are configurable and solid, their pricing is very reasonable and, as an added bonus, I get to familiarize myself with tech I'll be using for my current contract!

All good.

Once I'd made the decision to use AWS, becoming generally familiar with the products on offer was easy enough and it took me very little time to flesh out a design. I was ready to get coding!

Or so I thought...

Hurdle 1

Serverless. And possibly other frameworks, I'm pretty sure I looked at a few but it seems like forever ago. Serverless shows lots of promise, and I know people who swear by it, but it's only free up to a point and I immediately ran into strange issues with my first deployment. I found the interface surprisingly unhelpful, and it looks like once you're using it you're somewhat locked in.

Pass.

Hurdle 2

Cognito. Security first. Cognito sounds like a great solution, but once I'd gotten a handle on how it works I was severely disappointed by how limiting it is, and getting everything set up takes real developer and user effort (and I suffer enough from poor interface fatigue). After playing around with user pools and looking at various code samples, I realized that I'd rather allow users to register using their email addresses / phone numbers as 2nd-factor authentication (mailgun and twilio are both straightforward options), or use oauth providers like Facebook, Google and GitHub, and I certainly want to encourage my users to use strong, memorable passwords (easily enforced with zxcvbn, and why is this still a thing?!) which Cognito doesn't allow for.

You'll need to configure Lambda authorizers either way, so I really don't think Cognito adds much value.

Hurdle 3

Lambda / DynamoDB. Okay, so writing a lambda function is really easy, and guides and examples for code that reads to and writes from DynamoDB abound. Great! Except for the part where you need to test your functions before deploying them.

My first big mistake - and so far it's proved to be the most expensive one - was not understanding at this point that "testing locally" is simply not a feasible strategy for a cloud solution.

Hurdle 4

The first code I wrote for my project was effectively a lambda environment emulator to test my lambda functions. It was far from perfect, and it did take me a couple of hours to cobble together, but it did the job and I used it to successfully test lambda functions against DynamoDB running in Docker.

Hurdle 5

Lambda Layers. Why do most guides not touch on these? Why are there so few layers guides written for Javascript? It took me a little while to get a handle on layers and build a simple script to create them from package.json files, but as far as hurdles go this was a relatively short one.

Hurdle 6

Deployment. It's nice to have code running locally, but uploading it to the cloud? Configuring API Gateway was a mixed bag of Good Interface / Bad Interface, same with the Lambda functions, and what eventually stumped me was setting up IAM to make everything play nicely together. What's the opposite of intuitive? Not counter-intuitive, in this case, as I don't feel that that word evokes nearly enough frustration.

Anyway, it became abundantly clear at that point that manual deployment of AWS configurations and components is not a viable strategy.

Hurdle 7

A coworker introduced me to two tools that could supposedly Solve All My Problems: CDK and SAM. This seemed like a worthy rabbit-hole to crawl into, but I couldn't find any examples of the two working together!

I began to build my own little framework, one that would allow me to configure my stack using CDK, synthesize the CloudFormation templates, and test locally using SAM. Piece by piece I put together this wonderful little tool, hour by hour, first handling DynamoDB, then Lambda functions, then Lambda Layers...

It was at that point that realization dawned: not only are SAM and CDK not interoperable by design, but SAM does not, in fact, provide meaningful "local" testing. Sure, you can invoke your lambda functions on your local machine, but the intention is to invoke them against your deployed cloud infrastructure. Once I got that through my head, it was revelation time: testing in the cloud is cheaper and better than testing locally.

The Guide

If you're like me, and you intend your first foray into cloud computing to be simple, yet reasonably production-ready, CDK is the easiest way forward and it's completely free (assuming you don't factor in your time figuring it out as valuable, but that's what I'm here for!).

Over the course of the past couple of weeks, I've put together the aws-cdk-js-dev-guide. It's a work in progress (next stop, lambda authenticators!), but at the time of writing this guide it's functional enough to put together a simple stack that includes Lambda Layers, DynamoDB, functions using both of those, API Gateway routes to those functions and the IAM permissions that bind them.

And that's just the tip of the CDK iceberg.

Step 1 - Tooling

It is both valuable and necessary to go through the following steps prior to creating your first CDK project:

• Create a programmatic user in IAM with admin permissions
• If you're using Visual Studio Code (recommended), configure the aws toolkit
• Set up credentials with the profile ID "default"
• Get your 12 digit account ID from My Account in the AWS console
• Follow the CDK hello world tutorial

Step 2 - Creating a new CDK project

The first step to creating a CDK project is initializing it with cdk init, and a CDK project cannot be initialized if the project directory isn't empty. If you would like to use an existing project (like aws-cdk-js-dev-guide) as a template, bear in mind that you will have to rename the stack in multiple locations and it would probably be safer and easier to create a new project from scratch and copy and paste in whatever bits you need.

Step 3 - Stack Definition

There are many viable approaches to setting up stages, mine is to replicate my entire stack for development, testing and production. If my stack wasn't entirely serverless - if it included EC2 or Fargate instances, for example - then this approach might not be feasible from a cost point of view.

Stack definitions are located in the /lib folder, and this is where the stacks and all their component relationships are configured. Here you define your stacks programmatically, creating components and connecting them.



I find that the /lib folder is a good place to put your region configurations.



Once you have completed this, the code to synthesize the stack(s) is located in the /bin folder. If you intend, like I do, to deploy multiple replications of your stack, this is the place to configure that.



See the AWS CDK API documentation for reference.

Step 4 - Lambda Layers

I've got much more to learn about layers at the time of writing, but my present needs are simple: the ability to make npm packages available to my lambda functions. CDK's Code.fromAsset method requires the original folder (as opposed to an archive file, which is apparently an option), and that folder needs to have the following internal structure:

.
.nodejs
.nodejs/package.json
.nodejs/node_modules/*

You can manually create this folder anywhere in your project, maintain the package.json file and remember to run npm install and npm prune every time you update it... or you can just copy in the build-layers script, maintain a package.json file in the layers/src/my-layer-name directory and run the script as part of your build process.

Step 5 - Lambda Functions

There are a number of ways to construct lambda functions, I prefer to write mine as promises. There are two important things to note when putting together your lambda functions:

1. Error handling: if you don't handle your errors, lambda will handle them for you... and not gracefully. If you do want to implement your lambda functions as promises, as I have, try to use "resolve" to return your errors.

2. Response format: your response object MUST be in the following structure:

This example lambda demonstrates using a promise to return both success and failure responses:

Step 6 - Deployment

There are three steps to deploying and redeploying your stacks:

1. Build your project
   If you're using Typescript, run
   > tsc
   If you're using layers, run the build-layers script (or perform the manual steps)
   
2. Synthesize your project
   > cdk synth
   
3. Deploy your stacks
   > cdk deploy stack-name
   Note: you can deploy multiple stacks simultaneously using wildcards

At the end of deployment, you will be presented with your endpoints. Unless your lambda has its own routing configured - see the sample dynamodb API examples that follow - simply make your HTTP requests to those URLs as is.

Sample dynamodb API call examples:
• List objects
  GET https://xxxxxxxxxx.execute-api.us-east-1.amazonaws.com/prod/objects
• Get specific object
  GET https://xxxxxxxxxx.execute-api.us-east-1.amazonaws.com/prod/objects/b4e01973-053c-459d-9bc1-48eeaa37486e
• Create a new object
  POST https://xxxxxxxxxx.execute-api.us-east-1.amazonaws.com/prod/objects
• Update an existing object
  POST https://xxxxxxxxxx.execute-api.us-east-1.amazonaws.com/prod/objects/b4e01973-053c-459d-9bc1-48eeaa37486e

Step 7 - Debugging

Error reports from the API Gateway tend to hide useful details. If a function is not behaving correctly or is failing, go to your CloudWatch dashboard and find the log group for the function.

Step 8 - Profit!!!

I hope you've gotten good use of this guide! If you have any comments, questions or criticism, please leave them in the comments section or create issues at https://github.com/therightstuff/aws-cdk-js-dev-guide.

An improved (fairer) playlist shuffling algorithm

Lots of people find playlist shuffling insufficiently random for a variety of reasons, some of which have been addressed by the industry.

There's one aspect that my wife and I haven't seen, though, and that's making sure that no songs get "left behind" whenever a playlist is reshuffled, whether intentionally or by switching back-and-forth between playlists.

In an attempt to sow seeds, I've just put together an example of an improvement that can easily be applied to any of the existing shuffle algorithms.

The value of git (re)parenting

EDIT: I have subsequently learned about git merge --no-ff and no longer re-parent. But I'll leave this up in case someone finds it useful anyway.

I'm not a command-line person, I may have grown up with it but... let's just say I've developed an allergy. I want GUIs, I want simplicity and I want visual corroboration that I'm doing what I think I'm doing. I really don't see why I need to become an expert in every tool I use before it can be functional.

This is why I adore SourceTree, and (last time I used it) GitEye. Easier, more intuitive interface, and visualizations to help you see whether what you're doing makes sense. You're less likely to make mistakes!

I'm also (and for similar reasons) a huge fan of squashed merges in Git; one commit per feature / bugfix / hotfix, and (theoretically) the ability to go through the individual commits on my own time if I ever need to. I use interactive rebasing* a lot to achieve a similar result, but getting other devs to use it responsibly is a lot more trouble than teaching them to add --squashed to the merge command. The downside is that because I've become used to interactive rebasing to squash my commits before merging, I've also become used to using regular merges and seeing a neat line on the branch graph showing me where my merges originated. Squashed commits don't give you that. And that's sad. It's also problematic - this morning I freaked out because I thought the code in a branch I'd squash-merged into wasn't where it needed to be, all because I was relying on those parenting lines and didn't immediately think to do a diff**.

* Ironically, from the command-line. It's the one thing I find less intuitive and more risky using SourceTree.

** While we're talking about branch diffs, if you're using Bitbucket don't trust the branch diff - the UI uses a three-dot diff and what you actually want is a two-dot, ie git diff branch1..branch2

So, after much surfing around the internets and learning lots more about git's inner workings than I care to, I came across an elegant little solution and have wrapped it with a bash script in the hopes that it'll be found useful. All you have to do is run this script as follows:

./add_parent.sh TARGET_COMMIT_ID NEW_PARENT_COMMIT_ID

@mysql/xdevapi joy!

after hitting a wall last night with the existing node.js mysql packages, which for some reason have been years behind integrating with mysql 8's authentication, i discovered @mysql/xdevapi. the documentation's a bit heavy and not really comprehensive, but now that i've figured out how to use it, i'm very pleased with how it operates! unfortunately, i'm going to have to roll my own migrations management, but that's a small price to pay for being able to interface with the latest mysql databases in an intelligent way.

in the interests of reducing friction for other adopters, i've rolled some sample queries into my database creation script. enjoy!

UPDATE: i've subsequently learned that table joins have not been implemented, so to perform those you'd have to use the session.sql method and use it in the same way (the same promise chaining) as the CRUD methods. seems like a serious oversight, but whatever.

Handling emails with node.js and Mailparser

After sorting out mail forwarding and piping emails with postfix, I then needed to understand how to handle emails being POSTed to an API endpoint.

To parse an email with node.js, I recommend using Mailparser's simpleParser. I'm using express with bodyParser configured as follows:

app.use(bodyParser.json({
 limit : config.bodyLimit
}));

In your handler:

const express = require('express');
const router = express.Router();
const simpleParser = require('mailparser').simpleParser;

or

import { Router } from 'express';
import { simpleParser } from 'mailparser';

and then

api.post('/', (req, res) => {
    simpleParser(req)
        .then(parsed => {
            res.json(parsed);
        })
        .catch(err => {
            res.json(500, err);
        });
});

Mailparser is excellent, and documented, but the documentation assumes that we're familiar with the email format. Fortunately, oblac's example email exists for those of us who aren't!

To test, send the example email to the endpoint via curl:

curl --data-binary "@./example.eml" http://your-domain-name/api/email

or Postman (attach file to "binary").

And we're good to go!

FreeVote: anonymous Q&A app

I can't believe something like this doesn't exist already! It's far from a polished product, but it's already pretty functional. Feel free to send me feedback, it'll help me prioritize the long list of improvements I've already come up with. This was inspired by my coworkers, every retrospective we all write down answers to sensitive questions "anonymously" on folded papers that one team member gathers and reviews... this way we can do it truly anonymously and all see the results.

FreeVote

Hosting my own podcast

I've been having trouble with podcast garden, turns out they're a really unprofessional outfit; more than half the time their website is inaccessible, and they have poor customer service. Also, they've taken a lot more money off me than I agreed to and I've opened a dispute with PayPal, hopefully they'll sort this all out.

In the meanwhile, I've spent the last couple of days migrating to a custom solution, I honestly don't know what I'd do if I wasn't a software developer. The Shakespeare's Sonnets Exposed podcast is now being hosted right here on industrial curiosity, and not only am I no longer at the mercy of other people's incompetence, it's a much cheaper solution, too!

Now I just need to find the time to make my solution easily available to the public...

Effective Technical Interviewing with GitHub

Throughout my career I've encountered many problems with hiring processes, some mildly annoying, some decidedly infuriating, and some utterly baffling. Of all of those issues, few have pressed my buttons as consistently as those relating to how technical assessments are generally handled.

The cost of an ineffective technical interview to the company can be greater than the hours invested in interviewing or the lack of resources until the right candidate is found, with many regulations making it difficult to get rid of a low-performing employee and most situations requiring a learning curve that extends beyond legal probation periods.

The cost of a technical interview to the applicant can be even greater, as interviewing at multiple companies or when already employed leaves little time to do the assessments and it's usually a lot of effort, for no compensation, that produces nothing of value.

The bigger companies have found a reasonably good solution to the technical assessment, but phone screening and days full of whiteboard interviews are simply not affordable for the smaller players. The focus of this article is on technical assessments wherein the candidate is expected to produce a solution, or set of solutions, within a set period of time.

There are two major downsides to these coding assessments:

1. It is of little consequence whether this is done in an in-office setting or as a take-home assignment, it is very difficult to extract meaningful information about the strengths and weaknesses of a candidate from the results of an assessment done in this manner. There are many variables that may influence the results of this kind of an assessment, the candidate's ability to code under stressful conditions for a start, and usually there's not enough interaction with the candidate during the coding process to assess collaborative behaviour.

2. The work done in producing these solutions is, for all intents and purposes, wasted effort. Candidates are consistently asked to demonstrate their skills by producing arbitrary code that would not be of use or interest to anyone going forward.

In this article I'd like to address two possible strategies using GitHub that could significantly improve the effectiveness and overall outcomes of this important step in a candidate's application.

First strategy: Existing GitHub issues

The world of open-source is full of real problems that require solving, and nine times out of ten (for smaller organizations in particular) in software packages that are used by the interviewing company.

One possible strategy is to break down the technical skills you require from your candidates - not just the languages and frameworks, but problem spaces as well - and find open issues on GitHub with similar profiles.

Second Strategy: Porting an issue to GitHub

It is a common misconception that everything in a proprietary codebase must be kept under lock and key. With the right license, there are plenty of issues that a company might have that can be isolated from the codebase, ported to GitHub, and then reincorporated once solved.

Similarly to the first strategy, it is important to break down the technical skills you require from your candidates and find issues that would highlight their relevant strengths and weaknesses. The act of "open-sourcing" this code would be as simple as establishing the right license and creating a repo. Ideally, it would be code that others might find useful, too!

With this strategy, there would need to be a different problem for each individual candidate, as the moment one candidate has come up with a solution there would be little to stop another from taking a peek and little value in multiple developers repeating the same effort.

The advantage, obviously, is that work put into interviewing would directly benefit the company, but the moment you're benefitting from the candidate's efforts they would be entitled to at least some compensation, in addition to being able to show their work in other job applications.

When an applicant has successfully delivered a solution, you could compensate them directly, which may be preferable, or you could use Gitcoin, where you can set a bounty on your issues for immediate payment on the acceptance of a PR.

Either way, it'd be a small price to pay for development, the candidate would have earned a little money (which most candidates desperately need), and you'd have useful information for your hiring process.

Conclusion

Contributing to GitHub projects will show more than just a candidate's code quality; it can also show communication skills and collaborative behaviour. Additionally, choosing or defining projects with strict requirements for code conventions and unit tests will push your candidate more significantly than just mentioning that you'd like to see testing done.

An accepted PR (pull request) will make the world a (slightly) better place; it would be a good deed on your part, a good deed on the candidate's part, and the resulting effort would have actual value.

It is also far more likely that a candidate will be motivated to solve a real problem than a fictitious one, and their efforts will be rewarded by becoming a part of their portfolio. As somebody who has predominantly worked on proprietary code, I can testify to the utility of being able to direct a potential employer to my GitHub account!

With either strategy, everybody wins. For the developer, the technical assessment becomes real world experience that they can show off to others, and the employer gets to see real-world coding performed under real-world conditions.

Wins all around!