After successfully using a Typescript CDK project to deploy a python lambda on Thursday, I decided to spend some time this evening creating a Python CDK guide. It's very limited at the moment (just a simple function and a basic lambda layer), but it's a start!
ARCHIVE NOTICE
Saturday, 23 January 2021
Sunday, 25 October 2020
Keeping Track of your Technical Debt
The impact of technical debt
Over the years the concept of "technical debt" has become a phrase that can generate anxiety and a lack of trust, as well as setting up developers and their managers for failure. The metaphor might not be perfect (Ralf Westphal makes a strong case for treating it like an addiction), but I feel it's pretty apt if you think of the story of The Pied Piper of Hamelin - if you don't pay the piper promptly for keeping you alive, he'll come back to steal your future away.
Maybe I'm being a bit dramatic, but I value my time on this planet and over the course of two decades as a software developer in various sectors of the industry I have (along with countless others) paid with so much of my own precious lifetime (sleep hours, in particular) for others' (and occasionally my own) quick fixes, rushed decisions and hacky workarounds that it pains me to even think about it.
It's almost always worthwhile investing in doing things right the first time, but we rarely have the resources to invest and we are often unable to accurately predict what's right in the first place.
So let's at least find a way to mitigate the harm.
Why TODOs don't help
The traditional method of initiating technical debt is the TODO. You write a well-meaning (and hopefully descriptive) comment starting with TODO, and "Hey, presto!" you have a nice, easy way to find all those little things you meant to fix. Right? Except that's usually not the case. We are rarely able to make time to go looking for more things to do, and even when we can, with modern software practices it's unlikely that searching across all of our different repositories will be effective.
What generally ends up happening, then, is that we only come across TODOs by coincidence, when we happen to be working with the code around it, and the chances are that it'll be written by a different developer from a different time period and be explained in... suboptimal language.
"Why hasn't this been done?", you may well ask.
"Leave that alone! We don't remember why it works," you may well be told.
Track your technical debt with this one simple trick!
Saturday, 24 October 2020
Reading and writing regular expressions for sane people
Traditional regex: the do-it-yourself pattern
Kind regex: intention explained
Careful regex: a human-readable breakdown
Wednesday, 19 August 2020
Priority pinning in apt-preferences with different versions and architectures
I'm posting this because I've lost too many hours figuring it out myself, the documentation is missing several important notes and I haven't found any forum posts that really relate to this:
Question: How do I prioritize specific package versions for multiple architectures? In particular, I have a number of different packages which I would like to download for multiple architectures, and I would like to prioritize the versions so that if they're not explicitly provided, apt will try to get a version matching my arbitrary requirements (in my case, the current git branch name) and fall back to our develop branch versions.
eg. I would like to download package my-package for both i386 and amd64 architectures and I would like to pull the latest version that includes my-git-branch-name before falling back to the latest that includes develop.
Answer:
The official documentation is here.
1. In order to support multiple architectures, all packages being pinned must have their architecture specified, and there must be an entry for each architecture. A pinning for the package name without the architecture specified will only influence the default (platform) architecture:
Package: my-package
Pin: version /your regex here/
Pin-Priority: 1001
2. The entries are whitespace-sensitive, although no errors will be reported if you have whitespace. The following pinning will be silently disregarded:
Package: my-package:amd64
Pin: version /your regex here/
Pin-Priority: 1001
3. apt update must be called after updating the preferences file in order for them to be respected and after adding additional architectures using (for example) dpkg --add-architecture i386
The following excerpt from /etc/apt/preferences solves the stated problem:
Package: my-package:amd64
Pin: version /-my-git-branch-name-/
Pin-Priority: 1001
Package: my-package:i386
Pin: version /-my-git-branch-name-/
Pin-Priority: 1001
Package: my-package:amd64
Pin: version /-develop-/
Pin-Priority: 900
Package: my-package:i386
Pin: version /-develop-/
Pin-Priority: 900
It may be worthwhile noting that to download or install a package with a specified architecture and version use the command apt download package-name:arch=version
Monday, 23 September 2019
@mysql/xdevapi joy!
in the interests of reducing friction for other adopters, i've rolled some sample queries into my database creation script. enjoy!
UPDATE: i've subsequently learned that table joins have not been implemented, so to perform those you'd have to use the session.sql method and use it in the same way (the same promise chaining) as the CRUD methods. seems like a serious oversight, but whatever.
Friday, 23 August 2019
Handling emails with node.js and Mailparser
To parse an email with node.js, I recommend using Mailparser's simpleParser. I'm using express with bodyParser configured as follows:
app.use(bodyParser.json({
limit : config.bodyLimit
}));In your handler:const express = require('express');
const router = express.Router();
const simpleParser = require('mailparser').simpleParser;orimport { Router } from 'express';
import { simpleParser } from 'mailparser';and thenapi.post('/', (req, res) => {
simpleParser(req)
.then(parsed => {
res.json(parsed);
})
.catch(err => {
res.json(500, err);
});
});
Mailparser is excellent, and documented, but the documentation assumes that we're familiar with the email format. Fortunately, oblac's example email exists for those of us who aren't!To test, send the example email to the endpoint via curl:
curl --data-binary "@./example.eml" http://your-domain-name/api/email
or Postman (attach file to "binary").
And we're good to go!
Thursday, 22 August 2019
Mail forwarding and piping emails with Postfix for multiple domains
While I'm happily using mailgun for mail sending (after much frustration I threw in the towel trying to integrate DKIM packages with Postfix to get my outgoing emails secured) I was certain that I could at least have my mail server handle mail-forwarding for my multiple domains, and while that proved to be fairly straightforward I then tumbled down a rabbit-hole trying to get Postfix to pipe certain emails to a node.js script for processing.
- Here are the steps you'll need to take:
- Set up your A and MX records for your domain, the A record @ pointing to the IP address of the server you’re going to be receiving emails on and MX with the hostname @ and the value 10 mail.your-domain-name
If your mail server is not the same as your primary A record, simply create an additional A record mail pointing to the correct IP address. - sudo apt-get install postfix
Select "Internet Site" and enter your-domain-name (fully qualified) - sudo vi /etc/postfix/main.cf
- Add mail.your-domain-name to the list of mydestination values
- Append
virtual_alias_domains = hash:/etc/postfix/virtual_domains virtual_alias_maps = hash:/etc/postfix/virtual
to the end of the file
- sudo vi /etc/aliases
curl_email: "|curl --data-binary @- http://your-domain-name/email"
- sudo newaliases
- sudo vi /etc/postfix/virtual_domains
example.net #domain example.com #domain your-domain-name #domain
(the #domain fields suppress warnings) - sudo postmap /etc/postfix/virtual_domains
- sudo vi /etc/postfix/virtual
info@your-domain-name bob@gmail.com everyone@your-domain-name bob@gmail.com jim@gmail.com email_processor@your-domain-name curl_email@localhost @your-domain-name catchall@whereveryouwant.com ted@example.net jane@outlook.com
- sudo postmap /etc/postfix/virtual
- sudo /etc/init.d/postfix reload
You should be able to find your postfix logs at /var/log/mail.log. Good luck!
Friday, 17 August 2018
self-signed localhost ssl certificate on windows (for dummies)
[THIS ARTICLE IS OBSOLETE: you'll find better ones here and here]
today i needed to self-sign certificates, and while there are good guides available they make a lot of assumptions or use complicated tools. here's what i figured out this morning after a long struggle with scripts that windows doesn't like:
1. install openssh for windows, and make sure to remember where the installation directory is. there are a number of options available from the openssl wiki, shining light productions' version is the most official. download the default build (the larger installation file) paying attention to whether your system is 32-bit or 64-bit.
2. install babun (bash and zsh on windows for people who don't want to micromanage their software)
3. using babun, change to the openssl bin directory. run the following command from letsencrypt:
openssl req -x509 -out localhost.crt -keyout localhost.key -newkey rsa:2048 -nodes -sha256 -subj '/CN=localhost' -extensions EXT -config <( printf "[dn]\nCN=localhost\n[req]\ndistinguished_name = dn\n[EXT]\nsubjectAltName=DNS:localhost\nkeyUsage=digitalSignature\nextendedKeyUsage=serverAuth")
once installed in your app you'll be able to access with http or https - your browser will warn you that the certificate isn't signed but it's yours, so just accept it and get back to work!
Saturday, 11 August 2018
IIS Rewrite Rules Regex
1. Rewrite actions use regex matched groups by referencing {R:<group number>} eg. {R:1}
2. The order of the rules is critical, rewrites are then re-matched from scratch.
3. To prevent matches beginning with a word, use "^((?!theword).*)"
Wednesday, 28 March 2018
Azure Key Vault in C# for Dummies
Please note: I'm excited because I've finally managed to authenticate using a secret, it's probably more secure to use certificates but I'll get to that another time.
Step 1: Registered App
Under Azure Active Directory in the Azure Portal, select App registrations.
Add a New application registration, the application type being Web app / API and the Sign-on URL anything being any valid URL (just the format, it doesn't have to exist). If the name you enter isn't simple to remember then it would be a good idea to take a note of it for step 2.
Take note of the Application ID as that will be your Client ID for authentication, then select the Keys blade under Settings. Enter a Key description (preferably indicating the user or application that will be using this key), select a duration and Save. Immediately store the resulting value somewhere safe as it will never be displayed again.
I recommend adding yourself as an owner on the Owners blade (also under Settings), whatever else this may be good for it'll let you see the app registration immediately on the App registrations blade without having to select "All apps".
Step 2: Key Vault permissions
Open the Key Vault in the Azure Portal and select the Access policies blade under Settings. Click Add New and click on Select principal - you'll have to enter the full name of the registered app you created in the previous step in the search box before it'll show up, at which point you'll be able to select it.
You can either select an appropriate template from the top dropdown or choose Key, Secret or Certificate permissions manually. Don't worry about Authorized application at this stage.
IMPORTANT: pressing the OK button will add your new policy to the list, but it will not be saved! Be sure to click Save before continuing.
Step 3: Accessing the Key Vault from your Code
There are many different ways to authenticate, most of them obscure and undocumented. This is the simplest method, I've put the credentials in the code for clarity but I have faith that you'll store them somewhere more intelligent. Never store credentials in the codebase. Seriously. Just don't.
Sunday, 19 November 2017
'How to create an Azure SQL Database programmatically' with less frustration
In addition to the headache of setting up an Azure subscription and Azure Active Directory correctly, it took a silly amount of investigation and trial and error before I could figure out what values the code was expecting as the variable names (almost predictably) don't match their counterparts as displayed in the Azure portal.
- Instructions:
- Follow the instructions from the original code's page up to item 4 ("Add your variables to the program")
- Replace Program.cs with my modified code.
- Follow the instructions in the 4th item, using the modified code's documentation in case of confusion or ambiguity.
Friday, 12 May 2017
connecting node.js to the azure table storage emulator
- The npm azure-storage package instructions are found here, the emulator software is found here and the storage explorer is found here.
- Once the emulator has been installed, you'll need to start it. The init operation worked fine for me (I'm running SQL Server 2012 Express anyway), but the start operation failed and it took a while to realize that ports 10000 - 10002 (or is that 3?) need to be available; the software blocking could be anywhere from backup software or bittorrent to malware.
Good to know.
There doesn't appear to be any way to customize the ports used. - To verify that your emulator is running correctly, connect using the storage explorer.
- Select "Use a storage account name and key"
- Set the account name and authentication key (see point 6 below)
- Set the storage endpoints domain to "Other" with a value of 127.0.0.1
- Select "Use HTTP"
- The emulator runs on http NOT https, which shouldn't affect you once you've got your connection configured correctly. For some people the authentication requires setting your system time to UTC / GMT, for others it's setting the environment variable NODE_TLS_REJECT_UNAUTHORIZED to 0; the latter can be done in node.js with
process.env.NODE_TLS_REJECT_UNAUTHORIZED = 0;
- There are two ways to instantiate a table service object. Assuming
var azure = require('azure-storage');var tableSvc = azure.createTableService(account, key, '127.0.0.1:10002');
orprocess.env.AZURE_STORAGE_ACCOUNT = account; process.env.AZURE_STORAGE_ACCESS_KEY = key; process.env.AZURE_STORAGE_CONNECTION_STRING = connectionString; var tableSvc = azure.createTableService();
- The account name, authentication key and connection string are public, invariable and for emulation purposes only:
Account name: devstoreaccount1
Authentication Key:
Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
Connection String: UseDevelopmentStorage=true
Sunday, 23 April 2017
How to code Entity Framework programmable migrations
(And for poor me, too, I'm very lazy and I like being able to just press a button to make magic happen.)
The most important quirk of running migrations programmatically that you'll need to know before we begin is that the DbMigrator class, for reasons unclear, requires the Entity Framework Configuration class but uses it in a way that's incompatible with the Update-Database and Add-Migration scripts, even after manually configuring the ContextType and ContextKey properties. We'll set those anyway for consistency, but no matter where you store your migrations and however diligently you inform the DbMigrator (the MigrationsDirectory property was ignored too) you're going to be stuck with automatic migrations. And while we're fiddling with the Configuration class, be sure to make the internal Configuration class public.
Automatic migrations ignore the explicit migration files and detect changes between the context's models and the database. As with the Update-Database script, the database will be created if it hasn't been already. It will not function reliably unless the AutomaticMigrationDataLossAllowed property is set to true, and you'll also need to update the OnModelCreating method of your context class to allow for database calls when the models are out of sync to give you a chance to run the migrations:
Migrating up and down
Using automatic migrations to update your database is great, but what do you do if you need to roll back? DbMigrator requires a migration name to update to, and unlike the case with explicit migrations, the migration name is not established by the developer but is unique to the database in question.
Fortunately, the DbMigrator includes the GetDatabaseMigrations method, which returns a list of applied migrations; so while you won't be able to roll back to a predefined named state, you will be able to roll back to a previously run migration. Here it is important to note that the order of the migrations returned is not guaranteed, but the names begin with a timestamp so they're not too difficult to sort.
So rolling back the first of that list is as simple as
Seeding
Seeding must be performed manually after calling DbMigrator.Update(). The Configuration class' Seed method is protected, so add the following wrapper method to your Configuration class
and call it once the migration is complete.
Thursday, 26 January 2017
Passing data from C# to Node.js via Edge.js
public async Task<dynamic> Invoke(dynamic input) {
// echo input message
await doSomething(input.message);
}
Returning data is not trivial, however, because any kind of complex object I tried to return caused unreported exceptions.
Until now!
It turns out that Edge .js returns associative arrays if and only if they're correctly JSON formatted. I don't recall seeing this documented anywhere, but it makes perfect sense (20/20 hindsight). So in order to return an associative array to Node.js, make sure to use a Dictionary object where the first element type is a string; if you want to send a list of usernames with their IDs, for example, you would either use
Dictionary<string, string>
with the ID cast to string for transmission or
Dictionary<string, int>
with the ID / username columns switched from their normal positions.
Of course, one could go about creating more intricate objects using the dynamic type, but that's beyond the scope of this post.
Saturday, 26 November 2016
Oh! About that brilliant thing you did...
I've struggled with this in plenty of teams I've worked with in the past, but there's something particularly infuriating about what I've just struggled with. Microsoft's Entity Framework Code-First tools makes designing and updating databases automagic; the only reason I've had to use the database directly at all was to verify that the tables are being created and updated intelligently and that my test data is being stored correctly.
BUT.
I have been struggling for hours with self-referential foreign keys. Not because it's difficult to set up, but because the only helpful, readable instruction I've found online has been this answer here which I found after a lot of searching for "Entity Framework code-first self-referencing foreign keys" and coming across tutorials and MSDN documentation and forums (including stackoverflow) which are so dense and filled with any other kind of relations or answers that over-complicate things...
I salute those who make awesome software, and you people certainly know who you are. For the love of code that is elegant and holy, please share with the time-pressured slower kids in the class. We'll all win.
Monday, 29 July 2013
Comment Driven Coding
Background
I've always thought that I had this commenting thing down. When my teachers in high school and in university told me I needed to comment my code, I listened! They were satisfied, too. I took up tutoring whenever I couldn't find work, and I adamantly preached what I practised. Nobody was going to be able to call me an inconsiderate coder.That is, until I entered the army. My initial exposure to the comment style in an environment with high developer turnover and long-term maintenance responsibilities was like being picked up by a ship after days stranded at sea. For the first time I really understood the value of a well-placed, meaningful comment. I learned to expect comments whenever the code became even a little tricky, and I learned to be safe rather than sorry.
After a couple of years in the army, I transferred to the real deal: real-time avionics development. If the army comments were like a ship, this was like finding myself on dry land! I was shocked by the extreme attitude, and it took me a while to get to grips with it. For the first time in my life, I was writing comments for every single statement.
Every... single... statement. Let that sink in a moment. Even though each comment had to be meaningful - so instead of "add 1 to i" it would be "increment the loop counter" - this meant that there was far more comment text than code in any given source file. At first it was a bit of a headache, but soon the comments and the code began to blend and I found myself feeling far more comfortable with these complex systems than I'd ever been with any others. Nothing was left to the imagination, no guessing required. The things that didn't make any sense? They'd let you know precisely why.
After years of this I returned to the real world, the perpetually rushed, highly-stressed and highly caffeinated world of 25-hour work days led by marketing teams who promise the impossible to be delivered yesterday using codebases and frameworks that were built exclusively for the most antisocial misogynistic masochist enthusiasts and are generally worked on by whoever has the highest grades in the only-partially-related academic field of computer science.
These people do not have time for comments. They usually have a very specific idea of what comments look like, those unhelpful things that tell you what the code does. There's a popular opinion that good code is so readable that it doesn't need to be commented! But unfortunately, that's only partially true.
You see, and here I'm going to wind down on the personal history and get to the point of this post, good, clean code might tell you what it does but it doesn't tell you what it's supposed to do. It doesn't expose the logic of a group of statements, and it certainly doesn't provide an unambiguous guide to the code you're writing under the gun.
Method
Comment Driven Coding is summarized in four easy steps:1. Comment before you code
2. Comment the desired behaviour in English
3. Split your comments into logical steps.
4. Repeat until your comments describe the smallest reasonable description of logical behaviour.
One level above pseudo-code, use comments as a way to structure your code prior to implementation. This will handle all of the logic before you let syntax and details get in the way. Begin with the higher level functionality and drill-down until you've fleshed out the details.
[EXAMPLE FUNCTION INSPIRED BY THIS POST]
/* Reduced Sum Of Digits (RSOD) is calculated by repeatedly summing the digits of a number until a single digit is produced. */
function ReduceToRSOD(number) {
// return the RSOD
} // ReduceToRSOD
The first things to notice here are that we have explained RSOD clearly, we have explained the inner logic in the broadest sense possible and we have used a comment to explicitly mark the function of the closing brace. This last step may seem extreme, but as the number of braces and the size and complexity of the code increase it will become more difficult to keep track of the code blocks no matter how well the code is indented.
/* Reduced Sum Of Digits (RSOD) is calculated by repeatedly summing the digits of a number until a single digit is produced. */
function ReduceToRSOD(number) {
// if the number is comprised of a single digit, return it
// if the number is comprised of multiple digits
// separate the digits and add them together
// recursively apply the ReduceToRSOD function
} // ReduceToRSOD
By doing this using comments instead of code, logical errors or missed cases will be far easier to spot.
/* Reduced Sum Of Digits (RSOD) is calculated by repeatedly summing the digits of a number until a single digit is produced. */
function ReduceToRSOD(number) {
// return an error code if the number is a not positive integer
// if the number is comprised of a single digit, return it
// if the number is comprised of multiple digits
// separate the digits and add them together
// while the number has more digits
// add the last digit to a result variable
// remove the last digit from the number
// recursively apply the ReduceToRSOD function
} // ReduceToRSOD
The comments not only make the code readable, but they also function as the perfect tool for performing a code review. Does the comment logic make sense? Does the code do what the comments describe?
var INVALID_NUMBER = -1;
/* Reduced Sum Of Digits (RSOD) is calculated by repeatedly summing the digits of a number until a single digit is produced. */
function ReduceToRSOD(number) {
// return an error code if the number is a not positive integer
if ((parseInt(number)== Number.Nan) || (number < 0)) return INVALID_NUMBER;
// if the number is comprised of a single digit, return it
if (number < 10) return number;
// if the number is comprised of multiple digits
// separate the digits and add them together
var result = 0;
// while the number has more digits
while (number > 0) {
// add the last digit to a result variable
result += number % 10;
// remove the last digit from the number
number = (number - (number % 10)) / 10;
} // while the number has more digits
// recursively apply the ReduceToRSOD function
return ReduceToRSOD(result);
} // ReduceToRSOD
A well-written comment never gets old. The code might change, but what it's supposed to be doing will not.