Resolving private PyPI configuration issues

The Story

I've spent the last few days wrestling with the serverless framework, growing fonder and fonder of AWS' CDK at every turn. I appreciate that serverless has been, until recently, very necessary and very useful, but I feel confident in suggesting that when deployment mechanisms become more complex than the code they're deploying things aren't moving in the right direction.

That said, this post is about overcoming what for us was the final hurdle: getting Docker able to connect to a private PyPI repository for our Python lambdas and lambda layers.

Python is great, kind of, but is marred by two rather severe complications: the split between python and python3, and pip… so it's a great language, with an awful tooling experience. Anyway, our Docker container couldn't communicate with our PyPI repo, it took way too long to figure out why, here's what we learned:

The Solution

If you want to use a private PyPI repository without typing in your credentials at every turn, there are two options:

1.

~/.pip/pip.conf:

2.

~/.pip/pip.conf:



~/.netrc:



It is important that ~/.netrc has the same owner:group as pip.conf, and that its permissions are 0600 (chmod 0600 ~/.netrc).

What's not obvious - or even discussed anywhere - is that special characters are problematic and are handled differently by the two mechanisms.

In pip.conf, the password MUST be URL encoded.
In .netrc, the password MUST NOT be URL encoded.

The Docker Exception

For whatever reason, solution 2 (the combination of pip.conf and .netrc) does NOT work with Docker.

Conclusion

Amazon's CDK is excellent, and unless you have a very specific use-case that it doesn't support it really is worth trying out!

Oh! And that Python is Very Nice, but simply isn't nice enough to justify the cost of its tooling.

Handling emails with node.js and Mailparser

After sorting out mail forwarding and piping emails with postfix, I then needed to understand how to handle emails being POSTed to an API endpoint.

To parse an email with node.js, I recommend using Mailparser's simpleParser. I'm using express with bodyParser configured as follows:

app.use(bodyParser.json({
 limit : config.bodyLimit
}));

In your handler:

const express = require('express');
const router = express.Router();
const simpleParser = require('mailparser').simpleParser;

or

import { Router } from 'express';
import { simpleParser } from 'mailparser';

and then

api.post('/', (req, res) => {
    simpleParser(req)
        .then(parsed => {
            res.json(parsed);
        })
        .catch(err => {
            res.json(500, err);
        });
});

Mailparser is excellent, and documented, but the documentation assumes that we're familiar with the email format. Fortunately, oblac's example email exists for those of us who aren't!

To test, send the example email to the endpoint via curl:

curl --data-binary "@./example.eml" http://your-domain-name/api/email

or Postman (attach file to "binary").

And we're good to go!

Setting up Squiz Labs code sniffer for PHPStorm on Windows

PHPStorm is a brilliant piece of software that's highly extensible. One of the extensions that's particularly useful is Squiz Labs' PHP Code Sniffer. It wasn't quite apparent from other documentation I found online that it's really simple to set up for Windows! So with the help of a co-worker I put together the following tutorial:

To install the code sniffer for PHPStorm:

• Download and install PHP for Windows
• Download https://squizlabs.github.io/PHP_CodeSniffer/phpcs.phar
• Create a file called phpcs.bat in the C:\php directory
  @echo off
  "C:\php\php.exe" -f "C:\PATH_TO_DOWNLOADED_PHAR\phpcs.phar" -- %*
• In PHPStorm
• Open File->Settings
• Select Languages & Frameworks->PHP
• Add a local interpreter that points to C:\php\php.exe
• Select Languages & Frameworks->PHP->Code Sniffer
• Update the local configuration to point to C:\php\phpcs.bat
• increase the Tool process timeout if you're working on remote code
• Click Validate to ensure that the code sniffer is loaded correctly
• Click Apply
• Select Editor->Inspections
• Select the checkbox for PHP Code Sniffer validation
• Select PHP Code Sniffer validation
• Set the coding standard (you'll probably want PSR2)
• To run the code sniffer
• Press CTRL + ALT + SHIFT + I
• Type "code sniffer" and press enter
• Select the current file

And there you go!