Resolving private PyPI configuration issues

The Story

I've spent the last few days wrestling with the serverless framework, growing fonder and fonder of AWS' CDK at every turn. I appreciate that serverless has been, until recently, very necessary and very useful, but I feel confident in suggesting that when deployment mechanisms become more complex than the code they're deploying things aren't moving in the right direction.

That said, this post is about overcoming what for us was the final hurdle: getting Docker able to connect to a private PyPI repository for our Python lambdas and lambda layers.

Python is great, kind of, but is marred by two rather severe complications: the split between python and python3, and pip… so it's a great language, with an awful tooling experience. Anyway, our Docker container couldn't communicate with our PyPI repo, it took way too long to figure out why, here's what we learned:

The Solution

If you want to use a private PyPI repository without typing in your credentials at every turn, there are two options:

1.

~/.pip/pip.conf:

2.

~/.pip/pip.conf:



~/.netrc:



It is important that ~/.netrc has the same owner:group as pip.conf, and that its permissions are 0600 (chmod 0600 ~/.netrc).

What's not obvious - or even discussed anywhere - is that special characters are problematic and are handled differently by the two mechanisms.

In pip.conf, the password MUST be URL encoded.
In .netrc, the password MUST NOT be URL encoded.

The Docker Exception

For whatever reason, solution 2 (the combination of pip.conf and .netrc) does NOT work with Docker.

Conclusion

Amazon's CDK is excellent, and unless you have a very specific use-case that it doesn't support it really is worth trying out!

Oh! And that Python is Very Nice, but simply isn't nice enough to justify the cost of its tooling.