C# / OpenSSH RSA Encryption made easy

[EDIT: please see C# / OPENSSH RSA ENCRYPTION MADE EVEN EASIER]

The struggle to uncover the secrets of importing from and export to OpenSSH keys with Microsoft's .NET RSACryptoServiceProvider is real. It's possible but not practical to do this without BouncyCastle, which may or may not be well-documented (navigating their website is far from a joyful experience), but after trawling the web and playing around I've created the following gists that should be of assistance to anyone who needs to do this in a straightforward manner.
And if you want to share RSA keys between JavaScript and .NET platforms, well, you're going to need to do this.
Import and export RSA Keys between C# and PEM format using BouncyCastle
And just because the actual encryption and decryption are always annoying:
Simple RSA Encryption to and Decryption from Base64 encoded strings in C#

connecting node.js to the azure table storage emulator

Once again, solid software is rendered useless by a lack of documentation. I've just wasted way too much time connecting my node app to an emulated table, so I'm going to spell it out for you so that you don't have to do the same.

1. The npm azure-storage package instructions are found here, the emulator software is found here and the storage explorer is found here.
   
2. Once the emulator has been installed, you'll need to start it. The init operation worked fine for me (I'm running SQL Server 2012 Express anyway), but the start operation failed and it took a while to realize that ports 10000 - 10002 (or is that 3?) need to be available; the software blocking could be anywhere from backup software or bittorrent to malware.
   Good to know.
   There doesn't appear to be any way to customize the ports used.
   
3. To verify that your emulator is running correctly, connect using the storage explorer.
   1. Select "Use a storage account name and key"
   2. Set the account name and authentication key (see point 6 below)
   3. Set the storage endpoints domain to "Other" with a value of 127.0.0.1
   4. Select "Use HTTP"
      
4. The emulator runs on http NOT https, which shouldn't affect you once you've got your connection configured correctly. For some people the authentication requires setting your system time to UTC / GMT, for others it's setting the environment variable NODE_TLS_REJECT_UNAUTHORIZED to 0; the latter can be done in node.js with

   process.env.NODE_TLS_REJECT_UNAUTHORIZED = 0;
   
   

5. There are two ways to instantiate a table service object. Assuming

   var azure = require('azure-storage');

   1. var tableSvc = azure.createTableService(account, key, '127.0.0.1:10002');

      or

   2. process.env.AZURE_STORAGE_ACCOUNT = account;
      
      process.env.AZURE_STORAGE_ACCESS_KEY = key;
      
      process.env.AZURE_STORAGE_CONNECTION_STRING = connectionString;
      
      var tableSvc = azure.createTableService();

6. The account name, authentication key and connection string are public, invariable and for emulation purposes only:
   Account name: devstoreaccount1
   
   Authentication Key:
   Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
   
   Connection String: UseDevelopmentStorage=true

How to code Entity Framework programmable migrations

The internets are eerily quiet when it comes to running Entity Framework's migrations through code, which is odd because Entity Framework's Code First migrations are extremely powerful and calling them (or rolling them back) through the code seems an obvious choice to someone like me, who prefers to reduce the amount of complexity for the poor IT people handling deployment.

(And for poor me, too, I'm very lazy and I like being able to just press a button to make magic happen.)

The most important quirk of running migrations programmatically that you'll need to know before we begin is that the DbMigrator class, for reasons unclear, requires the Entity Framework Configuration class but uses it in a way that's incompatible with the Update-Database and Add-Migration scripts, even after manually configuring the ContextType and ContextKey properties. We'll set those anyway for consistency, but no matter where you store your migrations and however diligently you inform the DbMigrator (the MigrationsDirectory property was ignored too) you're going to be stuck with automatic migrations. And while we're fiddling with the Configuration class, be sure to make the internal Configuration class public.

Automatic migrations ignore the explicit migration files and detect changes between the context's models and the database. As with the Update-Database script, the database will be created if it hasn't been already. It will not function reliably unless the AutomaticMigrationDataLossAllowed property is set to true, and you'll also need to update the OnModelCreating method of your context class to allow for database calls when the models are out of sync to give you a chance to run the migrations:





Migrating up and down

Using automatic migrations to update your database is great, but what do you do if you need to roll back? DbMigrator requires a migration name to update to, and unlike the case with explicit migrations, the migration name is not established by the developer but is unique to the database in question.

Fortunately, the DbMigrator includes the GetDatabaseMigrations method, which returns a list of applied migrations; so while you won't be able to roll back to a predefined named state, you will be able to roll back to a previously run migration. Here it is important to note that the order of the migrations returned is not guaranteed, but the names begin with a timestamp so they're not too difficult to sort.



So rolling back the first of that list is as simple as



Seeding

Seeding must be performed manually after calling DbMigrator.Update(). The Configuration class' Seed method is protected, so add the following wrapper method to your Configuration class



and call it once the migration is complete.

Passing data from C# to Node.js via Edge.js

Edge.js is a phenomenal tool, but its documentation is a hit and miss.  Sending data to C# from Node.js is straightforward, and the JSON object is translated to a dynamic object automatically:

public async Task<dynamic> Invoke(dynamic input) {
   // echo input message
   await doSomething(input.message);
}

Returning data is not trivial, however, because any kind of complex object I tried to return caused unreported exceptions.

Until now!

It turns out that Edge .js returns associative arrays if and only if they're correctly JSON formatted. I don't recall seeing this documented anywhere, but it makes perfect sense (20/20 hindsight). So in order to return an associative array to Node.js, make sure to use a Dictionary object where the first element type is a string; if you want to send a list of usernames with their IDs, for example, you would either use

Dictionary<string, string>

with the ID cast to string for transmission or

Dictionary<string, int>

with the ID / username columns switched from their normal positions.

Of course, one could go about creating more intricate objects using the dynamic type, but that's beyond the scope of this post.

Web Worker callback design pattern

I couldn't find a way to post messages to a callback and define a handler for each individual message, and the following is what I've come up with. If you have suggestions or a better solution, let me know!

UPDATE: my goodness, looking back at my original post I wonder if I wasn't having a stroke. I've rewritten it and used gist to make it readable:

Oh! About that brilliant thing you did...

... if you write an ingenious piece of software that streamlines a process or makes everyone's lives easier, but you don't TELL anyone about it or explain concisely how to use it, then you may as well not have written it in the first place. Because it won't MATTER.

I've struggled with this in plenty of teams I've worked with in the past, but there's something particularly infuriating about what I've just struggled with. Microsoft's Entity Framework Code-First tools makes designing and updating databases automagic; the only reason I've had to use the database directly at all was to verify that the tables are being created and updated intelligently and that my test data is being stored correctly.

BUT.

I have been struggling for hours with self-referential foreign keys. Not because it's difficult to set up, but because the only helpful, readable instruction I've found online has been this answer here which I found after a lot of searching for "Entity Framework code-first self-referencing foreign keys" and coming across tutorials and MSDN documentation and forums (including stackoverflow) which are so dense and filled with any other kind of relations or answers that over-complicate things...

I salute those who make awesome software, and you people certainly know who you are. For the love of code that is elegant and holy, please share with the time-pressured slower kids in the class. We'll all win.

Without or without certificates - an idiot's guide to end-to-end web encryption

There are many cases where HTTPS / SSL isn't an option, and usually when that happens everything, including sensitive data, is sent plain text. Additionally, most websites still use passwords instead of external authentication and password policies and storage that were outdated years ago.

The risk is entirely unnecessary.

Implementing the most crucial elements of HTTPS with a combination of client and server security is non-trivial but inexpensive, and can protect your users and your organization from most types of attacks. The following protocol is based on existing protocols and algorithms - the first rule of crypto is "don't roll your own" - and as I am a crypto amateur I believe I can explain the steps in terms simple enough for most developers.

I have illustrated the implementation of this protocol in pure Javascript and Node.js here.

---

1. Session initiation

The only plaintext transmission is the initial transfer of the username / email address.

As a session initiation request, the client transmits the username / email address to the server, which is used to create a server-side session object identified by a token (preferably a GUID).

The server looks up the user's email address and any other authentication factor that can receive messages (ie other email addresses, mobile devices, etc) and sends them a verification code which will be used as a shared secret when authenticating. To this end the verification code must be stored in the session object.

This verification code must never be returned to the client directly.

The server responds to the session initiation request with the session identification token.

2. Session authentication

The first step to properly securing the communication is exchanging RSA public keys which will be used to protect the AES passphrases. Because the public keys must be exchanged before they can be used, the verification code sent to the user in the previous step will be the first AES passphrase.

AES 256 requires a passphrase 32 bytes (256 bits) long. We convert the verification code to 32 bytes by hashing it with the MD5 algorithm.

From this point on, every transmission will be encrypted using AES (specifically AES-256 in CTR mode). Every transmission must include the session token and an encrypted payload object. The payload object must include a timestamp, to ensure that the transmission was received in an acceptable amount of time, and a nonce, which is a randomly generated string used to prevent replay attacks.

 

The client generates an RSA private / public key pair and adds it to the payload. For browser calls, store the client keys in an HTML Local Storage Object.

At this point it is possible to include the user's password in the payload if one is required for authentication, although it is advisable to rely on other forms of authentication instead such as external applications (Duo, Authy), OAuth providers (Google, Facebook) or whatever else is available and practical with regards the user experience.

The payload is encrypted using the hashed verification code and is sent to the server along with the session token.

The server decrypts the payload using the hashed verification code for the session and validates the payload. Validating the payload includes testing that the timestamp isn't too old and the nonce against a list of nonces that the session has accepted.

If  validation is successful the session stores the client's public key and generates its own RSA key pair which will be used until the session expires.

The server generates a new AES passphrase and uses it to encrypt the session's public key. This will be returned to the client along with the AES passphrase encrypted using the client's public key. At this point only the client will be able to decrypt the AES passphrase protecting the session's public key.

3. Closure

The client decrypts the AES passphrase and stores the session's public key. From this point on, every transmission to the server will include the following:

a) the session token

b) a newly generated AES passphrase (or key) encrypted using the session's public key

c) the payload, including timestamp and nonce, encrypted using the generated AES passphrase

And from this point on the response format will be as follows:

a) a newly generated AES passphrase (or key) encrypted using the client's public key

b) the payload encrypted using the generated AES passphrase

---

Even when HTTPS is available, with the current computing power available the performance cost of this form of encryption is quite small and can add an extra layer of protection.

I have illustrated the implementation of this protocol in pure Javascript and Node.js here.

If you have any questions, suggestions or criticism, please drop me a line in the comments!

Write a destructor in PHP for fatal error handling

This just posted on stackoverflow.

I recently had trouble with this as I was trying to handle destruction specifically for the case where the server experiences a timeout and I wanted to include class data in the error log. I would receive an error when referencing &$this (although I've seen it done in a few examples, possibly a version issue or a symfony side-effect), and the solution I came up with was fairly clean:

    class MyClass
    {
        protected $myVar;

        /**
         * constructor, registers shutdown handling
         */
        public function __construct()
        {
            $this->myVar = array();

            // workaround: set $self because $this fails
            $self = $this;
            // register for error logging in case of timeout
            $shutdown = function () use (&$self) {
                $self->shutdown();
            };
            register_shutdown_function($shutdown);
        }

        /**
         * handle shutdown events
         */
        public function shutdown()
        {
            $error = error_get_last();
            // if shutdown in error
            if ($error['type'] === E_ERROR) {
                // write contents to error log
                error_log('MyClass->myVar on shutdown' . json_encode($this->myVar), 0);
            }
        }
     
        ...

Hope this helps!

Setting up Squiz Labs code sniffer for PHPStorm on Windows

PHPStorm is a brilliant piece of software that's highly extensible. One of the extensions that's particularly useful is Squiz Labs' PHP Code Sniffer. It wasn't quite apparent from other documentation I found online that it's really simple to set up for Windows! So with the help of a co-worker I put together the following tutorial:

To install the code sniffer for PHPStorm:

• Download and install PHP for Windows
• Download https://squizlabs.github.io/PHP_CodeSniffer/phpcs.phar
• Create a file called phpcs.bat in the C:\php directory
  @echo off
  "C:\php\php.exe" -f "C:\PATH_TO_DOWNLOADED_PHAR\phpcs.phar" -- %*
• In PHPStorm
• Open File->Settings
• Select Languages & Frameworks->PHP
• Add a local interpreter that points to C:\php\php.exe
• Select Languages & Frameworks->PHP->Code Sniffer
• Update the local configuration to point to C:\php\phpcs.bat
• increase the Tool process timeout if you're working on remote code
• Click Validate to ensure that the code sniffer is loaded correctly
• Click Apply
• Select Editor->Inspections
• Select the checkbox for PHP Code Sniffer validation
• Select PHP Code Sniffer validation
• Set the coding standard (you'll probably want PSR2)
• To run the code sniffer
• Press CTRL + ALT + SHIFT + I
• Type "code sniffer" and press enter
• Select the current file

And there you go!

Comment Driven Coding

This is undoubtedly my favourite trick, and oddly enough it's the one that drives experienced hackers round the bend!

Background

I've always thought that I had this commenting thing down. When my teachers in high school and in university told me I needed to comment my code, I listened! They were satisfied, too. I took up tutoring whenever I couldn't find work, and I adamantly preached what I practised. Nobody was going to be able to call me an inconsiderate coder.

That is, until I entered the army. My initial exposure to the comment style in an environment with high developer turnover and long-term maintenance responsibilities was like being picked up by a ship after days stranded at sea. For the first time I really understood the value of a well-placed, meaningful comment. I learned to expect comments whenever the code became even a little tricky, and I learned to be safe rather than sorry.

After a couple of years in the army, I transferred to the real deal: real-time avionics development. If the army comments were like a ship, this was like finding myself on dry land! I was shocked by the extreme attitude, and it took me a while to get to grips with it. For the first time in my life, I was writing comments for every single statement.

Every... single... statement. Let that sink in a moment. Even though each comment had to be meaningful - so instead of "add 1 to i" it would be "increment the loop counter" - this meant that there was far more comment text than code in any given source file. At first it was a bit of a headache, but soon the comments and the code began to blend and I found myself feeling far more comfortable with these complex systems than I'd ever been with any others. Nothing was left to the imagination, no guessing required. The things that didn't make any sense? They'd let you know precisely why.

After years of this I returned to the real world, the perpetually rushed, highly-stressed and highly caffeinated world of 25-hour work days led by marketing teams who promise the impossible to be delivered yesterday using codebases and frameworks that were built exclusively for the most antisocial misogynistic masochist enthusiasts and are generally worked on by whoever has the highest grades in the only-partially-related academic field of computer science.

These people do not have time for comments. They usually have a very specific idea of what comments look like, those unhelpful things that tell you what the code does. There's a popular opinion that good code is so readable that it doesn't need to be commented! But unfortunately, that's only partially true.

You see, and here I'm going to wind down on the personal history and get to the point of this post, good, clean code might tell you what it does but it doesn't tell you what it's supposed to do. It doesn't expose the logic of a group of statements, and it certainly doesn't provide an unambiguous guide to the code you're writing under the gun.

Method

Comment Driven Coding is summarized in four easy steps:
1. Comment before you code
2. Comment the desired behaviour in English
3. Split your comments into logical steps.
4. Repeat until your comments describe the smallest reasonable description of logical behaviour.
One level above pseudo-code, use comments as a way to structure your code prior to implementation. This will handle all of the logic before you let syntax and details get in the way. Begin with the higher level functionality and drill-down until you've fleshed out the details.

[EXAMPLE FUNCTION INSPIRED BY THIS POST]

/* Reduced Sum Of Digits (RSOD) is calculated by repeatedly summing the digits of a number until a single digit is produced. */
function ReduceToRSOD(number) {
  // return the RSOD
} // ReduceToRSOD

The first things to notice here are that we have explained RSOD clearly, we have explained the inner logic in the broadest sense possible and we have used a comment to explicitly mark the function of the closing brace. This last step may seem extreme, but as the number of braces and the size and complexity of the code increase it will become more difficult to keep track of the code blocks no matter how well the code is indented.

/* Reduced Sum Of Digits (RSOD) is calculated by repeatedly summing the digits of a number until a single digit is produced. */
function ReduceToRSOD(number) {
  // if the number is comprised of a single digit, return it
  // if the number is comprised of multiple digits
    // separate the digits and add them together
    // recursively apply the ReduceToRSOD function
} // ReduceToRSOD

By doing this using comments instead of code, logical errors or missed cases will be far easier to spot.

/* Reduced Sum Of Digits (RSOD) is calculated by repeatedly summing the digits of a number until a single digit is produced. */
function ReduceToRSOD(number) {
  // return an error code if the number is a not positive integer
  // if the number is comprised of a single digit, return it
  // if the number is comprised of multiple digits
    // separate the digits and add them together
    // while the number has more digits
      // add the last digit to a result variable
      // remove the last digit from the number
    // recursively apply the ReduceToRSOD function
} // ReduceToRSOD

The comments not only make the code readable, but they also function as the perfect tool for performing a code review. Does the comment logic make sense? Does the code do what the comments describe?
var INVALID_NUMBER = -1;

/* Reduced Sum Of Digits (RSOD) is calculated by repeatedly summing the digits of a number until a single digit is produced. */
function ReduceToRSOD(number) {
  // return an error code if the number is a not positive integer
  if ((parseInt(number)== Number.Nan) || (number < 0)) return INVALID_NUMBER;
  // if the number is comprised of a single digit, return it
  if (number < 10) return number;
  // if the number is comprised of multiple digits
    // separate the digits and add them together
    var result = 0;
    // while the number has more digits
    while (number > 0) {
      // add the last digit to a result variable
      result += number % 10;
      // remove the last digit from the number
      number = (number - (number % 10)) / 10;
    } // while the number has more digits
    // recursively apply the ReduceToRSOD function
    return ReduceToRSOD(result);
} // ReduceToRSOD

 

The reason this works so well is that even the brightest and most experienced programmers don't truly think in code. Our brains are wired for language and the easiest way to look at a problem is when it's expressed in words or images.

A well-written comment never gets old. The code might change, but what it's supposed to be doing will not.

 

At first glance it may look and sound like it'll take longer to write your code this way, but it really won't. It's quicker to write in natural language, and once the ideas have been broken down into their simplest logical components the code practically writes itself!

tic tac toe logic

the standard game of tic tac toe is played on a 3x3 board. this is the trivial case. the diagrams shown as examples apply to the trivial case. this method is appropriate for a game played on a larger board, hence the more generic definitions.

it is irrelevant whether both players are handled by the computer, or only one. each turn will see the definition of player and opponent switch, so if in the first turn the player is X and the opponent is O, then in the second turn the player will be O and the opponent X. in each turn the weight grid must be calculated anew.

the game grid

(1,1) | (1,2) | (1,3)
---------------------
(2,1) | (2,2) | (2,3)
---------------------
(3,1) | (3,2) | (3,3)

potential line cell (PL)

an empty cell which is on the same line as either an X or an O (not both on the same line), but if controlled by the same player (X if the other cells contain Xs, O if the other cells contain Os) will not finish the game

 X | PL| PL
-----------
   | PL|   
-----------
 O | PL| PL


in the above example (2,1) is on a line containing both X and O (1,3),(2,2),(3,3) are each found on distinct lines each containing one or the other

opponent's completion move cell (OC)

an empty cell which, if filled by the opponent will win him the game
   |   |   
-----------
 O | O | OC
-----------
   |   |   

in the above example the player is X

player's completion move cell (PC)

an empty cell which, if filled by the player will win him the game
   | X |   
-----------
   | PC|   
-----------
   | X |   

in the above example the player is X

the weights are calculated as follows:

+1

for a potential line cell

+10 (width * height + 1)

for an opponent's completion move cell (X)

+100 ((width * height + 1) ^ 2)

for the player's completion move cell (O)

the reasoning behind the weights is that we do not want the possibility, in the case of a larger grid, of a number of crossing potential lines (which are additive) to interfere with the priority of closing off an opponent's completion, or for a number of potential opponent completions to interfere with a player completion (winning the game). regardless of the size of the grid, (width * height + 1) and (width * height + 1)^2 will always be safe values. the basic method is fairly simple. we construct a weight grid with the same dimensions as the game grid to contain the weights, with each cell initialized to zero.

initial state for the turn
 game grid      weight grid
 X | X |         0 | 0 | 0 
-----------     -----------
   |   |         0 | 0 | 0 
-----------     -----------
 O |   |         0 | 0 | 0 

we then go over each line, modifying the line's cell's weights accordingly. in the figures below, it is player O's turn. if it was player X's turn, then cell (1,3) would be weighted 101 instead of 11 by the end of the process.

if the line contains more than one available cell, and contains either Os or Xs (in the trivial case, if the line contains exactly one filled cell), then any available cells are on the potential line, and their counterpart cells in the weight grid are each incremented by 1.

if the line contains exactly one available cell, and the rest of the line's cells are controlled by the opponent, then the available cell is incremented by (width * height + 1), or 10 in the trivial case.

if the line contains exactly one available cell, and the rest of the line's cells are controlled by the player, then the available cell is incremented by (width * height + 1)^2, or 100 in the trivial case.

row 1
 game grid      weight grid
 X | X |         0 | 0 | 10
-----------     -----------
   |   |         0 | 0 | 0 
-----------     -----------
 O |   |         0 | 0 | 0 

row 2
 game grid      weight grid
 X | X |         0 | 0 | 10
-----------     -----------
   |   |         0 | 0 | 0 
-----------     -----------
 O |   |         0 | 0 | 0 

row 3
 game grid      weight grid
 X | X |         0 | 0 | 10
-----------     -----------
   |   |         0 | 0 | 0 
-----------     -----------
 O |   |         0 | 1 | 1 

column 1
 game grid      weight grid
 X | X |         0 | 0 | 10
-----------     -----------
   |   |         0 | 0 | 0 
-----------     -----------
 O |   |         0 | 1 | 1 

column 2
 game grid      weight grid
 X | X |         0 | 0 | 10
-----------     -----------
   |   |         0 | 1 | 0 
-----------     -----------
 O |   |         0 | 2 | 1 

column 3
 game grid      weight grid
 X | X |         0 | 0 | 10
-----------     -----------
   |   |         0 | 1 | 0 
-----------     -----------
 O |   |         0 | 2 | 1 

diagonal 1
 game grid      weight grid
 X | X |         0 | 0 | 10
-----------     -----------
   |   |         0 | 2 | 0 
-----------     -----------
 O |   |         0 | 2 | 2 


diagonal 2
 game grid      weight grid
 X | X |         0 | 0 | 11
-----------     -----------
   |   |         0 | 3 | 0 
-----------     -----------
 O |   |         0 | 2 | 2 

once we have gone over each line (row, column, diagonal), we select the maximum weighted cell as the next move. in the case of a number of cells with the maximum weight, one can be selected at random.

we have now defined the computer's priorities as
1) finish the game by winning
2) prevent the opponent from winning
3) control the cell with the most completion options for either player

that was fun :)