Near the end of 2018, I was working on a project in which we had to generate ssh keys dynamically in code to be used in connecting to a number of remote servers. At some point, this stopped working and it was hard to figure out why. The public and private keys were being created, stored, and used in the right places, however, we constantly saw Permission denied (publickey) showing up in our logs. It was perplexing what had worked fine for months seemingly randomly stopped working. But in the world of software engineering, nothing is ever random.

Like any good software engineering team, we tried to break down the issue to pinpoint the problem spot.

• Can we reproduce it locally? √
• Does it happen on each team member's machine? √
• Can we manually generate an ssh key, add the public key to the remote server, and connect to it with the private key via normal ssh methods? √
• Is our manual usage the same as in the running app? Hmmm....

As we took a closer look and started breaking down our assumptions, we noticed a very important detail. The keys that had worked in the past all had began with:

-----BEGIN RSA PRIVATE KEY-----

...but our new keys all began with:

-----BEGIN OPENSSH PRIVATE KEY----- (!!!)

SSH Key Format

We came to find out that the default format for ssh-keygen (OpenSSH) had recently changed to something proprietary and was no longer compatible with a library we were using to connect to remote servers. The "key" for us to get an ssh key that worked with our solution, was to use the, now legacy, PEM (Privacy Enhanced Mail) format. Here is the command to do it:

ssh-keygen -m PEM

This change is described in the OpenSSH release notes for v7.8 on 8-24-2018:

ssh-keygen(1): write OpenSSH format private keys by default instead of using OpenSSL's PEM format. The OpenSSH format, supported in OpenSSH releases since 2014 and described in the PROTOCOL.key file in the source distribution, offers substantially better protection against offline password guessing and supports key comments in private keys. If necessary, it is possible to write old PEM-style keys by adding "-m PEM" to ssh-keygen's arguments when generating or updating a key.

Food for thought: this is where integration and end-to-end testing provides a lot of value. Upgrading versions of operating systems, dependencies, etc. might not show immediate side effects with day-to-day work or unit tests. Testing the whole system, in detail, would have yielded this problem earlier and helped us pinpoint "what changed" without as much digging.

Throughout history, there have been countless advances in all aspects of life. Generation after generation people have passed along what they have learned and accomplished to their family, friends, and neighbors. Without those exchanges of wisdom, this very article would likely not be written. We can find at the core of nearly every aspect of life a rabbit hole of how it came to be - Who developed it? Who thought of it? Where does it come from? How was it inspired? What was it built on? And what was THAT built on? etc. In one form or another, society has "paid it forward" over and over again; continually sharing the what, where, how, and why of ideas, manufactures, and methods.

There is no shortage of paying it forward it the tech world. There are numerous websites, forums, and blogs from which to pull material, read documentation, and to ask questions. Some of my favorites, and the most commonly used by my peers, include stackoverflow.com, css-tricks.com, baeldung.com, github.com, scotch.io, dzone.com, and countless Medium blogs. And that doesn't scratch the surface. To top it off, there are decades of goodwill in the form of open-source software that have enabled an explosion of capability. All of which I have benefited from in my software career.

It is because of the benefit that I have started this blog; as a way to "technically" pay it forward (pun intended). Over the past couple of years I have felt the need to share what I have learned. After enduring, on many occasions, hours of frustration trying to get something working, it typically ends by spinning around in my chair with both fists pumped high in success like a double Judd Nelson from The Breakfast Club.

I reached that nirvana due to others paying it forward. And now it's time for me to do so. I plan to write posts that include useful code and configuration nuggets and full solutions to a broad spectrum of software related topics including, but not limited to:

• Languages: JavaScript, TypeScript, Java, Groovy, C#, C++ and undoubtably, over time, others
• Front-end web development: HTML, CSS, App Frameworks and Libraries
• API development: RESTful Web Services, HATEOAS, Microservice Design, Spring, NodeJS
• Software Engineering: Data Modeling, SOLID Principles, Abstraction, Architecture Patterns
• Testing: Unit, Integration, and Functional tools, runners and libraries
• CI/CD: Jenkins and others
• Cloud Hosting, Containerization and Orchestration: Docker, Docker Compose, Kubernetes, Google Cloud Platform, AWS

You might be interested in this blog if you're:

• a middle or high school kid trying to figure out how to just run that little line of code that you saw in a tutorial where all the details were skipped and assumed you already knew the prerequisites.
• that college student who wants to learn how functions and classes of your unrealistic coding assignments apply to the "real world".
• wondering how your pet project prototype can be built into a reliable, maintainable, and affordably hosted app.
• looking for solutions to common problems due to lacking documentation and examples.
• a software engineering expert who wants to critique my posts, challenge solutions, and tell me how wrong I am (constructively, I hope).

I hope you find value in these posts and will consider, in turn, paying it forward.