Download & Install a SSL Cert into a Java keystore with keytool

Today I was notified our notification email mail server was changing hosts. So I made a list of the services that use the notify email address (e.g. notifications@domain.tld) – this email address is responsible for sending info to our network users which include updates for everything from issue tracking to password recovery (and then some). With security in mind all emails should be sent over SSL (the mail server supports SSLv3), but the problem is that the installed cert is self-signed; now I know it is a good cert – I generated it, we just don’t wanna fork over the $$ to have a root cert provider put their stamp of approval on it and it’s used for internal purposes only.

Now, if you generated a self-signed SSL cert and want to import that, just skip the “Download the SSL Certificate” section.

There’s Always Something to Mess Your Day Up

Normally this isn’t a big deal: just update the SMTPS credentials and be on your way. However, most of our service applications are Java-based. You’re thinking, “no biggy, just turn on the flag to trust all certs.” Not that easy; the options within our apps don’t have this fancy little checkbox. So I guess I have to do it the hard way: download the cert from the mail server and add it to the Java keystore, restart the service (bah, I have down time), and cross my fingers that it works.

Download the SSL Certificate

This one is pretty easy, and really straight forward (with the help of Didier Stevens’ quickpost)

Obviously the above dump isn’t exactly what you’ll get, but you get the idea… Also, notice the Ctrl+C up there, this is important, the openssl command hangs and you don’t need all the extra stuff, so just wait for the initial dump and cancel the script.

Next, copy the base64 encoded certificate to a .pem file. Don’t forget to include “—–BEGIN CERTIFICATE—–” and “—–END CERTIFICATE—–“. Just save it to something like host.domain.tld.pem. Really, just copy and paste from the terminal. For the idiots: in PuTTY just select it all with the left mouse button, and click on it with the left mouse button. This will copy it to your clipboard. Issue the command “nano host.domain.tld.pem” and right click in nano to paste. Ctrl^O to write out (write the file) and Ctrl^X to exit nano. Done.

Lastly, to figure out the host the certificate belongs to, run the following (this will also confirm if you’ve copied the PEM base64 over correctly):

This will show the lines which include “CN” (e.g. “…/CN=host.domain.tld/emailAddress…”). The CN parameter is the host/domain name that the certificate is registered under. This will be required information when importing with keytool.

Install the SSL Certificate with Java’s keytool into the keystore

There are a few things to accomplish in this section: find and locate the JRE you want to use, find the keytool script, find the trusted certificates file (cacerts), and execute a single-line command.

If there are several JRE’s on the system, figure out which one the application uses. For example, the standalone app I have installed has its own jre folder which contains ./bin/keytool, however, I also have a a system-wide Java installation. To expose all the keytools on your system use find / -name “keytool” …don’t use whereis, only registered applications appear with this command.

My setup looks something like this:

Since I need the service app to have the certificate trusted, I’ll use its own embedded jre keytool, “/opt/thirdpartyapp/jre/bin/keytool”.

Also, the cert needs to be installed into the trusted certs file, so within the particular java/jre installation there should also be a ./lib/security directory with a cacerts file.

TL;DR Working with keytool

Now with the previously retrieved information, here’s the bread’n’butter:

  • The import switch tells keytool we want to import the pem into the trusted certs file (cacerts).
  • The alias switch’s value should be the CN you found in the previous section (after running openssl x509…), this is the domain for which the cert was created for
  • The keystore switch is the path to the Java keystore file, usually cacerts, which stores trusted certificates
  • The file switch’s value is the path to the pem that we created in the previous section.

keytool will now attempt to import the cert. First it’ll prompt for a password, unless you know otherwise try “changeit” or “changeme” – these are widely used defaults. Once you provide the correct password it’ll dump out a bunch of information about the certificate it is importing and lastly ask you if you want to “Trust this certificate” – type “yes” and hit return: “Certificate was added to keystore” is presented.

Now restart the Java application (however it is you do that) and it’ll recognize the SMTPS connection (or what ever else you’re working towards, e.g. HTTPS, SFTP, POP3S etc.)

Piece of cake, after you do it a few times. (Just realized I think i change tenses and POV a couple times in this write up… oh well.)

Further Reading

…like at the bottom of each chapter in your school text books, don’t worry kids, there isn’t a chapter review or quiz.

Password-less SSH with Public/Private Keys

From the POV of an advanced *nix user setting up public and private keys for password-less SSH logins seems trivial. However, to a beginner/novice user this can be confusing. I’ll admit, as comfortable as I am with managing Linux servers, configuring RSA keys on the local and remote machines was messing with my head – until I got the hang of it. I still don’t have a complete top-to-bottom understanding of this when it comes to different versions of SSH and then some, but enough to jot down a note for future reference just in case. I’ll assume the reader has a basic knowledge of CLI, connecting to a remote server with SSH, and that the two (or more) devices in question are both Linux machines.

By the way, there are a lot of these guides out there, but i couldn’t find one that helped. I still had to screw around a bit to really the the hang of it. They were either all too in-depth, too brief, only covered certain parts, or simply just didn’t fit my needs.

Before we get started, there is an easier and less intrusive way to do this if you’re running cPanel. I’ll write about that later.

Use Case

Close your eyes, find your power animal – slide… imagine you’re logged into some nix box (we’ll call this one “foo.com” with user “jack”). Now you need to issue some commands on another remote machine (and we’ll call this one “bar.net” with user “jill”), so what do you do?

At this point, it will make the connection, and if is the first time ask if you want permanently trust the connection, in which you reply “yes” after careful consideration. SSH prompts you for a password, you type it in and you’re good to go. You start navigating the filesystem and ferociously execute commands.

Now this scenario is all fine and dandy, until you’re accessing the server so often that you get sick and tired of typing your 15 character password (with upper/lowercase letters, numbers and symbols). Perhaps a better excuse to not enter the password is that you have a non-interactive shell script that needs to make this connection temporarily, in which case you do not have the luxury of entering your password in the prompt; don’t even tell me you were thinking of embedding the password into the command with the p switch. Wish there was a better way? Let me show you the light.

Benefits of Using Keys with SSH

  • Doesn’t prompt for a password (duh)
  • Can be used with non-interactive/unmonitored scripts (play off of the previous bullet)
  • FAR more secure – this is probably the biggest reason you should be going with this approach regardless of the previous benefits
  • Establish a “trusted connection”
  • Help prevent brute force attacks
  • Read this article on “old” password-style authentication

TL;DR Lezzdo’t.

Remember jack and jill on foo.com and bar.net? Well they didn’t fetch a pail of water, instead they generated an RSA key with ssh-keygen. We’re getting back to that now… There are two types of encryption: DSA and RSA. DSA is supposedly faster but not as compatible (isn’t compatible w/ Protocol 1). RSA has higher encryption (4096) and is more compatible, but at the cost of speed. Up to you, but i’ll be sticking with RSA.

Configure SSH and SSHD on foo.com and bar.net

  •  Make sure that ssh is configured correctly on foo.com and bar.net: Use Protocol version 2, not 1 (you can use 1, but it is a PITA especially when you’re configuring multiple keys)
  • On bar.net in your sshd configuration (e.g., /etc/ssh/sshd_config) you’ll need the following configurations:
  • All other defaults (OOTB) should be good, maybe when you’re done you can set “PasswordAuthentication no” within bar.net’s sshd_config to force the usage of only key exchanges.

Generate a Key Pair on foo.com for Jack to Use

  1.  Create a .ssh directory in jacks’ home directory if it doesn’t exist yet, /home/jack or ~/ if you’re logged in as jack. actually from here on out, just assume you’re logged in as jack, if not, make sure you can sudo -u jack to make it seem like jack is issuing the commands. mkdir /home/jack/.ssh
  2. chown jack:jack /home/jack/.ssh
  3. chmod 700 /home/jack/.ssh
  4. step 3 is important, make sure the perms took: stat .ssh
  5. ssh-keygen -t rsa -C “the key for jill on bar dot net”
  6. the t switch tells ssh-keygen what crypt to use (t = type) and the C switch is just a comment to leave in the public key string (c = comment)
  7. Hit return
  8. Now it starts to generate a public/private key pair and asks you where to save giving you a suggestion (default): /home/jack/.ssh/id_rsa
  9. id_rsa is the default and you can stick with that, but if you want to store multiple, you’ll have to outsmart the keygen. remember, the computer is just a tool, its faster than you, but not nearly as smart. you need to be smarter than the computer if you’re not, this is a requirement unless you’re competing against an IQ of 150. before you hit enter, read step 10
  10. save it to the file: /home/jack/.ssh/id_rsa.jillbarnet
  11. take a moment and think, who is Jill Barnet?
  12. when prompted for a passphrase just keep it blank and hit return (twice)… disclaimer: this is NOT really a good idea. you should always have a passphrase for security purposes, but i’m lazy right now and don’t wanna go through the extra steps, if you’re really worried RTFM.
  13. notice /home/jack/.ssh/id_rsa.jillbarnet and /home/jack/.ssh/id_rsa.jillbarnet.pub
  14. DONE.

Configure Jack’s Account to Use Multiple Key Pairs

This is important if you followed the last steps exactly. We generated a specific key file. By default, ssh looks for a file called id_rsa. If we wouldn’t have specified a file, ssh-keygen would’ve written to that default id_rsa file and loaded it by default every time. This sucks when it comes to managing multiple keys, so forget the default functionality; we’re gunna rock this bitch.

  1. Create a file called config: touch /home/jack/.ssh/config
  2. chown jack:jack /home/jack/.ssh/config
  3. chmod 600 /home/jack/.ssh/config
  4. edit the file (i like nano, sorry @ most of you vi snobs): nano /home/jack/.ssh/config
  5. For each id_rsa.* entry we’ll need a Host and IdentityFile directive, all directives separated by at least a new line…

The Host directive means that only the next line (IndentifyFile) is applicable when connecting to that specific host. The IdentifyFile directive describes what file to use in that situation. So lemmie pick this apart really quick. I indent because it looks pretty.

  • Lines 1 and 2 mean that ssh should use the file id_rsa.jillbarnet when connecting to host bar.net.
  • Lines 3 and 4tell ssh to only use file id_rsa.rickybobbyorg when connecting to bobby.org (where did he come from? read the next section and find out)
  • Lines 5 and 6 is basically a catch all. The asterisk is a wildcard, it means, for any host that hasn’t matched yet, use the default identity id_rsa.

Hint: If those files don’t exist ssh won’t crap out at all, it’ll die gracefully during the lookup and continue normal operations (and ask for a password). So no worries there.

In theory you could combine multiple identity files under the same host directive, but i haven’t tried it and don’t take my word for it. You can also wildcard partial host names, e.g. Host *.domain.tld – this i did try.

Understanding the Private Key and Public Key

So here’s the confusing part, really try to pay attention. Remember step 13 (scrunched together letter B) in previous? Why did it generate two files when you only specified one?

Well this is where private key and public key comes into play. The private key is “id_rsa.jillbarnet” and the public key is “id_rsa.jillbarnet.pub”. The private key STAYS with Jack on foo.com, it should never be shared, that’s why it is “private” you I-D TEN TEE. The public key is the one you can whore out to other accounts; i’ll squash the myth right now: the public key is not specifically for jill over at bar.net, it can be used for ricky over at bobby.org, or at both places (jill@bar.net and ricky@bobby.org). The public key simply acts as a “trusted” or “authorized” user who provides the private key. Read that last sentence two or three more times.

The private key looks something like this:

And the public key looks something like this (a command with a base64 encoded string + your custom comment):

Cool? Cool. Now let’s prep jill’s home dir.

Prep Jill’s Home Dir Over at bar.net for Jack’s Key

Let’s assume jill’s home directory is /home/jill. Again assuming you’re logged in as user jill and jill belongs to group jill (in previous i assumed jack belonged to group jack).

  1. Make sure jill has a .ssh dir in her home directory, if not make one: mkdir /home/jill/.ssh
  2. Verify jill is the sole owner: chown jill:jill /home/jill/.ssh
  3. Make sure jill is the only one allowed to access that .ssh directory: chmod 700 /home/jill/.ssh
  4. make a file called authorized_keys: touch authorized_keys
    don’t know what touch does? RTFM: man touch. (just wanted to type “man touch”)
  5. set restrictive perms (that’s permissions, not a hair-do) for authorized_keys: chmod 700 authorized_keys
  6. At this point, in some envs/distros they require an authorized_keys2 file, just make sure it is identical to authorized_keys in terms of content and permissions. Or in my case with CentOS5, you don’t need it. so skip this step. I experienced this issue on a Debian box, fyi.
  7. Copy the contents of id_rsa.jillbarnet.pub (the one with the content that starts “ssh-rsa” then has the base64 encoded string w/ the comment) and append it to authorized_keys. There are a million-n-a-half ways to do this. Figure it out.
  8. For multiple entries in authorized_keys, just delimit them by new lines. so each line will start with “ssh_rsa”
  9. I feel like i’m forgetting something, but that should do it.

Test From Jack’s Account on foo.com

Granted you did everything correctly, it will log you in w/o a password prompt. If it asks for a password, you screwed up somewhere, try again, do not pass go, do not collect 200 bones. Note that restarting SSHD will NOT solve your problem, no service restart is required.

That’s All She Wrote

Well that’s it. So now you can connect to multiple hosts using multiple private keys for jack at foo.com (using multiple id_rsa files and the config file updates), and multiple people can connect to jill over at bar.net (by copying the public key to authorized_keys and delimiting by new lines). Any questions? Too bad, I closed my comments.

MaxMind GeoIP + PHP PECL

Overview

Getting around to writing a geo IP look-up script and needed to leverage maxmind’s geoip database. First things first, get the geoip lib installed and ready to go or else the dat file isn’t going to do me any good.

Initial Setup: First Try

At first I went with the PEAR route just because it wouldn’t require make installing’ing any dynamic PHP extensions (.so). Went to do a pear update-channels for the first time and got a boat load of eregi errors – off to a great start. Turns out CentOS 5.7 php53 (php 5.3.3) package w/ php-pear is not compatible – bastards! RedHat has an update available but is only available to their customers which doesn’t include me (since i go the free route of  centos). Did some extra googling and found this: php-pear-1.4.9-8.el5.noarch.rpm.html (version 8.el5 versus centos 5.7’s native version 6.el5). Did a little wget magic followed by a yum install <path to package> and ta-da, no more eregi errors. Did a pear install Net_GeoIP and it worked too.

After all that hard work and fidgeting around, I read this little line on MaxMind’s website:

PHP Extension on PECL
Download a PHP extension that allows you to embed the GeoIP C Library inside PHP for improved performance. There is a new fork with a more complete implementation.

Crap! I love good performance and I’d take a compiled lib over an interpreted one any day. PEAR headache: all for naught. Proceeded with a pear uninstall Net_GeoIP, then cried for a half an hour since i just wasted 15 minutes (for a total of 45 man minutes wasted, yes).

Next stop, PECL.

Initial Setup: Second Try (start tl;dr)

  1. Started with a pecl update-channels, and all went well. That’s a first.
  2. pecl search geoip – got results, sweet.
  3. pecl install geoip… got through maybe 40 lines of the ./configure and it bombed, can’t find a valid compiler
  4. no worries, do a little yum groupinstall “Development Tools” and now I’ve got a C compiler (as well as automake and a few other required tools)
  5. pecl install geoip… configure bombs again, this time I’m missing the geoip linux packages, and at this point i’m thinking, “how many friggin dependencies are there?” holy overhead batman.
  6. yum install geoip-devel, wait, no,yum install GeoIP-devel since yum is case sensitive – this gets me everything i need.
  7. 3rd times a charm: pecl install geoip – install completes, tells me to add “extension=geoip.so” so I do like the robot I am, doing what a linux prompt tells me to do.
  8. service httpd restart – cycling apache allows for the geoip section to appear in the phpinfo() call, not sure why this is (more mysteries!)

OK, I got the libs installed and the commands are asking for me to call them.

Installing the MaxMind database

  1. mkdir /usr/share/geoip
  2. wget and extract geoip db to /usr/share/geoip (http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz)
  3. chmod -R 755 /usr/share/geoip
  4. nano /etc/php.ini
  5. add a [geoip] config section:

  1. service httpd restart
  2. check the phpinfo() for custom_directory
Let the scripting begin. I’ll stop writing now; from this point forward it’s all about RTFM (link below…)

Resources