10.4 Secure Communication with SSH

Though you can easily communicate with your target using Telnet, it is a very insecure protocol and its vulnerabilities are widely documented. The user password, for instance, is transmitted in clear text from the client to the server. It would therefore be rather unprudent, and in most cases downright dangerous, to include a Telnet daemon in your product in the hopes of being able to remotely fix problems once the product is at the client's site. Instead, it would be much preferable to use a protocol that relies on strong encryption and other mechanisms to ensure the communication's confidentiality. The best way to do this currently is to use the SSH protocol and related tool suite. SSH uses public-key cryptography to guarantee end-to-end communication encryption while being fairly easy to use and deploy.

Because SSH is an IETF standard, there are a few competing implementations, some of which are proprietary commercial products. The main open source implementation is OpenSSH. Although there are other open source implementations, they are either very difficult to cross-compile or have dependencies that make them impractical for use in embedded Linux systems. I will therefore devote most of this section to discussing OpenSSH. We will briefly review the other open source implementations, because they may eventually become usable in embedded Linux systems.

An embedded system that can be accessed through SSH runs the same SSH server software usually run on a traditional server. Our discussion will therefore concentrate on the compilation of the SSH server for the target, and its setup, configuration, and use on the target. I will not cover aspects of how to set up and use any of the other SSH components.

If you are seriously considering using an SSH package in your target, I suggest you take a look at SSH, The Secure Shell: The Definitive Guide by Daniel Barrett and Richard Silverman (O'Reilly). It provides the in-depth coverage I cannot undertake here.

10.4.1 OpenSSH

OpenSSH is developed and maintained as part of the OpenBSD project. It is available from http://www.openssh.org/ under a composite BSD license (see the LICENSE file for the complete details) along with ample documentation and quite a few other resources. To use OpenSSH in your target, you also need two libraries: OpenSSL and zlib. OpenSSL is an open source implementation of the Secure Socket Layer (SSL) protocol and is available from http://www.openssl.org/ under a BSD-like license. The zlib compression library is the same as the one we discussed earlier in Section 7.1. Before building and installing OpenSSH, you must first build and install both these libraries in your host's cross-platform development framework. In addition, OpenSSH, OpenSSL, and zlib need to exist natively on your host. If you intend to link OpenSSH with uClibc, uClibc must be installed natively on your host. Before you go any further, make sure that the required components are properly installed on your workstation, since they are needed for compiling OpenSSH for the target.

For my example SYSM module, I used OpenSSH 3.5p1, OpenSSL 0.9.6g, and zlib 1.1.4. Because vulnerabilities are found from time to time in some of these packages, it is important to keep track of the versions you are using and to have upgrade plans, in case serious vulnerabilities are found in one of the versions you are using.

The OpenSSH package is hostile to cross-compilation. Among many other things, its configuration script attempts to run test samples it builds using the compiler specified with CC=. Because the applications built using the cross-compiler can't run on the host, the script stops and emits an error. As you'll see, we need to resort to a number of tricks to force the package's configuration scripts and Makefiles to build the software. The headers and libraries installed natively on the host are used extensively, for example, in fooling the package into compiling for the target. Though the instructions that follow try to be as complete as possible, you may need to put some effort into figuring out a few modifications for your own setup.

In the following, I discuss both OpenSSL and OpenSSH. Because the names of both packages differ by only one letter, it is easy to confuse the names while reading the text. Pay close attention to the last letter of each package name to avoid any confusion.

We will discuss how to build and install OpenSSL shortly. First, however, refer to Section 7.1 for instructions on how to build zlib. In contrast to these earlier instructions, you need to build zlib as a static library instead of a shared library. To do so, don't set the value of LDSHARED and don't use the - -shared option, as we did earlier, on the configure command line.

In the rest of this section, I will assume that you are building OpenSSH against glibc. If you would rather use uClibc, replace all references to ${TARGET_PREFIX} with references to ${PREFIX}/uclibc and all references to arm-linux-gcc with references to arm-uclibc-gcc. If you had not enabled shadow password support and C99 support when installing uClibc, you will need to reinstall uClibc with support for these features enabled. Also, you will need to install zlib in uClibc's directory. To do so, set the value of prefix to ${PREFIX}/uclibc instead of ${TARGET_PREFIX} when issuing the make install command for zlib.

With zlib properly built and installed, download and extract OpenSSL in your ${PRJROOT}/build-tools directory. Move to the package's directory to configure, compile, and install it:

$ ./config --prefix=${TARGET_PREFIX} compiler:arm-linux-gcc
$ make
$ make install

Here, the compiler is specified using the compiler: option instead of setting the value of CC. Once completed, all the components installed are found under the ${TARGET_PREFIX} directory. The compilation takes around 10 minutes on my hardware setup.

Once OpenSSL is installed, download and extract OpenSSH into your ${PRJROOT}/sysapps directory. Now move to OpenSSH's directory to proceed:

$ cd ${PRJROOT}/sysapps/openssh-3.5p1

To trick OpenSSH's configure script into successfully creating useful Makefiles, we need to pretend that we are actually configuring it for the host. We then use the Makefiles created by configure to build OpenSSH for the target. For this scheme to succeed, we must use a few fake file links. Mainly, we need to:

Create a symbolic link to the host's C compiler bearing the same name as the C compiler we generated for the target.
Create a symbolic link to the host's native OpenSSL headers.
Create a symbolic link to the host's native OpenSSL libraries.

On my development host, for example, the native OpenSSL headers and libraries are located in the /usr/local/ssl directory, and the C compiler is located in /usr/bin. Here are the preliminary steps I use for preparing the build of OpenSSH for my SYSM module:

$ export PATH=./:$PATH
$ which gcc
/usr/bin/gcc
$ ln -s /usr/bin/gcc ./arm-linux-gcc
$ ln -s /usr/local/ssl/include ./fake-include
$ ln -s /usr/local/ssl/lib ./fake-lib

I modified the PATH to force configure to start looking in the current directory first for all binaries. This enables me to trick it into using a compiler called arm-linux-gcc when this is really the host's own compiler. I can now run the configure script itself using the fake links:

$ CC=arm-linux-gcc CFLAGS=-I./fake-include LDFLAGS=-L./fake-lib \
> ./configure --host=$TARGET

The line above generates Makefiles that build dynamically linked binaries. To link the binaries statically, change the value of LDFLAGS to "-L./fake-lib -static".

The script's output resembles the output of other configure scripts we've seen before. At the end, however, it prints a summary of the configuration it has found:

OpenSSH has been configured with the following options:
                     User binaries: /usr/local/bin
                   System binaries: /usr/local/sbin
               Configuration files: /usr/local/etc
                   Askpass program: /usr/local/libexec/ssh-askpass
                      Manual pages: /usr/local/man/manX
                          PID file: /var/run
  Privilege separation chroot path: /var/empty
            sshd default user PATH: /usr/bin:/bin:/usr/sbin:/sbin:...
                    Manpage format: doc
                       PAM support: no
                KerberosIV support: no
                 KerberosV support: no
                 Smartcard support: no
                       AFS support: no
                     S/KEY support: no
              TCP Wrappers support: no
              MD5 password support: no
       IP address in $DISPLAY hack: no
          Use IPv4 by default hack: no
           Translate v4 in v6 hack: yes
                  BSD Auth support: no
              Random number source: OpenSSL internal ONLY
 
              Host: arm-unknown-linux-gnu
          Compiler: arm-linux-gcc
    Compiler flags: -I./fake-include -Wall -Wpointer-arith -Wno-un...
Preprocessor flags:
      Linker flags: -L./fake-lib
         Libraries:   -lutil -lz -lnsl  -lcrypto -lcrypt

This output indicates that the SSH software expects to operate through the root /usr and /var directories, which is fine since it will be running as root on the target. The most important parts, however, are the Compiler and the various flags fields at the bottom. In the output shown, the compiler name is the right one, and the include and library paths point to the fake entries I created earlier. Hence, I have succeeded in fooling configure into creating Makefiles that use filenames that I can control. To finish the trick, I can now remove the fake links I created earlier and create appropriate ones for my target:

$ rm arm-linux-gcc fake-include fake-lib
$ ln -s ${TARGET_PREFIX}/include ./fake-include
$ ln -s ${TARGET_PREFIX}/lib ./fake-lib

By removing the arm-linux-gcc file link, I am forcing the Makefiles to use the arm-linux-gcc command found in the PATH, which is the actual cross-compiler I built earlier for my target. Similarly, the library and header file path links I just created will force the Makefiles to use my target's actual libraries and header files. All is set now for building OpenSSH.

Before you issue the make command, you may need to hand-tweak some header files if the C library version you are using for your target is not the same as the one used on your host or if the sizes of the various C types on the target differ from those on the host. In my case, for example, I had to edit defines.h and config.h. In defines.h, I had to add an #if 0 and #endif around the definitions of _ _ss_family. In config.h, I had to do the same with the #define of HAVE_GETGROUPLIST. You will probably need to modify those files in your own way to get OpenSSH to compile. If you get errors at link time about missing symbols, this probably means you have to edit config.h and comment out the appropriate HAVE_ definition. Note that any modification to config.h will be lost if you run the configure script again.

With the headers having been properly modified, everything is ready for building OpenSSH:

$ make

The complete compilation takes less than five minutes on my hardware. The resulting SSH daemon—which is the binary we are most interested in for our target, as I said earlier—is fairly large. When compiled against glibc and stripped, the binary is around 1 MB in size if linked dynamically and 1.4 MB if linked statically. When compiled with uClibc and stripped, the binary is around 1 MB when linked dynamically and 1.1 MB when linked statically.

Copy the SSH daemon to your target's root filesystem and strip it:

$ cp ./sshd ${PRJROOT}/rootfs/usr/sbin
$ arm-linux-strip ${PRJROOT}/rootfs/usr/sbin/sshd

To run the daemon, you need a configuration file and a set of private and public keys. An example configuration file is already provided as part of the OpenSSH package, sshd_config. Copy this file to your target's root filesystem and customize it according to your needs:

$ mkdir -p ${PRJROOT}/rootfs/usr/local/etc
$ cp sshd_config ${PRJROOT}/rootfs/usr/local/etc

Also, you need to generate the keys for your target. There are three types of keys to generate, RSA1, RSA, and DSA, and each has a private and a public component. All keys will be located in the same directory as the daemon's configuration file, in the target's /usr/local/etc/ directory. Using your host's native OpenSSH tools, create the keys for your target:

$ ssh-keygen -t rsa1 -f ${PRJROOT}/rootfs/usr/local/etc/ssh_host_key
$ ssh-keygen -t rsa -f ${PRJROOT}/rootfs/usr/local/etc/ssh_host_rsa_key
$ ssh-keygen -t dsa -f ${PRJROOT}/rootfs/usr/local/etc/ssh_host_dsa_key

In addition, create /var entries on your target's root filesystem for OpenSSH:

$ mkdir -p ${PRJROOT}/rootfs/var/run ${PRJROOT}/rootfs/var/empty
$ su -m
Password:
# chown root:root ${PRJROOT}/rootfs/var/run ${PRJROOT}/rootfs/var/empty
# chmod 755 ${PRJROOT}/rootfs/var/empty
# exit

Finally, you need to have a "privilege separation" user on your target. This user isolates the connection to the outside world from the brain of the SSH daemon. Thus, if the connection is compromised, the remote party does not obtain root access to the system. To add the privilege separation user, first edit your target's /etc/group file and add the following line:

sshd:x:255:

Replace the 255 value with a group ID that is still available on your target. If you are using CRAMFS, remember that this number must be below 256. Now, edit your target's /etc/passwd file and add the privilege separation user:

sshd:x:501:255:sshd privsep:/var/empty:/bin/false

Also, edit your target's /etc/shadow file and add an entry for the privilege separation user:

sshd:*:11880:0:99999:7:-1:-1:0

Furthermore, you need to copy all the libraries sshd depends on to your target's root filesystem, if you had used dynamic linking. Use arm-uclibc-ldd to find the complete list of dependencies.

As with the earlier networking packages, edit your target's /etc/inittab file to start the sshd process:

::respawn:/usr/sbin/sshd -D

The -D flag is used to tell sshd not to fork from the shell and become a daemon. Hence, init can respawn it if it dies. This, however, is not a common occurrence, and any failure of sshd should be considered a serious bug and be properly investigated.

For further information on how to configure and operate OpenSSH, see the SSH, The Secure Shell: The Definitive Guide book mentioned earlier.

10.4.2 A Word on Other SSH Implementations

Apart from OpenSSH, there are a few other open source SSH implementations; most notably LSH and FreSSH. At the time of this writing, however, neither is fit for use in production embedded Linux systems.

LSH, for example, depends on packages that have their own dependencies. In particular, it depends on the GNU MP library, zlib, and liboop. The first two dependencies are tolerable. The problem is that liboop depends on glib, which in turn requires pkg-config. Moreover, the glib package doesn't lend itself well to cross-compiling. If you are using a host of the same architecture as the target, you may to consider compiling LSH statically and then using it on your target. If, as in most cases, your target isn't of the same architecture as your host, LSH isn't a practical choice at this time.

FreSSH, on the other hand, is a relatively new package that isn't as mature as the other open source SSH packages. Among other things, it lacks a configure script and requires extensive Makefile modifications to build. In addition, it can be built only against glibc. The compilation against uClibc requires a number of source code modifications. When compiled against glibc, the resulting SSH daemon's size is around 850 KB, which is very close to the size of the SSH daemon generated by OpenSSH.