9.6 Encrypting HKLM\SAM with SYSKEY

Like Unix, Windows 2000 and NT don't directly store user or machine passwords. Instead, they take the passwords and passes them through a scheme called a one-way function , or OWF. The OWF takes a password in and generates a new block of data that is related to, but doesn't contain, the password. The "OW" in OWF comes from the fact that it's not feasible to take the output of the OWF and "go backwards" to derive the original password. The output of the OWF is called a password hash. NT stores the password hashes instead of the password, so you can't steal the hash and use it directly in place of a password. Windows 2000 also stores hashed passwords for local user and computer accounts, as well as for backward compatibility with older Win9x and NT clients.

In the spring of 1997, an enterprising group of hackers from L0pht Heavy Industries (http://www.l0pht.com) publicized the fact that it was possible to get the password hashes from a SAM database (or by sniffing them over the network) and feed them to a password-cracking tool. These types of attacks have been known for many years in the Unix community, but their appearance in the Windows NT world generated a lot of headlines. In practical terms, the actual risk was significant. Even though only administrators have access to the SAM to get the OWF'ed passwords in the first place, the hashes could be recovered from backup tapes or ERDs, and they could be sniffed off the network.

Accordingly, Microsoft took a beating on the Internet and in the press for the perceived insecurity of the SAM password data. To provide a solution, Microsoft introduced a method of protecting the SAM data with strong encryption; the SYSKEY utility installs and controls this extra protective layer. SYSKEY is available with NT 4.0 SP3 and later, and it's installed and enabled by default in Windows 2000 (in fact, you can't turn it off ).

9.6.1 What SYSKEY Does

SYSKEY adds an extra layer of security to the password data stored in the SAM database by encrypting the hashed password data using a 128-bit system key. This key (Microsoft calls it the password encryption key , or PEK; so will I) is randomly generated when you install SYSKEY. Once your PEK is generated, NT uses it to encrypt and decrypt all password data (but not the ordinary account data) in the SAM. Because the data's encrypted, it's useless to any thief or cracker who might get it (and getting it still requires you to gain physical and administrative access to a domain controller). As a bonus, because the data is stored in encrypted form, it remains protected when it's backed up to an ERD or a tape.

Once the password data's encrypted, it's stored back into the SAM database, and services (including the local security authority, or LSA) that access the password data must depend on the kernel to decrypt it for them. For this to work, though, the kernel has to know what the PEK is at boot time: the SAM password information includes password data for system services that start when the machine's booted, in addition to the more mundane user password data.

In December 1999, a security team at BindView (http://www.bindview.com) found a vulnerability in SYSKEY's password encryption, making it much easier to attack. Microsoft immediately released a hotfix for NT 4 systems; the fix is included in Windows 2000 RC3 and later.

To accomplish this, SYSKEY stores the PEK. You might wonder how storing the PEK could possibly increase security; it seems foolish to store the master password used to encrypt the data that's supposed to be protected! The answer is simple: another key is used to encrypt the PEK. This second key is the system key, after which SYSKEY is named. SYSKEY supports three options for storing the system key and making it available to the system when it's needed to decrypt the PEK.

The first, and most secure, option allows you to store the system key on a floppy. When the machine's booted, the floppy must be present so the kernel can retrieve the system key and use it to decrypt the PEK. Without the right floppy, the machine cannot be booted into the version of Windows 2000 or NT that's protected by that floppy.^[3] This introduces a new single point of failure for your machines, so it's critical to keep backup copies of the floppy. In addition, the floppy serves as a token that allows access to the SAM data, so you must control who has access to it.

^[3] Each installation of the OS gets its own unique PEK. If you have multiple versions, or multiple copies, of Windows 2000/NT installed on a single computer, each has a unique PEK. If you have two installations on one machine and lose the system key disk for one of them, you can still boot the other one.

The next option is to store the system key encrypted with another key. This second key is generated from a passphrase you choose. Instead of inserting the system key floppy at boot time, a human must be present to type in the passphrase. The encrypted version of the system key is stored on the computer so that only the passphrase is required; there's no separate floppy or key disk involved.

Finally, you can choose to have the system key stored on the local machine. SYSKEY uses what Microsoft calls a "complex obfuscation algorithm" to hide the key. This is supposed to make it hard to compromise the system key. This reliance on "security through obscurity" offers considerably less security than the other available methods, but it has one saving grace: it allows unattended reboots, since the kernel can derive the PEK when needed without any human intervention. This is critical for some applications; only you can determine whether it's the best choice for your servers.

9.6.2 Before You Enable SYSKEY on Windows NT

As with most other NT components, it's tempting to rush out and install SYSKEY now that you know how it can add security to your machines' Registry data. However, in this case it pays to be cautious and make sure you've adequately planned deploying SYSKEY on your network. It's important to understand what SYSKEY protects you against and what additional problems it can potentially impose. Committing to using SYSKEY is not to be done lightly.

Every NT 4.0 workstation and server can run SYSKEY, and each machine can use any of the three system key storage options mentioned earlier. If you choose to use key floppies or passphrases, remember that the floppy or passphrase is just like an ERD: it's useful only on the machine where it was created, so you have one disk or passphrase for every protected machine. (You can cheat and use the same passphrase on all machines, though.)

First of all, let's start with the scariest problem: SYSKEY can make your system more secure, but it's a one-way trip. Once you enable strong encryption of the SAM account database, there's no way to turn off encryption and go back to the old unencrypted version (though you can use an ERD, as described later). In practice, as long as you keep your ERDs up to date, you won't need to go back to the unencrypted version.

The next problem is what security experts call the "steel lock, balsa-wood door" problem. If you have multiple domain controllers for one domain, and one uses SYSKEY but the others don't, you haven't added any security to your network. One machine's SAM database is protected, but--since all the other controllers have replicated copies--the data you want to secure is still easy to grab. Ideally, you should implement SYSKEY on every machine that has an accounts database. That means all domain controllers and any NT workstation machine that has local accounts.

The final difficulty posed by SYSKEY is the fact that it adds security by encrypting the data on your machine. As long as you retain access to the system key, NT can decrypt the PEK and use it to access the stored passwords. If you choose to use a passphrase or key floppy, and you lose or forget it, you'll have to restore from an ERD. If the only ERD you have was made after the Registry was encrypted, you still won't be able to get in! It's critical to safeguard the key floppy and make backup copies of it so a bad floppy doesn't take you out of production--but since the key floppy is a security component, you have to keep careful watch over it.

9.6.2.1 "What I tell you three times is true"

Microsoft recommends making a total of three ERDs when installing SYSKEY: one before installing the SYSKEY hotfix or service pack, one after installing it but before enabling SYSKEY, and one after the first reboot after installing SYSKEY. While this may seem excessive, making all three disks maximizes the likelihood that you can recover your machine if it crashes in the future:

• The post-SYSKEY ERD contains the encrypted version of your accounts database. As you add and remove accounts, keep this ERD up to date. As long as you have the system key (either stored on the computer or on a floppy), you can restore the account database, and the rest of the Registry, from the ERD.

• The pre-SYSKEY ERD holds a record of your unencrypted Registry. If you ever need to recover the machine but don't have the system key, you have two choices: reinstall NT and lose all of your account data, or recover the Registry from this ERD and lose any changes made after SYSKEY was activated.

• The preinstall ERD protects you from problems with the hotfix or service pack. In general, you should always update your ERD just before installing any service pack or hotfix; this gives you an escape hatch if you need to back out of a fix that actually made things worse on your machine.

Keep all three ERDs for each machine you might someday need to restore. At a minimum, that means you'll need them for every domain controller on your network, plus one set for any NT Workstation machine that has important local accounts.

9.6.2.2 Upgrading domain controllers

Microsoft also warns you about installing SYSKEY on your primary domain controller: if something goes wrong with the SYSKEY installation, or if you ever lose the system key for that one machine, no one will be able to log on to your domain! For domains with more than a few users, you should already have a backup domain controller anyway; if you don't, it's worth considering adding one.

The safest way to roll out SYSKEY for your Windows NT domain controllers in a multiple-domain or multiple-controller network is this:

If you have domains with only one controller, it's probably okay to dispense with these steps and just activate SYSKEY on the controller, as long as you have the recommended set of three ERDs.

9.6.3 Turning On SYSKEY Protection

You control SAM database encryption with the SYSKEY.EXE executable. As you might expect, only administrators may turn on system key protection. The first time you run SYSKEY, you see a dialog that warns you encryption can't be disabled once you turn it on; you see that warning again after you first enable encryption. There are only five controls in the window: the Encryption Enabled and Encryption Disabled radio buttons show the current state of system key encryption on this machine. You can change from disabled to enabled, but not vice versa. The standard OK and Cancel buttons work like they do in every other dialog. The Update button allows you to change the key storage method later (you'll see how to do that in Section 9.6.4).

The first step in activating SYSKEY is simple: click the Encryption Enabled radio button, then click the OK button. You then see a warning dialog reminding you that this conversion can't be undone and suggesting that you ensure that you've got a current ERD before proceeding. When you click OK in that dialog, you see the Account Database Key dialog (see Figure 9.5), which you use to tell SYSKEY where you want the system key stored after it's generated:

• If you want to use a passphrase to unlock the system key, click the Password Startup button and type your password into the Password and Confirm fields. You may enter up to 128 characters for a passphrase, and longer phrases are better. Unfortunately, SYSKEY doesn't enforce any minimum length restrictions on the password. Microsoft recommends at least 12 characters, but it's easy to come up with a longer password than that: pick two easy-to-remember adjectives and a noun, then string them together with punctuation or special characters (for instance, "galloping_sleepy#motorhome"). NT feeds the passphrase you enter to a special algorithm, which generates a 128-bit key from it.

Figure 9.5. Specifying the location for the system key

• If you want the system to generate a password on its own, click the System Generated Password radio button. In this mode, NT uses its own pseudorandom number generator^[4] to pick a random 128-bit system key. As you know, that key has to be stored somewhere. You get to choose where:

  ^[4] The great mathematician and computer scientist John von Neumann once said that if you rely on software to generate random numbers you're living in a state of sin. However, cryptographically strong pseudorandom generators (like the one NT uses) are only a little sinful.

• The Store Startup Key on Floppy Disk button instructs NT to keep the encrypted system key on a floppy. The key itself is stored in a file named StartKey.key. When you choose this option, you need a floppy handy to hold the key. Although it may be temptingly close, don't use your ERD to store the key: doing so concentrates both pieces of data needed to steal passwords on a single floppy.

• The Store Startup Key Locally button stores the obfuscated system key in HKLM\SYSTEM. When you choose this option, you can reboot the machine without having a human present.

Once you've selected the method you want, click the OK button. If you've chosen to store the key on a floppy, SYSKEY prompts you to insert a floppy and confirms that it's written the key to the disk. Otherwise, the key is silently updated and SYSKEY then exits.

The next time you boot the machine, SYSKEY protection will be in effect. This means that unless you're storing the system key locally, you have to be at the machine every time it's rebooted to either type in the passphrase or stick in the key floppy.

If you're in a hurry, you can use the -l flag with SYSKEY; this instructs it to silently generate a system key and store it on the local machine. This is a handy trick to use when setting up a new workstation or server; you can add the command to your ordinary setup scripts, then change the key storage method later when you have more time. This gives you immediate protection without any extra effort on your part.

9.6.4 Changing the Key Storage Method

Once you've installed and activated SYSKEY, you're not bound to your initial choice of key storage. You can run SYSKEY again at any time and change from one method to another. When you change methods, SYSKEY generates a new system key and stores it instead of reusing the old key; this helps protect your password data against compromise.

To change the key storage method for a machine, run SYSKEY and click the Update button. The Account Database Key dialog (shown earlier in Figure 9.5) then appears, and the radio button corresponding to the currently active storage method is active. To change to a new method, just click one of the other radio buttons, filling in the password if necessary.

Because SYSKEY generates a new key when you change storage methods, you must supply the old key as part of the change process. This means that what happens after you click "OK" depends on what storage method you were previously using. If you changed from "Store Startup Key Locally" to something else, SYSKEY can get the old key from HKLM\SYSTEM, so you don't have to do anything. If you're changing from storing the password on a floppy or being protected by a passphrase, SYSKEY requires you to provide the key disk or passphrase to continue. This prevents an attacker from changing the key, storing on a floppy, and stealing the floppy--thus rendering your machine unbootable.

Figure 9.6 shows the dialog that asks for the key disk, while Figure 9.7 shows the dialog requesting the current passphrase. If you supply the correct passphrase or floppy, SYSKEY displays a confirmation dialog to remind you that it's changed the system key, and the new key is stored using the method you've chosen. If you don't supply the right information, SYSKEY won't change anything.

Figure 9.6. The key disk dialog

Figure 9.7. The password dialog

9.6.5 Restoring a SYSKEY-Protected NT Registry

Chapter 3, described the mechanics of restoring a damaged Registry using a Windows NT ERD. To restore a machine protected with SYSKEY, you follow this same basic procedure, but there are a few new subtleties introduced as a result of SYSKEY 's presence. The golden rule for restoring a SYSKEY-protected machine is simple: use the correct ERD.

9.6.5.1 Restore SYSTEM and SAM hives

Even though the actual encrypted account information is stored in the HKLM\SAM subtree, the actual PEK, as well as all the other data SYSKEY needs to tell where the system key is stored, lives in HKLM\SYSTEM. To recover an encrypted account database, you must restore both the SYSTEM and SAM hives at the same time--not just SAM. If you don't do this, NT can't decrypt the Registry, either because it can't find the system key (if you don't restore SYSTEM) or because the key doesn't decrypt the data (if you don't restore SAM). Of course, you must restore these hives from the same ERD.

9.6.5.2 Get the right system components

If you got SYSKEY as the result of installing an NT 4.0 service pack, you may not have noticed that three existing system files were replaced as part of the update: winlogon.exe, samsrv.dll, and samlib.dll. These three files, along with syskey.exe, implement the account database protection. Their presence is required to enable the encrypted SAM data to be read and decrypted by system services that need it.

When you first install NT, it logs the versions of all the components you install in system.log. When you install service packs, hotfixes, or software such as Internet Explorer that replaces one or more system files, the installer application is supposed to update the entries in system.log so that it always reflects the current version of all DLLs, drivers, and other operating-system components.

What this means is that if you install SYSKEY as part of a hotfix or service pack, the system.log entries for winlogon.exe, samsrv.dll, and samlib.dll reflects the versions installed with SYSKEY, not the original versions you installed. If you want to restore your machine to its pre-SYSKEY state, you need to use the NT setup application's "Repair system files" option to restore the original versions from your NT CD or file server. However, you must be sure to restore the SAM and SYSTEM hives from the pre-SYSKEY ERD: if you revert to the original system components but leave the encrypted Registry in place, nothing will work right.

If you install SYSKEY but don't turn it on, the winlogon.exe, samsrv.dll, and samlib.dll files won't match your original installation. When you install the new versions of these files, they change the Registry format even when encryption is off. If you use NT setup to restore these three files to their original state by using your pre-SYSKEY ERD, you must also restore SAM and SYSTEM from the same ERD: if you don't, the old components won't be able to read the new Registry format.

9.6.5.3 Which ERD should I use?

Three ERDs is three more than most NT systems have, and so deciding which one to use may seem a little overwhelming. It's not hard, though: each ERD can put your system back into a particular state. Which one you use depends on what you want the restored system to have on it. Table 9.3 shows your options.

Table 9.3. ERD Restoration Table

To Revert to...	Use This ERD	Don't Forget...
System as it was before installing SYSKEY	Preinstall ERD	You may lose account database changes made since SYSKEY was installed. You must also choose "repair system files" in NT setup to restore the original versions of winlogon.exe, samsrv.dll, and samlib.dll. You can always fall back to this level, even without the system key.
System as it was after installing, but before activating, SYSKEY	Pre-SYSKEY ERD	You may lose account database changes made since SYSKEY was installed. You can always fall back to this level, even without the system key. When using this ERD, don't "repair system files" from CD.
System as it was after activating SYSKEY	Post-SYSKEY ERD	This preserves all account database changes since the ERD was updated. It requires presence of system key/passphrase on floppy or machine. When using this ERD, don't "repair system files" from CD.