gettimeofday() and time() should only be used to get the current time if the current wall-clock time is actually what you want. They should never be used to measure time or schedule an event X time into the future.

What's the problem?

gettimeofday() returns the current wall clock time and timezone. time() returns a subset of this info (only whole seconds, and not timezone).

Using these functions in order to measure the passage of time (how long an operation took) therefore seems like a no-brainer. After all, in real life you measure by checking your watch before and after the operation. The differences are:

1. Nobody sneaks in and changes your wristwatch when you're not looking

You usually aren't running NTP on your wristwatch, so it probably won't jump a second or two (or 15 minutes) in a random direction because it happened to sync up against a proper clock at that point.

Good NTP implementations try to not make the time jump like this. They instead make the clock go faster or slower so that it will drift to the correct time. But while it's drifting you either have a clock that's going too fast or too slow. It's not measuring the passage of time properly.

2. You're not stupid, the computer is

Doing something doesn't take less than 0 time. If you get <0 when you measure time, you'll realize something is wrong. A program will happily print that a ping reply came back before you sent it. Even if you check for time<0, the program that uses gettimeofday() still can't tell the difference between a 2 second delay and 3 seconds plus a time adjustment.

3. You are the limiting factor

In real life you are not expected to measure sub-second times. You can't measure the difference between 1.08 seconds and 1.03 seconds. This problem is mostly (but far from entirely) in the small scale.

What to use instead

The most portable way to measure time correctly seems to be clock_gettime(CLOCK_MONOTONIC, ...). It's not stable across reboots, but we don't care about that. We just want a timer that goes up by one second for each second that passes in the physical world.

So if you want to wait 10 seconds, then check the monotonic clock, add 10 seconds and wait until that time has come.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

#include <time.h>
#include <stdio.h>

/**                                                                             
 * sleep for `sec' seconds, without relying on the wall clock of time(2)        
 * or gettimeofday(2).                                                          
 *                                                                              
 * under ideal conditions is accurate to one microsecond. To get nanosecond     
 * accuracy, replace sleep()/usleep() with something with higher resolution     
 * like nanosleep() or ppoll().                                          
 */
void
true_sleep(int sec)
{
        struct timespec ts_start;
        struct timespec ts_end;

	clock_gettime(CLOCK_MONOTONIC, &ts_start);

        ts_end = ts_start;
        ts_end.tv_sec += sec;

        for(;;) {
                struct timespec ts_current;
                struct timespec ts_remaining;

                clock_gettime(CLOCK_MONOTONIC, &ts_current);

		ts_remaining.tv_sec = ts_end.tv_sec - ts_current.tv_sec;
                ts_remaining.tv_nsec = ts_end.tv_nsec - ts_current.tv_nsec;
                while (ts_remaining.tv_nsec > 1000000000) {
                        ts_remaining.tv_sec++;
                        ts_remaining.tv_nsec -= 1000000000;
                }
		while (ts_remaining.tv_nsec < 0) {
			ts_remaining.tv_sec--;
                        ts_remaining.tv_nsec += 1000000000;
                }

                if (ts_remaining.tv_sec < 0) {
                        break;
                }

                if (ts_remaining.tv_sec > 0) {
                        sleep(ts_remaining.tv_sec);
                } else {
                        usleep(ts_remaining.tv_nsec / 1000);
                }
        }
}

int
main()
{
        true_sleep(10);
}

The same method works if you want to schedule something in the future, or see how long something took. I've found that I very rarely actually need the wall clock time in programs. The only exceptions I can think of are when it should be store on disk (valid across reboot or with NFS across server) or when the time (not the time delta) should be shown to the user.

The case of "sleep" can actually be solved by clock_nanosleep(), but I wanted an example that could illustrate how to measure too.

If clock_gettime(CLOCK_MONOTONIC, ...) is not available on your system, then try to find a monotonic clock that is. Like gethrtime() or CLOCK_HIGHRES on Solaris. I have created a portable library for getting monotonic time.

The guilty ones

libpcap

The pcap_pkthdr struct (the "received packet" struct) contains a struct timeval ts that ruins our ability to measure the time it takes for the reply you get for some query you sent. They tell me the kernel supplies the timestamp, so it's not really libpcaps fault.

Calling clock_gettime() when libpcap gives you a packet has turned out to be useless, as the time difference between packet reception and the delivery to your program is too long and unstable. You're stuck with this wall-clock time until you fix all the kernels in the world and break binary compatibility with old libpcap programs.

ping

Try running a ping and setting the time in the past. Ping will freeze waiting until it thinks it's time for the next packet. Tried with iputils ping for Linux. A brief survey of the source code of other OpenSource pings show this:

• FreeBSD: Same as Linux
• NetBSD: Much code in common with FreeBSD. This too looks the same.
• Mac OSX: Same heritage, same fault
• OpenBSD: Seems OK because they (surprise surprise) ignore the C standard and do the work in a SIGALRM handler
• Solaris: Same as OpenBSD, except they don't support fractional second intervals

It seems that the only people who got the ping send scheduler right in this regard did it at the expense of not following the C standard, and nobody got the RTT calculation right. Solaris actually used a monotonic clock (gethrtime()) in other parts of the code, but not for RTT calculation.

I have sent a patch to fix this on Linux ping. It has not yet been applied to upstream.

Oracle database

If time was set backwards then Oracle would reboot the machine. Really. (bug ID seems to be 5015469)

When a leap second is inserted into the official time, such as 2008-12-31 23:59:60, all the other clocks are suddenly fast, and therefore adjust themselves backwards. On newyears day 2009 many people woke up to their Oracle servers having rebooted. Not simply shut down the oracle process, but actually rebooted the server.

F5

F5 BigIP load balancers seem to cut all active connections when time is set backwards. So if you use two NTP servers that don't have the same time, all your connections will be cut when your load balancer flips back and forth between the two upstream times.

Me

At least arping and gtping didn't handle this properly and could in some cases (not always) act like Linux ping. I have fixed both of them and it'll be in the next version. I had put off fixing them because I wanted a solution that solved packet timings as well as packet scheduling, but at least with arping that doesn't seem possible due to the libpcap issue mentioned above.

GTPing should now have this solved perfectly, and ARPing should only show incorrect times for replies when the wall clock changes or drifts, but the scheduler should still send one packet per interval.

Everyone else?

I'd be surprised to see any program handling this correctly. Please let me know if you find one.

Exception

If a wall-clock time is all you can get (such as from libpcap), then you're gonna have to use that. But you don't have to like it. And avoid wall time where at all possible.

Unrelated note

When I turn on the GPS I have in my car after it's been off for too long it asks me where I am and what time and date it is. Why would a GPS possibly want to ask me those things? Telling me where I am and what time it is is what it was made to do.

Update: As a couple of people have pointed out there is a "right" answer to this. It's not a problem for newer GPS units, but there it is.

Links

• Doug Coleman on Monotonic timers
• Portable library for getting monotonic time
• My feelings about GPSs
• Monotonic time in Python

What they are NOT

They're not SSH PubkeyAuthentication

In other words if your .pub file doesn't end in -cert.pub and you haven't used ssh-keygen -s, then you aren't using certificates.

Update (2012-09-12)

$ ssh-keygen -L -f id_rsa-cert.pub | grep Type
        Type: ssh-rsa-cert-v01@openssh.com user certificate

$ ssh-keygen -t v00 -s user-ca -I thomas@home -n thomas -V +52w id_rsa.pub

They're not SSL

They're not PEM, x509 ASN.1 or any other insane format

They're not easy to google for

What they do

Sign host keys

The authenticity of host 'xxx' can't be established.
RSA key fingerprint is xxx.
Are you sure you want to continue connecting (yes/no)? 

Sign user keys

Either or both of the above

Allow you to expire certs

Setting up host certificates

1. On the machine that you'll be storing your CA on: Create host CA key (host_ca & host_ca.pub):

   ssh-keygen -f host_ca

2. Sign existing host public key:

   ssh-keygen -s host_ca -I host_foo -h -n foo.bar.com -V +52w /etc/ssh/ssh_host_rsa_key.pub

   Copy /etc/ssh/ssh_host_rsa_key.pub from other servers and this command on those files too, and copy the resulting back to the server.
3. Configure server(s) to present certificate (/etc/ssh/sshd_config):

   HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub

   and restart sshd.
4. Put host CA in clients known_hosts, such as ~/.ssh/known_hosts or a system-wide one. The line should look something like this:

   @cert-authority *.bar.com ssh-rsa AAAAB3[...]== Comment

5. Remove clients entry (including aliases such as its IP address) for the host itself in ~/.ssh/known_hosts (if any)
6. Logging in to the correct name should now work and you should not be asked about the host key:

   ssh foo.bar.com

7. Logging in to the IP address of the machine should present the message: "Certificate invalid: name is not a listed principal"

Do NOT forget the -n switch when creating host certificates. Otherwise anyone who cracks one machine will be able to impersonate any other machine in the domain to users who trust this CA.

You may need to add something like this to your client config (~/.ssh/config):

Host * 
  HostKeyAlgorithms ssh-dss-cert-v01@openssh.com,ssh-dss 

Setting up user certificates

1. On the machine you'll be storing your CA on: Create user CA key (user_ca & user_ca.pub):

   ssh-keygen -f user_ca

2. For every server that the certificates should work on:
   • Copy user_ca.pub from the CA machine to /etc/ssh/ on the server.
   • Add user_ca.pub to servers /etc/ssh/sshd_config:

     TrustedUserCAKeys /etc/ssh/user_ca.pub

   And restart sshd.
3. For each user: Sign existing user pubkey key:

   ssh-keygen -s user_ca -I user_thomas -n thomas,thomas2 -V +52w /path/to/id_rsa.pub

   Give the resulting *-cert.pub file back to the user.
4. Log in as user thomas or thomas2 on the server, even though the server has never seen your key or cert before, only user_ca.pub.

Some notes

• Never forget -n when creating host certs.
• If you put your user CA in ~/.ssh/authorized_keys instead of configuring the server you don't need -n for user certs. If you're doing system-wide (as described) it's mandatory.
• The certificate validity time is in UTC
• You can revoke keys and certs with RevokeKeys
• -I just specifies the key ID. It doesn't have to follow the pattern I used.
• Print certificate information with ssh-keygen -L:

  $ ssh-keygen -L -f user-thomas-cert.pub
  user-thomas-cert.pub:
          Type: ssh-rsa-cert-v00@openssh.com user certificate
          Public key: RSA-CERT-V00 a4:b3:6d:e2:bd:4a:39:01:31:c9:05:43:db:78:f6:c9
          Signing CA: RSA a5:e5:20:8e:ea:ea:15:7e:c3:31:60:2d:6b:93:a0:6b
          Key ID: "user_thomas"
          Valid: from 2011-07-07T15:37:00 to 2012-07-05T15:38:11
          Principals: 
                  thomas
                  thomas2
          Critical Options: 
                  permit-agent-forwarding
                  permit-port-forwarding
                  permit-pty
                  permit-user-rc
                  permit-X11-forwarding
  

Links

• Email thread: getting host certificates working
• Serverfault: How to revoke an SSH certificate

Yubico and their products are awesome.

That pretty much sums up this blog post but I'm going to go on anyway. If you're thinking of introducing two-factor authentication to your company, or you're using something that's fundamentally broken (like RSA SecureID) you simply must at least take Yubikeys into consideration.

When I say that SecureID (and others) are fundamentally broken what I mean is that when (not if, as recent history has shown) RSA (the company) is broken into YOUR security is now compromised. When I first used SecureID and found out that you as a customer aren't in control of your own keys my first thought was "well that's just stupid". Why are you giving the keys to the kingdom to someone else?

Enter Yubikeys. They just beat SecureID in every way (almost). Benefits:

• Open specification.
• You can set your own keys (secrets) and don't have to show them to a third party who can lose them.
• They're cheap. 25 USD for one (yes you can buy just one if you want) and cheaper in bulk, of course. No server license.
• No server license. (this bears repeating). Yes there are products you can buy to get features and such, but with minimal coding you can actually get this up and running on your website / server logins for free!
• Looong OTP tokens. If you don't have some account locking feature on your servers you can brute-force 6 digits fairly quickly. OpenSSH by default allows 10 unauthenticated sessions with one attempt per second. You can script that to log in in less than a day.

• No clock in the device. While plugged in the token it generates will be a function of how long it's been powered up this session, but it doesn't have a battery-powered clock that you can sync with the server.
• No display on the device. Well, you wouldn't want to key in a really long token anyway, but if you can't insert USB into the terminal or want to use it to log in somewhere with your phone then you're screwed. I bet Yubico are working on some NFC to fix that though. (the RFID parts of some Yubikeys are something separate that I'm not including here)

So what can you do with a Yubikey?

Simple OTP generation

• The secret key (obviously)
• The time since this session started
• Number of tokens generated this session
• Number of sessions since device was programmed
• Key ID (user settable), so you know which secret key to use do decode the token.
• Some other less interesting fields. This list is intentionally not complete.

This waiting is not possible when using challenge-response mode (see below).

Symantec VIP

If you buy one for personal use I'd recommend buying the VIP key. It's the same price as the normal one but it comes pre-programmed in slot 1 with something you can use to log into your paypal account (once you associate the key with your account).

You can still use slot 2 for something else.

Static password

HOTP

Challenge-response mode

So what you need on your client is a program that can take the challenge from the server, send it to the Yubikey and then send the reply back to the server. If you are an online game developer this would just be something you put into your game. As long as the users have their Yubikey plugged into their machine they can log in and play.

I've created some scripts to use this for unix authentication that I'll be releasing in a week or two. You SSH to some server, the server says "Yubikey challenge: XXX[...]: ", you copy that challenge (just select it) and press a key combination that starts this script. The script takes the challenge from the clipboard, gets the reply from the Yubikey and "types" it. Challenge-responses aren't "typed" by the Yubikey so I'm doing that in the script. And presto, you get challenge-response Yubikey authentication with SSH and every other application without having to code it into your client.

YubiHSM

By the way, I don't care if you hash your passwords with unique salt for everyone. If you lose that database then you've lost the users' passwords.

YubiHSM is a USB-attached device that doesn't emulate a keyboard. It has several functions (including the ability to do the logic of a Yubikey authentication server) but to me the most interesting is protecting the password database.

In short, YubiHSM can give you an interface where you can use just two operations:

• Set password. This will not store anything on the YubiHSM but will give you back a blob that only the YubiHSM can decrypt.
• Check password. You supply a password and a blob and the YubiHSM says "yes" or "no". It does not give you the password.

Using yhsmpam I've used this to secure a unix machine. Think of it as moving from storing hashes in /etc/passwd to storing them in /etc/shadow, but this time moving from /etc/shadow to YubiHSM.

Unfortunately YubiHSM isn't for sale yet. Oooh! I just checked the website and it's going on sale on 18 August 2011. At 500 USD it's more expensive than the Yubikey, but then again you only need it for your authentication server, not for every user. 500 USD is NOT a lot for this kind of hardware. Before YubiHSM you'd have to cough up 15k+ USD to get this. PRE-ORDER NOW! Run, do not walk, to their online store.

Complaints

• No clock in the Yubikey
• Hard to use Yubikey with phone
• While all "normal" keyboard layouts work fine (including German), Dvorak will not work well with Yubikeys by default. Well, with my ChromeOS laptop it did work by default, but still.

Links

• Yubico
• pam_externalpass - PAM module I use to do more fancy auth.
• YHSMPAM - Backend for keeping UNIX passwords in YubiHSM
• YOracle - My own yubikey validator server.
• [Placeholder for when I release the challenge-response code]

The short version of the problem and solution I will describe is that while TCP gets up to speed fairly fast, and "fast enough" for many uses, it doesn't accelerate fast enough for short-lived connections such as web page requests. If I have 10Mbps connection and the server has 10Mbps to spare, why doesn't a 17kB web page transfer at 10Mbps from first to last byte? (that is, when excluding TCP handshake, HTTP request and server side page rendering)

This is pretty Linux-focused, but I'll add pointers for other OSs if I see them.

Short version

This will get a bit basic for some people, so here's the short version. Make sure you measure the effect of changing these settings. Don't just increase them and think "more is better".

ip route change default via x.x.x.x initrwnd 20

ip route change default via x.x.x.x initcwnd 20

ip route change default via x.x.x.x initcwnd 20 initrwnd 20

The path to enlightenment

In short, the number of bytes in flight in TCP is the lowest of the senders congestion window (cwnd) and the receiver window size. The window size is the receiver announcing how much it's ready to receive, and the sender will therefore not send any more than that. The cwnd is the sender trying to avoid causing congestion. (since TCP is two-way both sides have both, but think of it as one-way for now)

The window size is announced in every segment (TCP packets are called "segments") and is the receivers method of throttling. The cwnd is not sent across the network, but is a local sender-side throttling only.

Both of these default to "a few" multiples of the maximum size segments (MSS). The MSS is announced in the SYN and SYN|ACK packets and are adjusted so that a TCP segment fits in the path MTU (hopefully). Typical initial window size and cwnd is on the order of 4kB. Linux has an initial cwnd of 10*MSS nowadays, but that doesn't help if the receiver window is smaller that that. Both window size and cwnd will change as the connection progresses. How and why is a huge topic in itself. I'm focusing here on a specific problem of initial size and TCP slow start.

The problem I was having was that when I downloaded http://blog.habets.pp.se over a ~50ms RTT connection these values don't go up fast enough. During the ~300ms transaction (including everything) the sender stalls about 4-6 times waiting for the receiver to give the go-ahead. (I only have approximate numbers since the RTT between the servers I was testing was a bit unstable. Virtual servers and all that). The time between first and last byte was about 170ms. Really? ~50ms RTT and 17kB in 170ms? That's not good.

Ideally there should be two round trips involved. One where the client has sent SYN and is waiting for SYN|ACK, and the other when the client has sent the request and is waiting for the results. If this was a long-running connection (with HTTP keep-alive for example) this wouldn't be a problem, but since I'm looking at the first request in a new TCP connection it is.

Increase receiver window size

Since I couldn't find it in sysctl or /sys I looked at the source. net/ipv4/tcp_output.c has a function called tcp_select_initial_window(). The last parameter is __u32 init_rcv_win. Well, that was easy. Because I'm lazy I just compiled the kernel hardcoding it to 10 (it's in multiples of MSS). I started a lab virtual machine and sure enough the initial window is now a lot bigger. Still not seeing a reduction in round trip waits though, and it's as slow as before. At least now it's not the receivers fault. The window has lots of space left but the sender is quiet.

What is this dst_metric(dst, RTAX_INITRWND) that it was before I changed it to hardcoded 10 though? include/linux/rtnetlink.h which defines RTAX_INITRWND looks like mostly routing related stuff. Aha! Sure enough:

user@hostname$ ip route help
[...]
OPTIONS := FLAGS [ mtu NUMBER ] [ advmss NUMBER ]
       [ rtt TIME ] [ rttvar TIME ] [reordering NUMBER ]
       [ window NUMBER] [ cwnd NUMBER ] [ initcwnd NUMBER ]
       [ ssthresh NUMBER ] [ realms REALM ] [ src ADDRESS ]
       [ rto_min TIME ] [ hoplimit NUMBER ] [ initrwnd NUMBER ]
[...]  

Misc

• As far as I can see window size and cwnd can not be set per-socket by setsockopt() or similar. If they could this would cause people to write evil applications that don't play well with the rest of the Internet.
• Current cwnd, window size and other data can be seen with getsockopt(..., TCP_INFO, ...).
• An Argument for Increasing TCP’s Initial Congestion Window
• Google and Microsoft Cheat on Slow-Start. Should You?
• Linux kernel patch for adding setsockopt() for changing cwnd. Not needed. It was rejected for a reason.

Links

• Draft to increase TCP's initial window by default to 10 * MSS. This has been implemented in Linux.
• Increasing the TCP Initial Congestion Window on Windows 2008 Server R2

When connecting to a possibly hostile network I want to tunnel all traffic from my browser to some proxy I have set up on the Internet.

The obvious way to do this is with a proxy. The problem with that is that the traffic from the browser to the proxy is not encrypted. Even when you browse to secure SSL sites some traffic is being sent in the clear, such as the host name. That's not so bad, but I want to hide my HTTP traffic too.

Turns out that at least Chrome does support using SSL between the browser and the proxy, but you can't configure it directly. You have to use Proxy Auto Configuration to point to an HTTPS host:port.

Running OpenVPN is out in this case since I want it to work with everything, including Android and ChromeOS… and Windows (without installing "stuff"). Not that I've tried it with Windows yet, but it should work.

For added fun I wanted to authenticate to the proxy using a client certificate, so that I don't have to worry about remembering or losing passwords.

Before you set off to do the same thing I should warn you that either I haven't done it quite right or there is a bug in Chrome. If the network has a hickup so that the proxy or the .pac file is unavailable it will fall back to connecting directly. I want it to never fall back. That's kind of the point.

But all is not in vain. I also set up IPSec with these same certificates. There will be a follow-up post describing how I set this up to tunnel securely with both Chromebook and my phone.

This is what I did (caveat: some of this is from memory. If you find any errors please let me know):

1. Copy easyrsa 2.0 from openvpn sources

   This is a set of scripts to manage a CA. We'll be using this directory/set of scripts both to create a server and client certificates. But only the client certificates will be signed by our own CA.

2. Edit vars file

   I like to change KEY_SIZE to 2048, country and other stuff. These will be the defaults when creating certs.

3. Init easyrsa

   ./clean-all
   This is a one-time init. Don't run this script again.

4. Build the CA for the client certs

   ./build-ca

5. Copy CA cert to proxy server

   scp keys/ca.crt proxy:/etc/proxy-ssl/proxy-ca.crt

6. Build server key and signing request

   ./build-req proxy.foo.com

7. Get signed server cert

   Send keys/proxy.foo.com.csr to your real CA (e.g. cacert.org or Verisign). You'll get back proxy.foo.com.crt. If you do use cacert.org then make sure that your browser trusts their root by importing it.

8. Put proxy.foo.com cert and key on proxy server

   scp keys/proxy.foo.com.{crt,key} proxy:/etc/proxy-ssl/

9. Install squid proxy and stunnel on proxy server

   On Debian/Ubuntu: apt-get install squid3 stunnel

10. Configure & start stunnel

    1. set ENABLE=1 in /etc/defaults/stunnel.conf
    2. In /etc/stunnel/stunnel.conf

             cert=/etc/proxy-ssl/proxy.foo.com.crt
             key=/etc/proxy-ssl/proxy.foo.com.key
             verify=2
             CAfile=/etc/proxy-ssl/proxy-ca.crt
             [proxy]
             accept  = 12346
             connect = 3128

    3. /etc/init.d/stunnel4 start

11. Create .pac file somewhere on an https url

    function FindProxyForURL(url, host)
    {
          return "HTTPS proxy.foo.com:12346";
    }
    

    Make sure that the certificate of the https server is one that the browser will accept.

12. Set proxy autoconf to the url to this .pac file

    Spanner->Preferences->Under the Bonnet->Change proxy settings->Automatic Proxy Configuration
    Set the URL to where the file is, including "https://".

13. Try it now. It should fail with SSL errors

    It should fail because proxy doesn't accept your cert (you don't have a cert in the browser yet). Don't continue if you get some other error.

14. Create client cert signed by your own CA

    ./build-key my-proxy-key
    Don't set a password.

15. Convert the client key+cert to .p12, because that's what Android and Chrome wants

    openssl pkcs12 -export -in keys/my-proxy-key.crt -inkey keys/my-proxy-key.key -out my-proxy-key.p12
    It'll ask for a password that will only be used in the next step. No need to save it for later.

16. Import the client certificate into the browser

    On your Chrome (or chromebook) go Spanner->Preferences->Under the Bonnet->Certificate Manager->Your Certificates->Import (or "Import and Bind to Device").

17. Try to browse somewhere. Now it should work

    Make sure you're actually using the proxy. Go to www.whatismyip.com or something and make sure that it sees the IP of the proxy. It'll probably tell you that you're using a proxy. If it doesn't work then good luck. :-)

Problems

Like I said it seems that it falls back to connecting directly, even though the .pac file doesn't have a fallback mechanism configured. Stay tuned for the IPSec version.

This is a short howto on setting up TPM-backed SSL. This means that the secret key belonging to an SSL cert is protected by the TPM and cannot be copied off of the machine or otherwise inspected.

Meaning even if you get hacked the attackers cannot impersonate you, if you manage to kick them off or just shut down the server. The secret key is safe. It has never been outside the TPM and never will be.

This can be used for both client and server certs.

Prerequisites

• A TPM chip. Duh. May need to be turned on in the BIOS. Could be called "security chip" or something. If you don't have a TPM chip but still want to follow along (maybe add TPM support to some program) then you can install a TPM emulator. See links at the end on how to install a TPM emulator.
• A working CA that will sign your CSR. I will assume you're running your own CA, but you can send the CSR to someone else to sign if you want cert creating step.
• Installed "stuff":

  $ aptitude install tpm-tools libengine-tpm-openssl

Initialize TPM

• Make sure you can communicate with the TPM chip:

  $ tpm_version
    TPM 1.2 Version Info:
    Chip Version:        1.2.4.1
    Spec Level:          2
    Errata Revision:     2
    TPM Vendor ID:       INTC
    Vendor Specific data: 00040001 0003040e
    TPM Version:         01010000
    Manufacturer Info:   494e5443
  

• Clear the chip:

  $ tpm_clear --force
  $ tpm_setenable --enable --force
  $ tpm_setactive --active

  These steps may require reboots. They will probably tell you if that's the case.
• Take ownership and create SRK:

  $ tpm_takeownership -u -y
  Enter SRK password: <just press enter>
  Confirm password: <just press enter>

  Yes really, empty password. This password protects TPM operations, so worst case is that someone with shell access can put some load on your TPM chip. They can't use it to crack keys or anything. This is not a password that protects your secrets, they're already secure. You may want to supply an owner password. If so, leave out the -y option.

Create SSL certificate

• Use script from libengine-tpm-openssl to create key: create_tpm_key foo.key
• Create CSR for key: openssl req -keyform engine -engine tpm -key foo.key -new -out foo.csr
• Sign the CSR with your CA: openssl x509 -req -days 365 -in foo.csr -CA ca.crt -CAkey ca.key -CAserial serial -out foo.crt

Test it

$ openssl s_server -accept 12345 -cert /path/to/server.crt -key /path/to/server.key -CAfile CA/ca.crt -Verify 1

$ openssl s_client -keyform engine -engine tpm -connect server-name-here:12345 -cert foo.crt -key foo.key

Use in your code

Instead of using SSL_CTX_load_verify_locations(), use ENGINE_*() and SSL_CTX_use_PrivateKey()

Without TPM:

1
2
3
4

void loadpriv(SSL_CTX* ctx, const char* file)
{
  SSL_CTX_use_PrivateKey_file(ctx, file, SSL_FILETYPE_PEM);
}

With TPM:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

void loadpriv(SSL_CTX* ctx, const char* file)
{
  ENGINE_load_builtin_engines();
  ENGINE* engine = ENGINE_by_id("tpm");
  ENGINE_init(engine);

  UI_METHOD* ui_method = UI_OpenSSL();
  EVP_PKEY* pkey = ENGINE_load_private_key(engine, file, ui_method, NULL);
  SSL_CTX_use_PrivateKey(ctx, pkey);
}

Use in the real world

Huh. I haven't actually seen this implemented in any Open Source software (besides the openssl command line interface) Well, actually curl and stunnel4 appear that they could support it, but it's not clear how. Here is someone else wondering how to get stunnel to do it. It's from 2006 with no replies.

The only thing I have that uses the TPM chip is my Chromebook. Client certs there are protected by it by default.

Please leave comments if you know of any TPM support in Open Source stuff.

tlssh

1
2
3

PrivkeyEngine tpm
TpmSrkPassword
PrivkeyPassword secretpasswordhere

If PrivkeyEngine tpm or -E tpm is supplied the secret key will be assumed to be TPM protected and should be in the form of a "BEGIN TSS KEY BLOB" section instead of a PEM format "BEGIN RSA PRIVATE KEY" section.

Performance

Oh, it's slow. Really slow. Luckily it's just used in the handshake. I haven't done proper benchmarks, but let's just say you won't be using it to power your webmail. I may do some benchmarking in a coming blog post.

Update: It can do about 1.4 connections per second. See Benchmarking TPM backed SSL for details.

Misc notes

• The .key file for the TPM-backed cert contains the secret key encrypted with something that's in the TPM. It's not usable without the exact same TPM chip.
• You can choose to create a key in the normal way and then "convert" it to a TPM-backed key. It's not as secure since you can't say the key was never stored outside the TPM chip, but use the -w switch to create_tpm_key if you want to do this.
• OpenSSL is probably the most horrible, disgusting, undocumented crap library I've ever seen. Some of this can be blamed on SSL, X509 and ASN.1, but far from all.
• Emulated TPM is not completely useless. Since the emulator runs as root and the user has no insight into it you can have users be able to use private keys that they can't read or copy to other machines. (caveat: I don't know if the emulator is actually secure in this mode, but it can be made to be)
• My TPM chip doesn't seem to like importing (create_tpm_key -w foo.key) 4096 bit keys. 2048 was fine though.

Links

• trusted platforms module (TPM), openssl and ecryptfs tutorial. Another nice use of TPM. Also installation instructions for TPM emulator
• Resources from a TPM workshop
• Use TPM on Linux

As you can plainly see from this graph, my TPM chip can do approximately 1.4 SSL handshakes per second. A handshake takes about 0.7 seconds of TPM time, so when two clients are connecting the average connect time is 1.4 seconds. This means probably not useful on server side, but should be good for some client side applications.

To replicate the test, start a server:

openssl s_server -keyform engine -engine tpm -accept 12345 -cert foo.crt -key foo.key -tls1 -CAfile foo.crt -verify 1 -status

for n in $(seq 100); do time openssl s_client -tls1 -connect localhost:12345 < /dev/null >/dev/null 2>/dev/null;done 2> timelog

plot [1:] [0:2] '2' using (2/$1) w l title '2 clients','1' using (1/$1) w l title '1 client'

As you remember from long ago hashes are O(1) best case, but can be O(n) if you get hash collisions. And if you're adding n new entries that means O(n^2).

I thought I'd take a look at the hash_set/hash_map GNU C++ extension.

1
2
3
4
5
6
7
8

  inline size_t
  __stl_hash_string(const char* __s)
  {
    unsigned long __h = 0;
    for ( ; *__s; ++__s)
      __h = 5 * __h + *__s;
    return size_t(__h);
  }

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

#include<time.h>
#include<iostream>
#include<hash_set>

double getclock()
{
  struct timespec ts;
  clock_gettime(CLOCK_MONOTONIC, &ts);
  return ts.tv_sec + ts.tv_nsec / 1e9;
}

_GLIBCXX_BEGIN_NAMESPACE(__gnu_cxx)
template<> struct hash< ::std::string> {
  size_t operator()(const ::std::string& x) const {
    return hash<const char*>()(x.c_str());
  }
};
_GLIBCXX_END_NAMESPACE

int main()
{
  __gnu_cxx::hash_set<std::string> the_set;
  double start = getclock();

  while (std::cin.good()) {
    std::string line;
    getline(std::cin, line);

    the_set.insert(line);
  }
  std::cout << "Size: " << the_set.size() << std::endl
            << "Time: " << getclock()-start << std::endl;
}

$ ./generateCollisions 5 > coll-5
$ ./generateCollisions 6 > coll-6
$ wc -l coll-5 coll-6
  13826 coll-5
 149696 coll-6
 163522 total
$ seq -f "%06.0f" 13826 > num-5
$ seq -f "%06.0f" 149696 > num-6
$ ./load < num-5
Size: 13827
Time: 0.0148379
$ ./load < num-6
Size: 149697
Time: 0.135555
$ ./load < coll-5
Size: 13827
Time: 1.44663
$ ./load < coll-6
Size: 149697
Time: 212.009

Collision generator

Summary

Links

• 28C3: Effective Denial of Service attacks against web application platforms

If you split up code into different libraries you can get a diamond dependency problem. That is you have two parts of your code that depend on different incompatible versions of the same library.

Normally you shouldn't get in this situation. Only someone who hates their users makes a non backwards compatible change to a library ABI. You don't hate your users, do you?

(just kidding about hating your users.)

Disclaimer

I thought I'd dive into this problem as a weekend project. Don't rely on this article as a source of truth, but please correct me where I'm wrong. I'm not an expert in creating shared libraries, and it's much harder that it would first appear. The existence of libtool proves that.

Example project described can be found here.

Multiple versions of the same library

The lovely land of modern Unix will allow you to have multiple versions of the same library installed at the same time. That's not the problem. The problem is that when you load the libraries all the symbols are resolved inside the same namespace. There can't be two versions in use of the same function with the same name, even if they are in different libraries (well, it won't work the way you want it to). You can't choose which one to see from different code. (see breadth first search description here for details).

The scenario

In this scenario there are three separate libraries:

• libd1 - Calls functions in libd2 and libbase version 1. File name is simply libd1.so.
• libd2 - Calls functions in libd1 and libbase version 2. File name is simply libd2.so. (cross dependency between libd1 and libd2)
• libbase - This is the library that made non backwards compatible changes between version 1 and 2. Files are libbase.so.1.0 and libbase.so.2.0, with traditional symlinks from libbase.so.n. The important bit here is that the two versions are to be considered incompatible and must both be usable in the same process at the same time.

What we want to achieve is to have libd1 and libd2 be able to call functions in libbase, but to have them call different versions. Both libbase.so.1 and libbase.so.2 should be loaded and active (have their functions callable).

What normally happens

When you link an executable with -lfoo the linker finds libfoo.so and verifies that it (along with default libraries) contain all the symbols that are currently unresolved. Linking shared objects (DSO, .so files) is similar, except there's no requirement that all symbols are resolved. If an unresolved symbol exists in more than one library then only one is actually used. You can't therefore link two overlapping ABIs, even indirectly via an intermediate dependency. Well, you can, but are you comfortable with one of them being "hidden"? (which one is not important for this article)

Both versions of the library can therefore be loaded, but if they overlap then one will hide the other (in the overlap).

How to solve it

The hiding of one of the symbols in a name clash is done by the dynamic linker (ld.so) at load time (run time). Since symbols are referenced by name the only way to differentiate the two names is to force them to have different names.

This can be done in different ways; manually, or automatically. Both append some text to every symbol you want to differentiate.

Automatic (with --default-symver)

$ gcc -fPIC -Wall -pedantic   -c -o base1.o base1.c
$ gcc -shared \
		-Wl,--default-symver \
		-Wl,-soname,libbase.so.1 \
		-o libbase.so.1.0 base1.o
$ nm libbase.so.1.0
[...] 00000000000006f0 T base_print     <--- same name as without the special args
[...] 0000000000000000 A libbase.so.1   <--- this is new with --default-symver
[...]

Let's compile and link one of the libraries that uses libbase.so:

$ ldconfig -N -f ld.so.conf
$ ln -fs libbase.so.1 libbase.so  # library to link with
$ gcc -fPIC -Wall -pedantic   -c -o d1.o d1.c
$ gcc -Wl,-rpath=. -L. \
		-Wl,--default-symver \
		-Wl,-soname,libd1.so \
		-shared -o libd1.so d1.o -lbase
$ nm d1.o
[...]            U base_print
[...]            U d2_print
[...]
$ nm libd1.so
[...]            U base_print@@libbase.so.1
[...]            U d2_print
[...]
$ ldd libd1.so
[...]      libbase.so.1 => ./libbase.so.1
[...]

Also note that libd1.so knows that it's depending on libbase.so.1 (not plain libbase.so). This is the name given with the -soname option when linking libbase.so. So there's even more magic going on than I led on. And it's a good reason for having that option.

libd2.so is compiled similarly, except against version 2 of libbase.so. The program p is then linked to both libd1.so and libd2.so, which in turn pulls in both versions of libbase.so:

$ cc -Wl,-rpath=. -o p p.o -L. -Wl,-rpath=. -ld1 -ld2
$ ldd p
	linux-vdso.so.1 =>  (0x00007ffff41f8000)
	libd1.so => ./libd1.so (0x00007fe08b788000)
	libd2.so => ./libd2.so (0x00007fe08b586000)
	libc.so.6 => /lib/libc.so.6 (0x00007fe08b1e5000)
	libbase.so.1 => ./libbase.so.1 (0x00007fe08afe3000)
	libbase.so.2 => ./libbase.so.2 (0x00007fe08ade1000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fe08b98c000)
$ nm p | grep libd
                 U d1@@libd1.so
                 U d2@@libd2.so
$ ./p
base version 2> init
base version 1> init
d1> init
d2> init
d1()
 d2_print()
  base version 2> d1        <--- libd1 -> libd2 -> libbase version 2
d2()
 d1_print()
  base version 1> d2        <--- libd2 -> libd1 -> libbase version 1
d2> fini
d1> fini
base version 1> fini
base version 2> fini

Manual with --version-script

Instead of --default-syms one can use --version-script=base1.map and create the map file.

1
2
3
4

BASE1 {
      global: base_print;
      local: *;
};

Manual from within the code

asm(".symver base_print_foo,base_print@@BASE1");

Notes

• "@@" in a name means "this version and the default". A single "@" means just "this version".

Summary

… yet Unix still has much less DLL hell than Windows.

I aimed to provide an accessible view into shared libraries by solving a specific problem. For more in-depth information from people who know more about it than me see the links below. I've barely scratched the surface.

The compile time linker (ld) and the dynamic linker (ld.so) do a lot of magic. More than you'd expect if you haven't thought about it before.

Links

• Example project
• How To Write Shared Libraries, by Ulrich Drepper
• Program Library HOWTO - 3. Shared Libraries
• ld.so GNU linker/loader
• Gold readiness obstacle #1: Berkeley DB. Also see other linker related posts on that blog.
• It's not all gold that shines — Why underlinking is a bad thing. Nice illustrations of the problem.
• Autotools Mythbuster - Chapter 3. Building All Kinds of Libraries — libtool
• Why there are shared object files and a bunch of symlinks
• Binutils ld manual - 3.9 VERSION Command

A few months ago I was strolling in the Debian bug tracking system and found a curious bug filed against Arping, a program I maintain.

It said that unlike Arping 2.09, in Arping 2.11 the ARP cache was not updated after successful reply. I thought that was odd, since there's no code to touch the ARP cache, neither read nor write. Surely this behaviour hasn't changed?

I tried to reproduce the behaviour and sure enough, with Arping 2.09 the arp cache is updated, while with 2.11 it's not.

$ arp -na | grep 192.168.0.123
$ # --- First try Arping 2.11 ---
$ sudo ./arping-2.11 -c 1 192.168.0.123
ARPING 192.168.0.123
60 bytes from 00:22:33:44:55:66 (192.168.0.123): index=0 time=1.188 msec

--- 192.168.0.123 statistics ---
1 packets transmitted, 1 packets received,   0% unanswered (0 extra)
$ arp -na | grep 192.168.0.123
$ # --- Ok, that didn't change the ARP cache. Now try 2.09 ---
$ sudo ./arping-2.09 -c 1 192.168.0.123
ARPING 192.168.0.123
60 bytes from 00:22:33:44:55:66 (192.168.0.123): index=0 time=794.888 usec

--- 192.168.0.123 statistics ---
1 packets transmitted, 1 packets received,   0% unanswered (0 extra)
$ arp -na | grep 192.168.0.123
? (192.168.0.123) at 00:22:33:44:55:66 [ether] on wlan0

So what changed between 2.09 and 2.11?

$ git log --pretty=oneline arping-2.09..arping-2.11 | wc -l
43

• Switching to clock_gettime() (various patches). Read gettimeofday() should never be used to measure time for why.
• Switching to select() from poll()
• Adding support to use getifaddr() to find the correct output interface

Between Arping 2.09 and 2.10 I changed the interface finding code from an ugly hack of running /sbin/ip route get 1.1.1.1 to get the outgoing interface from the routing table. Since the output of the various "show me the routing table" commands are different in different OSs, I had to implement this subprocess (ugly) and parsing (ugly) several times. The new implementation uses getifaddrs() to traverse the interfaces programmatically.

The old code was still there as a fallback. It would never actually get used unless there's a Linux system out there that doesn't have getifaddrs(). It seems it was added to glibc 2.3 back in 2002. Anyway it was trivial to temporarily switch interface selection back to the old method. I confirmed that this was indeed what caused this change of behaviour.

Surely ip route get doesn't send an ARP request and populates the ARP cache when it gets the reply? No. So if ip route get 1.1.1.1 doesn't do it, and arping-2.11 1.1.1.1 doesn't do it, then surely ip route get 1.1.1.1 ; arping-2.11 1.1.1.1 doesn't do it?

Yes, yes it does. It seems ip route get 1.1.1.1 followed by arping-2.11 1.1.1.1 will cause 1.1.1.1 to show up in the ARP cache. And it doesn't matter if ip route get is run as an ordinary user or as root! (arping of course has to run as root or have NET_ADMIN capability). Only the exact address given to ip route get will be "open to be filled" by the second command, so it seems to be per address, and that ip route get will modify state in the kernel.

$ arp -na | grep 192.168.0.123
$ sudo ./arping-2.11 -i wlan0 -q -c 1 192.168.0.123
$ arp -na | grep 192.168.0.123
$ # --- Ok, still no entry in the ARP cache Now try running both commands ---
$ ip route get 192.168.0.123 ; sudo ./arping-2.11 -i wlan0 -q -c 1 192.168.0.123
192.168.0.123 dev wlan0  src 192.168.0.100 
    cache  mtu 1500 advmss 1460 hoplimit 64
$ arp -na | grep 192.168.0.123
? (192.168.0.123) at 00:22:33:44:55:66 [ether] on wlan0

I closed the bug since it's working as intended.

I have not dived into the kernel source to find the reason for this, but I may come back and update this post if and when I do.

Links

• Arping on github
• Arping home page
• Debian bug 667808: Arping 2.11 not update arp cache entry

To properly compile a static C++ binary on Linux you have to supply -static, -static-libgcc and -static-libstdc++ when linking.

That's fucked up.

Never EVER think that linking (at link time or runtime) is easy or obvious.

I link my current pet project with:

g++ -Wl,-z,now -Wl,-z,relro -pie -static-libstdc++

The binary then seems to work across the systems I currently want to run it on. Specifically it makes me able to run the binary compiled on Debian Testing on a Debian Stable installation.

Skipping that whole dynamic libraries thing is something Go got right.

Update: some clarification on why you'd want to compile statically

Let's start with the reason for wanting to compile static in the first place. While shared libraries are better in some aspects, "save RAM" is no longer a good reason for always compiling dynamically. There are reasons why you'd want dynamic linking still, but that aint it. The savings in RAM for the .text and .rodata sections (and unmodified .data, I guess) are negligible when compared to RAM footprint of interesting applications. There the interesting data is usually mmap()ed or simply read or generated. Shared libraries won't help you with that. KSM (Kernel Samepage Merging) will even help in doing reverse copy-on-write to take back even this loss.

Take /usr/bin/ssh for example.

$ ldd $(which ssh) | awk '{print $3}' | sed 1d | grep -v ^$ | xargs du -Dhcs | tail -1
4.6M total

No, the big benefit of shared libraries is the fact that you only have to upgrade in *one* place to have the code upgraded everywhere.

Where static linking wins is where you have control of "all" aspects of operations. That is, if there is a new version of openssl to fix a bug, it's feasible in your environment to recompile. You then also have the option to roll back, and roll back openssl to only one application in case there is an incompatibility. This won't, of course, help with dlopen()ed libraries.

Now back to what the options mean. "-static" is the option to ostensibly not require (assume existence and compatibility of) shared libraries on the target system. Turns out it's not that simple. "-static-libstdc++" is often (I dare almost say "usually") needed if the libstdc++ versions between build and target systems differ by a few years on Linux. Like I said I needed this to compile my C++11 stuff on Debian Testing (frozen to be stable, IIRC) and Debian Stable. One binary. Just scp it to the target and run. No dependencies.

As for libgcc it's mostly for cases where you can't count on access to *any* libraries (such as what you mentioned, as OS hacking), but isn't that why you gave "-static" in the first place? :-)

This post doesn't address other aspects of static linking, such as glibc being static-hostile, a whole topic in itself. Compiling static won't make everything "just work" either.

Last time I was at a hacker conference I for obvious reasons didn't want to connect to the local network. It's not just a matter of setting up some simple firewall rules, since the people around you are people who have and are inventing new and unusual attacks. Examples of this would be rogue IPv6 RA and NDs, and people who have actually generated their own signed root CAs. There's also the risk (or certainty) of having all your unencrypted traffic sniffed and altered.

For next time I've prepared a SheevaPlug computer I had laying around. I updated it to a modern Debian installation, added a USB network card, and set it up to provide always-on VPN. This could also be done using a raspberry pi, but I don't have one.

Always-on VPN is where you have NO network access unless your VPN is up, and then ALL traffic goes through the VPN. By setting up a plug computer as a VPN client you can just plug in an unprotected computer to the "inside" and you'll be protected against the attackers on your local network.

Here are my notes on setting this up. Some of it may not be useful to other people, but it was enough of a hassle looking this up the first time that I want to note it down in case I have to do it again.

WARNING: You may brick your device if you have a different device and/or don't know what you're doing.

1. Upgrade Uboot

   Download u-boot.kwb to your tftp server.

   Marvell>> version
   U-Boot 1.1.4 (Mar 19 2009 - 16:06:59) Marvell version: 3.4.16
   Marvell>> print ethaddr       # Write this down
   ethaddr=00:11:22:33:44:55
   
   setenv serverip 192.168.1.10 # IP of your TFTP server
   setenv ipaddr 192.168.1.20
   tftpboot 0x0800000 u-boot.kwb
   nand erase 0x0 0x60000
   nand write 0x0800000 0x0 0x60000
   reset
   setenv ethaddr 00:11:22:33:44:55
   saveenv
   reset
   

2. Install Debian

   Download the Debian installer uImage and uInitrd to the tftp server.

   setenv ipaddr 192.168.1.20
   setenv serverip 192.168.1.10
   tftpboot 0x00800000 uImage
   tftpboot 0x01100000 uInitrd
   setenv bootargs console=ttyS0,115200n8 base-installer/initramfs-tools/driver-policy=most
   bootm 0x00800000 0x01100000
   

   Then install normally.

3. Set up boot to load kernel and initrd from SD card

   setenv bootargs_console console=ttyS0,115200
   setenv bootcmd_mmc 'mmc init; ext2load mmc 0:1 0x00800000 /uImage; ext2load mmc 0:1 0x01100000 /uInitrd'
   setenv bootcmd 'setenv bootargs $(bootargs_console); run bootcmd_mmc; bootm 0x00800000 0x01100000'
   saveenv
   run bootcmd

4. Install some stuff

   mv /etc/motd{,.dist}
   apt-get install openssh-server screen mg openvpn git python-webpy sudo tcpdump tshark uptimed ntp ntpdate isc-dhcp-server arping pv gcc make kernel-package

5. Setup useful environment

   alias emacs=mg
   alias tl='sudo iptables -n -v --line -L'
   alias ls='ls --color'
   # set up .screenrc
   # visudo: thomas  ALL=NOPASSWD: ALL

6. Set up network interfaces

   /etc/network/interfaces

   iface eth0 inet static
       address 10.1.2.1
       netmask 255.255.255.0
   iface eth1 inet dhcp
       pre-up /.../bin/start
   iface eth2 inet dhcp
       pre-up /.../bin/start
   iface eth3 inet dhcp
       pre-up /.../bin/start

   (many interfaces since they may be swapped around, and the names are persistent by default)

7. Install OpenSSH user CA

   /etc/ssh/sshd_config:

   TrustedUserCAKeys /etc/ssh/user_ca.pub

   cat > /etc/ssh/user_ca.pub

8. Install dnetc

   Just for fun. Download.

9. Set up remote OpenVPN server

   Out of scope for this blog post.

10. Set up DHCP server

    /etc/dhcp/dhcpd.conf

    ddns-update-style none;
    option domain-name-servers 8.8.8.8, 8.8.4.4;
    default-lease-time 600;
    max-lease-time 7200;
    authoritative;
    log-facility local7;
    subnet 10.1.2.0 netmask 255.255.255.0 {
            range 10.1.2.11 10.1.2.250;
            option broadcast-address 10.1.2.255;
            option routers 10.1.2.1;
    }
    

11. Install firewalling scripts

    git clone git://github.com/ThomasHabets/profy.git

12. Set up OpenVPN

    Crate ovpn.conf config in cfg/ using ovpn.conf.template as a template. Symlink /etc/openvpn/ovpn.conf to that file.

13. Set noatime on all filesystems in /etc/fstab

14. Set up kernel build environment

    USB network drivers sucked in default kernel. I only got 2Mbps for the ones that even had drivers. 65% of CPU was interrupt handling, OpenVPN only about 5%.

    cd /usr/bin
    for i in ld objdump ar nm strip objcopy size; do
      ln -s {,arm-linux-gnueabi-}$i
    done

15. Build and install new kernel

    cp /boot/config* .config
    make menuconfig
    time fakeroot make-kpkg kernel_image
    dpkg -i ../kernel*.deb
    mkinitramfs -o /boot/initrd.img-3.7.6 3.7.6
    cd /boot
    mkimage -A ARM -O Linux -T Kernel -C none -a 0x00008000 -e 0x00008000 -n 3.7.6 -d vmlinuz-3.7.6  uImage-3.7.6
    mkimage -A ARM -O Linux -T RAMDisk -C gzip -a 0x00000000 -e 0x00000000 -n 3.7.6 -d initrd.img-3.7.6  uInitrd-3.7.6

16. Test the new kernel

    Copy new kernel to tftp server.

    setenv ipaddr 192.168.1.20
    setenv serverip 192.168.1.10
    tftpboot 0x00800000 uImage-3.7.6
    tftpboot 0x01100000 uInitrd-3.7.6
    bootm 0x00800000 0x01100000

17. If working, default to new kernel

    cd /boot
    mv uImage{,.dist}
    mv uInitrd{,.dist}
    ln -s uImage-3.7.6 uImage
    ln -s uInitrd-3.7.6 uInitrd

18. Profit

    I get about 14Mbps, with <50% CPU assigned to OpenVPN, and the rest curiously "idle". Keep in mind that the SheevaPlug is about 4 years old at this point.

Photo

Links

• Install Debian on SheevaPlug

I'm a big fan of hardware tokens for access. The three basic technologies where you have public key crypto are SSH, GPG and SSL. Here I will show how to use a Yubikey NEO to protect GPG and SSH keys so that they cannot be stolen or copied. (well, they can be physically stolen, of course).

Let's hope pkcs11 support is coming, so that SSH support improves and SSL keys can also be protected.

Parts of this howto are all but copied from YubiKey NEO and OpenPGP. I complete it with some details and the SSH parts.

GPG

GPG normally keeps your private key encrypted using your password. If your keyring is stolen someone can brute force your password and from there decrypt all your files. If someone steals your keyring you should revoke the key as soon as possible, but assuming this revokation gets to all interested parties this will only protect new messages from being encrypted to this key. Old encrypted files could be decrypted by the attacker. Signatures are a bit better off. Sure, you have to re-sign everything and redistributed the signatures, but not nearly as bad.

1. Install some dependencies

   sudo aptitude install gnupg gnupg-agent gpgsm

2. Install Yubikey personalizer

   Compile and install the Yubico library and the Yubikey personalization tool.

3. Switch Yubikey to HID/CCID (smartcard) mode

   $ ykpersonalize -m82
   Firmware version 3.0.1 Touch level 1536 Unconfigured
   
   The USB mode will be set to: 0x82
   
   Commit? (y/n) [n]: y

4. Set up USB device permissions

   While the Yubikey in normal keyboard mode will work without thinking about permissions, when used as a smartcard your user needs to have read/write access to the device. You have three options.
   • Set device owner or permissions manually: chmod 666 /dev/bus/usb/xxx/yyy
   • Set owner or permissions via udev in /etc/udev/rules.d/99-yubikeys.rules: SUBSYSTEMS=="usb", ATTRS{idVendor}=="1050", ATTRS{idProduct}=="0111", OWNER="thomas"
   • Be in whatever group defaults to owning the device on your system

5. Create GPG key pair

   $ gpg --card-edit
   
   Application ID ...: D2760001240102000000000000010000
   Version ..........: 2.0
   Manufacturer .....: test card
   Serial number ....: 00000001
   Name of cardholder: [not set]
   Language prefs ...: [not set]
   Sex ..............: unspecified
   URL of public key : [not set]
   Login data .......: [not set]
   Signature PIN ....: forced
   Key attributes ...: 2048R 2048R 2048R
   Max. PIN lengths .: 127 127 127
   PIN retry counter : 3 3 3
   Signature counter : 0
   Signature key ....: [none]
   Encryption key....: [none]
   Authentication key: [none]
   General key info..: [none]
   
   gpg/card> admin
   Admin commands are allowed
   
   gpg/card> generate
   
   Please note that the factory settings of the PINs are
      PIN = `123456'     Admin PIN = `12345678'
   You should change them using the command --change-pin
   
   [enter 12345678 and 123456 when asked for admin PIN and PIN]
   
   [fill in standard GPG key info]
   

   GPG using the newly created key should now work. It shouldn't even look special, except it will ask you for the PIN when needed, and won't work when the Yubikey NEO is not connected.

SSH

Turns out gpg-agent can act as an ssh-agent too. The reason for doing this is so that you can use your GPG key as an SSH key. It doesn't appear to support SSH certificates though, so I'm not ready to use it as my default one.

1. Get public key in SSH format

   $ gpg-agent --enable-ssh-support --daemon ssh-add -l
   2048 xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx cardno:000000000001 (RSA)
   $ gpg-agent --enable-ssh-support --daemon ssh-add -L
   ssh-add AAAA[...] cardno:000000000001

   Install this public key in ~/.ssh/authorized_keys of the remote machine.

2. Create SSH wrapper script for using gpg-agent

   #!/bin/sh
   exec gpg-agent --enable-ssh-support --daemon ssh "$@"

   You will be asked for your PIN code. Make sure you have a good PIN code, or no PIN code. Having the default of 123456 is intellectually dishonest.

   Using the key with SSH like this does not require setting up gnupg, or a keyring in ~/.gnupg or anything like that. Just install the required prerequisites, fix device permissions, and run the wrapper script.

Links

• Yubico blog describing setting up GPG with the NEO

1. Go has exceptions and return values for error

   Yes it does. Yes, it really really does. We can discuss this for hours but in the end it boils down to four points:

   • In Go some errors cause stack unrolling, with the possibility to for each step in the call stack register code that runs before the stack is unrolled further, or for the unrolling to stop. In programming languages, that's called "exceptions". Idiomatic use in a language of the feature doesn't affect what the feature is. Saying that Go doesn't have exceptions is like saying Go doesn't have NULL pointers (it has nil pointers).
   • There is no non-tautology definition of exceptions that includes the ones in Python, C++ and Java, but does not include panic/rescue in Go. Go on, try to language lawyer your way into a definition. In spoken/written languages we have words, and those words have meaning. And the word for this meaning is "exceptions".
   • The programmer must write exception-safe code, or it will be broken code.
   • encoding/json uses panic/rescue as exceptions internally. Thus proving that they are exceptions, and they're usable (arguably useful) as such.
   These three points feel like three stages of grief. Denial ("error codes only!"), bargaining ("What if we redefine the word "exception?") and eventually acceptance ("sigh, ok... what does this mean for my code?").

   defer isn't just a good idea. It's the law. If you want to do something while holding a lock then you have to lock, defer unlock, and then do whatever it is that needs to be done. Here's an incomplete list of things that can cause stack unroll, thus requiring cleanup to be done in defer:

   • Array or slice lookups
   • Acquiring a lock
   • Expected system files (like /dev/random) missing
   • Bugs in your code
   • Bugs in library code
   • Operating on channels

   This code is not safe! It can leave your program in a bad state.

   1 2 3

   mutex.Lock() fmt.Println(foo[0]) mutex.Unlock()

   Fine, it has exceptions and you have to write exception-safe code. So what?

   Do you know anybody who thinks mixing exceptions and error return values is a good idea? The reason Go gets away with it is that Go programmers stick fingers in their ears and scream "la la la, Go doesn't have exceptions" and thus avoid defending that choice.

   Fine, but I don't ever use rescue, so panic() becomes a graceful assert fail

   You may not rescue, but your framework might. In fact, the standard Go library HTTP handler parent will catch... I'm sorry, "rescue" a panic and prevent your program from crashing gracefully. All code in your HTTP handlers, and any library (yours or third party) now has to be exception safe. Idiomatic Go must be exception safe. It's not like idiomatic Erlang where you can assume your process will actually die (One can write bad code in any language, I'm specifically referring to idiomatic code here).

2. No way to run things at end of scope

   This code is bad:

   1 2 3 4 5 6 7 8 9 10 11

   mutex.Lock() defer mutex.Unlock() doSomethingRandom() for _, foo := range foos { file, err := os.Open(foo) if err != nil { log.Fatal(err) } defer file.Close() doSomething(file) }

   (and my GOD is it verbose)

   The reason is that no file is closed until the function returns. Now, how pretty is this code that fixes it?

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

   func() { mutex.Lock() defer mutex.Unlock() doSomethingRandom() }() for _, foo := range foos { func() { file, err := os.Open(foo) if err != nil { log.Fatal(err) } defer file.Close() doSomething(file) }() }

   C++ runs destructors at end of scope, enabling the lovely std::lock_guard. Python has the "with" statement, and Java (the weakest in the bunch in this one aspect, aside from Go) has "finally".

   The defer is needless typing. What do I want to do when my file object is unreachable? Close the file, obviously! Why do I have to tell Go that? My intent is obvious. C++ and CPython will do it automatically. It's so obvious that File objects actually have a finalizer that closes the file, but you don't know when or if that is called.

3. if ... := ...; <cond> { is crippled

   C syntax for assigning and checking for error in one line doesn't really work with multiple return values (well, it does, but it would use the comma operator. Not optimal). So when Go supplied this return-and-check syntax it seemed pretty nice. Until it collided with scoping rules. If the function returns more than one value then you either have to manually "var" all values and use "=", or do all work in an else handler.

   Bad code:

   1 2 3 4 5 6 7 8

   foo := 0 if len(*someFlag) < 0 { if foo, err := doSomething(*someFlag); err != nil { log.Fatalf("so that failed: %v", err) } else { log.Printf("it succeeded: %d", foo) } }

   Good code:

   1 2 3 4 5 6 7 8 9 10 11

   foo := 0 if len(*someFlag) < 0 { // Can't use := because then it'd create a new foo, // so need to define err manually. Oh, the verbosity! var err error if foo, err = doSomething(*someFlag); err != nil { log.Fatalf("so that failed: %v", err) } else { log.Printf("it succeeded: %d", foo) } }

4. Pointers satisfy interfaces

   So for any "handle" you get back from a function you now have to check if FooBar is a struct or an interface. "func NewFooBar() FooBar {...}". Should I pass FooBars around, or pointers to FooBars? I don't know that unless I look in a second place to see if FooBar is a struct or an interface.

5. Why do maps and channels need "make()", when all others types have valid nil values?

   I don't mean "why?" on a technical level. I know why nil slices are valid and nil maps aren't. But people give C++ grief all the time because of "B follows from A" excuses.

6. No interfaces or inheritance for variables

   This is sometimes annoying. I don't feel strongly about it though.

7. There are no compiler warnings, only errors! ... except there are warnings too

   There are no warnings in the compiler. Except it's so bloody useful to be able to have warnings, that it's instead been made into an external tool to do it. Which is less useful because it's one extra step you have to manually take.

So Go is not my favourite language. It comes in second after C++. And for many if not most projects I prefer to use Go.

I had this blog post lying around as a draft for a long time. I didn't think it was was "meaty" enough yet, but since I'm no longer a network consultant I don't think it'll become any meatier. So here it goes.

Here I will describe the process of L3-to-L2 mapping, or next-hop resolution and how it works with point-to-point circuits like PPP, ATM and Frame relay. It's the process of finding out what to actually do with a packet once the relevant routing table entry has been identified.

It's deceptively simpler than on a LAN segment, but since people generally learn Ethernet before they learn point-to-point nowadays I'm writing it anyway.

When a packet is to be sent to an address on the same subnet a L3-to-L2 mapping is done to look up the L2 destination address (if any) to apply.

The packet is then encapsulated in a L2 frame and sent out the interface.

On a normal Ethernet LAN segment ARP is used to look up L3-to-L2, and the frame will then carry that (L2) MAC address as its destination. The frame will then be received by (and only by) the intended destination.

In a point-to-point interface there is no L3-to-L2 lookup. Everything that is to go out an interface will just have the L2 encapsulation in question with no destination-specific values. There may be a circuit ID in the L2 header, but it's not specific to the L3 address. ATM & Frame relay are examples of technologies where there, in a point-to-point link, is a circuit ID but no L3-to-L2 lookup.

PPP is a Point-to-Point Protocol. Anything packets that are to go out a PPP interface will not need a destination address ("address" as in remote-host identifier, not address more akin to circuit ID which it can have).

Packets that arrive on the other end of a point-to-point interface will be routed according to the routing table. It doesn't matter that the sending host thinks the destination was directly attached. There is nothing specifying what the sending host thinks. There is only the source and the destination address (and other things you may choose to base routing decisions on, but that's beside the point).

If the destination address belongs to the receiving host then the packet is of course received and handled normally.

So the routing table is used for two things:

1. Specify out which interface the packet should be sent
2. Specify parameters for the L2-encapsulation, such as L2 destination address

Because we're talking about a point-to-point interface point two is not relevant, and the source host will simply push out the packet to the interface with a non-destination-specific L2-encapsulation and nothing more.

Some point-to-point devices try to emulate a LAN, and will send out ARP to what will eventually be a PPP connection. Unless you're running proxy-arp on the remote end this is not going to work. Nobody will answer your ARPs to other hosts on the fake subnet. I suggest that you try to fix your device or application so that it presents what it actually has, a point to point interface.

LAN emulation can be named NDIS, and I'd say you generally don't want it.

Example

R1:
int s0/0
  ip address 1.1.0.1 255.255.255.0

R2:
int lo0
  ip address 100.0.0.2 255.255.255.255
int s0/0
  ip unnumbered lo0
int s0/1
  ip unnumbered lo0
ip route 1.1.0.1 255.255.255.255 s0/0
ip route 1.1.0.3 255.255.255.255 s0/1

R3:
int s0/0
  ip address 1.1.0.3 255.255.255.0
ip route 0.0.0.0 0.0.0.0 1.1.0.2

Even though R1 and R3 are not on the same segment they are able to reach each other. R1 and R3 think that this is just one big happy /24 switched network, even though R2 is actually routing between the two /32s. Because R3 also has a default route pointing to the (nonexisting) 1.1.0.2 it's able to reach all other networks that R2 has routing to, like 100.0.0.2. It will not be able to ping its default gateway 1.1.0.2 though, since no router has that address configured on any interface. The address 1.1.0.2 is only used to choose the outgoing interface. No L2 destination address lookup is needed.

R1#show ip cef 1.1.0.3 detail 
1.1.0.0/24, version 12, epoch 0, attached, connected, cached adjacency to Serial0/0
0 packets, 0 bytes
  via Serial0/0, 0 dependencies
    valid cached adjacency

R2 was just one router in this example, but it could just as well have been a string of a hundred routers, connecting R1 and R3. R2 just routes between /32s.

Recently some time ago (this blog post has also been lying in draft for a while) someone came to me with a problem they had with a Cisco 7600. It felt sluggish and "show proc cpu" showed that the weak CPU was very loaded.

This is how I fixed it.

"show proc cpu history" showed that the CPU use had been high for quite a while, and too far back to check against any config changes. The CPU use of the router was not being logged outside of what this command can show.

"show proc cpu sorted" showed that almost all the CPU time was spent in interrupt mode. This is shown in after the slash in the first row of the output. 15% in this example:

  Router# show proc cpu sorted
  CPU utilization for five seconds: 18%/15%; one minute: 31%; five minutes: 42%
  PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process 
  198   124625752 909637916        137  0.87%  0.94%  0.94%   0 IP Input         
  [...]
  

Ok, so packets are being sent to the CPU because there's something special about them. What? You can sniff the RP and send it using ERSPAN to some UNIX box, and run tshark/wireshark there. But in some cases there is no UNIX box (or other sniffer recipient) than can peel off the ERSPAN header and look inside, and you have no machine more directly attached so that you can run SPAN or RSPAN.

Enter 'debug netdr capture rx'. It starts a sniff of punted packets and puts it into a small buffer. When the buffer is full it stops sniffing. If you're punting a lot of packets this buffer will of course fill fast. Then run "show netdr captured-packets" to see the packets in the buffer. It's safe to do on a live system.

When looking at the packets I saw no reason for them to be punted. It was normal IPv4 packets, TTL wasn't 0, it wasn't directed to the router itself, a route existed in the routing table for it, etc.. However, all of the packets were destined to addresses within the same /24. And sure enough, when tracerouting to the address it was obviously a routing loop.

I fixed the routing loop, and the CPU use dropped by ~10%. I then did a new sniff and found more routing loops that had been misconfigured at the same time. Eventually the interrupt CPU went down to 10%, which was normal for the features and traffic patterns in use. But the total CPU load was still at >90%.

"show proc cpu sorted" showed a whole bunch of "Virtual Exec" processes eating most of it. "show users" revealed that there were multiple logins by user "rancid". RANCID is a program that lots in now and then and runs a few commands to save configuration changes. Apparently these never had time to finish under the high load, and having 10 of these logged in at the same time caused quite a bit of load in itself, so I kicked them out with "clear user vty X".

The CPU went back to normal, and everyone lived happily ever after.

Side nodes

• I use the term "router" here loosely, since we all know the big secret behind the Cisco 7600.
• We couldn't turn off "ip unreachables" or even rate-limit the punting (mls rate-limit or COPP) because (long story short) that breaks stuff for us. Normally you can, though. But if you're using it as a "fix" then you may just be fixing the symptoms, not the problem. Had we done it here we would have lowered the CPU use, but the routing loop would still be there.

Links

• Catalyst 6500/6000 Switch High CPU Utilization
• Troubleshooting high CPU under interrupts on 7600 and 6500 boxes using "debug netdr" tool
• 6500-RP-Inband-SPAN
• 6500 SPAN the RP

Update 2: I have something I think will be better up my sleeve for using the TPM chip with SSH. Stay tuned. In the mean time, the below works.

Finally, I found out how to use a TPM chip to protect SSH keys. Thanks to Perry Lorier. I'm just going to note down those same steps, but with my notes.

I've written about hardware protecting crypto keys and increasing SSH security before:

• GPG and SSH with Yubikey NEO
• Benchmarking TPM backed SSL
• TPM backed SSL
• SSH certificates

Update: you need to delete /var/lib/opencryptoki/tpm/your-username/*.pem, because otherwise your keys will be migratable. I'm looking into how to either never generating these files, or making them unusable by having the TPM chip reject them. Update to come.

When I run this again on a completely blank system I'll add more exact outputs.

1. Install dependencies

   $ apt-get install trousers tpm-tools
   [...]

   Make sure tcsd is running and only listening on localhost.

   $ sudo netstat -nap | grep :30003.*LISTEN
     tcp    0      0 127.0.0.1:30003     0.0.0.0:*    LISTEN   3103/tcsd

2. Verify TPM chip

   Make sure tpm_version and tpm_selftest don't give errors.

   $ tpm_version
     TPM 1.2 Version Info:
     Chip Version:        1.2.4.1
     Spec Level:          2
     Errata Revision:     2
     TPM Vendor ID:       INTC
     Vendor Specific data: 00040001 0003040e
     TPM Version:         01010000
     Manufacturer Info:   494e5443
   $ tpm_selftest 
     TPM Test Results: 00000000

3. Initialise the TPM chip

   sudo tpm_takeownership If you've already taken ownership, but set an SRK password, then run sudo tpm_changeownerauth -s -r. Not all tools support SRK passwords, so you have to remove it, or in other words change it to the "well known password" that is 20 zero bytes (I'm not making that up).

   If it says Tspi_TPM_TakeOwnership failed then you may have to use tpm_clear to reset the chip to factory state and try again. This may involve cold reboots, enabling the TPM chip in the BIOS, and other joys.

   Run sudo pkcs11_startup; sudo service opencryptoki restart.

4. Add users to TPM groups

   Add all users who will be using the TPM chip to the 'tss' and 'pkcs11' groups. Don't forget to log out and in again, or run su $(whoami).

5. As every user, initialize the user's TPM PKCS#11 data store

   $ tpmtoken_init
   [...]

   If you get "Bus Error (core dumped)" then check for, and delete, any /var/lib/opencryptoki/tpm/your-user-name-here/.stmapfile file laying around.

6. Generate an RSA key pair

   $ pkcs11-tool --module=/usr/lib/opencryptoki/libopencryptoki.so.0 \
       --login --keypairgen -d 01 \
       -a "$(whoami)@$(hostname --fqdn) key" \
       --key-type rsa:2048
   [...]

7. Try some pkcs11-tool commands

   $ pkcs11-tool --module=/usr/lib/opencryptoki/libopencryptoki.so.0 -O
   Using slot 0 with a present token (0x0)
   Public Key Object; RSA 2048 bits
     label:      XXXX@XXXX
     ID:         XX
     Usage:      encrypt, verify, wrap
   $ pkcs11-tool --module /usr/lib/opencryptoki/libopencryptoki.so.0 -O --login
   Using slot 0 with a present token (0x0)
   Logging in to "IBM PKCS#11 TPM Token".
   Please enter User PIN: 
   Private Key Object; RSA 
     label:      XXXX@XXXX
     ID:         XX
     Usage:      decrypt, sign, unwrap
   warning: PKCS11 function C_GetAttributeValue(ALWAYS_AUTHENTICATE) failed: rv = CKR_ATTRIBUTE_TYPE_INVALID (0x12)
   [Thomas note: ignore this warning]
   Public Key Object; RSA 2048 bits
     label:      XXXX@XXXX
     ID:         XX
     Usage:      encrypt, verify, wrap

8. Set up simple SSH

   Extract the public part of the key in SSH format:

   $ ssh-keygen -D /usr/lib/opencryptoki/libopencryptoki.so.0
   ssh-rsa AAAAB3NzaHjn[...]uiimW

   Put it in an ~/.ssh/authorized_keys or something.

   $ ssh -I /usr/lib/opencryptoki/libopencryptoki.so.0 thomas@shell.example.com
   Enter PIN for 'IBM PKCS#11 TPM Token': 
   You have new mail.
   Last login: Wed Nov 13 22:14:11 2013 from XXXX.XXXX.com
   thomas@shell$ 

   Add this to your ~/.ssh/config:

   Host *
           PKCS11Provider /usr/lib/opencryptoki/libopencryptoki.so.0

   Or add it to your ssh-agent:

   $ ssh-add -s /usr/lib/opencryptoki/libopencryptoki.so.0
   Enter passphrase for PKCS#11: 
   Card added: /usr/lib/opencryptoki/libopencryptoki.so.0
   $ ssh-add -l
   2048 xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx /usr/lib/opencryptoki/libopencryptoki.so.0 (RSA)

9. If you want, set up SSH certificates

   Unlike when setting up Yubikey NEO-protected SSH keys, this method works with SSH certificates. It shouldn't be too much different from without TPM chips, but I haven't done it yet, myself.

Random notes

• Don't set an SRK password. Some tools don't specifying it (like tpmtoken_init), and only work with the default all-null SRK password.
• If you get a "No EK" error anywhere along the lines, try sudo tpm_createek
• The default "security officer" password is "87654321".
• ssh-add -D does not disconnect completely from the pkcs11 provider (the TPM chip), use ssh-add -e /usr/lib/opencryptoki/libopencryptoki.so.0 if ssh-add -s /usr/lib/opencryptoki/libopencryptoki.so.0 gives errors
• You can use the TPM chip with Chrome. See Loriers documentation for details.
• Another useful tool: /usr/sbin/pkcsconf -t

A Hardware Security Module (HSM) is any hardware that you can use for crypto operations without revealing the crypto keys. Specifically I'm referring to the Yubikey NEO and TPM chips, but it should apply to other kinds of special hardware that does crypto operations. I'll refer to this hardware as the "device" as the general term, below.

Some background

When describing the Yubikey NEO I'm specifically referring to its public key crypto features that I've previously blogged about, that enable using Yubikey NEO for GPG and SSH, not its OTP generating features.

To generate keys for these devices you have two options. Either you tell the device to generate a key using a built in random number generator, or generate the key yourself and "import" it to the device. In either case you end up with some handle to the key, so that you command the device to do a crypto operation using the key with a given handle.

This "handle" is often the key itself, but encrypted with a key that has never existed outside the device, and never will. For TPMs they are encrypted (wrapped) with the SRK key. The SRK is always generated inside the TPM chip.

TPM chips and Yubikey NEO both allow for importing and internal generation of keys. As does YubiHSM, which is a device that provides some other crypto primitives more geared at the server side.

Internal key generation for normal Yubikey OTP mode doesn't any sense, since the security model with these OTPs are a shared secret between the Yubikey and the authentication server. They key can't be extracted after being imported, but it has to exist outside the Yubikey too. So it by definition has to exist in at least two places, and at least temporarily has to exist in RAM.

So, which is it? Generate on device or import a key?

If the answer hadn't been "it depends", then this blog post would have been much shorter. They both have pros and cons.

Arguments for generating on the device

• The key has never been in RAM. Even if your computer was already hacked when you programmed the device, the keys are safe and don't have to be recalled.
• The key has never been on disk. There is no chance that part of the key generator was swapped out to disk while it had sensitive data in RAM. I list this point separately to stress that as soon as the key has ever existed on disk, it should be treated as being copied. While Gutmann hasn't been relevant for many years for modern hard drives, and a single overwrite on a spinning disk is probably safe against even the best adversaries, it's getting harder and harder to actually overwrite anything. With newer filesystems with journals and copy-on-write you can't overwrite data through the file system anymore. And even if you succeed on the file system layer sectors may be remapped on spinning platter disks, and wear leveling makes it pretty much impossible to overwrite all data on SSDs. It's easier, safer, and possibly cheaper (due to write cycle limits), to just destroy the drive and buy a new one.
• Cold boot attacks are prevented. If you generate on the CPU and in RAM someone with physical access could powercycle the machine and read out the contents of the memory. This is an unlikely attack, I'll admit.

Arguments for generating on CPU and importing into device

• Keys can be backed up before importing. You'll note that this is pretty much an attack against yourself. You can GPG the key to an encrypted file that can only be decrypted by the private key you store in your physically guarded safe. This point doesn't matter if your use case makes it cheap and easy to just regenerate the keys.
• The big benefit is that the random number generator is under your control. The entropy for generating the key can be taken from the device, RDRAND in your CPU, timings between keyboard pressings, network packets arriving, etc.. Since you can use (and /dev/random uses) many different sources, any one of them being bad is not a catastrophy. A hardware device can only use itself, and a deliberate or accidental flaw is much more serious.

Key handles

Like I mentioned earlier no matter how you generate your individual keys with a TPM chip, they are encrypted with the SRK key, and the "handles" are opaque blobs encrypted with this key. If the random number generator in the device is flawed, backdoored, or otherwise compromised, then an attacker can exploit this to decrypt your blobs, and thus get to your keys. If the TPM chip random number generator is bad then your key blobs can be broken. But it's not a complete break. As long as the keys themselves are of good quality then the attack only works if you have access to the blobs. You are still in a better situation than if you just had standard ~/.ssh/id_rsa files.

My recommendation

Short answer: generate the key on the device.

Motivation: I respect differing opinions (and have laid out arguments against my recommendation above), and may change my mind in the future. The reason I recommend generating the key on the device is that this is the only way in which you can say "this key has never existed, and will never exist, outside the device" (barring you doing something stupid like generate the key as a migratable key, something that's possible with TPM chips at least, and is even default and the only possible with opencryptoki).

• Do not generate migratable keys. Jeez!

My alternative recommendation

Neither way is bad, really. So if you prefer to generate the key outside the device, I recommend:

• Using /dev/random directly, not /dev/urandom or any homegrown thing.
• Generate the key only after the machine has been up for a while, and has had a chance to gather entropy.
• Never store the key on disk, even temporarily. Use /dev/shm if your key generator has to use files.
• Generate the key and import into the device from the same program, don't pass it along to an importer.
• Overwrite the key in memory when you're done importing it into the device. Keep in RAM for as short a time as possible
• Don't generate keys in a long-running daemon. Let the OS clean up after key generator.

Comments, suggestions, corrections and opinions welcome.

Not long after getting my TPM chip to protect SSH keys in a recent blog post, it started to become obvious that OpenCryptoKi was not the best solution. It's large, complicated, and, frankly, insecure. I dug in to see if I could fix it, but there was too much I wanted to fix, and too many features I didn't need.

So I wrote my own. It's smaller, simpler, and more secure. This post is about this new solution.

Why not Opencryptoki?

• It generates at least some keys in software. As I've explained earlier, I want to generate the keys in hardware.
• It generates migratable keys. This is hardcoded, and some people obviously want migratable keys (for backup purposes). So a fix would have to involve supporting both.
• Opencryptoki has no way to send such parameters from the command line key generator to the PKCS11 library. So not only would I have to implement the setting, but the whole settings subsystem.
• The code is big, because it supports a lot of features. Features I don't need or want. They get in the way of me as a user, and of me fixing the other issues.
• The code is of pretty poor quality. I encountered configuration problems causing segfaults, and that many (if not most) errors (like permission errors) give the error message "Incorrect PIN", because system calls are not checked for success. I can't trust code where a function that is meant to lock a file using open()+flock() tries to flock() file descriptor 2^32-1, because open() was not checked for success. (of course, flock() was not checked for success either, so it just continued happily without having locked anything!)

So what should the replacement be able to do?

• Generate 2048bit RSA non-migratable keys
• Use these keys to sign data (this is how SSH keys work)
• Interface with the PKCS11 API

How to use it

Once installed:

$ mkdir ~/.simple-tpm-pk11
$ stpm-keygen -o ~/.simple-tpm-pk11/my.key
[...]
$ echo key my.key > ~/.simple-tpm-pk11/config
$ echo -e "\nHost *\n    PKCS11Provider /usr/local/lib/libsimple-tpm-pk11.so.0.0.0" >> ~/.ssh/config
$ ssh-keygen -D /usr/local/lib/libsimple-tpm-pk11.so.0.0.0 | ssh shell.example.com tee -a .ssh/authorized_keys
$ ssh shell.example.com    # Unless you have an ssh-agent with other keys, this will use the hardware-protected key.

That's simple enough, isn't it? No harder than generating a software key with ssh-keygen.

How to install it

Also easy. The code itself is a simple ./configure && make && make install (first ./bootstrap.sh if you take the code from GIT directly).

Initialising the TPM chip is the step that may give you trouble, but should be simple to solve with at most a reboot or two in case you need to reset the TPM, to reclaim ownership.

You only need to do one thing: Take ownership. Ideally you only have to run tpm_takeownership -z and give it an "owner password", but you may get errors like:

• It asks for the old owner password. Solution: reset the TPM chip
• Actually, any other error: reset the TPM chip.
• Resetting TPM chip with tpm_clear fails: Power down (fully), then reset the chip from BIOS.

The code

git clone https://github.com/ThomasHabets/simple-tpm-pk11.git

I'll see if I can get a Debian Developer to help me get it into Debian.

Links

• git clone https://github.com/ThomasHabets/simple-tpm-pk11.git
• Chaps, a bigger, more ambitious project for PKCS11 and TPMs coming out of ChromeOS.

In my last blog post I described how to set up SSH with TPM-protected keys. This time I'll try to explain how it works.

I was very tired when I wrote this. I'll re-read it tomorrow to check if I got it right and clear.

SRK

The SRK is a public key pair that is the main secret inside the TPM chip. It is always generated by the chip, and the private key cannot be read or migrated.

In order to use the SRK key with any operation, the SRK password must be supplied. The SRK password is just an access password. It's not related to the key itself. The SRK password is usually set to the Well Known Secret (20 null characters), or sometimes the empty string, or something silly like "12345678".

There is not much point in having a good SRK password, since you probably have to store it on disk somewhere anyway, to allow TPM operations by daemons.

If you want a password then you probably want to set that per key, not chip-wide like the SRK password is.

Key generation

The stpm-keygen binary asks the TPM to generate a key, and the TPM hands back the public portion of the key, and a "blob" that has no meaning to anyone except the TPM. The blob is encrypted with the SRK, and the SRK never leaves the chip.

Key migration

At key generation time you can specify if you want the key to be migratable. If you do, then that allows the TPM operation "take this blob, decrypt the key and re-encrypt it with this other key, and give it back to me". It is meant for migrating a set of keys from one TPM to another, but there's no way to prove to the TPM that the new SRK is really from another TPM chip, so you could provide ANY new key and it'll hand you your migratable key. Key migration operations are locked behind the "owner password" of the TPM.

If you generated a key as non-migratable, then you can't change that attribute and later migrate it. (and vice versa)

Key migration is, in my opinion, a ridiculous concept, and should not be enabled when it can be avoided.

SSH auth operation

During an SSH handshake, the SSH client presents the public half of the keys it has access to. When the server sees a key that it would accept, it asks the SSH client to prove that it has the private key.

The SSH client proves to the server that it has the private key by signing some data (that's the SSH2 protocol. IIRC the SSH1 protocol instead encrypted or decrypted some data).

For TPM keys, the SSH client calls my module telling it to sign this data, and my module in turn hands the key blob and this data to the TPM chip for signing, and is returned the signature. The TPM chip then forgets everything about the key. The TPM chip is not a key store.

Links

• RFC4252 - The Secure Shell (SSH) Authentication Protocol