Configuring an IPsec VPN with Debian 3.1

WARNING: This post is really old, so the information here is likely obsolete!

This is yet another HOWTO on configuring a VPN link between two networks using the KAME IPsec implementation on Debian 3.1 (Sarge) with a 2.4.27, or 2.6.8 Linux kernel. Most of this (if not all) has been covered in great detail elsewhere, however I am so happy at pulling it off that I decided to write about it anyway. :)

Network Layout

(   --->   (
      LAN A                VPN Gateway A
      LAN B        --->    VPN Gateway B
(          (

The aim is to have the two LANs on either end communicating freely with each other across a VPN tunnel. In my test setup the VPN gateways do not have public IP addresses, but in practice they will, as the VPN tunnel will run over the Internet. Obviously, the gateways have two network interfaces – an internal interface (.254) and an external interface (.1)

Required software

I will assume that you already have a configured system running Debian 3.1 with a 2.4.x, or 2.6.x kernel (I have tried both.) The standard Debian kernel already comes with all the necessary options for IPsec, so there is no need to compile your own. You will also need to install the following packages: ipsec-tools, racoon and vim (the latter being the source of the xxd utility.)


You will need to have root privileges for the duration of this document.

Under Debian there are two ways you can configure racoon: the normal way, or through racoon-tool. When I first wrote this HOWTO I only covered the racoon-tool method, as it was easier. I have since switched to using the normal method, because as part of my job I work with different Linux distros, so I don't want to be limited to Debian here (as much as I like Debian.) You will have to make your own choice when you first install racoon, but you can change your mind at any time by using the following command:

dpkg-reconfigure racoon

First I will talk about configuring pre-shared keys and static routes, as both of these steps apply regardless of the racoon configuration method chosen.

Pre-shared keys

I will be using pre-shared keys because it is easiest, but you could also use SSL X.509 certificates. The same key is placed in the /etc/racoon/psk.txt file on both gateways. To generate the key I used the following command:

dd if=/dev/random count=24 bs=1 | xxd -ps

This will generate a nice long hex key. This key is then placed into the /etc/racoon/psk.txt file on both gateways. The hex key that goes into this file needs to be preceded by 0x to indicate that it is in hexadecimal format. Below is a sample key entry:     0x9cb1a7607b39265bb741f7280e857d0bb554349f8dda69af

This sample is from Gateway A, so, again, you will need to change the IP address when you edit this file on Gateway B. Make sure you generate your own key, rather than just copy and paste the sample.


The two private LANs will be able to use the VPN without any static routes, but if you want your gateways to use the VPN as well then you will need to add a special route (again slightly different on each gateway.) This is the route on on Gateway A:

ip route add dev eth0 src

This command and the changes that need to be made when adding the route on Gateway B are fairly self-explanatory. You will also need to enable IP forwarding either permanently in /etc/sysctl.conf, or temporarily, using:

echo 1 > /proc/sys/net/ipv4/ip_forward

If, like me, you are setting up a test VPN between two virtual gateways on a local area network, then you will need to add another pair of routes to the gateways, which will let them communicate with each other. Here is the route configured on Gateway A:

ip route add dev eth0

Again, this route will be slightly different on Gateway B. Obviously, you will need to substitute eth0 with the appropriate interface. The reason you only need these routes on a LAN, is that when you set up your VPN to go across the Internet your ISP will take care of directing your packets to the other gateway.

Configuring racoon using racoon-tool (Option 1)

When you install the racoon (the user-space IPsec daemon) package you will be given an option to use racoon-tool to configure it. Use this, unless you have good reason not to. The racoon-tool was written by the Debian package maintainer and it makes configuring racoon easier, by having its own configuration file that will then be used to generate the racoon configuration and the IPSec security policies.

You need to edit the /etc/racoon/racoon-tool.conf file. Obviously you will need to configure it on both gateways and the settings on one will be a reverse of the other. Here is the configuration for Gateway A:

  log: notify

  verify_identifier: on
  hash_algorithm[0]: sha1
  encryption_algorithm[0]: aes


  peers_identifier: address

  admin_status: enabled

After this all you need to do is start the racoon daemon like so:

/etc/init.d/racoon restart

Configuring racoon using the standard method (Option 2)

The traditional racoon configuration method is made of two stages: configuring racoon and then defining the IPSec security policies.

To configure racoon you will be editing the /etc/racoon/racoon.conf file on both gateways. Here is my configuration for Gateway A:

remote {
  exchange_mode main;
  proposal {
    encryption_algorithm aes;
    hash_algorithm sha1;
    authentication_method pre_shared_key;
    dh_group 2;

sainfo address any address any {
  pfs_group 2;
  encryption_algorithm aes;
  authentication_algorithm hmac_sha1;
  compression_algorithm deflate;

The security policies go into the /etc/ipsec-tools.conf file. Here is the file from Gateway A:


spdadd[any][any] any -P out ipsec

spdadd[any][any] any -P in ipsec

After this you need to load the security policies and then start the racoon daemon. I use the following command:

/etc/init.d/racoon stop && /etc/init.d/setkey restart && /etc/init.d/racoon start

That's it! If all went well you will now have a VPN tunnel between the two private LANs.

Firewall rules

Having a firewall is a good idea. Here is a basic firewall script I put together that will block most incoming connections and also prevent access to the internal LAN should the VPN daemon be stopped. This is the script that is run on Gateway A:


#Enable IP forwarding
echo 1 > /proc/sys/net/ipv4/ip_forward


$IPTABLES -t nat -F
$IPTABLES -t mangle -F


# Mark VPN packets
$IPTABLES -t mangle -A PREROUTING -i $EXT_IF -p esp -j MARK --set-mark 1 #VPN

$IPTABLES -t nat -A PREROUTING -s $REMOTE_LAN -i $EXT_IF -m mark --mark 1 -j ACCEPT
# Spoof protection

$IPTABLES -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT #SSH
$IPTABLES -A INPUT -i $EXT_IF -p udp -m udp --dport 500 -j ACCEPT #VPN
$IPTABLES -A INPUT -i $EXT_IF -m mark --mark 1 -j ACCEPT

$IPTABLES -A FORWARD -i $EXT_IF -m mark --mark 1 -j ACCEPT

This is a really basic firewall, therefore you will probably need to edit it to make it more suitable for your needs.


My article is, for the most part, little more than a fusion of the following excellent material:


The information above is accurate to my knowledge, however I provide no guarantees to this effect and consequently accept no liability whatsoever for any bad things that may happen as a result of the reader using this information in practice. Use at your own risk. Oh, and backup your data while you're at it.