Internet Protocol Security (IPsec) is a protocol suite for securing Internet Protocol (IP) communications by authenticating and encrypting each IP packet of a data stream. IPsec also includes protocols for establishing mutual authentication between agents at the beginning of the session and negotiation of cryptographic keys to be used during the session. IPsec can be used to protect data flows between a pair of hosts (e.g. computer users or servers), between a pair of security gateways (e.g. routers or firewalls), or between a security gateway and a host.

IPsec is a dual mode, end-to-end, security scheme operating at the Internet Layer of the Internet Protocol Suite or OSI model Layer 3. Some other Internet security systems in widespread use, such as Secure Sockets Layer (SSL), Transport Layer Security (TLS) and Secure Shell (SSH), operate in the upper layers of these models. Hence, IPsec can be used for protecting any application traffic across the Internet. Applications need not be specifically designed to use IPsec. The use of TLS/SSL, on the other hand, must typically be incorporated into the design of applications.

IPsec is a successor of the ISO standard Network Layer Security Protocol (NLSP). NLSP was based on the SP3 protocol that was published by NIST, but designed by the Secure Data Network System project of the National Security Agency (NSA).

IPSec Core Protocols

To support the activities above, a number of different components comprise the total package known as “IPSec”, as shown in Figure. The two main pieces are a pair of technologies sometimes called the core protocols of IPSec. These are the ones that actually do the work of encoding information to ensure security. They are:

IPSec Authentication Header (AH): This protocol provides authentication services for IPSec. What this means is that it allows the recipient of a message to verify that the supposed originator of a message was in fact the one that sent it. It also allows the recipient to verify that none of the data in the datagram has been changed by any intermediate devices en route. It also provides protection against so-called “replay” attacks, where a message is captured by an unauthorized user and re-sent.

Encapsulating Security Payload (ESP): The Authentication Header ensures integrity of the data in datagram, but not its privacy. When the information in a datagram is “for your eyes only”, it can be further protected using the ESP protocol, which encrypts the payload of the IP datagram.

IPSec Support Components

AH and ESP are commonly called “protocols”, though this is another case where the validity of this term is debatable. They are not really distinct protocols but are implemented as headers that are inserted into IP datagrams, as we will see. They thus do the “grunt work” of IPSec, and can be used together to provide both authentication and privacy. However, they cannot operate on their own. For them to function properly they need the support of several other protocols and services. The most important of these include:

Encryption/Hashing Algorithms: AH and ESP are generic and do not specify the exact mechanism used for encryption. This gives them the flexibility to work with a variety of such algorithms, and to negotiate which is used as needed. Two common ones used with IPSec are Message Digest 5 (MD5) and Secure Hash Algorithm 1 (SHA-1). These are also called hashing algorithms because they work by computing a formula called a hash based on input data and a key.

Security Policies and Associations, and Management Methods: Since IPSec provides flexibility in letting different devices decide how they want to implement security, some means is required to keep track of the security relationships between devices. This is done in IPSec using constructs called security policies and security associations, and by providing ways to exchange security association information (see below).

Key Exchange Framework and Mechanism: For two devices to exchange encrypted information they need to be able to share keys for unlocking the encryption. They also need a way to exchange security association information. In IPSec, a protocol called the Internet Key Exchange (IKE) provides these capabilities.

IPSec Architecture & Implementation

Bump in the Wire (BITW) Architecture

IPSec Modes: Transport and Tunnel

Transport Mode: IP header, IPSec headers (AH and/or ESP), IP payload (including transport header).

Tunnel Mode: New IP header, IPSec headers (AH and/or ESP), old IP header, IP payload.

Again, this is a simplified view of how IPSec datagrams are constructed; the reality is significantly more complex. The exact way that the headers are arranged in an IPSec datagram in both transport and tunnel modes depends on which version of IP is being used; IPv6 uses extension headers which must be arranged in a particular way when IPSec is used. The header placement also depends on which IPSec protocol is being used: AH or ESP. Note that it is also possible to apply both AH and ESP to the same datagram; if so, the AH header always appears before the ESP header.

There are thus three variables and eight basic combinations of mode (tunnel or transport), IP version (IPv4 or IPv6) and protocol (AH or ESP). The two topics on AH and ESP describe the four format combinations of transport/tunnel mode and IPv4/IPv6 applicable to each protocol. Note that ESP also includes an ESP trailer that goes after the data protected.