Encapsulation and overheads

Sending and receiving data - the basics#

When packets are sent from machine to machine they require certain headers in order that they may get to their destination.

This is a bit like putting a letter in an envelope. The original message is intact, and wrapped in details of where the message originated (sender address details) and where it needs to be delivered (the destination address).

Strip the envelope (the packet headers) and you're left with the message.

This is the principle at work in every computer network no matter how small or large.

In reality, most of the time the packet isn't being surrounded by a header and footer, although with some forms of encapsulation the packet actually has various trailing tags. These headers can stack up, as will be demonstrated.

LAN traffic - Ethernet#

When devices are on the same network (for these purposes we'll keep it simple and say they are all directly connected to the same network switch with cables), there are a couple of ways they can communicate.

The first is Ethernet - sometimes referred to as Layer 2 (from the OSI network model) - and uses MAC addresses. Each device interface has a unique hexadecimal string called a MAC (Media Access Control) address.

An Ethernet header on a packet consists of what is needed to transmit that packet to its destination while saying where its from.

Without getting into how broadcasting and switching works, the important concept here is that pretty much every packet you see on a network will have an Ethernet header, because even the more complex networks built on top, still need Ethernet when transmitting short distances between devices on the same network segments.

Ethernet headers make up 14 bytes, with an optional extra 4 for a VLAN tag. For the purposes of this documentation we will always keep them separate.

VLANs#

Applied to an Ethernet interface, VLANs allow us to segment traffic by tagging packets with a simple unique identifier for that particular network.

This means the packets from different logical networks can all co-exist on the same hardware switching platform. But packets from one VLAN can't see another.

VLAN header is 4 bytes.

1500 bytes payload#

The Ethernet standard 802.3 allows for a payload size of 1500 bytes plus Ethernet headers, but it can get confusing as to where you see the Ethernet headers or where they get stripped by the device sending the packet.

The standard means that when it comes to being able to observe packets in a packet capture (like TCPDump) the practical limit for the size is 1518

Payload + Ethernet + VLAN tag

1500 + 14 + 4 = 1518

IP#

Generally speaking, packets don't get anywhere (certainly on the internet) without an IP address. This is what most engineers are familiar with, and forms the basis of all routed networks, where source and destination IPs form the header for every packet.

Every router on the journey can see where it needs to send the packet, and the destination machine knows where it's from and therefore where to send any replies.

The internet is built on IP (after all it does stand for Internet Protocol) and most site to site networks (and LANs) are built on IP as well - broadly termed Layer 3 networks (again from the OSI model).

IP comes in two flavours - versions 4 and 6. This document ignore IPv6 for simple practical reasons that no customer has a need for it.

IPv4 headers are 20 bytes long, and form part of the Ethernet packet payload. By this, we mean Ethernet and its header (and the VLAN tag) don't care that there is another form of network occurring in its packet. The maximum payload size shrinks accordingly.

Traversing ISP networks#

Up until this point packets have been given the basic headers needed to get from computer to computer on the vast majority of networks, large and small.

Next we are going to see how ISPs multi-tenant their networks to provide WANs for their customers, and deliver internet access to individual sites.

MPLS#

MPLS literally labels packets in order to get them to their destination, keep them on the right network and optionally prioritise them in a rudimentary form.

Packets on an MPLS network will have a minimum of a 4 byte label (like the headers described above).

As soon as a network is multi-tenanted, so, for example, an ISP can sell multi-site WANs to their customers, at least another 4 byte label is needed. Networks inside of networks.

The MPLS labels get inserted between the Ethernet and IP headers and drop the maximum transmissible payload size that little bit more.

This is where we start introducing the term MTU - Maximum Transmissible Unit. In this case the IP MTU, which is 1492 with this MPLS network.

PPP#

Most broadband networks are delivered via an invisible PPP (Point to Point Protocol) tunnel, the only evidence of which is in the reduced MTU size and the credentials needed to login to get a connection on the broadband modem.

ISPs use PPP because you can assign an IP address to the end of it, and it requires authentication which can check against a billing server, offering basic user accounting management.

PPPoA - PPP over ATM is the default for ADSL based networks

PPPoE - PPP over Ethernet is used for more modern implementations such as FTTC and FTTP which run over a newer BT/Openreach network.

PPP has an 8 byte overhead which is why the MTU on broadband circuits is normally 1492 bytes.

Stripping headers#

Another important concept at this point is that headers can be stripped deliberately by interfaces.

The PPP or MPLS headers are stripped at the edges of those networks. Your PC on either type of connection won't see the MPLS labels or the PPP tunnel, just the packet that went inside them.

When looking at packet sizes, you can't just add up every single protocol and header that is in use over a path because they won't necessarily all get used at once.

TCP, UDP; protocol stack#

While we've said that all traffic on the internet and on everyday layer 3 networks all involve IPv4, there are a range of protocols that sit on the IP stack.

The most common are TCP and UDP. TCP for ensuring delivery, UDP for fire and forget.

These also have protocol overheads - headers needed for packet transmission.

Because TCP is more concerned about getting every packet there and in the right order, it has a large header size of 20 bytes.

UDP needs much less becuase it doesn't have any of the traffic cop elements, and so comes in at 8 bytes.

Both can be the basis for tunneling protocols in their own right.

What it does mean though is that the available payload shrinks again, so that if you are copying a file from your PC using TCP, then you need to take into account the extra bytes of overhead on each packet.

TCP MSS#

MSS standard for Maximum Segment Size. It's a setting that can be applied in certain situations to network interfaces whereby the TCP stream entering the interface gets told what MTU size to use and automatically adjusts accordingly.

TCP MSS is exactly that - its for TCP, not for other protocols such as UDP. This makes a significant difference if the application using the UDP has no path MTU discovery or fragmentation capabilities.

ICMP#

Essentially the ping protocol (but a lot more than that in reality), its always important to note that ICMP is a separate protocol to TCP and UDP.

That means when using a ping command to determine whether a network is traversible and what packet size it might take, you need to take into account that it won't have the extra 20 or 8 bytes of TCP or UDP.

ICMP itself though has a header size of 8 bytes.

IPsec - the clunky but ubiquitous VPN#

VPNs are point to point tunnels between devices on distant networks, creating virtual paths that seem closer and more simpler than they are.

They are a way of creating a WAN without expensive MPLS networking, and over internet circuits.

So far, none of the protocols mentioned have done anything with the payload data other than determine how large it is at maximum.

With VPNs we introduce encryption, and with that, more headers.

IPsec has a fairly convoluted handshake method, involves detailed and specific configuration information for every tunnel, and has a high overhead - as much as 57 bytes depending on the packet size.

Lots of (if not all other) SD-WAN technologies rely on IPsec as their foundational tunneling protocol.

Encapsulation#

Importantly, we've reached what we more usually refer to as encapsulation. The original packet is more or less sent intact across disparate networks, ready to be unpacked at its destination.

The networks the packet traverses have no idea of the ultimate source or destination endpoints for the flow of data, and cannot access the payload due to the encryption.

Because of this encapsulation, where the protocol headers reside starts becoming important, at least conceptually.

IPsec can only encapsulate layer 3 packets, so it can't encapsulation Ethernet frames. IPsec uses Encapsulating Security Payload as the protocol - ESP.

We are now at the point where various trailing headers are added, as part of the authentication and encryption methods, so the payload is in the middle of the packet.

Intelligent Network Fabric#

We create connections and complex networks with a number of different logical interfaces with different properties.

TAPs are our Ethernet tunnel, allowing us to connect virtually between EVX and NFR multiple times for each link, and still support Layer 2.

Tunnel Aggregation Groups (TAGs) are our primary form of aggregation and link grouping.

We use these in various combinations to create virtual hypernets for customerrs.

OpenVPN#

Our primary TAP technology is OpenVPN, a popular open source tunneling application.

OpenVPN has headers of 26 bytes, and operates over UDP.

Tunnel Aggregation Groups (TAGs)#

TAGs do not create headers or encapsulation the packets again, so you won't see the TAG illustrated with the packet headers and payload.

TAG technology merely distributes packets across different TAPs using various performance enhancing algorithms.

TAG VLANs#

TAGs are native Ethernet interfaces, and can have VLANs created on them. As standard we operate Management and Service VLANs to separate customer networks from engineer access and monitoring.

Ethernet frames with VLAN tags added on the EVX (or NFR) are sent across the TAPs on each connection, containing the packets sent from customer sites and internet and cloud locations.

Full packet path#

As shown in this diagram, packets are sent from customer networks and take the following path, through TAGs and their VLANs, into the TAPs and across PPP (if broadband).

There is an optional path for if packet reordering is needed, which adds another encapsulation through MLVPN.

Fragmentation & Path MTU Discovery#

Here we also show where we will fragment packets if they are too large - on the TAP interface. Every packet larger than 1464 will get fragmented at this point, and specifically not on the ISP network, where we can't control it.

Fragmentation is generally to be avoided. The TCP MSS will ensure that any TCP based application "should" keep the packet size small enough to get through.

Other applications have built in PMTUD - Path MTU Discovery. This means small test packets (possibly ICMP) are sent between application endpoints in order to determine the highest value of MTU. For those applications that can't rely on TCP, this is a good way of ensuring no packets are fragmented, or blocked entirely.

There is also a marker that can be set on packets so that they don't get fragmented. This is known as the DNF (Do Not Fragment) bit. If a router receives a packet that is too large, but it has the DNF bit set to true, then generally it will honour that marker and drop the packet rather than forwarding it.

Overheads combined#

The above path is summarised as follows - showing what Evolving Networks add, what is needed for the internet, and what the ISP requires.

This is our standard model, without any packet reordering.

Packet reordering#

If applications on the network are sensitive to out of order packets (like a lot of the Microsoft products and services), then we need to add an additional encapsulation which has the ability to reorder.

An extra encapsulation through MLVPN is performed (see interface flow diagram above).

Other considerations#

Application protocol headers#

Of course even the above is a simplified depiction, as the packet the customer sends will also have other headers such as IPv4, TCP, UDP etc. The sizes of these packets will vary according to the application involved, or the files being sent and received.

Some applications are very chatty and use small packets, others have large files that need to maximise packet size when transferring them. Some may have a fixed size, others may vary depending on the situation. But the total available throughput that is able to be achieved through a connection will depend on:

• the size of the pipe
• the MTU
• the applications in use
• jitter
• packet loss

Effects on throughput#

The overheads of all protocols take away from the total available size any packet can be while traversing a network connection. It's important to work out a realistic figure when calculating what your total available throughput should be on any given connection.

Remember that there are two sides to every connection, so just looking at the local connectivity to the internet isn't enough when diagnosing why a stream of data isn't maximising it. The destination endpoint of the data is just as important, along with every other hop in between.

Diagnosing throughput issues can be difficult and time consuming and must be approached carefully and methodically.

IPsec VPNs on top of everything else#

When you run a VPN service on top of a connection, you will again see how the MTU of the payload data shrinks again. This has a direct impact on the total available throughput or "goodput". When checking bandwidth, its vital to test the connection before applying the IPsec tunnel, so that you can clearly see how is available and stable.

IPsec packets are just like any other, so there won't be anything inherently modifying or slowing them down as opposed to any other traffic (unless you've chosen to with a customised QoS profile). Therefore its important to get the IPsec settings right on both VPN endpoints, as if, for example, the packets are too large and therefore fragment, you won't see the full bandwidth you are expecting.

Likewise, IPsec takes CPU cycles, in a much more considerable way than merely routing packets. Check your VPN endpoint to ensure it has the capability to process IPsec at the throughputs your need, while also carrying out the other security roles you have planned.

Applications with small packets#

There are some applications and services that will rarely ever be affected by the size of the available pipe or MTU. These include DNS and VoIP packets. Because they are always small packets, they will rarely hit an issue where tunnels within tunnels will cause them a problem.

If you do experience issue with applications with small packets, then its likely that the problem is not packet size, but something different such as congestion, packet loss, jitter or high latency.