IP and internet infrastructure: the big picture

Greg
Level Up Coding
Published in
9 min readDec 29, 2020

--

This blog post aims to create a big picture of the Internet protocol (IP) and the internetwork infrastructure allowing the transmission of IP packets.

Plan

  • Main IP transmission principle
  • IP packet anatomy
  • Routers
  • How are IP addresses attributed to LIRs (Local Internet Registries)?
  • Routing inside a LIR?
  • Routing between LIRs: BGP, peering, transit
  • Internet backbone (Tier 1 ISPs)

Main IP transmission principle

The Internet protocol allows two physically distant hosts to exchange (send and receive) data through networks composed of routers.

Exchanged data are encapsulated in “IP packets” which add meta-data in a header field.

The IP is a connectionless (no state between packets) and unreliable (packet delivery is not guaranteed nor the order of the potential delivery) protocol.

Possible example: if host 1 sends 4 packets in the following order: “P1 — P2 — P3 — P4” to host 2, the latter could receive: “P3 — P1 — P2”.

As you can see in the example above, packets have been received in an incorrect order and P4 has even been lost.

IP is a simple protocol which can be run on a wide variety of mediums (ethernet, wifi, cellular connection, …).

IP can even be used with pigeons!

IP packet anatomy

Below are the different fields of an IP packet (IPv4).

Figure 1 — Anatomy of an IP packet

Type of service: this field helps routers to classify the “importance” of the packet transmission (used by VoIP for instance).

Total length: this field is the length (in bytes) of the packet (header + data). The maximum value this field can take is 65.535 bytes. Big packets will have to be divided into smaller ones as the ethernet MTU (Maximum transmission unit) is 1500 bytes.

Fragment offset: this field is used to manage fragmentation of a packet; indeed an intermediate router could break a packet into smaller ones if needed.

TTL (Time To Live): this field is used to avoid IP datagrams from circulating indefinitely (in case of route loop). TTL value is typically 32 or 64 and is decremented by each router. If a TTL value is 0, the router will respond an ICMP packet (special IP packet) to the source host for information purpose.

Protocol: this field is used to specify an IP protocol number for routers (IP protocol ≠ port number !) e.g: ICMP = 1, TCP = 6, UDP = 17, IPv6 = 41, …

IP header checksum: this field is calculated and added by the source host which creates/sends the packet in order to avoid data corruption. Indeed each router when receiving a new packet will first begin by calculating/comparing the header checksum to verify the integrity of the content.

Routers

Routers are specialized (often linux-based) network computers (better performance / energy consumption) which have at least 2 network interfaces (see my blog post “from NIC to socket”) in order to connect 2 networks (or more).

Routers send packets to the correct network interface of a neighbor router in order for the packet to end up at the correct host. A packet can pass through a variable number of routers to reach the correct host.

The Linux command traceroute (or tracert on windows) shows the different ‘hops’ a packet can make from routers to routers in order to reach the correct host:

Figure 2 — Packets path from one host to Google’s servers

How does traceroute work ?

Traceroute sends IP packets (either UDP on Linux or ICMP ‘echo’ on Windows) sequentially.

The first packet sent has a TTL of 1 and this TTL value is incremented 1 by 1 for each following packet.

Routers encountered will decrement the TTL and return an ICMP “Time exceeded” response to the source if TTL is 0 (it will happen exactly once for each router).

Thus the source host will receive ICMP responses for all the routers and will display the different hops in the console.

Some “hops” have a ‘*’ in the result, it means the host didn’t receive an ICMP response in a certain time window because one of the intermediate routers may have been configured not to respond.

Routing algorithms

Routers store routing tables in their memory. These routing tables are frequently updated in order to know where incoming packets should be sent based on the IPs / CIDR.

Routing algorithm is a broad field which raises a lot of issues (efficiency, speed, path creation, …) .

It isn’t the scope of this article but you can take a look at this wikipedia page if you are interested.

How are IP addresses attributed to LIRs (Local Internet Registries) ?

Figure 3 — IP and Autonomous System hierarchy

Disclaimer: Through this post I will focus on ISP (Internet Service Provider), hence the words “AS” (Autonomous System), “LIR” (Local Internet Registry), “ISP” will all be used interchangeably.

Figure 4 — Global and simplified LIR scheme (values are incorrects)

In this scheme (figure 4), I represented a french LIR (SFR) and how it could connect its customers together and to the internet.

Once a LIR has bought some IP address blocks from its RIR (Regional Internet Registry), he can use them for its different Point of presence (PoP and customers.

We will discuss information about this scheme (BGP, IXP, …) in the rest of this blog post.

Routing inside a LIR ?

A LIR (Local Internet Registry) is a collection of networked systems that operate common routing protocols and are under the control of a single administration (wikipedia).

A LIR can select the algorithm he wants for its internal network.

Some routing algorithms for a single routing domain (also called Interior gateway protocols — IGP) are:

Once the IGP algorithm has been selected and configured, customers of the LIR are able to send and receive IP packets between them. However, customers of this LIR won’t be able to communicate with customers of other LIRs (at this stage).

Routing between LIRs: BGP, peering, transit

The Internet is made by Local Internet Registries — LIRs. These LIRs all have a unique AS number (to uniquely identify each other) and multiple sets of IP addresses they bought / rent from their RIR.

LIRs have to find a way to connect to each other and an algorithm to exchange informations about routes.

The routing algorithm used by all LIRs is the BGP (Border Gateway Protocol) algorithm.

There are two ways LIRs can communicate with BGP (we’ll see them in details after):

  • IP Transit
  • Peering

Most LIRs will use both methods and with redundancies.

BGP — Border Gateway Protocol

BGP associates address blocks with an Autonomous System number.

Technically speaking BGP is a “layer 4” routing protocol which runs on TCP port 179.

Oversimplified, thanks to regular updates made by AS routers composing a BGP network, each Autonomous System (LIR) connected to that network has an IP table where each existing IP address block is associated with the correct Autonomous System number. The table also contains the AS path it has to take to reach a specific address block (a path).

Figure 5 — Small portion of a BGP’s routing table

The Internet is a giant BGP network of connected LIRs each identified by its AS number. Hence every member of this BGP network can send and receive updates between them in order to know which path should be taken to reach every AS.

At this time there are around 800k entries in the BGP routing table (called RIB).

Full explanation of BGP click here

Type of BGP connections: peering or transit

A first method to share address blocks with other ISPs is to use BGP peering.

If we still consider the example of the figure 3 / 4, SFR can peer to other french LIRs such as “Bouygues” and “Free” by configuring BGP on some border routers. Once peered, these LIRs will share their address blocks and AS numbers together to send or receive traffic from the others.

Figure 6 — BGP peering

Important note: peering isn’t made between any LIR. Peering will generally be made between ISPs of the same size. None of the two ISPs will pay the other for the use of a peering connection as the traffic will flow both ways. It is a “win-win” deal between LIRs.

Peering connections can be made at an Internet eXchange Point (IXP).

Many LIRs of the same region (country or regional zone) will connect to others at an IXP.

A few IXPs exist per country in general.

With multiple peering connections with other LIRs, a LIR will be able to receive all the routes of the others LIRs and propagate its own.

A question you may ask:

How can an ISP be sure that he is able to reach every existing other ISPs (AS) in this kind of “peering jungle” ?

→ This is made possible thanks to a hierarchy which exists between the different ISPs allowing the emergence of an internet backbone.

Internet backbone (Tier 1 ISPs)

A few ISPs are at the core of the internet, these ISPs are called “Tier 1” networks.

A Tier 1 network is a network that only uses peering with every other Tier 1 ISPs.

Figure 7 — Tier 1 ISPs full-mesh peering

A Tier 1 network can access the whole internet for free as it is peered with every other Tier 1 network hence exchanging BGP updates with every other Tier 1.

There are around 18 Tier 1 ISPs in the world.
In France, only Orange is a Tier 1 network.

It means that every non-tier 1 ISPs don’t have access to the whole internet with peering only.
Non-tier 1 ISPs will have to create what is called an “IP transit” link with a Tier 1 network (or with a non-tier 1 network already having an “IP transit” link with a Tier 1 network).

Once the IP transit link has been established, the customer ISP of the transit provider ISP will be able to send some of its traffic to the transit provider ISP which will then be able to route the packets to the correct destination.
With an IP transit link, the customer ISP will also be able to promote all his address blocks to the entire internet through the transit provider ISP.
To create and use an IP Transit link, a customer ISP has to pay the transit provider ISP depending the amount of data he will consume.

IP Transit offers complete access to the internet.

An ISP which has both peering and IP transit connections (called Tier 2 ISP) can also sell IP transit links to other ISPs.

A Tier 2 ISP will create its own inter-LIRs connection web thanks to peering and transit IP.

Small ISPs called “Tier 3” won’t even peer with any other ISP and will only pay a transit provider ISP to send / receive traffic on the internet.

Figure 8 — Different types of ISPs (Tier 1, 2, 3)

The end !

This article took me some time to write because of the complexity needed to be summarized in order to get a short and big picture of the internet. I hope you enjoyed it, if you see any error don’t hesitate to let me know.

You can also follow me on twitter and check out the references below if you want to dive deeper into the subject.

References

https://en.wikipedia.org/wiki/Internet_transit
https://en.wikipedia.org/wiki/Internet_exchange_point
https://en.wikipedia.org/wiki/Tier_1_network
Complete website on BGP (great)
https://learn.nsrc.org/bgp/
BGP
https://www.noction.com/blog/routage-bgp-attribut-as-path
BGP for IP transit
https://ftp.registro.br/pub/gter/gter30/TutorialBGP/7%20-%20Transit.pdf
RIP explained (french)
https://www.youtube.com/watch?v=eRl0an-a3m0
BGP path vector protocol
https://www.youtube.com/watch?v=k6xMHFOc-rs
Optical fiber FTTH (french)
https://www.busyspider.fr/FTTH-Free-sur-le-chemin-de-la-fibre-optique-mini-lexique-en-images-equipements-termes-employes.ph

Images

Figure 1
VPNs Illustrated: Tunnels, VPNs, and IPsec, Jon C. Snader
Figure 2
https://www.geeksforgeeks.org/traceroute-command-in-linux-with-examples/
Figure 5
https://www.semanticscholar.org/paper/Data-Structure-Optimization-of-AS_PATH-in-BGP-Jiang/f41e9622e634823c6506bf0f5f543174e900d298
Figure 7
https://drpeering.net/FAQ/What-is-a-Tier-1-ISP.php
Figure 8
https://en.wikipedia.org/wiki/Tier_1_network

--

--