The Topology of the Internet and its Impact on Performance
The goal of this article is to describe the topology of the internet by describing how a device in the home connects to a destination such as a website in the general internet. We will cover the most common topologies and by no means profess to be exhaustive. We will then look at the impact this topology has on internet performance.
While we will avoid using a lot of jargon, we may sometimes be forced to use some terms of art in this article that, for simplicity, are defined in the article “Vocabulary of internet performance”.
Now to the topology …..
We can broadly divide the internet connections into a five parts:
The goal of this article is to describe the topology of the internet by describing how a device in the home connects to a destination such as a website in the general internet. We will cover the most common topologies and by no means profess to be exhaustive. We will then look at the impact this topology has on internet performance.
While we will avoid using a lot of jargon, we may sometimes be forced to use some terms of art in this article that, for simplicity, are defined in the article “Vocabulary of internet performance”.
Now to the topology …..
We can broadly divide the internet connections into a five parts:
- Inside the home: This includes the home wired and wireless (WiFi) connections that connect to the router and modem to connect to the local internet service provider (ISP).
- Local on-ramp connection: The local internet service provider is typically responsible for connecting the home to the general internet and is responsible for working with the local township to reach your home typically through some sort of cable (twisted pair, cable, fibre etc.). There are also options to connect via wireless or satellite.
- Long haul connections: Long haul internet service providers typically run the long distance internet cables that connect different metro areas via high capacity cables. These long haul internet service providers connect to other long haul internet service providers through “peering” relationships. In some cases, these companies are also the local internet service to your home.
- Local off-ramp connection: The other end of the connections are local internet service providers who connect the long haul networks to the companies that host the servers that are the home to web-sites and other internet destinations.
- Hosting services typically (though not necessarily) host the applications and websites that we access. Some services like Amazon web services, host sites for many companies worldwide, while some hosting services are small and host sites for local customers. Companies can choose to host their own servers and are responsible for connecting to the internet through a relationship with a local internet service provider.
How does the internet traffic flow?
Unlike traditional telephone networks, the internet traffic from the home to a destination may not take the same path each time. In fact, even during a single internet session such as visiting a website, the internet traffic may often take multiple paths due to network conditions. The decisions on how the traffic flows is decided by individual routers on the path based on complex real-time calculations.
Inside the home
The most common implementation is a cable modem or equivalent connecting from the outside to a router within the home which often doubles as a wireless access point (that provides WiFi within the home). There are devices in the market that combine the functionality of the modem, the router and a wireless access point.
All devices in the home connect to this router via WiFi or via a wired ethernet connection. The wired direct connection has the best performance characteristics. (Gamers are therefore well advised to use a wired connection directly to this router.) The quality of the WiFi connections are informed by the location of the router relative to the device that is accessing it. It is important to locate these routers in “good” locations to avoid interference.
There are new devices, such as Google WiFi and Eero, that spread WiFi through the home through “mesh” networks. They provide little devices spread around the home that talk to each other to provide the best WiFi experience. While there may be a slight hit in overall performance, they have the benefit of providing uniform WiFi coverage through the home.
There are other technologies that provide connections over power lines. These help avoid traversing WiFi through walls. Again, you should expect a performance hit for that convenience. In older homes, there may be further degradation due to the quality of the wiring.
In conclusion, once the configuration of a home network has been established, one would expect a fairly steady behavior. However, most devices within the home could use a periodic reboot to ensure that there is no degradation due to errant applications running in them.
Connecting inside and outside the home
As mentioned above, the internet service provider connects to the home via a modem that is either independent of the router or is combined with it. This modem is a single dividing line between the home and the local internet service provider. The internet service provider’s responsibility ends at this point and if they can “see” this modem over the internet, typically, their job is done.
Outside the home
Once the connection is outside the home the network is full of routers, switches and other technology to move traffic from one place to the other.
DNS: There is one key element of the network that is very relevant called the Domain Name Service (DNS) whose job it is to translate the destination URL that is provided by the user to an actual network IP address. As one can imagine, without the address there is nowhere to send the traffic. This service is typically provided by an ISP that you connect to. (There are third party DNS providers like Google that you can point your router to if one is savvy enough). A slow DNS can slow down the internet experience as well. Once a destination address has been established, the routers move the traffic in packets until they reach their destination.
The local connection: The quality of these connections and the path the packets take impact the performance. The one connection that is usually the most impactful is the one from the home to first router of a larger provider. This is because there is usually only one physical connection between the home and that router. Given there is no redundancy, any impact to the quality of this connection has an immediate impact on performance. Too many customers in the neighbourhood, a lot of simultaneous streaming movies, and the quality of the physical connections can all impact the experience. Local service providers normally build enough capacity to address the peaks in traffic, but occasionally have to add more. Any erratic behaviour of equipment in this critical path may require maintenance and/or replacement.
The long haul: Once the traffic reaches the larger networks the packets traverse multiple routers and connections that are potentially owned by several internet service providers before they reach their destination. Complex rules and logic determine the path an individual packet of information may take. Any problem in any of the connections causes the packet to be rerouted. There is typically a lot of redundancy built-in to avoid performance issues unless there is a very significant problem in the network, which does happen from time to time, but is rare.
The destination: Most popular internet destinations are hosted in data centers that have a lot of built in redundancy. Internet traffic arriving to these centers often use multiple physical links from one or more providers. These destinations, such as websites, are typically running on multiple machines simultaneously and are designed to handle varying traffic loads. While there are rare scenarios in which these destinations go down, it is more often the case that there are issues with the routing of traffic to them.
This is all great, but what should we take away from this?
Based on this high level view of the topology of the internet, one can draw the following conclusions:
A quick plug for monitor-io
Monitor-io provides visibility into the most common causes of negative internet performance by quickly identifying where the problems lie so that a subscriber can take the necessary actions. Among many other capabilities monitor-io:
Unlike traditional telephone networks, the internet traffic from the home to a destination may not take the same path each time. In fact, even during a single internet session such as visiting a website, the internet traffic may often take multiple paths due to network conditions. The decisions on how the traffic flows is decided by individual routers on the path based on complex real-time calculations.
Inside the home
The most common implementation is a cable modem or equivalent connecting from the outside to a router within the home which often doubles as a wireless access point (that provides WiFi within the home). There are devices in the market that combine the functionality of the modem, the router and a wireless access point.
All devices in the home connect to this router via WiFi or via a wired ethernet connection. The wired direct connection has the best performance characteristics. (Gamers are therefore well advised to use a wired connection directly to this router.) The quality of the WiFi connections are informed by the location of the router relative to the device that is accessing it. It is important to locate these routers in “good” locations to avoid interference.
There are new devices, such as Google WiFi and Eero, that spread WiFi through the home through “mesh” networks. They provide little devices spread around the home that talk to each other to provide the best WiFi experience. While there may be a slight hit in overall performance, they have the benefit of providing uniform WiFi coverage through the home.
There are other technologies that provide connections over power lines. These help avoid traversing WiFi through walls. Again, you should expect a performance hit for that convenience. In older homes, there may be further degradation due to the quality of the wiring.
In conclusion, once the configuration of a home network has been established, one would expect a fairly steady behavior. However, most devices within the home could use a periodic reboot to ensure that there is no degradation due to errant applications running in them.
Connecting inside and outside the home
As mentioned above, the internet service provider connects to the home via a modem that is either independent of the router or is combined with it. This modem is a single dividing line between the home and the local internet service provider. The internet service provider’s responsibility ends at this point and if they can “see” this modem over the internet, typically, their job is done.
Outside the home
Once the connection is outside the home the network is full of routers, switches and other technology to move traffic from one place to the other.
DNS: There is one key element of the network that is very relevant called the Domain Name Service (DNS) whose job it is to translate the destination URL that is provided by the user to an actual network IP address. As one can imagine, without the address there is nowhere to send the traffic. This service is typically provided by an ISP that you connect to. (There are third party DNS providers like Google that you can point your router to if one is savvy enough). A slow DNS can slow down the internet experience as well. Once a destination address has been established, the routers move the traffic in packets until they reach their destination.
The local connection: The quality of these connections and the path the packets take impact the performance. The one connection that is usually the most impactful is the one from the home to first router of a larger provider. This is because there is usually only one physical connection between the home and that router. Given there is no redundancy, any impact to the quality of this connection has an immediate impact on performance. Too many customers in the neighbourhood, a lot of simultaneous streaming movies, and the quality of the physical connections can all impact the experience. Local service providers normally build enough capacity to address the peaks in traffic, but occasionally have to add more. Any erratic behaviour of equipment in this critical path may require maintenance and/or replacement.
The long haul: Once the traffic reaches the larger networks the packets traverse multiple routers and connections that are potentially owned by several internet service providers before they reach their destination. Complex rules and logic determine the path an individual packet of information may take. Any problem in any of the connections causes the packet to be rerouted. There is typically a lot of redundancy built-in to avoid performance issues unless there is a very significant problem in the network, which does happen from time to time, but is rare.
The destination: Most popular internet destinations are hosted in data centers that have a lot of built in redundancy. Internet traffic arriving to these centers often use multiple physical links from one or more providers. These destinations, such as websites, are typically running on multiple machines simultaneously and are designed to handle varying traffic loads. While there are rare scenarios in which these destinations go down, it is more often the case that there are issues with the routing of traffic to them.
This is all great, but what should we take away from this?
Based on this high level view of the topology of the internet, one can draw the following conclusions:
- The broader internet infrastructure is pretty robust and, while possible and newsworthy, it is very unlikely to behave in a way to provide a bad internet experience when users visit popular sites. These are monitored rigorously and the problems here last for a very short amount of time.
- The most common culprits of bad internet performance are a lot closer to home and are
- Equipment and networks within the home
- The connection from the home to the broader internet provided by the internet service provider
- The Domain Name service, typically provided by the internet service provider
A quick plug for monitor-io
Monitor-io provides visibility into the most common causes of negative internet performance by quickly identifying where the problems lie so that a subscriber can take the necessary actions. Among many other capabilities monitor-io:
- Identifies if the problem is within the home or outside
- Identifies if a specific connection within the home is lost
- Continuously monitors the performance of the internet connections and provides trends and patterns of performance
