Cybertelecom Federal Internet Law & Policy An Educational Project

Internet Backbones :: Interconnection

Derived From: Characteristics and Competitiveness of the Internet Backbone Market, GAO-02-16 (Oct 2001)

To access the Internet, most residential users gain Internet access over DSL, cable, wireless or dial-up lines. For a residential customer, the ISP sends the user's Internet traffic on to the backbone network. To perform this function, ISPs obtain direct connections to one or more Internet backbone providers (connections to multiple backbone paths is known as multihoming and creates redundancy in order to protect against network outages). Small business users may also connect to a backbone network through an ISP, however, large businesses often purchase dedicated lines that connect directly to Internet backbone networks.

An ISP's traffic connects to a backbone provider's network at a facility known as a "point of presence." Backbone providers have points of presence in varied locations, although they concentrate these facilities in more densely-populated areas where Internet end users' demands for access are greatest. If an ISP or end user is far from a point of presence, it is able to reach distant points of presence over telecommunications lines. Figure 1 depicts two hypothetical Internet backbone networks that link at interconnection points and take traffic to and from residential users through ISPs and directly from large business users.

Once on an Internet backbone network, data packets that were created by the end user's computer are routed over the most best known available route and reassembled at their destination point; its possible for different packets to take different routes between end points. Internet packets are created and routed pursuant to the IETF standard the Internet Protocol (IP).

There are many Internet backbone providers offering service in the United States. A few of these are considered to be "Tier 1" backbone providers. See Wikipedia Tier 1 Providers

Regulation: Unlike telecommunications services, the provision of Internet backbone service is only lightly regulated by governmental communications agencies, usually only through ex post antitrust considerations. See discussion of backbone regulation.

NAPs and Private Interconnection Points

Two types of facilities are used for the exchange of data traffic by interconnected Internet backbone providers. The first type of facility, known as a "network access point" (NAP), enables numerous backbone providers to interconnect with each other at a common facility for the exchange of data traffic. Internet data traffic is also exchanged by backbone providers at "private" interconnections. See MCI WCOM/Sprint DOJ Complaint ¶ 25 (describing interconnection points)

A NAP facilitates the interconnection of multiple backbone providers. In the early to mid-1990s, the National Science Foundation (see NSFNET) designed and partially funded four NAPs, each of which was managed by a different company. Since that time, other interconnection points have been constructed, and for purposes of this report, the term NAPs refers to approximately major traffic exchange points that host backbone providers. Managed by different companies, NAPs are not uniform facilities; differences exist in terms of equipment, software, and data transmission rates.

1994 NSF awarded grants for the construction of four Network Access Points:

• New York (Sprint)
• Wash DC (MFS) (aka MAE EAST, acquired by WCOM, acquired by Verizon),
• Chicago (Ameritech) (aka MAE Central), (aquired by AT&T)
• San Jose (Pac Bell) (aka MAE West). [Living Internet: NSFNET] (aquired by AT&T)

Additional Major NAPs include (there are others not listed)

• MAE-Dallas (WCOM, acquired by Verizon [WCOM MCI Merger Order para 143])
• MAE-Los Angeles (WCOM, acquired by Verizon [WCOM MCI Merger Order para 143])

Although most backbone providers we interviewed use the NAPs, a few providers voiced concerns about them. In the first years of their existence, NAPs became congested with the rapid rate of growth in Internet traffic. Two of the providers with whom we spoke said that some NAPs were not well managed. Also, originally some NAP technology was not "scalable"- that is, beyond some level, it was very costly to increase the amount of traffic that could be exchanged at a NAP. If traffic exchange at a NAP became congested, service quality could be compromised. Two typical problems that congestion causes include latency (delay in the transmission of traffic) and packet loss (when transmitted data are actually lost and never reach their destination). For example, one backbone provider told us that the loss of packets at some NAPs had sometimes reached 50 percent.

The congestion and poor quality of connections at the NAPs led backbone providers to engage in another type of traffic exchange known as "private interconnection." Private interconnection refers to the exchange of traffic at a place other than a NAP. Usually, these private interconnections involve two companies entering into a bilateral agreement to exchange traffic; no third party manages the traffic exchange. The parties interconnect their networks at any feasible location, such as a facility of one of the providers. Because of the private nature of these agreements, the number of private interconnections that currently exist across the United States, according to one company representative, is not known. See MCI WCOM / Sprint Merger DOJ Complaint para 26.

Despite a variety of technological developments that have improved traffic flow at NAPs, we found that for the providers we interviewed, the majority of Internet traffic exchange occurs at private interconnection points. Of 17 backbone providers with whom we spoke, 15 used both NAPs and private interconnections; the remaining 2 used only private interconnections, avoiding the NAPs entirely. Slightly more than half of the 15 providers using both NAPs and private interconnection said they exchanged more than 80 percent of their traffic at private exchange points. Of the 17 companies that we met with, 10 provided estimates of how their mix of private interconnection and NAP use would likely change in the future. Nine of the 10 stated that they either plan less use of NAPs in the next few years or do not see their mix of NAPs and private interconnection changing; only one company said that it was likely to make greater use of NAPs in the future.

We found that some Internet backbone providers value several features of NAPs. For example, when a company interconnects at a NAP, it saves on equipment costs and administrative overhead. Representatives of two companies with whom we spoke noted that the NAPs play an important role in helping to keep the market for backbone service open for entry, and thus more competitive, because NAPs provide new backbone firms an efficient, low-cost method for exchanging traffic with numerous other providers.

A third alternative has emerged that is a hybrid of NAPs and private peering. A perceived problem with NAPs is that they tend to be hosted by a particular network which is also a party to the interconnection agreement. Companies such as Equinix have proposed a different model of a network neutral hosting center. The hosting center provides all necessary services. Networks come to the centers and create points of presence by collocating their equipment. Interconnection is achieved simply by running a new cable across the floor. It's like a NAP in that multiple networks come together at one center, but interconnection is private and it is not controlled by one of the parties to the agreement.

Peering and Transit

Independent of the type of facility at which backbone providers exchange traffic, two different types of financial arrangements exist among backbone providers for traffic exchanges. In a "peering" relationship, backbone providers exchange data destined only for each other's network generally without the imposition of a fee. Transit payments, which involve the payment by one backbone provider to another for the mutual exchange of traffic and for the delivery of traffic to other providers, have become more common with time.

When the commercial Internet began, only a few major backbone providers of relatively similar size existed, each of which sent and received roughly equal amounts of traffic. The similarities among these backbone firms led them to view each other as "peers." [See History of CIX] These providers elected to exchange traffic for free, rather than trying to measure the actual traffic exchanged and developing a payment method. In a peering arrangement, two backbone providers agree to exchange traffic destined only for each others' networks. As depicted in figure 2, the peering agreement between backbone provider A and backbone provider B only covers traffic going from A's network to B's network and vice versa. For backbone A to move traffic to backbone C's network under peering, it must have a peering agreement directly with backbone C. See MCI WCOM / Sprint Merger DOJ Complaint ¶ 24; AT&T / Bell South Merger Order 2007, para 123. [SBC / AT&T Merger Order ¶ 110 2005]

The Federal Communications Commission has described peering as follows:

"In a peering arrangement, two IBPs agree to exchange traffic that originates from an end user connected to one IBP and terminates with an end user connected to another IBP. A peering arrangement has two main characteristics. First, in general, peering is settlements-free, i.e., the IBPs do not charge each other for terminating traffic. Second, one peer will not allow traffic from another peer to transit its network to a third IBP. For example, if IBP A only has a peering arrangement with IBP B, and IBP B also has a peering arrangement with IBP C, then IBP B will not allow customers of IBP A to send traffic to or receive traffic from customers of IBP C. In order to provide access to the customers of IBP C, IBP A must either peer with IBP C or enter a transit agreement, as described below, with either IBP B or IBP C." [WCOM MCI Merger Order para 145]

Marcus indicates that some peering agreements can be for a fee. [Levy Slide 4 OECD 2011] [Marcus Slide 4 OECD 2011].

The decision to peer is increasingly premised on quality of performance concerns. "Perhaps most interesting, we found that some non-Tier 1 backbone providers do not want to peer with the largest backbone providers. For example, one provider spoke critically of the quality of peering connections and the quality of service provided between peers. Some stated that it is difficult to guarantee their own clients a certain level of service if they receive few guarantees themselves-a common occurrence under peering. Transit customers, however, do contract for a specified level of service for such items as "uptime"-the functioning of a network without impairment or failure." See [Krogfoss Slide11 OECD 2011 ("Today peering is primarily a performance tool rather than an economic tool")]

 

By the mid to late-1990s, another financial arrangement known as "transit" emerged. Transit and peering are distinctive in two key respects. First, while peering generally entails traffic exchange between two providers without payment, transit entails payment by one provider to another for carrying traffic. Transit agreements thus constitute a supplier-customer relationship between some backbone providers, much like the relationship between a backbone provider and a nonbackbone customer (such as an ISP). Second, when a backbone provider buys transit from another provider, it obtains not only access to the "supplier's" backbone network, but also access to any other backbone network with which its supplier peers. See MCI WCOM / Sprint Merger DOJ Complaint ¶ 23; AT&T / Bell South Merger Order 2007, para 123. [SBC / AT&T Merger Order ¶ 110 2005] Downstream networks can also acquire "partial transit" where they are acquiring access from the upstream provider only to some of the Internet. [Levy Slide 4 OECD 2011]

The FCC has described transit as follows:

A transit arrangement differs from peering in two respects. First, in contrast to a peering arrangement in which IBPs generally exchange traffic without charge, in a transit arrangement one IBP pays the other IBP to carry its traffic. The amount of this charge depends upon the capacity of the connection. Second, in contrast to a peering arrangement in which IBPs only terminate each other's traffic, in a transit arrangement an IBP agrees to deliver all Internet traffic that originates or terminates on the paying IBP regardless of the destination or source of that traffic. In the above example, if IBP A becomes a transit customer of IBP B, then as a paying customer of IBP B, IBP A is able to send traffic to and receive traffic from IBP C via IBP B's network. [WCOM MCI Merger Order para 145]

 

Regarding physical locations, however, both transit and peering take place at NAPs as well as at private interconnection points.

The negotiation of whether two networks agree to peer or transit involves an analysis of the value of the Interconnection. This value is determined by factors such as traffic volume, number of BGP routes exchanged, symmetry of interconnection, location of interconnection, customer base, ancillary services, quality of service, and technology. [Intl Internet Interconnection p. 7 2007] [Kissangou Slide 13] Peering generally happens when the value of the interconnection is balanced for both networks. Peering can happen between Tier 1 backbones, and it can happen locally between Tier 3 networks (local networks that peer keep traffic local and avoid transit and telecom costs).

There have been a number of notable instances where traffic loads between networks become unballanced, the networks fail to renegotiate a new interconnection agreement, and the networks de-peer.

Market Structure / Tiers

There is a segregation of backbone providers into "tiers." The top tier or "Tier 1" providers generally peer with each other and sell transit to smaller backbone providers. However, we found that smaller providers often peered with each other and were able, in some cases, to peer with larger providers. (this topology of the network is historically derived from the design of NSFNET) [a common definition of a Tier 1 provider is one that peers, but does not purchase transit from any other network] MCI WCOM / Sprint Merger DOJ Complaint paras 27-30.  ; AT&T / Bell South Merger Order 2007, para 123. [SBC / AT&T Merger Order ¶ 111 2005]

Tier 1 backbone service has been recognized as a separate relevant product market, and the geographic reach of that market is national. [MCI / WCOM Merger Order para 148 ("all parties appear to agree that the appropriate geographic market is nationwide")] [MCI WCOM / Sprint Merger DOJ Complaint para 31.] [AT&T / Bell South Merger Order 2007, paras 123-24 ("Tier 1 backbone services – the transporting and routing of packets between ISPs and large enterprise customers and Internet backbone networks – constitutes a separate relevant product market"; "we find it appropriate to evaluate Tier 1 backbone services at the national level")] [Verizon / MCI Merger Order ¶ 113, 115 2005] [ Verizon/MCI Order, 20 FCC Rcd at 18494, para. 113] [Cremer p 11]

"We find that Tier 1 backbone services—the transporting and routing of packets between ISPs and large enterprise customers and Internet backbone networks – constitutes a separate relevant product market. In this regard, we note key differences in quality and price between the transit and DIA services offered by Tier 1 and lower tier IBPs. For example, lower tier IBPs, ISPs, and multi-location enterprise customers typically seek service from a provider that can serve all their locations, and not all IBPs with POPs in a particular location will have such reach to all other locations. Only Tier 1 providers can offer such a high level of ubiquitous service. We find that there are no substitutes for these Tier 1 connectivity services sufficiently close to defeat or discipline a small but significant nontransitory increase in price." [SBC / AT&T Merger Order ¶ 112 2005]

"Consistent with Commission precedent and the DOJ’s previous findings, we analyze the market for Tier 1 IBPs using a national geographic market.342 As with special access, enterprise, and mass market services, we conclude that the relevant geographic market for Tier 1 IBP services is the customer’s location. We then aggregate locations where customers face similar competitive choices. Since all Tier 1 IBPs have extensive nationwide networks, we can aggregate Tier 1 customers throughout the United States since they effectively face the same choice of Tier 1 IBPs anywhere in the United States. Moreover, purchasers of Tier 1 Internet backbone service generally need the ability to connect at multiple locations throughout the United States. Consequently, we find it appropriate to aggregate customer locations and evaluate Tier 1 backbone services at the national level." [Verizon / MCI Merger Order ¶ 116 2005]

The market structure of the Internet is evolving significantly. Content Delivery Networks (CDNs) are taking on the role of backbone networks while access networks are acquiring their own national backbone networks. For example, both Google and Comcast now have substantial national network assets. ATLAS Internet Observatory, 2009 Annual Report The larger the network, the larger the incentive to deploy its own backbone infrastructure. [Marcus Slide6- 7 OECD 2011] [Marcus Slides 15-17 OECD 2011 ("any self supply is presumably cheaper than buying transit"]

Meanwhile, some lower tier network are electing to peer with other lower tier networks and bypass paying transit to upstream networks. Hurricane Electric says that this is creating a donut model of peering. [Levy Slide 12 OECD 2011 (noting that it can be quite expensive for a network to build its own peered network)] [Marcus Slide 17 OECD 2011 ("Any self-supply is presumably cheaper than buying transit")] [Krogfoss Slide 5 OECD 2011 ("50 global traceroute servers showed 60% of routes to large ASPs avoided T1 ISPs" (backbones)]]

The illustration in figure 3 shows backbone provider C as a transit customer of backbone provider B and backbone providers B and A as peers. In this case, traffic originating on backbone C can get to backbone B's network as well as to that of backbone A (with which backbone C does not have an independent relationship) because B will pass C's traffic off to A as part of its delivery of transit service to C. Thus, a smaller backbone provider generally need only buy transit from one or two large providers to achieve universal connectivity.

We found that it is generally not viewed as economical for a backbone provider to peer with a less geographically dispersed backbone provider. Thus, even if there were equal traffic flows, the larger provider will tend to carry traffic a further distance-which, according to a larger backbone provider we spoke with, ultimately means more costs are imposed on its infrastructure-when it peers with a provider with a smaller or less widely dispersed network.

Figures 4 and 5 illustrate this paradigm. In figure 4, backbone providers A and B are of similar size, and traffic between the two could be carried mostly by one backbone provider in one direction, but mostly by the other in the opposite direction.

In figure 5, backbone provider D is smaller than backbone provider C, with more limited points at which traffic can be brought onto the network. When backbones C and D exchange traffic, C must carry the traffic much farther on the return path before it can hand off the data packets to D. Therefore, C might consider D to be benefiting from C's network investment and thus, C would be more likely to view D as a customer purchasing access to its network than as a peer in traffic exchange.

The "tiering" of Internet backbone providers and the dual system of peering and transit agreements have caused controversies. Several of the non-Tier 1 backbone providers with whom we spoke expressed concerns about their inability to peer with the largest providers. In particular, we were told that the inability of non-Tier 1 providers to peer with Tier 1 providers puts smaller companies-which must therefore purchase transit service-at a competitive disadvantage. We were also told that peering policies should be made public.

Price Peering & Transit: Some backbone providers with whom we spoke stated that the market is competitive, and transit rates have been decreasing.

• Price of Transit has fallen and converging on cost of peering. [Krogfoss Slide11 OECD 2011]
• Fixed Broadband Revenue per subscriber is steady; unit price for transit is steadily declining; thus cost of carrying per bit of Internet traffic is declining. [Marcus Slides 15-17 OECD 2011]
• Price of transit varies based on geographic location and local competition. [Levy Slide 18 OECD 2011]

Parties engaged in interconnection [Marcus Slide 6 OECD 2011]

• Backbones
  • Global Internet Backbone Providers
  • National Internet Backbone Providers
  • Regional Internet Backbone Providers
• Access / Edge Networks
  • Public Access Internet Service Providers
  • Private networks (academic, corporate, government)
• Content
  • Content Delivery Networks
  • Content hosts (web server farms)
• Application Service Providers

Internet networks exist in a two sided market where they derive revenue from subscribers and from content networks paying transit. [Krogfoss Slide 7 OECD 2011] Marcus concludes that "Content providers make substantial payments for network connectivity, either through payments to Internet backbone providers, ISPs, and CDNs or through investment in infrastructure." [Marcus Slide 7 OECD 2011] Krogfoss states "Content provider revenue is shared as transit revenue to other ISPs" [Krogfoss Slide 7 OECD 2011]

Peering Disputes

• Cogent / Orange

• Comcast and Level3 2010

  • Level3 acquired a contract from Netflix to deliver Netflix content. Comcast claimed that this was a change, that Level3 had become the CDN of Netflix, and attempted to impose a surcharge.
  • Parties
    • Level 3 Comments on Comcast's Letter to the FCC, Level 3 Newsroom (Dec. 21, 2010)
    • Sena Fitzmaurice, Comcast Responds to Level 3's FCC Filing, Comcast Voices (Dec. 17, 2010) ("Level 3 proposes mandatory settlement-free peering for even radically unbalanced traffic -- not only 3:1 or 5:1, but also presumably even 100:1. In other words, to preserve its own business model which may be failing in the marketplace, Level 3 contends that it is perfectly fair to shift all the going-forward costs of sustaining exploding Internet growth onto one network in a two-network arrangement.")~
    • John Schanz, Comcast Continues Discussions with Level 3 -- Offers to Trail New Solutions, Comcast Voices (Dec. 17, 2010) ("We proposed a mutual and relatively modest investment that would allow us both to better understand the traffic, routing, and economic considerations. We also offered to keep the economics of the existing newly executed agreement at "no cost" until we mutually learned the actual costs of the new approach during this trial.") ~
    • Joe Waz, Comcast Answers Today's Traffic Questions, Comcast Voices (Dec. 14, 2010 ) ~
    • Comcast Corp., Comcast Voices Blog, 20 Q's - with Accurate A's - About Level 3's Peering Dispute (Dec. 7, 2010) ~
    • Sena Fitzmaurice, Comcast Responds to Level 3's Latest Press Release, Comcast Voices (Dec. 3, 2010) ~
    • Level 3, Press Release, Level 3 Releases Statement to Clarify Issues in Comcast/Level3 Interconnection Dispute (Dec. 3, 2010)
    • JT Ramsay, Comcast and Level 3: Internet Experts Bring Perspective, Comcast Voices (Dec. 1, 2011)
    • Level 3 Communications Issues Response to Comcast Statement, Level 3 (Nov. 30, 2010)
    • Comcast Corp., Correspondence with FCC Common Carrier Bureau Chief Sharon Gillette, (Nov. 30, 2010);
    • Joe Waz, Comcast's Letter to the FCC on Level 3, Comcast Voices (Nov. 30, 2010)
    • Joe Waz, 10 Facts About Peering, Comcast and Level 3, Comcast Voices (Nov. 30, 2010)
    • Joe Waz, Comcast Comments on Level 3, Comcast Voices (Nov. 29, 2010)
    • Level 3, Press Release, Level 3 Communications Issues Statement Concerning Comcast's Actions (Nov. 29, 2010) ("On November 19, 2010, Comcast informed Level 3 that, for the first time, it will demand a recurring fee from Level 3 to transmit Internet online movies and other content to Comcast’s customers who request such content. By taking this action, Comcast is effectively putting up a toll booth at the borders of its broadband Internet access network, enabling it to unilaterally decide how much to charge for content which competes with its own cable TV and Xfinity delivered content. This action by Comcast threatens the open Internet and is a clear abuse of the dominant control that Comcast exerts in broadband access markets as the nation’s largest cable provider. On November 22, after being informed by Comcast that its demand for payment was ‘take it or leave it,’ Level 3 agreed to the terms, under protest, in order to ensure customers did not experience any disruptions. ")
  • Daniel L. Brenner and Winston Maxwell, The Network Neutrality and the Netflix Dispute: Upcoming Challenges for Content Providers in Europe and the United States , 23 INTELL. PROP. & TECH. L.J. 3,5 (March 2011).
  • Letter Regarding Comcast Corporation and Level 3 Communications Dispute, From New America Foundation, Media Access Project,  Free Press, to FCC (Dec. 8, 2010)
  • Hearings
    • February 16, 2011, House Commerce Committee, Subcommittee on Communications and Technology, Network Neutrality and Internet Regulation: Warranted or More Economic Harm than Good? (Level3/Comcast Peering Dispute was raised as an issue)
  • Papers
  • News
    • Adam Rothchild, Peering Disputes: Comcast, Level 3 and You, Voxel (Dec. 2, 2010) (“In the other corner, we have the financially-challenged Level 3, who’s re-invented itself once again (as seems to happen once a year), shifting a lot of sales focus from wholesale IP transit and infrastructure to CDN. ”)
    • Nate Anderson, Comcast: We bent over backwards to help Level 3 (those bastards), ars technica Dec. 1, 2010
    • Daniel Golding, The Real Story Behind the Comcast Level 3 Battle, Gigaom Dec. 1, 2010
    • Susan Crawford, Bad Timing: Comcast, Netflix, NN, Cable Modems, and NBCU, Susan Crawford The Blog (Nov. 29, 2010) ("The takeaway from today:  No market forces are constraining Comcast – or any of the other major cable distributors, none of which compete with each other.  ")
    • Cecilia Kang, Level 3 Communications Calls Comcast Fees for Netflix Feeds Unfair, Wash. Post, Nov. 29, 2010
    • Brian Stelter, Netflix Partner Says Comcast 'Toll' Threatens Online Video Delivery, The New York Times, Nov. 29, 2010
• 2008 Sprint and Cogent
  • Om Malik, Cogent, Sprint Disconnect Networks, May Cause Web Slowdown, GigaOM (Oct. 30, 2008)
• Cogent and Level3
  • Papers
    • D(3)peered: Just the Facts Ma'am, Rensys, at NANOG 2005 (slides and real audio)
  • News
    • Level 3/Cogent agree new traffic deal, Register 10/28/2005
    • Level 3 And Cogent Reach Agreement On Equitable Peering Terms, Level3 10/28/2005
    • Report: Level 3 may end peering agreement, Bizjournal 10/18/2005
    • Internet Access Dispute Cut Off Some Businesses, Wash Post 10/14/2005
    • Customers Shocked By Level 3's Internet Disruption, Information Week 10/14/2005
    • Dispute threatens to snarl Internet, Boston Globe 10/7/2005
    • Finding a solution to the ISP wars, ZDNet 10/7/2005
    • Hissy fit brings down large chunk of the net, Inquirer 10/7/2005
    • Web Backbone Carrier Level 3 Narrows Loss, Mercury 2/5/2004
• 1997 Depeering
  • Sprint DePeers with Smalls ISPs
  • UUNET Depeering

Links

• Packet Clearing House Internet Exchange Directory

Web services provided by

:: Home :: About Us :: Contact Us :: Sitemap :: Discussion :: Disclaimer :: Search ::