Traffic cops and bottlenecks

The success of YouTube and the BBC's iPlayer is testing the ISPs' ability to deliver. E&T looks at their response to the bandwidth bulge.

When we log on to the Internet, we assume we'll be able to use any service on any website, whether it is reading a Web page or watching a high-quality video stream, whenever we want, at the fastest speeds that our connections allow. With government services moving online, there's also a growing expectation that citizens will use high-quality broadband access to interact with the state.

But networks and Internet service providers (ISPs) are feeling the strain of rising bandwidth demands. A couple of years ago the average home broadband user would rarely consume more than 2GB (gigabyte) of data a month. Now it's around 7GB, according to the UK ISP Plusnet, a BT subsidiary. Viewers who used broadband connections to watch the Beijing Olympic medal ceremonies generated a 25 to 40 per cent surge in traffic. ISPs didn't have the bandwidth to cope, says James Blessing, from the UK Internet Service Providers Association (ISPA). 

Depending on whom you ask, streaming media currently represents about 5 to 7 per cent of UK ISP traffic and has grown by around 180 per cent in the past year. While YouTube accounts for most of that traffic, BBC's catch-up TV service iPlayer is the fastest growing online video site, increasing its share of visits 50-fold over the last year, according to Hitwise, which analyses online usage, search and conversion behaviour.

The other villains of the piece are peer-to-peer (P2P) applications such as BitTorrent, which hog anything from 40 per cent of total bandwidth to 95 per cent overnight, depending on who you ask. P2P applications share large files by downloading them in small chunks from other downloaders' computers. When one user's computer finishes downloading a chunk of a file, it immediately starts uploading that chunk to other users.

A Blu-ray version of the film 'Iron Man' is 4.41GB, so an ISP's worst nightmare is to have a subscriber base full of P2P users who queue up five large movies for download and then go on holiday for two weeks. During that time the users' broadband connections will be saturated, downloading massive amounts of data until all five films are complete, and uploading even more data as the application shares chunks of these films with many others.


Most broadband access in the UK uses fixed-line DSL connections based on BT's network: four million customers are with BT Retail, and five million with ISPs who put their own multiplexers into one of BT's unbundled local-loop (LLU) exchanges. BT also provides a couple of million connections to other operators on a wholesale basis. Virgin Media, which owns a hybrid fibre/coax network (the last mile to the home is coax) has three or four million subscribers receiving broadband over cable.

Mobile broadband is also beginning to take off, with around two million users in the UK, according to Ofcom.

ISPs charge a flat monthly fee of between £10 and £20 to home users, who in turn expect unlimited usage and regular price cuts. When you consider that ISPs pay BT Wholesale around £8 a month for each copper connection (tail) to the subscriber's home, the economics of broadband provision don't seem to add up. That £8 fee gets the broadband signal from the subscriber's home to the telephone exchange and plugged into a port on the BT Wholesale multiplexer. BT then charges around £140/Mbit/s per month to put the traffic onto the ISP's network. This is based on BT subscribers' average usage of 15 to 25kbit/s, or 5 to 8GB of data a month, costing between £2.10 and £3.50 per subscriber, says Blessing.

"[The average] takes in a lot of users [downloading] 2GB and a smaller number at 2TB (terabyte). It's also skewed in terms of cost, as the usage is never 'average' but tends to be made up of individual usage graphs that overlap at peak times: 6pm to 10pm for residential, 9am to 5am for business," he says.

ISPs that use BT's local loop but provide their own multiplexers are charged about £4 per subscriber per month for the tail. A multiplexer costs around £3,000 to support 48 subscribers, and BT charges around £6,000 per year to have it in their exchange racks. The ISP then has to get the user's data onto their own network, at a cost that will depend on where the telephone exchange is and whether they are managing their own connection over someone else's fibre, or renting a connection service from BT Openreach. Both types of ISPs also need to add transit costs at around £10/Mbit/s per month.

This costing model relies on over-subscription, and works on the premise that users are just downloading Web pages, says Blessing.

"Not everyone is getting pages at the same time, so you have a spiky usage pattern," he says. "By the time you overlay thousands of spiky usage patterns you get a flat-ish line, nowhere near maximum capacity.

"Now that people are using broadband to listen to the radio, watch videos and make phone calls on Skype, as well as using virtual private networks (VPNs) to the office, ISPs see a big chunk of constant usage with some spikes on top."

Traffic lights

To reduce congestion on their topped-out networks, ISPs are imposing peak-time usage caps of typically 30 to 40GB per user per month and experimenting with clever traffic management technologies.

Traffic management or 'shaping' is used by most ISPs to ensure their networks run smoothly, with the simplest option being to randomly drop packets at times of network stress.

"The best providers will find a way to drop the unimportant packets first but in most cases, because you can't tell what the important packets are, you just drop them at random. That could be an email that has to be re-sent or a VoIP phone call, in which you'll notice the quality drop, or video streaming where you'll notice it rebuffering. If it's a Web page, it will take longer to download," says Blessing.

More controversial is the use of deep packet inspection (DPI) technology, which classifies traffic by protocol and throws away certain packets, such as those used in P2P connections, at times of high congestion.

In the US, Comcast, an ISP, was using DPI to terminate P2P traffic at peak times and got in trouble with the Federal Communications Commission which said these practices "discriminate against network management protocols rather than treating them equally".

Bell Canada's use of DPI has landed it a class-action lawsuit by Quebec's consumer watchdog, L'Union des Consommateurs, on the grounds that it misrepresented its "always-on constant high speed" service by slowing P2P traffic on the high-speed backbone to which Ontario and Quebec ISPs connect during peak hours. DPI was having an impact on VoIP telephones and also on VPN connections, affecting individuals' ability to work from home.

"What happened with Comcast was an unfortunate use of DPI which you can't blame the technology for. Comcast was trying to disguise their bandwidth management policies," says Klaus Mochalski, of the German DPI firm Ipoque. "If Comcast had been open, there wouldn't have been such fall-out."

In the UK, ISPs including BT are deploying DPI technology with much less fuss. Plusnet, for example, sells guaranteed quality of service for premium prices on the back of DPI, employing Arbor Networks' Ellacoya e30 switches to analyse traffic and Juniper ERX broadband routers to route it.

The e30 switches use a combination of DPI, TCP/IP port numbers and source IP addresses to determine the protocol being used for each packet. Depending on the kind of broadband subscription for which the user is paying, it allocates a priority to that packet. Subscribers pay for different qualities of service and can see their usage levels and upgrade if they're reaching a bandwidth limit.

Traffic is prioritised (queued) into four categories with the highest priority being used for real-time applications such as gaming and VoIP and the lowest priority for P2P and binary Usenet.

"At certain times of the day, rate limits are applied to certain types of traffic like P2P and Usenet to ensure a fair spread of the available bandwidth," says Neil Armstrong, Plusnet's products director.

"Each queue has a certain percentage of the bandwidth that it can take on the ERX and down the central pipes. If the amount of traffic going into the ERX is higher than the capacity of the pipe then the ERX will buffer traffic, and then drop it."

Providing the eyeballs

Consumers who won't pay for better service will end up paying indirectly through an increase in advertiser-funded material, their job being to provide the 'eyeballs' to view it. We're already seeing signs of this with Carphone Warehouse trying to use its broadband subscriber base to sell consumer electronics via Best Buy, and Sky partnering with Universal Music to provide access to music online.

The deal signed by BT, Virgin Media and Carphone last year with Phorm to try out its targeted advertising technology is in the same category. Phorm's Webwise technology uses DPI to track Web-surfing patterns to enable the ISP to target ads to individual users.

While potentially good for bringing in ad revenue, the move has generated complaints in the press about invasion of privacy. The Crown Prosecution Service (CPS) is now gathering evidence on the Phorm trials to see whether it is in the public interest to allow a private prosecution for breach of wiretapping laws.

In the US, a class action lawsuit has been filed against NebuAd, a company that tracks user behaviour, and 26 ISPs that tried out their service, on the grounds that they violated both privacy and computer fraud and abuse laws. These are beginning to be some pricey eyeballs…

Redistribution of wealth

The latest idea for sorting out the economics of consumer broadband is to change the way that content is delivered. A content publisher or broadcaster such as the BBC that wants to put material online will pay a content delivery network (CDN) company such as Akamai or Level 3 a per GB fee to deliver the material to ISP networks in a timely fashion, by replicating the content in local servers. The ISPs receive no income, but usually face extra traffic.

"CDNs are like a private fast lane on the Internet where we have our own partitioned bandwidth capacity and servers for moving content around," explains John Dillon, chief marketing officer of Velocix, a CDN company in Cambridge, UK.

But while a CDN provides a dedicated, private path to a point on the network that is close to the ISP, the content has to travel the last hop, to the ISP's servers, at the same speed as all the other traffic that's arriving on the public Internet highways. "At peak time, between 9pm to 11pm, it's still like hitting the M25 at rush hour and the result is much the same: your video is going to stop and start."

In the UK, Velocix is collaborating with the BBC, Carphone Warehouse, Virgin Media and Kingston Communications on a trial to put content distribution servers in the ISPs' Points of Presence (PoPs) to store local copies of the BBC's iPlayer contents much closer to the subscribers.

The benefit of the model is the way in which it redistributes revenue and cost more fairly among the different parties.

Velocix can charge the BBC lower rates per GB than a traditional CDN because it's delivering the content from within the ISP's network and not using its own transit links to the ISP each time an end user wants to see the content.

It should cost the BBC less to deliver iPlayer services, while providing better performance, because the video isn't travelling over multiple networks operated by different providers. And the ISPs should be happier because Velocix pays them a share of revenue for helping them deliver the content by having servers in their PoPs.

"We have to find a model where we can deliver at ever higher quality and levels of consumption while containing the costs," explained BBC spokesman Richard Cooper, speaking at the Streaming Media Europe Conference in October. "The Velocix model might be more of a capital investment than a service-based model with CDNs, but we're very open to trying out different technical solutions."

Virgin Media is also experimenting with another approach, a transmission method for content delivery using direct fibre links to content providers "to bring the content closer to the customer, like the spokes of a wheel", said Dale Barnes, head of advanced technology trials, at the Streaming Media Europe Conference.

"We have the same economic pressures as everyone else with a flat-rate fee so want to enhance the customer experience of streaming media while looking at what's economically viable."

Let's hope they've got it sorted out by the next Olympics.

Recent articles

Info Message

Our sites use cookies to support some functionality, and to collect anonymous user data.

Learn more about IET cookies and how to control them