FCC Hearings at Stanford: Towards a Consensus on ISP Transparency?

Yesterday, the FCC held a second hearing in its investigation of Comcast's use of forged RST packets to interfere with BitTorrent and other P2P applications. Free Press has a page linking to written testimony, statements, and audio and video recordings from the Stanford hearing.

At the previous hearing at Harvard Law School, Comcast attracted criticism for filling the auditorium with paid attendees. This time around, the telcos declined to participate at all. They sent proxies in their place: a conservative think tank called the Phoenix Center, freelance tech pundit George Ou, and one ISP: Lariat.net of Wyoming. It's a pity that ISPs aren't willing to participate in public debate about their own practices.

EFF has argued that the FCC should use its position of leadership to clarify that ISPs should, at the very least, provide adequate disclosure of any discriminatory network management practices that they deploy (we are also trying to get similar information by promoting independent testing of ISP networks with our Test Your ISP project). This kind of transparency is essential for a properly functioning marketplace: the public must be able to know when their software doesn't work because it's buggy, and when it doesn't work because of interference by an ISP. Without this information, users don't know which tech support line to raise hell with, whether they need to switch to new software, or whether they need to switch to a new ISP.

Transparency and responsiveness is also essential for application developers to understand the way that their applications will have to fit into ISPs' networks.

We were very pleased to see that requirements for disclosure and transparency seemed to command a near-consensus amongst the Commissioners and those testifying. The devil will be in the details, of course: will disclosures be informative enough for programmers to work with and for consumers to make good decisions?

One prevailing point of confusion in the discussion was the relationship between the lack of information about network traffic in general (eg, how much of Internet traffic is P2P? What kind of P2P?), the lack of information about Comcast's discriminatory network management practices (what percentage of BitTorrent seeds has Comcast been reseting? How has that varied at different times, and in different locations across the country?), and the lack of information about discrimination by other ISPs (Cox Communications, for instance, discloses that it uses "traffic prioritization" and "protocol filtering", but we don't know if its techniques are precisely the same as Comcast's, or whether it is planning to phase them out). These are all separate known unknowns and we know the FCC should look in different places if it wants to resolve them.

Another interesting question raised by Commissioner Tate was how an FCC disclosure obligation or principle would fit together with new software tools to test ISPs. We think the answer is that both are required: disclosures by ISPs and independent tests by the public are complimentary; neither of them will tell us everything we'd like to know about the network, and each of them will act as a cross-check for the other.

In the mean time, the threat of intervention by the FCC has caused Comcast to eat a great deal of humble pie. They're promising to work with BitTorrent Inc — we hope they'll also work with the wider Internet community — to find less discriminatory ways to manage their network.

In closing, we doubt that RST forgery will be the last "network management" practice to spark consternation and controversy. But we hope that in future, it won't take the best part of a year of wrangling and an FCC proceeding before transparency and common sense start to prevail.

[Permalink]

Software for Keeping ISPs Honest

Yesterday's announcement of a détente between Comcast and BitTorrent was great news. Unfortunately, the general problem of ISPs doing strange things to Internet traffic without telling their customers is likely to continue in the future. EFF and many other organizations are working on software to test ISPs for unusual (mis)behavior. In this detailed post, we have a round-up of the tools that are out there right now, and others that are in development...

[Read the entire post]

Comcast Reduces Discrimination, Plans To End It Altogether

Last month, shortly before the FCC held its first hearing in an investigation of Comcast's interference with BitTorrent and other P2P protocols, we noticed that Comcast was no longer injecting forged TCP RST packets in the simple tests we had been running on its cable network. Those tests had been showing interference through January 2008. Some sources with access to larger datasets informed us that the cable ISP was nonetheless still using RST packets against some BitTorrent sessions, just not the simple uses of BT and Gnutella that we had been testing. The status quo: Comcast is still interfering with P2P, but they are being more subtle about it.

Today, Comcast has announced that it will phase out its discrimination against P2P protocols entirely by the end of the year. According to the WSJ's coverage, the cable company is considering switching to non-discriminatory dynamic traffic shaping, which — as we've previously argued — is a much more responsible way of coping with network congestion. We're also pleased that Comcast is collaborating with the BitTorrent developers; we've been urging them to collaborate with the wider technical community for some time.

This is a big victory for common sense and a big victory for an Internet based on open standards, not the whims of major ISPs. But there's still more work to do.

In particular, the Internet community clearly needs to do a lot more testing for discrimination by the thousands of ISPs around the planet. EFF — and a number of other groups — have been working to build tools for those tests. In a follow-up post, we'll talk about projects that have already launched, and others that are in the pipeline.

[Update: The follow-up post is now online.]

[Permalink]

EFF to FCC: "Reasonable Network Management" Requires Transparency

In response to the FCC's inquiry into Comcast's interference with BitTorrent traffic, EFF filed comments yesterday urging the FCC to make it clear that ISPs must, at a minimum, adequately disclose their "network management" practices before they can hide behind the excuse of "reasonable network management."

The FCC has invited public comments regarding the Comcast BitTorrent blocking affair in response to two petitions: one filed by Vuze (formerly Azureus) and another filed by the Media Access Project, FreePress and Public Knowledge. (The recent public hearing in Boston, in which Comcast paid people to fill seats, was also part of this same proceeding.)

The central question in the proceeding is whether Comcast has violated the four neutrality principles set out in the FCC's Internet Policy Statement. It seems clear that Comcast's protocol-specific interference with BitTorrent traffic violates those neutrality principles. In response, Comcast (and other ISPs) have offered the excuse that it was all "reasonable network management" -- a catch-all exception to the FCC's neutrality principles.

In its comments to the FCC, EFF urges the agency to clarify that the "reasonable network management" exception to its neutrality principles should only apply where an ISP has adequately disclosed the existence and likely consequence to customers of its discriminatory practices. After all, if we believe that market forces are our first line of defense against unreasonable ISP behavior, those forces can only work if customers, competitors, innovators, and policy-makers know what the ISPs are up to. On that score, Comcast has obviously fallen short, issuing a series of denials, evasions, and half-truths for 10 months after its own customers caught them interfering with BitTorrent traffic. The FCC needs to send a message to Comcast and other ISPs that this is unacceptable.

[Permalink]

Time Warner Puts a Meter on the Internet

Time Warner Cable has confirmed that it will be rolling out metered pricing for Internet access in Beaumont, TX. Although the exact terms have apparently not been set, packages would reportedly offer between 5 gigabytes and 40 gigabytes a month, with the top plan costing roughly the same as the company’s current highest-speed service (~$50-60/month).

On balance, we think this is a fair choice among a bunch of bad options. Cable companies are struggling with an infrastructure that cannot meet the bandwidth needs of all its customers (in other words, they oversold their services). Providing transparent, metered access is certainly preferable to Comcast's arbitrary, undisclosed practice of selectively hobbling particular protocols. (Or, in the words of Harold Feld of the Media Access Project, "I think this beats outright lying about your limitations.")

Overall, business models that keep ISPs thinking of themselves as "pipe" rather than "content" are good. Better that your ISP worry about the tolls to pay for the highway, rather than scheming to force you to use their preferred offramps and eat in their preferred diners.

Transparency also encourages innovation and competition. Already, Verizon is gloating publicly, saying that its more modern FIOS fiber-optic service will not have caps ("step right up, no caps over here, folks"). This also may encourage new broadband technology providers to enter the market, as they will have another way to differentiate their offering from cable broadband ("real unlimited Internet, right this way").

But there are some serious potential drawbacks, too. First, if metered Internet access becomes widespread, it may discourage users from indulging in new, high-bandwidth activities, thereby foreclosing innovative new technologies and markets. For example, we might never have had a YouTube (or a Napster) if people were fretting about their bandwidth consumption.

Second, much will depend on the pricing of these new metered plans. The new plans could be used to bring basic broadband in at a lower price (good), or it could be used as a cover for price increases on existing customers (bad). And the pricing for "overages" should bear some relation to costs, rather than being exploited for windfall profits. (Broadband industry observer Dave Burstein has pointed out that the wholesale price to Time Warner for 40gb for a month amounts to about $3.)

Last word goes to Harold Feld: "The real solution, of course, is policies that build out more capacity so that it becomes too cheap to meter." Now if only we had a real national broadband policy to get us there.

[Permalink]

Scrutinizing Comcast's Apologists

EFF is continuing its research into Comcast's use of forged RST packets to interfere with their customers' BitTorrent connections. (Apparently the FCC is investigating, as well.) While Comcast has remained conspicuously silent about the technical details of its activities, a few networking engineers have tried to defend Comcast by proposing technical justifications for Comcast's interference activities.

One of the most energetic of these pundits is Richard Bennett, who has argued that Comcast deserves a "pat on the back and a gold star", not criticism, for injecting spoofed RST packets into their users' traffic. In this post we're going to examine and rebut his arguments...

[Read the entire post]

Comcast Needs to Come Clean

Over the last couple of days, Comcast has been telling the press that they're not interfering with their users' traffic, they're just "delaying" it. Let's examine that proposition for a moment. In our previous posts, we discussed Comcast's forging of TCP RST packets to kill users' connections on BitTottent, Gnutella and Lotus Notes. To see just how disingenuous Comcast is being, consider the following analogy:

Alice is trying to telephone Bob. Alice telephones Bob, and hears someone answer the phone in Bob's voice. They say "I'm sorry Alice, I don't want to talk to you", and hang up. Except, it wasn't actually Bob who answered the phone, it was Comcast using a special device to impersonate Bob's voice. Comcast might describe this as "delaying" Alice and Bob's conversation, on the theory that perhaps they'll keep calling each other until some day when Comcast isn't using their special device. They may also invoke the theory that Alice will call other people who are a lot like Bob, but aren't on Comcast's network, so her conversation will only be delayed.

If "delaying" traffic was Comcast's private intent, they were clearly making absurd and frequently incorrect assumptions about the protocols they were jamming. No doubt that is how they wound up blocking Lotus Notes. (On that subject: after the blog and media attention that followed our post pointing out that situation, Comcast may have finally listened to IBM/Lotus and taken steps to stop jamming Notes. We're happy they've done that, but we have little confidence that smaller companies or free/open source software developers would be able to get Comcast's attention when their protocols are broken by Comcast's packet spoofing.)

Another thing that Comcast has started suggesting to reporters — without giving any details — is that for obscure technical reasons, they can't always prevent network congestion by reducing the amount of bandwidth available to P2P protocols, so when the network is busy, they start jamming them instead. This is an interesting argument, but it's very hard to evaluate or refute it without any details! (A nice rhetorical trick, that.)

For now, let's be very generous to Comcast and assume what we think is false: let's assume that some P2P network or other actually triggers a flaw in the way cable modem networks are designed, so that it's hard for Comcast to keep plenty of the channel free for web surfers when there are enough P2P nodes around. In that case, rather than just spoofing packets and offering incredibly disingenuous denials after they've been caught red-handed, Comcast should come clean. They should explain what they're doing, and explain in precise and detailed terms why they're doing it. If they do that, the technical community will be able to evaluate their arguments properly, decide whether they've got any basis at all, and (we're just guessing here) explain to them how to solve their problem correctly and without arbitrarily jamming things.

That way, Comcast might not break the very thing they claim to be selling access to: the Internet.

[Permalink]

Comcast is also Jamming Gnutella (and Lotus Notes?)

Yesterday, we posted about some experiments showing that Comcast is forging packets in order to interfere with its customers' use of BitTorrent. There have been reports of strange things happening with other protocols, and we've been running some tests on two other file transfers protocols in particular — HTTP (which is used by the World Wide Web) and Gnutella. Comcast has also been strenuous in telling us, "we don't target BitTorrent". Perhaps not. Perhaps what they're doing is even worse.

In the limited tests we ran, we didn't see any interference with HTTP traffic. Comcast's network seems to behave correctly when you run a private web server and share a few of your photos or videos over it (we tested files up to about 25MB).

But when you try to run a Gnutella P2P node on your machine, things start getting strange. Gnutella operates in two stages: first of all, your node starts a conversation with other nodes on the network. Once that conversation is happening, nodes can say things to each other to organise searches for and downloads of files. We saw forged TCP reset packets that stop some of the nodes from being able to converse with each other in the first place.

Forged reset packets are normally the kind of thing that would only be present if a hacker was attacking your computer, but in this case, it's the ISP you pay money to each month that is sending them.

Strangely, the packet forgery only occurs when a non-Comcast node is trying to start a conversation with a Comcast customer's Gnutella node. If the Comcast customer starts the conversation, there is no Reset packet. This means that Comcast customers will not see Gnutella fail entirely — the network just doesn't work properly.

It isn't just BitTorrent and Gnutella that are affected. Kevin Kanarski has reported that Lotus Notes (a suite of software that many businesses use for email, calendaring and file sharing) is also being interfered with. We haven't tested this ourselves yet, but Kanarski's packet traces look a lot like the ones we've collected with BitTorrent and Gnutella.

When an ISP starts arbitrarily zapping some of the protocols that its customers use, they instantly endanger the cascade of innovation that the Internet has enabled. Before this kind of traffic jamming, anybody — huge businesses, small start-ups, college students and children in their bedrooms — could build new, innovative protocols on top of the Internet's TCP/IP platform.

If this type of conduct is allowed to continue, many innovators will have to get active assistance from an ISP in order to have their protocols allowed through the ISP's web of spoofing and forgery. Technologies like BitTorrent and Joost, which are used to distribute licensed movies and are in direct competition with Comcast's cable TV services, will be at Comcast's mercy.

It should also be remembered that in many parts of the United States, Comcast is a duopoly or even a monopoly provider of broadband Internet access. Competition might offer some protection against packet-forging ISPs, but under current market conditions, we can't depend on it.

[Permalink]

EFF tests agree with AP: Comcast is forging packets to interfere with user traffic

This morning the Associated Press reported that Comcast is interfering with users' ability to run file sharing applications over its network.

Since we spoke to Comcast last month and understood them to deny that they are doing this, we've been running our own tests.

The results of our tests have agreed with AP's. Comcast is forging TCP RST packets which cause connections to drop (a technique also used by Internet censorship systems in China). These packets cause software at both ends to believe, mistakenly, that the software on the other side doesn't want to continue communicating.

The TCP RST packet forging seems to be protocol-specific: as AP reported, it at least sometimes happens directly in response to specific BitTorrent protocol events. This contradicts Comcast's statement to us that their network management does not target or discriminate against particular protocols. The timing of the injected packets suggests that something on Comcast's network understands the BitTorrent protocol and treats it differently from other protocols.

We confirmed this by trying to download files from Comcast subscribers using BitTorrent. We disabled any firewall or NAT software and connected the machines at both ends directly to the Internet, and ran Wireshark, a packet capture tool. This allows us to see exactly what packets each end sent and exactly what packets each end receives. If ISPs between both points were not forging packets, no packets should have been received by one end that bear the other end's IP address but were not sent by it(*). (This is comparable to recording a telephone conversation at both ends and then comparing the recordings to see whether the phone company sent the conversation through faithfully and unmodified.)

Unfortunately, the resulting packet traces look drastically different from one another: each user routinely receives huge numbers of TCP RST packets that appear to have been sent by the other user. But the packet trace at the other end confirms that these packets were never transmitted; they must have been generated and injected by an ISP along the way.

How do we know that Comcast is responsible? Apart from the large number of reports accusing Comcast, increasingly accompanied by packet traces showing suspicious RST packets, we repeated the experiment with two different Comcast connections (one in San Francisco, and one in Oregon) and saw the RST packets appear in both cases. When the Comcast user in Oregon -- Robb Topolski, a source for the AP story and one of the first to carefully document the RST forging phenomenon -- switched on a VPN connection that caused his communications to be encrypted and routed through a different ISP, the RST packets completely disappeared.

Comcast keeps telling its users that the problems they're seeing are not its fault. It's time for Comcast to come clean about what it's doing and take its users' reports seriously.

(*) Note to network experts: IP fragmentation does allow for the possibility that a single packet sent by one end could arrive in multiple fragments, none of which was originally sent in fragmented form. However, we did not observe any fragmentation at all during our experiments and, in particular, the forged RST packets are clearly not fragments of packets sent by the other end.

[Permalink]

Comcast and BitTorrent

We've been receiving a lot of inquiries from people concerned about recent allegations that Comcast is interfering with its subscribers' use of the BitTorrent protocol, perhaps by using an appliance that disrupts BitTorrent sessions. Some of the people contacting us are Comcast subscribers who've had trouble with BitTorrent recently and think that they might be affected by the same problem.

Debugging network problems can be complicated because of the varieties of versions and configurations of client software and the number of places in the network where problems could occur. Most mysterious network errors aren't intentionally caused by anyone. But some ISPs and universities have experimented with appliances that block or disrupt particular traffic, such as VoIP or P2P file-sharing traffic.

On Wednesday, we spoke with Comcast to try to find out what was going on in this case. Comcast assured us that, while it does do some kinds of network management on its residential network, it isn't deliberately blocking, degrading, interfering with, or discriminating against particular protocols or kinds of traffic. (This is consistent with what Comcast told the press in August when these allegations were widely raised.) The company said that it isn't using network management techniques that are designed to disrupt anyone's use of BitTorrent (or any other application).

If anyone has repeatable indications that something else is going on, please let us know -- we'd be happy to bring it up with Comcast to try to get to the bottom of the situation.

[Permalink]