I caught up with a few senior executives at Comcast this afternoon who talked with me about the growing controversy surrounding the company's policy regarding P2P traffic. Right off the bat, they pointed to a follow-up article by Peter Svensson, the AP reporter who last week propelled Comcast's purported blockage of P2P uploads into the national spotlight.
For those of you who don't know, Svensson decided to test for himself whether isolated reports that Comcast is blocking BitTorrent uploads are true. He found that in three of four tries, his efforts to upload the Bible using BitTorrent's file-sharing application were indeed halted by Comcast's network. Following his original article, a storm of outrage erupted, sparking renewed calls for network neutrality regulations.
Svensson's follow-up article, however, retreats from the notion that Comcast "blocks" P2P uploading. Instead, Comcast "delays" P2P uploads, Svensson now writes, a nuance that the Comcast executives believe is critical. During heavy congestion, Comcast slows down P2P uploads by postponing the transaction. The system will, however, repeatedly keep trying to complete the upload until it has been completed.
Svensson writes:
The explanation is not inconsistent with the AP's tests. In one case, a BitTorrent file transfer was squelched, apparently by messages generated by Comcast, only to start 10 minutes later. Other tests were called off after around 5 minutes, while the transfers were still stifled.
Comcast is very pleased that Svensson admits that there really wasn't a blockage per se, although disagreements will surely crop up as to the definitional distinctions between a stop-and-start delay and a full-fledged blockage.
Whatever the case may be, Comcast is adamant that it has to somehow control the flow of P2P traffic on its network; otherwise, congestion and network failures will occur if the swarms consume too much capacity. "When P2P traffic comes at the point of degrading other services...we can't let those services be degraded because a small group uses a disproportionate share of bandwidth," one executive told me.
The small group in question are "seeders," people who spend a lot of time uploading content, but not a lot of time downloading content. As is widely understood by now, Comcast is very specifically focused on this group of users. Reports suggest that Comcast doesn't really mess with folks who download a lot of files (unless they do it excessively, which is another story).
Svensson experienced problems because he was uploading. If, however, he had been downloading at the same time, he likely wouldn't have experienced delays or delays to the same extent. In other words, users can download content or they can download and upload content simultaneously. The delays really kick in when the user is engaged solely in uploading content.
When I asked the Comcast folks why seeders are hit with the worst delays when, theoretically, more bandwidth is used when someone is both uploading and downloading content, they were "uncomfortable" talking about it, presumably because it might provide a roadmap for certain kinds of users to step-up their use of P2P. Speaking hypothetically, however, one executive said "we're worried about someone using an exceptional amount of bandwidth to feed the world's information."
Comcast made clear that its management of P2P is content-neutral. "This is completely content-agnostic. It's generic and fair based on contention [how many people are using the network at once]," one executive said.
I asked if Comcast is focused on uploaders because Hollywood is pressuring ISPs to do their part to nip content piracy in the bud. Nope, the executives said. It's strictly a bandwidth management issue.
I raised the issue of "reset packets" that contain spoofed IP addresses. These reset packets are like error messages and are what actually cause the P2P upload to stop and then delay. "We are working within the constraints of P2P protocol," one executive said. "The only way today that’s possible [slowing down P2P transmissions], is to work within the parameters of TCP" which dictate the use of reset packets. (As an aside, Princeton's Ed Felten suggests that using reset packets for this purpose is just bad for the Internet.)
Finally, I asked about the reported blocking by Comcast of Lotus Notes. On this point they were extremely apologetic. "It was absolutely unintentional and we had a bug in our software and it was quickly repaired."
After undergoing days of widespread drubbing for not explaining themselves very well, Comcast was very forthcoming with me and seemed sincerely interested in clearing up this mattter. It may be a little too late to yank back the hotheads for a calmer discussion of these issues, but at least Comcast is now trying to explain itself.
Cynthia Brumfield at 3:35 PM|Comments(3)
Comcast is BSing you. Here's how it works:
Comcast's network is broken up into local nodes that contain blocks of IPs. There is a Sandvine box at each node. Once general traffic is at a certain level within the node, the Sandvine box looks for IPs within that node that have a lot of connections and then spoofs the IP and sends TCP RST commands to both ends of each connection above a certain threshold. So if you're throttled by the Sandvine you can't connect to more than say, 10 IPs (I don't know that the actual number is, I think it's configurable.)
They "fixed" the Lotus Notes bug by adding a Lotus Notes profile to the DPI engine in the Sandvine boxes in a patch.
Now what about encrypted traffic? Well, you're screwed. If you're unwilling to allow Comcast to do DPI on your packets it will be throttled by the Sandvine box. This includes SSL, VPN, etc.
Needless to say, this is a TREMENDOUSLY bad idea that will not work. Right now, the quick and dirty solution is to shim your IP stack to ignore TCP reset packets on your BT port. This has the effect of creating tons of dead connections and generating even more traffic on Comcast's network. As we speak, Bittorent protocol developers are making a UDP version of the protocol, which will have it's own encrypted flow control mechanisms.
The solution for Comcast is to set up reflection servers that will prioritize BT traffic within Comcast's network to minimize the impact on their upstream pipes. Y'know, REAL traffic shaping.
Posted by: Robert Smith at December 26, 2007 11:40 PM
Very interesting.
I work at Comcast. I have an IT background and I heard from a network security associate of mine that he'd witnessed and was aware of this activity a few months ago. I'm not employed in a technical capacity there so I've been unable to test or investigate this claim.
Then NPR reported on Peter Svensson's AP story yesterday and I've been following along ever since.
Let me start by saying that Comcast is truly an excellent company to work for. I'm an entry level employee and I'm putting myself through school thanks to Comcast's generosity. I feel truly lucky to have found my job. It's a great place to work and being there has allowed me to pull myself from a fairly bleak place I'd by bad chance landslid into. The company's corporate leadership is smart and progressive and promulgates an interested customer/employee-centric ethos.
My initial reaction, if the traffic's spoofed, was: How does this guy know Comcast's spoofing the traffic? Thanks for allowing me to make such a misplaced assessment, NPR and your luddite recapitulation that bordered on describing electricity as being made of glowing wizards Macarenaing down the wires.
The Macarena will never leave us.
I think the type of bandwidth-leaching activity they're concerned about could be managed in a more savory way. Is IP spoofing dishonest? That's a question that could and should be avoided. But I think it's great that Comcast has entered a dialogue on this. It doesn't sound like what's going on is as evil or pernicious to good internet principles as we were first lead to believe.
Posted by: Matt Beringer at October 28, 2007 10:12 PM
While we're on the subject of p2p, I think the wave of the future is private p2p between friends. It's safe, it's 100% legal and it's more fun to swap stuff with people you know. A great application is GigaTribe: http://www.gigatribe.com
Posted by: John at October 26, 2007 6:45 AM
Blogs We Read
