Skype Journal

UPDATE:

From Julian Cain at 1:33 PM Pacific.

Hi Phil. I’d like to clear up the node to supernode relationship. Each supernode determines it’s client connection list to be full at 350 active tcp connections. This means a supernode is considered at maximum capacity at roughly ~350 client connections however they will and do handle more client connections up to a degree depending on system variables. ATM  there are ~58K supernodes "active" and ~116K "idle". My data during the outage listed clients and supernodes however I truncated the dataset to begin at about ~1.5 MM nodes. The decline is linear with online supernodes and clients and represents a "live" graph of the overlay undergoing "segmentation" due to a failure in the main supernode backbone.

CORRECTIONS:

First, while Cain’s data is correct, I mislabeled it. These are 1.4 million nodes, not supernodes. By 3:46PM Eastern in the US, the Skype network crash had been going on for many hours. So you’re seeing the last leg of the collapse. Cain labeled the data correctly; it was my transcription error.

Second, I should have recognized the error. I was confused by two things. All the other information and videos focused on supernode behavior, so I was myopic. I completely missed out on the time zone differences so I didn’t notice that this data really fit into a different place on the event timeline.

Thanks to several folks at Skype who urged me to dig a bit deeper and to anonymous commenter Chupacabra who wrote “1.4 million supernodes? Thats rubbish!”

My response to Chupacabra:

@Chupacabra, I thought that at first too. And you may be right since nobody at Skype is talking. So let’s do some back of the napkin calculations. In the olden days, say 2005 or so, it was said that a supernode could support between 500 and 1000 active nodes easily. On that fateful day last week, there were about 25 million accounts online when things began to go wrong. Very few people have more than one copy of Skype running, so let’s leave that to rounding error for the moment and say there were 25 million nodes running. So 1000 goes into 25,000,000 twenty-five thousand times. 50K supernodes if we say each supports only 500 nodes. 25 to 50K feel right to me.

Are we dividing into the right population? Skype has about 150 million active users over the course of a few weeks, about six times peak levels. So that would make the number of supernodes about 150K to 300K. Still far short of 1400K.

1.4 million supernodes is not what we’d expect.

So,

- Has the math changed? If Cain’s observations were correct, does a supernode now support, say, just 20 other nodes? That would be a massive drop in efficiency.

- Did Cain observe something else, not a supernode but, perhaps, a node that was capable of becoming a supernode?

- Did Cain only find nodes – not supernodes – that were visible from his vantage point in the network? A large but limited subset?

- Did Cain make a fundamental mistake in data collection and processing?

- Did I report it wrong?

I’d like to test his rig, but his response to a request for an interview:

“Hi, Thanks for the offer but I am not speaking to the press for the foreseeable future. Regards, Julian”

Almost 25 million accounts were online when Skype’s 1.4 million supernodes started leaving the cloud. It took 330 minutes for 98% of the supernodes to go offline, cutting off nearly all Skype users. Researcher Julian Cain set up a UDP probe to look at the Skype network as it crashed last week. At the bottom of the blackout, Cain demonstrated how his reverse engineered Skype client attempted to connect to the network and was rejected, like all the other Skype clients struggling to reconnect.

The chart shows 98% of the Skype supernodes leaving the Skype network over 5.5 hours (data).  Cain puts minimum viability for the p2p mesh at about 75K nodes.

Cain shot a short video after the network crash. He is showing his “fully functioning 3rd party Skype peer-to-peer stack during the global overlay outage.”

10:42 PM Eastern, still unable to connect with Skype. He shows traceroutes to Luxembourg supernodes operated by Skype, mostly pooled within the same IP ranges. Here’s a list of hard-coded Skype IP addresses from earlier this year. He shows his own reverse engineered, third-party implementation of a Skype client. “It’s the first ever.” He uses Skype’s login server as a bootstrap supernode. “Here I perform a UDP probe of all of the supernodes that ship with the Skype binary to check responsiveness.” He makes his own SkyLib (a core Skype messaging component) and pinged the Skype mega-supernode. He receives a NACK (negative acknowledgement) from the Skype supernode and connects to the network. The network drops his TCP/IP connection, just like any other Skype client in the outage.

Skype promised to report on its internal post-mortem this week. Let’s hope we get this level of disclosure.

More from Skype Journal on the outage:

• Skype Outage Coverage: What The Tech (video podcast)
• Quora Questions the 2010 Skype Blackout (20 questions)
• Complaining is your duty, not just a right
• Sample tweets about the outage in our SkypeLifeOnTwitter category
• The Skype Blackout of 2010 is Over (chart)(recap)
• Skype CEO: We apologize and will top you up (apology without the word sorry)
• Wishlist: Enterprise MegaSuperNode Appliances (architecture)
• First picture of a Skype #MegaSuperNode (paparazzi shot)
• 17.5 million: Skype restored dial tone twice as fast as in the 2007 outage (chart)
• First video message from Skype CEO Tony Bates (video, hoodie)
• 16.5 million online, 80% of normal says Skype (chart)
• Outage update: 20% recovery so far; 4.8 million seen online (chart)
• Good Morning, Mr. Bates (let’s talk)
• Skype Outage: Context (everyday is a Skype outage in Oman)
• Skype outage 2010: Bigger than 2007’s 48 hour blackout (150 million people served now, 45 then)
• Rise of the Mega-Supernodes! The Skype Network Outage of 2010 (chart)

What tools did you try when Skype went down? Did you come back to Skype? Chat with me on Skype. Call me at +1-510-343-5664 (Google Voice), follow on twitter @SkypeJournal(just the posts) and @evanwolf (everything). Visit our Skype Journal private technologist roundtable, one of the longest running public Skype chats, where we’re talking about this right now.