Sunday, July 01, 2012

How one extra second took down several of the biggest sites on the Internet

One second may not seem like much time to you, but for some websites, it was an eternity on Saturday night. The addition of a "leap second" to the Coordinated Universal Time at midnight Greenwich Mean Time wreaked havoc on a number of websites, including some pretty familiar names.

To put GMT into perspective, PDT - where the Pacific Region is currently - is GMT - 7 hours. Therefore, midnight would be 5 p.m. in the afternoon in the PDT time zone.

However, on Saturday night, many sites were affected adversely by the leap second, as the software platform they were running on, such as the Linux OS or the Java application platform, were unable to cope with the extra second. Sites affected by the second included all of Gawker Media's properties, 4chan, Fark, LinkedIn, Mozilla, Reddit, Stumbleupon, Yelp, and the Pirate Bay (the latter must have made the RIAA and MPAA happy, though), although that list is not complete.

Mozilla's Eric Ziegenhorn wrote a bug update entitled "Java is choking on leap second," in which he said, "Servers running java apps such as Hadoop and ElasticSearch and java doesn't appear to be working. We believe this is related to the leap second happening tonight becuase it happened at midnight GMT."

Reddit Tweeted the following: "We are having some Java/Cassandra issues related to the leap second at 5pm PST. We're working as quickly as we can to restore service."

It wasn't as though the leap second wasn't telegraphed. These sites were - or should have been - aware of it. Ironically, Gawker Media had posted a story entitled, “What Are You Going to Do With Your Extra Second?”

Not everyone was caught off-guard. Google site reliability engineer Christopher Pascoe wrote the following in a 2011 blog post: "The solution we came up with came to be known as the 'leap smear.' We modified our internal NTP servers to gradually add a couple of milliseconds to every update, varying over a time window before the moment when the leap second actually happens. This meant that when it became time to add an extra second at midnight, our clocks had already taken this into account, by skewing the time over the course of the day."

A leap second is occasionally added to Coordinated Universal Time (UTC) in order to keep its time of day synchronized with mean solar time. Without the occasional - and irregular - addition of a leap second, atomic clocks and actual mean solar time would drift apart.

10% Off Carrot Ink Products Today!

No comments: