christmas lights

"Slashdot Effect" on 2005 Christmas Lights Webcam

Shortcuts to stories and stats table and lessons learned.

Intro and Technical Notes

Once again, there are christmas lights at Komar.Org - what a surprise! There was a ton of traffic in 2004 so I figured I'd do a similar inbound traffic analysis for 2005 since it's fairly easy to do. The most noteable difference this year is that unlike the previous 3 years when it was a simulation, it's a 100% real for 2005. 3 webcams (3 more than previous years!) viewing 26,000 Christmas lights, (and 3 inflatables - Santa, Frosty, and ... Frankenstein) all controllable via the Internet using X10 technology ... and raising money for Celiac Disease Research ... an auto-immune disease that my kids have.

For those that don't know, Slashdot is the incredibly popular "News for Nerds" site and can bring a thundering herd of surfers to your site, all in a very short amount of time. I've been on Slashdot 5 times for my holidays lights so I've had a little bit of experience with this. The term can generically apply to other sites that generate a lotta traffic which can result in a DDoS - Distributed Denial of Service. Slashdot paid me a visit in 2002, 2003, and 2004 (3 times), but not in 2005 - ironic since it was for real this year..

While numerous folks have written about the Slashdot Effect, this is a little different in that not only is it a test of "digital" stuff like the web server, ISP bandwidth, Perl/CGI code, but also "analog" stuff as the the 3 ChristmasCams are FTP'ing up images and on/off commands are being sent to the X10 controls at the house. And oh yeah, you are turning a lotta lights ON & OFF - it certainly provided one heck of a light show for the neighbors! ;-)

The webcam page is about 50 KBytes of HTML/CSS that is very infrequently refreshed. There is the infamous Komar-Santa Cursor that is 12 KBytes along with a couple of KBytes of Javascript code. The control panel on the right side of the images meta-refreshes every 60 second and is about 15KBytes along with 5 GIF buttons that total less than another KByte. When you are viewing ALL of the webcams, each of the 352x240 small images is 10-15 KBytes ... they are actually resized on the fly (using the GD library) from the incoming 704x480 JPEG's that the webcams FTP over and are about 40-50 Kbytes. The images are meta-refreshed every 5 seconds. So while there is a lot going on, the file sizes are relative small, so raw bandwidth is probably not the limiting factor. My guess is either I'd run out of RAM or hit an Apache connection limit on the Linux box - neither has come close yet, so I don't know and I have not done benchmarking in this area.

In 2002, there was a single 1 Ghz Pentium box with 128 Mbytes of RAM on a T-1 connection. For 2005, there are FIVE web servers handling the load. They all run Linux/Apache (mod_perl rocks!) and are 3 GHz or so Pentium boxes with a Gbyte of memory - all 100Mbps connected. I.e. there is half a Gigabit of bandwidth available - BRING IT ON SLASHDOT! As noted above, Slashdot didn't show up - maybe there were afraid?!? ;-)

Two servers load balance the christmas movies since those are huge. The third server handles only ChristmasCam #1 and #2. The fourth server handles ChristmasCam #3 and a few misc. images like those buttons. And the fifth server handles the rest of komar.org. I.e. I could add one more server to fully load balance this real-time application, but that would be it.

Special thanx to Dominic at Vidahost who graciousely volunteered to help out. Dominic is a director of Thermal Degree, a UK-based web design and hosting company setup by two friends in early 2004. Vidahost, launched just over one year ago, is their hosting branch which offers shared and reseller hosting. Dominic also handled some mirroring of www.asiantsunamivideos.com, a collection of video footage from the tsunami of December 2004 which attracted thousands of visitors and nearly 1Gbit/sec of traffic. And he is also a member of the World Wind community at www.worldwindcentral.com.

Misc. Sites that mention the Christmas Webcam

Needless to say, the christmas lights webcam is talked about on countless blogs and web sites ... but I picked a few noteable/interesting one to show here. For the first 15 days of December and there were 18,255,169 hits on on Christmas related stuff at komar.org scattered among the 5 servers supporting this effort; i.e. it's averaging a little over a million hits/day.

Number of inbounds per site after being posted

The Apache Web Server generates copious logging data and one can use the raw data to "track" traffic from sites that link to you based on what is called the referrer (which spammers sometimes spoof for referrer log spamming) - I actually use this to do a customized body alert="message" from certain sites, and many folks appreciated that personal touch. It's actually quite easy to see when these sites posted about the Christmas stuff because there is a very noticeable uptick in traffic - all times are Mountain Standard Time (GMT-7) and the numbers below are definately LOW because some browsers block the referrer. In date/time order, we have:
Site - Time Posted 5 Min 10 Min 1 Hour 2 Hours 4 Hours 8 Hours 24 Hours 2 Days Week Month Site - Time Posted
BlogScoped - 28/1033 3 6 50 92 143 194 268 309 334 370 BlogScoped - 28/1033
DIGG - 1/2358 177 346 1,878 2,839 4,145 9.619 14,999 15,469 16,324 16,927 DIGG - 1/2358
MajorGeeks - 5/0714 7 16 123 229 389 643 1,068 1,264 1,408 1,450 MajorGeeks - 5/0714
WashPost - 6/0045 1 1 6 7 27 235 567 593 645 696 WashPost - 6/0045
Rocky - 10/0152 1 1 3 4 14 65 219 309 565 1,288 Rocky - 10/0152
FARK - 12/1631 664 1,361 6,683 10,748 16,381 22,029 31,364 33,804 34,399 34,694 FARK - 12/1631
DIGG - 20/0357 97 225 1,389 3,089 4,724 5,991 13,023 15,205 16,272 16,477 DIGG - 20/0357
Site - Time Posted 5 Min 10 Min 1 Hour 2 Hours 4 Hours 8 Hours 24 Hours 2 Days Week Month Site - Time Posted
Slashdot (2004) 781 1,604 11,699 21,651 35,895 53,720 90,607 94,830 98,054 117,210 Slashdot (2004)



Slashdot Effect Analysis - Misc. Interesting Tidbits/Ideas/Lessons Learned - SO FAR!

The 2005 optimization of the X10, Webcam, and mod_perl code is working well: While the web servers aren't exactly monster machines, they are so far handling the load well. For instance, despite 10,000 DIGG'ers the morning of December 2nd, it never skipped a beat ... and mod_perl just rocks. And I subsequently brought online that second ChristmasCam server.

Is DIGG the "next" Slashdot? 16,000+ inbounds in a day is nothing to sneeze at ... but at least compared to last year, Slashdot (and Fark) was five times that.

For the coders out there - be sure to think about and handle race conditions - they DO happen! If you have never dealt with race conditions, The Slashdot Effect is certainly a good test of your code. A classic (but flawed) approach to file locking is checking to see if a file exists, and if not, creating it ... but that is not an atomic operation - i.e. it's two different steps seperated by a small amount of time - so you better do it right! BTW, I use Perl Cookbook Recipe 7.11 to handle file locking and incrementing of a couple of counters - this has worked flawlessly ... although it did burp in previous years when Slashdot hit.

Is Alek ready for Christmas/2005? Wellllll ... you are never truly "ready" for Slashdot ... but the application is fairly tight and spread over 3 web servers - you could basically only add one more for load balancing this real-time application. So that is a cumulative 9.6 Ghz of computing power and 300 Mbps of bandwidth - holy mackeral! And all of the video's are redirected to two other servers (aggregrate of 200 Mbps) ... so that a lotta bits and bytes! I never got around to implementing Apache proxying which would certainly help ... and my Perl code could certainly be better ... but the error logs are clean.


Want more info and/or have a suggestion/idea for me? The Christmas FAQ has some more info as does my responses when folks Email Santa. And if those don't answer your questions and/or if you have an idea (or just want to send me an atta-boy), then send me an Email.


Go back to the main slashdot effect analysis page.