christmas lights

Slashdot Effect on 2004 Halloween Decorations and Webcam

I once again put out my halloween decorations in 2004 which ended up getting pounded as numerous web sites picked up on it, including Slashdot, which is the incredibly popular "News for Nerds" site and can bring a thundering herd of surfers to your site, all in a very short amount of time. I've written about the slashdot effect before (I was /.'ed in Christmas 2002 & 2003), and wondered how it compared to FARK, which is another heavily trafficked site ... wellll ... for Halloween/2004, I found out! ;-)

The Web Server (a 2.4 GHz Xeon with a GByte of RAM running pretty standard Linux with the Apache web server) generates copious logging data and one can use the raw data to "track" traffic from sites that link to you based on what is called the referrer (which spammers sometimes spoof for referrer log spamming) - I actually use this to do a customized body alert="message" from certain sites, so some folks were amazed I "knew" about it so soon! ;-) It's actually quite easy to see when these sites posted about the Halloween stuff because there is a very noticeable uptick in traffic - all times are Mountain Day Light Time (GMT-6) and the numbers below are definately LOW because some browsers block the referrer.

BTW, honorable mention goes to Kim Komando which picked it as their Kool Site on Friday, October 29th, 2004 and emailed her listeners ... but since there is no referrer, this is hard to track, but it was a bit of traffic ... plus she took my call on Saturday and we chatted briefly on the radio. And another honorable mention to the Earthcam folks as it was listed as one of their Spooky Webcams which also generated a constant stream of traffic. The the various search engines generated a bit of inbound traffic as did these funny guys from the Inquirer.


Number of inbounds per site after being posted

Site - Time Posted 5 Min 10 Min 1 Hour 2 Hours 4 Hours 8 Hours 24 Hours 2 Days Week Site - Time Posted
Slashdot - 1739 691 1,528 8,335 15,645 29,102 42,698 67,656 68,894 69,397 Slashdot - 1739
Ernie - 1104 47 104 674 1,224 2,446 4,701 8,842 12,892 19,164 Ernie - 1104
FARK - 1148 531 1,133 5,251 7,979 10,716 12,904 15,451 16,384 16,900 FARK - 1148
MajorGeeks - 1359 6 17 78 146 276 385 822 1,443 3,361 MajorGeeks - 1359
USA Today - 1216 4 9 75 175 274 395 838 868 924 USA Today - 1216

Some General "Slashdot Effect" Commentary

While numerous folks have written about the Slashdot Effect, this was a little different in that not only was it a test of "digital" stuff like the web server, ISP bandwidth, Perl/CGI code, but also "analog" stuff as the code has interfaces to various sensors and the webcam itself, plus you are turning a lotta lights ON & OFF - it certainly provided one heck of a light show for the neighbors! ;-)

It should be noted that the "digital" stuff has extensive logging (as did the the webcam daemon) but "analog" stuff does not. So I don't "know" if something goes wrong (for instance, sometimes the images just come back "bad") and you have to infer problems/breakdowns due to heavy load indirectly. Plus real-world analog stuff just isn't that fast, nor as "reliable", so I had coded things intentionally with a 1-second throttle with the hope that the analog side would not get overloaded - Fark "burned out" an X10 super socket, but I never did pop a circuit breaker!

The webcam page is about 20 KBytes of HTML/CSS and the webcam image varies between 10-20 KBytes ... so from a bandwidth point of view, it's a pretty lightweight page - hey, I designed it so even dial-up'ers can use it! While I missed the first hour or so of the Slashdotting (the most intense time), the server itself was fairly responsive to interactive ssh sessions and you can read more in the play-by-play that night. However, the web server was pretty sluggish ... and what "hurts" is that the webcam code is several thousand lines of Perl code (plus it slurps in the Perl Modules Benchmark (I shoulda pulled it for production), CGI::Carp, Fcntl, IPfree, Socket, and Time::HiRes) ... and I am NOT using mod_perl (one of these days!) ... so that's a lotta startup work for the web server/perl interpreter ... and then it runs my code ... which actually gets done fairly quickly - it uses signals to communicate with the "webcam daemon", so there's a sleep loop where it waits, but that isn't CPU intensive. I also have a flag to disable real-time DNS lookups and I probably should have set that, as those can take time.


Slashdot Effect Analysis - Misc. Interesting Tidbits/Ideas/Lessons Learned

The 2004 optimization of the X10, Webcam, and other "analog" devices worked well. In 2003, I had a 5-second throttle, but I looked closely at the various operations/commands and multi-threaded it as much as possible. One easy "trick" was that the weather data is actually from the previous query (it doesn't change that fast), so I just grab that data and queue up a re-request for the next person. If the webcam (this year's non-disclosure model is much quicker) is overloaded when handling someone else (signals are blocked during this time), I just grab the previous picture and say sorry, charlie. Similar optimizations were done for the other sensors, but again, these are all farmed off in parrellel rather than doing sequentially. In order to cycle the lights on & off faster than one/second, I'll have to use a different technology than X10 since it is relatively slow. BTW, several folks have asked when I'm going to stream video and I'll consider that when someone pays for an OC-3 line to my house and my web server at the ISP! ;-)

Which is "stronger" - Slashdot or Fark? Slashdot SMASHES Fark (as The Incredible HULK would say) ... no doubt about that. And the interesting thing that popped up from all this was Ernie is a serious player! ;-)

Insights into the Slashdot subscription program. While Slashdot is a "free" web site, you can pay for a subscription which also allows you to see stories earlier - i.e. beat the crowd and see into the "mysterious future" as they call it. The first referral was at 1719:24 (67.101.217.186 ), and there was a total of 68 inbound referrals from 47 unique IP's until it hit the publically viewable page at 1739:07 and the thundering herd showed up! ;-) Ironically, the Slashdot subscription FAQ says about 10-20 minutes early ... this was (basically) exactly 20 minutes.

Insights into the Fark subscription program - Total Fark. Fark also offers a premium "pay" service which among other things, allows you to see ALL submittals, not just those that are approved. The first referral from TotalFark was at 1035:56 (12.14.172.51) and there were a total of 134 inbounds from 122 unique IP's until it hit the main page at 1148:15. I don't know if that is "fast" or "slow" in terms of getting approved; but I would not be surprised if discussion among TotalFark'ers has some bearing on if Drew approves a story (?)

How many FARK/Slashdot/Ernie/etc. readers are there? The number of unique IP's was 70-80% of the numbers above (surprisingly consistant) which is a ballpark figure for number of readers - yea, I know there are proxy, DHCP, etc. issues. As mentioned, it's hard to say how popular the halloween webcam is for the referring web sites readers, so that certainly influences the numbers. It should be repeated again that Ernie "buried" the link in an article about baseball whereas the others clearly talked about what it was.

Alek had Google Adsense running - did I "clean up" from all the Slashdot traffic? Nobody asked me this year, but last year, one person asked me if I made a much money from the Google Adsense ads that are on the site. Unfortunately, the Terms of Service do not allow you to disclose (basically) ANY information. But you are only paid on click-thru's, not ad impressions. So while the later went up by (a LOT!), the click-thru percentage (no surprise) plummented ... and halloween lights/decorations/etc. aren't exactly high-value keywords! So lets just say that thanx to Slashdot in Christmas/2003 last year, the net "dollar delta" allowed me to take my wife out to a not-very fancy dinner, but probably not a movie. And thanx to ALL of the web sites above, this year, we went to a nicer place, ordered dessert, and caught that movie. Adsense is a cool program (those Google guys are smart), and while I'm sure others are making lotsa money with it, I do it to basically just pay the ISP bills and just general playing around.

For the coders out there - be sure to think about and handle race conditions - they DO happen! If you have never dealt with race conditions, The Slashdot Effect is certainly a good test of your code. A classic (but flawed) approach to file locking is checking to see if a file exists, and if not, creating it ... but that is not an atomic operation - i.e. it's two different steps seperated by a small amount of time - read more in my Christmas 2003 Slashdot Effect writeup. So I was using Perl Cookbook recipe 7.11 for file locking ... and there was a total of 683 times that the non-blocking attempt to lock the file failed ... so I would sleep a random amount of time from 0.1-0.2 seconds, and try 4 more times ... and even after all that, there were a total of 5 times when it failed to acquire a lock. The counters blew up one time, but I think that was a coding boo-boo on my part and I uncovered a few other corner cases - it was good that FARK hit me before Slashdot because the later is just a punishing test!

Would it be possible to get "early notification" of an incoming Slashdot? I don't log/track the referrer in real-time with the CGI Perl script, but this would not be difficult to add ... and then it's simply a matter of looking for the incoming "slashdot.org/" (main page) ... but as noted above, this would only give me 20 minutes notice. For Fark, you would look for incoming "totalfark.com" which would signal that it has been submitted, but I don't think there's any early indicator of when it has been approved except they all show up at once. Note that if I tracked this, it would be fairly easy to get/display "rate" information - i.e. 1,528 /.'ers in the last 10 minutes.

Is Alek ready for Christmas/2004? Wellllll ... you are never truly "ready" for Slashdot ... but I did "tighten" up the code a bit so it should handle things better. Should be the same 2.4 GHz Xeon ... and still no mod_perl. I haven't decided how many lights to put out for Christmas yet - I did 22,000 (which was almost 80 amps) in 2002 which is considerably more than the 3,700 lights (and 15 amps) that were out for Halloween, so I could be dimming the neighbors lights a bit! Be sure to come by and check it out in December/2004 - here is the main christmas lights page and/or you can just jump to the christmas webcam!


Want more info and/or have a suggestion/idea for me? The Christmas FAQ has some more info as does my responses when folks Email Santa. And if those don't answer your questions and/or if you have an idea (or just want to send me an atta-boy), then send me an Email.


Go back to the main slashdot effect analysis page.