The Christmas Lights & Webcam were humming along just fine until 1147 on Saturday, December 20th, 2003. At that minute, traffic increased a LOT; an article had been posted on Slashdot (http://slashdot.org/) that had pointers to the Christmas stuff. Slashdot bills itself as "News for Geeks" ... so imagine thousands of Geeks hitting your web site, all at the same time! ;-) This Slashdot Effect can bring the computer/ISP to "its knees" as it is basically a DDOS - Distributed Denial of Service ... which is what happened in Xmas/2002 - not only did the web server melt down, but a 40 amp circuit breaker providing power to the server popped on Christmas Day - yes, this really did happen!
While numerous folks have written about the Slashdot Effect, this was a little different in that not only was it a test of "digital" stuff like the web server, ISP bandwidth, Perl/CGI code, but also "analog" stuff as the code has interfaces to various sensors and the webcam itself, plus you are turning a lotta lights ON & OFF - it certainly provided one heck of a light show for the neighbors! ;-)
It should be noted that the "digital" stuff has extensive logging, whereas the "analog" stuff does not - you have to infer problems/breakdowns due to heavy load indirectly. And the real-world analog stuff just isn't that fast, nor as "reliable", so I had coded things intentionally with a 5-second throttle with the hope that the analog side would not get overloaded - I never did pop a circuit breaker!
Here is the actual Slashdot article and comments - as typical, the signal/noise ratio in the comments are not always that high/germane. One thing that complicates the Slashdot Effect Analysis is that there were 6 www.komar.org URL's referenced in the posting, with one of these being the top-level www.komar.org ... and all of the analysis below is ONLY the Xmas related stuff - rest assured the rest of the website got hammered pretty well too! The peak hourly rate for the entire web site was 62,924 hits between 1200-1300 ... a LOT more than I typically see, but certainly not bazillions of surfers/hits, like say the Mars lander. From a bandwidth point of view is I don't have any files/images over a MByte, and the vast majority are less than 100 KBytes.
The "good" news about the story hitting at 1147 is that the webcam was not active until 1700, although the software does provide a variety of analog data (mostly weather related) via various X10 sensors. A few spot checks of these showed they all continued to provide reasonable data. Things got a little bit more interesting at 1700 when the webcam (and webcontrol - i.e. turn the lights ON & OFF) became operational ... ;-)
BTW, another "big blog/news" site is Fark ... story never made the
front page of that, but it might have been interesting to see what this
coulda done ... i.e. who is more "powerful" - Slashdot or Fark?
Maybe that question will be answered at Xmas/2004!
if ( -e "$throttle_file") {
$throttled = "YES";
} else {
open(FILE_THROTTLE, ">$throttle_file") || warn "Can't open $throttle_file for write: $!";
print FILE_THROTTLE "$browser{'ip'}\n";
close(FILE_THROTTLE);
$throttled = "NO";
}
But there is an ever so slight window of opportunity where the second person
can check for the existance of that $throttle_file BEFORE it gets created.
flock is your friend - here's the actual snippet of Perl code from the
xmas_webcam:
if ( -e "$throttle_file") {
$throttled = "YES";
} else {
open(LOCK_THROTTLE, "$dir_locks/throttle") || warn "Can't open $dir_locks/throttle for write: $!";
if (flock(LOCK_THROTTLE, 2|4)) { # Exclusive write, non-blocking
open(FILE_THROTTLE, ">$throttle_file") || warn "Can't open $throttle_file for write: $!";
print FILE_THROTTLE "$browser{'ip'}\n";
close(FILE_THROTTLE);
$throttled = "NO";
} else {
print STDERR "xmas $program actually had an flock fail with $throttle_file at $date_time ...\n";
$throttled = "NO";
}
}
Note the non-blocking flock ... unlike say, an impending database write, there
isn't anything compelling and we don't want to block, so if it fails, we just
go ahead and say you are throttled. This actually happened 9 times - those were:
2003_12_15_12:44:55
2003_12_20_12:16:00
2003_12_20_16:56:47
2003_12_20_19:01:00
2003_12_20_19:30:01
2003_12_20_20:13:10
2003_12_20_20:41:00
2003_12_20_23:57:01
2003_12_21_22:29:18
Before anyone asks, I did NOT use the Perl HiRes Module to get sub-second
accuracy - didn't feel it was neccessary and didn't want to have the overhead
of pulling it in ... although note that the webcam timestamps the images
in hundredth of a second. One final thought - the peak rate for JUST the
CGI perl script was 11 times/second;
this is pretty darn fast for my little 'ol web server that I maintain as a
hobby (note that the total web server hit rate was probably about 10 times this)
- this is probably "slow" for some of the "big boys" who also have to
handle much more complicated transactions and really insure they are
done correct - think about that next time you are booking an airline
ticket online.
Want more info and/or have a suggestion/idea for me?
The Christmas FAQ has some
more info as does my responses when folks
Email Santa. And if
those don't answer your questions and/or if you have an idea (or just
want to send me an atta-boy), then
use this page to send me an Email.
Here is a summary of the Analog data from the Apache Web Server Logs:
388,851 Total Christmas related Web hits
135,943 Total Christmas related Page Views
6,680,000 KBytes of Christmas related text and images transferred
67,871 Hits on the Christmas Webcam
16,069 Hits on the xmas & christmas lights main page
20,864 Referrals from Slashdot
1,461 Referrals from Google
1,006 Referrals from www.uglychristmaslights.com - who said my lights were ugly?!? ;-)
865 Referrals from www.weirdlinks.com - and they are not ... too ... weird either!
Here is the data as recorded from the Christmas Webcam itself (webcam controllable from 1700-2400):
67,579 Total christmas webcam accesses (darn good correlation with Apache log data)
38,929 Most webcam accesses in a single day - December 20th (After Slashdot)
1,425 Most webcam accesses in a single day - December 03rd (Before Slashdot)
7,567 Most webcam accesses in a single hour - 1700-1800 on December 20th (After Slashdot)
274 Most webcam accesses in a single hour - 2300-2400 on December 13th (Before Slashdot)
238 Most webcam accesses in a single minute - 1702 on December 20th (After Slashdot)
14 Most webcam accesses in a single minute - 2242 on December 13th (Before Slashdot)
11 Most webcam accesses in a single SECOND - 1700:08 on December 20th (Slashdot'ers were waiting! ;-)
17,662 Number of time webcam was accessed within the 5 second throttle by more than one web surfer
21,801 Different/Unique IP addresses that accessed the webcam (2,068 were unresolveable)
184 Non-proxy hostname that hit it the most - m084-217.nv.iinet.net.au - this guy must like Christmas Lights! ;-)
83 Total different countries that accessed webcam - had very cool geolocation stuff working
48,214 Total hits from United States
5,373 2nd most popular country - Canada
2,328 3rd most popular country - Australia
7,919 Total "ACTION" requests to turn some/all lights ON/OFF
533 Turn ALL lights OFF (most popular change)
398 Turn ALL lights ON
8,720 Total "ACTION" requests to pan/zoom webcam
362 Pan Right (most popular change)
229 Pan Left
1,134 Move webcam to one of the "targets"
380 Most popular target - "Burnt Out!"
14,025 Different/unique images served up by xmas_webcam (some images are cached/re-used)
Go back to the main slashdot effect analysis page.