ForecastAdvisor Weather Forecast Accuracy Blog
Saturday, March 18, 2006
Is there an NWS web issue?
I love working with meteorologists. They perform such an important role in society. But their jobs are often underappreciated because they have to play so many roles...they are part teacher, part scientist, part artist. I enjoy providing tools that help them, and enjoy their criticism that helps me learn and grow and make the tools that ForecastWatch provides even better.
"Sandy in Arizona" left a comment on this blog. Thank you Sandy...it was the first comment (woohoo!!) :-). I hope you don't mind me talking about it here. If you would like to discuss further, please don't hesitate to contact me at any of the contact points mentioned on the websites.
Sandy made the comment:
I checked Tempe AZ...the NWS digital forecasts ranked number one while the NWS forecast ranked last...there would appear to be something wrong with your methodology. There almost always is with these types of sites.
Instead of asking us why the NWS digital forecast might be ranked differently than the NWS web forecast, Sandy proclaims that there "must be a problem with [our] methodology." I would certainly love to have a discussion about the methodology and uncover if there indeed is a problem. If there is, I would like to fix it. But with such a closed-minded statement like "There almost always is with these types of sites", I don't think there is much opportunity. The comment reeks of prejudice...ForecastWatch is just like all the other sites (BTW, what other sites? Can you send me some links please?) so why bother.
So let's look at Tempe, Arizona and see what the problem is [HINT: It's NOT a problem with the methodology, rather the NWS has a problem I think they need to fix].
The NWS forecasts that are graded for accuracy on ForecastAdvisor are the public forecasts available on weather.gov. The NWS Digital forecasts come from the SOAP interface to the NDFD. Forecasts on weather.gov are queried by zipcode, NDFD by latitude/longitude. ForecastWatch collects forecasts for all AWOS/ASOS observation sites, and maps the nearest/enclosing zipcode to each observation site.
For Tempe, Arizona, the forecast and observation site is actually Phoenix, Arizona. The AWOS/ASOS observing station is KPHX and the mapped zipcode is 85065. Zipcode 85065 encloses the AWOS/ASOS observing station, so for all intents and purposes they are equivalent as far as querying forecasts go.
So let's go to weather.gov and enter a couple of zipcodes. First, let's enter the zipcode of ForecastWatch's office, 43040. It returns a forecast for Marysville, Ohio. Just as expected. How about another random zipcode...68106 for Omaha, Nebraska. Again, perfect, the NWS forecast that shows up is for Omaha, Nebraska. No surprises so far.
Now lets enter the zipcode 85065 on weather.gov. Hmmm...that's odd, it returns a forecast for Quartzsite, Arizona. That's 128 miles from Phoenix. Let's try a nearby zipcode, say Scottsdale, Arizona (85260). Hmmm...Quartzsite again. Quartzsite sure does have a lot of zipcodes. Now let's look at querying from the region page (the Phoenix office website). When you query 85065, you get "not found" (though I can assure you the zipcode does exist). When you query 85260 you get Scottsdale, as expected.
What's happening then is that ForecastWatch is querying the NWS weather.gov site just like a user would, and when it queries zipcode 85065 it gets the forecast for Quartzsite, Arizona, 128 miles away. No wonder the NWS forecast shows as being so bad. It is! Shouldn't querying weather.gov for zipcode 85065 return the forecast for Phoenix, Arizona, not Quartzsite? It appears to me that there is a zipcode mapping problem. Most zipcodes appear to work just fine, but some, like 85065, do not. Can someone from the NWS comment? Thanks!
If "Sandy from Arizona" would have enquired if there was a problem with the methodology, instead of assuming it, unknown problems can be uncovered that may point in unexpected directions, lead to quality improvements, and discover things previously unknown.
I'll comment on Sandy's other comment ("same milk in the best looking bottle") in a future blog post.
Saturday, March 4, 2006
2005 Weather As Seen Through The Internet
Since March of last year, the amount of time it takes the ForecastWatch system to get and process weather forecasts has been recorded. I was doing some troubleshooting on the web weather forecast retrieval system and as part of that troubleshooting I looked at previous retrieval times.
When I created a graph of the retrieval times, there were a few very large, very curious spikes. Since I was looking at the retrieval and processing times for the public, web forecast component, there could be a number of explanations for the spikes.
They could be a result of problems with the network between ForecastWatch.com's computers and the websites of Accuweather, The Weather Channel, MyForecast, the National Weather Service and the like. If one of the websites was undergoing maintenance, or was having problems, that might also account for the up-tick.
The spikes, where it took longer to retrieve the weather forecasts, could also be because the weather websites were busy. If the web sites were very busy, if a lot of people were trying to access, say Accuweather.com, or Weather.com, it could slow things down for others as their computers become overloaded. If that were the case, they most likely became busy because people were interested in some big weather event affecting or going to affect the United States. On a "normal" weather day, you'd expect a "normal" amount of website visitors. But if something big were happening or expected to happen (a major snowfall or hurricane, for example) people who otherwise wouldn't be visiting, or would only be visiting once, would visit many times. Traffic would go up, and response times would go down.
Here is the graph showing the time it took to retrieve all web-based weather forecasts. These are weather forecasts offered to the public by the weather forecasting companies. ForecastWatch.com also receives non-public forecasts by various means, but their retrieval times aren't included here, since they are not on public web sites.
Click here for a larger version of this graph.
Immediately, you notice four huge spikes: one each on 3/11, 9/19-9/22, 10/20, and 12/5-12/7. All four of those dates can be linked to major weather events affecting a large number of people in the United States.
On 3/11/2005, an Alberta clipper dumped significant snow on New England.
From 9/19 through 9/22, Hurricane Rita was menacing the southern United States. First Florida, and then making land fall early on the 24th on the Texas/Louisiana border.
On 10/20, Hurricane Wilma became the strongest Atlantic hurricane ever recorded, and was heading for land fall near Cancun.
Finally, 12/5 through 12/7 there was a major snowstorm from Washington, DC to Boston.
But there are a couple of major weather events that weren't in the top four.
Notably, Hurricane Katrina didn't have the same web server impact as the some of the other major hurricanes. It could be because of when ForecastWatch pulls web forecasts (the evening). Or maybe after Katrina, people became more interested in future hurricanes because they were unfortunately reminded of their deadly power.
December 2005 January 2006 March 2006 June 2006 July 2006 August 2006 September 2006 October 2006 November 2006 December 2006 January 2007 February 2007 March 2009 September 2009 March 2010 April 2010 February 2011 April 2011 June 2011 February 2012 September 2012 Current Posts