Since a blog presents stories in reverse chronological order, those new coming into the blog will find my most recent stories first, even though they are intended to be later chapters. So, here is a chronological table of contents of the Experimental Warning Thoughts. I’ll update this and bump it to the top every once in a while.

Introduction

Warning Verification Pitfalls:

Part 1: Getting started

Part 2: 2x2x2!

Part 3: Let’s go fishing

Geospatial Verification Technique: Getting on the Grid

Creating Verification Numbers:

Part 1: The “Grid Point” Method

Part 2: Grid point scores for one event

Part 3: The “Truth Event” method

Part 4: Truth Event scores for one event

Part 5: Distribution of “Truth Event” statistics

The Benefits of Geospatial Warning Verification

Examining warning practices for QLCS tornadoes

Limitations of WarnGen polygons:

Part 1: Our storm escaped!

Part 2: Slide to the right

Part 3: But my threat area isn’t a 20 km square

Precise Threat Area Identification and Tracking:

Part 1: Let’s get digital

Part 2: Threats-In-Motion (TIM)

Part 3: How good can TIM be?

Warning Verification Pitfalls (continued):

Part 4: Inflating Probability Of Detection

Part 5: Inflating Lead Time

As finally promised as an “aside” in this blog entry, I will cover the issue of how using point observations can lead to a misrepresentation of the lead time of a warning.

Consider that one warning is issued, and a single severe weather report is received for that warning.  We have a POD = 1 (one report is warning, zero are unwarned), and an FAR = 0 (one warning is verified, zero warnings are false).  Nice!

How do we compute the lead time for this warning?  Presently, this is done by simply subtracting the warning issuance time from the time of the first report on the storm.  From this earlier blog post:

t_{warningBegins} = time that the warning begins

t_warningEnds= time that the warning ends

t_obsBegins = time that the observation begins

t_obsEnds= time that the observation ends

LEAD TIME (lt): t_obsBegins – t_{warningBegins} [HIT events only]

Let’s look at our scenario from the previous blog post:

For ease of illustration, I’m using the spatial scale to represent the time scale.  The warning begins at some time t_{warningBegins}, and the report is at a later time t_obsBegins.  The lead time is shown spatially in the figure, and in this case, it appears that the warning was issued with some appreciable lead time before the event at the reporting location occurs.

However, as we explained in the previous blog post, reports only represent a single sample of the severe weather event in space in time.  How can we be certain that the report above represents the location and time of the very first instance that the storm became severe?  In all but probably rare cases, it does not, and the storm became severe at some time prior to the time of that report.  This tells us that for pretty much every warning (hail and wind events at least), the computed lead times are erroneously too large!  Reality looks more like this:

ADDENDUM (1/10/2013):  Here is another way to view this so that the timeline of events is better illustrated.  In this next example, a warning is issued at t=0 minutes on a storm that is not yet severe, but expected to become severe in the next 10-20 minutes, hence hopefully providing that amount of lead time.  Let’s assume that the red contour in the storm indicates the area over which hail >1″ is falling, and when red appears, the storm is officially severe.  As the storm moves east, I’ve “accumulated” the severe hail locations into a hail swath (much like the NSSL Hail Swath algorithm works using multiple-radar/multiple-sensor data).  Only two storm reports were received on this storm, one at t=25 minutes after the warning was issued, and another at t=35 minutes.  That means this warning verified (was not a false alarm), and both reports were warned (two hits, no misses).  The lead times for each report were 25 and 35 minutes respectively, but official warning verification uses lead time to the first report known as the initial lead time.  Therefore, the lead time recorded for this warning would be 25 minutes, which is very respectable.   However, in this case, the storm was actually severe starting at t=10 minutes.  The lead time between the start of the warning and the start of severe weather was 15 minutes shorter than that officially recorded.

How can we be more certain of the actual lead times of our warnings?  By either gathering more reports on the storm (which isn’t always entirely feasible, although that may be improving with new weather crowdsourcing apps like mPING), or using proxy verification based on a combination of remotely-sensed data (like radar data) and actual reports.  Again, more on this later…

Greg Stumpf, CIMMS and NWS/MDL

I’m back after a too-lengthy absence from this blog.  I’ve been thinking about some experimental warning issues again lately, and have a few things to add to the blog regarding some more pitfalls of our current warning verification methodology.  I hinted on these in past posts, but would like to expand upon them.

Have you ever been amazed that some especially noteworthy severe weather days can produce record numbers of storm reports?  Let’s take this day for example, 1 July 2012:

Wow!  A whopping 522 wind reports and 212 hail reports.  That must have been an exceptionally-bad severe weather day.  (It actually was the day of the big Ohio Valley to East Coast derecho from last July, a very impactful event).

But what makes a storm report?  Somebody calls in, or uses some kind of software (e.g., Spotter Network), to report that the winds were X mph or the hail was Y inches in diameter from some location and at some time from within the severe thunderstorm.  But the severe weather event is actually impacting an area surrounding the location from which the report was generated, and has been and will occur over the time interval representing the lifetime of the storm.  It is highly unlikely that a hail report represented only a single stone falling at that location, or that the wind report represented a single wind gust local to that single location, and there were no other severe wind gusts anywhere else nor at any other time during the storm.  Each of these reports represent only a single sample of an event that covers a two-dimensional space over a time period.

If you recall from this blog entry, the official Probability Of Detection (POD) is computed to be the number of reports that were within warning polygons over the total number of reports (inside and outside polygons).  It’s easy to see that to effectively improve a office’s overall POD for a time period (e.g., one year), they only need to increase the number of reports that are covered by the warning polygons issued by that office during that time period.  One way to do this is to cast a wide net, and issue larger and longer-duration warning polygons.   But another way to artificially improve POD is to simply increase the number of reports within storms via aggressive report gathering.  Let’s consider a severe weather event like this one:

Look at all those (presumably) severe-sized hail stones.  We can make a report on each one, at each time they fell.  After about an hour of counting and collecting (before they all melted), this observer found 5,462 hail stones that were greater than 1″ in diameter.  Beautiful – the Probability Of Detection is going to go way up!  We can also count all the damaged trees as well to add hundreds of wind reports.  Do you see the problem here?  Are you getting tired of my extrapolations to infinity?  Yes, there are literally an infinite number of severe weather reports that can be gleaned from this event (technically, there is a finite number of severe-size hail stones fell in this storm, but who’s really counting that gigantic number?).  But let’s scale this back.  Here’s a scenario in which a particular warning is verified two different ways:

Each warning polygon verifies, so no false alarms.  For the scenario on the top, there is one hit added to all reports for the time period (maybe a year’s worth of warning), but for the bottom scenario, there are seven hits added to the statistics.

But wait, doesn’t the NWS Verification Branch filter storm reports that are in close proximity in space and time when computing warning statistics?  Wouldn’t those seven hits be reduced to a smaller amount?  They use a filter of 10 miles and 15 minutes to avoid my hypothetical over-reporting scenario.  But that really doesn’t address the issue entirely.  One can still try to fill every 10 mile and 15 minute window with a hail or wind report in order to maximize their POD.  But if you think about it, that’s not really a bad idea.  In essence, you are filling a grid with a 10 mile and 15 minute resolution with as much information known about the storm as possible.  But this works only if you also call into every 10-miles/15-minute grid point inside and outside every storm.  Forecasters rarely do this (and realistically can’t), because of workload issues, and because only one report within a warning polygon is all that is needed to avoid that warning from being labelled a false alarm (again, cast the wide net so that one can increase their chance of getting a report within the warning).

CORRECTION (1/10/2013):  I just learned that the 10 mile / 15 minute filtering was only done in the era of county-based warning verification, and is not done for storm-based verification.  Therefore, my arguments against the current verification methodology where hit rates and POD can be stacked by gathering more storm reports is further bolstered.  More information is in the NWS Directive on forecast and warning verification.

If we knew exactly what was happening within the storm at all times and locations at every grid point (in our case, every 1 km and 1 minute), we’d have a very robust verification grid to use for the geospatial warning verification methodology.  But we really don’t know exactly what’s is happening everywhere all the time because it is nearly impossible to collect all those data points.  The Severe Hazards Analysis and Verification Experiment (SHAVE) is attempting to improve on the report density in time and space.  But their resources are also finite, and they don’t have the staffing to call into every thunderstorm.  Their high-resolution data set is very useful, but limited to only the storms they’ve called.  What could we do to broaden the report database so that we have a better idea of the full scope of the impact of every storm?  One concept is proxy verification, in which some other remotely-sensed method is used to make a reasonable approximation of the coverage of severe weather within a storm, like so:

This set of verification data will have a degree of uncertainty associated with it, but the probability of the event isn’t zero, and is thus, useful.  It is also very amenable to the geospatial verification methodology already introduced in this blog series.  More on this later…

Greg Stumpf, CIMMS and NWS/MDL