{"id":2578,"date":"2011-10-10T17:13:08","date_gmt":"2011-10-10T22:13:08","guid":{"rendered":"https:\/\/hwt.nssl.noaa.gov\/ewp\/internal\/blog\/?p=2578"},"modified":"2017-06-23T16:42:24","modified_gmt":"2017-06-23T21:42:24","slug":"warning-verification-pitfalls-explained-part-1","status":"publish","type":"post","link":"https:\/\/inside.nssl.noaa.gov\/ewp\/2011\/10\/10\/warning-verification-pitfalls-explained-part-1\/","title":{"rendered":"Warning Verification Pitfalls Explained – Part 1: Getting started"},"content":{"rendered":"

Several years ago (in 2005), we hosted our first user group workshop on severe weather technology for NWS warning decision making.\u00a0 This workshop was held at NWS headquarters in Silver Spring, MD, and was attended by NWS management, severe weather researchers, technology specialists, and a “user group” of meteorologists from the field – local and regional NWS offices.\u00a0 The main objective of this workshop, and the second one that followed in 2007, was to review the current \u201cstate of the science and technology\u201d of NWS severe weather warning assistance tools, to identify gaps in the present methodologies and technologies, to gain expert feedback from the field (including \u201cstories\u201d from the front lines), to discuss the near-term and long-term future trends in R&D, and for field forecasters and R&D scientists to help pave the direction for new technological advances.\u00a0 Our invitations sought enthusiastic attendees who were interested in setting an aggressive agenda for change.\u00a0 We invited Dr. Harold Brooks (NSSL) to give a seminar at the workshop about how the NWS might go improving the system it uses to verify severe weather warnings.\u00a0 Many of the ideas I will present were borne out of Harold’s original presentation, and I will build upon them.<\/p>\n

So, to start, let’s look at how NWS warnings are verified today.\u00a0 As many of you know, the NWS transitioned to what is now known as polygon-based warnings about four years ago.\u00a0 Essentially, this means that warnings are now supposed to be drawn as polygons that represent the swath in which the forecaster thinks severe weather will occur during the duration of the warning, without regard to geo-political boundaries.\u00a0 In the past, warnings were county-based, even though severe storms don’t really care about that!\u00a0 It’s better to call the system “storm-based warnings”, since after all, counties are just differently-shaped polygons.<\/p>\n

\"storm-based-warnings\"<\/a><\/p>\n

But what really changed was not the shape of the warnings, but how warnings were verified.\u00a0 No longer was it required that each county covered in a warning received at least one storm report.\u00a0 Now, only one storm report is required to verify a single polygon warning.\u00a0 This sounded attractive since it meant that if a small storm-based warning touched several small portions of multiple counties, that there was no need to find a report in each of those county segments, reducing the workload required to gather such information.\u00a0 But, as I will show, there are flaws in that logic.<\/p>\n

How are forecasts verified for accuracy?\u00a0 One of the simplest ways to do this is via the use of a 2×2 contingency table.\u00a0 Each of the four cells in the matrix are explained:\u00a0 A verified forecast is called a HIT (cell A), and represents when an event did happen when the forecast said it would happen.\u00a0 An unverified forecast is called a FALSE ALARM (cell B) when a forecast was issued, but an event did not happen.\u00a0 An event that went unforecasted is called a MISS (Cell C).\u00a0 And finally, wherever and whenever events did not happen, when there was a forecast of no event (or no forecast at all), that is called a CORRECT NULL (Cell D).<\/p>\n

\"ctable\"<\/a><\/p>\n

Also from this table, one can derive a number of accuracy measures.\u00a0 The first is called Probability Of Detection (POD), which is the ratio of verified forecasts (HIT) to the number of all forecasts (HIT + MISS).\u00a0 Another is the False Alarm Ratio (FAR) or Probability Of False Alarm (POFA), which is the ratio of false forecasts (FALSE ALARM) to all forecasts of an event (HIT + FALSE ALARM).\u00a0 Finally, one can represent the combination of both POD and FAR into the Critical Success Index (CSI), which is the ratio of HIT to the sum of all HIT, MISS, and FALSE ALARM.\u00a0 CSI can be written both as a function of A, B, and C, and through algebraic manipulation, a function of POD and FAR.<\/p>\n

\"csi1\"<\/a><\/p>\n

In the next post, I will explain how NWS warnings are verified today, and how they use the 2×2 table and the above measures to derive their metrics.<\/p>\n

Greg Stumpf, CIMMS and NWS\/MDL<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

Several years ago (in 2005), we hosted our first user group workshop on severe weather technology for NWS warning decision making.\u00a0 This workshop was held at NWS headquarters in Silver… <\/p>\n

Read more \u00bb<\/a><\/p>\n","protected":false},"author":333,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[],"class_list":["post-2578","post","type-post","status-publish","format-standard","hentry","category-experimental-warning-thoughts"],"_links":{"self":[{"href":"https:\/\/inside.nssl.noaa.gov\/ewp\/wp-json\/wp\/v2\/posts\/2578","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/inside.nssl.noaa.gov\/ewp\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/inside.nssl.noaa.gov\/ewp\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/inside.nssl.noaa.gov\/ewp\/wp-json\/wp\/v2\/users\/333"}],"replies":[{"embeddable":true,"href":"https:\/\/inside.nssl.noaa.gov\/ewp\/wp-json\/wp\/v2\/comments?post=2578"}],"version-history":[{"count":2,"href":"https:\/\/inside.nssl.noaa.gov\/ewp\/wp-json\/wp\/v2\/posts\/2578\/revisions"}],"predecessor-version":[{"id":14744,"href":"https:\/\/inside.nssl.noaa.gov\/ewp\/wp-json\/wp\/v2\/posts\/2578\/revisions\/14744"}],"wp:attachment":[{"href":"https:\/\/inside.nssl.noaa.gov\/ewp\/wp-json\/wp\/v2\/media?parent=2578"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/inside.nssl.noaa.gov\/ewp\/wp-json\/wp\/v2\/categories?post=2578"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/inside.nssl.noaa.gov\/ewp\/wp-json\/wp\/v2\/tags?post=2578"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}