Is our over reliance on numerical weather models making us worse forecasters?

It seems there is a new profession in the field of weather forecasting. I like to call it modelology. Which is far different from the field of meteorology. There is a disturbing trend that seems to be infecting our field and, in my opinion, is making us worse at forecasting the weather.

Listen I love and know that numerical weather guidance or models have come a long way. They are getting better in both accuracy, reliability and more importantly in resolution. Though I would argue the resolution is getting better at a rate that is not equal to the accuracy. Which therein lies the problem if the data going in is the same, but you have higher resolution you actually have the potential for bigger errors.

I love models they are essential tools for my job.  I’m like a lot of weather geeks staying up to 1:00 am EST to see the 0Z runs of ECMWF model when a big storm is coming. Just like tools in any profession though they need to be used correctly by a skilled craftsman to make a forecast. They aren’t forecasts in and of themselves, thus why they are called “guidance.”  If you are a qualified carpenter and have a good saw, hammer and glue, you can make some beautiful things. Those tools don’t build that nice thing themselves and just like the carpenter this skill takes time and practice. You don’t need to have a degree either though it helps. Forecasting is a skill that takes time, practice and honestly lots of failures to get better. That’s why even a non-degree forecaster can become very skilled at forecasting over a Ph.D. in meteorology who doesn’t practice that skill on a daily basis.

With models becoming better, easier available and prettier to look at thanks to the bevy of model sites that have cropped up. Forecasters, both amateur and professional have strayed away from the skill of actually forecasting the weather and become glorified model regurgitators.  The model advancements have been great, but it’s like my carpenter friend who now uses an advanced power saw and a nail gun. He still must use his skills combined with these new tools to build his final product. The tools lying around with some wood don’t make a dang thing. I am hearing and seeing way too much of this, “well this model sucked”, or the “models missed this,” etc. Hey, remember they are models they aren’t forecasts. You issue a forecast, not the model.

The GPS problem

2016-01-27_11-22-23

Like other advancements in technology you still need to be involved. I’m sure you have read or heard a story of some knucklehead driving into a lake or down an abandoned fire road because their GPS navigation system told them to. I mean at some point when your GPS tells you to turn right, and there’s a wall there you stop. Right? Just this week I saw people going crazy on one single deterministic run of the ECMWF. No other piece of guidance was supporting this track or location. A whole bunch turned into a lake, or, in this case, the Atlantic Ocean. Why do we do this when meteorological common sense says otherwise?

ECMWF Meme

2016-01-27_11-18-59Another issue we have is the prevailing attitude that one model is superior to others. Yes, we all know over time the ECMWF has performed better especially in the long and mid-range than just about any other model. Does it always do better, though? No, it often fails as miserable as the GFS, yet many have acquired a bias towards one model. We remember only it’s “good” runs and failing to remember the bad ones.  Where now we are seeing people jumping on an ECMWF solution at seven days that they would never do with a GFS or CMC run. Remember Hurricane Joaquin? The ECMWF eventually got it correct quicker, but we forget at the beginning it too struggled with it like the GFS. Oh yeah, that run where the GFS nailed it 1st before the ECMWF, no one talks about that for some reason.

2016-01-27_11-06-17

Courtesy WeatherBell

2016-01-27_11-06-46

Courtesy WeatherBell

So how do we fix this?

  • Listen I love where our science, technology, and modeling is going. Both the GFS and ECMWF are going to get huge upgrades that go operational soon. Just remember, know where this data comes from. Stop looking at strictly derived snowfall maps and dive into the QPF in the model and look at the thicknesses, profiles, and mandatory temperature levels.  You’ll often find it’s not all snow it’s a mixed bag and it rarely if ever falls with a 10:1 snow to liquid ratio.
  • Keep a spreadsheet and verify your forecasts. This isn’t to brag when you are right; this is to figure out why you were wrong. I’ve done this for years, and it’s an eye opener how you recognized bias in your forecasts. Remember a busted forecast is only bad if you don’t learn from it.
  • Take your emotions out of a forecast. This is hard as a weather geek, really hard! If you like weather, don’t let it cloud or skew your forecast to something you want versus what is realistic
  • Forecast what is most likely over what could happen. Now I’m not saying don’t lay out all the possibilities, but lay them all out. Which means if you say there could be 1 foot of snow make sure also say it’s equally possible we get zero!
  • It’s okay to say we don’t really know. Tell people how confidence or not confident you are in your forecast. This is where modeling is huge they can give you confidence or take it away.
  • Always look at ensembles they help remove outliers and help you forecast what’s likely versus the extreme outliers.
  • Local knowledge and experience combined with climatology help tremulously. Analogs are great, has anything like this ever happened before?
  • Remember in a warming world the models may not be able to correctly simulate extreme events.

More Tips:

After posting this blog I received this really interesting paper abstract. In Canada they performed an experiment to see if forecasters could forecast without models. The results were shocking.

Project Phoenix – Optimizing the machine-person mix in high-impact weather forecasting

Patrick J. McCarthy, MSC, Winnipeg, MB, Canada; and W. Purcell and D. Ball

In the numerical model-dominated weather services of the world, do meteorologists still add value in weather forecasting? Project Phoenix was an innovative experiment designed to test this question. Conducted early in 2001 at the Prairie Storm Prediction Centre (PSPC) in Winnipeg, Manitoba, a team of three forecasters prepared noon and late afternoon public weather forecasts for the Canadian Provinces of Alberta, Saskatchewan and Manitoba. This Phoenix team began with the text forecasts produced by SCRIBE, Canada’s automated model-based forecast production software. From this starting point, the meteorologists produced their weather forecasts using only their analysis, diagnosis and prognosis skills by using real data, such as radar, satellite imagery, surface observations and upper air soundings. The Phoenix forecasters had no access to any numerical weather prediction data beyond the original SCRIBE text forecasts. The goal of the two week experiment was to discover if short-term forecasting techniques applied by meteorologists could achieve a significant improvement over the automated SCRIBE product, particulalry in terms of high-impact weather (HIW). The performance of the SCRIBE, Phoenix, and the official forecasts were verified against a comprehensive evaluation system.

The Phoenix meteorologists improved the forecasts beyond expectations during the two-week test. In a curious twist, the Phoenix forecasts also were significantly better than the official versions prepared by the PSPC. The PSPC repeated the experiment a few months later. The results confirmed the first test and Project Phoenix became a formal training program for all PSPC meteorologists. Over a dozen more groups completed a one-week version, including one limited to senior personnel and another team comprised of recent graduates. Every Phoenix group posted major improvements over the automated forecast products. As well, all the Phoenix teams achieved at least a slight improvement over the PSPC official forecasts. This training system has now been adopted by the Meteorological Service of Canada to position forecasters for HIW forecasting roles in the years ahead. This presentation examines Project Phoenix, the verification system, and offers an explanation for the results.

Link to paper: https://ams.confex.com/ams/22WAF18NWP/techprogram/paper_122657.htm

 

 

  • James Marusak

    What you’re describing can probably said a lot easier.

    Don’t treat the models like gods on Mount Olympus. They’re only gridded math problems with a couple quadrillion steps.

    Like you said, forecasting is a skill you pick up, not only with skill, but also instinct. But in a lot of company’s minds, when they actually look for mets to hire, a lot of them from my experience actually prefer you to have little instinct and skills, but just training on how to fundamentally use the models. And in more than a few small to mid-sized companies (and even a large one or two), in order to get out as many forecasts as possible, they will want you to just do it any way you can and get it semi-close, even if the way is improper and overly-leaning on using direct model guidance. It’s cheaper and faster, quality comes as it comes. Actually have some experience and know what you’re doing, that’s poison to some places as it means you actually take the time to do it right, and aren’t as cheap from a labor standpoint.

    and heck, actually take 5-10 minutes at the beginning for any forecast shift to see how your previous ones did? or ask someone how your forecasts worked out (for a more independent set of eyes)? want to get some strange looks and responses from people; like “it’s a waste of time”, or “what, you actually check your work?” I have gotten more than a few of those bad looks, and heck, it might have actually worked against me in a weather job as I knew some of the stuff being given to clients didn’t pass the smell test, with lows busting by 8-10 pm that were meant for 6 am. And of course on the update I had to try to fix messes like that, in the same areas, over and over. talk about frustrating.

    And as for learning forecasting like in the paper you cited, I only know of only like 2-3 college/university programs that have such a forecasting class in and of itself, whether optional or otherwise. Most places combine forecasting training with say synoptic met classes or such and generally gloss over the subject. You have all types of classes in programming and such that can be used to statistically blend models, but almost none on the skill of actually making a good forecast (and being able to make it customized for the client’s needs). That’s a reason that at least imho, I see forecasting as an art as well as a science. most places gloss over the science, but rarely take the time to teach the finer artistic standpoints (a lot like you mentioned, but others also that can be gone over in a different venue).

    It’s a mindset that needs to change in the enterprise, but one that unfortunately will be very hard to change as people to insist for quantity and price instead of quality. It reminds me of a couple of quotes I hear from time to time in the window factory I currently work at (to pay the bills until I can get back into a forecasting job).

    1: in any production outlet, you have 3 qualities, but in almost all instances, you can only get two of the three; quality, price and speed. want quality and a good price, it will take a while to get done. want cheap and fast, don’t expect quality. want quality and want it asap, it ain’t cheap.

    2: by Karl Albrecht: you seldom improve quality by cutting costs. but you can often cut costs by improving quality.