| |
| Robert Fildes | |
| Distinguished Emeritus Professor, Lancaster University |
How do you become a forecaster, or even better a Superforecaster? And why would you wish to take on such a role? For many years, forecasting as an academic discipline was a bit of an orphan. Were you an operations researcher, a statistician, or an econometrician? Each of these subject disciplines had their own requirements – in OR some of the premier institutions didn’t even have an advanced course which included forecasting so research posts were rare.
My own training as a statistician only included stochastic processes. It was only in the 1970s that the first course books focused on forecasting started to appear, stimulated by Box and Jenkins (1970). But forecasting as a research topic was still not part of the mainstream with few university departments majoring in it. This all began to change with the emergence of the International Institute of Forecasters and the founding of two academic journals, the International Journal of Forecasting and the Journal of Forecasting which through their published papers effectively defined the discipline. A crucial component of the Institutes success was an annual international symposium welcoming all forecasters from weather to demographics but particularly focusing on business and economics. It was and is a broad methodological church with mixed parentage from the core areas but also embracing operations and psychology. Now data science with AI and Machine Learning are at its forefront. This is a formidable range for any researcher to tackle.
"But there are good reasons to become a superforecaster, with shortages in academia but also in a wide range of commercial, software and public organizations.
"
The Forecasters’ Skill Set
The early career student will have a home in one of the core areas from data science and econometrics through to behavioural science or psychology. In addition to their core curriculum, surprise, the student needs to learn the basics of forecasting: benchmark methods, comparative validation and forecasting competitions, accuracy measures, model selection. Two core forecasting texts are: Ord et al. (2017) and more technical, Hyndman and Athanasopoulos (2018). Why do I state the obvious – because many researchers have grown up with a single methodological focus, they are often unaware of the forecasting canon. Many a machine learner still thinks that all that needs to be done is propose a new algorithm, unaware that exponential smoothing, some 70 years old, is still hard to beat when forecasting business data.
Outside academia the forecaster’s job may be research focussed seeking out improvements on current practice, but it may also involve the production and presentation of regular forecasts: this latter role relies on a quite different skill set. This brings up a second area of neglect, an understanding of organizational forecasting and the psychology of forecasting. For most students the best route to understanding is a placement in an organization to discover just why it is that a fancy new algorithm is neglected in favour of current practices which depend on the expertise of the forecasters in the organization. Knowing about different decision contexts and the data available will also stimulate ideas for a research topic, with on-line data and novel logistics systems as recent examples.
"In short, a deep understanding of the core disciplines employed in forecasting is not enough, the novice superforecaster has to delve into the practicalities of the subject.
"
What Pitfalls to Avoid?
As preparation for this wide variety of roles, the most important early pitfalls to avoid are (i) specializing too earlier, and (ii) not mastering one of the core areas (more would be too ambitious). Forecasting is not an abstract discipline but depends on its organizational context to be effective. Consequently, the novice needs a broad understanding of the context in which their specialist topic will prove valuable. The context determines both the data availability, the methods to explore and their constraints, and the loss functions, both theoretical and organizational that need to be adopted.
What makes a Superforecaster?
Superforecasting was first identified by Tetlock and colleagues (2016). The book of the same name drew lessons from a research study of around 80K longer-term political forecasts. They found that the more accurate forecasters shared characteristics in common, they were ‘cautious, humble, open minded, analytical – and good with numbers!’. For readers of this piece, we can take some of these requirements for granted. But what about open-mindedness: nowadays every problem, some believe, can be solved through ML methods, forgetting a key lesson from past forecasting research that simple methods often outperform their much more complicated competitors and, in addition, are much more acceptable to users. So, the most serious pitfall is also the most attractive to the student forecaster, to immerse themselves in the technology of a particular set of techniques, neglecting the fact that the broader discipline offers lessons that need to be absorbed. The other side of this coin is that for a job in organizations such as Google or Amazon, the forecaster will need to be thoroughly knowledgeable about the technical details and opportunities (probably in ML). So early on in a research training program, even to the extent of choosing a place of study, the student faces a dilemma: do they choose the deep dive into the technical or aim for a forecasting career more broadly? It requires the most difficult subjective judgmental forecast to decide which of the two will prove most satisfying. In making my own choice I realized I preferred breadth rather than depth (despite a highly specialized PhD) and that pointed the way to work in a business school committed to engaging with organizational problems rather than, for example, a data science department. And remember, it’s very hard for a university department of whatever colour to compete with large technical departments of the research-oriented technology companies. As an example, the winner of a major ‘forecasting competition’ worked for Uber (Smyl as discussed in Makridakis et al. (2018)).
"A stark choice then: Do you choose breath or depth?
"
The Future of Forecasters and Forecasting
Since the founding of the Institute in 1981, forecasting has seen dramatic changes, both in research communities but also in practice. Dawes et al. (1994) looked to the future of forecasting 30 years ago. They were remarkably prescient with three themes predominant then and now:
- The synergies (and problems) arising from the forecaster’s knowledge and the forecasting algorithm in use.
- The combining of and model selection from a range of algorithms.
- The importance of context in forecasting which encompasses the users, the decisions which depend on the forecasts and the data.
The last thirty years have seen major changes in the forecasting discipline, particularly with the advent of ML/AI methods. Some assume that there will emerge a ML/AI method which will render these concerns irrelevant. As yet there is no sign of such an all-encompassing solution. When the M5 forecasting competition was launched with its focus on ML methods, the winner turned out to be a combination of statistical and ML. More crucially, the majority of ML methods failed to outperform critical statistical benchmarks. While advances since then have been rapid, there is no evidence that a general ‘winner’ will be found including Large Language Models. I personally don’t believe there could be for the reasons given above. Ask yourself the question, can a general method designed for many different circumstances, however clever, prove most accurate despite the singular characteristics of the particular problem being faced?
"So these three areas remain topical and relevant today. But with novel development of new ML methods, now available to any would-be superforecaster, in particular Large Language Models, amplified by the ability to process vast amounts of data, new challenges will emerge and new pitfalls, not least the trap of getting caught up in the technology whilst ignoring the problem.
"
The role of humans in the forecasting process is a concept that is central to trustworthy AI, forecasting. Superforecasters of the future will have their work cut out for them to find ways to streamline both the information flows, that are increasingly more complex, and the value from the decisions supported by the forecasts. This gives tremendous opportunities for the forecasting field, both in innovation and impact. As a final note, these themes are stimulating and fun, not least because they require the researcher to consider a range of perspectives, and that it is perhaps the best training for a ‘superforecaster.’
"Robert Fildes is Distinguished Emeritus Professor of Management Science in the Management School, Lancaster University UK, Founding Director of the Lancaster Centre for Marketing Analytics and Forecasting and recipient of the Beale Medal, the UK Operational Research Society’s highest accolade. He was co-founder in 1981 of the Journal of Forecasting and in l985 of the International Journal of Forecasting. His current research interests are concerned with the comparative evaluation of different forecasting methods, the role of the forecaster, and the implementation of improved forecasting procedures and systems in organizations. He is well-versed in what superforecasters can be!
"
References
Box, G.E.P., Jenkins, G.M., 1970. Time Series Analysis: Forecasting and Control. Holden-Day, San Francisco.
Dawes, R., Fildes, R., Lawrence, M., Ord, K., 1994. The past and the future of forecasting research. International Journal of Forecasting 10, 151–159.
Hyndman, R.J., Athanasopoulos, G., 2018. Forecasting: Principles and Practice. OTexts. URL: https://otexts.com/fpp2/. Makridakis, S., Spiliotis, E., Assimakopoulos, V., 2018. The m4 competition: Results, findings, conclusion and way forward.
International Journal of Forecasting 34, 802–808. doi:10.1016/j.ijforecast.2018.06.001.
Ord, J.K., Fildes, R., Kourentzes, N., 2017. Principles of Business Forecasting. 2 ed., Wessex Press.
Acknowledgements: We would like to thank Kara Combs for taking time to review this article. Photo credit goes to Jakub Zerdicki for the header photo and Stephen Dawson for the footer photo.