Category Archives: forecasting turning points

Federal Reserve policies and impacts, financial forecasting, forecasting turning points, macroeconomic forecasting

Coming Attractions

Well, I have been doing a deep dive into financial modeling, but I want to get back to blogging more often. It gets in your blood, and really helps explore complex ideas.

So- one coming attraction here is going to be deeper discussion of the fractal market hypothesis.

Ladislav Kristoufek writes in a fascinating analysis (Fractal Markets Hypothesis and the Global Financial Crisis:Scaling, Investment Horizons and Liquidity) that,

“..it is known that capital markets comprise of various investors with very different investment horizons { from algorithmically-based market makers with the investment horizon of fractions of a second, through noise traders with the horizon of several minutes, technical traders with the horizons of days and weeks, and fundamental analysts with the monthly horizons to pension funds with the horizons of several years. For each of these groups, the information has different value and is treated variously. Moreover, each group has its own trading rules and strategies, while for one group the information can mean severe losses, for the other, it can be taken a profitable opportunity.”

The mathematician and discoverer of fractals Mandelbrot and investor Peters started the ball rolling, but the idea maybe seemed like a fad of the 1980’s and 1990s.

But, more and more,  new work in this area (as well as my personal research) points to the fact that the fractal market hypothesis is vitally important.

Forget chaos theory, but do notice the power laws.

The latest  fractal market research is rich in mathematics – especially wavelets, which figure in forecasting, but which I have not spent much time discussing here.

There is some beautiful stuff produced in connection with wavelet analysis.

For example, here is a construction from a wavelet analysis of the NASDAQ from another paper by Kristoufek

Wavlet1

The idea is that around 2008, for example, investing horizons collapsed, with long term traders exiting and trading becoming more and more short term. This is associated with problems of liquidity – a concept in the fractal market hypothesis, but almost completely absent from many versions of the so-called “efficient market hypothesis.”

Now, maybe like some physicists, I am open to the discovery of deep keys to phenomena which open doors of interpretation across broad areas of life.

Another coming attraction will be further discussion of forward information on turning points in markets and the business cycle generally.

The current economic expansion is growing long in tooth, pushing towards the upper historically observed lengths of business expansions in the United States.

The basic facts are there for anyone to notice, and almost sound like a litany of complaints about how the last crisis in 2008-2009 was mishandled. But China is decelerating, and the emerging economies do not seem positioned to make up the global growth gap, as in 2008-2009. Interest rates still bounce along the zero bound. With signs of deteriorating markets and employment conditions, the Fed may never find the right time to raise short term rates – or if they plunge ahead will garner virulent outcry. Financial institutions are even larger and more concentrated now than before 2008, so “too big to fail” can be a future theme again.

What is the best panel of financial and macroeconomic data to watch the developments in the business cycle now?

So those are a couple of topics to be discussed in posts here in the future.

And, of course, politics, including geopolitics will probably intervene at various points.

Initially, I started this blog to explore issues I encountered in real-time business forecasting.

But I have wide-ranging interests – being more of a fox than a hedgehog in terms of Nate Silver’s intellectual classification.

I’m a hybrid in terms of my skill set. I’m seriously interested in mathematics and things mathematical. I maybe have a knack for picking through long mathematical arguments to grab the key points. I had a moment of apparent prodigy late in my undergrad college career, when I took graduate math courses and got straight A’s and even A+ scores on final exams and the like.

Mathematics is time consuming, and I’ve broadened my interests into economics and global developments, working around 2002-2005 partly in China.

As a trivia note,  my parents were immigrants to the US from Great Britain , where their families were in some respects connected to the British Empire that more or less vanished after World War II and, in my father’s case, to the Bank of England. But I grew up in what is known as “the West” (Colorado, not California, interestingly), where I became a sort of British cowboy and subsequently, hopefully, have continued to mature in terms of attitudes and understanding.

Deep Questions, forecasting turning points

Our Next President is a Wrestling Giant – Trump

Greetings, and I thought you would all enjoy this bit of rough-and-tumble involving the leading Republican candidate so far for US President – Donald Trump.

Make sure you watch past the 42 second mark to see Trump lambast his billionaire buddy. 

So this is really happening. Trump apparently has hired people to work on his campaign for President, and he has taken an early lead over Scott Walker and Jeb Bush, and the other more minor candidates.

accuracy of forecasts, financial forecasting, forecasting turning points, stock market forecasts

Monday Morning Stock Forecasts May 18 – Highs and Lows for SPY, QQQ, GE, and MSFT

Look at it this way. There are lots of business and finance blogs, but how many provide real-time forecasts, along with updates on how prior predictions performed?

Here on BusinessForecastBlog – we roll out forecasts of the highs and lows of a growing list of securities for the coming week on Monday morning, along with an update on past performance.

It’s a good discipline, if you think you have discovered a pattern which captures some part of the variation in future values of a variable. Do the backtesting, but also predict in real-time. It’s very unforgiving.

Here is today’s forecast, along with a recap for last week (click to enlarge).TableMay18

There is an inevitable tendency to narrate these in “Nightly Business Report” fashion.

So the highs are going higher this week, and so are the lows, except perhaps for a slight drop in Microsoft’s low – almost within the margin of statistical noise. Not only that, but predicted increases in the high for QQQ are fairly substantial.

Last week’s forecasts were solid, in terms of forecast error, except Microsoft’s high came in above what was forecast. Still, -2.6 percent error is within the range of variation in the backtests for this security. Recall, too, that in the previous week, the forecast error for the high of MSFT was only .04 percent, almost spot on.

Since the market moved sideways for many securities, No Change forecasts were a strong competitor to the NPV (new proximity variable) forecasts. In fact, there was an 50:50 split. In half the four cases, the NPV forecasts performed better; in the other half, No Change forecasts had lower errors.

Direction of change predictions also came in about 50:50. They were correct for QQQ and SPY, and wrong for the two stocks.

Where is the Market Going?

This tool – forecasts based on the NPV algorithms – provides longer terms looks into the future, probably effectively up to one month ahead.

So in two weeks, I’m going to add that forecast to the mix. I think it may be important, incidentally, to conform to the standard practice of taking stock at the beginning of the month, rather than, say, simply going out four weeks from today.

To preview the power of this monthly NPV model, here are the backtests for the crisis months before and during the 2008 financial crisis.

SPYM2008

This is a remarkable performance, really. Once the crash really gets underway in late Summer-Fall 2008, the NPV forecast drops in a straight-line descent, as do the actual monthly highs. There are some turning points in common, too, between the two series. And generally, even at the start of the process, the monthly NPV model provides good guidance as to the direction and magnitude of changes.

Over the next two weeks, I’m collecting high frequency data to see whether I can improve these forecasts with supplemental information – such as interest rates spreads and other variables available on a weekly or monthly basis.

In closing, let me plug Barry Eichengreen’s article in Syndicate An Economics to Fit the Facts.

Eichengreen writes,

While older members of the economics establishment continue to debate the merits of competing analytical frameworks, younger economists are bringing to bear important new evidence about how the economy operates.

It’s all about dealing with the wealth of data that is being collected everywhere now, and much less about theoretical disputes involving formal models.

Finally, it’s always necessary to insert a disclaimer, whenever one provides real-time, actionable forecasts. This stuff is for informational and scientific purposes only. It is not intended to provide recommendations for specific stock trading, and what you do on that score is strictly your own business and responsibility.

Mountain climbing pic from Blink.

energy forecasting, forecasting turning points, futures markets

Do Oil and Gas Futures Forecast Oil and Gas Spot Prices?

I’m looking at evidence that oil and gas futures are useful in forecasting future prices. This is an important for reasons ranging from investment guidance to policy analysis (assessing the role of speculators in influencing current market prices).

So – what are futures contracts, where are they traded, and where do you find out about them?

A futures contract (long position) is an agreement to buy an amount of a commodity (oil or gas) at a specified price at the expiration of the contract. The seller (the party with a short position) agrees to sell the underlying commodity to the buyer at expiration at the fixed sales price. Futures contracts can be traded many times prior to the expiration date.

At the expiration of the contract, if the price of the contract is below the market or spot price at that time, the buyer makes money. Futures contracts also can be used to lock in prices, and hedge risk.

The New York Mercantile Exchange (NYMEX) maintains futures markets for oil and gas. Natural gas futures are based on delivery at the Henry Hub, Louisiana, a major crossroads for natural gas pipelines.

So there are futures contracts for 1 month, 2 month, and so forth, delivery dates.

Evidence Futures Predict Spot Prices

As noted by Menzie Chinn, a popular idea is that the futures price is the optimal forecast of the spot price is an implication of the efficient market hypothesis.

Nevertheless, the evidence for futures prices being unbiased estimators of future spot prices is mixed, despite widespread acceptance of the idea in central banks and the International Monetary Fund (IMF).

A recent benchmark study, Forecasting the Price of Oil, finds –

some evidence that the price of oil futures has additional predictive content compared with the current spot price at the 12-month horizon; the magnitude of the reduction in mean-squared prediction error (MSPE) is modest even at the 12-month horizon, however, and there are indications that this result is sensitive to fairly small changes in the sample period and in the forecast horizon. There is no evidence of significant forecast accuracy gains at shorter horizons, and at the long horizons of interest to policymakers, oil futures prices are clearly inferior to the no-change forecast.

Here, the “no-change forecast” can be understood and is sometimes also referred to as a “random walk forecast.”

Both Chinn and the Forecasting the Price of Oil chapter in the Handbook of Forecasting are good places for readers to check the extensive literature on this topic.

Hands-On Calculation

Forecasting is about computation and calculation, working with real data.

So I downloaded the Contract1 daily futures prices from the US EIA, a source which also provides the Henry Hub spot prices.

Natural gas contracts, for example, expire three business days prior to the first calendar day of the delivery month. Thus, the delivery month for Contract 1 in the US EIA tables is the calendar month following the trade date.

Here is a chart from the spreadsheet I developed.

FuturesDirectionCallChart1

I compared the daily spot prices and 1 month futures contract prices by date to see how often the futures prices correctly indicate the direction of change of the spot price at the settlement or delivery date, three days prior to the first calendar day of the delivery month. So, the April 14, 2014 spot price was $4.64 and the Contract1 futures closing price for that day was $4.56, indicating that the spot price in late May would be lower than the current spot price. In fact, the May 27th spot price was $4.56. So, in this case, not only was the predicted direction of change correct, but also the point estimate of the future spot price.

The chart above averages the performance of these daily forecasts of the future direction of spot prices over rolling 20 trading day windows.

From January through the end of September 2014, these averages score better than 50:50 about 71 percent of the time.

I have not calculated how accurate these one month natural gas futures are per se, but my guess is that the accuracies would be close.

However, clearly, a “no-change forecast” is incapable of indicating the future direction of changes in the gas spot price.

So the above chart and the associated information structure are potentially useful regardless of the point forecast accuracy. My explorations suggest additional information about direction and, possibly, even turning points in price, can be extracted from longer range gas futures contracts.

accuracy of forecasts, business trends, forecasting turning points, global business forecasts, predicting recessions

Surprising Revision of First Quarter GDP

I showed a relative this blog a couple of days ago, and, wanting “something spicy,” I pulled up The Record of Failure to Predict Recessions is Virtually Unblemished. The lead picture, as for this post, is Peter Sellers in his role as “Chauncey Gardiner” in Being There. Sellers played a simpleton mistaken for a savant, who would say things that everyone thought was brilliant, such as “There will be growth in the Spring.”

Well, last Wednesday, the US Bureau of Economic Analysis released a third revision of its estimate of the 1st quarter 2014 real GDP growth – down from an initial estimate of a positive .1 percent to -2.9 percent growth at an annual rate.

The BEA News Release says,

Real gross domestic product — the output of goods and services produced by labor and property located in the United States — decreased at an annual rate of 2.9 percent in the first quarter of 2014 according to the “third” estimate released by the Bureau of Economic Analysis….

The decrease in real GDP in the first quarter primarily reflected negative contributions from private inventory investment, exports, state and local government spending, nonresidential fixed investment, and residential fixed investment that were partly offset by a positive contribution from PCE. Imports, which are a subtraction in the calculation of GDP, increased.

Looking at this graph of quarterly real GDP growth rates for the past several years, it’s clear that a -2.9 percent quarter-over-quarter change is a significant size.

usgdpchartcustom

Again, macroeconomic forecasters were caught off guard.

In February of this year, the Survey of Professional Forecasters released its 1st Quarter 2014 consensus forecasts with numbers like –

SPF

Some SPF participants do predict 2014 overall will be a year of recession, as the following chart shows, but they are a tiny minority.

spfrange

A downward revision of almost 3 percentage points on the part of the BEA and almost 5 percent change for the median SPF forecast is poor performance indeed.

One hears things sped up in Q2, but on what basis I do not really know – and I am thinking of tracking key markets in future posts, such as housing, consumer spending, and so forth.

My feeling is that the quandary of the Fed – its desperate need to wind down asset purchases and restore interest rates to historic levels –creates an environment for a kind of “happy talk.”

Here’s some history on the real GDP.

USGDPnew

 

accuracy of forecasts, asset bubbles, Big Data, data reduction, financial forecasting, forecasting turning points, multivariate modeling, principal component models and forecasts, real estate forecasts

Forecasting Housing Markets – 3

Maybe I jumped to conclusions yesterday. Maybe, in fact, a retrospective analysis of the collapse in US housing prices in the recent 2008-2010 recession has been accomplished – but by major metropolitan area.

The Yarui Li and David Leatham paper Forecasting Housing Prices: Dynamic Factor Model versus LBVAR Model focuses on out-of-sample forecasts for house price indices for 42 metropolitan areas. Forecast models are built with data from 1980:01 to 2007:12. These models – dynamic factor and Large-scale Bayesian Vector Autoregressive (LBVAR) models – are used to generate forecasts of the one- to twelve- months ahead price growth 2008:01 to 2010:12.

Judging from the graphics and other information, the dynamic factor model (DFM) produces impressive results.

For example, here are out-of-sample forecasts of the monthly growth of housing prices (click to enlarge).

DFMhousing

The house price indices for the 42 metropolitan areas are from the Office of Federal Housing Enterprise Oversight (OFEO). The data for macroeconomic indicators in the dynamic factor and VAR models are from the DRI/McGraw Hill Basic Economics Database provided by IHS Global Insight.

I have seen forecasting models using Internet search activity which purportedly capture turning points in housing price series, but this is something different.

The claim here is that calculating dynamic or generalized principal components of some 141 macroeconomic time series can lead to forecasting models which accurately capture fluctuations in cumulative growth rates of metropolitan house price indices over a forecasting horizon of up to 12 months.

That’s pretty startling, and I for one would like to see further output of such models by city.

But where is the publication of the final paper? The PDF file linked above was presented at the Agricultural & Applied Economics Association’s 2011 Annual Meeting in Pittsburgh, Pennsylvania, July, 2011. A search under both authors does not turn up a final publication in a refereed journal, but does indicate there is great interest in this research. The presentation paper thus is available from quite a number of different sources which obligingly archive it.

Yarui

Currently, the lead author, Yarui Li, Is a Decision Tech Analyst at JPMorgan Chase, according to LinkedIn, having received her PhD from Texas A&M University in 2013. The second author is Professor at Texas A&M, most recently publishing on VAR models applied to business failure in the US.

Dynamic Principal Components

It may be that dynamic principal components are the new ingredient accounting for an uncanny capability to identify turning points in these dynamic factor model forecasts.

The key research is associated with Forni and others, who originally documented dynamic factor models in the Review of Economics and Statistics in 2000. Subsequently, there have been two further publications by Forni on this topic:

Do financial variables help forecasting inflation and real activity in the euro area?

The Generalized Dynamic Factor Model, One Sided Estimation and Forecasting

Forni and associates present this method of dynamic prinicipal componets as an alternative to the Stock and Watson factor models based on many predictors – an alternative with superior forecasting performance.

Run-of-the-mill standard principal components are, according to Li and Leatham, based on contemporaneous covariances only. So they fail to exploit the potentially crucial information contained in the leading-lagging relations between the elements of the panel.

By contrast, the Forni dynamic component approach is used in this housing price study to

obtain estimates of common and idiosyncratic variance-covariance matrices at all leads and lags as inverse Fourier transforms of the corresponding estimated spectral density matrices, and thus overcome(s)[ing] the limitation of static PCA.

There is no question but that any further discussion of this technique must go into high mathematical dudgeon, so I leave that to another time, when I have had an opportunity to make computations of my own.

However, I will say that my explorations with forecasting principal components last year have led to me to wonder whether, in fact, it may be possible to pull out some turning points from factor models based on large panels of macroeconomic data.

accuracy of forecasts, financial forecasting, forecasting research, forecasting turning points, global business forecasts, macroeconomic forecasting, many predictors

Credit Spreads As Predictors of Real-Time Economic Activity

Several distinguished macroeconomic researchers, including Ben Bernanke, highlight the predictive power of the “paper-bill” spread.

The following graphs, from a 1993 article by Benjamin M. Friedman and Kenneth N. Kuttner, show the promise of credit spreads in forecasting recessions – indicated by the shaded blocks in the charts.

CPTBspread

Credit spreads, of course, are the differences in yields between various corporate debt instruments and government securities of comparable maturity.

The classic credit spread illustrated above is the difference between six-month commercial paper rates and 6 month Treasury bill rates.

Recent Research

More recent research underlines the importance of building up credit spreads from metrics relating to individual corporate bonds , rather than a mishmash of bonds with different duration, credit risk and other characteristics.

Credit Spreads as Predictors of Real-Time Economic Activity: A Bayesian Model-Averaging Approach is key research in this regard.

The authors first note that,

the “paper-bill” spread—the difference between yields on nonfinancial commercial paper and comparable-maturity Treasury bills—had substantial forecasting power for economic activity during the 1970s and the 1980s, but its predictive ability vanished in the subsequent decade

They then acknowledge that credit spreads based on indexes of speculative-grade or “junk” corporate bonds work fairly well for the 1990s, but their performance is uneven.

Accordingly, Faust, Gilchrist, Wright, and Zakrajsek (GYZ) write that

In part to address these problems, GYZ constructed 20 monthly credit spread indexes for different maturity and credit risk categories using secondary market prices of individual senior unsecured corporate bonds.. [measuring]..the underlying credit risk by the issuer’s expected default frequency (EDF™), a market-based default-risk indicator calculated by Moody’s/KMV that is more timely that the issuer’s credit rating]

Their findings indicate that these credit spread indexes have substantial predictive power, at both short- and longer-term horizons, for the growth of payroll employment and industrial production. Moreover, they significantly outperform the predictive ability of the standard default-risk indicators, a result that suggests that using “cleaner” measures of credit spreads may, indeed, lead to more accurate forecasts of economic activity.

Their research applies credit spreads constructed from the ground up, as it were, to out-of-sample forecasts of

…real economic activity, as measured by real GDP, real personal consumption expenditures (PCE), real business fixed investment, industrial production, private payroll employment, the civilian unemployment rate, real exports, and real imports over the period from 1986:Q1 to 2011:Q3. All of these series are in quarter-over-quarter growth rates (actually 400 times log first differences), except for the unemployment rate, which is simply in first differences

The results are forecasts which significantly beat univariate (autoregressive) model forecass, as shown in the following table.

Cspreadresults

Here BMA is an abbreviation for Bayesian Model Averaging, the author’s method of incorporating these calculated credit spreads in predictive relationships.

Additional research validates the usefulness of credit spreads so constructed for predicting macroeconomic dynamics in several European economies –

We find that credit spreads and excess bond premiums, when used alongside monetary policy tightness indicators and leading indicators of economic performance, are highly significant for predicting the growth in the index of industrial production, employment growth, the unemployment rate and real GDP growth at horizons ranging from one quarter to two years ahead. These results are confirmed for individual countries in the euroarea and for the United Kingdom, and are robust to different measures of the credit spread. It is the unpredictable part associated with the excess bond premium that has greater influence on real activity compared to the predictable part of the credit spread. The implications of our results are that careful selection of the bonds used to construct the credit spreads, excluding those with embedded options and or illiquid secondary markets, delivers a robust indicator of financial market tightness that is distinct from tightness due to monetary policy measures or leading indicators of economic activity.

The Situation Today

A Morgan Stanley Credit Report for fixed income, released March 21, 2014, notes that

Spreads in both IG and HY are at the lowest levels we have seen since 2007, roughly 110bp for IG and 415bp for HY. A question we are commonly asked is how much tighter can spreads go in this cycle

So this is definitely something to watch. 

anatomy of an asset bubble, asset bubbles, celebrity forecasters, financial forecasts, financial risk management, forecasting asset bubbles, forecasting research, forecasting turning points, stock market forecasts

Didier Sornette – Celebrity Bubble Forecaster

Professor Didier Sornette, who holds the Chair in Entreprenuerial Risks at ETH Zurich, is an important thinker, and it is heartening to learn the American Association for the Advancement of Science (AAAS) is electing Professor Sornette a Fellow.

It is impossible to look at, say, the historical performance of the S&P 500 over the past several decades, without concluding that, at some point, the current surge in the market will collapse, as it has done previously when valuations ramped up so rapidly and so far.

S&P500recent

Sornette focuses on asset bubbles and has since 1998, even authoring a book in 2004 on the stock market.

At the same time, I think it is fair to say that he has been largely ignored by mainstream economics (although not finance), perhaps because his training is in physical science. Indeed, many of his publications are in physics journals – which is interesting, but justified because complex systems dynamics cross the boundaries of many subject areas and sciences.

Over the past year or so, I have perused dozens of Sornette papers, many from the extensive list at http://www.er.ethz.ch/publications/finance/bubbles_empirical.

This list is so long and, at times, technical, that videos are welcome.

Along these lines there is Sornette’s Ted talk (see below), and an MP4 file which offers an excellent, high level summary of years of research and findings. This MP4 video was recorded at a talk before the International Center for Mathematical Sciences at the University of Edinburgh.

Intermittent criticality in financial markets: high frequency trading to large-scale bubbles and crashes. You have to download the file to play it.

By way of précis, this presentation offers a high-level summary of the roots of his approach in the economics literature, and highlights the role of a central differential equation for price change in an asset market.

So since I know everyone reading this blog was looking forward to learning about a differential equation, today, let me highlight the importance of the equation,

dp/dt = cpd

This basically says that price change in a market over time depends on the level of prices – a feature of markets where speculative forces begin to hold sway.

This looks to be a fairly simple equation, but the solutions vary, depending on the values of the parameters c and d. For example, when c>0 and the exponent d  is greater than one, prices change faster than exponentially and within some finite period, a singularity is indicated by the solution to the equation. Technically, in the language of differential equations this is called a finite time singularity.

Well, the essence of Sornette’s predictive approach is to estimate the parameters of a price equation that derives, ultimately, from this differential equation in order to predict when an asset market will reach its peak price and then collapse rapidly to lower prices.

The many sources of positive feedback in asset pricing markets are the basis for the faster than exponential growth, resulting from d>1. Lots of empirical evidence backs up the plausibility and credibility of herd and imitative behaviors, and models trace out the interaction of prices with traders motivated by market fundamentals and momentum traders or trend followers.

Interesting new research on this topic shows that random trades could moderate the rush towards collapse in asset markets – possibly offering an alternative to standard regulation.

The important thing, in my opinion, is to discard notions of market efficiency which, even today among some researchers, result in scoffing at the concept of asset bubbles and basic sabotage of research that can help understand the associated dynamics.

Here is a TED talk by Sornette from last summer.

bagging, downloadable texts on statistics and forecasting, forecasting turning points, macroeconomic forecasting, random walk

The On-Coming Tsunami of Data Analytics

More than 25,000 visited businessforecastblog, March 2012-December 2013, some spending hours on the site. Interest ran nearly 200 visitors a day in December, before my ability to post was blocked by a software glitch, and we did this re-boot.

Now I have hundreds of posts offline, pertaining to several themes, discussed below. How to put this material back up – as reposts, re-organized posts, or as longer topic summaries?

There’s a silver lining. This forces me to think through forecasting, predictive and data analytics.

One thing this blog does is compile information on which forecasting and data analytics techniques work, and, to some extent, how they work, how key results are calculated. I’m big on computation and performance metrics, and I want to utilize the SkyDrive more extensively to provide full access to spreadsheets with worked examples.

Often my perspective is that of a “line worker” developing sales forecasts. But there is another important focus – business process improvement. The strength of a forecast is measured, ultimately, by its accuracy. Efforts to improve business processes, on the other hand, are clocked by whether improvement occurs – whether costs of reaching customers are lower, participation rates higher, customer retention better or in stabilization mode (lower churn), and whether the executive suite and managers gain understanding of who the customers are. And there is a third focus – that of the underlying economics, particularly the dynamics of the institutions involved, such as the US Federal Reserve.

Right off, however, let me say there is a direct solution to forecasting sales next quarter or in the coming budget cycle. This is automatic forecasting software, with Forecast Pro being one of the leading products. Here’s a YouTube video with the basics about that product.

You can download demo versions and participate in Webinars, and attend the periodic conferences organized by Business Forecast Systems showcasing user applications in a wide variety of companies.

So that’s a good solution for starters, and there are similar products, such as the SAS/ETS time series software, and Autobox.

So what more would you want?

Well, there’s need for background information, and there’s a lot of terminology. It’s useful to know about exponential smoothing and random walks, as well as autoregressive and moving averages.  Really, some reaches of this subject are arcane, but nothing is worse than a forecast setup which gains the confidence of stakeholders, and then falls flat on its face. So, yes, eventually, you need to know about “pathologies” of the classic linear regression (CLR) model – heteroscedasticity, autocorrelation, multicollinearity, and specification error!

And it’s good to gain this familiarity in small doses, in connection with real-world applications or even forecasting personalities or celebrities. After a college course or two, it’s easy to lose track of concepts. So you might look at this blog as a type of refresher sometimes.

Anticipating Turning Points in Time Series

But the real problem comes with anticipating turning points in business and economic time series. Except when modeling seasonal variation, exponential smoothing usually shoots over or under a turning point in any series it is modeling.

If this were easy to correct, macroeconomic forecasts would be much better. The following chart highlights the poor performance, however, of experts contributing to the quarterly Survey of Professional Forecasters, maintained by the Philadelphia Fed.

SPFcomp2

So, the red line is the SPF consensus forecast for GDP growth on a three quarter horizon, and the blue line is the forecast or nowcast for the current quarter (there is a delay in release of current numbers). Notice the huge dips in the current quarter estimate, associated with four recessions 1981, 1992, 2001-2, and 2008-9. A mere three months prior to these catastrophic drops in growth, leading forecasters at big banks, consulting companies, and universities totally missed the boat.

This is important in a practical sense, because recessions turn the world of many businesses upside down. All bets are off. The forecasting team is reassigned or let go as an economy measure, and so forth.

Some forward-looking information would help business intelligence focus on reallocating resources to sustain revenue as much as possible, using analytics to design cuts exerting the smallest impact on future ability to maintain and increase market share.

Hedgehogs and Foxes

Nate Silver has a great table in his best-selling The
Signal and the Noise on the qualities and forecasting performance of hedgehogs and foxes. The idea comes from a Greek poet, “The fox knows many little things, but the hedgehog knows one big thing.”

Following Tetlock, Silver finds foxes are multidisplinary, adaptable, self-critical, cautious, and empirical, tolerant of complexity. By contrast, the Hedgehog is specialized, sticks to the same approaches, stubbornly adheres to his model in spite of counter-evidence, is order-seeking, confident, and ideological. The evidence suggests foxes generally outperform hedgehogs, just as ensemble methods typically outperform a single technique in forecasting.

Message – be a fox.

So maybe this can explain some of the breadth of this blog. If we have trouble predicting GDP growth, what about forecasts in other areas – such as weather, climate change, or that old chestnut, sun spots? And maybe it is useful to take a look at how to forecast all the inputs and associated series – such as exchange rates, growth by global region, the housing market, interest rates, as well as profits.

And while we are looking around, how about brain waves? Can brain waves be forecast? Oh yes, it turns out there is a fascinating and currently applied new approach called neuromarketing, which uses headbands and electrodes, and even MRI machines, to detect deep responses of consumers to new products and advertising.

New Methods

I know I have not touched on cluster analysis and classification, areas making big contributions to improvement of business process. But maybe if we consider the range of “new” techniques for predictive analytics, we can see time series forecasting and analysis of customer behavior coming under one roof.

There is, for example, this many predictor thread emerging in forecasting in the late 1990’s and especially in the last decade with factor models for macroeconomic forecasting. Reading this literature, I’ve become aware of methods for mapping N explanatory variables onto a target variable, when there are M<N observations. These are sometimes called methods of data shrinkage, and include principal components regression, ridge regression, and the lasso. There are several others, and a good reference is The Elements of Statistical Learning, Data Mining, Learning and Prediction, 2nd edition, by Trevor Hastie, Robert Tibshirani, and Jerome Friedman. This excellent text is downloadable, accessible via the Tools, Apps, Texts, Free Stuff menu option located just to the left of the search utility on the heading for this blog.

There also is bagging, which is the topic of the previous post, as well as boosting, and a range of decision tree and regression tree modeling tactics, including random forests.

I’m actively exploring a number of these approaches, ginning up little examples to see how they work and how the computation goes. So far, it’s impressive. This stuff can really improve over the old approaches, which someone pointed out, have been around since the 1950’s at least.

It’s here I think that we can sight the on-coming wave, just out there on the horizon – perhaps hundreds of feet high. It’s going to swamp the old approaches, changing market research forever and opening new vistas, I think, for forecasting, as traditionally understood.

I hope to be able to ride that wave, and now I put it that way, I get a sense of urgency in keeping practicing my web surfing.

Hope you come back and participate in the comments section, or email me at [email protected]

bagging, forecasting turning points, random walk, Uncategorized

Changes to Businessforecastblog in 2014 – Where We Have Been, Where We Are Going

We’ve been struggling with a software glitch in WordPress, due to, we think, incompatibilities between plug-in’s and a new version of the blogging software. It’s been pretty intense. The site has been fully up, but there was no possibility of new posts, not even a notice to readers about what was happening. All this started just before Christmas and ended, basically, yesterday.

So greetings. Count on daily posts as rule, and I will get some of the archives accessible ASAP.

But, for now, a few words about my evolving perspective.

I came out of the trenches, so to speak, of sales, revenue, and new product forecasting, for enterprise information technology (IT) and, earlier, for public utilities and state and federal agencies. When I launched Businessforecastblog last year, my bias popped up in the secondary heading for the blog – with its reference to “data-limited contexts” – and in early posts on topics like “simple trending” and random walks.

longterm_study_of_market_trends

I essentially believed that most business and economic time series are basically one form or another of random walks, and that exponential smoothing is often the best forecasting approach in an applied context. Of course, this viewpoint can be bolstered by reference to research from the 1980’s by Nelson and Plosser and the M-Competitions. I also bought into a lazy consensus that it was necessary to have more observations than explanatory variables in order to estimate a multivariate regression. I viewed segmentation analysis, so popular in marketing research, as a sort of diversion from the real task of predicting responses of customers directly, based on their demographics, firmagraphics, and other factors.

So the press of writing frequent posts on business forecasting and related topics has led me to a learn a lot.

The next post to this blog, for example, will be about how “bagging” – from Bootstrap Aggregation – can radically reduce forecasting errors when there are only a few historical or other observations, but a large number of potential predictors. In a way, this provides a new solution to the problem of forecasting in data limited contexts.

This post also includes specific computations, in this case done in a spreadsheet. I’m big on actually computing stuff, where possible. I believe Elliot Shulman’s dictum, “you don’t really know something until you compute it.” And now I see how to include access to spreadsheets for readers, so there will be more of that.

Forecasting turning points is the great unsolved problem of business forecasting. That’s why I’m intensely interested in analysis of what many agree are asset bubbles. Bursting of the dot.com bubble initiated the US recession of 2001. Collapse of the housing market and exotic financial instrument bubbles in 2007 bought on the worst recession since World War II, now called the Great Recession. If it were possible to forecast the peak of various asset bubbles, like researchers such as Didier Sornette suggest, this would mean we would have some advance – perhaps only weeks of course – on the onset of the next major business turndown.

Along the way, there are all sorts of interesting sidelights relating to business forecasting and more generally predictive analytics. In fact, it’s clear that in the era of Big Data, data analytics can contribute to improvement of business processes – things like target marketing for customers – as well as perform less glitzy tasks of projecting sales for budget formulation and the like.

Email me at [email protected] if you want to receive PDF compilations on topics from the archives. I’m putting together compilations on New Methods and Asset Bubbles, for starters, in a week or so.