Do Oil and Gas Futures Forecast Oil and Gas Spot Prices?

I’m looking at evidence that oil and gas futures are useful in forecasting future prices. This is an important for reasons ranging from investment guidance to policy analysis (assessing the role of speculators in influencing current market prices).

So – what are futures contracts, where are they traded, and where do you find out about them?

A futures contract (long position) is an agreement to buy an amount of a commodity (oil or gas) at a specified price at the expiration of the contract. The seller (the party with a short position) agrees to sell the underlying commodity to the buyer at expiration at the fixed sales price. Futures contracts can be traded many times prior to the expiration date.

At the expiration of the contract, if the price of the contract is below the market or spot price at that time, the buyer makes money. Futures contracts also can be used to lock in prices, and hedge risk.

The New York Mercantile Exchange (NYMEX) maintains futures markets for oil and gas. Natural gas futures are based on delivery at the Henry Hub, Louisiana, a major crossroads for natural gas pipelines.

So there are futures contracts for 1 month, 2 month, and so forth, delivery dates.

Evidence Futures Predict Spot Prices

As noted by Menzie Chinn, a popular idea is that the futures price is the optimal forecast of the spot price is an implication of the efficient market hypothesis.

Nevertheless, the evidence for futures prices being unbiased estimators of future spot prices is mixed, despite widespread acceptance of the idea in central banks and the International Monetary Fund (IMF).

A recent benchmark study, Forecasting the Price of Oil, finds –

some evidence that the price of oil futures has additional predictive content compared with the current spot price at the 12-month horizon; the magnitude of the reduction in mean-squared prediction error (MSPE) is modest even at the 12-month horizon, however, and there are indications that this result is sensitive to fairly small changes in the sample period and in the forecast horizon. There is no evidence of significant forecast accuracy gains at shorter horizons, and at the long horizons of interest to policymakers, oil futures prices are clearly inferior to the no-change forecast.

Here, the “no-change forecast” can be understood and is sometimes also referred to as a “random walk forecast.”

Both Chinn and the Forecasting the Price of Oil chapter in the Handbook of Forecasting are good places for readers to check the extensive literature on this topic.

Hands-On Calculation

Forecasting is about computation and calculation, working with real data.

So I downloaded the Contract1 daily futures prices from the US EIA, a source which also provides the Henry Hub spot prices.

Natural gas contracts, for example, expire three business days prior to the first calendar day of the delivery month. Thus, the delivery month for Contract 1 in the US EIA tables is the calendar month following the trade date.

Here is a chart from the spreadsheet I developed.


I compared the daily spot prices and 1 month futures contract prices by date to see how often the futures prices correctly indicate the direction of change of the spot price at the settlement or delivery date, three days prior to the first calendar day of the delivery month. So, the April 14, 2014 spot price was $4.64 and the Contract1 futures closing price for that day was $4.56, indicating that the spot price in late May would be lower than the current spot price. In fact, the May 27th spot price was $4.56. So, in this case, not only was the predicted direction of change correct, but also the point estimate of the future spot price.

The chart above averages the performance of these daily forecasts of the future direction of spot prices over rolling 20 trading day windows.

From January through the end of September 2014, these averages score better than 50:50 about 71 percent of the time.

I have not calculated how accurate these one month natural gas futures are per se, but my guess is that the accuracies would be close.

However, clearly, a “no-change forecast” is incapable of indicating the future direction of changes in the gas spot price.

So the above chart and the associated information structure are potentially useful regardless of the point forecast accuracy. My explorations suggest additional information about direction and, possibly, even turning points in price, can be extracted from longer range gas futures contracts.

Speculators and Oil Prices

One of the more important questions in the petroleum business is the degree to which speculators influence oil prices.


If speculators can significantly move oil spot prices, there might be “overshooting” on the downside, in the current oil price environment. That is, the spot price of oil might drop more than fundamentals warrant, given that spot prices have dropped significantly in recent weeks and the Saudi’s may not reduce production, as they have in the past.

This issue can be rephrased more colorfully in terms of whether the 2008 oil price spike, shown below, was a “bubble,” driven in part by speculators, or whether, as some economists argue, things can be explained in terms of surging Chinese demand and supply constraints.

James Hamilton’s Causes and Consequences of the Oil Shock of 2007–08, Spring 2009, documents a failure of oil production to increase between 2005-2007, and the exponential growth in Chinese petroleum demand through 2007.

Hamilton, nevertheless, admits “the speed and magnitude of the price collapse leads one to give serious consideration to the alternative hypothesis that this episode represents a speculative price bubble that popped.”

Enter hedge fund manager Michael Masters stage left.

In testimony before the US Senate, Masters blames the 2007-08 oil price spike on speculators, and specifically on commodity index trading funds which held a quarter trillion dollars worth of futures contracts in 2008.

Hamilton characterizes Masters’ position as follows,

A typical strategy is to take a long position in a near-term futures contract, sell it a few weeks before expiry, and use the proceeds to take a long position in a subsequent near-term futures contract. When commodity prices are rising, the sell price should be higher than the buy, and the investor can profit without ever physically taking delivery. As more investment funds sought to take positions in commodity futures contracts for this purpose, so that the number of buys of next contracts always exceeded the number of sells of expiring ones, the effect, Masters argues, was to drive up the futures price, and with it the spot price. This “financialization” of commodities, according to Masters, introduced a speculative bubble in the price of oil.

Where’s the Beef?

If speculators were instrumental in driving up oil prices in 2008, however, where is the inventory build one would expect to accompany such activity? As noted above, oil production 2005-2007 was relatively static.

There are several possible answers.

One is simply that activity in the futures markets involve “paper barrels of oil” and that pricing of real supplies follows signals being generated by the futures markets. This is essentially Masters’ position.

A second, more sophisticated response is that the term structure of the oil futures markets changed, running up to 2008. The sweet spot changed from short term to long term futures, encouraging “ground storage,” rather than immediate extraction and stockpiling of inventories in storage tanks. Short term pricing followed the lead being indicated by longer term oil futures. The MIT researcher Parsons makes this case in a fascinating paper Black Gold & Fool’s Gold: Speculation in the Oil Futures Market.

..successful innovations in the financial industry made it possible for paper oil to be a financial asset in a very complete way. Once that was accomplished, a speculative bubble became possible. Oil is no different from equities or housing in this regard.

A third, more conventional answer is that, in fact, it is possible to show a direct causal link from activity in the oil futures markets to oil inventories, despite the appearances of flat production leading up to 2008.

Where This Leads

The uproar on this issue is related to efforts to increase regulation on the nasty speculators, who are distorting oil and other commodity prices away from values determined by fundamental forces.

While that might be a fine objective, I am more interested in the predictive standpoint.

Well, there is enough here to justify collecting a wide scope of data on production, prices, storage, reserves, and futures markets, and developing predictive models. It’s not clear the result would be most successful short term, or for the longer term. But I suspect forward-looking perspective is possible through predictive analytics in this area.

Top graphic from Evil Speculator.

Oil and Gas Prices II

One of the more interesting questions in applied forecasting is the relationship between oil and natural gas prices in the US market, shown below.


Up to the early 1990’s, the interplay between oil and gas prices followed “rules of thumb” – for example, gas prices per million Btu were approximately one tenth oil prices.

There is still some suggestion of this – for example, peak oil prices recently hit nearly $140 a barrel, at the same time gas prices were nearly $14 per million Btu’s.

However, generally, ratio relationships appear to break down around 2009, if not earlier, during the first decade of the century.

A Longer Term Relationship?

Perhaps oil and gas prices are in a longer term relationship, but one disturbed in many cases in short run time periods.

One way economists and ecommetricians think of this is in terms of “co-integrating relationships.” That’s a fancy way of saying that regressions of the form,

Gas price in time t = constant + α(oil price in time t) + (residual in time t)

are predictive. Here, α is a coefficient to be estimated.

Now this looks like a straight-forward regression, so you might say – “what’s the problem?”

Well, the catch is that gas prices and oil prices might be nonstationary – that is, one or another form of a random walk.

If this is so – and positive results on standard tests such as the augmented Dickey Fuller (ADR) and Phillips-Peron are widely reported – there is a big potential problem. It’s easy to regress one completely unrelated nonstationary time series onto another, getting an apparently significant result, only to find this relationship disappears in the forecast. In other words two random series can, by chance, match up to each other over closely, but that’s no guarantee they will continue to do so.

Here’s where the concept of a co-integrating relationship comes into play.

If you can show, by various statistical tests, that variables are cointegrated, regressions such as the one above are more likely to be predictive.

Well, several econometric studies show gas and oil prices are in a cointegrated relationship, using data from the 1990’s through sometime in the first decade of the 2000’s. The more sophisticated specify auxiliary variables to account for weather or changes in gas storage. You might download and read, for example, a study published in 2007 under the auspices of the Dallas Federal Reserve Bank – What Drives Natural Gas Prices?

But it does not appear that this cointegrated relationship is fixed. Instead, it changes over time, perhaps exemplifying various regimes, i.e. periods of time in which the underlying parameters switch to new values, even though a determinate relationship can still be demonstrated.

Changing parameters are shown in the excellent 2012 study by Ramberg and Parsons in the Energy Journal – The Weak Tie Between Natural Gas and Oil Prices.

The Underlying Basis

Anyway, there are facts relating to production and use of oil and natural gas which encourage us to postulate a relationship in their prices, although the relationship may shift over time.

This makes sense since oil and gas are limited or completely substitutes in various industrial processes. This used to be more compelling in electric power generation, than it is today. According to the US Department of Energy, there are only limited amounts of electric power still produced by generators running on oil, although natural gas turbines have grown in importance.

Still, natural gas is often produced alongside of and is usually dissolved in oil, so oil and natural gas are usually joint products.

Recently, technology has changed the picture with respect to gas and oil.

On the demand side, the introduction of the combined-cycle combustion turbine made natural gas electricity generation more cost effective, thereby making natural gas in electric power generation even more dominant.

On the demand side, the new technologies of extracting shale oil and natural gas – often summarized under the rubric of “fracking” or hydraulic fracturing – have totally changed the equation, resulting in dramatic increases in natural gas supplies in the US.

This leaves the interesting question of what sort of forecasting model for natural gas might be appropriate.

Oil and Gas Prices – a “Golden Swan”?

Crude oil prices plummeted last week, moving toward $80/Bbl for West Texas Intermediate (WTI) – the spot pricing standard commodity.


OPEC – the Organization of Petroleum Exporting Counties – is a key to trajectory of oil prices, accounting for about 40 percent of global oil output.

Media reports that the Saudi Arabian Kingdom, which is the largest producer in OPEC, is advising that it will not cut oil production at the current time. The US Energy Information Agency (EIA) has a graph on its website underlining the importance of Saudi production to global oil prices.


Officially, there is very little in the media to pin down the current Saudi policy, although, off-the-record, Saudi representatives apparently have indicated they could allow crude prices to drift between $80 and $90 a barrel for a couple of years. This could impact higher cost producers, such as Iran and burgeoning North American shale oil production.

At the same time, several OPEC members, such as Venezuela and Libya, have called for cuts in output to manage crude prices going forward. And a field jointly maintained by Saudi Arabia and Kuwait just has been shut down, ostensibly for environmental upgrades.

OPEC’s upcoming November 27 meeting in Vienna, Austria should be momentous.

US Oil Production

Currently, US oil production is running at 8.7 million barrels a day, a million barrels a day higher than in a comparable period of 2013, and the highest level since 1986.

The question of the hour is whether US production can continue to increase with significantly lower oil prices.

Many analysts echo the New York Times, which recently compared throttling back US petroleum activity to slowing a freight train.

Most companies make their investment decisions well in advance and need months to slow exploration because of contracts with service companies. And if they do decide to cut back some drilling, they will pick the least prospective fields first as they continue developing the richest prospects.

At the same time, the most recent data suggest US rig activity is starting to slip.

Economic Drivers

It’s all too easy to engage in arm-waving, when discussing energy supplies and prices and their relationship to the global economy.

Of course, we have supply and demand, as one basis. Supplies have been increasing, in part because of new technologies in US production and Libyan production coming back on line.

Demands have not been increasing, on the other hand, as rapidly as in the past. This reflects slowing growth in China and continuing energy conservation.

One imponderable is the influence of speculators on oil prices. Was there a “bubble” before 2009, for example, and could speculators drive oil prices significantly lower in coming months?

Another factor that is hard to assess is whether 2015 will see a recession in major parts of the global economy.

The US Federal Reserve has been planning on eliminating Quantitative Easing (QE) – its program of long-term bond purchases – and increasing the federal funds rate from its present level of virtually zero. Many believe that these actions will materially slow US and global economic growth. Coupled with the current deflationary environment in Europe, there have been increasing signs that factors could converge to trigger a recession sometime in 2015.

However, low energy prices usually are not part of the prelude for a recession, although they can develop after the recession takes hold.

Instead, prices at the pump in the US could fall below $3.00 a gallon, providing several hundred dollars extra in discretionary income over the course of a year. This, just prior to the Christmas shopping season.

So – if US oil production continues to increase and prices at the pump fall below $3.00, there will be jobs and cheap gas, a combination likely to forstall a downturn, at least in the US for the time being.

Top image courtesy of GameDocs

Forecasting in the Supply Chain

The Foresight Practitioner’s Conference held last week in on the campus of Ohio State University highlighted business gains in forecasting and the bottom line from integration across the supply chain.

Officially, the title of the Conference was “From S&OP to Demand-Supply Integration: Collaboration Across the Supply Chain.”
S&OP is an important practice in many businesses right now – Sales and Operations Planning. By itself it signifies business integration, but several speakers – starting off with Pete Alle of Oberweiss Dairy – emphasized the importance of linking the S&OP manager with the General Manager directly, and of his sponsorship and support.

Luke Busby described revitalization of an S&OP process for Steris – a medical technology leader focusing on infection prevention, contamination control, surgical and critical care technologies. Problems encountered were that the old process was spreadsheet driven, used minimal analytics, led to finger pointing – “Your numbers!”, was not comprehensive – not all products and plants included, and embodied divergent goals.

Busby had good things to say about software called Smoothie from Demand Works in facilitating the new Steris process. Busby described benefits from the new implementation at a high level of detail, including the ability, for example, to drill down and segment the welter of SKU’s in the company product lines.

I found the talk especially interesting because of its attention to organization detail, such as shown in the following slide.


But this was more than an S&OP Conference, as underlined by Dr. Mark A. Moon’s presentation From S&OP to True Business Integration. Moon, Head, Department of Marketing and Supply Chain Management, University of Tennessee, Knoxville, started his talk with the following telling slide –


Glen Lewis of the University of California at Davis and formerly a Del Monte Director spoke on a wider integration of S&OP with Green Energy practices, focusing mainly on time management of peak electric power demands.

Thomas Goldsby, Professor of Logistics Fisher College of Business who introduced the concept of the supply web (shown below), and co-presented with Alicia Hammersmith, GM for Materials, General Electric Aviation. I finally learned what 3D printing was.


Probably the most amazing part of the Conference for me was the Beer Game, led by James Hill, Associate Professor of Management Sciences at The Ohio State University Fisher College of Business. Several tables were set up in a big auditorium in the Business School, each with a layout of production, product warehousing, distributor warehouses, and retail outlets. These four positions were staffed by Conference attendees, many expert in supply chain management.

The objective was to minimize inventory costs, where shortfalls earned a double penalty. No communication was permitted along these fictive supply chains for beer. Demand was unknown at retail, but when discovered resulted in orders being passed back along the chain, where lags were introduced in provisioning. The upshot was that every table created the famous “bullwhip effect” of intensifying volatility of inventory back along the supply chain.

Bottom line was that if you want to become a hero in an organization short-term, find a way to reduce inventory, since that results in immediate increases in cash flow.

All very interesting. Where does forecasting fit into this? Good question, and that was discussed in open sessions.

A common observation was that relying on the field sales teams to provide estimates of future orders can lead to bias.

Video Friday on Steroids

Here is a list of the URL’s for all the YouTube and other videos shown on this blog from January 2014 through May of this year. I encourage you to shop this list, clicking on the links. There’s a lot of good stuff, including several  instructional videos on machine learning and other technical topics, a series on robotics, and several videos on climate and climate change.

January 2014

The Polar Vortex Explained in Two Minutes

NASA – Six Decades of a Warming Earth

“CHASING ICE” captures largest video calving of glacier

Machine Learning and Econometrics

Can Crime Prediction Software Stop Criminals?

Analytics 2013 – Day 1

The birth of a salesman

Economies Improve

Kaggle – Energy Applications for Machine Learning

2014 Outlook with Jan Hatzius

Nassim Taleb Lectures at the NSF

Vernon Smith – Experimental Markets



Forecast Pro – Quick Tour

February 2014

Stephen Wolfram’s Introduction to the Wolfram Language


Econometrics – Quantile Regression

Quantile Regression Example

Brooklyn Grange – A New York Growing Season

Getting in Shape for the Sport of Data Science

Machine Learning – Decision Trees

Machine Learning – Random Forests

Machine Learning – Random Forecasts Applications

Malcolm Gladwell on the 10,000 Hour Rule

Sornette Talk

Head of India Central Bank Interview

March 2014

David Stockman

Partial Least Squares Regression

April 2014

Thomas Piketty on Economic Inequality

Bonobo builds a fire and tastes marshmellows

Future Technology

May 2014

Ray Kurzweil: The Coming Singularity

Paul Root Wolpe: Kurzweil Critique

The Future of Robotics and Artificial Intelligence

Car Factory – KIA Sportage Assembly Line

10 Most Popular Applications for Robots

Predator Drones

The Future of Robotic Warfare

Bionic Kangaroo

Ping Pong Playing Robot

Baxter, the Industrial Robot


Stylized Facts About Stock Market Volatility

Volatility of stock market returns is more predictable, in several senses, than stock market returns themselves.

Generally, if pt is the price of a stock at time t, stock market returns often are defined as ln(pt)-ln(pt-1). Volatility can be the absolute value of these returns, or as their square. Thus, hourly, daily, monthly or other returns can be positive or negative, while volatility is always positive.

Masset highlights several stylized facts about volatility in a recent paper –

  • Volatility is not constant and tends to cluster through time. Observing a large (small) return today (whatever its sign) is a good precursor of large (small) returns in the coming days.
  • Changes in volatility typically have a very long-lasting impact on its subsequent evolution. We say that volatility has a long memory.
  • The probability of observing an extreme event (either a dramatic downturn or an enthusiastic takeoff) is way larger than what is hypothesized by common data generating processes. The returns distribution has fat tails.
  • Such a shock also has a significant impact on subsequent returns. Like in an earthquake, we typically observe aftershocks during a number of trading days after the main shock has taken place.
  • The amplitude of returns displays an intriguing relation with the returns themselves: when prices go down – volatility increases; when prices go up – volatility decreases but to a lesser extent. This is known as the leverage effect … or the asymmetric volatility phenomenon.
  • Recently, some researchers have noticed that there were also some significant differences in terms of information content among volatility estimates computed at various frequencies. Changes in low-frequency volatility have more impact on subsequent high-frequency volatility than the opposite. This is due to the heterogeneous nature of market participants, some having short-, medium- or long-term investment horizons, but all being influenced by long-term moves on the markets…
  • Furthermore, … the intensity of this relation between long and short time horizons depends on the level of volatility at long horizons: when volatility at a long time horizon is low, this typically leads to low volatility at short horizons too. The reverse is however not always true…

Masset extends and deepens this type of result for bull and bear markets and developed/emerging markets. Generally, emerging markets display higher volatility with some differences in third and higher moments.

A key reference is Rami Cont’s Empirical properties of asset returns: stylized facts and statistical issues which provides this list of features of stock market returns, some of which directly relate to volatility. This is one of the most widely-cited articles in the financial literature:

  1. Absence of autocorrelations: (linear) autocorrelations of asset returns are often insignificant, except for very small intraday time scales (~20 minutes) for which microstructure effects come into play.
  2. Heavy tails: the (unconditional) distribution of returns seems to display a power-law or Pareto-like tail, with a tail index which is finite, higher than two and less than five for most data sets studied. In particular this excludes stable laws with infinite variance and the normal distribution. However the precise form of the tails is difficult to determine.
  3. Gain/loss asymmetry: one observes large drawdowns in stock prices and stock index values but not equally large upward movements.
  4. Aggregational Gaussianity: as one increases the time scale t over which returns are calculated, their distribution looks more and more like a normal distribution. In particular, the shape of the distribution is not the same at different time scales.
  5. Intermittency: returns display, at any time scale, a high degree of variability. This is quantified by the presence of irregular bursts in time series of a wide variety of volatility estimators.
  6. Volatility clustering: different measures of volatility display a positive autocorrelation over several days, which quantifies the fact that high-volatility events tend to cluster in time.
  7. Conditional heavy tails: even after correcting returns for volatility clustering (e.g. via GARCH-type models), the residual time series still exhibit heavy tails. However, the tails are less heavy than in the unconditional distribution of returns.
  8. Slow decay of autocorrelation in absolute returns: the autocorrelation function of absolute returns decays slowly as a function of the time lag, roughly as a power law with an exponent β ∈ [0.2, 0.4]. This is sometimes interpreted as a sign of long-range dependence.
  9. Leverage effect: most measures of volatility of an asset are negatively correlated with the returns of that asset.
  10. Volume/volatility correlation: trading volume is correlated with all measures of volatility.
  11. Asymmetry in time scales: coarse-grained measures of volatility predict fine-scale volatility better than the other way round.

Just to position the discussion, here are graphs of the NASDAQ 100 daily closing prices and the volatility of daily returns, since October 1, 1985.


The volatility here is calculated as the absolute value of the differences of the logarithms of the daily closing prices.