The use of factual/counterfactual comparisons—that is, estimating the difference between what actually happened and what would have happened absent an event—is a fundamental tool in the economists’ work box in the context of contractual or regulatory disputes.1 If the event being analyzed has an impact on the input or output prices of energy market participants, determining what market prices would have been absent the event is a significant challenge that calls for expert industry knowledge and associated analyses. Such knowledge allows for the construction of robust models to parse the factual/counterfactual differences, and for understanding their impact on the recovery in contracts between involved parties or in discussions with regulators or other stakeholders. This article discusses some methodological considerations in creating and selecting an appropriate model to estimate counterfactual prices in energy markets.
Importance of Counterfactual Analysis in Energy Markets
Over the past decades, there are numerous examples of the importance of estimating how an energy market would have behaved absent an event. A recent such case is the June 2023 settlement process at the Federal Energy Regulatory Commission (FERC) involving penalties assessed by PJM Interconnection LLC (PJM), the largest U.S. grid operator, during Winter Storm Elliott.2 While the process is confidential, it is difficult to imagine that some form of counterfactual/but-for analyses is not informing a process where $1.8 billion is at stake. The importance of getting the counterfactual “right” is obvious in situations like this, and the process can involve modeling or otherwise considering a range of issues including, among others, fuel procurement, pricing, participant bidding and behavior, and the decision-making of regional transmission organizations (RTOs) in a counterfactual world.
Other instances where counterfactual estimation is relevant include local weather events, such as hurricanes and winter storms, which can have widespread impacts by affecting key production/consumption areas and pricing benchmarks (e.g., Hurricane Katrina, Winter Storm Uri).3 Even when they do not impact widely-used benchmarks, local weather events can have substantial implications on segmented markets where pricing is more localized—for instance, the effect of Winter Storm Elliot mentioned above on many PJM market participants.4
There are also more global, systemic events that can have a profound impact on multiple energy markets in very distinct geographical locations—recent examples include the COVID-19 pandemic and the Russian invasion of Ukraine in 2022. Many contractual and regulatory disputes may call for an objective, quantitative way to estimate the impact of such events in energy markets. In fact, virtually every time the prudence, propriety, lawfulness, or tariff compliance of a party is questioned and some form of adjustment or relief is requested, the generic fundamental tool used by economists is a counterfactual analysis.
Estimating Counterfactual Prices – Constructing Potential Models
Estimating what the price would have been in a given energy market had a certain event not occurred is a challenging exercise. Energy markets are inherently hard to model. Prices are often seasonal, volatile, prone to regime shifts, dependent on many different factors (including the prices of other energy sources), and—because supply and demand have limited responsiveness to changes in prices, particularly in the short-run—very sensitive to shocks. The overlay of business factors, state and federal regulatory regimes and rules, and the international nature of the financial and physical markets for some of the factors of production make this type of analysis in energy markets particularly challenging.5
However, with care and expertise, appropriate quantitative models can be developed to estimate counterfactual prices in many settings. A key first step is a clear articulation of the objective and metric to be used for assessing the appropriateness of each model—for instance, assessing whether the predictive ability of the counterfactual model is the most important element and, if so, determining the minimum requirements to provide a set of objective and defensible estimates.
The second step is specifying a set of models that may be able to answer the question being posed. Modeling prices in an economic context involves identifying supply and demand factors that are expected to impact prices, so that market dynamics are captured in a realistic way. While many market forces are common across energy markets, other market forces are unique or apply in unique ways to specific markets. For example, oil is one of the most integrated commodity markets, and prices across the globe typically move in tandem with reasonable predictability. Natural gas has been traditionally a more localized market but is now increasingly less so as the development of liquefied natural gas (LNG) markets results in increased international integration. Electricity prices depend on oil and natural gas prices, among others, as inputs for generating power. However, because of transportation constraints and requirements to meet demand in a timely fashion, the electricity market can be characterized as a local market, at least for very precise locational pricing. Gasoline markets are also more localized than oil markets, although a network of pipelines helps transmit price impacts over wider geographies. Coal prices are also generally dependent on location and, given coal’s role in electricity production, can interact with natural gas and electricity prices.
An additional step that can be important in modeling counterfactual prices is to develop an understanding of the channels through which the event in question is likely to impact prices. In this way, the quantitative modeling can be made independent of the event in question or checks can be added to ensure robustness of results. For example, if one expects a storm to affect prices by reducing the available capacity to produce or extract a certain commodity, it may be desirable that the quantitative models used to estimate the counterfactual prices do not rely on the actual available capacity during the event as an explanatory factor when attempting to model but-for prices.6 Rather, depending on the question at issue, the analysis developing the counterfactual scenario might need to account for the higher available capacity absent the storm.
The construction of pricing models can be informed and supported by a careful qualitative assessment of market and industry commentary. In particular, developing an understanding of the event based on qualitative sources can be valuable to ensure that the quantitative modeling exercise is being performed correctly and that the counterfactual estimates are credible and consistent with the existing evidence. In other words, both statistical and reasoned validation should be performed to support the models being considered.
Letting the Data Guide the Selection of a Pricing Model for Counterfactual Prices
One of the main challenges that economists face is the multitude of specifications that can conceivably be used when modeling prices. First, many variables can be included in the model, and typically some degree of data limitation exists. Second, there are different ways in which a given variable can be defined. Third, there may be a time lag in how certain variables affect prices that an economist may want to account for.
For example, when estimating natural gas prices in a given month, should one account for the quantity of gas that is in storage, net change of storage positions (time series) or the percentage utilization of storage facilities? Should one look at storage levels, whichever way they are defined, the month before, or the quarter before? What locational “breadth” should be used in considering storage, and storage constraints? Should one include industrial production and other economic indicators as explanatory variables, and if so, how should they be selected? What are the practical and theoretical limitations of the analytic approach, specification, and available data, and how can they be satisfied? There are many decisions to be made for which there is not always a clear theoretical justification.
Having created a large number of models and specifications grounded in economic principles, an economist can let the data inform the choice of the “best” model according to a selection criteria. Specifically, if there are enough historical data, one can test those models over a time period that was not used in the estimation (so-called out of sample testing) to see which model performs best when its predictions are compared to what actually happened.7 The model that is selected in this way can then be used to predict prices in the counterfactual period.8
This approach, although well-grounded in theory and econometric practice, is not a panacea. One needs to ensure that meaningful economic relations are being captured, and that the selected model is not just an artifact of data-snooping. Again, this is where a qualitative assessment of the results and a careful comparison to market and industry commentary can be valuable. Further, an underlying assumption is that the economic relationships captured by the model do not change fundamentally between the estimation or testing period and the period of interest when counterfactual prices are estimated; such concerns can be assuaged if the periods are not distant in time. Finally, while typically the primary function of such model is to estimate the impact of an event by calculating a but-for price, that is not its only function. It can also be used to estimate the duration of an event’s impact—when the but-for price gets close to the price levels actually observed in the market, this can indicate that the event that originally caused the price dislocation is no longer having a material impact on the markets.9
Conclusion
A carefully executed estimation of counterfactual prices, that is, prices that would be expected to prevail absent a given event, can be valuable in the context of commercial and regulatory disputes in the energy space. While a number of challenges can arise when performing such analysis, this article discusses one potential approach to arrive at an unbiased, robust set of counterfactual price estimates.
1For example, claims under business interruption insurance, or disputes around force majeure or acts of God/war clauses in contracts, may require estimating what would have been the earnings of a certain operation in the absence of an event.
2Keith Goldberg, “FERC Seeks Deal in Grid Operator’s Storm Penalty Fight,” Law 360, June 6, 2023; “FERC to Oversee Settlement Talks to Resolve PJM Winter Storm Elliott Complaints amid $1.8b in Charges,” Utility Dive, June 7, 2023, https://www.utilitydive.com/news/ferc-pjm-winter-storm-elliott-settlement-complaints-calpine-vistra/652282/.
3See, e.g., Kivanc Kirgiz, Michelle Burtis, and David A. Lunin, “Petroleum-Refining Industry Business Interruption Losses Due to Hurricane Katrina,” Journal of Business Valuation and Economic Loss Analysis 4, no. 2 (2009).
4See, e.g., “FERC to Oversee Settlement Talks to Resolve PJM Winter Storm Elliott Complaints amid $1.8b in Charges,” Utility Dive, June 7, 2023, https://www.utilitydive.com/news/ferc-pjm-winter-storm-elliott-settlement-complaints-calpine-vistra/652282/.
5A further complication can be the need to parse these effects into different “buckets” because of differentiated regulatory and contractual requirements, as different sources of impact can result in different allocations and recovery of costs and expenses among the impacted parties.
6In economics, an explanatory variable that is impacted by the effect that one intends to study is called endogenous. In this case, introducing an endogenous variable such as available capacity in the model predicting the counterfactual could result in that variable capturing some of the effect of the storm on prices, thereby reducing the impact that would be estimated when comparing the factual and counterfactual scenarios.
7In economics, an explanatory variable that is impacted by the effect that one intends to study is called endogenous. In this case, introducing an endogenous variable such as available capacity in the model predicting the counterfactual could result in that variable capturing some of the effect of the storm on prices, thereby reducing the impact that would be estimated when comparing the factual and counterfactual scenarios.
8
Most models rely on a linear relation between the dependent variable (i.e., the price being modeled) and a set of explanatory variables. Examples include pricing forecasting models used by many utilities. However, it can be appropriate to use more complex models that capture dynamic relations in a more explicit manner. Vector Autoregression (VAR) models allow for several relations to be modeled at the same time (e.g., natural gas prices and storage), and for contemporaneous and past realizations of one of the variables to impact the other (e.g., past storage levels impacting natural gas prices today). The testing itself may be made more robust by selecting varying out-of-sample data ranges as well as varying the underlying in-sample data used to estimate the models being tested.
9Note that this is not always the case. In some instances, the impact may actually be expected to reverse as a consequence of the economic dynamics associated with the event being analyzed. In other cases, there may be additional factors impacting observed prices that are not modeled in the counterfactual estimation.
The views expressed herein are solely those of the authors and do not necessarily represent the views of Cornerstone Research.
