A (preliminary) Intermediate Guide to How Prices are Set in the NEM

Back on 3rd August 2018, in a group effort involving a number in our team we posted  this “Beginner’s Guide to How Dispatch Works, and how Prices are Set in the NEM”.

It’s critical that readers understand that (as noted here) the NEM works with price-based dispatch where the marginal unit(s) set the price received by all generators.  As guest author Allan O’Neil noted here on January 11th:

“Fundamental to marginally priced markets is that prices are set by the last supply or demand reduction offer dispatched in order to balance supply and demand, after (in principle) arranging the volumes offered in order of increasing price. So it is the pricing of offers that fall at around the total demand level on the cumulative bid stack that is critical, rather than the average level of pricing across all market offers. ”

… so Allan goes on to say that:

“With the current supply-demand balance in the NEM, gas and hydro generators are frequently the marginal price setting plant”

… you can review Allan’s article, which focused more specifically on why prices through 2018 remained stubbornly high.

——————–

In the 6 months that have passed since our Beginner’s Guide was published, we have received numerous queries about when we might be posting the next edition – the Intermediate Guide, if you like.

In this post, we endeavour to to part of this, in that:

1)  We hope we deliver, in the following, a clear enough explanation of why the NEM operates (in the “real world”) at a much more complex level than that covered in the Beginner’s Guide.

1a)  In other words, we hope you understand, through this post, that reality is much more complex than simple theory.
1b)  Unfortunately, much of the discussion we’ve seen (e.g. on social media) about Price Setter seems to exist only down at that Beginner’s Guide level.

2)  However note that it’s not intended in this post that we actually fully explain all the nuances in these complexities.  We don’t have time to do this currently, but hope to return later to provide more of a fuller Intermediate Guide at some point in the months ahead (certainly after we’ve completed our Generator Report Card 2018, and the work that will flow on from that into our products like ez2view and NEMreview).

In recent times we’ve been fielding a growing number of calls from various people asking to know more about “who’s been setting the price” in the electricity market.  In part these questions have been spurred by the AEMO’s inclusion of trends in various reports (such as Figure 5 and Figure 6 in their Q4 2018 Quarterly Energy Dynamics report*)

As an aside, we’d like to note that these Quarterly Reviews are an excellent initiative, in our view.  Thanks to AEMO for delivering something that had been missing beforehand!  It seems pitched at about the right level (given the types of questions we keep hearing from new entrants, spectators and commentators).

Various others (such as Allan here) have also been referring to stats focused answering the same basic question.   These various analysts and publishers all seem to be basing their trends on interpretation of the data published in AEMO’s Price Setter files.

(A)  An introduction to AEMO’s Price Setter file

The AEMO started publishing these Price Setter files a number of years ago in response to Registered Participant requests to understand more detail about how NEMDE was actually determining price – because the price setting process is more complex than the simple bid stack analogies in our Beginner’s Guide would suggest.

These Price Setter files do a reasonable* job at explaining the multi-dimensional complexities involved in setting 45 prices in every dispatch interval (i.e. price for 5 regions, for Energy and 8 FCAS commodities).   However they are not simple.

Noting (only?) a “reasonable” job is not meant as a criticism – just a reflection that it’s actually a whole lot more complex than we might initially think, if our thinking is at Beginner’s Guide level – as I hope this article will illustrate quantitatively…..

We’ve been including these Price Setter files in our ez2view software (used by wholesale participants and others who have significant exposure to the wholesale market in some form) for a number of years, enabling clients to drill into particular dispatch intervals forensically to understand the specifics of what went on in those dispatch intervals of high interest.

However in more recent times, we have been investigating how else we might use these Price Setter files in other ways to help expand what we do in making complexity understandable to our broad range of clients:

1)  For a start, we are calculating (and trending) a measure of Aggregated “RRP Contribution Factor” on an individual DUID basis for inclusion inside our Generator Report Card 2018.  This work is currently underway (but not yet completed) so I can’t be definitive about specifically what has been done.

2)  Following from the completion of the Generator Report Card 2018, we will be taking the lessons we’ve learnt in dealing with a complete history of Price Setter data files over a number of years and applying these lessons in the enhancement of our existing software products in a number of ways:

2a)  For our ez2view (higher-end) product, we will be enhancing in some considerable depth – making full use of these insights.
2b)  Reflecting the more generalised analysis completed by our NEMreview clients (who tend more to be analysts sitting on the outside of the wholesale market, looking in) we will be applying the learnings we have made, but also striving to retain a level of clarity reflecting the fact that these clients may never have been involved within the wholesale market directly, so quite unfamiliar with Price Setter (and the complexities noted in this preliminary Intermediate Guide).

In this post I hope we clarify some of our reasoning in what we are doing.

If you have specific questions about the increasing number of ways in which we’re utilizing the raw data published in AEMO’s Price Setter files, please call us (+61 7 3368 4064) or use this Feedback Form.

(B)  Categories of Complexity in Price Setter

In the process of assembling the historical series of price setter files for our Generator Report Card 2018, we* have defined the following categories for getting some statistical sense of the level of complexity involved in the price setting process.

*  As we have been the ones to define the categories, readers should be aware that it’s not an “official” categorisation in that sense, and that someone else might have a somewhat different division of outcomes per dispatch interval and region (as we might ourselves, on a different day).  The purpose of this categorisation exercise is merely just to help explain the scale of the complexity, in terms of numbers of dispatch intervals involved.

Readers should be aware that we’re intending these categories to be used only for ENERGY prices (not any FCAS commodity), for each dispatch in each separate region.

Category Description
(1) Very Simple In this Very Simple category, we assume that the price for ENERGY in a Region is set equal to the marginal bid price band for ENERGY of a single DUID in that Region.

This “Very Simple” category is that most aligned with the level of explanation (and thinking) suited to our Beginner’s Guide – and also elsewhere at beginner’s level.

See below to see how few dispatch intervals actually fit into this simple model.  Based on discussions with many people in recent times, our suspicion is that this will be a surprise for a significant number of people.

(2) Simple As you’ll note below, there are well more than half the dispatch intervals that don’t fall into the “Very Simple” category.

On a reasonable number of occasions, this has been primarily because it’s more than one DUID involved in setting the price – where still:
• It’s only offers for Energy involved; and where
• It’s multiple DUIDs within a single station involved; and
• The DUIDs must be in the region in question; but where
• The RRP (Regional Reference Price) is just the ROP (Regional Override Price)

(3) Slightly Complex In the next category we take another step towards complexity, in that we look for cases where the Energy Price (in that region):
• Has a number of contributing DUIDs, but from different Stations;
• Which then might be in different Regions; but still
• Only offers for Energy; and still where
• The RRP (Regional Reference Price) is just the ROP (Regional Override Price)
.
.It’s important to note in this category that it’s almost certain, where multiple stations are involved, that the relative contributions of the DUIDs across these stations won’t be equal – and it might be that some make a fairly dominant contribution whilst others make smaller contributions to the ROP.
(4) Complex Next step along the complexity journey is where we acknowledge that:
• The Price for Energy might be set, at least in part, by Bids made for FCAS Services (i.e. because of co-optimisation).
• In this case it’s almost certain there will be Multiple stations involved (but only DUIDs involved in setting the price, as distinct from Category 5); and still where
• The RRP (Regional Reference Price) is just the ROP (Regional Override Price).
.
It’s been our experience that, in numerous conversations, that the co-optimisation process (and hence the effect that FCAS bids can have on the energy price) is frequently overlooked or forgotten.
(5) Very Complex This category will catch all the Dispatch Intervals not covered in the category above, and hence will include
• Instances where the Regional Reference Price is adjusted from the ROP because of a range of different reasons (e.g. Price is Capped, there is Load Shedding etc); and/or where
• Other factors (not just DUIDs) are involved in setting the price.

 

 

(C)  Examples of these Categories

Applying these categories above to the Price Setter files for all Regions for all Dispatch Intervals from 1st Jan 2018 through until sometime on 5th February 2019 reveals an informative distribution that is different for different regions, as follows:

Category QLD Region NSW Region VIC Region TAS Region SA Region
(1) Very Simple 4296 DIs

3.7% of time

11,469 DIs

10.0% of time

21,471 DIs

18.6% of time

47,503 DIs

41.2% of time

9,674 DIs

8.4% of time

(2) Simple 34,893 DIs

30.2% of time

30,540 DIs

26.5% of time

4,992 DIs

4.3% of time

0 DIs

0% of time

10,493 DIs

9.1% of time

(3) Slightly Complex 54,799 DIs

47.6% of time

52,901 DIs

45.9% of time

64,939 DIs

56.4% of time

16,427 DIs

14.1% of time

70,217 DIs

60.9% of time

(4) Complex 21,215 DIs

18.4% of time

20,272 DIs

17.6% of time

23,609 DIs

20.5% of time

51,476 DIs

44.7% of time

24,643 DIs

21.4% of time

(5) Very Complex 43 DIs

<1% of time

64 DIs

<1% of time

232 DIs

<1% of time

18 DIs

<1% of time

216 DIs

<1% of time

 

A couple of things jump out at me that I’d like to highlight for a start:

1)  For a start, we see a significant percentage of time (roughly 20% in mainland regions, and a massive 45% in Tassie) where bids to supply FCAS services are involved in the price set for Energy.

2)  In Category 1 for NSW and VIC we see a higher percentage of time than is the case for QLD.  Without looking in detail at the data, which is still to be done for the Generator Report Card 2018, my suspicion is that this is because the Snowy Hydro stations are operating as single DUID stations.  Similar scenario for most of the hydro generators in Tasmania.

3)  We see in all regions that more than 58% of the time (80% of the time in SA), the Price in that region is set by contributions from more than one Station.  Or, to flip that around, it’s significantly less than 40% of the time in all of the mainland regions that the price is set in the “Simple” or “Very Simple” way we talk about in the Beginner’s Guide (and which seems to be the common, but over-simplistic, understanding* of the average “NEM Spectator and Commentator”).

*  That’s an example of Villain no 4 in action, and an example where (I hope) this article will go some way to helping alleviate

To illustrate each of the 5 categories above, we’ve picked a single Region/DI combination from recent weeks to provide an example of what we mean in the categories – and used Time Travel in ez2view to dig forensically as follows:

(C1)  Example of Category 1 – Very Simple

It’s sometimes noted that us Queenslanders like the simpler life – though I note that it’s only 3.7% of the time that we see a “Very Simple” outcome in terms of a single DUID in the Queensland region setting the Energy Price for Queensland.

I thought I would start by highlighting one of the relatively rare examples that have occurred over the past 14 months – focusing specifically on the 22:00 dispatch interval back on 4th February – highlighted in this snapshot from ez2view in Time Travel mode:
Click on image for a full-screen view.

2019-02-04-at-22-00-ez2view-PriceSetter-Category1

In the case shown here, we see a case where a bid from Swanbank E sets the price directly in the Queensland region (Category 1 = Very Simple), and also propagates through to the other mainland regions to play the sole role (with dynamic inter-regional losses) in setting the price in NSW, VIC and SA (not Category 1 in those cases!)

(C2)  Example of Category 2 – Simple

Stepping up the complexity (though only slightly) to the “Simple” category – we expand the filter to allow multiple DUIDs at one station to set the price.  As an example of this we roll the clock forward a few hours to 03:00 on Tuesday 5th February 2019 in Victoria, when we see the following:
Click on image for a full-screen view.

2019-02-05-at-03-00-ez2view-PriceSetter-Category2

As seen in the table above, there’s a reasonable number of instances (close to 40% in some regions) where Price Setter outcomes are at this level, or Very Simple.

However that leaves a large number of instances (the majority!) that are not….

(C3)  Example of Category 3 – Slightly Complex

Given that I already invested the time to dig into the reasons why the price spiked in NSW unexpectedly in the 16:15 dispatch interval on Thursday 31st January 2019, I’m saving myself some time by linking into this here.

In that article, I included this snapshot from ez2view that highlighted (on the Price Setter widget) how Upper Tumut and Torrens B (units 1 and 2) were both involved in setting the price in NSW for that dispatch interval.  That’s:
(a)  Two different fuel types both involved – Hydro, and Gas
(b)  Units in two different regions involved – NSW, and SA via the interconnected NEM.

For those who like to look at aggregated data for “how much the [FuelTypeIHateorLike] has set the price”, there’s an open question about how you would classify this one.  Remember that DI outcomes are in this category or worse for >58% of the time.  Genuinely happy to take Your Suggestions on that one!

That particular example is also a good one to highlight how two things can be simultaneously true:

True #1 = the Price Setter file nominates both Torrens and Upper Tumut as being marginal units, and hence their bids setting the price; however

Also True #2 = my earlier article also highlighted a number of other factors contributing to why the price was what it was  (demand up, intermittent supply down, imports dropped (due in part to bidding behaviour, one of the comments notes), reduced capacity at Mt Piper 2, and the list would go on…).

Readers should take this as a note of caution about reading too much into Price Setter in isolation to all the other factors that contribute to making the price outcome what it is on all occasions.

(C4)  Example of Category 4 – Complex

As shown in the table above, a staggering 44% of the time we see that price outcomes for Energy in TAS are in the Complex category (for instance, that FCAS bids are involved in the way the Energy price is set).  Hence we thought it would be useful to highlight one of those particular cases – being 23:05 on 16th January 2019 via Time Travel in ez2view:
Click on image for a full-screen view.

2019-01-16-at-23-05-ez2view-PriceSetter-Category4

As per the annotation, there’s not enough time now to explain all of what went on within that dispatch interval for the price of Energy in Tassie, but three highlights are:

(a)  There’s no contribution in Price Setter from any ENERGY offers in Tassie (or Energy Supply anywhere!) to setting the Tassie Energy Price;

(b)  The bid by Hornsdale Power Reserve to suck power out of the grid as a load (in SA) plays a big role in setting the price for Energy in Tassie (and note that the contribution is more than the price ends up being).

(c) There are numerous FCAS bids (in Tassie and elsewhere) that are involved in making contributions to the Tassie Energy Price – some positive and some negative contributions, with the net effect being that it reduces the Total Band Cost back down towards the final RRP.

Yep, price setting can be Complex!  And it is so for close to 20% of all dispatch intervals on the mainland (44% of the time in TAS) …

(C5)  Example of Category 5 – Very Complex

… but in a (thankfully small) number of cases (<1% of the time in each region) it becomes even more complex than the above.

We see a total of 216 dispatch intervals for South Australia in the table above, so I have singled out one of these for the example here – being 15:20 on 29th January 2019 via Time Travel in ez2view:
Click on image for a full-screen view.

2019-01-29-at-15-20-ez2view-PriceSetter-Category5

I certainly don’t have time to explain all of what’s happening in this Very Complex dispatch interval for Price Setter in the South Australian region.

(D)  “Who’s involved in setting the price?”  is larger than just Price Setter

Readers should exercise caution, however, in not misconstruing what the “Price Setter” files mean.

For instance, they do not mean that these were the only factors involved in setting the price where it was.  I alluded to this in my comments about Category 3 above, and the example of why the price spiked in NSW unexpectedly in the 16:15 dispatch interval on Thursday 31st January 2019.

Hence please take caution to not become myopic in an obsessive scrutiny of “Price Setter” files whilst neglecting the broader factors that can all contribute to why a price outcome is what it is!

———

We’ll certainly be trying to provide that broader perspective in our Generator Report Card 2018.  Now back to the analysis for that…


About the Author

Paul McArdle
One of three founders of Global-Roam back in 2000, Paul has been CEO of the company since that time. As an author on WattClarity, Paul's focus has been to help make the electricity market more understandable.

8 Comments on "A (preliminary) Intermediate Guide to How Prices are Set in the NEM"

  1. Looks like how someone would explain economics or something that is really is too complex to work properly.
    Just my opinion.
    It would seem that the system has lost its way on price. Things used to be far more stable and the price much lower.
    Day by day its getting more and more all over the shop and the price just keep creeping up, yet i read how well everything is being run, and AEMO is doing such a wonderful job.

    • Agree on the complexity, not sure why there are separate prices in each when all states are connected.

      I think criticism of AEMO is a tough call, since they just have to deal with the policy, without being able to influence it or influence any investment decisions. Although I believe the CEO panders to certain interest groups unnecessarily and against the intent of the AEMO reports that are escalating the language around reliability and cost.

      There is no indication that cost of generation will come down at any stage.

      • Sorry Ben
        Did you read into something I’d written in this article that was a criticism of AEMO?
        Paul

      • Hello Ben,
        Who then has the control over the price. Someone must have some kind of input into it, I am saying that is AEMO, they are the Pilot.

        • John,
          On what basis are you saying AEMO is controlling the price? Have you read this article? AEMO dispatch generators according to their bids and (generally) in line with the market rules. What you are saying is like blaming the ASX for the stock market for being too high/low/unstable.

          Just because something is complex does not mean it cannot work properly.

  2. In Example 1, Very Simple, the ROP is $65.44. But then the Trading Price looks like $80.32. What give? I’m already lost. I’ll go back to the simple guide now. But this may be worth explaining here, no?

  3. Great articles – thanks for these resources.

    I have a burning question to the more complex cases. Assuming that the Increase field captures the marginal increase in energy for the ENOF and the marginal decrease in energy for the LDOF type entries (for the extra 1 MW), and then there is FCAS (Increase sums to 0 MW), what exactly does the Increase field capture for the tbslack (tie-break) type entries? And for MNSP type ones (Basslink maybe – although Basslink shows up as non-scheduled in AEMO’s registration list (https://www.aemo.com.au/-/media/Files/Electricity/NEM/Participant_Information/NEM-Registration-and-Exemption-List.xls))?

Leave a comment

Your email address will not be published.


*