Skip to content

Carbon intensity from WattTime at the time of usage - #302

Open
dpol1 wants to merge 6 commits into
mainfrom
6-watttime
Open

Carbon intensity from WattTime at the time of usage#302
dpol1 wants to merge 6 commits into
mainfrom
6-watttime

Conversation

@dpol1

@dpol1 dpol1 commented Oct 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Closes #6, for WattTime.

Adds optional modules which take the carbon intensity of a line item from the WattTime API, for the time of its usage. It replaces the yearly Ember or monthly NESO figures.

What it does

  • New modules watttime.aws.HistoricalCarbonIntensity and watttime.azure.HistoricalCarbonIntensity, for custom configurations only. As discussed in Module for accurate carbon intensity factors from ElectricityMaps / Watttime #6, they call the API while the job runs, with the user's own account, and keep the data in memory only.
    • The figures are WattTime's average operating emissions rate (AOER), converted from lbs/MWh to g/kWh. AOER only counts direct emissions from combustion.
    • The credentials come from the WATTTIME_USERNAME and WATTTIME_PASSWORD environment variables. On a cluster, set them on the executors.
  • WattTime regions: scripts/fetch_watttime_regions.sh maps each cloud region to its WattTime region with /v3/region-from-loc, from the coordinates in cloud_regions.json, and writes watttime/regions.csv.
  • API calls: /v3/historical returns the figures one calendar month (UTC) at a time, cached on each executor.
  • Usage period: AbstractHistoricalCarbonIntensity reads ChargePeriodStart / ChargePeriodEnd for FOCUS. The AWS subclass reads line_item_usage_start_date / line_item_usage_end_date for CUR, the Azure one the whole UTC day of Date. WattTime publishes AOER for each hour, so a line item gets the mean of the hourly figures within its period.
  • No figure: line items without a WattTime figure keep the value set by the earlier modules. When the module is used instead of ember.AverageCarbonIntensity and neso.MonthlyCarbonIntensity, as suggested in Module for accurate carbon intensity factors from ElectricityMaps / Watttime #6, these line items get no operational emissions. The docs also describe adding it after them. The output then mixes lifecycle figures with figures for direct emissions only.
  • Errors: the job stops on wrong credentials, on other API errors, or when the API cannot be reached.
  • RowColumn.getInstant reads a usage date from a Spark timestamp, a Spark date or a string, including the MM/dd/yyyy dates of Azure CSV exports.
  • Docs: new section in docs/src/modules.md, plus the overview table and the pipeline diagram. scripts/README.md describes the new script.

Written with the help of Claude Code 🤖

dpol1 added 3 commits October 23, 2026 10:32
@dpol1
Add a module reading the carbon intensity from WattTime
41028ce 
HistoricalCarbonIntensity sets CARBON_INTENSITY to the average
operating emissions rate (AOER) that the WattTime API publishes for the
grid region of the cloud region, averaged over the usage period of the
row. Credentials come from WATTTIME_USERNAME and WATTTIME_PASSWORD. The
WattTime region of each cloud region and the figures of each month are
fetched once per JVM. The usage period is read from the FOCUS columns,
else the CUR ones. Rows outside the coverage of WattTime keep their
previous value.
@dpol1
Read every report format in the WattTime module and document it
317fc4e 
The usage period comes from the FOCUS columns, the CUR columns or the
Azure date, which covers the whole day in UTC, through a new
RowColumn.getInstant that reads Spark timestamps, Spark dates and ISO
instants. A line item keeps its previous carbon intensity when its
cloud region has no coordinates, when it has no usable date, when its
usage ends before it starts or when it spans two months. The module
logs in again once when its token has expired. The WattTime URL is no
longer configurable. modules.md describes the module, its scope (AOER
excludes lifecycle emissions), the account it needs and how to
configure it.
@dpol1
Read Azure CSV dates and skip WattTime figures the account cannot read
291230e 
RowColumn.getInstant now also reads Spark instants, timestamps without
a time zone and the date strings getDate reads, such as the 06/23/2026
of Azure CSV exports, which shares the parsing with getDate. When the
account has no access to the figures of a WattTime region, its line
items keep the previous value and the module logs a warning. A failed
login stops the job with its own message, so that its HTTP status is
not taken for missing figures. The watttime package gets its
package-info.
jnioche
jnioche reviewed Oct 27, 2026
View reviewed changes

@jnioche jnioche left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

initial observations - thanks @dpol1

Comment thread src/main/java/com/digitalpebble/spruce/modules/watttime/HistoricalCarbonIntensity.java Outdated
Comment thread src/main/java/com/digitalpebble/spruce/modules/watttime/HistoricalCarbonIntensity.java Outdated
Comment thread src/main/java/com/digitalpebble/spruce/modules/watttime/AbstractWattTimeCarbonIntensity.java
@dpol1
dpol1 requested a review from jnioche October 28, 2026 10:03
@dpol1 dpol1 added the enhancement New feature or request label Oct 28, 2026
@dpol1 dpol1 added this to the 1.5 milestone Oct 28, 2026
@dpol1
dpol1 marked this pull request as ready for review October 28, 2026 10:03
@dpol1
Read the WattTime regions from a file and split the module by provider
8e848bb 
The WattTime region of each cloud region now comes from
watttime/regions.csv, generated by scripts/fetch_watttime_regions.sh,
instead of a call to /v3/region-from-loc while the job runs.

The module becomes an abstract class reading the usage period of FOCUS
reports, with a subclass for AWS, reading the CUR dates, and one for
Azure, reading the day of usage. The Javadoc of the average explains
why a line item gets the mean of the hourly figures.
jnioche
jnioche requested changes Oct 28, 2026
View reviewed changes

@jnioche jnioche left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @dpol1 see comments
I will run the code shortly

Comment thread docs/src/modules.md Outdated
Comment thread docs/src/modules.md Outdated
Comment thread docs/src/modules.md Outdated
@dpol1
dpol1 requested a review from jnioche October 29, 2026 07:43
@dpol1

dpol1 commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator Author

Added some parametrized tests where I can to cut duplication

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jnioche jnioche Awaiting requested review from jnioche

Assignees

No one assigned

Labels

enhancement New feature or request

Projects

None yet

Milestone

1.5

Development

Successfully merging this pull request may close these issues.

Module for accurate carbon intensity factors from ElectricityMaps / Watttime

2 participants

@dpol1 @jnioche