Skip to content

Add ChakraTS (YHat Labs) API wrapper and fev-bench results - #192

Open
shubham-forum wants to merge 1 commit into
autogluon:mainfrom
yhatlabs:add-chakrats-api
Open

shubham-forum wants to merge 1 commit into
autogluon:mainfrom
yhatlabs:add-chakrats-api

Conversation

@shubham-forum

Copy link
Copy Markdown

Issue #, if available: None

Description of changes:

  • Add the chakrats_api wrapper and pinned dependencies for ChakraTS, a hosted forecasting model from YHat Labs (https://huggingface.co/yhatlabs/ChakraTS). No weights are distributed; the wrapper reads the API key from YHAT_API_KEY, no credentials are included. This submission is for a closed-source API model as discussed with Oleksandr.

  • Add results for all 100 fev-bench tasks in benchmarks/fev_bench/results/chakrats_api.csv: SQL skill score 48.1 (seasonal-naive normalised). The results are for chakra-ts-fev, the fev-bench evaluation configuration of ChakraTS: no part of it was pretrained on any fev-bench dataset, so trained_on_datasets is empty and no task is flagged.

  • Add the models.yaml entry (closed-api, zero-shot, commercial use permitted).

  • The wrapper sends each window's items with all target columns, past covariates and known covariates, 100 items per call, and never mixes windows in one call.

Validation:

  • test/test_fev_bench_submissions.py passes for the new file and metadata.

  • The result CSV has 100 unique tasks and complete SQL, MASE, WAPE and WQL values.

  • The wrapper run against the API reproduces the submitted rows within 0.1% on every task we checked. We will send the maintainers an evaluation key privately. -t 3 runs the three ProEnFo tasks in about 5 minutes, -t 8 about 45 minutes, the full benchmark about few hours. The server scales to zero when idle, so the first call can take a couple of minutes; the wrapper retries.

  • We can also share the complete outputs of our own run privately, including the final forecasts for all 100 tasks, if that helps the review.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant