Repository navigation
Description
Describe the bug
Native casts to and from timestamps fail for values whose year is outside chrono's range (about ±262,143), while Spark supports timestamps up to year 294,247 and returns a result. This happens for TRY casts too, so try_cast raises instead of returning a value.
| Query | Comet |
|---|---|
CAST(ts AS DATE) |
CometNativeException: Cast error: Cannot convert TimestampMicrosecondType 9000000000000000000 to datetime. Comet only supports dates between Jan 1, 262145 BCE and Dec 31, 262143 CE |
CAST(ts AS STRING) |
same error |
try_cast(ts AS DATE) |
same error |
CAST(dt AS TIMESTAMP) |
native panic at spark-expr/src/conversion_funcs/temporal.rs:74:34 |
Steps to reproduce
As a SQL file test (spark/src/test/resources/sql-tests/...), default Spark 4.1 profile:
statement
CREATE TABLE tsr(ts timestamp, dt date) USING parquet
statement
INSERT INTO tsr VALUES
(timestamp_micros(9000000000000000000), date_from_unix_date(100000000)),
(timestamp_micros(0), date_from_unix_date(0))
query
SELECT CAST(ts AS DATE) FROM tsr
query
SELECT CAST(ts AS STRING) FROM tsr
query
SELECT try_cast(ts AS DATE) FROM tsr
query
SELECT CAST(dt AS TIMESTAMP) FROM tsrtimestamp_micros(9000000000000000000) is in year 287,000 or so, and date_from_unix_date(100000000) in year 275,000 or so; both are valid Spark values.
Expected behavior
The same results as Spark. Where the native kernel cannot represent the value, the serde should fall back to Spark rather than fail or panic.
Additional context
The error paths:
pre_timestamp_castinnative/spark-expr/src/utils.rs(as_datetime(...).ok_or_else(datetime_cast_err)), used by timestamp to string and timestamp to date for every eval mode.timestamp_ntz_to_timestampandcast_timestamp_to_ntzin the same file, same pattern.- Date to timestamp in
native/spark-expr/src/conversion_funcs/temporal.rs:epoch + chrono::Duration::days(d)panics. The comment above it says chrono's range covers everyi32day count, but it does not. test_cast_unsupported_timestamp_to_dateinconversion_funcs/cast.rsasserts the error fori64::MAX.
Found while auditing which expressions can safely run as TopK later keys in #7457 , which now treats every timestamp cast as able to fail. Suggested triage: priority:high (native panic), area:expressions.
Activity
tlfua commentedon Nov 6, 2026
A cast from
DATEtoSTRINGhas the same problem, raised in review on #7457 . Comet sends it to arrow-cast'svalue_to_string, whoseValueFormatter::writeignores thesafeoption, so a date beyond chrono's year range raises in every mode:Spark returns the date and its string. Comet fails with
Cast error: Failed to convert 100000000 to temporal for Date32(checked on Spark 4.1 with aCometSqlFileTestSuitefixture). Implicit casts such asconcat(s, dt)go the same way. #7457 now treats this cast as able to fail for TopK sort keys.kazuyukitanimura commentedon Nov 9, 2026
we are moving onto jiff FYI
fix: keep a native lambda body with a null map key or a date/time cas…