Support filesystem in cuIO paths - #23904
Draft
mhaseeb123 wants to merge 4 commits into
Draft
Conversation
Closes NVIDIA#20443 Only `read_parquet` accepted a pre-built fsspec `filesystem` object; every other reader and all writers accepted `storage_options` only. Add a `filesystem=` keyword to `read_csv`, `read_json`, `read_orc`, `read_avro`, `read_text`, `to_parquet`, `to_csv`, `to_json`, `to_orc`, `write_to_dataset` and `ParquetDatasetWriter`, plumbed through `ioutils` and validated by a shared `_validate_filesystem` helper. Co-Authored-By: Claude Opus 5 <[email protected]>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
mhaseeb123
changed the title
filesystem argument in more cuDF I/O pathsfilesystem in cuIO paths
Closes NVIDIA#23899 `read_parquet_metadata` took only `filepath_or_buffer`, so footer metadata could not be read from authenticated remote stores. Forward `storage_options` and `filesystem` to `get_reader_filepath_or_buffer` like the other readers. Co-Authored-By: Claude Opus 5 <[email protected]>
mhaseeb123
added
2 - In Progress
Remote reads route through `_prefetch_remote_buffers(method="all")`, which pulls whole files into host memory. That is pure waste for `read_parquet_metadata`, which needs only the footer, and `method="parquet"` does not help since it falls back to `_get_remote_bytes_all` when no columns or row groups are selected. Add a `parquet-footer` prefetcher that reads a 64 KiB tail, matching libcudf's LIBCUDF_PARQUET_METADATA_SIZE_HINT, trims it to the exact footer, and hands libcudf `PAR1` + footer + ender. libcudf locates the footer relative to the end of a source, so this parses identically while the buffer stays O(footer) instead of O(file): 2.8 KB rather than 7.5 MB on a 200k-row test file. Co-Authored-By: Claude Opus 5 <[email protected]>
The footer prefetcher called `fs.sizes()` to compute tail offsets, which costs an extra HEAD per file and adds a second sequential round trip before the ranges can be requested. Use suffix ranges instead: a negative `start` means "backwards from the end" per the fsspec `cat_file` contract, and clamps to the whole file when the file is shorter than the read. A short tail is therefore already the entire file, so the truncated-footer and non-Parquet cases can be detected without a size lookup. Measured on moto with 20 files: 20 requests instead of 40, all issued concurrently by `cat_ranges` on async filesystems. Co-Authored-By: Claude Opus 5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Under 🚧
Closes #20443
Closes #23899
This PR adds a new
filesystemparam to our orc, json, csv and parquet readers and writers to accept a pre-built fsspecfilesystemobject. Added validation by a shared_validate_filesystemhelper that keeps the existing "not at the same time asstorage_options" contract. Forread_parquet_metadata, the remote metadata reads also get aparquet-footerprefetcher which does a speculative read of 64KiB, trim to the exact footer, and hands libcudfPAR1 + footer + ender.Also two minor fixes:
_process_datasetnow passes its already-resolved filesystem tois_directory(), which previously inferredLocalFileSystemand answeredFalsefor every remote path; andParquetDatasetWriter.write_tableno longer dropsself.storage_optionsfor non-S3 remote paths.Checklist