[WIP] Added max_block and request_batch to pyfive.File and backend

[dwest77a]

Description

• Added max_block and request_batch as parameters to the pyfive.File declaration. This is relayed to the backend bulk_fsspec method which now uses the merge_requests utility from fsspec if max_block is defined. Request batch allows configuration via the fsspec cat_ranges method to restrict the number of simultaneous requests down from the default for remote connections of 1280.

Checklist

• This pull request has a descriptive title and labels
• This pull request has a minimal description (most was discussed in the issue, but a two-liner description is still desirable)
• Unit tests have been added (if codecov test fails)
• Any changed dependencies have been added or removed correctly (if need be)
• If you are working on the documentation, please ensure the current build passes
• All tests pass

[valeriupredoi]

thanks @dwest77a 🍺 Note that if you wish the PR to be truly draft, you can always convert it to a Draft PR. I approved the tests to run here, and if you don't mind, I'll also add you to PyActiveStorage 🍻

[dwest77a]

This relates to issue #257

[valeriupredoi]

@dwest77a the issue with pre-commit kicks in since you change the functional call and args:

131c139,141
<                 self._read_bulk_fsspec(fh, chunks, out, dtype)
---
>                 self._read_bulk_fsspec(
>                     fh, chunks, out, dtype, max_block=max_block, batch_size=batch_size
>                 )

posix is defined as attr only if not fh, and the same for _fh - not great in terms of coding, but also mypy is not catching that. I think you can safely ignore those, and to run the tests you can temporarily turn off the call to pre-commit in the Github Action workflow

[valeriupredoi]

@dwest77a I realized we've had quite a few of these [attr-defined] mypy barfs, that I had to manually allow - the way the code is written is a bit confusing to mypy, and having checked manually all is kosher, I told mypy to mind its own business in 702bf12 - note, though, it's still a good idea to run pre-commit install && pre-commit run -a when you done with code changes, so to catch any issues locally, and fix em 🍺

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.52%. Comparing base (667c468) to head (04164d0).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #256      +/-   ##
==========================================
+ Coverage   78.48%   78.52%   +0.04%     
==========================================
  Files          15       15              
  Lines        3416     3423       +7     
  Branches      546      547       +1     
==========================================
+ Hits         2681     2688       +7     
  Misses        593      593              
  Partials      142      142              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[dwest77a]

@valeriupredoi thanks, I'll look into the test coverage now.

[valeriupredoi]

Good man Dan 🍺

[dwest77a]

@valeriupredoi happy to reconfigure this in some way that doesn't use a hardcoded path, it does need to be remote to test the block and batch sizes though.

[bnlawrence]

I'd like to get the two other pull requests that affect file handling into main before doing this one ... just in case there is an interaction ... I'll try and get them done in the next couple of days, and then move to this one.