Reduce optimize worker memory by excluding full input list from recipe serialization

[senarvi]

Before submitting

• Was this discussed/agreed via a Github issue? I couldn't open an issue.
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

What does this PR do?

I would have opened an issue, but it's restricted, so I'll describe the issue here and propose a simple solution.

I run out of memory when trying to optimize the huge OpenImages dataset. A lot of memory is wasted, copying the input list to every worker. Under spawn multiprocessing, the optimize recipe is being pickled and sent to every worker process. Currently it contains the full inputs list, so memory is scaled with the number of workers. This change fixes that by customizing recipe serialization so that worker processes no longer receive that list.

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82%. Comparing base (8805bae) to head (29a0ccb).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@          Coverage Diff          @@
##            main    #890   +/-   ##
=====================================
  Coverage     82%     82%           
=====================================
  Files         65      65           
  Lines      13377   13381    +4     
=====================================
+ Hits       10949   10953    +4     
  Misses      2428    2428           

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.