[helipdx]

Cache: store resources by cache id, add db.xml format v2

Cache::hasEntries(), Cache::fillBlanks() and Cache::addResource() each walked the full QList<Resource> under cacheMutex for every ROM. Generating a gamelist for a large platform was therefore quadratic in the cache size, and because every scan held the same mutex, extra threads (-t) barely helped.

In memory

resources is now a QMap<QString, QList<Resource>> (cache id -> that ROM's resources), filled directly by read(). Per-ROM lookups go straight to that ROM's list, and there is no separate index to keep in sync after purge, vacuum, validate, edit or merge. Vacuum and validate also use set lookups instead of list scans. db.xml is written in cache id order instead of insertion order; the set of resources is unchanged.

On disk: format v2

<resources version="2">
  <game id="ABC123">
    <resource type="title" source="screenscraper" timestamp="...">Foo</resource>
    ...
  </game>
</resources>

• The id is stored once per game instead of on every resource (24.8 MB -> 19.6 MB on 151,500 resources).
• read() accepts both layouts, so existing caches load unchanged and are written as v2 on the next save.
• The 3.21.1 layout stays available as legacy-v1:
  • config.ini: cacheFormat="legacy-v1" in [main] or [<PLATFORM>] writes it on every save.
  • --cache convert:<FORMAT> rewrites db.xml once; without -p it converts all platforms.
• 3.21.1 and older read a v2 db.xml as empty, so CACHE.md and the changelog tell users to run --cache convert:legacy-v1 before going back to an older version.

Docs: CACHE.md, CLIHELP.md, CONFIGINI.md, config.ini.example, CHANGELOG.md (entry under 3.22.0 TBA) and the --cache help text.

Measurements

Synthetic snes cache, 15,000 ROMs / 151,500 resources, cache-mode generation with --flags pretend:

	master	map storage
-t 1	123 s	23.9 s
-t 8	95.5 s	5.0 s

Verification

• Synthetic cache (3,000 ROMs, resources shuffled across ids, orphaned ids), master and this branch run the same steps: generate, --cache edit (m, t, d, D, edit:new=title), purge:m=thegamesdb,t=cover, vacuum, merge:, validate with a stray file, report:missing=all, two -s import runs (refresh path), generate again. After each step the db.xml resource set and the cached media are identical; both gamelists, the output media, all 19 report files and quickid.xml are identical.
• legacy-v1 -> v2 -> legacy-v1 keeps the resource set identical; master reads and rewrites the legacy-v1 output unchanged; v2 -> v2 is byte-identical.
• cacheFormat in [main] and in a platform section, invalid values (warning for config.ini, exit 2 for --cache convert:), and convert across all platforms.
• test/ suite passes; test_settings covers cacheFormat.

🤖 Generated with Claude Code

[Gemba]

I was aware of that inefficiency. Thanks for the numbers.

I am not sold by the implementation, though:
It should be more ground up, where the cache data is loaded in a map right away. That would remove the extra index maintenance. I might consider to even change the XML storage format for that. With an option to write back the then legacy format (which is the current format as with 3.21.1 and before).

PS: Thanks for flagging your PR as AI assisted.

[helipdx]

Good, the side index was a workaround. I've pushed a commit that replaces it: resources is now a QMap<QString, QList<Resource>> (cache id -> that ROM's resources), filled directly by read(). Per-ROM lookups go straight to that ROM's list and there is nothing to invalidate after purge, vacuum, validate, edit or merge. Vacuum and validate also switched from list scans to set lookups. db.xml is now written in cache id order instead of insertion order; the set of resources is unchanged.

On a synthetic cache (3,000 ROMs, shuffled resources, orphans), master and this branch produce the same db.xml content, cached media, gamelist, output media and reports after generate, edit, purge, vacuum, merge, validate, report and import. Timing on 15,000 ROMs / 151,500 resources, cache-mode generation:

	master	index (first commit)	map (new commit)
-t 1	123 s	24.2 s	23.9 s
-t 8	95.5 s	5.4 s	5.2 s

Happy to squash before merge.

I've also added the new storage format to this PR, using the naming already in db.xml, quickid.xml and config.ini:

<resources version="2">
  <game id="ABC123">
    <resource type="title" source="screenscraper" timestamp="...">Foo</resource>
    ...
  </game>
</resources>

• The id is stored once per game instead of on every resource; on the set above db.xml goes from 24.8 MB to 19.6 MB.
• read() detects the version (no version attribute = the current format) and accepts both, so existing caches load without a migration step.
• config.ini: cacheFormat="legacy-v1" writes the 3.21.1-compatible format. It is allowed in [main] and platform sections, like cacheFolder. The default is v2.
• --cache convert:<FORMAT>, e.g. --cache convert:legacy-v1, rewrites db.xml once in the given format, following the merge:<PATH> / edit:new=<TYPE> pattern. Without -p it converts all platforms.
• Formats are named by version with a legacy- prefix for older ones, rather than a plain "legacy", so the names stay accurate when v2 is itself superseded.
• 3.21.1 reads a v2 db.xml as empty, so CACHE.md and the changelog tell users to run --cache convert:legacy-v1 before going back to an older version.

Docs (CACHE.md, CLIHELP.md, CONFIGINI.md, CHANGELOG.md, config.ini.example) and the settings test are updated, and the branch is merged with current master.

I assume game, version, cacheFormat and convert: are OK as names.