diff --git a/deployment/README.md b/deployment/README.md index 86c2639..95399e5 100644 --- a/deployment/README.md +++ b/deployment/README.md @@ -68,6 +68,12 @@ require manual attention. Original download archives remain available for seedin ## Optional cover resolver +SteamGridDB supplies portraits and hero backgrounds for PS5 and Switch, with +official Steam library assets as a fallback for missing games/images and verified +PlayStation Store products for configured console-only editions. Configure +its separate provider key in `/opt/botty-artwork/steamgriddb-api-key` (root:botty-artwork, +0640); see [key setup and cache behavior](botty-artwork/README.md). + The resolver listens on `127.0.0.1:9697`; Nginx exposes only its authenticated artwork route. It uses the **same API key** as the console's Prowlarr configuration. @@ -78,10 +84,13 @@ sudo apt-get install python3-pil sudo useradd --system --no-create-home --shell /usr/sbin/nologin botty-artwork sudo install -d -o root -g botty-artwork -m 0750 /opt/botty-artwork sudo install -m 0644 deployment/botty-artwork/server.py /opt/botty-artwork/server.py -sudo install -m 0644 deployment/botty-artwork/sources.json /opt/botty-artwork/sources.json +sudo install -m 0644 deployment/botty-artwork/steamgriddb-games.json /opt/botty-artwork/steamgriddb-games.json +sudo install -m 0644 deployment/botty-artwork/playstation-games.json /opt/botty-artwork/playstation-games.json sudo install -m 0644 deployment/botty-artwork/aliases.json /opt/botty-artwork/aliases.json sudo install -o root -g botty-artwork -m 0640 /dev/null /opt/botty-artwork/api-key sudoedit /opt/botty-artwork/api-key +sudo install -o root -g botty-artwork -m 0640 /dev/null /opt/botty-artwork/steamgriddb-api-key +sudoedit /opt/botty-artwork/steamgriddb-api-key sudo install -m 0644 deployment/botty-artwork/botty-artwork.service /etc/systemd/system/ sudo systemctl daemon-reload sudo systemctl enable --now botty-artwork @@ -90,9 +99,10 @@ sudo systemctl status botty-artwork Skip user creation if the account already exists. Enter only the API key into the key file; do not put it in shell arguments or source control. On updates, preserve -`api-key`, `aliases.json`, `sources.json` and `/var/cache/botty-artwork`; do not -repeat the command that creates an empty key file. Source mappings may have -local additions. See [resolver behavior](botty-artwork/README.md). +both key files, `aliases.json`, `steamgriddb-games.json`, `playstation-games.json` +and `/var/cache/botty-artwork`; merge new default mappings with operator additions. +Do not repeat the command that creates an empty key file. Legacy `sources.json` +mappings are left untouched. See [resolver behavior](botty-artwork/README.md). ## Updates and rollback diff --git a/deployment/botty-artwork/README.md b/deployment/botty-artwork/README.md index 58a34c0..de5348a 100644 --- a/deployment/botty-artwork/README.md +++ b/deployment/botty-artwork/README.md @@ -1,78 +1,138 @@ -# Cover resolver - -Run `server.py` as the dedicated `botty-artwork` user with Python 3 and Pillow, -using the systemd service and authenticated TLS nginx route shipped alongside it. -The API key remains in `/opt/botty-artwork/api-key`, never in this repository. - -Scene suffixes such as PROPER and INTERNAL are removed only when the input has -a PS5, CUSA/PPSA title-ID, [FPKG], UPDATE, `v` (for example `v1.02` -or `v10.02`), MULTI, or incl/including marker. The literal word `version` is -not a marker. Release detection and metadata truncation share the same pattern; -PS5, CUSA/PPSA IDs, and [FPKG] are also detected before bracket removal. -Platform markers accept word boundaries, including bracketed IDs and hyphen, -dot, or colon separators (for example `Proper-PPSA12345`). Scene suffixes -also accept these separators when a release marker is present, so -`Some Game-PROPER-PPSA12345` resolves to `Some Game`. UPDATE, `v`, MULTI, -and incl/including require preceding whitespace; `[FPKG]` does not. -Metadata and edition suffixes are -trimmed repeatedly in either order; unmarked canonical scene words are preserved. - -Resolution uses normalized game names (scene tags, editions, punctuation, accents, -Roman numerals, trademark symbols), Steam game search/library artwork, Wikipedia -exact article lookup followed by search, then a Steam store banner if no portrait -is available. Matches must preserve the game identity and sequel number. Provider -IDs, source URLs, canonical titles and outcomes are saved next to RGB images. -Positive results last thirty days. Confirmed misses last one hour; network/provider -errors are not negative-cached. Different titles can resolve concurrently; duplicate -lookups coalesce and network concurrency is bounded to four. - -Near-uniform provider images are rejected, including grey CDN placeholders returned -with HTTP 200. Existing cached placeholders are re-resolved rather than retained for -thirty days. Battlefield 6 and God of War Sons of Sparta have verified PlayStation -product mappings in the seed file. - -For regional names or verified exceptions, `/opt/botty-artwork/aliases.json` can map -a normalized title to a canonical game name. It is read on each request and survives -updates. Example: `{"regionalname":"Canonical Game"}`. No arbitrary image URL is -accepted from the client. HTTPS destinations and redirects are allowlisted, response -sizes and image dimensions are bounded. Images retain their complete composition. - -Merge the seed `aliases.json` into the installed alias file when deploying, preserving -operator entries. It covers the eFootball 2027 seasonal name, Sonic Unleashed's -unofficial recompilation port, and Resident Evil 4's explicit remake release name. -The latter uses the verified 2023 PlayStation product in `sources.json`. -It also restores the missing ampersands in The Bearer & The Last Flame and -Snoopy & The Great Mystery Club, and maps WILD HEARTS Karakuri Edition to its game. - -Verified PlayStation product pages can be registered in `/opt/botty-artwork/sources.json` -(mapping normalized game names to official product URLs; seed file provided). The -resolver reads Product JSON-LD, verifies the title and sequel, and downloads only -from allowlisted PlayStation origins. Publisher prefixes such as EA SPORTS and -trademark marks do not change identity. This mapping survives service upgrades. - -Release 1.5.2 includes verified PS5 product mappings for God of War Sons of Sparta, -Resident Evil 4 (2023), Infliction: Extended Cut, Snoopy & The Great Mystery Club, -and Crazy Chicken Shooter Edition. Keep `Shooter Edition` in Crazy Chicken's name: -it identifies the product and is not a generic edition suffix. - -To update an existing installation, retain rollback copies of `server.py`, -`aliases.json` and `sources.json`, replace the server code, and merge both seed -JSON objects into their installed counterparts. Preserve existing operator entries -outside the updated keys. Keep the API key, cache ownership and service user. -Restart only `botty-artwork` if the server code changed; mappings are read on each -request. An alias changes the canonical cache key automatically. For a newly -mapped product with the same canonical name, let its one-hour negative cache expire -or back up and remove only that title's empty RGB cache and matching JSON metadata. -On the console, failed lookups retry after one minute; change Explore pages and -return to request the artwork again. No console binary update is required. - -Coverage cannot be guaranteed for every release: games missing from both sources -retain an explicit title placeholder. Do not silently substitute another sequel. -The audit and cached metadata identify unresolved names for later corrections. - -References: -- https://partner.steamgames.com/doc/store/assets/libraryassets -- https://www.mediawiki.org/wiki/API:Pageimages - -Run `python3 test_server.py` to check matching, cache reuse, fallback, outage retry, -origin restrictions and persistent aliases without public network requests. +# Game artwork resolver + +SteamGridDB is the primary image provider for PS5 and Switch. If the game or +requested image is unavailable there, the resolver uses official Steam library +assets as a fallback. Operator-verified PlayStation Store products provide a +final fallback for the configured console-only releases/editions. The authenticated +`/artwork?title=...&kind=cover|hero` route returns RGB bytes to the console service: + +- `cover` prefers static **600×900** grids, also accepting **660×930** and + **342×482** portraits when needed. Every image is fitted into **160×240**; + landscape banners and square icons are excluded. +- `hero` selects static **1920×620**, **3840×1240** or **1600×650** heroes and fits + them into a **640×360** canvas while preserving the complete composition, with + blurred fill above/below wide artwork. Up to three top-ranked heroes are checked + to prefer illustration visible on the right, away from the UI text scrim. + +Steam fallback matches the complete game name, fetches typed game metadata from +`IStoreBrowseService/GetItems`, and uses `library_capsule[_2x]` or +`library_hero[_2x]`. Modern per-image hash paths are read from the metadata; old +hard-coded CDN paths can return 404 even for available artwork. Steam requires +no additional API key. Store banners, demos, DLC and unrelated annual editions +are never substituted for the game portrait. All providers share the same +160×240 / 640×360 output dimensions and persistent caches. + +For PlayStation Store, `/opt/botty-artwork/playstation-games.json` maps normalized +names to verified HTTPS product URLs (override with `BOTTY_PLAYSTATION_GAMES`). +NHL 27 and Dragon’s Dogma 2: Dark Arisen are included. The resolver checks the +JSON-LD product name/SKU and reads only that exact product’s media from the +embedded store data. `PORTRAIT_BANNER` supplies the cover; `GAMEHUB_COVER_ART`, +`EDITION_KEY_ART` and `BACKGROUND` supply the background. Concept/base-game, +upsell, screenshot, logo and square master images are excluded. This preserves +the actual Dark Arisen edition rather than selecting the original game’s art. +No PlayStation account, session or API key is required. Product media metadata +and resulting images persist in the existing caches. + +The app uses a blurred cover behind its text if neither provider has a hero. It keeps a +uniform title placeholder if there is no portrait; square Switch icons, Steam +banners and Wikipedia images are no longer substituted. + +## API key + +On the artwork host (the current installation uses SSH alias `test`), put only the +SteamGridDB API key in **`/opt/botty-artwork/steamgriddb-api-key`**. The service reads +it on each uncached API lookup; replacing the key needs no restart. + +```sh +sudo install -o root -g botty-artwork -m 0640 /dev/null /opt/botty-artwork/steamgriddb-api-key +sudoedit /opt/botty-artwork/steamgriddb-api-key +``` + +Run the first command only when creating the file; it would erase an existing key. +Generate a key at https://www.steamgriddb.com/profile/preferences/api . To use another +path, set `BOTTY_STEAMGRIDDB_KEY_FILE` in the systemd unit's environment. Keep this +key on the server, outside source control and console packages. The existing +`/opt/botty-artwork/api-key` is a separate key: it still authenticates the console's +Prowlarr/artwork proxy requests. + +## Matching and caches + +Release metadata, regions, editions, accents, punctuation and Roman numerals are normalized. +Switch scene suffixes (`NSW-GROUP`, `NSW Switch`) and PS5 VR markers are removed. +Only a parenthesized year disambiguates releases; numbers inside a game name, +such as Cyberpunk 2077, remain part of its identity. Canonical SteamGridDB names +are compared without release cleanup so editions, DLC and fan demakes stay distinct, +including qualifiers in parentheses. Only explicit release-year parentheses are +ignored when comparing names. +A match must preserve title/sequel identity. Multiple matching games (for example +a remake) require an explicit year in the canonical alias or a game-ID override. +Missing conjunctions and articles can be normalized and a distinctive subtitle +can be queried again, while the complete game identity still has to match. +No first-result fuzzy match is silently accepted. + +`/opt/botty-artwork/aliases.json` maps normalized names to canonical titles. A +`ps5:` prefix limits an alias to explicitly marked PS5 releases, allowing the +correct Resident Evil 4 / Demon’s Souls remake without affecting Switch names. +Optional `/opt/botty-artwork/steamgriddb-games.json` maps normalized titles to +positive numeric SteamGridDB game IDs, for example `{"astrobot": 5454475}`. These +operator files are read on lookup and survive upgrades. `sources.json` from the +previous resolver is preserved but no longer used. When changing an override, +back up/remove that title's `sgdb-v1` cache entries to apply it immediately. + +Successful Steam game metadata is cached alongside SteamGridDB matches. Removed Steam image URLs +trigger fresh asset metadata rather than pinning a stale hash directory. +Server matches, portraits and heroes persist in `/var/cache/botty-artwork`. +Positive images and game matches last thirty days; confirmed misses last an hour. +Authentication, rate limits and network errors are retried rather than saved as +missing artwork. A canonical-title lock coalesces concurrent cover/hero lookups and unique +temporary files make metadata/image publication atomic across processes; network +concurrency is bounded to four. Credentials never enter saved metadata. Images, +response sizes and HTTPS origins/redirects are checked, including rejecting +landscape grids, animated images and blank CDN tiles. + +The console service uses separate `cache/sgdb-covers-v1` and +`cache/sgdb-heroes-v2` directories under its root, bounded to 512 portraits and 128 +heroes. Full release names reach the resolver so scene markers and DLC Unlocker +suffixes can be cleaned correctly. The native app has its own persistent `/download0/botty-sgdb-covers-v1.cache` +and `/download0/botty-sgdb-backgrounds-v2.cache` files with the same capacities +(about 56 and 84 MiB). They work after closing/reopening the app, without a service +request for a cached image. Least recently used native entries are replaced; +checksums reject damaged or interrupted writes. Delete these two files with the +app closed to reset the native caches. The new namespaces automatically bypass +older mixed-format caches; those old files can be removed later with the app +closed. They are not deleted by this update. + +## Deployment and checks + +Use Python 3, Pillow, the dedicated `botty-artwork` user and the supplied systemd +unit. The listener stays on `127.0.0.1:9697`; nginx exposes only its authenticated +HTTPS route. Preserve both key files, aliases, game-ID overrides, ownership and +rollback copies when updating. Deploy the matching console service/native app to +enable Switch portraits and heroes throughout the UI. Do not interrupt active +extraction/compression or replace an open native title. + +Run `python3 deployment/botty-artwork/test_server.py` for deterministic provider, +matching, format, cache, secret-handling and outage checks, without public requests. +See https://www.steamgriddb.com/api/v2 for the provider API. + + +## Audit real Prowlarr results + +Keep raw Prowlarr responses and credentials in ignored private backups. Export a +sanitized JSON array containing only title strings or objects with a `title` field. +On the artwork host, run under the artwork service user: + +```sh +sudo -u botty-artwork python3 audit.py --resolver /opt/botty-artwork/server.py \ + --corpus /private/prowlarr-titles.json --images --output /private/artwork-audit.json +``` + +The audit checks every release, downloads/verifies the final portrait with +`--images`, and records matched game IDs, missing portraits, unresolved candidates +and request failures separately, including the actual image provider. It never downloads torrents or changes console +files. At most two workers run. Known multi-game library/collection packs are reported as `not-single-game`. +Reports contain no credentials or download URLs; +existing output files are not overwritten. Game-library bundles and releases for +other consoles can appear in category 1000; keep these distinct when interpreting +coverage of individual PS5/Switch games. Do not substitute a previous sequel or +a different game just to eliminate a missing-artwork result. diff --git a/deployment/botty-artwork/aliases.json b/deployment/botty-artwork/aliases.json index 823cc39..6c68778 100644 --- a/deployment/botty-artwork/aliases.json +++ b/deployment/botty-artwork/aliases.json @@ -5,5 +5,39 @@ "residentevil4remake": "Resident Evil 4 (2023 video game)", "thebearerthelastflame": "The Bearer & The Last Flame", "snoopythegreatmysteryclub": "Snoopy & The Great Mystery Club", - "wildheartskarakuriedition": "WILD HEARTS" + "wildheartskarakuriedition": "WILD HEARTS", + "ps5:residentevil4": "Resident Evil 4 (2023)", + "ps5:demonssouls": "Demon's Souls (2020)", + "ps5:kaothekangaroo": "Kao the Kangaroo (2022)", + "ps5:monopoly": "Monopoly 2024", + "dispatch": "Dispatch (2025)", + "constance": "Constance (2025)", + "blueysquestforthegoldpen": "Bluey's Quest for the Gold Pen (2025)", + "marioplusrabbidskingdombattle": "Mario + Rabbids Kingdom Battle", + "rayman30th": "Rayman: 30th Anniversary Edtion", + "akibalost": "AKIBA LOST - アキバロスト", + "viractalwillyoutrustyourparty": "VIractal", + "subwaysimulatorundergroundtrainridestationultimatedrivinggames": "SUBWAY SIMULATOR: UNDERGROUND TRAIN RIDE", + "nordicashesthecompletesaga": "Nordic Ashes", + "tombraideri3remasteredstarringlaracroft": "Tomb Raider I-III Remastered", + "metalgearsolidmastercollectionvolume1": "Metal Gear Solid: Master Collection Vol. 1", + "horizoncallofthemountain": "Horizon VR: Call of the Mountain", + "bloodraynerevamped": "BloodRayne", + "bloodrayne2revamped": "BloodRayne 2", + "crimsondesertenhanced": "Crimson Desert", + "legomarvelsuperheroes2": "LEGO Marvel Super Heroes 2", + "nintendo64nintendoswitchonlinemature17plus": "Nintendo 64: Nintendo Switch Online - Mature 17+", + "nintendo64nintendoswitchonline18plus": "Nintendo 64: Nintendo Switch Online 18+", + "thelastoricrufinalcut": "The Last Oricru", + "talesofeterniaremastered": "Tales of Eternia", + "fishingvalleyremastered": "Fishing Valley", + "theprinceoftennissweetschoolfestivallove40andmore": "The Prince of Tennis Sweet School Festival 0-40 and more...", + "guiltygearstrivenintendoswitcheditionggstadditionalcharactercolorpack": "Guilty Gear: Strive", + "guiltygearstrivenintendoswitcheditioncht": "Guilty Gear: Strive", + "castlevaniaanniversarycollectioneshop": "Castlevania Anniversary Collection", + "supernintendoentertainmentsystemnintendoswitchonlineinternaleshop": "Super Nintendo Entertainment System: Nintendo Switch Online", + "nintendoentertainmentsystemnintendoswitchonlineinternaleshop": "Nintendo Entertainment System: Nintendo Switch Online", + "minecraftnintendoswitcheditioneshop": "Minecraft: Nintendo Switch Edition", + "fc2027": "EA SPORTS FC 27", + "bluereflectionquartlet": "BLUE REFLECTION Quartet" } diff --git a/deployment/botty-artwork/audit.py b/deployment/botty-artwork/audit.py new file mode 100644 index 0000000..03da4d4 --- /dev/null +++ b/deployment/botty-artwork/audit.py @@ -0,0 +1,68 @@ +#!/usr/bin/env python3 +"""Audit a sanitized Prowlarr title corpus on the artwork host; never needs Prowlarr credentials.""" +import argparse, concurrent.futures, hashlib, importlib.util, json, pathlib, re, time, urllib.parse + + +def audit(resolver, row, images=False): + raw = row if isinstance(row, str) else row['title'] + title = resolver.canonical_title(raw) if hasattr(resolver, 'canonical_title') else resolver.clean_title(raw) + result = {'title': raw, 'canonical': title} + started = time.monotonic() + if hasattr(resolver, 'is_collection') and resolver.is_collection(title): + return {**result, 'status': 'not-single-game', 'elapsedMs': 0} + try: + key = hashlib.sha256(('sgdb-v1:' + title.casefold()).encode()).hexdigest() + if images: + result['coverBytes'] = len(resolver.cover(raw)) + if result['coverBytes']: + metadata = json.loads((resolver.CACHE / (key + '.cover.json')).read_text()) + match = metadata['match'] + result.update(status='covered', game=match['game'], provider=match['provider']) + result['elapsedMs'] = round((time.monotonic() - started) * 1000) + return result + ident = resolver.game(title, resolver.CACHE / (key + '.game.json')) + result['game'] = ident + if ident: + result.update(status='no-portrait' if images else 'matched', provider='SteamGridDB') + else: + # Resolve fallback metadata on cold caches as well; do not infer + # support from files left by a previous image request. + for candidates in (resolver.steam_candidates, resolver.playstation_candidates): + match = next(iter(candidates(title, 'cover')), None) + if match: + result.update(game=match['game'], provider=match['provider']) + break + if result.get('game'): + result['status'] = 'no-portrait' if images else 'matched' + else: + query = re.sub(r'\s*\([^)]*\)', '', title) + result['candidates'] = [{k: x.get(k) for k in ('id', 'name', 'release_date')} + for x in resolver.sgdb('search/autocomplete/' + urllib.parse.quote(query, safe=''))] + result['status'] = 'unmatched' if result['candidates'] else 'no-search-results' + except Exception as exc: + # Request exceptions can include URLs; only persist their class. + result.update(status='error', error=type(exc).__name__) + if hasattr(exc, 'code'): result['httpStatus'] = exc.code + result['elapsedMs'] = round((time.monotonic() - started) * 1000) + return result + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--resolver', default=str(pathlib.Path(__file__).with_name('server.py'))) + parser.add_argument('--corpus', required=True); parser.add_argument('--output', required=True) + parser.add_argument('--images', action='store_true'); parser.add_argument('--workers', type=int, choices=(1, 2), default=2) + args = parser.parse_args() + spec = importlib.util.spec_from_file_location('artwork', args.resolver) + resolver = importlib.util.module_from_spec(spec); spec.loader.exec_module(resolver) + rows = json.loads(pathlib.Path(args.corpus).read_text()) + with concurrent.futures.ThreadPoolExecutor(max_workers=args.workers) as pool: + results = list(pool.map(lambda row: audit(resolver, row, args.images), rows)) + counts = {s: sum(r['status'] == s for r in results) for s in sorted({r['status'] for r in results})} + report = {'resolver': resolver.RESOLVER, 'saved': time.time(), 'imagesChecked': args.images, + 'total': len(results), 'counts': counts, 'results': results} + with pathlib.Path(args.output).open('x') as out: json.dump(report, out, indent=2, ensure_ascii=False); out.write('\n') + print(json.dumps({'total': len(results), 'counts': counts, 'output': args.output})) + + +if __name__ == '__main__': main() diff --git a/deployment/botty-artwork/playstation-games.json b/deployment/botty-artwork/playstation-games.json new file mode 100644 index 0000000..0cf746f --- /dev/null +++ b/deployment/botty-artwork/playstation-games.json @@ -0,0 +1,4 @@ +{ + "nhl27": "https://store.playstation.com/en-us/product/UP0006-PPSA34063_00-NHL27STANDEDITON/", + "dragonsdogma2darkarisen": "https://store.playstation.com/en-us/product/UP0102-PPSA09663_00-DD2DA00000000000/" +} diff --git a/deployment/botty-artwork/server.py b/deployment/botty-artwork/server.py index 12de010..0e0c97a 100644 --- a/deployment/botty-artwork/server.py +++ b/deployment/botty-artwork/server.py @@ -1,30 +1,44 @@ #!/usr/bin/env python3 -"""Game cover resolver with bounded fetching, persistent matches and two providers.""" -import html, hashlib, hmac, io, json, os, pathlib, re, threading, time, unicodedata, urllib.parse, urllib.request +"""Game cover resolver with bounded fetching, persistent SteamGridDB portraits and backgrounds.""" +import html, hashlib, hmac, io, json, os, pathlib, re, threading, time, tempfile, unicodedata, urllib.error, urllib.parse, urllib.request from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer -from PIL import Image, ImageOps +from PIL import Image, ImageOps, ImageFilter, ImageStat CACHE = pathlib.Path(os.environ.get('BOTTY_ARTWORK_CACHE', '/var/cache/botty-artwork')) KEY = pathlib.Path(os.environ.get('BOTTY_ARTWORK_KEY_FILE', '/opt/botty-artwork/api-key')).read_text().strip() -SOURCES = pathlib.Path(os.environ.get('BOTTY_ARTWORK_SOURCES', '/opt/botty-artwork/sources.json')) +STEAMGRIDDB_KEY = pathlib.Path(os.environ.get('BOTTY_STEAMGRIDDB_KEY_FILE', '/opt/botty-artwork/steamgriddb-api-key')) +GAMES = pathlib.Path(os.environ.get('BOTTY_STEAMGRIDDB_GAMES', '/opt/botty-artwork/steamgriddb-games.json')) +PLAYSTATION_GAMES = pathlib.Path(os.environ.get('BOTTY_PLAYSTATION_GAMES', '/opt/botty-artwork/playstation-games.json')) ALIASES = pathlib.Path(os.environ.get('BOTTY_ARTWORK_ALIASES', '/opt/botty-artwork/aliases.json')) UA = 'Botty/0.3 (personal game library; https://github.com/Portablelle/Botty-Plus)' -HOSTS = {'store.playstation.com', 'image.api.playstation.com', 'en.wikipedia.org', 'upload.wikimedia.org', 'store.steampowered.com', 'cdn.akamai.steamstatic.com', 'shared.akamai.steamstatic.com', 'cdn.cloudflare.steamstatic.com'} +HOSTS = {'www.steamgriddb.com', 'cdn2.steamgriddb.com', 'cdn.steamgriddb.com'} +STEAM_CDNS = {'shared.akamai.steamstatic.com', 'shared.fastly.steamstatic.com'} +SIZES = {'cover': (160, 240), 'hero': (640, 360)} +RESOLVER = 12 LOCKS = [threading.Lock() for _ in range(32)] NETWORK = threading.BoundedSemaphore(4) Image.MAX_IMAGE_PIXELS = 12000000 def allowed(url): p = urllib.parse.urlsplit(url) - return p.scheme == 'https' and p.hostname in HOSTS and not p.username and p.port in (None, 443) + if p.scheme != 'https' or p.username or p.port not in (None, 443): return False + if p.hostname in HOSTS: return True + if p.hostname == 'store.playstation.com': return bool(re.fullmatch(r'/[a-z]{2}-[a-z]{2}/product/[A-Z0-9_-]+/?', p.path)) + if p.hostname == 'image.api.playstation.com': return bool(re.fullmatch(r'/vulcan/ap/rnd/\d{6}/\d{4}/[A-Za-z0-9_-]+\.(?:png|jpg|jpeg)', p.path)) + if p.hostname == 'store.steampowered.com': return p.path == '/api/storesearch/' + if p.hostname == 'api.steampowered.com': return p.path == '/IStoreBrowseService/GetItems/v1/' + if p.hostname in STEAM_CDNS: + return bool(re.fullmatch(r'/store_item_assets/steam/apps/\d+/(?:[0-9a-f]{40}/)?library_(?:capsule(?:_2x)?|600x900(?:_2x)?|hero(?:_2x)?)\.(?:jpg|png)', p.path)) + return False class Redirects(urllib.request.HTTPRedirectHandler): def redirect_request(self, req, fp, code, msg, headers, newurl): - if not allowed(newurl): raise ValueError('Untrusted artwork redirect') + if not allowed(newurl) or (req.has_header('Authorization') and urllib.parse.urlsplit(newurl).hostname != 'www.steamgriddb.com'): + raise ValueError('Untrusted artwork redirect') return super().redirect_request(req, fp, code, msg, headers, newurl) -def fetch(url, limit): +def fetch(url, limit, headers=None): if not allowed(url): raise ValueError('Untrusted artwork source') - with NETWORK, urllib.request.build_opener(Redirects).open(urllib.request.Request(url, headers={'User-Agent': UA}), timeout=4) as response: + with NETWORK, urllib.request.build_opener(Redirects).open(urllib.request.Request(url, headers={'User-Agent': UA, **(headers or {})}), timeout=4) as response: data = response.read(limit + 1) if len(data) > limit: raise ValueError('Artwork too large') return data @@ -32,8 +46,9 @@ def fetch(url, limit): def clean_title(title): """Remove release metadata while preserving scene words in unmarked names.""" title = title.replace('_', ' ') + title = re.sub(r'\((?:RF|USA?|EUR?|JP[N]?|ASIA|US-EU|eShop)\)', ' ', title, flags=re.I) title = re.sub(r'(?<=[A-Za-z])\.(?=[A-Za-z])', ' ', title) - platform_marker = r'\b(?:PS5|(?:CUSA|PPSA)\d+)\b' + platform_marker = r'\b(?:PS5|(?:CUSA|PPSA)\d+)\b|\s+(?:NSP|XCI)\b|\s+(?:Switch|NSW)(?=\s*(?:$|[-–—]|NSP\b|XCI\b|Switch\b|NSW\b|v\d|\[))' platform_release = bool(re.search(platform_marker + r'|\[FPKG\]', title, flags=re.I)) title = re.sub(r'\[[^\]]*\]', ' ', title) release_marker = re.compile(platform_marker + r'|\s+(?:incl\.?|including|v\d+(?:\.\d+)*|update|MULTI\d*)\b', re.I) @@ -42,139 +57,358 @@ def clean_title(title): # Alternate metadata and edition removal until stable, regardless of order. # Scene words are ambiguous in canonical names; require a release marker. while True: - trimmed = re.sub(r'\s+(?:DLC|add[ -]?on)(?:\s+add[ -]?on)?(?:\s+only)?\s*$', '', title, flags=re.I) + trimmed = re.sub(r'\s+(?:DLC|add[ -]?on)(?:\s+add[ -]?on)?(?:\s+(?:only|unlocker|pack|fix))?\s*$', '', title, flags=re.I) if release_name: trimmed = re.sub(r'[\s.:-]+(?:proper|repack|rerip|readnfo|internal)[\s.:-]*$', '', trimmed, flags=re.I) - trimmed = re.sub(r'\s+(?:the shores|(?:standard|digital deluxe|deluxe|ultimate|complete|gold|precious|anniversary|definitive|premium|special|ballpark|collectors?|game of the year) edition|directors? ?cut|goty)\s*$', '', trimmed, flags=re.I) + trimmed = re.sub(r'\s+(?:Nintendo Switch Edition|Master Collection Version|All Expansions Pack|Base Game(?:\s*\+)?|Digital Sound and Art Gallery)\s*$', '', trimmed, flags=re.I) + trimmed = re.sub(r'\s+(?:eShop|JPN|CHT|USA|EUR|DEMO)\s*$', '', trimmed, flags=re.I) if release_name else trimmed + trimmed = re.sub(r'\s+(?:PSVR2?|VR2)\s*$', '', trimmed, flags=re.I) if release_name else trimmed + trimmed = re.sub(r'\s+(?:the shores|(?:standard|digital deluxe|deluxe|ultimate|complete|gold|precious|infernal|midnight|legacy|console|anniversary|definitive|premium|special|ballpark|collectors?|game of the year) edition|directors? ?cut|goty)\s*$', '', trimmed, flags=re.I) if trimmed == title: break title = trimmed return re.sub(r'\s+', ' ', title).strip(' .-') +EXPLICIT_YEAR = r'\(((?:19|20)\d{2})(?:\s+video game)?\)' + +def without_release_year(title): + return re.sub(r'\s*' + EXPLICIT_YEAR, '', title, flags=re.I) + def normalize(title): - title = re.sub(r'\s*\([^)]*\)', '', title.casefold()).replace('&', ' and ').replace('™', '').replace('®', '').replace('+', ' and ') + title = without_release_year(title).casefold().replace('&', ' and ').replace('™', '').replace('®', '').replace('+', ' and ') title = re.sub(r'^ea\s+sports\s+', '', title) title = ''.join(c for c in unicodedata.normalize('NFKD', title) if not unicodedata.combining(c)) roman = {'ii':'2', 'iii':'3', 'iv':'4', 'v':'5', 'vi':'6', 'vii':'7', 'viii':'8', 'ix':'9', 'x':'10'} title = re.sub(r'\b(?:ii|iii|iv|v|vi|vii|viii|ix|x)\b', lambda m: roman[m.group()], title) return re.sub(r'[^\w]+', '', title) -def steam_candidates(title): - query = urllib.parse.urlencode({'term': title, 'l': 'english', 'cc': 'us'}) - data = json.loads(fetch('https://store.steampowered.com/api/storesearch/?' + query, 262144)) - for item in data.get('items', [])[:20]: - if normalize(clean_title(item.get('name', ''))) != normalize(title): continue - app = str(item.get('id', '')) - if not app.isdigit(): continue - yield {'provider': 'Steam', 'title': item['name'], 'id': app, - 'article': 'https://store.steampowered.com/app/' + app, - 'image': 'https://cdn.akamai.steamstatic.com/steam/apps/' + app + '/library_600x900.jpg'} - details = json.loads(fetch('https://store.steampowered.com/api/appdetails?appids=' + app + '&l=english', 1048576)).get(app, {}).get('data', {}) - if normalize(clean_title(details.get('name', ''))) == normalize(title) and details.get('header_image'): - yield {'provider': 'Steam', 'title': item['name'], 'id': app, 'kind': 'banner', - 'article': 'https://store.steampowered.com/app/' + app, 'image': details['header_image']} - - -def wikipedia_candidates(title): - capital = title.title() - capital = re.sub(r'\b(?:Ii|Iii|Iv|Vi|Vii|Viii|Ix|Nhl|Nba|Nfl)\b', lambda m: m.group().upper(), capital) - common = dict(action='query', format='json', formatversion=2, prop='pageimages', redirects=1, - piprop='thumbnail', pithumbsize=320, pilicense='any', pilimit=10) - queries = [dict(common, titles='|'.join(dict.fromkeys([title, capital, capital+' (video game)']))), - dict(common, generator='search', gsrsearch='intitle:' + title, gsrlimit=10, gsrnamespace=0)] - seen=set() - for query in queries: - data = json.loads(fetch('https://en.wikipedia.org/w/api.php?' + urllib.parse.urlencode(query), 262144)) - for page in data.get('query', {}).get('pages', []): - if normalize(page.get('title', '')) != normalize(title) or re.search(r'\((?:series|franchise)\)', page.get('title', ''), re.I): continue - image = page.get('thumbnail', {}).get('source') - if image and image not in seen: - seen.add(image) - yield {'provider': 'Wikipedia', 'title': page['title'], 'id': page.get('pageid'), - 'article': 'https://en.wikipedia.org/wiki/' + urllib.parse.quote(page['title']), 'image': image} - -def playstation_candidates(title): - # Operator-verified product identities, never a URL supplied by the client. - try: sources = json.loads(SOURCES.read_text()) +def match_key(title): + # Scene names commonly omit conjunctions and articles. Keep every number, + # subtitle and edition word; a partial title/sequel is never sufficient. + title = re.sub(r'(?<=\d)\s+plus\s+(?=\d)', ' ', title, flags=re.I) + return normalize(re.sub(r'\b(?:and|the)\b|[&+]', ' ', title, flags=re.I)) + +def canonical_title(raw): + title = clean_title(raw) + try: + aliases = json.loads(ALIASES.read_text()) + scoped = 'ps5:' + normalize(title) if re.search(r'\b(?:PS5|PPSA\d+)\b', raw, re.I) else '' + title = aliases.get(scoped, aliases.get(normalize(title), title)) + except (OSError, ValueError): pass + return title + +def is_collection(title): + # These Prowlarr releases contain many unrelated games, not one title. + return bool(re.match(r'^(?:(?:Nintendo\s+)?(?:MIG\s+)?Switch(?:\s+MIG)?\s+(?:Game\s+)?(?:Library|Collection)\b|Top\s+\d+\s+Switch\s+Games\b|XBOX\s*360\s+Peer Pressure\b)', title, re.I)) + +def sgdb(path): + try: key = STEAMGRIDDB_KEY.read_text().strip() + except OSError: raise RuntimeError('SteamGridDB API key is not configured') from None + if not re.fullmatch(r'[A-Za-z0-9_-]{16,128}', key): + raise RuntimeError('SteamGridDB API key is not configured') + data = json.loads(fetch('https://www.steamgriddb.com/api/v2/' + path, 1048576, + {'Authorization': 'Bearer ' + key})) + if not data.get('success') or not isinstance(data.get('data'), list): + raise RuntimeError('SteamGridDB request failed') + return data['data'] + + +def game(title, cache): + # Overrides disambiguate remakes/editions without accepting arbitrary URLs. + try: override = json.loads(GAMES.read_text()).get(normalize(title)) + except (OSError, ValueError): override = None + if override is not None: + if type(override) is not int or override <= 0: raise ValueError('Invalid SteamGridDB game ID') + return override + try: + previous = json.loads(cache.read_text()) + if previous.get('title') == title and time.time() - previous['saved'] < 2592000: + return previous['id'] + except (OSError, ValueError, KeyError, TypeError): pass + # Only an explicit parenthesized year disambiguates releases. "2077" in + # Cyberpunk and "2024" in Football Cup are parts of their actual names. + year = re.search(EXPLICIT_YEAR, title, re.I) + query = without_release_year(title) + queries = [query] + without_joiners = re.sub(r'\b(?:and|the)\b|[&+]', ' ', query, flags=re.I) + without_joiners = re.sub(r'\s+', ' ', without_joiners).strip() + if without_joiners != query: queries.append(without_joiners) + words = without_joiners.split() + if len(words) > 4: + tail = ' '.join(words[-3:]) + if tail not in queries: queries.append(tail) + matches = [] + for search in queries: + rows = [item for item in sgdb('search/autocomplete/' + urllib.parse.quote(search, safe='')) + if type(item.get('id')) is int and item['id'] > 0] + if year: + rows = [item for item in rows if time.gmtime(item.get('release_date') or 0).tm_year == int(year[1])] + # SGDB names are canonical, not scene filenames. Cleaning them merged + # originals with editions, DLC and even bracketed fan demakes. + matches = [item for item in rows if normalize(item.get('name', '')) == normalize(title)] + if not matches: + matches = [item for item in rows if match_key(item.get('name', '')) == match_key(title)] + if matches: break + ids = {item['id'] for item in matches} + if len(ids) != 1: return None # Never silently substitute another sequel/remake. + ident = ids.pop() + save_json(cache, {'title': title, 'id': ident, 'saved': time.time()}) + return ident + + +def candidates(ident, kind): + dimensions = '600x900,660x930,342x482' if kind == 'cover' else '1920x620,3840x1240,1600x650' + query = urllib.parse.urlencode({'dimensions': dimensions, 'types': 'static', + 'mimes': 'image/png,image/jpeg,image/webp', 'nsfw': 'false', 'humor': 'false'}) + rows = sgdb(('grids' if kind == 'cover' else 'heroes') + '/game/' + str(ident) + '?' + query) + for item in sorted(rows, key=lambda row: (kind != 'cover' or (row.get('width'), row.get('height')) == (600, 900), row.get('score', 0)), reverse=True): + if item.get('nsfw') or item.get('humor') or item.get('epilepsy') or item.get('animated'): continue + size = (item.get('width'), item.get('height')) + if size not in ([(600, 900), (660, 930), (342, 482)] if kind == 'cover' else [(1920, 620), (3840, 1240), (1600, 650)]): continue + if not allowed(item.get('url', '')) or urllib.parse.urlsplit(item['url']).hostname == 'www.steamgriddb.com': continue + yield {'provider': 'SteamGridDB', 'game': ident, 'id': item['id'], 'image': item['url'], + 'width': size[0], 'height': size[1], 'kind': kind} + + +def steam_candidates(title, kind, refresh=False): + """Use the publisher's library assets, never a cropped store banner.""" + cache = CACHE / (hashlib.sha256(('steam-game-v1:' + title.casefold()).encode()).hexdigest() + '.steam.json') + fields = ('library_capsule_2x', 'library_capsule') if kind == 'cover' else ('library_hero', 'library_hero_2x') + item = None + try: + previous = json.loads(cache.read_text()) + assets = previous['item'].get('assets', {}) + ttl = 2592000 if any(assets.get(field) for field in fields) else 3600 + if not refresh and previous.get('title') == title and time.time() - previous['saved'] < ttl: + item = previous['item'] + except (OSError, ValueError, KeyError, TypeError): pass + if item is None: + query = without_release_year(title) + url = 'https://store.steampowered.com/api/storesearch/?' + urllib.parse.urlencode({'term': query, 'l': 'english', 'cc': 'fr'}) + rows = json.loads(fetch(url, 1048576)).get('items', []) + rows = [row for row in rows if row.get('type') == 'app' and type(row.get('id')) is int and row['id'] > 0] + matches = [row for row in rows if normalize(row.get('name', '')) == normalize(title)] + if not matches: matches = [row for row in rows if match_key(row.get('name', '')) == match_key(title)] + ids = sorted({row['id'] for row in matches}) + if not ids or len(ids) > 10: return + request = {'ids': [{'appid': ident} for ident in ids], + 'context': {'language': 'english', 'country_code': 'FR', 'steam_realm': 1}, + 'data_request': {'include_assets': True, 'include_release': True}} + url = 'https://api.steampowered.com/IStoreBrowseService/GetItems/v1/?' + urllib.parse.urlencode({'input_json': json.dumps(request, separators=(',', ':'))}) + rows = json.loads(fetch(url, 1048576)).get('response', {}).get('store_items', []) + year = re.search(EXPLICIT_YEAR, title, re.I) + # GetItems exposes item_type (0 = app) separately from app subtype + # in type (0 = game, 1 = demo, 4 = DLC), not an app_type field. + matches = [row for row in rows if row.get('success') == 1 + and row.get('item_type', 0) == 0 and row.get('type') == 0 + and row.get('appid') in ids and match_key(row.get('name', '')) == match_key(title) + and (not year or time.gmtime(row.get('release', {}).get('steam_release_date') or 0).tm_year == int(year[1]))] + if len(matches) != 1: return + item = {key: matches[0][key] for key in ('appid', 'name')} + item['assets'] = matches[0].get('assets') or {} + save_json(cache, {'title': title, 'item': item, 'saved': time.time()}) + ident = item['appid']; assets = item['assets'] + template = assets.get('asset_url_format', '') + if not template: return + # Modern assets have a different hash directory for each library image. + if not re.fullmatch(r'steam/apps/' + str(ident) + r'/\$\{FILENAME\}(?:\?t=\d+)?', template): + raise ValueError('Unexpected Steam asset template') + seen = set() + for field in fields: + filename = assets.get(field, '') + if not filename or filename in seen: continue + seen.add(filename) + url = 'https://shared.akamai.steamstatic.com/store_item_assets/' + template.replace('${FILENAME}', filename) + if not allowed(url): raise ValueError('Unexpected Steam asset path') + yield {'provider': 'Steam', 'game': ident, 'id': ident, 'image': url, 'kind': kind} + + + +def playstation_candidates(title, kind): + # Only operator-verified product identities, never a URL supplied by the UI. + try: url = json.loads(PLAYSTATION_GAMES.read_text()).get(normalize(title)) except (OSError, ValueError): return - url = sources.get(normalize(title)) if not url: return - parsed = urllib.parse.urlsplit(url) - if parsed.hostname != 'store.playstation.com' or not re.fullmatch(r'/[a-z]{2}-[a-z]{2}/product/[A-Z0-9_-]+/', parsed.path): + if not allowed(url) or urllib.parse.urlsplit(url).hostname != 'store.playstation.com': raise ValueError('Invalid PlayStation product') - document = fetch(url, 2097152).decode('utf-8') - for body in re.findall(r']+type=["\']application/ld\+json["\'][^>]*>(.*?)', document, re.S | re.I): - product = json.loads(html.unescape(body)) - if product.get('@type') != 'Product' or normalize(clean_title(product.get('name', ''))) != normalize(title): continue - if isinstance(product.get('image'), str): - yield {'provider': 'PlayStation', 'title': product['name'], 'id': product.get('sku'), 'article': url, 'image': product['image']} - - -def usable_rgb(data): - if len(data) != 115200: return False - # Some CDNs return a successful JPEG response containing only a grey tile. - # Allow compression noise, but require actual contrast in at least one channel. + product_id = urllib.parse.urlsplit(url).path.rstrip('/').rsplit('/', 1)[-1] + cache = CACHE / (hashlib.sha256(('ps-game-v1:' + title.casefold()).encode()).hexdigest() + '.ps.json') + item = None + try: + previous = json.loads(cache.read_text()) + if previous.get('source') == url and previous.get('title') == title and time.time() - previous['saved'] < 2592000: + item = previous['item'] + except (OSError, ValueError, KeyError, TypeError): pass + if item is None: + document = fetch(url, 2097152).decode('utf-8') + # The JSON-LD SKU verifies the edition; concept/recommendation art can + # belong to the base game or another product and must not be selected. + verified = False + for body in re.findall(r']+type=["\']application/ld\+json["\'][^>]*>(.*?)', document, re.S | re.I): + product = json.loads(html.unescape(body)) + if (isinstance(product, dict) and product.get('@type') == 'Product' + and product.get('sku') == product_id + and match_key(clean_title(product.get('name', ''))) == match_key(title)): + verified = True; break + if not verified: raise ValueError('PlayStation product identity mismatch') + script = re.search(r']+id=["\']__NEXT_DATA__["\'][^>]*>(.*?)', document, re.S | re.I) + if not script: raise ValueError('PlayStation media metadata unavailable') + sections = json.loads(script[1])['props']['pageProps']['batarangs'] + for section in ('background-image', 'media-carousel'): + text = sections.get(section, {}).get('text', '') + embedded = re.search(r']+type=["\']application/json["\'][^>]*>(.*?)', text, re.S | re.I) + if not embedded: continue + data = json.loads(embedded[1]) + if data.get('args', {}).get('productId') != product_id: continue + product = data.get('cache', {}).get('Product:' + product_id, {}) + if (product.get('id') == product_id and match_key(clean_title(product.get('name', ''))) == match_key(title) + and isinstance(product.get('media'), list)): + item = {'id': product_id, 'media': product['media']}; break + if item is None: raise ValueError('PlayStation product media unavailable') + save_json(cache, {'title': title, 'source': url, 'item': item, 'saved': time.time()}) + roles = ('PORTRAIT_BANNER',) if kind == 'cover' else ('GAMEHUB_COVER_ART', 'EDITION_KEY_ART', 'BACKGROUND') + seen = set() + for role in roles: + for media in item['media']: + image = media.get('url', '') + if media.get('type') != 'IMAGE' or media.get('role') != role or image in seen: continue + if not allowed(image) or urllib.parse.urlsplit(image).hostname != 'image.api.playstation.com': + raise ValueError('Invalid PlayStation media source') + seen.add(image) + yield {'provider': 'PlayStation', 'game': item['id'], 'id': item['id'], 'image': image, + 'kind': kind, 'role': role, 'article': url} + + + +def usable_rgb(data, kind='cover'): + if len(data) != SIZES[kind][0] * SIZES[kind][1] * 3: return False return any(max(data[channel::3]) - min(data[channel::3]) > 8 for channel in range(3)) -def raster(candidate): - image = Image.open(io.BytesIO(fetch(candidate['image'], 2097152))).convert('RGB') + +def raster(candidate, kind='cover'): + image = Image.open(io.BytesIO(fetch(candidate['image'], 8388608))) + expected = (candidate.get('width'), candidate.get('height')) + valid_size = image.size in ([(300, 450), (600, 900), (1200, 1800)] if kind == 'cover' else [(1920, 620), (3840, 1240)]) if candidate.get('provider') == 'Steam' else image.size == expected + if candidate.get('provider') == 'PlayStation': + width, height = image.size + valid_size = (2 <= width <= 4096 and 2 <= height <= 4096 and + (abs(width / height - 2 / 3) < 0.02 if kind == 'cover' else 1.5 <= width / height <= 3.2)) + if getattr(image, 'n_frames', 1) != 1 or not valid_size: + raise ValueError('Unexpected artwork format') + image = image.convert('RGB') if not any(high - low > 8 for low, high in image.getextrema()): raise ValueError('Blank artwork placeholder') - data = ImageOps.pad(image, (160, 240), color=(18, 25, 35), method=Image.Resampling.LANCZOS).tobytes() - if not usable_rgb(data): raise ValueError('Blank artwork placeholder') + # Keep the entire hero: a central 16:9 crop can remove the actual illustration. + if kind == 'hero': + canvas = ImageOps.fit(image, SIZES[kind], method=Image.Resampling.LANCZOS).filter(ImageFilter.GaussianBlur(18)) + full = ImageOps.contain(image, SIZES[kind], method=Image.Resampling.LANCZOS) + canvas.paste(full, ((canvas.width-full.width)//2, (canvas.height-full.height)//2)) + data = canvas.tobytes() + else: + data = ImageOps.fit(image, SIZES[kind], method=Image.Resampling.LANCZOS).tobytes() + if not usable_rgb(data, kind): raise ValueError('Blank artwork placeholder') return data +def hero_quality(data): + # The app darkens its left side under text. Prefer art visible on the right, + # rather than a highly voted hero whose visible portion is almost flat colour. + image = Image.frombytes('RGB', SIZES['hero'], data) + return max(ImageStat.Stat(image.crop((image.width*3//5, image.height//5, image.width, image.height*4//5))).stddev) + + +def save_bytes(path, data): + # Unique siblings also protect shared provider metadata across processes. + temporary = None + try: + with tempfile.NamedTemporaryFile(dir=path.parent, prefix=path.name + '.', suffix='.tmp', delete=False) as stream: + temporary = pathlib.Path(stream.name) + stream.write(data) + temporary.replace(path) + finally: + if temporary is not None: temporary.unlink(missing_ok=True) + def save_json(path, data): - temp = path.with_suffix('.tmp-json'); temp.write_text(json.dumps(data)); temp.replace(path) + save_bytes(path, json.dumps(data).encode()) -def cover(title): - original = clean_title(title) - try: title = json.loads(ALIASES.read_text()).get(normalize(original), original) - except (OSError, ValueError): title = original +def cover(title, kind='cover'): + if kind not in SIZES: raise ValueError('Invalid artwork kind') + title = canonical_title(title) if not isinstance(title, str) or not title or len(title) > 200: raise ValueError('Invalid cover alias') - ident = hashlib.sha256(('v4:' + normalize(title)).encode()).hexdigest() - path = CACHE / (ident + '.rgb'); metadata = CACHE / (ident + '.json') - with LOCKS[int(ident[:2], 16) % len(LOCKS)]: + if is_collection(title): return b'' + # Fresh namespace: never reuse the previous mixed-provider images. + ident = hashlib.sha256((('sgdb-hero-v2:' if kind == 'hero' else 'sgdb-v1:') + title.casefold()).encode()).hexdigest() + game_ident = hashlib.sha256(('sgdb-v1:' + title.casefold()).encode()).hexdigest() + path = CACHE / (ident + '.' + kind + '.rgb'); metadata = CACHE / (ident + '.' + kind + '.json') + with LOCKS[int(game_ident[:2], 16) % len(LOCKS)]: try: previous = json.loads(metadata.read_text()) except (OSError, ValueError): previous = {} age = time.time() - path.stat().st_mtime if path.exists() else float('inf') - if path.exists() and (path.stat().st_size or previous.get('resolver') == 6) and age < (2592000 if path.stat().st_size else 3600): + if path.exists() and (previous.get('resolver') == RESOLVER or (previous.get('resolver') in (7, 8, 9, 10, 11) and path.stat().st_size > 0)) and age < (2592000 if path.stat().st_size else 3600): data = path.read_bytes() - if not data or usable_rgb(data): return data - try: previous = json.loads(metadata.read_text()) - except (OSError, ValueError): previous = {} - errors = []; deferred = [] - # Reuse the saved provider ID/image before doing name resolution again. - if previous.get('match'): + if not data or usable_rgb(data, kind): return data + errors = [] + match = previous.get('match') + if match and match.get('provider') == 'SteamGridDB' and match.get('kind') == kind: try: - result = raster(previous['match']); temporary = path.with_suffix('.tmp'); temporary.write_bytes(result); temporary.replace(path); return result + result = raster(match, kind) + save_bytes(path, result) + save_json(metadata, {**previous, 'resolver': RESOLVER, 'saved': time.time()}); return result except Exception as error: errors.append(type(error).__name__) - for provider in (playstation_candidates, steam_candidates, wikipedia_candidates): + selected = None + best_quality = -1 + try: + ident_game = game(title, CACHE / (game_ident + '.game.json')) + for index, candidate in enumerate(candidates(ident_game, kind) if ident_game else []): + if kind == 'hero' and index >= 3: break + try: result = raster(candidate, kind) + except Exception as error: errors.append(type(error).__name__); continue + quality = hero_quality(result) if kind == 'hero' else 100 + if quality > best_quality: selected = (candidate, result); best_quality = quality + if kind == 'cover' or quality >= 20: break + except Exception as error: errors.append(type(error).__name__) + if selected is None: try: - for candidate in provider(title): - if candidate.get('kind')=='banner': deferred.append(candidate); continue - try: result = raster(candidate) - except Exception as error: errors.append(type(error).__name__); continue - temporary = path.with_suffix('.tmp'); temporary.write_bytes(result); temporary.replace(path) - save_json(metadata, {'resolver': 6, 'requested': original, 'canonical': title, 'match': candidate, 'saved': time.time()}) - return result + # Resolve metadata again instead of pinning an expired saved URL. + refresh_needed = False + for candidate in steam_candidates(title, kind): + try: result = raster(candidate, kind) + except Exception as error: + errors.append(type(error).__name__) + refresh_needed |= isinstance(error, urllib.error.HTTPError) and error.code in (404, 410) + continue + selected = (candidate, result); break + if selected is None and refresh_needed: + for candidate in steam_candidates(title, kind, refresh=True): + try: result = raster(candidate, kind) + except Exception as error: errors.append(type(error).__name__); continue + selected = (candidate, result); break except Exception as error: errors.append(type(error).__name__) - for candidate in deferred: + if selected is None: try: - result = raster(candidate); temporary = path.with_suffix('.tmp'); temporary.write_bytes(result); temporary.replace(path) - save_json(metadata, {'resolver': 6, 'requested': original, 'canonical': title, 'match': candidate, 'saved': time.time()}); return result + for candidate in playstation_candidates(title, kind): + try: result = raster(candidate, kind) + except Exception as error: errors.append(type(error).__name__); continue + selected = (candidate, result); break except Exception as error: errors.append(type(error).__name__) - save_json(metadata, {'resolver': 6, 'requested': original, 'canonical': title, 'match': None, 'errors': errors, 'saved': time.time()}) - # Provider outages must not turn into permanent "no cover" results. - if errors: raise RuntimeError('Artwork providers temporarily unavailable') - path.write_bytes(b''); return b'' + if selected: + candidate, result = selected + save_bytes(path, result) + save_json(metadata, {'resolver': RESOLVER, 'canonical': title, 'match': candidate, 'saved': time.time()}) + return result + if errors: raise RuntimeError('Artwork temporarily unavailable') + # Confirmed misses expire after an hour. Authentication/outage errors above never do. + save_bytes(path, b'') + save_json(metadata, {'resolver': RESOLVER, 'canonical': title, 'match': None, 'saved': time.time()}) + return b'' class Handler(BaseHTTPRequestHandler): def log_message(self, *_): pass def do_GET(self): if not hmac.compare_digest(self.headers.get('X-Api-Key', ''), KEY): self.send_error(403); return - title = urllib.parse.parse_qs(urllib.parse.urlsplit(self.path).query).get('title', [''])[0] - if not title or len(title) > 200 or any(ord(c) < 32 for c in title): self.send_error(400); return - try: data = cover(title) + query = urllib.parse.parse_qs(urllib.parse.urlsplit(self.path).query) + title = query.get('title', [''])[0] + kind = query.get('kind', ['cover'])[0] + if kind not in SIZES or not title or len(title) > 200 or any(ord(c) < 32 for c in title): self.send_error(400); return + try: data = cover(title, kind) except Exception: self.send_error(503, 'Artwork temporarily unavailable'); return self.send_response(200); self.send_header('Content-Type', 'application/octet-stream') self.send_header('Content-Length', str(len(data))); self.end_headers(); self.wfile.write(data) diff --git a/deployment/botty-artwork/steamgriddb-games.json b/deployment/botty-artwork/steamgriddb-games.json new file mode 100644 index 0000000..14fc391 --- /dev/null +++ b/deployment/botty-artwork/steamgriddb-games.json @@ -0,0 +1,3 @@ +{ + "callofdutyblackops6": 5452948 +} diff --git a/deployment/botty-artwork/test_server.py b/deployment/botty-artwork/test_server.py index 725ab26..d8bc656 100644 --- a/deployment/botty-artwork/test_server.py +++ b/deployment/botty-artwork/test_server.py @@ -1,21 +1,27 @@ -import importlib.util, io, json, os, pathlib, tempfile, unittest +import concurrent.futures, importlib.util, io, json, os, pathlib, tempfile, threading, time, unittest +from unittest.mock import patch from PIL import Image class Covers(unittest.TestCase): def setUp(self): self.temp=tempfile.TemporaryDirectory();root=pathlib.Path(self.temp.name);(root/'key').write_text('test') + os.environ['BOTTY_STEAMGRIDDB_KEY_FILE']=str(root/'sgdb-key');(root/'sgdb-key').write_text('a'*32) + os.environ['BOTTY_PLAYSTATION_GAMES']=str(root/'ps-games.json') + os.environ['BOTTY_ARTWORK_ALIASES']=str(root/'aliases.json');os.environ['BOTTY_STEAMGRIDDB_GAMES']=str(root/'games.json') os.environ['BOTTY_ARTWORK_KEY_FILE']=str(root/'key');os.environ['BOTTY_ARTWORK_CACHE']=str(root) spec=importlib.util.spec_from_file_location('artwork',pathlib.Path(__file__).with_name('server.py'));self.module=importlib.util.module_from_spec(spec);spec.loader.exec_module(self.module) + self.steam_candidates=self.module.steam_candidates;self.module.steam_candidates=lambda *_:iter(()) def tearDown(self):self.temp.cleanup() - def test_exact_cover_and_cache(self): - image=io.BytesIO();picture=Image.new('RGB',(20,30),(20,60,80));picture.paste((220,180,100),(0,0,10,15));picture.save(image,format='PNG');calls=[] - def fetch(url,limit): - calls.append(url) - return json.dumps({'query':{'pages':[{'title':"Marvel's Demo",'thumbnail':{'source':'https://upload.wikimedia.org/test.png'}}]}}).encode() if 'api.php' in url else image.getvalue() - self.module.fetch=fetch - result=self.module.cover('marvels demo incl DLC');self.assertEqual(len(result),115200);self.assertEqual(self.module.cover('marvels demo incl DLC'),result);self.assertEqual(len(calls),3) - def test_never_substitutes_sequel(self): - self.module.fetch=lambda *_:json.dumps({'query':{'pages':[{'title':'Demo 2','thumbnail':{'source':'https://upload.wikimedia.org/test.png'}}]}}).encode() - self.assertEqual(self.module.cover('demo'),b'') + def test_audit_resolves_fallbacks_without_a_warm_image_cache(self): + spec=importlib.util.spec_from_file_location('artwork_audit',pathlib.Path(__file__).with_name('audit.py')) + audit=importlib.util.module_from_spec(spec);spec.loader.exec_module(audit) + for provider in ('Steam','PlayStation'): + with self.subTest(provider=provider), patch.object(self.module,'game',return_value=None), \ + patch.object(self.module,'steam_candidates',return_value=iter([{'game':123,'provider':'Steam'}] if provider=='Steam' else [])), \ + patch.object(self.module,'playstation_candidates',return_value=iter([{'game':'product','provider':'PlayStation'}] if provider=='PlayStation' else [])), \ + patch.object(self.module,'cover',side_effect=AssertionError('Metadata audit must not download images')): + result=audit.audit(self.module,'Fixture') + self.assertEqual(result['status'],'matched') + self.assertEqual(result['provider'],provider) def test_accents_and_punctuation(self): self.assertEqual(self.module.normalize("Ghost of Yōtei"), self.module.normalize("ghost of yotei")) self.assertEqual(self.module.normalize("Marvel's Wolverine"), self.module.normalize("marvels wolverine")) @@ -87,97 +93,354 @@ def test_punctuation_joined_scene_suffixes(self): self.assertEqual(self.module.normalize(self.module.clean_title(title)),self.module.normalize(title)) self.assertEqual(self.module.clean_title('Some Game-PROPER:REPACK-PPSA12345'),'Some Game') self.assertEqual(self.module.clean_title('Some Game Deluxe Edition-PROPER-PPSA12345'),'Some Game') - def test_steam_and_fallback(self): - image=io.BytesIO();picture=Image.new('RGB',(20,30),(20,60,80));picture.paste((220,180,100),(0,0,10,15));picture.save(image,format='PNG') - def fetch(url,limit): - if 'storesearch' in url:return json.dumps({'items':[{'id':1,'name':'Demo II'},{'id':2,'name':'Demo III'}]}).encode() - if '/apps/2/' in url:return image.getvalue() - raise AssertionError('Wrong sequel or unnecessary fallback') + def provider(self, missing_hero=False): + calls=[] + def fetch(url,limit,headers=None): + calls.append((url,headers)) + if '/api/v2/' in url: + self.assertEqual(headers,{'Authorization':'Bearer '+'a'*32}) + if '/search/' in url:rows=[{'id':42,'name':'Demo II','release_date':0},{'id':43,'name':'Demo III','release_date':0}] + else: + hero='/heroes/' in url + rows=[] if hero and missing_hero else [{'id':99,'width':1920 if hero else 600,'height':620 if hero else 900,'url':'https://cdn2.steamgriddb.com/'+('hero' if hero else 'cover')+'.png','score':5}] + return json.dumps({'success':True,'data':rows}).encode() + size=(1920,620) if 'hero' in url else (600,900) + image=Image.new('RGB',size,(20,60,80));image.paste((220,180,100),(0,0,size[0]//2,size[1]//2)) + out=io.BytesIO();image.save(out,format='PNG');return out.getvalue() self.module.fetch=fetch - self.assertEqual(len(self.module.cover('Demo 3')),115200) - def test_temporary_failure_retried(self): - self.module.steam_candidates=lambda _:iter([]) - def offline(_):raise OSError('offline') - self.module.wikipedia_candidates=offline - with self.assertRaises(RuntimeError):self.module.cover('Example') - self.module.wikipedia_candidates=lambda _:iter([{'provider':'Wikipedia','image':'https://upload.wikimedia.org/example.png'}]) - self.module.raster=lambda _:b'x'*115200 - self.assertEqual(len(self.module.cover('Example')),115200) - def test_persistent_alias(self): - aliases=pathlib.Path(self.temp.name)/'aliases.json';aliases.write_text(json.dumps({'regionalname':'Canonical Game'}));self.module.ALIASES=aliases - names=[] - def candidate(title):names.append(title);yield {'provider':'Steam','image':'https://cdn.akamai.steamstatic.com/example.jpg'} - self.module.steam_candidates=candidate;self.module.raster=lambda _:b'x'*115200 - self.assertEqual(len(self.module.cover('Regional Name')),115200);self.assertEqual(names,['Canonical Game']) - def test_verified_release_aliases(self): - self.module.ALIASES=pathlib.Path(__file__).with_name('aliases.json') - names=[] - def candidate(title):names.append(title);yield {'provider':'PlayStation','image':'fixture'} - self.module.playstation_candidates=candidate - self.module.raster=lambda _:bytes([20,40,60,200,180,160])*19200 - releases=[('eFootball 2027 PS5-PPSA03073[FPKG]','eFootball'), - ('Sonic Unleashed (recompiled) unofficial port PS5','Sonic Unleashed'), - ('sonic unleashed recompiled unofficial port','Sonic Unleashed'), - ('Resident Evil 4 remake PS5-PPSA07411[LIZARD]','Resident Evil 4 (2023 video game)'), - ('The Bearer The Last Flame PS5-PPSA18697[LIZARD]','The Bearer & The Last Flame'), - ('Snoopy The Great Mystery Club PS5-PPSA28795[FPKG]','Snoopy & The Great Mystery Club'), - ('WILD HEARTS Karakuri Edition PS5-PPSA07836[FPKG]','WILD HEARTS')] - for release,canonical in releases: - self.assertEqual(len(self.module.cover(release)),115200) - self.assertEqual(names,['eFootball','Sonic Unleashed','Resident Evil 4 (2023 video game)','The Bearer & The Last Flame','Snoopy & The Great Mystery Club','WILD HEARTS']) - aliases=json.loads(self.module.ALIASES.read_text()) - self.assertNotIn(self.module.normalize('eFootball 2026'),aliases) - self.assertNotIn(self.module.normalize('Resident Evil 3 remake'),aliases) - def test_playstation_identity(self): - sources=pathlib.Path(self.temp.name)/'sources.json';sources.write_text(json.dumps({'nhl27':'https://store.playstation.com/en-us/product/TEST/'}));self.module.SOURCES=sources - self.module.fetch=lambda *_:b'' - self.assertEqual(len(list(self.module.playstation_candidates('EA SPORTS™ NHL™ 27'))),1) - self.assertEqual(list(self.module.playstation_candidates('NHL 26')),[]) - self.module.fetch=lambda *_:b'' - self.assertEqual(list(self.module.playstation_candidates('NHL 27')),[]) - def test_origin_confinement(self): - with self.assertRaises(ValueError):self.module.fetch('http://localhost/private',100) - def test_sparta_release_uses_verified_playstation_source(self): - """Fetch the verified Sparta product and reject a deliberately wrong identity.""" - self.module.SOURCES=pathlib.Path(__file__).with_name('sources.json') + return calls + def test_portrait_hero_and_persistent_cache(self): + calls=self.provider() + cover=self.module.cover('Demo III PS5-PPSA12345');hero=self.module.cover('Demo III Switch NSP','hero') + self.assertEqual(len(cover),160*240*3);self.assertEqual(len(hero),640*360*3) + self.assertEqual(len([url for url,_ in calls if '/search/' in url]),1) + self.assertEqual(self.module.cover('Demo III','hero'),hero) + count=len(calls);self.assertEqual(self.module.cover('Demo III'),cover);self.assertEqual(len(calls),count) + self.assertTrue(any('dimensions=600x900' in url for url,_ in calls)) + metadata=''.join(p.read_text() for p in pathlib.Path(self.temp.name).glob('*.json')) + self.assertNotIn('a'*32,metadata);self.assertIn('SteamGridDB',metadata) + def test_missing_hero_does_not_poison_cover(self): + calls=self.provider(missing_hero=True) + self.assertEqual(self.module.cover('Demo III','hero'),b'') + self.assertEqual(len(self.module.cover('Demo III')),115200) + count=len(calls);self.assertEqual(self.module.cover('Demo III','hero'),b'');self.assertEqual(len(calls),count) + def test_missing_key_and_outage_are_retried(self): + self.module.STEAMGRIDDB_KEY.unlink() + with self.assertRaises(RuntimeError):self.module.cover('Demo III') + self.assertFalse(list(pathlib.Path(self.temp.name).glob('*.rgb'))) + self.module.STEAMGRIDDB_KEY.write_text('a'*32) + self.module.fetch=lambda *_:(_ for _ in ()).throw(OSError('offline')) + with self.assertRaises(RuntimeError):self.module.cover('Demo III') + self.provider();self.assertEqual(len(self.module.cover('Demo III')),115200) + def test_never_substitutes_sequel_or_ambiguous_remake(self): + self.module.sgdb=lambda _: [{'id':1,'name':'Demo II'},{'id':2,'name':'Demo III'},{'id':3,'name':'Demo III'}] + self.assertEqual(self.module.cover('Demo IV'),b'');self.assertEqual(self.module.cover('Demo III'),b'') + def test_year_and_manual_game_id(self): + self.module.sgdb=lambda _: [{'id':1,'name':'Demo','release_date':1104537600},{'id':2,'name':'Demo','release_date':1672531200}] + path=pathlib.Path(self.temp.name)/'match.json' + self.assertEqual(self.module.game('Demo (2023 video game)',path),2) + self.module.GAMES.write_text(json.dumps({'demo':123})) + self.assertEqual(self.module.game('Demo',path),123) + def test_aliases_and_switch_release_names(self): + self.assertEqual(self.module.clean_title('Demo III [0100000000000000] Switch NSP'), 'Demo III') + self.assertEqual(self.module.clean_title('1-2-Switch'), '1-2-Switch') + self.assertEqual(self.module.clean_title('Nintendo Switch Sports'), 'Nintendo Switch Sports') + self.assertEqual(self.module.clean_title('Castlevania.Belmonts.Curse.DLC.Unlocker.PS5'), 'Castlevania Belmonts Curse') + self.assertEqual(self.module.clean_title('Castlevania Belmonts Curse DLC Unlocker'), 'Castlevania Belmonts Curse') + self.module.ALIASES.write_text(json.dumps({'regionalname':'Demo III'}));self.provider() + self.assertEqual(len(self.module.cover('Regional Name')),115200) + def test_real_prowlarr_platform_suffixes(self): + cases = { + 'My Cozy Aquarium NSW-SUXXORS': 'My Cozy Aquarium', + 'Mario Kart 8 Deluxe (RF) PROPER NSW Switch': 'Mario Kart 8 Deluxe', + 'Yooka Laylee (eShop)(EU) NSW Switch': 'Yooka Laylee', + 'Bayonetta 2 NSW Switch': 'Bayonetta 2', + 'Dungeons 4 Nintendo Switch Edition Update v1 11 NSW-SUXXORS': 'Dungeons 4', + 'Hello Kitty Island Adventure Base Game + Update v1 0 NSW': 'Hello Kitty Island Adventure', + 'Castlevania Belmonts Curse Digital Sound and Art Gallery PS5-DUPLEX': 'Castlevania Belmonts Curse', + 'I Am Your Beast NSW-VENOM': 'I Am Your Beast', + 'Urban Jungle NSW-BREWS': 'Urban Jungle', + 'Football Cup 2024 NSW-SUXXORS': 'Football Cup 2024', + 'Castlevania Dominus Collection NSW-VENOM': 'Castlevania Dominus Collection', + 'Vertigo 2 VR2 PS5-DUPLEX': 'Vertigo 2', + 'BEYBLADE X EVOBATTLE v1 1 3 DLC Unlocker NSW-SUXXORS': 'BEYBLADE X EVOBATTLE', + } + for raw, expected in cases.items(): + with self.subTest(raw=raw): self.assertEqual(self.module.clean_title(raw), expected) + for canonical in ['Nintendo Switch Sports', '1-2-Switch', 'Everybody 1-2-Switch!', 'VR2']: + self.assertEqual(self.module.clean_title(canonical), canonical) + def test_real_prowlarr_matching(self): + cases = [ + ('Cyberpunk 2077', [{'id':5209422,'name':'Cyberpunk 2077','release_date':1607558400}], 5209422), + ('Football Cup 2024', [{'id':42,'name':'Football Cup 2024','release_date':1672531200}], 42), + ('Divinity Original Sin 2', [{'id':10425,'name':'Divinity: Original Sin 2'}, {'id':5264646,'name':'Divinity: Original Sin 2 - Definitive Edition'}],10425), + ('Hollow Knight Silksong',[{'id':1043,'name':'Hollow Knight: Silksong'}, {'id':5512976,'name':'Hollow Knight Silksong [Gameboy Demake]'}],1043), + ('Ratchet Clank Rift Apart',[{'id':5266059,'name':'Ratchet & Clank: Rift Apart'}],5266059), + ('Tony Hawks Pro Skater 3 4',[{'id':5480866,'name':"Tony Hawk's Pro Skater 3 + 4"}],5480866), + ('KINGDOM HEARTS HD 1 5 2 5 ReMIX',[{'id':5250081,'name':'Kingdom Hearts HD 1.5 + 2.5 ReMIX'}],5250081), + ('The Relic First Guardian',[{'id':5450064,'name':'The Relic: The First Guardian'}],5450064), + ] + for title, rows, expected in cases: + with self.subTest(title=title): + self.module.sgdb=lambda _, rows=rows:rows + self.assertEqual(self.module.game(title,pathlib.Path(self.temp.name)/str(expected)),expected) + self.module.sgdb=lambda _: [{'id':1,'name':'Demo II'}, {'id':2,'name':'Demo III'}] + self.assertIsNone(self.module.game('Demo IV',pathlib.Path(self.temp.name)/'missing')) + def test_query_joiners_retry_without_substituting_a_sequel(self): calls=[] - def fetch(url,limit): - calls.append(url) - return b'' + def search(path): + calls.append(path) + return [{'id':2,'name':'Ratchet & Clank: Rift Apart'}] if 'Ratchet%20Clank%20Rift' in path else [{'id':1,'name':'Ratchet & Clank'}] + self.module.sgdb=search + self.assertEqual(self.module.game('Ratchet and Clank Rift Apart',pathlib.Path(self.temp.name)/'game'),2) + self.assertEqual(len(calls),2) + def test_scoped_remake_alias_and_explicit_year(self): + self.module.ALIASES.write_text(json.dumps({'ps5:residentevil4':'Resident Evil 4 (2023)'})) + self.assertEqual(self.module.canonical_title('Resident Evil 4 PS5-DUPLEX'),'Resident Evil 4 (2023)') + self.assertEqual(self.module.canonical_title('Resident Evil 4 NSW-VENOM'),'Resident Evil 4') + self.module.sgdb=lambda _: [{'id':3168,'name':'Resident Evil 4','release_date':1105488000}, {'id':5332120,'name':'Resident Evil 4','release_date':1679616000}] + self.assertEqual(self.module.game('Resident Evil 4 (2023)',pathlib.Path(self.temp.name)/'game'),5332120) + self.assertIsNone(self.module.game('Resident Evil 4',pathlib.Path(self.temp.name)/'ambiguous')) + def test_alternate_portraits_are_uniform_and_prefer_standard(self): + self.module.sgdb=lambda _: [ + {'id':1,'width':660,'height':930,'score':100,'url':'https://cdn2.steamgriddb.com/one.png'}, + {'id':2,'width':600,'height':900,'score':1,'url':'https://cdn2.steamgriddb.com/two.png'}, + {'id':3,'width':342,'height':482,'score':10,'url':'https://cdn2.steamgriddb.com/three.png'}, + {'id':4,'width':1024,'height':1024,'score':100,'url':'https://cdn2.steamgriddb.com/square.png'}, + ] + self.assertEqual([c['id'] for c in self.module.candidates(42,'cover')],[2,1,3]) + for size in [(660,930),(342,482)]: + image=Image.new('RGB',size,(10,20,40));image.paste((200,30,10),(0,0,size[0]//2,size[1])) + source=io.BytesIO();image.save(source,format='PNG');self.module.fetch=lambda *_:source.getvalue() + self.assertEqual(len(self.module.raster({'image':'fixture','width':size[0],'height':size[1]})),115200) + def test_additional_actual_release_suffixes(self): + for raw,expected in [ + ('7 Days to Die Console Edition PS5-PPSA16645[FPKG]','7 Days to Die'), + ('Castlevania Belmonts Curse Midnight Edition NSW-VENOM','Castlevania Belmonts Curse'), + ('Castlevania Belmonts Curse DEMO NSW-VENOM','Castlevania Belmonts Curse'), + ('Frostpunk All Expansions Pack NSW-SUXXORS','Frostpunk'), + ('METAL GEAR SOLID 4 Guns of the Patriots Master Collection Version Incl DLC PS5-DUPLEX','METAL GEAR SOLID 4 Guns of the Patriots'), + ]: + with self.subTest(raw=raw): self.assertEqual(self.module.clean_title(raw),expected) + self.assertEqual(self.module.match_key('KINGDOM HEARTS HD 1 5 plus 2 5 ReMIX'),self.module.match_key('Kingdom Hearts HD 1.5 + 2.5 ReMIX')) + def test_collection_packs_do_not_match_an_arbitrary_game(self): + for raw in ['Switch Library Update Pack 270', 'Nintendo MIG Switch Game Collection / Vol 1', 'Nintendo Switch Game Library July 2019 Vol 1', 'Top 70 Switch Games of all time']: + with self.subTest(raw=raw): self.assertEqual(self.module.cover(raw),b'') + for raw in ['Castlevania Dominus Collection', 'Metal Gear Solid Master Collection Vol. 1', 'Nintendo Switch Sports']: + self.assertFalse(self.module.is_collection(raw)) + def test_search_tail_still_requires_the_complete_title(self): + self.module.sgdb=lambda path: [{'id':1,'name':'Other Game: Special Subtitle Name'}, {'id':2,'name':'Some Game with Special Subtitle Name'}] if path.endswith('Special%20Subtitle%20Name') else [] + self.assertEqual(self.module.game('Some Game with Special Subtitle Name',pathlib.Path(self.temp.name)/'game'),2) + def steam_provider(self, title='EA SPORTS FC 27', appid=4080220, returned_name=None, app_type=0): + self.module.steam_candidates=self.steam_candidates;self.module.sgdb=lambda _:[] + calls=[];assets={'asset_url_format':'steam/apps/'+str(appid)+'/${FILENAME}?t=123', + 'library_capsule_2x':'a'*40+'/library_capsule_2x.jpg', + 'library_capsule':'b'*40+'/library_capsule.jpg','library_hero':'c'*40+'/library_hero.jpg'} + def fetch(url,limit,headers=None): + calls.append((url,headers));self.assertIsNone(headers) + if '/storesearch/' in url: + return json.dumps({'items':[{'id':appid,'name':title,'type':'app'},{'id':99,'name':'EA SPORTS FC 26','type':'app'}]}).encode() + if '/GetItems/' in url: + return json.dumps({'response':{'store_items':[{'appid':appid,'name':returned_name or title,'item_type':0,'type':app_type,'success':1,'assets':assets,'release':{'steam_release_date':1790290781}}]}}).encode() + size=(1920,620) if 'library_hero' in url else (600,900) + im=Image.new('RGB',size,(10,20,40));im.paste((240,180,20),(0,0,size[0]//2,size[1])) + out=io.BytesIO();im.save(out,format='PNG');return out.getvalue() + self.module.fetch=fetch + return calls,assets + def test_steam_fallback_covers_and_heroes_with_modern_asset_hashes(self): + for title,appid in [('EA SPORTS FC 27',4080220),('BLUE REFLECTION Quartet',3825390)]: + with self.subTest(title=title): + calls,_=self.steam_provider(title,appid) + self.assertEqual(len(self.module.cover(title)),115200) + self.assertEqual(len(self.module.cover(title,'hero')),691200) + self.assertEqual(sum('/storesearch/' in u for u,_ in calls),1) + self.assertTrue(any('/'+str(appid)+'/'+'a'*40+'/library_capsule_2x.jpg' in u for u,_ in calls)) + count=len(calls);self.assertEqual(len(self.module.cover(title)),115200);self.assertEqual(len(calls),count) + metadata=[json.loads(p.read_text()) for p in pathlib.Path(self.temp.name).glob('*.cover.json')] + self.assertTrue(any(p['match']['provider']=='Steam' and p['match']['game']==appid for p in metadata)) + def test_steam_does_not_substitute_another_year_or_dlc(self): + self.steam_provider(returned_name='EA SPORTS FC 26') + self.assertEqual(self.module.cover('EA SPORTS FC 27'),b'') + # Actual GetItems app subtypes: both are still item_type=0 (app). + for title, subtype in [('BLUE REFLECTION Quartet',1),('NHL 27',4)]: + self.steam_provider(title=title,appid=3825390,app_type=subtype) + self.assertEqual(self.module.cover(title),b'') + + def test_steam_fallback_does_not_override_steamgriddb(self): + self.provider() + self.module.steam_candidates=lambda *_:self.fail('SteamGridDB must stay first') + self.assertEqual(len(self.module.cover('Demo III')),115200) + def test_invalid_steam_assets_and_square_images_are_rejected(self): + _,assets=self.steam_provider() + assets['library_capsule_2x']='../private.jpg' + with self.assertRaisesRegex(ValueError,'asset path'):list(self.module.steam_candidates('EA SPORTS FC 27','cover')) + for url in ['https://shared.akamai.steamstatic.com/store_item_assets/steam/apps/123/header.jpg', + 'https://api.steampowered.com/other-api', + 'https://shared.akamai.steamstatic.com.evil.test/store_item_assets/steam/apps/123/library_capsule.jpg']: + self.assertFalse(self.module.allowed(url)) + im=Image.new('RGB',(1024,1024),(10,20,40));out=io.BytesIO();im.save(out,format='PNG') + self.module.fetch=lambda *_:out.getvalue() + with self.assertRaisesRegex(ValueError,'Unexpected artwork format'):self.module.raster({'provider':'Steam','image':'fixture'}) + def test_steam_outage_does_not_become_a_confirmed_miss(self): + self.steam_provider();self.module.fetch=lambda *_:(_ for _ in ()).throw(OSError('offline')) + with self.assertRaises(RuntimeError):self.module.cover('EA SPORTS FC 27') + self.assertFalse(list(pathlib.Path(self.temp.name).glob('*.rgb'))) + self.steam_provider();self.assertEqual(len(self.module.cover('EA SPORTS FC 27')),115200) + def test_legacy_steam_library_names_and_missing_double_resolution(self): + _,assets=self.steam_provider(title='Afterdream',appid=1995880) + assets.update(library_capsule_2x='library_600x900_2x.jpg',library_capsule='library_600x900.jpg') + fetch=self.module.fetch;missing=[] + def request(url,*args): + if 'library_600x900_2x.jpg' in url: + missing.append(url);raise self.module.urllib.error.HTTPError(url,404,'missing',{},None) + return fetch(url,*args) + self.module.fetch=request + self.assertEqual(len(self.module.cover('Afterdream')),115200) + self.assertEqual(len(missing),1) + image=Image.new('RGB',(1200,1800),(10,20,40));image.paste((200,30,10),(0,0,600,1800)) + source=io.BytesIO();image.save(source,format='PNG');self.module.fetch=lambda *_:source.getvalue() + self.assertEqual(len(self.module.raster({'provider':'Steam','image':'fixture'})),115200) + def playstation_provider(self, name='NHL® 27 Standard Edition PS5', product_id='UP0006-PPSA34063_00-NHL27STANDEDITON'): + self.module.sgdb=lambda _:[] + url='https://store.playstation.com/en-us/product/'+product_id+'/' + self.module.PLAYSTATION_GAMES.write_text(json.dumps({'nhl27':url,'dragonsdogma2darkarisen':url})) + image='https://image.api.playstation.com/vulcan/ap/rnd/202610/0000/' + product={'id':product_id,'name':name,'media':[ + {'role':'MASTER','type':'IMAGE','url':image+'square.png'}, + {'role':'GAMEHUB_COVER_ART','type':'IMAGE','url':image+'hero.png'}, + {'role':'PORTRAIT_BANNER','type':'IMAGE','url':image+'portrait.png'}, + ]} + envelope={'args':{'productId':product_id},'cache':{'Product:'+product_id:product, + 'Concept:other':{'media':[{'role':'PORTRAIT_BANNER','type':'IMAGE','url':image+'wrong-game.png'}]}}} + ld={'@type':'Product','name':name,'sku':product_id} + calls=[] + def fetch(target,limit,headers=None): + calls.append(target);self.assertIsNone(headers) + if target==url: + text='' + nxt={'props':{'pageProps':{'batarangs':{'background-image':{'text':text}}}}} + return ('').encode() + self.assertNotIn('wrong-game',target);self.assertNotIn('square',target) + size=(720,1080) if 'portrait' in target else (1920,1080) + im=Image.new('RGB',size,(10,20,40));im.paste((230,160,20),(0,0,size[0]//2,size[1])) + out=io.BytesIO();im.save(out,format='PNG');return out.getvalue() + self.module.fetch=fetch + return calls,ld,product + def test_playstation_verified_portrait_and_product_specific_background(self): + for title,name,ident in [('NHL 27','NHL® 27 Standard Edition PS5','UP0006-PPSA34063_00-NHL27STANDEDITON'), + ('Dragons Dogma 2 Dark Arisen',"Dragon's Dogma 2: Dark Arisen",'UP0102-PPSA09663_00-DD2DA00000000000')]: + calls,_,_=self.playstation_provider(name,ident) + self.assertEqual(len(self.module.cover(title)),115200) + self.assertEqual(len(self.module.cover(title,'hero')),691200) + self.assertEqual(sum('store.playstation.com' in url for url in calls),1) + count=len(calls);self.assertEqual(len(self.module.cover(title)),115200);self.assertEqual(count,len(calls)) + meta=[json.loads(p.read_text()) for p in pathlib.Path(self.temp.name).glob('*.cover.json')] + self.assertTrue(any(x['match']['provider']=='PlayStation' and x['match']['game']==ident for x in meta)) + def test_playstation_does_not_accept_another_edition_or_untrusted_source(self): + _,ld,product=self.playstation_provider() + ld['sku']='wrong' + with self.assertRaisesRegex(ValueError,'identity mismatch'):list(self.module.playstation_candidates('NHL 27','cover')) + _,ld,product=self.playstation_provider(name="Dragon's Dogma 2") + with self.assertRaisesRegex(ValueError,'identity mismatch'):list(self.module.playstation_candidates('Dragons Dogma 2 Dark Arisen','cover')) + self.module.PLAYSTATION_GAMES.write_text(json.dumps({'nhl27':'https://evil.test/en-us/product/UP0006-PPSA34063_00-NHL27STANDEDITON/'})) + with self.assertRaisesRegex(ValueError,'Invalid PlayStation product'):list(self.module.playstation_candidates('NHL 27','cover')) + self.assertFalse(self.module.allowed('https://image.api.playstation.com.evil.test/vulcan/ap/rnd/202610/0000/image.png')) + def test_playstation_remains_after_steam(self): + self.steam_provider() + self.module.playstation_candidates=lambda *_:self.fail('PlayStation must not replace an available Steam image') + self.assertEqual(len(self.module.cover('EA SPORTS FC 27')),115200) + def test_parenthesized_fan_game_is_not_the_base_game(self): + self.module.sgdb=lambda _: [{'id':5512976,'name':'Hollow Knight Silksong (Gameboy Demake)'}] + self.assertIsNone(self.module.game('Hollow Knight Silksong',pathlib.Path(self.temp.name)/'game')) + self.assertNotEqual(self.module.match_key('Hollow Knight Silksong (Gameboy Demake)'),self.module.match_key('Hollow Knight Silksong')) + self.assertEqual(self.module.normalize('Resident Evil 4 (2023 video game)'),self.module.normalize('Resident Evil 4')) + self.steam_provider(title='Hollow Knight Silksong (Gameboy Demake)',appid=12) + self.assertEqual(self.module.cover('Hollow Knight Silksong'),b'') + def test_failed_saved_steam_url_refreshes_metadata(self): + calls,assets=self.steam_provider() + self.module.cover('EA SPORTS FC 27') + image=next(pathlib.Path(self.temp.name).glob('*.cover.rgb'));os.utime(image,(0,0)) + assets['library_capsule_2x']='d'*40+'/library_capsule_2x.jpg' + assets['library_capsule']='e'*40+'/library_capsule.jpg' + fetch=self.module.fetch + def request(url,*args): + if '/'+('a'*40)+'/' in url or '/'+('b'*40)+'/' in url: + raise self.module.urllib.error.HTTPError(url,404,'removed',{},None) + return fetch(url,*args) + self.module.fetch=request + self.assertEqual(len(self.module.cover('EA SPORTS FC 27')),115200) + self.assertEqual(sum('/GetItems/' in url for url,_ in calls),2) + metadata=json.loads(next(pathlib.Path(self.temp.name).glob('*.cover.json')).read_text()) + self.assertIn('d'*40,metadata['match']['image']) + def test_atomic_metadata_writes_use_unique_temporary_files(self): + target=pathlib.Path(self.temp.name)/'concurrent.json';barrier=threading.Barrier(2) + replace=pathlib.Path.replace;temporaries=[] + def synchronized_replace(source,destination): + temporaries.append(str(source));barrier.wait(timeout=5) + return replace(source,destination) + with patch.object(pathlib.Path,'replace',synchronized_replace): + with concurrent.futures.ThreadPoolExecutor(max_workers=2) as pool: + list(pool.map(lambda value:self.module.save_json(target,{'value':value}),[1,2])) + self.assertEqual(len(set(temporaries)),2) + self.assertIn(json.loads(target.read_text())['value'],[1,2]) + self.assertFalse(list(pathlib.Path(self.temp.name).glob('concurrent.json.*.tmp'))) + def test_cover_and_hero_coalesce_game_resolution(self): + calls=self.provider();game=self.module.game;barrier=threading.Barrier(2);gate=threading.Lock() + active=0;maximum=0 + def resolve(*args): + nonlocal active,maximum + with gate:active+=1;maximum=max(maximum,active) + try:time.sleep(.01);return game(*args) + finally: + with gate:active-=1 + self.module.game=resolve + def download(kind):barrier.wait(timeout=5);return self.module.cover('Demo III',kind) + with concurrent.futures.ThreadPoolExecutor(max_workers=2) as pool:images=list(pool.map(download,['cover','hero'])) + self.assertEqual([len(data) for data in images],[115200,691200]) + self.assertEqual(maximum,1) + self.assertEqual(sum('/search/' in url for url,_ in calls),1) + def test_untrusted_sources_and_redirects(self): + for url in ['http://localhost/private','https://evil.test/image','https://cdn2.steamgriddb.com.evil.test/a','https://cdn2.steamgriddb.com:444/a']: + self.assertFalse(self.module.allowed(url)) + req=self.module.urllib.request.Request('https://www.steamgriddb.com/api/v2/test',headers={'Authorization':'Bearer secret'}) + with self.assertRaises(ValueError):self.module.Redirects().redirect_request(req,None,302,'',{},'https://cdn2.steamgriddb.com/a') + def test_reject_landscape_grid_and_blank_image(self): + self.module.sgdb=lambda _: [{'id':1,'width':1920,'height':620,'url':'https://cdn2.steamgriddb.com/hero.png'}] + self.assertEqual(list(self.module.candidates(42,'cover')),[]) + out=io.BytesIO();Image.new('RGB',(600,900),(75,75,75)).save(out,format='PNG') + self.module.fetch=lambda *_:out.getvalue() + with self.assertRaisesRegex(ValueError,'Blank artwork'):self.module.raster({'image':'fixture','width':600,'height':900}) + def test_corruption_and_old_provider_cache_are_not_reused(self): + self.provider();self.module.cover('Demo III') + image=next(pathlib.Path(self.temp.name).glob('*.cover.rgb'));image.write_bytes(b'broken') + self.assertEqual(len(self.module.cover('Demo III')),115200) + meta=next(pathlib.Path(self.temp.name).glob('*.cover.json')) + data=json.loads(meta.read_text());data['resolver']=6;data['match']['provider']='Wikipedia';meta.write_text(json.dumps(data)) + image.write_bytes(bytes([75])*115200) + self.assertTrue(self.module.usable_rgb(self.module.cover('Demo III'))) + def test_resolver_upgrade_retries_misses_and_keeps_good_images(self): + self.provider();expected=self.module.cover('Demo III') + meta=next(pathlib.Path(self.temp.name).glob('*.cover.json')) + image=next(pathlib.Path(self.temp.name).glob('*.cover.rgb')) + value=json.loads(meta.read_text());value['resolver']=7;meta.write_text(json.dumps(value)) + fetch=self.module.fetch + self.module.fetch=lambda *_:self.fail('A valid persistent image must survive the matching upgrade') + self.assertEqual(self.module.cover('Demo III'),expected) self.module.fetch=fetch - title=self.module.clean_title('God of War Sons of Sparta PROPER PS5-PPSA28997[FPKG]') - candidates=list(self.module.playstation_candidates(title)) - self.assertEqual(len(candidates),1) - self.assertIn('UP9000-PPSA28997_00-SONSOFSPARTAPS50',calls[0]) - sources=json.loads(self.module.SOURCES.read_text()) - sources['godofwarghostofsparta']=sources['godofwarsonsofsparta'] - self.module.SOURCES=pathlib.Path(self.temp.name)/'sources.json' - self.module.SOURCES.write_text(json.dumps(sources)) - self.assertEqual(list(self.module.playstation_candidates('God of War Ghost of Sparta')),[]) - self.assertEqual(calls,[sources['godofwarsonsofsparta']]*2) - def test_blank_provider_image_falls_back(self): - blank=io.BytesIO();Image.new('RGB',(20,30),(75,75,75)).save(blank,format='JPEG') - picture=Image.new('RGB',(20,30),(20,60,80));picture.paste((220,180,100),(0,0,10,15));real=io.BytesIO();picture.save(real,format='PNG') - self.module.playstation_candidates=lambda _:iter([]) - self.module.steam_candidates=lambda _:iter([{'provider':'Steam','image':'blank'}]) - self.module.wikipedia_candidates=lambda _:iter([{'provider':'Wikipedia','image':'real'}]) - self.module.fetch=lambda url,_:blank.getvalue() if url=='blank' else real.getvalue() - data=self.module.cover('Battlefield 6');self.assertTrue(self.module.usable_rgb(data)) - metadata=list(pathlib.Path(self.temp.name).glob('*.json')) - self.assertEqual(json.loads(metadata[0].read_text())['match']['provider'],'Wikipedia') - def test_cached_grey_tile_is_replaced(self): - import hashlib - root=pathlib.Path(self.temp.name);ident=hashlib.sha256(b'v4:battlefield6').hexdigest() - # Includes the small JPEG variation observed in the real stale cache. - grey=bytes(73+i%9 for i in range(115200));(root/(ident+'.rgb')).write_bytes(grey) - (root/(ident+'.json')).write_text(json.dumps({'resolver':6})) - self.module.playstation_candidates=lambda _:iter([{'provider':'PlayStation','image':'real'}]) - expected=bytes([20,40,60,200,180,160])*19200 - self.module.raster=lambda _:expected - self.assertEqual(self.module.cover('Battlefield 6'),expected) - self.assertEqual(self.module.cover('Battlefield 6'),expected) - def test_placeholder_with_isolated_noise_is_rejected_after_resize(self): - picture=Image.new('RGB',(600,900),(75,75,75));picture.putpixel((300,450),(120,120,120)) - data=io.BytesIO();picture.save(data,format='PNG');self.module.fetch=lambda *_:data.getvalue() - with self.assertRaisesRegex(ValueError,'Blank artwork'):self.module.raster({'image':'fixture'}) + value['match']=None;meta.write_text(json.dumps(value));image.write_bytes(b'') + self.assertEqual(self.module.cover('Demo III'),expected) + def test_hero_preserves_edge_subjects(self): + image=Image.new('RGB',(1920,620),(240,220,20));image.paste((10,20,40),(0,0,300,620));image.paste((10,20,40),(1800,0,1920,620)) + source=io.BytesIO();image.save(source,format='PNG');self.module.fetch=lambda *_:source.getvalue() + data=self.module.raster({'image':'fixture','width':1920,'height':620},'hero') + rendered=Image.frombytes('RGB',(640,360),data) + self.assertEqual(rendered.getpixel((10,180)),(10,20,40)) + self.assertEqual(rendered.getpixel((630,180)),(10,20,40)) + def test_hero_prefers_visible_composition(self): + self.module.game=lambda *_:42 + self.module.candidates=lambda *_:iter([{'id':1},{'id':2}]) + flat=Image.new('RGB',(640,360),(240,220,20));flat.paste((10,20,40),(0,0,100,360)) + visible=Image.new('RGB',(640,360),(240,220,20));visible.paste((10,20,40),(500,0,640,360)) + self.module.raster=lambda candidate,kind: (flat if candidate['id']==1 else visible).tobytes() + self.assertEqual(self.module.cover('Demo III','hero'),visible.tobytes()) if __name__=='__main__':unittest.main() diff --git a/homebrew/botty-native/Makefile b/homebrew/botty-native/Makefile index 8307e93..c9eba8c 100644 --- a/homebrew/botty-native/Makefile +++ b/homebrew/botty-native/Makefile @@ -20,6 +20,11 @@ build/preview-tests/%.o: tests/%.cpp build/model-test: tests/model_test.cpp $(MODEL_OBJ) $(CXX) $(HOST_FLAGS) -MMD -MP $(filter %.cpp %.o,$^) -o $@ +build/artwork-worker-test: tests/artwork_worker_test.cpp $(MODEL_OBJ) + $(CXX) $(HOST_FLAGS) $(filter %.cpp %.o,$^) -o $@ +build/artwork-cache-test: tests/artwork_cache_test.cpp src/artwork_cache.hpp + @mkdir -p $(@D) + $(CXX) $(HOST_FLAGS) $< -o $@ build/keyboard-test: tests/keyboard_test.cpp build/host/keyboard.o $(CXX) $(HOST_FLAGS) -MMD -MP $(filter %.cpp %.o,$^) -o $@ build/native-update-test: tests/native_update_test.cpp $(MODEL_OBJ) @@ -29,10 +34,12 @@ build/action-client: tests/action_client.cpp $(MODEL_OBJ) build/keyboard-overlay-test: build/preview-tests/keyboard_overlay_test.o build/preview-tests/preview_stubs.o $(PREVIEW_OBJ) $(CXX) $(HOST_FLAGS) $(filter %.o,$^) -o $@ -host-test-build: build/model-test build/keyboard-test build/native-update-test build/keyboard-overlay-test build/action-client +host-test-build: build/artwork-cache-test build/artwork-worker-test build/model-test build/keyboard-test build/native-update-test build/keyboard-overlay-test build/action-client # Prerequisite compilation is bounded by the caller's -j; tests stay sequential. -test: build/model-test build/keyboard-test build/native-update-test build/keyboard-overlay-test +test: build/artwork-worker-test build/model-test build/keyboard-test build/native-update-test build/keyboard-overlay-test build/artwork-cache-test + ./build/artwork-worker-test + ./build/artwork-cache-test ./build/model-test ./build/keyboard-test ./build/native-update-test diff --git a/homebrew/botty-native/README.md b/homebrew/botty-native/README.md index 7851e9f..7ec2a7b 100644 --- a/homebrew/botty-native/README.md +++ b/homebrew/botty-native/README.md @@ -143,7 +143,8 @@ toast) and `error-toast`. Others include `offline`, `reconnecting`, `quit`, provide synthetic state for long-name and full-list layout checks, and `BOTTY_PREVIEW_COVERS` can name a directory of 160×240 RGB covers matched by title (`Astro Bot` loads `astro-bot.rgb`); titles without a file keep the -generated fallback. Font and illustration tools use Pillow; the recorded asset-generation +generated fallback. `BOTTY_PREVIEW_BACKGROUND` can name a 640×360 RGB hero +fixture for the selected game. Font and illustration tools use Pillow; the recorded asset-generation version is 12.0.0. Assets are already bundled. The font atlas holds Manrope at 600 (text) and 800 (titles) for sizes 20–76, plus the separators and arrows the interface uses. @@ -200,6 +201,33 @@ work; rest-mode support is not established. During a slow local refresh, the app keeps its last catalog with a Reconnecting indicator and disables mutations until a fresh response arrives. Transmission RPC failures preserve the previous downloads while extraction jobs continue updating. Transient failures retry automatically after one second; the receive timeout allows five seconds under disk load. +### Persistent game artwork + +PS5 and Switch prefer SteamGridDB portraits (decoded at 160×240) and heroes +(decoded at 640×360). Heroes appear behind Discover, game details and the Switch +library; a blurred cover fills in when no hero exists. Missing portraits retain +a uniform title placeholder. Both kinds are fetched through the authenticated +Botty service, so the provider API key stays on the server. Official Steam +library portraits/heroes provide a fallback when SteamGridDB has no matching +game or image, with the same output sizes and persistent caches. Verified +PlayStation Store products fill remaining configured console-only editions. + +The native app keeps up to 512 portraits in +`/download0/botty-sgdb-covers-v1.cache` (about 56 MiB) and 128 heroes in +`/download0/botty-sgdb-backgrounds-v2.cache` (about 84 MiB). Normalized game titles +share entries across rankings, searches and the library, independently of result +IDs, while keeping explicitly marked PS5 releases, Switch/emulation entries +and unmarked search results separate for platform-specific remakes. Older unscoped entries are treated as misses on the first lookup. A decoded RAM LRU retains 96 covers and 12 heroes (~18 MiB): returning +to a recent page publishes them in the same UI request, with no worker/disk wait. +Disk reads/writes run on the artwork worker; disk hits appear without contacting +the service, including after an app restart. The least recently used entries are +replaced; incomplete/corrupt images are fetched again. Unavailable storage falls +back to normal loading. Only successful image responses are saved. + +With the app closed, deleting the two cache files resets them. New filenames +bypass old mixed-provider images; previous caches are left untouched. Configure +the SteamGridDB key as described in [the artwork service guide](../../deployment/botty-artwork/README.md). + ### Library deletion For a moved game folder, Library → Options offers **Delete game**, with Cancel selected by default. This deletes the installed game files while keeping the torrent and original archives. Close the game and remove its home-screen entry first; mounted games are refused. The action requires a service advertising `libraryDeletionSupported`. Image-based games require manual unmounting/removal. **Remove from Activity** only hides an extraction row and is no longer offered in Library. diff --git a/homebrew/botty-native/docs/artwork-preview.png b/homebrew/botty-native/docs/artwork-preview.png new file mode 100644 index 0000000..a8aeda5 Binary files /dev/null and b/homebrew/botty-native/docs/artwork-preview.png differ diff --git a/homebrew/botty-native/src/artwork_cache.hpp b/homebrew/botty-native/src/artwork_cache.hpp new file mode 100644 index 0000000..a7eacc2 --- /dev/null +++ b/homebrew/botty-native/src/artwork_cache.hpp @@ -0,0 +1,158 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#pragma once +#include +#include +#include +#include +#include +namespace botty { +// One bounded file in the title's writable, persistent download directory. +// Fixed slots avoid directory scans and extra filesystem dependencies on PS5. +template class PersistentArtworkCache final { +public: + static constexpr unsigned capacity=Capacity, pixelBytes=PixelBytes, headerBytes=128; + using Key=std::array; + static Key key(const std::array& id,std::string_view title) noexcept { + Key out{}; + if(title.empty()){ + // A missing title must never share an entry with another game. + for(unsigned i=0;i(id[i]); + }else{ + // Rankings use random result IDs; retain their shared title identity + // within a platform, without sharing remake art with emulation. + std::array normalized{};unsigned length=0; + const bool emulation=id[0]=='m'&&id[1]==':';unsigned char platform=emulation?2:0; + const auto lower=[](unsigned char c){return c>='A'&&c<='Z'?static_cast(c+32):c;}; + const auto wordChar=[](unsigned char c){return (c>='a'&&c<='z')||(c>='A'&&c<='Z')||(c>='0'&&c<='9')||c>=128;}; + for(std::size_t at=0;at4&&lower(word[0])=='p'&&lower(word[1])=='p'&&lower(word[2])=='s'&&lower(word[3])=='a'; + for(unsigned i=4;ppsa&&i='0'&&word[i]<='9'; + if(equals("ps5")){ + if(!emulation)platform=1; + if(length)break; + continue; // Leading platform tags are not a game identity. + } + if(ppsa&&!emulation)platform=1; + else if(platform!=1&&(equals("nsw")||equals("nsp")||equals("xci")||equals("switch")))platform=2; + if(length&&length(hash>>(i*8)); + // Emulation and PS5 may resolve different releases of the same title. + out[104]=2;out[105]=platform;return out; + } + std::uint64_t hash=14695981039346656037ULL; + for(unsigned char c:title){hash^=c;hash*=1099511628211ULL;} + for(unsigned i=0;i<8;++i)out[96+i]=static_cast(hash>>(i*8)); + return out; + } +private: + struct Entry {Key key{};std::uint64_t age=0;std::uint32_t checksum=0;}; + std::array entries_{}; + const char* path_; + bool scanned_=false; + std::uint64_t clock_=0; + static std::uint32_t checksum(const void* data) noexcept { + auto* bytes=static_cast(data);std::uint32_t hash=2166136261U; + for(unsigned i=0;i header(const Entry& entry) noexcept { + std::array out{};std::memcpy(out.data(),"BC01",4); + std::memcpy(out.data()+4,entry.key.data(),entry.key.size()); + for(unsigned i=0;i<8;++i)out[116+i]=static_cast(entry.age>>(8*i)); + for(unsigned i=0;i<4;++i)out[124+i]=static_cast(entry.checksum>>(8*i)); + return out; + } + void scan() noexcept { + if(scanned_||!path_)return; + scanned_=true;FILE* file=std::fopen(path_,"rb");if(!file)return; + for(unsigned i=0;i bytes{}; + if(!seek(file,i)||std::fread(bytes.data(),1,bytes.size(),file)!=bytes.size())break; + if(std::memcmp(bytes.data(),"BC01",4))continue; + auto& entry=entries_[i];std::memcpy(entry.key.data(),bytes.data()+4,entry.key.size()); + for(unsigned b=0;b<8;++b)entry.age|=std::uint64_t(bytes[116+b])<<(8*b); + for(unsigned b=0;b<4;++b)entry.checksum|=std::uint32_t(bytes[124+b])<<(8*b); + // Reject damaged ages before advancing the logical LRU clock. + if(entry.age>UINT64_MAX/2){entry={};continue;} + if(entry.age>clock_)clock_=entry.age; + } + std::fclose(file); + } +public: + explicit PersistentArtworkCache(const char* path) noexcept:path_(path){} + bool load(const Key& key,void* pixels) noexcept { + scan();if(!path_)return false; + for(unsigned i=0;i invalid{}; + // Invalidate before replacing pixels. Interrupted writes remain misses. + bool ok=seek(file,slot)&&std::fwrite(invalid.data(),1,invalid.size(),file)==invalid.size()&&std::fflush(file)==0; + ok=ok&&std::fwrite(pixels,1,pixelBytes,file)==pixelBytes&&std::fflush(file)==0; + Entry next{key,++clock_,checksum(pixels)};const auto bytes=header(next); + ok=ok&&seek(file,slot)&&std::fwrite(bytes.data(),1,bytes.size(),file)==bytes.size(); + if(std::fclose(file)!=0)ok=false; + if(ok)entries_[slot]=next; + } +}; +using ArtworkCache=PersistentArtworkCache<>; +using BackgroundCache=PersistentArtworkCache<640*360*3,128>; +} + +namespace botty { +// A bounded decoded LRU keeps recent pages immediately available on the UI thread. +// Access is protected by Artwork's gate; it performs no filesystem or network work. +template class RetainedArtwork final { + struct Item {ArtworkCache::Key key{};std::array pixels{};std::uint64_t age=0;}; + std::array items_{}; + std::uint64_t clock_=0; +public: + bool load(const ArtworkCache::Key& key,void* output) noexcept { + for(auto& item:items_)if(item.age&&item.key==key){ + std::memcpy(output,item.pixels.data(),Bytes);item.age=++clock_;return true; + } + return false; + } + void store(const ArtworkCache::Key& key,const void* pixels) noexcept { + unsigned slot=0; + for(unsigned i=0;i& id,unsigned seed,unsigned fadeTop) noexcept { - if(const auto* pixels=coverFor(id))c.poster(532,0,1388,780,*pixels,0,255,true); + if(covers.backgroundReady&&covers.backgroundId==id)c.image(532,0,1388,780,covers.background,640,360,0,255,true); + else if(const auto* pixels=coverFor(id)){ + std::array blurred{}; + for(unsigned y=0;y<15;++y)for(unsigned x=0;x<10;++x)for(unsigned channel=0;channel<3;++channel){ + unsigned total=0;for(unsigned dy=0;dy<16;++dy)for(unsigned dx=0;dx<16;++dx)total+=(*pixels)[((y*16+dy)*160+x*16+dx)*3+channel]; + blurred[(y*10+x)*3+channel]=static_cast(total/256); + } + c.image(532,0,1388,780,blurred,10,15,0,255,true); + } else gamePattern(c,532,0,1388,780,seed); c.fade(532,0,287,780,bg,255,209,true);c.fade(819,0,533,780,bg,209,0,true); c.fade(0,fadeTop,1920,780-fadeTop,bg,0,255,false); @@ -1239,6 +1247,7 @@ void drawToast(Canvas& c,bool sheet) noexcept { } void drawEmulation(Canvas& c) noexcept { + if(const auto* selected=botty::emulationAt(catalog,model.filter,model.selected))hero(c,artworkId(*selected,'m'),model.selected,500); text(c,96,146,"Emulation",56,68,ink,bold); const char* filters[]={"Games","Updates / DLC","Available files"}; unsigned x=1100;for(unsigned i=0;i<3;++i)x+=chip(c,x,202,filters[i],model.filter==i)+12; @@ -1552,7 +1561,15 @@ bool draw(Canvas& c) noexcept { const unsigned first=(model.selected/6)*6; for(unsigned i=0;i<6&&first+iname.data();}} } - artwork.request(ids);if(artwork.read(covers))++displayRevision; + std::array backgroundId{};std::string_view backgroundName; + if(model.tab==Model::discover&&!model.searchResults&&model.selectedname.data();} + }else if(model.details){ + const auto item=focused();if(item.entry){backgroundId=artworkId(*item.entry,artSource(item));backgroundName=item.entry->name.data();} + } + artwork.request(ids,names,backgroundId,backgroundName);if(artwork.read(covers))++displayRevision; #ifdef BOTTY_HOST_PREVIEW // Cover fixtures are matched by title, so a cover never lands on another game: // "Astro Bot" loads /astro-bot.rgb when it exists. @@ -1569,6 +1586,12 @@ bool draw(Canvas& c) noexcept { FILE* file=std::fopen(path,"rb");if(!file)continue; covers.ready[i]=std::fread(covers.pixels[i].data(),1,covers.pixels[i].size(),file)==covers.pixels[i].size();std::fclose(file);covers.ids[i]=ids[i]; } + if(const char* path=std::getenv("BOTTY_PREVIEW_BACKGROUND");path&&backgroundId[0]&&(!covers.backgroundReady||covers.backgroundId!=backgroundId)){ + if(FILE* file=std::fopen(path,"rb")){ + covers.backgroundReady=std::fread(covers.background.data(),1,covers.background.size(),file)==covers.background.size(); + std::fclose(file);covers.backgroundId=backgroundId; + } + } #endif } if(exitRequested)return false; diff --git a/homebrew/botty-native/src/probe.cpp b/homebrew/botty-native/src/probe.cpp index 1a5f1cf..1f4a291 100644 --- a/homebrew/botty-native/src/probe.cpp +++ b/homebrew/botty-native/src/probe.cpp @@ -3,6 +3,7 @@ #include "platform.hpp" #include "json_flat.hpp" #include +#include #include #include #include @@ -336,9 +337,20 @@ void* Network::worker(void* context) noexcept { } void Artwork::start() noexcept {stop_.store(false);(void)platform::startWorker(worker,this,&thread_);} void Artwork::stop() noexcept {stop_.store(true);if(thread_){platform::joinWorker(thread_);thread_=nullptr;}} -void Artwork::request(const CoverIds& ids) noexcept { +void Artwork::request(const CoverIds& ids,const std::array& names,const std::array& backgroundId,std::string_view backgroundName) noexcept { + std::array keys{}; + for(unsigned i=0;i(context);unsigned handled=0; + auto& cache=self.cache_;auto& backgrounds=self.backgrounds_; + // Large pixel/HTTP buffers live outside the PS5 worker's bounded stack. + static std::array pixels; + static Response response; while(!self.stop_.load()){ - CoverIds ids;unsigned requestRevision; + CoverIds ids;std::array keys;unsigned revision; + std::array backgroundId;ArtworkCache::Key backgroundKey; while(self.gate_.test_and_set(std::memory_order_acquire))platform::sleep(1000); - ids=self.requested_;requestRevision=self.requestRevision_;self.gate_.clear(std::memory_order_release); - if(requestRevision==handled){platform::sleep(100000);continue;}handled=requestRevision; - // Scrolling shifts covers between slots: keep any that are already loaded. - static ArtworkPage previous; - std::array done{}; + ids=self.requested_;keys=self.requestedKeys_;revision=self.requestRevision_; + backgroundId=self.backgroundId_;backgroundKey=self.backgroundKey_;self.gate_.clear(std::memory_order_release); + if(revision==handled){platform::sleep(16000);continue;}handled=revision; + std::array done{};bool backgroundDone=!backgroundId[0],stale=false; while(self.gate_.test_and_set(std::memory_order_acquire))platform::sleep(1000); - previous=self.page_;self.page_.ids=ids;self.page_.ready.fill(false); - for(unsigned i=0;i encoded{};unsigned length=0; - constexpr char hex[]="0123456789ABCDEF"; - for(unsigned char ch:std::string_view(ids[i].data())){ + const auto download=[&](unsigned slot,bool background) noexcept { + const auto& id=background?backgroundId:ids[slot]; + std::array encoded{};unsigned length=0;constexpr char hex[]="0123456789ABCDEF"; + for(unsigned char ch:std::string_view(id.data())){ if((ch>='a'&&ch<='z')||(ch>='A'&&ch<='Z')||(ch>='0'&&ch<='9')||ch=='-')encoded[length++]=static_cast(ch); - else {encoded[length++]='%';encoded[length++]=hex[ch>>4];encoded[length++]=hex[ch&15];} + else{encoded[length++]='%';encoded[length++]=hex[ch>>4];encoded[length++]=hex[ch&15];} + } + char path[360];std::snprintf(path,sizeof(path),id[1]==':'?"/api/artwork?id=%s&kind=%s":"/api/explore/artwork?id=%s&kind=%s",encoded.data(),background?"hero":"cover"); + response.status=0;response.length=0; + const unsigned bytes=background?BackgroundCache::pixelBytes:ArtworkCache::pixelBytes; + const bool ready=get(path,token,response,platform::now()+14000000)==Probe::ready&&response.length==bytes; + if(ready){ + if(background)backgrounds.store(backgroundKey,response.body.data());else cache.store(keys[slot],response.body.data()); + std::memcpy(pixels.data(),response.body.data(),bytes); } - char path[340];std::snprintf(path,sizeof(path),ids[i][1]==':'?"/api/artwork?id=%s":"/api/explore/artwork?id=%s",encoded.data()); - static Response<160*240*3+1> response;response.status=0;response.length=0; - const bool ready=get(path,token,response,platform::now()+14000000)==Probe::ready&&response.length==160*240*3; + publish(slot,background,ready); + if(response.status==404){if(background)backgroundDone=true;else done[slot]=true;} + }; + for(unsigned attempt=0;attempt<20&&!self.stop_.load()&&!stale;++attempt){ + // Start the selected game's background without waiting for an entire cover row. + if(!self.stop_.load()&&!backgroundDone)download(0,true); + for(unsigned i=0;i(response.body[b]);self.page_.ready[i]=true;++self.page_.revision;} - self.gate_.clear(std::memory_order_release);if(stale)break; - } - bool complete=true;for(unsigned i=0;i,coverSlots> pixels{}; std::array ready{}; + std::array backgroundId{}; + std::array background{}; + bool backgroundReady=false; unsigned revision=0; }; +#if defined(__PROSPERO__) && !defined(BOTTY_HOST_PREVIEW) +inline constexpr const char* coverCachePath="/download0/botty-sgdb-covers-v1.cache"; +inline constexpr const char* backgroundCachePath="/download0/botty-sgdb-backgrounds-v2.cache"; +#else +inline constexpr const char* coverCachePath=nullptr; +inline constexpr const char* backgroundCachePath=nullptr; +#endif class Artwork final { + ArtworkCache cache_; + BackgroundCache backgrounds_; + RetainedArtwork retainedCovers_; + RetainedArtwork retainedBackgrounds_; std::atomic stop_{false}; std::atomic_flag gate_=ATOMIC_FLAG_INIT; CoverIds requested_{}; + std::array requestedKeys_{}; + std::array backgroundId_{}; + ArtworkCache::Key backgroundKey_{}; unsigned requestRevision_=0; ArtworkPage page_{}; void* thread_=nullptr; static void* worker(void*) noexcept; public: + explicit Artwork(const char* covers=coverCachePath,const char* backgrounds=backgroundCachePath) noexcept:cache_(covers),backgrounds_(backgrounds){} void start() noexcept; void stop() noexcept; - void request(const CoverIds&) noexcept; + void request(const CoverIds&,const std::array& names={},const std::array& backgroundId={},std::string_view backgroundName={}) noexcept; bool read(ArtworkPage&) noexcept; }; diff --git a/homebrew/botty-native/src/renderer.cpp b/homebrew/botty-native/src/renderer.cpp index 5bde4da..040e69b 100644 --- a/homebrew/botty-native/src/renderer.cpp +++ b/homebrew/botty-native/src/renderer.cpp @@ -597,13 +597,16 @@ void Canvas::clear(Color color) noexcept } void Canvas::poster(unsigned x,unsigned y,unsigned width,unsigned height,std::span rgb,unsigned radius,unsigned alpha,bool crop) noexcept { - if(rgb.size()!=160*240*3||!width||!height||!alpha)return; + image(x,y,width,height,rgb,160,240,radius,alpha,crop); +} +void Canvas::image(unsigned x,unsigned y,unsigned width,unsigned height,std::span rgb,unsigned sourceWidth,unsigned sourceHeight,unsigned radius,unsigned alpha,bool crop) noexcept { + if(sourceWidth<2||sourceHeight<2||rgb.size()!=std::size_t(sourceWidth)*sourceHeight*3||!width||!height||!alpha)return; if(radius>width/2)radius=width/2; if(radius>height/2)radius=height/2; // Source window in 1/256 pixels. Cover crops the long axis around its center. - std::uint64_t windowX=0,windowY=0,windowWidth=159*256,windowHeight=239*256; - if(crop&&std::uint64_t{width}*240>std::uint64_t{height}*160){windowHeight=std::uint64_t{159}*256*height/width;windowY=(239*256-windowHeight)/2;} - else if(crop){windowWidth=std::uint64_t{239}*256*width/height;windowX=(159*256-windowWidth)/2;} + std::uint64_t windowX=0,windowY=0,windowWidth=(sourceWidth-1)*256,windowHeight=(sourceHeight-1)*256; + if(crop&&std::uint64_t{width}*(sourceHeight-1)>std::uint64_t{height}*(sourceWidth-1)){windowHeight=std::uint64_t{sourceWidth-1}*256*height/width;windowY=((sourceHeight-1)*256-windowHeight)/2;} + else if(crop){windowWidth=std::uint64_t{sourceHeight-1}*256*width/height;windowX=((sourceWidth-1)*256-windowWidth)/2;} for(unsigned row=0;row255?255:alpha; if(radius){ @@ -615,11 +618,11 @@ void Canvas::poster(unsigned x,unsigned y,unsigned width,unsigned height,std::sp // Bilinear enlargement keeps covers smooth at TV viewing sizes. const unsigned sx=static_cast(windowX+(width>1?windowWidth*col/(width-1):0)); const unsigned sy=static_cast(windowY+(height>1?windowHeight*row/(height-1):0)); - const unsigned x0=sx/256,y0=sy/256,x1=x0<159?x0+1:x0,y1=y0<239?y0+1:y0; + const unsigned x0=sx/256,y0=sy/256,x1=x0(color),mask); diff --git a/homebrew/botty-native/src/renderer.hpp b/homebrew/botty-native/src/renderer.hpp index 9ab8ab6..83e2da5 100644 --- a/homebrew/botty-native/src/renderer.hpp +++ b/homebrew/botty-native/src/renderer.hpp @@ -43,6 +43,8 @@ class Canvas final // Crop fills the box like CSS "cover"; radius and alpha mask the result. void poster(unsigned x,unsigned y,unsigned width,unsigned height,std::span rgb, unsigned radius=0,unsigned alpha=255,bool crop=false) noexcept; + void image(unsigned x,unsigned y,unsigned width,unsigned height,std::span rgb, + unsigned sourceWidth,unsigned sourceHeight,unsigned radius=0,unsigned alpha=255,bool crop=false) noexcept; void gameCase(unsigned x,unsigned y,unsigned width,unsigned height,std::span rgb) noexcept; void clear(Color color) noexcept; void backdrop(bool subdued) noexcept; diff --git a/homebrew/botty-native/tests/artwork_cache_test.cpp b/homebrew/botty-native/tests/artwork_cache_test.cpp new file mode 100644 index 0000000..059ef67 --- /dev/null +++ b/homebrew/botty-native/tests/artwork_cache_test.cpp @@ -0,0 +1,83 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#include "artwork_cache.hpp" +#include +#include +#include +#include +#include +#include +using botty::ArtworkCache; +int main(){ + char directory[]="/tmp/botty-covers-XXXXXX";assert(mkdtemp(directory)); + const std::string path=std::string(directory)+"/covers.cache"; + std::array id{};std::snprintf(id.data(),id.size(),"m:fixture"); + const auto key=ArtworkCache::key(id,"Switch fixture"); + std::array first{},second{}; + std::snprintf(first.data(),first.size(),"ranking-seeders-id"); + std::snprintf(second.data(),second.size(),"ranking-recent-id"); + const auto shared=ArtworkCache::key(first,"Astro.Bot.PS5.Release-A"); + assert(shared==ArtworkCache::key(second,"ASTRO BOT PS5 Release-B")); + assert(shared!=ArtworkCache::key(second,"Another Game PS5 Release-B")); + assert(shared==ArtworkCache::key(first,"[PS5] Astro Bot")); + assert(shared!=ArtworkCache::key(first,"[PS5] Gran Turismo 7")); + assert(ArtworkCache::key(first,"[PS5]")!=ArtworkCache::key(second,"[PS5]")); + assert(ArtworkCache::key(first,"!!!")!=ArtworkCache::key(second,"!!!")); + second[0]='m';second[1]=':'; + assert(shared!=ArtworkCache::key(second,"Astro Bot PS5")); + const auto ps5Remake=ArtworkCache::key(first,"Resident Evil 4 PS5"); + const auto switchOriginal=ArtworkCache::key(second,"Resident Evil 4"); + assert(ps5Remake!=switchOriginal); + const auto unmarkedSearch=ArtworkCache::key(first,"Resident Evil 4"); + assert(ps5Remake!=unmarkedSearch); + std::array pixels{},out{};pixels.fill(73); + { + auto cache=std::make_unique(path.c_str()); + assert(!cache->load(key,out.data()));cache->store(key,pixels.data()); + } + auto cache=std::make_unique(path.c_str()); + assert(cache->load(key,out.data())&&out==pixels); + cache->store(shared,pixels.data()); + assert(cache->load(ArtworkCache::key(first,"Astro Bot PS5 Release-C"),out.data())&&out==pixels); + cache->store(ps5Remake,pixels.data()); + assert(!cache->load(switchOriginal,out.data())); + assert(!cache->load(unmarkedSearch,out.data())); + pixels.fill(29);cache->store(switchOriginal,pixels.data()); + assert(cache->load(switchOriginal,out.data())&&out==pixels); + pixels.fill(73);assert(cache->load(ps5Remake,out.data())&&out==pixels); + // Entries created before platform scoping must not be reused either. + auto legacy=ps5Remake;legacy[104]=1;legacy[105]=0; + assert(legacy!=ps5Remake); + // Reset the fixture before the capacity/LRU checks below. + cache.reset();std::remove(path.c_str()); + cache=std::make_unique(path.c_str());cache->store(key,pixels.data()); + assert(cache->load(key,out.data())&&out==pixels); // Reopen, including embedded NUL bytes. + assert(!cache->load(ArtworkCache::key(id,"Different title"),out.data())); + // Same-size damage and truncation both fall back to the service. + FILE* file=std::fopen(path.c_str(),"r+b");assert(file); + assert(std::fseek(file,ArtworkCache::headerBytes+10,SEEK_SET)==0);assert(std::fputc(0,file)!=EOF);std::fclose(file); + assert(!cache->load(key,out.data()));cache->store(key,pixels.data()); + assert(truncate(path.c_str(),ArtworkCache::headerBytes+10)==0); + assert(!cache->load(key,out.data()));cache->store(key,pixels.data()); + // Fill all slots, touch the first, then force an LRU eviction. + for(unsigned i=1;istore(ArtworkCache::key(id,"Fixture "+std::to_string(i)+" PS5"),pixels.data()); + } + assert(cache->load(key,out.data())); + std::snprintf(id.data(),id.size(),"j:overflow");cache->store(ArtworkCache::key(id,"Overflow PS5"),pixels.data()); + cache=std::make_unique(path.c_str()); + assert(cache->load(key,out.data())); + id.fill(0);std::snprintf(id.data(),id.size(),"j:1"); + assert(!cache->load(ArtworkCache::key(id,"Fixture 1 PS5"),out.data())); + struct stat info{};assert(stat(path.c_str(),&info)==0); + assert(info.st_size==ArtworkCache::capacity*(ArtworkCache::headerBytes+ArtworkCache::pixelBytes)); + ArtworkCache unavailable("/nonexistent/botty/covers");unavailable.store(key,pixels.data());assert(!unavailable.load(key,out.data())); + const auto heroPath=std::string(directory)+"/heroes.cache"; + auto heroPixels=std::make_unique>();heroPixels->fill(137); + auto heroOut=std::make_unique>(); + {auto heroes=std::make_unique(heroPath.c_str());heroes->store(key,heroPixels->data());} + auto heroes=std::make_unique(heroPath.c_str()); + assert(heroes->load(key,heroOut->data())&&*heroPixels==*heroOut); + assert(ArtworkCache::pixelBytes!=botty::BackgroundCache::pixelBytes); + std::remove(heroPath.c_str());std::remove(path.c_str());rmdir(directory); + std::puts("Artwork cache: persistence, identity, corruption, truncation, LRU and unavailable storage passed."); +} diff --git a/homebrew/botty-native/tests/artwork_worker_test.cpp b/homebrew/botty-native/tests/artwork_worker_test.cpp new file mode 100644 index 0000000..93e1b35 --- /dev/null +++ b/homebrew/botty-native/tests/artwork_worker_test.cpp @@ -0,0 +1,79 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#include "probe.hpp" +#include "platform.hpp" +#include +#include +#include +#include +#include +#include +#include +#include +namespace { +std::atomic calls{0};std::atomic online{true}; +thread_local std::string request,response;thread_local std::size_t offset=0; +std::string wire(const std::string& body){return "HTTP/1.1 200 OK\r\nContent-Length: "+std::to_string(body.size())+"\r\n\r\n"+body;} +} +namespace botty::platform { +std::uint64_t now() noexcept{return std::chrono::duration_cast(std::chrono::steady_clock::now().time_since_epoch()).count();} +void sleep(unsigned us) noexcept{std::this_thread::sleep_for(std::chrono::microseconds(us));} +void log(const char*) noexcept{} +int connectLocal() noexcept{++calls;request.clear();response.clear();offset=0;return online?1:-1;} +int send(int,const void* data,std::size_t size) noexcept{request.append(static_cast(data),size);return int(size);} +int receive(int,void* data,std::size_t size) noexcept{ + if(response.empty()){ + if(request.find("GET /api/bootstrap ")==0)response=wire(R"({"token":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"})"); + else{const bool hero=request.find("kind=hero")!=std::string::npos;assert(hero||request.find("kind=cover")!=std::string::npos);response=wire(std::string(hero?640*360*3:160*240*3,hero?'h':'c'));} + } + const auto count=std::min(size,response.size()-offset);std::memcpy(data,response.data()+offset,count);offset+=count;return int(count); +} +void closeSocket(int) noexcept{} +bool startWorker(void* (*fn)(void*),void* context,void** handle) noexcept{*handle=new std::thread([=]{fn(context);});return true;} +void joinWorker(void* handle) noexcept{auto* thread=static_cast(handle);thread->join();delete thread;} +} +int main(){ + char directory[]="/tmp/botty-artwork-worker-XXXXXX";assert(mkdtemp(directory)); + const auto coverPath=std::string(directory)+"/covers",heroPath=std::string(directory)+"/heroes"; + botty::CoverIds ids{};std::array names{}; + std::snprintf(ids[0].data(),ids[0].size(),"ranking-seeders");names[0]="Demo PS5"; + const auto await=[&](botty::Artwork& artwork){ + auto page=std::make_unique(); + for(unsigned i=0;i<500;++i){artwork.read(*page);if(page->ready[0]&&page->ids[0]==ids[0]&&page->backgroundReady&&page->backgroundId==ids[0]){ + assert(page->pixels[0][0]=='c'&&page->background[0]=='h');return; + }std::this_thread::sleep_for(std::chrono::milliseconds(5));} + assert(false&&"Artwork worker timed out"); + }; + { + auto artwork=std::make_unique(coverPath.c_str(),heroPath.c_str());artwork->start(); + artwork->request(ids,names,ids[0],names[0]);await(*artwork);assert(calls==3); + // Stop the worker entirely: a subsequent page must render in request()+read(), + // with neither disk work nor a background thread able to fill the page. + artwork->stop();online=false;calls=0; + artwork->request({}); + auto immediate=std::make_unique();artwork->read(*immediate);assert(!immediate->ready[0]); + artwork->request(ids,names,ids[0],names[0]); + assert(artwork->read(*immediate)&&immediate->ready[0]&&immediate->backgroundReady); + // Rankings can assign a different ID to the same game. No image/API fetch follows. + for(const char* id:{"ranking-completed","ranking-recent","ranking-seeders"}){ + std::snprintf(ids[0].data(),ids[0].size(),"%s",id);artwork->request(ids,names,ids[0],names[0]); + assert(artwork->read(*immediate)&&immediate->ready[0]&&immediate->backgroundReady); + assert(immediate->ids[0]==ids[0]&&immediate->backgroundId==ids[0]); + } + assert(calls==0); + std::snprintf(ids[0].data(),ids[0].size(),"search-unmarked");names[0]="Demo"; + artwork->request(ids,names,ids[0],names[0]); + assert(artwork->read(*immediate)&&!immediate->ready[0]&&!immediate->backgroundReady); + std::snprintf(ids[0].data(),ids[0].size(),"m:switch-demo");names[0]="Demo"; + artwork->request(ids,names,ids[0],names[0]); + assert(artwork->read(*immediate)&&!immediate->ready[0]&&!immediate->backgroundReady); + std::snprintf(ids[0].data(),ids[0].size(),"ranking-seeders");names[0]="Demo PS5"; + artwork->request(ids,names,ids[0],names[0]); + assert(artwork->read(*immediate)&&immediate->ready[0]&&immediate->backgroundReady); + artwork->stop(); + } + // A fresh worker/app instance must restore both images with the service offline. + auto artwork=std::make_unique(coverPath.c_str(),heroPath.c_str());artwork->start(); + artwork->request(ids,names,ids[0],names[0]);await(*artwork);assert(calls==0);artwork->stop(); + std::remove(coverPath.c_str());std::remove(heroPath.c_str());rmdir(directory); + std::puts("Artwork worker: shared ranking identities and offline persistent portraits/heroes passed."); +} diff --git a/homebrew/botty-native/tests/keyboard_overlay_test.cpp b/homebrew/botty-native/tests/keyboard_overlay_test.cpp index 7d75e09..6a38b3d 100644 --- a/homebrew/botty-native/tests/keyboard_overlay_test.cpp +++ b/homebrew/botty-native/tests/keyboard_overlay_test.cpp @@ -11,6 +11,12 @@ bool overlayFrame(Canvas& canvas) noexcept { using Buttons=botty::Buttons; unsigned edge=0; if(frame==0){ + // Near-identical source/destination aspect ratios must not underflow + // the centered source window (640×360 -> 1388×780 previously did). + std::array heroPixels{};heroPixels.fill(123); + canvas.image(532,0,1388,780,heroPixels,640,360,0,255,true); + canvas.image(532,0,1388,780,heroPixels,640,360,0,255,false); + model.tab=Model::activity;pad=1;workflow.add();textEntryState=TextEntryState::ready; notify(Toast::Kind::error,"Request failed","Fixture failure");edge=Buttons::circle; }else if(frame==1){ diff --git a/homebrew/botty/src/emulation.hpp b/homebrew/botty/src/emulation.hpp index 242be84..df55bea 100644 --- a/homebrew/botty/src/emulation.hpp +++ b/homebrew/botty/src/emulation.hpp @@ -13,7 +13,7 @@ namespace botty { // Eden owns metadata, keys, profiles and saves. Botty only inventories content. class Emulation { fs::path data_, install_; - struct Cached {std::array identity{},coverIdentity{};json data;std::string pixels,id,artworkId,kind;}; + struct Cached {std::array identity{},coverIdentity{};json data;std::string pixels,id,artworkId,artworkName,kind;}; std::shared_ptr displayLock_=std::make_shared(); mutable std::map display_; mutable std::string keysIdentity_; @@ -103,6 +103,11 @@ class Emulation { if(!modsByGame.count(baseId))modsByGame.emplace(baseId,discoverMods(base,baseId,config,settingsKnown)); } project(out,modsByGame); + // Receipts and base-game associations can provide a name without a NACP. + for(const auto& item:out["entries"]){ + const auto found=display_.find(item.at("path").get()); + if(found!=display_.end())found->second.artworkName=item.at("name").get(); + } } static void project(json& out,const std::map& modsByGame) { const auto version=[](const json& m){const auto label=m.value("versionLabel",std::string{});return label.empty()?"revision "+std::to_string(m.value("version",uint64_t(0))):"v"+label;}; @@ -251,6 +256,13 @@ class Emulation { try{enrich(out,root());}catch(...){out["error"]="Some game details are unavailable; files were left unchanged";} return out; } + std::string artworkTitle(const std::string& id) const { + std::lock_guard guard(*displayLock_); + const auto found=std::find_if(display_.begin(),display_.end(),[&](const auto& entry){return entry.second.id==id||entry.second.artworkId==id;}); + if(found==display_.end())return {}; + try{containedExisting(root(),found->first);if(sourceIdentity(found->first)!=found->second.identity)return {};}catch(...){return {};} + return found->second.artworkName; + } std::string artwork(const std::string& id) const { std::lock_guard guard(*displayLock_); auto found=std::find_if(display_.begin(),display_.end(),[&](const auto& entry){return entry.second.id==id||entry.second.artworkId==id;}); diff --git a/homebrew/botty/src/search.hpp b/homebrew/botty/src/search.hpp index 0cdc05c..7fbf7f7 100644 --- a/homebrew/botty/src/search.hpp +++ b/homebrew/botty/src/search.hpp @@ -155,35 +155,44 @@ class Search { for(const auto& row:torrents){if(count++==256)break;catalogTitles["t:"+std::to_string(row.at("id").get())]=row.value("name","");} count=0;for(const auto& row:jobs){if(count++==256)break;catalogTitles["j:"+row.at("id").get()]=row.value("name","");} } - std::string artwork(const Paths& paths,const std::string& id,bool catalog=false){ + std::string artwork(const Paths& paths,const std::string& id,bool catalog=false,const std::string& kind="cover"){ std::lock_guard guard(mutex);std::string title; - if(catalog){const auto found=catalogTitles.find(id);if(found!=catalogTitles.end())title=gameKey(found->second);} - else for(const auto& row:rows)if(row.at("id")==id)title=gameKey(row.value("name","")); + if(catalog){const auto found=catalogTitles.find(id);if(found!=catalogTitles.end())title=found->second;} + else for(const auto& row:rows)if(row.at("id")==id)title=row.value("name",""); + return artworkLocked(paths,title,kind); + } + std::string artworkTitle(const Paths& paths,const std::string& title,const std::string& kind="cover"){ + std::lock_guard guard(mutex);return artworkLocked(paths,title,kind); + } + std::string artworkLocked(const Paths& paths,std::string title,const std::string& kind){ + if(kind!="cover"&&kind!="hero")throw std::runtime_error("Invalid artwork kind"); if(title.empty())return {};if(title.size()>200)title.resize(200); + const auto bytes=kind=="hero"?640*360*3:160*240*3; + const auto cacheKey=kind+":"+title; unsigned long long hash=14695981039346656037ULL;for(unsigned char c:title){hash^=c;hash*=1099511628211ULL;} char key[17];std::snprintf(key,sizeof(key),"%016llx",hash); - const auto folder=paths.root/"cache/covers-v4",file=folder/(std::string(key)+".rgb"),meta=folder/(std::string(key)+".json"); - const auto found=covers.find(title);if(found!=covers.end()){if(!found->second.empty()||std::time(nullptr)-coverFailedAt[title]<60)return found->second;covers.erase(found);coverFailedAt.erase(title);} + const auto folder=paths.root/(kind=="hero"?"cache/sgdb-heroes-v2":"cache/sgdb-covers-v1"),file=folder/(std::string(key)+".rgb"),meta=folder/(std::string(key)+".json"); + const auto found=covers.find(cacheKey);if(found!=covers.end()){if(!found->second.empty()||std::time(nullptr)-coverFailedAt[cacheKey]<60)return found->second;covers.erase(found);coverFailedAt.erase(cacheKey);} if(coversLoading>=6)return "pending"; - if(covers.size()>=36){for(auto it=covers.begin();it!=covers.end();++it)if(it->second!="pending"){covers.erase(it);break;}} - try{auto saved=json::parse(readText(meta,4096));if(saved.at("title")==title&&std::time(nullptr)-saved.at("saved").get()<2592000){auto data=readText(file,160*240*3);if(data.size()==160*240*3){covers[title]=data;return data;}}}catch(...){} - covers[title]="pending";++coversLoading; - try{std::thread([this,paths,title,folder,file,meta]{ + if(covers.size()>=24){for(auto it=covers.begin();it!=covers.end();++it)if(it->second!="pending"){covers.erase(it);break;}} + try{auto saved=json::parse(readText(meta,4096));if(saved.at("title")==title&&std::time(nullptr)-saved.at("saved").get()<2592000){auto data=readText(file,bytes);if(data.size()==size_t(bytes)){covers[cacheKey]=data;return data;}}}catch(...){} + covers[cacheKey]="pending";++coversLoading; + try{std::thread([this,paths,title,folder,file,meta,kind,bytes,cacheKey]{ std::string data;try{std::string encoded;const char* hex="0123456789ABCDEF";for(unsigned char c:title){if(std::isalnum(c))encoded+=c;else{encoded+='%';encoded+=hex[c>>4];encoded+=hex[c&15];}} - data=fetch(paths,"/artwork?title="+encoded,160*240*3,12);if(data.size()!=160*240*3)data.clear(); + data=fetch(paths,"/artwork?title="+encoded+"&kind="+kind,bytes,12);if(data.size()!=size_t(bytes))data.clear(); }catch(...){} if(!data.empty())try{ fs::create_directories(folder);fs::permissions(folder,fs::perms::owner_all); const auto temp=file.string()+".tmp";{std::ofstream out(temp,std::ios::binary);out.write(data.data(),data.size());if(!out)throw std::runtime_error("Cover cache write failed");} fs::rename(temp,file);writeJson(meta,{{"title",title},{"saved",std::time(nullptr)}}); - // Bound disk usage to roughly 15 MiB; concurrent writers only prune old entries. + // Separate bounded namespaces: ~56 MiB of portraits and ~84 MiB of heroes. std::lock_guard cacheGuard(mutex);std::vector files; for(const auto& item:fs::directory_iterator(folder))if(item.path().extension()==".rgb")files.push_back(item.path()); std::sort(files.begin(),files.end(),[](const auto& a,const auto& b){return fs::last_write_time(a)128){auto oldest=files.front();fs::remove(oldest);oldest.replace_extension(".json");fs::remove(oldest);files.erase(files.begin());} + while(files.size()>size_t(kind=="hero"?128:512)){auto oldest=files.front();fs::remove(oldest);oldest.replace_extension(".json");fs::remove(oldest);files.erase(files.begin());} }catch(...){} - std::lock_guard finished(mutex);if(data.empty())coverFailedAt[title]=std::time(nullptr);covers[title]=std::move(data);--coversLoading; - }).detach();}catch(...){covers.erase(title);--coversLoading;return {};} + std::lock_guard finished(mutex);if(data.empty())coverFailedAt[cacheKey]=std::time(nullptr);covers[cacheKey]=std::move(data);--coversLoading; + }).detach();}catch(...){covers.erase(cacheKey);--coversLoading;return {};} return "pending"; } template void add(const Paths& paths,const std::string& id,RPC rpc){ diff --git a/homebrew/botty/src/server.cpp b/homebrew/botty/src/server.cpp index e005983..85a984b 100644 --- a/homebrew/botty/src/server.cpp +++ b/homebrew/botty/src/server.cpp @@ -895,14 +895,19 @@ int main(int argc,char** argv) { result["torrentRemovalSupported"]=true;result["exploreSupported"]=true;result["explore"]=explore.state(owned); reply(res,result); }); - server.Get("/api/explore/artwork",[](const auto& req,auto& res){std::lock_guard guard(lock);nativeUpdater->requireAdmission();auto data=explore.artwork(paths,req.get_param_value("id"));if(data=="pending"){res.status=202;res.set_content("Pending","text/plain");}else if(data.empty()){res.status=404;res.set_content("Unavailable","text/plain");}else res.set_content(data,"application/octet-stream");}); + server.Get("/api/explore/artwork",[](const auto& req,auto& res){std::lock_guard guard(lock);nativeUpdater->requireAdmission();const auto kind=req.has_param("kind")?req.get_param_value("kind"):"cover";auto data=explore.artwork(paths,req.get_param_value("id"),false,kind);if(data=="pending"){res.status=202;res.set_content("Pending","text/plain");}else if(data.empty()){res.status=404;res.set_content("Unavailable","text/plain");}else res.set_content(data,"application/octet-stream");}); // Resolve only known local entries or search IDs, never client-supplied URLs/titles. server.Get("/api/artwork",[](const auto& req,auto& res){ std::unique_lock guard(lock);nativeUpdater->requireAdmission(); const auto id=req.get_param_value("id"); + const auto kind=req.has_param("kind")?req.get_param_value("kind"):"cover"; + if(kind!="cover"&&kind!="hero")throw std::runtime_error("Invalid artwork kind"); std::string data; - if(id.rfind("m:",0)==0){guard.unlock();data=emulation.artwork(id.substr(2));if(data.empty())data=legacyEmulation.artwork(id.substr(2));} - else data=id.rfind("s:",0)==0?search.artwork(paths,id.substr(2)):explore.artwork(paths,id,true); + if(id.rfind("m:",0)==0){ + guard.unlock();auto title=emulation.artworkTitle(id.substr(2));if(title.empty())title=legacyEmulation.artworkTitle(id.substr(2)); + if(!title.empty())data=search.artworkTitle(paths,title,kind); + } + else data=id.rfind("s:",0)==0?search.artwork(paths,id.substr(2),false,kind):explore.artwork(paths,id,true,kind); if(data=="pending"){res.status=202;res.set_content("Pending","text/plain");} else if(data.empty()){res.status=404;res.set_content("Unavailable","text/plain");} else res.set_content(data,"application/octet-stream"); diff --git a/homebrew/botty/tests/emulation-test.cpp b/homebrew/botty/tests/emulation-test.cpp index b64bc33..c82956f 100644 --- a/homebrew/botty/tests/emulation-test.cpp +++ b/homebrew/botty/tests/emulation-test.cpp @@ -24,6 +24,15 @@ int main(){ const auto resilient=emulator.state();assert(resilient.at("entries").size()==1 && resilient.at("error")!=""); writeJson(malformed,{{"name","Downloaded fixture"},{"type","game"}}); assert(Emulation::entry(data,data/"roms/Game.xci","download").at("name")=="Downloaded fixture"); + writeJson(malformed,{{"verified",true},{"kind","roms"},{"name","Receipt game"}, + {"verifiedFileSize",fs::file_size(data/"roms/Game.xci")}, + {"verifiedMtime",fs::last_write_time(data/"roms/Game.xci").time_since_epoch().count()}, + {"titles",json::array()}}); + const auto named=emulator.state(); + for(const auto& item:named.at("entries"))if(item.at("kind")=="roms"){ + assert(item.at("name")=="Receipt game"); + assert(emulator.artworkTitle(item.at("artworkId"))=="Receipt game"); + } fs::remove(malformed); const auto id=state.at("entries").at(0).at("id").get();assert(id.size()==64);assert(emulator.resolve(id).at("kind")=="roms"); bool missing=false;try{emulator.launchPath(id);}catch(const std::exception&){missing=true;}assert(missing); diff --git a/homebrew/botty/tests/emulation_library.py b/homebrew/botty/tests/emulation_library.py index 6d280fe..9cec86b 100644 --- a/homebrew/botty/tests/emulation_library.py +++ b/homebrew/botty/tests/emulation_library.py @@ -47,6 +47,17 @@ def get(path,binary=False,status=200): except urllib.error.HTTPError as error:response=error assert response.status==status,(response.status,response.read()) value=response.read();return value if binary else json.loads(value) + def no_embedded_cover(ident): + # No remote artwork configuration in this fixture: never substitute square icons. + req=urllib.request.Request(f'http://127.0.0.1:{port}/api/artwork?id=m:'+ident,headers={'X-Botty-Token':token}) + for _ in range(100): + try: + with urllib.request.urlopen(req,timeout=5) as response: + assert response.status==202;response.read() + except urllib.error.HTTPError as error: + assert error.code==404;return + time.sleep(.02) + raise AssertionError('Artwork did not settle') def entries():return get('/api/state')['emulation']['entries'] try: for _ in range(200): @@ -56,7 +67,7 @@ def entries():return get('/api/state')['emulation']['entries'] rows=entries();selected=next(r for r in rows if r['kind']=='roms') legacy_entry=next(r for r in rows if r.get('source')=='existing-ps5') assert legacy_entry['name']=='Legacy fixture' - assert len(get('/api/artwork?id=m:'+legacy_entry['artworkId'],binary=True))==160*240*3 + no_embedded_cover(legacy_entry['artworkId']) assert selected['name']=='Mario Kart 8 Deluxe',selected assert selected['titleId']==TITLE assert selected['baseVersion']=='v1.0.0' and selected['updateVersion']=='v3.0.5' @@ -67,25 +78,21 @@ def entries():return get('/api/state')['emulation']['entries'] assert selected['library']['verified'] is False # display is not import verification orphan=next(r for r in rows if r['path'].endswith('unrelated.nsp')) assert orphan['dependencyStatus']=='base-game-missing' - pixels=get('/api/artwork?id=m:'+selected['id'],binary=True) - assert len(pixels)==160*240*3 - assert pixels[(120*160+80)*3:(120*160+80)*3+3]==bytes([255,0,0]) + no_embedded_cover(selected['id']) old_artwork=selected['artworkId'] hash_value=2166136261 for ch in game.name.encode():hash_value=((hash_value^ch)*16777619)&0xffffffff cover=root/'eden/covers'/f'{hash_value:08x}.tga';cover.parent.mkdir() - # Custom covers from the embedded launcher's cache take precedence. + # Embedded launcher artwork changes no longer replace uniform remote portraits. cover.write_bytes(bytes([0,0,2])+bytes(9)+bytes([2,0,2,0,24,32])+bytes([255,0,0])*4) selected=next(r for r in entries() if r['kind']=='roms') assert selected['artworkId']!=old_artwork - pixels=get('/api/artwork?id=m:'+selected['artworkId'],binary=True) - assert pixels[(120*160+80)*3:(120*160+80)*3+3]==bytes([0,0,255]) + no_embedded_cover(selected['artworkId']) # A valid TGA header with no pixels must fall back to the embedded icon, # never expose an uninitialized decoder buffer. cover.write_bytes(cover.read_bytes()[:18]) selected=next(r for r in entries() if r['kind']=='roms') - pixels=get('/api/artwork?id=m:'+selected['artworkId'],binary=True) - assert pixels[(120*160+80)*3:(120*160+80)*3+3]==bytes([255,0,0]) + no_embedded_cover(selected['artworkId']) get('/api/artwork?id=m:not-a-file',binary=True,status=404) # Settings changes do not require re-reading a multi-GB game. settings.write_text(json.dumps({'games':{TITLE:{'mods':False}}})) diff --git a/homebrew/botty/tests/search_integration.py b/homebrew/botty/tests/search_integration.py index a6746e9..14fc4c7 100644 --- a/homebrew/botty/tests/search_integration.py +++ b/homebrew/botty/tests/search_integration.py @@ -2,11 +2,17 @@ import hashlib, importlib.util, json, pathlib, socket, ssl, subprocess, tempfile, threading, time, urllib.request, urllib.parse from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from rtorrent_fixture import RtorrentFixture +from switch_fixtures import package HERE = pathlib.Path(__file__).resolve().parent spec = importlib.util.spec_from_file_location('fixtures', HERE/'make_fixtures.py') fixtures = importlib.util.module_from_spec(spec); spec.loader.exec_module(fixtures) with tempfile.TemporaryDirectory(prefix='botty-search-') as temp: root=pathlib.Path(temp); fixtures.build(root/'fixtures') + (root/'eden/roms').mkdir(parents=True);(root/'eden/keys').mkdir(parents=True) + (root/'eden/keys/prod.keys').write_text('# synthetic fixture') + (root/'eden/roms/demo.nsp').write_bytes(package(display=('Switch Fixture','1.0.0'))) + (root/'eden/updates').mkdir() + (root/'eden/updates/dlc.nsp').write_bytes(package('dlc')) complete=root/'downloads/complete'; complete.mkdir(parents=True) (complete/'sample.rar').write_bytes((root/'fixtures/app.rar').read_bytes()) size=(complete/'sample.rar').stat().st_size @@ -48,7 +54,8 @@ def do_GET(self): else:self.reply([dict(row,indexerId=2),dict(row,categories=[{'id':2000}]),row,dict(other,title='Other homebrew PS5 fixture',seeders=6),dict(row,title='Original homebrew PS5 fixture',grabs=4)]) elif url.path=='/artwork': cover_calls.append(self.path) - data=bytes([12,80,160])*(160*240);self.send_response(200);self.send_header('Content-Length',str(len(data)));self.end_headers();self.wfile.write(data) + hero=urllib.parse.parse_qs(url.query).get('kind')==['hero'] + data=bytes([12,80,160])*(640*360 if hero else 160*240);self.send_response(200);self.send_header('Content-Length',str(len(data)));self.end_headers();self.wfile.write(data) else: assert url.path in ['/1/download','/8/download'] and 'apikey' not in urllib.parse.parse_qs(url.query) data=remote_payload or b'd4:infod4:name7:fixtureee';self.send_response(200);self.send_header('Content-Length',str(len(data)));self.end_headers();self.wfile.write(data) @@ -165,13 +172,23 @@ def artwork(): with urllib.request.urlopen(req,timeout=2) as response: data=response.read();return data if response.status==200 else None assert until(artwork)==bytes([12,80,160])*(160*240) - def catalog_artwork(ident): - req=urllib.request.Request(f'http://127.0.0.1:{port}/api/artwork?id='+urllib.parse.quote(ident),headers={'X-Botty-Token':token}) + assert urllib.parse.parse_qs(urllib.parse.urlsplit(cover_calls[-1]).query)['title']==[browse['results'][0]['name']] # Preserve release markers for the resolver. + def catalog_artwork(ident,kind="cover"): + req=urllib.request.Request(f'http://127.0.0.1:{port}/api/artwork?id='+urllib.parse.quote(ident)+'&kind='+kind,headers={'X-Botty-Token':token}) with urllib.request.urlopen(req,timeout=2) as response: data=response.read();return data if response.status==200 else None assert state['catalogArtworkSupported'] assert until(lambda:catalog_artwork('t:1'))==bytes([12,80,160])*(160*240) assert until(lambda:catalog_artwork('s:'+results[0]['id']))==bytes([12,80,160])*(160*240) + switch=next(row for row in request('/api/state')['emulation']['entries'] if row['name']=='Switch Fixture') + assert until(lambda:catalog_artwork('m:'+switch['artworkId']))==bytes([12,80,160])*(160*240) + assert until(lambda:catalog_artwork('m:'+switch['artworkId'],'hero'))==bytes([12,80,160])*(640*360) + dlc=next(row for row in request('/api/state')['emulation']['entries'] if row['path'].endswith('/dlc.nsp')) + assert dlc['name']=='Switch Fixture' + # DLC has no Control NACP: resolve through its projected base-game name. + assert until(lambda:catalog_artwork('m:'+dlc['artworkId']))==bytes([12,80,160])*(160*240) + assert until(lambda:catalog_artwork('m:'+dlc['artworkId'],'hero'))==bytes([12,80,160])*(640*360) + assert until(lambda:catalog_artwork('t:1','hero'))==bytes([12,80,160])*(640*360) for invalid in ['t:999','j:unknown','https://example.com/image.jpg','../../outside']: try:catalog_artwork(invalid);raise AssertionError('Unknown artwork ID accepted') except urllib.error.HTTPError as e:assert e.code==404 diff --git a/packages/botty-native/botty-native-source.tar.gz b/packages/botty-native/botty-native-source.tar.gz index 1c28dee..d3ccde9 100644 Binary files a/packages/botty-native/botty-native-source.tar.gz and b/packages/botty-native/botty-native-source.tar.gz differ diff --git a/packages/botty-native/eboot.bin b/packages/botty-native/eboot.bin index 227e0a9..6395b3f 100644 Binary files a/packages/botty-native/eboot.bin and b/packages/botty-native/eboot.bin differ diff --git a/packages/botty-native/manifest.json b/packages/botty-native/manifest.json index 2597949..82b3b66 100644 --- a/packages/botty-native/manifest.json +++ b/packages/botty-native/manifest.json @@ -53,8 +53,8 @@ }, { "path": "eboot.bin", - "size": 33222252, - "sha256": "ffc310a4388d8dcd94ed45b1b51097333a2554833314a6924e938c506c1f4c02" + "size": 33234524, + "sha256": "5041b8c23ae9febfbfbac265b6700727eb74e381ff95f00a45612c307fae3bb5" }, { "path": "emulation-runtime.json", diff --git a/packages/botty-native/source/manifest.json b/packages/botty-native/source/manifest.json index 63cfb5d..acacdc6 100644 --- a/packages/botty-native/source/manifest.json +++ b/packages/botty-native/source/manifest.json @@ -1,73 +1,73 @@ { "schema": 1, "archive": "botty-native-source.tar.gz", - "size": 1176838696, - "sha256": "a957e699bcf9872665c77d142b4a3c51e68cc2729a1e24f215660e9db49f8ea6", + "size": 1176845715, + "sha256": "9f9d677e081c92e2b1a77c271328e25b11acfbc855f653787db18c89b535df9d", "parts": [ { "path": "source/native-source.part001", "size": 94371840, - "sha256": "c66b7f551857b439c3e1dccfd0baa0c363c7d3aef0deb6a284afd08e6ceca1b8" + "sha256": "1955b2714d70755317a00a50393b050a089cdaae6438978350ffd19f984b5bd4" }, { "path": "source/native-source.part002", "size": 94371840, - "sha256": "f603f5d109b57a4a0709dffe5bc24cc30a560a4d4450467168bcf6470a9d3215" + "sha256": "1d0bbe2e7224ea7b868f1b09ed4119ec10fc9067c777cf9a4674a328de022d17" }, { "path": "source/native-source.part003", "size": 94371840, - "sha256": "b90e25c46acb0f6e4c2a75f2f3cf7584cc4fd8194ff37b8ae6ccd637845f10ba" + "sha256": "2ac51b47471b8acc2c4fa0099883dc8fb6f7089f9acd0052c4bb6a7c22f1ddaf" }, { "path": "source/native-source.part004", "size": 94371840, - "sha256": "4d150c56abe16b358fe18461ca5979a717fbe13ef857eb16d84b6403d42b4208" + "sha256": "7cc4d9409b5e65370e778d92d37e5bc5c45585ef211106c40898f272059ed29d" }, { "path": "source/native-source.part005", "size": 94371840, - "sha256": "93c5e0be846ea02b4db4855fbecfded48c28a085978db1286e46c18b35cc6516" + "sha256": "7aa0a3f9c2a880dca026acfa365dbe64554ce6d0115ce827762bb8edef922398" }, { "path": "source/native-source.part006", "size": 94371840, - "sha256": "7fb00442056bc03e6692d6cff38bf00c4bc58dc43cf0366eb06bb3f67729fdaa" + "sha256": "bb6b8c6d8b5d100e901ed3d0520d640ed4613ce7886a72e4b196d5678a3cdf8c" }, { "path": "source/native-source.part007", "size": 94371840, - "sha256": "57af7434ec21cad6665d3a3090811bac2c0cb381ebc3ceebcae392a7c036d65b" + "sha256": "f3352228eb7c2624ee48755a7d4b78c95fd5a2503ec2b08e5e57404dac2833cb" }, { "path": "source/native-source.part008", "size": 94371840, - "sha256": "ddf739e5428bce50195ad3697ee910c0b3fb399f4097032bd14059ccd4e0a72c" + "sha256": "8eab1f5dcc623159f6164cc6e713abdb4cb4a2efc355cec02a6e4466b816f3e0" }, { "path": "source/native-source.part009", "size": 94371840, - "sha256": "b3acb82e893e4e89d347c8036f79e9cb15fc8143741a534c291b0be8f689993e" + "sha256": "8ae21ea06aa405c937e04702050e39870a79b1d210787ae3741dbe96e1cf06e4" }, { "path": "source/native-source.part010", "size": 94371840, - "sha256": "ae57cb377448e3a78fa01fcc9bc715ac9f44ac3df2a914161ab91c27134034b6" + "sha256": "24cd4ea4148ee430238afcdfa0661b380970dd160c0d2bc0470fd1e25824f34f" }, { "path": "source/native-source.part011", "size": 94371840, - "sha256": "b780fb494f1c2aadb6abf378a97bed3be3ec2570111e185ef53d44c74566f052" + "sha256": "c5125b1609a6eb8b114f65bb5e6eef6915d9e2e084db51392c369569b3049e5b" }, { "path": "source/native-source.part012", "size": 94371840, - "sha256": "63b9f81e2116e34d083c75a4af2f38caebf22405f2f7326919f2a5afd3305bc7" + "sha256": "c7fc8bbebcd14630162fb9c2bf4fab6437073ce035c60bcae2a37eedb41aaa3e" }, { "path": "source/native-source.part013", - "size": 44376616, - "sha256": "ba63c6d9b8c2111c439acadd8fb5a37d1f7b8b2b2681faa0a765dc7f3fca98ce" + "size": 44383635, + "sha256": "b89b1c42ac418269c459669a90ecf5910b36df522f48c643d44f43c521eade70" } ] } diff --git a/packages/botty-native/source/native-source.part001 b/packages/botty-native/source/native-source.part001 index dc12963..9dcb7e0 100644 Binary files a/packages/botty-native/source/native-source.part001 and b/packages/botty-native/source/native-source.part001 differ diff --git a/packages/botty-native/source/native-source.part002 b/packages/botty-native/source/native-source.part002 index 760d9d2..7584df8 100644 Binary files a/packages/botty-native/source/native-source.part002 and b/packages/botty-native/source/native-source.part002 differ diff --git a/packages/botty-native/source/native-source.part003 b/packages/botty-native/source/native-source.part003 index ec70446..eb37f57 100644 Binary files a/packages/botty-native/source/native-source.part003 and b/packages/botty-native/source/native-source.part003 differ diff --git a/packages/botty-native/source/native-source.part004 b/packages/botty-native/source/native-source.part004 index 7965dfa..fa52b88 100644 Binary files a/packages/botty-native/source/native-source.part004 and b/packages/botty-native/source/native-source.part004 differ diff --git a/packages/botty-native/source/native-source.part005 b/packages/botty-native/source/native-source.part005 index 2c71850..013cb24 100644 Binary files a/packages/botty-native/source/native-source.part005 and b/packages/botty-native/source/native-source.part005 differ diff --git a/packages/botty-native/source/native-source.part006 b/packages/botty-native/source/native-source.part006 index 602da2c..4286332 100644 Binary files a/packages/botty-native/source/native-source.part006 and b/packages/botty-native/source/native-source.part006 differ diff --git a/packages/botty-native/source/native-source.part007 b/packages/botty-native/source/native-source.part007 index 91b599a..bce684c 100644 Binary files a/packages/botty-native/source/native-source.part007 and b/packages/botty-native/source/native-source.part007 differ diff --git a/packages/botty-native/source/native-source.part008 b/packages/botty-native/source/native-source.part008 index 6e02018..db2c5b7 100644 Binary files a/packages/botty-native/source/native-source.part008 and b/packages/botty-native/source/native-source.part008 differ diff --git a/packages/botty-native/source/native-source.part009 b/packages/botty-native/source/native-source.part009 index b6acdfa..2bb7f7c 100644 Binary files a/packages/botty-native/source/native-source.part009 and b/packages/botty-native/source/native-source.part009 differ diff --git a/packages/botty-native/source/native-source.part010 b/packages/botty-native/source/native-source.part010 index 85e42fe..93f4b7d 100644 Binary files a/packages/botty-native/source/native-source.part010 and b/packages/botty-native/source/native-source.part010 differ diff --git a/packages/botty-native/source/native-source.part011 b/packages/botty-native/source/native-source.part011 index b6d80ae..a2fa19a 100644 Binary files a/packages/botty-native/source/native-source.part011 and b/packages/botty-native/source/native-source.part011 differ diff --git a/packages/botty-native/source/native-source.part012 b/packages/botty-native/source/native-source.part012 index 0b33989..b652196 100644 Binary files a/packages/botty-native/source/native-source.part012 and b/packages/botty-native/source/native-source.part012 differ diff --git a/packages/botty-native/source/native-source.part013 b/packages/botty-native/source/native-source.part013 index e112aca..9375a27 100644 Binary files a/packages/botty-native/source/native-source.part013 and b/packages/botty-native/source/native-source.part013 differ diff --git a/packages/botty-release.json b/packages/botty-release.json index 7060a80..758e106 100644 --- a/packages/botty-release.json +++ b/packages/botty-release.json @@ -22,8 +22,8 @@ "botty-native/assets/nebula.rgb": "86e016792eb91775163b9ffea8691314ee3a96ddeb1acacfdcc2d60a210caff1", "botty-native/assets/ui-font.bin": "d7cc26de2042e04bc70bf9a41dce63eebae0ac3c9d15441bc14290a29ee06352", "botty-native/assets/vault.rgba": "22ce003865a105b63b886c71f81d9550e6189d4051bc8df88322fd09f984c15a", - "botty-native/botty-native-source.tar.gz": "31a760ab048240c12ad32b82253275e48cb88a292ab3267bde00528f2a0a6d49", - "botty-native/eboot.bin": "ffc310a4388d8dcd94ed45b1b51097333a2554833314a6924e938c506c1f4c02", + "botty-native/botty-native-source.tar.gz": "59bd2f86286c4dd43cb77cefb8ec18ff94a034ec4c622c3b78f5d46c3ab7a066", + "botty-native/eboot.bin": "5041b8c23ae9febfbfbac265b6700727eb74e381ff95f00a45612c307fae3bb5", "botty-native/emulation-runtime.json": "88c2ef2f6edb5de8ff0e9c940ceaddea2ad2f09c502de62b1ddc76dcf488f033", "botty-native/emulation/core-homebrew.nro": "b3c95e1c0dba8080fd172d1554af33506126ed0f74c18fb42b4ed3b9ea9032c2", "botty-native/emulation/eboot.bin": "7d1a01dfa7b1ea8fb49b006ca818c98289592e10ae6f11eb322a72b2b261fbae", @@ -102,36 +102,36 @@ "botty-native/emulation/ultracam/botw/exefs/subsdk3": "1216d58cb3eb8028a905a3c88b104a1338b48cff2b1dfeac571bff7b9826dac6", "botty-native/emulation/ultracam/botw/romfs/UltraCam/maxlastbreath.ini": "8ff291f955068298d1210212ace9787f9701f3cf7a8f704f7531b59b0f37f80a", "botty-native/emulation/ultracam/manifest.json": "44f8aaf6be386becc77138a7ef8385cf7b999bd4185c29346d83fe1afa01f7c9", - "botty-native/manifest.json": "397e1484a828ff9c5b6000ee77525f9d65cf21a39a29919d91c49e43a9a58119", + "botty-native/manifest.json": "c3152e9a9d04b52b632ee0d3bb495dbe1c3ffce04aac22cbebdc55599cebc5e6", "botty-native/sce_module/libc.prx": "e6ff45d16adf687855cc3b33b0c8a4132b6504360b221e0a34c7e99fb3ba0036", "botty-native/sce_sys/icon0.png": "b04611b29f7fbd980d6ea550379df4cc897d077bed06b69eaaa59ec58ea2987d", "botty-native/sce_sys/param.json": "e1f57cd04902e9461c92c4d232e4247dbec757762ed993c43ab01cfc2cb428a1", "botty-native/sce_sys/pic0.dds": "1005350b9f1feecdd6eb1006c389a6bd738d31b1090325deecaf799c8c695724", "botty-native/sce_sys/snd0.at9": "4284d15a2bae57739c944355c917ab11fce7b18d9b47054e2c7e3746a3c57218", - "botty-native/source/manifest.json": "16f632e4fb925ac788258d1415b5afc98af0c9e54a34dbfc3034b81716a17362", - "botty-native/source/native-source.part001": "c66b7f551857b439c3e1dccfd0baa0c363c7d3aef0deb6a284afd08e6ceca1b8", - "botty-native/source/native-source.part002": "f603f5d109b57a4a0709dffe5bc24cc30a560a4d4450467168bcf6470a9d3215", - "botty-native/source/native-source.part003": "b90e25c46acb0f6e4c2a75f2f3cf7584cc4fd8194ff37b8ae6ccd637845f10ba", - "botty-native/source/native-source.part004": "4d150c56abe16b358fe18461ca5979a717fbe13ef857eb16d84b6403d42b4208", - "botty-native/source/native-source.part005": "93c5e0be846ea02b4db4855fbecfded48c28a085978db1286e46c18b35cc6516", - "botty-native/source/native-source.part006": "7fb00442056bc03e6692d6cff38bf00c4bc58dc43cf0366eb06bb3f67729fdaa", - "botty-native/source/native-source.part007": "57af7434ec21cad6665d3a3090811bac2c0cb381ebc3ceebcae392a7c036d65b", - "botty-native/source/native-source.part008": "ddf739e5428bce50195ad3697ee910c0b3fb399f4097032bd14059ccd4e0a72c", - "botty-native/source/native-source.part009": "b3acb82e893e4e89d347c8036f79e9cb15fc8143741a534c291b0be8f689993e", - "botty-native/source/native-source.part010": "ae57cb377448e3a78fa01fcc9bc715ac9f44ac3df2a914161ab91c27134034b6", - "botty-native/source/native-source.part011": "b780fb494f1c2aadb6abf378a97bed3be3ec2570111e185ef53d44c74566f052", - "botty-native/source/native-source.part012": "63b9f81e2116e34d083c75a4af2f38caebf22405f2f7326919f2a5afd3305bc7", - "botty-native/source/native-source.part013": "ba63c6d9b8c2111c439acadd8fb5a37d1f7b8b2b2681faa0a765dc7f3fca98ce", + "botty-native/source/manifest.json": "3475fbdbfc1cf623e366ae5d6a7df2435cef02e655dfa1cf5fe9047e94b84a7a", + "botty-native/source/native-source.part001": "1955b2714d70755317a00a50393b050a089cdaae6438978350ffd19f984b5bd4", + "botty-native/source/native-source.part002": "1d0bbe2e7224ea7b868f1b09ed4119ec10fc9067c777cf9a4674a328de022d17", + "botty-native/source/native-source.part003": "2ac51b47471b8acc2c4fa0099883dc8fb6f7089f9acd0052c4bb6a7c22f1ddaf", + "botty-native/source/native-source.part004": "7cc4d9409b5e65370e778d92d37e5bc5c45585ef211106c40898f272059ed29d", + "botty-native/source/native-source.part005": "7aa0a3f9c2a880dca026acfa365dbe64554ce6d0115ce827762bb8edef922398", + "botty-native/source/native-source.part006": "bb6b8c6d8b5d100e901ed3d0520d640ed4613ce7886a72e4b196d5678a3cdf8c", + "botty-native/source/native-source.part007": "f3352228eb7c2624ee48755a7d4b78c95fd5a2503ec2b08e5e57404dac2833cb", + "botty-native/source/native-source.part008": "8eab1f5dcc623159f6164cc6e713abdb4cb4a2efc355cec02a6e4466b816f3e0", + "botty-native/source/native-source.part009": "8ae21ea06aa405c937e04702050e39870a79b1d210787ae3741dbe96e1cf06e4", + "botty-native/source/native-source.part010": "24cd4ea4148ee430238afcdfa0661b380970dd160c0d2bc0470fd1e25824f34f", + "botty-native/source/native-source.part011": "c5125b1609a6eb8b114f65bb5e6eef6915d9e2e084db51392c369569b3049e5b", + "botty-native/source/native-source.part012": "c7fc8bbebcd14630162fb9c2bf4fab6437073ce035c60bcae2a37eedb41aaa3e", + "botty-native/source/native-source.part013": "b89b1c42ac418269c459669a90ecf5910b36df522f48c643d44f43c521eade70", "botty/LICENSE": "3972dc9744f6499f0f9b2dbf76696f2ae7ad8af9b23dde66d6af86c9dfb36986", "botty/NOTICE.md": "1809e80ab148ce8369192e65dee425eff1c45511f270cb65306610a3971503fa", - "botty/botty-manager.elf": "2559bfd938957c1081e9cd4db3f1c7715a81b4ec6a582b4671bdfdd235a8a9be", - "botty/botty-source.tar.gz": "5e19d6c867a55b343b416df40be1cd14e5ca30185b2c981ac3499191c1491be5", + "botty/botty-manager.elf": "4703ecb622e5adb6be9b98466c9259adb45f86ed7c0cf8b304b3fc3cf8e541ac", + "botty/botty-source.tar.gz": "cfa013a0526685842ec2ad5a3b6ae649edf6f9a4da310cd70792c9db18f5113d", "botty/cacert.pem": "d5945f3f21330ef0cd1dad15ccbf010b33088026539cd424add4c2a1e8205e0b", "botty/game-compressor-NOTICE.md": "20a9a00d6e88b2aab616c96b910aab4846363b744e2782d9617c81ba6e270ccc", "botty/game-compressor-source.tar.gz": "3c3fb3a5dd18fd117583151108ceca23b6e65e1c6decb473cdc1af1b5ac14629", "botty/game-compressor.elf": "6712948e7d4dff3be8c0a515dae4bb29465d63b049d371ae0404b231ba17c8cf", "botty/icon0.png": "f84f3a673dd9b53b2b7b3608058cbd2b983dc9cb649dbe306d53f36e205ea080", - "botty/manifest.json": "fedef0113985d997c9c39102fd0429a66f4946a5c37ca83e511e02ea5b3a8880", + "botty/manifest.json": "8bd33fa299626f0941a2c6c0068646d28210f4f6147434652bac871414a0fd30", "botty/ui/app.js": "752d7a9e4ebde1a8277a961a7f57ef4c92e50ca4b81ee3689fd991fd36d34615", "botty/ui/index.html": "1090cd6e1b435f4ad29dd1b0a64f128cc2fa3cc5b1d63644235e76eb1970dce9", "botty/ui/style.css": "9ecb5da30d595865d1cc28d8192616b48b3ac1aee80c3bc952276c8141f7e8b5", diff --git a/packages/botty/botty-manager.elf b/packages/botty/botty-manager.elf index 6d4b8f1..fd084ad 100755 Binary files a/packages/botty/botty-manager.elf and b/packages/botty/botty-manager.elf differ diff --git a/packages/botty/botty-source.tar.gz b/packages/botty/botty-source.tar.gz index e425038..e832cc1 100644 Binary files a/packages/botty/botty-source.tar.gz and b/packages/botty/botty-source.tar.gz differ diff --git a/packages/botty/manifest.json b/packages/botty/manifest.json index 2b55173..927b639 100644 --- a/packages/botty/manifest.json +++ b/packages/botty/manifest.json @@ -8,8 +8,8 @@ "files": [ { "path": "botty-manager.elf", - "size": 16294744, - "sha256": "2559bfd938957c1081e9cd4db3f1c7715a81b4ec6a582b4671bdfdd235a8a9be" + "size": 16295056, + "sha256": "4703ecb622e5adb6be9b98466c9259adb45f86ed7c0cf8b304b3fc3cf8e541ac" }, { "path": "icon0.png",