[syphar]

The super high server load today got me to look into where certain endpoints spend time again.

And one thing was odd:

This is happening when our source-browser inspects the list of files in the source archive to generate the directory listing.

There seem to be crates where the source archive has >10k files:

           name           |   version   | source_file_count
--------------------------+-------------+-------------------
 shrust                   | 0.0.4       |             80827
 skia-bindings            | 0.14.2      |             63739
 skia-bindings            | 0.17.0      |             62966
 skia-bindings            | 0.15.0      |             62344
 skia-bindings            | 0.16.0      |             62344
 skia-bindings            | 0.14.4      |             62344
 shrust                   | 0.0.3       |             57486
 tsrun                    | 0.1.16      |             56217
 tsrun                    | 0.1.15      |             56217

Using these I wrote a benchmark, and could reproduce the issue when I had parallel clients on the same sqlite db.

So I changed the logic so we're using sqlite functions to do the work:

 With 16 concurrent readers, time until all finish:

   Listing          Before    After
  ━━━━━━━━━━━━━━━  ━━━━━━━━  ━━━━━━━
   Root             1.16 s    51 ms
  ───────────────  ────────  ───────
   Nested folder    252 ms    34 ms

or even faster.

[syphar]

r? @GuillaumeGomez

[GuillaumeGomez]

Nice, thanks!