Repository navigation
Bug: Reports scraper worked, but it doesn't show log nor update database #30
Description
Which service is this about?
Morph
What happened?
After Installed latest morph-cli from source, it now reports Scraper didn't output anything, but it ran successfully. but neither updates data.sqlite nor reports actual scraper output.
ianh@ben:.../example_ruby_scraper (main)$ echo 'select rowid,* from data' | sqlite3 --json data.sqlite
[{"rowid":38,"name":"Example Domain"}]
ianh@ben:.../example_ruby_scraper (main)$ md5sum data.sqlite
bff4ecc9a65c97ec18c1eb80ff87abb4 data.sqlite
ianh@ben:.../example_ruby_scraper (main)$ morph
Uploading 30.41 KiB (including data.sqlite)...
Scraper didn't output anything, but it ran successfully.
ianh@ben:.../example_ruby_scraper (main)$ echo 'select rowid,* from data' | sqlite3 --json data.sqlite
[{"rowid":38,"name":"Example Domain"}]
ianh@ben:.../example_ruby_scraper (main)$ md5sum data.sqlite
bff4ecc9a65c97ec18c1eb80ff87abb4 data.sqlite
Steps to reproduce
As above
What did you expect to happen?
It to report the system and stdout lines reported when it is run on https://morph.io/ianheggie-oaf/example_ruby_scraper
Injecting configuration and compiling...
[1G [1G-----> Ruby app detected
[1G-----> Installing bundler 2.3.10
[1G-----> Removing BUNDLED WITH version in the Gemfile.lock
[1G-----> Compiling Ruby
[1G-----> Using Ruby version: ruby-3.2.2
[1G-----> Installing dependencies using bundler 2.3.10
[1G Running: BUNDLE_WITHOUT='development:test' BUNDLE_PATH=vendor/bundle BUNDLE_BIN=vendor/bundle/bin BUNDLE_DEPLOYMENT=1 bundle install -j4
[1G Fetching gem metadata from https://rubygems.org/.........
[1G Fetching https://github.com/openaustralia/scraperwiki-ruby.git
[1G Fetching public_suffix 7.0.5
[1G Fetching base64 0.3.0
[1G Using bundler 2.4.10
[1G Fetching connection_pool 3.0.2
[1G Fetching domain_name 0.6.20240107
[1G Installing base64 0.3.0
[1G Installing connection_pool 3.0.2
[1G Installing public_suffix 7.0.5
[1G Fetching mutex_m 0.3.0
[1G Fetching logger 1.7.0
[1G Installing domain_name 0.6.20240107
[1G Installing mutex_m 0.3.0
[1G Installing logger 1.7.0
[1G Fetching mime-types-data 3.2026.0701
[1G Fetching net-http-digest_auth 1.4.1
[1G Fetching nkf 0.3.0
[1G Installing net-http-digest_auth 1.4.1
[1G Fetching racc 1.8.1
[1G Fetching webrick 1.9.2
[1G Installing mime-types-data 3.2026.0701
[1G Installing nkf 0.3.0 with native extensions
[1G Installing racc 1.8.1 with native extensions
[1G Installing webrick 1.9.2
[1G Fetching webrobots 0.1.2
[1G Installing webrobots 0.1.2
[1G Fetching sqlite3 2.2.0 (x86_64-linux-gnu)
[1G Fetching rubyntlm 0.6.5
[1G Installing rubyntlm 0.6.5
[1G Fetching net-http-persistent 4.0.8
[1G Installing net-http-persistent 4.0.8
[1G Fetching httpclient 2.9.0
[1G Installing httpclient 2.9.0
[1G Fetching addressable 2.9.0
[1G Installing sqlite3 2.2.0 (x86_64-linux-gnu)
[1G Installing addressable 2.9.0
[1G Fetching http-cookie 1.1.6
[1G Installing http-cookie 1.1.6
[1G Fetching mime-types 3.7.0
[1G Installing mime-types 3.7.0
[1G Fetching sqlite_magic 0.0.6
[1G Installing sqlite_magic 0.0.6
[1G Using scraperwiki 3.0.1 from https://github.com/openaustralia/scraperwiki-ruby.git (at morph_defaults@fc50176)
[1G Fetching nokogiri 1.17.2 (x86_64-linux)
[1G Installing nokogiri 1.17.2 (x86_64-linux)
[1G Fetching mechanize 2.14.0
[1G Installing mechanize 2.14.0
[1G Bundle complete! 5 Gemfile dependencies, 24 gems now installed.
[1G Gems in the groups 'development' and 'test' were not installed.
[1G Bundled gems are installed into `./vendor/bundle`
[1G Bundle completed (6.56s)
[1G Cleaning up the bundler cache.
[1G Removing bundler (2.3.10)
[1G-----> Detecting rake tasks
[1G
[1G
[1G [1G-----> Discovering process types
[1G Procfile declares types -> scraper
Injecting scraper and running...
Running ruby 3.2.2 on Linux d51edb93b01a 6.14.3-x86_64-linode168 #1 SMP PREEMPT_DYNAMIC Mon Apr 21 19:47:55 EDT 2025 x86_64 x86_64 x86_64 GNU/Linux
19: Example Domain
Browser, device, or OS (if relevant)
tested with bash terminal on Ubuntu 24.04.4 LTS x86_4
Screenshots or error messages
ianh@ben:.../example_ruby_scraper (main)$->(130) mise ls
Tool Version Source Requested
ruby 3.2.2 ~/Projects/OpenAustralia/my_repos/example_ruby_scraper/Gemfile 3.2.2
Anything else?
Installed by:
ianh@ben:.../morph-cli (main)$->(1) be rake -T
rake build # Build morph-cli-0.2.5.gem into the pkg directory
rake build:checksum # Generate SHA512 checksum of morph-cli-0.2.5.gem into the checksums directory
rake clean # Remove any temporary products
rake clobber # Remove any generated files
rake install # Build and install morph-cli-0.2.5.gem into system gems
rake install:local # Build and install morph-cli-0.2.5.gem into system gems without network access
rake release[remote] # Create tag v0.2.5 and build and push morph-cli-0.2.5.gem to rubygems.org
ianh@ben:.../morph-cli (main)$ be rake build
morph-cli 0.2.5 built to pkg/morph-cli-0.2.5.gem.
ianh@ben:.../morph-cli (main)$ be rake install
morph-cli 0.2.5 built to pkg/morph-cli-0.2.5.gem.
morph-cli (0.2.5) installed.
ianh@ben:.../morph-cli (main)$ morph
Uploading 65.28 KiB...
Scraper didn't output anything, but it ran successfully.
ianh@ben:.../morph-cli (main)$ git log | head -20
commit ca6c5dc44775f4393c8eb8b4ccee1f1dd0989cf8 (HEAD -> main, origin/main, origin/HEAD)
Merge: 9b6ba2b 3dfbb07
Author: Ben Fairless <[email protected]>
Date: Mon Aug 24 13:45:35 2026 +0800
Merge pull request #27 from openaustralia/feature/7-warn-when-scraper-produces-no-output
Tell the user when a scraper runs but outputs nothing
No response
Activity
[-]Bug: Reports scraper worked, but it doesn't show log nore update database[/-][+]Bug: Reports scraper worked, but it doesn't show log nor update database[/+]ianheggie-oaf commentedon Aug 24, 2026
@benrfairless - this brings back memories of last time I tried it :(
benrfairless commentedon Aug 24, 2026
@ianheggie-oaf this relies on openaustralia/morph#1521 which has been merged but not deployed. Can you deploy this for me and try again?
benrfairless commentedon Aug 26, 2026
I've tracked this down. The scraper never ran at all, and the CLI's new "ran successfully" message was masking a server-side failure.
Root cause
#26 (commit 907b895) changed the CLI to gzip-compress the tarred code before uploading. Server-side support for that only landed in openaustralia/morph@9019144 ("Accept gzip-compressed code uploads in run API"), committed at 2026-08-24 04:11 UTC, about two and a half hours before this issue was filed. Production morph.io wasn't running it, so
Archive::Tar::Minitar.unpackchoked on the gzip bytes. Becauserun_remotestreams viaActionController::Live, the 200 status was already committed when the unpack raised, and theensureblock just closed the stream, so the client got a clean, empty HTTP 200. The CLI then read zero output lines as "scraper ran but printed nothing" thanks to #27, and reported success.That also explains the database: the run never executed, so nothing changed server-side either. (Note the CLI has never downloaded
data.sqliteback after a run; it only uploads it.)Evidence
A differential test against production morph.io, uploading the same one-line scraper (
puts 'DIFF-HARNESS-OK') twice, varying only compression:stdout: DIFF-HARNESS-OKFixes
internaloutbuildstep lines, so an entirely empty response means the run never happened. The CLI now warns that something has gone wrong on the server and exits non-zero instead of claiming success.internalerrstream. Found while testing: the server emits{"stream":"internalerr",...}(e.g. "Scraper didn't create an SQLite database..."), andMorphCLI.lograisedUnknown streamon it, so even a working run of a scraper that doesn't write a database would crash the current CLI. It now goes to stderr.Fixes 2 and 3 are implemented with regression specs and verified end-to-end against production (the CLI now exits 1 with "The server at https://morph.io didn't send anything back. Something has probably gone wrong on the Morph server." instead of the misleading success message). PR to follow.