Uprading from version 0.6.17 to 0.7.0 requires a rescan?

[got3nks]

I recently upgraded chaind from version 0.6.17 to 0.7.0.

{"level":"info","service":"chaindb","impl":"postgresql","target_version":9,"time":"2023-02-10T00:02:21+01:00","message":"Upgrading database"}
{"level":"info","service":"chaindb","impl":"postgresql","current":1,"total":1,"time":"2023-02-10T00:02:21+01:00","message":"Running upgrade function"}
{"level":"info","service":"chaindb","impl":"postgresql","target_version":10,"time":"2023-02-10T00:02:21+01:00","message":"Upgrading database"}
{"level":"info","service":"chaindb","impl":"postgresql","current":1,"total":2,"time":"2023-02-10T00:02:21+01:00","message":"Running upgrade function"}
{"level":"info","service":"chaindb","impl":"postgresql","current":2,"total":2,"time":"2023-02-10T00:02:21+01:00","message":"Running upgrade function"}
{"level":"info","service":"chaindb","impl":"postgresql","target_version":11,"time":"2023-02-10T00:02:21+01:00","message":"Upgrading database"}
{"level":"info","service":"chaindb","impl":"postgresql","current":1,"total":1,"time":"2023-02-10T00:02:21+01:00","message":"Running upgrade function"}
{"level":"info","service":"chaindb","impl":"postgresql","time":"2023-02-10T00:02:21+01:00","message":"Upgrade complete"}

I checked the db updates 9,10,11 in the source file postgresql/upgrader.go and I see requiresRefetch is not set to true. Nevertheless beaconcommittees is taking very long to catch up. Is it a bug or a rescan is actually needed?

In the PostgreSQL pg_stat_activity I see these queries being executed:

      INSERT INTO t_attestations(f_inclusion_slot
                                ,f_inclusion_block_root
                                ,f_inclusion_index
                                ,f_slot
                                ,f_committee_index
                                ,f_aggregation_bits
                                ,f_aggregation_indices
                                ,f_beacon_block_root
                                ,f_source_epoch
                                ,f_source_root
                                ,f_target_epoch
                                ,f_target_root
                                ,f_canonical
                                ,f_target_correct
                                ,f_head_correct
						  )
      VALUES($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15)
      ON CONFLICT (f_inclusion_slot,f_inclusion_block_root,f_inclusion_index) DO
      UPDATE ....

[mcdee]

There should be no rescanning, inserting attestations is part of updating the blocks.

Best thing to do is to look at two snapshots of t_metadata 7 minutes apart and seeing what is changing. Over that time period you would expect most items to have updated.

[got3nks]

This is what I see in t_metadata, so it basically restarted from 0? Before the upgrade chaind was fully sync'd with mainnet. The chaind upgrade started more than 12 hours ago.

"schema"	"{""version"": 11}"
"summarizer.standard"	"{""latest_epoch"": 0, ""last_validator_day"": -1, ""latest_block_epoch"": 180099, ""latest_validator_epoch"": 0, ""periodic_validator_rollups"": true}"
"blocks.standard"	"{""latest_slot"": 5763327}"
"proposerduties.standard"	"{""latest_epoch"": 180229}"
"validators.standard"	"{""latest_epoch"": 180228, ""latest_balances_epoch"": 0}"
"beaconcommittees.standard"	"{""latest_epoch"": 180229}"
"finalizer.standard"	"{""latest_epoch"": 0, ""latest_canonical_slot"": 5763232}"
"synccommittees.standard"	"{""latest_period"": 704}"

Also the DB space is growing a lot, will the space be reclaimed at the end of the upgrade process, or the new DB schema is storing more data? It's already grown by 400 GB!

[got3nks]

After 7 minutes only proposerduties, beaconcommittees and validators is moving.

"schema"	"{""version"": 11}"
"summarizer.standard"	"{""latest_epoch"": 0, ""last_validator_day"": -1, ""latest_block_epoch"": 180099,""latest_validator_epoch"": 0, ""periodic_validator_rollups"": true}"
"blocks.standard"	"{""latest_slot"": 5763327}"
"proposerduties.standard"	"{""latest_epoch"": 180231}"
"validators.standard"	"{""latest_epoch"": 180231, ""latest_balances_epoch"": 0}"
"beaconcommittees.standard"	"{""latest_epoch"": 180231}"
"finalizer.standard"	"{""latest_epoch"": 0, ""latest_canonical_slot"": 5763232}"
"synccommittees.standard"	"{""latest_period"": 704}"

[mcdee]

What does SELECT MAX(f_epoch) FROM t_epoch_summaries return?

[got3nks]

It returns null. But I'm not using summaries. This is my chaind.yml.

# log-level is the base log level of the process.
# 'info' should be a suitable log level, unless detailed information is
# required in which case 'debug' or 'trace' can be used.
log-level: debug
# log-file specifies that log output should go to a file.  If this is not
# present log output will be to stderr.
log-file: /home/chaind/chaind.log
chaindb:
  # url is the URL of the PostgreSQL database.
  url: postgres://x:x@localhost:5432
  max-connections: 16
# eth2client contains configuration for the Ethereum 2 client.
eth2client:
  # log-level is the log level of the specific module.  If not present the base log
  # level will be used.
  log-level: debug
  # address is the address of the beacon node.
  address: localhost:5052
# eth1client contains configuration for the Ethereum 1 client.
eth1client:
  # address is the address of the Ethereum 1 node.
  address: localhost:8545
# blocks contains configuration for obtaining block-related information.
blocks:
  # enable states if this module will be operational.
  enable: true
  # address is a separate connection for this module.  If not present then
  # chaind will use the eth2client connection.
  address: localhost:5052
  # start-slot is the slot from which to start.  chaind should keep track of this itself,
  # however if you wish to start from a later slot this can be set.
#  start-slot: 4930000
  # refetch will refetch block data from a beacon node even if it has already has a block
  # in its database.
  # refetch: false
# validators contains configuration for obtaining validator-related information.
validators:
  enable: true
  # balances contains configuration for obtaining validator balances.  This is
  # a separate configuration flag for two reasons.  First, it can take a long
  # time to retrieve this information.  Second, the information can be
  # derived from the data obtained by the other modules.
  balances:
    enable: false
# beacon-committees contains configuration for obtaining beacon committee-related
# information.
beacon-committees:
  enable: true
# proposer-duties contains configuration for obtaining proposer duty-related
# information.
proposer-duties:
  enable: true
# finalizer updates tables with information available for finalized states.
finalizer:
  enable: true
# eth1deposits contains information about transactions made to the deposit contract
# on the Ethereum 1 network.
eth1deposits:
  enable: false
  # start-block is the block from which to start fetching deposits.  chaind should
  # keep track of this itself, however if you wish to start from a different block this
  # can be set.
  # start-block: 500

[mcdee]

If you aren't summarizing then that all looks up-to-date to me. What exactly do you think is still catching up?

[mcdee]

(I suspect that the finalizer may be taking a while, but once that is finished you should see blocks increase again.)

[got3nks]

If you aren't summarizing then that all looks up-to-date to me. What exactly do you think is still catching up?

Disk space growing, disk I/O at 100%, blocks not increasing. It's been 14 hours now.

In the new version are summaries disabled by default or I should add any line in the chaind.yml configuration file?

[mcdee]

Summaries are not disabled by default but if you don't gather validator balances they should not kick in. But if you want to be sure you can add:

summarizer:
  enable: false

to your chaind configuration file.

It's possible that some sort of summarizing is going on, you can check the relevant tables:

SELECT COUNT(*) FROM t_epoch_summaries;
SELECT COUNT(*) FROM t_validator_epoch_summaries;
SELECT COUNT(*) FROM t_block_summaries;

If these are all 0 then it means the summarizer is not running. If it is then you should set the configuration above and restart chaind.

If not, then my guess is that the finalizer may still be running, if you're seeing lots of attestations being updated for older blocks that would be indicative of this being what is going on.

[got3nks]

Only t_block_summaries returned a non-zero value (692535), but I see it's not taking much disk space so I'm fine with that.

Finally chaind is fully sync'd again. It took around 17 hours. I don't understand why attestations (finalizer) have been rescanned for old blocks after the upgrade but it's ok.

I also see postgresql autovacuum is running now, so I hope the extra space will be reclaimed. The DB grew by over 550GB after the upgrade.