Skip to content

Cluster upgrade fails when multiple topology hosts resolve to same physical server #2730

Description

@zhenemola

Bug Report

Please answer these questions before submitting your issue. Thanks!

  1. What did you do?
    A TiDB cluster with the following topology:
  • 3 physical servers with IPs in two subnets: 172.xx.xx.n and 192.168.0.n
  • tiproxy_servers using 192.168.0.n addresses
  • All other components (PD, TiKV, TiDB, CDC, monitoring) using 172.xx.xx.n addresses
  • The 192.168.0.n and 172.xx.xx.n addresses resolve to the same physical machines (e.g., 192.168.0.212 = 172.xx.xx.212)

The monitored components (node_exporter, blackbox_exporter) are configured with a shared deploy_dir: , which is identical across all hosts in the topology.

tiup cluster upgrade <cluster_name> v7.1.9-0.2

  1. What did you expect to see?

Cluster upgrade to complete successfully without errors.

  1. What did you see instead?

Upgrade fails with error:

cannot remove '/tdata1/tidb-deploy/monitored-9100/bin/node_exporter-v1.5.0-d529d2a-linux-amd64.tar.gz',no such file or directory

seems like : TiUP treats the two IP addresses (172.xx.xx.n and 192.168.0.n) as separate hosts. When upgrading, two concurrent tasks deploy node_exporter to the same physical path on the same machine. The first task's command deletes the tar.gz, then the second task's fails with "no such file".

workaround: specify -c 1 to run upgrades serially and bypass this issue.

  1. What version of TiUP are you using (tiup --version)?

v1.16.4

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions