parser.py doesn't work without Nvidia / CUDA

[JuKu]

Hi together,

I do a research project about MPI / OpenMPI 5.0.8 (with Peruse) and we try to trace things with ucTrace.
I used a slurm system without Nvidia / CUDA with 8 Nodes and ran LULESH 2.0 LULESH 2.0 with it:

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --tasks-per-node=8
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=2G
#SBATCH --output=lulesh-%j.out
#SBATCH --error=lulesh-%j.err
export LD_PRELOAD=/home/<USERNAME>/git/so-files-lib/preload.so
# This folder contains the built .so files from ucTrace folder
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/<USERNAME>/git/so-files-lib/
srun /home/<USERNAME>/git/LULESH/lulesh2.0

Everything worked and I got the 8 .asd.gz files.

After that, I tried to execute parse.py to parse the log files:

python3 parse.py my-outputs/ -p my_output.pkl -n 0

Badly, I got an error while the script was parsing the COMMS:

Traceback (most recent call last):
  File "/home/<USERNAME>/ucTrace/parser/parse.py", line 2017, in <module>
    main()
    ~~~~^^
  File "/home/<USERNAME>/ucTrace/parser/parse.py", line 1966, in main
    all_data, parsed_comms = parse_folder(args.folder_path, args)
                             ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/<USERNAME>/ucTrace/parser/parse.py", line 1793, in parse_folder
    match_ucp_send_recv(parsed_comms, all_data)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/<USERNAME>/ucTrace/parser/parse.py", line 1253, in match_ucp_send_recv
    recvs = tmp_recvs[target_pid]
            ~~~~~~~~~^^^^^^^^^^^^
KeyError: "Error decoding hex string-- 'ascii' codec can't decode byte 0xf0 in position 8"

How can I fix this?
Thanks in advance!

[JuKu]

I tried it again with 8 Nodes and 1 Task per CPU:

#!/bin/bash
#SBATCH --nodes=8
#SBATCH --tasks-per-node=1
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=2G
#SBATCH --output=lulesh-%j.out
#SBATCH --error=lulesh-%j.err
export LD_PRELOAD=/home/<USERNAME>/git/so-files-lib/preload.so
# This folder contains the built .so files from ucTrace folder
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/<USERNAME>/git/so-files-lib/
srun /home/<USERNAME>/git/LULESH/lulesh2.0

Now I got a different error message:

[...]
Processed 7/8 files (87.50%)
optimized file n1494.<HOSTNAME>--3111259.log.asd.gz
<HOSTNAME>--3111259.log.asd.gz -- ifaces
<HOSTNAME>--3111259.log.asd.gz -- eps
<HOSTNAME>--3111259.log.asd.gz -- comms
<HOSTNAME>--3111259.log.asd.gz -- connections
<HOSTNAME>--3111259.log.asd.gz -- mpi rank
<HOSTNAME>--3111259.log.asd.gz -- ucp sends
<HOSTNAME>--3111259.log.asd.gz -- ucp recvs
<HOSTNAME>--3111259.log.asd.gz converting ifaces
Processed 8/8 files (100.00%)
PARSING ALL COMMS
Traceback (most recent call last):
  File "/home/<USERNAME>/ucTrace/parser/parse_backup.py", line 2017, in <module>
    main()
    ~~~~^^
  File "/home/<USERNAME>/ucTrace/parser/parse_backup.py", line 1966, in main
    all_data, parsed_comms = parse_folder(args.folder_path, args)
                             ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/<USERNAME>/ucTrace/parser/parse_backup.py", line 1763, in parse_folder
    parse_all_comms(all_data, parsed_comms)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/<USERNAME>/ucTrace/parser/parse_backup.py", line 855, in parse_all_comms
    len(target_pid) > 0
AssertionError: target pid 0, tl: rc_mlx5, comm: Comm(time=Time(sec=1782903061, nsec=522987813), time_took=5112, time_start=Time(sec=1782903061, nsec=522987813), time_end=Time(sec=1782903061, nsec=522992925), success=True, size=248, func='uct_ep_am_short', rkey='null', extra={'call_stack': '{\n  "call_stack": [\n    "/home/<USERNAME>/git/so-files-lib/preload.so(snoop_uct_send_proxy+0xe2) [0x7ffff7fb9072]",\n    "/home/<USERNAME>/git/build/lib/libucp.so.0(+0x95e98) [0x7ffff76b6e98]",\n    "/home/<USERNAME>/git/build/lib/libucp.so.0(ucp_wireup_replay_pending_requests+0x96) [0x7ffff76d2956]",\n    "/home/<USERNAME>/git/build/lib/libucp.so.0(ucp_wireup_eps_progress+0x143) [0x7ffff76d2b13]",\n    "/home/<USERNAME>/git/build/lib/libucs.so.0(+0x22db7) [0x7ffff73bcdb7]",\n    "/home/<USERNAME>/git/build/lib/libucp.so.0(ucp_worker_progress+0x2a) [0x7ffff766efea]",\n    "/home/<USERNAME>/git/ompi/build-openmpi/lib/libopen-pal.so.80(opal_progress+0x34) [0x7ffff772a634]",\n    "/home/<USERNAME>/git/ompi/build-openmpi/lib/libmpi.so.40(ompi_request_default_wait_all+0x255) [0x7ffff7cb8c55]",\n    "/home/<USERNAME>/git/ompi/build-openmpi/lib/libmpi.so.40(PMPI_Waitall+0x8f) [0x7ffff7d01b9f]",\n    "/home/<USERNAME>/git/LULESH/lulesh2.0() [0x40fe64]",\n    "/home/<USERNAME>/git/LULESH/lulesh2.0() [0x402b3d]",\n    "/lib64/libc.so.6(+0x295d0) [0x7ffff74295d0]",\n    "/lib64/libc.so.6(__libc_start_main+0x80) [0x7ffff7429680]",\n    "/home/<USERNAME>/git/LULESH/lulesh2.0() [0x404f85]"\n  ]\n}', 'copy_buffer': '0x91e770', 'extra_info': 'SNOOP_UCT_COMM_EXTRA_AMINFO:2'}, ep_ptr='9919312', name='rc_mlx5', if_ptr=7024832, local_ptr={'device_id': -1, 'device_type': 'Host', 'ptr': 9561968}, target_ptr=None)

[JuKu]

If I replaced line 852 in parse.py:

# if len(target_pid) == 0:
            #     target_pid = "?"
            assert (
                len(target_pid) > 0
            ), f"target pid 0, tl: {find_comm_iface(comm, pid, all_data).tl_name}, comm: {comm}"

with:

if len(target_pid) == 0:
                target_pid = "?"

then this worked. But then I get another error message.

[emrgncr]

Hello, can you please contact me at egencer@umd.edu and attach the log files you obtained? I will try to reproduce the issue.

[JuKu]

Oh, I completely oversaw your answer!
Of course I will do!
Thank you in advance!

[JuKu]

After recompiling everything (including UCX, OpenMPI and so on), I get a new error message from the parse.py script:

[...]
optimized file n1234.<censored>.hpc.<censored>--2214119.log.asd.gz
n1234.<censored>.hpc.<censored>--2214119.log.asd.gz -- ifaces
n1234.<censored>.hpc.<censored>--2214119.log.asd.gz -- eps
n1234.<censored>.hpc.<censored>--2214119.log.asd.gz -- comms
n1234.<censored>.hpc.<censored>--2214119.log.asd.gz -- connections
n1234.<censored>.hpc.<censored>--2214119.log.asd.gz -- mpi rank
n1234.<censored>.hpc.<censored>--2214119.log.asd.gz -- ucp sends
n1234.<censored>.hpc.<censored>--2214119.log.asd.gz -- ucp recvs
n1234.<censored>.hpc.<censored>--2214119.log.asd.gz converting ifaces
Processed 6/6 files (100.00%)
PARSING ALL COMMS
Traceback (most recent call last):
  File "/home/<username>/ucTrace/parser/parse.py", line 2017, in <module>
    main()
    ~~~~^^
  File "/home/<username>/ucTrace/parser/parse.py", line 1966, in main
    all_data, parsed_comms = parse_folder(args.folder_path, args)
                             ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/<username>/ucTrace/parser/parse.py", line 1763, in parse_folder
    parse_all_comms(all_data, parsed_comms)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/<username>/ucTrace/parser/parse.py", line 784, in parse_all_comms
    assert len({i[0] for i in b}) == 1, b
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: []