Cuda out of memory when distributed training using multipy node

[Adnios]

Dear all.

I am trying to training resnet50 in parallel. I am using the MXNet v1.7.0. And I use the code in the directory example/image-classification/train_imagenet.py.

When training resnet50 with 4 nodes, it success. But failed with 6 nodes. Each node is equipped with 4 V100. In our code, we only use 1 GPU per node. The command to training is following.

# using 4 node (1 GPU per node). It success. 32 input images per GPU
python ~/wangjian/mxnet-1.7.0/tools/launch.py --launcher ssh -H 4hosts -n 4 --p3 python ../image-classification/train_imagenet.py --gpus 0 --data-train ~/imagenet/train.rec  --data-val ~/imagenet/val.re    c --network resnet --num-layers 50 --num-epochs 1 --batch-size 128 --kv-store dist_sync


# sing 6 node (1 GPU per node). It failed. 32 input images per GPU
python ~/wangjian/mxnet-1.7.0/tools/launch.py --launcher ssh -H 6hosts -n 6 --p3 python ../image-classification/train_imagenet.py --gpus 0 --data-train ~/imagenet/train.rec  --data-val ~/imagenet/val.    rec --network resnet --num-layers 50 --num-epochs 1 --batch-size 192 --kv-store dist_sync

error message: