I'm curious if the full dataset must be fully memory resident for the training pipeline to work, is that how this works?
If so, that's really unfortunate because i'd like to continue to scale the data up.
Training CAMELTrack on a custom dataset with about 180 videos, resulting in a roughly 6GB pklz training state file.
When i try to train CAMELTrack with this, the training job dies pretty early in training, with "Killed" message, which i believe is the system running out of system/CPU memory. I have 64GB system RAM.
I ended up having to shrink my dataset by thresholding detections which allowed me to train.
Andrew
I'm curious if the full dataset must be fully memory resident for the training pipeline to work, is that how this works?
If so, that's really unfortunate because i'd like to continue to scale the data up.
Training CAMELTrack on a custom dataset with about 180 videos, resulting in a roughly 6GB pklz training state file.
When i try to train CAMELTrack with this, the training job dies pretty early in training, with "Killed" message, which i believe is the system running out of system/CPU memory. I have 64GB system RAM.
I ended up having to shrink my dataset by thresholding detections which allowed me to train.
Andrew