A question about the code

[Victor-818]

In Reconstruction/gsfixer/cogvideo/inference.py, lines 105–118, control_pixel_values is normalized to [-1, 1] via (frames - 0.5) * 2. However, first_image and last_image are then converted by (image * 255).astype(np.uint8) directly in the [-1, 1] range without mapping back to [0, 1] first. This appears to produce incorrect pixel values for the VGGT and DINO inputs.

Here is the relevant code snippet:

control_pixel_values = (frames - 0.5) * 2
control_pixel_values = control_pixel_values.permute(1, 0, 2, 3).unsqueeze(0)
ref_first_last_pixel_values = torch.cat([control_pixel_values[:, :, 0, :, :].unsqueeze(2), control_pixel_values[:, :, -1, :, :].unsqueeze(2)], dim=2)

ref_first_last_image_path = []  # for vggt
first_image = control_pixel_values[:, :, 0, :, :].squeeze(0).cpu().clone().permute(1, 2, 0).numpy()
first_image = (first_image * 255).astype(np.uint8)
first_image = Image.fromarray(first_image)
ref_first_last_image_path.append(first_image)
last_image = control_pixel_values[:, :, -1, :, :].squeeze(0).cpu().clone().permute(1, 2, 0).numpy()
last_image = (last_image * 255).astype(np.uint8)
last_image = Image.fromarray(last_image)
ref_first_last_image_path.append(last_image)
vggt_images = load_and_preprocess_images_(ref_first_last_image_path).to(self.opts.device)

dino_latents = self.image_encoder(vggt_images).last_hidden_state[:, 5:, :].to(self.opts.weight_dtype)
output_list, patch_start_idx = self.vggt.aggregator.forward(vggt_images.unsqueeze(0))
vggt_latents = output_list[-1][:, :, patch_start_idx:, :].squeeze(0).to(self.opts.weight_dtype)

The correct inverse transform should be ((image + 1) / 2 * 255) or equivalent. Could you please take a look and help clarify whether this is indeed a bug? Thanks!

[Victor-818]

I have another question: is the COLMAP point cloud derived from training views only, or from all available views?

In FSGS, the dense MVS point cloud used for initialization is reconstructed only from the training views. I've read the dataset parser in Reconstruction/datasets/colmap_debug.py, and it seems to directly load the COLMAP result from the data path — I couldn't find any additional initialization step beyond that.

I'm wondering if you followed the same approach as FSGS here, or whether the COLMAP result includes all views (including test/novel views).

Thanks for clarifying!

[flow0314]

Hi Bee-droid818,
For the first question: There may not be a bug. However, my implementation of this part of the code was not elegant enough at the time and can be improved.
For the second question: We use the COLMAP point cloud from training views only to ensure a fair evaluation of sparse-view reconstruction. You can take a look at colmap_debug.py: we explicitly handle the sparsity constraints within the ‘Dataset’ class by utilizing the ‘filter_points_sparse’ function (defined in line 354) to filter points based on visibility.

[Victor-818]

Hi Bee-droid818, For the first question: There may not be a bug. However, my implementation of this part of the code was not elegant enough at the time and can be improved. For the second question: We use the COLMAP point cloud from training views only to ensure a fair evaluation of sparse-view reconstruction. You can take a look at colmap_debug.py: we explicitly handle the sparsity constraints within the ‘Dataset’ class by utilizing the ‘filter_points_sparse’ function (defined in line 354) to filter points based on visibility.

Thank you for the clarification!