Will you expand the seg_query_token to NUM_OBJECT_QUERIES when preparing input embeddings?

[zhang-yige]

I noticed that a seg_query was defined in advance in the code:

self.seg_query = nn.Parameter(
            torch.zeros([self.mask_decoder_cfg.MODEL.MASK_FORMER.NUM_OBJECT_QUERIES, self.config.hidden_size]))

Then add it to input_embeds when encountering SEG_TOKEN in prepare_inputs_labels_for_multimodal and concat_image_seg_cls_embeds functions:

elif chunk_len == 1 and chunk[0] == SEG_TOKEN_INDEX:
                cur_new_input_embeds.append(seg_query)
                cur_new_seg_query_mask.append(torch.ones(seg_query.shape[0]))

I find that in the dataset, each answer contains only one [SGE] special token, so can I take it that you extended the length of input_id, and it was extended to MASK_FORMER.NUM_OBJECT_QUERIES (which is defined as 100 or 50) tokens at [SGE]?