Hi, thank you for releasing MultiWorld and the RoboFactory dataset!
I am currently working with the TwoRobotsStackCube data and I have a question about the temporal alignment between the RGB videos and the action arrays.
For example, for one episode I found:
-episodeXXXXXX_action.npy: shape (2, T, 8)
-episodeXXXXXX_head_camera_global_video.mp4: T frames
Could you please clarify the transition convention used when exporting the dataset?
Specifically, does:
action[:, t, :]
represent the action applied from video_frame[t] to obtain video_frame[t+1], i.e.
observation[t] + action[t] → observation[t+1]?
Or is the action aligned with the observation resulting from that action, meaning that a one-frame shift is required for action-conditioned prediction?
Also, were the RGB frames or actions temporally resampled/interpolated during the RoboFactory → MultiWorld dataset export?
Thank you very much!
Hi, thank you for releasing MultiWorld and the RoboFactory dataset!
I am currently working with the TwoRobotsStackCube data and I have a question about the temporal alignment between the RGB videos and the action arrays.
For example, for one episode I found:
-episodeXXXXXX_action.npy: shape (2, T, 8)
-episodeXXXXXX_head_camera_global_video.mp4: T frames
Could you please clarify the transition convention used when exporting the dataset?
Specifically, does:
action[:, t, :]
represent the action applied from video_frame[t] to obtain video_frame[t+1], i.e.
observation[t] + action[t] → observation[t+1]?
Or is the action aligned with the observation resulting from that action, meaning that a one-frame shift is required for action-conditioned prediction?
Also, were the RGB frames or actions temporally resampled/interpolated during the RoboFactory → MultiWorld dataset export?
Thank you very much!