Discrepancy Between State and Action in Dataset (i+1’th state != i’th action)

[Leesangoh]

Hi, thank you for your great work on this project!

I’ve been reviewing your datasets and noticed a potential inconsistency:
the (i+1)’th state is not equal to the i’th action.

From what I can tell, the action seems to represent the absolute coordinate of the robot.
If that’s the case, I would expect the (i+1)’th state to match the i’th action.

For example, in the bridge and fractal datasets, this relationship holds true — although the action is a delta value, adding the i’th action to the i’th state exactly yields the (i+1)’th state. However, VLABench appear not to follow this rule.

In my opinion, this discrepancy may cause instability when running robots on VLABench, leading to warnings such as:

WARNING:absl:Failed to converge after 99 steps: err_norm=0.104117

Could you please clarify whether this behavior is intentional, or if there might be an issue with the dataset?

[Shiduo-zh]

Hi, that’s a great question! As you mentioned, the common way to represent state-action transitions is indeed through a frame-to-frame relationship, i.e., action[i] = state[i+1] or state[i] + action[i] = state[i+1].

In our current dataset, the difference comes from the fact that the action is recorded as the target end-effector position from the planner, while the state reflects the actual position reached by the robot. I checked this discrepancy using several replay formulations, including:
• state[1:] - state[:-1]
• action - state
• action[1:] - action[:-1]

From actual trajectory replay results, I found that both action - state and state[1:] - state[:-1] yield almost negligible differences. So this mismatch does not materially affect the experimental outcomes.

However, this issue does exist, and I agree it is better to store actions in a more standard form. In future updates, I will adjust the action recording format to align directly with state transitions.

Thanks a lot for pointing this out — really appreciate the thoughtful feedback!