Robot Demonstration Data for Research: Synchronization, Labels, Resets, and Reproducibility

A robot that can be teleoperated is not yet a research platform. The machine moves when an operator moves it, which is the easy part. The hard part is capturing what happened in a form another researcher can replay, understand, and compare. Demonstration data that is missing its time alignment, its task labels, or its failed attempts is data that cannot be reused, and a pipeline built on it will not survive a change of operator, room, or robot. The requirements below are about making the data reproducible, not about making the robot capable.

Define the Research Task and Data Use

The data requirements follow from what the data is for, and different uses need different records.

Behavior cloning trains a policy to imitate a demonstrated action, so it needs the observations and the actions, aligned in time. Policy evaluation tests a trained policy, which needs consistent starting conditions and a way to score success. A recording study documents what happened for later analysis, which needs a complete and honest record, including the failures. These uses overlap, but they emphasize different things: imitation cares about the action signal, evaluation cares about the initial conditions, and recording cares about completeness.

Write down the intended use before designing the collection. A dataset for imitation learning has to capture the operator's action, not just the robot's final motion, because the learning target is the action that produced the motion. A dataset for evaluation has to capture the initial state precisely, because the same policy can score differently from a different start. Do not treat the volume of data as a proxy for its usefulness: a hundred unlabeled, unsynchronized episodes are worth less than ten that are fully described. The use case decides the fields.

Specify Sensors, Actions, and Time Alignment

The core of a demonstration dataset is the temporal alignment of what the robot sensed and what it was told to do.

List the sensors: cameras and their positions, joint encoders, any force or tactile sensing, and any tool state. List the action signals: joint commands, end-effector targets, gripper commands, and any operator input. Then specify how they are synchronized in time. Cameras run at one rate, joint data at another, and a task instruction may be a single event. If these are not aligned to a common clock, a later analysis will not know whether a camera frame shows the moment before or after an action.

To formalize supplier-accessible state and command interfaces, use the robot interface control document guide; this article defines the research dataset built from those signals.

LeRobotDataset v3.0 is a documented robotics-learning dataset format, not a mandatory industry standard. Its structure separates metadata, chunked Parquet data, and video shards while associating records with episodes and tasks; adopt its exact version and field conventions only if they fit the intended pipeline. It is a useful reference for what "synchronized" means in practice, and you should confirm the schema you adopt against your own toolchain.

Record Teleoperation and Human Assistance

The actions a person takes during a demonstration are part of the data, and leaving them out loses the very thing that makes the data useful.

In teleoperation, a human drives the robot, so the robot commands the human issues are part of the action signal the policy is meant to learn. For imitation learning, specify the action representation the policy will learn, such as robot joint commands, end-effector targets, or teleoperation commands, and align it with the observations. Recording only the final end pose omits the action sequence needed to learn the behavior. Beyond the primary teleoperation, there are interventions: the operator resets the scene, fixes a dropped object, or assists mid-task. Each of these is a human action, and it changes what the episode contains.

Record who did what. Note when the operator intervened, what they did, and whether the episode was a clean demonstration or an assisted one. Projects such as ALOHA and ACT illustrate how demonstration recording, robot action representation, and downstream policy training are connected. Their published task results belong to the tested tasks and setups, so your dataset record should describe interventions and resets independently of any headline success rate. A dataset that silently mixes clean and assisted episodes, without labeling which is which, cannot be used to measure how much of the behavior came from the policy and how much from the person. Label the human involvement; do not hide it.

Preserve Task Labels and Initial Conditions

What makes one episode comparable to another is that the task and the starting conditions are recorded, not assumed.

A task label says what the demonstration was meant to accomplish, and the success definition says how that was judged. The initial conditions describe the world at the start: where the objects were, how they were arranged, the lighting, and anything else that could affect the outcome. Without these, two episodes that look similar may not be comparable, and a policy trained on them may learn a correlation that does not hold.

Record the initial conditions systematically, and define success unambiguously. For evaluation, the initial conditions define the test, so a policy that scores well from one arrangement may fail from another; the record is what makes the score meaningful. NIST's robotic assembly and manipulation test efforts illustrate why defined tasks, repeatable starting conditions, and transparent outcome measures matter. Use them as a methodological reference rather than claiming that a particular task-board count or assembly-factor list is mandatory for your research dataset. For your evaluation, repeat the same task under recorded conditions and report variation alongside success counts; one successful episode is not a performance distribution.

Review Access, Consent, and Storage Limits

Data that includes people, images, or identifiable spaces carries obligations, and the collection design has to respect them.

A camera in a working room may capture faces, badges, screens, or other people. A teleoperation setup may record an operator's hands and voice. These are personal data, and their collection, storage, and sharing are governed by the rules that apply to your institution, your participants, and your jurisdiction. The design decision is not whether to record, but what to record, who may access it, how long it is kept, and how it is de-identified.

Plan this before collection, not after. Decide what the camera sees and whether it can be framed to avoid unnecessary personal information. Decide who holds the data and under what access control. Decide the retention period and the deletion rule. Where the data leaves the institution, confirm the terms under which it may be shared. The point is to avoid capturing and storing identity information that the research does not need, and to handle what is captured according to the applicable rules. This is a data-governance task, and it belongs in the design alongside the sensors and the labels.

The robot software licensing and data rights guide covers access and permitted reuse questions that must be resolved before sharing captured demonstrations.

Build a Small Reproducibility Trial

The proof of a data pipeline is that someone else can reproduce a small piece of it, so run a trial that checks the pipeline end to end.

Pick a small task, collect a few episodes with the full field set, and then try to replay and inspect them as a different person would. Does the replay align in time? Are the labels and initial conditions complete? Can a second person understand what each episode contains without asking the first? The trial is not about training a good model, and it does not promise any training result; it is about whether the data is usable.

Record the trial: the fields collected, the synchronization method, the labels, the initial conditions, and the problems found. Keep the failed and assisted episodes in the record, because a pipeline that only works for clean demonstrations has not been tested against the case it will meet in practice. Where a format such as LeRobotDataset is used, note it as the community format it is, and confirm it fits your toolchain. The deliverable of the trial is a documented pipeline and the evidence that it reproduces, not a claim about model performance.

Before buying a research platform, prove that one demonstration can be understood and replayed. Share your required sensor streams, action representation, clock strategy, episode metadata and intervention labels. AIsirRobot can coordinate manufacturer SDK and export-format questions through integration planning and coordination so your research team can identify what the platform exposes and what the dataset pipeline must supply separately.