Guide · Robot learning
How to build a LeRobot dataset from your robot logs
A LeRobot dataset is how you feed robot demonstrations into training. Here is what one contains, how to build it, and how to source the best episodes from the fleet logs you already have — not just fresh teleoperation.
What a LeRobot dataset is
LeRobot is Hugging Face's open-source library for real-world robot learning. Its dataset format — the LeRobot dataset — is a standard way to store episodes of a robot performing a task, so that policies like ACT and diffusion policies can be trained on them. Camera frames are stored as video, tabular data as Parquet, and datasets are commonly shared on the Hugging Face Hub.
What is inside one
A dataset is a set of episodes; an episode is a sequence of frames. Each frame usually holds:
observation.images.*— one or more camera views for that moment.observation.state— the state vector, e.g. joint positions.action— the action taken at that frame.timestamp,frame_index,episode_index— bookkeeping.
Dataset-level metadata records the frame rate, the feature schema, and the episode count. The diversity and quality of the episodes decides how well a policy learns — a sharp, varied set beats a huge, repetitive one.
How to build one
The standard path is four steps:
- Collect demonstrations of the task, often by teleoperating the robot.
- Record synchronized frames, states, and actions for each episode.
- Write them into the LeRobot dataset format (and optionally push to the Hub).
- Train a policy on the dataset.
The tooling is rarely the hard part. The bottleneck is getting enough good, varied episodes of the right task — which is where your existing logs come in.
Source episodes from logs you already have
If your fleet already records logs — for example as MCAP files — the demonstrations and informative failures you want are already in there. Collecting everything fresh is slower than mining what you have. The catch is finding the right moments across a whole fleet: thousands of files, petabytes of data, and the good episodes scattered through them.
That is what Coldstack does. You point it at your bucket of logs; it builds a compact index on object storage; then you search the whole fleet in one query — by metadata, by signal ("a force spike during a grasp"), and by meaning ("the gripper slipping on a transparent object") — and export the matches straight to a LeRobot-compatible dataset. Raw logs never leave your bucket.
Questions
What is a LeRobot dataset?
A LeRobot dataset is a standardized format for robot-learning data, used by Hugging Face's open-source LeRobot library. It stores episodes — short recordings of a robot performing a task — where each frame carries the observations (camera images and state) and the action taken, at a fixed frame rate. Camera frames are stored as video and the tabular data as Parquet, and datasets are commonly shared on the Hugging Face Hub. Policies like ACT and diffusion policies are trained on this format.
What is inside a LeRobot dataset?
Each episode is a sequence of frames. A frame typically holds one or more camera images (observation.images.*), a state vector (observation.state, e.g. joint positions), the action taken, and bookkeeping fields like timestamp, frame_index, and episode_index. Dataset-level metadata records the frame rate, the feature schema, and the number of episodes. The quality and diversity of the episodes matters more than the raw count.
How do you build a LeRobot dataset?
The usual path is: collect demonstrations of the task (often by teleoperating the robot), record synchronized frames, states, and actions for each episode, write them into the LeRobot dataset format, and optionally push the dataset to the Hugging Face Hub. You then train a policy on it. The bottleneck is rarely the tooling — it is getting enough good, varied episodes of the right task.
Can you build a LeRobot dataset from existing fleet logs?
Yes, and it is often faster than collecting everything fresh. If your fleet already records logs (for example as MCAP files), the good demonstrations and informative failures are already in there — you just have to find them. The hard part is searching a whole fleet's logs for the right moments, then exporting those episodes into the LeRobot format.
How do you find the right episodes across a whole fleet?
You search the logs three ways at once: by metadata (which robot, when, which task), by time-series signal (a force spike, a joint limit), and by semantic image or text match (what the scene looked like). Coldstack indexes a fleet's logs on object storage and lets you run that search in one query, then export the matches straight to a LeRobot-compatible dataset — with the raw logs staying in your own bucket.