Last Updated: April 23, 2026

Data Collection for Embodied AI: Teleoperation, Sim-to-Real and Industrial Datasets in 2026

Data is the bottleneck for embodied AI, not compute. Training a robot policy requires synchronized action-state streams that simply do not exist at internet scale. Three sources currently feed that demand. Human teleoperation yields high-quality demonstrations at the cost of operator time. Sim-to-real transfer scales episode count but introduces a fidelity gap. Large open datasets pool contributions from dozens of labs and platforms.

Why Data Is the Embodied AI Bottleneck

Compute has followed a predictable scaling curve. GPU clusters capable of training billion-parameter vision-language-action (VLA) models are commercially accessible today. What has not scaled is the data those models need: paired observations and actions recorded during physical robot manipulation, with enough task and embodiment diversity to produce policies that generalize.

A language model can ingest a trillion tokens from the web in weeks. A robot manipulation dataset has to be generated episode by episode, each one requiring a physical robot, a real or simulated environment, a task definition, and either a human operator or a scripted demonstration. The International Federation of Robotics (IFR) estimates over 3.9 million industrial robots were operating globally in 2024, yet the largest open manipulation datasets contain on the order of one million episodes. The gap is not closing fast.

The challenge compounds when you consider the embodiment gap: a policy trained on a 6-DoF single-arm manipulator does not transfer cleanly to a bimanual humanoid or a mobile base with an arm. Every new robot form factor effectively resets data requirements. Action chunking approaches (where a policy outputs multi-step action sequences rather than single steps) reduce inference latency but do not reduce the need for diverse training coverage.

According to the Open X-Embodiment paper (Padalkar et al., 2023, Google DeepMind et al.), cross-embodiment training on diverse robot data can improve policy generalization beyond what single-embodiment datasets provide. EVST addresses this by evaluating cross-embodiment dataset quality as part of its internal assessment process when selecting AI-augmented motion planning for its full-range 3–800 kg robot lineup.

The Three Main Data Sources: Strengths and Limits

Teleoperation

Teleoperation captures expert-quality demonstrations on the target robot hardware. The operator controls the robot remotely, through leader-follower arms, VR headsets, exoskeleton suits, or handheld interfaces, while the system records every joint angle, gripper state, force reading, and camera frame in synchrony. Because the data comes from a human performing the actual task on the actual robot, the action-state correspondence is exact and the embodiment gap is zero.

The limitation is throughput. Skilled teleoperators produce roughly 5–50 episodes per hour depending on task complexity, and data quality degrades when operators fatigue. Teleoperation also does not scale to failure recovery without deliberate protocol design: if operators always succeed, the policy never learns to recover from perturbations.

Sim-to-Real Transfer

Physics simulators (MuJoCo, NVIDIA Isaac Sim, Isaac Lab, PyBullet) let researchers generate millions of robot episodes at near-zero marginal cost. A single GPU cluster can run thousands of parallel simulation instances overnight. The cost shows up at deployment: policies trained in simulation often fail when faced with real-world contact dynamics, sensor noise, and visual variation that the simulator did not replicate accurately. Closing this sim-to-real gap is an active research area.

Internet Video and Passive Observation

Large vision-language models pre-trained on internet video bring general scene understanding that helps with task grounding. Physical Intelligence’s pi0 architecture and similar VLA designs use video pre-training to bootstrap manipulation skills before fine-tuning on robot demonstrations. The limit is that video provides no action labels: a video shows a hand grasping an object but does not record the force applied or the joint angles of an equivalent robot arm. Inverse dynamics models can partially infer actions from video, but the resulting pseudo-labels carry noise.

Teleoperation Hardware and Protocols

The choice of teleoperation interface shapes both data quality and collection throughput. Below are the systems most active in 2025–2026 research and commercial programs.

Leader-Follower Arms: ALOHA and Mobile ALOHA

ALOHA (A Low-cost Open-source Hardware System for Bimanual Teleoperation) was developed at Stanford and uses two inexpensive leader arms that the operator moves by hand, while two matched follower arms on the robot replicate the motion in real time. The bimanual design captures two-handed tasks that single-arm systems cannot. Mobile ALOHA extends the platform onto a mobile base. ALOHA 2, released in 2024, improved joint tracking fidelity and operator ergonomics. Hardware cost is sub-$20,000 for the leader-follower pair, which has driven adoption across dozens of academic labs.

Wrist-Mounted Interfaces: UMI

The Universal Manipulation Interface (UMI), from Stanford and Columbia, mounts a fisheye camera and encoder on a handheld gripper. The operator physically demonstrates the task in the environment while the gripper records end-effector pose and grasp state. UMI decouples data collection from the robot: demonstrations can be collected in the field and later retargeted to different robot embodiments. This makes it particularly useful for collecting in-the-wild manipulation data outside a lab setting.

VR and Head-Mounted Displays

Apple Vision Pro and Meta Quest 3 are both being explored as teleoperation interfaces. The appeal is bimanual six-DoF hand tracking with intuitive operator control. The engineering challenge is retargeting: human hand kinematics (five fingers, 27 joints) must map to robot end-effector commands (typically 1-DoF grippers or dexterous hands with 12–16 DoF). Without careful retargeting, the recorded demonstrations contain operator intent but not directly executable robot actions.

Exoskeleton and Whole-Body Capture

For humanoid robots that need full-body motion data (including locomotion combined with manipulation), exoskeleton suits capture whole-body kinematics at sampling rates above 100 Hz. AgiBot uses an exoskeleton-based approach for collecting humanoid manipulation data for its AgiBot World dataset. Fourier GR-1 teleoperation similarly relies on a wearable capture rig matched to the robot’s joint configuration.

Figure and Large-Scale Commercial Fleets

Figure AI operates a proprietary teleop fleet across its BMW manufacturing deployment. The data flywheel model, in which each deployed robot collects new episodes that feed back into training, is also central to Tesla Optimus’s development strategy. These closed-loop systems accumulate industrial data at a rate that open academic datasets cannot match.

Teleoperation Platform Comparison

Note: throughput figures are approximate ranges drawn from published research and researcher accounts. Individual operator skill and task complexity affect results significantly.

Public Datasets: Open X-Embodiment, DROID, BridgeData, RoboSet, RH20T, AgiBot World

The open dataset ecosystem has expanded rapidly since 2022. Below is an overview of the most cited collections as of early 2026.

Open X-Embodiment (OXE) and RT-X

Assembled by Google DeepMind with 33 contributing institutions, OXE aggregates over 1 million episodes across more than 22 robot types. The associated RT-X models demonstrated that training on pooled cross-embodiment data outperforms single-embodiment baselines on generalization benchmarks. OXE is released under a mixture of CC-BY and Apache 2.0 licenses depending on the contributing institution. Access is through the RT-X website and HuggingFace.

DROID

DROID (Distributed Robot Interaction Dataset) was collected across 13 North American research labs using a standardized Franka Panda setup with an ALOHA-style teleoperation interface. It contains approximately 76,000 episodes of diverse household and tabletop manipulation tasks. The standardized hardware across sites makes DROID one of the cleanest large-scale manipulation datasets for behavior cloning and RLHF experiments.

BridgeData V2

BridgeData V2, from UC Berkeley and Stanford, contains approximately 60,000 demonstrations across a wide variety of tabletop manipulation tasks. It uses a WidowX arm and is notable for its task diversity (over 70 distinct task categories), which makes it a common fine-tuning baseline for VLA models. Released under CC-BY 4.0.

RoboSet

RoboSet, from IIT Delhi and collaborators, provides multi-task kitchen manipulation data collected with a Franka Panda. It includes RGB-D observations and language annotations for each episode. RoboSet is designed specifically to support language-conditioned policy evaluation, making it useful for testing instruction-following generalization.

RH20T

RH20T (Robot-Human 20 Tasks) is a large-scale contact-rich manipulation dataset featuring synchronized robot proprioception, RGB video, depth, tactile, and audio streams. The multimodal richness makes RH20T a reference for researchers working on policies that need contact sensing, not just visual observation.

AgiBot World

AgiBot World is a humanoid manipulation dataset produced by AgiBot, a Chinese robotics company, using its exoskeleton-based data collection pipeline. It focuses on full-body humanoid manipulation tasks and represents one of the first large-scale datasets specifically targeting humanoid form factors at industrial scale. AgiBot World is available under an open research license for non-commercial use.

Public Embodied AI Dataset Reference

Episode counts and license terms sourced from dataset papers and official repositories as of early 2026. Verify current terms before use.

Sim-to-Real Stack: Isaac Sim, MuJoCo, Cosmos, and Domain Randomization

Simulation scales data generation. The practical question is which simulator to use and how to close the gap back to physical hardware.

MuJoCo

MuJoCo (Multi-Joint dynamics with Contact) remains the standard for contact-rich manipulation research. Its physics engine handles rigid body dynamics, soft contacts, and tendon-actuated systems with computational efficiency that allows thousands of parallel training episodes on a single GPU cluster. The open-source release under Google DeepMind in 2022 accelerated adoption significantly. Most academic behavior cloning and RLHF benchmarks use MuJoCo environments.

NVIDIA Isaac Sim and Isaac Lab

Isaac Sim runs on NVIDIA’s Omniverse platform and provides GPU-accelerated physics with high-fidelity rendering. Isaac Lab (previously OmniIsaacGymEnvs) is the reinforcement learning framework built on Isaac Sim, optimized for massively parallel policy training. A key advantage is photorealistic rendering: Isaac Sim’s path-traced visuals reduce the visual domain gap compared with MuJoCo’s simplified renderers, which matters for training vision-based policies.

NVIDIA Cosmos

Cosmos is NVIDIA’s world foundation model platform, announced in late 2024. Rather than simulating physics from first principles, Cosmos generates photorealistic video of robot and object interactions from text or image prompts, trained on large video datasets. The promise is generating synthetic training data that looks real enough to close the visual sim-to-real gap without hand-crafted domain randomization. As of early 2026, Cosmos is being explored by several robot learning labs as a data augmentation layer on top of real demonstrations.

Gaussian Splatting and Scene Reconstruction

3D Gaussian Splatting reconstructs photorealistic scenes from a small set of RGB images in minutes rather than hours. For robot training, this means a lab can scan a new work environment, reconstruct a Gaussian scene, and generate novel viewpoints or object configurations for data augmentation, without requiring a physics simulator at all. This approach is being integrated into several sim-to-real pipelines as a low-cost visual domain randomization tool.

Domain Randomization and System Identification

Domain randomization randomizes simulation parameters (object textures, lighting conditions, friction coefficients, robot joint stiffness) during training, forcing the policy to be resilient to variation rather than fitted to a single simulated configuration. System identification takes the complementary approach: it measures the real robot’s physical parameters (mass, inertia, joint compliance) precisely and uses those values in simulation to minimize the gap before randomization.

Curriculum learning applies randomization progressively. A policy first trains in a narrow, clean simulation, then gradually encounters more variation as performance improves. This staged approach often converges faster than applying full randomization from the start.

Data Quality vs. Quantity Trade-offs

More data is not automatically better. The composition of a dataset shapes what a trained policy can and cannot do.

Task diversity matters more than episode count when the goal is a generalist policy. A dataset of 100,000 episodes of the same pick-and-place task produces a specialist. A dataset of 10,000 episodes spread across 50 task types can produce a policy that transfers to new tasks after few-shot fine-tuning.

Embodiment diversity matters for cross-robot transfer. OXE’s value is precisely that it spans arm types, mobile platforms, and dexterous hands. A policy pre-trained on OXE and fine-tuned on a new robot typically outperforms one trained from scratch on the same new robot’s data alone, a result consistent across multiple published studies.

Episode length and failure recovery are underappreciated. Datasets collected by skilled teleoperators who rarely fail produce policies that do not know how to recover from perturbations. Deliberately injecting failure states, and demonstrations of recovery, substantially improves real-world resilience. Some labs use adversarial perturbations during teleoperation sessions to generate this data.

According to industry observations from multiple robot learning lab reports in 2024–2025, the marginal value of additional episodes declines sharply once a dataset covers a task at roughly 500–1,000 demonstrations, while the marginal value of new task types remains high up to thousands of distinct tasks. EVST takes this into account when structuring pilot data capture cells, prioritizing task breadth over repetitive episode accumulation.

Industrial-Grade Data Considerations

Factory data collection imposes requirements that academic teleoperation setups typically do not address.

Deterministic logging is the first requirement. In a production cell, the timestamp on a joint encoder reading must be traceable to a wall-clock reference with sub-millisecond accuracy. EtherCAT 1 kHz fieldbus control, standard on industrial robots (including those in EVST’s lineup), provides the deterministic timing foundation that makes multi-sensor synchronization tractable. A system running at 1 kHz generates 1,000 joint-state samples per second per axis; aligning that stream with camera frames at 30–60 fps and force-torque data at 500 Hz requires a disciplined hardware timestamping architecture, not software buffering.

Safety labels are required when episodes include near-miss events, e-stop activations, or force-limit violations. These labels are necessary both for compliance and for training policies that understand safety-relevant states. Industrial controllers that log e-stop causes and joint-torque exceedances at the hardware level make this labeling tractable.

Cycle-time annotation connects robot episodes to manufacturing outcomes. Knowing that episode 4,712 completed the assembly task in 8.3 seconds with zero quality flags ties the robot learning data to the MES system and allows selection of high-quality demonstrations for training.

MES-linked metadata is the bridge. When a robot’s data logging system writes episode files with MES job IDs, part types, and quality outcomes attached, the resulting dataset can be filtered programmatically. Episodes associated with rework, scrap, or operator override can be excluded or treated as failure demonstrations.

In practice, when retrofitting a pilot cell for data capture, the most common bottleneck is not the robot’s logging capability but the cell’s network architecture. Industrial ethernet segments that were designed for PLC-to-robot communication at sub-kHz rates are not always sized for the bandwidth of continuous multi-camera video plus proprioception streams. A cell running two 1080p cameras at 30 fps plus six-axis force data plus joint states generates roughly 150–300 MB per minute of raw episode data. Planning the storage and network infrastructure before deployment avoids the retrofitting cost later.

Labeling Pipelines and Tooling

Raw episode recordings are not training-ready. They need language annotations, object segmentation masks, and in some cases reward labels.

Language Annotation

Language-conditioned VLA models need a natural-language description of each episode’s task. For small datasets this is done manually; for datasets above 10,000 episodes, automatic annotation using vision-language models (GPT-4V, Gemini, or specialized captioning models) becomes necessary. Quality control requires human review of a sample to catch systematic errors in the auto-annotator.

Object Segmentation and Tracking

Policies that reason about object state benefit from per-episode segmentation masks identifying the manipulated objects. Segment Anything Model (SAM 2) from Meta makes interactive segmentation tractable at dataset scale, reducing annotation time from hours to minutes per episode when combined with a tracking propagation pass.

Reward Shaping for RLHF

When episodes include both success and failure demonstrations, reward labeling enables reinforcement learning from human feedback (RLHF) fine-tuning. Human annotators rank episode outcomes, and those preferences train a reward model that shapes policy updates. This pipeline is used in Physical Intelligence’s pi0 development and is being explored for industrial task training where binary success/failure labels are already available from MES.

Privacy, IP, and Dataset Licensing

Open dataset contributions raise intellectual property questions that industrial OEMs need to resolve before participating.

The main license categories in use are CC-BY (attribution required, commercial use permitted), CC-BY-NC (non-commercial only), and Apache 2.0 (permissive, including commercial use and model weight redistribution). Proprietary datasets (Tesla Optimus, Figure’s fleet data, Boston Dynamics) are not shared externally.

For factory OEMs, the IP concern is process exposure: an episode showing a specific jigging configuration, tool path, or fixturing design may reveal proprietary manufacturing methods. The standard mitigation is to strip visual context to the end-effector level, contribute joint-trajectory streams only (no camera data), or use synthetic renderings of the manipulation trajectory against a generic background. Several emerging consortium models offer legal frameworks for anonymized industrial data contribution under mutual non-disclosure agreements combined with Apache 2.0 data licenses.

According to analysis by Stanford CRFM and collaborators in the 2024 robot learning survey, proprietary data silos represent the single largest structural barrier to embodied AI progress beyond current benchmarks. Industrial OEMs that develop internal data governance frameworks now will be positioned to participate in dataset consortia as those programs formalize over the next 18–24 months.

How Factory OEMs Can Contribute Data

Industrial automation suppliers (including FANUC, ABB, KUKA, and certified turnkey integrators such as EVST) sit at the intersection of robot hardware and real production environments. That position makes them natural contributors to the embodied AI data ecosystem if the operational and legal framework is in place.

Pilot Cells with Teleoperation Overlay

The most accessible entry point is a pilot cell where the production robot is fitted with a teleoperation overlay: a leader-follower arm that an operator uses to demonstrate variations of the production task. These demonstrations are collected at low volume (50–500 episodes) and are sufficient for fine-tuning a pre-trained VLA model onto the specific production task. The pilot cell approach does not require changes to the production cell’s safety certification because the teleoperation system operates as an input device to the robot controller, not as a parallel control path.

Retrofit Logging on Production Robots

Existing production robots can be retrofitted with lightweight logging modules that record joint state, end-effector pose, and gripper force at EtherCAT rates, writing timestamped episode files to an edge server during production runs. This requires no change to the robot’s control program. The logged data captures real production manipulation behavior, which is more task-representative than lab demonstrations. EVST’s industrial controllers, operating at 1 kHz EtherCAT cycle times and certified to CE/SGS/TUV standards, provide the deterministic timing architecture that this logging approach depends on. The data can be cleaned, anonymized, and formatted for standard robot learning frameworks without expensive reintegration work.

Anonymized Dataset Program Participation

Emerging OEM dataset programs (including AgiBot World’s open contribution track and several consortium discussions at ICRA 2025) accept de-identified industrial manipulation data. OEMs with IATF16949 certification, like EVST, have existing quality management processes that translate well to the metadata integrity requirements these programs specify: traceable timestamps, documented collection protocols, and validated sensor calibration records.

Global Field Coverage as a Data Advantage

An OEM with field engineers across 100+ countries can collect embodied AI training data from genuinely diverse manufacturing environments (different workpiece geometries, ambient conditions, operator demographics, and process variations). This geographic diversity is valuable because policies trained on it generalize better than those trained on data from a single lab or facility. EVST’s explosion-proof certified robots and full-range 3–800 kg lineup mean that data collection can extend to environments such as chemical processing, hazardous material handling, and extreme-temperature manufacturing that most academic datasets do not cover at all.

Key Industry Data Points

According to the International Federation of Robotics (IFR) World Robotics 2024 report, over 3.9 million industrial robots were in operation globally in 2023, yet the largest open robot manipulation datasets contain fewer than 2 million episodes combined, illustrating the scale gap between deployed hardware and available training data. EVST addresses this by exploring retrofit logging architectures that convert production robot fleets into passive data collection assets.

According to the Open X-Embodiment paper (Padalkar et al., Google DeepMind et al., 2023), models trained on cross-embodiment data from 22+ robot types showed measurably better generalization on held-out tasks than models trained on any single embodiment’s data alone. EVST addresses this by structuring its internal data evaluation framework to weight embodiment diversity alongside episode count when assessing dataset utility for its production-line fine-tuning evaluations.

According to industry observations from multiple robot learning labs in 2024–2025, the visual sim-to-real gap accounts for the majority of policy failure cases in manipulation tasks that involve specular or transparent objects, which standard domain randomization techniques do not reliably cover. EVST addresses this by monitoring photorealistic simulation tools (specifically NVIDIA Cosmos and Gaussian Splatting pipelines) as candidate augmentation layers for its AI-enhanced motion planning roadmap.

Related Guides

• Embodied AI and VLA Models in Industrial Robotics: What Factory Engineers Need to Know. Covers the architecture of vision-language-action models and their deployment constraints on production hardware.
• Humanoid Robots in Industrial Manufacturing 2026. Examines which tasks are genuinely addressable by humanoid platforms versus conventional industrial arms.
• Humanoid Robots vs. Industrial Robot Arms for Factory Use in 2026. A direct comparison of capability, cost, and integration complexity between the two form factors.
• Complete Guide to Cobots: Types, Selection and Applications in 2026. The foundational reference for collaborative robot selection across payload, application, and environment requirements.

Frequently Asked Questions

Why is embodied AI data collection harder than collecting data for language models?

Language models train on text that already exists on the internet at enormous scale. Embodied AI robots must collect action-paired sensorimotor data: synchronized joint angles, gripper forces, camera frames, and task context, all recorded during physical manipulation. That data does not exist at web scale and is expensive to generate. Each teleoperated episode typically takes 1–10 minutes of skilled operator time to produce a single training trajectory, and quality degrades if the operator is unfamiliar with the robot’s dynamics.

What is the difference between sim-to-real transfer and domain randomization in robot training?

Sim-to-real transfer is the broad goal: train a policy in simulation and deploy it on a physical robot without performance collapse. Domain randomization is one technique to achieve that goal. It randomly varies simulation parameters — lighting, textures, friction coefficients, object masses — during training so the policy learns to handle real-world variation rather than overfitting to a single simulated appearance. Other methods include domain adaptation, photorealistic rendering (as in NVIDIA Cosmos), and system identification.

What is Open X-Embodiment and how many robot types does it cover?

Open X-Embodiment is a large-scale open robot learning dataset assembled by Google DeepMind with 33 contributing research institutions. It aggregates over 1 million robot episodes across more than 22 robot embodiments — single-arm manipulators, bimanual rigs, mobile bases, and dexterous hands. The RT-X models trained on OXE demonstrated that cross-embodiment training improves policy generalization compared with single-embodiment baselines.

How can a factory OEM contribute robot data to embodied AI training without exposing proprietary process IP?

The most practical approach is anonymized episode contribution: strip process-identifying metadata (part numbers, customer references, cycle-time targets) and contribute joint-trajectory and gripper-state streams only. Several dataset programs — including AgiBot World and emerging OEM consortium initiatives — accept de-identified industrial manipulation data under Apache 2.0 or CC-BY terms. Logging infrastructure with deterministic EtherCAT timestamps ensures episode quality meets dataset intake standards.

What teleoperation data collection system works best for industrial manipulation tasks?

For industrial manipulation the leading options are ALOHA 2 (bimanual leader-follower arms, low cost, open-source), UMI (wrist-mounted, decoupled from the robot, highly portable), and exoskeleton-based capture for higher-payload or full-body tasks. VR rigs using Apple Vision Pro or Meta Quest 3 are gaining traction for intuitive bimanual capture but require retargeting pipelines to translate human hand kinematics to robot joint space. The right choice depends on the target robot’s degrees of freedom, payload, and whether the task requires two hands.

Last Updated: April 23, 2026