Experience is
the prompt

1Experience prompting

In 2020, GPT-3 showed that a language model could take on a new task from a few examples in its prompt [1]. This is in-context learning. In this post we use the term broadly, for any behavior a model picks up or adapts from experience at the moment of use, without any change to its weights. Robot models have begun to show a narrow form of it. Show them a few demonstrations of a new task, and they can sometimes carry it out [2, 3]. CO01 widens it in two ways. Its context is not limited to demonstrations, and its demonstrations are not limited to its own body or viewpoint.

For CO01, a demonstration is only the start of the lesson. A person can show the task with their own hands, let the robot try, and give feedback along the way. The robot uses all of it together, so its next attempt draws on more than imitation.

Experience prompting is how we put that experience in front of CO01. Each kind tells it something different about what to do next. A demonstration shows the goal. A failed attempt shows what to change. A correction adjusts a motion already under way. Figure 1 shows how this compares with what other robot policies learn from.

We want people to put our robots to work on tasks we never anticipated. For that, teaching a new task has to become part of using the robot, and part of building it. The people who use a robot should be able to show it a new task on the spot, and the team developing it should be able to add one without a new round of data collection and training. That makes learnability part of performance. What a robot can already do matters, and so does how easily anyone can teach it something new.

2Experience prompting in action

Here we change the experience CO01 receives and watch what it does with each kind. Section 3 puts them together in one long task.

To check that the demonstration really steers the robot, rather than the robot doing something it already knows, we set up scenes with two equally sensible goals and change only the demonstration: the same buttons pressed in a different order, or the same towel folded into a triangle or in half (Figure 2).

Next, we move further from training: scenes from an exact match to a different tool doing the same job (S0–S3), and tasks from familiar to never seen (N0–N2).

Finally, the biggest win from a demonstration in context is speed. We fine-tune the same pretrained model on more and more demonstrations of the same unseen tasks and count the time from collecting them to a working policy (Figure 4).

The benefit already shows up in training: with its earlier attempts in context, the model reaches the other model's best loss after about 4,600 training steps instead of 12,500 (Figure 5a). On the robot, we pick tasks where the first attempt often fails, allow up to five attempts, and compare the robot's own earlier attempts in context with a failed attempt from an unrelated task and with no history at all (Figure 5b).

Guidance sticks because the policy's context holds its interaction history within the episode, and across sessions the orchestrator saves preferences like this to memory.

3Putting it together: a long task

Making a cup of milk tea at a drink station takes many steps over several minutes, which makes it a useful first test of how the pieces of CO01 fit together. A person made the drink once at the station, and that recording became the prompt. What follows is an early proof of concept, not a finished capability. It shows how CO01 approaches a long recording: breaking it into subtasks, following them in order and drawing on the kinds of experience from Section 2 along the way. Making long tasks dependable is a focus of our next release.

Splitting a robot into a slow planner and a fast controller is not new [9, 10, 11, 12], and general-purpose models already show that robots can learn from experience in context [13, 14]. What sets CO01 apart is what passes between the two. Most hierarchical systems send a short language command, which says what to do but loses how. As Polanyi put it, "we can know more than we can tell" [15]. Few people could explain in words how to tuck in a fitted sheet, but a ten-second clip shows it. So the orchestrator hands over the experience itself, and the policy, trained to read it, does the interpreting.

4Building experience

A model only learns to read its context if, during training, the context helps it predict what comes next. Language models get this almost for free, because the web is full of documents in which earlier text explains later text, and in-context learning shows up when related examples arrive in bursts [16, 17]. Robot data rarely looks like that. Pipelines sample one episode at a time, and the sessions in which a robot fails and tries again, or a person steps in, usually get trimmed away. We built CO01's data around exactly that structure.

None of this needs a new learning algorithm. What matters is which data sits next to which in the training stream, and how varied that stream is [23].

5A scalable backbone that learns to perceive

Everything above runs on the policy's video backbone. We pretrain it on a broad mix of first-person video of people, robot data from many embodiments, handheld-gripper recordings [18], simulation and curated web video. At enough scale, video models pick up useful priors about the physical world [24]. But predicting how video unfolds does not, on its own, make every cue a policy needs easy to use. Geometry, object identity and motion can stay tangled together in the representation.

So we also train the backbone to perceive, as recent work shows a video-generation backbone can [25]: dense tasks (depth, surface normals, referring segmentation and optical flow) and sparse read-outs (3D body pose, hand pose, camera motion and visual grounding). This makes geometric, semantic and motion cues explicit for the policy. Figure 10 compares this one backbone with specialist models, and Video 6 shows what it reads from individual clips.

We expect better perception to mean better action, and we track it as pretraining grows along two axes. The first is the amount of data; we have processed roughly 50k hours so far. Figure 11a tracks two results on new task sets as it grows: the policy fine-tuned on 800 demonstrations from them, and CO01 given one demonstration in context with no task-specific training.

The second is task diversity. Figure 11b tracks in-context success as the in-context training data covers more distinct tasks, from a few dozen to several thousand, against the same model given only a language instruction. Studies in other domains find broad task coverage key to learning new tasks in context [23].

6Looking ahead

CO01 is an early look at what we are building. From a single demonstration, it already picks up skills it never saw in training. It is not yet as reliable as a policy fine-tuned on plenty of data, small errors still add up over long tasks, and the orchestrator sometimes picks the wrong demonstration. We expect these gaps to keep closing as training brings in more experience, and more varied experience. We also plan to carry CO01 beyond the tabletop to whole-body dexterity, starting with bimanual loco-manipulation. Today's general-purpose models still struggle most with this kind of task [33], and it is easier to show than to describe, which is where learning from experience should help.

The orchestrator is the part of CO01 we have explored least, and it has the most room to grow. Next, it could decide what experience to gather in the first place: ask for a demonstration when it is unsure, set up its own practice when a step keeps failing, or search other robots' sessions for one that fits.

In-context learning is also a kind of short-term memory. What CO01 learns from a correction lasts only as long as the context does. The orchestrator's memory keeps preferences across sessions, but not skills. The next step is to move what each robot learns in context into shared weights, continuously, so that every robot in a fleet benefits from what any one of them experiences. Silver and Sutton call this the era of experience, in which agents learn mainly from their own interaction with the world [34]. Robots may get there first, because their data already arrives as experience. Every deployment produces demonstrations, attempts and corrections, whether or not anyone keeps them. CO01 is built to keep them.

We are looking for partners to work with on CO01. If you want a robot that learns from you, get in touch. If you want to build one, we are hiring. The work spans in-context and continual learning, multimodal representation learning and experiments on real robots.

References

1. Brown, T. et al. Language Models are Few-Shot Learners. NeurIPS 2020. arXiv:2005.14165
2. Generalist AI. GEN-1.5: Embodied Foundation Models are One-Shot Learners. 2026. generalistai.com
3. Skild AI. Introducing S1: In-Context Learning for Robotics. 2026. skild.ai
4. Duan, Y. et al. One-Shot Imitation Learning. NeurIPS 2017. arXiv:1703.07326
5. Wood, D., Bruner, J. S. and Ross, G. The Role of Tutoring in Problem Solving. Journal of Child Psychology and Psychiatry 17(2), 1976.
6. OpenAI. GPT-6 Astra: A New Generation of Intelligence. 2026. openai.com
7. Li, S. et al. Unified Video Action Model. 2025. arXiv:2503.00200
8. Dyna Robotics. Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models. 2026. dyna.co
9. Shi, L. X. et al. Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models. ICML 2025. arXiv:2502.19417
10. NVIDIA (Bjorck, J. et al.). GR00T N1: An Open Foundation Model for Generalist Humanoid Robots. 2025. arXiv:2503.14734
11. Figure AI. Helix: A Vision-Language-Action Model for Generalist Humanoid Control. 2025. figure.ai
12. Dyna Robotics. Dyna-2.1: A Physical Agent for End-to-End Workflows. 2026. dyna.co
13. Liu, F. et al. RoboICL: Embodied In-Context Learning with GPT-6 Astra. 2026. arXiv:2609.34261v1
14. Cheng, D. et al. In-Context Robot Learning with VLM Agents. 2026. arXiv:2609.19138v1
15. Polanyi, M. The Tacit Dimension. University of Chicago Press, 1966.
16. Chan, S. C. Y. et al. Data Distributional Properties Drive Emergent In-Context Learning in Transformers. NeurIPS 2022. arXiv:2205.05055
17. Raparthy, S. C. et al. Generalization to New Sequential Decision Making Tasks with In-Context Learning. ICML 2024. arXiv:2312.03801
18. Chi, C. et al. Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots. RSS 2024. arXiv:2402.10329
19. Laskin, M. et al. In-context Reinforcement Learning with Algorithm Distillation. ICLR 2023. arXiv:2210.14215
20. Ross, S., Gordon, G. and Bagnell, J. A. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. AISTATS 2011. arXiv:1011.0686
21. Kelly, M., Sidrane, C., Driggs-Campbell, K. and Kochenderfer, M. J. HG-DAgger: Interactive Imitation Learning with Human Experts. ICRA 2019. arXiv:1810.02890
22. Physical Intelligence. π0.7: a Steerable Model with Emergent Capabilities. 2026. pi.website
23. Raventós, A., Paul, M., Chen, F., Ganguli, S. Pretraining Task Diversity and the Emergence of Non-Bayesian In-Context Learning for Regression. NeurIPS 2023. arXiv:2306.15063
24. Wiedemer, T. et al. Video Models are Zero-Shot Learners and Reasoners. 2025. arXiv:2509.20328
25. Wang, L., Zhang, C., Kabra, R. et al. Video Generation Models are General-Purpose Vision Learners. 2026. arXiv:2607.09024
26. Wang, J. et al. VGGT: Visual Geometry Grounded Transformer. CVPR 2025. arXiv:2503.11651
27. He, J. et al. Lotus-2: Advancing Geometric Dense Prediction with Powerful Image Generative Model. 2025. arXiv:2512.01030
28. Lin, H. et al. Depth Anything 3: Recovering the Visual Space from Any Views. ICLR 2026. arXiv:2511.10647
29. Wang, Y., Lipson, L. and Deng, J. SEA-RAFT: Simple, Efficient, Accurate RAFT for Optical Flow. ECCV 2024. arXiv:2405.14793
30. Bagchi, A. et al. ReferEverything: Towards Segmenting Everything We Can Speak of in Videos. ICCV 2025. arXiv:2410.23287
31. Lin, S. et al. Robust High-Resolution Video Matting with Temporal Guidance. WACV 2022. arXiv:2108.11515
32. Shen, Z. et al. World-Grounded Human Motion Recovery via Gravity-View Coordinates. SIGGRAPH Asia 2024. arXiv:2409.06662
33. SenseNova Joint Research Team (Cai, Z. et al.). Has GPT-6-Astra Achieved Embodied Intelligence? 2026. arXiv:2610.22471
34. Silver, D. and Sutton, R. S. Welcome to the Era of Experience. Preprint of a chapter in Designing an Intelligence, MIT Press, 2025. PDF

Citation

@misc{co01_2026,
  title  = {Experience is the Prompt: Introducing CO01},
  author = {CO01 Team},
  year   = {2026},
  month  = {October},
  note   = {COCO Matrix Research blog}
}