Practical overview
When it fits
Use RoboTTT when building a robotics learning prototype in Python and you want to add memory or test-time-training components to an existing policy. The README offers both standalone wrapper and policy-integration examples.
Who it is for
- Robotics researchers exploring memory and test-time training in learning systems.
- Python developers who already have a policy model and want to integrate reusable memory components.
Documented uses
- Install the library and build a standalone memory wrapper for action-chunk sequences.
- Wrap a policy model with RoboTTT and configure which policy modules and arguments participate.
- Set up an example masked-loss training call and sample actions across multiple steps.
- Experiment with predicting a next-state latent representation alongside action generation.
Requirements & limits
The public GitHub repository specifies the MIT license. The README gives a pip installation command.
- The library supplies components rather than a complete robot-control application; policy and input configuration remain part of the integration.
- The examples use PyTorch and 512-dimensional memory components, so existing model dimensions and interfaces matter.
- Fast Weight Product Key Memory is shown as an alternative memory integration through fwPKMWrapper.
- The README demonstrates optional next-latent prediction from state representations, with configurable loss weighting and token slicing.
Check current requirements in the official documentation. Source-based guidance does not mean OpenViST has run or independently tested this project.
Install RoboTTT and assemble a standalone memory wrapper
Set up the library and construct the README's example TTTWrapper with a two-layer PyTorch memory network.
Before you start
- Python environment with pip
- PyTorch
- A 512-dimensional memory network, as in the example
Steps
Install the package with the command below.
pip install robo-tttImport
MemoryKeyValueBindandTTTWrapperfromrobo_ttt, along with PyTorch modules.Create the example two-layer MLP using Linear, GELU, and Linear layers.
Pass the network to
MemoryKeyValueBind(512, memory_network)and use that memory to createTTTWrapper(512, memory=memory).
Expected outcome
A standalone TTTWrapper configured with the example memory module, ready to receive attended action chunks in a Python program.
Guide limitations
- The guide covers installation and construction only; it does not verify execution or describe a complete robot policy or deployment.
- The example uses a 512-dimensional memory; other dimensions require matching configuration.
Wrap a policy with RoboTTT for sequence training and sampling
Configure the README's RoboTTT wrapper around a MimicVideo policy, including the shown training and rollout calls.
Before you start
- RoboTTT and PyTorch
- A policy model such as MimicVideo
- Video hidden states, joint-state inputs, actions, and prompt token IDs in shapes compatible with the configured policy
Steps
Create a memory network, MemoryKeyValueBind, and TTTWrapper as shown in the README.
Instantiate the example MimicVideo policy and pass it, the TTT wrapper, module paths, and time-related argument names to RoboTTT.
Provide sequence inputs and a loss mask to the model for the illustrated training pass, then call
loss.backward().For sampling, provide initial video hidden states and joint state to
model.sample, specifying steps and batch size as shown.
Expected outcome
A configured policy wrapper with example calls for masked-loss training and stepwise action sampling.
Guide limitations
- The example depends on the separate MimicVideo model and compatible inputs; the README does not provide a general-purpose policy or robot hardware setup.
- The illustrated dimensions and module paths are specific to the example and may need adaptation for another policy.
Sources & review
Source record details
- Source availability
- Active
- Sources checked
- 12 October 2026
- Original website title
- lucidrains/robo_ttt · Official homepage title · retained in its source language