For AI agents: the public content index is available at https://we0.ai/llms.txt, and the English article bundle is available at https://we0.ai/llms-full.txt.
For AI agents: the public content index is available at https://we0.ai/llms.txt, and the English article bundle is available at https://we0.ai/llms-full.txt.
For AI agents: the complete content index is available at https://we0.ai/llms.txt, the full English article bundle is available at https://we0.ai/llms-full.txt, and this page is available as Markdown at https://we0.ai/articles/infinihand-streaming-3d-hand-motion-from-53c55163.md.
A head-mounted camera captures how people manipulate objects, but turning that recording into useful 3D motion is difficult. The hands move,...
A head-mounted camera captures how people manipulate objects, but turning that recording into useful 3D motion is difficult. The hands move, the camera moves, fingers disappear behind objects, and two hands can obscure each other.
InfiniHand combines hand localization, detailed hand reconstruction, and camera-motion estimation in a streaming framework. Its goal is to place successive hand poses in a shared world coordinate system, making the motion more useful for large-scale annotation and potential embodied-learning workflows.
The work comes from researchers affiliated with Shanghai Artificial Intelligence Laboratory, Shanghai Jiao Tong University, and several other universities. Its project page provides synchronized visual comparisons and a 3D motion viewer.
A cropped hand image is useful for identifying finger articulation, but it contains limited information about where the hand sits in the surrounding scene. Likewise, a movement across the image can result from a moving hand, a turning head, or both.
Traditional pipelines often combine a hand detector, a pose estimator, a camera-tracking system, and an alignment stage. Errors in one stage can propagate: a missed hand changes the reconstruction, and an inaccurate camera path changes the apparent global motion.
InfiniHand shares geometric context across these tasks. A pretrained LingBot-Map backbone extracts full-frame features, which support left- and right-hand localization. The hand-reconstruction component combines those features with hand-centered appearance information to recover finer articulation.
Its hand representation uses MANO parameters for pose, shape, and orientation. Auxiliary 2D landmark supervision helps constrain how predicted 3D hands project back into the image. Camera estimates then allow the reconstructed hands to be expressed in the same world frame across time.
InfiniHand method figure, reproduced in the QbitAI/BAAI report.
The researchers gathered approximately 5,000 hours of public egocentric video from nine datasets, including ARCTIC, HOT3D, and EgoDex. Combining these sources requires more than concatenating files: hand order, coordinate conventions, and annotation types can differ.
The preprocessing pipeline standardizes available images, hand parameters, joints, masks, and bounding boxes. Where possible, it fits missing MANO parameters from 3D joints. Sequences with camera trajectories can also supply world-space supervision.
Annotation quality is a separate concern. The team uses Sapiens 2 to predict 2D hand landmarks and compares them with the original annotations projected into the image. Persistent disagreement can indicate wrong associations, bad projections, or implausible poses.
According to the paper, this process removes roughly 30–40% of candidate training data, with uncertain cases reserved for manual inspection. The figure describes the candidate-cleaning process; it should not be used to infer a precise retained-hour total that the authors do not provide here.
Training then proceeds in two stages:
This arrangement lets datasets with different annotation coverage contribute to the parts of the model they can supervise.
InfiniHand processes incoming video in windows while retaining geometric memory. Its context includes initial reference information, recent detailed features, and compressed information from older observations.
The implementation overlaps two prediction tasks: one worker processes the upcoming window through the backbone, while another decodes hand and camera predictions for the preceding window. Reusing features and reducing idle time supports higher throughput.
The paper reports 11.19 FPS prediction throughput, compared with 5.48 FPS for HaWoR, a 2.04-fold improvement under the reported comparison. This is not a claim of universal real-time operation at 30 FPS, nor a guaranteed speed on arbitrary hardware.
For longer sequences, a separate worker performs sparse bundle adjustment to refine the camera trajectory. A binary keyframe pool retains recent observations together with selected earlier frames. That background optimization has additional computational cost beyond the reported prediction throughput.
Camera-space evaluation covers 34 ARCTIC scenes, 10 HOT3D scenes, 166 HOI4D scenes, and 100 EgoDex scenes. The researchers compare against several existing reconstruction methods.
Describe your idea once, and We0 AI can generate a showcase site, pages, and CMS, then help you attract customers and traffic after launch.
One complete project generation for free registration
Best for trying one complete generation flow and seeing a first project draft quickly.
The report highlights PA-p, a measure that incorporates pose error and a penalty for missed detections. Lower values are better in the reported comparison:
| Dataset and metric | Comparison method | InfiniHand | Reported reduction |
|---|---|---|---|
| ARCTIC PA-p | ViDiHand: 9.82 mm | 7.72 mm | 21.4% |
| EgoDex PA-p | ViDiHand: 17.22 mm | 7.29 mm | 57.7% |
| EgoDex world-space W-MPJPE | Dyn-HaMR: 78.41 mm | 28.77 mm | 63.3% |
These rows concern different datasets and metrics; they should not be averaged into a single accuracy claim. InfiniHand also reports the lowest W-MPJPE among the methods compared on ARCTIC, HOT3D, and EgoDex.
Aligned qualitative comparison: blue denotes InfiniHand and green denotes the reference. Alignment is part of the visualization and should be considered when interpreting apparent agreement.
The source also discusses natural-scene Ego4D examples involving occlusion and object interaction. One illustrated offline comparison uses external hand localization for InfiniHand. Readers should keep that setup separate from the unified model's localization claims; the interactive project demos provide the relevant visual context.
Recovering 3D human demonstrations can help prepare data for dexterous manipulation, motion transfer, and imitation-learning research. Turning reconstructed hands into robot training data still requires quality checks and adaptation to the robot's physical structure.
The authors identify several limitations. LingBot-Map does not inherently recover metric scale, so an auxiliary alignment model is needed and can introduce errors. Fast viewpoint changes, severe occlusion, and intermittent hand visibility remain difficult because suitable training data is limited.
Monocular supervision can also carry time-varying scale drift into training. Sparse refinement helps, but it does not make every recovered trajectory an exact physical measurement.
It estimates hand locations, MANO hand parameters, and camera trajectories from egocentric video. These estimates support hand motion represented in a shared world coordinate system.
Camera-space hand motion changes with the camera's movement. A shared world frame makes it easier to compare hand trajectories across frames while accounting for the moving viewpoint.
The proposed framework includes its own localization component. However, the source identifies an offline qualitative example that uses external hand localization, so readers should check the setup of each displayed comparison.
No. It is prediction throughput under the authors' reported evaluation. Background sparse bundle adjustment incurs additional work, and performance depends on the execution configuration.
The researchers compare projected annotations with image-based 2D hand landmark predictions and reject sequences with persistent inconsistencies. They report removing roughly 30–40% of candidate data and reserving ambiguous cases for manual checks.
The paper presents reconstruction and annotation research, not a validated direct robot-control system. Robot use would require retargeting, physical constraints, and task-specific verification.
InfiniHand combines scene geometry, detailed hand appearance, and camera estimation to recover hand motion from first-person video. Its streaming design and data-cleaning pipeline target the practical challenge of annotating large video collections.
The reported accuracy and throughput improvements are promising research results, with important conditions around scale recovery and background refinement. The output is a useful reconstruction estimate that still needs validation for downstream physical tasks.
Start from one sentence and have a complete website in minutes.
