Very Recent Progress in 3D Hand Tasks

A complete paper list for hand pose can be found here. This repo focuses on some subject areas, ideas, and works. Although some papers invloved do not study towards 3D hand scope, they are conceived presenting relavent and inspiring ideas. If there is any mistake or carelessness, please feel free to open an issue or send me email.

Contents

• Contents
• TODO
• Overview
• Datasets
• Papers
  • Hand Shape/Pose
    • Forward Kinematics
    • Inverse Kinematics
    • Graph
    • Transformer
    • 2.5D
    • UV
    • SDF
    • Neural Rendering
    • Dynamics
    • Weak Supervision
    • Temporal
    • Cross-Modal
  • Hand-Object Interaction
    • Kinematics
    • Graph
    • SDF
    • Neural Rendering
    • Dynamics
    • Weak Supervision
    • Temporal
    • Joint Optimzation
  • Hand-Hand Interaction
    • Kinematics
    • 2.5D
    • Transformer
    • GAN
    • Graph
  • Full Body Reconstruction
    • Kinematics
    • Inverse Kinematics
    • SDF
    • Neural Rendering
    • Grasp
    • In-Sence
• Tools
  • Model
  • MoCap
• Benchmarks
  • FreiHAND
  • RHD

TODO

• Dtasets
• Papers
• Tools
• Benchmarks

Overview

Datasets

This list presents recent datasets with properties of data size, shape annotation, hand-hand interaction (Hand int.), hand-object interaction (Obj. int.), motion sequence, synthetic, and physical attributes (Phy. att.).

Dataset	Link	Size	Shape	Hand int.	Obj. int.	Motion	Synthetic	Phy. att.
CBF	link	25K	✅		✅	✅		✅
OakInk	link	50K	✅		✅	✅
Sign-Language	link	22801
MuViHand	link	402K				✅	✅
H2O	link	571K	✅	✅	✅	✅
HanCO	link	860K	✅			✅
H2O	link	5M	✅	✅	✅	✅
H2O-3D	link	62K	✅	✅	✅	✅
GRAB	link	1.6M	✅	✅	✅	✅
DexYCB	link	528K	✅		✅	✅
YoutubeHand	link	47K	✅
YCB-Affordance	link	134K	✅		✅	✅
HO3D	link	77K	✅		✅	✅
ContactPose	link	2.9M	✅	✅	✅	✅
FreiHAND	link	134K	✅
ObMan	link	154K	✅		✅		✅
Interhand2.6M	link	2.6M	✅	✅		✅
RHD	link	44K					✅
H3D	link	22K	✅
MHP	link	80K
MVHM	link	320K	✅				✅
Dexter+Object	link	3K			✅	✅
EgoDexter	link	3K				✅
STB	link	36K	✅			✅
FPHA	link	105K				✅
Tzionas et al.	link	36K		✅		✅
GANerated Hands	link	331K			✅	✅
SynthHands	link	220K			✅		✅

Papers

The below indices are references to the above overview.

Hand Shape/Pose

Forward Kinematics

• [H.FK.11] HandOccNet: Occlusion-Robust 3D Hand Mesh Estimation Network. CVPR22. [PDF] [Code]
  JoonKyu Park, Yeonguk Oh, Gyeongsik Moon, Hongsuk Choi, Kyoung Mu Lee

• [H.FK.10] RealisticHands: A Hybrid Model for 3D Hand Reconstruction. arXiv21. [PDF]
  Michael Seeber, Martin R. Oswald, Roi Poranne

• [H.FK.9] Contrastive Representation Learning for Hand Shape Estimation. arXiv21. [PDF] [Project] [Code]
  Christian Zimmermann, Max Argus, Thomas Brox

• [H.FK.8] Hand Image Understanding via Deep Multi-Task Learning. ICCV21. [PDF]
  Xiong Zhang, Hongsheng Huang, Jianchao Tan, Hongmin Xu, Cheng Yang, Guozhu Peng, Lei Wang, Ji Liu

• [H.FK.7] DeepHandMesh: Weakly-supervised Deep Encoder-Decoder Framework for High-fidelity Hand Mesh Modeling from a Single RGB Image. ECCV20. [PDF] [Project]
  Gyeongsik Moon, Takaaki Shiratori, Kyoung Mu Lee

• [H.FK.6] Knowledge as Priors: Cross-Modal Knowledge Generalization for Datasets without Superior Knowledge. CVPR20. [PDF]
  Long Zhao, Xi Peng, Yuxiao Chen, Mubbasir Kapadia, Dimitris N. Metaxas

• [H.FK.5] FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images. ICCV19. [PDF] [Project]Code]
  Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, Thomas Brox

• [H.FK.4] End-to-end Hand Mesh Recovery from a Monocular RGB Image. ICCV19. [PDF] [Code]
  Xiong Zhang, Qiang Li*, Wenbo Zhang, Wen Zheng*

• [H.FK.3] 3D Hand Shape and Pose from Images in the Wild. CVPR19. [PDF] [code]
  Adnane Boukhayma, Rodrigo de Bem, Philip H.S. Torr

• [H.FK.2] Pushing the Envelope for RGB-based Dense 3D Hand Pose Estimation via Neural Rendering. CVPR19. [PDF]
  Seungryul Baek, Kwang In Kim, Tae-Kyun Kim

• [H.FK.1] GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB. CVPR18. [PDF] [Supp] [Project]
  Franziska Mueller, Florian Bernard, Oleksandr Sotnychenko, Dushyant Mehta, Srinath Sridhar, Dan Casas, Christian Theobalt

Inverse Kinematics

• [H.IK.4] HandTailor: Towards High-Precision Monocular 3D Hand Recovery. arXiv21. [PDF] [Code]
  Jun Lv, Wenqiang Xu, Lixin Yang, Sucheng Qian, Chongzhao Mao, Cewu Lu

• [H.IK.3] HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation. CVPR21. [PDF] [Code]
  Jiefeng Li, Chao Xu, Zhicun Chen, Siyuan Bian, Lixin Yang, Cewu Lu

• [H.IK.2] BiHand: Recovering Hand Mesh with Multi-stage Bisected Hourglass Networks. BMVC20. [PDF] [Code]
  Lixin Yang, Jiasen Li, Wenqiang Xu, Yiqun Diao, Cewu Lu

• [H.IK.1] Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data. CVPR20. [PDF] [Project] [Code]
  Yuxiao Zhou, Marc Habermann, Weipeng Xu, Ikhsanul Habibie, Christian Theobalt, Feng Xu

Graph

• [H.GH.10] MobRecon: Mobile-Friendly Hand Mesh Reconstruction from Monocular Image. CVPR22. [PDF] [Code]
  Xingyu Chen, Yufeng Liu, Yajiao Dong, Xiong Zhang, Chongyang Ma, Yanmin Xiong, Yuan Zhang, Xiaoyan Guo

• [H.GH.9] JGR-P2O: Joint Graph Reasoning based Pixel-to-Offset Prediction Network for 3D Hand Pose Estimation from a Single Depth Image. ECCV20 [PDF] [Code]
  Linpu Fang, Xingyan Liu, Li Liu, Hang Xu, Wenxiong Kang

• [H.GH.8] Point2Skeleton: Learning Skeletal Representations from Point Clouds. CVPR21. [PDF] [Code]
  Cheng Lin, Changjian Li, Yuan Liu, Nenglun Chen, Yi-King Choi, Wenping Wang

• [H.GH.7] DC-GNet: Deep Mesh Relation Capturing Graph Convolution Network for 3D Human Shape Reconstruction, MM21. [PDF]
  Shihao Zhou, Mengxi Jiang, Shanshan Cai, Yunqi Lei

• [H.GH.6] MVHM: A Large-Scale Multi-View Hand Mesh Benchmark for Accurate 3D Hand Pose Estimation. WACV21. [PDF]
  Liangjian Chen, Shih-Yao Lin, Yusheng Xie, Yen-Yu Lin, Xiaohui Xie

• [H.GH.5] Camera-Space Hand Mesh Recovery via Semantic Aggregation and Adaptive 2D-1D Registration. CVPR21. [PDF] [Code]
  Xingyu Chen, Yufeng Liu, Chongyang Ma, Jianlong Chang, Huayan Wang, Tian Chen, Xiaoyan Guo, Pengfei Wan, Wen Zheng

• [H.GH.4] Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose. ECCV20. [PDF] [Code]
  Hongsuk Choi, Gyeongsik Moon, Kyoung Mu Lee

• [H.GH.3] Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild. CVPR20. [PDF] [Project]
  Dominik Kulon, Riza Alp Güler, Iasonas Kokkinos, Michael Bronstein, Stefanos Zafeiriou

• [H.GH.2] Exploiting Spatial-temporal Relationships for 3D Pose Estimation via Graph Convolutional Networks. ICCV19. [PDF]
  Yujun Cai, Liuhao Ge, Jun Liu, Jianfei Cai, Tat-Jen Cham, Junsong Yuan, and Nadia Magnenat Thalmann

• [H.GH.1] 3D Hand Shape and Pose Estimation from a Single RGB Image. CVPR19. [PDF] [Project] [Code]
  Liuhao Ge, Zhou Ren, Yuncheng Li, Zehao Xue, Yingying Wang, Jianfei Cai, Junsong Yuan

Transformer

• [H.TR.6] Spatial-Temporal Parallel Transformer for Arm-Hand Dynamic Estimation. CVPR22. [PDF]
  Shuying Liu, Wenbin Wu, Jiaxian Wu, Yue Lin

• [H.TR.5] GraFormer: Graph Convolution Transformer for 3D Pose Estimation. arXiv21. [PDF] [Code]
  Weixi Zhao, Yunjie Tian, Qixiang Ye, Jianbin Jiao and Weiqiang Wang

• [H.TR.4] Mesh Graphormer. ICCV21. [PDF] [Code]
  Kevin Lin, Lijuan Wang, Zicheng Liu

• [H.TR.3] End-to-End Human Pose and Mesh Reconstruction with Transformers. CVPR21. [PDF] [Code]
  Kevin Lin, Lijuan Wang, Zicheng Liu

• [H.TR.2] Hand-Transformer: Non-Autoregressive Structured Modeling for 3D Hand Pose Estimation. ECCV20. [PDF]
  Lin Huang, Jianchao Tan, Ji Liu, and Junsong Yuan

• [H.TR.1] Epipolar Transformers. CVPR20. [PDF] [Code]
  Yihui He, Rui Yan*, Shoou-I Yu, Katerina Fragkiadaki*

2.5D

• [H.VX.7] Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstruction. ICCV21. [PDF] [Code]
  Xiao Tang, Tianyu Wang, Chi-Wing Fu

• [H.VX.6] Weakly-Supervised 3D Human Pose Learning via Multi-view Images in the Wild. CVPR20. [PDF]
  Umar Iqbal, Pavlo Molchanov, Jan Kautz

• [H.VX.5] HandVoxNet++: 3D Hand Shape and Pose Estimation using Voxel-Based Neural Networks. arXiv21. [PDF]
  Jameel Malik, Soshi Shimada, Ahmed Elhayek, Sk Aziz Ali, Christian Theobalt, Vladislav Golyanik, Didier Stricker

• [H.VX.4] Exploiting Learnable Joint Groups for Hand Pose Estimation. AAAI21. [PDF] [Code] Moran Li, Yuan Gao, Nong Sang

• [H.VX.3] I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image. ECCV20. [PDF] [Code]
  Gyeongsik Moon, Kyoung Mu Lee

• [H.VX.2] HandVoxNet: Deep Voxel-Based Network for 3D Hand Shape and Pose Estimation from a Single Depth Map. CVPR20. [PDF]
  Jameel Malik, Ibrahim Abdelaziz, Ahmed Elhayek, Soshi Shimada, Sk Aziz Ali, Vladislav Golyanik, Christian Theobalt, Didier Stricker

• [H.VX.1] Hand Pose Estimation via Latent 2.5D Heatmap Regression. ECCV18. [PDF]
  Umar Iqbal, Pavlo Molchanov, Thomas Breuel, Juergen Gall, Jan Kautz

UV

• [H.UV.2] I2UV-HandNet: Image-to-UV Prediction Network for Accurate and High-fidelity 3D Hand Mesh Modeling. ICCV21. [PDF]
  Ping Chen, Yujin Chen, Dong Yang, Fangyin Wu, Qin Li, Qingpei Xia, Yong Tan

• [H.UV.1] HTML: A Parametric Hand Texture Model for 3D Hand Reconstruction and Personalizationm. ECCV20. [PDF] [Project]
  Neng Qian, Jiayi Wang, Franziska Mueller, Florian Bernard, Vladislav Golyanik, Christian Theobalt

SDF

• [H.SDF.1] A Skeleton-Driven Neural Occupancy Representation for Articulated Hands. 3DV21. [PDF]
  Korrawe Karunratanakul, Adrian Spurr, Zicong Fan, Otmar Hilliges, Siyu Tang

Neural Rendering

• [H.NR.1] LISA: Learning Implicit Shape and Appearance of Hands. CVPR22. [PDF]
  Enric Corona, Tomas Hodan, Minh Vo, Francesc Moreno-Noguer, Chris Sweeney, Richard Newcombe, Lingni Ma

Dynamics

• [H.DN.4] Neural MoCon: Neural Motion Control for Physically Plausible Human Motion Capture. CVPR22. [PDF] [Project]
  Buzhen Huang, Liang Pan, Yuan Yang, Jingyi Ju, Yangang Wang

• [H.DN.3] PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision. CVPR22. [PDF] [Code]
  Kehong Gong, Bingbing Li, Jianfeng Zhang, Tao Wang, Jing Huang, Michael Bi Mi, Jiashi Feng, Xinchao Wang

• [H.DN.2] SimPoE: Simulated Character Control for 3D Human Pose Estimation. ICCV21. [PDF] [Project]
  Ye Yuan, Shih-En Wei, Tomas Simon, Kris Kitani, Jason Saragih

• [H.DN.1] Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesis. NeurIPS20. [PDF] [Project] [Code]
  Ye Yuan, Kris Kitani

Weak Supervision

• [H.WS.14] End-to-end Weakly-supervised Multiple 3D Hand Mesh Reconstruction from Single Image. arXiv22. [PDF]
  Jinwei Ren, Jianke Zhu, Jialiang Zhang

• [H.WS.13] Consistent 3D Hand Reconstruction in Video via Self-Supervised Learning. arXiv22. [PDF]
  Zhigang Tu, Zhisheng Huang, Yujin Chen, Di Kang, Linchao Bao, Bisheng Yang, Junsong Yuan

• [H.WS.12] Weakly-supervised 3D Hand Pose Estimation from Monocular RGB Images. ECCV18. [PDF]
  Yujun Cai, Liuhao Ge, Jianfei Cai, and Junsong Yuan

• [H.WS.11] Aligning Latent Spaces for 3D Hand Pose Estimation. ICCV19. [PDF]
  Linlin Yang, Shile Li, Dongheui Lee, Angela Yao

• [H.WS.10] Semi-Supervised 3D Hand Shape and Pose Estimation with Label Propagation. arXiv21. [PDF]
  Samira Kaviani, Amir Rahimi, Richard Hartley

• [H.WS.9] Self-supervised Transfer Learning for Hand Mesh Recovery from Binocular Images. ICCV21. [PDF]
  Zheng Chen, Sihan Wang, Yi Sun, Xiaohong Ma

• [H.WS.8] SO-HandNet: Self-Organizing Network for 3D Hand Pose Estimation with Semi-supervised Learning. ICCV19. [PDF]
  Yujin Chen, Zhigang Tu, Liuhao Ge, Dejun Zhang, Ruizhi Chen, Junsong Yuan

• [H.WS.7] Self-Supervised 3D Hand Pose Estimation from Monocular RGB via Contrastive Learning. ICCV21. [PDF]
  Adrian Spurr, Aneesh Dahiya, Xucong Zhang, Xi Wang, Otmar Hilliges

• [H.WS.6] Adversarial Motion Modelling helps Semi-supervised Hand Pose Estimation. arXiv21. [PDF]
  Adrian Spurr, Pavlo Molchanov, Umar Iqbal, Jan Kautz, Otmar Hilliges

• [H.WS.5] SemiHand: Semi-supervised Hand Pose Estimation with Consistency. ICCV21. [PDF]
  Linlin Yang, Shicheng Chen, Angela Yao

• [H.WS.4] Model-based 3D Hand Reconstruction via Self-Supervised Learning. CVPR21. [PDF]
  Yujin Chen, Zhigang Tu, Di Kang, Linchao Bao, Ying Zhang, Xuefei Zhe, Ruizhi Chen, Junsong Yuan

• [H.WS.3] Weakly-Supervised 3D Hand Pose Estimation via Biomechanical Constraints. ECCV20. [PDF]
  Adrian Spurr, Umar Iqbal, Pavlo Molchanov, Otmar Hilliges, Jan Kautz

• [H.WS.2] Dual Grid Net: hand mesh vertex regression from single depth maps. ECCV20. [PDF]
  Chengde Wan, Thomas Probst, Luc Van Gool, Angela Yao

• [H.WS.1] Adaptive Wasserstein Hourglass for Weakly Supervised Hand Pose Estimation from Monocular RGB. MM20. [PDF]
  Yumeng Zhang, Li Chen, Yufeng Liu, Junhai Yong, Wen Zheng

Temporal

• [H.TP.8] Capturing Humans in Motion: Temporal-Attentive 3D Human Pose and Shape Estimation from Monocular Video. CVPR22. [PDF] [Project]
  Wen-Li Wei, Jen-Chun Lin, Tyng-Luh Liu, Hong-Yuan Mark Liao

• [H.TP.7] SmoothNet: A Plug-and-Play Network for Refining Human Poses in Videos. arXiv21. [PDF]
  Ailing Zeng, Lei Yang, Xuan Ju, Jiefeng Li, Jianyi Wang, Qiang Xu

• [H.TP.6] Multi-View Video-Based 3D Hand Pose Estimation. arXiv21. [PDF] [Code]
  Leyla Khaleghi, Alireza Sepas Moghaddam, Joshua Marshall, Ali Etemad

• [H.TP.5] TravelNet: Self-supervised Physically Plausible Hand Motion Learning from Monocular Color Images. ICCV21. [PDF]
  Zimeng Zhao, Xi Zhao and Yangang Wang

• [H.TP.4] Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Video. CVPR21. [PDF] [Code]
  Hongsuk Choi, Gyeongsik Moon, Ju Yong Chang, Kyoung Mu Lee

• [H.TP.3] Temporal-Aware Self-Supervised Learning for 3D Hand Pose and Mesh Estimation in Videos. WACV21. [PDF]
  Liangjian Chen, Shih-Yao Lin, Yusheng Xie, Yen-Yu Lin, Xiaohui Xie

• [H.TP.2] Adaptive Computationally Efficient Network for Monocular 3D Hand Pose Estimation. ECCV20. [PDF]
  Zhipeng Fan, Jun Liu, Yao Wang

• [H.TP.1] SeqHAND: RGB-Sequence-Based 3D Hand Pose and Shape Estimation. ECCV20. [PDF]
  John Yang, Hyung Jin Chang, Seungeui Lee, Nojun Kwak

Cross-Modal

• [H.CM.2] Knowledge as Priors: Cross-Modal Knowledge Generalization for Datasets without Superior Knowledge. CVPR20. [PDF]
  Long Zhao, Xi Peng, Yuxiao Chen, Mubbasir Kapadia, Dimitris N. Metaxas

• [H.CM.1] Cross-modal Deep Variational Hand Pose Estimation. CVPR18. [PDF]
  Adrian Spurr, Jie Song, Seonwook Park, Otmar Hilliges

Hand-Object Interaction

Kinematics

• [HO.FK.6] OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction. CVPR22. [PDF] [Code]
  Lixin Yang, Kailin Li Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, Cewu Lu

• [HO.FK.5] HO-3D_v3: Improving the Accuracy of Hand-Object Annotations of the HO-3D Dataset. arXiv21. [PDF]
  Shreyas Hampali, Sayan Deb Sarkar, Vincent Lepetit

• [HO.FK.4] Unsupervised Domain Adaptation with Temporal-Consistent Self-Training for 3D Hand-Object Joint Reconstruction. arXiv21. [PDF]
  Mengshi Qi, Edoardo Remelli, Mathieu Salzmann, Pascal Fua

• [HO.FK.3] DexYCB: A Benchmark for Capturing Hand Grasping of Objects. CVPR21. [PDF] [Project] [Code]
  Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S. Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, Jan Kautz, Dieter Fox

• [HO.FK.2] HOnnotate: A method for 3D Annotation of Hand and Objects Poses. CVPR20. [PDF] [Project] [Code]
  Shreyas Hampali, Mahdi Rad, Markus Oberweger, Vincent Lepetit

• [HO.FK.1] Learning joint reconstruction of hands and manipulated objects. CVPR19. [PDF] [Project] [Code]
  Yana Hasson, Gül Varol, Dimitris Tzionas, Igor Kalevatykh, Michael J. Black, Ivan Laptev, and Cordelia Schmid

Graph

• [HO.GH.3] Collaborative Learning for Hand and Object Reconstruction with Attention-guided Graph Convolution. CVPR22. [PDF]
  Tze Ho Elden Tse, Kwang In Kim, Ales Leonardis, Hyung Jin Chang

• [HO.GH.2] H2O: Two Hands Manipulating Objects for First Person Interaction Recognition. ICCV21. [PDF] [Project] [Code]
  Taein Kwon, Bugra Tekin, Jan Stuhmer, Federica Bogo, Marc Pollefeys

• [HO.GH.1] HOPE-Net: A Graph-based Model for Hand-Object Pose Estimation. CVPR20. [PDF] [Code]
  Bardia Doosti, Shujon Naha, Majid Mirbagheri, David Crandall

SDF

• [HO.SDF.2] What’s in your hands? 3D Reconstruction of Generic Objects in Hands. CVPR22. [PDF] [Project] [Code]
  Yufei Ye, Abhinav Gupta, Shubham Tulsiani

• [HO.SDF.1] Grasping Field: Learning Implicit Representations for Human Grasps. 3DV20. [PDF] [Code]
  Korrawe Karunratanakul, Jinlong Yang, Yan Zhang, Michael Black, Krikamol Muandet, Siyu Tang

Neural Rendering

• [HO.NR.1] NeuralHOFusion: Neural Volumetric Rendering Under Human-Object Interactions. CVPR22. [PDF]
  Yuheng Jiang, Suyi Jiang, Guoxing Sun, Zhuo Su, Kaiwen Guo, Minye Wu, Jingyi Yu, Lan Xu

Dynamics

• [HO.DN.3] D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactions. CVPR22. [PDF] [Project]
  Sammy Christen, Muhammed Kocabas, Emre Aksan, Jemin Hwangbo, Jie Song, Otmar Hilliges

• [HO.DN.2] DexMV: Imitation Learning for Dexterous Manipulation from Human Videos. ICCVW21. [PDF] [Project]
  Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, Xiaolong Wang

• [HO.DN.1] Stability-driven Contact Reconstruction From Monocular Color Images. CVPR22. [PDF]
  Zimeng Zhao, Binghui Zuo, Wei Xie, Yangang Wang

Weak Supervision

• [HO.WS.2] Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time. CVPR21. [PDF] [Project] [Code]
  Shaowei Liu, Hanwen Jiang*, Jiarui Xu, Sifei Liu, Xiaolong Wang*

• [HO.WS.1] Weakly-supervised Domain Adaptation via GAN and Mesh Model for Estimating 3D Hand Poses Interacting Objects. CVPR20. [PDF] [Code]
  Seungryul Baek, Kwang In Kim, Tae-Kyun Kim

Temporal

• [HO.TP.2] Towards unconstrained joint hand-object reconstruction from RGB videos. arXiv21. [PDF] [Project] [Code]
  Yana Hasson, Gül Varol, Ivan Laptev, Cordelia Schmid

• [HO.TP.1] Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstruction. CVPR20. [PDF] [Project] [Code]
  Yana Hasson, Bugra Tekin, Federica Bogo, Ivan Laptev, Marc Pollefeys, Cordelia Schmid

Joint Optimzation

• [HO.OP.9] ContactOpt: Optimizing Contact to Improve Grasps. CVPR21. [PDF] [Code]
  Patrick Grady, Chengcheng Tang, Christopher D. Twigg, Minh Vo, Samarth Brahmbhatt, Charles C. Kemp

• [HO.OP.8] CHORE: Contact, Human and Object REconstruction from a single RGB image. arXiv22. [PDF]
  Xianghui Xie, Bharat Lal Bhatnagar, Gerard Pons-Moll

• [HO.OP.7] Understanding Egocentric Hand-Object Interactions from Hand Pose Estimation [PDF]
  Yao Lu, Walterio W. Mayol-Cuevas

• [HO.OP.6] Reconstructing Hand-Object Interactions in the Wild. ICCV21. [PDF] [Code]
  Zhe Cao, Ilija Radosavovic*, Angjoo Kanazawa, Jitendra Malik*

• [HO.OP.5] ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesis. CVPR22. [PDF] [Code]
  Kailin Li, Lixin Yang, Xinyu Zhan, Jun Lv, Wenqiang Xu, Jiefeng Li, Cewu Lu

• [HO.OP.4] Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild. ECCV20. [PDF] [Code]
  Jason Y. Zhang, Sam Pepose, Hanbyul Joo, Deva Ramanan, Jitendra Malik, Angjoo Kanazawa

• [HO.OP.3] GanHand: Predicting Human Grasp Affordances in Multi-Object Scenes. CVPR20. [PDF] [Code]
  Enric Corona, Albert Pumarola, Guillem Alenya, Francesc Moreno-Noguer, Gregory Rogez

• [HO.OP.2] Hand-Object Contact Consistency Reasoning for Human Grasps Generation. ICCV21. [PDF] [Project]
  Hanwen Jiang, Shaowei Liu, Jiashun Wang, Xiaolong Wang

• [HO.OP.1] CPF: Learning a Contact Potential Field to Model the Hand-object Interaction. ICCV21. [PDF] [Code]
  Lixin Yang, Xinyu Zhan, Kailin Li, Wenqiang Xu, Jiefeng Li, Cewu Lu

Hand-Hand Interaction

Kinematics

• [HH.FK.5] Constraining Dense Hand Surface Tracking with Elasticity. TOG20. [PDF] Breannan Smith, Chenglei Wu, He Wen, Patrick Peluse, Yaser Sheikh, Jessica Hodgins, Takaaki Shiratori

• [HH.FK.4] Keypoint Transformer: Solving Joint Identification in Challenging Hands and Object Interactions for Accurate 3D Pose Estimation. CVPR22. [PDF] [Project]
  Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, Vincent Lepetit

• [HH.FK.3] Coherent Reconstruction of Multiple Humans from a Single Image. CVPR20 [PDF] [Project] [Code]
  Wen Jiang, Nikos Kolotouros, Georgios Pavlakos, Xiaowei Zhou, Kostas Daniilidis

• [HH.FK.2] Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinements. 3DV21. [PDF] [Project] [Code]
  Yu Rong, Jingbo Wang, Ziwei Liu, Chen Change Loy

• [HH.FK.1] RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video. SIGGRAPHAsia20 [PDF] [Project]
  Jiayi Wang, Franziska Mueller, Florian Bernard, Suzanne Sorli, Oleksandr Sotnychenko, Neng Qian, Miguel A. Otaduy, Dan Casas, and Christian Theobalt

2.5D

• [HH.VX.4] End-toend detection and pose estimation of two interacting hands. ICCV21. [PDF]
  Dong Uk Kim, Kwang In Kim, Seungryul Baek

• [HH.VX.3] Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-pixel Part Segmentation. 3DV21. [PDF]
  Zicong Fan, Adrian Spurr, Muhammed Kocabas, Siyu Tang, Michael J. Black, Otmar Hilliges

• [HH.VX.2] Interacting Two-Hand 3D Pose and Shape Reconstruction from Single Color Image. ICCV21. [PDF] [Project] [Code]
  Baowen Zhang, Yangang Wang, Xiaoming Deng, Yinda Zhang, Ping Tan, Cuixia Ma and Hongan Wang

• [HH.VX.1] InterHand2.6M: A New Large-scale Dataset and Baseline for 3D Single and Interacting Hand Pose Estimation from a Single RGB Image. ECCV20. [PDF] [Project] [Code]
  Gyeongsik Moon, Shoou-i Yu, He Wen, Takaaki Shiratori, Kyoung Mu Lee

Transformer

GAN

• [HH.GAN.2] H2O: A Benchmark for Visual Human-human Object Handover Analysis. ICCV21. [PDF]
  Ruolin Ye, Wenqiang Xu, Zhendong Xue, Tutian Tang, Yanfeng Wang, Cewu Lu

• [HH.GAN.1] Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamics. CVPR21. [PDF] [Project] [Code]
  Evonne Ng, Hanbyul Joo, Shiry Ginosar, Trevor Darrell

Graph

• [HH.GH.1] Interacting Attention Graph for Single Image Two-Hand Reconstruction. CVPR22. [PDF] [Code]
  Mengcheng Li, Liang An, Hongwen Zhang, Lianpeng Wu, Feng Chen, Tao Yu, Yebin Liu

Full Body Reconstruction

Kinematics

• [FBR.FK.6] Lightweight Multi-person Total Motion Capture Using Sparse Multi-view Cameras. ICCV21. [PDF]
  Yuxiang Zhang, Zhe Li, Liang An, Mengcheng Li, Tao Yu, Yebin Liu

• [FBR.FK.5] Monocular Total Capture: Posing Face, Body, and Hands in the Wild. CVPR19. [PDF] [Project] [Code]
  Donglai Xiang, Hanbyul Joo, Yaser Sheikh

• [FBR.FK.4] Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies [PDF]
  Hanbyul Joo, Tomas Simon, Yaser Sheikh

• [FBR.FK.3] Expressive Body Capture: 3D Hands, Face, and Body from a Single Image. CVPR19. [PDF] [Project] [Code]
  Georgios Pavlakos, Vasileios Choutas*, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, Michael J. Black*

• [FBR.FK.2] FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integration. arXiv21. [PDF] [Code]
  Yu Rong, Takaaki Shiratori, Hanbyul Joo

• [FBR.FK.1] Monocular Expressive Body Regression through Body-Driven Attention. ECCV20. [PDF] [Project] [Code]
  Vasileios Choutas, Georgios Pavlakos, Timo Bolkart, Dimitrios Tzionas , Michael J. Black

Inverse Kinematics

• [FBR.IK.2] Pose2Pose: 3D Positional Pose-Guided 3D Rotational Pose Prediction for Expressive 3D Human Pose and Mesh Estimation. arXiv20. [PDF]
  Gyeongsik Moon, Kyoung Mu Lee

• [FBR.IK.1] Monocular Real-time Full Body Capture with Inter-part Correlations. CVPR21. [PDF]
  Yuxiao Zhou, Marc Habermann, Ikhsanul Habibie, Ayush Tewari, Christian Theobalt, Feng Xu

SDF

• [FBR.SDF.9] SNARF: Differentiable Forward Skinning for Animating Non-Rigid Neural Implicit Shapes. ICCV21. [PDF] [Code] [Project]
  Xu Chen, Yufeng Zheng, Michael J. Black, Otmar Hilliges, Andreas Geiger

• [FBR.SDF.8] NASA: Neural Articulated Shape Approximation. ECCV20. [PDF] [Code]
  Boyang Deng, JP Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, Andrea Tagliasacchi

• [FBR.SDF.7] Locally Aware Piecewise Transformation Fields for 3D Human Mesh Registration. CVPR21. [PDF] [Code]
  Shaofei Wang, Andreas Geiger, Siyu Tang

• [FBR.SDF.6] LEAP: Learning Articulated Occupancy of People. CVPR21 [PDF] [Code]
  Marko Mihajlovic, Yan Zhang, Michael J. Black, Siyu Tang

• [FBR.SDF.5] Monocular Real-Time Volumetric Performance Capture. ECCV20. [PDF] [Code]
  Ruilong Li, Yuliang Xiu, Shunsuke Saito, Zeng Huang, Kyle Olszewski, Hao Li

• [FBR.SDF.4] ARCH: Animatable Reconstruction of Clothed Humans. CVPR20. [PDF]
  Zeng Huang, Yuanlu Xu, Christoph Lassner, Hao Li, Tony Tung

• [FBR.SDF.3] Geo-PIFu: Geometry and Pixel Aligned Implicit Functions for Single-view Human Reconstruction. NeurIPS20. [PDF] [Code]
  Tong He, John Collomosse, Hailin Jin, Stefano Soatto

• [FBR.SDF.2] PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization. CVPR20. [PDF] [Code]
  Shunsuke Saito, Tomas Simon, Jason Saragih, Hanbyul Joo

• [FBR.SDF.1] PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization. CVPR19 [PDF] [Code]
  Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, Hao Li

Neural Rendering

• [FBR.NR.9] Structured Local Radiance Fields for Human Avatar Modeling. CVPR22. [PDF] [Project]
  Zerong Zheng, Han Huang, Tao Yu, Hongwen Zhang, Yandong Guo, Yebin Liu

• [FBR.NR.8] Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control. SIGGRAPHAsia21. [PDF]
  [Project] Lingjie Liu, Marc Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, Christian Theobalt

• [FBR.NR.7] Editable Free-viewpoint Video Using a Layered Neural Representation. SIGGRAPH21. [PDF] [Project] [Code]
  Jiakai Zhang, Xinhang Liu, Xinyi Ye, Fuqiang Zhao, Yanshun Zhang, Minye Wu, Yingliang Zhang, Lan Xu, Jingyi Yu

• [FBR.NR.6] NeuMan: Neural Human Radiance Field from a Single Video. arXiv22. [PDF]
  Wei Jiang, Kwang Moo Yi, Golnoosh Samei, Oncel Tuzel, Anurag Ranjan

• [FBR.NR.5] HumanNeRF: Generalizable Neural Human Radiance Field from Sparse Inputs. arXiv21. [PDF] [Project]
  Fuqiang Zhao, Wei Yang, Jiakai Zhang, Pei Lin, Yingliang Zhang, Jingyi Yu, Lan Xu

• [FBR.NR.4] HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video. arXiv22. [PDF] [Project]
  Chung-Yi Weng, Brian Curless, Pratul P. Srinivasan, Jonathan T. Barron, Ira Kemelmacher-Shlizerman

• [FBR.NR.3] Vid2Actor: Free-viewpoint Animatable Person Synthesis from Video in the Wild. arXiv20. [PDF] [Project]
  Chung-Yi Weng, Brian Curless, Ira Kemelmacher-Shlizerman

• [FBR.NR.2] Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies. ICCV21 [PDF] [Code] [Project]
  Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, Hujun Bao

• [FBR.NR.1] Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans. CVPR21. [PDF] [Code] [Project]
  Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, Xiaowei Zhou

Grasp

• [FBR.RG.3] Stochastic Whole-Body Grasping with Contact. arXiv21. [PDF] [Project] [Code]
  Yan Wu, Jiahao Wang, Yan Zhang, Siwei Zhang, Otmar Hilliges, Fisher Yu, Siyu Tang

• [FBR.RG.2] GOAL: Generating 4D Whole-Body Motion for Hand-Object Grasping. CVPR22. [PDF] [Project]
  Omid Taheri, Vasileios Choutas, Michael J. Black, Dimitrios Tzionas

• [FBR.GP.1] GRAB: A Dataset of Whole-Body Human Grasping of Objects. ECCV20 [PDF] [Project] [Code]
  Omid Taheri, Nima Ghorbani, Michael J. Black, Dimitrios Tzionas

In-Sence

• [FBR.IS.5] Stochastic Scene-Aware Motion Prediction. ICCV21. [PDF] [Project] [Code]
  Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, Michael Black

• [FBR.IS.4] PLACE: Proximity Learning of Articulation and Contact in 3D Environments. 3DV20. [PDF] [Project] [Code]
  Siwei Zhang, Yan Zhang, Qianli Ma, Michael J. Black, Siyu Tang

• [FBR.IS.3] Populating 3D Scenes by Learning Human-Scene Interaction. CVPR21. [PDF] [Project] [Code]
  Mohamed Hassan, Partha Ghosh, Joachim Tesch, Dimitrios Tzionas, Michael J. Black

• [FBR.IS.2] The Wanderings of Odysseus in 3D Scenes. CVPR22. [PDF] [Project]
  Yan Zhang Siyu Tang

• [FBR.IS.1] Generating 3D People in Scenes without People. CVPR20. [PDF] [Code]
  Yan Zhang, Mohamed Hassan, Heiko Neumann, Michael J. Black, Siyu Tang

Tools

Model

• [T.MD.6] NIMBLE
• [T.MD.5] HTML
• [T.MD.4] mano
• [T.MD.3] manotorch
• [T.MD.2] manopth
• [T.MD.1] spheremesh

MoCap

• [T.MC.6] AlphaPose
• [T.MC.5] EasyMocap
• [T.MC.4] MediaPipe
• [T.MC.3] frankmocap
• [T.MC.2] HandMotionViewer
• [T.MC.1] openpose

Benchmarks

FreiHAND

Method	Backbone	PAMPJPE (mm)	PAMPVPE (mm)	F@5	F@15
MobRecon [H.GH.10]	ResNet50	5.7	5.9	0.784	0.986
Graphormer [H.TR.4]	Transformer	5.9	6.0	0.765	0.987
Tang et al. [H.VX.7]	ResNet50	6.7	6.7	0.724	0.981
METRO [H.TR.3]	Transformer	6.7	6.8	0.717	0.981
I2UV-HandNet [H.UV.2]	ResNet50	6.8	6.9	0.707	0.977
CMR [H.GH.5]	ResNet50⏳	6.9	7.0	0.715	0.977
HIU-DMTL [H.RG.8]	Custom	7.1	7.3	0.699	0.974
PeCLR [H.WS.7]	ResNet152	7.3	-	-	-
CMR [H.GH.5]	ResNet18⏳	7.4	7.5	0.687	0.973
I2L-MeshNet [H.VX.3]	ResNet50⏳	7.4	7.6	0.681	0.973
Pose2Mesh [H.GH.4]	−	7.7	7.8	0.674	0.969
Pose2Pose [FB.IK.2]	ResNet50	7.8	7.8	0.661	0.970
Li et al. [H.VX.4]	Hourglass⏳	8.0	-	-	-
YotubeHand [H.GH.3]	ResNet50	8.4	8.6	0.614	0.966
Pose2Pose [FB.IK.2]	ResNet18	8.6	8.6	0.621	0.962
FreiHAND [H.MN.5]	ResNet50	11.0	10.9	0.516	0.934
ObMan [HO.MN.1]	ResNet18	13.3	13.3	0.429	0.907
Boukhayma et al. [H.MN.3]	ResNet50	35.0	13.2	0.427	0.895

⏳: Hourglass stacked structure

RHD