Rao FU [娆]

Graphics/Vision/Robotics

Hello,

a bit about me:

I'm a postdoctoral scholar at the Berkeley AI Research (BAIR) Lab, where I'm fortunate to work with Prof. Jitendra Malik and Prof. Angjoo Kanazawa. I earned my PhD in Computer Science at Brown University, advised by Prof. Srinath Sridhar and Prof. Daniel Ritchie. I was a visiting PhD at MPI Informatics, spent an amazing time working with Prof. Christian Theobalt and Dr. Rishabh Dabral in Germany. I was named a Microsoft Research Fellow 2026 and an MIT EECS Rising Star 2025.

I was a research intern at Amazon FAR, Microsoft Research Asia, Autodesk AI Research, and Meta llama team.

Prior to this, I earned my B.Eng degree from the University of the Chinese Academy of Sciences, advised by Prof. Lin Gao and Prof. Xilin Chen. I also had a wonderful time at the University of Southern California and the University of California San Diego, mentored by Prof. Hao Su.

勿戳！
Don't poke my cheeks!
¡no me toques las mejillas!
मेरे गाल मत दबाओ!
আমার গালে খোঁচা দিও না!
não cutuque minhas bochechas!
не тыкай меня в щеки!
私の頬をつつかないで！
ਮੇਰੀਆਂ ਗੱਲ੍ਹਾਂ ਨੂੰ ਨਾ ਮਾਰੋ!
唔好戳我嘅脸颊！
đừng chọc má tôi!
నా బుగ్గలు కుట్టకు!
yanaklarımı dürtme!
내 뺨을 찌르지 마!
ne touchez pas mes joues!
stoß mir nicht in die Wangen!
என் கன்னங்களை குத்தாதே!
میرے گالوں کو مت مارو!
ojo nyocot pipiku!
non toccarmi le guance!
لا تكز خدي!

a bit about my research...

My research sits at the intersection of 3D vision, computer graphics, and robotics, where I work to enable machines to perceive multisensory spatial signals and dexterously interact with the world in human-like ways.

More concretely, I work on:

From Words to Worlds: Transforming language into physical reality. We explore how spoken and written words can be used to build spatial mental maps and guide intelligent actions.
Senses in the World: Learning through sight, touch, and movement. We research how machines can use multisensory input to understand their environment and interact with it naturally.
Capture the World: Digitizing reality with precision. We develop advanced methods to capture high-quality physical data, creating the foundation needed to train smarter AI systems.

and more about me...

Email to
Say Hi

CV/
Resume

Google
Scholar

LinkedIn
Profile

B.E.@
UCAS

Postdoc@
BAIR

Ph.D.@
Brown

My Cat
Inocchi

Arts
Todo...

News

[Sept. 2026] Successfully defended my thesis (From Words to Worlds: Bridging Linguistic and Spatial Intelligence)! OMG! Thanks to my committee, my collaborators, and everyone who has been so supportive! I'm lucky to have you all!
[June 2026] I'm excited to join Amazon FAR as a research intern! See you at FAR!
[Apr. 2026] Honored to be named as one of the Microsoft Research Fellows 2026! Thank you, Microsoft Research!
[Jan. 2026] Our 1st Sense of Space workshop proposal got aceepted to CVPR 2026!
[Oct. 2025] I'm visiting the University of California, San Diego for this fall 2025 and spring 2026. See you in the beautiful San Diego!
[Oct. 2025] Invited talk at UPenn GRASP. Click for video. Topic: From Words to Worlds!
[Sept 2025] Honored to be named as one of the Rising Stars in EECS — many thanks to the search committee for this recognition!
[July 2025] Invited talk at MPI Intelligent Systems. Topic: Capturing Dexterity!
[June 2025] Invited talk at Meta Robotics Group. Topic: Capturing Dexterity!

[May 2025] I'm visiting MPI Informatics this summer. See you in Germany!
[April 2025] Our 9th HANDS workshop proposal got aceepted to ICCV 2025!
[April 2025] Invited talk at MIT, Multisensory Intelligence Group. Topic: Capturing Dexterity!
[March 2025] Invited talk at UC Berkeley, Computer Vision Groups. Topic: Capturing Dexterity!
[March 2025] Invited talk at Columbia RoboPIL. Topic: Capturing Dexterity!
[March 2025] Invited talk at NYU Immersive Computing Lab. Topic: Capturing Dexterity!
[Feb 2025] Invited talk at NYC Vision Day. Topic: GigaHands!
[July 2024] OpenAI Research Access Granted. Thank you, OpenAI!
[Dec 2023] Oral Presentation at NECV 2023. See you at Dartmouth!
[Aug 2023] I'm attending Siggraph 2023 for fun: See you in Los Angeles!
[June 2023]I'm excited to join Meta Research as a research scientist intern this summer : See you in Menlo Park!
[Dec 2022] Oral Presentation at NECV 2022. See you at MIT!
[Nov 2023] I'm attending NeurIPS 2022 with ShapeCrafter: See you in New Orleans!
[Nov 2022] Invited talk at ICT Turing Seminar.
[May 2022]I'm excited to join Autodesk Research as a research intern this summer.

Selected Research Paper

OPENTOUCH: Bringing Full-Hand Touch to Real-World Interaction
R. Song*, J.Z. Li*, R. Fu*, D. Murphy, K.C. Zhou, R. Shiv, Y.Q. Li, H.Y. Xiong, C.E. Owens, Y.L. Du, Y.Y. Luo, X.Y. Cheng, A. Torralba, W. Matusik, P.P. Liang
preprint 2025
To bridge the gap between visual perception and physical interaction, we present OpenTouch, the first in-the-wild egocentric full-hand tactile dataset, containing 5.1 hours of synchronized video-tactile-pose data and over 2,900 curated clips collected across 14 everyday environments and around 800 objects in 14 general categories.
[paper] [project]

DyTact: Capturing Dynamic Contacts in Hand-Object Manipulation.
C.X. Yan, A. Xing, C. Pokhariya, R. Fu, S. Sridhar.
3DV 2026 (Oral)
We capture accurate dynamic contact (invisible property) in complex hand-object manipulation by accurately modeling the visible signals.
[paper] [project]

GigaHands: A Massive Annotated Dataset of Bimanual Hand Activities.
R. Fu*, D.X. Zhang*, A. Jiang, W.J. Fu, A. Funk, D. Ritchie, S. Sridhar.
CVPR 2025 (Highlight)
We introduce GigaHands, a massive annotated dataset capturing 34 hours of bimanual hand activities, totaling 14k motion clips derived from 183 million frames paired with 84k text annotations. The scale and diversity of GigaHands enable broad applications, including text-driven action synthesis, hand motion captioning, and dynamic radiance field reconstruction.
[paper] [project]

AnyHome: Open-Vocabulary Generation of Structured and Textured 3D Homes.
R. Fu*†, Z.H. When*, Z.C. Liu*, S. Sridhar. (†Corresponding Author)
ECCV 2024
We introduce AnyHome, a framework that translates open-vocabulary descriptions, ranging from simple labels to elaborate paragraphs, into well-structured and textured 3D indoor scenes at a house-scale.
[paper] [project]

Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning.
R. Fu, J.Y. Liu, X.L Chen, Y.X. Nie, W.H. Xiong.
WACV 2025
We build a 3D-VLM that understand and reason in interactive 3D scenes.
[paper]

CharacterMixer: Rig-Aware Interpolation of 3D Characters
X. Zhan, R. Fu, D. Ritchie.
Eurographics 2024
We present CharacterMixer, a system for blending two rigged 3D characters with different mesh and skeleton topologies while maintaining a rig throughout interpolation.
[paper] [project]

CLIP-Sculptor: Zero-Shot Generation of High-Fidelity and Diverse Shapes from Natural Language.
A. Sanghi, R. Fu, V. Liu, K. Willis, H. Shayani, A.H. Khasahmadi, S. Sridhar, D. Ritchie.
CVPR 2023
We introduce CLIP-Sculptor, a method to address these limitations by producing high-fidelity and diverse 3D shapes without the need for (text, shape) pairs for training.
[paper] [project]

ShapeCrafter: A Recursive Text-Conditioned 3D Shape Generation Model.
R. Fu, X. Zhan, Y.W. Chen, D. Ritchie, S. Sridhar.
NeurIPS 2022
We present ShapeCrafter, a neural network for recursive text-conditioned 3D shape generation. Our method supports shape editing, extrapolation, and can enable new applications in human-machine collaboration for creative design.
[paper] [project]

HRFormer: High-Resolution Transformer for Dense Prediction.
Y.H. Yuan, R. Fu, L. Huang, W.H. Lin, C. Zhang, X.L. Chen, J.D. Wang.
NeurIPS 2021
We present a High-Resolution Transformer (HRFormer) that learns high-resolution representations for dense prediction tasks. We demonstrate the effectiveness of the HRFormer on both human pose estimation and semantic segmentation tasks.
[paper] [project]

NeuralODF: Learning Omnidirectional Distance Fields for 3D Shape Representation.
T. Houchens, C.Y. Lu, S. Duggal, R. Fu, S. Srinath.
Technical Report
We propose Omnidirectional Distance Fields (ODFs), a new 3D shape representation that encodes geometry by storing the distance to the object’s surface from any 3D position in any viewing direction. We also introduce efficient forward mapping algorithms for transforming ODFs to and from common 3D representations.
[paper]

ROSA-Net: Rotation-Robust Structure-Aware Network for Fine-Grained 3D Shape Retrieval.
R. Fu, Y.C. Zhang, J. Yang, J.W. Sun, F.L. Zhang, Y.K. Lai, L. Gao.
CVM 2024
We introduce ROSAN, which learns rotation robust 3D shape descriptors and achieve accurate results on the task of fine-grained 3D object retrieval. We also build and publish a new 3D shape dataset for fine-grained 3D shape retrieval.
[paper] [project] [dataset]

Academic Journey

UNIVERSITY OF CALIFORNIA, BERKELEY

Sept. 2026 – Present