Walking Cat

Rao FU [娆]

Graphics/Vision/Robotics

 Hello,

a bit about me:

I'm a PhD student in Computer Science at Brown University. I'm now visiting MPI Informatics this summer. My research sits at the intersection of 3D vision, computer graphics, and robotics, where I work to enable machines to imagine and interact with the world in human-like ways.

More concretely, I work on:

  1. From Words to Worlds: How abstract, symbolic input (language) can be transformed into spatial mental models to generate worlds and actions?.
  2. Senses in the World: How embodied, physical input (vision, touch, and proprioception) defines and grounds mental models for action.
  3. Capture the World: How to effectively capture high-quality real-world physical signals to train such systems?
I'm fortunate to work with Prof. Srinath Sridhar and Prof. Daniel Ritchie at Brown IVL & Brown VC Group.

Previously, I have intern at Microsoft Research Asia, Autodesk AI Research, and Meta llama team.

Prior to this, I earned my Bachelor of Engineering degree from the University of the Chinese Academy of Sciences, advised by Prof. Lin Gao and Prof. Xilin Chen. I also had a wonderful time at the University of Southern California and the University of California San Diego, mentored by Prof. Hao Su.

Your Image

勿戳！
Don't poke my cheeks!
¡no me toques las mejillas!
मेरे गाल मत दबाओ!
আমার গালে খোঁচা দিও না!
não cutuque minhas bochechas!
не тыкай меня в щеки!
私の頬をつつかないで！
ਮੇਰੀਆਂ ਗੱਲ੍ਹਾਂ ਨੂੰ ਨਾ ਮਾਰੋ!
唔好戳我嘅脸颊！
đừng chọc má tôi!
నా బుగ్గలు కుట్టకు!
yanaklarımı dürtme!
내 뺨을 찌르지 마!
ne touchez pas mes joues!
stoß mir nicht in die Wangen!
என் கன்னங்களை குத்தாதே!
میرے گالوں کو مت مارو!
ojo nyocot pipiku!
non toccarmi le guance!
لا تكز خدي!
and more about me...
email_button

Email to
Say Hi

cv_button

CV/
Resume

google_button

Google
Scholar

lkin_icon

LinkedIn
Profile

ucas_button

B.E.@
UCAS

brown_button

Ph.D.@
Brown

cat_button

My Cat
Inocchi

art_button

Arts
Todo...

Future Innovator On Deck 🚀
Open to 2025-2026 Postdocs & Industry Positions 🌍🤝

News
Selected Research Paper
 DyTact: Capturing Dynamic Contacts in Hand-Object Manipulation.
C.X. Yan, A. Xing, C. Pokhariya, R. Fu*, S. Sridhar.
ArXiv preprint 2025 
We capture accurate dynamic contact in complex hand-object manipulation.
[paper] [project]
GigaHands: A Massive Annotated Dataset of Bimanual Hand Activities.
R. Fu*, D.X. Zhang*, A. Jiang, W.J. Fu, A. Funk, D. Ritchie, S. Sridhar.
CVPR 2025 (Highlight)
We introduce GigaHands, a massive annotated dataset capturing 34 hours of bimanual hand activities, totaling 14k motion clips derived from 183 million frames paired with 84k text annotations. The scale and diversity of GigaHands enable broad applications, including text-driven action synthesis, hand motion captioning, and dynamic radiance field reconstruction.
[paper] [project]
AnyHome: Open-Vocabulary Generation of Structured and Textured 3D Homes.
R. Fu*†, Z.H. When*, Z.C. Liu*, S. Sridhar. (†Corresponding Author)
ECCV 2024
We introduce AnyHome, a framework that translates open-vocabulary descriptions, ranging from simple labels to elaborate paragraphs, into well-structured and textured 3D indoor scenes at a house-scale.
[paper] [project]
Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning.
R. Fu, J.Y. Liu, X.L Chen, Y.X. Nie, W.H. Xiong.
WACV 2025 
We build a 3D-VLM that understand and reason in interactive 3D scenes.
[paper]
CharacterMixer: Rig-Aware Interpolation of 3D Characters
X. Zhan, R. Fu*, D. Ritchie.
Eurographics 2024
We present CharacterMixer, a system for blending two rigged 3D characters with different mesh and skeleton topologies while maintaining a rig throughout interpolation.
[paper] [project]
CLIP-Sculptor: Zero-Shot Generation of High-Fidelity and Diverse Shapes from Natural Language.
A. Sanghi, R. Fu, V. Liu, K. Willis, H. Shayani, A.H. Khasahmadi, S. Sridhar, D. Ritchie.
CVPR 2023
We introduce CLIP-Sculptor, a method to address these limitations by producing high-fidelity and diverse 3D shapes without the need for (text, shape) pairs for training.
[paper] [project]
ShapeCrafter: A Recursive Text-Conditioned 3D Shape Generation Model.
R. Fu, X. Zhan, Y.W. Chen, D. Ritchie, S. Sridhar.
NeurIPS 2022
We present ShapeCrafter, a neural network for recursive text-conditioned 3D shape generation. Our method supports shape editing, extrapolation, and can enable new applications in human-machine collaboration for creative design.
[paper] [project]
HRFormer: High-Resolution Transformer for Dense Prediction.
Y.H. Yuan, R. Fu, L. Huang, W.H. Lin, C. Zhang, X.L. Chen, J.D. Wang.
NeurIPS 2021
We present a High-Resolution Transformer (HRFormer) that learns high-resolution representations for dense prediction tasks. We demonstrate the effectiveness of the HRFormer on both human pose estimation and semantic segmentation tasks.
[paper] [project]
NeuralODF: Learning Omnidirectional Distance Fields for 3D Shape Representation.
T. Houchens, C.Y. Lu, S. Duggal, R. Fu, S. Srinath.
Technical Report
We propose Omnidirectional Distance Fields (ODFs), a new 3D shape representation that encodes geometry by storing the distance to the object’s surface from any 3D position in any viewing direction. We also introduce efficient forward mapping algorithms for transforming ODFs to and from common 3D representations.
[paper]
ROSA-Net: Rotation-Robust Structure-Aware Network for Fine-Grained 3D Shape Retrieval.
R. Fu, Y.C. Zhang, J. Yang, J.W. Sun, F.L. Zhang, Y.K. Lai, L. Gao.
CVM 2024
We introduce ROSAN, which learns rotation robust 3D shape descriptors and achieve accurate results on the task of fine-grained 3D object retrieval. We also build and publish a new 3D shape dataset for fine-grained 3D shape retrieval.
[paper] [project] [dataset]
Education
BROWN UNIVERSITY
Sept.2021 – Present
  • Title: Computer Science Ph.D. Student
  • Concentration: Vision, Graphics and Robotics
UNIVERSITY OF CHINESE ACADEMY OF SCIENCES
Sept. 2017-Jul. 2021
  • Degree: Bachelor Engineering in Computer Science.
  • Outstanding Thesis Awards.
  • National Inspirational Scholarship(5%).
UNIVERSITY OF CALIFORNIA, SAN DIEGO
May. 2020-Aug. 2020
  • Title: Visiting Scholar.
UNIVERSITY OF SOUTHERN CALIFORNIA
Jan. 2020-May. 2020
  • Title: Visiting Student.
BEIJING NATIONAL DAY SCHOOL
Sept. 2011-Jul. 2017
  • Title: Student.
Employment
Research intern: Meta
May. 2023 – Sept. 2023
Group: llama Team
Menlo Park, CA, US
  • Research on Large 3D Visual-Language Model.
Research intern: Autodesk Inc.
May. 2022 – Dec. 2022
Group: AI Lab
Toronto, Canada
  • Research on Lauguage-guided 3D Shape Generation.
Research intern: Microsoft Research, Asia
Mar. 2021 – July. 2021
Group: Speech Group, Visual Computing Group
Beijing, China
  • Research on High-Resolution Machine Perception Backbone
Research intern: University of California, San Diego
May. 2020 – Nov. 2020
Advisor: Prof. Hao Su
San Diego, CA, USA
  • Research on Robot Grasping with Two-Finger Grippers and Humanoid Robot Grippers.
Research intern: Institute of Computing Technology, CAS.
Jul. 2019 – Nov. 2020
Advisor: Prof. Lin Gao
Beijing, China
  • Research on Fine-grained 3D Shape Retrieval.
  • Research on Talking Head Generation.

Buy me a Boba:

If you like my homepage and think it's useful, welcome to buy me a Boba via Wechat or Paypal down here:
© 2025 by Rao FU