Rao FU
Computer Vision/Graphics/Robotics
a bit about me:
I'm a PhD student in Computer Science at Brown University. My research sits at the intersection of 3D vision, computer graphics, and robotics, where I work to enable machines to imagine and interact with the world in human-like ways.
More concretely, I ask:
- What representations effectively bridge verbal descriptions and inherently non-verbal spatial concepts?
- Which sensing modalities are needed for robots to replicate human-level dexterity?
- How do we capture, curate, and structure the rich, multi-modal datasets required to train such systems?
Previously, I have enriched my research experience as an intern at Microsoft Research Asia, Autodesk AI Research, and Meta GenAI.
Prior to this, I earned my Bachelor of Engineering degree from the University of the Chinese Academy of Sciences, advised by Prof. Lin Gao and Prof. Xilin Chen. I also had a wonderful time at the University of Southern California and the University of California San Diego, mentored by Prof. Hao Su.
and more about me...
News
- [April 2025] Invited talk at MIT, Multisensory Intelligence Group. Topic: Capturing Dexterity!
- [March 2025] Invited talk at UC Berkeley, Computer Vision Groups. Topic: Capturing Dexterity!
- [March 2025] Invited talk at Columbia RoboPIL. Topic: Capturing Dexterity!
- [March 2025] Invited talk at NYU Immersive Computing Lab. Topic: Capturing Dexterity!
- [Feb 2025] Invited talk at NYC Vision Day. Topic: GigaHands!
- [July 2024] OpenAI Research Access Granted. Thank you, OpenAI!
- [Dec 2023] Oral Presentation at NECV 2023. See you at Dartmouth!
- [Aug 2023] I'm attending Siggraph 2023 for fun: See you in Los Angeles!
- [June 2023]I'm excited to join Meta Research as a research scientist intern this summer : See you in Menlo Park!
- [Dec 2022] Oral Presentation at NECV 2022. See you at MIT!
- [Nov 2023] I'm attending NeurIPS 2022 with ShapeCrafter: See you in New Orleans!
- [Nov 2022] Invited talk at ICT Turing Seminar.
- [May 2022]I'm excited to join Autodesk Research as a research intern this summer.
Selected Research Paper
R. Fu*, D.X. Zhang*, A. Jiang, W.J. Fu, A. Funk, D. Ritchie, S. Sridhar.
CVPR 2025 (Highlight)
We introduce GigaHands, a massive annotated dataset capturing 34 hours of bimanual hand activities, totaling 14k motion clips derived from 183 million frames paired with 84k text annotations. The scale and diversity of GigaHands enable broad applications, including text-driven action synthesis, hand motion captioning, and dynamic radiance field reconstruction.
[paper] [project]
R. Fu*†, Z.H. When*, Z.C. Liu*, S. Sridhar. (†Corresponding Author)
ECCV 2024
We introduce AnyHome, a framework that translates open-vocabulary descriptions, ranging from simple labels to elaborate paragraphs, into well-structured and textured 3D indoor scenes at a house-scale.
[paper] [project]
A. Sanghi, R. Fu, V. Liu, K. Willis, H. Shayani, A.H. Khasahmadi, S. Sridhar, D. Ritchie.
CVPR 2023
We introduce CLIP-Sculptor, a method to address these limitations by producing high-fidelity and diverse 3D shapes without the need for (text, shape) pairs for training.
[paper] [project]
R. Fu, X. Zhan, Y.W. Chen, D. Ritchie, S. Sridhar.
NeurIPS 2022
We present ShapeCrafter, a neural network for recursive text-conditioned 3D shape generation. Our method supports shape editing, extrapolation, and can enable new applications in human-machine collaboration for creative design.
[paper] [project]
Y.H. Yuan, R. Fu, L. Huang, W.H. Lin, C. Zhang, X.L. Chen, J.D. Wang.
NeurIPS 2021
We present a High-Resolution Transformer (HRFormer) that learns high-resolution representations for dense prediction tasks. We demonstrate the effectiveness of the HRFormer on both human pose estimation and semantic segmentation tasks.
[paper] [project]
T. Houchens, C.Y. Lu, S. Duggal, R. Fu, S. Srinath.
Technical Report
We propose Omnidirectional Distance Fields (ODFs), a new 3D shape representation that encodes geometry by storing the distance to the object’s surface from any 3D position in any viewing direction. We also introduce efficient forward mapping algorithms for transforming ODFs to and from common 3D representations.
[paper]
R. Fu, Y.C. Zhang, J. Yang, J.W. Sun, F.L. Zhang, Y.K. Lai, L. Gao.
CVM 2024
We introduce ROSAN, which learns rotation robust 3D shape descriptors and achieve accurate results on the task of fine-grained 3D object retrieval. We also build and publish a new 3D shape dataset for fine-grained 3D shape retrieval.
[paper] [project] [dataset]
Education
BROWN UNIVERSITY
Sept.2021 – Present
- Title: Computer Science Ph.D. Student
- Concentration: Vision, Graphics and Robotics
UNIVERSITY OF CHINESE ACADEMY OF SCIENCES
Sept. 2017-Jul. 2021
- Degree: Bachelor Engineering in Computer Science.
- Outstanding Thesis Awards.
- National Inspirational Scholarship(5%).
UNIVERSITY OF CALIFORNIA, SAN DIEGO
May. 2020-Aug. 2020
- Title: Visiting Scholar.
UNIVERSITY OF SOUTHERN CALIFORNIA
Jan. 2020-May. 2020
- Title: Visiting Student.
BEIJING NATIONAL DAY SCHOOL
Sept. 2011-Jul. 2017
- Title: Student.
Employment
Research intern: Meta
May. 2023 – Sept. 2023
Group: GenAI
Menlo Park, CA, US
- Research on Large 3D Visual-Language Model.
Research intern: Autodesk Inc.
May. 2022 – Dec. 2022
Group: AI Lab
Toronto, Canada
- Research on Lauguage-guided 3D Shape Generation.
Research intern: Microsoft Research, Asia
Mar. 2021 – July. 2021
Group: Speech Group, Visual Computing Group
Beijing, China
- Research on High-Resolution Machine Perception Backbone
Research intern: University of California, San Diego
May. 2020 – Nov. 2020
Advisor: Prof. Hao Su
San Diego, CA, USA
- Research on Robot Grasping with Two-Finger Grippers and Humanoid Robot Grippers.
Research intern: Institute of Computing Technology, CAS.
Jul. 2019 – Nov. 2020
Advisor: Prof. Lin Gao
Beijing, China
- Research on Fine-grained 3D Shape Retrieval.
- Research on Talking Head Generation.
Buy me a Boba:
If you like my homepage and think it's useful, welcome to buy me a Boba via Wechat or Paypal down here:
© 2024 by Rao FU