About
I am a Senior Research Scientist at Adobe Research. I got my PhD in Computer Science from Columbia University and BEng from HKUST.
I am interested in building novel creative tools, using the latest advances from vision, graphics, machine learning, and HCI. My past research and engineering has been recognized by two ACM UIST Best Paper Awards 🏅 (2022 and 2017), an Emmy Award 🏆 for Technology and Engineering (2020), two Adobe MAX Sneaks Demos ✨(2019 , 2020), an Adobe Research Fellowship 🎓 (2017), a NVIDIA PhD Fellowship Finalist (2017), a Shapeways Educational Grant 🧧 (2016), and an HKUST academic achievement medal 🥇 (2013). I have served as international program committee members for Eurographics, Graphics Interface, and ACM Multimedia, and as reviewers for many top academic conferences including UIST, CHI, SIGGRAPH, CVPR, ICCV, etc. From 2019-2022, I served as the Adobe Fellowship program co-chair, recognizing outstanding PhD students conducting exceptional research.
I’m always looking for interns and collaborators to work on research projects leading to publications/patents/product features. Please reach out to chat.
In my personal time, I host two podcasts -- Not Just Research (Youtube or Audio Podcast) in English and 李丁聊天室(Youtube and Audio Podcast) in Chinese. The topics I cover are quite diverse, mostly about research and some about life/investment/etc.
Selected Publications (full list)
PodReels: Human-AI Co-Creation of Video Podcast Teasers
Sitong Wang, Zheng Ning, Anh Truong, Mira Dontcheva, Dingzeyu Li, Lydia B Chilton
DIS 2024
[5-minute video]
[30-second video]
[html]
[pdf]
ReelFramer: Co-creating News Reels on Social Media with Generative AI
Sitong Wang, Samia Menon, Tao Long, Keren Henderson, Dingzeyu Li, Kevin Crowston, Mark Hansen, Jeffrey V Nickerson, Lydia B Chilton
CHI 2024
[video]
[html]
[pdf]
CrossA11y: Identifying Video Accessibility Issues via Cross-modal Grounding
Xingyu "Bruce" Liu, Ruolin Wang, Dingzeyu Li, Xiang "Anthony" Chen, Amy Pavel
UIST 2022
Best Paper Award
[pdf]
[project]
[video]
Beyond Subtitles: Captioning and Visualizing Non-speech Sounds to Improve Accessibility of User-Generated Videos
Oliver Alonzo, Hijung Valentina Shin, Dingzeyu Li
ASSETS 2022
[pdf]
[project]
[presentation video]
MakeItTalk: Speaker-Aware Talking Head Animation
Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis, Dingzeyu Li
SIGGRAPH Asia 2020
[arxiv]
[video]
[code]
[MAX Sneaks demo]
[TechCrunch]
[Adobe blog]
(Public) Product Impacts, Tech Transfers, and Demos
Project Blink, AI-powered video editing on the web
Shipped at Adobe MAX 2022 as Adobe Labs project
[Project]
Contributors: Blink team. I led the multi-modal visual search feature,
including AI/ML research, ML data ingestion, frontend engineering.
Animate from Audio, Adobe Express.
Shipped in Adobe Exprses 2022, AI/ML Feature
One of the most highly used quick actions.
[Live Demo]
[Adobe HelpX page]
[Video Demo]
Lead Researcher: Dingzeyu Li
Speech-Aware Animation
Shipped in Adobe Character Animator 2020, Sensei-powered ML Feature, runs faster than real-time in all desktop platforms.
[Official Release Note]
[Public Beta Release Note]
Press Coverage:
[Forbes Coverage]
[9to5Mac]
[VentureBeat]
[EnterpriseTalk]
[Animation World Network]
[Computer Graphics World]
[BlogCritics]
[ProVideo Coalition]
[Animation Magazine]
Lead Researcher/Developer: Dingzeyu Li
Collaborators: Yang Zhou
Project On the Beat: An AI-powered music video editing tool for synchronizing body movements to beats
Adobe MAX Sneaks Demo 2020
[video]
[Protocol]
[Adobe blog]
Presenter: Yang Zhou
Collaborators: Dingzeyu Li, Jun Saito, Deepali Aneja, Jimei Yang
Project Sweet Talk: Audio Driven Facial Animation from Single Image
Adobe MAX Sneaks Demo 2019
[video]
[TechCrunch]
[Adobe blog]
Presenter: Dingzeyu Li
Collaborators: Yang Zhou, Jose Echevarria, Eli Shechtman
Robust Noise-Resilient Automatic Lipsync and Interactive Adjustment
Shipped in Adobe Character Animator 2019
[Release Note]
Lead Researcher/Developer: Dingzeyu Li
Physics-Aware 3D Shape Drop to Ground
Shipped in Adobe Dimension 2019
Lead Researcher/Developer: Dingzeyu Li
Research Interns/Students Mentored
- Sitong Wang, Columbia University
- 2023 -- Topic: AI tool for podcast teaser creation
- Ning Zheng, University of Notre Dame
- 2023 -- Topic: productized research on LLM + video authoring
- Xingyu (Bruce) Liu, UCLA
- 2022 -- Topic: Tools to Improve Video Accessibility
- Oliver Alonzo, Rochester Institute of Technology
- 2021 -- Topic: Tools for Making Videos Accessible to Deaf and Hard-of-Hearing People
- Next: Assistant Professor @ Creighton University
- Yapeng Tian, University of Rochester
- 2021 -- Topic: Tools for Making Videos Accessible to the Blind and Visually Impaired
- 2019 -- Topic: Deep Audio Prior & Multimodal Video Parsing
- Next: Assistant Professor @ University of Texas at Dallas
- Yujia Wang, Beijing Institute of Technology
- 2020 -- Topic: Automatic Audio Description Synthesis for the Blind and Visually Impaired
- 2019 -- Topic: Automatic Scene-Aware Background Music Synthesis
- Arda Senocak, KAIST
- 2020 -- Topic: Causality Analysis in Audiovisual Understanding
- Henrique Maia, Columbia University,
- 2020 -- Topic: Snooping the GPU and Neural Network via Magnetic Side Channel
- 2018 -- Topic: Invisible Tags in 3D Printing via Layer-by-Layer Nature
- Yang Zhou, UMass Amherst
- 2020 -- Topic: Audio-driven Upper Body Gesture Synthesis
- 2019 -- Topic: Audio-driven Facial and Talking Head Synthesis
- Next: Research Scientist @ Adobe Research
- Zhenyu Tang, University of Maryland,
- 2019 -- Topic: Room Acoustic Analysis with Deep Learning
- Next: Research Scientist @ Tiktok / Bytedance
- Avinash S. Nair, Columbia University
- 2017 -- Topic: Invisible Tags in 3D Printing via Subsurface Scattering
- Next: Cruise
Patents
See most updated list here.Granted:
- Rendering Scene-Aware Audio Using Neural Network-Based Acoustic Analysis. Filed in 2019, Granted in 2021. [link]
- Style-Aware Audio-Driven Talking Head Animation From a Single Image. Filed in 2020, Granted in 2022. [link]
- Segmentation and Hierarchical Clustering of Video. Filed in 2020, Granted in 2022. [link]
Filed:
- Re-Timing a Video Sequence to an Audio Sequence Based on Motion and Audio Beat Detection. Filed in 2021
- Thumbnail Video Segmentation Identifying Thumbnail Locations for a Video. Filed in 2020
- Interacting With Hierarchical Clusters of Video Segments Using a Metadata Search. Filed in 2020
- Refining Image Acquisition Data Through Domain Adaptation. Filed in 2020
- Snap Point Video Segmentation Identifying Selection Snap Points for a Video. Filed in 2020
- Interacting With Semantic Video Segments Through Interactive Tiles. Filed in 2020
- Selecting and Performing Operations on Hierarchical Clusters of Video Segments. Filed in 2020
- Interacting With Hierarchical Clusters of Video Segments Using a Metadata Panel. Filed in 2020
- Interacting With Hierarchical Clusters of Video Segments Using a Video Timeline. Filed in 2020
- Hierarchical Segmentation of Screen Captured, Screencasted, or Streamed Video. Filed in 2020
- Hierarchical Segmentation Based on Voice-Activity. Filed in 2020