Scantalk: Talking Heads from Unregistered Scans


Federico Nocentini1 *    Thomas Besnier2 *    Claudio Ferrari3    Sylvain Arguillère4    Mohamed Daoudi2,5    Stefano Berretti1   

1Media Integration and Communication Center (MICC), University of Florence, Italy   

2Univ. Lille, CNRS, Centrale Lille, UMR 9189 CRIStAL, F-59000 Lille, France   

3Department of Architecture and Engineering, University of Parma, Italy   

4Univ. Lille, CNRS, UMR 8524 Laboratoire Paul Painlevé, Lille, F-59000, France   

5IMT Nord Europe, Institut Mines-Télécom, Centre for Digital Systems   

arXiv Code
Model architecture

Abstract

Speech-driven 3D talking heads generation has emerged as a significant area of interest among researchers, presenting numerous challenges. Existing methods are constrained by animating faces with fixed topologies, wherein point-wise correspondence is established, and the number and order of points remains consistent across all identities the model can animate. In this work, we present ScanTalk, a novel framework capable of animating 3D faces in arbitrary topologies including scanned data. Our approach relies on the DiffusionNet architecture to overcome the fixed topology constraint, offering promising avenues for more flexible and realistic 3D animations. By leveraging the power of DiffusionNet, ScanTalk not only adapts to diverse facial structures but also maintains fidelity when dealing with scanned data, thereby enhancing the authenticity and versatility of generated 3D talking heads. Through comprehensive comparisons with state-of-the-art methods, we validate the efficacy of our approach, demonstrating its capacity to generate realistic talking heads comparable to existing techniques. While our primary objective is to develop a generic method free from topological constraints, all state-of-the-art methodologies are bound by such limitations.

Method

Model architecture

Examples

Paper and supplementary material

Paper

F. Nocentini, T. Besnier, C. Ferrari, S. Arguillère, M. Daoudi, S. Berretti
Scantalk: Talking Heads from Unregistered Scans

Paper (supplementary at the end) | BibTeX 

@misc{nocentini2024scantalk,
      title={ScanTalk: 3D Talking Heads from Unregistered Scans},
      author={Federico Nocentini and Thomas Besnier and Claudio Ferrari and Sylvain Arguillere and Stefano Berretti and Mohamed Daoudi},
      year={2024},
      eprint={2403.10942},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Acknowledgements

This work is supported by the ANR project Human4D ANR-19-CE23-0020 and by the IRP CNRS project GeoGen3DHuman . This work was also partially supported by ``Partneariato FAIR (Future Artificial Intelligence Research) - PE00000013, CUP J33C22002830006" funded by NextGenerationEU through the italian MUR within NRRP, project DL-MIG.