SemiUHPE: Semi-Supervised Unconstrained Head Pose Estimation in the Wild
arXiv 2024.04 (under review)
✅ The used Body-Head Joint Detector is our previous project BPJDet.
✅ SemiUHPE is much more superior than our previous project DirectMHP.

  • Huayi Zhou
    Shanghai Jiao Tong University (SJTU)
  • Fei Jiang
    Chongqing Academy of Science and Technology
  • Hongtao Lu
    Shanghai Jiao Tong University (SJTU)
overview

The unconstrained head pose estimation results of our SemiUHPE on wild challenging heads (e.g., heavy blur, extreme illumination, severe occlusion, atypical pose, and invisible face).

Short Phone Video 1 Short Phone Video 2 Short Phone Video 3

Abstract

overview

Existing head pose estimation datasets are either composed of numerous samples by non-realistic synthesis or lab collection, or limited images by labor-intensive annotating. This makes deep supervised learning based solutions compromised due to the reliance on generous labeled data. To alleviate it, we propose the first semi-supervised unconstrained head pose estimation (SemiUHPE) method, which can leverage a large amount of unlabeled wild head images. Specifically, we follow the recent semi-supervised rotation regression, and focus on the diverse and complex head pose domain. Firstly, we claim that the aspect-ratio invariant cropping of heads is superior to the previous landmark-based affine alignment, which does not fit unlabeled natural heads or practical applications where landmarks are often unavailable. Then, instead of using an empirically fixed threshold to filter out pseudo labels, we propose the dynamic entropy-based filtering by updating thresholds for adaptively removing unlabeled outliers. Moreover, we revisit the design of weak-strong augmentations, and further exploit its superiority by devising two novel head-oriented strong augmentations named pose-irrelevant cut-occlusion and pose-altering rotation consistency. Extensive experiments show that SemiUHPE can surpass SOTAs with remarkable improvements on public benchmarks under both front-range and full-range.

SemiUHPE vs. previous SOTAs

overview

Visualizations of front-range or full-range head pose estimation (HPE). Our method SemiUHPE is more robust to severe occlusion, atypical pose and invisible face.

overview

Qualitative results of our method (3rd line) and DAD-3DNet (2nd line) on heads from DAD-3DHeads test-set (1st line), which never appeared during SSL training.


More Quantitative Results of SemiUHPE

Real Classrooms.

Recorded Vlogs.

Citation

Acknowledgements

We acknowledge the effort from authors of human-related datasets including CrowdHuman, COCOHumanParts and DAD-3DHeads. These datasets make researches and downstream applications about Semi-Supervised Unconstrained Head Pose Estimation in the Wild possible.

The website template was borrowed from Jon Barron and Zip-NeRF.