Back

Disclaimer: These are my personal notes on this paper. I am in no way related to this paper. All credits go towards the authors.

DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection

June 18, 2020 - Paper Link - Tags: Deepfake, Detection, Survey

Summary

Survey of various face manipulation methods including deepfakes.

Notes

• Paper covers four areas: entire face synthesis (completely generated faces), identity swap (deepfakes), attribute manipulation (change something, ex, age), and expression swap (source-target expression swap).
• Entire Face Synthesis
• Each fake image contains a specific GAN fingerprint, just like natural images can be identified by a device-based fingerprint (PRNU). The fingerprint spans different architectures and different instances. [ 1, 2, 3]
• Neves et al. presented the iFakeFaceDB database, consisting of fake images as well as images that had their GAN fingerprints removed via GANprintR (GAN fingerprint Removal). Figure 2 shows an example.
• Table 2 summarizes the detection approaches for face synthesis
• Identity Swap (Deepfakes)
• Datasets
• Table 3 summarizes identity swap datasets that are public available.
• The FaceForensics++ is considered a first generation dataset (because it looks fake to be honest).
• Google added DeepFakeDetection to the FaceForensics++ database. They look a lot better, however, Google did not release the corresponding models.
• Detection Methods
• Table 4 highlights the results. Deep learning features, Image + Temporal features, and Facial Regions features performed the best.
• Inconsistencies between lip movements and audio speech
• Eye color, missing reflections, and missing details in the eye and teeth areas
• Head poses
• Facial expressions + head movements
• Eye blinking
• Artifact detection
• Mesoscopic properties of the image
• Classifier + steganalysis features with triplet loss over SVM classification
• XceptionNet
• Capsule networks
• Spatial and motion information based on 3DCNN
• Temporal-aware pipeline consisting of a CNN and a RNN
• Temporal discrepancies across frames
• Detection Methods Summary
• All show poor generalisation results to unseen databases
• Poor results on the second generation deepFake databases

Interesting References

• From Faceforensics++: compression and resizing operations are generally automatically used on social networks.

Citation: Tolosana, Ruben, et al. "Deepfakes and beyond: A survey of face manipulation and fake detection." arXiv preprint arXiv:2001.00179 (2020).