Designing Human Comparisons for Subject Consistency

Authors

  • Austin Warren Author
  • Simon Collins Author
  • Jeffrey Mason Author

Abstract

Human assessment of subject consistency is sensitive to the references, questions, and viewing conditions presented to observers. A preference for an attractive clip may not answer whether it preserves a particular person. This conceptual article develops a comparative study in which identity, instruction execution, and general appearance receive separate judgments. It uses ARGUS to motivate evaluation with multiple reference views. No participants are recruited and no preference rates are reported. The proposed design makes a human comparison answer a specific preservation question.

References

  1. Meng, Zijie, et al. "Argus: Stacked multi-view identity mosaic injection for subject-preserving video generation." arXiv preprint arXiv:2606.11670 (2026).
  2. White, Halbert. “A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity.” Econometrica, vol. 48, no. 4, 1980, pp. 817–838. https://doi.org/10.2307/1912934.
  3. Cameron, A. Colin, and Douglas L. Miller. “A Practitioner’s Guide to Cluster-Robust Inference.” Journal of Human Resources, vol. 50, no. 2, 2015, pp. 317–372. https://doi.org/10.3368/jhr.50.2.317.
  4. Greenland, Sander, Judea Pearl, and James M. Robins. “Causal Diagrams for Epidemiologic Research.” Epidemiology, vol. 10, no. 1, 1999, pp. 37-48. https://doi.org/10.1097/00001648-199901000-00008.
  5. Benjamini, Yoav and Yosef Hochberg. “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.” Journal of the Royal Statistical Society Series B: Statistical Methodology, vol. 57, no. 1, 1995, pp. 289-300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x.
  6. Roberts, David R., et al. “Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure.” Ecography, vol. 40, no. 8, 2017, pp. 913-929. https://doi.org/10.1111/ecog.02881.
  7. Varma, Sudhir and Richard Simon. “Bias in error estimation when using cross-validation for model selection.” BMC Bioinformatics, vol. 7, no. 1, 2006. https://doi.org/10.1186/1471-2105-7-91.
  8. Wilkinson, Mark D., et al. "The FAIR Guiding Principles for Scientific Data Management and Stewardship." Scientific Data, vol. 3, 2016, article 160018. https://doi.org/10.1038/sdata.2016.18.
  9. Sculley, D., et al. "Hidden Technical Debt in Machine Learning Systems." Advances in Neural Information Processing Systems, vol. 28, 2015. https://papers.nips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
  10. Bland, J. Martin, and Douglas G. Altman. "Statistical Methods for Assessing Agreement between Two Methods of Clinical Measurement." The Lancet, vol. 327, no. 8476, 1986, pp. 307-310. https://doi.org/10.1016/S0140-6736(86)90837-8.
  11. Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. “On Calibration of Modern Neural Networks.” Proceedings of the 34th International Conference on Machine Learning, PMLR, vol. 70, 2017, pp. 1321–1330. https://proceedings.mlr.press/v70/guo17a.html.
  12. Geifman, Yonatan, and Ran El-Yaniv. “Selective Classification for Deep Neural Networks.” Advances in Neural Information Processing Systems, vol. 30, 2017. https://arxiv.org/abs/1705.08500.
  13. Ovadia, Yaniv, et al. “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift.” Advances in Neural Information Processing Systems, vol. 32, 2019. https://arxiv.org/abs/1906.02530.
  14. Huang, Gary B., Marwan Mattar, Tamara Berg, and Erik Learned-Miller. "Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments." ECCV Workshop on Faces in Real-Life Images, 2008. https://cs.umass.edu/~elm/papers/Huang_eccv2008-lfw.pdf.
  15. Schroff, Florian, et al.. "FaceNet: A Unified Embedding for Face Recognition and Clustering" arXiv preprint arXiv:1503.03832, 2015. https://arxiv.org/abs/1503.03832.
  16. Deng, Jiankang, et al.. "ArcFace: Additive Angular Margin Loss for Deep Face Recognition" arXiv preprint arXiv:1801.07698, 2018. https://arxiv.org/abs/1801.07698.
  17. Wang, Hao, et al.. "CosFace: Large Margin Cosine Loss for Deep Face Recognition" arXiv preprint arXiv:1801.09414, 2018. https://arxiv.org/abs/1801.09414.
  18. Cao, Qiong, et al.. "VGGFace2: A dataset for recognising faces across pose and age" arXiv preprint arXiv:1710.08092, 2017. https://arxiv.org/abs/1710.08092.
  19. Kemelmacher-Shlizerman, Ira, et al.. "The MegaFace Benchmark: 1 Million Faces for Recognition at Scale" arXiv preprint arXiv:1512.00596, 2015. https://arxiv.org/abs/1512.00596.
  20. Kärkkäinen, Kimmo and Jungseock Joo. "FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age" arXiv preprint arXiv:1908.04913, 2019. https://arxiv.org/abs/1908.04913.
  21. Ruiz, Nataniel, et al.. "DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation" arXiv preprint arXiv:2208.12242, 2022. https://arxiv.org/abs/2208.12242.
  22. Ye, Hu, et al.. "IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models" arXiv preprint arXiv:2308.06721, 2023. https://arxiv.org/abs/2308.06721.
  23. Huang, Ziqi, et al.. "VBench: Comprehensive Benchmark Suite for Video Generative Models" arXiv preprint arXiv:2311.17982, 2023. https://arxiv.org/abs/2311.17982.
  24. Caron, Mathilde, et al.. "Emerging Properties in Self-Supervised Vision Transformers" arXiv preprint arXiv:2104.14294, 2021. https://arxiv.org/abs/2104.14294.
  25. Radford, Alec, et al.. "Learning Transferable Visual Models From Natural Language Supervision" arXiv preprint arXiv:2103.00020, 2021. https://arxiv.org/abs/2103.00020.
  26. Zhang, Richard, et al.. "The Unreasonable Effectiveness of Deep Features as a Perceptual Metric" arXiv preprint arXiv:1801.03924, 2018. https://arxiv.org/abs/1801.03924.
  27. Unterthiner, Thomas, et al.. "Towards Accurate Generative Models of Video: A New Metric & Challenges" arXiv preprint arXiv:1812.01717, 2018. https://arxiv.org/abs/1812.01717.
  28. Wang, Zhou, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. "Image Quality Assessment: From Error Visibility to Structural Similarity." IEEE Transactions on Image Processing, vol. 13, no. 4, 2004, pp. 600-612. https://doi.org/10.1109/TIP.2003.819861.
  29. Gebru, Timnit, et al.. "Datasheets for datasets" Communications of the ACM, 2021. https://doi.org/10.1145/3458723.
  30. Pedregosa, Fabian, et al. "Scikit-learn: Machine Learning in Python." Journal of Machine Learning Research, vol. 12, 2011, pp. 2825-2830. https://jmlr.org/papers/v12/pedregosa11a.html.

Downloads

Published

2026-09-17

Issue

Vol. 2 (2026): Archive issue of 2026-09-17

Section

Articles

License

Copyright (c) 2026 Austin Warren, Simon Collins, Jeffrey Mason

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.