Sampling Conditions and Paired Judgments in Released Midjourney Scores

Authors

  • Adrian Bryant Author
  • Christian Crawford Author
  • Owen Thompson Author

Abstract

The two released Midjourney condition labels in AGIQA-3K differ strongly in their paired human quality scores. Across 296 shared prompts, lowstep minus normal averages -1.239 quality units and -0.495 alignment units. Quality decreases in 287 pairs and increases in nine. These are descriptive comparisons of published rows, not a new intervention on a current generation service. The analysis makes the prompt pairing and incomplete table support explicit while keeping perceptual quality and instruction alignment as separate observed outcomes.

References

  1. Meng, Zijie. "Decoupling Semantics from Distortions: Multi-Scale Two-Stream Vision-Language Alignment for AI-Generated Image Quality Assessment." arXiv preprint arXiv:2606.16799 (2026).
  2. White, Halbert. “A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity.” Econometrica, vol. 48, no. 4, 1980, pp. 817–838. https://doi.org/10.2307/1912934.
  3. Cameron, A. Colin, and Douglas L. Miller. “A Practitioner’s Guide to Cluster-Robust Inference.” Journal of Human Resources, vol. 50, no. 2, 2015, pp. 317–372. https://doi.org/10.3368/jhr.50.2.317.
  4. Greenland, Sander, Judea Pearl, and James M. Robins. “Causal Diagrams for Epidemiologic Research.” Epidemiology, vol. 10, no. 1, 1999, pp. 37-48. https://doi.org/10.1097/00001648-199901000-00008.
  5. Benjamini, Yoav and Yosef Hochberg. “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.” Journal of the Royal Statistical Society Series B: Statistical Methodology, vol. 57, no. 1, 1995, pp. 289-300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x.
  6. Roberts, David R., et al. “Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure.” Ecography, vol. 40, no. 8, 2017, pp. 913-929. https://doi.org/10.1111/ecog.02881.
  7. Varma, Sudhir and Richard Simon. “Bias in error estimation when using cross-validation for model selection.” BMC Bioinformatics, vol. 7, no. 1, 2006. https://doi.org/10.1186/1471-2105-7-91.
  8. Wilkinson, Mark D., et al. "The FAIR Guiding Principles for Scientific Data Management and Stewardship." Scientific Data, vol. 3, 2016, article 160018. https://doi.org/10.1038/sdata.2016.18.
  9. Sculley, D., et al. "Hidden Technical Debt in Machine Learning Systems." Advances in Neural Information Processing Systems, vol. 28, 2015. https://papers.nips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
  10. Bland, J. Martin, and Douglas G. Altman. "Statistical Methods for Assessing Agreement between Two Methods of Clinical Measurement." The Lancet, vol. 327, no. 8476, 1986, pp. 307-310. https://doi.org/10.1016/S0140-6736(86)90837-8.
  11. Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. “On Calibration of Modern Neural Networks.” Proceedings of the 34th International Conference on Machine Learning, PMLR, vol. 70, 2017, pp. 1321–1330. https://proceedings.mlr.press/v70/guo17a.html.
  12. Geifman, Yonatan, and Ran El-Yaniv. “Selective Classification for Deep Neural Networks.” Advances in Neural Information Processing Systems, vol. 30, 2017. https://arxiv.org/abs/1705.08500.
  13. Ovadia, Yaniv, et al. “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift.” Advances in Neural Information Processing Systems, vol. 32, 2019. https://arxiv.org/abs/1906.02530.
  14. Li, Chunyi, et al.. "AGIQA-3K: An Open Database for AI-Generated Image Quality Assessment" arXiv preprint arXiv:2306.04717, 2023. https://arxiv.org/abs/2306.04717.
  15. Radford, Alec, et al.. "Learning Transferable Visual Models From Natural Language Supervision" arXiv preprint arXiv:2103.00020, 2021. https://arxiv.org/abs/2103.00020.
  16. Ke, Junjie, et al.. "MUSIQ: Multi-scale Image Quality Transformer" arXiv preprint arXiv:2108.05997, 2021. https://arxiv.org/abs/2108.05997.
  17. Hosu, Vlad, et al.. "KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessment" arXiv preprint arXiv:1910.06180, 2019. https://arxiv.org/abs/1910.06180.
  18. Lin, Hanhe, et al.. "KADID-10k: A Large-scale Artificially Distorted IQA Database" 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX), 2019. https://doi.org/10.1109/qomex.2019.8743252.
  19. Wang, Zhou, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. "Image Quality Assessment: From Error Visibility to Structural Similarity." IEEE Transactions on Image Processing, vol. 13, no. 4, 2004, pp. 600-612. https://doi.org/10.1109/TIP.2003.819861.
  20. Zhang, Richard, et al.. "The Unreasonable Effectiveness of Deep Features as a Perceptual Metric" arXiv preprint arXiv:1801.03924, 2018. https://arxiv.org/abs/1801.03924.
  21. Mittal, A., et al.. "Making a “Completely Blind” Image Quality Analyzer" IEEE Signal Processing Letters, 2013. https://doi.org/10.1109/lsp.2012.2227726.
  22. Talebi, Hossein and Peyman Milanfar. "NIMA: Neural Image Assessment" arXiv preprint arXiv:1709.05424, 2017. https://arxiv.org/abs/1709.05424.
  23. Kirstain, Yuval, et al.. "Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation" arXiv preprint arXiv:2305.01569, 2023. https://arxiv.org/abs/2305.01569.
  24. Wu, Xiaoshi, et al.. "Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis" arXiv preprint arXiv:2306.09341, 2023. https://arxiv.org/abs/2306.09341.
  25. Caron, Mathilde, et al.. "Emerging Properties in Self-Supervised Vision Transformers" arXiv preprint arXiv:2104.14294, 2021. https://arxiv.org/abs/2104.14294.
  26. Ruiz, Nataniel, et al.. "DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation" arXiv preprint arXiv:2208.12242, 2022. https://arxiv.org/abs/2208.12242.
  27. Ye, Hu, et al.. "IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models" arXiv preprint arXiv:2308.06721, 2023. https://arxiv.org/abs/2308.06721.
  28. Heusel, Martin, et al.. "GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium" arXiv preprint arXiv:1706.08500, 2017. https://arxiv.org/abs/1706.08500.
  29. Gebru, Timnit, et al.. "Datasheets for datasets" Communications of the ACM, 2021. https://doi.org/10.1145/3458723.
  30. Pedregosa, Fabian, et al. "Scikit-learn: Machine Learning in Python." Journal of Machine Learning Research, vol. 12, 2011, pp. 2825-2830. https://jmlr.org/papers/v12/pedregosa11a.html.

Downloads

Published

2026-09-17

Issue

Vol. 2 (2026): Archive issue of 2026-09-17

Section

Articles

License

Copyright (c) 2026 Adrian Bryant, Christian Crawford, Owen Thompson

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.