Game-Weighted Evidence for Interactive Rendering: A Reanalysis of Released Atari Scores

Authors

  • Oscar Nelson Author
  • Mason Peterson Author
  • Austin Barnes Author

Abstract

A benchmark average can change simply because some games contribute more runs. In two released Bigger, Better, Faster score files, equal weighting of 55 games lowers the main configuration's mean normalized score from 1.832 to 1.484. A game-balanced comparison on a compressed score scale still favors that configuration, with an estimated difference of 0.466 and a game-bootstrap interval of 0.322 to 0.627. These are secondary calculations on published agent results. They motivate a practical distinction for interactive rendering research: report visual quality, control performance, and the distribution of evaluated environments separately.

References

  1. Meng, Zijie, et al. "Make a game: A novel paradigm for interactive game rendering." ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2026.
  2. White, Halbert. “A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity.” Econometrica, vol. 48, no. 4, 1980, pp. 817–838. https://doi.org/10.2307/1912934.
  3. Cameron, A. Colin, and Douglas L. Miller. “A Practitioner’s Guide to Cluster-Robust Inference.” Journal of Human Resources, vol. 50, no. 2, 2015, pp. 317–372. https://doi.org/10.3368/jhr.50.2.317.
  4. Greenland, Sander, Judea Pearl, and James M. Robins. “Causal Diagrams for Epidemiologic Research.” Epidemiology, vol. 10, no. 1, 1999, pp. 37-48. https://doi.org/10.1097/00001648-199901000-00008.
  5. Benjamini, Yoav and Yosef Hochberg. “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.” Journal of the Royal Statistical Society Series B: Statistical Methodology, vol. 57, no. 1, 1995, pp. 289-300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x.
  6. Roberts, David R., et al. “Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure.” Ecography, vol. 40, no. 8, 2017, pp. 913-929. https://doi.org/10.1111/ecog.02881.
  7. Varma, Sudhir and Richard Simon. “Bias in error estimation when using cross-validation for model selection.” BMC Bioinformatics, vol. 7, no. 1, 2006. https://doi.org/10.1186/1471-2105-7-91.
  8. Wilkinson, Mark D., et al. "The FAIR Guiding Principles for Scientific Data Management and Stewardship." Scientific Data, vol. 3, 2016, article 160018. https://doi.org/10.1038/sdata.2016.18.
  9. Sculley, D., et al. "Hidden Technical Debt in Machine Learning Systems." Advances in Neural Information Processing Systems, vol. 28, 2015. https://papers.nips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
  10. Bland, J. Martin, and Douglas G. Altman. "Statistical Methods for Assessing Agreement between Two Methods of Clinical Measurement." The Lancet, vol. 327, no. 8476, 1986, pp. 307-310. https://doi.org/10.1016/S0140-6736(86)90837-8.
  11. Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. “On Calibration of Modern Neural Networks.” Proceedings of the 34th International Conference on Machine Learning, PMLR, vol. 70, 2017, pp. 1321–1330. https://proceedings.mlr.press/v70/guo17a.html.
  12. Geifman, Yonatan, and Ran El-Yaniv. “Selective Classification for Deep Neural Networks.” Advances in Neural Information Processing Systems, vol. 30, 2017. https://arxiv.org/abs/1705.08500.
  13. Ovadia, Yaniv, et al. “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift.” Advances in Neural Information Processing Systems, vol. 32, 2019. https://arxiv.org/abs/1906.02530.
  14. Bellemare, Marc G., et al.. "The Arcade Learning Environment: An Evaluation Platform for General Agents" arXiv preprint arXiv:1207.4708, 2012. https://arxiv.org/abs/1207.4708.
  15. Agarwal, Rishabh, et al.. "Deep Reinforcement Learning at the Edge of the Statistical Precipice" arXiv preprint arXiv:2108.13264, 2021. https://arxiv.org/abs/2108.13264.
  16. Cobbe, Karl, et al.. "Leveraging Procedural Generation to Benchmark Reinforcement Learning" arXiv preprint arXiv:1912.01588, 2019. https://arxiv.org/abs/1912.01588.
  17. Brockman, Greg, et al.. "OpenAI Gym" arXiv preprint arXiv:1606.01540, 2016. https://arxiv.org/abs/1606.01540.
  18. Ha, David and Jürgen Schmidhuber. "World Models" arXiv preprint arXiv:1803.10122, 2018. https://arxiv.org/abs/1803.10122.
  19. Hafner, Danijar, et al.. "Dream to Control: Learning Behaviors by Latent Imagination" arXiv preprint arXiv:1912.01603, 2019. https://arxiv.org/abs/1912.01603.
  20. Micheli, Vincent, et al.. "Transformers are Sample-Efficient World Models" arXiv preprint arXiv:2209.00588, 2022. https://arxiv.org/abs/2209.00588.
  21. Alonso, Eloi, et al.. "Diffusion for World Modeling: Visual Details Matter in Atari" arXiv preprint arXiv:2405.12399, 2024. https://arxiv.org/abs/2405.12399.
  22. Valevski, Dani, et al.. "Diffusion Models Are Real-Time Game Engines" arXiv preprint arXiv:2408.14837, 2024. https://arxiv.org/abs/2408.14837.
  23. Bruce, Jake, et al.. "Genie: Generative Interactive Environments" arXiv preprint arXiv:2402.15391, 2024. https://arxiv.org/abs/2402.15391.
  24. Zhang, Richard, et al.. "The Unreasonable Effectiveness of Deep Features as a Perceptual Metric" arXiv preprint arXiv:1801.03924, 2018. https://arxiv.org/abs/1801.03924.
  25. Unterthiner, Thomas, et al.. "Towards Accurate Generative Models of Video: A New Metric & Challenges" arXiv preprint arXiv:1812.01717, 2018. https://arxiv.org/abs/1812.01717.
  26. Wang, Zhou, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. "Image Quality Assessment: From Error Visibility to Structural Similarity." IEEE Transactions on Image Processing, vol. 13, no. 4, 2004, pp. 600-612. https://doi.org/10.1109/TIP.2003.819861.
  27. Schwarzer, Max, et al.. "Bigger, Better, Faster: Human-level Atari with human-level efficiency" arXiv preprint arXiv:2305.19452, 2023. https://arxiv.org/abs/2305.19452.
  28. Sutton, Richard S., and Andrew G. Barto. Reinforcement Learning: An Introduction. 2nd ed., MIT Press, 2018. http://incompleteideas.net/book/the-book-2nd.html.
  29. Levine, Sergey, Aviral Kumar, George Tucker, and Justin Fu. “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems.” arXiv, 2020, arXiv:2005.01643. https://arxiv.org/abs/2005.01643.
  30. Kumar, Aviral, Aurick Zhou, George Tucker, and Sergey Levine. “Conservative Q-Learning for Offline Reinforcement Learning.” Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 1179–1191. https://arxiv.org/abs/2006.04779.

Downloads

Published

2026-08-03

Issue

Vol. 2 (2026): Archive issue of 2026-08-03

Section

Articles

License

Copyright (c) 2026 Oscar Nelson, Mason Peterson, Austin Barnes

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.