Scene Identity and Leakage in Driving-Video Benchmarks

Authors

  • Adam Pearson Author
  • Jacob Evans Author
  • Wesley Wilson Author

Abstract

Different driving clips can share the same recording, location, objects, or source frames even when their filenames differ. Evaluation separation should therefore be defined at the scene level relevant to the generalization claim. This article proposes a provenance-based leakage audit for multi-view driving generation, motivated by OmniDrive. No training corpus or benchmark is newly searched. The design distinguishes known shared observations from unverified exposure and explains how a clean file split can remain insufficient evidence for transfer to genuinely new driving scenes.

References

  1. Meng, Zijie, et al. "OmniDrive: An LLM-Choreographed Multi-Agent World Model with Unified Latent Co-Compression for Multi-View Driving Video Generation." arXiv preprint arXiv:2606.17536 (2026).
  2. White, Halbert. “A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity.” Econometrica, vol. 48, no. 4, 1980, pp. 817–838. https://doi.org/10.2307/1912934.
  3. Cameron, A. Colin, and Douglas L. Miller. “A Practitioner’s Guide to Cluster-Robust Inference.” Journal of Human Resources, vol. 50, no. 2, 2015, pp. 317–372. https://doi.org/10.3368/jhr.50.2.317.
  4. Greenland, Sander, Judea Pearl, and James M. Robins. “Causal Diagrams for Epidemiologic Research.” Epidemiology, vol. 10, no. 1, 1999, pp. 37-48. https://doi.org/10.1097/00001648-199901000-00008.
  5. Benjamini, Yoav and Yosef Hochberg. “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.” Journal of the Royal Statistical Society Series B: Statistical Methodology, vol. 57, no. 1, 1995, pp. 289-300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x.
  6. Roberts, David R., et al. “Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure.” Ecography, vol. 40, no. 8, 2017, pp. 913-929. https://doi.org/10.1111/ecog.02881.
  7. Varma, Sudhir and Richard Simon. “Bias in error estimation when using cross-validation for model selection.” BMC Bioinformatics, vol. 7, no. 1, 2006. https://doi.org/10.1186/1471-2105-7-91.
  8. Wilkinson, Mark D., et al. "The FAIR Guiding Principles for Scientific Data Management and Stewardship." Scientific Data, vol. 3, 2016, article 160018. https://doi.org/10.1038/sdata.2016.18.
  9. Sculley, D., et al. "Hidden Technical Debt in Machine Learning Systems." Advances in Neural Information Processing Systems, vol. 28, 2015. https://papers.nips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
  10. Bland, J. Martin, and Douglas G. Altman. "Statistical Methods for Assessing Agreement between Two Methods of Clinical Measurement." The Lancet, vol. 327, no. 8476, 1986, pp. 307-310. https://doi.org/10.1016/S0140-6736(86)90837-8.
  11. Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. “On Calibration of Modern Neural Networks.” Proceedings of the 34th International Conference on Machine Learning, PMLR, vol. 70, 2017, pp. 1321–1330. https://proceedings.mlr.press/v70/guo17a.html.
  12. Geifman, Yonatan, and Ran El-Yaniv. “Selective Classification for Deep Neural Networks.” Advances in Neural Information Processing Systems, vol. 30, 2017. https://arxiv.org/abs/1705.08500.
  13. Ovadia, Yaniv, et al. “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift.” Advances in Neural Information Processing Systems, vol. 32, 2019. https://arxiv.org/abs/1906.02530.
  14. Geiger, Andreas, Philip Lenz, and Raquel Urtasun. "Are We Ready for Autonomous Driving? The KITTI Vision Benchmark Suite." 2012 IEEE Conference on Computer Vision and Pattern Recognition, 2012, pp. 3354-3361. https://doi.org/10.1109/CVPR.2012.6248074.
  15. Caesar, Holger, et al.. "nuScenes: A multimodal dataset for autonomous driving" arXiv preprint arXiv:1903.11027, 2019. https://arxiv.org/abs/1903.11027.
  16. Yu, Fisher, et al.. "BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning" arXiv preprint arXiv:1805.04687, 2018. https://arxiv.org/abs/1805.04687.
  17. Sun, Pei, et al.. "Scalability in Perception for Autonomous Driving: Waymo Open Dataset" arXiv preprint arXiv:1912.04838, 2019. https://arxiv.org/abs/1912.04838.
  18. Luiten, Jonathon, et al.. "HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking" arXiv preprint arXiv:2009.07736, 2020. https://arxiv.org/abs/2009.07736.
  19. Zhang, Yifu, et al.. "ByteTrack: Multi-Object Tracking by Associating Every Detection Box" arXiv preprint arXiv:2110.06864, 2021. https://arxiv.org/abs/2110.06864.
  20. Weng, Xinshuo, et al.. "AB3DMOT: A Baseline for 3D Multi-Object Tracking and New Evaluation Metrics" arXiv preprint arXiv:2008.08063, 2020. https://arxiv.org/abs/2008.08063.
  21. Unterthiner, Thomas, et al.. "Towards Accurate Generative Models of Video: A New Metric & Challenges" arXiv preprint arXiv:1812.01717, 2018. https://arxiv.org/abs/1812.01717.
  22. Wang, Zhou, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. "Image Quality Assessment: From Error Visibility to Structural Similarity." IEEE Transactions on Image Processing, vol. 13, no. 4, 2004, pp. 600-612. https://doi.org/10.1109/TIP.2003.819861.
  23. Zhang, Richard, et al.. "The Unreasonable Effectiveness of Deep Features as a Perceptual Metric" arXiv preprint arXiv:1801.03924, 2018. https://arxiv.org/abs/1801.03924.
  24. Hafner, Danijar, et al.. "Dream to Control: Learning Behaviors by Latent Imagination" arXiv preprint arXiv:1912.01603, 2019. https://arxiv.org/abs/1912.01603.
  25. Ha, David and Jürgen Schmidhuber. "World Models" arXiv preprint arXiv:1803.10122, 2018. https://arxiv.org/abs/1803.10122.
  26. Ye, Hu, et al.. "IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models" arXiv preprint arXiv:2308.06721, 2023. https://arxiv.org/abs/2308.06721.
  27. Huang, Ziqi, et al.. "VBench: Comprehensive Benchmark Suite for Video Generative Models" arXiv preprint arXiv:2311.17982, 2023. https://arxiv.org/abs/2311.17982.
  28. Radford, Alec, et al.. "Learning Transferable Visual Models From Natural Language Supervision" arXiv preprint arXiv:2103.00020, 2021. https://arxiv.org/abs/2103.00020.
  29. Gebru, Timnit, et al.. "Datasheets for datasets" Communications of the ACM, 2021. https://doi.org/10.1145/3458723.
  30. Levine, Sergey, Aviral Kumar, George Tucker, and Justin Fu. “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems.” arXiv, 2020, arXiv:2005.01643. https://arxiv.org/abs/2005.01643.

Downloads

Published

2026-10-06

Issue

Vol. 2 (2026): Archive issue of 2026-10-06

Section

Articles

License

Copyright (c) 2026 Adam Pearson, Jacob Evans, Wesley Wilson

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.