Journal of Evidence and Innovation
Original publication date: 2026-10-06
Archive migration: 2026-10-30
Zachary Sullivan — Suriname; Bruce Powell — Congo; Joseph Duncan — Congo
One generated clip cannot reveal how reliably a method preserves a subject across repeated attempts. This article proposes a repeated-sampling design that separates variation over subjects, instructions, and generation draws. ARGUS supplies the motivating reference-conditioned task. No repeated generations are performed. The analysis plan retains failures and selection effort, reports uncertainty at the appropriate level, and treats favorable best-of-many outputs as evidence about a search workflow rather than as an unqualified description of ordinary single-attempt behavior.
Stochastic generation can produce different identity outcomes for the same subject and instruction. A favorable example may therefore be a typical result, a rare success, or the product of extensive selection. Without an attempt record, these possibilities are difficult to distinguish. Evaluation needs repeated observations and a declared sampling plan if it aims to describe reliability rather than simply demonstrate that one acceptable output can be produced under some run conditions.
ARGUS uses structured reference evidence for subject-preserving video (Meng et al., 2026b). That setting motivates repeated-sampling evaluation, but the present article does not estimate its generation variance. It proposes a study design and reports no new scores. The question is how to attribute observed variation to the sampled subjects, instructions, reference sets, and generation draws, while keeping the interpretation of a single attempt separate from the performance of a selection-assisted workflow.
Identity evidence has a sampling structure before any recognition score is calculated. LFW defines a particular pair-verification protocol (Huang et al., 2008), and the documented scikit-learn loader makes its prescribed files accessible through a reproducible software interface (Pedregosa et al., 2011). Neither source turns a pair count into a count of independent subjects or images. Dataset documentation should identify how references, identities, comparisons, and splits were assembled (Gebru et al., 2021). For generated video, the analogous record should distinguish a new person from another reference photograph, another trajectory, or another frame. Each adds evidence along a different dimension. A transparent report should preserve those links rather than flattening every observation into one score table. The resulting metadata can support a useful audit even when no images or generated clips are available. Such an audit describes coverage and organization; it does not itself establish recognition accuracy, demographic representativeness, or identity preservation in an unobserved video.
A proposed experiment would fix a set of tasks and declare the number of attempts per task before generation. It would record random-state information where the interface exposes it, together with model versions and all relevant sampling settings. A repeated seed is not necessarily a guarantee of identical output across changed environments, so reproducibility should refer to the recorded software conditions. Failed and incomplete requests should remain in the attempt table rather than vanish before scoring begins.
Each attempt would receive separate outcomes for recognizable identity, instruction execution, temporal quality, and assessment coverage. Continuous identity scores could be retained alongside declared decisions. This permits a distinction between numerical fluctuation and consequential failure. If the evaluator is itself stochastic or unstable, repeated measurement of fixed clips can examine that additional source of variation. It should not be confused with generation variation, since the two call for different controls and potential remedies.
The sampling hierarchy would link draws to subject, instruction, reference set, and any repeated experimental condition. Variance summaries should state which factors are fixed and which are sampled. Many draws from a few easy tasks can precisely describe those tasks while providing limited evidence about new subjects. A study should therefore allocate effort according to its intended generalization claim rather than maximize the raw number of scored frames or pairwise comparisons among outputs.
A large collection of measurements does not guarantee a large collection of independent cases. Repeated observations can share unrecorded acquisition conditions or a common source, making row-level uncertainty misleading. White's covariance estimator addresses heteroskedasticity within a specified regression (White, 1980), while cluster methods address a declared form of grouped dependence (Cameron and Miller, 2015). Neither approach repairs a poorly defined outcome or establishes representativeness of the observed sample. A report should identify the number of groups, their relative sizes, and the reason for selecting the grouping level. When only a few independent groups are available, apparently precise numerical intervals warrant particular caution. This is a limitation of the evidence structure, rather than a reason to count correlated measurements as fresh observations.
Discriminative identity representations address a different question from the design of the test population. FaceNet learns an embedding for verification and related recognition tasks (Schroff et al., 2015). ArcFace (Deng et al., 2018) and CosFace (Wang et al., 2018) develop margin-based approaches to discriminative face learning. These methods do not determine which reference views should be supplied to a generator. Dataset choices also alter the challenge: VGGFace2 emphasizes pose and age variation (Cao et al., 2017), MegaFace increases the scale of recognition distractors (Kemelmacher-Shlizerman et al., 2015), and FairFace studies representation across annotated demographic groups (Kärkkäinen and Joo, 2019). Their relevance is to separate distinct kinds of coverage. Many comparisons of a small reference set cannot replace missing pose variation or a different population of people. A study should state which identity measurement it uses, how thresholds are selected, and what the represented cases actually cover. Strong performance on one protocol does not establish universal validity of the embedding across generated or stylized subjects.
The protected assessment population should be defined before investigators inspect detailed failures. Once a collection of cases guides repeated revisions, it has become development information even if its filenames still carry an evaluation label. Cross-validation for dependent data treats the grouping of observations as a substantive design decision (Roberts et al., 2017). Research on model selection likewise shows why tuning and final error estimation should not reuse the same information (Varma and Simon, 2006). A practical record therefore distinguishes fitting data, diagnostic examples, validation data, and the final external assessment. Where only a fixed public table is available, a descriptive analysis remains legitimate, but it should not be renamed a held-out prediction experiment without generating and assessing predictions under a protected split.
An unselected-attempt result and a best-of-many result describe different workflows. The latter can be useful when users genuinely inspect several candidates, but its selection rule and effort should be included. Choosing the most favorable clip using the same identity metric later reported can create an optimistic view of independent validity. A separate endpoint or blinded human assessment can help determine whether score-based selection improves recognizable preservation rather than only the selected instrument's numerical output.
The analysis could show the distribution of task-level outcomes and the relationship between additional attempts and accepted outputs. It should avoid implying that repeated attempts provide unlimited improvement or that a small observed sample identifies a universal variance ceiling. Confidence intervals would reflect the declared sampling design and dependence among draws. Where the number of independent subjects is small, that limitation should remain visible even if the number of generated clips is large.
Comparisons should use matched attempt budgets or explicitly model the practical difference in search opportunity. They should also retain tasks for which all attempts fail or remain unassessable. Excluding those tasks would condition the comparison on having already found a successful generation. A transparent report can then distinguish average single-attempt performance, variability across attempts, and the cost of obtaining a usable result, each of which answers a useful but different question about the workflow's reliability.
A subject-generation pipeline can succeed at using a reference without succeeding at every identity requirement. DreamBooth's model adaptation (Ruiz et al., 2022) and IP-Adapter's image prompting (Ye et al., 2023) offer examples of reference use with different costs and information access. CLIP (Radford et al., 2021) and DINO (Caron et al., 2021) provide further visual representations whose similarity scores may emphasize other aspects of content. The evaluation should therefore define its target before choosing a convenient feature. If the intended target is a recognizable moving person, that target needs appropriate observations across the relevant views and expressions. A high generic image similarity can instead reflect scene layout or texture. Matching reference budgets and documenting the evaluator's representation permit those explanations to be compared. The objective is to connect a claimed conditioning advantage to the particular identity evidence the resulting output actually preserves.
A proposed explanation should be distinguishable from alternative explanations using observations the study can actually collect. Diagramming the relationships among selection, exposure, measurement, and outcome makes that requirement explicit (Greenland et al., 1999). The diagram does not establish its own correctness, but it prevents adjustment choices from remaining invisible. The same transparency is needed when investigators inspect many endpoints before choosing a narrative. Controlling false discovery concerns the behavior of a specified testing family (Benjamini and Hochberg, 1995); it is not a general certificate of scientific validity. Primary endpoints, secondary descriptions, and future mechanism tests should be named separately. Readers can then assess whether an attractive pattern was predicted, explored, or merely noticed after the available comparisons were examined.
Replacing an expensive judgment with an automated score changes the measurement process. A positive correlation is evidence of co-movement, but it does not establish that the two procedures can be used interchangeably (Bland and Altman, 1986). A calibrated forecast has a different requirement: its stated probabilities should match relevant observed frequencies (Guo et al., 2017). Neither property follows merely from a favorable overall ranking. A useful comparison preserves paired measurements, identifies the conditions in which disagreement matters, and keeps scale transformations explicit. If the intended judgment concerns recognizability, playability, or physical compatibility, the proxy should be assessed against that judgment directly. Agreement on easy cases cannot supply evidence about the difficult conditions absent from the comparison.
Subject preservation is one dimension of a larger video-generation task. VBench explicitly motivates separating evaluation dimensions (Huang et al., 2023), and the visual literature offers structural (Wang et al., 2004), perceptual (Zhang et al., 2018), and temporal-distribution measurements (Unterthiner et al., 2018). A study should choose among them according to the claim and available reference information. When identity, motion, and visual quality move in different directions, the disagreement should be analyzed rather than automatically resolved by a weighted total. The weighting itself expresses a preference that may differ across applications. Preserving per-frame and per-clip records allows alternative interpretations to be evaluated later. It also makes coverage visible when some poses or occlusions prevent valid scoring. A generator can demonstrate a useful, limited preservation capability without having established universal reliability across all subjects, durations, and visual conditions.
A system's reliable operating region may be narrower than the collection of inputs an interface accepts. Selective prediction separates performance on accepted cases from the fraction of cases accepted (Geifman and El-Yaniv, 2017). Reporting only the former can hide a substantial loss of coverage. Uncertainty estimates may also change under distribution shift (Ovadia et al., 2019), so familiar-condition confidence is not automatically evidence about new subjects, scenes, or control histories. A useful evaluation records rejected, ambiguous, and unsupported cases alongside successful ones. This makes it possible to distinguish limited but honest capability from a system that answers broadly with unreliable certainty. Any proposed deferral rule still requires assessment on its intended input population; it is not validated merely by being described.
The unit of provenance should be fine enough to reconstruct an important disagreement. Reusable identifiers can connect a reported score to its input evidence and the relevant model and evaluator versions (Wilkinson et al., 2016). Pipeline dependencies should be treated as part of the experiment because apparently minor changes can alter output eligibility or metric behavior (Sculley et al., 2015). A report that preserves only aggregate numbers prevents this reconstruction. Keeping per-case results and documented transformations enables alternative weighting and error analysis without repeating expensive generation. Such a record also helps identify whether two papers compare the same scientific objects or merely use similar names for different releases. Versioned evidence makes progress easier to assess across successive evaluations.
Repeated-sampling evaluation needs complete attempt records and a hierarchy that separates subjects, tasks, references, and draws. The proposed design reports no new variance estimate. It makes reliability and selection-assisted performance distinct, while retaining failures and uncertainty. Future preservation results can then describe how often a method succeeds under a declared budget rather than treating one favorable clip or a best-of-many score as a complete account of ordinary behavior.