Human-AI Agreement in Citation Validation of an AI-Generated Subspecialty Anesthesiology Knowledge Base: Cross-Sectional Paired Validation Study

Introduction

Evaluating citations in a clinical knowledge resource requires checking both the cited article and the information attached to it. We compared AI and human judgments on 570 citation records across 19 subspecialties of a bilingual anesthesiology knowledge base. Earlier studies have assessed reference existence and bibliographic details [1-6], including mismatches in cited content [5,6], or compared citation credibility and accuracy [7]. Statement-support research has examined whether source text supports a claim and compared automated assessments with physicians’ judgments [8]. Scripted synthetic examples have distinguished apparent evidence support from correctness [9]. A guideline study assessed citation relevance, metadata, existence, and publication in peer-reviewed journals [10]. In our resource, each record connects an article to a clinical concept and summaries in English and Chinese. A source may be real yet fail to answer the assigned clinical question, or a summary may alter an outcome, qualification, or translated term. We therefore distinguished article existence, relevance, scientific accuracy, and translation fidelity.

We used OpenEvidence to obtain candidate references and bilingual summaries for this resource. Retrieval-augmented generation supplies retrieved material to a language model, but generated answers can still contain errors [11]. A biomedical citation-recommendation system combines retrieval and reranking [12]. Previous research describes OpenEvidence as a biomedical literature retrieval-augmented generation system [13], although our records did not identify its backend implementation. Clinical evaluations have tested document or online support in neurology [14] and the relevance and quality of evidence in clinical answers [15]. Other tasks include guideline concordance [16], cardiology questions with and without retrieval support [17], treatment plans [18], repeated management recommendations [19], and anesthesia decisions [20]. Physician assessments have also included citation characteristics [21], while a pediatric question study compared input formats [22]. A spine-surgery citation audit tested OpenEvidence version 2.7.12 in October 2025 and scored article existence and bibliographic accuracy [23]. Our citations were generated in 2026, and the OpenEvidence version was not recorded. Evidence-summary evaluations have separately assessed main-point coverage, errors, and discussion of quality and context [24]. These tasks inform our evaluation, but their percentages describe different outputs and rating rules rather than a common benchmark for judgments on this corpus.

We paired overall verdicts from a separate web-enabled AI checking workflow with human subspecialty reviewers’ judgments. Human ratings provided a reference for comparison, not independently established correct answers. We first compared whether AI and human reviewers agreed on retaining a citation, treating PASS and CONDITIONAL as one group and FAIL as the other. We then examined whether both assessors gave exactly the same overall rating: PASS, CONDITIONAL, or FAIL. For example, an AI PASS paired with a human CONDITIONAL counted as agreement in the 2-group comparison but as disagreement in the 3-category comparison. We also compared AI and human ratings for article existence, relevance, scientific accuracy, and translation fidelity wherever both ratings were available. To inform specialist review, we compared 2 AI alert rules: flagging only AI FAIL ratings or flagging both AI CONDITIONAL and FAIL ratings. For each rule, we counted how many citations it flagged and how many citations rated FAIL by human reviewers it left unflagged. We also repeated the AI assessment on a sample of 112 citations and compared the repeat AI ratings with the original AI ratings for the same citations. To support metascience and research integrity, we documented evaluation choices, retained source locations, search events, and recovery records, and released structured ratings for statistical recomputation. Reporting guidance addresses AI tools, prompts, roles, and content checks [25], and raters, sampling, procedures, and statistics in agreement studies [26]. These records can inform specialist checks of source support and the reasons for judgments.

Methods

Study Design, Setting, Temporal Structure, and Units of Analysis

This cross-sectional paired study compared AI and human ratings of citations from the Chinese American Society of Anesthesiology Digital Library, a member-authenticated English-Chinese anesthesiology platform. The knowledge base was organized into 28 subspecialties, each containing clinical topics termed concepts. For each concept, a standardized query to OpenEvidence requested 10 PubMed-indexed papers and a structured synthesis. Each citation record linked an article reference to its clinical concept and summaries in English and Chinese.

We selected 19 of the 28 subspecialties across 4 study-defined tiers and sampled 30 citation records from each, yielding 570 records. For each sampled record, we paired the overall rating from the AI checking workflow with the overall rating from the assigned human reviewer, producing 570 matched AI-human rating pairs. We also compared the paired ratings available for article existence, relevance, scientific accuracy, and translation fidelity. Stage 1 (AI) and stage 2 (human) denote the 2 assessment streams and do not imply that one preceded the other.

Ten stage 1 batches began on 5 local dates between April 28 and May 3, 2026. Eighteen stage 2 forms covering 19 subspecialties were dated between April 10 and June 7, 2026; form dates neither identify scoring starts nor establish a complete scoring window. The ranges overlapped; 5 subspecialties had form dates before stage 1, and scoring starts were unrecorded. Item 5 of the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) checklist (Multimedia Appendix 1) maps these dates and limitations. The 30-per-subspecialty target reflected a feasible volunteer workload, without prospective end point–specific power or precision calculations; sample size was separate from random-start selection. The study was not prospectively registered; no dated design-phase plan documented prespecification of the binary end point or coefficient set before data collection or outcome inspection. Institutional record 2026D000620 documents the ethics determination, not registration or an analysis protocol.

Knowledge-Base Construction, Citation Generation, and Sampling

Concept Generation

From December 2025 to March 2026, the principal investigator (PI) developed subspecialty matrices with Google Gemini, OpenAI ChatGPT, and Anthropic Claude over 3-5 iterative rounds. The 28-subspecialty inventory comprised 1667 concepts. Concept-generation rules required topics to fall within a subspecialty’s 6-module taxonomy and the scope of clinical anesthesiology and to be addressable in peer-reviewed medical literature; basic science without clinical application was excluded. Retrospective PI clarification defined structural completeness as 10 recognizable, nonempty primary citation–summary entries, regardless of numbering or file size. Historical size/marker screens were not equivalent, so no full-library completion rate is reported. This definition concerns structure, not validity. PI review addressed 6-module adherence, broad subspecialty scope, and within-module duplicates, but not concept-by-concept clinical review; verbatim transcripts were not retained.

Citation Generation

A script sequentially submitted the standardized bilingual queries through the investigator’s authenticated OpenEvidence user-facing session and saved outputs locally. Retained records identify neither the automation framework, application programming interface access, nor the provider backend. Sampling-frame files were generated March 20-April 3, 2026; library-wide querying continued through April 18, 2026. Later regeneration in 4 sampled subspecialties left the frozen frame unchanged. With no retained provider version, user-selected mode, or underlying model identifier, we identify the corpus by interface and query period. Multimedia Appendix 2 reports the retained query and deployment metadata.

Citation-Level Sampling

Nineteen of 28 subspecialties were selected across 4 study-defined tiers: all tier 1 high-acuity (4/4) and tier 2 large surgical (5/5), 6 out of 13 tier 3 medium surgical, and 4 out of 6 tier 4 small or cross-disciplinary subspecialties. The historical sampler used 19 retained operational N values that summed to 1166, giving 11,660 nominal requested slots (10 per concept count). These sampling counts neither estimate revision-stage completeness nor identify the highest original file number. Within each subspecialty, k=round(10N/30); a retained random start r from 1 through k generated positions r+jk for j=0 through 29. Five sequences exceeded nominal 10N but identified existing files because original numbering differed from operational N counts. These overruns were distinct from substitutions. The tier 3/4 subspecialty draw cannot be independently reproduced because no seed, script, or time stamp was retained. Multimedia Appendix 2 lists N, k, r, and overruns for all 19 subspecialties. If a planned position failed the operational usability check, selection advanced to the next usable concept, retaining the citation slot. Retained records and revision-stage source-file readback identified 57 forward substitutions: 1 missing file, 53 outputs with fewer than 10 primary citation–summary entries, and 3 with 10 entries whose numbering of the historical check was deemed ineligible. Frozen replacements were not reversed; no backward or wraparound fallback occurred. Rounded intervals, uncompressed original file numbers, and forward-only substitutions produced unequal selection probabilities.

AI Citation-Validation Workflow and Tool-Use Audit

Model and Access

Each of the 570 citations received one final AI verdict from a web-enabled command-line workflow. All frozen production records from April 28 to May 3, 2026, recorded the requested/selected identifier claude-opus-4-7, corresponding to Anthropic Claude Opus 4.7 [27]. Without a retained interface version or provider-reported runtime identity, this documents the configuration, not an independently verified backend identity.

Production Schedule and Calibration

Production comprised nine 60-citation batches and one 30-citation batch; 4 batches totaling 210 citations began May 3, 2026. A separate 10-citation AI-only calibration run on April 27, 2026, lacked human ratings, was excluded from the 570 pairs, and yielded neither an agreement statistic nor a calibration threshold.

Rubric

The workflow assessed existence, relevance, scientific accuracy, and translation fidelity. AI criterion 1 was binary (PASS or FAIL); a FAIL fixed the AI overall rating at FAIL. Criteria 2-4 were still recorded when criterion 1 failed but could neither reverse the overall FAIL rating nor establish existence. Written key-issue fields held comments, not additional criteria. Multimedia Appendix 3 shows the sequence. Table 1 compares shared domains and stream-specific options.

Prompt Configuration

All 10 production batches referenced 1 system-prompt file with the same recorded Secure Hash Algorithm 256-bit hash, establishing cross-batch file stability only. Multimedia Appendix 4 reproduces it. Per-citation records document tool use and recovery but contain no prompt or independent proof of prompt delivery.

Tool-Use Audit

Each evaluation wrote a JSON record with invocation/result counts, WebSearch (ie, the web-search tool) queries, first-attempt compliance, and enforcement-retry status. The workflow specified 1 enforcement retry after a parseable response without WebSearch; if neither attempt invoked WebSearch, it assigned ERROR. Separately, parse recovery allowed up to 4 total attempts for unreadable structured output. Enforcement outcomes, recovered parse retries, and duration outliers were recorded separately. These records identify workflow steps, not retrieval completeness or judgment correctness.

Human Validation, Assignment, and Blinding

Panel and Assignment

Ten US-based validators (9-institution final-affiliation count) completed 18 forms covering 19 subspecialties. JL and YB shared a Beth Israel Deaconess Medical Center affiliation; the other 8 validators represented 8 additional institutions. HM and JW were senior authors and did not serve as stage 2 validators. Thus, the 12 authors included 10 validators representing 9 institutions. Convenience recruitment opened April 7, 2026; the first external validator agreed on April 10, 2026. Written instructions were provided without shared training or calibration. Each validator received 1-4 subspecialties within their stated expertise (30-120 citations). Each citation belonged to 1 subspecialty and had 1 assigned validator, making their effects inseparable and validator summaries descriptive.

Ratings and Reconciliation

Each citation had one human overall verdict; none had a second human rating, so human-human reliability could not be estimated. Multimedia Appendix 5 reproduces the blank Expert Citation Validation Form, version 2.1 (April 2026). Revision-stage reconciliation addressed variation in instrument use through a documented hierarchy that prioritized explicit citation-level verdicts. One validator returned workbook comments without verdict symbols for 60 citations in 2 subspecialties. The PI derived analytic verdicts under this hierarchy; each derived non-PASS verdict had a supporting comment. These were reconciled comments, not independently recorded form ratings; no retrospective form was created. Later PI-framed color clarification only reconciled existing annotations.

Blinding

Non-PI validators covered 17 subspecialties (510 citations). Five (150 citations) had form dates preceding stage 1, indicating that citation-level AI outputs were unavailable on those dates. For the other 12 (360 citations), nondisclosure of AI criterion/overall ratings rests on retrospective PI confirmation; scoring-start dates and assignment-package transmission, receipt, and acknowledgment dates were not retained. The PI rated “pediatric anesthesia and pharmacology” (60 citations) after reviewing stage 1 outputs and was unblinded. Neither the AI-FAIL audit, whose sampling frame used AI verdicts, nor revision-stage issue adjudication was independent or blinded.

Administrative Sign-Off

On August 13, 2026, 1 validator signed 2 previously dated forms; the dates remained unchanged, and comparison with retained pre-sign-off copies found no scoring changes.

Outcomes and Timing of Analysis Decisions

Rating Framework

The 4 domains were shared, but the instructions and recording procedures differed. The human form included definitions, random-start/substitution instructions, 30 scoring blocks, notes, qualitative assessment, and sign-off; the AI prompt required per-citation WebSearch, evidence addresses, structured output, and tool-compliance handling. Criterion 4 asked both streams to compare the clinical meaning of the Chinese and English summaries for each sampled output. Human ratings reflected subspecialty judgment, without a prospective terminology-control protocol or separate independent adjudication to establish preservation of all nuance in the original concepts. Human criterion 1 specified PASS/FAIL, although the generic grid displayed MINOR for every criterion. One submitted criterion 1 MINOR was retained without recoding or exclusion in 570 records. The form defined overall PASS using criteria 1-3 and called translation optional in its citation log, although the scoring instructions included criterion 4. The unchanged blank form preserves these inconsistencies; recorded ratings were unchanged.

Citation-Use Decisions and Timing

Operational curation treated PASS/CONDITIONAL as USABLE and FAIL as requiring correction or replacement, without a written standard operating procedure. Binary USABLE versus NOT-USABLE became the primary post hoc end point during the initial presubmission analysis, after collection and inspection. Three-category, criterion-level, and alert-rule analyses had no prospectively specified priority. The April 2026 Expert Citation Validation Form documented 4 domains. A protocol added to Google Drive on April 21, 2026, with an April 9, 2026, source-file modification time stamp, named Cohen κ as secondary before AI evaluation and paired-analysis completion, but not before the earliest human rating or any data inspection. This superseded document was neither a registration nor a statistical analysis plan; neither artifact fixed the binary end point or later coefficient set. An April 28, 2026, manuscript retained κ while paired results were pending. Gwet first-order agreement coefficient (AC1) was added after recognizing unequal category use. In the retained manuscript sequence ordered by source-file modification time stamps, the first reviewed file containing AC1 and Brennan-Prediger carries a June 7, 2026, time stamp. Revision-stage plan version 4.1 records an August 13, 2026, freeze but contains August 14-15, 2026, amendments; it is not an untouched August 13, 2026, byte snapshot and governs revision reanalysis only. Multimedia Appendix 6 gives this bounded chronology.

Statistical Analysis

Statistics Reported

Observed agreement is the proportion of citations receiving identical ratings from both assessment streams. Binary analyses report observed agreement, unweighted Cohen κ, Brennan-Prediger agreement, and AC1, all calculated from the same ratings. Three-category analyses report exact agreement, unweighted and linearly weighted Cohen κ, Brennan-Prediger agreement, and AC1. As the coefficients account for chance differently, we avoid strong/weak agreement labels. Unweighted Cohen κ follows the original 2-rater formulation [28]. We also calculated criterion-specific unweighted Cohen κ; each available pair contained 1 AI and 1 human rating, and no Fleiss κ was computed. CIs used the tier-preserving subspecialty-cluster percentile bootstrap described below, rather than a large-sample analytic standard-error formula.

Testing and Intervals

We estimated agreement and uncertainty without null-hypothesis tests, a prespecified α level, or P values. For agreement, coefficient, and alert-rule estimates requiring inferential intervals, we used 2-sided 95% CIs from the specified percentile-bootstrap procedures. Limits were the 2.5th and 97.5th percentiles of the resampled estimates. Complete-corpus counts, leave-one-subspecialty/validator-out analyses, and temporal reproducibility summaries have no inferential intervals. No causal exposures, predictors, or confounding-adjustment models were specified.

Available Pairs

All 570 AI records had 4 criterion ratings. Pairs were missing only for blank or AMBIGUOUS human entries; other available ratings remained analyzed. Criteria 1-4 denominators were 570, 569, 568, and 565, respectively. The single human criterion 1 MINOR rating, disallowed by the rubric, was retained as submitted.

Alert Rules

Threshold A flagged AI CONDITIONAL/FAIL and cleared PASS; threshold B flagged only AI FAIL. Sensitivity was the proportion of 74 human-FAIL citations that were flagged. Specificity was the proportion of 496 human not-FAIL citations that were unflagged (421 PASS and 75 CONDITIONAL). Neither threshold was evaluated as a standalone decision rule.

Grouped Uncertainty

Citations from the same concept stayed together within their subspecialty; concept effects were not separately estimated. Primary and most sensitivity intervals used a tier-preserving subspecialty-cluster percentile bootstrap (B=10,000; seed 20260814). Each replicate sampled subspecialties with replacement within tiers, retaining counts of 4/5/6/4, carried all rows from each selected subspecialty, and recalculated the statistics. With only 19 subspecialty groups, these approximate uncertainty intervals account for grouping but do not guarantee nominal coverage or support population-wide inference. A separate validator-level sensitivity percentile bootstrap used the same B and seed, resampling 10 observed validators with replacement and all their assigned subspecialties/rows. It cannot separate validator from subspecialty effects; convenience recruitment precludes population-wide rater inference.

Sensitivity Analyses

Post hoc analyses compared human category use with AI as a common comparator, resampled validators, omitted 1 subspecialty/validator at a time, excluded the PI, compared substituted and nonsubstituted citations, and counted each validator’s human-overall CONDITIONAL ratings. Revision analyses repeated binary, 3-category, criterion-level, and threshold A/B outcomes in 2 subsets: excluding all 120 WC/YB assignments as a proxy for exception-style recording, and restricting to validators with at least one final human-overall rating in each category: PASS, CONDITIONAL, and FAIL. Two-sided 95% CIs used the same tier-preserving subspecialty-cluster percentile bootstrap (B=10,000; seed 20260814), preserving each subset’s observed subspecialties and tier counts. Undefined and degenerate replicates were audited. Validator/specialty effects remain inseparable; AI was a comparator, not truth, and population-rater inference was not supported.

Citation-Issue Classification

Revision assigned each citation either no issue or a primary category from 8 post hoc levels: (1) nonexistent or fabricated citation; (2) identifier mismatch or metadata cross-assignment; (3) wrong-article or concept mismatch; (4) verification gap; (5) content misrepresentation; (6) translation error; (7) outdated source or temporal-currency concern; and (8) upstream concept-generation concern. When multiple categories applied, all individual labels were retained, and the lowest numbered category became primary. Labels could overlap and were not summed to estimate prevalence.

The post hoc classification began with the ratings and comments from the original human reviewers. Of the 570 citations, 389 had a human overall PASS rating, human PASS ratings on all 4 criteria, and no retained human comments identifying an issue. These 389 citations were assigned “no issue” by rule; the PI did not individually recheck their sources during this classification. The PI individually reviewed the remaining 181 records because at least one of these conditions was not met. Of these, 154 were assigned to primary categories 1-5 and included in the combined count, 16 were assigned only to primary categories 6-8, and 11 had no assigned issue. The complete-frozen-corpus count has no inferential CI. It includes verification gaps, and adjudication was neither independent nor blinded, so it does not estimate objectively established error or hallucination prevalence. Revision review of retained annotations selected examples without changing classifications; it was not a prespecified qualitative analysis with independent coding and cannot establish frequencies, prevalence, consensus, or causes.

Temporal Reproducibility and Process Audit

During revision, we used the recovered AI citation-checking workflow to reassess 112 citations. We compared these repeat AI ratings with the original AI ratings for the same citations. The sample was selected by original AI overall rating: 50 PASS, 45 CONDITIONAL, and all 17 FAIL citations. Including all 17 FAIL citations allowed us to examine whether every original AI FAIL rating was repeated. The ordered sampling lists and selection record were retained. We compared whether the original and repeat AI ratings matched exactly across PASS, CONDITIONAL, and FAIL, and whether they matched after combining PASS and CONDITIONAL into 1 group. We reported unweighted results for the 112 citations. As the sample included all original FAIL citations but only some PASS and CONDITIONAL citations, we also reported secondary estimates weighted to the proportions of these rating groups in the full 570-citation corpus. These summaries were descriptive, without inferential CIs.

We requested the same claude-opus-4-7 model identifier for the repeat assessment. However, missing interface-version and provider runtime records prevented confirmation of the model version actually used. Web search results and rankings were neither frozen nor controlled. The comparison therefore assessed the repeatability of the AI and retrieval workflow under later conditions, rather than the model alone or an exact reproduction of the original run.

Ethical Considerations

On June 30, 2026, Beth Israel Deaconess Medical Center’s Committee on Clinical Investigations determined under record 2026D000620 that this activity was not human research, requiring no further committee review or approval. This was not an exemption; participant-informed consent and a waiver were inapplicable. Citation-level records and professional assessments contain no patient data or clinical records. Validators are named in the manuscript, reader-facing supplements, and the released row-level dataset. These intentionally identifiable professional-assessment data were released with documented authorization and exact-file approval from all 10 validators [29]. Completed forms, signatures, contact information, institution fields, form dates, raw free-text comments, and provider/tool records were excluded; Multimedia Appendix 5 contains only the blank instrument. Validators received no study-specific compensation. No patient or platform user is identifiable in the analytic figure or reader-facing supplements.

Results

Cohort and Analytic Completeness

The cohort comprised 570 citations with paired AI-human evaluations, 30 per sampled subspecialty (19 total). Ten US-based validators from 9 institutions recorded assessments on 18 forms. One validator covered 4 subspecialties, 6 covered 2 each, and 3 covered 1 each. The PI rated 2 subspecialties, and the other validators rated the remaining 17. Each subspecialty had 1 validator, preventing separation of validator and subspecialty effects.

Human overall verdicts were 421 PASS, 75 CONDITIONAL, and 74 FAIL (total 570); AI verdicts were 296 PASS, 257 CONDITIONAL, and 17 FAIL (total 570). Six citations lacked complete 4-criterion records—4 had a blank human entry and 2 had an AMBIGUOUS entry—but retained their overall verdicts. Three of these 6 had determinate overall outcomes because another criterion was FAIL. Paired-rating denominators were 570, 569, 568, and 565 for criteria 1 through 4, respectively.

Post hoc revision-stage classification assigned 154 out of 570 (27.0%) citations to primary levels 1-5: 1 nonexistent or fabricated citation, 1 identifier mismatch or metadata cross-assignment, 106 wrong-article or concept mismatches, 33 verification gaps, and 13 content misrepresentations. Of 181 individually adjudicated records, 16 received only levels 6-8, and 11 received no assigned issue. Of the 400 citations with no assigned issue, 389 were classified from existing records without individual source rereview because they had no retained human-validator issue text, a human overall PASS, and PASS on all 4 criteria; 11 arose from adjudication. The classification includes verification gaps, was neither independent nor blinded, and is not an error-prevalence estimate; no inferential CI is reported. Tables 2 and 3 summarize cohort composition and classifications; overall verdict counts are reported above. Figure 1 shows overall verdicts by subspecialty.

‎

Overall Human-AI Agreement

For the binary end point, PASS and CONDITIONAL were classified as USABLE and FAIL as NOT-USABLE. Cross-classifying AI rows against human columns gave 484 citations rated USABLE by both streams, 69 rated USABLE by AI and NOT-USABLE by the human stream, 12 rated NOT-USABLE by AI and USABLE by the human stream, and 5 rated NOT-USABLE by both. The proportion rated USABLE by AI was 553 out of 570 (97.0%), and the proportion rated USABLE by human validators was 496 out of 570 (87.0%). Binary exact agreement was 489 out of 570 (85.8%; 95% CI 79.12%-91.75%). Cohen κ was 0.0645 (95% CI −0.0103 to 0.1639), AC1 was 0.8334 (95% CI 0.7388-0.9084), and Brennan-Prediger agreement was 0.7158 (95% CI 0.5825-0.8351). The coefficients differed because they account for chance in different ways when the assessment streams use rating categories in unequal proportions. They are reported together, without privileging one coefficient or attaching verbal strength labels.

Three-category exact agreement was 271 out of 570 (47.5%; 95% CI 41.93%-53.16%). Combining PASS and CONDITIONAL counted 218 mismatched pairs among the 489 binary agreements. Three-category coefficients were unweighted Cohen κ 0.0519 (95% CI −0.0112 to 0.1121), linearly weighted Cohen κ 0.0828 (95% CI 0.0087-0.1523), Brennan-Prediger 0.2132 (95% CI 0.1289-0.2974), and AC1 0.2944 (95% CI 0.2043-0.3841). Tables 4-6 present the binary matrix and both coefficient sets. Binary disagreements were unequal in direction (69 vs 12 citations), but their causes were not determined; this does not establish that AI or human validators were systematically more cautious or lenient.

Criterion-Level Agreement and Disagreement Pattern

Observed agreement was numerically highest for criteria 1 and 4 and lower for criteria 2 and 3 (Table 7). Nearly all criterion 1 ratings were PASS, and the FAIL sets did not overlap; its agreement (ie, 563/570, 98.8%) therefore does not establish concordance on rare FAIL ratings. Exact agreement was 563 out of 570 (98.8%) for criterion 1 (95% CI 97.72%-99.65%); 323 out of 569 (56.8%) for relevance (95% CI 51.75%-61.62%); 440 out of 568 (77.5%) for scientific accuracy (95% CI 72.63%-81.83%); and 529 out of 565 (93.6%) for translation fidelity (95% CI 91.23%-95.95%). Corresponding unweighted Cohen κ estimates were −0.0050 (95% CI −0.0097 to 0.0000), 0.1329 (95% CI 0.0737 to 0.1910), 0.0508 (95% CI −0.0165 to 0.1149), and 0.0678 (95% CI −0.0310 to 0.1773), respectively.

Criterion 1 was binary, although the generic human grid displayed MINOR for every criterion. AI recorded 566 PASS, 0 MINOR, and 4 FAIL (total 570); humans recorded 567 PASS, 1 out-of-rubric MINOR, and 2 FAIL (total 570). The FAIL sets did not overlap. The 7 disagreements comprised 4 AI FAIL/human PASS, 2 AI PASS/human FAIL, and 1 AI PASS/human MINOR. The out-of-rubric MINOR was retained unchanged and was not treated as a hallucination flag. Criterion 1 κ is descriptive only: of 10,000 bootstrap replicates, 29 were undefined, and 623 were defined but degenerated because ratings were nearly constant. It is not interpreted as meaningful below-chance agreement. Mapping ratings to the revision-stage classification cannot establish which stream was correct because that classification used human-validation evidence as its operational reference.

Vascular anesthesia, airway management, regional and acute pain, and intraoperative monitoring each had 30 human-PASS citations (120 total). Their zero κ values and zero-to-zero intervals reflect constant human ratings, not an absence of agreement. Using AI as a common comparator, PASS, CONDITIONAL, and FAIL correlations across 19 subspecialties were r=−0.004, −0.297, and −0.003, respectively; the CONDITIONAL correlation was weakly negative. These comparisons are descriptive. Multimedia Appendix 7 reports signed gaps and validator-pooled summaries. Human-overall CONDITIONAL counts and proportions, in neutral source order, were WC 8 out of 60 (13%), SH 0 out of 120 (0%), YQ 14 out of 60 (23%), GQ 11 out of 60 (18%), XZ 4 out of 30 (13%), YZ 18 out of 60 (30%), XF 0 out of 30 (0%), JX 0 out of 30 (0%), JL (PI) 18 out of 60 (30%), and YB 2 out of 60 (3%), totaling 75 out of 570. These are not rankings; validator and subspecialty effects remain inseparable. Retained annotations illustrated plausible locations of citation issues but were not a prespecified, independently coded qualitative dataset; they neither changed classifications nor supported frequency, consensus, or causal inference.

Citations Flagged or Missed by the Alert Rules

Using human FAIL as the operational reference, threshold A flagged AI CONDITIONAL or FAIL: 274 out of 570 citations were flagged, and 296 were cleared. Sensitivity was 46 out of 74 (62%; 95% CI 48.27%-74.42%); 28 out of 74 (a false-negative rate of 38%) were missed (95% CI 25.58%-51.73%). Specificity was 268 out of 496 (54.0%; 95% CI 49.35%-58.90%), with all 421 human PASS and 75 CONDITIONAL citations in the denominator. Positive predictive value was 46 out of 274 (16.8%; 95% CI 7.89%-26.27%); negative predictive value was 268 out of 296 (90.5%; 95% CI 85.19%-95.16%). Of the 296 cleared citations, 28 (9.5%) were human FAIL (95% CI 4.84%-14.81%). The flag rate was 274 out of 570 (48.1%; 95% CI 42.98%-52.98%).

Threshold B flagged only AI FAIL: 17 out of 570 (3%) citations (95% CI 1.93%-4.04%). It captured 5 out of 74 human-FAIL records and missed 69 out of 74 (93%; 95% CI 86.36%-97.70%). Sensitivity was 6.76% (95% CI 2.30%-13.64%) and specificity was 484 out of 496 (97.6%; 95% CI 96.30%-98.80%). In each bootstrap replicate, the missed proportion equaled one minus sensitivity; its interval therefore complements the sensitivity end points in reverse order. Table 8 compares citations flagged and human-FAIL citations missed. Neither threshold was a standalone decision rule or citation-accuracy measure.

Sensitivity Analyses, Temporal Reproducibility, and Process Audit

Excluding all 120 WC/YB assignments left 450 citations in 15 subspecialty clusters. Validator assignment was a proxy for the exception-style recording workflow, not a citation-level identification of its use. Binary agreement was 396 out of 450 (88%; 95% CI 81.33%-93.56%), with Cohen κ=0.1155 (95% CI 0.0269-0.2253); 3-category exact agreement was 225 out of 450 (50%; 95% CI 44.22%-55.56%), with κ=0.0723 (95% CI −0.0031 to 0.1421). Restricting to validators with at least one final human-overall PASS, CONDITIONAL, and FAIL retained WC, YQ, XZ, YZ, JL (PI), and YB (330 citations; 11 clusters). Binary agreement was 263 out of 330 (79.7%; 95% CI 72.12%-86.36%), with κ=0.0315 (95% CI −0.0356 to 0.1012); 3-category exact agreement was 159 out of 330 (48.2%; 95% CI 41.21%-54.85%), with κ=0.1010 (95% CI 0.0254-0.1783). In both revision-stage subsets, 3-category agreement remained near the full-corpus estimate, while coefficients diverged; binary agreement varied with subgroup composition. Agreement depended on whether 2 or 3 categories were compared, and no single binary percentage described every subgroup. Multimedia Appendix 7 provides complete coefficient families, criterion and threshold A/B results, intervals, and bootstrap accounting. Multimedia Appendix 8 maps reporting items from the 3 cited reporting frameworks [25,26,30] to manuscript locations without evaluating adherence or assigning a grade.

Tables 9 and 10 summarize the remaining sensitivity and influence analyses. Resampling validators produced wider uncertainty intervals without changing the point estimates. Excluding the PI and omitting 1 subspecialty or validator at a time did not identify a single assignment whose removal eliminated the observed binary pattern. These analyses still cannot separate validator effects from the mix of specialties or eliminate bias from overlapping investigator roles. Future studies could examine whether substitutions affect validation results. The comparison of 57 substituted with 513 nonsubstituted citations could not isolate such an effect.

In the stratified 112-citation rerun, sample-unweighted exact 3-category and binary reproducibility were 91 out of 112 (81.3%) and 107 out of 112 (95.5%), respectively; secondary estimates weighted to the original verdict distribution were 82.84% and 99.12% (analyzed n=112), respectively (Table 11). Original verdicts persisted for 43 out of 50 PASS, 36 out of 45 CONDITIONAL, and 12 out of 17 FAIL citations; criterion 1 FAIL persisted in 2 out of 4. These descriptive summaries have no inferential CIs. The audit (Table 12) recorded first-attempt WebSearch for 570 out of 570 citations, 0 enforcement retries, 0 ERROR verdicts, 20 recovered parse retries, 3 duration outliers, and 1108 invocations (mean 1.9439). These records demonstrate tool use and processing recovery, not retrieval correctness, completeness, or correct model interpretation.

Discussion

Principal Findings

AI and human reviewers could agree to retain a citation while disagreeing about whether it still needed checking. Our binary comparison grouped PASS and CONDITIONAL together. For example, an AI PASS paired with a human CONDITIONAL counted as agreement in the binary analysis but as disagreement when PASS, CONDITIONAL, and FAIL were compared separately. Higher binary agreement therefore does not mean that both assessors considered a citation acceptable without further checking. This distinction remained in subset analyses, although binary agreement varied with the citations included. Individual checks helped identify what needed attention: relevance and scientific accuracy showed more disagreement than article existence and translation fidelity. Relevance asks whether the article addresses the clinical question; scientific accuracy asks whether the summary preserves its findings and qualifications. An existing article may provide unsuitable support for a statement, or its findings may be summarized misleadingly. Disagreements could reflect different interpretations of evidence or applications of rating rules; our comparisons cannot identify the explanation for an individual citation. Specialists can examine the source passage, the linked clinical statement, and the clinical context. Reporting these checks alongside overall ratings helps direct that examination. The retained examples below illustrate what such examination can reveal.

The choice of agreement measure affects how readers interpret the same AI-human ratings. Most binary matches were citations that both streams classified as USABLE. Percent agreement counts how often ratings matched, without adjusting for category use [31]. Cohen κ compares observed agreement with the agreement expected under its chance model, based on the marginal use of each rating category by the 2 streams. This reference level can be high when both streams often assign the same category, so many matching ratings can coexist with a low κ. AC1 uses a different chance-agreement calculation and responds differently when 1 rating is much more common [31,32]. A higher AC1 should not be interpreted as a more accurate measurement of the same quantity. We also reported Brennan-Prediger agreement, without choosing a preferred coefficient. Several κ CIs included 0, leaving uncertainty about agreement beyond the coefficient’s reference level; this does not mean that the assessors never agreed. None of these measures establishes citation accuracy because human judgments were not independently established as correct answers [33,34]. Reporting rating definitions, category frequencies, measures, and CIs together lets readers examine how conclusions depend on analysis choices. Such reporting also helps future studies explain why agreement estimates differ.

The alert-rule comparison asks which citations AI would bring to a specialist’s attention and which would remain outside the review list. Flagging only AI FAIL produced a shorter list but left most human FAIL ratings unflagged. Adding AI CONDITIONAL captured more human FAIL ratings, although not all, and selected more citations for checking. The number selected must therefore be considered alongside what remains unflagged. An AI flag identifies a citation for examination, not a confirmed error; an unflagged citation is not confirmed correct. AI flags may help organize specialist review if unflagged citations also receive appropriate scrutiny. We did not measure the time needed to review either list or determine which rule would be more efficient. Future evaluations should assess the effort required and the problems found and corrected, including among unflagged citations.

Comparison With Previous Studies

Comparisons with earlier citation studies must begin with what each study checked. An overall score cannot distinguish confirmation of article identity from judgment of whether content supports a clinical statement. Earlier audits included article existence and citation details [1-7], while statement-source research examined whether the source text supported an attached claim [8]. Scripted conceptual examples separated apparent evidence support from correctness [9]. Other studies assessed relevance, metadata, existence, and peer-reviewed publication status [10], or main points, errors, quality assessment, and context in evidence summaries [24]. Building on these distinctions, our paired evaluations applied different checks to the same bilingual citation records. The paired ratings show which aspects of a citation AI and human reviewers judged differently.

Retained comments and citation materials preserve details that a rating label alone cannot convey. One human comment noted that a paper addressed transversus abdominis plane block safety and median dosage rather than the assigned question about maximum dose. In a stored bilingual summary, “stroke volume” appeared as “卒中容积” rather than the cardiac term “每搏量.” The summary for US-013-03 described tricuspid annular plane systolic excursion ≥1.7 cm as suggesting right-ventricular dysfunction, whereas the cited guideline defines normal M-mode tricuspid annular plane systolic excursion as >1.70 cm [35]. These materials allow reviewers to examine the specific question, terminology, or threshold, including a meaningful translation concern despite higher overall translation agreement. They identify concrete checks, not issue frequencies, the cause of the overall disagreement pattern, or which assessment stream was superior.

The examples above illustrate why the type of concern matters for further citation review. Harada’s [9] conceptual audit used scripted examples to distinguish clinical correctness, evidence support, and numeric consistency. Colder Carras et al [24] documented inaccurate terminology and a summary of the wrong publication in AI-generated annotations. Our post hoc classification addressed related concerns in retained records from a working bilingual knowledge base. The PI distinguished article or concept mismatch, insufficient evidence for verification, and misrepresentation of source content, alongside concerns about translation, outdated sources, and the original clinical concept. These distinctions can help specialists decide whether to recheck the chosen article, seek supporting passages, or examine a summary or translation for correction. Retaining overlapping labels preserves details that an overall verdict alone cannot convey. The classification was unblinded and not independently repeated, so it describes identified review concerns rather than confirmed error prevalence. Its contribution is to make the reasons for further review explicit and provide candidate criteria for future evaluation by independent reviewers.

The generation setting and evaluation unit provide another basis for comparison. Previous studies used restricted website searches [13], supplied documents or online material [14,17], and citation-recommendation systems [12]. Others assessed clinical answers alongside source usefulness or citation characteristics [15,21]; pediatric perioperative work separately assessed answer quality, citation authenticity, and clinical safety [36]. Here, OpenEvidence generated candidate citations and bilingual summaries, while a separate AI checking workflow and human reviewers evaluated the same sampled citation records. Each record linked an article to a clinical concept and English-Chinese summaries, with article existence explicitly included among the checks. This adds evidence about judgments made during citation curation in a working resource; it does not make different retrieval and clinical-answer tasks directly comparable as performance tests.

Interpretation within this study also depends on how the assessments were arranged. The 2 streams assigned criterion 1 FAIL to different citations, providing candidates for further checking rather than evidence of complementary detection of true errors. Each subspecialty had 1 assigned human reviewer, so reviewer and subspecialty differences could not be separated, consistent with methodological work on nested rater designs [37]. Comparisons of category use against the AI ratings may help identify assignments for rubric review but cannot rank reviewer strictness or determine which stream was correct.

Methodological Innovation and Practical Implications

The methodological contribution combines evaluations of the same 570 citation records from a working bilingual anesthesiology knowledge base covering 19 subspecialties. Each citation received 1 AI and 1 assigned human overall judgment. We compared whether both assessors would retain a citation and whether they assigned the same PASS, CONDITIONAL, or FAIL rating. We also compared their available ratings for article existence, relevance, scientific accuracy, and translation fidelity. The separate post hoc classification described recorded review concerns, while alert rules identified citations selected for further checking and human FAIL ratings left outside that selection. Together, these comparisons describe how often judgments agree and which aspects of a citation may warrant closer examination.

Repeat AI assessment and retained process records add information about whether judgments remain stable and how the workflow was executed. In the later run, decisions to retain or remove citations repeated more often than exact PASS, CONDITIONAL, or FAIL ratings, and some original FAIL ratings changed. Following individual ratings therefore helps identify judgments that may warrant another look when an AI workflow is used again. The requested model identifier was unchanged, but the model actually used could not be independently confirmed, and search results were not held fixed. The comparison therefore describes the combined AI and retrieval workflow at a later time. Related research found that more repeatable antibiotic recommendations were not necessarily more clinically correct [38]. Repeated requests for laboratory reference intervals produced responses whose consistency depended on the model version and laboratory test [39]. We also retained checks confirming that the saved prompt file was unchanged across batches, source locations, search records, and records of how output problems were handled. These records give later reviewers a starting point for examining the documented steps associated with a changed or disputed judgment.

The framework can guide a review process in which AI identifies citations for attention and specialists judge their source support. Brini and Leung’s [40] tutorial emphasizes human oversight suited to the task, checking whether tools are valid for their intended use, and transparent documentation. Screening studies show that selection can change with prompt wording [41,42] and the way eligibility criteria are expressed [43]. Reported efficiency findings included measured reductions in screening time [41], estimated human time savings [42], and calculated reductions in a second reviewer’s workload [43]. Disagreements with reference screening judgments remained [41-43]. Clinical studies also show why assistance needs to be evaluated in its intended setting: a physician trial found no significant improvement in diagnostic reasoning with large language model access [44], radiology research found different effects across readers and tasks [45], and a systematic review found variable outcomes from human-large language model collaboration [46]. Applied to citation curation, our framework would retain AI ratings and recorded concerns alongside each citation. A specialist could use these records to locate the relevant source passage, check the clinical claim and bilingual summary, and decide whether to retain, correct, or replace the citation.

Implications for Metascience and Research Integrity

This study has a metascientific contribution by illustrating how rating definitions and agreement measures influence the interpretation of the same AI-human citation evaluations. Applying established methods in a working bilingual knowledge base makes these evaluation choices and their consequences visible. For research integrity, explicit criteria and retained reasons and process records provide a basis for examining documented judgments. Public structured ratings allow others to recalculate the reported statistics. These resources support scrutiny of the evaluation but do not independently establish citation correctness. The public release excludes raw comments and provider/tool records, and the available records cannot reproduce the original AI and retrieval conditions. Within these limits, the study offers a practical example of documenting citation evaluation and sharing its structured results, which may help other researchers plan and interpret similar assessments.

Limitations

First, each citation had only 1 human overall rating, with no independently double-scored subset, consensus panel, or human-human agreement estimate. Human ratings were a comparison reference, not independent confirmation of citation correctness [33]. One convenience-recruited validator per subspecialty prevents separation of reviewer and subspecialty effects. Comparisons of human and AI category use remain descriptive because AI performance may differ by subspecialty. Records do not confirm that criteria left unmarked on forms recording only exceptions were checked as explicitly as criteria with recorded PASS ratings. Excluding 120 assignments therefore uses validator assignment as a proxy for workflow differences, without confirming how each citation was assessed. Overlapping citation sets assessed by multiple validators across subspecialties would help separate sources of disagreement [37]. Consensus would still require source-passage checks and rules for resolving disputes, without guaranteeing correctness.

Second, for 12 subspecialties assessed by other validators (360 citations), evidence that AI ratings were withheld rests on the PI’s later confirmation. Records of scoring-start dates and when review materials were sent were not retained. The PI saw stage 1 outputs before rating 2 subspecialties (60 citations). These ratings, the AI-FAIL audit, and revision-stage classification were unblinded; the latter 2 were not independent. Excluding PI ratings cannot remove potential bias from the PI’s design, corpus construction, workflow operation, validation, audit, and classification roles. Original file numbers did not directly match nominal sampling counts; the tier 3/4 selection seed, script, and time stamp were missing. No registration or preregistration record was found. The binary end point and full coefficient set were not fixed before data inspection. Volunteer workload determined sample size, without agreement-specific power or precision calculations. With only 19 subspecialty groups and convenience recruitment, bootstrap intervals may not achieve their intended coverage and cannot establish generalizability to all subspecialties or validators.

Third, 1 unblinded investigator classified the expanded retained records during revision; no independent investigator repeated the classification. Most no-issue assignments used existing ratings without individually rechecking sources, so concerns may remain unrecorded. The combined count includes verification gaps, and 1 primary category may summarize several overlapping concerns. The 4 domains did not systematically assess outdated sources or poorly specified clinical concepts; passing ratings therefore leave these questions unresolved. Selected examples were not part of a prespecified qualitative study or independent coding process and cannot establish frequencies or causes.

Fourth, retained records allow statistical recalculation but cannot recreate the original run. Missing information includes provider confirmation of the actual model, original provider responses, complete command-line and search-result records, the software environment, and a frozen search index. The later assessment tested the AI and retrieval workflow together under uncontrolled search conditions. A newer model was released after the primary AI evaluation but within the dates on some human review forms [47]; these dates do not establish the full scoring period. Frozen records identify the requested/selected claude-opus-4-7 identifier, but provide no later-model identifier or verified runtime identity. Findings may not apply to newer systems. Repeat evaluations should document the requested generation system and retrieval environment.

Fifth, the bilingual prompt and criterion 4 specified Chinese-English summary comparison without prespecified terminology rules or separate independent bilingual adjudication. The human form described translation’s role in the overall rating inconsistently and called it optional in the citation log. Ratings were preserved, but consistent interpretation across validators cannot be confirmed. The rubric incompletely covered subspecialty nuance, overlooked better sources, and failed to account for the effects of omitted literature on summaries. We did not measure how much relevant literature the searches retrieved; concept-generation transcripts and complete transmission timing were unavailable. Upstream review checked structure, not every clinical concept. We assessed citation curation, not correction time, prospective review burden, learner knowledge, clinical decisions, or patient outcomes. One account, interface, and anesthesiology corpus limit generalizability. Future work needs multilingual expert adjudication, independently assembled relevant sources, system versions documented over time, and prospective workflow outcomes across institutions, specialties, and interfaces.

Conclusions

AI-generated citations need to be evaluated both as bibliographic references and as evidence supporting clinical statements. This study brings together overall AI-human judgments and specific checks of article existence, clinical relevance, scientific accuracy, and translation fidelity in a working bilingual anesthesiology knowledge base. These complementary comparisons help distinguish agreement on a broad decision from specific concerns that require further review. Repeat AI assessment adds information about rating stability, while subsequent human review and a separate post hoc 8-category classification describe concerns that the 3 overall verdicts alone cannot explain. Together, these components can help specialists determine what needs checking, whether source passages support the attached claims, and whether summaries or translations require correction. Retained assessment and process records support later examination of documented judgments, and public structured ratings enable statistical recalculation and further analysis. The contribution to metascience and research integrity lies in making evaluation choices and their supporting records inspectable and potentially useful for future work as AI-assisted citation use expands.