<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="static/style.xsl"?><OAI-PMH xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/ http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd"><responseDate>2026-11-22T22:20:00Z</responseDate><request verb="GetRecord" identifier="oai:digital.lib.washington.edu:1773/45512" metadataPrefix="dim">https://digital.lib.washington.edu/server/oai/request</request><GetRecord><record><header><identifier>oai:digital.lib.washington.edu:1773/45512</identifier><datestamp>2026-02-15T22:31:16Z</datestamp><setSpec>com_1773_4888</setSpec><setSpec>col_1773_4936</setSpec></header><metadata><dim:dim xmlns:dim="http://www.dspace.org/xmlns/dspace/dim" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:doc="http://www.lyncode.com/xoai" xsi:schemaLocation="http://www.dspace.org/xmlns/dspace/dim http://www.dspace.org/schema/dim.xsd">
   <dim:field mdschema="dc" element="contributor" qualifier="advisor">Xia, Fei</dim:field>
   <dim:field mdschema="dc" element="contributor" qualifier="author" authority="c85ee99e-16b6-40bc-9063-b87a4917b7d5" confidence="300">Zhu, Haotian</dim:field>
   <dim:field mdschema="dc" element="date" qualifier="accessioned">2020-04-30T17:43:47Z</dim:field>
   <dim:field mdschema="dc" element="date" qualifier="issued">2020-04-30</dim:field>
   <dim:field mdschema="dc" element="date" qualifier="submitted">2020</dim:field>
   <dim:field mdschema="dc" element="identifier" qualifier="other">Zhu_washington_0250O_21188.pdf</dim:field>
   <dim:field mdschema="dc" element="identifier" qualifier="uri">http://hdl.handle.net/1773/45512</dim:field>
   <dim:field mdschema="dc" element="description">Thesis (Master's)--University of Washington, 2020</dim:field>
   <dim:field mdschema="dc" element="description" qualifier="abstract">This thesis explores and compares statistical significance tests frequently used in comparing Natural Language Processing (NLP) system performance in several aspects. We begin by establishing the fundamentals of the NLP system performance comparison and formulating it into four major tasks specific to NLP. Each statistical significance test is explained in great detail with its assumptions explicated and testing procedure outlined. We stress the importance of verifying test assumptions before conducting a test. In addition, we examine the effect size and statistical power and discuss their significance in the statistical significance testing in NLP. By considering potential dependencies within a test set, the block bootstrap is introduced and employed to calibrate the statistical significance testing for comparing performance of two systems on average. Four case studies with both simulated and real data, of which the complexity of data dependency varies, are presented to illustrate the process of properly using a statistical significance test in comparing NLP system performance under different settings. We then proceed to discussion from different perspectives, with some open issues such as cross-domain comparison and the violation of i.i.d. assumption, which expects further studies. In conclusion, this thesis advocates the proper use of statistical significance testing in comparing NLP system performance and the reporting of the comparison results in more transparency and completeness.</dim:field>
   <dim:field mdschema="dc" element="format" qualifier="mimetype">application/pdf</dim:field>
   <dim:field mdschema="dc" element="language" qualifier="iso">en_US</dim:field>
   <dim:field mdschema="dc" element="rights">none</dim:field>
   <dim:field mdschema="dc" element="subject">Effect size</dim:field>
   <dim:field mdschema="dc" element="subject">Power analysis</dim:field>
   <dim:field mdschema="dc" element="subject">Significance testing</dim:field>
   <dim:field mdschema="dc" element="subject">Linguistics</dim:field>
   <dim:field mdschema="dc" element="subject">Statistics</dim:field>
   <dim:field mdschema="dc" element="subject" qualifier="other">Linguistics</dim:field>
   <dim:field mdschema="dc" element="title">On the Statistical Significance Testing for Natural Language Processing</dim:field>
   <dim:field mdschema="dc" element="type">Thesis</dim:field>
   <dim:field mdschema="dc" element="embargo" qualifier="terms">Restrict to UW for 1 year -- then make Open Access</dim:field>
   <dim:field mdschema="dc" element="embargo" qualifier="lift">2021-04-30T17:43:47Z</dim:field>
   <dim:field mdschema="others" element="access-status">open.access</dim:field>
</dim:dim>
</metadata></record></GetRecord></OAI-PMH>