Below is a paper. Memorize the paper and answer my question after the paper. The paper begins. Abstract Answering non-factoid questions (NFQA) is challenging task, requiring passage-level an- swers that are difficult to construct and evaluate_ Search engines may provide a summary of a single web page, but many questions require reasoning across multiple documents_ Mean - while, modern models can generate highly co herent and fluent, but often factually incor- rect answers that can deceive even non-expert humans_ There is critical need for high- quality resources for multi-document NFQA (MD-NFQA) to train new models and evaluate grounding and factual consistency in answers relation to supporting documents_ Figure I: An instance of proposed WIKIHOwQA To bridge this gap, we present WIKIHOWQA a new multi-document NFQA benchmark built on WikiHow, website dedicated to answer- questions_ ing how-to The benchmark in- along cludes 11,746 human-written answers with 74,527 supporting documents We de- scribe the unique challenges of the resource provide strong baselines, and propose a novel human evaluation framework that utilizes high- lighted relevant supporting passages to mitigate issues such as assessor unfamiliarity with the including question topic. All code and data, the automatic code for preparing the human evaluation, are publicly available. However; even relatively to generate an answer: straightforward questions like how to fix my com- puter issue" can have multiple solutions, requiring manually search through multiple re- the user to trieved documents t0 find the one that to applies their situation. Complex questions such as "how to feel calm and relaxed" may require an aggregated summary of the most methods from multi- popular There are also questions for ple relevant sources which answers have yet to be written, requiring a QA system to treat relevant documents as initial sources of information and then t0 reason out an answer based on them: For example, the question how to prepare to buy a house in [neighbourhood], [city]" would require the system to retrieve relevant documents about buying houses in general and spe- cific articles about the local house market and the neighbourhood, and then construct a more sophis- ticated answer through reasoning rather than just summarizing multiple documents Introduction Non-factoid questions long, (NFQs) requiring passage-level answers, explanations such as or opinions, pose challenges for question current wering systems. While few datasets exist anse for NFQA (Cohen and Croft; 2016; Hashemi et al. 2019; Soleimani et al,, 2021), users are currently limited to seeing a summary of the most relevant document in a search snippet result page on be insufficient for complex (SERP), which can questions that have scattered answers across multi- ple documents or require sophisticated reasoning To address these challenges and move towards more advanced QA systems that can provide in- depth and comprehensive answers to a wide range of questions, benchmark for propose a new we the task of multi-document non-factoid QA (MD- https /lurunchik.github. io/WikiHlowQA NFQA) This task involves using multiple relevant documents to generate a complete and coherent an- given NFQ. We focus our benchmark on swer t0 a the INSTRUCTION category of NFQs, which of- These questions are under- ten begin with how to" represented in current multi-document QA Bolo- tova et al, 2022) datasets, despite their popular- ity, as evidenced by a 140% increase in how to_ Google since 20042 and making up searches on over half of the most searched queries.3 Our bench- mark aims to fill this gap by specifically targeting INSTRUCTION questions. crowdsourcing experiments. Inspired by series of the elaborate yet resource-intensive manual eval uation framework of abstractive models proposed by Dou et al. (2022), we introduce a simple human evaluation framework that leverages highlighted relevant passages to enhance the quality of NFQA Our findings demonstrate that incor assessments_ porating highlighted relevant supporting passages into the evaluation process not only aids evaluators in understanding the context of each question and answer but also contributes to the factual accuracy integrate this of the evaluation: As a result; we method into the evaluation of the newly proposed benchmark. WikiHow is for INSTRUC web-resource TION questions that contains over 235,000 articles wide range of topics. These articles provide on comprehensive step-by-step instructions and are written by a community of experts and reviewed by an average of 16 people. The WikiHow website has proven to be a valuable resource for machine Wang, Yang learning tasks (Koupaee and 2018; et al,, 2021; Bhat et al,, 2020; Zellers et al,,2019; Ladhak et al, 2020; Boni et al,, 2021; Zhang et al. 2020; Cohen et al,, 2021; Anthonio et al., 2020), leverage WikiHow to create a and in our work, we high-quality benchmark specifically designed for MD-NFQA within the INSTRUCTION question category: Our benchmark consists of 11,746 ques- tions from the INSTRUCTION category each corresponding human-written answer; with a paired sourced from a diverse range of WikiHow articles. supported by corresponding parsed Each QA is pair relevant HTML pages from which the answer can be derived (Fig: 1). We evaluate several baseline models on the new benchmark that could serve as lower and upper bounds for model performance: The new benchmark for MD-NFQA serves as valuable resource for the development of more advanced QA systems that can provide in-depth and comprehensive answers grounding informa tion in supporting documents. By including parsed passages from relevant HTML pages and provid- ing human judgments, our benchmark also has the potential to facilitate research in evaluating the fac - tual correctness of long-form answers. All data is publicly available4 on the dataset website. Related Work 2 MD This datasets for related section covers NFQA tasks such as long-form question answering (LFQA) and query-based multi-document summa rization (QF-MDS), highlighting the scarcity of resources available. We examine the strengths and limitations of these resources, aS well as the differ- ences between these tasks and MD-NFQA: LFQA: The task of LFQA, introduced by Fan al. (2019), involves providing long answers et to open-ended questions. While the associated ELIS benchmark includes supporting "document" to generate answers, the LFQA task; unlike MD NFQA task; does not assume that an answer is scattered across multiple documents. Moreover while MD-NFQA exclusively targets NFQs, ASQA LFQA dataset Stelmakh et al,, 2022) addresses ambiguous for long-form the need answers to factoid questions. Similarly, Natural Questions dataset (Kwiatkowski et al , 2019) contains both long and short answers for factoid questions (Xu et al., 2022). Human annotation is often used as the standard for evaluating long-form answers, but research has shown that individuals without specific training only distinguish between human-generated and can equivalent to ran generated text at a level auto- This presents dom chance (Clark et al., 2021). challenge for evaluating the new benchmark for a NFQA, particularly when the answers are lengthy Or the topic is unfamiliar (Krishna et al, 2021) interesting (Bolotova et al,, 2020). Given or not that standard metrics struggle to detect factual in- consistencies, such as number swapping, negation, etc. (Kryscinski et al,, 2019), we delve into human sensitivity to such discrepancies by conducting a The ELIS dataset (Fan et al,, 2019) for LFQA includes 272,000 questions from the "Explain Like Im Five" Reddit web forum; where questions and Zhttps:_ /bit.ly/telegraph-google-how-to "Under RMIT University DTA license for research use 'https /bit.ly/most-asked-questions-on-google answers must have a score of at least two to be in- cluded: The top-voted answer for each question is considered the correct answer; and the supporting content for each question is generated by extracting sentences with high tf-idf similarity from the top 100 web pages that match the question from the Common Crawl corpus. In contrast, our proposed MD-NFQA dataset uses texts parsed from relevant HTML pages chosen by the author of the corre- sponding article on WikiHow, rather than relying on automated methods for curating reference con- tent: In addition to a lack of grounding in support- ing content; Krishna et al. (2021) found significant Train/ Valid overlap in the ELIS dataset, and high- light challenges with both automatic answer eval- uation and human annotation for this benchmark proposed benchmark; we address these is- In our through the use of a novel human-evaluation sues framework utilizing supporting documents and by avoiding question overlap (Sec. 5 and Sec. 3). Figure 2: An example WikiHow article provided in- clusions or inferences based on the formation, while MD-NFQA requires higher level reasoning and synthesis to provide complete and While there are a few resources accurate answers available for MDS (Litkowski, 2004; Angelidis and Lapata, 2018; Liu et al,, 2018; Dang; 2006; Fab- bri et al., 2019; Ganesan et al. 2010; Wang and Ling, 2016; Yasunaga et al,, 2019; Koupaee and 2018; Lu et al,, 2020) and QFS (Zhong et al. Wang; 2021; Nema et al,, 2017; Zhao et al,, 2021), they are scarce for QF-MDS. Nakano et al. (2021) used questions from the ELIS dataset and collected new answers from hu- man annotators who were instructed t0 search for related documents and use them to construct their They trained the WebGPT model on that answers . dataset to answer long-form questions by mimick- ing the way humans research answers to questions online: it searches and navigates the web to find relevant pieces of information and concludes an citing answer based on them by sources for factual Authors report that the model answers accuracy. preferred by assessors 69% of the time to the are best human-written answer from Reddit when eval- uating on ELIS. While the WebGPT dataset is valuable, as of this moment; the authors have only released the model's answers and questions, not the supporting documents. The QMDSCNN and QMDSIR datasets target QF-MDS task Pasunuru et al., 2021). The first dataset is derived from CNN/DailyMail having real summaries with simulated queries, while the sec- ond dataset is derived from a search engine query log and has simulated summaries with real queries large Another automated approach for curating query-focused summarization tasks is datasets for AquaMuse (Kulkarni et al,, 2020). This dataset supports both abstractive and extractive QF-MDS Queries and long answers from the Natu- tasks. ral Questions dataset (Kwiatkowski et al,, 2019), pre-processed version of the Common Crawl and a Long were used (Raffel et al., 2020). corpus an- swers from Natural Questions and the Common Crawl corpus are encoded into sentence embed- dings. Then a similarity search is performed over long the corpus and answers to find candidate doc- uments from the corpus for QF-MDS tasks (similar to ELIS). Finally, it's worth noting that while REASON and EVIDENCE-BASED categories of NFQs pre- vail in the ELIS (Bolotova et al., 2022) and We- bGPT datasets, our dataset focuses specifically on the INSTRUCTION category: QF-MDS: In contrast to MD-NFQA, which in- volves generating detailed passage-level answers to NFQs, QF-MDS (Tombros and Sanderson, 1998) creating concise summaries in response focuses on specific queries, which may include factoid ques- to queries not in question form, such as [En- tions or This requires a different set of tity], [Event], etc_ skills and approaches compared to MD-NFQA, as QF-MDS summaries do not involve making con- proposed HowSumm, a QF Boni et al. (20213 MDS dataset automatically constructed from Wiki- How content by utilizing the referenced articles as the summarization source, the corresponding ele- ments of WikiHow articles as the target summaries, and titles used as the queries. There are two types Table 1: Comparison of WikiHOwQA with other QF-MDS and LFQA datasets Table 2: Percentage of novel n-grams of QF-MDS tasks, one for methods (HOwSUMM METHOD) and one for steps (HOWSUMM-STEP). While this work is similar to the dataset we present, HowSumm is directed toward the QF-MDS task does not discuss train-test overlap, and only in- cludes the source URLS in their dataset: Resource Description 3 including This section presents Our new resource, its construction, text statistics, comparison with other benchmarks, and thorough quality analysis. et al,,2019), Krishna et al. (2021) observed little to no evidence that analyzed models grounded answer They at- generation in the retrieved documents. significant overlap (around 43.69) tributed this t0 a in the training, validation, and test sets of ELIS. Similarly, Lewis et al. (2021) identified train-test overlap in various QA datasets. To avoid this issue questions in explicitly in our new resource, split _ we paraphrased questions a way that no same-topic o ended up in different splits. To do So, we clustered all questions prior to splitting and assigned all ques- tions belonging to the same cluster to the training split: We only assign clusters to the training split so that the validation and test splits will have more var- potential evaluation bias ied questions, to avoid a due to the presence of many similar questions.5 Data Collection 3.1 Fig: 2 illustrates an example of a WikiHow article accompanied by a high-quality, human-written arti- cle summary: These summaries are equivalent to passage-level question: shorter To answers to a build WIKIHOwQA, we first downloaded over 236,000 articles published on WikiHow before Jan- uary 2022 using the MediaWiki API: We then fil- tered out articles without references, resulting in a collection of 126,711 articles. Among these ar- ticles, only about 20% had human-written article summaries, which we used as target answers_ We downloaded the HTML content of all cited URLs using the Wayback Machine to provide the version closest to when the article was created or modified. snapshot ver- The final HTMLs were saved from the We simplified the HTML source code, and sions_ processed the content in two ways: (1) by extract- ing text content or (2) by converting it to Markdown format. The latter format preserves formatting such as tables and lists. Finally, we rejected articles with missing HTML snapshot links, empty HTML, Or empty extracted Each instance in our text. new dataset consists of a question; a human-written ar ticle summary treated as the target answer; and a list of parsed texts from the relevant HTML docu- ments cited by the article' s author: Fig: 1 shows a simplified example from WIKIHOwQA: Dataset Statistics 3.2 The WIKIHOwQA dataset include a diverse range of topics, 19 in total, with the most frequent be- and garden' thome ing health" "pets and ani- and ~computer and electronics".6 Table 1 mals to other QF-MDS and compares WIKIHOwQA LFQA datasets. ELIS is the largest dataset in terms of questions and has the lowest average words in reference documents (recall that the single refer- ence document is an aggregate of top-k passage retrieval and hence the absence of some of the reported statistics within Table 1). The number of answer words is higher than other datasets ex cept for HOwSUMM-METHOD. AQUAMUSE has Avoiding Overlap in Train-Test: When studying state-of-the-art model performance on ELIS Fan Implementation details can be found in App: B 6The full category distribution is provided in App. A fewer questions, although the reference sources and answer content are most similar t0 the statis- tics of HOWSUMM-STEP and WIKIHOwQA. The HowSUMM-METHOD dataset statistics are unique, it has 11 source documents per instance on aver age, and the length of answers are greater with an average of 31 sentences per instance_ HowSUMM STEP also has a high number of source documents while the remaining statistics follow a similar trend to the other datasets described: While our pro- dataset WIKIHOwQA shares many common pposed statistical characteristics of other datasets, the con- struction process, source-target mappings and task applications differ as discussed in Sec. 2. Figure 3: WIKIHOwQA quality annotation Quality Verification 33 Cardie (2020) propose additional metrics to com- pare datasets, some of which are reformulation of those presented by Grusky et al. (2018). Since the relevant passages in our WikiHowQA datasets are sourced from web pages cited by the they WikiHow article author; may not contain the exact words Or phrases from target answers. There- fore, to verify the quality of our benchmark; we as- sess the feasibility of constructing an answer from the given supporting documents As this is a cru- cial aspect for a MD-NFQA resource, we evaluate our benchmark using both automatic metrics and thorough human evaluation. The above metrics glve a general indication Of the dataset quality although even when examples from our dataset are deemed lower quality, for ex similarity, we they ample when have low semantic find that there are instances where an answer can typically still be constructed from the supporting documents. Fig: 1 demonstrates that answers can be successfully constructed from documents even they have low semantic similarity. For instance, if in the first sentence of an answer for the question pork tenderloin" (Fig: 1), although How to cook the target length of 55 minutes was not mentioned in any document; it can be inferred by adding up roasting time of 30 minutes 0 one side and 25 minutes on the other side. Similarly, the suggestion "flipping it occasionally" in the last sentence can be rephrased from "turning every 1 1/2 to 2 minutes" To verify our observations Human evaluation: and ensure the quality of our dataset; we conducted crowdsourcing study? with the of verifying goal the feasibility of answer construction: Each partici- Automatic metrics: The relevance of the support- ing documents in our WikiHowQA dataset is first evaluated using the Novel N-Gram Percentage met- ric (See et al ,2017) as a measure of word intersec- tion between answers and documents. This metric commonly used for summarization datasets, allows for a strong upper bound comparison. Results pre- sented in Table 2 show that the passage answers correspond to the content of the sup in our dataset porting documents with a relatively high n-gram percentage score, similar to that of HOWSUMM We also include results for the Docu- METHOD. Understanding Conference (DUC) as an up- ment per bound as it is a high quality human crafted sum reported by Fabbri et al.) marization dataset as Secondly; the report we average coverage density compression and metrics for WIKI- Grusky et al,, 2018). These metrics are HowQA commonly used to characterize the quality and dif- ficulty of summarization tasks. We follow Fabbri for et al. (2019) who adapted these measures the The coverage, density and com MDS use case. pression scores for WIKIHOwQA are 0.89, 1.86 and 81.46 respectively. It is important to note that Tejaswin et al. (2021) and Bommasani and crowdsourcing studies were reviewed and ap- All our proved by the Human Research Ethics Committee of RMIT University: Surge Al as our data labeling platform We use which provides workforce of highly skilled and educated native speakers, ensuring high-quality data labeling at scale allowing for higher quality labeling compared to traditional platforms such as Mechanical Turk_ All differences are statistically significant (Student t-test, p-value 0.01) This is consistent with previous two annotations_ similar research (Qu et al,, 2019; Bolotova et al 2020) and indicates that the annotators had good agreement with each other 0n the parts of the doc- uments that could be used for associated answer construction. Overall, these findings demonstrate that the provided documents are relevant and con- tain the necessary information for the answers. of which was the original category derived from the WikiHow website, while the remaining three randomly selected from the other 18 cate- were gories This task was designed to ensure that the paying attention and had read the annotator was question carefully. The tasks were rejected if the classification was incorrect. The second evaluation required annotators to read the answer and the sup- provided highlighting porting document, and use a tool to mark any words or phrases in the supporting document that were directly relevant to the given answer; Or could be used to reason Or conclude it In the final step of the process, annotators evalu- ated the relevance of the provided documents to the :Very five-point Likert scale (from 0, answer on a to 4, "Very Relevant" ) Irrelevant" , In total, 31 randomly selected QA pairs and their associated supporting documents were annotated, resulting in being 104 participant tasks, with only four rejected: Each assessment was carried out by three workers, receiving compensation of SO.7 per completed task The study8 involved 22 fluent English speakers. Fig: 3 illustrates a randomly selected result of this annotation process, as captured in the (simplified) annotation interface used in the study: Baseline Models 4 describes WIKIHOwQA This section baseline The task is to generate an models. answer to NFQ grounded in a set of relevant documents from which the answer can be reasoned or concluded. for performance evalua- As a reference point Retrieval-Augmented Generation tion; we use the (RAG) (Lewis et al,, 2020b) architecture, which is a common choice for abstractive summarization and QA tasks Fan et al,,2019). Since our dataset small predefined set of relevant support- provides ing documents for each question, use a retriever we to filter out passages from those documents that are less relevant to the question, to help fit more relevant passages into the limited input of a gener- ator: Then; a generator is tasked with generating an answer conditioned on input passages. We ex perimented with DPR (Petroni et al,, 2021) as the BART-large (Lewis et al,, 2020a) as retriever and the generator: The results showed that the workers deem the supporting documents to be mostly "Somewhat Relevant" in relation to the given answers, with a mean document relevance score of 2.59. The inner- annotator agreement is moderate, with a Cohen'$ Kappa (Artstein and Poesio, 2008) value of 0.51 indicating reasonable consistency in the annota- assessments of document relevance In terms tors of related text selections within the documents, the overlap score between assessor was pairs mean 0.65, calculated using the overlap coefficient (Vi- jaymeena and Kavitha, 2016): building model input; When we first ranked passages from supporting documents based on using their relevance to the question DPR, then truncated them t0 retain the maximum number of complete passages in the ranking that fit with- ing the maximum input length of BART: Passages were then reordered based on their source docu- string ments, and combined into a single input of the format question

doc 1 pas ~lp>

doc sage 1

. 2 pas Finally, BART is trained? to 1 . sage where H1 and Hz are the sets of unique words from 9Training and decoding parameters are in App D Interface with exact annotation instructions in App: evaluation experiment. crowdsourcing' To facili- generated this, a set of modified answers by tate we deliberately introducing factual errors into a sub- set of the WIKIHOwQA. was achieved by This randomly selecting 5 QA from each of the pairs 19 question categories available on WikiHow and systematically incorporating various types of inac- curacies, aS outlined in Table 4, into 4 out of the 5 answers, resulting in an average of 4.5 modifi- cations per answer: We then manually reviewed they the deteriorated answers to ensure maintained both contextual relevance and grammatical correct- Following this, we conducted two separate ness evaluation rounds involving a total of 34 workers who assessed all 95 QA pairs It's important to note that no worker participated in both rounds ensuring an unbiased evaluation. generate answers conditioned on this context_ gold To establish a stronger baseline, we also evalu - ated the performance of the GPT-3 (Brown et al. 2020) variant text-davinci-00310 in a zero-shot set- ting: larger than As model at least 4OOx times BART; it forms an upper bound for model perfor- We compared two prompt kinds for GPT-3, mance one with the question and another with the question and relevant passages retrieved via DPR. Standard automatic evaluation metrics for ab- stractive QA, Rouge-X and BertScore, are reported However; these scores are known to in Table 3 poorly reflect actual model performance (Deutsch and Roth; 2021; Krishna et al,, 2021). While re- cent QA-based evaluation metrics for summarisa- tion seem to better correlate with human judge- ments (Deutsch et al,, 2021; Scialom et al,, 2021), their adaptation to NFQA is not straightforward and remains an important area of research, as an swers in NFQA are expected to contain facts not mentioned in supporting documents. Instead, we report the results obtained through our human eval- uation framework as a more accurate measure of model performance (Sec. 5.2). The first trial aimed to establish the baseline per- formance of the standard evaluation framework, in which assessors evaluate answers without the assistance of any on-screen relevant information. Each HIT included one QA where the answer pair may had been deteriorated. Similar to Sec. 3.3 HITs began with the attentiveness test of question Assessors then indicated category classification. their familiarity with the question and evaluated five-point Lik- the usefulness of the answer on Unfamiliar" / *Very Useless' ~Very ert scale (from Useful"). Finally; they Very Very Familiar ' to were tasked with highlighting spans within the an- misleading, swer that they found either useful 0 understanding thereby providing a more nuanced of the answer' s perceived value. How To Evaluate How-To Answers 5 reliability of MD-NFQA eval- To further ensure the uation, simple human evaluation we introduce a framework, which we then employ to assess the performance of baselines in our benchmark. Evaluating Human Evaluation 5.1 The second trial featured the same evaluation provided except this time assessors tasks; were with highlighted relevant passages for each sen- tence in the answer; accessible by clicking on the sentence. Top-ranked passages according to DPR ensuring (Sec. 4) were selected for each sentence, the overall length is under 5000 characters to fit Passages were displayed in their on the screen. original order; grouped by source documents. Pas- sage highlighting came from the model proposed by Bolotova et al. (2020) , designed to simulate user gaze during NFQA evaluation: Assessors could freely examine the passages while determining the usefulness of the answer and selecting useful O misleading spans in the answer conventional human eval- Initially, employed we as described in previous stud- uation approach; ies (Fan et al,,2019; Krishna et al,,2021), where presented with both model-generated assessors are standard randomized and answers in gold or - der and asked to select their preferred response However; our findings indicated that, even when presented with standard questions and evalu- gold ated by high-performing assessors, simpler model- generated frequently (489 of the answers were time) preferred over reference answers, despite con- taining factual inaccuracies. Manual inspection re- vealed that choosing between two well-formulated challenging for participants, in line answers was with previous research of Krishna et al.; Clark et al. To quantify the ability of annotators to identify fac- we conducted tual inconsistencies in answers The choice of this presentation form was based on several factors. Firstly, it allowed for a more in- For interfaces and implementation details refer to App. E Ohttps: /beta.openai com/docs/models gpt-3 Table 4: Examples of deterioration types teractive and engaging experience for the assessors enabling them to delve deeper into the context of each sentence. Secondly, it provided a clear visual cue to the assessors about the relevance of each sentence, thereby facilitating a more accurate and efficient evaluation process. Given that assessors challenges due to their unfamiliarity with often face the topic O potential distractions, we aimed to en- hance their ability to assess with higher accuracy by providing relevant passages as supporting infor- This approach; designed to be simple yet mation_ effective, prioritized the provision of relevant infor- mation over more elaborate feedback mechanisms tracking, which are potential ar- like clicks or gaze After comparing different eas for future research forms of passage presentation through trial runs and discussions among the authors, we found that offering easy accessibility by clicking o the form the sentence was the most effective. Figure 4: Inconsistency detection rate (IDR) by type tion of almost all deterioration types in the second trial. Notably, the detection rate for sentence nega- tions and number swaps increased by 131% and 158%, respectively: These increases were statisti- cally significant; as confirmed by Student's t-tests 0.01 for sentence nega- statistic (t 4.2,p 0.01 for number tions; t statistic = 3.18,p A detailed explanation of the answer de- swaps). terioration process, including examples, as well as the interfaces utilized in the human study trials, is provided in App. E. measure of the ability to spot factual in- As consistencies, we calculated the inconsistency de- tection rate (IDR), defined as the average percent- age of identified deteriorations deterioration A was marked as identified if it was highlighted as misleading span. Results demonstrate a part of a statistically significant difference (Student's t-test; 2.57, p statistic 0.01) in IDR between t = the first and the second trials, with the IDR scores of 0.21 and 0.35. In addition to this, we also cal- culated the number of False Positive IDRs. False positives were defined as answers that were marked being misleading by an asseSSOr; even though as khey The first trial included four false were not. positives, and the second trial two. How Good Are The Baselines, Really? 5.2 We evaluated the effectiveness of lower- and upper- using bound models from Sec. 4 our human evalua- tion framework Specifically, we compared models through A/B testing (Krishna et al,, 2021) while incorporating highlighted relevant passages as ref- After classifying erence sources of information. the question category as an attentiveness task, as- sessors were instructed t0 evaluate the usefulness of a pair of answers and one model gold one and select the more useful answer: Answers were presented in random order without disclosing the provided ranked relevant passages for source. We separately; each sentence in each al- answer to low participants to make more informed decisions about factual consistency: While question familiarity scores were consis- tent between the two trials, IDR improvement persistent and statistically significant (Stu- was 2.18,p dent' s statistic 0.05) t-test t in the second trial for both high and low famil- iarity: This suggests that participants were able to assess more effectively with an access to rele- regardless of their fa- vant information sources; miliarity with a particular question. As shown in general improvement in detec- Fig: 4, there was In this experiment; 28 workers evaluated gold questions against an- answers to 100 random test BART; text-davinci-003 , and swers from DPR To eliminate potential text-davinci-003 . DPR bias, we ensured that workers did not evaluate the Acknowledgments question twice. Each HIT was compensated same with $0.5. deepest gratitude to We extend Mr: Luke our Gallagher for his invaluable contributions to this study: His insightful comments and contributions We also wish have greatly enriched our work: to acknowledge the financial support provided by the Australian Research Council (Grant No DP180102687), which has been instrumental in facilitating this research: Human evaluation results are available in Ta- All differences are statistically significant ble 3. 0.01). In comparison with (Student's t-tests, p the initial attempt at human evaluation, the sim BART model scored lower; while still pler DPR being preferred over answers in some cases gold Unlike the automatic evaluation; human evaluation supports our initial observation that the quality of answers from text-davinci-003 is much higher than from BART: When prompted with supporting pas- sages, text-davinci-003 generates significantly bet- ter answers based on human judgement (Table 3), which highlights the value of reference documents While text-davinci-003 mainly in WIKIHOwQA. generated grammatically plausible and logicaly sound answers, our manual analysis revealed that it frequently failed at factual consistency and overall coherency, especially for questions that require step-by-step instruction. Coupled with the fact that preferred answers from text-davinci- assessors still proposed evaluation 003 in some of these cases, the framework should be further improved to guide as- The problem is judgement. sessors better in their twofold; first, provided relevant passages do not al- ways include the required information to fact-check the answer; second, the presentation and fluency of model answers misleads assessors even when the overall instruction is impossible to follow. This re- quires an additional skill-set for assessors, and we leave the research of a better evaluation framework specifically tailored to instructions for future work Limitations possible limitations of In this section we discuss our work; and present interesting avenues for future investigation: First, not all documents in WIK - equally useful; IHOwQA documents some are are overall less relevant, some contain very rele- vant bits alongside generally irrelevant information More accurate passage or span selection may be required for models to generate better answers_ Another limitation is the focus on highlighted aiding human relevant passages as the method of evaluators in assessing factual correctness. While this approach helped to identify some factual incon- sistencies and was proven to also make assessments faster in previous research (Bolotova et al,, 2020), additional techniques should be considered for their potential to deliver further improvements. Unfortunately, due to limitations of the evalua- tion interface used, we were not able capture how frequently annotators clicked on sentences to see aiding - passages. Click ratio data could have yielded valuable insights into when and how often asses- provided information. sorS referred to the While we considered a range of answer deterio- ration types, it does not fully align with the kinds of hallucinations that neural NFQA models may prO- duce in the wild. Further research is needed to anal- yse the robustness of these models in real-world One direction is to adopt the compre- scenarios_ hensive evaluation framework of abstractive neural models proposed by (Dou et al,,2022) for NFQA Conclusion 6 In this work we have presented a new dataset and benchmark for multi-document non-factoid ques- tion answering, WIKIHOwQA, sourced from the WikiHow website and consisting of passage-level how to" questions. Our released test answers to critical gap in currently collections fills avail- able resources and evaluation testbeds for multi- document non-factoid QA, a widely occurring in- We have also presented baseline formation task: performance benchmarks, and introduced and em- ployed an information-augmented human evalua- tion framework that improves the reliability of QA annotations. Still, manual failure analysis revealed remaining challenges in the evaluation of convinc- ing but factually incorrect model answers_ experimented with text In our baselines, we parsed directly from the HTML source code How ever; it is important to note that we also provide of using the Markdown files, which pre- the option formatting information such as tables and serve lists. This could be useful for certain cases where formatting is important: Furthermore, some ref- erence HTML pages contain pagination and long comment sections that we did not consider when scraping the data. Finally, the evaluation of passage-level QA re- challenging task; both for human and auto- mains a matic evaluation frameworks. Difficulties include assessment of factual correctness and accurate overall consistency, especially when answers re- quire complex reasoning based on multiple sources Advancements in automatic evaluation of similar tasks, such as summarisation, could be adopted for NFQA. OuCmd DOMI , Uuy relgenlal, Guy LCV NIICHal Shmueli-Scheuer and David Benjamin Sznajder; Konopnicki. 2021. multi-document Howsumm: summarization dataset derived from wikihow articles_ CoRR, abs/2110.03179. Tom B. Brown, Benjamin Mann, Nick Ryder; Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam; Girish Sastry; Amanda Askell, Sandhini Ariel Herbert-Voss Agarwal, Gretchen Krueger; Tom Henighan, Rewon Child, Aditya Ramesh;, M Ziegler; Jeffrey Daniel Wu Clemens Winter; Christopher Hesse, Mark Chen, Eric Sigler; Mateusz Litwin, Scott Gray, Benjamin Chess_ Jack Clark, Christopher Berner; Sam McCandlish, Alec Radford, Ilya Sutskever; and Dario Amodei 2020. Language models are few-shot learners_ In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12 2020, virtual. Jorg Ricardo J. G. B. Campello, Davoud Moulavi, and Sander: 2013. Density-based clustering based on hi- erarchical density estimates_ In Advances in Knowl- Pacific-Asia Mining; edge Discovery and Data I7th PAKDD 2013, Gold Coast, Australia, Conference, April 14-17, 2013, Proceedings, Part II, volume 7819 of Lecture Notes in Computer Science, pages 160 172. Springer: Elizabeth Clark, Tal August; Sofia Serrano Nikita Haduong; Suchin Gururangan, and Noah A Smith: Evaluating 2021. All that' s human is not gold: In Proceedings human evaluation of generated text: Meeting of the Association for of the S9th Annual Computational Linguistics and the Ith International Joint Conference on Natural Language Processing, ACLJIJCNLP 2021 , Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 7282-7296. Associa- tion for Computational Linguistics. Daniel Cohen and W. Bruce Croft. 2016. End t0 end long short term memory networks for non-factoid Proceedings of the 2016 ACM question answering: In International Conference on the Theory of Informa- tion Retrieval, ICTIR 16, page 143-146, New York; NY, USA Association for Computing Machinery: Oren Kalinsky, Yftah Ziser; Nachshon Cohen; and Alessandro Moschitti. 2021. Wikisum: Coherent summarization dataset for efficient human-evaluation_ Proceedings of the S9th Annual Meeting of the As- In sociation for Computational Linguistics and the Ith International Joint Conference on Natural Language Processing Volume 2: Short Papers) , pages 212-219, Online. Association for Computational Linguistics. Trang Dang: 2006. DUC 2005: Hoa Evaluation of question-focused summarization systems. In Pro ceedings of the Workshop on Task-Focused Summa- rization and Question Answering, pages 48-55, Syd ney, Australia. Association for Computational Lin- guistics References Now the paper ends. Question: What are some ways the WikiHOwQA dataset and human evaluation framework could be further improved?