Tootfinder

Improving Factual Accuracy of Neural Table-to-Text Output by Addressing Input Problems in ToTTo
Barkavi Sundararajan, Somayajulu Sripada, Ehud Reiter
https://arxiv.org/abs/2404.04103

Improving Factual Accuracy of Neural Table-to-Text Output by Addressing Input Problems in ToTTo
Neural Table-to-Text models tend to hallucinate, producing texts that contain factual errors. We investigate whether such errors in the output can be traced back to problems with the input. We manually annotated 1,837 texts generated by multiple models in the politics domain of the ToTTo dataset. We identify the input problems that are responsible for many output errors and show that fixing these inputs reduces factual errors by between 52% and 76% (depending on the model). In addition, we obse…

This judge is a fascist in the pocket of Trump. She should be removed from this case and impeached. https://mastodon.social/@JoshuaHolland/112400339850273355

That is a heck of a quote. Simple. Factual. Poignant. Powerful. True.
From: @…
https://press.coop/@nytimes/112055921492425764

The New York Times :press: (@nytimes@press.coop)
Attached: 1 image How a Sudden Halt to In Vitro Fertilization Shook Alabama Couples Many suffered setbacks after the state supreme court ruled that frozen embryos were children. “Someone else’s opinion changed my future,” one woman said. #press https://www.nytimes.com/2024/03/07/health/ivf-alabama-families.html?utm_source=press.coop

This https://arxiv.org/abs/2309.03685 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csAI_…

PyGraft: Configurable Generation of Synthetic Schemas and Knowledge Graphs at Your Fingertips
Knowledge graphs (KGs) have emerged as a prominent data representation and management paradigm. Being usually underpinned by a schema (e.g., an ontology), KGs capture not only factual information but also contextual knowledge. In some tasks, a few KGs established themselves as standard benchmarks. However, recent works outline that relying on a limited collection of datasets is not sufficient to assess the generalization capability of an approach. In some data-sensitive fields such as education…

Anthropic announces Claude 3 Opus, Sonnet, and Haiku, aiming to improve factual accuracy; Opus and Sonnet are available now, and Haiku in the coming weeks (Rachel Metz/Bloomberg)
https://www.bloomberg.com/news/articles/20

AI Startup Anthropic Launches New Models for Chatbot Claude
The startup says new versions of Claude will be twice as likely to answer a question correctly.

FaaF: Facts as a Function for the evaluation of RAG systems
Vasileios Katranidis, Gabor Barany
https://arxiv.org/abs/2403.03888 https://

FaaF: Facts as a Function for the evaluation of RAG systems
Factual recall from a reference source is crucial for evaluating the performance of Retrieval Augmented Generation (RAG) systems, as it directly probes into the quality of both retrieval and generation. However, it still remains a challenge to perform this evaluation reliably and efficiently. Recent work has focused on fact verification via prompting language model (LM) evaluators, however we demonstrate that these methods are unreliable in the presence of incomplete or inaccurate information. …

How can we evaluate the factual accuracy of long answers from LLMs? Researchers from DeepMind / Stanford demonstrate a strategy that uses LLMs search to assess factuality: it's more accurate than human evaluation and 20x cheaper. h/t Marc Lanctot on Threads arxiv.org/abs/2403.18802

This https://arxiv.org/abs/2403.01432 has been replaced.
link: https://scholar.google.com/scholar?q=a

Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge
Large language models (LLMs) memorize a vast amount of factual knowledge, exhibiting strong performance across diverse tasks and domains. However, it has been observed that the performance diminishes when dealing with less-popular or low-frequency concepts and entities, for example in domain specific applications. The two prominent approaches to enhance the performance of LLMs on low-frequent topics are: Retrieval Augmented Generation (RAG) and fine-tuning (FT) over synthetic data. This paper e…

This https://arxiv.org/abs/2311.16842 has been replaced.
link: https://scholar.google.com/scholar?q=a

RELIC: Investigating Large Language Model Responses using Self-Consistency
Large Language Models (LLMs) are notorious for blending fact with fiction and generating non-factual content, known as hallucinations. To address this challenge, we propose an interactive system that helps users gain insight into the reliability of the generated text. Our approach is based on the idea that the self-consistency of multiple samples generated by the same LLM relates to its confidence in individual claims in the generated texts. Using this idea, we design RELIC, an interactive syst…

PRobELM: Plausibility Ranking Evaluation for Language Models
Zhangdie Yuan, Chenxi Whitehouse, Eric Chamoun, Rami Aly, Andreas Vlachos
https://arxiv.org/abs/2404.03818

PRobELM: Plausibility Ranking Evaluation for Language Models
This paper introduces PRobELM (Plausibility Ranking Evaluation for Language Models), a benchmark designed to assess language models' ability to discern more plausible from less plausible scenarios through their parametric knowledge. While benchmarks such as TruthfulQA emphasise factual accuracy or truthfulness, and others such as COPA explore plausible scenarios without explicitly incorporating world knowledge, PRobELM seeks to bridge this gap by evaluating models' capabilities to prioritise pl…

AI-generated biographies, often riddled with gross grammatical and factual errors, are appearing for sale online soon after the death of well-known people (Elizabeth A. Harris/New York Times)
https://www.nytimes.com/2024/02/18/books/ai-books-biographies.html

A Celebrity Dies, and New Biographies Pop Up Overnight. The Author? A.I.
Books — often riddled with gross grammatical and factual errors — are appearing for sale online soon after the death of well-known people.

The Oz government’s digital ID bill has become a lightning rod for conspiracy theories and fearmongering, despite it being “world’s best practice” and “completely sane,” privacy and security experts say.
Several Coalition and crossbench politicians claimed the bill was an attempt to introduce a Chinese Communist Party-style social credit score and a measure to control the public, despite no evidence or factual basis, after it passed the Senate.

How the digital ID bill became a lightning rod for conspiracies
The Albanese government's digital ID Bill has become a lightning rod for conspiracy theories, despite it being “completely sane”.

Discovered this because the author wrote a factual piece for Labour List - "Best Labour leaders we never had: Anthony Greenwood"
https://labourlist.org/2023/08/best-labour-leaders-we-never-had-anthony-greenwood/

This https://arxiv.org/abs/2311.16842 has been replaced.
link: https://scholar.google.com/scholar?q=a

RELIC: Investigating Large Language Model Responses using Self-Consistency
Large Language Models (LLMs) are notorious for blending fact with fiction and generating non-factual content, known as hallucinations. To address this challenge, we propose an interactive system that helps users gain insight into the reliability of the generated text. Our approach is based on the idea that the self-consistency of multiple samples generated by the same LLM relates to its confidence in individual claims in the generated texts. Using this idea, we design RELIC, an interactive syst…

Self-Consistent Decoding for More Factual Open Responses
Christopher Malon, Xiaodan Zhu
https://arxiv.org/abs/2403.00696 https://arxi…

Self-Consistent Decoding for More Factual Open Responses
Self-consistency has emerged as a powerful method for improving the accuracy of short answers generated by large language models. As previously defined, it only concerns the accuracy of a final answer parsed from generated text. In this work, we extend the idea to open response generation, by integrating voting into the decoding method. Each output sentence is selected from among multiple samples, conditioning on the previous selections, based on a simple token overlap score. We compare this "S…

I don't know if you've read this yet (not the book, the review) but it is an absolute masterpiece of how to deliver a completely devastating, but cordial and factual, takedown. Honestly, the last takedown I saw this powerful was Susan Fowler's blog post on Uber.

On Truthful Item-Acquiring Mechanisms for Reward Maximization
Liang Shan, Shuo Zhang, Jie Zhang, Zihe Wang
https://arxiv.org/abs/2402.14540 https://…

On Truthful Item-Acquiring Mechanisms for Reward Maximization
In this research, we study the problem that a collector acquires items from the owner based on the item qualities the owner declares and an independent appraiser's assessments. The owner is interested in maximizing the probability that the collector acquires the items and is the only one who knows the items' factual quality. The appraiser performs her duties with impartiality, but her assessment may be subject to random noises, so it may not accurately reflect the factual quality of the items. …

Een mooi overzichtsartikel met de feiten op een rijtje als het gaat om de oorsprong van het coronavirus. #labLeak
https://www.commondreams.org/opinion/covid-19…

What Might the US Owe the World for Covid-19?
A US-funded laboratory origin of Covid-19 would certainly constitute the most significant case of governmental gross negligence in history. The people of the world deserve transparency and factual answers on vital questions.

It really is a step backwards for most folk when #AI credentials mean more than factual accuracy.
A simple lookup table would have correctly shown that #Debian #Linux 10.x is called buster (trixie is the name of t…

A look at the US DOJ's lawsuit against Apple, comparisons to Microsoft suit in the 90's, basic factual errors, why it is clearly a political case, and more (Steven Sinofsky/Hardcore Software)
https://hardcoresoftware.learningbyshipping…

216. United States v. Apple (Complaint)
March 21, 2024 the United States filed an antitrust complaint against Apple. The filing is a new approach to antitrust and hinges on mostly how Apple chose to design the iPhone from the very start.

This https://arxiv.org/abs/2307.12371 has been replaced.
link: https://scholar.google.com/scholar?q=a

PSentScore: Evaluating Sentiment Polarity in Dialogue Summarization
Automatic dialogue summarization is a well-established task with the goal of distilling the most crucial information from human conversations into concise textual summaries. However, most existing research has predominantly focused on summarizing factual information, neglecting the affective content, which can hold valuable insights for analyzing, monitoring, or facilitating human interactions. In this paper, we introduce and assess a set of measures PSentScore, aimed at quantifying the preserv…

A good read from a smart guy.
Worth reading just to get factual scoop about QUOTE
"information wants to be free"
https://medium.com/@johnmarkoff/introduction-28fd6f7df6e0l

This https://arxiv.org/abs/2301.10389 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csIR_…

Counterfactual Editing for Search Result Explanation
Search Result Explanation (SeRE) aims to improve search sessions' effectiveness and efficiency by helping users interpret documents' relevance. Existing works mostly focus on factual explanation, i.e. to find/generate supporting evidence about documents' relevance to search queries. However, research in cognitive sciences has shown that human explanations are contrastive i.e. people explain an observed event using some counterfactual events; such explanations reduce cognitive load and provide a…

This https://arxiv.org/abs/2306.15328 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_sta…

Simulating counterfactuals
Counterfactual inference considers a hypothetical intervention in a parallel world that shares some evidence with the factual world. If the evidence specifies a conditional distribution on a manifold, counterfactuals may be analytically intractable. We present an algorithm for simulating values from a counterfactual distribution where conditions can be set on both discrete and continuous variables. We show that the proposed algorithm can be presented as a particle filter leading to asymptotical…

This https://arxiv.org/abs/2403.01548 has been replaced.
link: https://scholar.google.com/scholar?q=a

In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation
Large language models (LLMs) frequently hallucinate and produce factual errors, yet our understanding of why they make these errors remains limited. In this study, we delve into the underlying mechanisms of LLM hallucinations from the perspective of inner representations, and discover a salient pattern associated with hallucinations: correct generations tend to have sharper context activations in the hidden states of the in-context tokens, compared to the incorrect ones. Leveraging this insight…

And here is the trigger event, I reckon. The opinion writer we believe is basically a sock puppet for the media proprietor is back – and making some plainly false factual claims.
The easiest to prove: that in investigating their identity, News24 did not provide them with an opportunity to comment.
That is the work of a moment to take before the Press Ombud, at which point Independent Media has to choose between apologising or storming out again.
https://web.archive.org/web/20240315072705/https://www.iol.co.za/sundayindependent/analysis/debunking-the-lies-a-response-to-news24s-special-projects-propaganda-779c9b95-930a-4b70-b845-f710f63c897a

Google so badly wants to disown "Bard", it is ascribing it to competitors.
Me: What are the names of your top competitor offerings?
Gemini: [1]
Me: Are you sure about # 3?
Gemini: [2, yup it's all on me 🤦‍♂️]
https://gemini.google.com/share/c24cc7

This https://arxiv.org/abs/2312.08976 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csSE_…

Dynamic Retrieval-Augmented Generation
Current state-of-the-art large language models are effective in generating high-quality text and encapsulating a broad spectrum of world knowledge. These models, however, often hallucinate and lack locally relevant factual data. Retrieval-augmented approaches were introduced to overcome these problems and provide more accurate responses. Typically, the retrieved information is simply appended to the main request, restricting the context window size of the model. We propose a novel approach for …

About 50 NPR staffers sign a letter to NPR leadership calling for a public rebuke of the "factual inaccuracies and elisions" in Uri Berliner's essay (Ben Mullin/@benmullin)
https://twitter.com/benmullin/status/1780635242170028359

Ben Mullin (@BenMullin) on X
new: About 50 NPR employees sign a letter to CEO Katherine Maher and top editor Edith Chapin calling for, among other things, a public rebuke of the “factual inaccuracies and elisions” in Uri Berliner’s Free Press essay.

FLAME: Factuality-Aware Alignment for Large Language Models
Sheng-Chieh Lin, Luyu Gao, Barlas Oguz, Wenhan Xiong, Jimmy Lin, Wen-tau Yih, Xilun Chen
https://arxiv.org/abs/2405.01525

FLAME: Factuality-Aware Alignment for Large Language Models
Alignment is a standard procedure to fine-tune pre-trained large language models (LLMs) to follow natural language instructions and serve as helpful AI assistants. We have observed, however, that the conventional alignment process fails to enhance the factual accuracy of LLMs, and often leads to the generation of more false facts (i.e. hallucination). In this paper, we study how to make the LLM alignment process more factual, by first identifying factors that lead to hallucination in both align…

This https://arxiv.org/abs/2403.18802 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csCL_…

Long-form factuality in large language models
Large language models (LLMs) often generate content that contains factual errors when responding to fact-seeking prompts on open-ended topics. To benchmark a model's long-form factuality in open domains, we first use GPT-4 to generate LongFact, a prompt set comprising thousands of questions spanning 38 topics. We then propose that LLM agents can be used as automated evaluators for long-form factuality through a method which we call Search-Augmented Factuality Evaluator (SAFE). SAFE utilizes an …

This https://arxiv.org/abs/2310.14894 has been replaced.
link: https://scholar.google.com/scholar?q=a

Local Universal Explainer (LUX) -- a rule-based explainer with factual, counterfactual and visual explanations
Explainable artificial intelligence (XAI) is one of the most intensively developed area of AI in recent years. It is also one of the most fragmented with multiple methods that focus on different aspects of explanations. This makes difficult to obtain the full spectrum of explanation at once in a compact and consistent way. To address this issue, we present Local Universal Explainer (LUX), which is a rule-based explainer that can generate factual, counterfactual and visual explanations. It is ba…

Dataset Artefacts are the Hidden Drivers of the Declining Disruptiveness in Science
Vincent Holst, Andres Algaba, Floriano Tori, Sylvia Wenmackers, Vincent Ginis
https://arxiv.org/abs/2402.14583

Dataset Artefacts are the Hidden Drivers of the Declining Disruptiveness in Science
Park et al. [1] reported a decline in the disruptiveness of scientific and technological knowledge over time. Their main finding is based on the computation of CD indices, a measure of disruption in citation networks [2], across almost 45 million papers and 3.9 million patents. Due to a factual plotting mistake, database entries with zero references were omitted in the CD index distributions, hiding a large number of outliers with a maximum CD index of one, while keeping them in the analysis [1…

And here is the trigger event, I reckon. The opinion writer we believe is basically a sock puppet for the media proprietor is back – and making some plainly false factual claims.
The easiest to prove: that in investigating their identity, News24 did not provide them with an opportunity to comment.
That is the work of a moment to take before the Press Ombud, at which point Independent Media has to choose between apologising or storming out again.
https://web.archive.org/web/20240315072705/https://www.iol.co.za/sundayindependent/analysis/debunking-the-lies-a-response-to-news24s-special-projects-propaganda-779c9b95-930a-4b70-b845-f710f63c897a

Filing: OpenAI rebuts Elon Musk's lawsuit, saying OpenAI didn't violate its founding agreement because "there is no founding agreement, or any agreement at all" (Rachel Metz/Bloomberg)
https://www.bloomberg.com/news/articles/20

OpenAI Calls Elon Musk’s Claims ‘Incoherent’ in Court Filing
OpenAI rebutted Elon Musk’s claims against it in a court filing, saying that the Tesla Inc. billionaire’s claims “rest on convoluted — often incoherent — factual premises.”

Semantic Data for Humanities and Social Sciences (SDHSS): an Ecosystem of CIDOC CRM Extensions for Research Data Production and Reuse
Francesco BerettaLARHRA, LARHRA PHN
https://arxiv.org/abs/2402.07531

Semantic Data for Humanities and Social Sciences (SDHSS): an Ecosystem of CIDOC CRM Extensions for Research Data Production and Reuse
Given the challenge of giant knowledge graphs created by major eco-nomic actors, which could virtually replace research in the Humani-ties and Social Sciences (HSS) in responding to public concerns, thequestion arises of how to increase the value of research data throughtheir publication and networking, applying the FAIR principles. Bothan epistemological and a semantic analysis show that the most rel-evant part of research data is factual information, understood as arepresentation of the objec…

WikiFactDiff: A Large, Realistic, and Temporally Adaptable Dataset for Atomic Factual Knowledge Update in Causal Language Models
Hichem Ammar Khodja, Fr\'ed\'eric B\'echet, Quentin Brabant, Alexis Nasr, Gw\'enol\'e Lecorv\'e
https://arxiv.org/abs/2403.14364

WikiFactDiff: A Large, Realistic, and Temporally Adaptable Dataset for Atomic Factual Knowledge Update in Causal Language Models
The factuality of large language model (LLMs) tends to decay over time since events posterior to their training are "unknown" to them. One way to keep models up-to-date could be factual update: the task of inserting, replacing, or removing certain simple (atomic) facts within the model. To study this task, we present WikiFactDiff, a dataset that describes the evolution of factual knowledge between two dates as a collection of simple facts divided into three categories: new, obsolete, and static…

This https://arxiv.org/abs/2402.18045 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csCL_…

Multi-FAct: Assessing Multilingual LLMs' Multi-Regional Knowledge using FActScore
Large Language Models (LLMs) are prone to factuality hallucination, generating text that contradicts established knowledge. While extensive research has addressed this in English, little is known about multilingual LLMs. This paper systematically evaluates multilingual LLMs' factual accuracy across languages and geographic regions. We introduce a novel pipeline for multilingual factuality evaluation, adapting FActScore(Min et al., 2023) for diverse languages. Our analysis across nine languages …

Analyizing the Conjunction Fallacy as a Fact
Tomas Veloz, Olha Sobetska
https://arxiv.org/abs/2402.13615 https://arxiv.org/pdf/2402.1…

Analyizing the Conjunction Fallacy as a Fact
Since the seminal paper by Tversky and Kahneman, the conjunction fallacy has been the subject of multiple debates and become a fundamental challenge for cognitive theories in decision-making. In this article, we take a rather uncommon perspective on this phenomenon. Instead of trying to explain the nature or causes of the conjunction fallacy (intensional definition), we analyze its range of factual possibilities (extensional definition). We show that the majority of research on the conjunction …

This https://arxiv.org/abs/2402.18045 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csCL_…

Multi-FAct: Assessing Multilingual LLMs' Multi-Regional Knowledge using FActScore
Large Language Models (LLMs) are prone to factuality hallucination, generating text that contradicts established knowledge. While extensive research has addressed this in English, little is known about multilingual LLMs. This paper systematically evaluates multilingual LLMs' factual accuracy across languages and geographic regions. We introduce a novel pipeline for multilingual factuality evaluation, adapting FActScore(Min et al., 2023) for diverse languages. Our analysis across nine languages …

This https://arxiv.org/abs/2403.11169 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csCL_…

Correcting misinformation on social media with a large language model
Real-world misinformation can be partially correct and even factual but misleading. It undermines public trust in science and democracy, particularly on social media, where it can spread rapidly. High-quality and timely correction of misinformation that identifies and explains its (in)accuracies has been shown to effectively reduce false beliefs. Despite the wide acceptance of manual correction, it is difficult to be timely and scalable, a concern as technologies like large language models (LLM…

HyperMono: A Monotonicity-aware Approach to Hyper-Relational Knowledge Representation
Zhiwei Hu, V\'ictor Guti\'errez-Basulto, Zhiliang Xiang, Ru Li, Jeff Z. Pan
https://arxiv.org/abs/2404.09848

HyperMono: A Monotonicity-aware Approach to Hyper-Relational Knowledge Representation
In a hyper-relational knowledge graph (HKG), each fact is composed of a main triple associated with attribute-value qualifiers, which express additional factual knowledge. The hyper-relational knowledge graph completion (HKGC) task aims at inferring plausible missing links in a HKG. Most existing approaches to HKGC focus on enhancing the communication between qualifier pairs and main triples, while overlooking two important properties that emerge from the monotonicity of the hyper-relational gr…

This https://arxiv.org/abs/2404.11184 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csCL_…

FIZZ: Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document
Through the advent of pre-trained language models, there have been notable advancements in abstractive summarization systems. Simultaneously, a considerable number of novel methods for evaluating factual consistency in abstractive summarization systems has been developed. But these evaluation approaches incorporate substantial limitations, especially on refinement and interpretability. In this work, we propose highly effective and interpretable factual inconsistency detection method metric Fact…

This https://arxiv.org/abs/2402.13919 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csCL_…

SYNFAC-EDIT: Synthetic Imitation Edit Feedback for Factual Alignment in Clinical Summarization
Large Language Models (LLMs) such as GPT & Llama have demonstrated significant achievements in summarization tasks but struggle with factual inaccuracies, a critical issue in clinical NLP applications where errors could lead to serious consequences. To counter the high costs and limited availability of expert-annotated data for factual alignment, this study introduces an innovative pipeline that utilizes >100B parameter GPT variants like GPT-3.5 & GPT-4 to act as synthetic experts to generate h…

This https://arxiv.org/abs/2402.17097 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csCL_…

Re-Ex: Revising after Explanation Reduces the Factual Errors in LLM Responses
Mitigating hallucination issues is a key challenge that must be overcome to reliably deploy large language models (LLMs) in real-world scenarios. Recently, various methods have been proposed to detect and revise factual errors in LLM-generated texts, in order to reduce hallucination. In this paper, we propose Re-Ex, a method for post-editing LLM-generated responses. Re-Ex introduces a novel reasoning step dubbed as the factual error explanation step. Re-Ex revises the initial response of LLMs u…

Memory-Augmented Generative Adversarial Transformers
Stephan Raaijmakers, Roos Bakker, Anita Cremers, Roy de Kleijn, Tom Kouwenhoven, Tessa Verhoef
https://arxiv.org/abs/2402.19218

Memory-Augmented Generative Adversarial Transformers
Conversational AI systems that rely on Large Language Models, like Transformers, have difficulty interweaving external data (like facts) with the language they generate. Vanilla Transformer architectures are not designed for answering factual questions with high accuracy. This paper investigates a possible route for addressing this problem. We propose to extend the standard Transformer architecture with an additional memory bank holding extra information (such as facts drawn from a knowledge ba…

FIT-RAG: Black-Box RAG with Factual Information and Token Reduction
Yuren Mao, Xuemei Dong, Wenyi Xu, Yunjun Gao, Bin Wei, Ying Zhang
https://arxiv.org/abs/2403.14374

FIT-RAG: Black-Box RAG with Factual Information and Token Reduction
Due to the extraordinarily large number of parameters, fine-tuning Large Language Models (LLMs) to update long-tail or out-of-date knowledge is impractical in lots of applications. To avoid fine-tuning, we can alternatively treat a LLM as a black-box (i.e., freeze the parameters of the LLM) and augment it with a Retrieval-Augmented Generation (RAG) system, namely black-box RAG. Recently, black-box RAG has achieved success in knowledge-intensive tasks and has gained much attention. Existing blac…

SIFiD: Reassess Summary Factual Inconsistency Detection with LLM
Jiuding Yang, Hui Liu, Weidong Guo, Zhuwei Rao, Yu Xu, Di Niu
https://arxiv.org/abs/2403.07557

SIFiD: Reassess Summary Factual Inconsistency Detection with LLM
Ensuring factual consistency between the summary and the original document is paramount in summarization tasks. Consequently, considerable effort has been dedicated to detecting inconsistencies. With the advent of Large Language Models (LLMs), recent studies have begun to leverage their advanced language understanding capabilities for inconsistency detection. However, early attempts have shown that LLMs underperform traditional models due to their limited ability to follow instructions and the …

SIFiD: Reassess Summary Factual Inconsistency Detection with LLM
Jiuding Yang, Hui Liu, Weidong Guo, Zhuwei Rao, Yu Xu, Di Niu
https://arxiv.org/abs/2403.07557

SIFiD: Reassess Summary Factual Inconsistency Detection with LLM
Ensuring factual consistency between the summary and the original document is paramount in summarization tasks. Consequently, considerable effort has been dedicated to detecting inconsistencies. With the advent of Large Language Models (LLMs), recent studies have begun to leverage their advanced language understanding capabilities for inconsistency detection. However, early attempts have shown that LLMs underperform traditional models due to their limited ability to follow instructions and the …

Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledge
Xin Zhao, Naoki Yoshinaga, Daisuke Oba
https://arxiv.org/abs/2403.05189 …

Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledge
Acquiring factual knowledge for language models (LMs) in low-resource languages poses a serious challenge, thus resorting to cross-lingual transfer in multilingual LMs (ML-LMs). In this study, we ask how ML-LMs acquire and represent factual knowledge. Using the multilingual factual knowledge probing dataset, mLAMA, we first conducted a neuron investigation of ML-LMs (specifically, multilingual BERT). We then traced the roots of facts back to the knowledge source (Wikipedia) to identify the ways…

Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledge
Xin Zhao, Naoki Yoshinaga, Daisuke Oba
https://arxiv.org/abs/2403.05189 …

Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledge
Acquiring factual knowledge for language models (LMs) in low-resource languages poses a serious challenge, thus resorting to cross-lingual transfer in multilingual LMs (ML-LMs). In this study, we ask how ML-LMs acquire and represent factual knowledge. Using the multilingual factual knowledge probing dataset, mLAMA, we first conducted a neuron investigation of ML-LMs (specifically, multilingual BERT). We then traced the roots of facts back to the knowledge source (Wikipedia) to identify the ways…

FactCheck Editor: Multilingual Text Editor with End-to-End fact-checking
Vinay Setty
https://arxiv.org/abs/2404.19482 https://arxiv.org/pdf/2404.19482
arXiv:2404.19482v1 Announce Type: new
Abstract: We introduce 'FactCheck Editor', an advanced text editor designed to automate fact-checking and correct factual inaccuracies. Given the widespread issue of misinformation, often a result of unintentional mistakes by content creators, our tool aims to address this challenge. It supports over 90 languages and utilizes transformer models to assist humans in the labor-intensive process of fact verification. This demonstration showcases a complete workflow that detects text claims in need of verification, generates relevant search engine queries, and retrieves appropriate documents from the web. It employs Natural Language Inference (NLI) to predict the veracity of claims and uses LLMs to summarize the evidence and suggest textual revisions to correct any errors in the text. Additionally, the effectiveness of models used in claim detection and veracity assessment is evaluated across multiple languages.

UFO: a Unified and Flexible Framework for Evaluating Factuality of Large Language Models
Zhaoheng Huang, Zhicheng Dou, Yutao Zhu, Ji-rong Wen
https://arxiv.org/abs/2402.14690

UFO: a Unified and Flexible Framework for Evaluating Factuality of Large Language Models
Large language models (LLMs) may generate text that lacks consistency with human knowledge, leading to factual inaccuracies or \textit{hallucination}. Existing research for evaluating the factuality of LLMs involves extracting fact claims using an LLM and verifying them against a predefined fact source. However, these evaluation metrics are task-specific, and not scalable, and the substitutability of fact sources in different tasks is under-explored. To address these challenges, we categorize f…

Editing Knowledge Representation of Language Lodel via Rephrased Prefix Prompts
Yuchen Cai, Ding Cao, Rongxi Guo, Yaqin Wen, Guiquan Liu, Enhong Chen
https://arxiv.org/abs/2403.14381

Editing Knowledge Representation of Language Lodel via Rephrased Prefix Prompts
Neural language models (LMs) have been extensively trained on vast corpora to store factual knowledge about various aspects of the world described in texts. Current technologies typically employ knowledge editing methods or specific prompts to modify LM outputs. However, existing knowledge editing methods are costly and inefficient, struggling to produce appropriate text. Additionally, prompt engineering is opaque and requires significant effort to find suitable prompts. To address these issues…

This https://arxiv.org/abs/2401.17809 has been replaced.
link: https://scholar.google.com/scholar?q=a

SWEA: Changing Factual Knowledge in Large Language Models via Subject Word Embedding Altering
Model editing has recently gained widespread attention. Current model editing methods primarily involve modifying model parameters or adding additional modules to the existing model. However, the former causes irreversible damage to Large Language Models (LLMs), while the latter incurs additional inference overhead and fuzzy vector matching is not always reliable. To address these issues, we propose an expandable Subject Word Embedding Altering (SWEA) framework, which finds the fused embeddings…

This https://arxiv.org/abs/2404.05904 has been replaced.
initial toot: https://mastoxiv.page/@arXiv_csCL_…

The Hallucinations Leaderboard -- An Open Effort to Measure Hallucinations in Large Language Models
Large Language Models (LLMs) have transformed the Natural Language Processing (NLP) landscape with their remarkable ability to understand and generate human-like text. However, these models are prone to ``hallucinations'' -- outputs that do not align with factual reality or the input context. This paper introduces the Hallucinations Leaderboard, an open initiative to quantitatively measure and compare the tendency of each model to produce hallucinations. The leaderboard uses a comprehensive set…

This https://arxiv.org/abs/2310.01045 has been replaced.
link: https://scholar.google.com/scholar?q=a

Tool-Augmented Reward Modeling
Reward modeling (a.k.a., preference modeling) is instrumental for aligning large language models with human preferences, particularly within the context of reinforcement learning from human feedback (RLHF). While conventional reward models (RMs) have exhibited remarkable scalability, they oft struggle with fundamental functionality such as arithmetic computation, code execution, and factual lookup. In this paper, we propose a tool-augmented preference modeling approach, named Themis, to address…

Clue-Instruct: Text-Based Clue Generation for Educational Crossword Puzzles
Andrea Zugarini, Kamyar Zeinalipour, Surya Sai Kadali, Marco Maggini, Marco Gori, Leonardo Rigutini
https://arxiv.org/abs/2404.06186

Clue-Instruct: Text-Based Clue Generation for Educational Crossword Puzzles
Crossword puzzles are popular linguistic games often used as tools to engage students in learning. Educational crosswords are characterized by less cryptic and more factual clues that distinguish them from traditional crossword puzzles. Despite there exist several publicly available clue-answer pair databases for traditional crosswords, educational clue-answer pairs datasets are missing. In this article, we propose a methodology to build educational clue generation datasets that can be used to …