NAVER LABS Europe seminars are open to the public. This seminar is virtual and requires registration.
Date: 17th October 2024, 5:00 pm (CEST)
How to properly build a metric to evaluate AI-generated text?
About the speaker: Wenda Xu is finishing his PhD with the NLP group of the UC Santa Barbara, advised by Prof. William Wang and Prof. Lei Li. He is also a visiting scholar at CMU Language Technologies Institute. His research interests lie in the area of LLM evaluation and alignment (at pre-training, post-training and inference stages). In one sentence, he works on learning metrics that can assess LLM’s generation quality and align LLM with well defined feedback. His recent works were published at top conferences, including ICLR, AAAI, EMNLP, ACL, and NAACL.
Abstract: As Large Language Models (LLMs) continue to advance in natural language generation (NLG), the need for effective evaluation metrics becomes increasingly critical. This talk addresses three key challenges in constructing metrics for NLG: 1) High Cost of Human Annotations: We propose a self-supervised approach that eliminates the need for extensive human annotations, enabling to learn a model-based metric that strongly correlates with human judgments while being more generalizable. 2) Lack of Explainability: We introduce INSTRUCTSCORE, a fine-grained explainable metric that provides detailed annotations of error locations, severity levels, and error types. This offers a more informative and actionable assessment of generated text quality. 3) Self-Bias in LLM-based Metrics: We formally define and investigate the phenomenon of LLM self-bias, the tendency of a model to favor its own generations over others. Through extensive experiments, we demonstrate the prevalence of self-bias and propose strategies to mitigate it. By addressing these challenges, we aim to develop explainable, unbiased, and (almost) human-free metrics that effectively evaluate the quality of AI-generated text.