Generative AI

5Study of State-of-the-Art Performance Metrics in NLP: Specifically for Text Summarization in the Medical Domain Using the SumPubMed Dataset

M. Sridhar¹, Mohammad Irfan^2* and Sukumar Kathirmani³

¹NSB Academy, Business School, Bangalore, India

²Christ University, Bangalore, India

³Quelit Innovations Pvt Ltd., Chennai, India

Abstract

Text summarization is becoming very important given the number of documents produced each year across domains. In this paper we explore the various traditional metrics for text summarization, such as ROUGE, BLEU, METEOR, etc., and look at improving the performance of the existing metric by taking the state-of-the-art untrained metric SUPERT, and clubbing it with a readability score and a penalty for long summaries. The SUMPUBMED dataset was used for this research and a BERT extractive summarizer was used for generating the summaries. It was found that using a readability score with an unsupervised metric such as SUPERT helped in assessing the quality of the summary more accurately than earlier metrics. We compared the metrics such as SUPERT scores and BERT scores with and without involving the human annotated summaries in the SUMPUBMED dataset and found that untrained metrics perform better than when involving a reference annotated summary.

Keywords: NLP, text summarization, metrics, BERT scores, SUPERT

5.1 Introduction

Text summarization is becoming very important given the number of documents produced each year across domains. Given the volumes ...