Darius Knowledge Hub

Large Language Models (LLMs)

2 min read

Large Language Models (LLMs)

Created: =dateformat(this.file.ctime,"dd MMM yyyy, hh:mm a") | Modified: =dateformat(this.file.mtime,"dd MMM yyyy, hh:mm a") Tags: knowledge

Overview

Introduction

Perplexity Metric (PPL)

  • metric to determine the accuracy of next token prediction for language models. where lower perplexity, better the model
  • the metric applies specifically to classical language models (sometimes called autoregressive or causal language models) and is not well defined for masked language models like BERT
  • defined as the exponentiated average negative log-likelihood of a sequence
  • Intuitively, it can be thought of as an evaluation of the model’s ability to predict uniformly among the set of specified tokens in a corpus.
  • the tokenization procedure has a direct impact on a model’s perplexity which should always be taken into consideration when comparing different models.
  • Evaluation Metrics for Language Modeling

Theoretical References

Papers

Articles

Courses

Code References

Methods

Tools, Frameworks

Graph View

Backlinks