Skip to Content
ReferenceEvalsCompleteness

CompletenessMetric ✅

The CompletenessMetric class evaluates how thoroughly an LLM’s output covers the key elements present in the input. It analyzes nouns, verbs, topics, and terms to determine coverage and provides a detailed completeness score.

Basic Usage ✅

import { CompletenessMetric } from "@kastrax/evals/nlp";
 
const metric = new CompletenessMetric();
 
const result = await metric.measure(
  "Explain how photosynthesis works in plants using sunlight, water, and carbon dioxide.",
  "Plants use sunlight to convert water and carbon dioxide into glucose through photosynthesis."
);
 
console.log(result.score); // Coverage score from 0-1
console.log(result.info); // Object containing detailed metrics about element coverage

measure() Parameters ✅

input:

string
The original text containing key elements to be covered

output:

string
The LLM's response to evaluate for completeness

Returns ✅

score:

number
Completeness score (0-1) representing the proportion of input elements covered in the output

info:

object
Object containing detailed metrics about element coverage
string[]

inputElements:

string[]
Array of key elements extracted from the input
string[]

outputElements:

string[]
Array of key elements found in the output
string[]

missingElements:

string[]
Array of input elements not found in the output
object

elementCounts:

object
Count of elements in input and output

Element Extraction Details ✅

The metric extracts and analyzes several types of elements:

  • Nouns: Key objects, concepts, and entities
  • Verbs: Actions and states (converted to infinitive form)
  • Topics: Main subjects and themes
  • Terms: Individual significant words

The extraction process includes:

  • Normalization of text (removing diacritics, converting to lowercase)
  • Splitting camelCase words
  • Handling of word boundaries
  • Special handling of short words (3 characters or less)
  • Deduplication of elements

Scoring Details ✅

The metric evaluates completeness through linguistic element coverage analysis.

Scoring Process

  1. Extracts key elements:

    • Nouns and named entities
    • Action verbs
    • Topic-specific terms
    • Normalized word forms

  2. Calculates coverage of input elements:

    • Exact matches for short terms (≤3 chars)
    • Substantial overlap (>60%) for longer terms

Final score: (covered_elements / total_input_elements) * scale

Score interpretation

(0 to scale, default 0-1)

  • 1.0: Complete coverage - contains all input elements
  • 0.7-0.9: High coverage - includes most key elements
  • 0.4-0.6: Partial coverage - contains some key elements
  • 0.1-0.3: Low coverage - missing most key elements
  • 0.0: No coverage - output lacks all input elements

Example with Analysis ✅

import { CompletenessMetric } from "@kastrax/evals/nlp";
 
const metric = new CompletenessMetric();
 
const result = await metric.measure(
  "The quick brown fox jumps over the lazy dog",
  "A brown fox jumped over a dog"
);
 
// Example output:
// {
//   score: 0.75,
//   info: {
//     inputElements: ["quick", "brown", "fox", "jump", "lazy", "dog"],
//     outputElements: ["brown", "fox", "jump", "dog"],
//     missingElements: ["quick", "lazy"],
//     elementCounts: { input: 6, output: 4 }
//   }
// }

Related ✅

Last updated on
BiasContentSimilarity