Entity and Relation Extraction using OntoGPT and SPIRES

Bart Kleijngeld | March 3rd, 2024

Goal

Populate a knowledge graph from unstructured text with as much structure as we can

  • Entity and relation extraction from unstructured text
  • Grounding these terms on ontologies (aka entity linking)

SPIRES

Structured Prompt Interrogation And Recursive Extraction Of Semantics (SPIRES): A Method For Populating Knowledge Bases Using Zero-Shot Learning

Given a detailed, user-defined knowledge schema and an input text, SPIRES recursively performs prompt interrogation against an LLM to obtain a set of responses matching the provided schema. SPIRES uses existing ontologies and vocabularies to provide identifiers for matched elements.

Developed by the Monarch Initiative

  • Field of Bio Informatics
  • Involvement of Berkeley Lab

How it Works

Overview

  • Recursive prompting
    • For each attribute
  • Entity grounding
    • Services that provide LD URIs from ontologies

OntoGPT

  • Implementation of SPIRES
  • Model-agnostic, but easiest is to use OpenAI GPT
    • we use OpenAI GPT3.5 Turbo

Demo

Extraction of music artist information from Miles Davis Wikipedia page excerpt

Notes

  • Scalable method
  • High-level
    • Abstracts away the prompting
  • Relatively cumbersome schema due to how SPIRES works
  • Great grounding performance
    • let's have a look...

Performance

Current SPIRES accuracy is comparable to the mid-range of existing Relation Extraction (RE) methods, but greatly surpasses an LLM’s native capability of grounding entities with unique identifiers

This grounding method offers far more consistent mapping to unique identifiers than hallucination-prone LLM querying alone

Benchmarks: Entity Grounding

Precision of grounding 100 randomly selected entities

GPT3.5T (SPIRES) GPT4T (SPIRES) GPT3.5T GPT4T
GO 0.98 0.97 0.03 -
MUNDO 0.97 0.18 ≤ 0.01 ≤ 0.01
EMADA 1.00 1.00 wrong ontology -

Benchmarks: NER

Precision/recall for NER including grounding

Challenges and questions

  • Token limit exceeded very quickly with OpenAI
    • How many requests will this take?
  • Flaky: very different outcomes and performance with different models
  • How well does it perform, offset against impact on resources?
  • Very promising results
  • How to determine what to do next?

1. SSH into `bart-k-semantics` 2. ~/ontogpt-kg $ poetry shell 3. ~/ontogpt-kg $ ontogpt extract -i m.txt -t person `person` is the schema for music artist extraction which I've added. Make sure to generate a new Pydantic schema whenever you change the person schema: gen-pydantic ~/.cache/pypoetry/virtualenvs/ontogpt-kg-ipkDGH2g-py3.10/lib/python3.10/site-packages/ontogpt/templates/person.yaml

* With large general purpose LLMs such as these I can imagine a large probability for finding false positives (such as the EMADA wrong ontology choice by GPT3.5T) * Grounding with dedicated annotator services provides the precise scope/context to do it right, as in: if recall succeeds it's likely accurate.