Knowledge Graph Ingestion — Concept Extraction Standards

This document describes the industry-standard methodologies used by the agent-utilities Knowledge Graph to automatically extract, organize, and align concepts from ingested codebases and research papers.

Standards Applied

SKOS — Simple Knowledge Organization System (W3C)

Specification: W3C SKOS Reference

SKOS is the W3C standard for representing taxonomies, classification schemes, and controlled vocabularies in RDF. This repository use SKOS vocabulary semantics for concept relationships within the Knowledge Graph:

SKOS Property	KG Edge Type	Meaning
skos:prefLabel	name property	Primary human-readable label
skos:broader	BROADER edge	Parent concept in hierarchy
skos:narrower	NARROWER edge	Child concept in hierarchy
skos:related	RELATED_CONCEPT edge	Non-hierarchical association
skos:ConceptScheme	scheme property	The project this concept belongs to
skos:hasTopConcept	Top-level concept nodes	Root concepts of a scheme

ConceptScheme Per Project

Each ingested codebase produces one ConceptScheme identified by the project name. For example, agent-utilities produces the scheme agent-utilities with its 5 pillar top concepts: ORCH, KG, AHE, ECO, OS.

External codebases (e.g., networkx, pydantic-ai) produce their own concept schemes with hierarchies auto-generated from package structure.

SSSOM — Standardized Semantic Sets of Mappings

Specification: SSSOM GitHub

SSSOM is the standard for representing concept mappings between different ontologies or concept schemes. This repository use SSSOM-style metadata on ANALOGOUS_TO edges:

SSSOM Field	KG Property	Purpose
subject_id	Source node ID	Concept from source scheme
object_id	Target node ID	Concept from target scheme
predicate_id	Edge type	Relationship type (e.g., ANALOGOUS_TO)
mapping_justification	source property	How the mapping was discovered
confidence	score property	Similarity/confidence score

CodeTaxo — LLM-Driven Code Taxonomy Expansion (ACL 2024)

CodeTaxo is a method for using LLMs to expand taxonomies from codebases. The key insight is that code-language prompts (docstrings, function names, module structure) provide strong signals for concept hierarchies.

This repository's 3-tier extraction pipeline implements the CodeTaxo approach:

Concept Extraction Pipeline

Tier 1: Structural Extraction (No LLM Required)

Parses the codebase structure to extract initial concept nodes:

• Package tree → Top-level hierarchy (e.g., agent_utilities/ → agent-utilities scheme)
• Module names → Mid-level concepts (e.g., knowledge_graph/ → "Knowledge Graph")
• Class/function names → Leaf concepts from public API
• README sections → Theme extraction
• CONCEPT:ID tags → Explicit concept declarations in code comments

Output: Concept nodes with extraction_method: "structural", connected by BROADER/NARROWER edges reflecting the package hierarchy.

Tier 2: Semantic Clustering (LLM-Powered)

For codebases with >20 modules, uses the LLM to cluster related modules into higher-level domain concepts:

1. Embed all module names + docstrings
2. Cluster via semantic similarity
3. LLM names each cluster (e.g., "authentication", "data pipeline", "orchestration")
4. Creates Concept nodes at level=0 (top-level themes) with extraction_method: "llm"

Pydantic Schema Enforcement: Uses pydantic-ai output_type for grammar-constrained JSON decoding — zero regex parsing, zero JSON errors.

Tier 3: Cross-Reference Alignment (LLM-Powered)

Compares extracted concepts against existing KG concepts and research paper topics:

1. For each new concept, find similar existing Concept and Article nodes
2. Create ANALOGOUS_TO edges with SSSOM-style metadata
3. Discover cross-domain innovations via analogy matching

OWL Reasoning Integration

After concept extraction and alignment, the OWL reasoning engine runs a promote → reason → downfeed cycle:

1. Promote: Exports KG edges to the OWL backend
2. Reason: Applies transitive and symmetric closure rules
3. Downfeed: Writes inferred relationships back to the KG

This discovers implicit relationships that weren't directly extracted. For example, if A ANALOGOUS_TO B and B ANALOGOUS_TO C, the reasoner infers A ANALOGOUS_TO C.

Triggering OWL Reasoning

The promote → reason → downfeed cycle is implemented in agent_utilities/knowledge_graph/core/owl_bridge.py and runs automatically as one of the consolidated maintenance _tick_* jobs (see core/engine_tasks.py::_maintenance_scheduler_loop) behind the background throttle gate. There is no dedicated MCP action to invoke it directly.

Analysis output

Synthesis and deep extraction intentionally request freeform Markdown. The current analyzer returns an explicit stage envelope plus the generated text; this avoids model-specific structured-output failures while keeping the outer tool response stable and inspectable.

MCP Tool Reference

Tool	Action/Mode	Layer	LLM Required
graph_search	mode='discover'	Native retrieval	No
graph_analyze	action='synthesize'	Retrieval + synthesis	Yes
graph_analyze	action='deep_extract'	Retrieval + deep extraction	Yes
graph_analyze	action='background_research'	Async retrieval + synthesis + extraction	Yes
graph_analyze	action='enrichment_coverage'	Coverage gauge	No
graph_ingest	action='rebuild_indexes'	HNSW index mgmt	No

Note: OWL reasoning has no standalone MCP action — it runs on the consolidated maintenance scheduler (see Triggering OWL Reasoning above).

See also: Vector Index Lifecycle for details on HNSW index management, the drop→ingest→build cycle, and search performance tiers.

Environment Configuration

The KG MCP server relies on the unified agent-utilities configuration for LLM connectivity. Ensure the operator's ~/.config/agent-utilities/config.json is configured:

{
  "chat_models": [
    {
      "id": "qwen/qwen3.8-27b",
      "provider": "openai",
      "intelligence_level": "normal"
    }
  ],
  "embedding_models": [
    {
      "id": "embedding-model",
      "provider": "openai",
      "base_url": "https://embedding-api.example.test/v1"
    }
  ]
}

Authentication is handled through a runtime credential reference in AgentConfig, resolved from explicit process injection, the private XDG runtime-secret source, or the configured durable secret backend.