GNNs in the LLM Era | Graph Machine Learning

1. Why Combine Graphs and LLMs?

LLMs are great at: language, reasoning over text, few-shot generalization, world knowledge in parameters.

LLMs are weak at: precise multi-hop relational traversal, combinatorial structure (symmetry, connectivity), updating factual graphs cheaply, guaranteed consistency over structured KBs.

GNNs are great at: encoding connectivity, permutation-invariant structure, inductive generalization on graphs, efficient inference on fixed topology.

GNNs are weak at: open-ended language, zero-shot reasoning, tasks needing broad world knowledge not in node features.

Intuition: LLM is the linguist. GNN is the map. You need both when your data is a network of text things - documents citing documents, products with descriptions, genes with papers, entities in a knowledge graph.

2. Text-Attributed Graphs (TAGs)

Each node carries raw text: paper abstract, product title, wiki paragraph, code docstring. This is the most common LLM+GNN setting in 2024–2026 research.

Pipeline options:

Freeze LLM → embed → GNN: cheap, fast GNN training
Finetune LLM + GNN end-to-end: best quality, expensive
Adapter layers: LoRA on LLM + light GNN on top

Refer: Chen et al. - Text-Attributed Graph Representation Learning (TAGLab)

3. LLM as Node Feature Encoder

# Step 1: embed each node's text with a frozen LM
from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
lm = AutoModel.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")

def embed_texts(texts):
    inputs = tokenizer(texts, padding=True, truncation=True, return_tensors="pt")
    with torch.no_grad():
        out = lm(**inputs)
    return out.last_hidden_state[:, 0]  # CLS token

data.x = embed_texts(node_text_list)  # shape [num_nodes, dim]

# Step 2: GNN on top  -  structure refines text embeddings
model = GAT(in_dim=data.x.size(1), hidden=256, out_dim=num_classes)

Why GNN after LLM? Two papers with similar abstracts may belong to different topics if one cites ML and one cites biology - citations (edges) disambiguate.

Why not LLM alone? Feeding all neighbors' text into context blows the context window on high-degree nodes and misses global graph structure.

4. GNN as Retriever for RAG

Standard RAG retrieves text chunks by vector similarity. GraphRAG-style systems retrieve by graph structure:

Build KG or document graph (entities, citations, sections)
GNN or graph embedding scores relevant subgraph / nodes
Pass retrieved nodes' text to LLM for answer generation

GNN captures multi-hop relations vector search misses: "papers that cite papers that cite X."

5. GraphRAG

Refer: Edge et al. - From Local to Global: A GraphRAG Approach (Microsoft, 2024)

GraphRAG pipeline:

Extract entities and relations from documents with LLM
Build knowledge graph
Community detection on graph (Leiden algorithm)
Summarize each community with LLM → hierarchical summaries
Query: retrieve relevant communities + entities, LLM answers with global context

Why it works: Plain RAG fails on "what are the main themes across this entire corpus?" - too many chunks. GraphRAG pre-clusters the corpus into a graph and summarizes communities, giving the LLM a map of the whole dataset.

6. LLM Prompting vs Learned GNN

Approach	How	Pros	Cons
Text-only prompt	Describe edges in natural language to LLM	No training, flexible	Context limit, hallucination, slow, expensive at scale
GNN	Learned structure encoding	Fast inference, scales to millions of nodes	Needs labels/data, no language reasoning
Hybrid	GNN retrieves + LLM reasons	Best of both in production	More engineering

When prompting is enough: one-off analysis on small graphs (< 100 nodes), exploratory QA, prototyping.

When you need GNN: million-node graphs, real-time recommendation, repeated queries, training data available.

7. Agents + Knowledge Graphs

LLM agents use tools (search, SQL, code). A knowledge graph becomes a structured tool:

Agent plans query → GNN/KG embedding finds entry points → traverses relations → LLM synthesizes answer
Reduces hallucination - facts grounded in KG triples
GNN can rank which entities to explore next (learned policy vs greedy traversal)

Related: Graph-ToolFormer, Think-on-Graph, KG-Agent lines of work (2024–2025).

8. Multimodal: Molecules + Language

Molecules are graphs (atoms = nodes, bonds = edges). LLMs understand SMILES strings and natural language descriptions.

GNN encodes 3D/2D structure and bond symmetry equivariance (EGNN, SchNet)
LLM encodes text descriptions and property questions
Joint models (MolT5, Text2Mol, MoMu): align GNN graph embedding space with LLM text embedding space

For drug discovery: GNN still wins on property prediction from structure alone. LLM wins on "explain why this molecule might bind to this protein in plain English."

9. Who Wins: LLM, GNN, or Both?

Task	Best approach
Cora node classification (bag-of-words)	GNN or MLP - LLM overkill
OGBN-Arxiv (paper titles + citations)	LLM embed + GNN
Molecule property prediction	Equivariant GNN (LLM optional for text)
Enterprise doc QA over corpus	GraphRAG (LLM + graph)
Recommendation (user-item graph)	GNN / classical CF - LLM for cold-start text only
KG question answering	GNN retriever + LLM reader
"Summarize themes in 10K papers"	GraphRAG - LLM alone struggles

10. Future: Graph Foundation Models

Like LLMs pretrain on text, graph foundation models pretrain on many graphs then finetune:

GraphMAE, GraphGPT, OneForAll, GFM: self-supervised pretraining on diverse graphs
LLM as graph tokenizer: describe nodes/edges in text, single LM handles all modalities
Unified orchestration: LLM plans, GNN executes structured computation, LLM explains

The trend is not "GNN replaces LLM" or vice versa - it is specialized modules in one system: LLM for language and reasoning, GNN for relational structure.