1. Loss Functions
| Task | Loss | Notes |
|---|---|---|
| Node classification | Cross-entropy | Only on labeled train nodes |
| Multi-label | BCE with logits | Nodes can have multiple labels |
| Link prediction | BCE / margin ranking | Need negative samples |
| Graph regression | MSE / MAE | On graph-level readout |
| Self-supervised | Contrastive (InfoNCE) | GraphMAE, DGI - pretrain then finetune |
2. Full-Batch vs Mini-Batch
Full-batch: entire graph in memory, one gradient step per epoch. Works for Cora (2.7K nodes). Fails for OGBN-Products (2.4M nodes).
Mini-batch: sample subset of nodes/edges per step. Required for large graphs.
3. Neighbor & Subgraph Sampling
- Neighbor sampling (GraphSAGE): sample fixed fanout per layer, e.g. [25, 10] - 25 neighbors layer 1, 10 layer 2
- GraphSAINT: sample random subgraphs (random walk, forest fire)
- Cluster-GCN: partition graph into clusters, batch clusters
- FastGCN: importance sampling of nodes
from torch_geometric.loader import NeighborLoader
loader = NeighborLoader(
data,
num_neighbors=[25, 10],
batch_size=1024,
input_nodes=data.train_mask,
)4. Training Link Prediction
- Split edges into train/val/test
- Train GNN on train edges only
- Positive pairs: held-out edges
- Negative pairs: random non-edges (sample 1:1 or 1:k ratio)
- Score: dot product or MLP on $(\mathbf{h}_i, \mathbf{h}_j)$
Never include validation/test edges in the message-passing graph during training.
5. Optimization Tricks
- Adam lr 0.01 (GCN) to 0.001 (GAT) - always tune
- Weight decay 5e-4 standard for citation networks
- Dropout 0.5 on embeddings (GCN paper default)
- Early stopping on validation metric
- Gradient clipping for deep/unstable models
- Mixed precision (AMP) for large graph batches
6. Debugging Checklist
- Does a label propagation or MLP baseline work? If MLP beats GNN, structure may not help.
- Check train accuracy - can't fit train = bug or too few layers
- Verify masks and edge splits for leakage
- Plot loss curve - diverging = lr too high
- Check isolated nodes - get zero or self-loop-only messages
- Compare 2 vs 4 layers - val drops at 4 = oversmoothing
7. Frameworks
| Framework | Strength |
|---|---|
| PyTorch Geometric | Research default, huge model zoo |
| DGL | Large-scale, heterogeneous, distributed |
| PyG + OGB | Standardized benchmarks |
| DeepSNAP | Bridge NetworkX and PyG |