Interview: Deep GNNs & Limitations

Stacking many GNN layers makes all node embeddings converge to similar values. Mathematically, repeated low-pass filtering minimizes Dirichlet energy $\mathbf{h}^T\mathbf{L}\mathbf{h}$. Nodes become indistinguishable - classification fails.

Information from distant nodes must pass through low-degree bottleneck edges, compressed into fixed-size vectors. Exponential information loss with distance. Fix: graph rewiring, virtual nodes, graph transformers, subgraph GNNs.

Concatenate or aggregate embeddings from all layers $\mathbf{h}_i^{(1)}, \ldots, \mathbf{h}_i^{(L)}$ so the final representation keeps both local (shallow) and global (deep) information instead of only the last layer.

Residual connections, Jumping Knowledge, PairNorm/GraphNorm, DropEdge regularization, and limiting depth to 2-4 layers on most graphs. Graph transformers for global context without deep MPNN.

Start with 2. Try 3-4 for molecules needing wider receptive fields. If validation drops with more layers, you hit oversmoothing. Match depth to the diameter of relevant structure, not "deeper is always better."

H2GCN (separate ego and neighbor features), GPR-GNN (learnable signed hop weights), FAGCN (signed attention). Standard GCN assumes homophily and fails when neighbors have different labels.

NGNN, ESAN - operate on subgraphs around each node to exceed 1-WL expressiveness. Can distinguish graphs that standard MPNNs cannot. More compute but stronger discrimination.