tp4-feature-learning

Run python train.py --dataset omniglot --num-ways 5 --num-shots 1 --inf from the TP4MAML/meta directory after installing both requirements.txt files. The --inf flag switches the metalearner from standard MAML to InfMAML using GP or NTK limits.

The GP limit treats the network as a Gaussian Process with kernel regression, the NTK limit captures gradient-based training dynamics via the neural tangent kernel, and muP enables feature learning. This skill provides InfGP1LP and InfNTK1LP models for the first two regimes.

Build the C binaries with make in the Word2Vec directory, prepare data with create-text8-data.sh, then run train-text8-inf.sh. The infinite-width scripts set hyperparameters appropriate for infinite-width training, and evaluate.sh scores the embeddings.

InfSGD subclasses torch.optim.Optimizer and accepts standard parameter groups, so it works with regular nn.Module models. It applies learning rate scaling conventions appropriate for infinite-width NTK or muP parameterizations, with InfMultiStepLR as its compatible scheduler.

Infinite-width models like InfGP1LP and InfNTK1LP perform prediction through kernel computations rather than learned weight matrices. They accumulate kernel values between training and test points using functions like F00ReLU, the arc-cosine ReLU kernel.