36 lessons in Machine Learning: Build Your Own Language Model

Predict the next characterCounting pairsFrom counts to written namesOne number for a language modelPairs it never sawThe same model as a networkTraining the bigram networkIt rediscovered countingMore context, more empty cellsA window of contextEmbeddingsThe neural language modelThe first loss is a checkUnits stuck at the edgeSlopes into the embeddingsTraining the language modelWhat it writes nowThe scoreboard so farAveraging what came beforeWeights from scoresQueries and keysValues: one head of attentionDivide by the square rootSeveral heads at onceWhere each letter sitsThe engine, on whole arraysA straight path through the blocksKeeping the numbers in rangeAdam: a step for every weightYour transformer, trainedTemperatureTrain it on your own textTokens: merging common pairsWhat your model is, and is notTrain it properly, on the graphics chipHow the search box guesses what you type

Built to be understood — and remembered.

Every idea is taught with motivation and a worked example before the drills, and an FSRS spaced-repetition engine schedules each review for the day its model predicts you would forget it. A short placement check finds what you already know, so you start Machine Learning: Build Your Own Language Model exactly where it's useful.

Start Machine Learning: Build Your Own Language Model →

Related Ml subjects

Machine Learning: The Whole Game Machine Learning: Learning from Data, in Code Machine Learning: Neural Networks from Scratch Machine Learning: Deep Learning, Seeing and Remembering Machine Learning: Doing It Honestly