Z2H Video 1, round 2, lots of clarity [Post #12, Day 27]

I'm about two thirds of my way through my second pass of Video 1 and I am understanding a lot of the details. After my first pass I wanted to go back through the manual backpropagation segment especially as I didn't fully understand it, now I feel that I do. I was able to work through the numbers by hand, by drawing the computation graphs out on gridded graph paper and working out the gradients by hand, it was fun!

Here's the progress of the video as I have it organized in my mind:

1. Intro to micrograd
2. Understanding derivatives
3. Object-oriented programming (OOP) basics (e.g. creating a class, objects, and Python dunder methods)
4. Forward pass through a computation graph visualized using the graphviz library
5. Backpropagation by hand! Including the key node (Andrej said understand this and you've got the key to backpropagation) and chain rule
6. tanh, backpropagating through tanh
7. Codifying backpropagation piece by piece (so we don't have to work through the computation graph by hand)
8. Topological sort
9. Breaking up tanh (composite operation) into its parts (atomic operations)
10. Introduction to PyTorch, PyTorch autograd gradients agree with our micrograd gradients
11. Structure of a neural network, building a neuron that we can forward pass through

Next up is building a layer of neurons.

1. Building a layer, then building an MLP (3 input neurons, 2 hidden layers of 4 input neurons each, 1 output neuron) and making a simple forward pass and calling draw_dot()
2. Loss, Mean Squared Error Loss
3. Gradient descent to optimize neural net: forward pass and calculate loss, backward pass, nudge (i.e. update parameters), forward pass and calculate loss, backward pass, nudge (i.e. update parameters)...
4. Wrap gradient descent into a nice loop, correct "not gradient flushing" bug
5. Summarize, "we have a blob of neural stuff (simulated neural tissue), and we can make it do arbitrary things, and that's the power of neural networks"
6. Walk through micrograd to show it's similar to what we accomplished, more complex classification demo example
7. Finding tanh implementation in PyTorch, adding new LEGO blocks (i.e. functions like Legendre polynomial to PyTorch)
8. Outtakes, "micrograbbed"

Done with Z2H Video 1 second pass at 22:12 on 02/02/25! I see there are exercises I can try using Google Colab, plan to have a go at those tomorrow.