Speaker: Barry Burd
See the table of contents
Opener
- Imagine have to find lowest point on a line but can only see a few steps ahead or behind you. One dimensional problem.
- On a mountain, finding lowest point but in two dimensions.
- LLM does that but with many more dimensions – ex: billions
- Uses a lot of tricks, not just extending the one dimensional problem
Problem
- Training GPT-3 required 10K NVIDIA GPUs
- PyTorch is highly optimized 0 biuldt in libraries, deep integration with GPU hardware (NVIDIA CUDA)
- Apple has GPU stack
- Want to do with Java
Solution
- HAT (Heterogeneous Acceleration Toolkit)
- Work in progress
- Part of project Babylon
- Code models/reflection
- Barry’s goal: algorithms to run on these
Deeplearning4j (ND4j)
- CUDA support
- No MDS support
- Arrays stored off-heap (outside JVM)
- Several arrays can point to several subarrays of same data.
What LLM does
- After analyzing a possible incomplete string, the LLM decides to add the string’s next token.
- Too many words to predict word
- Characters too granular because missing meaning.
Tokens
- I’ve groked Heinelin’s work as tokens:
*I
've
gro
k
ked
Hein
lein
's
work
s- Token id is a number that goes with the tokens
- Token is sequence of characters that occur together frequently enough using byte-pair encoding.
- Supposing string is “a b r a c a d a b r a” the tokens are: a, b, c, d,r . Then observer token pair ab appears frequently so ab is also a token. Now have “ab r a c a d ab r a” with “ab” added to vocabulary. Then repeat and see ab and r appear next to each other so “abr a c a d abr a” with “abr” added to token list. Then “abra”
- Python library tiktoken
- Java library JTokkkit
Brain
- 86 billion neurons in human brain
- Dendrite – input from another cell
- Soma – cell body
- Axon – output to other cells
- Oversimplified: Imagine cell body multiplies each input by a certain weight (different per cell) and adds them. That’s like multiplying a vector and a matrix
Math terms
- Vector – array/list of numbers. Can represent a point in n-dimensional space. Usually visualized as an arrow from origin to that point. ex: 1526 dimensions is 1526 number in vector
- Matrix – rectangular array of vectors. Turns one vector into another
- Tensor – stack of matrixes. Array of array of matrixes. Not important here.
- Dot product of two vectors – multiply elements in same spot in each vector and add them up.
- Matrix multiplication – had nice animation
ND4j
- N dimensions for J.
- Knows how to do vector/matrix math
Embedding
- Each token gets assigned to an arbitrary vector at first. This is the token embedding
- Picture adjusting bunny years antenna and how it only works when you are touching it. Walking away breaks the reception.
- Each number in the initial arbitrary vector is like a dial that needs to be tuned
Gradient Descent
- Normally millions of minimum points and easy to get stuck in point that is a local minimum. Looks like at lowest point in all directions, but there is a lower one elsewhere. LLM training is meant to avoid that pitfall
- Eclipse DeepLearning4J – can configure neural network and make a model
Vector meaning
- Similar vectors are similar semantics
- Applying related vectors to others should have consistent semantic meaning
- RGB for colors are vectors with three points representing the colors.
- Dot product of Cyan (0/255/255) and Red (255/0/0) because nothing in common
- Add positional embedding to token embedding so know where in sentence. (Add to each element with different scale so know which part goes with which). These combined are in the input embedding
Attention
- Attention is all you need https://arxiv.org/abs/1706.03762
- Attention examples: grammatical structure, meaning, word order
- Long range dependency – like a pronoun that refers to something many words away
- Attention helps focus on important parts, ex: “The cat sat on the mat”. What’s on the me (the mat) at that point.. Has to know.
- Apply key matrix to cat and a query matrix to mat. Key matrix offers info. Query matrix is what you want to know.
- Start with random values. Then multiply by tokens. Then take dot product and get a number to see what predicts about next word. See how far prediction off from what it actually is. Apply to surrounding neighbors and go in direction that makes error less. Repeat a very large number of times
- A lot of this can be done in parallel
Feed forward
- Linear – straight – ex: 2x + 3y (2 and 3 are the knobs to tune)
- Non-linear – wavy – more dimensions
- Languages isn’t linear by nature
- “great, terrific meal” – can add up
- “not good” – not flips meaning of sentence. So can’t just predict the next word.
Universal approximation theorem
- Imagine a wavy line as a series of bumps from the ground to that line
- The more granular the bump, the more accurate the result.
- Each bump can be represented by a linear formula
- Apply GeLU (non linear) function to make the bumps
Experiment
- Translated Karpathy’s 253 line LLM into Java with a list of baby names
- Took about a minute to train.
- Generated new names. A couple legit but most random looking
My take
This was great. Tokens and critical concepts get used without defining them and we take for granted. So being able to think about it was informative and helpful. I like that Barry showed math but said ok to not to understand. [it was understandable]