Speaker: Leah Berg
For more, see the table of contents.
Notes
- Night job – https://www.datasciencerebalanced.com
Data Leakage
- Also known as leakage or target leakage
- Different meaning for information security (data leaking to outside organization)
- Can be difficult to spot
- Training data includes info about test.
- Model trained on info not available in production
How models learn
- Split data into training data and test data.
- Test data – data model has never seen before and makes sure model gets is right
- Can also have an optional validation set
- Randomly pick whether data points are training or test data. – Called random train/test split
- More training data than test data
Don’t include data from the future
- Using a random split of time series data doesn’t work because model has learned about future data.
- Better to use a sliding window. Use first few months to predict next month. Then add that next value and predict one after. And keep going. Adding up error gives you accuracy of model.
- This works because model only knows about data before one asked to predict.
- Create timeline for when events happen. That way you make sure you aren’t using data from before the prediction
- Don’t always know where/when data was created. Important to understand business process
Don’t randomly split groups
- Have some data from the group you are then predicting
- Problem when new student shows up so prediction will be bad
- scikit-learn has GroupShuffleSplit() to get full group in same set – testing or training
Don’t forget your data is a snapshot
- In school, have pristine data set.
- In real world, data is always changing.
- Could tell model about data that occurred after prediction. Again think about data on timeline
Don’t randomly split data when retraining
- Want to use same training/test data on production and challenger models to see which better.
- One has already seen data points during training that you are testing so you don’t know if it is better.
- Challenger model can get more data that wasn’t available originally. Ok to split new data into test/train as long as original data part is split same way.
Split data immediately
- Risky to rescale before split because data isn’t represented same way. Min/max can vary if split after
- Run normalization on different sets of data
- Before split, do analysis with business, exploratory data analysis. Split data before start modeling
Use Cross Validation
- KFold Validation – split training data into K parts
- ex: 3 fold validation – two parts stay as training and one is validation. The test data remains as test data and is kept separate for final evaluation.
- The validation set is for an initial test.
- Gives more options to train model
Be Skeptical of High Performance
- If validation much higher than train/test, suspicious.
- If train/test/validation sets are all high/the same, suspicious.
Use scikit-learn pipeline
- Helps avoid leaking test data into training data
Check for features correlated with target
- If another attribute has a high match with what looking for, make sure not mixing up correlation/causation.
- Also, avoid timeline errors for reverse causation. Ex: the thing you are looking for causes, something else
My take
Great talk. Almost all of this was new to me. It was understandable and I learned a lot.