# Jeremy Jordan > Thoughts, ideas, and new things I've learned. Public Ghost content for AI and LLM tooling. This file includes a bounded export of public pages first, then recent public posts. Append `.md` to any post or page URL to get the content in Markdown (for example, `/example-post.md`). ## Pages ### About me. URL: https://www.jeremyjordan.me/about/ Last updated: 2026-03-26T20:59:39.000Z Hi👋 I'm Jeremy, welcome to my blog. I'm a machine learning engineer currently working on information retrieval and agent memory at Nvidia. I love the process of learning and building a better understanding of our world. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2020/01/IMG_0969.jpg) I started this blog for two purposes: to write about what I'm learning in my own words, and to share my perspectives and reflections with others and my future self. The former reason serves mainly to reinforce what I'm learning (you don't really know something unless you can teach it!), while the latter is simply a way for me to share what reflections I've found helpful for me and store a snapshot of my views over the course of my life. Many of the blog posts are a living document, so I'll come back and edit things, add new info, switch out the graphics, etc. These are also sometimes pretty informal posts, more of a quick brain dump than a work of literary art. I've chosen to write in public so that a) the content may help someone else and b) someone might disagree and we can have a discussion. I'm a big believer in open, frank critique of ideas and I believe the best way to grow an idea is to attack it. Don't take what I write on here for granted - ask yourself if what I'm talking about makes sense to you. If it doesn't, speak up! Let's have a discussion. ##### A little bit about myself I love sailing. I have a small Sunfish that I'll take out on the lakes around Raleigh. I also have a stand-up paddleboard that I'll take out when the winds aren't strong enough for sailing. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/09/GOPR8041.JPG) ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/09/GOPR8057.JPG) Spending time out on the water is a great way to temporarily escape from the crazy world of tech. I like to stay active and take care of my body. In order to establish a baseline level of fitness, I started doing [one sprint triathlon every week](https://www.jeremyjordan.me/triathlons/). On top of that, I enjoy playing soccer and tennis with friends. I'm a huge adventurer, and I love to travel. I've taken three trips to Europe visiting Spain, Italy, France, Switzerland, Denmark, Norway, and the Netherlands, and I have an extensive list of places/cultures I'd like to see/experience. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/09/IMG_6475.jpg) Hiking in Norway My interests span from engineering to design to neuroscience and many fields in between - my curiosity fuels an insatiable love for learning new things. I'm also fascinated by entrepreneurship and wish to make my impact through creating businesses that add value to our society. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/12/me.png) Go Wolfpack! (I graduated from NC State University) ### My favorite quotes. URL: https://www.jeremyjordan.me/quotes/ Last updated: 2019-08-24T22:32:35.000Z Here's a collection of some of my favorite quotes: #### Life "Do not fear mistakes. There are none." ― Miles Davis “Every man I meet is my superior in some way, and in that I learn from him.” ― Ralph Waldo Emerson “A master in the art of living draws no sharp distinction between his work and his play; his labor and his leisure; his mind and his body; his education and his recreation. He hardly knows which is which. He simply pursues his vision of excellence through whatever he is doing, and leaves others to determine whether he is working or playing. To himself, he always appears to be doing both.” ― Yvon Chouinard “What man actually needs is not a tensionless state but rather the striving and struggling for some goal worthy of him. What he needs is not the discharge of tension at any cost, but the call of a potential meaning waiting to be fulfilled by him.” ― Victor Frankl "Learn to seek purpose more than you do pleasure. Pleasure does not change your life, it makes discomfort bearable. Purpose makes discomfort worth it. Don’t let comfort be your first virtue." ― Brianna Wiest “You don't have to see the whole staircase. Just take the first step.” ― Martin Luther King “What lies behind us and what lies before us are tiny matters compared to what lies within us.” ― Ralph Waldo Emerson "To love, ultimately, is to have the willingness to interpret someone’s (on surface) not very appealing behavior, in order to find more benevolent reasons why it may be unfolding. In other words, to love someone is to apply charity and generosity of interpretation." ― Alain de Botton > **The Man in the Arena** > > **It is not the critic who counts**; not the man who points out how the strong man stumbles, or where the doer of deeds could have done them better. **The credit belongs to the man who is actually in the arena**, whose face is marred by dust and sweat and blood; who strives valiantly; who errs, who comes short again and again, **because there is no effort without error and shortcoming**; but who does actually strive to do the deeds; who knows great enthusiasms, the great devotions; who spends himself in a worthy cause; who at the best knows in the end the triumph of high achievement, and who at the worst, **if he fails, at least fails while daring greatly**, so that his place shall never be with those cold and timid souls who neither know victory nor defeat. – Theodore Roosevelt “I am always doing that which I cannot do, in order that I may learn how to do it.” – Pablo Picasso "You can’t stop the waves, but you can learn to surf." – Jon Kabat-Zinn "Each man is questioned by life; and he can only answer to life by answering for his own life; to life he can only respond by being responsible." ― Victor Frankl "Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them." ― Frank Herbert, Dune “I am strong because I've been weak. I am fearless because I've been afraid. I am wise because I've been foolish.” ― Unknown “The purpose of life is not to be happy. It is to be useful, to be honorable, to be compassionate, to have it make some difference that you have lived and lived well.” ― Ralph Waldo Emerson > A grandfather is talking with his grandson and he says *there are two wolves inside of us* which are always at war with each other. One of them is a **good wolf** which represents things like **kindness, bravery, and love**. The other is a **bad wolf**, which represents things like **greed, hatred, and fear**. The grandson stops and thinks about it for a second then he looks up at his grandfather and says, “Grandfather, which one wins?” > > The grandfather quietly replies, "the one you feed." “Holding on to anger is like grasping a hot coal with the intent of throwing it at someone else; you are the one who gets burned.” ― Buddha “I cannot remember the books I've read any more than the meals I have eaten; even so, they have made me.” ― Ralph Waldo Emerson “A ship in harbour is safe, but that is not what ships are built for.” – William G.T. Shedd “The man who asks a question is a fool for a minute, the man who does not ask is a fool for life.” – Confucius "There are many things of which a wise man might wish to be ignorant" – Ralph Waldo Emerson > **Passage from Gilles Deleuze in Negotiations** > > …we’re riddled with pointless talk, insane quantities of words and images. Stupidity’s never blind or mute. So it’s not a problem of getting people to express themselves but of providing little gaps of solitude and silence in which they might eventually find something to say. Repressive forces don’t stop people expressing themselves but rather force them to express themselves; **what a relief to have nothing to say, the right to say nothing, because only then is there a chance of framing the rare, and ever rarer, thing that might be worth saying.** “Life is a series of natural and spontaneous changes. Don’t resist them; that only creates sorrow. Let reality be reality. Let things flow naturally forward in whatever way they like.” ― Lao Tzu "There is only one thing that makes a dream impossible to achieve: the fear of failure." – Paulo Coelho "And now you say I changed... Like I'm doing all of this to stay the same?" – Anderson Paak > **That Man is a Success by Ralph Waldo Emerson** > > "To laugh often and much; to win the respect of intelligent people and the affection of children; > to earn the appreciation of honest critics and endure the betrayal of false friends; to appreciate beauty; > to find the best in others; > to leave the world a bit better whether by a healthy child, a garden patch or a redeemed social condition; > to know even one life has breathed easier because you lived. > This is to have succeeded." “As with our colleges, so with a hundred ‘modern improvements;’ there is an illusion about them; there is not always a positive advance. The devil goes on exacting compound interest to the last for his early share and numerous succeeding investments in them. Our inventions are wont to be pretty toys, which distract our attention from serious things. They are but improved means to an unimproved end, an end which it was already but too easy to arrive at...” ― Henry David Thoreau “God grant me the serenity to accept the things I cannot change, the courage to change the things I can, and the wisdom to know the difference.” "They tried to bury us. They didn't know we were seeds." ― Mexican Proverb > Buddha was well known for his ability to respond to evil with good. There was a man who knew about his reputation and he traveled miles and miles and miles to test Buddha. When he arrived and stood before Buddha, he verbally abused him constantly; he insulted him; he challenged him; he did everything he could to offend Buddha. Buddha was unmoved, he simply turned to the man and said, “May I ask you a question?” The man responded with, “Well, what?” Buddha said, “If someone offers you a gift and you decline to accept it, to whom then does it belong?” The man said, “Then it belongs to the person who offered it.” Buddha smiled, “That is correct. So if I decline to accept your abuse, does it not then still belong to you?” The man was speechless and walked away. “You can be the ripest, juiciest peach in the world, and there’s still going to be somebody who hates peaches.” "The future is here, it's just not evenly distributed." - William Gibson “I saw my life branching out before me like the green fig tree in the story. From the tip of every branch, like a fat purple fig, a wonderful future beckoned and winked. One fig was a husband and a happy home and children, and another fig was a famous poet and another fig was a brilliant professor, and another fig was Ee Gee, the amazing editor, and another fig was Europe and Africa and South America, and another fig was Constantin and Socrates and Attila and a pack of other lovers with queer names and offbeat professions, and another fig was an Olympic lady crew champion, and beyond and above these figs were many more figs I couldn't quite make out. I saw myself sitting in the crotch of this fig tree, starving to death, just because I couldn't make up my mind which of the figs I would choose. I wanted each and every one of them, but choosing one meant losing all the rest, and, as I sat there, unable to decide, the figs began to wrinkle and go black, and, one by one, they plopped to the ground at my feet.” ― Sylvia Plath #### Ideas “The mind, once stretched by a new idea, never returns to its original dimensions.” ― Ralph Waldo Emerson "Enthusiastic skepticism is not the enemy of boundless optimism. " ― Astro Teller “Vulnerability is the birthplace of innovation, creativity, and change.” ― Brene Brown “I never came upon any of my discoveries through the process of rational thinking.” — Albert Einstein "The only true wisdom is in knowing you know nothing." — Socrates #### Leadership “If you want to build a ship, don’t drum up the men to gather wood, divide the work, and give orders. Instead, teach them to yearn for the vast and endless sea.” "Doing the right thing today will pay dividends tomorrow." -Me "A lot of people have gone further than they thought they could because someone else thought they could." – Zig Ziglar "Great leaders are almost always great simplifiers, who can cut through argument, debate and doubt to offer a solution everybody can understand." – Colin Powel “When they say you can't, they show you their limits, not yours.” – Kevin Keenoo “Honesty is a very expensive gift. Don’t expect it from cheap people.” ― Warren Buffett #### Stoicism “Until we have begun to go without them, we fail to realize how unnecessary many things are. We've been using them not because we needed them but because we had them.” ― Seneca “Luck is what happens when preparation meets opportunity.” ― Seneca “If a man knows not to which port he sails, no wind is favorable.” ― Seneca “He suffers more than necessary, who suffers before it is necessary.” ― Seneca “As long as you live, keep learning how to live.” ― Seneca “The impediment to action advances action. What stands in the way becomes the way.” ― Marcus Aurelius "Everything we hear is an opinion, not a fact. Everything we see is a perspective, not the truth." ― Marcus Aurelius #### Business > For some, I realize, business is the all-out pursuit of profits, period, full stop, but for us business was no more about making money than being human is about making blood. Yes, the human body needs blood. It needs to manufacture red and white cells and platelets and redistribute them evenly, smoothly, to all the right places on time, or else. But that day-to-day business of the human body isn’t our mission as human beings. It’s a basic process that enables our higher aims, and life always strives to transcend the basic processes of living — and at some point in the late 1970s, I did, too. I redefined winning, expanded it beyond my original definition of not losing, of merely staying alive. That was no longer enough to sustain me, or my company. We wanted, as all great businesses do, to create, to contribute, and we dared to say so aloud. When you make something, when you improve something, when you deliver something, when you add some new thing or service to the lives of strangers, making them happier, or healthier, or safer, or better, and when you do it all crisply and efficiently, smartly, the way everything should be done but so seldom is- you’re participating more fully in the whole grand human drama. More than simply alive, you’re helping others to live more fully, and if that’s business, all right, call me a businessman. - Phil Knight ### Business Ideas URL: https://www.jeremyjordan.me/business-ideas/ Last updated: 2016-10-22T22:23:03.000Z _No content available._ ### My favorite talks. URL: https://www.jeremyjordan.me/my-favorite-talks/ Last updated: 2021-03-21T23:44:47.000Z Go Against the Grain with David Heinemeier Hansson The Education of a Reluctant Businessman with Yvon Chouinard (CEO of Patagonia) How To Speak by Patrick Winston Andrew Ng: Artificial Intelligence is the New Electricity Scott Young on the Best Way to Learn Anything Dan Ariely: On Dating & Relationships | Talks at Google Simple Made Easy: Rich Hickey [Link](https://www.infoq.com/presentations/Simple-Made-Easy?ref=jeremyjordan.me) Why You Will Marry the Wrong Person - Alain de Botton The power of vulnerability | Brené Brown The unexpected benefit of celebrating failure | Astro Teller On being wrong | Kathryn Schulz Steven Kotler: "The Rise of Superman: Decoding the Science of Ultimate Human Performance" The Art of Stress-Free Productivity: David Allen at TEDxClaremontColleges Dave Evans: "Designing Your Life" | Talks at Google Robert Greene: "Mastery" | Talks at Google Beau Lotto: "Deviate" | Talks at Google High Line Slack Line with Good Line Brené Brown: Why Your Critics Aren't The Ones Who Count John Collison: Putting Startup Success in Perspective Why is 'x' the unknown? | Terry Moore Are we in control of our decisions? | Dan Ariely Simon Sinek: Understanding The Game We're Playing Simon Sinek: Why Leaders Eat Last How great leaders inspire action | Simon Sinek Simon Sinek: If You Don't Understand People, You Don't Understand Business Olivia Fox Cabane: "Charisma, Leadership and the Imposter Syndrome" | Talks at Google Why Trust Is Worth It Hamming, "You and Your Research" ### Data Science Topics URL: https://www.jeremyjordan.me/data-science/ Last updated: 2023-05-07T14:05:49.000Z This page contains most of the topics I've covered in a self-set curriculum as I study the field of data science (with a strong focus on machine learning). Bullets without a link are topics that I plan to get to, but will not post an article on in the immediate future. Links labeled "coming soon" are posts currently in progress. [Courses I've taken](https://www.jeremyjordan.me/data-science-courses/) ## Machine Learning [Machine Learning Overview](https://www.jeremyjordan.me/machine-learning-overview/) #### The General ML Framework - [Organizing machine learning projects: project management guidelines](https://www.jeremyjordan.me/ml-projects-guide/) - [Building machine learning products: a problem well-defined is a problem half-solved.](https://www.jeremyjordan.me/ml-requirements/) - [Preparing data for a machine learning model](https://www.jeremyjordan.me/preparing-data-for-a-machine-learning-model/) - [Feature selection](https://www.jeremyjordan.me/feature-selection/) - [Evaluating a machine learning model](https://www.jeremyjordan.me/evaluating-a-machine-learning-model/) - [Hyperparameter tuning](https://www.jeremyjordan.me/hyperparameter-tuning/) - [Learning from imbalanced data](https://www.jeremyjordan.me/imbalanced-data/) - [Effective testing for machine learning systems](https://www.jeremyjordan.me/testing-ml/) - [A simple solution for monitoring ML systems](https://www.jeremyjordan.me/ml-monitoring/) - Building machine learning pipelines #### Machine Learning Models ###### Classification Classification algorithms are used when you have a dataset of observations where we'd like to use the features associated with an observation to predict its class. Example: Predict the type of flower when provided information on sepal length, sepal width, color, petal width, and petal length. - [Naive Bayes](https://www.jeremyjordan.me/naive-bayes-classification/) - [Logistic Regression](https://www.jeremyjordan.me/logistic-regression/) - [Decision Trees](https://www.jeremyjordan.me/decision-trees/) - [K-Nearest Neighbors](https://www.jeremyjordan.me/k-nearest-neighbors/) - [Support Vector Machines](https://www.jeremyjordan.me/support-vector-machines/) - [Random Forests](https://www.jeremyjordan.me/random-forests/) - [Boosted Trees](https://www.jeremyjordan.me/boosted-trees/) ###### Regression Regression algorithms are used when you have a dataset of observations where you'd like to use the features to predict a continuous output. Example: Predict the price of a house using the following features: sq ft, number of rooms, zip code, age of house, school district. - [Linear Regression](https://www.jeremyjordan.me/linear-regression/) - [Polynomial Regression](https://www.jeremyjordan.me/polynomial-regression/) - [Decision Trees](https://www.jeremyjordan.me/decision-trees/) - [K-Nearest Neighbors](https://www.jeremyjordan.me/k-nearest-neighbors/) - [Random Forests](https://www.jeremyjordan.me/random-forests/) - [Boosted Trees](https://www.jeremyjordan.me/boosted-trees/) - Gaussian Process Regression ###### Clustering Clustering is a popular technique to find groups or segments in your data that are similar. This is an unsupervised learning algorithm in the sense that you don't train the algorithm and give it examples for what you'd like it to do, you just let the clustering algorithm explore the data and provide you with new insights. - [K-means clustering](https://www.jeremyjordan.me/grouping-data-points-with-k-means-clustering/) - [Soft clustering with Gaussian mixture models](https://www.jeremyjordan.me/gaussian-mixed-models/) - Density-based spatial clustering of applications with noise (DBSCAN) ###### Dimensionality Reduction When we're building machine learning models, sometimes we deal with datasets with well over 1,000 or even 10,000 dimensions. While this allows us to account for many features, these features are often redundant. Ideally, due to the curse of dimensionality, we'd like to limit our data to capture the true signal in the data and ignore the noise. Dimensionality reduction is one technique to reduce the dimension of our feature-space while maintaining the maximum amount of information. Dimensionality reduction is also very convenient for visualizing higher-dimensional data sets in two or three dimensions. [This paper](https://web.archive.org/web/20170216052444/https://e-reports-ext.llnl.gov/pdf/240921.pdf) provides a great overview of the different techniques available for dimensionality reduction. - [Principal Component Analysis](https://www.jeremyjordan.me/principal-components-analysis/) - [Autoencoders](https://www.jeremyjordan.me/autoencoders/) - [Variational autoencoders](https://www.jeremyjordan.me/variational-autoencoders/) - t-SNE - UMAP ###### Neural Networks Neural networks are one of the most popular approaches to machine learning today, achieving impressive performance on a large variety of tasks. Often referred to as the "universal function approximator", this approach is very flexible to learning a variety of tasks. - Foundation - [Introduction and network representation](https://www.jeremyjordan.me/intro-to-neural-networks/) - [Activation functions](https://www.jeremyjordan.me/neural-networks-activation-functions/) - Training - [Training (weight optimization) using backpropagation](https://www.jeremyjordan.me/neural-networks-training/) - [Gradient descent](https://www.jeremyjordan.me/gradient-descent/) - [Setting the learning rate of your neural network](https://www.jeremyjordan.me/nn-learning-rate/) - [Deep neural networks: preventing overfitting](https://www.jeremyjordan.me/deep-neural-networks-preventing-overfitting/) - [Batch normalization](https://www.jeremyjordan.me/batch-normalization/) - Convolutional neural networks - [Introduction to convolutional neural networks](https://www.jeremyjordan.me/convolutional-neural-networks/) - [Common architectures in convolutional neural networks](https://www.jeremyjordan.me/convnet-architectures/) - Image segmentation - [Semantic image segmentation](https://www.jeremyjordan.me/semantic-segmentation/) - Instance image segmentation - [Evaluating image segmentation models](https://www.jeremyjordan.me/evaluating-image-segmentation-models/) - Object detection - [One stage methods: YOLO and SSD](https://www.jeremyjordan.me/object-detection-one-stage/) - Two stage methods: Faster R-CNN - Evaluating object detection models - Facial recognition - Recurrent neural networks - [Introduction to recurrent neural networks](https://www.jeremyjordan.me/introduction-to-recurrent-neural-networks/) - Gated recurrent units: Introducing intentional memory - Long short term memory networks: Learning what to remember and what to forget - [Attention mechanisms](https://www.jeremyjordan.me/attention/) - [Transformer networks](https://www.jeremyjordan.me/transformer-architecture/) - Transfer learning - Image recognition - Natural language processing - One-shot learning - Siamese networks ###### Reinforcement Learning Reinforcement learning is an approach to machine learning where agents are rewarded to accomplish some task. "Good" behavior is reinforced via a reward, so this approach can more realistically be considered a method of reward maximization. [This book](https://s3-us-west-1.amazonaws.com/udacity-dlnfd/suttonbookdraft2018jan1.pdf?ref=jeremyjordan.me) is the canonical resource for learning RL. - [Overview of reinforcement learning](https://www.jeremyjordan.me/overview-of-reinforcement-learning/) - [Planning in a stochastic environment](https://www.jeremyjordan.me/markov-decision-process/) - [Learning in a stochastic environment](https://www.jeremyjordan.me/rl-learning-methods/) - [Implementations of Monte Carlo and temporal difference learning methods](https://www.jeremyjordan.me/rl-learning-implementations/) - [Generalizing value functions for large state-spaces](https://www.jeremyjordan.me/rl-generalizing-value-functions/) #### Machine Learning Applications - Building a recommendation system with collaborative filtering ###### Natural Language Processing - Preprocessing text data for NLP - [TF-IDF Vectorization](https://www.jeremyjordan.me/identifying-related-bodies-of-text-using-tf-idf-vectorization/) ## Data Visualization The following links are external links to useful resources. At this time, I haven't written any blog posts on data visualizations but wanted to save a few external posts for future reference. - [Effectively Using Matplotlib - Chris Moffitt](http://pbpython.com/effective-matplotlib.html?ref=jeremyjordan.me) - [Visualization with Matplotlib - Jake VanderPlas](https://jakevdp.github.io/PythonDataScienceHandbook/04.00-introduction-to-matplotlib.html?ref=jeremyjordan.me) - [Fundamentals of Data Visualization](https://serialmentor.com/dataviz/?ref=jeremyjordan.me) ## Data Acquisition and Wrangling - [Basics of SQL](https://www.jeremyjordan.me/sql/) ### Data science and machine learning courses. URL: https://www.jeremyjordan.me/data-science-courses/ Last updated: 2020-07-11T17:29:57.000Z Below I've curated a list of online courses I've used to learn more about data science and machine learning. ## Udacity - [Intro to Data Analysis](https://www.udacity.com/course/intro-to-data-analysis--ud170?ref=jeremyjordan.me) - [Machine Learning Engineer Nanodegree](https://www.udacity.com/course/machine-learning-engineer-nanodegree--nd009t?ref=jeremyjordan.me) ## Coursera - [Machine Learning Course by Stanford University](https://www.coursera.org/learn/machine-learning?ref=jeremyjordan.me) - [Deep Learning Specialization by deeplearning.ai](https://www.coursera.org/specializations/deep-learning?ref=jeremyjordan.me) - Neural Networks and Deep Learning - Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization - Structuring Machine Learning Projects - Convolutional Neural Networks - Sequence Models - [Machine Learning with TensorFlow on Google Cloud Platform Specialization](https://www.coursera.org/specializations/machine-learning-tensorflow-gcp?ref=jeremyjordan.me) - How Google does Machine Learning - Launching into Machine Learning - Intro to TensorFlow - Feature Engineering - Art and Science of Machine Learning - [Architecting with Google Kubernetes Engine](https://www.coursera.org/specializations/architecting-google-kubernetes-engine?ref=jeremyjordan.me) - Google Cloud Platform Fundamentals: Core Infrastructure - Architecting with Google Kubernetes Engine: Foundations - Architecting with Google Kubernetes Engine: Workloads - Architecting with Google Kubernetes Engine: Production ## Fast AI - [Practical Deep Learning For Coders](https://course.fast.ai/?ref=jeremyjordan.me) - [Cutting Edge Deep Learning for Coders](http://course18.fast.ai/part2.html?ref=jeremyjordan.me) ## University courses - Stanford CS229: Machine Learning - [Notes](http://cs229.stanford.edu/syllabus.html?ref=jeremyjordan.me) - [Lecture videos](https://www.youtube.com/watch?v=UzxYlbK2c7E&list=PLA89DCFA6ADACE599&ref=jeremyjordan.me) - Stanford CS231n: Convolutional Neural Networks for Visual Recognition - [Notes](http://cs231n.github.io/?ref=jeremyjordan.me) - [Lecture videos](https://www.youtube.com/playlist?list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv&ref=jeremyjordan.me) - MIT 6.034: Artificial Intelligence - [Lecture videos](https://www.youtube.com/watch?v=TjZBTDzGeGg&list=PLUl4u3cNGP63gFHB6xb-kVBiQHYe%5F4hSi&ref=jeremyjordan.me) - UCL Advanced Topics 2015 (COMPM050/COMPGI13): Reinforcement Learning - [Notes](http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html?ref=jeremyjordan.me) - [Lecture videos](https://www.youtube.com/watch?v=2pWv7GOvuf0&ref=jeremyjordan.me) ## Industry-led courses - Full Stack Deep Learning - [Website](https://course.fullstackdeeplearning.com/?ref=jeremyjordan.me) --- ## Other interesting courses Courses I've found useful but have not rigorously completed. [MIT 18.06: Linear Algebra](https://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-spring-2010/?ref=jeremyjordan.me) [Essence of linear algebra](https://www.youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE%5Fab&ref=jeremyjordan.me) [Richard Hamming: The Art of Doing Science and Engineering](https://www.youtube.com/watch?v=AD4b-52jtos&list=PL2FF649D0C4407B30&ref=jeremyjordan.me) [Harvard CS50](https://cs50.harvard.edu/?ref=jeremyjordan.me) ### Startup graveyard. URL: https://www.jeremyjordan.me/startup-graveyard/ Last updated: 2018-06-11T18:28:41.000Z I've worked on a lot of things. Many of them failed. Here's a list. - **BoostOurTeam** - A fundraising platform for high school sports teams. The neat idea here was that coaches could share key stats and highlights from each game and allow donors to pledge a certain amount of money according to a game statistic (ie. $1 per touchdown). Unfortunately, most teams fundraise early on in the season to handle upfront costs like equipment, so it wasn't always feasible for teams to get their funding over the course of the season. - **Aquanix** - Nature has all kinds of nifty tricks up her sleeve, and there's a whole field known as biomimicry founded on understanding these tricks and repurposing them for other problems. We were trying to mimic the [water collection properties of spider silk](https://www.nature.com/articles/nature08729?ref=jeremyjordan.me) to build a material capable of hyper-efficient dehumidification. Currently, HVAC systems bring in outdoor air and dehumidify it by overcooling it (which essentially squeezes water out of the air) and then heating it back up to a more comfortable indoor temperature. We imagined placing humidity "filters" in HVAC systems instead of wasting energy overcooling and reheating air. We were able to design materials fairly well to control water transport (aggregated micro water droplets into larger water droplets which could roll of the surface into a collection pipe), but nucleation (converting water from a gas in the air to a liquid on the surface of our screen) proved more challenging. - **Kast Clothing** - Clothes are designed to fit the average person, that's how you sell the highest volume of clothes. That kinda sucks though if you're not average and you're left with clothes that don't *quite* fit you just right. Our whole idea here was to allow customers to take a 3D scan of their body and algorithmically generate patterns for clothing which would fit their body perfectly. The 3D laser scanners we used were expensive and hard to scale, although with advances in computer vision this might become feasible with simple cameras (which are cheap!). I'd love to revisit this one day. - **Mobius Labs** - Financial exchanges use something called an orderbook to keep track of buyers and sellers, and these orderbooks are used to match interested parties and facilitate a trade between a buyer and seller. Sometimes these books are imbalanced where you have more interested seller than buyers, or vice versa. We worked on building machine learning models which could forecast short term price fluctuations based on this orderbook data, which would then be used for algorithmic trading. It turns out financial data can be really noisy. I wrote about this experience in more detail [here](https://www.jeremyjordan.me/mobius/). I'm sure there will be more to add in the future :) ### Saltzer and Schroeder's design principles URL: https://www.jeremyjordan.me/security-design/ Last updated: 2019-12-10T01:38:24.000Z 1. **Economy of mechanism**: Keep the design as simple and small as possible. 2. **Fail-safe defaults**: Base access decisions on permission rather than exclusion. 3. **Complete mediation**: Every access to every object must be checked for authority. 4. **Open design**: The design should not be secret. 5. **Separation of privilege**: Where feasible, a protection mechanism that requires two keys to unlock it is more robust and flexible than one that allows access to the presenter of only a single key. 6. **Least privilege**: Every program and every user of the system should operate using the least set of privileges necessary to complete the job. 7. **Least common mechanism**: Minimize the amount of mechanism common to more than one user and depended on by all users. 8. **Psychological acceptability**: It is essential that the human interface be designed for ease of use, so that users routinely and automatically apply the protection mechanisms correctly. 9. **Work factor**: Compare the cost of circumventing the mechanism with the resources of a potential attacker. 10. **Compromise recording**: It is sometimes suggested that mechanisms that reliably record that a compromise of information has occurred can be used in place of more elaborate mechanisms that completely prevent loss. Reproduced from [this paper](http://web.mit.edu/Saltzer/www/publications/protection/Basic.html?ref=jeremyjordan.me) for convenience. I first learned about these principles from Nicolas Papernot's talk on [security and privacy of machine learning](https://www.papernot.fr/marauder%5Fmap.pdf?ref=jeremyjordan.me). ### Talks and Presentations URL: https://www.jeremyjordan.me/talks/ Last updated: 2023-05-08T22:02:46.000Z **Powering large-scale similarity search with deep learning** *Dec 2018 – Deep Learning RTP Meetup* - [Slides](https://docs.google.com/presentation/d/1bcZwgm9Cd7llICVv-itXAT2O3KPyhErm/edit?usp=sharing&ouid=107936521982821287195&rtpof=true&sd=true&ref=jeremyjordan.me) - [Meetup](https://www.meetup.com/deep-learning-rtp/events/nvgwrpyxqbnc/?ref=jeremyjordan.me) **Paper discussion on "mixup: Beyond Empirical Risk Minimization"** *Feb 2019 – Deep Learning RTP Meetup* - [Slides](https://docs.google.com/presentation/d/1Wvncia-KDvDhcVrPn6Un6YILFz9Cw6H2/edit?usp=sharing&ouid=107936521982821287195&rtpof=true&sd=true&ref=jeremyjordan.me) - [Meetup](https://www.meetup.com/deep-learning-rtp/events/lzdvpqyzdbnb/?ref=jeremyjordan.me) **8 lessons learned building threat detection systems as an MLE** *Oct 2022 – MLOps Learners Community Meetup* - [Slides](https://docs.google.com/presentation/d/1UMF4mysIeEShsj3o0CYUbxbnhZKoiPFeoTjGYbOee%5FI/edit?ref=jeremyjordan.me#slide=id.g16286b535dd%5F0%5F0) - [Talk](https://www.youtube.com/watch?v=uk9ugHHrZps&ref=jeremyjordan.me) **Building Threat Detection Systems: An MLE's Perspective** *Nov 2022 – MLOps Podcast* - [Talk](https://www.youtube.com/watch?v=13nOmMJuiAo&ref=jeremyjordan.me) **ML doesn't always replace rules, sometimes they work together** *Dec 2022 – NormConf* - [Slides](https://docs.google.com/presentation/d/1xjMPmZqKmwNIYKELMjIsoQjYGx0vVR7VaNZ3sEqlAO4/edit?usp=sharing&ref=jeremyjordan.me) - [Talk](https://youtu.be/gXe9iXNTuDc?ref=jeremyjordan.me) ### Triathlons 🏊 🚴 🏃 URL: https://www.jeremyjordan.me/triathlons/ Last updated: 2026-07-31T02:26:59.000Z I've been completing one sprint triathlon (750m swim, 20km bike, 5km run) every week since May 2023\. I usually split the three activities into separate workouts; I'm mostly focused on ensuring that I hit my distance goals over the course of the week, and I'm not too strict on time splits for each activity. One nice thing about trying to "keep the streak alive" and do this *every* week, is that it's a great motivator to ensure that I'm not pushing myself to the point of injury. I view this weekly ritual as a nice solid baseline for my physical health. ## Common questions **Are you training for an event?** Nope! My main objective is to build a *consistent* habit of exercise so I can live a healthy life and age gracefully. When I'm old and gray I'd like to still have an active lifestyle with plenty of lean muscle mass. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2024/02/Screenshot-2024-02-24-at-10.11.33-AM.png) Image source: [Chronic exercise preserves lean muscle mass in masters athletes](https://pubmed.ncbi.nlm.nih.gov/22030953/?ref=jeremyjordan.me) Of course I still might sign up for the occasional triathlon competition too for the fun of it, it's just not the main goal. **Do you plan to increase your distances?** Probably not. I want this to be a feasible goal that I can complete every week for decades. On weeks where I travel out of town, I might end up doing all three activities (swimming, biking, and running) in the same day after returning home from a trip. In the future, when life gets busier raising a family, I'd like to still be able to carve out time for my triathlons. By sticking with the [sprint triathlon](https://en.wikipedia.org/wiki/Triathlon?ref=jeremyjordan.me#Race%5Fformats) distances, I can get my baseline level of activity for the week done in \~2-3 hours. This also saves more time for me to be active in other ways (e.g. soccer and tennis) throughout the week! **What if you're injured?** If I have an injury where exercise would prolong the healing, I will try to adapt until I heal (e.g. go for a long walk instead of a run). However, trying to keep this consistent weekly habit has turned out to be a good motivator to stay injury free! I've been more intentional about stretching and strength training to try to avoid injuries. ### Random software I love to use URL: https://www.jeremyjordan.me/random-software/ Last updated: 2024-04-28T01:08:22.000Z A collection of little software programs that I enjoy on my MacOS devices. ### BlackHole: Audio Loopback Driver This software allows you to easily record system audio. You can also [configure your sound settings](https://github.com/ExistentialAudio/BlackHole/wiki/Multi-Output-Device?ref=jeremyjordan.me) to output both to BlackHole (to record the audio) and your speakers (to hear what's being recorded). [GitHub - ExistentialAudio/BlackHole: BlackHole is a modern macOS audio loopback driver that allows applications to pass audio to other applications with zero additional latency.BlackHole is a modern macOS audio loopback driver that allows applications to pass audio to other applications with zero additional latency. - ExistentialAudio/BlackHole![](https://github.githubassets.com/assets/pinned-octocat-093da3e6fa40.svg)GitHubExistentialAudio![](https://repository-images.githubusercontent.com/211381291/44f8b580-75b2-11ea-9438-4fc514e081bb)](https://github.com/ExistentialAudio/BlackHole?ref=jeremyjordan.me) ### BrainWorkshop: Dual n-back training Dual n-back training is a cognitive training exercise designed to help [improve working memory, focus, and problem-solving skills](https://www.pnas.org/doi/abs/10.1073/pnas.0801268105?ref=jeremyjordan.me). The [original application](https://brainworkshop.sourceforge.net/tutorial.html?ref=jeremyjordan.me) has not been maintained and no longer works on the latest MacOS devices, but there's a fork available on Github which has continued development. [GitHub - brain-workshop/brainworkshop: Continued development of the popular brainworkshop gameContinued development of the popular brainworkshop game - brain-workshop/brainworkshop![](https://github.githubassets.com/assets/pinned-octocat-093da3e6fa40.svg)GitHubbrain-workshop![](https://opengraph.githubassets.com/ac52f94e6965b76848a06d763ce87ad3d4bb312a8b6ed5bda4fd04d73e8a9ab0/brain-workshop/brainworkshop)](https://github.com/brain-workshop/brainworkshop?ref=jeremyjordan.me) After cloning the repo, you can create a virtual environment, install the requirements, and start the application by running `python brainworkshop.py`. For convenience I created a `.command` file that I can click on to easily open the app. ``` cd ~/Desktop echo "cd ~/Documents/Code/brainworkshop && source .venv/bin/activate && python brainworkshop.py" > dualnback.command chmod +x dualnback.command ``` You can drag an icon to replace the default icon by right clicking the file and selecting "Get Info" (or just `⌘+I` after selecting the file) and dragging a new image over the default icon. I also found it helpful to go to my Terminal Settings -> Profiles -> Shell and selecting *"When the shell exits: Close if the shell exited cleanly"*. ## Posts ### Exploring the age of continuous work. URL: https://www.jeremyjordan.me/continuous-work/ Last updated: 2026-06-17T14:02:39.000Z A few weeks ago, I was swimming laps after work, reflecting on my day. This eventually led to a spark of inspiration for a potential project, and I spent the rest of my workout mulling over this idea. As I biked home, I opened up my voice notes app and proceeded to do a verbal dump of my brainstorming session at the pool. When I got home, I copied a cleaned up version of my voice note transcript into a Google Doc and refined the vision. I spun up Codex, set a goal, pasted in my vision, and Codex has been working to build out this vision ever since. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2026/06/00-thinking-in-life.png) This blog post explores a weird but powerful new way of building software that I've started to refer to as **continuous work**. The premise is straightforward: establish an ambitious goal with a clear vision and task a coding agent to work nonstop, around the clock, building towards that goal. I'll share how I've adapted my development process to explore this new mode of working, and peek into a future where this becomes the predominant way that people collaborate on ambitious projects. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2026/06/agent_loop-4.gif) The relentless drive of an agent working through item after item: editing code, running tests, reasoning about how the code should look, and checking each change against the vision and principles before moving on to the next item. ## The core principle Rule #1 of continuous work is that **your main agent thread should never be idle**; this means that we need to create the conditions for the agent to work without stopping. It starts with an ambitious goal; if the goal is too simple, agents will simply complete the task and then stop and wait for further instruction (which breaks rule #1). ℹ️ Setting a `/goal` puts the agent in a loop: do some work, check against the goal's criteria, decide what's next, repeat. It keeps going on its own until the goal is met, it's blocked, or the budget runs out. This also means that you are no longer in the driver's seat; you need to **reorient your development process to allow the agent to work autonomously**. The agent will work relentlessly while you sleep, eat dinner with your family, go to the gym, and generally live your life. In order to keep the agent on a productive path, I have settled into a pattern where I'll periodically check in, see what work it's recently completed, notice when it may be heading in the wrong direction, and provide periodic nudges to steer the agent towards the path that better aligns with my vision. Humans become the guide, not the gatekeeper. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2026/06/01-periodic-course-correction-with-goal.png) ## Keeping a pulse on progress Figuring out the right way to check in on the work has been an interesting challenge, given that it's not practical to read every line of code the agent added to or deleted from the repo. I've settled into a pattern where I instruct the agent to maintain documentation that represents a higher-level vantage point for me to inspect. Some of these documents include: - a **roadmap** to document what work has been completed and what the agent is planning to work on next - the shared **vision** for what we're working towards - a technical **specification** for how the overall system works alongside some architecture diagrams - an **experiment log** for targeted explorations (e.g. working on performance improvements) ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2026/06/04-gauge-instrument-desk.png) These documents serve as an effective entrypoint for me to understand the state of the system and allow me to prioritize which parts of the codebase I should more closely inspect. ## Steering the agent When I do want to take a closer look, I'll usually open up a side thread where I can ask the agent questions, understand the context behind certain technical decisions, and share my input on the direction I'd like to go. Doing this in a side thread allows the main agent thread to stay focused on relentless execution, adhering to the core principle we previously established. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2026/06/agent_loop-5.gif) ℹ️ A side thread forks the main conversation — preserving its full history as context — so you can ask questions or talk through direction without disturbing the main thread. Codex calls it `/side`; Claude Code calls it `/fork` (now `/branch`). As I gain a better understanding of the current state of things and the reasoning behind the decisions that got us here, I will sometimes want to redirect the agent to explore a new direction. To do this, I'll share my feedback and ask the side thread agent to write a concise "design nudge" to share in the main thread and steer the main agent. This can involve multiple iterations of back and forth as we work to crisply define the guidance to share with the main agent. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2026/06/03-side-thread-nudge.png) I've learned that the most useful feedback isn't just about the specific thing the agent is working on, it's about the principles behind it. When I share feedback with the agent, I try to explain my thinking in terms of durable principles that the agent can reuse when making future decisions. It's incredibly satisfying to give some feedback and share the principled reasoning and see the agent using that same line of reasoning when solving a problem two days later. ℹ️ In order to remember feedback for the future, the agent will commonly document the principles from my feedback into its [AGENTS.md](https://agents.md/?ref=jeremyjordan.me) file. While principles are useful for teaching the agent how I want it to think, guardrails help catch the ways it tends to go wrong. I'm not going to be monitoring each and every line of code the agent writes, so it's important to expose signals it can use to detect issues and correct course on its own. I've found Rust to be a great language for coding agents: the compiler is strict enough that whole classes of bugs become precise compile errors, giving the agent fast, automatic feedback it can act on before anything reaches me. On top of that, I've set up a set of "quality gates" the agent runs after completing each unit of work: - standard unit and integration tests help prevent behavioral regressions, - [cargo-crap](https://minikin.me/blog/cargo-crap?ref=jeremyjordan.me) helps fight against untested complexity, - differential tests are useful in comparing output against a known reference (e.g. comparing a slow but trusted method against a performance-optimized method), - and I've also had the agent write its own linting tools to enforce feedback I've given around areas like code organization. Building out this kind of environment is a big part of what makes continuous work sustainable: the richer the signals around the agent, the more it can catch and correct on its own — and the longer it keeps making real progress while I'm off doing something else. ## The shifting nature of work With this new style of continuous work, I feel like one of the most important things humans can do is **spend more time thinking and learning**. Let the agents build and prototype and experiment with new ideas; our role shifts to building a deeper connection to the problem, continuing to learn about the world, and figuring out the right questions to ask. That kind of thinking starts with the goal itself. For this project, I thought hard about the problem and the general shape of the solution I wanted to see. Then I handed the agent a clearly specified goal to build toward. I sometimes see people recommend having an agent write their goal, but I think there's real value in humans thinking deeply about what we want the agents to build. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2026/06/02-agent-robot-refiner.png) To me, this presents an optimistic view of the future. I love learning! Having more time for it is a gift. And you don't need to be tied to a desk to think deeply; it doesn't matter where your body is, it matters where your mind is. The idea behind this whole project came to me while I was swimming laps at the pool. You can work through a hard problem on a walk in the forest or in your favorite cozy spot, and I think that's a delightful thing. ## Leaning in Suppose we lean into this new style of work and push the frontier. What could this look like? One emerging pattern shifts the main agent from executor to orchestrator: instead of writing all the code itself, it breaks the goal into bounded tasks, hands them to subagents, and stitches the results back together. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2026/06/05-orchestrator-subagents.png) There have been times when I've taken on the orchestrator role myself, and it can be incredibly taxing. Handing that coordination to a main agent lets us scale the system's productivity without over-saturating our mental bandwidth. From there, the system scales as a function of the main agent's capabilities and speed. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2026/06/06-subagent-merge-loom-v4-1.png) ## Collaboration in the age of continuous work Scaling up the system's output is one interesting dimension to explore, but I think an equally interesting one is scaling up the team steering the agentic system. The same main thread that one person watches and nudges can become something a whole team watches, discusses, and shapes together. When humans collaborate, they share ideas and diverse perspectives, ask questions to seek clarity, and push back to refine a design. The vision comes out stronger and clearer for it. Perhaps, as the nature of our work shifts, we'll place an even higher value on collaboration with other humans. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2026/06/07-roundtable-factory-overlook.png) Right now the coding agents are largely designed for personal interactions, but I can envision a future where an entire team can observe the main thread as it works continuously towards a shared vision; different people can dig into different areas, discuss amongst the team, and shape nudges to share with the main agent. ## Closing thoughts This style of working is new and weird and it can feel uncomfortable to hand control over to the agents. I don't think we're at a point where this works well for every problem, but for a certain shape of problem it can be very powerful. I've been running this experiment alongside my normal day-to-day work, where I use coding agents on bounded tasks that end in a pull request. In that mode, I'll go back and forth with the agent on exactly the right approach, shaping the code until it's how I want it. In continuous work, I let that go and trust the agent to find its own way, even when it isn't exactly what I'd have chosen. Moving between the two creates real cognitive dissonance. How do we best leverage something that is extremely powerful, but not perfect? In The Art of Doing Science and Engineering, Richard Hamming wrote about the shift from analog to digital computing; he observed that the people who helped drive progress were the ones who recognized the new thing as new and studied it on its own terms. Continuous work feels like one of those moments, and I think it's worth exploring. ## Acknowledgements Thank you to Kalpesh Sutaria for showing me what's possible when you task agents with ambitious goals and set up the right oversight to keep them on track. Thank you to Ian (伊恩) for creating and sharing the [ian-xiaohei-illustrations](https://github.com/helloianneo/ian-xiaohei-illustrations?ref=jeremyjordan.me) skill that I used to make the illustrations in this blog post. ### Training extremely large neural networks across thousands of GPUs. URL: https://www.jeremyjordan.me/distributed-training/ Last updated: 2025-09-08T03:09:54.000Z Over the past few years, we've seen an incredible rate of scaling when it comes to training deep neural networks. Today's large language models, for instance, require an immense amount of compute and data to reach state of the art performance. This scale has been made possible by techniques that allow us to **distribute** the training of these neural networks across a large cluster of computers, all working in concert to apply gradient descent optimization to a model at massive scale. 🔍 For example, [Llama 3.1 405B](https://ai.meta.com/research/publications/the-llama-3-herd-of-models/?ref=jeremyjordan.me) was trained using $3.8 \\times 10^{25}$ FLOPs (floating point operations) of compute using Nvidia's H100 GPUs. These GPUs are capable of $1.979 \\times 10^{15}$ FLOPs per second using FP16 tensor cores ([source](https://resources.nvidia.com/en-us-tensor-core/nvidia-tensor-core-gpu-datasheet?ref=jeremyjordan.me)), which would mean, assuming perfect utilization and ignoring memory limitations, it would take ***\~609** ***years** to fully train the model on a single GPU. Obviously, it wouldn't make sense to do this, which is why Meta used 16,000 H100 GPUs (worth hundreds of millions of dollars) to train the model in 54 days. In this blog post, we'll discuss techniques such as data and model parallelism which allow us to distribute the model training process across a large cluster of machines. ## First, let's revisit the basics Before we dive into distributed training techniques, let's quickly refresh on the fundamentals of [how a neural network is trained](https://www.jeremyjordan.me/neural-networks-training/). Neural networks are effectively defined by a set of parameterized transformations of our data in order to map the input data to an expected output. We start off with randomly initialized parameters and iteratively refine their values through a [gradient descent](https://www.jeremyjordan.me/gradient-descent/) optimization process. This gradient descent optimization requires us to define a **loss function** which describes how close the model's *actual* output values are to the *expected* output. We can then compute the partial derivative of our loss function with respect to each parameter in the model (collectively referred to as the **gradient**) to describe how changes to our model parameters would affect the computed loss. Using this information, we make small updates to each parameter in the model in the direction that would result in a lower value to our computed loss. The magnitude of our update to each parameter is dictated by a **learning rate**, which effectively controls how "fast" we're able to converge on an optimal set of parameters. ## Scaling training to larger batches In practice, instead of using the *entire* dataset to compute the gradient during each optimization step, we typically use mini-batches of data, which allows for more frequent updates to our model while still providing a good approximation of the overall gradient. Smaller batches provide noisier estimates of the gradient; which can be faster to compute but may require more optimization steps to converge on a target loss value. Larger batches, on the other hand, give us a better approximation of the true gradient (leading to smoother optimization dynamics and allowing for us to use a higher learning rate) at the cost of being more expensive to compute per batch. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2024/09/Screenshot-2024-09-29-at-4.42.00-PM.png) An illustration from [An Empirical Model of Large-Batch Training](https://arxiv.org/pdf/1812.06162.pdf?ref=jeremyjordan.me) (Figure 2) demonstrating the improved training dynamics of low-variance estimates of the gradient, allowing us to use a higher learning rate and resulting in fewer optimization steps to reach the desired minimum point. Moreover, as you increase the batch size you'll hit a point where it gives a very close approximation to the true gradient and increasing the batch size further past this point yields minimal benefit (e.g. it's more expensive to compute but doesn't meaningfully improve the estimate of our gradient). As discussed in [An Empirical Model of Large-Batch Training](https://arxiv.org/pdf/1812.06162.pdf?ref=jeremyjordan.me), this transition happens at different batch sizes depending on your dataset and the complexity of your modeling task. The paper also provides some guidelines for determining at what batch size this transition will happen based on the per-sample gradients of your input data. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2024/09/Screenshot-2024-09-29-at-4.53.02-PM.png) An illustration from [An Empirical Model of Large-Batch Training](https://arxiv.org/pdf/1812.06162.pdf?ref=jeremyjordan.me) (Figure 3) demonstrating the two regimes of ****perfect scaling**, where you can scale the learning rate linearly with batch size to speed up training, and ****ineffective scaling**, where the optimal learning rate starts to scale sub-linearly with respect to batch size. One very impactful result of this paper is that they demonstrate you can often scale to **extremely** large batches before hitting this transition point for modern day datasets - referencing optimal batch sizes in the tens of thousands of images for ImageNet and up to millions of samples per batch for some reinforcement learning tasks. It's also important to note that the transition point (referred to as *gradient noise scale* in the paper) will *increase* during training as the model's output improves and the gradients become more informative. As a result, you can often increase the batch size as your model trains, such as what was done for [Llama 3.1 405B](https://ai.meta.com/research/publications/the-llama-3-herd-of-models/?ref=jeremyjordan.me): > ...we use an initial batch size of 4M tokens and sequences of length 4,096, and double these values to a batch size of 8M sequences of 8,192 tokens after pre-training 252M tokens. We double the batch size again to 16M after pre-training on 2.87T tokens. We found this training recipe to be very stable: we observed few loss spikes and did not require interventions to correct for model training divergence. ## Pushing past the limits of a single machine If you want to scale your model training to larger batches, however, you're very quickly going to hit the limits of what you can do on a single GPU. Specifically, you'll experience the infamous `RuntimeError: CUDA out of memory` message that most machine learning engineers know all too well. Let's take a second to enumerate the various things that we need to keep in memory while training a model. - **Model parameters**: the learnable weights of our model. - **Optimizer states**: the exact state that you need to track depends on what optimizer you're using; for example, if you're using [AdamW](https://pytorch.org/docs/stable/generated/torch.optim.AdamW.html?ref=jeremyjordan.me), you'll need to track the first and second momentum estimates in addition to the model parameters. - **Model activations**: this will vary based on the architecture of your network and batch size, but can significantly impact memory usage. This information is required for [backpropagation](https://www.jeremyjordan.me/neural-networks-training) to allow us to *efficiently* compute our gradients. - **Gradients**: stored for each parameter of the model, same memory footprint as the model parameters. - **Input data**: the batch of input data to be passed to the model, the memory footprint depends on the size and type of data being modeled. You can look at the overall memory footprint during model training, broken down by category, using [PyTorch's memory profiler](https://pytorch.org/docs/stable/profiler.html?ref=jeremyjordan.me). ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2024/09/memory_footprint.png) The memory footprint during training of a small Transformer model on random data for 5 steps. ### Tricks to reduce memory consumption There are some tricks that we can employ to squeeze more out of single GPU training or generally reduce the overall memory footprint during training, but these tricks can only take us so far and often trade off reducing memory consumption by requiring more computation. - **Gradient accumulation** allows us to scale to larger effective batch sizes by processing smaller batches sequentially. Instead of computing gradients on the full batch at once (which would require storing all activations in memory), we add up the gradients from each small batch before updating the model parameters. This reduces memory usage but requires more forward/backward passes. - **Activation checkpointing** allows us to "forget" (i.e. release from memory) some of the activations during the forward pass and recompute the values when needed during the backward pass. - **CPU offloading** allows us to transfer some of the state to CPU so we don't have to hold everything in GPU RAM. While CPU operations are slower than GPU operations, moving less frequently accessed data to CPU memory can help us stay within our GPU memory constraints. All three of these techniques effectively trade extra computation time for reduced memory usage (and there's still a limit to how much they can help in a single GPU setting). In order to *efficiently* scale up to larger model sizes and ever-growing datasets while still training the model in a reasonable amount of time, we need to **distribute** our computation over a cluster of machines. There's two main axes by which we can "split up" the model training process to work across multiple machines: - **Data parallelism**: this approach splits the input batch across multiple GPUs, where each GPU has its own copy of the model. Each GPU processes its portion of the data independently, then all GPUs work together to combine their results and update the model. This helps us handle larger batches of data without running into memory limits from input data and activations. - **Model parallelism**: when a model is too large to fit on a single GPU (considering parameters, optimizer states, gradients, and activations), we can split it across multiple GPUs in two ways: (1) by dividing the layers across different GPUs, or (2) by splitting individual layers themselves across GPUs. For the most demanding scenarios, advanced techniques like [**fully sharded data parallelism**](https://arxiv.org/abs/2304.11277?ref=jeremyjordan.me) combine both approaches to push the limits of what's possible with distributed training. ## Data parallelism Let's dive into how data parallelism works in practice. The core idea is simple: we want to process larger batches of data by spreading the work across multiple GPUs. To make this work in practice, these GPUs need to coordinate during the training process to ensure that each GPU maintains an **identical** copy of a model while training (resulting in one final trained model at the end of training). Let's consider an example with 4 GPUs where we want to process a batch of 1024 samples in each step of our training loop: - Each GPU gets a copy of the model - The batch is split into 4 chunks of 256 samples each - Each GPU independently processes its portion of the data - The GPUs must then coordinate to update the model weights 0:00 /0:37 1× The first three steps are straightforward - each GPU can compute its predictions and gradients **independently**. However, if each GPU were to update its model copy using only its local gradients, the models would slowly drift apart as they learn from different portions of data. To keep the models synchronized, we need to communicate the gradient information across all GPUs before updating the model parameters on each device. ✍️ Distributed training relies on several key communication primitives: ****scatter** (distribute different data to each GPU), ****gather** (collect all data to one GPU), ****broadcast** (share same data to all), and ****reduce** (combine data to one GPU). ****All-gather** and ****all-reduce** are variants where the result is shared with all GPUs rather than just one. To keep our gradients synchronized in data parallel training, we use the **all-reduce** communication primitive. Here's how it works: - Each GPU computes gradients on its portion of data - All GPUs participate in an all-reduce operation to sum the individual local gradients - Each GPU takes this gradient reduction (sum) and computes the average gradient by dividing by the total number of GPUs - Each GPU then applies an identical update using this average gradient, keeping the models synchronized The all-reduce operation has highly-optimized implementations in common distributed communication libraries to minimize communication overhead between GPUs. Instead of every GPU sending its gradients to a central coordinator that averages the gradients and sending the results back to each device (which would create a bottleneck), libraries use clever communication patterns that maximize bandwidth utilization across the available network topology. 0:00 /0:48 1× An animation of the ****ring all-reduce algorithm** for efficiently sharing the local gradients across all GPUs. This algorithm aims to ****maximize the communication bandwidth** between GPUs by simultaneously sharing "chunks" of the gradient vector between GPUs until all GPUs have the full state. The values of the individual chunks are summed as they're accumulated across GPUs. Data parallelism allows us to **scale up the effective batch size** by distributing the input data across multiple GPUs. However, this approach requires storing a complete copy of the model on each GPU. For today's largest models with hundreds of billions of parameters, a single GPU simply doesn't have enough memory to store the model, making it necessary to distribute the model itself across multiple devices. ## Model parallelism For scenarios where the model is too large to fit on a single GPU, we need model parallelism to distribute the parameters across multiple GPU devices. As we discussed previously, there's two main ways we can split up the model: we can either distribute the various layers across different GPUs (pipeline parallelism) or split individual layers themselves across GPUs (tensor parallelism). Let's examine how each approach works and the trade-offs involved. #### Pipeline parallelism In order to explain pipeline parallelism, let's consider a model architecture with 16 layers. In order to distribute these model parameters across 4 GPUs, we'd place layers 1-4 on GPU0, layers 5-8 on GPU1, layers 9-12 on GPU2 and layers 13-16 on GPU3\. For a given input batch during training we'd load the input data on GPU0 and pass it through the first four layers, then we'd take the activations of layer 4 and send it as input to layer 5 on the next GPU, then taking the activations of layer 8 and passing it along until we've reached the end of the network and have a final prediction. From here, we can compute our loss and start the backwards pass. During the backwards pass, we propagate gradients through our pipelined model as follows: 1. On GPU3 (layers 13-16), we start with the gradient of the loss with respect to the model output $\\frac{\\partial L}{\\partial Y}$. We compute the gradients for each layer on GPU3 and end up with $\\frac{\\partial L}{\\partial A\_{12}}$ where $A\_{12}$ represents the activation matrix output from layer 12 (input to layer 13). 2. This gradient signal $\\frac{\\partial L}{\\partial A\_{12}}$ is then sent to GPU2, which is used to compute gradients for layers 9-12 , ultimately producing $\\frac{\\partial L}{\\partial A\_{8}}$ to be sent to GPU1 3. The process continues, with GPU1 using $\\frac{\\partial L}{\\partial A\_{8}}$ to compute gradients for layers 5-8 4. Finally, GPU0 receives $\\frac{\\partial L}{\\partial A\_4}$ and computes gradients for layers 1-4. This sequential processing across GPUs creates a challenge in GPU utilization. Consider what happens when processing a batch of data: 1. Initially, only GPU0 is active, processing its layers 2. Then GPU1 starts working, but GPU0 goes back to an idle state 3. Then GPU2 starts, while GPU0 and GPU1 are idle 4. Finally GPU3 processes its layers while the other GPUs wait 5. During the backward pass, the GPUs activate in reverse order, each processing while others remain idle This leads to significant GPU idle time - a phenomenon known as the "pipeline bubble" because of the gaps in GPU utilization it creates. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2025/02/Screenshot-2025-02-17-at-10.56.53-AM.png) A visualization from "[GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism](https://arxiv.org/abs/1811.06965?ref=jeremyjordan.me)" depicting the "pipeline bubble" challenge. The white space represents idle time where the GPU devices are not being utilized. One way to mitigate this issue is to split a batch into multiple **micro-batches**, allowing different GPUs to work on different micro-batches at the same time. Instead of GPU0 initially processing the entire input batch at once, we'll have it compute a subset (i.e. micro-batch) of the input and pass on those activations to GPU1 to compute the activations of the next layers. This allows GPU0 to compute the next subset of inputs *while* GPU1 is computing the first subset of inputs. This staggering of computation allows more GPUs to work in parallel, increasing the overall device utilization and reducing the size of the "pipeline bubble". Each device accumulates the computed gradients across all micro-batches before applying an update to the model weights at the end of the pipeline execution. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2025/02/Screenshot-2025-02-17-at-10.57.15-AM.png) A visualization from "[GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism](https://arxiv.org/abs/1811.06965?ref=jeremyjordan.me)" depicting the reduction in idle GPU time as a result of splitting a batch into micro-batches. In other variants of pipeline parallelism, model weights can be updated after each micro-batch, resulting in even near-perfect GPU utilization across devices as a result of more dense interleaving of forward and backward passes from the various micro-batches. However, this increased utilization comes at the cost of greater inconsistency as a result of different micro-batches seeing different versions of the model weights. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2025/02/Screenshot-2025-02-17-at-12.03.02-PM.png) A visualization from "[PipeDream: Fast and Efficient Pipeline Parallel DNN Training](https://arxiv.org/abs/1806.03377?ref=jeremyjordan.me)" showing how we can completely remove the pipeline bubble if you remove the constraint of synchronously updating the model weights. In summary, pipeline parallelism introduces a simple approach to splitting a large model across GPUs that can be applied to any model architecture without special consideration of layer computations. The main challenge with this approach is ensuring that we can achieve sufficiently high utilization to be able to benefit from the parallel computation. #### Tensor parallelism In order to further increase compute and memory efficiency, tensor parallelism offers an approach which splits the underlying mathematical operations of an individual layer across multiple GPUs. This approach allows easier parallelization of compute (compared to pipeline parallelism) at the cost of increased communication between GPUs. Let's examine how tensor parallelism works for linear layers, which form the foundation of neural networks and account for the majority of parameters in modern architectures. A linear layer performs the operation $Y = XW$, where: - $X$ is the input tensor with shape $\[\\text{batch\_size}, \\text{input\_dim}\]$ - $W$ is the weight matrix with shape $\[\\text{input\_dim}, \\text{output\_dim}\]$ - $Y$ is the output tensor with shape $\[\\text{batch\_size}, \\text{output\_dim}\]$ There are two main ways we can partition this matrix multiplication across devices: column and row-wise partitioning. ###### Column Partitioning In column partitioning, we split the weight matrix $W$ along its output dimension. This means each GPU stores and computes only a subset of the output features. 0:00 /0:29 1× **Forward Pass:** 1. The input $X$ is broadcast to all GPUs 2. Each GPU $i$ computes its portion of the output using its slice of weights: $Y\_i = XW\_i$ 3. The partial outputs are concatenated via an all-gather operation: $Y = \[Y\_0|Y\_1|...|Y\_{n-1}\]$ **Backward Pass:** 1. Each GPU receives the slice of output gradients $\\frac{\\partial L}{\\partial Y\_i}$ corresponding to its portion of $W$ 2. Gradients for the local weights are computed: $\\frac{\\partial L}{\\partial W\_i} = X^T \\frac{\\partial L}{\\partial Y\_i}$ 3. The gradients with respect to $X$ require combining information across GPUs via all-reduce: $\\frac{\\partial L}{\\partial X} = \\sum\\limits\_{i} \\frac{\\partial L}{\\partial W\_i}$. These gradients represent how much the activations from the previous layer contributed to the overall loss and are needed to continue the backward pass through earlier layers in the network. ➡️ This approach requires an ****all-gather** in the forward pass to concatenate partial outputs, and an ****all-reduce** in the backward pass to compute gradients with respect to $X$. ###### Row partitioning In row partitioning, we split the weight matrix $W$ along its input dimension. This means each GPU works with only a subset of the input features. 0:00 /0:31 1× **Forward Pass:** 1. Each GPU receives only the portion of $X$ corresponding to its rows of $W$ 2. Each GPU computes partial outputs: $Y\_i = X\_iW\_i$ 3. The partial outputs are combined via all-reduce: $Y = \\sum\\limits\_{i} Y\_i$ **Backward Pass:** 1. The full gradient $\\frac{\\partial L}{\\partial Y}$ is broadcast to all GPUs 2. Each GPU computes gradients for its local weights: $\\frac{\\partial L}{\\partial W\_i} = X\_i^T \\frac{\\partial L}{\\partial Y}$ 3. Each GPU computes gradients for its portion of $X$: $\\frac{\\partial L}{\\partial X\_i} = \\frac{\\partial L}{\\partial Y}W\_i^T$. These portions must be combined via all-gather to reconstruct the full $\\frac{\\partial L}{\\partial X}$ tensor needed to continue the backward pass through earlier layers. ➡️ This approach has a complementary communication pattern - an ****all-reduce** in the forward pass to sum partial outputs, and an ****all-gather** in the backward pass to reconstruct the full gradient tensor for $X$. ###### Optimizing Communication Through Clever Partitioning Tensor parallelism requires substantial communication between GPUs, but this overhead can be minimized through careful choices in how we partition consecutive linear layers, as discussed in [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/abs/1909.08053?ref=jeremyjordan.me) where this technique was first introduced. Consider a sequence of two linear layers with a nonlinear activation function (like GeLU) between them: 1. If we use column partitioning (split weights along the output dimension) for the first layer, each GPU can independently apply the activation function to its portion of the output 2. By then using row partitioning (split weights along the input dimension) for the second layer, each GPU can directly consume its local activations without requiring any communication 3. Finally, we can apply an all-reduce across the devices to combine the final outputs ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2025/02/mlp_mp_2.png) A visualization from [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/abs/1909.08053?ref=jeremyjordan.me) showing how clever partitioning of the weight matrices $A$ and $B$ allow for efficient distributed processing and reduce the communication overhead. In this scenario, we only need a single all-reduce operation $g$ in the forward pass (to combine the results after the second linear layer) and a single all-reduce operation $f$ in the backward pass (to compute gradients with respect to the first layer's input activations), overall reducing the number of synchronization points by 50% compared to a naive implementation. ## Putting it all together The parallelization strategies we've discussed offer complementary approaches for distributed training that can be combined to maximize training efficiency and scale. However, because these techniques have different communication patterns, the optimal balance and configuration of the different types of parallelism are influenced by your training cluster's network topology. Modern GPU clusters typically have a hierarchical network structure: GPUs within a node are connected by very high-bandwidth links (like NVLink), while cross-node communication happens over relatively slower network connections (like InfiniBand). For example, [Llama 3.1 405B](https://ai.meta.com/research/publications/the-llama-3-herd-of-models/?ref=jeremyjordan.me) was trained using: > ...up to 16K H100 GPUs, each running at 700W TDP with 80GB HBM3, using Meta's Grand Teton AI server platform. Each server is equipped with eight GPUs and two CPUs. Within a server, the eight GPUs are connected via NVLink... each rack hosts 16 GPUs split between two servers and connected by a \[network switch\] … 192 such racks are \[connected\] to form a pod of 3,072 GPUs … eight such pods within the same datacenter building are \[connected\] to form a \[total cluster size\] of 24K GPUs. Taking this typical hierarchical network topology into consideration: - **Data parallelism** helps us process larger batches by distributing them across many nodes. Since it only requires synchronization during gradient averaging, it remains efficient even with slower cross-node communication. - **Tensor parallelism** works best for GPUs with high-speed interconnects, making it ideal for splitting large matrix operations across GPUs within the same node where frequent communication won't become a bottleneck. - **Pipeline parallelism** provides an efficient way to split model layers across nodes, minimizing cross-node communication by only needing to pass activations between sequential stages. Continuing with our example, we can see that [Llama 3.1 405B](https://ai.meta.com/research/publications/the-llama-3-herd-of-models/?ref=jeremyjordan.me) was trained using tensor parallelism of 8, pipeline parallelism of 16, and data parallelism ranging from 8 to 128 as the researchers adjusted the batch size during training. At its peak, the model training was distributed across 16,384 GPUs. ## Conclusion Data, pipeline, and tensor parallelism have enabled researchers and engineers to push the limits of model training to an incredible scale resulting in some seriously impressive capabilities. In addition to these foundational techniques, we've also seen techniques introduced such as [context parallelism](https://arxiv.org/abs/2411.01783?ref=jeremyjordan.me) (for training on long sequence lengths) and [expert parallelism](https://arxiv.org/abs/2101.03961?ref=jeremyjordan.me) (for training sparse models) in order to further improve the scale and efficiency of training state of the art models. Moreover, these innovations aren't slowing down – the GPU hardware for training gets better every year and new algorithmic improvements (such as [DualPipe](https://github.com/deepseek-ai/DualPipe?ref=jeremyjordan.me) introduced in [DeepSeek-V3 Technical Report](https://arxiv.org/abs/2412.19437?ref=jeremyjordan.me)) continue to improve the efficiency of large training runs. ## References **Papers** - [PyTorch Distributed: Experiences on Accelerating Data Parallel Training](https://arxiv.org/abs/2006.15704?ref=jeremyjordan.me) - [GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism](https://arxiv.org/abs/1811.06965?ref=jeremyjordan.me) - [PipeDream: Fast and Efficient Pipeline Parallel DNN Training](https://arxiv.org/abs/1806.03377?ref=jeremyjordan.me) - [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/abs/1909.08053?ref=jeremyjordan.me) - [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://arxiv.org/abs/2104.04473?ref=jeremyjordan.me) - [The Ultra-Scale Playbook: Training LLMs on GPU Clusters](https://huggingface.co/spaces/nanotron/ultrascale-playbook?ref=jeremyjordan.me) - [The Llama 3 Herd of Models](https://ai.meta.com/research/publications/the-llama-3-herd-of-models/?ref=jeremyjordan.me) **Talks** - [Model vs Data Parallelism in Machine Learning | Mark Saroufim](https://www.youtube.com/watch?v=ZkWs89H7baA&ref=jeremyjordan.me) - [Distributed Data Parallel in PyTorch Tutorial Series](https://www.youtube.com/watch?v=-K3bZYHYHEA&list=PL%5FlsbAsL%5Fo2CSuhUhJIiW0IkdT5C2wGWj&ref=jeremyjordan.me) - [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM | Jared Casper](https://www.youtube.com/watch?v=gHaNUcS1%5FO4&ref=jeremyjordan.me) - [Fully Sharded Data Parallel (FSDP) and Pipeline Parallelism in 3D with Vision Pro | Will Falcon](https://www.youtube.com/watch?v=kgOJHAdNnGU&ref=jeremyjordan.me) - [Stanford CS231N | Spring 2025 | Lecture 11: Large Scale Distributed Training](https://www.youtube.com/watch?v=9MvD-XsowsE&ref=jeremyjordan.me) ### Understanding the Transformer architecture for neural networks URL: https://www.jeremyjordan.me/transformer-architecture/ Last updated: 2024-04-07T16:56:48.000Z In a previous post, we discussed a key innovation in sequence to sequence neural network architectures: [the attention mechanism](https://www.jeremyjordan.me/attention/). This innovation significantly improved performance for encoder-decoder style [recurrent neural network](https://www.jeremyjordan.me/introduction-to-recurrent-neural-networks/) architectures by removing a bottleneck passing information from the encoder network to the decoder network. As a quick recap, before the introduction of attention we would typically summarize the entire input sequence in a single fixed-size vector to pass along as context to the decoder network. The attention mechanism allows us to merge a variable-length sequence of vectors into a fixed-size context vector at *each* time step in the decoder network. This provides significantly more information bandwidth to the decoder via a learnable mechanism which is capable of determining which time-steps in the input sequence are relevant at each time-step being generated in the output sequence. Let's stop and think about that at a high-level. We have a mechanism which allows us to take a variable-length sequence and merge this information together to output a fixed-size vector. Doesn't that sound a lot like to role of a recurrent neural network? We introduced recurrence as a way to learn from variable-length sequences; but in an effort to squeeze more performance out of those recurrent networks, we accidentally stumbled across another way to process variable-length sequences. This begs the question, what if we tried to get rid of the recurrent layers and simply used attention everywhere? This is the premise behind a seminal paper from 2017, [Attention Is All You Need](https://arxiv.org/abs/1706.03762?ref=jeremyjordan.me), which introduced the Transformer architecture for neural networks. Let's dive in and explore that idea in this blog post. ### Overview - [Swapping out recurrent layers with attention](#replacing-recurrence) - [Generalizing the attention mechanism](#attention-mechanisms) - [Additive attention](#additive) - [Dot product (multiplicative) attention](#dot-product) - [Scaled dot product attention](#scaled-dot-product) - [Defining the Transformer architecture](#architecture) - [Multi-head attention](#multi-head-attention) - [Encoder](#encoder) - [Decoder](#decoder) - [Embeddings](#embeddings) - [Overall architecture](#overall-architecture) - [Benefits of an attention-only architecture](#benefits) - [Resources](#resources) ## Swapping out recurrent layers with attention Let's first walk through what exactly it means to "replace the recurrent layers with attention". Recall that a standard recurrent neural network architecture involves evolving a hidden state across a sequence of inputs. We process each time step by combining information from the current input with information from the previous hidden state. This allows us to pass information along the temporal dimension as we process our sequence. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/04/recurrent-network-1.png) The above visualization shows the "unrolled" representation of our network, but if we were to look at a single time step it becomes more clear that we're only processing the immediately previous hidden state and the current input. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/05/recurrent-single-time-step.png) This forces us to evolve a summary of the *entire input sequence* (represented by the previous hidden state) as we process the sequence step by step. What if we could instead leverage an attention mechanism to look back at every previous time step when building our representation of the current time step? ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/04/attend-over-all-time-steps.png) With this approach, we can flexibly look back across the sequence to retrieve relevant information as we process the next item in the input sequence. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/04/replacing-recurrence-with-attention-1.png) Notice how there is no longer information flowing *through* the hidden state representation like we had in the unrolled recurrent neural network; in this new architecture, each time-step instead looks back across the entire input sequence. In some applications, we want to see the entire sequence at each time step, not just the preceding time-steps. With recurrent neural networks we typically achieved this with a [bidirectional recurrent layer](https://www.jeremyjordan.me/introduction-to-recurrent-neural-networks/#bidirectional) where we had one recurrent layer processing the input in the forward direction and one recurrent layer processing the input in the backward direction, concatenating these outputs together. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/04/bidirectional-rnn.png) We can achieve this same functionality with an attention-only architecture by simply allowing each time step to attend across *all* time steps in the sequence. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/04/bidirectionality-for-free.png) Further, we can still leverage the attention mechanism for its original design, passing information between an encoder and decoder model. In this case we have attention from two different perspectives: - **self attention**: looking across the current sequence being processed (e.g. an encoder network looking across the input sequence being encoded, such as the example shown above) - **cross attention**: looking across a related sequence (e.g. the decoder network looking across a sequence of representations from the encoder network) We describe the type of attention based on the relationship between the sequence that we're currently processing and the sequence that we're using as context. ## Generalizing the attention mechanism With this conceptual understanding in place, let's spend some time digging deeper into how the attention mechanism looks across multiple time steps and combines relevant information into a fixed-size context vector. At a high level, the attention mechanism: - compares a *reference* vector against a *candidate* vector to determine a relevance score between the two vectors - performs the above calculation for a set of candidate vectors - normalizes all of the computed relevance scores (e.g. softmax) - uses the normalized scores to take a weighted combination of the candidate vectors This provides us with a single "context" vector which highlights relevant information from across the sequence, relative to the current time-step. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/05/output.gif) In their paper introducing the Transformer architecture, [Vaswani et al](https://arxiv.org/abs/1706.03762?ref=jeremyjordan.me) refer to the [original attention mechanism implementation](https://arxiv.org/abs/1409.0473?ref=jeremyjordan.me) as "additive attention" and compare this to [another proposed mechanism](https://arxiv.org/abs/1508.04025v5?ref=jeremyjordan.me) known as "dot-product (multiplicative) attention". The authors then build on this second mechanism to introduce a variant known as "scaled dot product attention". Let's discuss these various implementations and show how they relate to the high-level structure presented above. #### Additive attention I [previously introduced](https://www.jeremyjordan.me/attention/#training) the attention model as a small neural network which maps $\\left\[ \\text{reference} ,\\ \\text{candidate} \\right\] \\rightarrow \\text{relevance score}$ where our input is a concatenation of the reference and candidate vectors. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/04/attention-model--additive--3.jpg) A ****really** tiny attention network for the purposes of demonstration. However, instead of concatenating these vectors together and using a single weights matrix to project the input into the hidden layer, we could also project the reference and candidate vectors separately, and then simply add the two projections. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/04/matrix-operations-equivalence-2.jpg) A visualization of matrix operations to demonstrate $Wx^{\\left\[ \\text{ref} ,\\text{cand} \\right\] }=Wx^{\\text{ref} }+Ux^{\\text{cand} }$ It's advantageous to frame this as two projections which are added together, since the same set of candidate vectors will need to be projected for each reference vector. If we separate these projections, we can cache the results of $Ux^{\\text{cand} }$ and reuse this projection in cases where we attend over the same set of candidate vectors multiple times (e.g. cross attention where a decoder autoregressively generates an output sequence and attends over the same sequence of vectors from our encoder representation at each time step). With this reframing, we will define our additive attention model as: $$\\text{relevance} \\left( x^{\\text{ref} },\\ x^{\\text{cand} }\\right) =v^{\\intercal }\\tanh \\left( Wx^{\\text{ref} }+Ux^{\\text{cand} }\\right) = e\_{i,j}$$ $$\\text{attention} \\left( x^{\\text{ref} },X^{\\text{cand} }\\right) =\\text{softmax} \\left( \\begin{matrix}e\_{i,1}\\\\ \\ldots \\\\ e\_{i,n}\\end{matrix} \\right) X^{\\text{cand} }$$ where $W$ and $U$ are our weights matrices to project from the input to the hidden layer and $v$ represents the weights used to project from the hidden layer to our output. We use $\\tanh$ as our nonlinearity in this case but in theory you could swap this out for your nonlinearity of choice. $X^{\\text{cand}}$ denotes our set of candidate vectors as a matrix. These candidate vectors are weighted by the normalized relevance scores and combined to produce our final context vector. This is the attention model introduced by [Bahdanau et al](https://arxiv.org/abs/1409.0473?ref=jeremyjordan.me) (see appendix section A.1.2) with our inputs renamed to use slightly more generic terminology. #### Dot product (multiplicative) attention Instead of using a small neural network to compute the relevance score between a reference and candidate vector, a simpler approach might be to take the [dot product](https://en.wikipedia.org/wiki/Dot%5Fproduct?ref=jeremyjordan.me) of the two vectors; this dot product gives us a direct measure of similarity between two vectors. As a quick reminder, the dot product between two vectors is calculated as: $$\\left\[ \\begin{matrix}a\_{1}\\\\ a\_{2}\\\\ a\_{3}\\end{matrix} \\right\] \\cdot \\left\[ \\begin{matrix}b\_{1}\\\\ b\_{2}\\\\ b\_{3}\\end{matrix} \\right\] =a\_{1}b\_{1}+a\_{2}b\_{2}+a\_{3}b\_{3}$$ If two vectors are aligned in the same direction with similar magnitudes the dot product will be a large positive number. If two vectors are pointing in opposite directions with similar magnitudes, the dot product will be a large negative number. If two vectors are perpendicular, their dot product is zero. This attention model was originally introduced by [Luong et al](https://arxiv.org/abs/1508.04025v5?ref=jeremyjordan.me) where attention scores are simply computed by the dot product of our reference and candidate vectors (in their case, the vectors were hidden states of encoder and decoder LSTM networks). [Vaswani et al](https://arxiv.org/abs/1706.03762?ref=jeremyjordan.me) make a slight modification to this technique, where rather than directly computing the dot product between our reference and candidate vectors, we'll opt to use two learnable weights matrices to project $x^{\\text{ref} }$ and $x^{\\text{cand} }$ into a new representation subspace, similar to what we did for additive attention. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/04/dot-product-attention.png) The projection of our reference vector is referred to as our ****query**. The query vector represents the information that we're searching for across our set of candidate vectors. The projection of our candidate vector is referred to as a ****key**. This represents the information contained within a specified candidate vector. The dot product between these two vectors helps us find keys that a relevant for a given query. This transformation into a new representation subspace allows us to extract/highlight information from the input vectors before computing their relevance as determined by the dot product. This also allows us to project the inputs into multiple representation subspaces such that our model can attend to different characteristics of the input in parallel; this is known as *multi-headed attention* which we'll discuss further in a later section. Finally, rather than using the candidate vectors directly to perform a weighted combination of our candidate vectors (based on the normalized relevance scores), we'll instead introduce a third projection to use in producing our weighted combination across the candidate vectors. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/05/output-projection-1.png) The projection of our candidate vector used in producing the final weighted combination is known as the ****value.** This represents the information from our candidate vector that we wish to express for a given query. In summary, we'll define dot product attention as: $$\\text{relevance} \\left( x^{\\text{ref} },\\ x^{\\text{cand} }\\right) =W\_{\\text{query}}x^{\\text{ref} }\\cdot W\_{\\text{key}}x^{\\text{cand} }=q\\cdot k = e\_{i,j}$$ $$\\text{attention} \\left( x^{\\text{ref} },X^{\\text{cand} }\\right) =\\text{softmax} \\left( \\begin{matrix}e\_{i,1}\\\\ \\ldots \\\\ e\_{i,n}\\end{matrix} \\right) W\_{\\text{value} }X^{\\text{cand} }$$ where our weights matrices $W$ are used to project the input into our query, key, and value vectors and the corresponding relevance scores are denoted as $e\_{i,j}$. Here, $i$ denotes our index into a matrix of reference vectors $X^{\\text{ref}}$ and $j$ represents the index into the matrix of candidate vectors $X^{\\text{cand} }$. For simplicity, we focus the attention calculation on a single reference vector but in practice this can be done in parallel as a single matrix operation treating every vector in our sequence as a reference vector. #### Scaled dot product attention Recall that our relevance scores, $e\_{i,j}$, for a given set of candidate vectors are normalized using the softmax function to produce our final attention weights, $a\_{i,j}$. $$a\_{i,j}=\\frac{\\exp \\left( e\_{i,j}\\right)}{\\sum^{K}\_{k=1}\\exp \\left( e\_{i,k}\\right)}$$ Further recall that our dot product between the query and key vectors is computed as $q\_{1}k\_{1}+q\_{2}k\_{2}+\\ldots +q\_{d}k\_{d}$. If we were to assume that our two vectors were normally distributed (mean of 0 and variance of 1), we would expect their dot product to have a mean of 0 and a variance of $d$. Thus, as the dimensionality $d$ grows larger, so does the variance of the dot product. Unfortunately, large magnitudes of our relevance scores push the softmax function into regions of extremely small gradients, which attenuates our learning signal during training. To counteract this effect, [Vaswani et al](https://arxiv.org/abs/1706.03762?ref=jeremyjordan.me) opt to scale the dot products by $\\frac{1}{\\sqrt{d\_{k}} }$ before normalizing them with the softmax function. With this variant of attention, we compute the relevance score as $$\\text{relevance} \\left( x^{\\text{ref} },\\ x^{\\text{cand} }\\right) =\\frac{1}{\\sqrt{d\_{k}} } \\left( W\_{\\text{query} }x^{\\text{ref} }\\cdot W\_{\\text{key} }x^{\\text{cand} }\\right) =\\frac{1}{\\sqrt{d\_{k}} } \\left( q\\cdot k\\right)$$ where $d\_{k}$ denotes the dimensionality of our query and key vectors. Putting this all together, we can visualize a scaled dot product attention layer using self-attention to process a sequence of inputs. This visual just shows the computation to produce the context vector for the first time step in the sequence, but you can imagine a similar computation happening in parallel for all time steps. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/05/output-8.gif) ## Defining the Transformer architecture Let's now discuss the Transformer architecture as presented by [Vaswani et al](https://arxiv.org/abs/1706.03762?ref=jeremyjordan.me). This architecture was constructed for the task of machine translation and leverages the typical encoder-decoder approach where the encoder processes a sentence in the input language and the decoder uses representations from the encoder to generate a sentence in the target language. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/05/output-2.gif) The overall Transformer architecture is mainly a composition and stacking of just a few building blocks: - scaled dot product attention, - residual connections, - layer normalization, - and feed forward networks. Stacking multiple attention layers on top of each other has the effect of increasing the receptive field. The first attention layer produces context vectors based on interactions between pairs from the original sequence. The second attention layer produces context vectors based on pairs of pairs of the original sequence. As we continue to stack more attention layers on top of each other, we gain a wider perspective considering multiple levels of interactions between items in the original sequence. #### Multi-head attention For the attention layers, the authors leverage a **multi-headed attention** implementation where we perform multiple queries in parallel for each time step. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/05/output-4.gif) With this approach, we end up with a set of context vectors at each time step in the sequence. These context vectors represent different attention summaries of the same sequence which provides us with multiple perspectives on the same input. The vectors are then concatenated together and passed through a final linear layer. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/05/multi-head-attention.png) A visualization of multi-head attention from the original paper. #### Encoder An encoder block is defined as: - **multi-head (self) attention**: queries are computed for each step in the input sequence and are compared against keys computed across all steps in the input sequence - **residual connection and layer normalization**: the input embeddings are added to context vectors produced from the self-attention layer to create a residual connection, and normalized across the layer dimension (e.g. the output vectors for each sequence in the batch will be separately normalized to have zero mean and unit variance) - **feed forward**: we then pass the normalized output from our attention sublayer through a linear projection, ReLU activation, and another linear projection - **residual connection and layer normalization**: finally, we add the projection from the feed forward sublayer to the normalized output from our attention sublayer, and once again normalize this vector to have zero mean and unit variance The encoder model is comprised of 6 of these blocks stacked on top of each other. #### Decoder A decoder block is defined as: - **multi-head (masked self) attention**: queries are computed for each step in the output sequence and are compared against keys computed across all steps in the output sequence, due to the fact output sequence is intended to be generated autoregressively we apply a causal mask to the computed relevance scores (limiting the model to only attend across *previous* time steps) ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/05/output-9.gif) A visualization of causal masking for self-attention. - **residual connection and layer normalization**: the embeddings of our output sequence are added to context vectors produced from the causal self-attention layer to create a residual connection, and normalized across the layer dimension - **multi-head (cross) attention**: we use the output of the previous self-attention sublayer to generate a second set of queries for each step in the output sequence, the keys and values are computed from the output of the final encoder block across all steps in the input sequence ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/05/output-10.gif) A visualization of cross attention, where the keys and values are computed on a separate sequence from the queries. - **residual connection and layer normalization**: the context vectors from the *causal self-attention* sublayer are added to context vectors produced from the *cross-attention* sublayer to create a residual connection, and normalized across the layer dimension - **feed forward**: we then pass the normalized output from our attention sublayer through a linear projection, ReLU activation, and another linear projection - **residual connection and layer normalization**: finally, we add the projection from the feed forward sublayer to the normalized output from our cross-attention sublayer, and once again normalize this vector to have zero mean and unit variance The decoder model is comprised of 6 of these blocks stacked on top of each other. #### Embeddings Our input and output sequences are ultimately represented as a sequence of token ids (integers). To prepare these sequences for our model, we will leverage the standard technique of using this token id as an index to look up a vector of learnable parameters from an embeddings matrix. However, we must also inject some additional information into these embeddings so that the model can learn positional relationships between tokens in our input and output sequences. This is because the attention mechanism doesn't have any notion of spatial relationships between tokens in the sequence. Our query at each time step simply looks across the set of keys and determines which keys are most relevant to our current time step; if we shuffled up the order of the sequence this would have no effect on our pairwise relevance comparisons. However, we know intuitively that token order *should* matter, so the authors develop a technique known as **positional encoding** to inject information about the order of the tokens in our sequence. We encode the position $pos$ in a sequence using a combination of $sin$ and $cos$ functions (alternating these functions for each dimension $i$ in the $d$-dimensional embedding vector). $$PE\_{\\left( pos,2i\\right) }=sin(\\frac{pos}{10000^{\\frac{2i}{d} }} )$$ $$PE\_{\\left( pos,2i+1\\right) }=cos(\\frac{pos}{10000^{\\frac{2i}{d} }} )$$ These position encodings are then added to our embedding vectors before being passed to the encoder or decoder models. 💡 ****Note:** Many subsequent Transformer-based architectures have opted to inject positional information in different ways. One [promising approach](https://arxiv.org/abs/2108.12409?ref=jeremyjordan.me) (ALiBi) advocates for removing the positional encoding entirely from our embeddings and rather introducing an inductive bias to our relevance scores to give higher weight to nearby tokens. #### Overall architecture Putting all this together, we have the Transformer architecture. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/05/transformer-architecture.png) A visualization of the overall Transformer architecture from the original paper. 💡 ****Note**: Many subsequent Transformer-based architectures have opted to to move layer normalization to be performed **before* the attention/feed forward layers instead of **after*. This has led to [more stable training dynamics](https://arxiv.org/abs/2002.04745?ref=jeremyjordan.me). ## Benefits of an attention-only architecture One of the biggest benefits of replacing recurrence with attention is that we now have a fully **parallelizable** architecture. Whereas recurrence requires us to process a sequence one time-step at a time (due to the fact that we rely on the previous hidden state to process the next item in a sequence), an attention-only architecture does not have such a limitation. You can use a single matrix operation to project queries, keys, and values for every time-step in your sequence, simultaneously attending across the sequence for every time-step in parallel. This allows us to more efficiently leverage accelerated compute infrastructure (e.g. GPUs) and train models faster. The attention mechanism also allows our gradient signal to flow through the network more efficiently during backpropagation. With recurrent neural networks, if you needed propagate a gradient signal from $t=3$ back to $t=0$, you would have to back-propagate through all of the recurrent hidden layers between each of these time-steps. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/05/backprop-recurrent.png) However, with the attention mechanism we can directly back-propagate our gradient signal across an arbitrarily number of time steps using a constant number of operations. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/05/backprop-attention.png) These **shorter path lengths** make it easier for the model to learn long-term dependencies between items in our sequence. Finally, the attention mechanism has an inherent level of **interpretability** as a result of inspecting the attention weights. That is, we can use the computed attention weights to understand which tokens across the sequence were deemed relevant and used to produce a context vector for any given token in the sequence. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/05/attention-weights-visualization.png) An example from the Transformers paper showing how visualizing the attention weights can help aid in understanding how the model processes our input sequence. ## Resources **Papers** - [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473?ref=jeremyjordan.me) - [Effective Approaches to Attention-based Neural Machine Translation](https://arxiv.org/abs/1508.04025v5?ref=jeremyjordan.me) - [Attention Is All You Need](https://arxiv.org/abs/1706.03762?ref=jeremyjordan.me) 🐐 - [On Layer Normalization in the Transformer Architecture](https://arxiv.org/abs/2002.04745?ref=jeremyjordan.me) - [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://arxiv.org/abs/2108.12409?ref=jeremyjordan.me) (replace positional encoding) - The author gives a nice overview of the paper in [this talk](https://www.youtube.com/watch?v=Pp61ShI9VGc&ref=jeremyjordan.me) **Lectures** - 3Blue1Brown Series - [But what is a GPT? Visual intro to Transformers](https://www.youtube.com/watch?v=wjZofJX0v4M&ref=jeremyjordan.me) - [Visualizing Attention, a Transformer's Heart](https://www.youtube.com/watch?v=eMlx5fFNoYc&ref=jeremyjordan.me) - Sebastian Raschka's lectures on this topic - [L19.4.1 Using Attention Without the RNN: A Basic Form of Self-Attention](https://youtu.be/i%5FpfHD4P%5Fwg?ref=jeremyjordan.me) - [L19.4.2 Self-Attention and Scaled Dot-Product Attention](https://youtu.be/0PjHri8tc1c?ref=jeremyjordan.me) - [L19.4.3 Multi-Head Attention](https://youtu.be/A1eUVxscNq8?ref=jeremyjordan.me) - [L19.5.1 The Transformer Architecture](https://youtu.be/tstbZXNCfLY?ref=jeremyjordan.me) - Andrej Karpathy's deep dive + live coding - [Let's build GPT: from scratch, in code, spelled out](https://youtu.be/kCc8FmEb1nY?ref=jeremyjordan.me) **Blog posts** - [The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/?ref=jeremyjordan.me) - [The Annotated Transformer](http://nlp.seas.harvard.edu/annotated-transformer/?ref=jeremyjordan.me) - [Understanding and Coding the Self-Attention Mechanism of Large Language Models From Scratch](https://sebastianraschka.com/blog/2023/self-attention-from-scratch.html?ref=jeremyjordan.me) - [A walkthrough of transformer architecture code](https://github.com/markriedl/transformer-walkthrough?ref=jeremyjordan.me) **Other** - [Calculus (OpenStax) 12.3: The Dot Product](https://math.libretexts.org/Bookshelves/Calculus/Calculus%5F%28OpenStax%29/12%3A%5FVectors%5Fin%5FSpace/12.03%3A%5FThe%5FDot%5FProduct?ref=jeremyjordan.me) - [LLM Visualization](https://bbycroft.net/llm?ref=jeremyjordan.me) - This is a really impressive interactive visualization. *Thanks to James Black, Tim Hopper, and ChatGPT for pointing me to `ffmpeg` and helping me figure out the right settings to use to create GIFs from my diagrams. I hope you found the animations to be useful!* ### Understanding the attention mechanism in sequence models URL: https://www.jeremyjordan.me/attention/ Last updated: 2025-02-18T03:36:46.000Z In this blog post, we'll discuss a key innovation in sequence-to-sequence model architectures: the **attention mechanism**. This architecture innovation dramatically improved model performance for sequence-to-sequence tasks such as machine translation and text summarization. Moreover, the success of this attention mechanism led to the seminal paper, ["Attention Is All You Need"](https://arxiv.org/abs/1706.03762?ref=jeremyjordan.me), which introduced the Transformer model architecture. I'll discuss the [Transformer model architecture](https://www.jeremyjordan.me/transformer-architecture/) in subsequent posts, but I feel that it's useful to start by building a foundational understanding of the attention mechanism itself (which is the focus of this post). With that said, let's dive right in! ### Overview - [A quick refresher: sequence to sequence models](#seq2seq) - [The context vector bottleneck](#bottleneck) - [Introducing the attention mechanism](#mechanism) - [Training an attention model](#training) - [Benefits of attention: shorter path lengths](#benefits) - [Conclusion](#conclusion) - [Resources](#resources) ## A quick refresher: sequence to sequence models Let us consider the sequence modeling task where we have a variable-length input sequence and we're expected to predict a variable-length output sequence. A common example of this type of task is machine translation, where the input might be "i love you" (English) and the expected output might be "te amo" (Spanish). Notice how the input length (3 words) doesn't necessarily match the output length (2 words). This poses a challenging sequence prediction task since we can't simply train a model which learns to map each input token to its corresponding output token; there isn't a 1:1 mapping! [Cho et al](https://arxiv.org/abs/1406.1078?ref=jeremyjordan.me) introduced the RNN Encoder–Decoder neural network architecture for this type of sequence modeling where the variable-length input is encoded into a fixed-size vector (using a [recurrent neural network](https://www.jeremyjordan.me/introduction-to-recurrent-neural-networks/)) which is subsequently decoded into another variable-length output sequence (using a second recurrent neural network). ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/03/rnn-encoder-decoder-3.png) A visualization of the proposed architecture from [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://arxiv.org/abs/1406.1078?ref=jeremyjordan.me). [Sutskever et al](https://arxiv.org/abs/1409.3215?ref=jeremyjordan.me) introduced a similar architecture that builds on this approach using LSTM networks to encode and decode variable-length sequences. In this architecture, however, we simply take the last hidden state of the encoder and use that to initialize the decoder rather than passing it as context to every step during decoding. Also note how the input sequence is reversed in order to make the optimization problem easier, we'll dig deeper into this architecture choice in a later section. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/02/Screenshot-2023-02-20-at-9.18.23-PM.png) A visualization of the proposed architecture from [Sequence to Sequence Learning with Neural Networks](https://arxiv.org/abs/1409.3215?ref=jeremyjordan.me). While these two architectures had some slight differences, the overall approach was the same. We encode our variable-length input sequence into a fixed-size vector which provides a summary of the entire input sequence. This summary is then provided as context to the decoder when generating the output sequence. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/02/encoder-decoder-model-1.png) ## The context vector bottleneck Although the fixed-size context vector enables us to flexibly learn sequence-to-sequence tasks such as machine translation or text summarization, let's consider the implications of this architecture design. A fixed-size vector implies that there's a fixed amount of [information](https://en.wikipedia.org/wiki/Information%5Ftheory?ref=jeremyjordan.me) that we can store in our summary of the input sequence. For the sake of an example, let's assume that our context vector can store 100 bits of information. Let's also assume that each time-step in the input sequence contributes equally to the information stored in the context vector. For an input sequence of 5 tokens, each token would contribute 20 bits of information to the context vector. However, for an input sequence of 10 tokens, each token would only contribute 10 bits of information to the context vector. Through this example one can see how performance may degrade in sequence-to-sequence models for longer input sequences. As the size of the input grows, we can store less and less information per token in the input sequence. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/02/bottleneck-1.png) ## Introducing the attention mechanism In order to address this context bottleneck challenge, [Bahdanau et al](https://arxiv.org/abs/1409.0473?ref=jeremyjordan.me) introduced a neural network architecture which allows us to build a unique context vector for every decoder time-step based on different weighted aggregations across all of the encoder hidden states. Whereas in the previous hypothetical example we only had 100 bits of information to summarize the entire input sequence, with this new approach we can provide a hypothetical 100 bits of information about the input sequence at ***each step*** of the decoder model. In the example visualized below, you can see how we take a weighted combination across all of the encoder hidden states to produce our context vector, rather than simply using the final hidden state. These weights are determined by an **attention mechanism** which determines the encoder hidden states that are *relevant* given the current decoder hidden state. We'll discuss more about how this attention mechanism is trained in the next section. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/02/attention-1-1.png) Notice how the attention weights in the example above place most of the weight on the last token of the input sequence when generating the first token of the output sequence. If you recall our running example of translating "i love you" to Spanish, we're effectively aligning the related tokens of "i love **you**" and "**te** amo". [Bahdanau](https://arxiv.org/search/cs?searchtype=author&query=Bahdanau%2C+D&ref=jeremyjordan.me)[ et al](https://arxiv.org/abs/1409.0473?ref=jeremyjordan.me) refer to this attention mechanism as an ***alignment model*** since it allows the decoder to search across the input sequence for the *most relevant* time-steps regardless of the temporal position of the decoder model. When decoding the second token, you can see that we have an entirely different weighted combination of contexts from the encoder hidden states, which helps align the "**i love** you" with "te **amo**" (in case you don't speak Spanish, "amo" is the *first person singular* conjugation of the Spanish word for *love*). ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/02/attention-2-1.png) As the decoder model continues to generate the output sequence, we'll continue to compare the current decoder hidden state against all of our encoder hidden states to produce a unique context vector. ## Training an attention model Now that you understand how the attention weights are *used* to produce a unique context vector at each step in the decoder model, let's dive into the architecture and training process for our attention mechanism. As we discussed previously, the attention model is effectively comparing a given decoder hidden state with a specific encoder hidden state to determine the *relevance* between these two vectors. One way to compute this relevance score is to concatenate the encoder hidden state and decoder hidden state vectors and pass them through a multi-layer perception. The hidden layer of our MLP model can learn to identify relationships between certain dimensions in the encoder hidden state and the decoder hidden state. The output of this model is effectively answering the question: "how **relevant** is the information in this *encoder* hidden state for computing the next output in the sequence, given my current *decoder* hidden state?" ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/02/attention-model-1.png) As a concrete example, the figure below visualizes how the attention weights are computed for the first time-step in the decoder model. Notice how we compare the previous decoder hidden state to *every* encoder hidden state in our sequence. The attention model outputs a "relevance" score (**ei,j**) for each combination, which we can normalize via the softmax function to obtain our attention weights (**αi,j**). ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/03/computing-relevance.png) This attention mechanism is trained **jointly** with the rest of the sequence-to-sequence network, since we can backpropagate through the entire calculation of our attention weights and corresponding context vector. This means that the attention model is optimized to find relevant hidden states for the *specific* sequence-to-sequence task at hand. In other words, the attention mechanism for a machine translation model might learn to find different patterns than the attention mechanism for a text summarization model, depending on what's most useful for the task at hand. ## Benefits of attention: shorter path lengths In a previous section we touched on [Sutskever et al](https://arxiv.org/abs/1409.3215?ref=jeremyjordan.me)'s architecture decision to reverse the input sequence in order to make the optimization problem easier. In fact, they state "the simple trick of reversing the words in the source sentence is **one of the key technical contributions** of this work." Let's dig in to why this may be the case. Let us assume that, on average, the position of a token in the input sequence is roughly equivalent to its relevant position in the output sequence. For example, in the English to Spanish translation of "this is an example" to "esto es un ejemplo", the first word "this" of the input sequence can be translated directly to the first word "esto" of the output sequence. Recall that a standard encoder-decoder model architecture (without attention) uses the *last* hidden state of the encoder model as the context vector. This means that the information from the first time-step of the input sequence must be preserved in the hidden state passed through all of the time-steps in the input sequence. This information is then used to decode the first time-step of the output sequence. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/02/path-length-1-2.png) The path from an input token and it's corresponding output token in a standard encoder-decoder model is visualized in red. If we instead reverse the order of the input sequence, the information from the first time-step in the input has a shorter **path length** through the unrolled recurrent networks to the corresponding output. In general, *shorter path lengths are desired* because it's easier for gradient signals to propagate. Neural networks have historically struggled with learning longer-term dependencies (read: patterns across longer path lengths) due to the vanishing/exploding gradient problem coupled with the fact that we can only store a limited amount of information in our hidden state. In section 3.3 the authors state: > By reversing the words in the source sentence, **the average distance between corresponding words in the source and target language is unchanged**. However, the first few words in the source language are now very close to the first few words in the target language, so the problem’s minimal time lag is greatly reduced. Thus, **backpropagation has an easier time “establishing communication” between the source sentence and the target sentence**, which in turn results in substantially improved overall performance. – [Sequence to Sequence Learning with Neural Networks](https://arxiv.org/abs/1409.3215?ref=jeremyjordan.me) The trick of reversing the input sequence is effectively making it easier to pass information from the earlier parts of the input sequence while making it harder to pass information from the later parts of the input sequence. As a quick aside, this tradeoff makes sense when you consider the fact that our decoder model *autoregressively* generates the output vector. By making it easier for the model to start off with good initial output sequence predictions (by decreasing the path length to the initial input tokens), we're reducing the likelihood of introducing errors early on which would continue to accumulate as you generate the rest of the output sequence. While this does appear to make the optimization problem easier, reversing the input sequence doesn't have an effect on the average distance between related tokens in the input and output sequences. Let's contrast this with an encoder-decoder model architecture which leverages attention, where **every** time-step in the input sequence now has an equal path length to each time-step in the output sequence. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2023/02/shorter-path-lengths-2.png) Notice how the information from the encoder's first hidden state is **directly available** to the decoder; our encoder model no longer needs to remember all of the details from this first time step throughout the entire input sequence. Because the attention mechanism introduces shorter path lengths at every time step in the input sequence, **we no longer need to make a tradeoff** regarding which portions of the input sequence are most important for the decoder; the entire sequence is now easier to optimize as a result of shorter path lengths. ## Conclusion Standard encoder-decoder sequence models struggle from an information bottleneck passing information from the encoder to the decoder. The attention mechanism helps alleviate this bottleneck by allowing the decoder to search across the entire input sequence for information at each step during the output sequence generation. This is accomplished by introducing a small attention model which learns to compute a relevance score between an encoder hidden state and a decoder hidden state. From these relevance scores, we can produce a unique weighted combination of encoder hidden states as context for each step of the decoder model. ## Resources **Papers** - [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://arxiv.org/abs/1406.1078?ref=jeremyjordan.me) - [Sequence to Sequence Learning with Neural Networks](https://arxiv.org/abs/1409.3215?ref=jeremyjordan.me) - [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473?ref=jeremyjordan.me) **Lectures** - [Andrew Ng – Attention Model Intuition](https://www.youtube.com/watch?v=SysgYptB198&ref=jeremyjordan.me) - [Andrew Ng – Attention Model](https://www.youtube.com/watch?v=quoGRI-1l0A&ref=jeremyjordan.me) **Blog posts** - [Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention)](https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/?ref=jeremyjordan.me) ### Managing your machine learning infrastructure as code with Terraform URL: https://www.jeremyjordan.me/terraform/ Last updated: 2023-03-05T03:21:04.000Z Let's say you want to deploy a recommender system at your company. A [typical architecture](https://medium.com/nvidia-merlin/recommender-systems-not-just-recommender-models-485c161c755e?ref=jeremyjordan.me) might include a set of **inference servers** to run your embedding and ranking models, an **approximate nearest neighbor** **index** to select a set of candidate items that match your query, a **database** to retrieve features about your selected candidate items, and a **server** that runs some business logic and filters your set of candidate items. These servers, databases, and the networking configurations that allow them to communicate with each other comprise the **infrastructure** of your machine learning system. In this blog post, I'll provide an introduction to Terraform and how you can use it to provision the infrastructure layer of a machine learning system. ### Overview - [What is infrastructure as code?](#what) - [Benefits of infrastructure as code](#benefits) - [A conceptual introduction to Terraform](#concept) - [Typical workflow deploying infrastructure with Terraform](#workflow) - [Managing ML infrastructure with Terraform](#ml%5Finfrastructure) - [Best practices](#best%5Fpractices) - [Design guide for creating modules](#module%5Fdesign) - [Debugging](#debugging) - [Resources](#resources) ## What is infrastructure as code? Terraform and similar infrastructure-as-code tools ([CloudFormation](https://aws.amazon.com/cloudformation/?ref=jeremyjordan.me), [Pulumi](https://www.pulumi.com/?ref=jeremyjordan.me), etc.) provide a mechanism for **declaratively** defining your infrastructure via a set of configuration files. This allows us to focus on *what* infrastructure we want to deploy (e.g. three servers with a load balancer) without needing to worry about *how* to provision these resources (e.g. first spin up three servers, then create a load balancer, then attach the servers to the load balancer). These configuration files are stored in a version-controlled code repository which enables the same set of collaboration patterns that we already follow as part of the software development lifecycle. For example, if you wanted to propose changes to your team's infrastructure, you might create a new branch, update the relevant configuration files, and open a pull request to be reviewed by your team. ## Benefits of infrastructure as code When you first started working with cloud services, you probably just created whatever resources you needed directly in the AWS/GCP/Azure console. However, as the size of your organization or the number of projects grow, this becomes difficult to manage at scale. Infrastructure-as-code tools simplify your life in a number of ways: - **Easier to keep track of resources.** You don’t have to remember which resources you spun up for a given project, it’s all tracked in a centralized location. Spin up/down resources for a project with a single command. - **Create an audit log of changes to your infrastructure over time.** By tracking our desired infrastructure state *as code* in a git repository, we can easily see how our infrastructure has evolved and query for the desired infrastructure state at a specific point in time. - **Repeatable across multiple environments.** Creating reusable components helps ensure that your development environment accurately mirrors your production environment. - **Automation improves efficiency and reliability.** By codifying how we provision and manage infrastructure for our projects, we can reduce the potential for human error. ## A conceptual introduction to Terraform Let's first spend some time to discuss the common set of abstractions in Terraform's design that allows us to flexibly create infrastructure for whatever configuration we might need. Terraform communicates with a wide variety of infrastructure **providers** which are responsible for creating, updating, and deleting the set of resources that we use to deploy our applications. Terraform has providers for generic cloud infrastructure companies (e.g. AWS, GCP, Azure) as well as more focused, higher-level infrastructure services (e.g. Auth0, DataDog, Snowflake, and *many* more). If you happen to come across an infrastructure component that isn't already available via [Terraform's registry of providers](https://registry.terraform.io/browse/providers?ref=jeremyjordan.me), you can always [create your own provider](https://www.hashicorp.com/resources/creating-terraform-provider-for-anything?ref=jeremyjordan.me) as long as there's an API you can communicate with to create, read, update, and delete resources. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2022/06/provider.png) Terraform **providers** are versioned code which can communicate (via APIs) with external infrastructure providers. A **resource** is the smallest unit of infrastructure managed by a given provider; each provider will have a different set of documented resources supported for you to deploy. For example, using the AWS provider you can spin up an [EC2 machine](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/instance?ref=jeremyjordan.me) (server) or an [RDS database](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/db%5Finstance?ref=jeremyjordan.me). Using the Snowflake provider, you can configure your [data warehouse](https://registry.terraform.io/providers/Snowflake-Labs/snowflake/latest/docs/resources/warehouse?ref=jeremyjordan.me). These are just a few examples, each provider's documentation page will list the full set of available resources. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2022/06/resource_concept.png) Here, the gray boxes in the top row represent the actual resources that we've specified in our configuration. Terraform providers are responsible for reconciling the *actual state* in each of those environments to match the *desired state* specified in our configuration. Let's take a look at a minimal example of a Terraform configuration to see what this looks like in practice. ```text provider "aws" { region = "us-west-2" } resource "aws_instance" "my_example_server" { ami = "ami-005e54dee72cc1d00" instance_type = "t2.micro" tags = { name = "example" owner = "jeremy" } } ``` For a closer look at Terraform's syntax, check out: [Terraform configuration: quick reference](https://www.jeremyjordan.me/terraform-quick-reference). Here, we're using the AWS provider to configure a single EC2 instance. We could save this configuration as a file named `main.tf`, run `terraform apply`, and Terraform would get to work [reading in your AWS credentials](https://registry.terraform.io/providers/hashicorp/aws/latest/docs?ref=jeremyjordan.me#authentication-and-configuration) and communicating with AWS to spin up our requested instance. We'll revisit this process for actually spinning up/down resources in the next section, but there's a few more Terraform concepts that I want to cover first. The above example looks pretty simple, but it's likely that you'll need to provision more than a single EC2 instance for your infrastructure needs. *I mean, have you seen how many different services AWS puts in their [reference architecture diagrams](https://aws.amazon.com/architecture/reference-architecture-diagrams?ref=jeremyjordan.me)?* Terraform **modules** allow us to group together common resources, making it easier to reuse across projects or environments. A module is defined as a folder or directory containing one or more Terraform files (e.g. files with the `.tf` extension) which specify a set of providers and resources that we'd like to deploy. In order to facilitate module reuse, we can define **variables** to flexibly configure certain resource attributes and **outputs** to provide useful information that other resources/modules may need to know about. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2022/06/example-module.png) A minimal example of a Terraform module. As an example, suppose we want to standardize how servers are provisioned at our company. We can do this with Terraform by creating a `terraform-cloud-server` module which uses a (fictitious) `cloud` provider to create some example resources. We'll define two resource specifications: - a web server which *always uses Linux* as the operating system - a log stream which retains the server's log output for *90 days* ```text resource "cloud_server" "server" { operating_system = "linux" memory = var.server_memory cpus = var.server_cpus log_stream = cloud_log_stream.logs.id } resource "cloud_log_stream" "logs" { name = "${var.project}_log_stream" retention_in_days = 90 } ``` main.tf In this scenario, `operating_system` and `retention_in_days` will be consistent across all of the servers created using this module. However, we still may want to configure some values, such as the server memory, depending on the requirements of each individual server. For these values, we can create **variables** to pass configuration from the module level to individual resources within the module (as seen above). ```text variable "project" { description = "Provide a unique name for your project" type = string } variable "server_cpus" { description = "Number of CPUs requested for the server" type = number default = 2 } variable "server_memory" { description = "Amount of memory requested for the server" type = string default = "4 GiB" } ``` variables.tf And finally, we may want to know some information about the low-level resources that are created as part of this `terraform-cloud-server` module. We can expose this information by specifying **outputs**. ```text output "server_ip_address" { description = "IP address for the server" value = cloud_server.server.ip_address } ``` outputs.tf Putting this all together, our Terraform module would look something like: ```text ├── terraform-cloud-server ├── README.md ├── main.tf ├── outputs.tf └── variables.tf ``` If we wanted to use this module in our infrastructure configuration, we'd define a `module` block which specifies where our module is defined and provides all of the necessary variables to configure the underlying resources. ```text module "standard_server" { source = "../path/to/terraform-cloud-server" server_cpus = 4 server_memory = "8 GiB" } ``` In some cases, you may want to query information from a given provider when configuring a resource. For example, suppose we want to use an existing IP network (and its corresponding configuration) when assigning an IP address to our cloud server. A **data resource** allows you to query a given provider for information about *existing* components that might be useful when configuring new resources in your module. ```text data "vpc" "selected" { id = var.vpc_id } # e.g. use the data resource when configuring other resources resource "cloud_server" "server" { ... subnet = data.vpc.selected.private_subets[0] } ``` Putting this all together, we can see that our *infrastructure as code* is largely comprised of ***modules*** (reusable groups of resource definitions) and ***configurations*** (a specification of the resources/modules that we want to deploy in a given environment). ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2022/07/terraform_block.png) **Note**: the "modules" and "configurations" that I refer to above are both *technically* Terraform modules, since a module is just a directory of Terraform files, but I find it useful to draw a distinction between the two groups since they each have a different ***intention***. ## Typical workflow deploying infrastructure with Terraform Now that we've covered the basic concepts, let's walk through the standard workflow for ***deploying*** your infrastructure with Terraform. #### Specify your desired state The first step is to specify the collection of resources (and modules) that you want to provision. We do this by defining a series of configuration blocks in one or more Terraform files within the same directory. For more details on how to specify these configuration blocks, you can either review the [Terraform documentation](https://www.terraform.io/language?ref=jeremyjordan.me) (more likely to be up-to-date over time) or check out the [Terraform configuration: quick reference](https://www.jeremyjordan.me/terraform-quick-reference) that I put together. Building on the previous example using the (fictitious) `cloud` provider, a configuration for `project_a` might look something like: ```text module "database" { source = "../path/to/terraform-cloud-database-module" name = "project_a_database" cpu_limit = 4 memory_limit = "16 GiB" backup_daily = true } module "app_server" { source = "../path/to/terraform-cloud-app-server-module" replicas = 5 cpu_limit = 2 memory_limit = "1 GiB" container_image = "my_org/project_a:latest" } resource "cloud_domain_name" "domain" { name = "app.example.com" target = module.app_server.load_balancer.ip_address } ``` An example project configuration using fictitious modules and resources. **Note**: It's generally best to deploy your infrastructure as separate, logical components rather than one giant pile of Terraform configurations. This enables you to create/destroy resources for `project_a` without having any affect on `project_b`'s resources. ```text terraform/ configuration/ development/ staging/ production/ account_base/ main.tf # resources you only need to create once per environment region_a/ region_base/ main.tf # resources you only need to create once per region project_a/ main.tf # project-specific resources project_b/ main.tf # project-specific resources region_b/ ... ``` A typical organization pattern for Terraform configurations. #### Build an execution plan Once we've specified our desired infrastructure state, we'll run `terraform plan` from the same directory as our configuration files. Terraform will get to work communicating with the necessary providers in order to figure out the actual state of our infrastructure, how that differs from our desired state, and the set of changes we need to make in order to achieve our desired state. Terraform will output this plan to the console, sharing the set of resources will get created, modified in place, or deleted if we execute the plan. ```text $ terraform plan Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # module.database.cloud_database_cluster.cluster will be created + resource "cloud_database_cluster" "cluster" { + id = (known after apply) + name = "project_a_database" ... } ... Plan: 10 to add, 0 to change, 0 to destroy. ``` In order for Terraform to know: - which resources need to be created, - which resources already exist and may simply need updated, - and which resources are no longer necessary, Terraform maintains a **state file** which describes a mapping between resources defined in our configuration and resources that exist in the real world. This state file also contains other useful metadata such as resource dependencies, which helps Terraform know which *order* to safely destroy resources which are no longer needed. Most teams will configure Terraform to save this state file in a remote data store (e.g. Amazon S3) so that there's a single source of truth. This also allows us to set up state file **locking** (e.g. keeping track of locks in a DynamoDB table) so that multiple people can't be updating the same set of resources at the same time. #### Apply your execution plan Once you've reviewed the plan and confirmed that it looks correct, you can run `terraform apply` to execute the plan described in the previous step. Terraform will then perform the following steps: 1. Place a lock on the state file to prevent any other Terraform processes from modifying the same set of resources. 2. Create a new execution plan, unless you explicitly provide the output of a `terraform plan` that you ran previously. 3. Follow the above execution plan, leveraging the providers to create, update, and delete the appropriate resources in a safe order. 4. Update the state file to reflect changes made to our set of resources. 5. Remove the lock from the state file so that other Terraform processes can make changes when needed. #### Clean up resources you're no longer using Once we're ready to tear down the resources defined in your Terraform configuration, you can simply run `terraform destroy` from the same directory as our configuration files. Terraform will leverage its understanding of resource dependencies in order to delete the resources in a safe order. ## Managing ML infrastructure with Terraform Let's now walk through a standard machine learning model lifecycle and discuss how we might support the requisite infrastructure components with Terraform. > my generic mental model for the various components involved in operating a machine learning system. [pic.twitter.com/CtL9sAGd6R](https://t.co/CtL9sAGd6R?ref=jeremyjordan.me) > > — Jeremy Jordan (@jeremyjordan) [April 17, 2022](https://twitter.com/jeremyjordan/status/1515821782426607625?ref%5Fsrc=twsrc%5Etfw&ref=jeremyjordan.me) See this tweet as a high-level example of the various infrastructure components we may need to operationalize a machine learning model in our product. We'll need the ability to define a batch job which runs a training job for our model, so we'll go ahead and define a `terraform-aws-batch-job` module which can (1) be triggered either based on an event or a cron schedule, (2) spin up the necessary compute to train our model, (3) connect to the appropriate data sources for generating our training and validation datasets, and (4) spin down the necessary compute once our job completes. This training job will need to log key metrics and output the trained model artifacts into a model registry. We'll probably need to create a database and a cloud object store to save model metadata and artifacts respectively. Further, we might configure the model registry to automate a deployment rollout when a user updates a registered model's stage from shadow mode to production. This may make calls out to a service such as Sagemaker to [update the model artifact being served](https://mlflow.org/docs/latest/python%5Fapi/mlflow.sagemaker.html?ref=jeremyjordan.me#mlflow.sagemaker.SageMakerDeploymentClient.update%5Fdeployment) at a given endpoint. In this case, we may create some of the initial Sagemaker infrastructure using Terraform but mark certain attributes as `ignore_changes` in the `lifecycle` block (see more on this below) in order to allow our model registry to dynamically update the model configuration. ## Best practices #### Generate documentation for your Terraform modules In order to provide a succinct summary of the modules you create, you can use [terraform-docs](https://terraform-docs.io/user-guide/introduction/?ref=jeremyjordan.me) to automatically generate a description of a module's variables and outputs, in addition to the resources and modules it creates. ```text terraform-docs markdown table \ --output-file README.md \ --output-mode inject . ``` #### Tag your resources Some providers allow you to specify tags/labels for your resources as a method of organization. It's a good practice to leverage these tags to store information such as resource ownership, which projects a resource may belong to, how this resource should be categorized from a billing perspective, or other key information. ```text resource "resource_type" "resource_name" { # resource-specific configuration ... tags = { Owner = "my-team" Project = "my-project" Environment = "development" Billing = "some-cost-center" } } ``` #### Monitor for drift between your desired and actual infrastructure states Terraform only checks for discrepancies between the actual state and desired state of your infrastructure when you run commands. It's entirely possible that someone has made changes to your infrastructure outside of Terraform (e.g. manual actions in the AWS console), so the desired state that you track in a code repository isn't always an accurate depiction of what's deployed. It's usually a good practice to periodically check for "infrastructure drift"; you can use tools such as [driftctl](https://github.com/snyk/driftctl?ref=jeremyjordan.me) to automate this process. In order to bring Terraform configuration back in line with reality: - Run `terraform plan -refresh-only` to **compare** Terraform’s state file with the actual current infrastructure - Update the configuration files to match the current state (assuming you want to keep the resources which were created outside of Terraform) - Run `terraform import` to **update Terraform’s state file** and associate the updated configuration with the already-created infrastructure You can run `terraform apply -replace="resource_type.resource_name"` to target a specific resource to recreate in the event that someone manually changed its configuration. #### Safeguard critical resources from accidental downtime Some `terraform apply` plan executions can introduce the possibility of downtime, which may not always be ideal for resources that need to be highly available. The Terraform `lifecycle` block allows you to specify some degree of control over how resources can be updated. ```text resource "resource_type" "resource_name" { lifecycle { prevent_destroy = true # prevent accidental deletion of resources create_before_destroy = true # avoid downtime when replacing a resource ignore_changes = [attr] # ignore differences between config and state } } ``` ## Design guide for creating modules Modules are incredibly useful for deploying the same set of infrastructure components across projects, regions, or environments. The first rule of Terraform modules is that ***you should use them***. There's a wide variety of [ready-to-use Terraform modules](https://registry.terraform.io/browse/modules?ref=jeremyjordan.me) developed by the community; it's likely that a module already exists for some of your infrastructure needs. However, it's also likely that you'll end up needing to build some Terraform modules that are more specific to how your company provisions infrastructure. Here are some tips to keep in mind when building Terraform modules: - **Keep modules focused on a single purpose.** Modules should be used to group a set of resources that are always deployed together. - **Group resources together that change at a similar frequency.** By keeping short-lived resources separate from long-lived resources, we can reduce risk associated with making changes to our infrastructure. - **Consider the sensitivity of the infrastructure resources being grouped.** A networking module might have a different set of security implications as compared to an application server module. It's best to keep highly privileged resources separate so that we can design with the principle of least access in mind. - **Take a hierarchical approach when designing modules.** In order to get the most reuse out of your modules, it can be helpful to define a base set of modules which define lower-level infrastructure components which can then be referenced across a set of higher-level modules. ## Debugging There are a number of areas where things can go wrong when managing infrastructure with Terraform. Here are a few of those scenarios, as well as some quick tips for identifying and resolving issues. **You have syntax errors in your configuration files.** Run `terraform fmt` to surface syntax errors. You can also configure your IDE to run the formatter every time you save a file. **You may improperly configured some of your resources.** Run `terraform validate` to check your configuration in the context of the infrastructure provider. If you want to interactively test your configuration expressions, you can run `terraform console`. **Your state file has been corrupted.** Run `terraform state list [options] [address]` to inspect the state file for a given resource or module. Alternatively, you can run `terraform show` to view the entire state file. **Your state file has a stale lock.** Terraform uses state file locking in order to prevent multiple users from overwriting the same set of resources at the same time. For the S3 remote state configuration, you can configure a locking table with DynamoDB where Terraform will simply write some lock metadata to a specified key in your table (and remove it when the lock is released). If Terraform ends up in a weird state, you may need to manually remove this lock with `terraform force-unlock`. **There's a bug in a provider that you're using.** Capture logs from the providers when executing a plan or apply. ```text export TF_LOG_CORE=TRACE export TF_LOG_PROVIDER=TRACE export TF_LOG_PATH=logs.txt ``` ## Resources **Tutorials** - [Official Terraform tutorials](https://learn.hashicorp.com/terraform?ref=jeremyjordan.me) - There's a lot of great content here, the only problem is it's **a lot** which is why I decided it might be useful to write a condensed introduction to Terraform (this blog post). - [HashiCorp - Recommended Enterprise Patterns](https://learn.hashicorp.com/collections/terraform/recommended-patterns?ref=jeremyjordan.me) - [Purpose of Terraform State](https://www.terraform.io/language/state/purpose?ref=jeremyjordan.me) - [Troubleshoot Terraform](https://learn.hashicorp.com/tutorials/terraform/troubleshooting-workflow?ref=jeremyjordan.me) - [Terraform Best Practices: Key Concepts](https://www.terraform-best-practices.com/key-concepts?ref=jeremyjordan.me) **Blog posts** - [Terraform Import - Leaving Click Ops Behind](https://earthly.dev/blog/terraform-lambda/?ref=jeremyjordan.me) **Tools** - [Terraform Provider Iterative (TPI)](https://registry.terraform.io/providers/iterative/iterative/latest/docs?ref=jeremyjordan.me) - The team at iterative.ai has created a Terraform provider which makes it really easy to create the infrastructure needed for running ML tasks, including best practices by default such as using spot instances and spinning down resources once the task successfully finishes. [GitHub - jeremyjordan/terraform-blog-examples: Code snippets from my Terraform blog postCode snippets from my Terraform blog post. Contribute to jeremyjordan/terraform-blog-examples development by creating an account on GitHub.![](https://github.com/fluidicon.png)GitHubjeremyjordan![](https://opengraph.githubassets.com/ab09fcfee054352ab02889699efa51719f5875e047355ae8a09cfac0d3b4163f/jeremyjordan/terraform-blog-examples)](https://github.com/jeremyjordan/terraform-blog-examples?ref=jeremyjordan.me) Check out this repository containing the code examples from this blog post. ### Acknowledgements *Thanks to Vicki Boykis, Jérémie Vallée, John Huffman, and Adam Laiacano for reading early drafts of this blog post and providing helpful feedback.* ### Terraform configuration: quick reference URL: https://www.jeremyjordan.me/terraform-quick-reference/ Last updated: 2023-03-05T03:27:40.000Z This page contains a quick reference for writing Terraform configuration. For a conceptual introduction to Terraform and managing your infrastructure as code, read [this blog post](https://www.jeremyjordan.me/terraform). *Disclaimer: for the most up-to-date and detailed information, check out the official Terraform documentation.* [Overview - Configuration Language | Terraform by HashiCorpYou can use the Terraform language to write configuration files that tell Terraform how to manage a collection of infrastructure.![](https://www.terraform.io/favicon.ico)Terraform by HashiCorp![](https://www.terraform.io/img/og-image.png)](https://www.terraform.io/language?ref=jeremyjordan.me) *Jump to:* - [Configuration blocks](#configuration) - [Terraform](#terraform) - [Provider](#provider) - [Resource](#resource) - [Variable](#variable) - [Output](#output) - [Module](#module) - [Data](#data) - [Locals](#locals) - [Dynamic expressions](#expressions) - [Conditional statements](#conditionals) - [Wildcards](#wildcards) ## Configuration blocks #### Terraform Configuration block that allows you to specify various Terraform settings (such as required versions for Terraform and/or its providers). ```text terraform { required_providers { # e.g. include the AWS provider and require a specific major version aws = { source = "hashicorp/aws" version = ">= 4.0.0, < 5.0.0" } } required_version = ">= 1.2.3" } ``` #### Provider Versioned code which is used to communicate with an external infrastructure provider. ```text provider "provider_name" { # provider-specific configuration # e.g. what region should this provider create your resources in? } ``` #### Resource Smallest unit of infrastructure managed by a provider. ```text resource "resource_type" "resource_name" { # resource-specific configuration } ``` - The `resource_type` always starts with the prefix `{provider}_` - You can reference a given resource in other parts of your Terraform configuration using the format `{resource_type}.{resource_name}` - There's a small set of meta-arguments which you can use for any resource: - `count`: create multiple instances of the same resource specification - `for_each`: configure a set of resources based on a provider iterable - `provider`: override the default provider configuration (e.g. changing the region to deploy a specific resource) - `depends_on`: specify hidden dependencies between resources that Terraform isn't able to infer - `lifecycle`: configuration around resource lifecycle management (e.g. create a new version of the resource before destroying the old one) #### Variable Allow end users to specify values (to be used in resource creation) when defining a module. ```text variable "variable_name" { type = string description = "" # optional default = "" sensitive = true validation { condition = ... error_message = "" } } ``` - Usually defined in a separate file, `variables.tf`, for readability - These values are constant, they cannot be changed during planning or execution - You can reference a given variable as `var.{variable_name}` - Supported types: - simple (`string`, `bool`, `number`) - collection (`map`, `list`, `set`) - structural (`tuple`, `object`) - You can optionally add one or more validation conditions (e.g. to make sure a resource matches AWS resource naming requirements). This can be helpful for ["failing fast"](https://scalefactory.com/blog/2021/10/13/failing-faster-with-terraform/?ref=jeremyjordan.me) when you provide improper values. - By default, variables are **not** treated as sensitive. You can specify `sensitive = true` in order to prevent Terraform from outputing these values to the console. #### Output Expose a specific attribute of a resource within a module. ```text output "output_name" { description = "" value = resource_type.resource_name.attribute # optional sensitive = true precondition { condition = ... error_message = "" } } ``` - Usually defined in a separate file, `outputs.tf`, for readability - You can reference a module's output as `{module_name}.{output_name}` - You can optionally add one or more preconditions to check before returning an output value (e.g. to check that a certificate status is set to `ISSUED`). This can be used as a last line of defense to validate your assumptions before returning data about a given resource. - By default, variables are **not** treated as sensitive. You can specify `sensitive = true` in order to prevent Terraform from outputing these values to the console. #### Module A logical grouping of resources which can be configured and deployed together. ```text module "module_name" { source = "../path/to/module" # optional version = "" } ``` - Encourages reuse of configuration and provides consistency across your infrastructure stack - There’s a small set of universal “meta-arguments” that you can use for any module: - `count`: create multiple instances of the same module specification - `for_each`: configure a set of modules based on a provider iterable - `provider`: override the default provider configuration (e.g. changing the region to deploy the resources in your module) - `depends_on`: specify hidden dependencies between your module and other resources that Terraform isn't able to infer - Modules can be referenced from local filepaths or remote repositories (Terraform registry, Github, S3) In addition to using existing modules available on the Terraform registry, you can also define your own. A module can be defined as a set of Terraform configuration files within a single directory. ```text . ├── LICENSE ├── README.md ├── main.tf ├── variables.tf ├── outputs.tf ``` An example Terraform module definition. #### Data Retrieve information from an external infrastructure provider. ```text data "data_resource_type" "data_resource_name" { # parameters used to retrieve information } ``` #### Locals Used to provide a succinct or readable name for a Terraform expression which may be referenced multiple times. ```text locals { # e.g. define a set of required tags to add to all resources required_tags = { project = var.project_name, environment = var.environment } tags = merge(var.resource_tags, local.required_tags) # e.g. define a common suffix for use in multiple resource definitions name_suffix = "${var.project_name}-${var.environment}" } ``` - Can be referenced as `locals.{attribute}` - Dynamic expressions (e.g. using Terraform functions) are allowed, providing more flexibility than you have available in Terraform variables - It's recommended you use locals sparringly, keeping in mind the trade-offs between readability and DRY principles - In other words, don't make someone constantly scroll up to the `locals` block in order to figure out how a given resource is being configured ## Dynamic expressions #### Conditional statements You can express conditional statements in Terraform, although I find the syntax a bit odd. ```text attribute = (conditional_expression ? value_if_true : value_if_false) ``` Interpret `?` as “then” and `:` as “else” when reading. The `conditional_expression` itself is defined with common syntax: | Symbol | Meaning | | ------ | --------- | | != | not equal | | \== | equal | #### Wildcards You can use a splat `*` expression to reference a given attribute over a list of resources. ```text resource "aws_instance" "app_servers" { count = 5 ... } output "private_addresses" { value = aws_instance.app_servers[*].private_dns } ``` #### Comprehensions You can create lists and maps using syntax similar to list/dictionary comprehensions in Python. The general syntax is `[for item in iterable: value]` for a list and `{for item in iterable: key => value}` for a map. You can also add conditional statements, such as in the example below. ``` locals { tags = tomap({ foo = "123", bar = "456", empty = "" }) } value = { for key, value in local.tags : key => value if value != "" } ``` ### A simple solution for monitoring ML systems. URL: https://www.jeremyjordan.me/ml-monitoring/ Last updated: 2023-03-05T03:25:59.000Z This blog post aims to provide a simple, open-source solution for monitoring ML systems. We'll discuss industry-standard monitoring tools and practices for software systems and how they can be adapted to monitor ML systems. To illustrate this, we'll use a scikit-learn model trained on the [UCI Wine Quality dataset](https://archive.ics.uci.edu/ml/datasets/wine+quality?ref=jeremyjordan.me) and served via FastAPI (see Github repo [here](https://github.com/jeremyjordan/ml-monitoring?ref=jeremyjordan.me)). We'll collect metrics from the server using Prometheus and visualize the results in a Grafana dashboard. All of the services will be deployed on a Kubernetes cluster; if you're not familiar with Kubernetes, feel free to take a quick read through my [introduction to Kubernetes](https://www.jeremyjordan.me/kubernetes/) blog post. ### Overview - [Why is monitoring important?](#why-monitor) - [What should we be monitoring?](#what-monitor) - [Monitoring a wine quality prediction model: a case study.](#case-study) - [Deploying a model with FastAPI](#deploy) - [Instrumenting our model service with metrics](#instrument) - [Capturing metrics with Prometheus](#prometheus) - [Visualizing results in Grafana](#grafana) - [Simulating production traffic with Locust](#locust) - [Going beyond a simple monitoring solution](#advanced-monitoring) - [Purpose-built monitoring tools for ML models](#monitoring-products) - [Best practices for monitoring](#best-practices) - [Resources](#resources) ## Why is monitoring important? It's a well-accepted practice to monitor software systems so that we can understand performance characteristics, react quickly to system failures, and ensure that we're upholding our [Service Level Objectives](https://sre.google/sre-book/service-level-objectives/?ref=jeremyjordan.me). Monitoring systems can help give us confidence that our systems are running smoothly and, in the event of a system failure, can quickly provide appropriate context when diagnosing the root cause. When deploying machine learning models, we still have the same set of concerns discussed above. However, we'd also like to have confidence that our model is making **useful** predictions in production. There are many reasons why a model can fail to make useful predictions in production: - The underlying data distribution has shifted over time and the model has gone stale. - The production data stream contains edge cases (not seen during model development) where the model performs poorly. - The model was misconfigured in its production deployment. In all of these scenarios, the model could still make a "successful" prediction from a service perspective, but the predictions will likely not be useful. Monitoring our machine learning models can help us detect such scenarios and intervene (e.g. trigger a model retraining/deployment pipeline). ## What should we be monitoring? At a high level, there's three classes of metrics that we'll want to track and monitor. **Model metrics** - Prediction distributions - Feature distributions - Evaluation metrics (when ground truth is available) **System metrics** - Request throughput - Error rate - Request latencies - Request body size - Response body size **Resource metrics** - CPU utilization - Memory utilization - Network data transfer - Disk I/O ## Monitoring a wine quality prediction model: a case study. Throughout the rest of this blog post, we'll walk through the process of instrumenting and monitoring a scikit-learn model trained on the [UCI Wine Quality dataset](https://archive.ics.uci.edu/ml/datasets/wine+quality?ref=jeremyjordan.me). This model is trained to predict a wine's quality on the scale of 0 (lowest) to 10 (highest) based on a number of chemical attributes. At a high level, we'll: 1. Create a containerized REST service to expose the model via a prediction endpoint. 2. Instrument the server to collect metrics which are exposed via a separate metrics endpoint. 3. Deploy Prometheus to collect and store metrics. 4. Deploy Grafana to visualize the collected metrics. 5. Finally, we'll simulate production traffic using Locust so that we have some data to see in our dashboards. Feel free to clone [this Github repository](https://github.com/jeremyjordan/ml-monitoring?ref=jeremyjordan.me) and follow along yourself. All of the instructions to deploy these components on your own cluster are provided in the `README.md` file. ### Deploying a model with FastAPI If you look in the `model/` directory of the repo linked previously, you'll see a couple files. - `train.py` contains a simple script to produce a serialized model artifact. - `app/api.py` defines a few routes for our model service including a model prediction endpoint and a health-check endpoint. - `app/schemas.py` defines the expected schema for the request and response bodies in the model prediction endpoint. - `Dockerfile` lists the instructions to package our REST server as a container. We can deploy this server on our Kubernetes cluster using the manifest defined in `kubernetes/models/`. ### Instrumenting our service with metrics In order to monitor this service, we'll need to collect and expose metrics data. We'll go into more details in the subsequent section, but for now our goal is to capture "metrics" and expose this data via a `/metrics` endpoint on our server. For FastAPI servers, we can do this using `prometheus-fastapi-instrumentator`. This library includes FastAPI middleware that collects metrics for each request and exposes the metric data to a specified endpoint. For our example, we'll capture some of the metrics included in the library (request size, response size, latency, request count) as well as one custom-defined metric (our regression model's output). You can see this configuration defined in `model/app/monitoring.py`. After deploying our model service on the Kubernetes cluster, we can port forward to a pod running the server and check out the metrics endpoint running at [127.0.0.1:3000/metrics](http://127.0.0.1:3000/metrics?ref=jeremyjordan.me). ``` kubectl port-forward service/wine-quality-model-service 3000:80 ``` ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2021/01/metrics_endpoint.png) An example of the data typically found at a /metrics endpoint. **Note**: many of the framework-specific serving libraries offer the ability to expose a metrics endpoint out of the box. However, I'm not sure how you can define custom (model-specific) metrics to be logged using these serving platforms. - [PyTorch Serve](https://pytorch.org/serve/metrics%5Fapi.html?ref=jeremyjordan.me) - [Tensorflow Serving](https://www.tensorflow.org/tfx/serving/serving%5Fconfig?ref=jeremyjordan.me#monitoring%5Fconfiguration) - [NVIDIA Triton Inference Server](https://github.com/triton-inference-server/server/blob/r20.12/docs/metrics.md?ref=jeremyjordan.me) ### Capturing metrics with Prometheus After exposing our metrics at a specified endpoint, we can use Prometheus to collect and store this metric data. We'll deploy Prometheus onto our Kubernetes cluster using `helm`, see the **Setup** section in the `README.md` file for full instructions. Prometheus is an open-source monitoring service with a focus on reliability. It is responsible for collecting metrics data from our service endpoints and efficiently storing this data to be later queried. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2021/01/prometheus_architecture.jpg) Prometheus refers to endpoints containing metric data as **targets** which can be discovered either through service discovery or static configuration. In our example, we'll use *service discovery* to enable Prometheus to discover which targets should be scraped. We can do this by creating a `ServiceMonitor` resource for our wine quality prediction service. This resource specification is included in the `kubernetes/models/wine_quality.yaml` manifest. This resource must be defined in the same namespace that Prometheus is running in. You can see all of the services configured to be discovered by Prometheus by running: ``` kubectl get servicemonitor -n monitoring ``` You'll notice that in addition to collecting metrics from our wine quality prediction service, Prometheus has already been configured to collect metrics from the Kubernetes cluster itself. Prometheus will scrape the metrics data at each of these endpoints at a specified interval (every 15 seconds by default). There are four supported data types for metrics. - **Counter**: a single value that monotonically increases over time (e.g. for counting number of requests made) - **Gauge**: a single value that can increase or decrease over time (e.g. for tracking current memory utilization) - **Summary**: a collection of values aggregated by `count` and `sum` (e.g. for calculating average request size) - **Histogram**: a collection of values aggregated into buckets (e.g. for tracking request latency) You can read more about these metric types [here](https://prometheus.io/docs/concepts/metric%5Ftypes/?ref=jeremyjordan.me). Each metric has a name and an optional set of labels (key-value pairs) to describe the observed value. These metrics are designed such that they can be aggregated at different timescales for [more efficient storage](https://prometheus.io/docs/prometheus/latest/storage/?ref=jeremyjordan.me#compaction). This metric data can be queried by other services (such as a Grafana) which make requests to the HTTP server. **Note**: Prometheus uses a PULL mechanism for collecting metrics, where services simply expose the metrics data at an endpoint and Prometheus collects the data. Some other monitoring services (e.g. AWS CloudWatch) use a PUSH mechanism where each service collects and sends its own metrics to the monitoring server. You can read about the tradeoffs for each approach [here](https://giedrius.blog/2019/05/11/push-vs-pull-in-monitoring-systems/?ref=jeremyjordan.me). ### Visualizing results in Grafana Now that our metrics are being collected and stored in Prometheus, we need a way to visualize the data. Grafana is often paired with Prometheus to provide the ability to create dashboards from the metric data. In fact, our `helm` install of the Prometheus stack included Grafana out of the box. You can check out the Grafana dashboard by port-forwarding to the service and visiting [127.0.0.1:8000](http://127.0.0.1:8000/?ref=jeremyjordan.me). ``` kubectl port-forward service/prometheus-stack-grafana 8000:80 -n monitoring ``` We can create visualizations by making queries to the Prometheus data source. Prometheus uses a query language called [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/?ref=jeremyjordan.me). Admittedly, it can take some time to get used to this query language. After reading through the official documentation, I'd recommend watching [PromQL for Mere Mortals](https://www.youtube.com/watch?v=hTjHuoWxsks&ref=jeremyjordan.me) to get a better understanding of the query language. The repository for this blog post contains a pre-built dashboard (see `dashboards/model.json`) for the wine quality prediction service which you can import and play around with. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2021/01/dashboard.png) ### Simulating production traffic with Locust At this point, we have a model deployed as a REST service which is instrumented to export metric data. This metric data is being collected by Prometheus and we can visualize the results in a Grafana dashboard. However, in order to see something interesting in the dashboard we'll want to simulate production traffic. We'll use `locust`, a Python load testing framework, to make requests to our model service and simulate production traffic. This behavior is defined in `load_tests/locustfile.py` where we define three tasks: - make a request to our health check endpoint - choose a random example from the wine quality dataset and make a request to our prediction service - choose a random example from the wine quality dataset, corrupt the data, and make a bad request to our prediction service The manifests to deploy this load test to our cluster can be found in `kubernetes/load_tests/`. ## Going beyond a simple monitoring solution In the wine quality prediction model case study, we only tracked a single model metric: the prediction distributions. However, as I mentioned previously there's more model-related metrics that we may want to track such as feature distributions and evaluation metrics. These remaining concerns introduce a few new challenges: - The Prometheus time series database was designed to store metric data, not features for ML models. This is not the right technology choice to store feature data for tracking their fluctuations over time. - Model evaluation metrics require a feedback signal containing the ground truth which is not available at inference time. When it comes to logging feature distributions, there's a range of approaches you can take. The simplest approach for monitoring feature distributions might be to deploy a **drift-detection service** alongside the model. This service would fire (and increase a counter metric) when feature drift is detected. For additional visibility into your production data, you can log a **statistical profile** of the features observed in production. This allows for a compact summary of data observed in production and can help you identify when data distributions are shifting. The information logged will vary depending on the type of data: - For *tabular* data, you can directly log features used through statistical measures such as histograms and value counts. - For *text* data, you can log metadata about the text such as character length, count of numeric characters, count of alphabetic characters, count of special characters, average word length, etc. - For *image* data, you can log metadata about the image such as average pixel value per channel, image width, image height, aspect ratio, etc. For full visibility on your production data stream, you can **log the full feature payload**. This is not only useful for monitoring purposes, but also in data collection for labeling and future model training. If your deployment stack already supports saving production data, you can leverage this data source (typically a data lake) for monitoring whether or not the production data distribution is drifting away from the training data distribution. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2021/01/model_drift_evaluation_workflow-1024x281.png) An example architecture provided by [Databricks](https://databricks.com/blog/2019/09/18/productionizing-machine-learning-from-deployment-to-drift-detection.html?ref=jeremyjordan.me) for saving and monitoring production data. In order to monitor model evaluation metrics, we'll need to persist model prediction results alongside a unique identifier which is included in the model inference response. This unique identifier will be used when we **asynchronously receive feedback** from the user or some upstream system regarding a prediction we made. Once we've received feedback we can calculate our evaluate metric and log the results via Prometheus to be displayed in the model dashboard. ### Purpose-built monitoring tools for ML models If you're looking for out-of-the-box support for some of these more advanced model monitoring use cases, there's a growing number of ML monitoring services that you can use. **WhyLogs** is an [open-source library](https://whylogs.readthedocs.io/en/latest/?ref=jeremyjordan.me) for capturing statistical profiles of production data streams. **Seldon** has developed an [open-source library](https://docs.seldon.io/projects/seldon-core/en/latest/?ref=jeremyjordan.me) which runs on Kubernetes and uses a similar technology stack (Prometheus, Grafana, Elasticsearch) as discussed in this blog post. **Fiddler** offers a slick [monitoring](https://www.fiddler.ai/ml-monitoring?ref=jeremyjordan.me) product with a dashboard built specifically for monitoring models and understanding predictions on production data. *Check out [this pull request](https://github.com/jeremyjordan/ml-monitoring/pull/1?ref=jeremyjordan.me) to see how I integrated Fiddler with the wine quality prediction model from this blog post.* **Amazon Sagemaker** offers a [model monitoring suite](https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor.html?ref=jeremyjordan.me) which integrates well if you're already deploying models on Sagemaker. ## Best practices for monitoring **Prometheus** - Avoid storing high-cardinality data in labels.Every unique set of labels for is treated as a distinct time series, high-cardinality data in labels can drastically increase the amount of data being stored. As a general rule, try to keep the cardinality for a given metric (number of unique label-sets) under 10. - Metric names should have a suffix describing the unit (e.g. `http_request_duration_****seconds**`) - Use base units when recording values (e.g. seconds instead of milliseconds). - Use standard [Prometheus exporters](https://prometheus.io/docs/instrumenting/exporters/?ref=jeremyjordan.me) when available. **Grafana** - Ensure your dashboards are easily discoverable and consistent by design. - Use template variables instead of hardcoding values or duplicating charts. - Provide appropriate context next to important charts. - Keep your dashboards in source control. - Avoid duplicating dashboards. ## Resources **Blog posts** - [Production Machine Learning Monitoring: Outliers, Drift, Explainers & Statistical Performance](https://towardsdatascience.com/production-machine-learning-monitoring-outliers-drift-explainers-statistical-performance-d9b1d02ac158?ref=jeremyjordan.me) - [Monitoring Machine Learning Models in Production: A Comprehensive Guide](https://christophergs.com/machine%20learning/2020/03/14/how-to-monitor-machine-learning-models/?ref=jeremyjordan.me) - [whylogs: Embrace Data Logging Across Your ML Systems](https://medium.com/whylabs/whylogs-embrace-data-logging-a9449cd121d?ref=jeremyjordan.me) - [Building dashboards for operational visibility](https://aws.amazon.com/builders-library/building-dashboards-for-operational-visibility/?ref=jeremyjordan.me) - [How to Make Dashboards Using a Product Thinking Approach](https://shopify.engineering/make-dashboards-using-product-thinking-approach?ref=jeremyjordan.me) - [How does a Prometheus Histogram work?](https://www.robustperception.io/how-does-a-prometheus-histogram-work?ref=jeremyjordan.me) **Talks** - [Fool-Proof Kubernetes Dashboards for Sleep-Deprived Oncalls - David Kaltschmidt](https://www.youtube.com/watch?v=YE2aQFiMGfY&ref=jeremyjordan.me) - [How Prometheus Monitoring works | Prometheus Architecture explained](https://www.youtube.com/watch?v=h4Sl21AKiDg&ref=jeremyjordan.me) - [PromQL for Mere Mortals](https://www.youtube.com/watch?v=hTjHuoWxsks&ref=jeremyjordan.me) ### Acknowledgements *Thanks to Goku Mohandas, John Huffman, Shreya Shankar, and Binal Patel for reading early drafts of this blog post and providing feedback.* ### Effective testing for machine learning systems. URL: https://www.jeremyjordan.me/testing-ml/ Last updated: 2025-02-28T03:08:51.000Z *Working as a core maintainer for [PyTorch Lightning](https://github.com/PyTorchLightning/pytorch-lightning?ref=jeremyjordan.me), I've grown a strong appreciation for the value of tests in software development. As I've been spinning up a new project at work, I've been spending a fair amount of time thinking about how we should test machine learning systems. A couple weeks ago, one of my coworkers sent me [a fascinating paper](https://homes.cs.washington.edu/~marcotcr/acl20%5Fchecklist.pdf?ref=jeremyjordan.me) on the topic which inspired me to dig in, collect my thoughts, and write this blog post.* In this blog post, we'll cover what testing looks like for traditional software development, why testing machine learning systems can be different, and discuss some strategies for writing effective tests for machine learning systems. We'll also clarify the distinction between the closely related roles of evaluation and testing as part of the model development process. By the end of this blog post, I hope you're convinced of both the extra work required to effectively test machine learning systems and the value of doing such work. ## What's different about testing machine learning systems? In traditional software systems, humans write the logic which interacts with data to produce a desired behavior. Our software tests help ensure that this **written logic** aligns with the actual expected behavior. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2020/08/Screen-Shot-2020-08-09-at-4.45.29-PM.png) However, in machine learning systems, humans provide desired behavior as examples during training and the model optimization process produces the logic of the system. How do we ensure this **learned logic** is going to consistently produce our desired behavior? ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2020/08/Screen-Shot-2020-08-09-at-4.47.06-PM.png) Let's start by looking at the best practices for testing traditional software systems and developing high-quality software. A typical software testing suite will include: - **unit tests** which operate on atomic pieces of the codebase and can be run quickly during development, - **regression tests** replicate bugs that we've previously encountered and fixed, - **integration tests** which are typically longer-running tests that observe higher-level behaviors that leverage multiple components in the codebase, and follow conventions such as: - don't merge code unless all tests are passing, - always write tests for newly introduced logic when contributing code, - when contributing a bug fix, be sure to write a test to capture the bug and prevent future regressions. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2020/08/Group-5-1.jpg) A typical workflow for software development. When we run our testing suite against the new code, we'll get a report of the specific behaviors that we've written tests around and verify that our code changes don't affect the expected behavior of the system. If a test fails, we'll know which specific behavior is no longer aligned with our expected output. We can also look at this testing report to get an understanding of how extensive our tests are by looking at metrics such as **[code coverage](https://en.wikipedia.org/wiki/Code%5Fcoverage?ref=jeremyjordan.me)**. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2020/08/Group-1.png) An example output from a traditional software testing suite. Let's contrast this with a typical workflow for developing machine learning systems. After training a new model, we'll typically produce an evaluation report including: - performance of an established metric on a validation dataset, - plots such as precision-recall curves, - operational statistics such as inference speed, - examples where the model was most confidently incorrect, and follow conventions such as: - save all of the hyper-parameters used to train the model, - only promote models which offer an improvement over the existing model (or baseline) when evaluated on the same dataset. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2020/08/Group-3-1.png) A typical workflow for model development. When reviewing a new machine learning model, we'll inspect metrics and plots which summarize model performance over a validation dataset. We're able to compare performance between multiple models and make relative judgements, but we're not immediately able to characterize specific model behaviors. For example, figuring out *where* the model is failing usually requires additional investigative work; one common practice here is to look through a list of the top most egregious model errors on the validation dataset and manually categorize these failure modes. Assuming we write behavioral tests for our models (discussed below), there's also the question of whether or not we have enough tests! While traditional software tests have metrics such as the lines of code covered when running tests, this becomes harder to quantify when you shift your application logic from lines of code to parameters of a machine learning model. Do we want to quantify our test coverage with respect to the input data distribution? Or perhaps the possible activations inside the model? [Odena et al](http://proceedings.mlr.press/v97/odena19a/odena19a.pdf?ref=jeremyjordan.me). introduce one possible metric for coverage where we track the model logits for all of the test examples and quantify the area covered by radial neighborhoods around these activation vectors. However, my perception is that as an industry we don't have a well-established convention here. In fact, it feels like that testing for machine learning systems is in such early days that this question of test coverage isn't really being asked by many people. ## What's the difference between model testing and model evaluation? While reporting evaluation metrics is certainly a good practice for quality assurance during model development, I don't think it's sufficient. Without a granular report of specific behaviors, we won't be able to immediately understand the nuance of how behavior may change if we switch over to the new model. Additionally, we won't be able to track (and prevent) behavioral regressions for specific failure modes that had been previously addressed. This can be especially dangerous for machine learning systems since often times failures happen silently. For example, you might improve the overall evaluation metric but introduce a regression on a critical subset of data. Or you could unknowingly add a gender bias to the model through the inclusion of a new dataset during training. We need more nuanced reports of model behavior to identify such cases, which is exactly where model testing can help. For machine learning systems, we should be running model evaluation and model tests in parallel. - **Model evaluation** covers metrics and plots which summarize performance on a validation or test dataset. - **Model testing** involves explicit checks for behaviors that we expect our model to follow. Both of these perspectives are instrumental in building high-quality models. In practice, most people are doing a combination of the two where evaluation metrics are calculated automatically and some level of model "testing" is done [manually through error analysis](https://www.coursera.org/learn/machine-learning-projects/lecture/GwViP/carrying-out-error-analysis?ref=jeremyjordan.me) (i.e. classifying failure modes). Developing model tests for machine learning systems can offer a systematic approach towards error analysis. ## How do you write model tests? In my opinion, there's two general classes of model tests that we'll want to write. - **Pre-train** **tests** allow us to identify some bugs early on and short-circuit a training job. - **Post-train tests** use the trained model artifact to inspect behaviors for a variety of important scenarios that we define. ### Pre-train tests There's some tests that we can run without needing trained parameters. These tests include: - check the shape of your model output and ensure it aligns with the labels in your dataset - check the output ranges and ensure it aligns with our expectations (eg. the output of a classification model should be a distribution with class probabilities that sum to 1) - make sure a single gradient step on a batch of data yields a decrease in your loss - make [assertions about your datasets](https://greatexpectations.io/?ref=jeremyjordan.me) - check for label leakage between your training and validation datasets The main goal here is to identify some errors early so we can avoid a wasted training job. ### Post-train tests However, in order for us to be able to understand model behaviors we'll need to test against trained model artifacts. These tests aim to **interrogate the logic learned during training** and provide us with a behavioral report of model performance. > **Paper highlight**: [Beyond Accuracy: Behavioral Testing of NLP Models with CheckList](https://homes.cs.washington.edu/~marcotcr/acl20%5Fchecklist.pdf?ref=jeremyjordan.me) The authors of the above paper present three different types of model tests that we can use to understand behavioral attributes. **Invariance Tests** Invariance tests allow us to describe a set of perturbations we should be able to make to the input *without* affecting the model's output. We can use these perturbations to produce pairs of input examples (original and perturbed) and **check for consistency** in the model predictions. This is closely related to the concept of data augmentation, where we apply perturbations to inputs during training and preserve the original label. For example, imagine running a sentiment analysis model on the following two sentences: - Mark was a great instructor. - Samantha was a great instructor. We would expect that simply changing the name of the subject doesn't affect the model predictions. **Directional Expectation Tests** Directional expectation tests, on the other hand, allow us to define a set of perturbations to the input which *should* have a *predictable* effect on the model output. For example, if we had a housing price prediction model we might assert: - Increasing the number of bathrooms (holding all other features constant) should not cause a drop in price. - Lowering the square footage of the house (holding all other features constant) should not cause an increase in price. Let's consider a scenario where a model fails the second test - taking a random row from our validation dataset and decreasing the feature `house_sq_ft` yields a higher predicted price than the original label. This is surprising as it doesn't match our intuition, so we decide to look further into it. We realize that, without having a feature for the house's neighborhood/location, our model has learned that smaller units tend to be more expensive; this is due to the fact that smaller units from our dataset are more prevalent in cities where prices are generally higher. In this case, the *selection* of our dataset has influenced the model's logic in unintended ways - this isn't something we would have been able to identify simply by examining performance on a validation dataset. **Minimum Functionality Tests (aka data unit tests)** Just as software unit tests aim to isolate and test atomic components in your codebase, data unit tests allow us to quantify model performance for specific cases found in your data. This allows you to identify critical scenarios where prediction errors lead to high consequences. You may also decide to write data unit tests for failure modes that you uncover during error analysis; this allows you to "automate" searching for such errors in future models. Snorkel also has introduced a very similar approach through their concept of [slicing functions](https://www.snorkel.org/use-cases/03-spam-data-slicing-tutorial?ref=jeremyjordan.me). These are programatic functions which allow us to identify subsets of a dataset which meet certain criteria. For example, you might write a slicing function to identify sentences less than 5 words to evaluate how the model performs on short pieces of text. ### Organizing tests In traditional software tests, we typically organize our tests to mirror the structure of the code repository. However, this approach doesn't translate well to machine learning models since our logic is structured by the parameters of the model. The authors of the CheckList paper linked above recommend structuring your tests around the "skills" we expect the model to acquire while learning to perform a given task. For example, a sentiment analysis model might be expected to gain some understanding of: - vocabulary and parts of speech, - robustness to noise, - identifying named entities, - temporal relationships, - and negation of words. For an image recognition model, we might expect the model to learn concepts such as: - object rotation, - partial occlusion, - perspective shift, - lighting conditions, - weather artifacts (rain, snow, fog), - and camera artifacts (ISO noise, motion blur). ### Model development pipeline Putting this all together, we can revise our diagram of the model development process to include pre-train and post-train tests. These tests outputs can be displayed alongside model evaluation reports for review during the last step in the pipeline. Depending on the nature of your model training, you may choose to automatically approve models provided that they meet some specified criteria. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2020/08/Group-7.png) A proposed workflow for developing high-quality models. ## Conclusion Machine learning systems are trickier to test due to the fact that we're not explicitly writing the logic of the system. However, automated testing is still an important tool for the development of high-quality software systems. These tests can provide us with a behavioral report of trained models, which can serve as a systematic approach towards error analysis. Throughout this blog post, I've presented "traditional software development" and "machine learning model development" as two separate concepts. This simplification made it easier to discuss the unique challenges associated with testing machine learning systems; unfortunately, the real world is messier. Developing machine learning models also relies on a large amount of "traditional software development" in order to process data inputs, create feature representations, perform data augmentation, orchestrate model training, expose interfaces to external systems, and much more. Thus, effective testing for machine learning systems requires **both** a traditional software testing suite (for model development infrastructure) and a model testing suite (for trained models). If you have experience testing machine learning systems, please reach out and share what you've learned! *Thank you, Xinxin Wu, for sending me the paper which inspired me to write this post! Additionally, I'd like to thank John Huffman, [Josh Tobin](http://josh-tobin.com/?ref=jeremyjordan.me), and [Andrew Knight](https://automationpanda.com/?ref=jeremyjordan.me) for reading earlier drafts of this post and providing helpful feedback.* ## Further reading **Papers** - [Beyond Accuracy: Behavioral Testing of NLP Models with CheckList](https://homes.cs.washington.edu/~marcotcr/acl20%5Fchecklist.pdf?ref=jeremyjordan.me) - [TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing](http://proceedings.mlr.press/v97/odena19a/odena19a.pdf?ref=jeremyjordan.me) **Blog posts** - [How to Test Machine Learning Code and Systems](https://eugeneyan.com/writing/testing-ml/?ref=jeremyjordan.me) by Eugene Yan - [Snorkel Intro Tutorial: **Data Slicing*](https://www.snorkel.org/use-cases/03-spam-data-slicing-tutorial?ref=jeremyjordan.me) - [How to Trust Your Deep Learning Code](https://krokotsch.eu/cleancode/2020/08/11/Unit-Tests-for-Deep-Learning.html?ref=jeremyjordan.me) by Tilman Krokotsch **Talks** - [MLOps Chat: How Should We Test ML Models? with Data Scientist Jeremy Jordan](https://www.youtube.com/watch?v=k0naEYedv5I&feature=youtu.be&ref=jeremyjordan.me) - [Unit Testing for Data Scientists - Hanna Torrence](https://www.youtube.com/watch?v=Da-FL%5F1i6ps&ref=jeremyjordan.me) - [Trey Causey: Testing for Data Scientists](https://www.youtube.com/watch?v=GEqM9uJi64Q&ref=jeremyjordan.me) - [Bay Area NLP Meetup: Beyond Accuracy Behavioral Testing of NLP Models with CheckList](https://www.youtube.com/watch?v=x7lhb7ASyu0&feature=emb%5Ftitle&ref=jeremyjordan.me) **Code** - [CheckList](https://github.com/marcotcr/checklist?ref=jeremyjordan.me) - [Great Expectations](https://github.com/great-expectations/great%5Fexpectations?ref=jeremyjordan.me) - [Chex](https://github.com/deepmind/chex?ref=jeremyjordan.me) (testing library for Jax) ### An introduction to Kubernetes. URL: https://www.jeremyjordan.me/kubernetes/ Last updated: 2023-03-05T03:26:44.000Z This blog post will provide an introduction to Kubernetes so that you can understand the motivation behind the tool, what it is, and how you can use it. In a follow-up post, I'll discuss how we can leverage Kubernetes to power data science workloads using more concrete (data science) examples. However, it helps to first build an understanding of the fundamentals - which is the focus of this post. **Prerequisites**: I'm going to make the assumption that you're familiar with container technologies such as Docker. If you don't have experience building and running container images, I suggest starting [here](https://unsupervisedpandas.com/data-science/docker-for-data-science/?ref=jeremyjordan.me) before you continue reading this post. ## Overview *Here's what we'll discuss in this post. You can click on any top-level heading to jump directly to that section.* - [What is the point of Kubernetes?](#objective) - [Design principles.](#design) - Declaritive - Distributed - Decoupled - Immutable - [Basic objects in Kubernetes.](#objects) - Pod - Deployment - Service - Ingress - Job - [How? The Kubernetes control plane.](#control-plane) - Master node - API server - etcd - scheduler - controller-manager - Worker nodes - kubelet - kube-proxy - [When should you not use Kubernetes?](#nah) - [Resources](#resources) --- ## What is the point of Kubernetes? Kubernetes is often described as a **container orchestration** platform. In order to understand what exactly that means, it helps to revisit the purpose of containers, what's missing, and how Kubernetes fills that gap. > Note: You will also see Kubernetes referred to by its [numeronym](https://en.wikipedia.org/wiki/Numeronym?ref=jeremyjordan.me), **k8s**. It means the same thing, just easier to type. **Why do we love containers?** Containers provide a lightweight mechanism for isolating an application's environment. For a given application, we can specify the system configuration and libraries we want installed without worrying about creating conflicts with other applications that might be running on the same physical machine. We encapsulate each application as a *container image* which can be executed reliably on any machine\* (as long as it has the ability to run container images), providing us the portability to enable smooth transitions from development to deployment. Additionally, because each application is self-contained without the concern of environment conflicts, it's easier to place multiple workloads on the same physical machine and achieve higher resource (memory and CPU) utilization - ultimately lowering costs. **What's missing?** However, what happens if your container dies? Or even worse, what happens if the machine running your container fails? Containers do not provide a solution for *fault tolerance*. Or what if you have multiple containers that need the ability to communicate, how do you enable networking between containers? How does this change as you spin up and down individual containers? Container *networking* can easily become an entangled mess. Lastly, suppose your production environment consists of multiple machines - how do you decide which machine to use to run your container? **Kubernetes as a container orchestration platform.** We can address many of the concerns mentioned above using a container orchestration platform. > The director of an orchestra holds the **vision** for a musical performance and **communicates** with the musicians in order to **coordinate** their individual instrumental contributions to achieve this overall vision. As the architect of a system, your job is simply to **compose the music** (specify the containers to be run) and then hand over control to the orchestra director (container orchestration platform) to achieve that vision. ![orchestration](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/11/orchestration.gif) [Image credit](https://giphy.com/gifs/kennedycenter-nationalsymphonyorchestra-noseda-nso-classicalmusic-orchestra-3o8dFCeY6O5NyLVDYQ?ref=jeremyjordan.me) A container orchestration platform manages the entire lifecycle of individual containers, spinning up and shutting down resources as needed. If a container shuts down unexpectedly, the orchestration platform will react by launching another container in its place. On top of this, the orchestration platform provides a mechanism for applications to communicate with each other even as underlying individual containers are created and destroyed. Lastly, given (1) a set of container workloads to run and (2) a set of machines on a cluster, the container orchestrator examines each container and determines the optimal machine to schedule that workload. To understand why this can be valuable, [watch Kelsey Hightower explain](https://youtu.be/u%5FiAXzy3xBA?t=1067&ref=jeremyjordan.me) (17:47-20:55) the difference between automated deployments and container orchestration using an example game of Tetris. --- ## Design principles. Now that we understand the motivation for container orchestration in general, let's spend some time to discuss the motivating design principles behind Kubernetes. It helps to understand these principles so that you can use the tool *as it was intended to be used*. ### Declarative Perhaps the most important design principle in Kubernetes is that we simply define the *desired state* of our system and let Kubernetes automation work to ensure that the *actual state* of the system reflects these desires. This absolves you of the responsibility of fixing most things when they break; you simply need to state what your system *should* look like in an ideal state. Kubernetes will detect when the actual state of the system doesn't meet these expectations and it will intervene on your behalf to fix the problem. This enables our systems to be **self-healing** and react to problems without the need for human intervention. The "state" of your system is defined by a collection of *objects*. Each Kubernetes object has (1) a *specification* in which you provide the desired state and (2) a *status* which reflects the current state of the object. Kubernetes maintains a list of all object specifications and constantly polls each object in order to ensure that its status is equal to the specification. If an object is unresponsive, Kubernetes will spin up a new version to replace it. If a object's status has drifted from the specification, Kubernetes will issue the necessary commands to drive that object back to its desired state. ### Distributed For a certain operating scale, it becomes necessary to architect your applications as a distributed system. Kubernetes is designed to provide the infrastructural layer for such distributed systems, yielding clean abstractions to build applications on top of a collection of machines (collectively known as a cluster). More specifically, Kubernetes provides a *unified* interface for interacting with this cluster such that you don't have to worry about communicating with each machine individually. ![distributed_systems](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/11/distributed_systems.png) ### Decoupled It is [commonly recommended](https://www.redhat.com/en/resources/cloud-native-container-design-whitepaper?ref=jeremyjordan.me) that containers be developed with a *single concern* in mind. As a result, developing containerized applications lends itself quite nicely to the [microservice architecture](https://martinfowler.com/articles/microservices.html?ref=jeremyjordan.me) design pattern, which recommends "designing software applications as suites of independently deployable services." The abstractions provided in Kubernetes naturally support the idea of decoupled services which can be scaled and updated independently. These services are logically separated and communicate via well-defined APIs. This logical separation allows teams to deploy changes into production at a higher velocity since each service can operate on independent release cycles (provided that they respect the existing API contracts). ### Immutable infrastructure In order to achieve the most benefit from containers and container orchestration, you should be deploying immutable infrastructure. This is, rather than logging in to a container on a machine to make changes (eg. updating a library), you should build a new container image, deploy the new version, and terminate the older version. As you transition across environments during the life-cycle of a project (development -> testing -> production) you should *use the same container image* and only modify configurations external to the container image (eg. by mounting a config file). This becomes very important since **containers are designed to be ephemeral**, ready to be replaced by another container instance at any time. If your original container had mutated state (eg. manual configuration) but was shut down due to a failed healthcheck, the new container spun up in its place would not reflect those manual changes and could potentially break your application. When you maintain immutable infrastructure, it also becomes much easier to roll back your applications to a previous state (eg. if an error occurs) - you can simply update your configuration to use an older container image. --- ## Basic objects in Kubernetes. Previously, I mentioned that we describe our *desired state* of the system through a collection of Kubernetes **objects**. Up until now, our discussion of Kubernetes has been relatively abstract and high-level. In this section, we'll dig into more specifics regarding how you can deploy applications on Kubernetes by covering the basic objects available in Kubernetes. Kubernetes objects can be defined using either YAML or JSON files; these files defining objects are commonly referred to as *manifests*. It's a good practice to keep these manifests in a version controlled repository which can act as the single source of truth as to what objects are running on your cluster. ### Pod The **Pod** object is the fundamental building block in Kubernetes, comprised of one or more (tightly related) containers, a shared networking layer, and shared filesystem volumes. Similar to containers, pods are designed to be ephemeral - there is no expectation that a *specific, individual pod* will persist for a long lifetime. ![pod](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/11/pod.png) You won't typically explicitly create Pod objects in your manifests, as it's often simpler to use higher level components which manage Pod objects for you. ### Deployment A **Deployment** object encompasses a collection of pods defined by a template and a replica count (how many copies of the template we want to run). You can either set a specific value for the replica count or use a separate Kubernetes resource (eg. a horizontal pod autoscaler) to control the replica count based on system metrics such as CPU utilization. > Note: The Deployment object's controller actually creates another object, a ReplicaSet, under the hood. However, this is abstracted away from you as the user. ![deployment](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/11/deployment.png) While you can't rely on any *single* pod to stay running indefinitely, you can rely on the fact that the cluster will always try to have $n$ pods available (where $n$ is defined by your specified replica count). If we have a Deployment with a replica count of 10 and 3 of those pods crash due to a machine failure, 3 more pods will be scheduled to run on a different machine in the cluster. For this reason, *Deployments are best suited for stateless applications* where Pods are able to be replaced at any time without breaking things. The following YAML file provides an annotated example of how you might define a Deployment object. In this example, we want to run 10 instances of a container which serves an ML model over a REST interface. ![deployment_spec](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/11/deployment_spec.png) > Note: In order for Kubernetes to know how compute-intensive this workload might be, we should also provide [resource limits](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/?ref=jeremyjordan.me) in the pod template specification. Deployments also allow us to specify how we would like to roll out updates when we have new versions of our container image; [this blog post](https://blog.container-solutions.com/kubernetes-deployment-strategies?ref=jeremyjordan.me) provides a good overview of your different options. If we wanted to override the defaults we would include an additional `strategy` field under the object `spec`. Kubernetes will make sure to gracefully shut down Pods running the old container image and spin up new Pods running the new container image. ![deployment_update](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/11/deployment_update.png) ### Service Each Pod in Kubernetes is assigned a unique IP address that we can use to communicate with it. However, because Pods are ephemeral, it can be quite difficult to send traffic to your desired container. For example, let's consider the Deployment from above where we have 10 Pods running a container serving a machine learning model over REST. How do we reliably communicate with a server if the set of Pods running as part of the Deployment can change at any time? This is where the **Service** object enters the picture. A Kubernetes Service provides you with a stable endpoint which can be used to direct traffic to the desired Pods even as the exact underlying Pods change due to updates, scaling, and failures. Services know which Pods they should send traffic to based on *labels* (key-value pairs) which we define in the Pod metadata. > Note: This [blog post](https://www.asykim.com/blog/deep-dive-into-kubernetes-external-traffic-policies?ref=jeremyjordan.me) does a nice job explaining how traffic is actually routed. ![service](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/11/service-1.png) In this example, our Service sends traffic to all healthy Pods with the label `app="ml-model"`. The following YAML file provides an example for how we might wrap a Service around the earlier Deployment example. ![service_spec](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/11/service_spec.png) ### Ingress While a Service allows us to expose applications behind a stable endpoint, the endpoint is only available to internal cluster traffic. If we wanted to expose our application to traffic external to our cluster, we need to define an **Ingress** object. ![ingress](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/11/ingress.png) The benefit of this approach is that you can select which Services to make publicly available. For example, suppose that in addition to our Service for a machine learning model, we had a UI which leveraged the model's predictions as part of a larger application. We may choose to only make the UI available to public traffic, preventing users from being able to query the model serving Service directly. ![ingress_2](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/11/ingress_2.png) The following YAML file defines an Ingress object for the above example, making the UI publicly accessible. ![ingress_spec](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/11/ingress_spec.png) ### Job The Kubernetes objects I've described up until this point can be composed to create reliable, long-running services. In contrast, the **Job** object is useful when you want to perform a discrete task. For example, suppose we want to retrain our model daily based on the information collected from the previous day. Each day, we want to spin up a container to execute a predefined workload (eg. a `train.py` script) and then shut down when the training finishes. Jobs provide us the ability to do exactly this! If for some reason our container crashes before finishing the script, Kubernetes will react by launching a new Pod in its place to finish the job. For Job objects, the "desired state" of the object is completion of the job. The following YAML defines an example Job for training a machine learning model (assuming the training code is defined in `train.py`). ![job_spec](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/11/job_spec.png) > Note: This Job specification will only execute a single training run. If we wanted to execute this job daily, we could define a CronJob object instead. ### ...and many more. The objects discussed above are certainly not an exhaustive list of resource types available in Kubernetes. Some other objects that you'll most likely find useful when deploying applications include: - Volume: for managing directories mounted onto Pods - Secret: for storing sensitive credentials - Namespace: for separating resources on your cluster - ConfigMap: for specifying application configuration values to be mounted as a file - HorizontalPodAutoscaler: for scaling Deployments based on the current resource utlization of existing Pods - StatefulSet: similar to a Deployment, but for when you need to run a stateful application --- ## How? The Kubernetes control plane. By this point, you're probably wondering how Kubernetes is capable of taking all of our object specifications and actually executing these workloads on a cluster. In this section we'll discuss the components that make up the Kubernetes **control plane** which govern how workloads are executed, monitored, and maintained on our cluster. Before we dive in, it's important to distinguish two classes of machines on our cluster: - A **master node** contains most of the components which make up our control plane that we'll discuss below. In most moderate-sized clusters you'll only have a single master node, although it is possible to have multiple master nodes for high availability. If you use a cloud provider's managed Kubernetes service, they will typically abstract away the master node and you will not have to manage or pay for it. - A **worker node** is a machine which actually runs our application workloads. It is possible to have multiple different machine types tailored to different types of workloads on your cluster. For example, you might have some GPU-optimized nodes for faster model training and then use CPU-optimized nodes for serving. When you define object specifications, you can specify a preference as to what type of machine the workload gets assigned to. Now let's dive in to the main components on our master node. When you're communicating with Kubernetes to provide a new or updated object specification, you're talking to the **API server**. ![api-server](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/11/api-server.png) More specifically, the API server validates requests to update objects and acts as the unified interface for questions about our cluster's current state. However, the *state* of our cluster is stored in **etcd**, a distributed key-value store. We'll use etcd to persist information regarding: our cluster configuration, object specifications, object statuses, nodes on the cluster, and which nodes the objects are assigned to run on. ![etcd](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/11/etcd.png) > Note: etcd is the only stateful component in our control plane, all other components are stateless. Speaking of where objects should be run, the **scheduler** is in charge of determining this! The scheduler will ask the API server (which will then communicate with etcd) which objects which haven't been assigned to a machine. The scheduler will then determine which machines those objects should be assigned to and will reply back to the API server to reflect this assignment (which gets propagated to etcd). ![scheduler](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/11/scheduler.png) The last component on the master node that we'll discuss in this post is the **controller-manager**, which monitors the state of a cluster through the API server to see if the current state of the cluster aligns with our desired state. If the actual state differs from our desired state, the controller-manager will make changes via the API server in an attempt to drive the cluster towards the desired state. The controller-manager is defined by a collection of **controllers**, each of which are responsible for managing objects of a specific resource type on the cluster. At a very high level, a controller will watch a specific resource type (eg. deployments) stored in etcd and create specifications for the pods which should be run to acheive the object's desired state. It is then the controller's responsibility for ensuring that these pods stay healthy while running and are shut down when needed. To summarize what we've covered so far... ![master-node](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/11/master-node.png) Next, let's discuss the control plane components which are run on worker nodes. Most of the resources available on our worker nodes are spent running our actual applications, but our nodes do need to know which pods they should be running and how to communicate with pods on other machines. The two final components of the control plane that we'll discuss cover exactly these two concerns. The **kubelet** acts as a node's "agent" which communicates with the API server to see which container workloads have been assigned to the node. It is then responsible for spinning up pods to run these assigned workloads. When a node first joins the cluster, kubelet is responsible for announcing the node's existence to the API server so the scheduler can assign pods to it. ![kubelet](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/11/kubelet.png) Lastly, **kube-proxy** enables containers to be able to communicate with each other across the various nodes on the cluster. This component handles all the networking concerns such as how to forward traffic to the appropriate pod. Hopefully, by this point you should be able to start seeing the whole picture for how things operate in a Kubernetes cluster. All of the components interact through the API server and we store the state of our cluster in etcd. There are various components which write to etcd (via the API server) to make changes to the cluster and nodes on the cluster listen to etcd (via the API server) to see which pods it should be running. ![full_picture](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/11/full_picture-1.png) The overall system is designed such that failures have minimal impact on the overall cluster. For example, if our master node went down, none of our applications would be immediately affected; we would just be prevented from making any further changes to the cluster until a new master node is brought online. --- ## When should you not use Kubernetes? As with every new technology, you'll incur an overhead cost of adoption as you learn how it works and how it applies to the applications that you're building. It's a reasonable question to ask "do I really need Kubernetes?" so I'll attempt to provide some example situations where the answer might be no. - You can run your workload on a single machine. *(Kubernetes can be viewed as a platform for building distributed systems, but you shouldn't build a distributed system if you don't need one!)* - Your compute needs are light. *(In this case, the compute spent on the orchestration framework is relatively high!)* - You don't need high availability and can tolerate downtime. - You don't envision making a lot of changes to your deployed services. - You already have an effective tool stack that you're satisfied with. - You have a monolithic architecture and don't plan to [separate it into microservices](https://martinfowler.com/bliki/MonolithFirst.html?ref=jeremyjordan.me). *(This goes back to using the tool as it was intended to be used.)* - You read this post and thought "holy shit this is complicated" rather than "holy shit this is useful". --- ## Gratitude Thank you to Derek Whatley and Devon Kinghorn for teaching me most of what I know about Kubernetes and answering my questions as I've been trying to wrap my head around this technology. Thank you to Goku Mohandas, John Huffman, Dan Salo, and Zack Abzug for spending your time to review an early version of this post and provide thoughtful feedback. And lastly, thank you to Kelsey Hightower for all that you've contributed to the Kubernetes community - your talks have helped me understand the bigger picture and gave me confidence that I could learn about this topic. --- ## Resources Here are some resources that I found to be useful when learning about Kubernetes. **Blogs** - [Julia Evans - Reasons Kubernetes is cool](https://jvns.ca/blog/2017/10/05/reasons-kubernetes-is-cool/?ref=jeremyjordan.me) - [Julia Evans - A few things I've learned about Kubernetes](https://jvns.ca/blog/2017/06/04/learning-about-kubernetes/?ref=jeremyjordan.me) (Julia's zines were a big inspiration for my visuals explaining the Kubernetes control plane) - [Jessie Frazelle - You might not need Kubernetes](https://blog.jessfraz.com/post/you-might-not-need-k8s/?ref=jeremyjordan.me) - [Matt Rogish - Is Kubernetes Overkill?](https://medium.com/faun/is-kubernetes-overkill-ac7796d18b6f?ref=jeremyjordan.me) - [Major Trends in the 2019 Data & AI Landscape](https://mattturck.com/2019trends/?ref=jeremyjordan.me) - [Introduction to cloud-native applications](https://docs.microsoft.com/en-us/dotnet/architecture/cloud-native/introduction?ref=jeremyjordan.me) and [defining cloud native](https://docs.microsoft.com/en-us/dotnet/architecture/cloud-native/definition?ref=jeremyjordan.me) - [Kubernetes Best Practices 101](https://github.com/diegolnasc/kubernetes-best-practices?ref=jeremyjordan.me) **Videos** - [Kelsey Hightower - Kuberenetes for Pythonistas](https://www.youtube.com/watch?v=u%5FiAXzy3xBA&ref=jeremyjordan.me) discusses the motivation for Kubernetes and provides an example of running a Python application. - [Kubernetes by Kelsey Hightower](https://www.youtube.com/watch?v=8SvQqZNP6uo&ref=jeremyjordan.me) introduces the core components of Kubernetes and how they work together, with the API server at the core. - [Kubernetes The Easy Way!](https://www.youtube.com/watch?v=kOa%5FllowQ1c&ref=jeremyjordan.me) presents a developer-centric workflow for building products and leveraging Kubernetes as the infrastructure. - [Kubernetes in Real Life - Ian Crosby](https://www.youtube.com/watch?v=UUt7SuG3nW4&ref=jeremyjordan.me) - [Kubernetes Design Principles: Understand the Why - Saad Ali, Google](https://www.youtube.com/watch?v=ZuIQurh%5FkDk&ref=jeremyjordan.me) - [Kubernetes Deconstructed: Understanding Kubernetes by Breaking It Down - Carson Anderson, DOMO](https://www.youtube.com/watch?v=90kZRyPcRZw&ref=jeremyjordan.me) - [From COBOL to Kubernetes: A 250 Year Old Bank's Cloud-Native Journey - Laura Rehorst](https://www.youtube.com/watch?v=uRvKGZ%5FfDPU&feature=youtu.be&ref=jeremyjordan.me) **Books** - [Kubernetes: Up and Running, Second Edition](https://azure.microsoft.com/en-us/resources/kubernetes-up-and-running/?ref=jeremyjordan.me) \- free book! - [Kubernetes for Machine Learning, Deep Learning & AI](https://mapr.com/ebook/kubernetes-for-machine-learning-deep-learning-and-ai/assets/Cloud%5FEB%5FKubernetes%5FMLDL%5FSPONSOR.pdf?ref=jeremyjordan.me) ### Building machine learning products: a problem well-defined is a problem half-solved. URL: https://www.jeremyjordan.me/ml-requirements/ Last updated: 2023-03-05T03:48:02.000Z Previously, I wrote about [organizing machine learning projects](https://www.jeremyjordan.me/ml-projects-guide/) where I presented the framework that I use for building and deploying models. However, that framework operates on the implicit assumption that you already know generally what your model should do. In this post, we'll dig deeper into how to **develop the requirements for a machine learning project** when you're given a vague problem to solve. Some questions that we'll address include: - What specific task should our model be automating? - How does the user interact with the model? - What information should we expose to the user? **Note:* Sometimes machine learning projects can be very straightforward; your stakeholders define an API specification stating the inputs to the system and the desired outputs and you agree that the task seems feasible. These projects typically support existing products with existing "intelligence" solutions - your task is to simply encapsulate the "intelligence" task using machine learning in lieu of the existing solution (ie. rule-based systems, mechanical turk workers, etc).* Throughout this blog post, I'll use the following problem statement as a running example. > We're building an application to help people keep their photos organized. Our target user base are casual smartphone photographers who have a large number of photos in their camera roll. We want to make it easier for these users to find photos of interest. Notice how vague that problem statement is - what defines a photo of interest? There's a multitude of ways we could address this task and without understanding the problem in more detail we won't know which direction to take. At this point in time, we have insufficient information to specify an objective function for training a model. Understanding the problem and developing the requirements isn't something you typically get right on the first attempt; this is often an *iterative process* where we initially define a set of coarse requirements and refine the detail as we gain more information. ![design-process](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/09/design-process.png) **Jump to:** - [Understand the problem from the perspective of the user](#understand) - [Mock out your machine learning model and iterate on the user experience](#iterate) - [Develop a shared language with your project stakeholders](#language) - [Win by shipping](#shipping) - [Conclusion](#conclusion) --- ## Understand the problem from the perspective of the user. The first step towards establishing any set of requirements for a project is understanding the problem you're setting out to solve. No one knows the problem better than the intended user - the person we are attempting to solve the problem for. **Perform informational interviews with end users to understand their perspective.** The further removed you are from the end user, the less likely you are to solve the actual problem they're experiencing. Do you remember playing the telephone game as a kid, where you have a chain of people and try to deliver a message from one end to the other by asking each person to transmit the message to the person next to them? Usually the message at the end is very different from the original message. In order to ensure you're solving the right problem, you'll want to be able to empathize with the people who currently experience that problem. For the case of our photo app example, this would involve speaking with casual smartphone photographers and understanding how they use their app. - When they're searching through their camera roll, what are they typically looking for? - Are they Instagram influencers who take 100 photos of the same thing and are trying to find the *best* photo to post? - Are they nature photographers looking to share the scenic views from their latest camping trip? - Are they recent parents trying to document the development of their newborn baby? - Are they often looking for photos with a *specific* friend to share on social media with a happy birthday wish? - What are their current strategies for finding the photos of interest? - Do they know when the photo was taken? - Do they flip through their camera roll photo by photo or scroll through a series of thumbnails? - Are they typically searching for a photo of a specific person? - Do they mentally "chunk" photos together and search chunk by chunk? What's the criteria for chunking? At this stage you don't want to start prescribing a solution, you're simply trying to understand the problem. However, it can be good to consider the capabilities of machine learning systems and ask questions which may eventually guide your scoping of a solution. For example, let's consider the motivation behind asking the first set of questions. - Are they Instagram influencers who take 100 photos of the same thing and are trying to find the *best* photo to post? *Perhaps we want to consider clustering similar photos and then applying a ranking model to help them find the best photo to share.* - Are they nature photographers looking to share the scenic views from their latest camping trip? *Do we need to consider additional metadata such as the GPS coordinates of where the photo was taken to enrich the data representation?* - Are they recent parents trying to document the development of their newborn baby? *Perhaps we want to perform action recognition on the photographs to help the parents find the moment where their newborn walked for the first time or did a cute dance.* - Are they often looking for photos with a *specific* friend to share on social media with a happy birthday wish? *Maybe we'll need to use facial recognition to identify the user's friends.* Even though you might be considering solutions, it's important that the conversations with users are focused on the problems they experience. You won't ask them about their thoughts on specific solutions until the next stage. **Subject yourself to the problem.** As a machine learning practitioner, it can be tempting to jump right into training a model to learn some task. However, it's often very instructive to first force yourself to perform the task manually. Additionally, this helps you *empathize* with the users. Pay close attention to how *you* solve the task, as this might inform what features might be important to include when you do train a model to perform the task. For example, after speaking with some casual smartphone photographers you might construct a couple photo albums and go through the tasks described during your interviews. What strategies did *you* find effective for finding content more quickly? --- ## Mock out your machine learning model and iterate on the user experience. After (and **only** after) getting a better understanding of the problem, you'll want to start sketching out the set of possible solutions and approaches that you could take. It's generally a good idea to think broadly at this stage rather than prematurely honing in on your first decent idea. At this stage, we're trying to elicit the *desired user experience*, as this can ultimately drive the requirements of the project. **Prototype and iterate on the user experience using design tools to communicate possible solutions.** There's something magical about seeing something concretely that has the ability to elicit tangible feedback from your users. When speaking in the abstract sense, it's possible for both you and the other stakeholders to have different understandings whilst under the illusion that you're in agreement. However, these latent misunderstandings often become very clear when you reduce an abstract idea to practice. For example, I used a design tool called Figma to sketch out a couple different ways we might help users find photos of interest more quickly. The goal of these sketches is to spark discussion with our stakeholders and/or users in an attempt to start narrowing down the possible solution space. ![four-ideas](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/09/four-ideas.png) These tools allow you to easily stitch together user flows and import data to use in your mockups. **Fake the machine learning component with "Wizard-of-Oz" experiments.** Building a machine learning model takes a significant amount of work. You have to acquire data for training, decide on a model architecture, ensure the model is performing at a sufficient level, etc. Our goal here is to validate the *utility* of a model without actually going through all of the effort involved in building it. These types of experiments can be especially useful when there's still uncertainty surrounding how users will interact with your model. Apple researchers [wrote about one such study](https://arxiv.org/abs/1904.01664?ref=jeremyjordan.me) when deciding whether their digital assistant should mimic the conversational style of the user. Their core hypothesis was that users would prefer digital assistants that match their level of chattiness when conversing with the agent. Rather than training a machine learning model that would allow them to modulate the "chattiness" level in the agent's response, they first tested this hypothesis by faking the digital assistant component and having humans in a separate room follow a script pretending to be the digital assistant. **Figure out how to establish trust with the user.** Ideally you'd like to design your product such that user interactions can improve your model, which in turn improve the user experience. This is commonly referred to as the "data flywheel" in machine learning products. However, in order to source meaningful interactions from your users, you may need to first establish trust with the user that those interactions will indeed improve their experience. For example, if we decided that we would perform facial recognition and allow you to search for photos with a specific person present, you'd likely require the user to assign identities to collections of photos with a common face. In order to motivate and incentivize users, they need to feel as if their effort in identifying faces in photos is meaningful. One common technique used here is to *show the model's effort* in an unobtrusive manner which informs the user how it's working. Continuing the facial recongition example, you might allow the user to toggle a setting which draws bounding boxes around the identified faces in photos. ![faces_toggle](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/09/faces_toggle.png) --- ## Develop a shared language with your project stakeholders. At this point, you're likely to have a decent grasp on the approach you'll take to solve the presented problem. However, much work still lies ahead! It'll be important that you *communicate effectively* with stakeholders as you work to build out your solution; this starts with speaking a common language. Perhaps a number of users leverage a technique of *chunking* photos together when searching, while you typically refer to that same technique as *clustering*. Without converging on a shared language, it's all too easy to talk right past one another without realizing you're actually on the same page. **Over-communicate and ask a lot of clarifying questions at the onset.** This can include asking questions when you're already pretty sure what the answer will be, sometimes you may be surprised. **Present ideas and progress often.** It's important to share progress with your stakeholders and hold discussion to ensure you're still heading in the right direction. As you present your work, it can be helpful to discuss things such as model metrics which you're using to evaluate performance and *why we should care* about those metrics; keeping things simple combined with repeated exposure can go a long way here. --- ## Win by shipping. Getting your product in the hands of your users is one of the best ways to validate your ideas. Quicker iterations allow you to validate more ideas, which in turn allows you to fine-tune your solution offering in order to maximize value to the user. Further, incremental deployments help ensure that you're heading in the right direction as you work to build out the full solution. ![giphy](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/09/giphy.gif) In my post on [organizing machine learning projects](https://www.jeremyjordan.me/ml-projects-guide/), I presented a diagram for model development which represents the iterative nature of the work. ![ml-development-cycle-1](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/09/ml-development-cycle-1.png) I also called out [Stephen Merity's advice](https://twitter.com/Smerity/status/1095490777860304896?ref=jeremyjordan.me) and Martin Zinkevich's [Rule #4 of Machine Learning](http://martin.zinkevich.org/rules%5Fof%5Fml/rules%5Fof%5Fml.pdf?ref=jeremyjordan.me), which both advocate for initially deploying simple models in the spirit of winning by shipping. In this context, *"winning by shipping"* involves making quick iterations through the outermost development loop. It can take getting a few projects under your belt to fully appreciate this wisdom. When I initially published the diagram a year ago, I tended to view progression through this iterative flow as a [stage-gate process](https://en.wikipedia.org/wiki/Phase-gate%5Fprocess?ref=jeremyjordan.me) where you might loop back to an earlier step if you gain more information, but you still *advance* through the cycle in the order that sections appear on the diagram. However, after joining the machine learning team at Proofpoint, I've learned from the team that there can be immense value in shortcutting parts of the development process (eg. see *shadow mode deployment* below) in order to gain insights from deploying models on production data. **Deploy a baseline model on production data as soon as possible.** Deploying your model on production data can be enlightening. Often times, the "live data" varies in unexpected ways from the data you've collected to use during development (often referred to as "train/test skew" or "production data drift"). As a countermeasure, it's often a good idea to deploy a simple model *on production data* as soon as possible. Depending on the consequence of wrong predictions, you might choose to deploy this simple model in "shadow mode" (don't actually use the predictions) or as a canary deployment on a small subset of your users. The goal of this deployment is to observe and characterize the types of errors that the simple model makes, which can inform further model improvements. ![shadow-mode-1](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/09/shadow-mode-1.png) You may choose to initially skip some steps in the model development process in order to prioritize a quick first iteration and gain insights. Deploying a simple model with urgency also helps ensure that the additional engineering work required to run a machine learning model is done upfront. This allows you to *deploy* your incremental model improvements (enabling quick iterations) rather than slaving away trying to build the perfect model in isolation. **Deliver value incrementally and quickly.** If you're automating a system, start with the simplest task and deploy a solution quickly. Let data guide the priority order of further automation. In other words, automate first according to the "happy path" (assume nothing goes wrong) and then categorize the times where control is reverted to a human. Going back to our photo app example, there are *many* opportunities to apply machine learning models to the product. Ideally, we'd like to start at the intersection of a model which is simple to develop and provides value to users. For example, we might decide to initially deploy a facial recognition model given the fact that there are a number of open source implementations that we could use and user studies provided us with some level of evidence that this model would be valuable as part of the product. **Measure time to results, not results.** Sometimes it can be tricky to get into the mindset of delivering value quickly. It's easy to think that in order to delight the user, we need to give them the *perfect* product. Measuring *time to results* rather than the results themselves instills a culture of quick iterations which ultimately enables your team to deliver better products, winning by shipping. --- ## Conclusion Often times we're presented with vague problems and tasked with developing a "machine learning solution." By spending the effort to properly scope your project and define its requirements up front, you can establish a foundation for *smooth iterations* through the model development loop as we work towards a final solution. ![full-diagram](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/09/full-diagram.png) Now go build great things! --- *The insights shared in this blog post are shaped by my experience working on teams building and deploying machine learning models in products. On my current team at Proofpoint, we frequently discuss how we can refine our process to deliver value quickly on machine learning projects; these conversations and shared experiences are invaluable. Thank you to [John Berryman](http://jnbrymn.com/?ref=jeremyjordan.me), [Dan Salo](https://dancsalo.github.io/?ref=jeremyjordan.me), John Huffman, and Michelle Carney for reading early drafts of this post and providing feedback.* ## External Resources **Blog posts** - [Machine learning, faster](https://medium.com/@neal%5Flathia/machine-learning-faster-ce404dc4cd2?ref=jeremyjordan.me) (introduced me to the concept of "measure time to results, not results") - [Data Jujitsu: The art of turning data into product](http://radar.oreilly.com/2012/07/data-jujitsu.html?ref=jeremyjordan.me) - [What you need to know about product management for AI](https://www.oreilly.com/radar/what-you-need-to-know-about-product-management-for-ai/?ref=jeremyjordan.me) - [Practical Skills for The AI Product Manager](https://www.oreilly.com/radar/practical-skills-for-the-ai-product-manager/?ref=jeremyjordan.me) - [Product Driven Machine Learning](https://hackernoon.com/product-driven-machine-learning-and-parking-tickets-in-nyc-4a3b74cfe496?ref=jeremyjordan.me) - [Human-Centered Machine Learning](https://medium.com/google-design/human-centered-machine-learning-a770d10562cd?ref=jeremyjordan.me) - [Design in a world where Machines are Learning](http://www.r2d3.us/talks/design-in-a-world-where-machines-are-learning/?ref=jeremyjordan.me) - [When are we going to start designing AI with purpose?](https://uxdesign.cc/when-are-we-going-to-start-designing-ai-with-purpose-e196f986974b?ref=jeremyjordan.me) - [User Research Makes Your AI Smarter](https://medium.com/microsoft-design/user-research-makes-your-ai-smarter-70f6ef6eb25a?ref=jeremyjordan.me) - [What is the best way to write a Product Requirements Document?](https://www.vindhyac.com/posts/best-prd-templates-from-companies-we-adore/?ref=jeremyjordan.me) **Case studies** - [Control and Simplicity in the Age of AI](https://design.google/library/control-and-simplicity/?ref=jeremyjordan.me) - [150 successful machine learning models: 6 lessons learned at Booking.com](https://blog.acolyer.org/2019/10/07/150-successful-machine-learning-models/?ref=jeremyjordan.me) **Talks** - [Triptech: A Method for Evaluating Early Design Concepts](https://www.youtube.com/watch?v=O5Gx1CFp0-Y&ref=jeremyjordan.me) - [AI x Design - Youtube Playlist](https://www.youtube.com/playlist?list=PLUW5utnrTMQc2VMdpf8cPz9qQpHGWL-iu&ref=jeremyjordan.me) **Guides** - [Guidelines for Human-AI Interaction](https://www.microsoft.com/en-us/research/uploads/prod/2019/01/Guidelines-for-Human-AI-Interaction-camera-ready.pdf?ref=jeremyjordan.me) (and also check out the pretty infographic [here](https://www.microsoft.com/en-us/research/uploads/prod/2019/03/AI%5FGuidelines%5FPoster%5FPrintQuality.pdf?ref=jeremyjordan.me)) - [AI Meets Design](http://aimeets.design/?ref=jeremyjordan.me) - [Google's People+AI Guidebook: Designing human-centered AI products](https://pair.withgoogle.com/?ref=jeremyjordan.me) - [Designing Machine Learning](https://designwith.ml/?ref=jeremyjordan.me) - [Human-centered AI cheat-sheet](https://uxdesign.cc/human-centered-ai-cheat-sheet-1da130ba1bab?ref=jeremyjordan.me) **People to follow** - Michelle Carney [@michellercarney](https://twitter.com/michellercarney?ref=jeremyjordan.me) works at the intersection of UX+ML and leads the [MLUX meetup group](https://www.youtube.com/channel/UC8WlnMDt6LdqoZhnPakagXw?ref=jeremyjordan.me). - Jess Holbrook [@jessholbrook](https://medium.com/@jessholbrook?ref=jeremyjordan.me) is the co-lead of Google’s People + AI Research team; he has a ton of great articles on Medium. - Nadia Piet [@NadiaPiet](https://twitter.com/NadiaPiet?ref=jeremyjordan.me) works as a freelance design consultant; she put together the phenomenal [aimeets.design](http://aimeets.design/?ref=jeremyjordan.me) toolkit referenced above. ### Introduction to recurrent neural networks. URL: https://www.jeremyjordan.me/introduction-to-recurrent-neural-networks/ Last updated: 2023-03-05T03:50:18.000Z **Jump to:** - [Overview](#overview) - [Evolving a hidden state over time](#evolving) - [Common structures of recurrent networks](#common%5Fstructures) - [Bidirectionality](#bidirectional) - [Limitations](#limitations) - [Further reading](#further%5Freading) ## Overview Previously, I've written about [feed-forward neural networks](https://www.jeremyjordan.me/intro-to-neural-networks/) as a generic function approximator and [convolutional neural networks](https://www.jeremyjordan.me/convolutional-neural-networks/) for efficiently extracting local information from data. In this post, I'll discuss a third type of neural networks, recurrent neural networks, for learning from sequential data. For some classes of data, the order in which we receive observations is important. As an example, consider the two following sentences: 1. "I'm sorry... it's not you, it's me." 2. "It's not me, it's you... I'm sorry." These two sentences are communicating quite different messages, but this can only be interpreted when considering the sequential order of the words. Without this information, we're unable to disambiguate from the collection of words: `{'you', 'sorry', 'me', 'not', 'im', 'its'}`. Recurrent neural networks allow us to formulate the learning task in a manner which considers the sequential order of individual observations. ## Evolving a hidden state over time In this section, we'll build the intuition behind recurrent neural networks. We'll start by reviewing standard feed-forward neural networks and build a simple mental model of how these networks learn. We'll then build on that to discuss how we can extend this model to a sequence of related inputs. Recall that neural networks perform a series of layer by layer transformations to our input data. The hidden layers of the network form *intermediate representations* of our input data which make it easier to solve the given task. ![Screen-Shot-2019-05-27-at-11.39.48-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/05/Screen-Shot-2019-05-27-at-11.39.48-AM.png) This is demonstrated in the example below. Observe how our input space is warped into one which allows for a linear decision boundary to cleanly separate the two classes. At a high level, you can think of the hidden layers as **"useful representations"** of the original input data. ![netvis](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/05/netvis.png) [Image credit](https://colah.github.io/posts/2015-09-NN-Types-FP/?ref=jeremyjordan.me) Now let's consider how we can leverage this insight for a sequence of related observations. Let's first focus on the initial value in the sequence. As we calculate the forward pass through the network, we build a "useful representation" of our input in the hidden layers (the activations in these layers define our **hidden state**), continuing on to calculate an output prediction for the initial time-step. ![intuition-1](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/05/intuition-1.png) When considering the next time-step in the sequence, we want to leverage any information we've already extracted from the sequence. ![intuition-2](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/05/intuition-2.png) In order to do this, our next hidden state will be calculated as a **combination** of the previous hidden state and latest input. ![intuition-3](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/05/intuition-3.png) The basic method for combining these two pieces of information is shown below; however, there exist other more advanced methods that we'll discuss later (gated recurrent units, long short-term memory units). Here, we have one set of weights $w\_{ih}$ to transform the input to a hidden layer representation and a second set of weights $w\_{hh}$ to bring along information from the previous hidden state into the next time-step. ![intuition-4](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/05/intuition-4.png) We can continue performing this *same calculation* of incorporating new information to update the value of the hidden state for an *arbitrarily long sequence* of observations. ![intuition-5](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/05/intuition-5.png) By always remembering the previous hidden state, we're able to chain a sequence of events together. This also allows us to backpropagate errors to earlier timesteps during training, often referred to as "backpropagation through time". ![Screen-Shot-2019-06-01-at-12.21.53-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/06/Screen-Shot-2019-06-01-at-12.21.53-PM.png) ## Common structures of recurrent networks One of the benefits of recurrent neural networks is the ability to handle arbitrary length inputs and outputs. This flexibility allows us to define a broad range of tasks. In this section, I'll discuss the general architectures used for various sequence learning tasks. **One to many** RNNs are used in scenarios where we have a single input observation and would like to generate an arbitrary length sequence related to that input. One example of this is image captioning, where you feed in an image as input and output a sequence of words to describe the image. For this architecture, we take our prediction at each time step and feed that in as input to the next timestep, iteratively generating a sequence from our initial observation and following predictions. ![rnn-one-to-many](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/11/rnn-one-to-many.png) [Image credit](https://stanford.edu/~shervine/teaching/cs-230/cheatsheet-recurrent-neural-networks?ref=jeremyjordan.me#overview) **Many to one** RNNs are used to look across a sequence of inputs and make a single determination from that sequence. For example, you might look at a sequence of words and predict the sentiment of the sentence. Generally, this structure is used when you want to perform classification on sequences of data. ![rnn-many-to-one](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/11/rnn-many-to-one.png) [Image credit](https://stanford.edu/~shervine/teaching/cs-230/cheatsheet-recurrent-neural-networks?ref=jeremyjordan.me#overview) **Many to many (same)** RNNs are used for tasks in which we would like to predict a label for each observation in a sequence, sometimes referred to as dense classification. For example, if we would like to detect named entities (person, organization, location) in sentences, we might produce a label for every single word denoting whether or not that word is part of a named entity. As another example, you could feed in a video (sequence of images) and predict the current activity in frame. ![rnn-many-to-many-same](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/11/rnn-many-to-many-same.png) [Image credit](https://stanford.edu/~shervine/teaching/cs-230/cheatsheet-recurrent-neural-networks?ref=jeremyjordan.me#overview) **Many to many (different)** RNNs are useful for translating a sequence of inputs into a different but related sequence of outputs. In this case, both the input and the output can be arbitrary length sequences and the input length might not always be equal to the output length. For example, a machine translation model would be expected to translate "how are you" (input) into "cómo estás" (output) even though the sequence lengths are different. ![rnn-many-to-many-different](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/11/rnn-many-to-many-different.png) [Image credit](https://stanford.edu/~shervine/teaching/cs-230/cheatsheet-recurrent-neural-networks?ref=jeremyjordan.me#overview) ## Bidirectionality One of the weaknesses of a ordinary recurrent neural networks is that we can only use the set of observations which we have already seen when making a prediction. As an example, consider training a model for named entity recognition. Here, we want the model to output the start and end of phrases which contain a named entity. Consider the following two sentences: > "I can't believe that Teddy Roosevelt was your great grandfather!" > "I can't believe that Teddy bear is made out of chocolate!" However, if you only read the input sequence from left to right, it's hard to tell whether or not you should mark "Teddy" as the start of a name. ![Screen-Shot-2019-06-03-at-10.42.54-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/06/Screen-Shot-2019-06-03-at-10.42.54-PM.png) Ideally, our model output would look something like this when reading the first sentence (roughly following the [inside–outside–beginning tagging](https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning%5F%28tagging%29?ref=jeremyjordan.me) format). ![Screen-Shot-2019-06-03-at-10.47.38-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/06/Screen-Shot-2019-06-03-at-10.47.38-PM.png) When determining whether or not a token is the start of a name, it would sure be helpful to see which tokens follow after it; a **bidirectional** recurrent neural network provides exactly that. Here, we process the sequence reading from left-to-right and right-to-left in parallel and then combine these two representations such that at any point in a sequence you have knowledge of the tokens which came before **and** after it. ![Screen-Shot-2019-06-03-at-11.12.52-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/06/Screen-Shot-2019-06-03-at-11.12.52-PM.png) We have one set of recurrent cells which process the sequence from left to right... ![Screen-Shot-2019-06-03-at-11.13.23-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/06/Screen-Shot-2019-06-03-at-11.13.23-PM.png) ... and another set of recurrent cells which process the sequence from right to left. ![Screen-Shot-2019-06-03-at-11.13.43-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/06/Screen-Shot-2019-06-03-at-11.13.43-PM.png) Thus, at any given time-step we have knowledge of all of the tokens which came before the current time-step **and** all of the tokens which came after that time-step. ## Limitations One key component that I glanced over previously is that the recurrent layer's weights are **shared** across time-steps. This provides us with the flexibility to process arbitrary length sequences, but also introduces a unique challenge when training the network. For a concrete example, suppose you've trained a recurrent neural network as a language model (predict the next word in a sequence). As you're generating text, it might be important to know whether the current word is inside quotation marks. Let's assume this is true and consider the case where our model makes a wrong prediction because it wasn't paying attention to whether or not the current time-step is inside quotation marks. Ideally, you want a way to send back a signal to the earlier time-step where we entered the quotation mark to say "pay attention!" to avoid the same mistake in the future. Doing so requires sending our error signal back through **many time-steps**. (As an aside, Karpathy has a [famous blog post](http://karpathy.github.io/2015/05/21/rnn-effectiveness/?ref=jeremyjordan.me) which shows that a character-level RNN language model can indeed pay attention to this detail.) ![Screen-Shot-2019-06-08-at-5.02.21-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/06/Screen-Shot-2019-06-08-at-5.02.21-PM.png) [Image credit](http://karpathy.github.io/2015/05/21/rnn-effectiveness/?ref=jeremyjordan.me) Let's consider what the backpropagation step would look like to send this signal to earlier time-steps. ![Screen-Shot-2019-06-08-at-5.18.39-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/06/Screen-Shot-2019-06-08-at-5.18.39-PM.png) As a reminder, the [backpropagation algorithm](https://www.jeremyjordan.me/neural-networks-training/) states that we can define the relationship between a given layer's weights and the final loss using the following expression: $$ \\frac{{\\partial E\\left( w \\right)}}{{\\partial w^{(l)}}} = {\\left( {{\\delta ^{(l + 1)}}} \\right)^T}{a^{(l)}} $$ where ${\\delta ^{(l)}}$ (our "error" term) can be calculated as: $$ {\\delta ^{(l)}} = {\\delta ^{(l + 1)}}{w ^{(l)}}f'\\left( {{a^{(l)}}} \\right) $$ This allows to efficiently calculate the gradient for any given layer by reusing the terms already computed at layer $l+1$. However, notice how there's a term for the weight matrix, ${w ^{(l)}}$, included in the computation at every layer. Now recall that I earlier mentioned recurrent layers share weights across time-steps. This means that the **same exact value** is being mulitplied every time we perform this layer by layer backpropagation through time. Let's suppose one of the weights in our matrix is 0.5 and we're attempting to send a signal back 10 time-steps. By the time we've backpropagated to $t-10$, we've multiplied the overall gradient expression by $0.5 \\cdot 0.5 \\cdot 0.5 \\cdot 0.5 \\cdot 0.5 \\cdot 0.5 \\cdot 0.5 \\cdot 0.5 \\cdot 0.5 \\cdot 0.5 = 0.00098$. This has the effect of drastically reducing the magnitude of our error signal! This phenomenon is known as the **"vanishing gradient" problem** which makes it very hard to learn using a vanilla recurrent neural network. The same problem can occur when the weight is greater than one, introducing an exploding gradient, although this is slightly easier to manage thanks to a technique known as gradient clipping. In following posts, we'll look at two common variations of the standard recurrent cell which alleviate this problem of a vanishing gradient. ## Further reading **Papers** - [Learning Long-Term Dependencies with Gradient Descent is Difficult](http://ai.dinfo.unifi.it/paolo//ps/tnn-94-gradient.pdf?ref=jeremyjordan.me) - [On the difficulty of training Recurrent Neural Networks](https://arxiv.org/pdf/1211.5063.pdf?ref=jeremyjordan.me) - [A Simple Way to Initialize Recurrent Networks of Rectified Linear Units](https://arxiv.org/abs/1504.00941?ref=jeremyjordan.me) **Lectures/Notes** - [Stanford CS231n: Lecture 10 | Recurrent Neural Networks](https://www.youtube.com/watch?v=6niqTuYFZLQ&ref=jeremyjordan.me) - [Stanford CS231n Winter 2016: Lecture 10: Recurrent Neural Networks, Image Captioning, LSTM](https://www.youtube.com/watch?v=yCC09vCHzF8&ref=jeremyjordan.me) - [Stanford CS224n: Lecture 8: Recurrent Neural Networks and Language Models](https://www.youtube.com/watch?v=6niqTuYFZLQ&ref=jeremyjordan.me) - [Stanford CS230: Recurrent Neural Networks Cheatsheet](https://stanford.edu/~shervine/teaching/cs-230/cheatsheet-recurrent-neural-networks?ref=jeremyjordan.me) - [MIT 6.S094: Recurrent Neural Networks for Steering Through Time](https://www.youtube.com/watch?v=nFTQ7kHQWtc&ref=jeremyjordan.me) - [University of Toronto CSC2535: Lecture 10 | Recurrent neural networks](https://www.cs.toronto.edu/~hinton/csc2535/notes/lec10new.pdf?ref=jeremyjordan.me) **Blog posts** - [The Unreasonable Effectiveness of Recurrent Neural Networks](http://karpathy.github.io/2015/05/21/rnn-effectiveness/?ref=jeremyjordan.me) ### Scaling nearest neighbors search with approximate methods. URL: https://www.jeremyjordan.me/scaling-nearest-neighbors-search-with-approximate-methods/ Last updated: 2023-04-06T00:58:29.000Z **Jump to:** - [What is nearest neighbors search?](#what) - [K-d trees](#kd%5Ftree) - [Quantization](#quantization) - [Product quantization](#pq) - [Handling multi-modal data](#multimodal) - [Locally optimized product quantization](#lopq) - [Common datasets](#datasets) - [Further reading](#further%5Freading) ## What is nearest neighbors search? In the world of deep learning, we often use neural networks to learn **representations of objects as vectors**. We can then use these vector representations for a myriad of useful tasks. To give a concrete example, let's consider the case of a facial recognition system powered by deep learning. For this use case, the objects are images of people's faces and the task is to identify whether or not the person in a submitted photo matches a person in a database of known identities. We'll use a neural network to build vector representations of all of the images; then, performing facial recognition is as simple as taking the vector representation of a submitted image (the query vector) and **searching for similar vectors** in our database. Here, we define similarity as vectors which are close together in vector-space. (How we actually train a network to produce these vector representations is outside of the scope of this blog post.) ![Screen-Shot-2019-01-27-at-10.00.13-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/01/Screen-Shot-2019-01-27-at-10.00.13-PM.png) All of these vectors were extracted from a ResNet50 model. Notice how the values in the query vector are quite similar to the vector in the top left of known identities. The process of finding vectors that are close to our query is known as **nearest neighbors** search. A naive implementation of nearest neighbors search is to simply calculate the distance between the *query vector* and every vector in our collection (commonly referred to as the *reference set*). However, calculating these distances in a brute force manner quickly becomes infeasible as your reference set grows to millions of objects. ![Screen-Shot-2019-01-29-at-9.57.21-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/01/Screen-Shot-2019-01-29-at-9.57.21-PM.png) Imagine if Facebook had to compare each face in a new photo against **all** of its users *every* time it suggested who to tag, this would be computationally infeasible! A class of methods known as **approximate nearest neighbors** search offer a solution to our scaling dilemma by partitioning the vector space in a clever way such that we only need to examine a small subset of the overall reference set. ![Screen-Shot-2019-01-29-at-10.04.09-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/01/Screen-Shot-2019-01-29-at-10.04.09-PM.png) Approximate methods alleviate this computational burden by cleverly partitioning the vectors such that we only need to focus on a small subset of objects. In this blog post, I'll cover a couple of techniques used for approximate nearest neighbors search. This post will not cover approximate nearest neighbors methods exhaustively, but hopefully you'll be able to understand how people generally approach this problem and how to apply these techniques in your own work. In general, the approximate nearest neighbor methods can be grouped as: - Tree-based data structures - Neighborhood graphs - Hashing methods - Quantization ## K-dimensional trees The first approximate nearest neighbors method we'll cover is a tree-based approach. K-dimensional trees generalize the concept of a [binary search tree](https://medium.com/basecs/leaf-it-up-to-binary-trees-11001aaf746d?ref=jeremyjordan.me) into multiple dimensions. The general procedure for growing a k-dimensional tree is as follows: - pick a random dimension from your k-dimensional vector - find the median of that dimension across all of the vectors in your current collection - split the vectors on the median value - repeat! A toy 2-dimensional example is visualized below. At the top level, we select a random dimension (out of the two possible dimensions, $x\_0$ and $x\_1$) and calculate the median. Then, we follow the same procedure of picking a dimension and calculating the median for *each path* independently. This process is repeated until some stopping criterion is satisfied; each leaf node in the tree contains a subset of vectors from our reference set. ![Screen-Shot-2019-01-29-at-12.35.09-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/01/Screen-Shot-2019-01-29-at-12.35.09-AM.png) We can view how the two-dimensional vectors are partitioned at each level of the k-d tree in the figure below. Take a minute to verify that this visualization matches what is described in the tree above. ![Screen-Shot-2019-01-30-at-11.16.21-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/01/Screen-Shot-2019-01-30-at-11.16.21-PM.png) In order to see the usefulness of this tree, let's now consider how we could use this data structure to perform an approximate nearest neighbor query. As we walk down the tree, notice how the highlighted area (the area in vector space that we're interested in) shrinks down to a small subset of the original space. (I'll use the level 4 subplot for this example.) ![Screen-Shot-2019-01-29-at-10.21.41-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/01/Screen-Shot-2019-01-29-at-10.21.41-PM.png) At the top level, we look at the first dimension of the query vector and ask whether or not its value is greater than or equal to 1\. Since 4 is greater than 1, we walk down the "yes" path to the next level down. We can safely ignore any of the nodes that follow the first "no" path. ![Screen-Shot-2019-01-29-at-10.22.10-PM-1](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/01/Screen-Shot-2019-01-29-at-10.22.10-PM-1.png) Now we look at the second dimension of the vector and ask whether its value is greater than or equal to 0\. Since -2 is less than 0, we now walk down the "no" path. Notice again how the area of interest in our overall vector-space continues to shrink. ![Screen-Shot-2019-01-29-at-10.22.43-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/01/Screen-Shot-2019-01-29-at-10.22.43-PM.png) Finally, once we reach the bottom of the tree we are left with a collection of vectors. Thankfully, this is a small subset relative to the overall size of the reference set, so calculating the distance between the query vector and each vector in this subset is computationally feasible. ![Screen-Shot-2019-01-29-at-10.23.05-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/01/Screen-Shot-2019-01-29-at-10.23.05-PM.png) K-d trees are popular due to their simplicity, however, this technique struggles to perform well when dealing with high dimensional data. Further notice how we only returned vectors which are found in the same cell as the query point. In this example, the query vector happened to fall in the middle of a cell, but you could imagine a scenario where the query vector lies near the edge of a cell and we miss out on vectors which lie just outside of the cell. ## Quantization Another approach to the approximate nearest neighbors problem is to collapse our reference set into a smaller collection of representative vectors. We can find these "representative" vectors by simply running the [K-means algorithm](https://www.jeremyjordan.me/grouping-data-points-with-k-means-clustering/) on our data. In the literature, this collection of "representative" vectors is commonly referred to as the **codebook**. ![Screen-Shot-2019-01-29-at-11.29.48-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/01/Screen-Shot-2019-01-29-at-11.29.48-PM.png) The right figure displays a [Voronoi diagram](https://en.wikipedia.org/wiki/Voronoi%5Fdiagram?ref=jeremyjordan.me) which essentially partitions the space according to the set of points for which a given centroid is closest. We'll then "map" all of our data onto these centroids. By doing this, we can represent our reference set of a couple hundred vectors with only 7 representative centroids. This greatly reduces the number of distance computations we need to perform (only 7!) when making an nearest neighbors query. ![Screen-Shot-2019-01-29-at-11.30.19-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/01/Screen-Shot-2019-01-29-at-11.30.19-PM.png) We can then maintain an inverted list to keep track of all of the original objects in relation to which centroid represents the quantized vector. ![Screen-Shot-2019-02-02-at-11.45.21-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/02/Screen-Shot-2019-02-02-at-11.45.21-PM.png) You can optionally retrieve the full vectors for all of the ids maintained in the inverted list for a given centroid, calculating the true distances between each vector and our query. This is a process known as **re-ranking** and can improve your query performance. Similar to before, let's now look at how we can use this method to perform a query. For a given query vector, we'll calculate the distances between the query vector and each centroid in order to find the closest centroid. We can then look up the centroid in our inverted list in order to find all of the nearest vectors. ![Screen-Shot-2019-01-30-at-12.00.35-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/01/Screen-Shot-2019-01-30-at-12.00.35-AM.png) Unfortunately, in order to get good performance using quantization, you typically need to use very large numbers of centroids for quantization; this impedes on original goal of alleviating the computational burden of calculating too many distances. ### Product quantization Product quantization addresses this problem by first subdividing the original vectors into subcomponents and then quantizing (ie. running K-means on) each subcomponent separately. A single vector is now represented by a collection of centroids, one for each subcomponent. To illustrate this, I've provided two examples. In the 8D case, you can see how our vector is divided into subcomponents and each subcomponent is represented by some centroid value. However, the 2D example shows us the benefit of this approach. In this case, we can only split our 2D vector into a maximum of two components. We'll then quantize each dimension separately, squashing all of the data onto the horizontal axis and running k-means and then squashing all of the data onto the vertical axis and running k-means again. We find 3 centroids for each subcomponent with a total of 6 centroids. However, the total set of all possible quantized states for the overall vector is the **Cartesian product** of the subcomponent centroids. ![Screen-Shot-2019-01-30-at-11.22.40-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/01/Screen-Shot-2019-01-30-at-11.22.40-PM.png) In other words, if we divide our vector into $m$ subcomponents and find $k$ centroids, we can represent $k^m$ possible quantizations using only $km$ vectors! The chart below shows how many centroids are needed in order to get 90% of the top 5 search results correct for an approximate nearest neighbors query. Notice how using product quantization ($m>1$) vastly reduces the number of centroids needed to represent our data. One of the reasons why I love this idea so much is that we've effectively turned [the curse of dimensionality](https://en.wikipedia.org/wiki/Curse%5Fof%5Fdimensionality?ref=jeremyjordan.me) into something highly beneficial! ![Screen-Shot-2019-01-30-at-11.24.00-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/01/Screen-Shot-2019-01-30-at-11.24.00-PM.png) [Image credit](https://loukratz.info/talks/2018-08-09-lopq?ref=jeremyjordan.me) ### Handling multi-modal data Product quantization alone works great when our data is distributed relatively evenly across the vector-space. However, in reality our data is usually multi-modal. To handle this, a common technique involves first training a coarse quantizer to roughly "slice" up the vector-space, and then we'll run product quantization on each individual coarse cell. ![Screen-Shot-2019-01-31-at-12.13.25-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/01/Screen-Shot-2019-01-31-at-12.13.25-AM.png) Below, I've visualized the data that falls within a single coarse cell. We'll use product quantization to find a set of centroids which describe this local subset of data, and then repeat for each coarse cell. Commonly, people encode the vector residuals (the difference between the original vector and the closest coarse centroid) since the residuals tend to have smaller magnitudes and thus lead to less lossy compression when running product quantization. In simple terms, we **treat each coarse centroid as a local origin** and run product quantization on the data with respect to the *local origin* rather than the global origin. ![Screen-Shot-2019-02-03-at-10.16.01-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/02/Screen-Shot-2019-02-03-at-10.16.01-PM.png) **Pro-tip:** If you want to scale to *really* large datasets you can use product quantization as both the coarse quantizer *and* the fine-grained quantizer within each coarse cell. See [this paper](https://cache-ash04.cdn.yandex.net/download.yandex.ru/company/cvpr2012.pdf?ref=jeremyjordan.me) for the details. ### Locally optimized product quantization The ideal goal for quantization is to develop a codebook which is (1) concise and (2) highly representative of our data. More specifically, we'd like all of the vectors in our codebook to represent dense regions of our data in vector-space. A centroid in a low-density area of our data is inefficient at representing data and introduces high distortion error for any vectors which fall in its Voronoi cell. One potential way we can attempt to avoid these inefficient centroids is to add an alignment step to our product quantization. This allows for our product quantizers to better cover the local data for each coarse Voronoi cell. We can do this by applying a transformation to our data such that we minimize our quantization distortion error. One simple way to minimize this quantization distortion error is to simply apply [PCA](https://www.jeremyjordan.me/principal-components-analysis/) in order to mean-center the data and rotate it such that the axes capture most of the variance within the data. Recall my earlier example where we ran product quantization on a toy 2D dataset. In doing so, we effectively squashed all of the data onto the horizontal axis and ran k-means and then repeated this for the vertical axis. By rotating the data such that the axes capture most of the variance, we can more effectively cover our data when using product quantization. ![Screen-Shot-2019-01-31-at-11.48.58-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/02/Screen-Shot-2019-01-31-at-11.48.58-PM.png) This technique is known as **locally optimized product quantization**, since we're manipulating the local data within each coarse Voronoi cell in order to optimize the product quantization performance. The authors who introduced this technique have a great illustrative example of how this technique can better fit a given set of vectors. ![Screen-Shot-2019-01-31-at-11.52.37-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/02/Screen-Shot-2019-01-31-at-11.52.37-PM.png) This blog post glances over (c) Optimized Product Quantization which is the same idea of aligning our data for better product quantization performance, but the alignment is performed globally instead of aligning local data in each Voronoi cell independently. [Image credit](http://image.ntua.gr/iva/files/lopq.pdf?ref=jeremyjordan.me) #### A quick sidenote regarding PCA alignment The authors who introduced product quantization noted that the technique works best when the vector subcomponents had similar variance. A nice side effect of doing PCA alignment is that during the process we get a matrix of eigenvalues which describe the variance of each principal component. We can use this to our advantage by allocating principal components into buckets of equal variance. ![Screen-Shot-2019-02-03-at-11.32.59-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/02/Screen-Shot-2019-02-03-at-11.32.59-PM.png) ## Common datasets - [GLOVE word vectors](https://nlp.stanford.edu/projects/glove/?ref=jeremyjordan.me) - [SIFT image descriptors](http://corpus-texmex.irisa.fr/?ref=jeremyjordan.me) ## Further reading **Papers** - [Product quantization for nearest neighbor search](https://lear.inrialpes.fr/pubs/2011/JDS11/jegou%5Fsearching%5Fwith%5Fquantization.pdf?ref=jeremyjordan.me) - [Optimized Product Quantization for Approximate Nearest Neighbor Search](https://www.robots.ox.ac.uk/~vgg/rg/papers/ge%5F%5Fcvpr2013%5F%5Foptimized.pdf?ref=jeremyjordan.me) (see also this [presentation](http://kaiminghe.com/cvpr13/cvpr13opq%5Fppt.pdf?ref=jeremyjordan.me)) - [Locally Optimized Product Quantization for Approximate Nearest Neighbor Search](http://image.ntua.gr/iva/files/lopq.pdf?ref=jeremyjordan.me) - [The Inverted Multi-Index](https://cache-ash04.cdn.yandex.net/download.yandex.ru/company/cvpr2012.pdf?ref=jeremyjordan.me) I didn't cover binary codes in this post - but I should have! I *may* come back and edit the post to include more information soon. Until then, enjoy this paper. - [SUBIC: A supervised, structured binary code for image search](https://arxiv.org/pdf/1708.02932.pdf?ref=jeremyjordan.me) **Lectures** - [Victor Lavrenko - KNN Lectures](https://www.youtube.com/watch?v=gHdbqsDK9YY&list=PLBv09BD7ez%5F48heon5Az-TsyoXVYOJtDZ&ref=jeremyjordan.me) **Blog posts/talks** - [Nearest Neighbor Indexes for Similarity Search](https://www.pinecone.io/learn/vector-indexes/?ref=jeremyjordan.me) - [Scaling Visual Search with Locally Optimized Product Quantization](https://loukratz.info/talks/2018-08-09-lopq?ref=jeremyjordan.me) - [Indexing Billions of Text Vectors](https://0x65.dev/blog/2019-12-07/indexing-billions-of-text-vectors.html?ref=jeremyjordan.me) - [Product Quantizers for k-NN Tutorial Part 1](https://mccormickml.com/2017/10/13/product-quantizer-tutorial-part-1/?ref=jeremyjordan.me) - [Product Quantizers for k-NN Tutorial Part 2](https://mccormickml.com/2017/10/22/product-quantizer-tutorial-part-2/?ref=jeremyjordan.me) **Libraries and Github repos** - [Facebook AI Similarity Server (FAISS)](https://github.com/facebookresearch/faiss?ref=jeremyjordan.me) - [Spotify Annoy](https://github.com/spotify/annoy?ref=jeremyjordan.me) - [Yahoo LOPQ](https://github.com/yahoo/lopq?ref=jeremyjordan.me) - [Non-Metric Space Library (NMSlib)](https://github.com/nmslib/nmslib?ref=jeremyjordan.me) - [Scikit-learn Nearest Neighbors](https://scikit-learn.org/stable/modules/neighbors.html?ref=jeremyjordan.me) - [Github: ANN Benchmarks](https://github.com/erikbern/ann-benchmarks?ref=jeremyjordan.me) ### Organizing machine learning projects: project management guidelines. URL: https://www.jeremyjordan.me/ml-projects-guide/ Last updated: 2023-03-05T03:51:53.000Z The goal of this document is to provide a common framework for approaching machine learning projects that can be referenced by practitioners. If you build ML models, this post is for you. If you collaborate with people who build ML models, I hope that this guide provides you with a good perspective on the common project workflow. Knowledge of machine learning is assumed. ## Overview This overview intends to serve as a project "checklist" for machine learning practitioners. Subsequent sections will provide more detail. **Project lifecycle** Machine learning projects are highly iterative; as you progress through the ML lifecycle, you’ll find yourself iterating on a section until reaching a satisfactory level of performance, then proceeding forward to the next task (which may be circling back to an even earlier step). Moreover, a project isn’t complete after you ship the first version; you get feedback from real-world interactions and redefine the goals for the next iteration of deployment. ![ml-development-cycle](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/09/ml-development-cycle.png) 1. [**Planning and project setup**](#planning) - Define the task and scope out requirements - Determine project feasibility - Discuss general model tradeoffs (accuracy vs speed) - Set up project codebase 2. [**Data collection and labeling**](#data) - Define ground truth (create labeling documentation) - Build data ingestion pipeline - Validate quality of data - Label data and ensure ground truth is well-definend - Revisit Step 1 and ensure data is sufficient for the task 3. [**Model exploration**](#exploration) - Establish baselines for model performance - Start with a simple model using initial data pipeline - Overfit simple model to training data - Stay nimble and try many parallel (isolated) ideas during early stages - Find SoTA model for your problem domain (if available) and reproduce results, then apply to your dataset as a second baseline - Revisit Step 1 and ensure feasibility - Revisit Step 2 and ensure data quality is sufficient 4. [**Model refinement**](#refinement) - Perform model-specific optimizations (ie. hyperparameter tuning) - Iteratively debug model as complexity is added - Perform error analysis to uncover common failure modes - Revisit Step 2 for targeted data collection and labeling of observed failure modes 5. [**Testing and evaluation**](#testing) - Evaluate model on test distribution; understand differences between train and test set distributions (how is “data in the wild” different than what you trained on) - Revisit model evaluation metric; ensure that this metric drives desirable downstream user behavior - Write tests for: - Input data pipeline - Model inference functionality - Model inference performance on validation data - Explicit scenarios expected in production (model is evaluated on a curated set of observations) 6. [**Model deployment**](#deployment) - Expose model via a REST API - Deploy new model to small subset of users to ensure everything goes smoothly, then roll out to all users - Maintain the ability to roll back model to previous versions - Monitor live data and model prediction distributions 7. [**Ongoing model maintenance**](#maintenance) - Understand that changes can affect the system in unexpected ways - Periodically retrain model to prevent model staleness - If there is a transfer in model ownership, educate the new team **Team roles** A typical team is composed of: - **data engineer** (builds the data ingestion pipelines) - **machine learning engineer** (train and iterate models to perform the task) - **software engineer** (aids with integrating machine learning model with the rest of the product) - **project manager** (main point of contact with the client) --- ## Planning and project setup It may be tempting to skip this section and dive right in to "just see what the models can do". Don't skip this section. All too often, you'll end up wasting time by delaying discussions surrounding the project goals and model evaluation criteria. Everyone should be working toward a common goal from the start of the project. It's worth noting that defining the model task is not always straightforward. There's often many different approaches you can take towards solving a problem and it's not always immediately evident which is optimal. If your problem is vague and the modeling task is not clear, jump over to my post on [defining requirements for machine learning projects](https://www.jeremyjordan.me/ml-requirements/) before proceeding. ### Prioritizing projects *Ideal: project has high impact and high feasibility.* Mental models for evaluating project impact: - Look for places where cheap prediction drives large value - Look for complicated rule-based software where we can learn rules instead of programming them When evaluating projects, it can be useful to have a common language and understanding of the differences between traditional software and machine learning software. Andrej Karparthy's [Software 2.0](https://medium.com/@karpathy/software-2-0-a64152b37c35?ref=jeremyjordan.me) is recommended reading for this topic. **Software 1.0** - Explicit instructions for a computer written by a programmer using a *programming language* such as Python or C++. A human writes the logic such that when the system is provided with data it will output the desired behavior. ![Screen-Shot-2019-05-07-at-9.45.18-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/05/Screen-Shot-2019-05-07-at-9.45.18-PM.png) **Software 2.0** - Implicit instructions by providing data, "written" by an optimization algorithm using *parameters* of a specified model architecture. The system logic is learned from a provided collection of data examples and their corresponding desired behavior. ![Screen-Shot-2019-05-07-at-9.44.55-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/05/Screen-Shot-2019-05-07-at-9.44.55-PM.png) See [this talk](https://www.youtube.com/watch?v=zywIvINSlaI&ref=jeremyjordan.me) for more detail. A quick note on Software 1.0 and Software 2.0 - these two paradigms are **not** mutually exclusive. Software 2.0 is usually used to scale the **logic** component of traditional software systems by leveraging large amounts of data to enable more complex or nuanced decision logic. ![Screen-Shot-2019-05-10-at-9.52.20-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/05/Screen-Shot-2019-05-10-at-9.52.20-PM.png) For example, [Jeff Dean talks](https://twimlai.com/twiml-talk-124-systems-software-machine-learning-scale-jeff-dean/?ref=jeremyjordan.me) (at 27:15) about how the code for Google Translate used to be a very complicated system consisting of \~500k lines of code. Google was able to simplify this product by leveraging a machine learning model to perform the core logical task of translating text to a different language, requiring only \~500 lines of code to describe the model. However, this model still requires some "Software 1.0" code to process the user's query, invoke the machine learning model, and return the desired information to the user. In summary, machine learning can drive large value in applications where decision logic is difficult or complicated for humans to write, but relatively easy for machines to learn. On that note, we'll continue to the next section to discuss how to evaluate whether a task is "relatively easy" for machines to learn. ### Determining feasibility Some useful questions to ask when determining the feasibility of a project: - Cost of data acquisition - How hard is it to acquire data? - How expensive is data labeling? - How much data will be needed? - Cost of wrong predictions - How frequently does the system need to be right to be useful? - Are there scenarios where a wrong prediction incurs a large cost? - Availability of good published work about similar problems - Has the problem been reduced to practice? - Is there sufficient literature on the problem? - Are there pre-trained models we can leverage? - Computational resources available both for training and inference - Will the model be deployed in a resource-constrained environment? - What are the latency requirements for the model? ### Specifying project requirements Establish a single value optimization metric for the project. Can also include several other [satisficing](https://en.wikipedia.org/wiki/Satisficing?ref=jeremyjordan.me) metrics (ie. performance thresholds) to evaluate models, but can only **optimize** a single metric. *Example:* - Optimize for accuracy - Prediction latency under 10 ms - Model requires no more than 1gb of memory - 90% coverage (model confidence exceeds required threshold to consider a prediction as valid) The optimization metric may be a weighted sum of many things which we care about. Revisit this metric as performance improves. Some teams may choose to ignore a certain requirement at the start of the project, with the goal of revising their solution (to meet the ignored requirements) after they have discovered a promising general approach. Decide at what point you will ship your first model. > Some teams aim for a “neutral” first launch: a first launch that explicitly deprioritizes machine learning gains, to avoid getting distracted. — [Google Rules of Machine Learning](http://martin.zinkevich.org/rules%5Fof%5Fml/rules%5Fof%5Fml.pdf?ref=jeremyjordan.me) The motivation behind this approach is that the first deployment should involve a simple model with focus spent on building the proper machine learning pipeline required for prediction. This allows you to deliver value quickly and avoid the trap of spending too much of your time trying to ["squeeze the juice."](http://karpathy.github.io/2019/04/25/recipe/?ref=jeremyjordan.me#6-squeeze-out-the-juice) ### Setting up a ML codebase A well-organized machine learning codebase should modularize data processing, model definition, model training, and experiment management. Example codebase organization: ```bash configs/ baseline.yaml latest.yaml data/ docker/ project_name/ api/ app.py models/ base.py simple_baseline.py cnn.py datasets.py train.py experiment.py scripts/ ``` `data/` provides a place to store raw and processed data for your project. You can also include a `data/README.md` file which describes the data for your project. `docker/` is a place to specify one or many Dockerfiles for the project. Docker (and other container solutions) help ensure consistent behavior across multiple machines and deployments. `api/app.py` exposes the model through a REST client for predictions. You will likely choose to load the (trained) model from a [model registry](https://mlflow.org/docs/latest/model-registry.html?ref=jeremyjordan.me) rather than importing directly from your library. `models/` defines a collection of machine learning models for the task, unified by a common API defined in `base.py`. These models include code for any necessary data preprocessing and output normalization. `datasets.py` manages construction of the dataset. Handles data pipelining/staging areas, shuffling, reading from disk. `experiment.py` manages the experiment process of evaluating multiple models/ideas. This constructs the dataset and models for a given experiment. `train.py` defines the actual training loop for the model. This code interacts with the optimizer and handles logging during training. See other examples [here](https://github.com/jeremyjordan/data-science-template?ref=jeremyjordan.me), [here](https://github.com/ml-tooling/ml-project-template?ref=jeremyjordan.me), [here](https://github.com/cmawer/reproducible-model?ref=jeremyjordan.me) and [here](https://drivendata.github.io/cookiecutter-data-science/?ref=jeremyjordan.me#directory-structure). --- ## Data collection and labeling An ideal machine learning pipeline uses data which labels itself. For example, Tesla Autopilot has a model running that predicts when cars are about to [cut into your lane](https://www.youtube.com/watch?v=Ucp0TTmvqOE&feature=youtu.be&t=7809&ref=jeremyjordan.me). In order to acquire labeled data in a systematic manner, you can simply observe when a car changes from a neighboring lane into the Tesla's lane and then rewind the video feed to label that a car is about to cut in to the lane. As another example, suppose Facebook is building a model to predict user engagement when deciding how to order things on the newsfeed. After serving the user content based on a prediction, they can monitor engagement and turn this interaction into a labeled observation without any human effort. However, just be sure to think through this process and ensure that your "self-labeling" system won't get stuck in a *feedback loop* with itself. For many other cases, we must manually label data for the task we wish to automate. The quality of your data labels has a *large* effect on the upper bound of model performance. > Here is a real use case from work for model improvement and the steps taken to get there: > > \- Baseline: 53% > \- Logistic: 58% > \- Deep learning: 61% > \- \*\*Fixing your data: 77%\*\* > > Some good ol' fashion "understanding your data" is worth it's weight in hyperparameter tuning! > > — Alex Gude (@alex\_gude) [April 24, 2019](https://twitter.com/alex%5Fgude/status/1121138827601383426?ref%5Fsrc=twsrc%5Etfw&ref=jeremyjordan.me) Most data labeling projects require multiple people, which necessitates labeling **documentation**. Even if you're the only person labeling the data, it makes sense to document your labeling criteria so that you maintain consistency. One tricky case is where you decide to change your labeling methodology after already having labeled data. For example, in the Software 2.0 talk mentioned previously, Andrej Karparthy [talks about](https://www.youtube.com/watch?v=zywIvINSlaI&feature=youtu.be&t=20m43s&ref=jeremyjordan.me) data which has no clear and obvious ground truth. ![lane_lines](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/09/lane_lines.png) *[Image credit](https://www.figure-eight.com/wp-content/uploads/2018/06/TRAIN%5FAI%5F2018%5FAndrej%5FKarpathy%5FTesla.pdf?ref=jeremyjordan.me)* If you run into this, *tag "hard-to-label" examples* in some manner such that you can easily find all similar examples should you decide to change your labeling methodology down the road. Additionally, you should **version your dataset** and associate a given model with a dataset version. *Tip: After labeling data and training an initial model, look at the observations with the largest error. These examples are often poorly labeled.* **Active learning** Active learning is useful when you have a large amount of unlabeled data and you need to decide what data you should label. Labeling data can be expensive, so we'd like to limit the time spent on this task. *As a counterpoint, if you can afford to label your entire dataset, you probably should. Active learning adds another layer of complexity.* > "The main hypothesis in active learning is that if a learning algorithm can choose the data it wants to learn from, it can perform better than traditional methods with substantially less data for training." - [DataCamp](https://www.datacamp.com/community/tutorials/active-learning?ref=jeremyjordan.me) General approach: 1. Starting with an unlabeled dataset, build a "seed" dataset by acquiring labels for a small subset of instances 2. Train initial model on the seed dataset 3. Predict the labels of the remaining unlabeled observations 4. Use the uncertainty of the model's predictions to prioritize the labeling of remaining observations **Leveraging weak labels** However, tasking humans with generating ground truth labels is expensive. Often times you'll have access to large swaths of unlabeled data and a limited labeling budget - how can you maximize the value from your data? In some cases, your data can have information which provides a noisy estimate of the ground truth. For example, [if you're categorizing Instagram photos, you might have access to the hashtags used in the caption of the image](https://code.fb.com/ml-applications/advancing-state-of-the-art-image-recognition-with-deep-learning-on-hashtags/?ref=jeremyjordan.me). Other times, you might have subject matter experts which can help you develop heuristics about the data. [Snorkel](https://hazyresearch.github.io/snorkel/?ref=jeremyjordan.me) is an interesting project produced by the Stanford DAWN (Data Analytics for What’s Next) lab which formalizes an approach towards combining many noisy label estimates into a probabilistic ground truth. I'd encourage you to check it out and see if you might be able to leverage the approach for your problem. --- ## Model exploration **Establish performance baselines on your problem.** Baselines are useful for both establishing a lower bound of expected performance (simple model baseline) and establishing a target performance level (human baseline). - Simple baselines include out-of-the-box scikit-learn models (i.e. logistic regression with default parameters) or even simple heuristics (always predict the majority class). Without these baselines, it's impossible to evaluate the value of added model complexity. - If your problem is well-studied, search the literature to approximate a baseline based on published results for very similar tasks/datasets. - If possible, try to estimate human-level performance on the given task. Don't naively assume that humans will perform the task perfectly, a lot of simple tasks are [deceptively hard](http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/?ref=jeremyjordan.me)! **Start simple and gradually ramp up complexity.** This typically involves using a simple model, but can also include starting with a simpler version of your task. > Before doing anything intelligent with "AI", do the unintelligent version fast and at scale. > At worst you understand the limits of a simplistic approach and what complexities you need to handle. > At best you realize you don't need the overhead of intelligence. > > — Smerity (@Smerity) [February 13, 2019](https://twitter.com/Smerity/status/1095490777860304896?ref%5Fsrc=twsrc%5Etfw&ref=jeremyjordan.me) **Once a model runs, overfit a single batch of data.** Don't use regularization yet, as we want to see if the unconstrained model has sufficient capacity to learn from the data. - [Practical Advice for Building Deep Neural Networks](https://pcc.cs.byu.edu/2017/10/02/practical-advice-for-building-deep-neural-networks/?ref=jeremyjordan.me) (see case study on overfitting an initial model) **Survey the literature.** Search for papers on Arxiv describing model architectures for similar problems and speak with other practitioners to see which approaches have been most successful in practice. Determine a *state of the art* approach and use this as a baseline model (trained on your dataset). **Reproduce a known result.** If you're using a model which has been well-studied, ensure that your model's performance *on a commonly-used dataset* matches what is reported in the literature. **Understand how model performance scales with more data.** Plot the model performance as a function of increasing dataset size for the baseline models that you've explored. Observe how each model's performance scales as you increase the amount of data used for training. --- ## Model refinement Once you have a general idea of successful model architectures and approaches for your problem, you should now spend much more focused effort on squeezing out performance gains from the model. **Build a scalable data pipeline.** By this point, you've determined which types of data are necessary for your model and you can now focus on engineering a performant pipeline. **Apply the bias variance decomposition to determine next steps.** Break down error into: irreducible error, avoidable bias (difference between train error and irreducible error), variance (difference between validation error and train error), and validation set overfitting (difference between test error and validation error). - If training on a (known) different distribution than what is available at test time, consider having *two validation subsets*: val-train and val-test. The difference between val-train error and val-test error is described by distribution shift. - *Addressing underfitting*: 1. Increase model capacity 2. Reduce regularization 3. Error analysis 4. Choose a more advanced architecture (closer to state of art) 5. Tune hyperparameters 6. Add features - *Addressing overfitting*: 1. Add more training data 2. Add regularization 3. Add data augmentation 4. Error analysis 5. Tune hyperparameters 6. Reduce model size - *Addressing distribution shift*: 1. Perform error analysis to understand nature of distribution shift 2. Synthesize data (by augmentation) to more closely match the test distribution 3. Apply domain adaptation techniques **Use coarse-to-fine random searches for hyperparameters.** Start with a wide hyperparameter space initially and iteratively hone in on the highest-performing region of the hyperparameter space. - [Hyperparameter tuning for machine learning models](https://www.jeremyjordan.me/hyperparameter-tuning/). **Perform targeted collection of data to address current failure modes.** Develop a systematic method for analyzing errors of your current model. Categorize these errors, if possible, and collect additional data to better cover these cases. ### Debugging ML projects Why is your model performing poorly? - Implementation bugs - Hyperparameter choices - Data/model fit - Dataset construction *Key mindset for DL troubleshooting: pessimism.* In order to complete machine learning projects efficiently, **start simple** and gradually increase complexity. Start with a solid foundation and build upon it in an incremental fashion. *Tip: Fix a [random](https://towardsdatascience.com/properly-setting-the-random-seed-in-machine-learning-experiments-7da298d1320b?ref=jeremyjordan.me) [seed](https://pytorch.org/docs/stable/notes/randomness.html?ref=jeremyjordan.me) to ensure your model training is reproducible.* Common bugs: > oh: 5) you didn't use bias=False for your Linear/Conv2d layer when using BatchNorm, or conversely forget to include it for the output layer .This one won't make you silently fail, but they are spurious parameters > > — Andrej Karpathy (@karpathy) [July 1, 2018](https://twitter.com/karpathy/status/1013245864570073090?ref%5Fsrc=twsrc%5Etfw&ref=jeremyjordan.me) ### Discovering failure modes Use clustering to uncover failure modes and improve error analysis: - Select all incorrect predictions. (Optionally, sort your observations by their calculated loss to find the most egregious errors.) - Run a clustering algorithm such as DBSCAN across selected observations. - Manually explore the clusters to look for common attributes which make prediction difficult. Categorize observations with incorrect predictions and determine what best action can be taken in the model refinement stage in order to improve performance on these cases. --- ## Testing and evaluation If you haven't already written tests for your code yet, you should write them at this point. Different components of a ML product to test: - **Training system** processes raw data, runs experiments, manages results, stores weights. - *Required tests*: - Test the full training pipeline (from raw data to trained model) to ensure that changes haven't been made upstream with respect to how data from our application is stored. These tests should be run nightly/weekly. - **Prediction system** constructs the network, loads the stored weights, and makes predictions. - *Required tests*: - Run inference on the validation data (already processed) and ensure model score does not degrade with new model/weights. This should be triggered every code push. - You should also have a quick functionality test that runs on a few important examples so that you can quickly (<5 minutes) ensure that you haven't broken functionality during development. These tests are used as a sanity check as you are writing new code. - Also consider scenarios that your model might encounter, and develop tests to ensure new models still perform sufficiently. The "test case" is a scenario defined by the human and represented by a curated set of observations. - *Example: For a self driving car, you might have a test to ensure that the car doesn't turn left at a yellow light. For this case, you may run your model on observations where the car is at a yellow light and ensure that the prediction doesn't tell the car to proceed forward.* - **Serving system** exposed to accept "real world" input and perform inference on production data. This system must be able to scale to demand. - *Required monitoring*: - Alerts for downtime and errors - Check for distribution shift in data ![test_infra](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/09/test_infra.png) *[Image credit](https://ai.google/research/pubs/pub46555?ref=jeremyjordan.me)* ### Evaluating production readiness [The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction](https://ai.google/research/pubs/pub46555?ref=jeremyjordan.me) *Data:* - Feature expectations are captured in a schema. - All features are beneficial. - No feature’s cost is too much. - Features adhere to meta-level requirements. - The data pipeline has appropriate privacy controls. - New features can be added quickly. - All input feature code is tested. *Model:* - Model specs are reviewed and submitted. - Offline and online metrics correlate. - All hyperparameters have been tuned. - The impact of model staleness is known. - A simple model is not better. - Model quality is sufficient on important data slices. - The model is tested for considerations of inclusion. *Infrastructure:* - Training is reproducible. - Model specs are unit tested. - The ML pipeline is integration tested. - Model quality is validated before serving. - The model is debuggable. - Models are canaried before serving. - Serving models can be rolled back. *Monitoring:* - Dependency changes result in notification. - Data invariants hold for inputs. - Training and serving are not skewed. - Models are not too stale. - Models are numerically stable. - Computing performance has not regressed. - Prediction quality has not regressed. --- ## Model deployment Be sure to have a versioning system in place for: - Model parameters - Model configuration - Feature pipeline - Training dataset - Validation dataset A common way to deploy a model is to package the system into a Docker container and expose a REST API for inference. **Canarying**: Serve new model to a small subset of users (ie. 5%) while still serving the existing model to the remainder. Check to make sure rollout is smooth, then deploy new model to rest of users. **Shadow mode:** Ship a new model alongside the existing model, still using the existing model for predictions but storing the output for both models. Measuring the delta between the new and current model's predictions will give an indication for how drastically things will change when you switch to the new model. --- ## Ongoing model maintenance [Hidden Technical Debt in Machine Learning Systems](https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf?ref=jeremyjordan.me) (quoted below, emphasis mine) A primer on concept of technical debt: > As with fiscal debt, there are often sound strategic reasons to take on technical debt. **Not all debt is bad, but all debt needs to be serviced.** Technical debt may be paid down by refactoring code, improving unit tests, deleting dead code, reducing dependencies, tightening APIs, and improving documentation. The goal is not to add new functionality, but to enable future improvements, reduce errors, and improve maintainability. **Deferring such payments results in compounding costs.** Hidden debt is dangerous because it compounds silently. Machine learning projects are not complete upon shipping the first version. If you are "handing off" a project and transferring model responsibility, it is extremely important to talk through the required model maintenance with the new team. > Developing and deploying ML systems is relatively fast and cheap, but maintaining them over time is difficult and expensive. ### Maintenance principles **CACE principle: Changing Anything Changes Everything** Machine learning systems are tightly coupled. Changes to the feature space, hyper parameters, learning rate, or any other "knob" can affect model performance. *Specific mitigation strategies:* - Create model validation tests which are run every time new code is pushed. - Decompose problems into *isolated* components where it makes sense to do so. **Undeclared consumers of your model may be inadvertently affected by your changes.** > "Without access controls, it is possible for some of these consumers to be undeclared consumers, consuming the output of a given prediction model as an input to another component of the system." If your model and/or its predictions are widely accessible, other components within your system may grow to depend on your model without your knowledge. Changes to the model (such as periodic retraining or redefining the output) may negatively affect those downstream components. *Specific mitigation strategies:* - Control access to your model by making outside components request permission and signal their usage of your model. **Avoid depending on input signals which may change over time.** Some features are obtained by a table lookup (ie. word embeddings) or simply an input pipeline which is outside the scope of your codebase. When these external feature representations are changed, the model's performance can suffer. *Specific mitigation strategies:* - Create a versioned copy of your input signals to provide stability against changes in external input pipelines. These versioned inputs can be specified in a model's configuration file. **Eliminate unnecessary features.** Regularly evaluate the effect of removing individual features from a given model. A model's feature space should only contain relevant and important features for the given task. There are many strategies to determine feature importances, such as leave-one-out cross validation and feature permutation tests. Unimportant features add noise to your feature space and should be removed. *Tip: Document deprecated features (deemed unimportant) so that they aren't accidentally reintroduced later.* **Model performance will likely decline over time.** As the input distribution shifts, the model's performance will suffer. You should plan to periodically retrain your model such that it has always learned from recent "real world" data. --- This guide draws inspiration from the [Full Stack Deep Learning Bootcamp](https://fullstackdeeplearning.com/?ref=jeremyjordan.me), [best practices released by Google](http://martin.zinkevich.org/rules%5Fof%5Fml/rules%5Fof%5Fml.pdf?ref=jeremyjordan.me), my personal experience, and conversations with fellow practitioners. Find something that's missing from this guide? Let me know! ## External Resources **Blog posts** - [A Recipe for Training Neural Networks](https://karpathy.github.io/2019/04/25/recipe/?ref=jeremyjordan.me) - [How to put machine learning models into production](https://stackoverflow.blog/2020/10/12/how-to-put-machine-learning-models-into-production/?ref=jeremyjordan.me) - [Designing collaborative AI](https://medium.com/@Ben%5FReinhardt/designing-collaborative-ai-5c1e8dbc8810?ref=jeremyjordan.me) (clever product design can reduce model performance requirements) - [Data project checklist - Jeremy Howard](https://www.fast.ai/2020/01/07/data-questionnaire/?ref=jeremyjordan.me) - [Checklist for debugging neural networks](https://towardsdatascience.com/checklist-for-debugging-neural-networks-d8b2a9434f21?ref=jeremyjordan.me) - [Troubleshooting Deep Neural Networks](http://josh-tobin.com/troubleshooting-deep-neural-networks.html?ref=jeremyjordan.me) - [Nitpicking Machine Learning Technical Debt](https://matthewmcateer.me/blog/machine-learning-technical-debt/?ref=jeremyjordan.me) - [Accelerate Machine Learning with Active Learning](https://becominghuman.ai/accelerate-machine-learning-with-active-learning-96cea4b72fdb?ref=jeremyjordan.me) **Papers** - [Managing Machine Learning Projects](https://d1.awsstatic.com/whitepapers/aws-managing-ml-projects.pdf?ref=jeremyjordan.me) - [Active Learning Literature Survey](http://burrsettles.com/pub/settles.activelearning.pdf?ref=jeremyjordan.me) **Case studies** - [Using machine learning to predict what file you need next](https://dropbox.tech/machine-learning/content-suggestions-machine-learning?ref=jeremyjordan.me) - [A better clickthrough rate: How Pinterest upgraded everyone’s favorite engagement metric](https://medium.com/pinterest-engineering/a-better-clickthrough-rate-how-pinterest-upgraded-everyones-favorite-engagement-metric-27f6fa6cba14?ref=jeremyjordan.me) **Talks** - [Leading Data Science Teams: A Framework To Help Guide Data Science Project Managers - Jeffrey Saltz](https://www.youtube.com/watch?v=C5JElgliTeE&ref=jeremyjordan.me) - [An Only One Step Ahead Guide for Machine Learning Projects - Chang Lee](https://www.youtube.com/watch?v=7D8unG3XMzU&ref=jeremyjordan.me) - An entertaining talk discussing advice for approaching machine learning projects. This talk will give you a "flavor" for the details covered in this guide. - [Microsoft Research: Active Learning and Annotation](https://www.youtube.com/watch?v=FE1r7%5FSQq6Y&ref=jeremyjordan.me) ### An overview of object detection: one-stage methods. URL: https://www.jeremyjordan.me/object-detection-one-stage/ Last updated: 2023-03-05T03:53:38.000Z In this post, I'll discuss an overview of deep learning techniques for **object detection** using [convolutional neural networks](https://www.jeremyjordan.me/convolutional-neural-networks/). Object detection is useful for understanding what's in an image, describing both *what* is in an image and *where* those objects are found. In general, there's two different approaches for this task – we can either make a fixed number of predictions on grid (one stage) **or** leverage a proposal network to find objects and then use a second network to fine-tune these proposals and output a final prediction (two stage). In this blog post, I'll discuss the one-stage approach towards object detection; a follow-up post will then discuss the two-stage approach. Each approach has its own strengths and weaknesses, which I'll discuss in the respective blog posts. **Jump to:** - [Understanding the task](#understanding) - [Direct object prediction](#direct%5Fprediction) - [Predictions on a grid](#grid) - [Non-maximum suppression](#nonmax%5Fsuppression) - [YOLO: You Only Look Once](#yolo) - [SSD: Single Shot Detection](#ssd) - [Addressing object imbalance with focal loss](#focal) - [Common datasets and competitions](#datasets) - [Further reading](#further%5Freading) ## Understanding the task The goal of object detection is to recognize instances of a predefined set of object classes (e.g. {people, cars, bikes, animals}) and describe the locations of each detected object in the image using a *bounding box*. Two examples are shown below. ![Screen-Shot-2018-06-29-at-2.52.10-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/06/Screen-Shot-2018-06-29-at-2.52.10-PM.png) [Example images are taken from the PASCAL VOC dataset.](http://host.robots.ox.ac.uk/pascal/VOC/?ref=jeremyjordan.me) We'll use rectangles to describe the locations of each object, which may lead to imperfect localizations due to the shapes of objects. An alternative approach would be [image segmentation](https://www.jeremyjordan.me/semantic-segmentation/) which provides localization at the pixel-level. ## Direct object prediction This blog post will focus on model architectures which directly predict object bounding boxes for an image in a **one-stage** fashion. In other words, there is no intermediate task (as we'll discuss later with region proposals) which must be performed in order to produce an output. This leads to a simpler and faster model architecture, although it can sometimes struggle to be flexible enough to adapt to arbitrary tasks (such as mask prediction). #### Predictions on a grid In order to understand what's in an image, we'll feed our input through a [standard convolutional network](https://www.jeremyjordan.me/convnet-architectures/) to build a **rich feature representation** of the original image. We'll refer to this part of the architecture as the *"backbone" network*, which is usually pre-trained as an image classifier to more cheaply learn how to *extract features from an image*. This is a result of the fact that data for image classification is easier (and thus cheaper) to label as it only requires a single label as opposed to defining bounding box annotations for each image. Thus, we can train on a very large labeled dataset (such as ImageNet) in order to learn good feature representations. ![backbone architecture](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/07/Screen-Shot-2018-07-01-at-11.25.31-PM.png) After pre-training the backbone architecture as an image classifier, we'll remove the last few layers of the network so that our backbone network outputs a collection of stacked feature maps which describe the original image in a *low spatial resolution* albeit a *high feature (channel) resolution*. In the example below, we have a 7x7x512 representation of our observation. Each of the 512 feature maps describe different characteristics of the original image. ![grid output](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/07/Screen-Shot-2018-07-01-at-11.29.17-PM.png) We can relate this 7x7 grid back to the original input in order to understand what each grid cell represents relative to the original image. ![feature map to receptive field](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/07/Screen-Shot-2018-07-02-at-10.42.33-PM.png) We can also determine *roughly* where objects are located in the coarse (7x7) feature maps by observing which grid cell contains the center of our bounding box annotation. We'll assign this grid cell as being "responsible" for detecting that specific object. ![bounding box center](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/07/Screen-Shot-2018-07-10-at-11.04.51-PM.png) In order to detect this object, we will add another convolutional layer and learn the kernel parameters which combine the context of all 512 feature maps in order to produce an activation corresponding with the grid cell which contains our object. ![convolution operation](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/07/Screen-Shot-2018-07-02-at-9.52.11-PM.png) If the input image contains multiple objects, we should have multiple activations on our grid denoting that an object is in each of the activated regions. ![multiple objects](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/07/Screen-Shot-2018-07-02-at-9.54.18-PM.png) However, we cannot sufficiently describe each object with a single activation. In order to fully describe a detected object, we'll need to define: - The likelihood that a grid cell contains an object ($p\_{obj}$) - Which class the object belongs to ($c\_1$, $c\_2$, ..., $c\_C$) - Four bounding box descriptors to describe the $x$ coordinate, $y$ coordinate, width, and height of a labeled box ($t\_x$, $t\_y$, $t\_w$, $t\_h$) Thus, we'll need to learn a convolution filter for *each* of the above attributes such that we produce $5 + C$ output channels to describe a *single bounding box* at each grid cell location. This means that we'll learn a set of weights to look across all 512 feature maps and determine which grid cells are likely to contain an object, what classes are likely to be present in each grid cell, and how to describe the bounding box for possible objects in each grid cell. ![one bounding box](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/07/Screen-Shot-2018-07-02-at-10.00.39-PM.png) The full output of applying $5 + C$ convolutional filters is shown below for clarity, producing one bounding box descriptor for each grid cell. ![bounding box all grid cells](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/07/Screen-Shot-2018-07-05-at-1.05.06-PM.png) However, some images might have multiple objects which "belong" to the same grid cell. We can alter our layer to produce $B(5 + C)$ filters such that we can predict $B$ bounding boxes for each grid cell location. ![two bounding boxes](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/07/Screen-Shot-2018-07-02-at-10.13.04-PM.png) Visualizing the full convolutional output of our $B(5 + C)$ filters, we can see that our model will always produce a fixed number of $N \\times N \\times B$ predictions for a given image. We can then filter our predictions to only consider bounding boxes which has a $p\_{obj}$ above some defined threshold. ![full grid one object](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/07/Screen-Shot-2018-07-02-at-10.58.21-PM.png) Because of the convolutional nature of our detection process, **multiple objects can be detected in parallel**. However, we also end up predicting for a large number grid cells where no object is found. Although we can filter these bounding boxes out by their $p\_{obj}$ score, this introduces quite a large imbalance between the predicted bounding boxes which contain an object and those which do not contain an object. ![full grid two objects](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/07/Screen-Shot-2018-07-02-at-10.58.30-PM.png) The two models I'll discuss below both use this concept of **"predictions on a grid"** to detect a fixed number of possible objects within an image. In the respective sections, I'll describe the nuances of each approach and fill in some of the details that I've glanced over in this section so that you can actually implement each model. #### Non-maximum suppression The "predictions on a grid" approach produces a fixed number of bounding box predictions for each image. However, we would like to filter these predictions in order to only output bounding boxes for objects that are actually likely to be in the image. Moreover, we want a single bounding box prediction for each object detected. We can filter out most of the bounding box predictions by only considering predictions with a $p\_{obj}$ above some defined confidence threshold. However, we still may be left with multiple high-confidence predictions describing the *same* object. Thus, we need a method for removing redundant object predictions such that each object is described by a single bounding box. To accomplish this, we'll use a technique known as **non-max suppression**. At a high level, this technique will look at highly overlapping bounding boxes and suppress (or discard) all of the predictions except the highest confidence prediction. ![NMS steps](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/07/Screen-Shot-2018-07-10-at-9.46.29-PM.png) We'll perform non-max suppression on **each class separately**. Again, the goal here is to remove *redundant* predictions so we shouldn't be concerned if we have two predictions that overlap if one box is describing a person and the other box is describing a bicycle. However, if two bounding boxes with high overlap are both describing a person, it's likely that these predictions are describing the same person. ## YOLO: You Only Look Once The YOLO model was first published (by Joseph Redmon et al.) in 2015 and subsequently revised in two following papers. In each section, I'll discuss the specific implementation details and refinements that were made to improve performance. #### Backbone network The original YOLO network uses a modified GoogLeNet as the backbone network. Redmond later created a new model named DarkNet-19 which follows the general design of a $3 \\times 3$ filters, doubling the number of channels at each pooling step; $1 \\times 1$ filters are also used to periodically compress the feature representation throughout the network. His latest paper introduces a new, larger model named DarkNet-53 which offers improved performance over its predecessor. All of these models were first pre-trained as image classifiers before being adapted for the detection task. In the second iteration of the YOLO model, Redmond discovered that using higher resolution images at the end of classification pre-training improved the detection performance and thus adopted this practice. Adapting the classification network for detection simply consists of removing the last few layers of the network and adding a convolutional layer with $B(5 + C)$ filters to produce the $N \\times N \\times B$ bounding box predictions. #### Bounding boxes (and concept of anchor boxes) The first iteration of the YOLO model **directly predicts** all four values which describe a bounding box. The $x$ and $y$ coordinates of each bounding box are defined relative to the top left corner of each grid cell and normalized by the cell dimensions such that the coordinate values are bounded between 0 and 1\. We define the boxes width and height such that our model predicts the *square-root* width and height; by defining the width and height of the boxes as a square-root value, differences between large numbers are less significant than differences between small numbers (confirm this visually by looking at a plot of $y = \\sqrt {x}$). Redmond chose this formulation because “small deviations in large boxes matter less than in small boxes" and thus when calculating our loss function we would like the emphasis to be placed on getting small boxes more exact. The bounding box width and height are normalized by the image width and height and thus are also bounded between 0 and 1\. An L2 loss is applied during training. This formulation was later revised to introduce the concept of a **bounding box prior**. Rather than expecting the model to directly produce unique bounding box descriptors for each new image, we will define a collection of bounding boxes with varying aspect ratios which embed some prior information about the shape of objects we're expecting to detect. Redmond offers an approach towards discovering the best aspect ratios by doing k-means clustering (with a custom distance metric) on all of the bounding boxes in your training dataset. In the image below, you can see a collection of 5 bounding box priors (also known as anchor boxes) for the grid cell highlighted in yellow. With this formulation, each of the $B$ bounding boxes explicitly specialize in detecting objects of a specific size and aspect ratio. ![bounding box prior](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/07/download--3-.png) *Note: Although it is not visualized, these anchor boxes are present for each cell in our prediction grid.* Rather than directly predicting the bounding box dimensions, we'll reformulate our task in order to simply predict the *offset* from our bounding box prior dimensions such that we can fine-tune our predicted bounding box dimensions. This reformulation makes the prediction task easier to learn. ![dimension refinement](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/07/Screen-Shot-2018-07-09-at-12.36.54-PM.png) For similar reasons as originally predicting the square-root width and height, we'll define our task to predict the *log offsets* from our bounding box prior. #### Objectness (and assigning labeled objects to a bounding box) In the first version of the model, the "objectness" score $p\_{obj}$ was trained to approximate the Intersection over Union (IoU) between the predicted box and the ground truth label. When we calculate our loss during training, we'll match objects to whichever bounding box prediction (on the same grid cell) has the highest IoU score. For unmatched boxes, the only descriptor which we'll include in our loss function is $p\_{obj}$. After the addition bounding box priors in YOLOv2, we can simply assign labeled objects to whichever anchor box (on the same grid cell) has the highest IoU score with the labeled object. In the third version, Redmond redefined the "objectness" target score $p\_{obj}$ to be 1 for the bounding boxes with highest IoU score for each given target, and 0 for all remaining boxes. However, we will not include bounding boxes which have a high IoU score (above some threshold) but not the highest score when calculating the loss. In simple terms, it doesn't make sense to punish a good prediction just because it isn't the *best* prediction. #### Class labels Originally, class prediction was performed at the *grid cell* level. This means that a single grid cell could not predict multiple bounding boxes of different classes. This was later revised to predict class for each bounding box using a softmax activation across classes and a cross entropy loss. Redmond later changed the class prediction to use sigmoid activations for multi-label classification as he found a softmax is not necessary for good performance. This choice will depend on your dataset and whether or not your labels overlap (eg. "golden retriever" and "dog"). #### Output layer The first YOLO model simply predicts the $N \\times N \\times B$ bounding boxes using the output of our backbone network. In YOLOv2, Redmond adds a weird skip connection splitting a higher resolution feature map across multiple channels as visualized below. ![weird skip connection](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/07/Screen-Shot-2018-07-08-at-11.31.01-PM.png) The weird "skip connection from higher resolution feature maps" idea that I don't like. Fortunately, this was changed in the third iteration for a more standard feature pyramid network output structure. With this method, we'll alternate between outputting a prediction and upsampling the feature maps (with skip connections). This allows for predictions that can take advantage of finer-grained information from earlier in the network, which helps for detecting small objects in the image. ![FPN structure](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/07/Screen-Shot-2018-07-09-at-11.21.03-AM.png) [Image credit](https://arxiv.org/abs/1612.03144?ref=jeremyjordan.me) ## SSD: Single Shot Detection The SSD model was also published (by Wei Liu et al.) in 2015, shortly after the YOLO model, and was also later refined in a subsequent paper. In each section, I'll discuss the specific implementation details for this model. #### Backbone network A VGG-16 model, pre-trained on ImageNet for image classification, is used as the backbone network. The authors make a few slight tweaks when adapting the model for the detection task, including: replacing fully connected layers with convolutional implementations, removing dropout layers, and replacing the last max pooling layer with a dilated convolution. #### Bounding boxes (and concept of anchor boxes) Rather than using k-means clustering to discover aspect ratios, the SSD model manually defines a collection of aspect ratios (eg. {1, 2, 3, 1/2, 1/3}) to use for the $B$ bounding boxes at each grid cell location. For each bounding box, we'll predict the *offsets* from the anchor box for both the bounding box coordinates ($x$ and $y$) and dimensions (width and height). We'll use ReLU activations trained with a Smooth L1 loss. #### Objectness (and assigning labeled objects to a bounding box) One major distinction between YOLO and SSD is that SSD *does not* attempt to predict a value for $p\_{obj}$. Whereas the YOLO model predicted the probability of an object and then predicted the probability of each class given that there was an object present, the SSD model attempts to directly predict the probability that a class is present in a given bounding box. When calculating the loss, we'll match each ground truth box to the anchor box with the highest IoU — defining this box with being "responsible" for making the prediction. However, we'll also match the ground truth boxes with any other anchor boxes with an IoU above some defined threshold (0.5) in the same light of not punishing good predictions simply because they weren't the best. We can always rely on non-max suppression at inference time to filter out redundant predictions. #### Class labels As I mentioned previously, the class predictions for SSD bounding boxes *are not* conditioned on the fact that an object is present. Thus, we directly predict the probability of each class using a softmax activation and cross entropy loss. Because we don't explicitly predict $p\_{obj}$, it's important to have a class for "background" so that we can predict when no object is present. Due to the fact that most of the boxes will belong to the "background" class, we will use a technique known as "hard negative mining" to sample negative (no object) predictions such that there is at most a 3:1 ratio between negative and positive predictions when calculating our loss. #### Output layer To allow for predictions at multiple scales, the SSD output module progressively downsamples the convolutional feature maps, intermittently producing bounding box predictions (as shown with the arrows from convolutional layers to the predictions box). ![multi scale output](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/07/Screen-Shot-2018-07-09-at-12.26.30-PM.png) ## Addressing object imbalance with focal loss As I mentioned earlier, we often end up with a large amount of bounding boxes in which no object is contained due to the nature of our "predictions on a grid" approach. Although we can easily filter these boxes out after making a fixed set of bounding box predictions, there is still a (foreground-background) class imbalance present which can introduce difficulties during training. This is especially difficult for models which don't separate prediction of objectness and class probability into two separate tasks, and instead simply include a "background" class for regions with no objects. Researchers at Facebook proposed adding a scaling factor to the standard cross entropy loss such that it places more the emphasis on "hard" examples during training, preventing easy negative predictions from dominating the training process. ![focal loss](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/07/Screen-Shot-2018-07-10-at-10.34.22-PM.png) [Image credit](https://arxiv.org/abs/1708.02002?ref=jeremyjordan.me) As the researchers point out, easily classified examples can incur a non-trivial loss for standard cross entropy loss ($\\gamma=0$) which, summed over a large collection of samples, can easily dominate the parameter update. The ${\\left( {1 - {p\_t}} \\right)^\\gamma }$ term acts as a tunable scaling factor to prevent this from occuring. As the paper points out, "with $\\gamma=2$, an example classified with $p\_t = 0.9$ would have 100X lower loss compared with CE and with $p\_t = 0.968$ it would have 1000X lower loss." ## Common datasets and competitions Below I've listed some common datasets that researchers use when evaluating new object detection models. - [PASCAL VOC 2012 Detection Competition](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/index.html?ref=jeremyjordan.me) - [COCO 2018 Stuff Object Detection Task](http://cocodataset.org/?ref=jeremyjordan.me#detection-2018) - [ImageNet Object Detection Challenge](https://www.kaggle.com/c/imagenet-object-detection-challenge?ref=jeremyjordan.me) - [Google AI Open Images - Object Detection Track](https://www.kaggle.com/c/google-ai-open-images-object-detection-track?ref=jeremyjordan.me) - [Vision Meets Drones: A Challenge](http://www.aiskyeye.com/views/index?ref=jeremyjordan.me) ## Further reading **Papers** - [Deep Learning for Generic Object Detection: A Survey](https://arxiv.org/abs/1809.02165v1?ref=jeremyjordan.me) - YOLO - [You Only Look Once: Unified, Real-Time Object Detection](https://arxiv.org/abs/1506.02640?ref=jeremyjordan.me) - [YOLO9000: Better, Faster, Stronger](https://arxiv.org/abs/1612.08242?ref=jeremyjordan.me) - [YOLOv3: An Incremental Improvement](https://arxiv.org/abs/1804.02767?ref=jeremyjordan.me) - SSD - [SSD: Single Shot MultiBox Detector](https://arxiv.org/abs/1512.02325?ref=jeremyjordan.me) - [DSSD: Deconvolutional Single Shot Detector](https://arxiv.org/abs/1701.06659?ref=jeremyjordan.me) (I didn't discuss this in the blog post but it's worth the read) - [Focal Loss for Dense Object Detection](https://arxiv.org/abs/1708.02002?ref=jeremyjordan.me) - [An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution](https://arxiv.org/abs/1807.03247?ref=jeremyjordan.me) (see relevant section on object detection) - [Explainer video](https://www.youtube.com/watch?v=8yFQc6elePA&ref=jeremyjordan.me) **Lectures** - [Stanford CS 231n: Lecture 11 | Detection and Segmentation](https://www.youtube.com/watch?v=nDPWywWRIRo&t=1967s&ref=jeremyjordan.me) **Blog posts** - [Understanding deep learning for object detection](http://zoey4ai.com/2018/05/12/deep-learning-object-detection/?ref=jeremyjordan.me) - [Real-time Object Detection with YOLO, YOLOv2 and now YOLOv3](https://medium.com/@jonathan%5Fhui/real-time-object-detection-with-yolo-yolov2-28b1b93e2088?ref=jeremyjordan.me) **Frameworks and GitHub repos** - [Luminoth](https://github.com/tryolabs/luminoth?ref=jeremyjordan.me) - [Darkflow](https://github.com/thtrieu/darkflow?ref=jeremyjordan.me) **Tools for labeling data** - [Computer Vision Annotation Tool (CVAT)](https://github.com/opencv/cvat?ref=jeremyjordan.me) - [LabelImg](https://github.com/tzutalin/labelImg?ref=jeremyjordan.me) - [Microsoft's Visual Object Tagging Tool](https://github.com/Microsoft/VoTT?ref=jeremyjordan.me) - [Taggerine](https://github.com/tryolabs/taggerine?ref=jeremyjordan.me) ### Evaluating image segmentation models. URL: https://www.jeremyjordan.me/evaluating-image-segmentation-models/ Last updated: 2018-12-18T05:13:51.000Z When evaluating a standard machine learning model, we usually classify our predictions into four categories: true positives, false positives, true negatives, and false negatives. However, for the dense prediction task of image segmentation, it's not immediately clear what counts as a "true positive" and, more generally, how we can evaluate our predictions. In this post, I'll discuss common methods for evaluating both semantic and instance segmentation techniques. ## Semantic segmentation Recall that the task of [semantic segmentation](https://www.jeremyjordan.me/semantic-segmentation/) is simply to predict the class of each pixel in an image. ![Screen-Shot-2018-05-21-at-10.44.23-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-21-at-10.44.23-PM.png) [Image credit](https://arxiv.org/abs/1611.09326?ref=jeremyjordan.me) Our prediction output shape matches the input's spatial resolution (width and height) with a channel depth equivalent to the number of possible classes to be predicted. Each channel consists of a binary mask which labels areas where a specific class is present. #### Intersection over Union The Intersection over Union (IoU) metric, also referred to as the Jaccard index, is essentially a method to quantify the percent overlap between the target mask and our prediction output. This metric is closely related to the Dice coefficient which is often used as a [loss function](https://www.jeremyjordan.me/semantic-segmentation/#loss) during training. Quite simply, the IoU metric measures the number of pixels common between the target and prediction masks divided by the total number of pixels present across *both* masks. $$ IoU = \\frac{{target \\cap prediction}}{{target \\cup prediction}} $$ As a visual example, let's suppose we're tasked with calculating the IoU score of the following prediction, given the ground truth labeled mask. ![target_prediction](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/target_prediction.png) The **intersection** ($A \\cap B$) is comprised of the pixels found in both the prediction mask *and* the ground truth mask, whereas the **union** ($A \\cup B$) is simply comprised of all pixels found in either the prediction *or* target mask. ![intersection_union](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/intersection_union.png) We can calculate this easily using Numpy. ```python intersection = np.logical_and(target, prediction) union = np.logical_or(target, prediction) iou_score = np.sum(intersection) / np.sum(union) ``` The IoU score is calculated for each class separately and then **averaged over all classes** to provide a global, mean IoU score of our semantic segmentation prediction. #### Pixel Accuracy An alternative metric to evaluate a semantic segmentation is to simply report the percent of pixels in the image which were correctly classified. The pixel accuracy is commonly reported for each class separately as well as globally across all classes. When considering the per-class pixel accuracy we're essentially evaluating a binary mask; a true positive represents a pixel that is correctly predicted to belong to the given class (according to the target mask) whereas a true negative represents a pixel that is correctly identified as not belonging to the given class. $$ accuracy = \\frac{{TP + TN}}{{TP + TN + FP + FN}} $$ This metric can sometimes provide misleading results when the class representation is small within the image, as the measure will be biased in mainly reporting how well you identify negative case (ie. where the class is not present). ## Instance segmentation Instance segmentation models are a little more complicated to evaluate; whereas semantic segmentation models output a single segmentation mask, instance segmentation models produce a collection of local segmentation masks describing each object detected in the image. As such, evaluation methods for instance segmentation are quite similar to that of object detection, with the exception that we now calculate IoU of masks instead of bounding boxes. ![semantic_instance-2](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/semantic_instance-2.png) [Image credit](https://engineering.matterport.com/splash-of-color-instance-segmentation-with-mask-r-cnn-and-tensorflow-7c761e238b46?ref=jeremyjordan.me) #### Calculating Precision To evaluate our collection of predicted masks, we'll compare each of our predicted masks with each of the available target masks for a given input. - A **true positive** is observed when a prediction-target mask pair has an IoU score which exceeds some predefined threshold. - A **false positive** indicates a predicted object mask had no associated ground truth object mask. - A **false negative** indicates a ground truth object mask had no associated predicted object mask. ![Screen-Shot-2018-05-30-at-12.01.19-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-30-at-12.01.19-PM.png) **Precision** effectively describes the *purity* of our positive detections relative to the ground truth. Of all of the objects that we predicted in a given image, how many of those objects actually had a matching ground truth annotation? $$ Precision = \\frac{TP}{TP + FP} $$ **Recall** effectively describes the *completeness* of our positive predictions relative to the ground truth. Of all of the objected annotated in our ground truth, how many did we capture as positive predictions? $$ Recall = \\frac{TP}{TP + FN} $$ However, in order to calculate the prediction and recall of a model output, we'll need to define what constitutes a **positive detection**. To do this, we'll calculate the IoU score between each (prediction, target) mask pair and then determine which mask pairs have an IoU score *exceeding a defined threshold value*. ![IoU-comparison](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/IoU-comparison.png) However, computing a single precision and recall score at the specified IoU threshold does not adequately describe the behavior of our model's full precision-recall curve. Instead, we can use **average precision** to effectively integrate the area under a precision-recall curve. Let's use the precision-recall curve below as an example. ![calculated-PR-curve](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/12/calculated-PR-curve-1.png) First, we'll adjust our curve such that the precision at a given point $r$ is adjusted to the maximum precision for recall greater than $r$. ![adjusted-PR-curve](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/12/adjusted-PR-curve-1.png) Then, we'll simply calculate the area under the curve by numerical integration. This method replaces an [older approach of averaging over a range of recall values](https://www.youtube.com/watch?v=yjCMEjoc%5FZI&ref=jeremyjordan.me). *Note that the precision-recall curve will likely not extend out to perfect recall due to our prediction thresholding according to each mask IoU.* As an example, the [Microsoft COCO challenge](http://cocodataset.org/?ref=jeremyjordan.me#detection-eval)'s primary metric for the detection task evaluates the average precision score using IoU thresholds ranging from 0.5 to 0.95 (in 0.05 increments). ![threshold_ranges](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/threshold_ranges.png) For prediction problems with multiple classes of objects, this value is then averaged over all of the classes. ### An overview of semantic image segmentation. URL: https://www.jeremyjordan.me/semantic-segmentation/ Last updated: 2023-03-05T03:54:04.000Z In this post, I'll discuss how to use [convolutional neural networks](https://www.jeremyjordan.me/convolutional-neural-networks/) for the task of **semantic image segmentation**. Image segmentation is a computer vision task in which we label specific regions of an image according to what's being shown. > "What's in this image, and where in the image is it located?" **Jump to:** - [Representing the task](#representing) - [Constructing an architecture](#constructing) - [Methods for upsampling](#upsampling) - [Fully convolutional networks](#fully%5Fconvolutional) - [Adding skip connections](#skip%5Fconnections) - [Advanced U-Net variants](#advanced%5Funet) - [Dilated convolutions](#dilated%5Fconvolutions) - [Defining a loss function](#loss) - [Common datasets and segmentation competitions](#datasets) - [Further reading](#further%5Freading) More specifically, the goal of semantic image segmentation is to label *each pixel* of an image with a corresponding **class** of what is being represented. Because we're predicting for every pixel in the image, this task is commonly referred to as **dense prediction**. ![Screen-Shot-2018-05-17-at-7.42.16-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-17-at-7.42.16-PM.png) An example of semantic segmentation, where the goal is to predict class labels for each pixel in the image. [(Source)](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/?ref=jeremyjordan.me#devkit) One important thing to note is that we're not separating *instances* of the same class; we only care about the category of each pixel. In other words, if you have two objects of the same category in your input image, the segmentation map does not inherently distinguish these as separate objects. There exists a different class of models, known as *instance segmentation* models, which *do* distinguish between separate objects of the same class. Segmentation models are useful for a variety of tasks, including: - **Autonomous vehicles** We need to equip cars with the necessary perception to understand their environment so that self-driving cars can safely integrate into our existing roads. ![deeplabcityscape](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/deeplabcityscape.gif) A real-time segmented road scene for autonomous driving. [(Source)](https://www.youtube.com/watch?v=ATlcEDSPWXY&ref=jeremyjordan.me) - **Medical image diagnostics** Machines can augment analysis performed by radiologists, greatly reducing the time required to run diagnositic tests. ![chest xray](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-23-at-7.17.43-PM.png) A chest x-ray with the heart (red), lungs (green), and clavicles (blue) are segmented. [(Source)](https://arxiv.org/abs/1701.08816?ref=jeremyjordan.me) ## Representing the task Simply, our goal is to take either a RGB color image ($height \\times width \\times 3$) or a grayscale image ($height \\times width \\times 1$) and output a segmentation map where each pixel contains a class label represented as an integer ($height \\times width \\times 1$). ![input to label](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-17-at-9.02.15-PM.png) *Note: For visual clarity, I've labeled a low-resolution prediction map. In reality, the segmentation label resolution should match the original input's resolution.* Similar to how we treat standard categorical values, we'll create our **target** by one-hot encoding the class labels - essentially creating an output channel for each of the possible classes. ![one hot](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-16-at-9.36.00-PM.png) A prediction can be collapsed into a segmentation map (as shown in the first image) by taking the `argmax` of each depth-wise pixel vector. We can easily inspect a target by overlaying it onto the observation. ![overlay](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-16-at-9.36.38-PM.png) When we overlay a *single channel* of our target (or prediction), we refer to this as a **mask** which illuminates the regions of an image where a specific class is present. ## Constructing an architecture A naive approach towards constructing a neural network architecture for this task is to simply stack a number of convolutional layers (with `same` padding to preserve dimensions) and output a final segmentation map. This directly learns a mapping from the input image to its corresponding segmentation through the successive transformation of feature mappings; however, it's quite computationally expensive to preserve the full resolution throughout the network. ![Screen-Shot-2018-05-19-at-12.32.20-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-19-at-12.32.20-PM.png) [Image credit](http://cs231n.stanford.edu/slides/2017/cs231n%5F2017%5Flecture11.pdf?ref=jeremyjordan.me) Recall that for deep convolutional networks, earlier layers tend to learn low-level concepts while later layers develop more high-level (and specialized) feature mappings. **In order to maintain expressiveness, we typically need to increase the number of feature maps (channels) as we get deeper in the network.** This didn't necessarily pose a problem for the task of image classification, because for that task we only care about *what* the image contains (and not where it is located). Thus, we could alleviate computational burden by periodically downsampling our feature maps through pooling or strided convolutions (ie. compressing the spatial resolution) without concern. However, for image segmentation, we would like our model to produce a *full-resolution* semantic prediction. One popular approach for image segmentation models is to follow an **encoder/decoder structure** where we *downsample* the spatial resolution of the input, developing lower-resolution feature mappings which are learned to be highly efficient at discriminating between classes, and the *upsample* the feature representations into a full-resolution segmentation map. ![Screen-Shot-2018-05-16-at-10.33.29-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-16-at-10.33.29-PM.png) [Image credit](http://cs231n.stanford.edu/slides/2017/cs231n%5F2017%5Flecture11.pdf?ref=jeremyjordan.me) #### Methods for upsampling There are a few different approaches that we can use to *upsample* the resolution of a feature map. Whereas pooling operations downsample the resolution by summarizing a local area with a single value (ie. average or max pooling), "unpooling" operations upsample the resolution by distributing a single value into a higher resolution. ![Screen-Shot-2018-05-19-at-12.54.50-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-19-at-12.54.50-PM.png) [Image credit](http://cs231n.stanford.edu/slides/2017/cs231n%5F2017%5Flecture11.pdf?ref=jeremyjordan.me) However, **transpose convolutions** are by far the most popular approach as they allow for us to develop a *learned upsampling*. ![Screen-Shot-2018-05-19-at-3.12.51-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-19-at-3.12.51-PM.png) [Image credit](http://cs231n.stanford.edu/slides/2017/cs231n%5F2017%5Flecture11.pdf?ref=jeremyjordan.me) Whereas a typical convolution operation will take the dot product of the values currently in the filter's view and produce a single value for the corresponding output position, a transpose convolution essentially does the opposite. For a transpose convolution, we take a single value from the low-resolution feature map and multiply all of the weights in our filter by this value, projecting those weighted values into the output feature map. ![Screen-Shot-2018-05-21-at-11.01.29-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-21-at-11.01.29-PM.png) A simplified 1D example of upsampling through a transpose operation. [(Source)](http://cs231n.stanford.edu/slides/2017/cs231n%5F2017%5Flecture11.pdf?ref=jeremyjordan.me) For filter sizes which produce an overlap in the output feature map (eg. 3x3 filter with stride 2 - as shown in the below example), the overlapping values are simply added together. Unfortunately, this tends to produce a checkerboard artifact in the output and is undesirable, so it's best to ensure that your filter size does not produce an overlap. ![padding_strides_transposed](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/padding_strides_transposed--1-.gif) Input in blue, output in green. [(Source)](https://github.com/vdumoulin/conv%5Farithmetic?ref=jeremyjordan.me) #### Fully convolutional networks The approach of using a "fully convolutional" network trained end-to-end, pixels-to-pixels for the task of image segmentation was introduced by [Long et al.](https://arxiv.org/abs/1411.4038?ref=jeremyjordan.me) in late 2014\. The paper's authors propose adapting existing, well-studied *image classification* networks (eg. AlexNet) to serve as the encoder module of the network, appending a decoder module with transpose convolutional layers to upsample the coarse feature maps into a full-resolution segmentation map. ![Screen-Shot-2018-05-20-at-9.53.20-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-20-at-9.53.20-AM.png) [Image credit (with modification)](https://arxiv.org/abs/1411.4038?ref=jeremyjordan.me) The full network, as shown below, is trained according to a pixel-wise cross entropy loss. ![Screen-Shot-2018-05-16-at-10.34.02-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-16-at-10.34.02-PM.png) [Image credit](https://arxiv.org/abs/1411.4038?ref=jeremyjordan.me) However, because the encoder module reduces the resolution of the input by a factor of 32, the decoder module **struggles to produce fine-grained segmentations** (as shown below). ![Screen-Shot-2018-05-20-at-10.15.09-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-20-at-10.15.09-AM.png) The paper's authors comment eloquently on this struggle: > Semantic segmentation faces an inherent tension between semantics and location: global information resolves **what** while local information resolves **where**... Combining fine layers and coarse layers lets the model make local predictions that respect global structure. ― [Long et al.](https://arxiv.org/abs/1411.4038?ref=jeremyjordan.me) #### Adding skip connections The authors address this tension by slowly upsampling (in stages) the encoded representation, adding "skip connections" from earlier layers, and summing these two feature maps. ![FCN-8s](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-20-at-12.26.53-PM.png) [Image credit (with modification)](https://arxiv.org/abs/1411.4038?ref=jeremyjordan.me) These skip connections from earlier layers in the network (prior to a downsampling operation) should provide the necessary detail in order to reconstruct accurate shapes for segmentation boundaries. Indeed, we can recover more fine-grain detail with the addition of these skip connections. ![Screen-Shot-2018-05-20-at-12.10.25-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-20-at-12.10.25-PM.png) [Ronneberger et al.](https://arxiv.org/abs/1505.04597?ref=jeremyjordan.me) improve upon the "fully convolutional" architecture primarily through **expanding the capacity of the decoder** module of the network. More concretely, they propose the **U-Net architecture** which "consists of a contracting path to capture context and a **symmetric** expanding path that enables precise localization." This simpler architecture has grown to be very popular and has been adapted for a variety of segmentation problems. ![U Net](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-20-at-1.46.43-PM.png) [Image credit](https://arxiv.org/abs/1505.04597?ref=jeremyjordan.me) *Note: The original architecture introduces a decrease in resolution due to the use of `valid` padding. However, some practitioners opt to use `same` padding where the padding values are obtained by image reflection at the border.* Whereas [Long et al.](https://arxiv.org/abs/1411.4038?ref=jeremyjordan.me) (FCN paper) reported that data augmentation ("randomly mirroring and “jittering” the images by translating them up to 32 pixels") did not result in a noticeable improvement in performance, [Ronneberger et al.](https://arxiv.org/abs/1505.04597?ref=jeremyjordan.me) (U-Net paper) credit data augmentations ("random elastic deformations of the training samples") as a key concept for learning. It appears as if **the usefulness (and type) of data augmentation depends on the problem domain**. #### Advanced U-Net variants The standard U-Net model consists of a series of convolution operations for each "block" in the architecture. As I discussed in my post on [common convolutional network architectures](https://www.jeremyjordan.me/convnet-architectures/), there exist a number of more advanced "blocks" that can be substituted in for stacked convolutional layers. [Drozdzal et al.](https://arxiv.org/abs/1608.04117?ref=jeremyjordan.me) swap out the basic stacked convolution blocks in favor of **residual blocks**. This residual block introduces short skip connections (within the block) alongside the existing long skip connections (between the corresponding feature maps of encoder and decoder modules) found in the standard U-Net structure. They report that the short skip connections allow for faster convergence when training and allow for deeper models to be trained. Expanding on this, [Jegou et al.](https://arxiv.org/abs/1611.09326?ref=jeremyjordan.me) proposed the use of **dense blocks**, still following a U-Net structure, arguing that the "characteristics of DenseNets make them a very good fit for semantic segmentation as they *naturally induce skip connections and multi-scale supervision*." These dense blocks are useful as they carry low level features from previous layers directly alongside higher level features from more recent layers, allowing for highly efficient feature reuse. ![FC DenseNet](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-20-at-3.42.24-PM.png) [Image credit (with modification)](https://arxiv.org/abs/1611.09326?ref=jeremyjordan.me) One very important aspect of this architecture is the fact that the upsampling path *does not* have a skip connection between the input and output of a dense block. The authors note that because the "upsampling path *increases* the feature maps spatial resolution, the linear growth in the number of features would be too memory demanding." Thus, only the *output* of a dense block is passed along in the decoder module. ![Screen-Shot-2018-05-21-at-10.44.23-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-21-at-10.44.23-PM.png) The FC-DenseNet103 model acheives [state of the art results](https://arxiv.org/abs/1611.09326?ref=jeremyjordan.me) (Oct 2017) on the CamVid dataset. #### Dilated/atrous convolutions One benefit of downsampling a feature map is that it *broadens the receptive field* (with respect to the input) for the following filter, given a constant filter size. Recall that this approach is more desirable than increasing the filter size due to the parameter inefficiency of large filters (discussed [here](https://arxiv.org/abs/1512.00567?ref=jeremyjordan.me) in Section 3.1). However, this broader context comes at the cost of reduced spatial resolution. **Dilated convolutions** provide alternative approach towards gaining a wide field of view while preserving the full spatial dimension. As shown in the figure below, the values used for a dilated convolution are spaced apart according to some specified *dilation rate*. ![dilation](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/dilation.gif) [Image credit](https://github.com/vdumoulin/conv%5Farithmetic?ref=jeremyjordan.me) [Some architectures](https://arxiv.org/abs/1511.07122?ref=jeremyjordan.me) swap out the last few pooling layers for dilated convolutions with successively higher dilation rates to maintain the same field of view while preventing loss of spatial detail. However, it is often [still too computationally expensive](https://arxiv.org/abs/1606.00915?ref=jeremyjordan.me) to completely replace pooling layers with dilated convolutions. ## Defining a loss function The most commonly used loss function for the task of image segmentation is a **pixel-wise cross entropy loss**. This loss examines *each pixel individually*, comparing the class predictions (depth-wise pixel vector) to our one-hot encoded target vector. ![cross entropy](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-24-at-10.46.16-PM.png) Because the cross entropy loss evaluates the class predictions for each pixel vector individually and then averages over all pixels, we're essentially asserting equal learning to each pixel in the image. This can be a problem if your various classes have unbalanced representation in the image, as training can be dominated by the most prevalent class. [Long et al.](https://arxiv.org/abs/1411.4038?ref=jeremyjordan.me) (FCN paper) discuss weighting this loss for each **output channel** in order to counteract a class imbalance present in the dataset. Meanwhile, [Ronneberger et al.](https://arxiv.org/abs/1505.04597?ref=jeremyjordan.me) (U-Net paper) discuss a loss weighting scheme for each **pixel** such that there is a higher weight at the border of segmented objects. This loss weighting scheme helped their U-Net model segment cells in biomedical images in a *discontinuous* fashion such that individual cells may be easily identified within the binary segmentation map. ![pixel loss weights](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-21-at-10.53.04-PM.png) Notice how the binary segmentation map produces clear borders around the cells. [(Source)](https://arxiv.org/abs/1505.04597?ref=jeremyjordan.me) --- Another popular loss function for image segmentation tasks is based on the [Dice coefficient](https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice%5Fcoefficient?ref=jeremyjordan.me), which is essentially a measure of overlap between two samples. This measure ranges from 0 to 1 where a Dice coefficient of 1 denotes perfect and complete overlap. The Dice coefficient was originally developed for binary data, and can be calculated as: $$ Dice = \\frac{{2\\left| {A \\cap B} \\right|}}{{\\left| A \\right| + \\left| B \\right|}} $$ where ${\\left| {A \\cap B} \\right|}$ represents the common elements between sets A and B, and $\\left| A \\right|$ represents the number of elements in set A (and likewise for set B). For the case of evaluating a Dice coefficient on predicted segmentation masks, we can approximate ${\\left| {A \\cap B} \\right|}$ as the element-wise multiplication between the prediction and target mask, and then sum the resulting matrix. ![intersection](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/intersection-1.png) Because our target mask is binary, we effectively zero-out any pixels from our prediction which are not "activated" in the target mask. For the remaining pixels, we are essentially penalizing low-confidence predictions; a higher value for this expression, which is in the numerator, leads to a better Dice coefficient. In order to quantify $\\left| A \\right|$ and $\\left| B \\right|$, [some researchers](https://arxiv.org/abs/1608.04117?ref=jeremyjordan.me) use the simple sum whereas [other researchers](https://arxiv.org/abs/1606.04797?ref=jeremyjordan.me) prefer to use the squared sum for this calculation. I don't have the practical experience to know which performs better empirically over a wide range of tasks, so I'll leave you to try them both and see which works better. ![cardinality](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-25-at-5.53.46-PM.png) In case you were wondering, there's a 2 in the numerator in calculating the Dice coefficient because our denominator "double counts" the common elements between the two sets. In order to formulate a loss function which can be minimized, we'll simply use $1 - Dice$. This loss function is known as the **soft Dice loss** because we directly use the predicted probabilities instead of thresholding and converting them into a binary mask. With respect to the neural network output, the numerator is concerned with the *common activations* between our prediction and target mask, where as the denominator is concerned with the quantity of activations in each mask *separately*. This has the effect of normalizing our loss according to the size of the target mask such that the soft Dice loss does not struggle learning from classes with lesser spatial representation in an image. ![soft dice](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-24-at-10.50.59-PM.png) A soft Dice loss is calculated for each class separately and then averaged to yield a final score. An example implementation is provided below. ## Common datasets and segmentation competitions Below, I've listed a number of common datasets that researchers use to train new models and benchmark against the state of the art. You can also explore previous Kaggle competitions and read about how winning solutions implemented segmentation models for their given task. **Datasets** - [PASCAL VOC 2012 Segmentation Competition](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/index.html?ref=jeremyjordan.me) - [COCO 2018 Stuff Segmentation Task](http://cocodataset.org/?ref=jeremyjordan.me#stuff-2018) - [BDD100K: A Large-scale Diverse Driving Video Database](http://bair.berkeley.edu/blog/2018/05/30/bdd/?ref=jeremyjordan.me) - [Cambridge-driving Labeled Video Database (CamVid)](http://mi.eng.cam.ac.uk/research/projects/VideoRec/CamVid/?ref=jeremyjordan.me) - [Cityscapes Dataset](https://www.cityscapes-dataset.com/?ref=jeremyjordan.me) - [Mapillary Vistas Dataset](https://www.mapillary.com/dataset/vistas?ref=jeremyjordan.me) - [ApolloScape Scene Parsing](http://apolloscape.auto/scene.html?ref=jeremyjordan.me) **Past Kaggle Competitions** - [2018 Data Science Bowl](https://www.kaggle.com/c/data-science-bowl-2018?ref=jeremyjordan.me) - Read about the [first place solution.](https://www.kaggle.com/c/data-science-bowl-2018/discussion/54741?ref=jeremyjordan.me) - [Carvana Image Masking Challenge](https://www.kaggle.com/c/carvana-image-masking-challenge?ref=jeremyjordan.me) - Read about the [first place solution.](https://arxiv.org/abs/1801.05746?ref=jeremyjordan.me) - [Dstl Satellite Imagery Feature Detection](https://www.kaggle.com/c/dstl-satellite-imagery-feature-detection?ref=jeremyjordan.me) - Read about the [third place solution.](https://arxiv.org/abs/1706.06169?ref=jeremyjordan.me) ## Further Reading Papers - [Fully Convolutional Networks for Semantic Segmentation](https://arxiv.org/abs/1605.06211?ref=jeremyjordan.me) - [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://arxiv.org/abs/1505.04597?ref=jeremyjordan.me) - [The Importance of Skip Connections in Biomedical Image Segmentation](https://arxiv.org/abs/1608.04117?ref=jeremyjordan.me) - [The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation](https://arxiv.org/abs/1611.09326?ref=jeremyjordan.me) - [Multi-Scale Context Aggregation by Dilated Convolutions](https://arxiv.org/abs/1511.07122?ref=jeremyjordan.me) - [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://arxiv.org/abs/1606.00915?ref=jeremyjordan.me) - [Rethinking Atrous Convolution for Semantic Image Segmentation](https://arxiv.org/abs/1706.05587?ref=jeremyjordan.me) - [Evaluation of Deep Learning Strategies for Nucleus Segmentation in Fluorescence Images](https://www.biorxiv.org/content/early/2018/05/31/335216?ref=jeremyjordan.me) Lectures - [Stanford CS231n: Detection and Segmentation](https://youtu.be/nDPWywWRIRo?ref=jeremyjordan.me) - [Lecture Slides](http://cs231n.stanford.edu/slides/2017/cs231n%5F2017%5Flecture11.pdf?ref=jeremyjordan.me) Blog posts - [Mat Kelcey's (Twitter Famous) Bee Detector](http://matpalm.com/blog/counting%5Fbees/?ref=jeremyjordan.me) - [Semantic Image Segmentation with DeepLab in TensorFlow](https://ai.googleblog.com/2018/03/semantic-image-segmentation-with.html?ref=jeremyjordan.me) - [Going beyond the bounding box with semantic segmentation](https://thegradient.pub/semantic-segmentation/?ref=jeremyjordan.me) - [U-Net Case Study: Data Science Bowl 2018](https://medium.com/@keremturgutlu/semantic-segmentation-u-net-part-1-d8d6f6005066?ref=jeremyjordan.me) - [Lyft Perception Challenge: 4th place solution](https://nikolasent.github.io/proj/comp2.html?ref=jeremyjordan.me) Image labeling tools - [labelme: Image Polygonal Annotation with Python](https://github.com/wkentaro/labelme?ref=jeremyjordan.me) Useful Github repos - [Pytorch implementations](https://github.com/meetshah1995/pytorch-semseg?ref=jeremyjordan.me) ### Lessons learned from attempting to launch a startup. URL: https://www.jeremyjordan.me/mobius/ Last updated: 2021-09-14T03:50:34.000Z In Fall 2017, I made the decision to walk down the entrepreneurial path and dedicate a full-time effort towards launching a startup venture. I secured a healthy seed round of funding from a local angel investor and recruited three of my peers to join me in this effort. By Summer 2018, we decided to put the project on an indefinite pause. This blog post discusses what I set out to accomplish, what happened, and what I've learned from the experience. # The big idea The venture was founded on a simple premise: **we could use machine learning to forecast short-term fluctuations** in market prices **based on volume imbalances in order books**. Specifically, we were focused on the cryptoasset markets as these exchanges offered free, real-time data feeds to the order books and fulfilled trades. An **order book** essentially maintains current unfulfilled orders for a given asset. The exchanges reference these order books in order to *match interested buyers and sellers*. If you're interested in trading something, you would publish your intentions to this order book which would be executed according to the type of order placed. New orders which can be matched with an existing order on the book are executed immediately and *remove* that volume from the order book. ![Order-book](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Order-book.png) To gain an understanding for how this order book works, you can reference the following cartoon. ##### An illustrated guide to exchange order books ![OB cartoon](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-08-at-9.33.12-AM-1.png#full) Because the order book contains a collection of orders waiting to be matched, it essentially *displays the schedule for supply and demand* as a function of price. That is, you can see how many people have stated that they are willing to buy/sell when the asset moves to a given price. The market price changes when the volume at the "best available price" level is exhausted. As I watched how the order book and market prices evolved over a period of time, I began to notice predictable patterns emerge. ![patterns](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-08-at-10.10.26-AM.png) Feel free to check out a [live feed of the BTC-USD order book](https://www.gdax.com/trade/BTC-USD?ref=jeremyjordan.me) and see if you can spot any of the patterns mentioned alongside the corresponding price action. It's also worth noting that [cryptoassets](https://www.jeremyjordan.me/blockchain-introduction/) are a unique asset class given that the **underlying assets being traded are publicly inspectable in real-time**. For example, you can [listen in on the Bitcoin network](http://statoshi.info/dashboard/db/transactions?ref=jeremyjordan.me) and capture statistics such as the number of participants in the network, the total value being transacted on the network, and other useful metrics. Compare this to a US public equity which only releases internal information about the company in their quarterly reports. Given this radical openness and transparency, the asset class seems uniquely posed for quantitative, data-driven trading. After securing funding and recruiting a team, we initially budgeted 2 to 3 months to build an initial proof of concept and further evaluate the feasibility of the idea. # Building the proof of concept As we set out to build the proof of concept, the initial days consisted of architecting and clarifying the exact needs and requirements of our technical infrastructure. We discussed various machine learning models and approaches, and specified the desired characteristics of our models. Because the cryptoasset market is rapidly evolving, we wanted models which supported continuous "online" learning. We also posited that this would help safeguard our models from being manipulated by bad actors who could ["spoof" the order book data](https://en.wikipedia.org/wiki/Spoofing%5F%28finance%29?ref=jeremyjordan.me); a model capable of continuous learning could evolve to not be susceptible to repeated manipulation attempts. As the weeks passed, we unknowingly succumbed to [**scope creep**](https://en.wikipedia.org/wiki/Scope%5Fcreep?ref=jeremyjordan.me) as our designs for a minimum viable product inched closer toward designs for a robust scalable product. I've realized the importance of building *forcing functions* into projects to specifically protect against scope creep. Looking back, it's now clear that **we had operated under the naive assumption that our fundamental hypothesis was correct** (after all, it *seemed* like it was the right idea) which led us to focus our time addressing auxiliary problems such as scalability and performance optimizations. If we had instead focused *all of our energy* on testing our [riskiest (fundamental) assumption](https://hackernoon.com/the-mvp-is-dead-long-live-the-rat-233d5d16ab02?ref=jeremyjordan.me), we would have discovered its flaws much more quickly. # Discovering flaws in our fundamental assumption After spending a fair amount of time designing and iterating on our data infrastructure, we settled down to perform some more rigorous analysis of the data being collected by the real-time market feeds. Here's what we learned: - Most of the observed **price fluctuations were too small to be profitable** after accounting for trading fees. - The data was **highly imbalanced**, where most periods of time the proper action was no action. When looking at snapshots of the order book once per second, *profitable buy and sell opportunities* were observed in \~2% of the snapshots. - The **signal wasn't as clear as it seemed**. When I had manually been watching the exchange data for patterns, I was subject to confirmation bias where I was only attune to the patterns when they *did* manifest, not when they didn't. The order book data in fact contains quite a bit of noise as traders can cancel their orders at any given time. For machine learning models, datasets that are inherently more difficult to learn from see an *amplification* in the learning challenge when a class imbalance is introduced. The initial hypothesis was that we could use machine learning to forecast short-term price fluctuations based on the order book data. After validating the idea, we concluded that the idea was *not likely feasible*, at least not in the continuous, automated fashion we had imagined with machine learning. I had started working on this problem as a side project as I was looking to apply what I have been learning about (machine learning) to address a messy, real world problem. The problem was attractive as it appeared to offer a wealth of data which was freely accessible - however, I did not accurately gauge the challenge of learning from this firehose of market data. # Moving forward When we discovered the flaws in our fundamental assumption, we had two main options: pivot or halt the project. After much deliberation, we eventually decided to halt this project. We each had our own reasons for reaching this conclusion, I'll only discuss my personal reasons. At this stage in life, I’m focused on optimizing for learning and continuing to invest in myself. I’ve realized that your *environment* is a critical factor in this growth, so I’ve decided to embed myself in an environment where I can work alongside more senior engineers and gain experience working with teams who have a history of building machine learning products. I'm 22 years old, I have a long career ahead of me. My insatiable desire to build things has cultivated an interest in both engineering and entrepreneurship, and I'm sure I'll walk down the entrepreneurial path once again later in my career. However, my current focus is on *mastering machine learning* and gaining practical work experience; I feel as if this focus will be beneficial for the *long-term trajectory* of my career. > "The average age of founders of the most successful startups—those with growth in the top 1% of their industry—was 45." [Source](https://work.qz.com/1260465/the-most-successful-startups-have-founders-over-the-age-of-40/?ref=jeremyjordan.me) I learned a lot my from experience exploring this venture and I'm grateful for the challenges that I encountered. Here's to the next adventure. ### Common architectures in convolutional neural networks. URL: https://www.jeremyjordan.me/convnet-architectures/ Last updated: 2018-10-20T16:37:16.000Z In this post, I'll discuss commonly used architectures for convolutional networks. As you'll see, almost all CNN architectures follow the same general design principles of successively applying convolutional layers to the input, periodically downsampling the spatial dimensions while increasing the number of feature maps. While the classic network architectures were comprised simply of stacked convolutional layers, modern architectures explore new and innovative ways for constructing convolutional layers in a way which allows for more efficient learning. Almost all of these architectures are based on a repeatable unit which is used throughout the network. These architectures serve as general design guidelines which machine learning practitioners will then adapt to solve various computer vision tasks. These architectures serve as **rich feature extractors** which can be used for image classification, object detection, image segmentation, and many other more advanced tasks. Classic network architectures (included for historical purposes) - [LeNet-5](#lenet5) - [AlexNet](#alexnet) - [VGG 16](#vgg16) Modern network architectures - [Inception](#inception) - [ResNet](#resnet) - [ResNeXt](#resnext) - [DenseNet](#densenet) ## LeNet-5 Yann Lecun's LeNet-5 model was developed in 1998 to identify handwritten digits for zip code recognition in the postal service. This pioneering model largely introduced the convolutional neural network as we know it today. **Architecture** ![Screen-Shot-2018-04-16-at-11.34.51-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/04/Screen-Shot-2018-04-16-at-11.34.51-AM.png) Convolutional layers use a subset of the previous layer's channels for each filter to reduce computation and force a break of symmetry in the network. The subsampling layers use a form of average pooling. **Parameters:** 60,000 **Paper:** [Gradient-based learning applied to document recognition](http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf?ref=jeremyjordan.me) ## AlexNet AlexNet was developed by Alex Krizhevsky et al. in 2012 to compete in the ImageNet competition. The general architecture is quite similar to LeNet-5, although this model is considerably larger. The success of this model (which took first place in the 2012 ImageNet competition) convinced a lot of the computer vision community to take a serious look at deep learning for computer vision tasks. **Architecture** ![AlexNet-CNN-architecture-layers](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/04/AlexNet-CNN-architecture-layers.png) **Parameters:** 60 million **Paper:** [ImageNet Classification with Deep Convolutional Neural Networks](https://www.nvidia.cn/content/tesla/pdf/machine-learning/imagenet-classification-with-deep-convolutional-nn.pdf?ref=jeremyjordan.me) ## VGG-16 The VGG network, introduced in 2014, offers a deeper yet simpler variant of the convolutional structures discussed above. At the time of its introduction, this model was considered to be very deep. **Architecture** ![vgg16](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/04/vgg16.png) **Parameters:** 138 million **Paper:** [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://arxiv.org/abs/1409.1556?ref=jeremyjordan.me) --- ## Inception (GoogLeNet) In 2014, researchers at Google introduced the Inception network which took first place in the 2014 ImageNet competition for classification and detection challenges. The model is comprised of a basic unit referred to as an "Inception cell" in which we perform a series of convolutions at different scales and subsequently aggregate the results. In order to save computation, 1x1 convolutions are used to reduce the input channel depth. For each cell, we learn a set of 1x1, 3x3, and 5x5 filters which can learn to extract features at different scales from the input. Max pooling is also used, albeit with "same" padding to preserve the dimensions so that the output can be properly concatenated. ![inception cell](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/04/Screen-Shot-2018-04-17-at-10.12.35-AM.png) These researchers published a follow-up paper which introduced more efficient alternatives to the original Inception cell. Convolutions with large spatial filters (such as 5x5 or 7x7) are beneficial in terms of their expressiveness and ability to extract features at a larger scale, but the computation is disproportionately expensive. The researchers pointed out that a 5x5 convolution can be more cheaply represented by two stacked 3x3 filters. ![3x3 vs 5x5](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/04/Screen-Shot-2018-04-17-at-5.32.45-PM.png) Whereas a $5 \\times 5 \\times c$ filter requires $25c$ parameters, two $3 \\times 3 \\times c$ filters only require $18c$ parameters. In order to most accurately represent a 5x5 filter, we shouldn't use any nonlinear activations between the two 3x3 layers. However, it was discovered that "linear activation was always inferior to using rectified linear units in all stages of the factorization." It was also shown that 3x3 convolutions could be further deconstructed into successive 3x1 and 1x3 convolutions. ![Screen-Shot-2018-04-17-at-7.07.13-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/04/Screen-Shot-2018-04-17-at-7.07.13-PM.png) Generalizing this insight, we can more efficiently compute an $n \\times n$ convolution as a $1 \\times n$ convolution followed by a $n \\times 1$ convolution. **Architecture** ![inception-model](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/04/inception-model.png) In order to improve overall network performance, two auxiliary outputs are added throughout the network. It was later discovered that the earliest auxiliary output had no discernible effect on the final quality of the network. The addition of auxiliary outputs primarily benefited the end performance of the model, converging at a slightly better value than the same network architecture without an auxiliary branch. It is believed the addition of auxiliary outputs had a regularizing effect on the network. A revised, deeper version of the Inception network which takes advantage of the more efficient Inception cells is shown below. ![inception-v3-model](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/04/inception-v3-model.png) **Parameters:** 5 million (V1) and 23 million (V3) **Papers:** - [Going deeper with convolutions](https://arxiv.org/abs/1409.4842?ref=jeremyjordan.me) - [Rethinking the Inception Architecture for Computer Vision](https://arxiv.org/abs/1512.00567?ref=jeremyjordan.me) ## ResNet Deep residual networks were a breakthrough idea which enabled the development of much deeper networks (hundreds of layers as opposed to tens of layers). Its a generally accepted principle that deeper networks are capable of learning more complex functions and representations of the input which should lead to better performance. However, many researchers observed that adding more layers eventually had a negative effect on the final performance. This behavior was not intuitively expected, as explained by the authors below. > Let us consider a shallower architecture and its deeper counterpart that adds more layers onto it. There exists a solution *by construction* to the deeper model: the added layers are *identity* mapping, and the other layers are copied from the learned shallower model. The existence of this constructed solution indicates that **a deeper model should produce no higher training error than its shallower counterpart**. But experiments show that our current solvers on hand are unable to find solutions that are comparably good or better than the constructed solution (or unable to do so in feasible time). This phenomenon is referred to by the authors as the *degradation* problem - alluding to the fact that although better parameter initialization techniques and batch normalization allow for deeper networks to *converge*, they often converge at a higher error rate than their shallower counterparts. In the limit, simply stacking more layers degrades the model's ultimate performance. The authors propose a remedy to this degradation problem by introducing *residual blocks* in which intermediate layers of a block learn a residual function with reference to the block input. You can think of this residual function as a refinement step in which we learn how to adjust the input feature map for higher quality features. This compares with a "plain" network in which each layer is expected to learn new and distinct feature maps. In the event that no refinement is needed, the intermediate layers can learn to gradually adjust their weights toward zero such that the residual block represents an identity function. ![residual unit](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/04/Screen-Shot-2018-04-16-at-6.29.19-PM.png) Note: [It was later discovered](https://arxiv.org/abs/1603.05027?ref=jeremyjordan.me) that a slight modification to the original proposed unit offers better performance by more efficiently allowing gradients to propagate through the network during training. ![revised residual unit](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/04/Screen-Shot-2018-04-17-at-10.36.21-PM.png) **Wide residual networks** Although the original ResNet paper focused on creating a network architecture to enable deeper structures by alleviating the degradation problem, [other researchers have since pointed out](https://arxiv.org/abs/1605.07146?ref=jeremyjordan.me) that increasing the network's width (channel depth) can be a more efficient way of expanding the overall capacity of the network. **Architecture** ![Screen-Shot-2018-04-16-at-6.30.05-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/04/Screen-Shot-2018-04-16-at-6.30.05-PM.png) Each colored block of layers represent a series of convolutions of the same dimension. The feature mapping is periodically downsampled by strided convolution accompanied by an increase in channel depth to preserve the time complexity per layer. Dotted lines denote residual connections in which we project the input via a 1x1 convolution to match the dimensions of the new block. The diagram above visualizes the ResNet 34 architecture. For the ResNet 50 model, we simply replace each two layer residual block with a three layer bottleneck block which uses 1x1 convolutions to reduce and subsequently restore the channel depth, allowing for a reduced computational load when calculating the 3x3 convolution. ![Screen-Shot-2018-04-16-at-5.47.38-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/04/Screen-Shot-2018-04-16-at-5.47.38-PM.png) **Parameters:** 25 million (ResNet 50) **Papers:** - [Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385?ref=jeremyjordan.me) - [Identity Mappings in Deep Residual Networks](https://arxiv.org/abs/1603.05027?ref=jeremyjordan.me) - [Wide Residual Networks](https://arxiv.org/abs/1605.07146?ref=jeremyjordan.me) ## ResNeXt The ResNeXt architecture is an extension of the deep residual network which replaces the standard residual block with one that leverages a "*split-transform-merge*" strategy (ie. branched paths within a cell) used in the Inception models. Simply, rather than performing convolutions over the full input feature map, the block's input is projected into a series of lower (channel) dimensional representations of which we separately apply a few convolutional filters before merging the results. ![Screen-Shot-2018-04-18-at-11.46.24-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/04/Screen-Shot-2018-04-18-at-11.46.24-PM.png) This idea is quite similar to [*group convolutions*](https://blog.yani.io/filter-group-tutorial/?ref=jeremyjordan.me), which was an idea proposed in the [AlexNet paper](https://www.nvidia.cn/content/tesla/pdf/machine-learning/imagenet-classification-with-deep-convolutional-nn.pdf?ref=jeremyjordan.me) as a way to share the convolution computation across two GPUs. Rather than creating filters with the full channel depth of the input, the input is split channel-wise into groups with each as shown below. ![Screen-Shot-2018-04-18-at-11.47.29-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/04/Screen-Shot-2018-04-18-at-11.47.29-PM.png) It was discovered that using grouped convolutions led to a degree of *specialization* among groups where separate groups focused on different characteristics of the input image. ![Screen-Shot-2018-04-19-at-10.07.03-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/04/Screen-Shot-2018-04-19-at-10.07.03-AM.png) The ResNeXt paper refers to the number of branches or groups as the **cardinality** of the ResNeXt cell and performs a series of experiments to understand relative performance gains between increasing the cardinality, depth, and width of the network. The experiments show that increasing cardinality is more effective at benefiting model performance than increasing the width or depth of the network. The experiments also suggest that "residual connections are helpful for *optimization*, whereas aggregated transformations are (helpful for) *stronger representations*." **Architecture** ![Screen-Shot-2018-04-19-at-12.08.53-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/04/Screen-Shot-2018-04-19-at-12.08.53-AM.png) The ResNeXt architecture simply mimicks the ResNet models, replacing the ResNet blocks for the ResNeXt block. **Paper:** [Aggregated Residual Transformations for Deep Neural Networks](https://arxiv.org/abs/1611.05431?ref=jeremyjordan.me) ## DenseNet The idea behind dense convolutional networks is simple: **it may be useful to reference feature maps from earlier in the network**. Thus, each layer's feature map is concatenated to the input of *every successive layer* within a dense block. This allows later layers within the network to *directly* leverage the features from earlier layers, encouraging feature reuse within the network. The authors state, "concatenating feature-maps learned by *different layers* increases variation in the input of subsequent layers and improves efficiency." ![Screen-Shot-2018-04-19-at-5.15.59-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/04/Screen-Shot-2018-04-19-at-5.15.59-PM.png) When I first came across this model, I figured that it would have an absurd number of parameters to support the dense connections between layers. However, because the network is capable of directly using any previous feature map, the authors found that they could work with very small output channel depths (ie. 12 filters per layer), vastly *reducing* the total number of parameters needed. The authors refer to the number of filters used in each convolutional layer as a "growth rate", $k$, since each successive layer will have $k$ more channels than the last (as a result of accumulating and concatenating all previous layers to the input). When compared with ResNet models, DenseNets are reported to acheive better performance with less complexity. ![Screen-Shot-2018-04-19-at-7.27.08-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/04/Screen-Shot-2018-04-19-at-7.27.08-PM.png) **Architecture** ![Screen-Shot-2018-04-19-at-12.14.09-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/04/Screen-Shot-2018-04-19-at-12.14.09-PM.png) For a majority of the experiments in the paper, the authors mimicked the general ResNet model architecture, simply swapping in the *dense block* as the repeated unit. **Parameters:** - 0.8 million (DenseNet-100, k=12) - 15.3 million (DenseNet-250, k=24) - 40 million (DenseNet-190, k=40) **Paper:** [Densely Connected Convolutional Networks](https://arxiv.org/abs/1608.06993?ref=jeremyjordan.me) **Video:** [CVPR 2017 Best Paper Award: Densely Connected Convolutional Networks](https://www.youtube.com/watch?v=-W6y8xnd--U&ref=jeremyjordan.me) ## Further reading [An Analysis of Deep Neural Network Models for Practical Applications](https://arxiv.org/abs/1605.07678?ref=jeremyjordan.me) \- [Corresponding blog post](https://towardsdatascience.com/neural-network-architectures-156e5bad51ba?ref=jeremyjordan.me) ### Variational autoencoders. URL: https://www.jeremyjordan.me/variational-autoencoders/ Last updated: 2023-03-05T03:55:59.000Z In my [introductory post](https://www.jeremyjordan.me/autoencoders/) on autoencoders, I discussed various models (undercomplete, sparse, denoising, contractive) which take data as input and discover some latent state representation of that data. More specifically, our input data is converted into an *encoding vector* where each dimension represents some learned attribute about the data. The most important detail to grasp here is that our encoder network is outputting a *single value* for each encoding dimension. The decoder network then subsequently takes these values and attempts to recreate the original input. A variational autoencoder (VAE) provides a *probabilistic* manner for describing an observation in latent space. Thus, rather than building an encoder which outputs a single value to describe each latent state attribute, we'll formulate our encoder to describe a probability distribution for each latent attribute. ## Intuition To provide an example, let's suppose we've trained an autoencoder model on a large dataset of faces with a encoding dimension of 6\. An ideal autoencoder will learn descriptive attributes of faces such as skin color, whether or not the person is wearing glasses, etc. in an attempt to describe an observation in some compressed representation. ![Screen-Shot-2018-03-16-at-10.24.11-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/Screen-Shot-2018-03-16-at-10.24.11-PM.png) In the example above, we've described the input image in terms of its latent attributes using a single value to describe each attribute. However, we may prefer to represent each latent attribute as a range of possible values. For instance, what *single value* would you assign for the smile attribute if you feed in a photo of the Mona Lisa? Using a variational autoencoder, we can describe latent attributes in probabilistic terms. ![Screen-Shot-2018-06-20-at-2.46.16-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/06/Screen-Shot-2018-06-20-at-2.46.16-PM.png) With this approach, we'll now represent *each latent attribute* for a given input as a probability distribution. When decoding from the latent state, we'll randomly sample from each latent state distribution to generate a vector as input for our decoder model. ![Screen-Shot-2018-06-20-at-2.47.56-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/06/Screen-Shot-2018-06-20-at-2.47.56-PM.png) *Note: For variational autoencoders, the encoder model is sometimes referred to as the *recognition model* whereas the decoder model is sometimes referred to as the *generative model*.* By constructing our encoder model to output a range of possible values (a statistical distribution) from which we'll randomly sample to feed into our decoder model, we're essentially enforcing a continuous, smooth latent space representation. For any sampling of the latent distributions, we're expecting our decoder model to be able to accurately reconstruct the input. Thus, values which are nearby to one another in latent space should correspond with very similar reconstructions. ![Screen-Shot-2018-06-20-at-2.48.42-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/06/Screen-Shot-2018-06-20-at-2.48.42-PM.png) ## Statisical motivation Suppose that there exists some hidden variable $z$ which generates an observation $x$. ![Screen-Shot-2018-03-17-at-1.31.39-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/Screen-Shot-2018-03-17-at-1.31.39-PM.png) We can only see $x$, but we would like to infer the characteristics of $z$. In other words, we’d like to compute $p\\left( {z|x} \\right)$. $$ p\\left( {z|x} \\right) = \\frac{{p\\left( {x|z} \\right)p\\left( z \\right)}}{{p\\left( x \\right)}} $$ Unfortunately, computing $p\\left( x \\right)$ is quite difficult. $$ p\\left( x \\right) = \\int {p\\left( {x|z} \\right)p\\left( z \\right)dz} $$ This usually turns out to be an [intractable distribution](https://stats.stackexchange.com/questions/4417/intractable-posterior-distributions?ref=jeremyjordan.me). However, we can apply [varitational inference](https://arxiv.org/pdf/1601.00670.pdf?ref=jeremyjordan.me) to estimate this value. Let's approximate $p\\left( {z|x} \\right)$ by another distribution $q\\left( {z|x} \\right)$ which we'll define such that it has a tractable distribution. If we can define the parameters of $q\\left( {z|x} \\right)$ such that it is very similar to $p\\left( {z|x} \\right)$, we can use it to perform approximate inference of the intractable distribution. Recall that the KL divergence is a measure of difference between two probability distributions. Thus, if we wanted to ensure that $q\\left( {z|x} \\right)$ was similar to $p\\left( {z|x} \\right)$, we could minimize the KL divergence between the two distributions. $$ \\min KL\\left( {q\\left( {z|x} \\right)||p\\left( {z|x} \\right)} \\right) $$ Dr. Ali Ghodsi goes through a full derivation [here](https://youtu.be/uaaqyVS9-rM?t=19m42s&ref=jeremyjordan.me), but the result gives us that we can minimize the above expression by maximizing the following: $$ {E\_{q\\left( {z|x} \\right)}}\\log p\\left( {x|z} \\right) - KL\\left( {q\\left( {z|x} \\right)||p\\left( z \\right)} \\right) $$ The first term represents the reconstruction likelihood and the second term ensures that our learned distribution $q$ is similar to the true prior distribution $p$. To revisit our graphical model, we can use $q$ to infer the possible hidden variables (ie. latent state) which was used to generate an observation. We can further construct this model into a neural network architecture where the encoder model learns a mapping from $x$ to $z$ and the decoder model learns a mapping from $z$ back to $x$. ![Screen-Shot-2018-03-17-at-11.31.15-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/Screen-Shot-2018-03-17-at-11.31.15-PM.png) Our loss function for this network will consist of two terms, one which penalizes reconstruction error (which can be thought of maximizing the reconstruction likelihood as discussed earlier) and a second term which encourages our learned distribution ${q\\left( {z|x} \\right)}$ to be similar to the true prior distribution ${p\\left( z \\right)}$, which we'll assume follows a unit Gaussian distribution, for each dimension $j$ of the latent space. $$ {\\cal L}\\left( {x,\\hat x} \\right) + \\sum\\limits\_j {KL\\left( {{q\_j}\\left( {z|x} \\right)||p\\left( z \\right)} \\right)} $$ ## Implementation In the previous section, I established the statistical motivation for a variational autoencoder structure. In this section, I'll provide the practical implementation details for building such a model yourself. Rather than directly outputting values for the latent state as we would in a standard autoencoder, the encoder model of a VAE will output parameters describing a distribution for each dimension in the latent space. Since we're assuming that our prior follows a normal distribution, we'll output *two* vectors describing the mean and variance of the latent state distributions. If we were to build a true multivariate Gaussian model, we'd need to define a covariance matrix describing how each of the dimensions are correlated. However, we'll make a simplifying assumption that our covariance matrix only has nonzero values on the diagonal, allowing us to describe this information in a simple vector. Our decoder model will then generate a latent vector by sampling from these defined distributions and proceed to develop a reconstruction of the original input. ![Screen-Shot-2018-03-18-at-12.24.19-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/Screen-Shot-2018-03-18-at-12.24.19-AM.png) However, this sampling process requires some extra attention. When training the model, we need to be able to calculate the relationship of each parameter in the network with respect to the final output loss using a technique known as [backpropagation](https://www.jeremyjordan.me/neural-networks-training/). However, we simply cannot do this for a *random sampling* process. Fortunately, we can leverage a clever idea known as the "reparameterization trick" which suggests that we randomly sample $\\varepsilon$ from a unit Gaussian, and then shift the randomly sampled $\\varepsilon$ by the latent distribution's mean $\\mu$ and scale it by the latent distribution's variance $\\sigma$. ![Screen-Shot-2018-03-18-at-4.36.34-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/Screen-Shot-2018-03-18-at-4.36.34-PM.png) With this reparameterization, we can now optimize the *parameters* of the distribution while still maintaining the ability to randomly sample from that distribution. ![Screen-Shot-2018-03-18-at-4.39.41-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/Screen-Shot-2018-03-18-at-4.39.41-PM.png) *Note: In order to deal with the fact that the network may learn negative values for $\\sigma$, we'll typically have the network learn $\\log \\sigma$ and exponentiate this value to get the latent distribution's variance.* ## Visualization of latent space To understand the implications of a variational autoencoder model and how it differs from standard autoencoder architectures, it's useful to examine the latent space. [This blog post](https://towardsdatascience.com/intuitively-understanding-variational-autoencoders-1bfe67eb5daf?ref=jeremyjordan.me) introduces a great discussion on the topic, which I'll summarize in this section. The main benefit of a variational autoencoder is that we're capable of learning *smooth* latent state representations of the input data. For standard autoencoders, we simply need to learn an encoding which allows us to reproduce the input. As you can see in the left-most figure, focusing only on reconstruction loss *does* allow us to separate out the classes (in this case, MNIST digits) which should allow our decoder model the ability to reproduce the original handwritten digit, but there's an uneven distribution of data within the latent space. In other words, there are areas in latent space which don't represent *any* of our observed data. ![Screen-Shot-2018-03-18-at-7.22.24-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/Screen-Shot-2018-03-18-at-7.22.24-PM.png) [Image credit](https://towardsdatascience.com/intuitively-understanding-variational-autoencoders-1bfe67eb5daf?ref=jeremyjordan.me) (modified) On the flip side, if we only focus only on ensuring that the latent distribution is similar to the prior distribution (through our KL divergence loss term), we end up describing *every* observation using the same unit Gaussian, which we subsequently sample from to describe the latent dimensions visualized. This effectively treats every observation as having the same characteristics; in other words, we've failed to describe the original data. However, when the two terms are optimized simultaneously, we're encouraged to describe the latent state for an observation with distributions close to the prior but deviating when necessary to describe salient features of the input. ![Screen-Shot-2018-06-20-at-2.51.06-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/06/Screen-Shot-2018-06-20-at-2.51.06-PM.png) When I'm constructing a variational autoencoder, I like to inspect the latent dimensions for a few samples from the data to see the characteristics of the distribution. I encourage you to do the same. ![6-vae](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/6-vae.png) If we observe that the latent distributions appear to be very tight, we may decide to give higher weight to the KL divergence term with a parameter $\\beta>1$, encouraging the network to learn broader distributions. This simple insight has led to the growth of a new class of models - disentangled variational autoencoders. As it turns out, by placing a larger emphasis on the KL divergence term we're also implicitly enforcing that the learned latent dimensions are uncorrelated (through our simplifying assumption of a diagonal covariance matrix). $$ {\\cal L}\\left( {x,\\hat x} \\right) + \\beta \\sum\\limits\_j {KL\\left( {{q\_j}\\left( {z|x} \\right)||N\\left( {0,1} \\right)} \\right)} $$ ## Variational autoencoders as a generative model By sampling from the latent space, we can use the decoder network to form a generative model capable of creating new data similar to what was observed during training. Specifically, we'll sample from the prior distribution ${p\\left( z \\right)}$ which we assumed follows a unit Gaussian distribution. The figure below visualizes the data generated by the decoder network of a variational autoencoder trained on the MNIST handwritten digits dataset. Here, we've sampled a grid of values from a two-dimensional Gaussian and displayed the output of our decoder network. ![vae-sample](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/vae-sample.png) As you can see, the distinct digits each exist in different regions of the latent space and smoothly transform from one digit to another. This smooth transformation can be quite useful when you'd like to interpolate between two observations, such as this recent example where [Google built a model for interpolating between two music samples](https://magenta.tensorflow.org/music-vae?ref=jeremyjordan.me). ## Further reading Lectures - [Ali Ghodsi: Deep Learning, Variational Autoencoder (Oct 12 2017)](https://www.youtube.com/watch?v=uaaqyVS9-rM&ref=jeremyjordan.me) - [UC Berkley Deep Learning Decall Fall 2017 Day 6: Autoencoders and Representation Learning](https://www.youtube.com/watch?v=R3DNKE3zKFk&ref=jeremyjordan.me) - [Stanford CS231n: Lecture on Variational Autoencoders](https://youtu.be/5WoItGTWV54?t=26m32s&ref=jeremyjordan.me) Blogs/videos - [Building Variational Auto-Encoders in TensorFlow (with great code examples)](https://danijar.com/building-variational-auto-encoders-in-tensorflow/?ref=jeremyjordan.me) - [Building Autoencoders in Keras](https://blog.keras.io/building-autoencoders-in-keras.html?ref=jeremyjordan.me) - [Variational Autoencoders - Arxiv Insights](https://www.youtube.com/watch?v=9zKuYvjFFS8&ref=jeremyjordan.me) - [Intuitively Understanding Variational Autoencoders](https://towardsdatascience.com/intuitively-understanding-variational-autoencoders-1bfe67eb5daf?ref=jeremyjordan.me) - [Density Estimation: A Neurotically In-Depth Look At Variational Autoencoders](http://ruishu.io/2018/03/14/vae/?ref=jeremyjordan.me) - [Pyro: Variational Autoencoders](http://pyro.ai/examples/vae.html?ref=jeremyjordan.me) - [Under the Hood of the Variational Autoencoder](http://blog.fastforwardlabs.com/2016/08/22/under-the-hood-of-the-variational-autoencoder-in.html?ref=jeremyjordan.me) - [With Great Power Comes Poor Latent Codes: Representation Learning in VAEs](https://towardsdatascience.com/with-great-power-comes-poor-latent-codes-representation-learning-in-vaes-pt-2-57403690e92b?ref=jeremyjordan.me) - [Kullback-Leibler Divergence Explained](https://www.countbayesie.com/blog/2017/5/9/kullback-leibler-divergence-explained?ref=jeremyjordan.me) - [Neural Discrete Representation Learning](https://avdnoord.github.io/homepage/vqvae/?ref=jeremyjordan.me) Papers/books - [Deep learning book (Chapter 20.10.3): Variational Autoencoders](http://www.deeplearningbook.org/contents/generative%5Fmodels.html?ref=jeremyjordan.me) - [Variational Inference: A Review for Statisticians](https://arxiv.org/pdf/1601.00670.pdf?ref=jeremyjordan.me) - [A tutorial on variational Bayesian inference](http://www.robots.ox.ac.uk/~sjrob/Pubs/vbTutorialFinal.pdf?ref=jeremyjordan.me) - [Auto-Encoding Variational Bayes](https://arxiv.org/abs/1312.6114?ref=jeremyjordan.me) - [Tutorial on Variational Autoencoders](https://arxiv.org/abs/1606.05908?ref=jeremyjordan.me) Papers on my reading list - [Neural Discrete Representation Learning](https://arxiv.org/abs/1711.00937?ref=jeremyjordan.me) - [Early Visual Concept Learning with Unsupervised Deep Learning](https://arxiv.org/abs/1606.05579?ref=jeremyjordan.me) - [Multimodal Unsupervised Image-to-Image Translation](https://arxiv.org/abs/1804.04732?ref=jeremyjordan.me) - [Video from paper](https://www.youtube.com/watch?v=ab64TWzWn40&ref=jeremyjordan.me) ### Introduction to autoencoders. URL: https://www.jeremyjordan.me/autoencoders/ Last updated: 2018-03-19T04:25:45.000Z Autoencoders are an unsupervised learning technique in which we leverage neural networks for the task of **representation learning**. Specifically, we'll design a neural network architecture such that we *impose a bottleneck in the network which forces a *compressed* knowledge representation of the original input*. If the input features were each independent of one another, this compression and subsequent reconstruction would be a very difficult task. However, if some sort of structure exists in the data (ie. correlations between input features), this structure can be learned and consequently leveraged when forcing the input through the network's bottleneck. ![Screen-Shot-2018-03-06-at-3.17.13-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/Screen-Shot-2018-03-06-at-3.17.13-PM.png) As visualized above, we can take an unlabeled dataset and frame it as a supervised learning problem tasked with outputting $\\hat x$, a **reconstruction of the original input** $x$. This network can be trained by minimizing the *reconstruction error*, ${\\cal L}\\left( {x,\\hat x} \\right)$, which measures the differences between our original input and the consequent reconstruction. The bottleneck is a key attribute of our network design; without the presence of an information bottleneck, our network could easily learn to simply memorize the input values by passing these values along through the network (visualized below). ![Screen-Shot-2018-03-06-at-6.09.05-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/Screen-Shot-2018-03-06-at-6.09.05-PM.png) A bottleneck constrains the amount of information that can traverse the full network, forcing a learned compression of the input data. *Note: In fact, if we were to construct a linear network (ie. without the use of nonlinear [activation functions](https://www.jeremyjordan.me/neural-networks-activation-functions/) at each layer) we would observe a similar dimensionality reduction as observed in PCA. [See Geoffrey Hinton's discussion of this here.](https://www.coursera.org/learn/neural-networks/lecture/JiT1i/from-pca-to-autoencoders-5-mins?ref=jeremyjordan.me)* The ideal autoencoder model balances the following: - Sensitive to the inputs enough to accurately build a reconstruction. - Insensitive enough to the inputs that the model doesn't simply memorize or overfit the training data. This trade-off forces the model to maintain only the variations in the data required to reconstruct the input without holding on to redundancies within the input. For most cases, this involves constructing a loss function where one term encourages our model to be sensitive to the inputs (ie. reconstruction loss ${\\cal L}\\left( {x,\\hat x} \\right)$) and a second term discourages memorization/overfitting (ie. an added regularizer). $$ {\\cal L}\\left( {x,\\hat x} \\right) + regularizer $$ We'll typically add a scaling parameter in front of the regularization term so that we can adjust the trade-off between the two objectives. In this post, I'll discuss some of the standard autoencoder architectures for imposing these two constraints and tuning the trade-off; in a follow-up post I'll discuss [variational autoencoders](https://www.jeremyjordan.me/variational-autoencoders/) which builds on the concepts discussed here to provide a more powerful model. #### Undercomplete autoencoder The simplest architecture for constructing an autoencoder is to constrain the number of nodes present in the hidden layer(s) of the network, limiting the amount of information that can flow through the network. By penalizing the network according to the reconstruction error, our model can learn the most important attributes of the input data and how to best reconstruct the original input from an "encoded" state. Ideally, this encoding will **learn and describe latent attributes of the input data**. ![Screen-Shot-2018-03-07-at-8.24.37-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/Screen-Shot-2018-03-07-at-8.24.37-AM.png) Because neural networks are capable of learning nonlinear relationships, this can be thought of as a more powerful (nonlinear) generalization of [PCA](https://www.jeremyjordan.me/principal-components-analysis/). Whereas PCA attempts to discover a lower dimensional hyperplane which describes the original data, autoencoders are capable of [learning nonlinear manifolds](http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/?ref=jeremyjordan.me) (a manifold is defined in *simple* terms as a continuous, non-intersecting surface). The difference between these two approaches is visualized below. ![Screen-Shot-2018-03-07-at-8.52.21-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/Screen-Shot-2018-03-07-at-8.52.21-AM.png) For higher dimensional data, autoencoders are capable of learning a complex representation of the data (manifold) which can be used to describe observations in a lower dimensionality and correspondingly decoded into the original input space. ![LinearNonLinear](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/LinearNonLinear.png) [Image credit](https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/content/unsupervised%5Flearning.html?ref=jeremyjordan.me) An undercomplete autoencoder has no explicit regularization term - we simply train our model according to the reconstruction loss. Thus, our only way to ensure that the model isn't memorizing the input data is the ensure that we've sufficiently restricted the number of nodes in the hidden layer(s). For deep autoencoders, we must also be aware of the *capacity* of our encoder and decoder models. Even if the "bottleneck layer" is only one hidden node, it's still possible for our model to memorize the training data provided that the encoder and decoder models have sufficient capability to learn some arbitrary function which can map the data to an index. Given the fact that we'd like our model to discover latent attributes within our data, it's important to ensure that the autoencoder model is not simply learning an efficient way to memorize the training data. Similar to supervised learning problems, we can employ various forms of regularization to the network in order to encourage good generalization properties; these techniques are discussed below. #### Sparse autoencoders Sparse autoencoders offer us an alternative method for introducing an information bottleneck without *requiring* a reduction in the number of nodes at our hidden layers. Rather, we'll construct our loss function such that we penalize *activations* within a layer. For any given observation, we'll encourage our network to learn an encoding and decoding which only relies on activating a small number of neurons. It's worth noting that this is a different approach towards regularization, as we normally regularize the *weights* of a network, not the activations. A generic sparse autoencoder is visualized below where the opacity of a node corresponds with the level of activation. It's important to note that the individual nodes of a trained model which activate are *data-dependent*, different inputs will result in activations of different nodes through the network. ![Screen-Shot-2018-03-07-at-1.50.55-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/Screen-Shot-2018-03-07-at-1.50.55-PM.png) One result of this fact is that **we allow our network to sensitize individual hidden layer nodes toward specific attributes of the input data**. Whereas an undercomplete autoencoder will use the entire network for every observation, a sparse autoencoder will be forced to selectively activate regions of the network depending on the input data. As a result, we've limited the network's capacity to memorize the input data without limiting the networks capability to extract features from the data. This allows us to consider the latent state representation and regularization of the network *separately*, such that we can choose a latent state representation (ie. encoding dimensionality) in accordance with what makes sense given the context of the data while imposing regularization by the sparsity constraint. There are two main ways by which we can impose this sparsity constraint; both involve measuring the hidden layer activations for each training batch and adding some term to the loss function in order to penalize excessive activations. These terms are: - **L1 Regularization**: We can add a term to our loss function that penalizes the absolute value of the vector of activations $a$ in layer $h$ for observation $i$, scaled by a tuning parameter $\\lambda$. $$ {\\cal L}\\left( {x,\\hat x} \\right) + \\lambda \\sum\\limits\_i {\\left| {a\_i^{\\left( h \\right)}} \\right|} $$ - **KL-Divergence**: In essence, KL-divergence is a measure of the difference between two probability distributions. We can define a sparsity parameter $\\rho$ which denotes the average activation of a neuron over a collection of samples. This expectation can be calculated as $ {{\\hat \\rho }\_ j} = \\frac{1}{m}\\sum\\limits\_{i} {\\left\[ {a\_i^{\\left( h \\right)}\\left( x \\right)} \\right\]} $ where the subscript $j$ denotes the specific neuron in layer $h$, summing the activations for $m$ training observations denoted individually as $x$. In essence, by constraining the average activation of a neuron over a collection of samples we're encouraging neurons to only fire for a subset of the observations. We can describe $\\rho$ as a Bernoulli random variable distribution such that we can leverage the KL divergence (expanded below) to compare the ideal distribution $\\rho$ to the observed distributions over all hidden layer nodes $\\hat \\rho$. $$ {\\cal L}\\left( {x,\\hat x} \\right) + \\sum\\limits\_{j} {KL\\left( {\\rho ||{{\\hat \\rho }\_ j}} \\right)} $$ *Note: A [Bernoulli distribution](https://en.wikipedia.org/wiki/Bernoullidistribution?ref=jeremyjordan.me) is "the probability distribution of a random variable which takes the value 1 with probability $p$ and the value 0 with probability $q=1-p$". This corresponds quite well with establishing the probability a neuron will fire.* The KL divergence between two Bernoulli distributions can be written as $\\sum\\limits\_{j = 1}^{{l^{\\left( h \\right)}}} {\\rho \\log \\frac{\\rho }{{{{\\hat \\rho }\_ j}}}} + \\left( {1 - \\rho } \\right)\\log \\frac{{1 - \\rho }}{{1 - {{\\hat \\rho }\_ j}}}$. This loss term is visualized below for an ideal distribution of $\\rho = 0.2$, corresponding with the minimum (zero) penalty at this point. ![KLPenaltyExample-1](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/KLPenaltyExample-1.png) #### Denoising autoencoders So far I've discussed the concept of training a neural network where the input and outputs are identical and our model is tasked with reproducing the input as closely as possible while passing through some sort of information bottleneck. Recall that I mentioned we'd like our autoencoder to be sensitive enough to recreate the original observation but insensitive enough to the training data such that the model learns a generalizable encoding and decoding. Another approach towards developing a generalizable model is to slightly corrupt the input data but still maintain the uncorrupted data as our target output. ![Screen-Shot-2018-03-09-at-10.20.44-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/Screen-Shot-2018-03-09-at-10.20.44-AM.png) With this approach, **our model isn't able to simply develop a mapping which memorizes the training data because our input and target output are no longer the same**. Rather, the model learns a vector field for mapping the input data towards a lower-dimensional manifold (recall from my earlier graphic that a manifold describes the high density region where the input data concentrates); if this manifold accurately describes the natural data, we've effectively "canceled out" the added noise. ![Screen-Shot-2018-03-09-at-10.12.59-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/Screen-Shot-2018-03-09-at-10.12.59-PM.png) [Image credit](https://arxiv.org/abs/1211.4246?ref=jeremyjordan.me) The above figure visualizes the vector field described by comparing the reconstruction of $x$ with the original value of $x$. The yellow points represent training examples prior to the addition of noise. As you can see, the model has learned to adjust the corrupted input towards the learned manifold. It's worth noting that this vector field is typically only well behaved in the regions where the model has observed during training. In areas far away from the natural data distribution, the reconstruction error is both large and does not always point in the direction of the true distribution. ![Screen-Shot-2018-03-10-at-10.17.44-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/Screen-Shot-2018-03-10-at-10.17.44-AM.png) [Image credit](https://arxiv.org/abs/1211.4246?ref=jeremyjordan.me) #### Contractive autoencoders One would expect that **for very similar inputs, the learned encoding would also be very similar**. We can explicitly train our model in order for this to be the case by requiring that the *derivative of the hidden layer activations are small* with respect to the input. In other words, for small changes to the input, we should still maintain a very similar encoded state. This is quite similar to a denoising autoencoder in the sense that these small perturbations to the input are essentially considered noise and that we would like our model to be robust against noise. Put in [other words](https://arxiv.org/abs/1211.4246?ref=jeremyjordan.me) (emphasis mine), "denoising autoencoders make the *reconstruction function* (ie. decoder) resist small but finite-sized perturbations of the input, while contractive autoencoders make the *feature extraction function* (ie. encoder) resist infinitesimal perturbations of the input." Because we're explicitly encouraging our model to learn an encoding in which similar inputs have similar encodings, we're essentially forcing the model to learn how to **contract** a neighborhood of inputs into a smaller neighborhood of outputs. Notice how the slope (ie. derivative) of the reconstructed data is essentially zero for local neighborhoods of input data. ![Screen-Shot-2018-03-10-at-12.25.43-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/Screen-Shot-2018-03-10-at-12.25.43-PM.png) [Image credit](https://arxiv.org/abs/1211.4246?ref=jeremyjordan.me) (modified) We can accomplish this by constructing a loss term which penalizes large *derivatives* of our *hidden layer activations* with respect to the input training examples, essentially penalizing instances where a small change in the input leads to a large change in the encoding space. In fancier mathematical terms, we can craft our regularization loss term as the squared Frobenius norm ${\\left\\lVert A \\right\\rVert\_F}$ of the Jacobian matrix ${\\bf{J}}$ for the hidden layer activations with respect to the input observations. A Frobenius norm is essentially an L2 norm for a matrix and the Jacobian matrix simply represents all first-order partial derivatives of a vector-valued function (in this case, we have a vector of training examples). For $m$ observations and $n$ hidden layer nodes, we can calculate these values as follows. $${\\left\\lVert A \\right\\rVert\_F}= \\sqrt {\\sum\\limits\_{i = 1}^m {\\sum\\limits\_{j = 1}^n {{{\\left| {{a\_{ij}}} \\right|}^2}} } } $$ J\=\[ δ a 1 ( h ) ( x ) δ x 1 ⋯ δ a 1 ( h ) ( x ) δ x m ⋮ ⋱ ⋮ δ a n ( h ) ( x ) δ x 1 ⋯ δ a n ( h ) ( x ) δ x m \] Written more succinctly, we can define our complete loss function as $$ {\\cal L}\\left( {x,\\hat x} \\right) + \\lambda {\\sum\\limits\_i {\\left\\lVert {{\\nabla \_ x}a\_i^{\\left( h \\right)}\\left( x \\right)} \\right\\rVert} ^2} $$ where ${{\\nabla\_x}a\_i^{\\left( h \\right)}\\left( x \\right)}$ defines the gradient field of our hidden layer activations with respect to the input $x$, summed over all $i$ training examples. ## Summary An autoencoder is a neural network architecture capable of discovering structure within data in order to develop a compressed representation of the input. Many different variants of the general autoencoder architecture exist with the goal of ensuring that the compressed representation represents *meaningful* attributes of the original data input; typically the biggest challenge when working with autoencoders is getting your model to actually learn a meaningful and generalizable latent space representation. Because autoencoders *learn* how to compress the data based on attributes (ie. correlations between the input feature vector) *discovered from data during training*, these models are typically only capable of reconstructing data similar to the class of observations of which the model observed during training. Applications of autoencoders include: - Anomaly detection - Data denoising (ex. images, audio) - Image inpainting - Information retrieval ## Further reading Lectures/notes - [Unsupervised feature learning - Stanford](http://ufldl.stanford.edu/tutorial/unsupervised/Autoencoders/?ref=jeremyjordan.me) - [Sparse autoencoder - Andrew Ng CS294A Lecture notes](https://web.stanford.edu/class/cs294a/sparseAutoencoder%5F2011new.pdf?ref=jeremyjordan.me) - [UC Berkley Deep Learning Decall Fall 2017 Day 6: Autoencoders and Representation Learning](https://www.youtube.com/watch?v=R3DNKE3zKFk&ref=jeremyjordan.me) Blogs/videos - [Building Autoencoders in Keras](https://blog.keras.io/building-autoencoders-in-keras.html?ref=jeremyjordan.me) - [Neural Networks, Manifolds, and Topology - Chris Olah](http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/?ref=jeremyjordan.me) Papers/books - [Deep learning book (Chapter 14): Autoencoders](http://www.deeplearningbook.org/contents/autoencoders.html?ref=jeremyjordan.me) - [What Regularized Auto-Encoders Learn from the Data Generating Distribution](https://arxiv.org/abs/1211.4246?ref=jeremyjordan.me) ### Setting the learning rate of your neural network. URL: https://www.jeremyjordan.me/nn-learning-rate/ Last updated: 2023-03-05T03:56:30.000Z In previous posts, I've discussed how we can train neural networks using [backpropagation](https://www.jeremyjordan.me/neural-networks-training/) with [gradient descent](https://www.jeremyjordan.me/gradient-descent/). One of the key hyperparameters to set in order to train a neural network is the *learning rate* for gradient descent. As a reminder, this parameter scales the magnitude of our weight updates in order to minimize the network's loss function. If your learning rate is set too low, training will progress very slowly as you are making very tiny updates to the weights in your network. However, if your learning rate is set too high, it can cause undesirable divergent behavior in your loss function. I'll visualize these cases below - if you find these visuals hard to interpret, I'd recommend reading (at least) the first section in my post on [gradient descent](https://www.jeremyjordan.me/gradient-descent/). ![Goldilocks of learning rates](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/Screen-Shot-2018-02-24-at-11.47.09-AM.png) So how do we find the optimal learning rate? > 3e-4 is the best learning rate for Adam, hands down. > > — Andrej Karpathy (@karpathy) [November 24, 2016](https://twitter.com/karpathy/status/801621764144971776?ref%5Fsrc=twsrc%5Etfw&ref=jeremyjordan.me) Perfect! I guess my job here is done. Well... not quite. > (i just wanted to make sure that people understand that this is a joke...) > > — Andrej Karpathy (@karpathy) [November 24, 2016](https://twitter.com/karpathy/status/801694597009113088?ref%5Fsrc=twsrc%5Etfw&ref=jeremyjordan.me) (Humor yourself by reading through that thread after finishing this post.) The loss landscape of a neural network (visualized below) is a function of the network's parameter values quantifying the "error" associated with using a specific configuration of parameter values when performing inference (prediction) on a given dataset. This loss landscape can look quite different, even for very similar network architectures. The images below are from a paper, [Visualizing the Loss Landscape of Neural Nets](https://arxiv.org/abs/1712.09913?ref=jeremyjordan.me), which shows how residual connections in a network can yield an smoother loss topology. ![Screen-Shot-2018-02-26-at-10.50.53-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/Screen-Shot-2018-02-26-at-10.50.53-PM.png) [Image credit](https://www.cs.umd.edu/~tomg/projects/landscapes/?ref=jeremyjordan.me) The optimal learning rate will be dependent on the topology of your loss landscape, which is in turn dependent on both your model architecture and your dataset. While using a default learning rate (ie. the defaults set by your deep learning library) may provide decent results, you can often improve the performance or speed up training by searching for an optimal learning rate. I hope you'll see in the next section that this is quite an easy task. ## A systematic approach towards finding the optimal learning rate Ultimately, we'd like a learning rate which results is a *steep decrease* in the network's loss. We can observe this by performing a simple experiment where we gradually increase the learning rate after each mini batch, recording the loss at each increment. This gradual increase can be on either a linear or exponential scale. For learning rates which are too low, the loss may decrease, but at a very shallow rate. When entering the optimal learning rate zone, you'll observe a quick drop in the loss function. Increasing the learning rate further will cause an increase in the loss as the parameter updates cause the loss to "bounce around" and even diverge from the minima. Remember, the best learning rate is associated with the *steepest* drop in loss, so we're mainly interested in analyzing the slope of the plot. ![lr_finder](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/lr_finder.png) You should set the range of your learning rate bounds for this experiment such that you observe **all three** phases, making the optimal range trivial to identify. This technique was proposed by Leslie Smith in [Cyclical Learning Rates for Training Neural Networks](https://arxiv.org/abs/1506.01186?ref=jeremyjordan.me) and evangelized by Jeremy Howard in [fast.ai's course](http://course.fast.ai/index.html?ref=jeremyjordan.me). ## Setting a schedule to adjust your learning rate during training Another commonly employed technique, known as **learning rate annealing**, recommends starting with a relatively high learning rate and then gradually lowering the learning rate during training. The intuition behind this approach is that we'd like to traverse quickly from the initial parameters to a range of "good" parameter values but then we'd like a learning rate small enough that we can explore the "deeper, but narrower parts of the loss function" (from [Karparthy's CS231n notes](http://cs231n.github.io/neural-networks-3/?ref=jeremyjordan.me#annealing-the-learning-rate)). If you're having a hard time picturing what I just mentioned, recall that too high of a learning rate can cause the parameter update to "jump over" the ideal minima and subsequent updates will either result in a continued noisy convergence in the general region of the minima, or in more extreme cases may result in divergence from the minima. The most popular form of learning rate annealing is a *step decay* where the learning rate is reduced by some percentage after a set number of training epochs. ![stepdecay](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/stepdecay.png) More generally, we can establish that it is useful to define a **learning rate schedule** in which the learning rate is updating during training according to some specified rule. #### Cyclical learning rates In the previously mentioned paper, [Cyclical Learning Rates for Training Neural Networks](https://arxiv.org/abs/1506.01186?ref=jeremyjordan.me), Leslie Smith proposes a cyclical learning rate schedule which varies between two bound values. The main learning rate schedule (visualized below) is a triangular update rule, but he also mentions the use of a triangular update in conjunction with a fixed cyclic decay or an exponential cyclic decay. ![Screen-Shot-2018-02-25-at-8.44.49-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/Screen-Shot-2018-02-25-at-8.44.49-PM.png) [Image credit](https://github.com/bckenstler/CLR?ref=jeremyjordan.me) *Note: At the end of this post, I'll provide the code to implement this learning rate schedule. Thus, if you don't care to understand the mathematical formulation you can skim past this section.* We can write the general schedule as $$ {\\eta\_t} = {\\eta\_{\\min }} + \\left( {{\\eta\_{\\max }} - {\\eta\_{\\min }}} \\right)\\left( {\\max \\left( {0,1 - x} \\right)} \\right) $$ where $x$ is defined as $$ x = \\left| {\\frac{{iterations}}{stepsize} - 2\\left( {cycle} \\right) + 1} \\right| $$ and $cycle$ can be calculated as $$ cycle = floor\\left( 1 + {\\frac{{iterations}}{{2\\left( {stepsize} \\right)}}} \\right) $$ where $\\eta\_{\\min }$ and $\\eta\_{\\max }$ define the bounds of our learning rate, $iterations$ represents the number of completed mini-batches, $stepsize$ defines one half of a cycle length. As far as I can gather, $1-x$ should always be positive, so it seems the $\\max$ operation is not strictly necessary. In order to grok how this equation works, let's progressively build it with visualizations. For the visuals below, the triangular update for 3 full cycles are shown with a step size of 100 iterations. Remember, one iteration corresponds with one mini-batch of training. Foremost, we can establish our "progress" during training in terms of half-cycles that we've completed. We measure our progress in terms of half-cycles and not full cycles so that we can acheive symmetry within a cycle (this should become more clear as you continue reading). ![grok_step1](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/grok_step1.png) Next, we compare our half-cycle progress to the number of half-cycles which will be completed at the end of the current cycle. At the beginning of a cycle, we have two half-cycles yet to be completed. At the end of a cycle, this value reaches zero. ![grok_step2](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/grok_step2.png) Next, we'll add 1 to this value in order to shift the function to be centered on the y-axis. Now we're showing our progress within a cycle with reference to the half-cycle point. ![grok_step3](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/grok_step3.png) At this point, we can take the absolute value to acheive a triangular shape within each cycle. This is the value that we assign to $x$. ![grok_step4](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/grok_step4.png) However, we'd like our learning rate schedule to start at the minumum value, increasing to the maximum value at the middle of a cycle, and then decrease back to the minimum value. We can accomplish this by simply calculating $1-x$. ![grok_step5](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/03/grok_step5.png) We now have a value which we can use to modulate the learning rate by adding some fraction of the learning rate range to the minimum learning rate (also referred to as the base learning rate). Smith writes, the main assumption behind the rationale for a cyclical learning rate (as opposed to one which *only* decreases) is "that **increasing the learning rate might have a short term negative effect and yet achieve a longer term beneficial effect**." Indeed, his paper includes several examples of a loss function evolution which temporarily deviates to higher losses while ultimately converging to a lower loss when compared with a benchmark fixed learning rate. To gain intuition on why this short-term effect would yield a long-term positive effect, it's important to understand the desirable characteristics of our converged minimum. Ultimately, we'd like our network to learn from the data in a manner which *generalizes* to unseen data. Further, a network with good generalization properties should be robust in the sense that small changes to the network's parameters don't cause drastic changes to performance. With this in mind, it makes sense that [sharp minima lead to poor generalization](https://arxiv.org/abs/1609.04836?ref=jeremyjordan.me) as small changes to the parameter values may result in a drastically higher loss. By allowing for our learning rate to *increase* at times, we can "jump out" of sharp minima which would temporarily increase our loss but may ultimately lead to convergence on a more desirable minima. *Note: Although the "flat minima for good generalization" is widely accepted, you can read a good counter-argument [here](https://arxiv.org/abs/1703.04933?ref=jeremyjordan.me).* Additionally, **increasing the learning rate can also allow for "more rapid traversal of saddle point plateaus."** As you can see in the image below, the gradients can be very small at a saddle point. Because the parameter updates are a function of the gradient, this results in our optimization taking very small steps; it can be useful to increase the learning rate here to avoid getting stuck for too long at a saddle point. ![Saddle point](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/Saddle_point.svg.png) [Image credit](https://en.wikipedia.org/wiki/Saddle%5Fpoint?ref=jeremyjordan.me#/media/File:Saddle%5FPoint%5Fbetween%5Fmaxima.svg) (with modification) *Note: A saddle point, by definition, is a critical point in which some dimensions observe a local minimum while other dimensions observe a local maximum. Because neural networks can have thousands or even millions of parameters, it's unlikely that we'll observe a true local minimum across all of these dimensions; saddle points are much more likely to occur. When I referred to "sharp minima", realistically we should picture a saddle point where the minimum dimensions are very steep while the maximum dimensions are very wide (as shown below).* ![sharp_minima](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/sharp_minima.png) #### Stochastic Gradient Descent with Warm Restarts (SGDR) A similar approach cyclic approach is known as **stochastic gradient descent with warm restarts** where an aggressive annealing schedule is combined with periodic "restarts" to the original starting learning rate. ![cosine_annealing](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/cosine_annealing.png) We can write this schedule as $$ {\\eta\_t} = \\eta\_{\\min }^i + \\frac{1}{2}\\left( {\\eta\_{\\max }^i - \\eta\_{\\min }^i} \\right)\\left( {1 + \\cos \\left( {\\frac{{{T\_{current}}}}{{{T\_i}}}} \\pi \\right) } \\right) $$ where ${\\eta\_t}$ is the learning rate at timestep $t$ (incremented each mini batch), ${\\eta\_{\\max}^i}$ and ${\\eta\_{\\min }^i}$ define the range of desired learning rates, $T\_{current}$ represents the number of epochs since the last restart (this value is calculated at every iteration and thus can take on fractional values), and $T\_{i}$ defines the number of epochs in an cycle. Let's try to break this equation down. This annealing schedule relies on the cosine function, which varies between -1 and 1\. ${\\frac{T\_{current}}{T\_i}}$ is capable of taking on values between 0 and 1, which is the input of our cosine function. The corresponding region of the cosine function is highlighted below in green. By adding 1, our function varies between 0 and 2, which is then scaled by $\\frac{1}{2}$ to now vary between 0 and 1\. Thus, we're simply taking the minimum learning rate and adding some fraction of the specified learning rate range (${\\eta\_{\\max }^i - \\eta\_{\\min }^i}$). Because this function starts at 1 and decreases to 0, the result is a learning rate which starts at the maximum of the specified range and decays to the minimum value. Once we reach the end of a cycle, $T\_{current}$ resets to 0 and we start back at the maximum learning rate. ![cosine](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/cosine.png) The authors note that this learning rate schedule can further be adapted to: 1. Lengthen the cycle as training progresses. 2. Decay ${\\eta\_{\\max }^i}$ and ${\\eta\_{\\min }^i}$ after each cycle. By drastically increasing the learning rate at each restart, we can essentially exit a local minima and continue exploring the loss landscape. *Neat idea: By snapshotting the weights at the end of each cycle, researchers were able to [build an ensemble of models at the cost of training a single model](https://arxiv.org/abs/1704.00109?ref=jeremyjordan.me). This is because the network "settles" on various local optima from cycle to cycle, as shown in the figure below.* ![Snapshot Ensembles: Train 1, get M for free](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/Screen-Shot-2018-02-26-at-9.12.57-PM.png) ## Implementation Both finding the optimal range of learning rates and assigning a learning rate schedule can be implemented quite trivially using Keras [Callbacks](https://keras.io/callbacks/?ref=jeremyjordan.me). #### Finding the optimal learning rate range We can write a Keras Callback which tracks the loss associated with a learning rate varied linearly over a defined range. #### Setting a learning rate schedule **Step Decay** For a simple step decay, we can use the `LearningRateScheduler` Callback. **Cyclical Learning Rate** To apply the cyclical learning rate technique, we can reference [this repo](https://github.com/bckenstler/CLR?ref=jeremyjordan.me) which has already implemented the technique in the paper. In fact, this repo is cited in the paper's appendix. **Stochastic Gradient Descent with Restarts** **Snapshot Ensembles** To apply the "Train 1, get M for free" technique, you can reference [this repo](https://github.com/titu1994/Snapshot-Ensembles?ref=jeremyjordan.me). ## Further reading - [Stanford CS231n: Annealing the learning rate](http://cs231n.github.io/neural-networks-3/?ref=jeremyjordan.me#annealing-the-learning-rate) - [Cyclical Learning Rates for Training Neural Networks](https://arxiv.org/abs/1506.01186?ref=jeremyjordan.me) - [Exploring loss function topology with cyclical learning rates](https://arxiv.org/abs/1702.04283?ref=jeremyjordan.me) - [SGDR: Stochastic Gradient Descent with Warm Restarts](https://arxiv.org/abs/1608.03983?ref=jeremyjordan.me) - [Snapshot Ensembles: Train 1, get M for free](https://arxiv.org/abs/1704.00109?ref=jeremyjordan.me) - (extension of concept) [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs ](https://arxiv.org/abs/1802.10026?ref=jeremyjordan.me) - [Visualizing the Loss Landscape of Neural Nets](https://arxiv.org/abs/1712.09913?ref=jeremyjordan.me) - [Optimization for Deep Learning Highlights in 2017: Sebastian Ruder](http://ruder.io/deep-learning-optimization-2017/?ref=jeremyjordan.me) - [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://arxiv.org/abs/1609.04836?ref=jeremyjordan.me) - [Sharp Minima Can Generalize For Deep Nets](https://arxiv.org/abs/1703.04933?ref=jeremyjordan.me) - [The Two Phases of Gradient Descent in Deep Learning](https://medium.com/intuitionmachine/the-peculiar-behavior-of-deep-learning-loss-surfaces-330cb741ec17?ref=jeremyjordan.me) - [Recent Advances for a Better Understanding of Deep Learning − Part I](https://towardsdatascience.com/recent-advances-for-a-better-understanding-of-deep-learning-part-i-5ce34d1cc914?ref=jeremyjordan.me) ### Learning from imbalanced data. URL: https://www.jeremyjordan.me/imbalanced-data/ Last updated: 2025-03-03T02:15:45.000Z In this blog post, I'll discuss a number of considerations and techniques for dealing with imbalanced data when training a machine learning model. The blog post will rely heavily on a sklearn contributor package called [imbalanced-learn](http://contrib.scikit-learn.org/imbalanced-learn/stable/?ref=jeremyjordan.me) to implement the discussed techniques. Training a machine learning model on an imbalanced dataset can introduce unique challenges to the learning problem. Imbalanced data typically refers to a classification problem where the number of observations per class is not equally distributed; often you'll have a large amount of data/observations for one class (referred to as the *majority class*), and much fewer observations for one or more other classes (referred to as the *minority classes*). For example, suppose you're building a classifier to classify a credit card transaction a fraudulent or authentic - you'll likely have 10,000 authentic transactions for every 1 fraudulent transaction, that's quite an imbalance! To understand the challenges that a class imbalance imposes, let's consider two common ways we'll train a model: tree-based logical rules developed according to some splitting criterion, and parameterized models updated by gradient descent. When building a tree-based model (such as a decision tree), our objective is to find logical rules which are capable of taking the full dataset and separating out the observations into their different classes. In other words, we'd like each split in the tree to increase the purity of observations such that the data is filtered into homogeneous groups. If we have a majority class present, the top of the decision tree is likely to learn splits which separate out the majority class into pure groups at the expense of learning rules which separate the minority class. ![Screen-Shot-2018-02-12-at-10.06.36-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/Screen-Shot-2018-02-12-at-10.06.36-PM.png) For a more concrete example, here's a decision tree trained on the wine quality dataset used as an example later on in this post. The field `value` represents the number of observations for each class in a given node. ![download-1](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/download-1.png) Similarly, if we're updating a parameterized model by gradient descent to minimize our loss function, we'll be spending most of our updates changing the parameter values in the direction which allow for correct classification of the majority class. In other words, many machine learning models are subject to a **frequency bias** in which they place more emphasis on learning from data observations which occur more commonly. It's worth noting that not all datasets are affected equally by class imbalance. Generally, for easy classification problems in which there's a clear separation in the data, class imbalance doesn't impede on the model's ability to learn effectively. However, datasets that are inherently more difficult to learn from see an amplification in the learning challenge when a class imbalance is introduced. ## Metrics When dealing with imbalanced data, standard classification metrics do not adequately represent your models performance. For example, suppose you are building a model which will look at a person's medical records and classify whether or not they are likely to have a rare disease. An accuracy of 99.5% might look great until you realize that it is correctly classifying the 99.5% of healthy people as "disease-free" and *incorrectly* classifying the 0.5% of people which **do** have the disease as healthy. I discussed this in my post on [evaluating a machine learning model](https://www.jeremyjordan.me/evaluating-a-machine-learning-model/), but I'll provide a discussion here as well regarding useful metrics when dealing with imbalanced data. **Precision** is defined as the fraction of relevant examples (true positives) among all of the examples which were predicted to belong in a certain class. $${\\rm{precision}} = \\frac{{{\\rm{true \\hspace{2mm} positives}}}}{{{\\rm{true \\hspace{2mm} positives + false \\hspace{2mm} positives}}}}$$ **Recall** is defined as the fraction of examples which were predicted to belong to a class with respect to all of the examples that truly belong in the class. $${\\rm{recall}} = {\\rm{ }}\\frac{{{\\rm{true \\hspace{2mm} positives}}}}{{{\\rm{true \\hspace{2mm} positives + false \\hspace{2mm} negatives}}}}$$ The following graphic does a phenomenal job visualizing the difference between precision and recall. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Precisionrecall.svg.png) [*Image credit*](https://en.wikipedia.org/wiki/Precision%5Fand%5Frecall?ref=jeremyjordan.me) We can further combine these two metrics into a single value by calcuating the f-score as defined below. $${F \_\\beta } = \\left( {1 + {\\beta ^2}} \\right)\\frac{{{\\rm{precision}} \\cdot {\\rm{recall}}}}{{\\left( {{\\beta ^2} \\cdot {\\rm{precision}}} \\right) + {\\rm{recall}}}}$$ The $\\beta$ parameter allows us to control the tradeoff of importance between precision and recall. $\\beta < 1$ focuses more on precision while $\\beta > 1$ focuses more on recall. Another common tool used to understand a model's performance is a Receiver Operating Characteristics (ROC) curve. An ROC curve visualizes an algorithm's ability to discriminate the positive class from the rest of the data. We'll do this by plotting the True Positive Rate against the False Positive Rate for varying prediction thresholds. $${\\rm{TPR}} = {\\rm{ }}\\frac{{{\\rm{true \\hspace{2mm} positives}}}}{{{\\rm{true \\hspace{2mm} positives + false \\hspace{2mm} negatives}}}}$$ $${\\rm{FPR}} = {\\rm{ }}\\frac{{{\\rm{false \\hspace{2mm} positives}}}}{{{\\rm{false \\hspace{2mm} positives + true \\hspace{2mm} negatives}}}}$$ For classifiers which only produce factor outcomes (ie. directly output a class), there exists a fixed TPR and FPR for a trained model. However, other classifiers, such as logistic regression, are capable of giving a probabilistic output (ie. the chance that a given observation belongs to the positive class). For these classifiers, we can specify the probability threshold by which above that amount we'll predict the observation belongs to the positive class. ![Screen-Shot-2018-02-15-at-12.41.38-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/Screen-Shot-2018-02-15-at-12.41.38-PM.png) [Image credit](https://en.wikipedia.org/wiki/Receiver%5Foperating%5Fcharacteristic?ref=jeremyjordan.me) and [Image credit](http://scikit-learn.org/stable/auto%5Fexamples/model%5Fselection/plot%5Froc.html?ref=jeremyjordan.me#sphx-glr-auto-examples-model-selection-plot-roc-py) If we set a very low value for this probability threshold, we can increase our True Positive Rate as we'll be more likely to capture all of the positive observations. However, this can also introduce a number of false positive classifications, increasing our False Positive Rate. Intuitively, there exists a tradeoff between maximizing our True Positive Rate and minimizing our False Positive Rate. The ideal model would correctly identify all positive observations as belonging to the positive class (TPR=1) and would not incorrectly classify negative observations as belonging to the positive class (FPR=0). ![roc_cutoff-1](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/11/roc_cutoff-1.gif) [*Image credit*](https://github.com/dariyasydykova/open%5Fprojects/blob/master/ROC%5Fanimation/cutoff.gif?ref=jeremyjordan.me) This tradeoff can be visualized [in this demonstration](http://www.navan.name/roc/?ref=jeremyjordan.me) in which you can adjust the class distributions and classification threshold. The **area under the curve (AUC)** is a single-value metric for which attempts to summarize an ROC curve to evaluate the quality of a classifier. As the name implies, this metric approximates the area under the ROC curve for a given classifier. Recall that the ideal curve hugs the upper lefthand corner as closely as possible, giving us the ability to identify all true positives while avoiding false positives; this ideal model would have an AUC of 1\. On the flipside, if your model was no better than a random guess, your TPR and FPR would increase in parallel to one another, corresponding with an AUC of 0.5. ``` import matplotlib.pyplot as plt from sklearn.metrics import roc_curve, roc_auc_score preds = model.predict(X_test) fpr, tpr, thresholds = roc_curve(y_test, preds, pos_label=1) auc = roc_auc_score(y_test, preds) fig, ax = plt.subplots() ax.plot(fpr, tpr) ax.plot([0, 1], [0, 1], color='navy', linestyle='--', label='random') plt.title(f'AUC: {auc}') ax.set_xlabel('False positive rate') ax.set_ylabel('True positive rate') ``` ## Class weight One of the simplest ways to address the class imbalance is to simply provide a weight for each class which places more emphasis on the minority classes such that the end result is a classifier which can learn equally from all classes. To calculate the proper weights for each class, you can use the sklearn utility function shown in the example below. ``` from sklearn.utils.class_weight import compute_class_weight weights = compute_class_weight('balanced', classes, y) ``` In a tree-based model where you're determining the optimal split according to some measure such as decreased entropy, you can simply scale the entropy component of each class by the corresponding weight such that you place more emphasis on the minority classes. As a reminder, the entropy of a node can be calculated as $$ \\mathop - \\sum \\limits\_i^{} {p\_i}{\\log}\\left( {{p\_i}} \\right) $$ where $p\_i$ is the fraction of data points within class $i$. In a gradient-based model, you can scale the calculated loss for each observation by the appropriate class weight such that you place more significance on the losses associated with minority classes. As a reminder, a common loss function for classification is the categorical cross entropy (which is very similar to the above equation, albeit with slight differences). This may be calculated as $$ -\\sum\_i y\_i \\log \\hat{y}\_ i $$ where $y\_i$ represents the true class (typically a one-hot encoded vector) and $\\hat{y}\_ i$ represents the predicted class distribution. ## Oversampling Another approach towards dealing with a class imbalance is to simply alter the dataset to remove such an imbalance. In this section, I'll discuss common techniques for *oversampling* the minority classes to increase the number of minority observations until we've reached a balanced dataset. #### Random oversampling The most naive method of oversampling is to randomly sample the minority classes and simply duplicate the sampled observations. With this technique, it's important to note that you're artificially reducing the variance of the dataset. #### SMOTE However, we can also use our existing dataset to synthetically generate new data points for the minority classes. Synthetic Minority Over-sampling Technique (SMOTE) is a technique that generates new observations by interpolating between observations in the original dataset. For a given observation $x\_i$, a new (synthetic) observation is generated by interpolating between one of the k-nearest neighbors, $x\_{zi}$. $$x\_{new} = x\_i + \\lambda (x\_{zi} - x\_i)$$ where $\\lambda$ is a random number in the range $\\left\[ {0,1} \\right\]$. This interpolation will create a sample on the line between $x\_{i}$ and $x\_{zi}$. ![sphx_glr_plot_illustration_generation_sample_001](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/02/sphx_glr_plot_illustration_generation_sample_001.png) [Image credit](http://contrib.scikit-learn.org/imbalanced-learn/stable/%5Fimages/sphx%5Fglr%5Fplot%5Fillustration%5Fgeneration%5Fsample%5F0011.png?ref=jeremyjordan.me) This algorithm has three options for selecting which observations, $x\_i$, to use in generating new data points. 1. `regular`: No selection rules, randomly sample all possible $x\_i$. 2. `borderline`: Separates all possible $x\_i$ into three classes using the k nearest neighbors of each point. - *noise*: all nearest-neighbors are from a different class than $x\_i$ - *in danger*: at least half of the nearest neighbors are of the same class as $x\_i$ - *safe*: all nearest neighbors are from the same class as $x\_i$ 3. `svm`: Uses an SVM classifier to identify the support vectors (samples close to the decision boundary) and samples $x\_i$ from these points. #### ADASYN Adaptive Synthetic (ADASYN) sampling works in a similar manner as SMOTE, however, the number of samples generated for a given $x\_i$ is proportional to the number of nearby samples which **do not** belong to the same class as $x\_i$. Thus, ADASYN tends to focus solely on outliers when generating new synthetic training examples. ## Undersampling Rather than oversampling the minority classes, it's also possible to achieve class balance by *undersampling* the majority class - essentially throwing away data to make it easier to learn characteristics about the minority classes. #### Random undersampling As with oversampling, a naive implementation would be to simply sample the majority class at random until reaching a similar number of observations as the minority classes. For example, if your majority class has 1,000 observations and you have a minority class with 20 observations, you would collect your training data for the majority class by randomly sampling 20 observations from the original 1,000\. As you might expect, this could potentially result in removing key characteristics of the majority class. #### Near miss The general idea behind near miss is to only the sample the points from the majority class necessary to distinguish between other classes. **NearMiss-1** select samples from the majority class for which the average distance of the N *closest* samples of a minority class is smallest. ![sphx_glr_plot_illustration_nearmiss_0011](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/02/sphx_glr_plot_illustration_nearmiss_0011.png) [Image credit](http://contrib.scikit-learn.org/imbalanced-learn/stable/under%5Fsampling.html?ref=jeremyjordan.me) **NearMiss-2** select samples from the majority class for which the average distance of the N *farthest* samples of a minority class is smallest. ![sphx_glr_plot_illustration_nearmiss_0021](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/02/sphx_glr_plot_illustration_nearmiss_0021.png) [Image credit](http://contrib.scikit-learn.org/imbalanced-learn/stable/under%5Fsampling.html?ref=jeremyjordan.me) #### Tomeks links A Tomek’s link is defined as two observations of different classes ($x$ and $y$) such that there is no example $z$ for which: $$d(x, z) < d(x, y) \\text{ or } d(y, z) < d(x, y)$$ where $d()$ is the distance between the two samples. **In other words, a Tomek’s link exists if two observations of different classes are the nearest neighbors of each other.** In the figure below, a Tomek’s link is illustrated by highlighting the samples of interest in green. ![sphx_glr_plot_illustration_tomek_links_0011](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/02/sphx_glr_plot_illustration_tomek_links_0011.png) [Image credit](http://contrib.scikit-learn.org/imbalanced-learn/stable/under%5Fsampling.html?ref=jeremyjordan.me) For this undersampling strategy, we'll remove any observations *from the majority class* for which a Tomek's link is identified. Depending on the dataset, this technique won't actually achieve a balance among the classes - it will simply "clean" the dataset by removing some noisy observations, which may result in an easier classification problem. As I discussed earlier, most classifiers will still perform adequately for imbalanced datasets as long as there's a clear separation between the classifiers. Thus, by focusing on removing noisy examples of the majority class, we can improve the performance of our classifier even if we don't necessarily balance the classes. #### Edited nearest neighbors EditedNearestNeighbours applies a nearest-neighbors algorithm and “edit” the dataset by removing samples which do not agree “enough” with their neighboorhood. For each sample in the class to be under-sampled, the nearest-neighbours are computed and if the selection criterion is not fulfilled, the sample is removed. This is a similar approach as Tomek's links in the respect that we're not necessarily focused on actually achieving a class balance, we're simply looking to remove noisy observations in an attempt to make for an easier classification problem. ## Example To demonstrate these various techniques, I've trained a number of models on the [UCI Wine Quality dataset](https://archive.ics.uci.edu/ml/datasets/wine+quality?ref=jeremyjordan.me) where I've generated my target by asserting that observations with a quality rating less than or equal to 4 are "low quality" wine and observations with a quality rating greater than or equal to 5 are "high quality" wine. As you can see, this has introduced an imbalance between the two classes. ![data visualization](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/download.png) I'll provide the notebook I wrote to explore these techniques in a [Github repo](https://github.com/jeremyjordan/imbalanced-data?ref=jeremyjordan.me) if you're interested in exploring this further. I highly encourage you to check out this notebook and perform the same experiment on a different dataset to see how it compares - let me know in the comment section! #### Decision tree ##### Oversampling ![dt1](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/download--1-.png) ##### Undersampling ![dt2](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/download--2-.png) #### Random forest ##### Oversampling ![rf1](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/download--3--1.png) ##### Undersampling ![rf2](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/download--4--1.png) #### Logistic regression ##### Oversampling ![lr1](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/lr1.png) ##### Undersampling ![lr2](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/lr2.png) #### SVM Classifier ##### Oversampling ![svm1](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/svm1.png) ##### Undersampling ![svm2](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/svm2.png) #### KNN ##### Oversampling ![knn1](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/knn1.png) ##### Undersampling ![knn2](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/knn2.png) #### AdaBoost Classifier ##### Oversampling ![ab1](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/ab1.png) ##### Undersampling ![ab2](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/ab2.png) #### Simple neural network ##### Oversampling ![nn1](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/nn1.png) ##### Undersampling ![nn2](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/02/nn2.png) ## Further reading - [Learning from Imbalanced Data - Literature Review](http://ieeexplore.ieee.org/document/5128907/?ref=jeremyjordan.me) - [Learning from Imbalanced Classes](https://svds.com/learning-imbalanced-classes/?ref=jeremyjordan.me) - [Learning from imbalanced data: open challenges and future directions](https://link.springer.com/article/10.1007/s13748-016-0094-0?ref=jeremyjordan.me) - [Handling imbalanced datasets in machine learning](https://towardsdatascience.com/handling-imbalanced-datasets-in-machine-learning-7a0e84220f28?ref=jeremyjordan.me) ### Normalizing your data (specifically, input and batch normalization). URL: https://www.jeremyjordan.me/batch-normalization/ Last updated: 2025-03-03T02:14:13.000Z In this post, I'll discuss considerations for normalizing your data - with a specific focus on neural networks. In order to understand the concepts discussed, it's important to have an understanding of [gradient descent](https://www.jeremyjordan.me/gradient-descent/). As a quick refresher, when training neural networks we'll feed in observations and compare the expected output to the true output of the network. We'll then use gradient descent to update the parameters of the model in the direction which will minimize the difference between our expected (or ideal) outcome and the true outcome. In other words, we're attempting to minimize the error we observe in our model's predictions. The exact manner by which we update our model parameters will depend on the variant of gradient descent optimization techniques we select (stochastic gradient descent, RMSProp, Adam, etc.) but all of these update techniques will scale the magnitude of our parameter update by a learning rate. This rate ensures that we aren't changing the parameter too drastically such that we overshoot our update and fail to find the optimal value. ![Screen-Shot-2018-01-21-at-10.15.28-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/01/Screen-Shot-2018-01-21-at-10.15.28-PM.png) Ultimately, gradient descent is a search among a loss function surface in an attempt to find the values for each parameter such that the loss function is minimized. In other words, we're looking for the lowest value on the loss function surface. In my post on [gradient descent](https://www.jeremyjordan.me/gradient-descent/), I discussed a few advanced techniques for efficiently updating our parameter values such that we can avoid getting stuck at saddle points. However, we can also improve the actual topology of our loss function by ensuring all of the parameters exist on the same scale. *Note: Understanding the topology of loss functions, and how network design affects this topology, is a [current area of research](https://arxiv.org/abs/1712.09913?ref=jeremyjordan.me) in the field.* The important thing to remember throughout this discussion is that our loss function surface is characterized by the parameter values in the network. When visualizing this topology, each parameter will represent a dimension of which a range of values will have a resulting affect on the value of our loss function. Unfortunately, this becomes rather tricky to visualize once you extend beyond two parameters (a dimension characterized by each parameter, and the third dimension representing the value of the loss function). ![opt1](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/01/opt1.gif) In the above image, we're visualizing the loss function of a model parameterized by two weights (the x and y dimensions) with the z dimension representing the corresponding "error" (loss) of the network. This 3D visualization is often also represented by a 2D contour plot. ## What's the problem with unnormalized data? Let's take a second to imagine a scenario in which you have a very simple neural network with two inputs. The first input value, $x\_1$, varies from 0 to 1 while the second input value, $x\_2$, varies from 0 to 0.01\. Since your network is tasked with learning how to *combine* these inputs through a series of linear combinations and nonlinear activations, the parameters associated with each input will also exist on different scales. Unfortunately, this can lead toward an awkward loss function topology which places more emphasis on certain parameter gradients. ![Screen-Shot-2018-01-23-at-2.27.20-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/01/Screen-Shot-2018-01-23-at-2.27.20-PM.png) *More discussion on this subject found [here](https://www.coursera.org/learn/neural-networks/lecture/1Vq6C/a-bag-of-tricks-for-mini-batch-gradient-descent?ref=jeremyjordan.me).* By normalizing all of our inputs to a standard scale, we're allowing the network to *more quickly* learn the optimal parameters for each input node. Additionally, it's useful to ensure that our inputs are roughly in the range of -1 to 1 to avoid weird [mathematical artifacts associated with floating point number precision](https://www.youtube.com/watch?v=PZRI1IfStY0&ref=jeremyjordan.me). In short, computers lose accuracy when performing math operations on really large or really small numbers. Moreover, if your inputs and target outputs are on a completely different scale than the typical -1 to 1 range, the default parameters for your neural network (ie. learning rates) will likely be ill-suited for your data. #### Implementation It's a common practice to scale your data inputs to have zero mean and unit variance. This is known as the [standard scaler](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html?ref=jeremyjordan.me#sklearn.preprocessing.StandardScaler) approach. However, you may opt for a different normalization strategy. For example, it's common for image data to simply be scaled by 1/255 so that the pixel intensity range is bound by 0 and 1. ## Batch normalization Normalizing the input of your network is a well-established technique for improving the convergence properties of a network. A few years ago, a technique known as [batch normalization](https://arxiv.org/abs/1502.03167?ref=jeremyjordan.me) was proposed to extend this improved loss function topology to more of the parameters of the network. ![Screen-Shot-2018-01-24-at-1.12.52-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/01/Screen-Shot-2018-01-24-at-1.12.52-PM.png) If we were to consider the above network as an example, normalizing our inputs will help ensure that our network can effectively learn the parameters in the first layer. However, consider the fact that the second layer of our network accepts the activations from our first layer as input. Thus, by extending the intuition established in the previous section, one could posit that normalizing *these* values will help the network more effectively learn the parameters in the second layer. ![Screen-Shot-2018-01-24-at-1.16.09-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/01/Screen-Shot-2018-01-24-at-1.16.09-PM.png) By ensuring the activations of *each* layer are normalized, we can simplify the overall loss function topology. This is especially helpful for the hidden layers of our network, since the distribution of unnormalized activations from previous layers will change as the network evolves and learns more optimal parameters. Thus, by normalizing each layer, we're introducing a level of [orthogonality](https://www.coursera.org/learn/machine-learning-projects/lecture/FRvQe/orthogonalization?ref=jeremyjordan.me) between layers - which generally makes for an easier learning process. #### Implementation To summarize, we'd like to normalize the activations of a given layer such that we improve learning of the weights which connect the next layer. In practice, people will typically normalize the value of ${z^{\\left\[ l \\right\]}}$ rather than ${a^{\\left\[ l \\right\]}}$ - although sometimes debated whether we should normalize before or after activation. Given a vector of linear combinations from the previous layer ${z^{\\left\[ l \\right\]}}$ for each observation $i$ in a dataset, we can calculate the mean and variance as: $$ \\mu = \\frac{1}{m}\\sum\\limits\_i {z\_i^{\\left\[ l \\right\]}} $$ $$ {\\sigma ^2} = \\frac{1}{m}{\\sum\\limits\_i {\\left( {z\_i^{\\left\[ l \\right\]} - \\mu } \\right)} ^2} $$ Using these values, we can normalize the vectors ${z^{\\left\[ l \\right\]}}$ as follows. $$z\_{norm}^{\\left( i \\right)} = \\frac{{{z^{\\left( i \\right)}} - \\mu }}{{\\sqrt {{\\sigma ^2} + \\varepsilon } }}$$ We add a very small number $\\epsilon$ to prevent the chance of a divide by zero error. However, it may not be the case that we always want to normalize $z$ to have zero mean and unit variance. In fact, this would perform poorly for some activation functions such as the sigmoid function. Thus, we'll allow our normalization scheme to learn the optimal distribution by scaling our normalized values by $\\gamma$ and shifting by $\\beta$. $${{\\tilde z}^{\\left( i \\right)}} = \\gamma z\_{norm}^{\\left( i \\right)} + \\beta $$ In other words, we've now allowed the network to normalize a layer into whichever distribution is most optimal for learning. One result of batch normalization is that we no longer need a bias vector for a batch normalized layer given that we are already shifting the normalized $z$ values with the $\\beta$ parameter. *Note: $\\mu$ and ${\\sigma ^2}$ are calculated on a per-batch basis while $\\gamma$ and $\\beta$ are learned parameters used across all batches.* ## Summary Ultimately, batch normalization allows us to build deeper networks without the need for exponentially longer training times. This is a result of introducing orthogonality between layers such that we avoid shifting distributions in activations as the parameters in earlier layers are updated. ## Further reading - [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://arxiv.org/abs/1502.03167?ref=jeremyjordan.me) - [How Does Batch Normalization Help Optimization? (No, It Is Not About Internal Covariate Shift)](https://arxiv.org/abs/1805.11604?ref=jeremyjordan.me) - [CS231n Winter 2016: Lecture 5: Neural Networks Part 2](https://www.youtube.com/watch?v=gYpoJMlgyXA&feature=youtu.be&list=PLkt2uSq6rBVctENoVBg1TpCC7OQi31AlC&t=3079&ref=jeremyjordan.me) - [Understanding the backward pass through Batch Normalization Layer](http://kratzert.github.io/2016/02/12/understanding-the-gradient-flow-through-the-batch-normalization-layer.html?ref=jeremyjordan.me) - [Layer Normalization](https://arxiv.org/abs/1607.06450?ref=jeremyjordan.me) > The paper that introduced Batch Norm [https://t.co/vkT0LioKHc](https://t.co/vkT0LioKHc?ref=jeremyjordan.me) combines clear intuition with compelling experiments (14x speedup on ImageNet!!) > > So why has 'internal covariate shift' remained controversial to this day? > > Thread 👇 [pic.twitter.com/L0BBmo0q4t](https://t.co/L0BBmo0q4t?ref=jeremyjordan.me) > > — David Page (@dcpage3) [September 11, 2019](https://twitter.com/dcpage3/status/1171867587417952260?ref%5Fsrc=twsrc%5Etfw&ref=jeremyjordan.me) ### New Year's Resolutions 2018 URL: https://www.jeremyjordan.me/new-years-resolutions-2018/ Last updated: 2023-03-05T03:57:56.000Z After revisiting my [2017 resolutions](https://www.jeremyjordan.me/new-years-resolutions-2017/) and evaluating how well I adhered each resolution, I'd like to set forth my resolutions for the coming year. This year, I'll set more measurable goals so that I can more effectively evaluate my performance at the end of this year. Just as I did for 2017, I'll revisit these goals and resolutions at the end of the year to evaluate my performance. ## Find a better work/life balance I really enjoy the work that I do, so much so that I sometimes find myself having a hard time pulling away from it. In general, I resonate with Chouinard's thoughts on a work/life balance. > “A master in the art of living draws no sharp distinction between his work and his play; his labor and his leisure; his mind and his body; his education and his recreation. He hardly knows which is which. He simply pursues his vision of excellence through whatever he is doing, and leaves others to determine whether he is working or playing. To himself, he always appears to be doing both.” ― Yvon Chouinard However, I'd like to instill a few gentle forcing functions to ensure a balanced life. - Continue to make time for learning. - Don't spend my entire day in front of a computer screen. - Escape into nature for periodic reflection. ## Read 20 books this year Last year, I was able to read roughly [1 book per month](https://www.jeremyjordan.me/books/). In the beginning of the year, I had a really solid habit of reading every night but as the year progressed my habit slipped and my reading pace slowed down. I'm setting a measurable metric this year to ensure that I stay on pace throughout the year with my reading goals. ## Post at least one article biweekly on this blog I'm off to a late start with this one, but plan to stick to this as best I can. In fact, this goal is largely a forcing function to hold me accountable for continuing to make time for learning - since I learn a great deal when writing posts on this blog. This year will probably continue to be heavily focused on machine learning concepts; I also hope to start writing about specific tips about designing infrastructure for machine learning ventures, something I'm learning as I work to get one off the ground. This said, I'd also like to get back and write more musings on life and my experiences. ## Find/grow the machine learning community in the Triangle area I spent the second half of 2017 studying machine learning, but I largely spent this time learning independently from resources on the internet and engaging in online communities. These online communities are great - I'm able to connect with people from all over the world with similar interests and diverse backgrounds. However, this year I'd really like to get more involved with my local community. ## Get a top 5% ranking in at least 3 Kaggle competitions After spending many hours honing my craft as a machine learning practioner, I'd like to see how I stack up in the world's largest data science playground, Kaggle. ### Hyperparameter tuning for machine learning models. URL: https://www.jeremyjordan.me/hyperparameter-tuning/ Last updated: 2018-12-05T01:30:25.000Z When creating a machine learning model, you'll be presented with design choices as to how to define your model architecture. Often times, we don't immediately know what the optimal model architecture should be for a given model, and thus we'd like to be able to explore a range of possibilities. In true machine learning fashion, we'll ideally ask the machine to perform this exploration and select the optimal model architecture automatically. Parameters which define the model architecture are referred to as **hyperparameters** and thus this process of searching for the ideal model architecture is referred to as *hyperparameter tuning*. These hyperparameters might address model design questions such as: - What degree of [polynomial features](https://www.jeremyjordan.me/polynomial-regression/) should I use for my [linear model](https://www.jeremyjordan.me/linear-regression/)? - What should be the maximum depth allowed for my [decision tree](https://www.jeremyjordan.me/decision-trees-for-classification/)? - What should be the minimum number of samples required at a leaf node in my decision tree? - How many trees should I include in my [random forest](https://www.jeremyjordan.me/ensemble-learning/)? - How many neurons should I have in my [neural network](https://www.jeremyjordan.me/neural-networks-representation/) layer? - How many layers should I have in my neural network? - What should I set my learning rate to for gradient descent? I want to be absolutely clear, **hyperparameters are not model parameters** and they cannot be directly trained from the data. *Model parameters* are learned during training when we optimize a loss function using something like [gradient descent](https://www.jeremyjordan.me/gradient-descent/).The process for learning parameter values is shown generally below. ![Screen-Shot-2017-11-02-at-1.28.26-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/11/Screen-Shot-2017-11-02-at-1.28.26-PM.png) Whereas the model parameters specify how to transform the input data into the desired output, the hyperparameters define how our model is actually structured. Unfortunately, there's no way to calculate “which way should I update my hyperparameter to reduce the loss?” (ie. gradients) in order to find the optimal model architecture; thus, we generally resort to experimentation to figure out what works best. In general, this process includes: 1. Define a model 2. Define the range of possible values for all hyperparameters 3. Define a method for sampling hyperparameter values 4. Define an evaluative criteria to judge the model 5. Define a cross-validation method Specifically, the various hyperparameter tuning methods I'll discuss in this post offer various approaches to Step 3. ### Model validation Before we discuss these various tuning methods, I'd like to quickly [revisit](https://www.jeremyjordan.me/evaluating-a-machine-learning-model/) the purpose of splitting our data into training, validation, and test data. The ultimate goal for any machine learning model is to learn from examples in such a manner that the model is capable of generalizing the learning to new instances which it has not yet seen. At a very basic level, you should train on a subset of your total dataset, holding out the remaining data for evaluation to gauge the model's ability to generalize - in other words, "how well will my model do on data which it hasn't directly learned from during training?" ![Screen-Shot-2017-10-31-at-7.42.50-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/10/Screen-Shot-2017-10-31-at-7.42.50-PM.png) When you start exploring various model architectures (ie. different hyperparameter values), you also need a way to evaluate each model's ability to generalize to unseen data. However, if you use the testing data for this evaluation, you'll end up "fitting" the model architecture to the testing data - losing the ability to truely evaluate how the model performs on unseen data. This is sometimes referred to as "data leakage". To mitigate this, we'll end up splitting the total dataset into three subsets: training data, validation data, and testing data. The introduction of a validation dataset allows us to evaluate the model on different data than it was trained on and select the best model architecture, while still holding out a subset of the data for the final evaluation at the end of our model development. ![Screen-Shot-2017-10-31-at-7.43.02-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/10/Screen-Shot-2017-10-31-at-7.43.02-PM.png) You can also leverage more advanced techniques such as K-fold cross validation in order to essentially combine training and validation data for both learning the model parameters and evaluating the model without introducing data leakage. ### Hyperparameter tuning methods Recall that I previously mentioned that the hyperparameter tuning methods relate to how we sample possible model architecture candidates from the space of possible hyperparameter values. This is often referred to as "searching" the hyperparameter space for the optimum values. In the following visualization, the $x$ and $y$ dimensions represent two hyperparameters, and the $z$ dimension represents the model's score (defined by some evaluation metric) for the architecture defined by $x$ and $y$. *Note: Ignore the axes values, I borrowed this image as noted and the axis values don't correspond with logical values for the hyperparameters.* ![hyperparameter_space](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/11/hyperparameter_space.png) Photo by [SigOpt](https://blog.sigopt.com/posts/common-problems-in-hyperparameter-optimization?ref=jeremyjordan.me) If we had access to such a plot, choosing the ideal hyperparameter combination would be trivial. However, calculating such a plot at the granularity visualized above would be prohibitively expensive. Thus, we are left to blindly explore the hyperparameter space in hopes of locating the hyperparameter values which lead to the maximum score. For each method, I'll discuss how to search for the optimal structure of a random forest classifer. [Random forests](https://www.jeremyjordan.me/ensemble-learning/) are an ensemble model comprised of a collection of [decision trees](https://www.jeremyjordan.me/decision-trees-for-classification/); when building such a model, two important hyperparameters to consider are: - How many estimators (ie. decision trees) should I use? - What should be the maximum allowable depth for each decision tree? ###### Grid search Grid search is arguably the most basic hyperparameter tuning method. With this technique, we simply build a model for each possible combination of all of the hyperparameter values provided, evaluating each model, and selecting the architecture which produces the best results. For example, we would define a list of values to try for both `n_estimators` and `max_depth` and a grid search would build a model for each possible combination. Performing grid search over the defined hyperparameter space ``` n_estimators = [10, 50, 100, 200] max_depth = [3, 10, 20, 40] ``` would yield the following models. ``` RandomForestClassifier(n_estimators=10, max_depth=3) RandomForestClassifier(n_estimators=10, max_depth=10) RandomForestClassifier(n_estimators=10, max_depth=20) RandomForestClassifier(n_estimators=10, max_depth=40) RandomForestClassifier(n_estimators=50, max_depth=3) RandomForestClassifier(n_estimators=50, max_depth=10) RandomForestClassifier(n_estimators=50, max_depth=20) RandomForestClassifier(n_estimators=50, max_depth=40) RandomForestClassifier(n_estimators=100, max_depth=3) RandomForestClassifier(n_estimators=100, max_depth=10) RandomForestClassifier(n_estimators=100, max_depth=20) RandomForestClassifier(n_estimators=100, max_depth=40) RandomForestClassifier(n_estimators=200, max_depth=3) RandomForestClassifier(n_estimators=200, max_depth=10) RandomForestClassifier(n_estimators=200, max_depth=20) RandomForestClassifier(n_estimators=200, max_depth=40) ``` Each model would be fit to the training data and evaluated on the validation data. As you can see, this is an **exhaustive** sampling of the hyperparameter space and can be quite inefficient. ![grid_search](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/11/grid_search.gif) Photo by [SigOpt](https://blog.sigopt.com/posts/common-problems-in-hyperparameter-optimization?ref=jeremyjordan.me) ###### Random search Random search differs from grid search in that we longer provide a discrete set of values to explore for each hyperparameter; rather, we provide a statistical distribution for each hyperparameter from which values may be randomly sampled. We'll define a sampling distribution for each hyperparameter. ``` from scipy.stats import expon as sp_expon from scipy.stats import randint as sp_randint n_estimators = sp_expon(scale=100) max_depth = sp_randint(1, 40) ``` We can also define how many iterations we'd like to build when searching for the optimal model. For each iteration, the hyperparameter values of the model will be set by sampling the defined distributions above. The `scipy` distributions above may be sampled with the `rvs()` function - feel free to explore this in Python! One of the main theoretical backings to motivate the use of random search in place of grid search is the fact that for most cases, hyperparameters are not *equally* important. > A Gaussian process analysis of the function from hyper-parameters to validation set performance reveals that **for most data sets only a few of the hyper-parameters really matter, but that different hyper-parameters are important on different data sets**. This phenomenon makes grid search a poor choice for configuring algorithms for new data sets. - [Bergstra, 2012](http://jmlr.csail.mit.edu/papers/volume13/bergstra12a/bergstra12a.pdf?ref=jeremyjordan.me) In the following example, we're searching over a hyperparameter space where the one hyperparameter has significantly more influence on optimizing the model score - the distributions shown on each axis represent the model's score. In each case, we're evaluating nine different models. The grid search strategy blatantly misses the optimal model and spends redundant time exploring the unimportant parameter. During this grid search, we isolated each hyperparameter and searched for the best possible value while holding all other hyperparameters constant. For cases where the hyperparameter being studied has little effect on the resulting model score, this results in wasted effort. Conversely, the random search has much improved exploratory power and can focus on finding the optimal value for the important hyperparameter. ![grid_vs_random](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/11/grid_vs_random.png) Photo by [Bergstra, 2012](http://jmlr.csail.mit.edu/papers/volume13/bergstra12a/bergstra12a.pdf?ref=jeremyjordan.me) As you can see, this search method works best under the assumption that not all hyperparameters are equally important. While this isn't always the case, the assumption holds true for most datasets. ![random_search](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/11/random_search.gif) Photo by [SigOpt](https://blog.sigopt.com/posts/common-problems-in-hyperparameter-optimization?ref=jeremyjordan.me) ###### Bayesian optimization The previous two methods performed individual experiments building models with various hyperparameter values and recording the model performance for each. Because each experiment was performed in isolation, it's very easy to parallelize this process. However, because each experiment was performed in isolation, we're not able to use the information from one experiment to improve the next experiment. Bayesian optimization belongs to a class of *sequential model-based optimization* (SMBO) algorithms that allow for one to use the results of our previous iteration to improve our sampling method of the next experiment. We'll initially define a model constructed with hyperparameters $\\lambda$ which, after training, is scored $v$ according to some evaluation metric. Next, we use the previously evaluated hyperparameter values to compute a posterior expectation of the hyperparameter space. We can then choose the optimal hyperparameter values according to this posterior expectation as our next model candidate. We iteratively repeat this process until converging to an optimum. We'll use a Gaussian process to model our prior probability of model scores across the hyperparameter space. This model will essentially serve to use the hyperparameter values $\\lambda\_{1,...i}$ and corresponding scores $v\_{1,...i}$ we've observed thus far to approximate a continuous score function over the hyperparameter space. This approximated function also includes the degree of certainty of our estimate, which we can use to identify the candidate hyperparameter values that would yield the largest expected improvement over the current score. The formulation for expected improvemenet is known as our acquisition function, which represents the posterior distribution of our score function across the hyperparameter space. ![Bayesian_optimization](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/11/Bayesian_optimization.gif) Photo by [SigOpt](https://blog.sigopt.com/posts/common-problems-in-hyperparameter-optimization?ref=jeremyjordan.me) *Note: these visualizations were provided by [SigOpt](https://blog.sigopt.com/posts/common-problems-in-hyperparameter-optimization?ref=jeremyjordan.me), a company that offers a Bayesian optimization product. It's not likely a coincidence that the visualized hyperparamter space is such that Bayesian optimization performs best.* ### Further reading - [Random Search for Hyper-Parameter Optimization](http://jmlr.csail.mit.edu/papers/volume13/bergstra12a/bergstra12a.pdf?ref=jeremyjordan.me) - [Tuning the hyper-parameters of an estimator](http://scikit-learn.org/stable/modules/grid%5Fsearch.html?ref=jeremyjordan.me) - [A Conceptual Explanation of Bayesian Hyperparameter Optimization for Machine Learning](https://towardsdatascience.com/a-conceptual-explanation-of-bayesian-model-based-hyperparameter-optimization-for-machine-learning-b8172278050f?ref=jeremyjordan.me) - [Common Problems in Hyperparameter Optimization](https://blog.sigopt.com/posts/common-problems-in-hyperparameter-optimization?ref=jeremyjordan.me) - [Gilles Louppe | Bayesian optimization with Scikit-Optimize](https://www.youtube.com/watch?v=DGJTEBt0d-s&ref=jeremyjordan.me) - [Hyperparameter Optimization with Keras](https://towardsdatascience.com/hyperparameter-optimization-with-keras-b82e6364ca53?ref=jeremyjordan.me) - [A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning](http://haikufactory.com/files/bayopt.pdf?ref=jeremyjordan.me) - [Population based training of neural networks](https://deepmind.com/blog/population-based-training-neural-networks/?ref=jeremyjordan.me) Hyperparameter optimization libraries (free and open source): - [Ray.tune: Hyperparameter Optimization Framework](http://ray.readthedocs.io/en/latest/tune.html?ref=jeremyjordan.me) - [Optuna](https://optuna.org/?ref=jeremyjordan.me) - [Hyperopt](https://github.com/hyperopt/hyperopt?ref=jeremyjordan.me) - [Polyaxon](https://github.com/polyaxon/polyaxon?ref=jeremyjordan.me) - [Talos](https://github.com/autonomio/talos?ref=jeremyjordan.me) - [BayesianOptimization](https://github.com/fmfn/BayesianOptimization?ref=jeremyjordan.me) - [Metric Optimization Engine](https://github.com/Yelp/MOE?ref=jeremyjordan.me) - [Spearmint](https://github.com/HIPS/Spearmint?ref=jeremyjordan.me) - [GPyOpt](https://github.com/SheffieldML/GPyOpt?ref=jeremyjordan.me) - [Scikit-Optimize](https://scikit-optimize.github.io/?ref=jeremyjordan.me#skopt.Optimizer) Hyperparameter optimization libraries (everybody's favorite commerial library): - [SigOpt](https://sigopt.com/?ref=jeremyjordan.me) Implementation examples: - [Bayesian optimisation for smart hyperparameter search](http://betatim.github.io/posts/bayesian-hyperparameter-search/?ref=jeremyjordan.me) - [Bayesian optimization with scikit-learn](https://thuijskens.github.io/2016/12/29/bayesian-optimisation/?ref=jeremyjordan.me) - [Bayesian optimization with hyperopt](https://thuijskens.github.io/2017/05/12/pydata-london/?ref=jeremyjordan.me) ### What the heck is blockchain? URL: https://www.jeremyjordan.me/blockchain-introduction/ Last updated: 2018-02-19T14:45:29.000Z Lately, I've been talking more and more about blockchain and its potential impact. As I've been learning more about the technology and sharing what I've learned with my friends, I've decided it would be useful to write an introductory post to the technology, paving the way for subsequent posts on how this technology may be used to create value in our society. At a high level, blockchain allows us to store information in a resilient manner that is tamper-resistant. Because of its structure, blockchains remove the need to place trust in some central authority to maintain and secure our data. In this post, we'll discuss how to store information in a blockchain and what makes it resistant to fraudulent activity and data tampering. ### Hashing functions The first concept we must discuss in order to talk about blockchains is a hash function, specifically, a cryptographic hash function. Hash functions are used to map data of arbitrary size to data of a fixed size (referred to as the hash). Cryptographic hash functions are a special type of hash functions which easily allows for someone to map input data to a hash, but prevents someone from being able to recreate the input data given a hash value. In this sense, a cryptographic hash function is considered a "one-way" function. The SHA256 algorithm is a popular cryptographic hashing function that is used widely across many industries to keep data (such as passwords) secure. You can play with a demo [here](https://anders.com/blockchain/hash.html?ref=jeremyjordan.me) which will calculate the hash of any input data you provide. For example, if your input data is "This is a test." the resulting hash will always be "a8a2f6ebe286697c527eb35a58b5539532e9b3ae3b64d4eb0a46fb657b41562c". ![Screen-Shot-2017-10-14-at-10.28.37-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/10/Screen-Shot-2017-10-14-at-10.28.37-PM.png) ### Creating a block Blockchains are made of individual blocks that each contain data. For the case of a digital asset such as bitcoin, this data will be transactions between people sending some quantity of the asset to one another. Each block contains some header information (discussed in the following section), the data to be stored, and something called **nonce**. The nonce is an arbitrary number that we combine with our data stored in the block to slightly alter the resulting hash. When someone refers to "mining" with respect to the blockchain, what they're really referring to is searching for the right nonce value such that our resulting hash meets some criteria. For example, we might require that the hash *must* start with four zeros; in reality, we could set our criteria to any number of things, the important part is that we're making the miner search (ie. work) for a nonce that will yield a hash meeting our criteria. Similar to the previous section, you can explore what a block is with [this](https://anders.com/blockchain/block.html?ref=jeremyjordan.me) demo. Go ahead and input some data, and then change the nonce value, observing what happens to the resulting hash. When you click the "Mine" button the computer will start with an initial nonce value of 1, and increment the nonce until the resulting hash meets the specified criteria. In this demo, we're looking for a nonce value that results in a hash beginning with four zeros. ![Screen-Shot-2017-10-14-at-11.07.09-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/10/Screen-Shot-2017-10-14-at-11.07.09-PM.png) Because the cryptographic hash function is considered a "one-way" function, there's no way to reverse engineer the function to calculate the proper nonce value. Thus, we must simply brute force guess the right nonce. This brute force guessing takes work, and finding the right nonce value serves as a **"proof of work"**. This is an important part of the process, one that we'll revisit later. A block is considered valid if the hash meets a specified criteria. ### Chaining blocks together The real value emerges when we chain blocks together as a linked list. Each block has a header which contains information on the block's sequence number and the hash of the previous block. If we include the previous hash (combined with the data and nonce value of the current block) when calculating the hash of the current block, we can create a dependency chain between blocks. ![Screen-Shot-2017-10-15-at-12.25.48-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/10/Screen-Shot-2017-10-15-at-12.25.48-AM.png) As new information flows in, we create a new block and append it to this linked list. Recall that the hash of a block depends on three things: 1. The hash of the previous block. 2. The data of the current block. 3. The nonce of the current block. \[{\\rm{SHA256}}\\left( {{\\rm{Prev\\hspace{1mm}hash, data, nonce}}} \\right) \\to {\\rm{Hash}}\] If we were to update the data of the middle block (highlighted below in purple), this would also affect the resulting hash of the block. Moreover, because the blocks are linked together by including the hash of the previous block, the following block is now changed as well (highlighted in yellow). After making a change to one block, we break the hash criteria for that block, and every subsequent block, to be considered valid (the hash no longer contains four zeros, in this example). ![Screen-Shot-2017-10-15-at-11.09.25-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/10/Screen-Shot-2017-10-15-at-11.09.25-AM.png) If we wanted to make a change to a block in the blockchain, we'd need to re-"mine" the block in order to find the proper nonce such that our hash meets the specified criteria. And because the blocks are linked, we'd need to find a new nonce value (by re-"mining") for every subsequent block such that all of the updated hash values meet the specified criteria. Thus, it is possible to change data stored on a blockchain, but it would take work to find the new nonce values in order for our hash to meet the specified criteria. This introduces a **cost** for changing data in the blockchain. I encourage you to play around with the [demo](https://anders.com/blockchain/blockchain.html?ref=jeremyjordan.me) to see how changing information in one block affects the subsequent blocks. ### A distributed network Up until now, we've discussed a somewhat annoying way of keeping track of information. Why must we subject ourselves to so much work each time we create a block? Thus far, nothing we've discussed is "tamper-proof", we've just made it laborious (and thus costly) to go back and make changes to the blockchain. What's the big deal? So far we've discussed the blockchain from the perspective of a single player, but the true blockchain protocol requires a *network* of participants, each maintaining their own blockchain. Thus, our ledger of data must be **distributed** across a network. ![Screen-Shot-2017-10-15-at-11.19.27-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/10/Screen-Shot-2017-10-15-at-11.19.27-PM.png) Thus, each member of the network is responsible for keeping their blockchain up to date and accurate by communicating with other participants on the network and listening for new blocks. ![Screen-Shot-2017-10-16-at-12.10.05-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/10/Screen-Shot-2017-10-16-at-12.10.05-AM.png) Network participants can broadcast new information to be stored on the blockchain. Block creators (ie. miners) listen for new information being broadcast and collect this information into a block, going to work searching for the proper nonce value for the hash to meet its criteria. Once a block creator has discovered the proper nonce, they broadcast this to the network and network participants add this block to the end of their chain. You can easily verify that the block creator expended work to create this block by confirming the block hash meets the specified criteria. Additionally, the block creator is typically rewarded for their work which provides the incentive to continue mining. Participants can exit and enter the network at any time; upon entering or re-entering the network, they simply adopt the most trusted blockchain found on the network and listen for new blocks broadcasted by block creators. Additionally, if at some point your blockchain conflicts with someone else's blockchain on the network, you defer to the longest chain as the true ledger. We consider the longest chain on the network to be the most trusted because that chain has had the most work put into it. By doing so, we've introduced a way to reach **decentralized consensus**. Now let's revisit this concept of "proof of work" and consider why we might subject ourselves to so much work in order to add or change information to the blockchain. Let's suppose that you broadcast a transaction to be stored on the blockchain and one of the miners decides to change this transaction. For a concrete example, suppose you were trying to send your mother some money, but the miner decides to rewrite the transaction such that you send the miner money instead of your mother. The miner could make this change to the data and then get to work finding the right nonce such that the block is considered valid, broadcasting their block to the network once they find the right nonce. ![Screen-Shot-2017-10-16-at-12.12.08-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/10/Screen-Shot-2017-10-16-at-12.12.08-AM.png) However, as long as there are other miners on the network that are not in coordination with the malevolent miner, they'll broadcast a different block to be added to the chain (one containing the true information). ![Screen-Shot-2017-10-16-at-12.12.22-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/10/Screen-Shot-2017-10-16-at-12.12.22-AM.png) Now that there's a conflict, we must wait until one chain grows to be longer than the other. In order for the malevolent miner to maintain the longest chain, they would have to work at breakneck pace furiously mining new blocks in order to keep up with the pace of the valid blockchain that all of the other miners are contributing to. At a certain point, it simply becomes infeasible for a single entity to maintain this fraudulent chain unless they control close to 50% of the mining network. Thus, because we've required a certain amount of work to be put into mining a block, we're able to make fraudulent activity cost prohibitive. Further, because the miners will only receive a reward for blocks that they contribute to the trusted blockchain, they're incentivized to maintain the most accurate blockchain themselves. The fact that the blockchain is replicated and distributed across many participants in the network is an important one. Rather than trusting one central authority, we've now developed a system that allows us to place our trust into a network. ### Consensus protocols The "proof of work" scheme discussed in this post is loosely based on the consensus protocol for bitcoin (the hash criteria was simplified for this introductory post). However, it's worth noting that there exists a whole family of consensus protocols for determining what to trust in a decentralized network. The two main approaches today for a consensus protocol are proof of work and proof of stake, the differences between the two approaches are discussed [here](https://ethereum.stackexchange.com/a/214?ref=jeremyjordan.me). ### Summary The blockchain protocol has introduced a way to securely keep track of information without the need to place trust in a central authority. This protocol combines computer networking, cryptographic techniques, and economic incentives to introduce a method for maintaining a distributed, decentralized ledger that remains secure as long as there are sufficient participants in the network. This approach of securely storing information on a network can be used for anything from keeping track of financial transactions (Bitcoin) to hosting code that can be run autonomously (Ethereum). A large number of industries are now looking at the technology to see how they can take advantage of its benefits. ### Further reading In future posts, I'll take a look at bitcoin, smart contracts, and token economies. Until then, enjoy these resources to continue exploring blockchain technologies. - [Original bitcoin paper](https://bitcoin.org/bitcoin.pdf?ref=jeremyjordan.me) - [History of blockchain developments](https://queue.acm.org/detail.cfm?id=3136559&ref=jeremyjordan.me) - [Blockchain demo by Anders Brownworth](https://anders.com/blockchain/?ref=jeremyjordan.me) - [Blockchains don’t scale. Not today, at least. But there’s hope.](https://hackernoon.com/blockchains-dont-scale-not-today-at-least-but-there-s-hope-2cb43946551a?ref=jeremyjordan.me) - [A Letter to Jamie Dimon - and anyone else still struggling to understand cryptocurrencies](https://blog.chain.com/a-letter-to-jamie-dimon-de89d417cb80?ref=jeremyjordan.me) - [Hackernoon: 3 steps to understanding blockchain](https://hackernoon.com/3-steps-to-understanding-blockchain-8a285572daa3?ref=jeremyjordan.me) - [Making Sense of “Cryptoeconomics”](https://hackernoon.com/making-sense-of-cryptoeconomics-5edea77e4e8d?ref=jeremyjordan.me) - [A Hitchhiker’s Guide to Consensus Algorithms](https://hackernoon.com/a-hitchhikers-guide-to-consensus-algorithms-d81aae3eb0e3?ref=jeremyjordan.me) - [Learn Blockchains by Building One](https://hackernoon.com/learn-blockchains-by-building-one-117428612f46?ref=jeremyjordan.me) ### Generalizing value functions for large state spaces. URL: https://www.jeremyjordan.me/rl-generalizing-value-functions/ Last updated: 2017-10-26T18:15:30.000Z Up until now, we've discussed the concept of a value function primarily as a lookup table. As our agent visits specific state-action pairs and continues to explore an environment, we update the value of that state-action pair independent of any other state-action pairs. When we'd like to know the value, we simply query our lookup table and report the value. However, this makes an incredibly difficult case for learning in environments with large state spaces. For example, the game of Go has $10^{170}$ possible states; it would be impractical to build a lookup table for such an environment - both due to the size required to store such a table and the amount of training required to visit all of the states. Even more challenging, some environments (such as the case for robotics) exist in a continuous state space. Clearly, a lookup table is not a practical model for representing our value function. Ideally, we'd have a model that can generalize concepts of value across similar states. In order to do this, we'll need to develop a set of features that adequately describe the states in such a manner that we can identify similar states based on their features. In short, we're looking for a more efficient representation of state than simply viewing each one in isolation. We can then develop a parametric model, such as a linear addition of features or a neural network, which can learn to regress the value of a state given its features. The parameters of this model can be trained to approximate our current estimates for the value of states given our agent's experience. ### Function approximation In earlier posts, we've mainly been focused on estimating the action value function, $Q\\left( {S,A} \\right)$, given that it allows us to perform model-free learning. Thus, we'll need to develop a feature representation that adequately describes state-action pairs. x( S,A )\=\[ x 1 ( S,A ) ⋮ x n ( S,A ) \] ${\\bf{x}}$ is a vector of length $n$ which contains $n$ features that describe a given state-action pair. These features will be in the input to our value function model. For the simple case of a linear model, the value of a state-action pair will be calculated as $$ \\hat q\\left( {S,A} \\right) = {x\_1}\\left( {S,A} \\right){\\theta\_1} + ... + {x\_n}\\left( {S,A} \\right){\\theta\_n} $$ or more succinctly as $$ \\hat q\\left( {S,A,\\theta } \\right) = {\\mathbf{x}}{\\left( {S,A} \\right)^{\\text{T}}}\\theta $$ where $\\theta$ represents the vector of learned parameters of our function approximator. There's three main types of value function approximation: 1. Given a state, what is the value of that state? 2. Given a state and action, what is the value of that state-action pair? 3. Given a state, what is the value of all possible actions taken from that state? ![Screen-Shot-2017-10-02-at-6.10.05-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/10/Screen-Shot-2017-10-02-at-6.10.05-PM.png) Each of these models are defined by a set of parameters, $\\theta$ (also sometimes referred to as weights, $w$). With this value function approximator, we'll input a feature vector that describes a state or state-action pair and output the approximated value. In the following section, I'll discuss how we can train these model parameters to most closely match our best estimate of the value function. ### Training the parameters of our value function approximator Suppose that we had were able to compare the true value of states to our approximated value. If this were the case, we would have a supervised learning environment where we could simply establish a loss function and adjust our parameters to minimize this loss. One popular optimization method is [gradient descent](https://www.jeremyjordan.me/gradient-descent/), a topic I've covered previously. We can use this algorithm to find a set of parameters which exist at a local minimum of the loss function. As a reminder, gradient descent allows us to find a local minimum by iteratively adjusting the parameters of a function in the negative direction of the function gradient. The parameter update may be written as $$ \\Delta \\theta = - \\alpha {\\nabla\_\\theta }J\\left( \\theta \\right) $$ where $\\alpha$ denotes the step size for the update. Let's use mean squared error as a loss function to compare the true value function, ${q\_\\pi }\\left( {S,A} \\right)$, and our approximate value function, $\\hat q\\left( {S,A,\\theta } \\right)$, characterized by parameters $\\theta$. Specifically, we'll define our loss function, $J\\left( \\theta \\right)$ as the expectation of the squared error, where the expectation is the mean value. $$ J\\left( \\theta \\right) = \\mathbb{E}\\left\[ {{{\\left( {{q\_\\pi }\\left( {S,A} \\right) - \\hat q\\left( {S,A,\\theta } \\right)} \\right)}^2}} \\right\] $$ We can calculate the gradient of our cost function using the chain rule. $$ {\\nabla\_\\theta }J\\left( \\theta \\right) = -2\\left( {{q\_\\pi }\\left( {S,A} \\right) - \\hat q\\left( {S,A,\\theta } \\right)} \\right){\\nabla\_\\theta }\\hat q\\left( {S,A,\\theta } \\right) $$ Plugging this in to our gradient descent algorithm, we can find the final rule for our parameter update. $$ \\Delta \\theta = \\alpha \\left( {{q\_\\pi }\\left( {S,A} \\right) - \\hat q\\left( {S,A,\\theta } \\right)} \\right){\\nabla\_\\theta }\\hat q\\left( {S,A,\\theta } \\right) $$ *Note: the two negative signs canceled out and the constant was absorbed into our step size, $\\alpha$.* As an example, let's look at the case where our value function approximator is a linear model. $$ \\hat q\\left( {S,A,\\theta } \\right) = {\\mathbf{x}}{\\left( {S,A} \\right)^{\\text{T}}}\\theta $$ Thus, our cost function may be written as $$ J\\left( \\theta \\right) = \\mathbb{E}\\left\[ {{{\\left( {{q\_\\pi }\\left( {S,A} \\right) - {\\mathbf{x}}{{\\left( {S,A} \\right)}^{\\text{T}}}\\theta } \\right)}^2}} \\right\] $$ where we've substituted our model in for ${\\hat q}$. The gradient of ${\\hat q}$ is simply our feature vector, ${\\nabla\_\\theta }\\hat q = {{\\mathbf{x}}\\left( {S,A} \\right)}$. Thus, the update rule for a linear model is simply the product of our step size, prediction error, and feature values. $$ \\Delta \\theta = \\alpha \\left( {{q\_\\pi }\\left( {S,A} \\right) - \\hat q\\left( {S,A,\\theta } \\right)} \\right){\\mathbf{x}}\\left( {S,A} \\right) $$ ###### Updating towards a target Unfortunately, we can't use the gradient descent equations presented above because reinforcement learning isn't a supervised learning problem; we don't know the true value function. However, we've previously covered [methods for estimating the value function](https://www.jeremyjordan.me/rl-learning-methods/) such as Monte Carlo learning, temporal difference learning, and a hybrid approach. As a reminder, we can iteratively update our estimate of the value function as our agent explores an environment and gains experience. Thus, we can substitute a target value from one of our value function estimation methods for ${q\_\\pi }$ in the gradient descent algorithm described previously. 1. For Monte Carlo learning, the target is the return, $G\_t$. $$ \\Delta {\\theta ^{MC}} = \\alpha \\left( {{G\_t} - \\hat q\\left( {S,A,\\theta } \\right)} \\right){\\nabla\_\\theta }\\hat q\\left( {S,A,\\theta } \\right) $$ The return, $G\_t$, represents the future sequence of rewards observed after an agent visited a state-action pair. This is an unbiased, albeit noisy, sample of the true value. Given data for an episode $$ \\left\\langle {\\left( {{S\_1},{A\_1}} \\right),{G\_1}} \\right\\rangle ,\\left\\langle {\\left( {{S\_2},{A\_2}} \\right),{G\_2}} \\right\\rangle ,...,\\left\\langle {\\left( {{S\_T},{A\_T}} \\right),{G\_T}} \\right\\rangle $$ we can apply supervised learning to fit a model which maps the features describing a state-action pair to the resulting return. 1. For $TD\\left( 0 \\right)$ learning, the target is ${{R\_{t}} + \\gamma \\hat q\\left( {S\_{t+1},A\_{t+1},\\theta } \\right)}$. $$ \\Delta {\\theta ^{TD\\left( 0 \\right)}} = \\alpha \\left( {\\left( {{R\_{t}} + \\gamma \\hat q\\left( {S\_{t+1},A\_{t+1},\\theta } \\right)} \\right) - \\hat q\\left( {S,A,\\theta } \\right)} \\right){\\nabla\_\\theta }\\hat q\\left( {S,A,\\theta } \\right) $$ The target, ${{R\_{t}} + \\gamma \\hat q\\left( {S\_{t+1},A\_{t+1},\\theta } \\right)}$, is a biased sample of the true value. Given set of training data $$ \\left\\langle {\\left( {{S\_1},{A\_1}} \\right),{R\_1} + \\gamma Q\\left( {{S\_2},{A\_2},\\theta } \\right)} \\right\\rangle ,\\left\\langle {\\left( {{S\_2},{A\_2}} \\right),{R\_2} + \\gamma Q\\left( {{S\_3},{A\_3},\\theta } \\right)} \\right\\rangle ,...,\\left\\langle {\\left( {{S\_T},{A\_T}} \\right),{R\_T}} \\right\\rangle $$ we can similarly fit a model which maps the features describing a state-action pair to the resulting TD target. 1. For $TD\\left( \\lambda \\right)$ learning, we discussed targets for both a forward-view and backward-view perspective. The forward-view target, ${q\_t^\\lambda }$, is simply the weighted sum of all $n$-step returns for values of $n$ ranging from $n=1,...,T$ where $T$ signifies the terminal state. $$ \\Delta {\\theta ^{TD\\left( \\lambda \\right)}} = \\alpha \\left( {q\_t^\\lambda - \\hat q\\left( {S,A,\\theta } \\right)} \\right){\\nabla\_\\theta }\\hat q\\left( {S,A,\\theta } \\right) $$ The backward-view target combines a TD error, $\\delta\_t$, with the elibility of state-action pairs to be updated. However, we now define our eligilibity trace in terms of our features (by looking at ${\\nabla\_\\theta }\\hat q$) rather than storing an eligibility for each individual state-action pair. $$ {\\delta\_t} = {R\_t} + \\gamma \\hat q\\left( {{S\_{t + 1}},{A\_{t + 1}},\\theta } \\right) - \\hat q\\left( {{S\_t},{A\_t},\\theta } \\right) $$ $$ {E\_t} = \\gamma \\lambda {E\_{t - 1}} + {\\nabla\_\\theta }\\hat q\\left( {{S\_t},{A\_t},\\theta } \\right) $$ $$ \\Delta {\\theta ^{TD\\left( \\lambda \\right)}} = \\alpha {\\delta\_t}{E\_t} $$ ### Incremental updates We can use stochastic gradient descent, an online optimization technique that updates parameters on a single observation, to train the model parameters at each step as our agent explores the environment. Essentially, we'll use our model to predict the value of taking some action in a given state, proceed to take that action and build a more informed estimate of the value of the given state-action pair, and then use gradient descent to update our parameters to minimize that difference. We'll continue to do this following the trajectory of our agent for the duration of the exploration. Because we're performing a parameter update on single observations, it's somewhat of a noisy (stochastic) optimization process, but it allows us to immediately improve our model after each step of experience and the model eventually converges to a local optimum in most cases. However, this approach often fails to converge when using a non-linear function approximator, such as a neural network. ### Batch updates Although we can reach convergence in some instances of incremental gradient descent updates, it's not a very efficient use of our experiences given that we essentially throw away each transition after updating the parameters. In fact, a better way to leverage this experience is to store it in a dataset, $D$, as a collection of individual transitions $\\left( {{s\_t},{a\_t},{r\_t},{s\_{t + 1}}} \\right)$. We can then proceed to randomly sample this dataset, referred to as **replay memory**, performing stochastic gradient descent to optimize the parameters of our value function approximator until we've reached convergence under a process known as **experience replay**. In essence, we're able to get more out of our data by going over it multiple times (ie. replaying transitions) rather than discarding transitions after only one update. In an incremental approach, the transitions used to update our parameters are highly correlated and can cause our parameter optimization to blow up (ie. fail to converge). However, with experience replay we randomly sample from a collection of transitions, breaking the correlation between observations that were present along a given trajectory. This has the effect of stabilizing the parameter optimization and yields much better results. ### Fixed targets For targets based on the temporal difference approach, it's important to note that these targets are based on the approximated value function which we are attempting to optimize. To avoid any potential feedback loops that would destabilize training, it's best to optimize the parameters of ${\\hat q}$ against a fixed target that uses parameters from some previous point in training. Thus, we'll need to store two sets of parameters: the parameters of the approximated value function after $i$ training iterations, ${\\theta\_i}$, and the fixed parameters, ${\\theta\_{fixed}}$, to be used when calculating our target. This set of fixed parameters will be updated periodically during training to improve to accuracy of our targets. For example, the cost function for a Q-learning algorithm will use the fixed parameters, ${\\theta\_{fixed}}$, in calculating the target, while optimizing ${\\theta\_i}$ with gradient descent. $$ J\\left( \\theta \\right) = {\\mathbb{E}\_ {{s\_t},{a\_t},{r\_t},{s\_{t + 1}}\~{D\_i}}}\\left\[ {{{\\left( {r + \\gamma \\mathop {\\max Q\\left( {s',a',{\\theta\_{fixed}}} \\right) - Q\\left( {s,a,{\\theta\_i}} \\right)}\\limits\_{a'} } \\right)}^2}} \\right\] $$ ### Further reading - [RL Course by David Silver - Lecture 6: Value Function Approximation](https://www.youtube.com/watch?v=UoPei5o4fps&ref=jeremyjordan.me) ### Implementations of Monte Carlo and Temporal Difference learning. URL: https://www.jeremyjordan.me/rl-learning-implementations/ Last updated: 2018-02-03T16:11:40.000Z In the [previous post](https://www.jeremyjordan.me/rl-learning-methods/), I discussed two different learning methods for reinforcement learning, Monte Carlo learning and temporal difference learning. I then provided a unifying view by considering $n$-step TD learning and establishing hybrid learning method, $TD\\left( \\lambda \\right)$. These methods are mainly focused on how to approximate a value function from experience. I introduced a new value function, $Q\\left( {s,a} \\right)$, that looks at the value of state-action pairs in order to achieve model-free learning. For all approaches, learning takes place by **policy iteration**, as shown below. ![Screen-Shot-2017-09-27-at-11.18.39-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/09/Screen-Shot-2017-09-27-at-11.18.39-PM.png) We start off by following an initial policy and explore the environment. As we observe the results of our actions, we can update our estimate of the true value function that describes the environment. We can then update our policy by querying the best actions to take according to our newly estimated value function. Ideally, this cycle continues until we've sufficiently explored the environment and have converged on an optimal policy. In this post, I'll discuss specific implementations of the Monte Carlo and TD approach to learning. However, I first must introduce the concept of **on-policy** and **off-policy** learning. A method is considered **on-policy** if the agent follows the same policy which it is iteratively improving. However, there are cases where it would be more useful to have the agent follow a *behavior policy* while learning a *target policy*. Such methods, where the agent doesn't follow the same policy that it is learning, are considered **off-policy**. ### Monte Carlo learning The Monte Carlo approach approximates the value of a state-action pair by calculating the mean return from a collection of episodes. ###### GLIE Monte-Carlo GLIE Monte-Carlo is an on-policy learning method that learns from complete episodes. For each state-action pair, we keep track of how many times the pair has been visited with a simple counter function. $$ N\\left( {{S\_t},{A\_t}} \\right) \\leftarrow N\\left( {{S\_t},{A\_t}} \\right) + 1 $$ For each episode, we can update our estimated value function using an incremental mean. $$ Q\\left( {{S\_t},{A\_t}} \\right) \\leftarrow Q\\left( {{S\_t},{A\_t}} \\right) + \\frac{1}{{N\\left( {{S\_t},{A\_t}} \\right)}}\\left( {{G\_t} - Q\\left( {{S\_t},{A\_t}} \\right)} \\right) $$ Here, $G\_t$ either represents the return from time $t$ when the agent first visited the state-action pair, or the sum of returns from each time $t$ that the agent visited the state-action pair, depending on whether you are using first-visit or every-visit Monte Carlo. We'll adopt an $\\epsilon$-greedy policy with $\\epsilon=\\frac{1}{k}$ where $k$ represents the number of episodes our agent has learned from. ### Temporal difference learning The temporal difference approach approximates the value of a state-action pair by comparing estimates at two points in time (thus the name, temporal difference). The intuition behind this method is that we can formulate a better estimate for the value of a state-action pair after having observed some of the reward that our agent accumulates after having visited a state and performing a given action. ###### Sarsa Sarsa is an on-policy $TD\\left(0 \\right)$ learning method. The name refers to the State-Action-Reward-State-Action sequence that we use to update our action value function, $Q\\left( {s,a} \\right).$ We start off in state $S$ and perform action $A$. This state-action pair yields some reward $R$ and brings us to a new state, $S'$. We then follow our policy to perform action $A'$ and look up the value of this future trajectory according to $Q\\left( {S',A'} \\right)$ (taking advantage of the fact that we define the value of a state-action pair to incorporate the value of the entire future sequence of states following our policy). ![Screen-Shot-2017-09-28-at-6.47.32-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/09/Screen-Shot-2017-09-28-at-6.47.32-PM.png) Photo by [Richard Sutton](http://ufal.mff.cuni.cz/~straka/courses/npfl114/2016/sutton-bookdraft2016sep.pdf?ref=jeremyjordan.me) We'll define our policy as $\\epsilon$-greedy with respect to $Q\\left( {s,a} \\right)$, which means that our agent will have a $1 - \\epsilon$ probability of performing the optimal action in state $S'$ and an $\\epsilon$ probability of performing a random exploratory action. The value function update equation may be written as $$ Q\\left( {S,A} \\right) \\leftarrow Q\\left( {S,A} \\right) + \\alpha \\left( {\\left( {R + \\gamma Q\\left( {S',A'} \\right)} \\right) - Q\\left( {S,A} \\right)} \\right) $$ where we include one step of experience combined with a discounted estimate for future returns as our TD target. ###### Q-learning Q-learning is an off-policy $TD\\left(0 \\right)$ learning method where the agent's next action is chosen according to a behavior policy but we evaluate state-action pairs according to a target policy. This has the effect of allowing our agent to explore while maintaining the highest possible accuracy when comparing our expected state-action value to a target. We'll define our **behavior policy**, the policy that determines the agent's actions, as $\\epsilon$-greedy with respect to $Q\\left( {s,a} \\right)$. However, we'll also define a **target policy** as greedy with respect to $Q\\left( {s,a} \\right)$. Rather than looking at a sequence of State-Action-Reward-State-Action, we instead take a sequence of State-Action-Reward-State and refer to our target policy for the subsequent action. Because our target policy is greedy, there is no stochastic behavior ($\\epsilon$ exploration) present to degrade the accuracy of our TD target. ![Screen-Shot-2017-09-28-at-6.27.05-PM-1](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/09/Screen-Shot-2017-09-28-at-6.27.05-PM-1.png) Photo by [Richard Sutton](http://ufal.mff.cuni.cz/~straka/courses/npfl114/2016/sutton-bookdraft2016sep.pdf?ref=jeremyjordan.me) The value function update equation for Q-learning may be written as $$ Q\\left( {S,A} \\right) \\leftarrow Q\\left( {S,A} \\right) + \\alpha \\left( {\\left( {R + \\gamma \\mathop {\\max }\\limits\_{a'} Q\\left( {S',A'} \\right)} \\right) - Q\\left( {S,A} \\right)} \\right) $$ where we similarly include one step of experience combined with a discounted estimate for future returns as our TD target; however, we've removed the exploratory behavior from our estimate for future returns in favor of selecting the optimal trajectory, increasing the accuracy of our TD target. ### Hybrid approach to learning ###### $n$-step Sarsa So far, we've discussed methods which seek a more informed valuation of a state-action pair by two main approaches, a one-step lookahead combined with a guess (the temporal difference approach) and a full sequence lookahead (the Monte Carlo approach). Let us now consider the spectrum of algorithms that lie between these two methods by expanding our Sarsa implementation to include $n$-step returns. The general Sarsa algorithm considers the case of $n=1$, where our more informed valuation of a state-action pair $q\_t^{\\left( 1 \\right)}$ at time $t$ may be written as: $$ q\_t^{\\left( 1 \\right)} = {R\_t} + \\gamma Q\\left( {{S\_{t + 1}}} \\right) $$ At the other end of the spectrum, the Monte Carlo approach, which considers a full sequence of observed rewards until the agent reaches a terminal timestep $T$, may be written as: $$ q\_t^{\\left( \\infty \\right)} = {R\_t} + \\gamma {R\_{t + 1}} + ... + {\\gamma ^{T - 1}}{R\_T} $$ However, we could develop $q\_t^{\\left( n \\right)}$ to generally represent a target value which incorporates $n$ steps of experience combined with a guess for the value of the future trajectory from that point onward. $$ q\_t^{\\left( n \\right)} = {R\_t} + \\gamma {R\_{t + 1}} + {\\gamma ^{n - 1}}{R\_{t + n - 1}} + {\\gamma ^n}Q\\left( {{S\_{t + n}}} \\right) $$ ![Screen-Shot-2017-09-26-at-11.39.13-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/09/Screen-Shot-2017-09-26-at-11.39.13-AM.png) Photo by [Richard Sutton](http://ufal.mff.cuni.cz/~straka/courses/npfl114/2016/sutton-bookdraft2016sep.pdf?ref=jeremyjordan.me) The $n$-step Sarsa implementation is an on-policy method that exists somewhere on the spectrum between a temporal difference and Monte Carlo approach. The value function update equation may be written as $$ Q\\left( {S,A} \\right) \\leftarrow Q\\left( {S,A} \\right) + \\alpha \\left( {q\_t^{\\left( n \\right)} - Q\\left( {S,A} \\right)} \\right) $$ where $q\_t^{\\left( n \\right)}$ is the general $n$-step target we defined above. ###### Sarsa$\\left(\\lambda\\right)$ In the last post, I discussed a general method for combining the information from all of the $n$-step returns into a single collective return by calculating the geometric sum of the returns. Recall that this approach is more robust than simply choosing the optimal $n$-step return, given the fact that the optimal $n$-step return varies according to the situation. Using a weight of $\\left( {1 - \\lambda } \\right){\\lambda ^{n - 1}}$, we can efficiently combine all $n$-step returns into a target value, $q\_t^\\lambda$. $$ q\_t^\\lambda = \\left( {1 - \\lambda } \\right)\\sum\\limits\_{n = 1}^\\infty {{\\lambda ^{n - 1}}} q\_t^{\\left( n \\right)} $$ The forward-view Sarsa$\\left(\\lambda\\right)$ simply adjusts our estimated value function towards the $q\_t^\\lambda$ target. $$ Q\\left( {{S\_t},{A\_t}} \\right) \\leftarrow Q\\left( {{S\_t},{A\_t}} \\right) + \\alpha \\left( {q\_t^\\lambda - Q\\left( {{S\_t},{A\_t}} \\right)} \\right) $$ This method will, for each state-action pair, combine the returns from a one-step lookahead, two-step lookahead, ..., $n$-step lookahead, all the way to a full reward sequence lookahead, aggregate the returns by a geometric sum, and adjust the original estimate towards this new aggregated lookahead. However, we need to develop a backward-view algorithm in order to enable our agent to learn online during each episode. Otherwise, we'd need to wait until the agent reaches a terminal state to do the forward-view algorithm described above. To do this, we'll need to replace our $n$-step aggregation approach with eligibility traces, a way of keeping track of which previously visited states are likely to be responsible for the reward accumulated (covered in more detail in the previous post). As covered in the last post, we'll initialize all eligibilities to zero, increment the eligibility of visited states, and decay all eligibilities at each time step. $$ {E\_0}\\left( {s,a} \\right) = 0 $$ $$ {E\_t}\\left( {s,a} \\right) = \\gamma \\lambda {E\_{t - 1}}\\left( {s,a} \\right) + {\\bf{1}}\\left( {{S\_t} = s,{A\_t} = a} \\right) $$ Our value function is updated in proportion to both the TD error and the eligilibilty trace. $$ Q\\left( {s,a} \\right) \\leftarrow Q\\left( {s,a} \\right) + \\alpha {\\delta \_ t}{E\_ t}\\left( {s,a} \\right) $$ This method is an on-policy approach, meaning that next actions are selected according to the learned policy. ### Further reading - [RL Course by David Silver - Lecture 4: Model-Free Prediction](https://www.youtube.com/watch?v=PnHCvfgC%5FZA&ref=jeremyjordan.me) - [RL Course by David Silver - Lecture 5: Model Free Control](https://www.youtube.com/watch?v=0g4j2k%5FGgc4&ref=jeremyjordan.me) - [Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto](http://ufal.mff.cuni.cz/~straka/courses/npfl114/2016/sutton-bookdraft2016sep.pdf?ref=jeremyjordan.me) - [Intuitive RL: Intro to Advantage-Actor-Critic (A2C)](https://hackernoon.com/intuitive-rl-intro-to-advantage-actor-critic-a2c-4ff545978752?ref=jeremyjordan.me) ### Learning in a stochastic environment. URL: https://www.jeremyjordan.me/rl-learning-methods/ Last updated: 2017-10-26T17:42:04.000Z Previously, I discussed how we can use the [Markov Decision Process](https://www.jeremyjordan.me/markov-decision-process) for planning in stochastic environments. For the process of planning, we already have an understanding of our environment via access to information given by the transfer function and reward function. In other words, we **defined** the world and simply wanted to calculate the optimal method of completing some task. For the case of reinforcement learning, our agent will **explore** and **learn** about the world as it assumes different states within the environment. You can think of **planning** as the process of taking a model (a fully defined state space, transition function, and reward function) as input and outputting a policy on how to act within the environment, whereas **reinforcement learning** is the process of taking a collection of individual events (a transition from one state to another and the resulting reward) as input and outputting a policy on how to act within the environment. These individual transitions are collected as the agent explores various states within the environment. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/08/Screen-Shot-2017-08-18-at-10.22.47-PM.png) ### Defining the value of an action [Previously](https://www.jeremyjordan.me/markov-decision-process), we defined equations to represent the utility and optimal policy for a state. $$U\\left( s \\right) = R\\left( s \\right) + \\gamma \\mathop {\\max }\\limits \_a \\sum\\limits \_{s'} {T\\left( {s,a,s'} \\right)} U\\left( {s'} \\right)$$ $$\\pi {\\left( s \\right)^\*} = \\arg \\mathop {\\max }\\limits \_a \\sum\\limits \_{s'} {T\\left( {s,a,s'} \\right)} U\\left( {s'} \\right)$$ Given a specified state, we're evaluating the different actions we could take to proceed into the next state, and choosing the action which yields the greatest benefit (ie. transfers the agent to the most valuable state). Thus, it would make sense to develop an expression which directly represents the value for performing an action in a given state. Specifically, we'll define a new function, $Q\\left( {s,a} \\right)$, that evaluates the value of existing in a state, performing an action which transitions us to a new state, and then proceeding to follow some policy thereafter. Similar to the utility function for states, we can define this as an expectation of accumulated reward - the only difference is that we're now enforcing the agent's first action. $$ Q\\left( {s,a} \\right) = E\\left\[ {\\sum\\limits\_ {t = 0}^\\infty {{\\gamma ^t}R\\left( {{S\_ t}} \\right)|{S\_ 0} = s,{A\_ 0} = a} } \\right\] $$ Next, let's expand this summation. $$ Q\\left( {s,a} \\right) = E\\left\[ {R\\left( {{S\_ t}} \\right) + \\gamma R\\left( {{S\_ {t + 1}}} \\right) + {\\gamma ^2}R\\left( {{S\_ {t + 2}}} \\right) + ...|{S\_ t} = s,{A\_ t} = a} \\right\] $$ Because we've instructed our agent to perform a specific action in state $S$, the state $S\_{t + 1}$ depends on what action we take. All of the succeeding states are determined by following the optimal policy. Thus, we could group this sequence of states together, $$ Q\\left( {s,a} \\right) = E\\left\[ {R\\left( {{S\_ t}} \\right) + \\gamma \\left( {R\\left( {{S\_ {t + 1}}} \\right) + {\\gamma}R\\left( {{S\_ {t + 2}}} \\right) + ...} \\right)|{S\_ t} = s,{A\_ t} = a} \\right\] $$ where action $a$ brings us to state $S\_ {t + 1}$ and the agent proceeds adhering to its policy. We can write this in terms of $Q$ by enforcing that the agent takes the best perceived action once it has arrived in state $S\_ {t + 1}$, and continuing to proceed similarly thereafter. $$ Q\\left( {s,a} \\right) = E\\left\[ {R\\left( {{S\_ t}} \\right) + \\gamma \\left( {\\mathop {\\max }\\limits\_ {q'} Q\\left( {{S\_ {t + 1}},{A\_ {t + 1}}} \\right)} \\right)|{S\_ t} = s,{A\_ t} = a} \\right\] $$ We can further simplify this expectation expression into our final expression for the value of a state-action pair. $$ Q\\left( {s,a} \\right) = R\\left( s \\right) + \\gamma \\sum\\limits\_ {s'} {T\\left( {s,a,s'} \\right)} \\mathop {\\max Q\\left( {s',a'} \\right)}\\limits\_ {q'} $$ While this is a formal definition of the true value for $Q\\left( {s,a} \\right)$, in practice we'll develop methods for approximating $Q\\left( {s,a} \\right)$ (ideally in a manner which converges to the true value function) from an agent's experience. With this value function, $Q\\left( {s,a} \\right)$, we can redefine our original equations for utility and policy in much simpler terms. $$ U\\left( s \\right) = \\mathop {\\max }\\limits\_ a Q\\left( {s,a} \\right) $$ $$ \\pi {\\left( s \\right)^\*} = \\mathop {\\arg \\max }\\limits\_ a Q\\left( {s,a} \\right) $$ Let's just take a moment to appreciate how beautifully simple these expressions have become. The utility of a state is simply defined as the value of the best action we can take in that state, and the policy simply mandates that our agent take the most valuable action. ### Learning from the environment To reiterate, the goal of reinforcement learning is to develop a policy in an environment where the dynamics of the system are unknown. Our agent must explore its environment and learn a policy from its experiences, updating the policy as it explores to improve the behavior of the agent. We can accomplish this by iteratively following our policy, updating the value function as our agent gains more experience, and then improving our policy based on the updated value function. This is known as **policy iteration**. As it turns out, we cannot use the definition of our optimal policy with respect to the **state value** function, $U\\left( {s} \\right)$, because the definition includes knowledge of the environment model. $$\\pi {\\left( s \\right)^\*} = \\arg \\mathop {\\max }\\limits \_a \\sum\\limits \_{s'} {T\\left( {s,a,s'} \\right)} U\\left( {s'} \\right)$$ Specifically, we do not have access to the transition function, $T\\left( {s,a,s'} \\right)$, for the environment. Alas, we cannot compute the expected value of the succeeding state without knowledge of the system dynamics. Fortunately, if we instead use the **action value** function, $Q\\left( {s,a} \\right)$, we do not require knowledge of the system dynamics in order to improve the policy. $$ \\pi {\\left( s \\right)^\*} = \\mathop {\\arg \\max }\\limits\_ a Q\\left( {s,a} \\right) $$ To improve our policy for a given state, we simply need to update our action value function as we experience transitions within the environment, and query the best action to take according to $Q\\left( {s,a} \\right)$. As the agent explores its environment, it will remember the value of the actions it has taken, allowing our agent to **learn** an approximation for the system dynamics. ### Incremental learning This next section will discover different approaches to approximating $Q\\left( {s,a} \\right)$ as our agent explores the environment. This represents our way of "storing" previous experiences to remember the value of actions. Using this updated value function, we can recalculate our optimal policy by querying for the action with the highest value in each state. We'll first discuss a method for determining the value of states and actions given the experience it has collected thus far, and then extend this value approximation to develop a method of *incrementally* updating our estimated value for each state-action pair as we explore the environment. Recall, earlier we asserted that the value of existing in a state and performing an action was a combination of the reward collected in the current state, the reward collected in the next state visited as a result of the action, and an accumulation of rewards proceeding optimally thereafter where future rewards were discounted. While we do not immediately know the optimal policy and thus cannot simply "proceed optimally thereafter", **we can use what we've experienced to develop a more accurate representation of** $Q\\left( {s,a} \\right)$, **which in turn helps us develop a more optimal policy to follow**. Upon sufficient explorations and iterations, we can arrive at the optimal policy. ###### Monte Carlo approach For Monte Carlo learning, we use a collection of complete episodes to approximate the value function. An episode is defined as the agent journey from the initial state to the terminal state, so this approach only works when your environment has a concrete ending. Let's revisit the gridworld example with a more complex environment. As a reminder, the agent can take an action to move up, down, left, or right and the green and red states are terminal states. However, we no longer know the reward for any of the states until we've visited them nor do we know how the transfer function determines the agent's next state (as we did when discussing the [Markov Decision Process](https://www.jeremyjordan.me/markov-decision-process/)). Each colored path is an episode which represents the path an agent took to reach a terminal state. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/08/Screen-Shot-2017-08-19-at-2.27.30-PM.png) Previously, we defined $Q\\left( {s,a} \\right)$ as the expectation of the accumulated reward starting from a given state and performing a given action. $$ Q\\left( {s,a} \\right) = E\\left\[ {R\\left( {{S\_ t}} \\right) + \\gamma R\\left( {{S\_ {t + 1}}} \\right) + {\\gamma ^2}R\\left( {{S\_ {t + 2}}} \\right) + ...|{S\_ t} = s,{A\_ t} = a} \\right\] $$ For Monte Carlo approximation, we'll replace this expectation with the mean value of accumulated rewards over all episodes. These accumulated rewards are sometimes referred to collectively as a return, defined as ${G\_t}$ below. $$ {G\_t} = {R\_t} + \\gamma {R\_{t + 1}} + ... + {\\gamma ^{T - 1}}{R\_T} $$ For example, suppose we wanted to calculate the value of starting in the state highlighted in yellow, and then taking an action to move right. There were two episodes in which we visited this state and proceeded by moving right. *Actually, it's important to note that we don't know what action the agent took during each episode unless I explicitly provided that information, since the environment exhibits stochastic behavior and our transfer function doesn't always bring the agent to the intended state. For simplicity's sake, in this example I will also suppose that our agent took the action to move right for both episodes.* ![Screen-Shot-2017-10-01-at-3.08.14-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/10/Screen-Shot-2017-10-01-at-3.08.14-PM.png) We can calculate the value of existing in the highlighted state and moving right for both episodes by the discounted accumulation of reward until the agent reaches a terminal state. Then, we can approximate a general value for moving right from the highlighted state by taking the mean of all episodic returns. When evaluating all of the states within an environment, there's two general ways to approach this. One approach is to consider the future trajectory of a state from the point of first visit of the state for each episode; this is known as First Visit Monte Carlo Evaluation. We only consider trajectories from a state for the first visit by the agent. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/08/Screen-Shot-2017-08-19-at-9.52.52-PM.png) The second approach is to consider the future trajectories of a state for all times that the state is visited for each episode; this is known as Every Visit Monte Carlo Evaluation. Notice how in this case we also consider the gray trajectory starting from the highlighted state after the blue trajectory passes back through the state. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/08/Screen-Shot-2017-08-19-at-9.53.02-PM.png) Earlier, we established our approximation for the action-value function as the mean return for each state-action pair over all episodes. In order to learn after each episode, we can rewrite our calculation for the mean in an incremental fashion. $$ Q\\left( {s,a} \\right) = {\\mu\_ k} = \\frac{1}{k}\\sum\\limits\_ {i = 1}^k {{G^{\\left( i \\right)}}\\left( {s,a} \\right)} $$ First, we'll pull our latest episode outside of the summation. $$ {\\mu\_k} = \\frac{1}{k}\\left( {{G^{\\left( k \\right)}}\\left( {s,a} \\right) + \\sum\\limits\_{i = 1}^{k - 1} {{G^{\\left( i \\right)}}\\left( {s,a} \\right)} } \\right) $$ We can then express the summation as a repeated mean. $$ {\\mu\_k} = \\frac{1}{k}\\left( {{G^{\\left( k \\right)}}\\left( {s,a} \\right) + \\left( {k - 1} \\right){\\mu\_{k - 1}}} \\right) $$ With a little more rearrangement, we arrive at a definition for the mean which we can update incrementally using the current episodic state-action values combined with the total approximated state-action values. $$ {\\mu\_k} = {\\mu\_{k - 1}} + \\frac{1}{k}\\left( {{G^{\\left( k \\right)}}\\left( {s,a} \\right) - {\\mu\_{k - 1}}} \\right) $$ Each time we visit a state in an episode, we can total the reward accumulated from that state onward and upon reaching a terminal state, we can update our value function approximation. We can also incrementally calculate a running mean (forgetting older values over time) by substituting $\\frac{1}{k}$ for a constant value, $\\alpha$, in cases where the environment state change over time. This results in more recent experiences holding more weight when approximating the true action value function. $$ {\\mu\_k} = {\\mu\_{k - 1}} + \\alpha \\left( {{G^{\\left( k \\right)}}\\left( {s,a} \\right) - {\\mu\_{k - 1}}} \\right) $$ Recall that ${\\mu\_k}$ is our estimate for the true value function, ${{Q}\\left( {s,a} \\right)}$, given what we've learned from $k$ episodes. In general terms, this update equation allows us to compare our previous estimate, ${\\mu\_ {k - 1}}$, to the return from the latest episode, $G^{\\left( k \\right)}$, and update our guess accordingly. $$ Q\\left( {s,a} \\right) \\leftarrow Q\\left( {s,a} \\right) + \\alpha \\left( {{G^{\\left( k \\right)}}\\left( {s,a} \\right) - Q\\left( {s,a} \\right)} \\right) $$ ###### Temporal difference approach Initially, our estimated value function is arbitrarily defined - often we assume that every state-action pair has zero value, although you can also initialize the value function to a small positive number in order to encourage exploration (known as "optimism in the face of uncertainty"). However, as our agent explores an environment and accumulates experiences, we can use these experiences to update our guess of the true value function. The value function represents the reward of completing an action, and accumulating future rewards by following a set policy. Thus, our estimated value function signals how much reward we expect to accumulate from entering a state and performing some action. In the Monte Carlo approach previously discussed, we simulated complete episodes and would then back-calculate the value of each state-action pair executed during the episode. Since we ran the simulation to completion (ie. our agent reached a terminal state), we could easily calculate the future accumulated reward from any point during the simulation. However, the Monte Carlo approach requires us to run complete episodes before we can update our estimated value function. In Monte Carlo learning, we would update the value of a state-action pair by comparing the historical mean to the episodic return, and take a step in the direction which would minimize this difference. Temporal difference learning provides an *online* learning approach, allowing us to update the estimated value function during a single episode. As the name implies, we update our value function by looking at the difference of the estimated value of a state-action pair at two different points in time (temporal difference). More concretely, we'll compare a current estimate for the value of a state-action pair to a more informed future estimate after the agent has gained more experience within the environment. For example, upon arriving at some state we'll look at the reward the agent immediately collects plus the expected future returns upon entering a subsequent state, $s'$, via performing some action, and compare this with our previous estimate for the value of this state-action pair. $$ Q\\left( {s,a} \\right) \\leftarrow Q\\left( {s,a} \\right) + \\alpha \\left( {\\left( {{R\_t} + \\gamma Q\\left( {s',a'} \\right)} \\right) - Q\\left( {s,a} \\right)} \\right) $$ The expression ${R\_t} + \\gamma Q\\left( {s',a'}\\right)$ is known as the **TD target**, and represents a better approximation for $Q\\left( {s,a} \\right)$ given the fact that it replaces an estimated reward received from existing in state $s$ with the actual reward. In this instance, the TD target is one step ahead of the state-action pair value that we're updating, but this can be extended to look $n$ steps ahead when approximating $Q\\left( {s,a} \\right)$. Comparing the TD target to the previous approximation for $Q\\left( {s,a} \\right)$ yields what is known as the **TD error**, $\\delta\_t $; our update function takes a step in the direction which minimizes the TD error. $$ \\delta\_t = {\\left( {{R\_t} + \\gamma Q\\left( {s',a'} \\right)} \\right) - Q\\left( {s,a} \\right)} $$ This approach to learning allows us to make updates before we have knowledge of the final outcome, which is clearly advantageous when learning in a non-terminating environment. ###### A hybrid approach To extend our TD expression to look $n$ steps ahead, we simply need to update our TD target. Remember, the TD target represents a more informed approximation for the value of a state-action pair by incorporating observed future returns. For $n=1$, we take one step forward, observe the reward and combine it with our estimate of future returns from that point onward, and assign this as the TD target for the original state. For $n=2$, we follow the same procedure expect this time we take two steps forward, observing the reward collected at each state. ![Screen-Shot-2017-09-26-at-11.39.13-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/09/Screen-Shot-2017-09-26-at-11.39.13-AM.png) Photo by [Richard Sutton](http://ufal.mff.cuni.cz/~straka/courses/npfl114/2016/sutton-bookdraft2016sep.pdf?ref=jeremyjordan.me) In this diagram, white circles represent states, black circles represent reward collected, and the square represents a terminal state. Generally, the TD target for $TD\\left( n \\right)$ is as follows: $$ {R\_t} + \\gamma {R\_{t + 1}} + ... + {\\gamma ^{n - 1}}{R\_{t + n - 1}} + {\\gamma ^n}Q\\left( {s',a'} \\right) $$ If we were to look all the way forward to $n = \\infty$, we would arrive at the Monte Carlo approach for learning, where we simply look at the discounted sum of rewards after leaving some state $s$ via action $a$. As we increase $n$, we're including more observations from the actual environment which reduces the bias in our estimate (see following section on comparing learning methods). However, because each next state is a result of a stochastic process, the variance increases. For different environments, there exist different values of $n$ which can most efficiently propagate information backwards. A more robust approach would be to use a TD target which efficiently considers all $n$ at once. This more robust approach is referred to as the $TD\\left( \\lambda \\right)$ approach which combines all $n$-step returns as a geometric weighted sum, decayed by a factor of $\\lambda$, as shown in the diagram below. We specifically choose to use a geometric sum for the sake of computational efficiency. ![Screen-Shot-2017-09-26-at-12.16.45-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/09/Screen-Shot-2017-09-26-at-12.16.45-PM.png) Photo by [Richard Sutton](http://ufal.mff.cuni.cz/~straka/courses/npfl114/2016/sutton-bookdraft2016sep.pdf?ref=jeremyjordan.me) Setting $\\lambda = 0$, only the one-step return has a non-zero weight and thus this is equivalent to the $TD\\left( 0 \\right)$ method. Setting $\\lambda = 1$, only the full return sequence has a non-zero weight and thus this is equivalent to the Monte Carlo method. The figure below shows weighting for each of the $n$-step returns from $n=1$ (the one-step return) through $n=\\infty$ (the full return sequence), according to various values of $\\lambda$. ![Screen-Shot-2017-09-30-at-12.10.33-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/09/Screen-Shot-2017-09-30-at-12.10.33-PM.png) The $TD\\left( \\lambda \\right)$ target, $G\_t^\\lambda$, which aggregates $n$-step returns for all values of $n$, may be written as $$G\_t^\\lambda = \\left( {1 - \\lambda } \\right)\\sum\\limits\_{n = 1}^\\infty {{\\lambda ^{n - 1}}} G\_t^{\\left( n \\right)} $$ where $G\_t^{\\left( n \\right)}$ represents the $n$-step return. The result is a more robust value approximation model. Tuning the parameter $\\lambda$ generalizes to more cases much better than you would observe tuning $n$. Up until now, I've been discussing temporal difference learning from the perspective of a forward-view, where we assign the value of a state-action pair by observing what happens in the following steps. ![Screen-Shot-2017-09-26-at-8.47.34-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/09/Screen-Shot-2017-09-26-at-8.47.34-AM.png) Photo by [Richard Sutton](http://ufal.mff.cuni.cz/~straka/courses/npfl114/2016/sutton-bookdraft2016sep.pdf?ref=jeremyjordan.me) However, in order to implement this forward-view approach, we'd need to wait until the end of each episode before we perform our updates. Fortunately, we can reframe this perspective to look at a backward-view that would allow us to update previously visited states upon sufficient exploration. For example, in the case of $TD\\left(n\\right)$ learning, each step we take along a trajectory allows us to update the value of the state $n$ steps prior to our current state. ![Screen-Shot-2017-09-26-at-8.47.48-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/09/Screen-Shot-2017-09-26-at-8.47.48-AM.png) Photo by [Richard Sutton](http://ufal.mff.cuni.cz/~straka/courses/npfl114/2016/sutton-bookdraft2016sep.pdf?ref=jeremyjordan.me) In doing this, we can send back the longer-term result of actions in previous states. However, this introduces the **credit assignment problem**. When considering updates for multiple $n$-steps back, how do we know which previous actions had an influence on the reward accumulated at the current timestep? A *frequency heuristic* would assign credit to the states which we had previously visited most frequently. This would be assuming that the states we visited most often are the most likely influence on our current outcome. A *recency heuristic* would assign credit to the states which we had visited most recently. This would be assuming that our current outcomes are most likely influenced by the states we visited most recently. An **eligibility trace** combines both of these heuristics by increasing the credit assigned to a state by a fixed amount every time the state is visited, and slowly decaying that credit over time as shown in the photo below. ![Screen-Shot-2017-09-28-at-1.14.57-PM-1](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/09/Screen-Shot-2017-09-28-at-1.14.57-PM-1.png) Photo by [Richard Sutton](http://ufal.mff.cuni.cz/~straka/courses/npfl114/2016/sutton-bookdraft2016sep.pdf?ref=jeremyjordan.me) As our agent explores the environment, we update our state-action value function in the direction which minimizes TD error, scaled by the eligibility trace for each state updated. This allows us to focus our updates on the states which we believe resulted in future returns. We'll initialize the eligibility for all state-action pairs to be zero at time $t=0$. $$ {E\_0}\\left( {s,a} \\right) = 0 $$ At each time step, all state-action pair eligibility traces are decayed by a factor of $\\gamma$. Further, we increment the eligibility of a state-action pair by one each time the pair is visited. $$ {E\_t}\\left( {s,a} \\right) = \\gamma \\lambda {E\_{t - 1}}\\left( {s,a} \\right) + {\\bf{1}}\\left( {{S\_t} = s,{A\_t} = a} \\right) $$ This allows us to perform updates to state-action pairs, at each timestep, in proportion to their eligibility, ${E\_t}\\left( {s,a} \\right)$, and the TD error, $\\delta \_t$. $$ {\\delta\_ t} = {R\_ {t + 1}} + \\gamma Q\\left( {{S\_ {t + 1}},{A\_ {t + 1}}} \\right) - Q\\left( {{S\_ t},{A\_ t}} \\right) $$ $$ Q\\left( {s,a} \\right) \\leftarrow Q\\left( {s,a} \\right) + \\alpha {\\delta \_ t}{E\_ t}\\left( {s,a} \\right) $$ While our TD error compares only a one-step lookahead to our current estimate, we have the capability to send back error to any state-action pair with a non-zero eligibility. ### Comparing learning methods ###### Bias versus variance trade-off When performing Monte Carlo learning, we updated our estimated value function based solely on experience from previous episodes; the value of a given state-action pair is calculated as the mean return over all episodes. Thus, we introduce zero bias with this approach. However, the returns from any given episode are an accumulation of stochastic transitions, which can introduce a high level of variance into our estimated value function. On the contrary, when updating our estimated value function using temporal difference learning we use one step of experience combined with an estimate of the future returns from that point onward (collectively known as the TD target). In this case, our estimation of future returns introduces bias; however, this approach has much less variance since we only have one stochastic transition. ###### Markov environments In both learning methods (TD and Monte Carlo), we update our estimate of the value function by comparing our estimate to some target, and we adjust our estimate to be closer to the target. In **TD learning**, our target is defined as $$ {R\_t} + \\gamma Q\\left( {s',a'} \\right) $$ for the case of $TD\\left( 0 \\right)$. In **Monte Carlo learning**, our target is defined as the episodic return $$ {R\_t} + \\gamma {R\_{t + 1}} + ... + {\\gamma ^\\infty }{R\_\\infty } $$ where we simply discount all of the returns subsequent to a state until the terminal state is reached. The Markov property mandates that the conditional probability of future states depend only on the current state, not on the sequence of states which preceded it. Earlier, when looking at the [value of sequences of states](https://www.jeremyjordan.me/markov-decision-process/), we used this property to collapse the entire future value of a trajectory into a value assigned to a single state. The value of any given state implicitly accounted for the future reward accumulated when following some policy. In the temporal difference approach for learning, we similarly assigned our TD target as an estimated value that considers the future sequence of states. Because TD learning exploits this Markov property, it is generally more efficient than Monte Carlo learning in Markov environments. ### Epsilon-greedy exploration As our agent explores the environment and builds an understanding for the value of its actions, we gain the ability to make "smarter" decisions. However, if we prematurely optimize our policy before fully exploring the environment, we may converge to a sub-optimal policy. Suppose you're presented two doors, which you can open and keep whatever reward lies on the other side. You open Door A and find nothing. Next, you open Door B and get $5\. Given what you've learned so far, the "optimal" policy would instruct you to continue opening Door B and forget that Door A exists entirely. However, if this occurs in a stochastic environment, how can we be so sure that there's no value in opening Door A after only one try? The simplest way to address this problem is known as epsilon-greedy exploration. The greedy action represents that which our policy would recommend for optimal results, and we'll follow this policy with a probability of $1 - \\epsilon$. However, we'll resort to random choice with a probability of $\\epsilon$, allowing us to sometimes make "stupid" choices for the sake of exploration. Often, these random explorations will discover a more valuable state than our policy had knowledge of, and we can adjust our policy accordingly. However, while this $\\epsilon$-greedy policy is very valuable for encouraging the agent to explore an environment, ultimately we'd like to converge on a greedy policy that always makes the optimal decision (ie. without random exploration). This trade-off is known generally as **exploration versus exploitation**, which refers to the balance between encouraging your agent to explore the environment and encouraging your agent to make smart decisions within the environment. One idea to address this tradeoff is **Greedy in the Limit with Infinite Exploration (GLIE)**. GLIE mandates that 1) all state-action pairs are explored infinitely many times and 2) the policy eventually converges on a greedy policy. We could apply the principles of GLIE to an $\\epsilon$-greedy policy by simply decaying $\\epsilon$ over time. ### Summary In this post, we introduced $Q\\left( {s,a} \\right)$ as a way for determining the optimal policy without knowledge of the underlying system dynamics. This allows us to learn from the environment, rather than just plan. We then discussed two main approaches for updating $Q\\left( {s,a} \\right)$ as our agent explores an environment: Monte Carlo - Learn from complete episodes. - High variance, zero bias. - Full lookahead. Temporal Difference - Learn from incomplete/ongoing episodes. - High bias, low variance. - Short-term lookahead combined with a guess. Lastly, we introduced a technique which presents a hybrid of the two methods, which leads to more efficient learning. In my next post, I'll discuss specific implementations of these general methods. ### Further reading - [RL Course by David Silver - Lecture 4: Model-Free Prediction](https://www.youtube.com/watch?v=PnHCvfgC%5FZA&ref=jeremyjordan.me) - [RL Course by David Silver - Lecture 5: Model Free Control](https://www.youtube.com/watch?v=0g4j2k%5FGgc4&ref=jeremyjordan.me) - [Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto](http://ufal.mff.cuni.cz/~straka/courses/npfl114/2016/sutton-bookdraft2016sep.pdf?ref=jeremyjordan.me) Note: In his lectures, David Silver assigns reward as the agent leaves a given state. In my blog posts, I assign reward as the agent enters a state, as it is what makes most sense to me. In one of his lectures he makes this distinction and states that either way is valid, it's just a matter of notation. Thus, if you're following along with David Silver's notes, you might find small deviations between his equations and the ones on this blog. If you're confused, feel free to ask questions in the comments section. ### Overview of reinforcement learning. URL: https://www.jeremyjordan.me/overview-of-reinforcement-learning/ Last updated: 2017-10-04T12:58:08.000Z Reinforcement learning is a method of learning where we teach the computer to perform some task by providing it with feedback as it performs actions. This is different from supervised learning in that we don't explicitly provide correct and incorrect examples of how the task should be completed, we simply tell the computer when it is doing a good job along the way. Reinforcement learning is also distinct from unsupervised learning because we are providing the computer with some level of feedback, even if we aren't providing explicit examples. This method of learning involves an **agent** (ie. computer/robot) exploring an **environment** (a physical or virtual world) to complete some **task** that is taught by incentivizing the agent with a **reward**. The agent learns a **policy** which dictates the best action to take given that agent's current state. In the next few posts on this subject I'll discuss common techniques used for reinforcement learning. This post will mainly serve to introduce some key concepts that will come up as I discuss reinforcement learning techniques. ### Planning vs learning **Planning** refers to finding the optimal set of actions to take in an environment, in order to complete some task (or more truly, maximize reward), when the conditions and states of the environment are known. For this case, you can directly calculate the optimal policy before the agent even moves. You can consider this as developing a policy with a "God's eye" view of the system. **Learning** refers to finding the optimal set of actions within an environment to complete a task when the agent has no previous knowledge of the environment. Thus, the agent must explore the environment and learn the best actions to take as it goes. For the case of learning, the agent's policy improves over time as it explores the environment. This is more akin to a person interacting with the world, they're limited to what they experience through their own observations and they are born knowing nothing. ### Exploration vs exploitation For the task of learning, you'd like to **explore** the environment so that you can learn the best actions to take within the environment. While you explore, you take random actions just to see where you end up. However, as you're learning, you develop an idea for what actions are better to take than others - it would make sense that you'd like to **exploit** what you've learned to choose good actions over bad actions. Ultimately, you must find a balance between choosing to explore the environment more and using what you've learned so far. To view this tradeoff in another light, suppose you visit a new Italian restaurant in town and the first meal you order was incredibly delicious. The next time you go back, do you choose to try something else at the risk of it not being as good, or do you choose to stick with the dish you already know is delicious? In essence, this is the tradeoff between exploration and exploitation. ### What's coming next In the following [posts](https://www.jeremyjordan.me/data-science/#reinforcementlearning), I'll dive into the mathematical models behind reinforcement learning. ### Interesting articles - [The hippocampus as a 'predictive map'](https://deepmind.com/blog/hippocampus-predictive-map/?ref=jeremyjordan.me) ### SQL for data analysis. URL: https://www.jeremyjordan.me/sql/ Last updated: 2018-02-04T03:38:29.000Z As a data scientist, you deal with a lot of data. For small datasets, maybe you just store this information in a CSV file and load it into [Pandas](https://pandas.pydata.org/?ref=jeremyjordan.me). However, this isn't really a scalable solution and won't do too well if you're constantly updating and inserting new data. You need a database; somewhere to hold your information that allows for easy inserting, updating, and reading. There are two main approaches to storing data in a database: **relational databases** store data as a row in tables with a pre-defined structure, while **non-relational databases** store data independently as a document. This post will focus on accessing data from relational databases. Relational database tables are very similar to Pandas dataframes, each row corresponds with a certain object and columns are used to store different features of that object. For example, a website might have a table in their database that stores its users' information (username, first name, last name, etc.). | id | first\_name | last\_name | username | password | | -- | ----------- | ---------- | -------------- | --------------- | | 1 | Jeremy | Jordan | password | password | | 2 | John | Snow | winteriscoming | braceyourselves | | 3 | Anderson | Paak | yeslawd | breezylovejoy | *Note: Websites should *never* store your password in plain text.* When we want to get information from a relational database, we do so by querying (asking) for it. Specifically, we use a **structured query language (SQL)** to speak with the database and access data. A **schema** defines how information is stored within the table, essentially describing what type of information is stored in each column. SQL has the ability to search, filter, and combine data that might be stored across multiple tables. In this post, I'll discuss the common SQL commands used for data analysis. ### Quick links for reference ###### Basic SQL Commands - [SELECT/FROM](#selectfrom): specifies the columns and table of interest - [LIMIT](#limit): caps the query results - [ORDER BY](#orderby): specify the order results are returned in - [WHERE](#where): filters the data using a comparison or logical operator - [LIKE](#like): used to match on similar values instead of strictly identical values - [BETWEEN](#between): only select rows in the specified range - [IN](#in): specify a list of values to include - [AND](#and): select rows which satisfy two specified conditions - [OR](#or): select rows which satisfy one of the specified conditions - [NOT](#not): select rows which do not satisfy the specified condition - [IS NULL](#isnull): select rows with null values in specified column ###### Intermediate SQL Commands - Aggregation functions (column based) - [COUNT(column)](#count): counts number of rows in a specified column - [SUM(column)](#sum): adds all the values in a specified column - [MIN/MAX(column)](#minmax): returns lowest/highest value in a specified column - [AVG(column)](#avg): calculates the average value in a specified column - [GROUP BY](#groupby): segments records into groups - [HAVING](#having): used to filter groups using a comparison or logical operator - [DISTINCT](#distinct): returns only unique values, no duplicates - [CASE (WHEN/THEN/ELSE/END)](#case): if/then logic for SQL - Data joins - [JOIN (ON)](#join): combining data from multiple tables - [INNER JOIN ](#innerjoin): exclude all unmatched records - [LEFT JOIN](#leftjoin): exclude unmatched records found in the joined table - [RIGHT JOIN](#rightjoin): exclude unmatched records found in the original table - [FULL OUTER JOIN](#fulloutjoin): return all records, matched and unmatched from each table - [UNION (ALL)](#union): combine results from multiple queries - [SQL commands order of operation](#orderoperations) - [SQL data types (external link)](https://www.w3schools.com/sql/sql%5Fdatatypes%5Fgeneral.asp?ref=jeremyjordan.me) --- ### Basic SQL Commands There are two commands which are absolutely required for every SQL query, what data you want (`SELECT`) and where that data is stored (`FROM`). With the `SELECT` command, you can specify the **columns** of a table that you'd like to retrieve while `FROM` specifies what **table** you're interested in retrieving data from. The "\*" symbol is shorthand for "all", so if you wanted to grab all of the users' fields from the example table above you'd perform a query like `SELECT * FROM users`. The `AS` keyword will create an alias for a column in the data retrieved. ``` SELECT first_name AS "First Name" FROM users ``` You can also combine values from different columns (within the same row). ``` SELECT unread_count + read_count AS total_count FROM emails ``` To limit the size of your query results, you can use the command `LIMIT`, followed by the sample size, at the end of your query. ``` SELECT * FROM users LIMIT 100 ``` You can use the command `ORDER BY` to specify the column to sort the returned results. The default order is ascending, but you can add the keyword `DESC` after specifying the column to return a descending order of results. You can also specify multiple columns to order by. ``` SELECT * FROM users ORDER BY year, month DESC ``` If you'd like to filter your results, you can specify which results you'd like using the `WHERE` command, followed by the column to filter on and a comparison or logical expression. For example, you could query the database for records where the user's age is greater than 21. ``` SELECT * FROM users WHERE age > 21 ``` ###### Logical operators If you want to filter a collection of similar records, you can specify the similarity using `LIKE` (or `ILIKE` to ignore case in strings for some flavors of SQL). The `%` symbol can be used as a wildcard where it could represent any character or set of characters. To limit the wildcard to only one character, you can use the `_` symbol. ``` SELECT * FROM users WHERE name LIKE 'Bill%' -- returns for Bill, Billy, Billie, etc. ``` To access data from a specific time period, a certain age group, or any other range you might be interested in, you can use the `BETWEEN` command to specify a range of results to filter by. Both upper and lower bounds are included in the search results. ``` SELECT * FROM users WHERE age BETWEEN 20 AND 30 ``` For even more specific ranges, you can specify every value to include using the `IN` command. ``` SELECT * FROM users WHERE age IN (20, 22, 24, 26) ``` `AND` and `OR` are used to combine multiple expressions when specifying a filter on the data, where `AND` only returns values which meet all specifications while `OR` will return values where at least one of the specifications is met. `NOT` can be used to negate any expression. `IS NULL` will check to see whether or not the data in a specified column is null. Typically it is used in combination with the `NOT` keyword to only access records which do not contain null values. ``` SELECT * FROM users WHERE age IS NOT NULL ``` --- ### Intermediate SQL Commands ###### Aggregation functions `COUNT(column)` is used to return the number of rows containing non-null values in a given column. You can return the total number of rows by either counting on the index column or using `*`. ``` -- returns total number of users SELECT COUNT(*) FROM users -- returns number of users where last_name is not blank SELECT COUNT(last_name) FROM users ``` `SUM(column)` provides a summation of all numeric values in a specified column. `MIN(column)` provides the minimum value within a column. For columns containing text, 'A' is considered the lowest value. `MAX(column)` provides the maximum value within a column. For columns containing text, 'Z' is considered the highest value. `AVG(column)` calculates the arithmetic mean for numeric columns, ignoring null values. --- So far, we've only used the aggregation functions on the entire dataset. However, it is possible to segment the data into groups and aggregate each group independently. Suppose you have a table that has the daily stock price of a company for the past few years, and you'd like to look at the monthly stock price average over this time period. You could accomplish this by grouping the records by year and month (`GROUP BY`), and then aggregate the individual records in each group to find the average (`AVG`). ``` SELECT year, month, AVG(daily_open) AS monthly_avg FROM google_stock_price GROUP BY year, month ORDER BY month, year ``` Whereas the previously discussed `WHERE` command can be used to filter **individual records**, `HAVING` is used to filter **groups**. For example, you could query the stock prices database to report the monthly average only for the months which had high price fluctuations. The same set of comparison and logical operators can be used for filtering groups. Aggregator functions (in this case, `MAX` and `MIN`) operate individually for each group. ``` SELECT year, month, AVG(daily_open) AS monthly_avg FROM google_stock_price GROUP BY year, month HAVING MAX(daily_open) - MIN(daily_open) > 50 ORDER BY month, year ``` If your table has repeated values across records, you can use `DISTINCT` to return a set of unique values, or unique combinations of values in the case of using `DISTINCT` for multiple columns. You can also place `DISTINCT` inside an aggregator function to perform aggregations only on the unique set of values. ``` -- example using DISTINCT for multiple columns SELECT DISTINCT year, month FROM google_stock_price -- example using DISTINCT within an aggregator function (suppose there are multiple stock price records per day) SELECT year, month, COUNT(DISTINCT day) AS days_in_month FROM google_stock_price ``` SQL is capable of performing "if/then" logic during queries using `CASE`. The general structure is to provide instructions on what to do if the case is true (`CASE` \--> `THEN`), what to do if the case is not true (`ELSE` \--> `END` or just `END`). For multiple "if" statements, you can specify what to do for each case (`WHEN` \--> `THEN`). Case statements may be used in both `SELECT` and `GROUP BY` statements. ``` SELECT first_name, last_name, CASE WHEN age < 18 THEN 'child' WHEN age >= 18 THEN 'adult' ELSE NULL END AS legal_age_classification FROM users -- count the number of records in each age bracket SELECT CASE WHEN age >= 7 AND age < 22 THEN 'gen_z' WHEN age >= 22 AND age < 38 THEN 'gen_y' WHEN age >= 38 AND age < 53 THEN 'gen_x' WHEN age >= 53 THEN 'baby_boomer' ELSE 'unassigned' END AS age_group, COUNT(1) AS count FROM users GROUP BY age_group -- reorient the age bracket count horizontally (like a pivot table) SELECT COUNT(CASE WHEN age >= 7 AND age < 22 THEN 1 ELSE NULL END) AS gen_z_count, COUNT(CASE WHEN age >= 22 AND age < 38 THEN 1 ELSE NULL END) AS gen_y_count, COUNT(CASE WHEN age >= 38 AND age < 53 THEN 1 ELSE NULL END) AS gen_x_count, COUNT(CASE WHEN age >= 53 THEN 1 ELSE NULL END) AS baby_boomer_count FROM users ``` ###### Combining data from different sources Data **joins** are used to combine related information stored across multiple tables. A foreign key is used to describe how a record in one table relates to information in another table. There are many different ways to combine information across two tables, as discussed below. These approaches define how we'll treat records which don't have a connection relating it to records in the other table. To understand how the following joins discussed work, check out this [interactive demo](http://joins.spathon.com/?ref=jeremyjordan.me). Note: If both tables have the same column name, you need to create alias for the columns in the results. This is accomplished with `SELECT column AS alias`. In order to combine data from multiple tables, there must be a common field relating records in one table to records in the other. If this is the case, we can combine the records by using `JOIN` to specify the table containing related information and `ON` to provide the relationship between the two tables. This is basically saying, when a record in the original table matches a record in the join table, combine the two records. You can also provide additional logic in the `ON` statement to further control what records are joined. Lastly, you can also specify multiple foreign keys to be matched on; if there are two columns that are common across both tables, joining on both of these keys can sometimes increase accuracy of your query. ``` SELECT * FROM users JOIN emails ON users.id = emails.user_id ``` Pro tip: You can also create an alias for your tables by including the alias after the table name. ``` SELECT * FROM really_long_name_for_table_of_users users -- here, users is an alias JOIN emails ON users.id = emails.user_id AND users.join_date > '20150214' -- example of additional logic used in the ON statement ``` By default, SQL performs an **inner join**. For this approach, only results which are related across the two tables are returned (excluding all other records). You can also explicitly define an inner join with `INNER JOIN` instead of `JOIN`, although the two commands are synonymous. All joins returns rows that are matched via the join criteria, however, an inner join excludes all unmatched rows. It is possible, however, to include records even if they weren't matched with any records in the other table - this is known as an **outer join**. The unmatched records will contain null values for values which would have normally been retrieved from the joined table. A **left join** (`LEFT JOIN`) returns all unmatched records in the original table (specified via `FROM`), in addition to the matched records. The unmatched records will contain null values for the fields that were retrieved from the second table (specified via `JOIN`). A **right join** (`RIGHT JOIN`) does the opposite, returning all unmatched records in the second table (specified via `JOIN`), in addition to the matched records. The unmatched records will contain null values for the fields that could not be located in the original table (specified via `FROM`). A **full outer join** (`FULL JOIN`) returns all records, matched and unmatched, from both tables. --- You can combine multiple queries, stacking the results of one on top of the other, by creating a `UNION`. By default, `UNION` will not repeat any identical rows in the results. If you want the full addition of all results across queries, you must specify that with `UNION ALL`. In order for two queries to be compatible for a union, they must have the same number of columns and each column should hold the same data type across both queries. ``` SELECT * FROM table1 UNION ALL SELECT * FROM table2 ``` ### Order of operations When building an SQL query, it's important to keep in mind the order that commands are executed. People often say that lexical order and logical order of SQL commands differ; what they mean is that the way we would form a query in plain English (lexical) doesn't always match the order of information (logical) a computer needs to perform the query. ###### Execution 1. FROM 2. ON 3. WHERE 4. GROUP BY 5. Aggregation functions (COUNT, SUM, MIN/MAX, AVG) 6. HAVING 7. SELECT 8. DISTINCT 9. UNION, INTERSECT, EXCEPT 10. ORDER BY 11. LIMIT, TOP ## Practice [https://pgexercises.com/questions/basic/](https://pgexercises.com/questions/basic/?ref=jeremyjordan.me) ### Convolutional neural networks. URL: https://www.jeremyjordan.me/convolutional-neural-networks/ Last updated: 2019-04-02T14:34:21.000Z In my [introductory post](https://www.jeremyjordan.me/neural-networks-representation/) on neural networks, I introduced the concept of a neural network that looked something like this. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-26-at-1.44.58-PM.png) As it turns out, there are [many different neural network architectures](http://www.asimovinstitute.org/neural-network-zoo/?ref=jeremyjordan.me), each with its own set of benefits. The architecture is defined by the type of layers we implement and how layers are connected together. The neural network above is known as a **feed-forward network** (also known as a multilayer perceptron) where we simply have a series of fully-connected layers. Today, I'll be talking about **convolutional neural networks** which are used heavily in image recognition applications of machine learning. Convolutional neural networks provide an advantage over feed-forward networks because they are capable of considering locality of features. Consider the case where we'd like to build an neural network that could recognize handwritten digits. For example, given the following 4 by 4 pixel image as input, our neural network should classify it as a "1". ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-26-at-4.25.52-PM.png) > Images are simply a matrix of values corresponding with the intensity of light (white for highest intensity, black for lowest intensity) at each pixel value. Grayscale images have a single value for each pixel while color images are typically represented by light intensity values for red, green, and blue at each pixel value. Thus, a 400 by 400 pixel image has the dimensions $\\left\[ {400 \\times 400 \\times 1} \\right\]$ for a grayscale image and $\\left\[ {400 \\times 400 \\times 3} \\right\]$ for a color image. When using images for machine learning models, we'll typically rescale the light intensity values to be bound between 0 and 1. ![8_digits](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/06/8_digits.gif) [Image credit](https://medium.com/@ageitgey/machine-learning-is-fun-part-3-deep-learning-and-convolutional-neural-networks-f40359318721?ref=jeremyjordan.me) ### Revisiting feed-forward networks First, let's examine what this would look like using a feed-forward network and identify any weaknesses with this approach. Foremost, we can't directly feed this image into the neural network. A feed-forward network takes a vector of inputs, so we must flatten our 2D array of pixel values into a vector. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-26-at-4.26.01-PM.png) Now, we can treat each individual pixel value as a feature and feed it into the neural network. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-27-at-12.07.11-AM.png) Unfortunately, we lose a great deal of information about the picture when we convert the 2D array of pixel values into a vector; specifically, we lose the spatial relationships within the data. Just ask yourself, could you identify the following numbers in flattened vector form without rearranging the pixels back into a 2D array in your head? ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-26-at-4.30.29-PM.png) ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-26-at-4.30.37-PM.png) Don't kid yourself, you can't. The spatial arrangement of features (pixels) is important because we see in a relativistic perspective. This is where convolutional neural networks shine. ### Convolution layers A convolution layer defines a window by which we examine a subset of the image, and subsequently scans the entire image looking through this window. As you'll see below, we can parameterize the window to *look for specific features* (e.g. edges) within an image. This window is also sometimes called a filter, since it produces an output image which focuses *solely on the regions of the image which exhibited the feature* it was searching for. The output of a convolution is referred to as a **feature map**. *Note: Windows, filters, and kernels all refer to the same thing with respect to convolutional neural networks; don't get confused if you see one term being used instead of another.* For example, the following window searches for vertical lines in the image. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-26-at-5.15.35-PM.png) We overlay this filter onto the image and linearly combine (and then [activate](https://www.jeremyjordan.me/neural-networks-activation-functions/)) the pixel and filter values. The output, shown on the right, identifies regions of the image in which a vertical line was present. We could do a similar process using a filter designed to find horizontal edges to properly characterize all of the features of a "4". ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/08/Screen-Shot-2017-08-23-at-5.05.59-PM.png) We'll scan each filter across the image calculating the linear combination (and subsequent activation) at each step. In the figure below, an input image is represented in blue and the convolved image is represented in green. You can imagine each pixel in the convolved layer as a neuron which takes all of the pixel values *currently in the window* as inputs, linearly combined with the corresponding weights in our filter. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/no_padding_no_strides.gif) [*Image credit*](https://github.com/vdumoulin/conv%5Farithmetic?ref=jeremyjordan.me) You can also pad the edges of your images with 0-valued pixels as to fully scan the original image and preserve its complete dimensions. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/same_padding_no_strides.gif) [*Image credit*](https://github.com/vdumoulin/conv%5Farithmetic?ref=jeremyjordan.me) As an example, I took an image of myself sailing and applied four filters, each of which look for a certain type of edge in the photo. You can see the resulting output below. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/download--7-.png) ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/download--8-.png) ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/download--9-.png) As shown in the example above, a typical convolutional layer will apply multiple filters, each of which output a *feature mapping* of the input signifying the (spatial) locations where a feature is present. Notice how some feature mappings are more useful than others; in the below example, the "right diagonal" feature mapping, which searches for right diagonals in the image, is essentially a dark image which signifies that the feature is not present. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-26-at-6.13.41-PM.png) In practice, we don't explicitly define the filters that our convolutional layer will use; we instead parameterize the filters and let the network learn the best filters to use during training. We do, however, define how many filters we'll use at each layer. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-26-at-6.32.58-PM.png) We can stack layers of convolutions together (ie. perform convolutions on convolutions) to learn more intricate patterns within the features mapped in the previous layer. This allows our neural network to identify general patterns in early layers, then focus on the patterns within patterns in later layers. Let's revisit the feed-forward architecture one more time and compare it with convolutional layers now that we have a better understanding of what's going on inside them. A feed-forward network connects every pixel with each node in the following layer, ignoring any spatial information present in the image. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-27-at-12.07.33-AM.png) By contrast, a convolutional architecture looks at local regions of the image. In this case, a 2 by 2 filter with a stride of 2 (more on strides below) is scanned across the image to output 4 nodes, each containing localized information about the image. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-27-at-12.09.53-AM.png) The four nodes can be combined to form the "pixels" of a feature map, where the feature extracted is dependent on the parameters of our filter. Here, spatial information can be preserved. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-27-at-12.08.01-AM.png) We can scan the image using multiple filters to generate multiple feature mappings of the image. Each feature mapping will reveal the parts of the image which express the given feature defined by the parameters of our filter. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-27-at-12.08.10-AM.png) In general, a convolution layer will transform an input into a stack of feature mappings of that input. Depending on how you set your padding, you may or may not reduce the size of your input. The depth of the feature map stack depends on how many filters you define for a layer. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-27-at-9.43.59-AM.png) Each filter will have a defined width and height, but the filter depth should match that of the input. For earlier examples where we looked at grayscale images, our filter had a `depth=1`. However, if we wanted to convolve a color image, for example, we would need a filter with `depth=3`, containing a filter for each of the three RGB color channels. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/04/Screen-Shot-2018-04-15-at-11.20.46-PM.png) *Note: The convolution operation may be more accurately describing as calculating the cross correlation between the image input and a given filter. In pure mathematical terms, a convolution involves flipping the kernel matrix, but since we're simply learning parameter values, this operation doesn't add any value.* ### Deep convolutional networks I mentioned earlier that we can stack multiple layers of convolutions (each followed by a nonlinear activation) to learn more intricate patterns. I'll now provide a more concrete example of this. Let's start with a question... how would you describe a *square* to someone who has never seen the shape before? Here's one possible answer: > A square is comprised of two sets of parallel lines, all of equal length, which intersect orthogonal to one another. In this response, we've described the abstract concept of a **square** using lower-level concepts such as lines and angles. We could use these same *lower-level concepts* to describe a plethora of shapes. When we [visualize convolutional networks](https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/5.4-visualizing-what-convnets-learn.ipynb?ref=jeremyjordan.me), we find that the same phenomenon occurs. The network learns to use earlier layers to extract low-level features and then combine these features into more rich representations of the data. This means that later layers of the network are more likely to contain specialized feature maps. Here, I've shown how we could combine successive convolutional operations such that we can detect the presence of a square in the original input. Although I've hard-coded the filter values for this example, the filter values in a convolutional layer are *learned*. ![Screen-Shot-2018-05-12-at-12.57.58-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-12-at-12.57.58-PM.png) Layers in a convolutional network follow the same general principle, using parameterized filters to transform the input into useful representations. ![Screen-Shot-2018-05-12-at-12.54.40-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/05/Screen-Shot-2018-05-12-at-12.54.40-PM.png) *Reminder: the number of channels in your filter must match the number of channels in your input.* As you progress into deeper layers of a network, it's very common to see an increase in the *channel depth* (number of feature maps); as feature maps become more specialized, representing higher-level abstract concepts, we simply need more feature maps to represent the input. Consider the fact that a small set of lines can be combined into a large number of shapes, and these shapes can be combined into an even larger number of objects. If our goal is to build a model with general representational power, we'll likely need to grow our feature maps accordingly. ### Pooling layers A **pooling layer** can be used to compress spatial information of our feature mappings. We'll still scan across the image using a window, but this time our goal is to compress information rather than extract certain features. Similar to convolutional layers, we'll define the window size, and stride. Padding is not commonly used for pooling layers. ###### Max pooling In max pooling, we simply return the maximum value (highlighted in yellow) present inside the window for each scanning location. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-27-at-11.43.19-AM.png) You can similarly define layers for **average pooling** or **minimum pooling** when necessary, although max pooling is the most commonly used pooling layer. ###### Global pooling Global pooling is a more extreme case, where we define our window to match the dimensions of the input, effectively compressing the size of each feature mapping to a single value. This is useful for building convolutional neural networks in which need to be able to accept a variety of input sizes, as it compresses your feature representation from an arbitrary $w \\times h \\times c$ into a fixed $1 \\times 1 \\times c$ feature map. ### Strided convolutions Strided convolutions are an alternative method for downsampling the spatial resolution of our feature mappings which allows for a degree of *learned* downsampling rather than explicitly defining how to summarize a window (such as the case for max/average/min pooling). The *stride length* of a convolution defines how many steps we take when sliding our window (filter) across an image. A standard convolution has a stride length of one, moving the filter one unit as we scan across the input. ![same_padding_no_strides](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/04/same_padding_no_strides.gif) [*Image credit*](https://github.com/vdumoulin/conv%5Farithmetic?ref=jeremyjordan.me) Alternatively, if we were to define a stride length of two, we would move the filter by two units for each calculation. ![padding_strides](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/04/padding_strides.gif) [*Image credit*](https://github.com/vdumoulin/conv%5Farithmetic?ref=jeremyjordan.me) Notice how this has the effect of reducing the dimensions of the output feature map, relative to the input. You can calculate the dimensions of a feature map according to the following formula $$ \\left( {\\frac{{{n\_{width}} - {f\_{width}} + 2{p\_{width}}}}{{{s\_{width}}}} + 1,\\frac{{{n\_{height}} - {f\_{height}} + 2{p\_{height}}}}{{{s\_{height}}}} + 1} \\right) $$ where $n$ is the input dimension, $f$ is the filter size, $p$ is the padding, $s$ is the stride length. ### Putting it all together Convolutional neural networks (also called ConvNets) are typically comprised of convolutional layers with some method of periodic downsampling (either through pooling or strided convolutions). The convolutional layers create feature mappings which serve to explain the input in different ways, while the pooling layers compress the spatial dimensions, reducing the number of parameters needed to extract features in following layers. For image classification, we eventually reduce this representation of our original input into a deep stack of 1 by 1 (single value) representations of the original input. At this time, we can now feed this information into a few fully-connected layers to determine the likelihood of each output class being present in the image. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/cnn.png) [*Image credit*](https://www.clarifai.com/technology?ref=jeremyjordan.me) ### Further reading Lectures/notes - [Stanford CS231: Introduction to Convolutional Neural Networks for Visual Recognition](https://www.youtube.com/watch?v=vT1JzLTH4G4&list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv&t=0s&index=2&ref=jeremyjordan.me) - [Stanford CS231 Notes: Convolutional Neural Networks](http://cs231n.github.io/convolutional-networks/?ref=jeremyjordan.me) - [Stanford CS231 Notes: Visualizing what ConvNets learn](http://cs231n.github.io/understanding-cnn/?ref=jeremyjordan.me) - [Deep Learning for Computer Vision (Andrej Karpathy, OpenAI)](https://www.youtube.com/watch?v=u6aEYuemt0M&ref=jeremyjordan.me) - [Stanford Stats 385: Theoretical analysis of convnets](https://www.youtube.com/watch?v=uIVPo5eyhE0&ref=jeremyjordan.me) Blogs/talks - [François Chollet: Visualizing what convnets learn](https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/5.4-visualizing-what-convnets-learn.ipynb?ref=jeremyjordan.me) - [A visual and intuitive understanding of deep learning](https://www.youtube.com/watch?v=Oqm9vsf%5FhvU&ref=jeremyjordan.me) - [Backpropagation for convolutional neural networks](http://www.jefkine.com/general/2016/09/05/backpropagation-in-convolutional-neural-networks/?ref=jeremyjordan.me) - [Applied Deep Learning - Part 4: Convolutional Neural Networks](https://towardsdatascience.com/applied-deep-learning-part-4-convolutional-neural-networks-584bc134c1e2?ref=jeremyjordan.me) - [Fifteen questions and answers about the use of convolutional neural networks as a model of the visual system.](https://neurdiness.wordpress.com/2018/05/17/deep-convolutional-neural-networks-as-models-of-the-visual-system-qa/?ref=jeremyjordan.me) - [Intuitively Understanding Convolutions for Deep Learning](https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1?ref=jeremyjordan.me) Papers - [A guide to convolution arithmetic for deep learning](https://arxiv.org/abs/1603.07285?ref=jeremyjordan.me) - [Visualizing and Understanding Convolutional Networks](https://arxiv.org/abs/1311.2901?ref=jeremyjordan.me) - [Yann LeCun’s original introduction of CNNs](http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf?ref=jeremyjordan.me) Demos - [Browser-based convnet with inspectable layers](https://deeplearnjs.org/demos/imagenet/?ref=jeremyjordan.me) - [Handwritten digits interactive demo and visualization of convolutional neural networks](http://scs.ryerson.ca/~aharley/vis/conv/?ref=jeremyjordan.me) Visualization tools \-[Keras Visualization Toolkit](https://github.com/kitamura-felipe/keras-vis?ref=jeremyjordan.me) \-[Tensorflow Lucid](https://github.com/tensorflow/lucid?ref=jeremyjordan.me) For a more detailed discussion of the implementations of convolutions, see [cuDNN: Efficient Primitives for Deep Learning](https://arxiv.org/pdf/1410.0759.pdf?ref=jeremyjordan.me). ### Deep neural networks: preventing overfitting. URL: https://www.jeremyjordan.me/deep-neural-networks-preventing-overfitting/ Last updated: 2018-07-30T03:26:43.000Z In previous posts, I've [introduced the concept of neural networks](https://www.jeremyjordan.me/neural-networks-representation/) and discussed how we can train [neural networks](https://www.jeremyjordan.me/neural-networks-training/). For these posts, we examined neural networks that looked like this. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-26-at-11.30.04-AM.png) However, many of the modern advancements in neural networks have been a result of stacking many hidden layers. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-26-at-11.30.12-AM.png) This deep stacking allows us to learn more complex relationships in the data. However, **because we're increasing the complexity of the model, we're also more prone to potentially overfitting our data**. In this post, I'll discuss common techniques to leverage the power of deep neural networks without falling prey to overfitting. ### Early stopping Arguably, the simplest technique to avoid overfitting is to watch a [validation curve](https://www.jeremyjordan.me/evaluating-a-machine-learning-model/) while training and stop updating the weights once your validation error starts increasing. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-25-at-3.55.30-PM.png) > **Reminder:** During each iteration of training, we perform forward propagation to compute the outputs and backward propagation to compute the errors; one complete iteration over all of the training data is known as an **epoch**. It is common to report evaluation metrics after each epoch so that we can watch the evolution of our neural network as it trains. ### Parameter Regularization Think back to the overfitting example for linear regression, and recall that this model performs poorly because it suffers from high variance. That is, for small changes in the input value, we see wild fluctuations in the corresponding output. This same problem can occur in neural networks where inputs are linearly combined with parameters to pass information between layers; when these parameters are excessively large, they amplify small changes in the input. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/download--3-.png) One way to combat this is by penalize parameters in our cost function to prevent parameters from becoming excessively large during training. We can do this via a number of expressions, the two most common are L1 and L2 norms. **L1 regularization** \[\\lambda \\sum\\limits\_{i = 1}^n {\\left| {{\\theta\_i}} \\right|} \] This expression is commonly used for feature selection as it tends to produce sparse parameter vectors where only the important features take on non-zero values. **L2 regularization** \[\\lambda \\sum\\limits\_{i = 1}^n {\\theta\_i^2} \] This expression doesn't tend to push less important weights to zero and typically produces better results when training a model. For example, if we select binary cross-entropy with L1 regularization as our loss function, the total expression would be \[J\\left( \\theta \\right) = - \\frac{1}{m}\\sum\\limits \_{i = 1}^m {\\left\[ {{y \_i}\\ln {{\\hat y} \_i} + \\left( {1 - {y \_i}} \\right)\\ln \\left( {1 - {{\\hat y} \_i}} \\right)} \\right\]} + \\lambda \\sum\\limits \_{i = 1}^n {\\left| {{\\theta \_i}} \\right|} \] where the first term in the cost function represents the binary cross-entropy and the second term is our regularization expression. $\\lambda$ controls the degree to which we choose to penalize large parameters. Why do we need to worry about parameters growing excessively large? A neuron's output (activation) is a function of activating a linear combination of inputs and weights to that neuron. \[a = g\\left( {{\\theta \_0} + {\\theta \_1}{x \_1} + {\\theta \_2}{x \_2} + ... + {\\theta \_n}{x \_n}} \\right)\] Suppose the case where we have an output neuron which performs binary classification of whether or not an image contains a cat. The output neuron should produce a value close to 1 for cases where image contains a cat and produce values close to 0 for cases where the image does not contain a cat. We measure the performance of this neural network by comparing the output of the neuron to the true label. The true label will always be either 0 or 1, thus our model is incentivized to produce outputs as close to 0 or as close to 1 as possible. One way to accomplish this is by simply increasing the magnitude of the weights. As you can see below, large parameter values have the effect of pushing input closer to 0 and 1; in effect, this arbitrarily increases the confidence in our predictions. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/download--6-.png) In summary, our model might let parameters grow excessively large to reduce training error but we'd like to prevent this to reduce the variance of our model. We accomplish this by adding a penalty term for large parameter values in our cost function. ### Dropout [Ensemble learning](https://www.jeremyjordan.me/ensemble-learning/) is a very intuitive approach to addressing the problem of overfitting. If the problem of overfitting is due to our model learning too much from the training data, we can fix that by training multiple models, each of which learn from the data in different ways. If each component model learns a relationship from the data that contains the true signal with some addition of noise, a combination of models should maintain the relationship of the signal within the data while averaging out the noise. However, training neural networks is computationally expensive - some neural networks take days to train! Training an ensemble of neural networks? Yeah, that sounds like a terrible idea. But the concept of ensemble learning to address the overfitting problem still sounds like a good idea... this is where the idea of dropout saves the day for neural networks. Using dropout, we can build multiple representations of the relationship present in the data by randomly dropping neurons from the network during training. We use a Bernoulli distribution to decide which neurons should be dropped, defining the probability that any given neuron will be dropped as a parameter. The result is a "thinned" network. *Note: In order to prevent scale problems at test time, the activations of the remaining neurons are upscaled by the dropout factor.* ![Screen-Shot-2017-10-24-at-3.26.46-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/10/Screen-Shot-2017-10-24-at-3.26.46-PM.png) Recomputing dropout during training allows us to build multiple, unique models to learn from the data. You can perform dropout on any layer, visible or hidden. ![Screen-Shot-2017-10-24-at-3.25.06-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/10/Screen-Shot-2017-10-24-at-3.25.06-PM.png) Unlike other ensemble learners (ex. random forests) where you must keep the collection of models for later use, once you have completed training you can combine the learned parameters from the component models to build a single network with all neurons present. > If you ever need a definition of dropout that is both concise and accurate: [pic.twitter.com/D3M7cWbfXY](https://t.co/D3M7cWbfXY?ref=jeremyjordan.me) > > — Smerity (@Smerity) [March 31, 2018](https://twitter.com/Smerity/status/980175898119778304?ref%5Fsrc=twsrc%5Etfw&ref=jeremyjordan.me) Dropout has brought significant advances to modern neural networks and it considered one of the most powerful techniques to avoid overfitting. ### Further reading - [Dropout: A Simple Way to Prevent Neural Networks from Overfitting](https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf?ref=jeremyjordan.me) ### Planning in a stochastic environment. URL: https://www.jeremyjordan.me/markov-decision-process/ Last updated: 2017-10-26T14:45:13.000Z In this post, I'll be discussing how to calculate the best set of actions to complete a task whilst operating in a known environment, otherwise known as **planning**. For this scenario, we have complete knowledge over the system's dynamics including the reward of each state and a probabilistic model for how the agent changes its state. This post will establish the groundwork for eventually discussing reinforcement learning, where our agent doesn't have access to the underlying system dynamics. ### Markov Decision Process The Markov Decision Process is a method for planning in a stochastic environment. Whereas we cannot control or optimize the randomness that occurs, we can optimize our actions within a random environment. Planning for stochastic environments is much more difficult than planning for a deterministic environment; given the randomness present, there's a degree of uncertainty surrounding the results of our actions. Specifically, planning refers to figuring out a set of actions to complete a given task. The Markov Decision Process is useful framework for directly solving for the best set of actions to take in a random environment. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/08/Screen-Shot-2017-08-09-at-10.15.47-AM.png) An **environment** used for the Markov Decision Process is defined by the following components: - An **agent** is the object within the environment who is encouraged to complete a given task. - The agent assumes a **state** within the environment and can traverse the environment by changing its state. - The agent can change its state by performing an **action** from a defined set of possible actions. - A **transfer function** defines how the agent enters its new state after completing an action. This transfer function is a probabilistic distribution of potential states, introducing stochastic behavior into the environment. Further, this transfer function only depends on the *current* state - it does not require the entire history of previous actions. This upholds what is known as the [Markov property](https://en.wikipedia.org/wiki/Markov%5Fproperty?ref=jeremyjordan.me). - A **reward function** serves to encourage the agent to complete the given task. Given this environment, we can develop a **policy** which describes the best action to take for any given state in the environment. As an agent traverses its environment, it will refer to the policy in order to perform the next action. ### A simple example To provide a concrete example of the defined environment above, take a look a the following image. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/08/Screen-Shot-2017-08-07-at-12.03.32-PM.png) Each square in this figure represents a possible state (excluding the black square, let's say this is an infinitely high barrier). An agent starts at the bottom left and can move from state to state with the following set of actions: up, down, left, right. If the agent is at one of the squares on the perimeter of the environment and tries to take a step outside of the environment, it simply runs into a wall and stays in its current state. The top right square is the desired end state, rewarding the agent with a score of +1 and ending the "game". The square right below it also ends the "game", but rewards the agent with a penalty of -1. ###### Introducing stochastic behavior to the environment Now let's define our transfer function. In a deterministic world, if the agent was instructed (by the policy) to move "up" it would do just that. However, suppose your agent is an angsty teenager and only listens to instructions 80% of the time. 10% of the time, the agent moves 90 degrees to the left of the instructed action and 10% of the time the agent moves 90 degrees to the right of the instructed action. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/08/Screen-Shot-2017-08-07-at-12.20.47-PM.png) Here, the blue arrow represents the *instructed action* whereas all of the arrows represent possible outcomes. This introduces a degree of uncertainty, which is useful for more than just modeling angsty teenagers. For example, if you're relying on sensor data to provide information for an [autonomous robot](https://www.youtube.com/watch?v=9QMZQkKuYjo&ref=jeremyjordan.me), you need to take the noise from a sensor into account when using that information. Modeling stochastic behavior allows us to build more dependable (robust) systems by understanding the undependable nature of our information. Now that we've introduced stochastic behavior, note how your optimal set of actions will change. Originally, the shortest sequence from start to finish in this environment was either {up, up, right, right, right} or {right, right, up, up, right}. However, now that we've introduced the transfer function where our agent only performs the expected action 80% of the time, the former route seems like a "safer" route than the latter because it is less likely that we'll end up accidentally entering the red state. ### Defining rewards The reward function largely controls the behavior of our agent. This function maps a reward to every state in the environment; thus, for the above example we'll have a reward for the white states in addition to the predefined reward for the green and red states. If we'd like to encourage the agent to reach the green state in the shortest amount of time possible, we can assign a small penalty (negative reward) for the remaining states. The following image displays the optimal policy (denoted by blue arrows) for an environment where each white state has a reward of -0.04. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/08/Screen-Shot-2017-08-07-at-2.49.16-PM.png) At the risk of introducing redundant terminology in the field, I'll make a distinction between the two types of rewards we've seen thus far. - **Objective state rewards** drive the end behavior of our agent to complete a given task. In our example, this is 1) go to the green square and 2) avoid the red square. - **Latent state rewards** define how the agent operates in the environment on its journey to achieve the objective rewards. This is the reward given for each white state in our example to encourage our agent to reach the end state quickly. We can observe the effect of our reward function on the optimal policy by changing the latent state reward value. If we make each state have a latent reward that is too large of penalty (in order to encourage the agent to end the game), our agent isn't so careful to avoid the red state. The penalty for existing is harsher than that of entering the red state, so our agent becomes suicidal. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/08/Screen-Shot-2017-08-07-at-2.59.28-PM.png) If we make each state have a positive latent reward, the agent has no incentive to ever end the game and aimlessly wanders through the environment collecting the latent reward at each of the white states. In this event, our agent becomes a hedonist and doesn't care about accomplishing the goal. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/08/Screen-Shot-2017-08-09-at-10.44.57-AM.png) Thus, finding the right reward function is critical to properly encourage your agent to complete a task. ### Utility of a state One could intuit that it might be helpful to be able to calculate the usefulness of a sequence of states, such that our "view of the future" isn't limited to our direct next action. For example, rather than blindly walking around in an environment guided by the next immediate reward in your neighboring states, you'd like for it to see what's ahead and walk in the direction of positive rewards and away from negative rewards, even if those states aren't a direct neighbor of your current state. Suppose your agent is in the following environment where the gold star represents some large reward. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/08/Screen-Shot-2017-08-09-at-10.57.15-AM.png) Clearly, you'd like your agent to take actions to move closer to the large reward. However, if we're only choosing our actions by the rewards of the next possible states, the agent is clueless as to where to go. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/08/Screen-Shot-2017-08-09-at-10.57.29-AM.png) We can calculate the usefulness (or utility) of a sequence of states as a sum of rewards accumulated at each state. By considering the utility of a sequence of states when deciding which next action to take, we can introduce a long-term view of the result of our immediate actions. We'd also like our model to favor more immediate rewards in this stochastic environment because later rewards aren't guaranteed (we might not end up at the expected state due to the stochastic nature of our transfer function). Thus our model for the utility of a sequence of states will discount future rewards. More formally, we can define our utility for a given sequence of states as $$U\\left( {{S\_ 0},{S\_ 1},...,{S\_ n}} \\right) = {\\gamma ^0}R\\left( {{S\_ 0}} \\right) + {\\gamma ^1}R\\left( {{S\_ 1}} \\right) + ... + {\\gamma ^n}R\\left( {{S\_ n}} \\right) = \\sum\\limits\_ {t = 0}^n {{\\gamma ^t}R\\left( {{S\_ t}} \\right)} $$ where $\\gamma$ is a number between 0 and 1 that acts as a discount factor. For states further into the future, the reward will be discounted to a larger degree. Now that we've established a measure of value for a given **sequence of states**, let's consider how we can use this to define the value of a **specific state**. Remember, we'd like the value of a state to represent both the immediate reward and a measure of "heading in the right direction" (refer back to the above figures) to accumulate future rewards; we're interested in the sequence of future states so that we can determine a measure of "heading in the right direction". It's important to make a distinction that heading in the right direction only matters if you continue to head in the right direction. If you were to take a step in the right direction and then get completely sidetracked, there would be little value (ie. reward) accumulated in the sequence of states that would follow. Thus, when considering whether a given state is a step in the right direction (by calculating the accumulated future rewards from the sequence of states that follow), we assume that our agent would proceed in a consistent manner from the new state forward in accordance with some specified policy. To summarize what we've established thus far, the **utility** of a state is the reward we expect to receive in the current state, plus the rewards we expect to accumulate proceeding thereafter following a given policy. We frame this definition as an expectation because the environment is stochastic and we can't definitively say what states our agent will visit in the future. $$U\\left( s \\right) = E\\left\[ {\\sum\\limits\_ {t = 0}^\\infty {{\\gamma ^t}R\\left( {{S\_ t}} \\right)|{S\_ 0} = s} } \\right\]$$ Further, the value of the next state encompasses the value of the infinite sequence of states that follow. Thus, we can define the utility of a state solely in terms of its current reward and the expectation of the utility of the following state. Let's see how this works. First, let's expand the summation in our previous definition. Remember, our policy determines how the agent changes state; if no policy is specified we'll assume the agent is following an optimal policy. $$U\\left( s \\right) = E\\left\[ {R\\left( {{S\_ t}} \\right) + {\\gamma}R\\left( {{S\_ {t+1}}} \\right) + {\\gamma ^2}R\\left( {{S\_ {t+2}}} \\right) + ...|{S\_ t} = s} \\right\]$$ Next, we'll group together all of the discounted future reward terms and factor out $\\gamma$. $$U\\left( s \\right) = E\\left\[ {R\\left( {{S\_ t}} \\right) + \\gamma \\left( {R\\left( {{S\_ {t+1}}} \\right) + {\\gamma}R\\left( {{S\_ {t+2}}} \\right) + ...} \\right)|{S\_ t} = s} \\right\]$$ Now, it is evident that we can rewrite the expression inside the parentheses as the utility of the next state. $$U\\left( {{S\_{t + 1}}} \\right) = E\\left\[ {R\\left( {{S\_{t + 1}}} \\right) + \\gamma R\\left( {{S\_{t + 2}}} \\right) + {\\gamma ^2}R\\left( {{S\_{t + 3}}} \\right) + ...} \\right\]$$ $$U\\left( s \\right) = E\\left\[ {R\\left( {{S\_ t}} \\right) + \\gamma \\left( {U\\left( {{S\_ {t + 1}}} \\right)} \\right)|{S\_ t} = s} \\right\]$$ With a little further rearrangement, we can now clearly observe the two components previously mentioned that compose the utility of a state. $$U\\left( s \\right) = E\\left\[ {R\\left( {{S\_ t}} \\right)|{S\_ t} = s} \\right\] + \\gamma E\\left\[ {U\\left( {{S\_ {t + 1}}} \\right)} \\right\]$$ In summary, we have defined a function for calculating utility where we feed in a state as input and the utility function returns the value for entering the given state. ### Defining an optimal policy The policy represents the instructions given to an agent, based on the information available at the agent's current state. We can define the **optimal policy** as the policy $\\pi$ which maximizes the expected utility of the sequence of states that result from following the policy. Note that this is an expected utility because there is no guarantee that our agent's actions will bring it to the intended state. $${\\pi^\* } = \\arg \\mathop {\\max }\\limits \_\\pi E\\left\[ {\\sum\\limits \_{t = 0}^n {{\\gamma ^t}R\\left( {{S \_t}} \\right)|\\pi } } \\right\]$$ As we discussed in the previous section, the **utility** of a state represents the expected value of the current state's reward plus the infinite summation of discounted future rewards accumulated by following a given policy. Here, we're asserting that our agent will follow the optimal policy. $$U\\left( s \\right) = E\\left\[ {\\sum\\limits \_{t = 0}^\\infty {{\\gamma ^t}R\\left( {{S \_t}} \\right)|\\pi ^\* ,{S \_0} = s} } \\right\]$$ This utility can be viewed as a long-term benefit of following a sequence of actions, whereas the reward function defines the immediate benefit of existing in a given state. By focusing on utility, we can follow a policy which might cause short term negative results but result in more value in the long term. We can consider the optimal policy for a given state as the action that yields the greatest sum of weighted utilities for all of the possible outcomes (where the weighting term is the likelihood of entering state $s'$ from state $s$, as defined by the transfer function). $$\\pi {\\left( s \\right)^\*} = \\arg \\mathop {\\max }\\limits \_a \\sum\\limits \_{s'} {T\\left( {s,a,s'} \\right)} U\\left( {s'} \\right)$$ ### The Bellman equation Let's revisit the final expression we arrived at for defining the utility function. $$U\\left( s \\right) = E\\left\[ {R\\left( {{S\_ t}} \\right)|{S\_ t} = s} \\right\] + \\gamma E\\left\[ {U\\left( {{S\_ {t + 1}}} \\right)} \\right\]$$ First, we can simplify the first term since the reward for the current state is deterministic. $$ U\\left( s \\right) = R\\left( {{S\_ t}} \\right) + \\gamma E\\left\[ {U\\left( {{S\_ {t + 1}}} \\right)} \\right\] $$ Next, let's consider the expectation for the utility of the next state. Recall that we'll choose our action to proceed forward according to the optimal policy. From our current state, there exist a range of possible next states as defined by our transfer function. The expected value from this range of possibilities may be written as a sum, over all possible next states, of the utility of a given next state weighted by the probability of the transfer function placing the agent in that state. $$ E\\left\[ {U\\left( {{S\_ {t + 1}}} \\right)} \\right\] = \\sum\\limits\_ {{S\_ {t + 1}}} {T\\left( {{S\_ t},a,{S\_ {t + 1}}} \\right)} U\\left( {{S\_ {t + 1}}} \\right) $$ In order to follow the optimal policy, we'll take the action which maximizes our expected utility of the next state. $$ \\mathop {\\max }\\limits\_ a \\sum\\limits\_ {s'} {T\\left( {s,a,s'} \\right)} U\\left( {s'} \\right)$$ *(I'll commonly use $s$ to denote the current state and $s'$ to denote the next state. Just remember that $s$ is equivalent to $S\_ t$ and $s'$ is equivalent to $S\_ {t + 1}$.)* Substituting this new expression in to our previous definition of the utility function, we obtain the **Bellman equation**. $$U\\left( s \\right) = R\\left( s \\right) + \\gamma \\mathop {\\max }\\limits \_a \\sum\\limits \_{s'} {T\\left( {s,a,s'} \\right)} U\\left( {s'} \\right)$$ ### Dynamic programming for finding the optimal policy We have an $n$ equations representing the utility of the $n$ states in our environment, and we have $n$ unknowns - the utility of each state. For a set of linear equations, this would be trivial to solve. Unfortunately, the max operator introduces a non-linearity that prevents us from directly solving a system of linear equations. However, we can approximate the true utilities by the following procedure: 1. Start with an initial guess for all utilities. 2. Update the utility for each state by looking at the current state's reward value and using the previously estimated utilities for all other states. 3. Repeat until convergence. In essence, we're recursively solving the following expression. $${{\\hat U}}\\left( s \\right) = R\\left( s \\right) + \\gamma \\sum\\limits \_{s'} {T\\left( {s,a,s'} \\right)} {{\\hat U}}\\left( {s'} \\right)$$ This iterative process has the effect of propagating out the true rewards from each state. Because we're adding the true reward during each iteration, the expression for utility is increasingly "more true" at each next iteration. This process is known as **value iteration**. You can see this process visualized in an interactive demo [here](http://cs.stanford.edu/people/karpathy/reinforcejs/gridworld%5Fdp.html?ref=jeremyjordan.me) (click "Toggle Value Iteration"). To find the optimal policy, we need to find the action of any given state which will yield a transition to the state of highest possible utility. Thus, in order to find the optimal policy, we only need to have the relative value of utilities such that our policy can find the action which brings us to the best possible state. Taking this into consideration, we can focus on directly searching for the optimal policy rather than performing value iteration and then inferring the best policy. This is known as **policy iteration**. 1. Start with an initial guess for the policy. 2. Given the policy, ${\\pi}$, calculate the corresponding utilities of the states. $${U}\\left( s \\right) = R\\left( s \\right) + \\gamma \\sum\\limits \_{s'} {T\\left( {s,{\\pi \_t}\\left( s \\right),s'} \\right)} {U}\\left( {s'} \\right)$$ 3. Recalculate the policy given the updated utilities from the previous step. $${\\pi} = \\arg \\mathop {\\max }\\limits \_a \\sum\\limits \_{s'} {T\\left( {s,a,s'} \\right)} {U}\\left( {s'} \\right)$$ ### Further reading - [UC Berkeley CS188 - Project 3: Reinforcement Learning](http://ai.berkeley.edu/reinforcement.html?ref=jeremyjordan.me) - [RL Course by David Silver - Lecture 2: Markov Decision Process](https://www.youtube.com/watch?v=lfHX2hHRMVQ&ref=jeremyjordan.me) - [RL Course by David Silver - Lecture 3: Planning by Dynamic Programming](https://www.youtube.com/watch?v=Nd1-UUMVfz4&ref=jeremyjordan.me) - [Visual representation of Markov Chains](http://setosa.io/ev/markov-chains/?ref=jeremyjordan.me) - [What is the difference between all types of Markov Chains?](https://math.stackexchange.com/questions/22982/what-is-the-difference-between-all-types-of-markov-chains?ref=jeremyjordan.me) ### Evaluating a machine learning model. URL: https://www.jeremyjordan.me/evaluating-a-machine-learning-model/ Last updated: 2018-08-25T19:53:03.000Z So you've built a machine learning model and trained it on some data... now what? In this post, I'll discuss how to evaluate your model, and practical advice for improving the model based on what we learn evaluating it. I'll answer questions like: - How well is my model doing? Is it a useful model? - Will training my model on more data improve its performance? - Do I need to include more features? ### The train/test/validation split The most important thing you can do to properly evaluate your model is to not train the model on the entire dataset. I repeat: do not train the model on the entire dataset. I talked about this in my post on [preparing data for a machine learning model](https://www.jeremyjordan.me/preparing-data-for-a-machine-learning-model/) and I'll mention it again now because it's that important. A typical train/test split would be to use 70% of the data for training and 30% of the data for testing. As I discussed previously, it's important to use new data when evaluating our model to prevent the likelihood of overfitting to the training set. However, sometimes it's useful to evaluate our model as we're building it to find that best parameters of a model - but we can't use the test set for this evaluation or else we'll end up selecting the parameters that perform best on the test data but maybe not the parameters that generalize best. To evaluate the model while still building and tuning the model, we create a third subset of the data known as the validation set. A typical train/test/validation split would be to use 60% of the data for training, 20% of the data for validation, and 20% of the data for testing. I'll also note that it's very important to shuffle the data before making these splits so that each split has an accurate representation of the dataset. ### Metrics In this session, I'll discuss common metrics used to evaluate models. ###### Classification metrics When performing classification predictions, there's four types of outcomes that could occur. - **True positives** are when you predict an observation belongs to a class and it actually does belong to that class. - **True negatives** are when you predict an observation does not belong to a class and it actually does not belong to that class. - **False positives** occur when you predict an observation belongs to a class when in reality it does not. - **False negatives** occur when you predict an observation does not belong to a class when in fact it does. These four outcomes are often plotted on a confusion matrix. The following confusion matrix is an example for the case of binary classification. You would generate this matrix after making predictions on your test data and then identifying each prediction as one of the four possible outcomes described above. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-21-at-9.57.38-AM.png) You can also extend this confusion matrix to plot multi-class classification predictions. The following is an example confusion matrix for classifying observations from the Iris flower dataset. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/download--1-.png) [*Image credit*](http://scikit-learn.org/stable/auto%5Fexamples/model%5Fselection/plot%5Fconfusion%5Fmatrix.html?ref=jeremyjordan.me) The three main metrics used to evaluate a classification model are accuracy, precision, and recall. **Accuracy** is defined as the percentage of correct predictions for the test data. It can be calculated easily by dividing the number of correct predictions by the number of total predictions. \[{\\rm{accuracy = }}\\frac{{{\\rm{correct \\hspace{2mm} predictions}}}}{{{\\rm{all \\hspace{2mm} predictions}}}}\] **Precision** is defined as the fraction of relevant examples (true positives) among all of the examples which were predicted to belong in a certain class. \[{\\rm{precision}} = \\frac{{{\\rm{true \\hspace{2mm} positives}}}}{{{\\rm{true \\hspace{2mm} positives + false \\hspace{2mm} positives}}}}\] **Recall** is defined as the fraction of examples which were predicted to belong to a class with respect to all of the examples that truly belong in the class. \[{\\rm{recall}} = {\\rm{ }}\\frac{{{\\rm{true \\hspace{2mm} positives}}}}{{{\\rm{true \\hspace{2mm} positives + false \\hspace{2mm} negatives}}}}\] The following graphic does a phenomenal job visualizing the difference between precision and recall. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Precisionrecall.svg.png) [*Image credit*](https://en.wikipedia.org/wiki/Precision%5Fand%5Frecall?ref=jeremyjordan.me) Precision and recall are useful in cases where classes aren't evenly distributed. The common example is for developing a classification algorithm that predicts whether or not someone has a disease. If only a small percentage of the population (let's say 1%) has this disease, we could build a classifier that always predicts that the person does not have the disease, we would have built a model which is 99% accurate and 0% useful. However, if we measured the recall of this useless predictor, it would be clear that there was something wrong with our model. In this example, recall ensures that we're not overlooking the people who have the disease, while precision ensures that we're not misclassifying too many people as having the disease when they don't. Obviously, you wouldn't want a model that incorrectly predicts a person has cancer (the person would end up in a painful and expensive treatment process for a disease they didn't have) but you also don't want to incorrectly predict a person does not have cancer when in fact they do. Thus, it's important to evaluate both the precision and recall of a model. Ultimately, it's nice to have one number to evaluate a machine learning model just as you get a single grade on a test in school. Thus, it makes sense to combine the precision and recall metrics; the common approach for combining these metrics is known as the f-score. \[{F \_\\beta } = \\left( {1 + {\\beta ^2}} \\right)\\frac{{{\\rm{precision}} \\cdot {\\rm{recall}}}}{{\\left( {{\\beta ^2} \\cdot {\\rm{precision}}} \\right) + {\\rm{recall}}}}\] The $\\beta$ parameter allows us to control the tradeoff of importance between precision and recall. $\\beta < 1$ focuses more on precision while $\\beta > 1$ focuses more on recall. When I was searching for other examples to explain the tradeoff between precision and recall, I came across the following article discussing [using machine learning to predict suicide](https://qz.com/1001968/artificial-intelligence-can-now-predict-suicide-with-remarkable-accuracy/?ref=jeremyjordan.me). In this case, we'd want to put much more focus on the model's recall than its precision. It would be much less harmful to have an intervention with someone who was not actually considering suicide than it would be to miss someone who was considering suicide. However, precision is still important because you don't want **too** many instances where your model predicts false positives or else you have the case of "The Model Who Cried Wolf" ([this is the reference for those not familiar with the story](https://en.wikipedia.org/wiki/The%5FBoy%5FWho%5FCried%5FWolf?ref=jeremyjordan.me)). *Note: The article only reports the accuracy of these models, not the precision or recall! By now you should know that accuracy alone is not too informative regarding a model's effectiveness. The original paper published, however, does present the precision and recall.* ###### Regression metrics Evaluation metrics for regression models are quite different than the above metrics we discussed for classification models because we are now predicting in a continuous range instead of a discrete number of classes. If your regression model predicts the price of a house to be $400K and it sells for $405K, that's a pretty good prediction. However, in the classification examples we were only concerned with whether or not a prediction was correct or incorrect, there was no ability to say a prediction was "pretty good". Thus, we have a different set of evaluation metrics for regression models. **Explained variance** compares the variance within the expected outcomes, and compares that to the variance in the error of our model. This metric essentially represents the amount of variation in the original dataset that our model is able to explain. $$ EV\\left( {{y\_{true}},{y\_{pred}}} \\right) = 1 - \\frac{{Var\\left( {{y\_{true}} - {y\_{pred}}} \\right)}}{{{y\_{true}}}} $$ **Mean squared error** is simply defined as the average of squared differences between the predicted output and the true output. Squared error is commonly used because it is agnostic to whether the prediction was too high or too low, it just reports that the prediction was incorrect. \[{\\rm{MSE}}\\left( {{y \_{true}},{y \_{pred}}} \\right) = \\frac{1}{{{n \_{samples}}}}{\\sum {\\left( {{y \_{true}} - {y \_{pred}}} \\right)} ^2}\] The **R2 coefficient** represents the proportion of variance in the outcome that our model is capable of predicting based on its features. \[{R^2}\\left( {{y \_{true}},{y \_{pred}}} \\right) = 1 - \\frac{{{{\\sum {\\left( {{y \_{true}} - {y \_{pred}}} \\right)} }^2}}}{{{{\\sum {\\left( {{y \_{true}} - \\bar y} \\right)} }^2}}}\] \[\\bar y = \\frac{1}{{{n \_{samples}}}}\\sum {{y \_{true}}} \] ### Bias vs Variance The ultimate goal of any machine learning model is to learn from examples and generalize some degree of knowledge regarding the task we're training it to perform. Some machine learning models provide the framework for generalization by suggesting the underlying structure of that knowledge. For example, a linear regression model imposes a framework to learn linear relationships between the information we feed it. However, sometimes we provide a model with too much pre-built structure that we limit the model's ability to learn from the examples - such as the case where we train a linear model on a exponential dataset. In this case, our model is **biased** by the pre-imposed structure and relationships. Models with high bias pay little attention to the data presented; this is also known as **underfitting**. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/download--2-.png) It's also possible to bias a model by trying to teach it to perform a task without presenting all of the necessary information. If you know the constraints of the model are not biasing the model's performance yet you're still observed signs of underfitting, it's likely that you are not using enough features to train the model. On the other extreme, sometimes when we train our model it learns *too much* from the training data. That is, our model captures the noise in the data in addition to the signal. This can cause wild fluctuations in the model that does not represent the true trend; in this case, we say that the model has high **variance**. In this case, our model does not generalize well because it pays too much attention to the training data without consideration for generalizing to new data. In other words, we've **overfit** the model to the training data. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/download--3-.png) In summary, a model with high bias is limited from learning the true trend and underfits the data. A model with high variance learns too much from the training data and overfits the data. The best model sits somewhere in the middle of the two extremes. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/download--4-.png) Next, I'll discuss two common tools that are used to diagnosed whether a model is susceptible to high bias or variance. ###### Validation curves As we discussed in the previous section, the goal with any machine learning model is **generalization**. Validation curves allow us to find the sweet spot between underfitting and overfitting a model to build a model that generalizes well. A typical validation curve is a plot of the model's error as a function of some model hyperparameter which controls the model's tendency to overfit or underfit the data. The parameter you choose depends on the specific model you're evaluating; for example, you might choose to plot the degree of polynomial features (typically, this means you have polynomial features up to this degree) for a linear regression model. Generally, the chosen parameter will have some degree of control over the model's complexity. On this curve, we plot both the training error and the validation error of the model. Using both of these errors combined, we can diagnose whether a model is suffering from high bias or high variance. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-21-at-2.30.04-PM.png) In the region where both the training error and validation error are high, the model is subject to high bias. Here, it was not able to learn from the data and it performing poorly. In the region where the training error and validation error diverge, with the training error staying low and validation error increasing, we're beginning to see the effects of high variance. The training error is low because we're overfitting the data and learning too much from the training examples, while the validation error remains high because our model isn't able to generalize from the training data to new data. ###### Learning curves The second tool we'll discuss for diagnosing bias and variance in a model is learning curves. Here, we'll plot the error of a model as a function of the number of training examples. Similar to validation curves, we'll plot the error for both the training data and validation data. If our model has **high bias**, we'll observe fairly quick convergence to a high error for the validation and training datasets. If the model suffers from high bias, training on more data will do very little to improve the model. This is because models which underfit the data pay little attention to the data, so feeding in more data will be useless. A better approach to improving models which suffer from high bias is to consider adding additional features to the dataset so that the model can be more equipped to learn the proper relationships. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-21-at-5.12.23-PM.png) If our model has **high variance**, we'll see a gap between the training and validation error. This is because the model is performing well for the training data, since it has been overfit to that subset, and performs poorly for the validation data since it was not able to generalize the proper relationships. In this case, feeding more data during training can help improve the model's performance. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-21-at-5.18.22-PM.png) ### Other practical advice Another common thing I'll do when evaluating classifier models is to [reduce the dataset](https://www.jeremyjordan.me/principal-components-analysis/) into two dimensions and then plot the observations and decision boundary. Sometimes it's helpful to visually inspect the data and your model when evaluating its performance. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/sphx_glr_plot_iris_001.png) [*Image credit*](http://scikit-learn.org/stable/auto%5Fexamples/svm/plot%5Firis.html?ref=jeremyjordan.me) ### Further reading - [37 Reasons why your Neural Network is not working](https://blog.slavv.com/37-reasons-why-your-neural-network-is-not-working-4020854bd607?ref=jeremyjordan.me) - [Improving the Validation and Test Split](https://changhsinlee.com/better-validation-test/?ref=jeremyjordan.me) ### Neural networks: training with backpropagation. URL: https://www.jeremyjordan.me/neural-networks-training/ Last updated: 2025-03-03T02:08:53.000Z In my first post on neural networks, I discussed a [model representation for neural networks](https://www.jeremyjordan.me/neural-networks-representation/) and how we can feed in inputs and calculate an output. We calculated this output, layer by layer, by combining the inputs from the previous layer with weights for each neuron-neuron connection. I mentioned that we'd talk about how to find the proper weights to connect neurons together in a future post - this is that post! ### Overview In the previous post I had just assumed that we had magic prior knowledge of the proper weights for each neural network. In this post, we'll actually figure out how to get our neural network to "learn" the proper weights. However, for the sake of having somewhere to start, let's just initialize each of the weights with random values as an initial guess. We'll come back and revisit this random initialization step later on in the post. Given our randomly initialized weights connecting each of the neurons, we can now feed in our matrix of observations and calculate the outputs of our neural network. This is called forward propagation. Given that we chose our weights at random, our output is probably not going to be very good with respect to our expected output for the dataset. So where do we go from here? Well, for starters, let's define what a "good" output looks like. Namely, we'll develop a **cost function** which penalizes outputs far from the expected value. Next, we need to figure out a way to change the weights so that the cost function improves. Any given path from an input neuron to an output neuron is essentially just a composition of functions; as such, we can use partial derivatives and the chain rule to **define the relationship between any given weight and the cost function**. We can use this knowledge to then leverage [gradient descent](https://www.jeremyjordan.me/gradient-descent/) in updating each of the weights. ### Pre-requisites When I was first learning backpropagation, many people tried to abstract away the underlying math (derivative chains) and I was never really able to grasp what the heck was going on until watching [Professor Winston's lecture](https://www.youtube.com/watch?v=uXt8qF2Zzfo&ref=jeremyjordan.me) at MIT. I hope that this post gives a better understanding of backpropagation than simply "this is the step where we send the error backwards to update the weights". In order to fully grasp the concepts discussed in this post, you should be familiar with the following: **Partial derivatives** This post is going to be a bit dense with a lot of partial derivatives. However, it is my hope that even a reader without prior knowledge of multivariate calculus can still follow the logic behind backpropagation. If you're not familiar with calculus, $\\frac{{\\partial f\\left( x \\right)}}{{\\partial x}}$ will probably look pretty foreign. You can interpret this expression as "how does $f\\left( x \\right)$ change as I change $x$?" This will be useful because we can ask questions like "How does the cost function change when I change this parameter? Does it increase or decrease the cost function?" in search of the optimal parameters. If you'd like to brush up on multivariate calculus, check out [Khan Academy's lessons](https://www.khanacademy.org/math/multivariable-calculus?ref=jeremyjordan.me) on the subject. **Gradient descent** To prevent this post from getting too long, I've separated the topic of gradient descent into another post. If you're not familiar with the method, be sure to read about it [here](https://www.jeremyjordan.me/gradient-descent/) and understand it before continuing this post. **Matrix multiplication** [Here](https://www.khanacademy.org/math/precalculus/precalc-matrices/multiplying-matrices-by-matrices/v/matrix-multiplication-intro?ref=jeremyjordan.me) is a quick refresher from Khan Academy. ### Starting simple To figure out how to use gradient descent in training a neural network, let's start with the simplest neural network: one input neuron, one hidden layer neuron, and one output neuron. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/10/Screen-Shot-2018-10-08-at-11.05.44-PM.png) To show a more complete picture of what's going on, I've expanded each neuron to show 1) the linear combination of inputs and weights and 2) the nonlinear activation of this combination. It's easy to see that the forward propagation step is simply a series of functions where the output of one feeds as the input to the next. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-16-at-1.42.55-PM.png) ###### Defining "good" performance in a neural network Let's define our cost function to simply be the squared error. $$J\\left( \\theta \\right) = {\\raise0.5ex\\hbox{$\\scriptstyle 1$} \\kern-0.1em/\\kern-0.15em \\lower0.25ex\\hbox{$\\scriptstyle 2$}}{\\left( {y - {{\\rm{a}}^{(3)}}} \\right)^2}$$ There's a myriad of cost functions we could use, but for this neural network squared error will work just fine. Remember, we want to evaluate our model's output with respect to the target output in an attempt to minimize the difference between the two. ### Relating the weights to the cost function In order to minimize the difference between our neural network's output and the target output, we need to know how the model performance changes with respect to each parameter in our model. In other words, we need to define the relationship (read: partial derivative) between our cost function and each weight. We can then update these weights in an iterative process using gradient descent. $$\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\theta \_1}}} = ?$$ $$\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\theta \_2}}} = ?$$ Let's look at $\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\theta \_2}}}$ first. Keep the following figure in mind as we progress. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-16-at-2.16.54-PM.png) Let's take a moment to examine how we could express the relationship between $J\\left( \\theta \\right)$ and $\\theta \_2$. Note how $\\theta \_2$ is an input to ${z^{(3)}}$, which is an input to ${{\\rm{a}}^{(3)}}$, which is an input to $J\\left( \\theta \\right)$. When we're trying to compute a derivative of this sort, we can use the chain rule to solve. As a reminder, the chain rule states: $$\\frac{\\partial }{{\\partial z}}p\\left( {q\\left( z \\right)} \\right) = \\frac{{\\partial p}}{{\\partial q}}\\frac{{\\partial q}}{{\\partial z}}$$ Let's apply the chain rule to solve for $\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\theta \_2}}}$. $$\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\theta \_2}}} = \\left( {\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {{\\rm{a}}^{(3)}}}}} \\right)\\left( {\\frac{{\\partial {{\\rm{a}}^{(3)}}}}{{\\partial z}}} \\right)\\left( {\\frac{{\\partial z}}{{\\partial {\\theta \_2}}}} \\right)$$ By similar logic, we can find $\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\theta \_1}}}$. $$\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\theta \_1}}} = \\left( {\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {{\\rm{a}}^{(3)}}}}} \\right)\\left( {\\frac{{\\partial {{\\rm{a}}^{(3)}}}}{{\\partial {z^{(3)}}}}} \\right)\\left( {\\frac{{\\partial {z^{(3)}}}}{{\\partial {{\\rm{a}}^{(2)}}}}} \\right)\\left( {\\frac{{\\partial {{\\rm{a}}^{(2)}}}}{{\\partial {z^{(2)}}}}} \\right)\\left( {\\frac{{\\partial {z^{(2)}}}}{{\\partial {\\theta \_1}}}} \\right)$$ For the sake of clarity, I've updated our neural network diagram to visualize these chains. Make sure you are comfortable with this process before proceeding. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-16-at-4.44.08-PM.png) ### Adding complexity Now let's try this same approach on a slightly more complicated example. Now, we'll look at a neural network with two neurons in our input layer, two neurons in one hidden layer, and two neurons in our output layer. For now, we'll disregard the bias neurons that are missing from the input and hidden layers. > Let's take a second to go over the notation I'll be using so you can follow along with these diagrams. The **superscript** (1) **denotes what layer the object is in** and the **subscript denotes which neuron** we're referring to **in a given layer**. For example $a\_1^{(2)}$ is the activation of the first neuron in the second layer. For the **parameter values** $\\theta$, I like to read them like a **mailing label** \- the first value denotes what neuron the input is being sent **to** in the next layer, and the second value denotes **from** what neuron the information is being sent. For example, ${\\theta \_{21}^{(2)}}$ is used to send input **to** the 2nd neuron, **from** the 1st neuron in layer 2\. The superscript denoting the layer corresponds with where the input is coming from. This notation is consistent with the matrix representation we discussed in my post on neural networks representation. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-17-at-11.13.18-AM.png) Let's expand this network to expose all of the math that's going on. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-17-at-2.47.27-PM.png) Yikes! Things got a little more complicated. I'll walk through the process for finding one of the partial derivatives of the cost function with respect to one of the parameter values; I'll leave the rest of the calculations as an exercise for the reader (and post the end results below). Foremost, we'll need to revisit our cost function now that we're dealing with a neural network with more than one output. Let's now use the mean squared error as our cost function. $$J\\left( \\theta \\right) = \\frac{1}{{2m}}\\sum{{{\\left( {{y \_i} - {\\rm{a}} \_i^{(2)}} \\right)}^2}} $$ *Note: If you're training on multiple observations (which you always will in practice), we'll also need to perform a summation of the cost function over all training examples. For this cost function, it's common to normalize by $\\frac{1}{{m}}$ where $m$ is the number of examples in your training dataset.* Now that we've corrected our cost function, we can look at how changing a parameter affects the cost function. Specifically, I'll calculate $\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{11}^{(1)}}}$ in this example. Looking at the diagram, $\\theta \_{11}^{(1)}$ affects the output for both $a \_1^{(3)}$ and $a \_2^{(3)}$. Because our cost function is a summation of individual costs for each output, we can calculate the derivative chain for each path and simply add them together. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-18-at-10.40.52-AM.png) The derivative chain for the blue path is: $$ \\left( {\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\rm{a}} \_1^{(3)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_1^{(3)}}}{{\\partial z \_1^{(3)}}}} \\right)\\left( {\\frac{{\\partial z \_1^{(3)}}}{{\\partial {\\rm{a}} \_1^{(2)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_1^{(2)}}}{{\\partial z \_1^{(2)}}}} \\right)\\left( {\\frac{{\\partial z \_1^{(2)}}}{{\\partial \\theta \_{11}^{(1)}}}} \\right) $$ The derivative chain for the orange path is: $$ \\left( {\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\rm{a}} \_2^{(3)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_2^{(3)}}}{{\\partial z \_2^{(3)}}}} \\right)\\left( {\\frac{{\\partial z \_2^{(3)}}}{{\\partial {\\rm{a}} \_1^{(2)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_1^{(2)}}}{{\\partial z \_1^{(2)}}}} \\right)\\left( {\\frac{{\\partial z \_1^{(2)}}}{{\\partial \\theta \_{11}^{(1)}}}} \\right)$$ Combining these, we get the total expression for $\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{11}^{(1)}}}$. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-18-at-10.53.41-AM.png) I've provided the remainder of the partial derivatives below. Remember, we need these partial derivatives because they describe how changing each parameter affects the cost function. Thus, we can use this knowledge to change all of the parameter values in a way that continues to decrease the cost function until we converge on some minimum value. ###### Layer 2 Parameters $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{11}^{(2)}}} = \\left( {\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\rm{a}} \_1^{(3)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_1^{(3)}}}{{\\partial z \_1^{(3)}}}} \\right)\\left( {\\frac{{\\partial z \_1^{(3)}}}{{\\partial \\theta \_{11}^{(2)}}}} \\right) $$ $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{12}^{(2)}}} = \\left( {\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\rm{a}} \_1^{(3)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_1^{(3)}}}{{\\partial z \_1^{(3)}}}} \\right)\\left( {\\frac{{\\partial z \_1^{(3)}}}{{\\partial \\theta \_{12}^{(2)}}}} \\right) $$ $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{21}^{(2)}}} = \\left( {\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\rm{a}} \_2^{(3)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_2^{(3)}}}{{\\partial z \_2^{(3)}}}} \\right)\\left( {\\frac{{\\partial z \_2^{(3)}}}{{\\partial \\theta \_{21}^{(2)}}}} \\right) $$ $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{22}^{(2)}}} = \\left( {\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\rm{a}} \_2^{(3)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_2^{(3)}}}{{\\partial z \_2^{(3)}}}} \\right)\\left( {\\frac{{\\partial z \_2^{(3)}}}{{\\partial \\theta \_{22}^{(2)}}}} \\right)$$ ###### Layer 1 Parameters $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{11}^{(1)}}} = \\left( {\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\rm{a}} \_1^{(3)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_1^{(3)}}}{{\\partial z \_1^{(3)}}}} \\right)\\left( {\\frac{{\\partial z \_1^{(3)}}}{{\\partial {\\rm{a}} \_1^{(2)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_1^{(2)}}}{{\\partial z \_1^{(2)}}}} \\right)\\left( {\\frac{{\\partial z \_1^{(2)}}}{{\\partial \\theta \_{11}^{(1)}}}} \\right) + \\left( {\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\rm{a}} \_2^{(3)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_2^{(3)}}}{{\\partial z \_2^{(3)}}}} \\right)\\left( {\\frac{{\\partial z \_2^{(3)}}}{{\\partial {\\rm{a}} \_1^{(2)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_1^{(2)}}}{{\\partial z \_1^{(2)}}}} \\right)\\left( {\\frac{{\\partial z \_1^{(2)}}}{{\\partial \\theta \_{11}^{(1)}}}} \\right) $$ $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{12}^{(1)}}} = \\left( {\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\rm{a}} \_1^{(3)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_1^{(3)}}}{{\\partial z \_1^{(3)}}}} \\right)\\left( {\\frac{{\\partial z \_1^{(3)}}}{{\\partial {\\rm{a}} \_1^{(2)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_1^{(2)}}}{{\\partial z \_1^{(2)}}}} \\right)\\left( {\\frac{{\\partial z \_1^{(2)}}}{{\\partial \\theta \_{12}^{(1)}}}} \\right) + \\left( {\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\rm{a}} \_2^{(3)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_2^{(3)}}}{{\\partial z \_2^{(3)}}}} \\right)\\left( {\\frac{{\\partial z \_2^{(3)}}}{{\\partial {\\rm{a}} \_1^{(2)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_1^{(2)}}}{{\\partial z \_1^{(2)}}}} \\right)\\left( {\\frac{{\\partial z \_1^{(2)}}}{{\\partial \\theta \_{12}^{(1)}}}} \\right) $$ $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{21}^{(1)}}} = \\left( {\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\rm{a}} \_1^{(3)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_1^{(3)}}}{{\\partial z \_1^{(3)}}}} \\right)\\left( {\\frac{{\\partial z \_1^{(3)}}}{{\\partial {\\rm{a}} \_2^{(2)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_2^{(2)}}}{{\\partial z \_2^{(2)}}}} \\right)\\left( {\\frac{{\\partial z \_2^{(2)}}}{{\\partial \\theta \_{21}^{(1)}}}} \\right) + \\left( {\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\rm{a}} \_2^{(3)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_2^{(3)}}}{{\\partial z \_2^{(3)}}}} \\right)\\left( {\\frac{{\\partial z \_2^{(3)}}}{{\\partial {\\rm{a}} \_2^{(2)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_2^{(2)}}}{{\\partial z \_2^{(2)}}}} \\right)\\left( {\\frac{{\\partial z \_2^{(2)}}}{{\\partial \\theta \_{21}^{(1)}}}} \\right) $$ $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{22}^{(1)}}} = \\left( {\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\rm{a}} \_1^{(3)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_1^{(3)}}}{{\\partial z \_1^{(3)}}}} \\right)\\left( {\\frac{{\\partial z \_1^{(3)}}}{{\\partial {\\rm{a}} \_2^{(2)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_2^{(2)}}}{{\\partial z \_2^{(2)}}}} \\right)\\left( {\\frac{{\\partial z \_2^{(2)}}}{{\\partial \\theta \_{22}^{(1)}}}} \\right) + \\left( {\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\rm{a}} \_2^{(3)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_2^{(3)}}}{{\\partial z \_2^{(3)}}}} \\right)\\left( {\\frac{{\\partial z \_2^{(3)}}}{{\\partial {\\rm{a}} \_2^{(2)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_2^{(2)}}}{{\\partial z \_2^{(2)}}}} \\right)\\left( {\\frac{{\\partial z \_2^{(2)}}}{{\\partial \\theta \_{22}^{(1)}}}} \\right) $$ Woah there. We just went from a neural network with 2 parameters that needed 8 partial derivative terms in the previous example to a neural network with 8 parameters that needed 52 partial derivative terms. This is going to quickly get out of hand, especially considering many neural networks that are used in practice are much larger than these examples. Fortunately, upon closer inspection many of these partial derivatives are repeated. If we're smart about how we approach this problem, we can dramatically reduce the computational cost of training. Further, it'd really be a pain in the ass if we had to manually calculate the derivative chains for every parameter. Let's look at what we've done so far and see if we can generalize a method to this madness. ### Generalizing a method Let's examine the partial derivatives above and make a few observations. We'll start with looking at the partial derivatives with respect to the parameters for layer 2\. Remember, the parameters in for layer 2 are combined with the activations in layer 2 to feed as inputs into layer 3. ###### Layer 2 Parameters Let's analyze the following expressions; I encourage you to solve the partial derivatives as we go along to convince yourself of my logic. $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{11}^{(2)}}} = \\left( {\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\rm{a}} \_1^{(3)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_1^{(3)}}}{{\\partial z \_1^{(3)}}}} \\right)\\left( {\\frac{{\\partial z \_1^{(3)}}}{{\\partial \\theta \_{11}^{(2)}}}} \\right) $$ $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{12}^{(2)}}} = \\left( {\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\rm{a}} \_1^{(3)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_1^{(3)}}}{{\\partial z \_1^{(3)}}}} \\right)\\left( {\\frac{{\\partial z \_1^{(3)}}}{{\\partial \\theta \_{12}^{(2)}}}} \\right) $$ $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{21}^{(2)}}} = \\left( {\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\rm{a}} \_2^{(3)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_2^{(3)}}}{{\\partial z \_2^{(3)}}}} \\right)\\left( {\\frac{{\\partial z \_2^{(3)}}}{{\\partial \\theta \_{21}^{(2)}}}} \\right) $$ $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{22}^{(2)}}} = \\left( {\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\rm{a}} \_2^{(3)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_2^{(3)}}}{{\\partial z \_2^{(3)}}}} \\right)\\left( {\\frac{{\\partial z \_2^{(3)}}}{{\\partial \\theta \_{22}^{(2)}}}} \\right)$$ Foremost, it seems as if the columns contain very similar values. For example, the **first column** contains partial derivatives of the cost function with respect to the neural network outputs. In practice, this is the difference between the expected output and the actual output (and then scaled by $m$) for each of the output neurons. The **second column** represents the derivative of the activation function used in the output layer. Note that for each layer, neurons will use the same activation function. A homogenous activation function within a layer is necessary in order to be able to leverage matrix operations in calculating the neural network output. Thus, the value for the second column will be the same for all four terms. The first and second columns can be combined as $\\delta \_i^{(3)}$ for the sake of convenience later on down the road. It is not immediately evident why this would be helpful, but you'll see as we go backwards another layer why this is useful. People will often refer to this expression as the "error" term which we use to "send back error from the output throughout the network". We'll see why this is the case soon. Each neuron in the network will have a corresponding $\\delta$ term that we'll solve for. $$\\delta \_i^{(3)} = \\frac{1}{m}\\left( {{y \_i} - a \_i^{(3)}} \\right)f'\\left( {{a^{(3)}}} \\right)$$ The **third column** represents how the parameter of interest changes with respect to the weighted inputs for the current layer; when you calculate the derivative this corresponds with the activation from the previous layer. I'd also like to note that the first two partial derivative terms seem concerned with the first output neuron (neuron 1 in layer 3) while the last two partial derivative terms seem concerned with the second output neuron (neuron 2 in layer 3). This is evident in the $\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\rm{a}}\_i^{(3)}}}$ term. Let's use this knowledge to rewrite the partial derivatives using the $\\delta$ expression we defined above. $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{11}^{(2)}}} = \\delta \_1^{(3)}\\left( {\\frac{{\\partial z \_1^{(3)}}}{{\\partial \\theta \_{11}^{(2)}}}} \\right) $$ $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{12}^{(2)}}} = \\delta \_1^{(3)}\\left( {\\frac{{\\partial z \_1^{(3)}}}{{\\partial \\theta \_{12}^{(2)}}}} \\right) $$ $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{21}^{(2)}}} = \\delta \_2^{(3)}\\left( {\\frac{{\\partial z \_2^{(3)}}}{{\\partial \\theta \_{21}^{(2)}}}} \\right) $$ $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{22}^{(2)}}} = \\delta \_2^{(3)}\\left( {\\frac{{\\partial z \_2^{(3)}}}{{\\partial \\theta \_{22}^{(2)}}}} \\right) $$ Next, let's go ahead and calculate the last partial derivative term. As we noted, this partial derivative ends up representing activations from the previous layer. *Note: I find it helpful to use the expanded neural network graphic in the previous section when calculating the partial derivatives.* $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{11}^{(2)}}} = \\delta \_1^{(3)}{\\rm{a}} \_1^{(2)} $$ $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{12}^{(2)}}} = \\delta \_1^{(3)}{\\rm{a}} \_2^{(2)} $$ $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{21}^{(2)}}} = \\delta \_2^{(3)}{\\rm{a}} \_1^{(2)} $$ $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{22}^{(2)}}} = \\delta \_2^{(3)}{\\rm{a}} \_2^{(2)} $$ It appears that we're combining the "error" terms with activations from the previous layer to calculate each partial derivative. It's also interesting to note that the indices $j$ and $k$ for $\\theta \_{jk}$ match the combined indices of $\\delta \_j^{(3)}$ and ${\\rm{a}} \_k^{(2)}$. $$\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{jk}^{(2)}}} = \\delta \_j^{(3)}{\\rm{a}} \_k^{(2)}$$ Let's see if we can figure out a matrix operation to compute all of the partial derivatives in one expression. ${\\delta ^{(3)}}$ is a vector of length $j$ where $j$ is equal to the number of output neurons. δ (3) \=\[ y 1 − a 1 (3) y 2 − a 2 (3) ⋯ y j − a j (3) \] f ′ ( a (3) ) ${\\rm{a}}^{(2)}$ is a vector of length $k$ where $k$ is the number of neurons in the previous layer. The values of this vector represent activations of the previous layer calculated during forward propagation; in other words, it's the vector of inputs to the output layer. $$ {a^{(2)}} = \\left\[ {\\begin{array}{\* {20}{c}} {a \_1^{(2)}}&{a \_2^{(2)}}& \\cdots &{a \_k^{(2)}} \\end{array}} \\right\] $$ Multiplying these vectors together, we can calculate all of the partial derivative terms in one expression. ∂J( θ ) ∂ θ ij (2) \=\[ δ 1 (3) δ 2 (3) \]\[ a 1 (2) a 2 (2) \]\=\[ δ 1 (3) a 1 (2) δ 1 (3) a 2 (2) δ 2 (3) a 1 (2) δ 2 (3) a 2 (2) \] To summarize, see the graphic below. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-18-at-11.13.30-PM.png) *Note: Technically, the first column should also be scaled by $m$ to be an accurate derivative. However, my focus was to point out that this is the column where we're concerned with the difference between the expected and actual outputs. The $\\delta$ term on the right has the full derivative expression.* ###### Layer 1 Parameters Now let's take a look and see what's going on in layer 1. Whereas the weights in layer 2 only directly affected one output, the weights in layer 1 affect all of the outputs. Recall the following graphic. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-18-at-10.40.52-AM.png) This results in a partial derivative of the cost function with respect to a parameter now becoming a summation of different chains. Specifically, we'll have a derivative chain for every $\\delta$ we calculated in the next layer forward. Remember, we started at the end of the network and are working our way backwards through the network. Thus, the next layer forward represents the $\\delta$ values that we previously calculated. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-19-at-12.25.31-PM.png) As we did for layer 2, let's make a few observations. The **first two columns** (of each summed term) corresponds with a $\\delta \_j^{(3)}$ calculated in the next layer forward (remember, we started at the end of the network and are working our way back). The **third column** corresponds with some parameter that connects layer 2 to layer 3\. If we considered $\\delta \_j^{(3)}$ to be some "error" term for layer 3, and each derivative chain is now a summation of these errors, then this third column allows us to weight each respective error. Thus, the first three terms combined represent some measure of the proportional error. We'll also redefine $\\delta$ for all layers excluding the output layer to include this combination of weighted errors. $$ \\delta \_j^{(l)} = f'\\left( {{a^{(l)}}} \\right)\\sum\\limits \_{i = 1}^n {\\delta \_i^{(l + 1)}\\theta \_{ij}^{(l)}} $$ We could write this more succinctly using matrix expressions. $$ {\\delta ^{(l)}} = {\\delta ^{(l + 1)}}{\\Theta ^{(l)}}f'\\left( {{a^{(l)}}} \\right) $$ *Note: It's standard notation to denote vectors with lowercase letters and matrices as uppercase letters. Thus, $\\theta$ represents a vector while $\\Theta$ represents a matrix.* The **fourth column** represents the derivative of the activation function used in the current layer. Remember that for each layer, neurons will use the same activation function. Lastly, the **fifth column** represents various inputs from the previous layer. In this case, it is the actual inputs to the neural network. Let's take a closer look at one of the terms, $\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{11}^{(1)}}}$. Foremost, we established that the first two columns of each derivative chains were previously calculated as $\\delta \_j^{(3)}$. $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{11}^{(1)}}} = \\delta \_1^{(3)}\\left( {\\frac{{\\partial z \_1^{(3)}}}{{\\partial {\\rm{a}} \_1^{(2)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_1^{(2)}}}{{\\partial z \_1^{(2)}}}} \\right)\\left( {\\frac{{\\partial z \_1^{(2)}}}{{\\partial \\theta \_{11}^{(1)}}}} \\right) + \\delta \_2^{(3)}\\left( {\\frac{{\\partial z \_2^{(3)}}}{{\\partial {\\rm{a}} \_1^{(2)}}}} \\right)\\left( {\\frac{{\\partial {\\rm{a}} \_1^{(2)}}}{{\\partial z \_1^{(2)}}}} \\right)\\left( {\\frac{{\\partial z \_1^{(2)}}}{{\\partial \\theta \_{11}^{(1)}}}} \\right) $$ Further, we established the the third columns of each derivative chain was a parameter that acted to weight each of the respective $\\delta$ terms. We also established that the fourth column was the derivative of the activation function. $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{11}^{(1)}}} = \\delta \_1^{(3)}\\theta \_{11}^{(2)}f'\\left( {{a^{(2)}}} \\right)\\left( {\\frac{{\\partial z \_1^{(2)}}}{{\\partial \\theta \_{11}^{(1)}}}} \\right) + \\delta \_2^{(3)}\\theta \_{21}^{(2)}f'\\left( {{a^{(2)}}} \\right)\\left( {\\frac{{\\partial z \_1^{(2)}}}{{\\partial \\theta \_{11}^{(1)}}}} \\right) $$ Factoring out $\\left( {\\frac{{\\partial z\_1^{(2)}}}{{\\partial \\theta \_{11}^{(1)}}}} \\right)$, $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{11}^{(1)}}} = \\left( {\\frac{{\\partial z \_1^{(2)}}}{{\\partial \\theta \_{11}^{(1)}}}} \\right)\\left( {\\delta \_1^{(3)}\\theta \_{11}^{(2)}f'\\left( {{a^{(2)}}} \\right) + \\delta \_2^{(3)}\\theta \_{21}^{(2)}f'\\left( {{a^{(2)}}} \\right)} \\right) $$ we're left with our new definition of $\\delta\_j^{(l)}$. Let's go ahead and substitute that in. $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{11}^{(1)}}} = \\left( {\\frac{{\\partial z \_1^{(2)}}}{{\\partial \\theta \_{11}^{(1)}}}} \\right)\\left( {\\delta \_1^{(2)}} \\right) $$ Lastly, we established the fifth column (${\\frac{{\\partial z\_1^{(2)}}}{{\\partial \\theta\_{11}^{(1)}}}}$) corresponded with an input from the previous layer. In this case, the derivative calculates to be $x\_1$. $$ \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{11}^{(1)}}} = \\delta \_1^{(2)}{x \_1} $$ Again let's figure out a matrix operation to compute all of the partial derivatives in one expression. $\\delta ^{(2)}$ is a vector of length $j$ where $j$ is the number of neurons in the current layer (layer 2). We can calculate $\\delta ^{(2)}$ as the weighted combination of errors from layer 3, multiplied by the derivative of the activation function used in layer 2. δ (2) \=\[ δ 1 (3) δ 2 (3) \]\[ θ 11 (2) θ 12 (2) θ 21 (2) θ 22 (2) \] f ′ ( a (2) )\=\[ δ 1 (2) δ 2 (2) \] $x$ is a vector of input values. Multiplying these vectors together, we can calculate all of the partial derivative terms in one expression. ∂J( θ ) ∂ θ ij (1) \=\[ δ 1 (2) δ 2 (2) \]\[ x 1 x 2 \]\=\[ δ 1 (2) x 1 δ 1 (2) x 2 δ 2 (2) x 1 δ 2 (2) x 2 \] If you've made it this far, congrats! We just calculated all of the partial derivatives necessary to use gradient descent and optimize our parameter values. In the next section, I'll introduce a way to visualize the process we've just developed in addition to presenting an end-to-end method for implementing backpropagation. If you understand everything up to this point, it should be smooth sailing from here on out. ### Backpropagation In the last section, we developed a way to calculate all of the partial derivatives necessary for gradient descent (partial derivative of the cost function with respect to all model parameters) using matrix expressions. In calculating the partial derivatives, we started at the end of the network and, layer by layer, worked our way back to the beginning. We also developed a new term, $\\delta$, which essentially serves to represent all of the partial derivative terms that we would need to reuse later and we progress, layer by layer, backwards through the network. *Note: Backpropagation is simply a method for calculating the partial derivative of the cost function with respect to all of the parameters. The actual optimization of parameters (training) is done by gradient descent or another more advanced optimization technique.* Generally, we established that you can calculate the partial derivatives for layer $l$ by combining $\\delta$ terms of the next layer forward with the activations of the current layer. $$ \\frac{\\partial J(\\theta)}{\\partial \\theta\_{ij}^{(l)}} = \\left( \\delta^{(l+1)} \\right)^T a^{(l)} $$ ###### Backpropagation visualized Before defining the formal method for backpropagation, I'd like to provide a visualization of the process. First, we have to compute the output of a neural network via forward propagation. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-19-at-4.52.13-PM.png) Next, we compute the ${\\delta ^{(3)}}$ terms for the last layer in the network. Remember, these $\\delta$ terms consist of all of the partial derivatives that will be used again in calculating parameters for layers further back. In practice, we typically refer to $\\delta$ as the "error" term. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-19-at-5.05.21-PM.png) ${\\Theta ^{(2)}}$ is the matrix of parameters that connects layer 2 to 3\. We multiply the error from the third layer by the inputs in the second layer to calculate our partial derivatives for this set of parameters. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-19-at-5.03.13-PM.png) Next, we "send back" the "error" terms in the exact same manner that we "send forward" the inputs to a neural network. The only difference is that time, we're starting from the back and we're feeding an error term, layer by layer, backwards through the network. Hence the name: backpropagation. The act of "sending back our error" is accomplished via the expression ${\\delta ^{(3)}}{\\Theta ^{(2)}}$. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-19-at-5.05.05-PM-1.png) ${\\Theta ^{(1)}}$ is the matrix of parameters that connects layer 1 to 2\. We multiply the error from the second layer by the inputs in the first layer to calculate our partial derivatives for this set of parameters. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-19-at-5.03.32-PM.png) For every layer except for the last, the "error" term is a linear combination of parameters connecting to the next layer (moving forward through the network) and the "error" terms of that next layer. This is true for all of the hidden layers, since we don't compute an "error" term for the inputs. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-20-at-12.37.21-AM.png) The last layer is a special case because we calculate the $\\delta$ values by directly comparing each output neuron to its expected output. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-19-at-11.21.18-PM.png) ###### A formalized method for implementing backpropagation Here, I'll present a practical method for implementing backpropagation through a network of layers $l=1,2,...,L$. 1. Perform forward propagation. 2. Compute the $\\delta$ term for the output layer. $${\\delta ^{(L)}} = \\frac{1}{m}\\left( {y - {a^{(L)}}} \\right)f'\\left( {{a^{(L)}}} \\right)$$ 3. Compute the partial derivatives of the cost function with respect to all of the parameters that feed into the output layer, ${\\Theta ^{(L - 1)}}$. $$\\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial \\theta \_{ij}^{(L-1)}}} = {\\left( {{\\delta ^{(L)}}} \\right)T}{a{(L-1)}}$$ 1. Go back one layer. $l = l - 1$ 2. Compute the $\\delta$ term for the current hidden layer. $${\\delta ^{(l)}} = {\\delta ^{(l + 1)}}{\\Theta ^{(l)}}f'\\left( {{a^{(l)}}} \\right)$$ 3. Compute the partial derivatives of the cost function with respect to all of the parameters that feed into the current layer. $$\\frac{\\partial J(\\theta)}{\\partial \\theta\_{ij}^{(l)}} = \\left( \\delta^{(l+1)} \\right)^T a^{(l)}$$ 1. Repeat 4 through 6 until you reach the input layer. ### Revisiting the weights initialization When we started, I proposed that we just randomly initialize our weights in order to have a place to start. This allowed us to perform forward propagation, compare the outputs to the expected values, and compute the cost of our model. It's actually very important that we initialize our weights to random values so that we can break the symmetry in our model. If we had initialized all of our weights to be equal to be the same, every neuron in the next layer forward would be equal to the same linear combination of values. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-20-at-12.05.13-AM.png) By this same logic, the $\\delta$ values would also be the same for every neuron in a given layer. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-20-at-12.05.23-AM.png) Further, because we calculate the partial derivatives in any given layer by combining $\\delta$ values and activations, all of the partial derivatives in any given layer would be identical. As such, the weights would update symmetrically in gradient descent and multiple neurons in any layer would be useless. This obviously would not be a very helpful neural network. Randomly initializing the network's weights allows us to break this symmetry and update each weight individually according to its relationship with the cost function. Typically we'll assign each parameter to a random value in $\\left\[ { - \\varepsilon ,\\varepsilon } \\right\]$ where $\\varepsilon$ is some value close to zero. ### Putting it all together After we've calculated all of the partial derivatives for the neural network parameters, we can use gradient descent to update the weights. In general, we defined gradient descent as $${\\theta \_i}: = {\\theta \_i} + \\Delta {\\theta \_i}$$ where $\\Delta {\\theta \_i}$ is the "step" we take walking along the gradient, scaled by a learning rate, $\\eta$. $$\\Delta {\\theta \_i} = - \\eta \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\theta \_i}}}$$ We'll use this formula to update each of the weights, recompute forward propagation with the new weights, backpropagate the error, and calculate the next weight update. This process continues until we've converged on an optimal value for our parameters. During each iteration we perform forward propagation to compute the outputs and backward propagation to compute the errors; one complete iteration is known as an epoch. It is common to report evaluation metrics after each epoch so that we can watch the evolution of our neural network as it trains. ### Further reading - [The Matrix Calculus You Need For Deep Learning](http://parrt.cs.usfca.edu/doc/matrix-calculus/index.html?ref=jeremyjordan.me) - [How the backpropagation algorithm works](http://neuralnetworksanddeeplearning.com/chap2.html?ref=jeremyjordan.me) - [Stanford cs231n: Backpropagation, Intuitions](http://cs231n.github.io/optimization-2/?ref=jeremyjordan.me) - [CS231n Winter 2016: Lecture 4: Backpropagation, Neural Networks 1](https://www.youtube.com/watch?v=i94OvYb6noo&ref=jeremyjordan.me) - [Lectures on Deep Learning](https://github.com/ndrplz/deep%5Flearning%5Flectures?ref=jeremyjordan.me) - [Yes you should understand backprop](https://medium.com/@karpathy/yes-you-should-understand-backprop-e2f06eab496b?ref=jeremyjordan.me) - [Building blocks of neural networks](https://www.coursera.org/learn/neural-networks-deep-learning/lecture/uGCun/building-blocks-of-deep-neural-networks?ref=jeremyjordan.me) - And in case you just gave up on backpropagation... [Deep Learning without Backpropagation](https://iamtrask.github.io/2017/03/21/synthetic-gradients/?ref=jeremyjordan.me) ### Gradient descent. URL: https://www.jeremyjordan.me/gradient-descent/ Last updated: 2025-03-03T01:59:36.000Z Gradient descent is an optimization technique commonly used in training machine learning algorithms. Often when we're building a machine learning model, we'll develop a cost function which is capable of measuring how well our model is doing. This function will penalize any error our model makes (by assigning a cost) with respect to the current parameter values. Thus, by minimizing the cost function we can find the optimal parameters that yield the best model performance. Common cost functions: - Mean squared error - Cross-entropy loss (log loss) Let's start off with an extremely simple example of a model with only one parameter. We can plot an arbitrary cost function, $J\\left( \\theta \\right)$, as a function of $\\theta$. Viewing this plot, it is easy to see what the best parameter for the model is (labeled as $\\theta \_{optimal}$) as it leads to the least amount of error possible in the model. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-14-at-6.25.00-PM.png) However, it would be computationally expensive to calculate the cost function for every possible parameter value. Instead, let's start with an initial guess and iteratively refine it. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-14-at-6.31.10-PM.png) Remember, we don't actually know what the plotted cost function looks like (imagine how computationally expensive it would be for a model with 100 parameters!), we only know what the cost function is at our initial guess. Fortunately, we constructed our cost function so that it was continuous and differentiable. Thus, we also know the instantaneous slope of the cost function at our current position. For multivariate functions, we cn calculate the gradient which represents the instantaneous slope of the function with respect to each parameter. (Hence the name of our optimization technique, gradient descent.) We can find the minimum point of a function by iteratively taking steps in the opposite direction of the instantaneous slope. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-14-at-6.50.01-PM.png) Next, we'll reevaluate the cost function and its gradient at our new parameter value... ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-14-at-6.51.54-PM.png) and then iteratively repeat this process of taking a step in the opposite direction of the gradient and then reevaluating the cost function and its gradient. Generally, we can define the process of gradient descent as $${\\theta \_i}: = {\\theta \_i} + \\Delta {\\theta \_i}$$ where $$\\Delta {\\theta\_i} = - \\alpha \\frac{{\\partial J\\left( \\theta \\right)}}{{\\partial {\\theta\_i}}}$$ $\\Delta {\\theta\_i}$ is the "step" we take walking along the gradient, and we set a learning rate, $\\alpha$, to control the size of our steps. For the case of a univariate cost function, we can only move along the cost function by moving forwards or backwards in one dimension. However, for multivariate cost functions, you can move along the cost function in any number of directions. Conveniently, the gradient of the cost function corresponds with the direction of greatest descent. Thus, we can extend our approach to a multivariate cost function as an efficient way to optimize our model's parameters. The following figure displays a gradient descent optimization of a linear regression model, where two parameters are optimized by finding the minimum value on the 3D cost function (represented by a contour plot). ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/linreg2d.png) Some people [describe gradient descent](https://www.youtube.com/watch?v=29PmNG7fuuM&ref=jeremyjordan.me) as follows: > Say you're on the top of a mountain and you want to figure out how to get to the bottom. With gradient descent, you'll simply look around in all possible directions and take a step in the steepest downhill direction. Then you'll look around again and take another step in the steepest downhill direction. You'll continue this process until you get to the bottom. However, gradient descent is actually quite a bit more elegant in that we **don't** have to look around in every direction to find the steepest downhill direction. We can find this direction in one swift *calculation* of the gradient! As you reach a local optima, the slope will approach zero. Once the slope of your current parameter reaches zero, your parameter value will stop updating. While this is generally useful as it results in convergence and signifies that we can stop the iterative process, it can pose a problem when the gradient descent finds a local optimum (see example in the figure below). To prevent gradient descent from getting stuck at local optima (which may not be the global optimum), we can use a momentum term which allows our search to overcome local optima. Further, in some cases we can develop cost functions which are proven to have a convex space, guaranteeing that we only have one optimum, the global minimum. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-14-at-8.37.42-PM.png) ### Mini batch gradient descent In general, when computing the cost function we look at the loss associated with each training example and then sum these values together for an overall cost of the *entire* dataset. This is the most basic form of gradient descent, also known as **batch gradient descent** since we compute the cost in one large batch computation. However, for big datasets this can take a long time to compute. Moreover, it begs the question: do we really need to see *all* of the data before making improvements to our parameter values? **Mini batch gradient descent** allows us to split our training data into mini batches which can be processed individually. When each mini batch is processed, we look at the cost function relative to the data from the current mini batch and update our parameters accordingly. Then we continue to do this iterating over all of the mini batches until we've seen the whole dataset. One full cycle through the dataset is referred to as an *epoch*. ![Screen-Shot-2017-11-15-at-7.26.05-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/11/Screen-Shot-2017-11-15-at-7.26.05-PM.png) This allows us to improve our model parameters at *each* mini batch iteration. If we split our data into 100 mini batches, this would allow us to take 100 steps towards the global optimum rather than just 1 step had we elected to use batch gradient descent. This said, because we're updating our parameters on a more narrow view of the data, we introduce a degree of variance into our optimization process. Although we will generally still follow the direction towards the global minimum, we're no longer guarunteed that each step will bring us closer to the optimal parameter values. At the extreme case, we can set a mini batch size equal to one where we perform an update on a single training observation. This is known as **stochastic gradient descent**. On the other extreme, a batch size equal to the number of training examples would represent batch gradient descent. Mini batch gradient descent lies somewhere in the middle of that spectrum, with common batch sizes including: 64, 128, 256, and 512\. Finding the optimal batch size will yield the fastest learning. ### Advanced techniques Now that you have a basic understanding for the gradient descent process, we can discuss more advanced optimization algorithms that promise faster learning than standard gradient descent. Ultimately, the following techniques are adaptations to the standard gradient descent equation in hopes of improving convergence properties. I introduced the concept of gradient descent with an example finding the optimal value (lowest cost) for a single parameter. However, some machine learning models contain millions of parameters, meaning we're optimizing a cost function over millions of dimensions. These cost functions can be quite complex. Ideally, we want to find the *global minimum* of the function, which represents the best possible parameter values to optimize our selected evaluation metric, given the design of our model. Depending on where we start, we'll encounter local minimum and saddlepoints along the way. Fortunately, because we're dealing with such a high dimensional space, the odds of having a minimum value in all dimensions is quite low, so saddle points are much more common. ![minmaxsaddle](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/11/minmaxsaddle.png) Photo by [Rong Ge](http://www.offconvex.org/2016/03/22/saddlepoints/?ref=jeremyjordan.me) Unforunately, saddle points can slow down or halt the optimization process as shown in the examples below. As you can see, the stochastic gradient descent prematurely converges to a suboptimal value. The other points are all variations of the standard gradient descent that allow for faster training with improved convergence. ![Long-Valley---Imgur](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/11/Long-Valley---Imgur.gif) Photo by [Alec Radford](http://www.denizyuret.com/2015/03/alec-radfords-animations-for.html?ref=jeremyjordan.me) Recall that for gradient descent we're taking a step along the gradient in *each* dimension. Notice how in the first animation we get stuck oscillating around the local minimum of one dimension and aren't able to continue downward because we're at the local maximum of the other dimension (and thus the gradient is close to zero). Because our step size in a given dimension is determined by the gradient value, we're slowed down in the presence of local optima. ![Saddle-Point---Imgur](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/11/Saddle-Point---Imgur.gif) Photo by [Alec Radford](http://www.denizyuret.com/2015/03/alec-radfords-animations-for.html?ref=jeremyjordan.me) ###### Momentum One approach to this problem is to use a **moving average gradient** instead of the immediate gradient at each time step. By using an average gradient over the previous $x$ steps, we're able to keep *momentum* when we approach local optima. As you approach the optima your immediate gradient will get smaller, but because you're averaging over $x$ timesteps, we're don't slow down as drastically as we would if we only used the immediate gradient value. This said, it's still possible to get stuck at a saddle point if we only have momentum in the direction of the minimum. You can see this in the first animation, where the momentum optimizer fails to exit the local optima. In order to implement momentum, we'll keep track (incrementally) of the moving average gradient, where $\\beta$ is a hyperparameter of the optimization scheme. $\\beta$ essentially controls how far back we'll look when calculating our moving average. For example, a common value for $\\beta$ is 0.9 which tracks the moving average of the last 10 timesteps. $${v\_{dW}} = \\beta {v\_{dW}} + \\left( {1 - \\beta } \\right)dW$$ $${v\_{db}} = \\beta {v\_{db}} + \\left( {1 - \\beta } \\right)db$$ *Note: Some formulas for moving average will have a bias correction, but this implementation typically disregards it.* This is sometimes referred to as the velocity term. Next, when updating the gradients, we'll use the velocity term in place of the immediate gradient value. $$W = W - \\alpha {v\_{dW}}$$ $$b = b - \\alpha {v\_{db}}$$ Momentum offers improvements navigating ravines, reducing back and forth oscillations (because in a moving average those would cancel out) and focusing on the steepest descent path. ###### RMSProp As mentioned previously, momentum can still have its shortfalls. Geoff Hinton introduced an alternative optimization scheme, RMSProp, in a Coursera course. This technique aims to prevent local optima from slowing our convergence process by adaptively scaling the learning rate in each dimension according to the exponentially weight average of the gradient. We keep track of the exponentially weighted average of the gradient in each dimension using the following equation, where $\\beta$ again represents a hyperparameter that controls how far back we look when computing the exponential moving average. $${s\_{dW}} = \\beta {s\_{dW}} + \\left( {1 - \\beta } \\right)d{W^2}$$ $${s\_{db}} = \\beta {s\_{db}} + \\left( {1 - \\beta } \\right)d{b^2}$$ We can then update our paramters using the standard form of gradient descent, but scale the learning rate $\\alpha$ by our exponentially weighted average. $$W = W - \\frac{\\alpha }{{\\sqrt {{s\_{dW}}} }}dW$$ $$b = b - \\frac{\\alpha }{{\\sqrt {{s\_{db}}} }}db$$ This has the effect of dampening out steep gradients as to reduce oscillations within ravines during our journey to the global optimum. ###### Adam Adam, which stands for Adaptive Moment Estimation, combines the two previous ideas of using a moving average gradient (momentum) and adaptive learning rates (RMSProp). These two values respectively represent the estimates of the [first and second moment](https://onlinecourses.science.psu.edu/stat414/node/193?ref=jeremyjordan.me) of the gradients, hence the name of the technique. We'll calculate the first and second moments, $v$ and $s$, using two separate $\\beta$ values. $${v\_{dW}} = {\\beta\_1}{v\_{dW}} + \\left( {1 - {\\beta\_1}} \\right)dW$$ $${s\_{dW}} = {\\beta\_2}{s\_{dW}} + \\left( {1 - {\\beta\_2}} \\right)d{W^2}$$ $${v\_{db}} = {\\beta\_1}{v\_{db}} + \\left( {1 - {\\beta\_1}} \\right)db$$ $${s\_{db}} = {\\beta\_2}{s\_{db}} + \\left( {1 - {\\beta\_2}} \\right)d{b^2}$$ For this technique, we do apply bias correction to the moving averages. $$v\_{dW}^{corrected} = \\frac{{{v\_{dW}}}}{{1 - \\beta\_1^t}}$$ $$v\_{db}^{corrected} = \\frac{{{v\_{db}}}}{{1 - \\beta\_1^t}}$$ $$s\_{dW}^{corrected} = \\frac{{{s\_{dW}}}}{{1 - \\beta\_2^t}}$$ $$s\_{db}^{corrected} = \\frac{{{s\_{db}}}}{{1 - \\beta\_2^t}}$$ We can then update our parameter values as follows. $$W = W - \\frac{\\alpha }{{\\sqrt {s\_{dW}^{corrected}} + \\varepsilon }}v\_{dW}^{corrected}$$ $$b = b - \\frac{\\alpha }{{\\sqrt {s\_{db}^{corrected}} + \\varepsilon }}v\_{db}^{corrected}$$ *Note: This update rule will have an added $\\epsilon$ in the denominator to prevent divide by zero errors. A typical value for $\\epsilon$ is $10^8$.* For this optimization scheme, we'll typically use $\\beta\_1=0.9$, and $\\beta\_2 =0.999$ as the initial hyperparamter values. ### Further reading - [Setting the learning rate of your neural network](https://www.jeremyjordan.me/nn-learning-rate/) - [An overview of gradient descent optimization algorithms: Sebastian Ruder](http://ruder.io/optimizing-gradient-descent/index.html?ref=jeremyjordan.me) - [Optimization: Stochastic Gradient Descent](http://ufldl.stanford.edu/tutorial/supervised/OptimizationStochasticGradientDescent/?ref=jeremyjordan.me) - [The Marginal Value of Adaptive Gradient Methods in Machine Learning](https://arxiv.org/abs/1705.08292?ref=jeremyjordan.me) - [More advanced gradient descent techniques](https://www.youtube.com/watch?v=nhqo0u1a6fw&ref=jeremyjordan.me) - [An Interactive Tutorial on Numerical Optimization](http://www.benfrederickson.com/numerical-optimization/?ref=jeremyjordan.me) - [Learning: tips for neural networks](http://cs231n.github.io/neural-networks-3/?ref=jeremyjordan.me) - [Intro to optimization in deep learning: Momentum, RMSProp and Adam](https://medium.com/paperspace/intro-to-optimization-in-deep-learning-momentum-rmsprop-and-adam-8335f15fdee2?ref=jeremyjordan.me) ### Principal components analysis (PCA). URL: https://www.jeremyjordan.me/principal-components-analysis/ Last updated: 2025-03-03T02:10:18.000Z Principal components analysis (PCA) is the most popular dimensionality reduction technique to date. It allows us to take an $n$-dimensional feature-space and reduce it to a $k$-dimensional feature-space while maintaining as much information from the original dataset as possible in the reduced dataset. Specifically, PCA will create a new feature-space that aims to capture as much variance as possible in the original dataset; I'll elaborate on this later in this post. *Note: We're only dealing with the feature-space and not any corresponding labels or output. Dimensionality reduction is an *unsupervised learning* technique that is agnostic to the features' labels.* Let's start by addressing the burning question: how the heck do we reduce the dimensionality of our dataset? A naive approach might be to simply perform [feature selection](https://www.jeremyjordan.me/feature-selection/) and keep only a subset of the original features. However, although we *are* reducing the dimensionality (we have less features), we're doing so by *deleting* information that might be valuable. A better approach would be to find a way to *compress* the information. The ultimate goal with dimensionality reduction is to find the most compressed representation possible to accurately describe the data. If all of the features within your dataset were completely independent signals, this would be very hard. However, it's often the case that you will have redundancies present in your feature-space. You may find that many features are simply indicators of some latent signal that we haven't directly observed. As such, we would expect that indicators of the same latent signal would be correlated in some manner. For example, if we were trying to reduce the feature-space of a dataset that contained information about housing prices, features such as number of rooms, number of bedrooms, number of floors, and number of bathrooms all might be indicators of the *size* of the house. One might argue that these latent signals are the **principal components** which make up our dataset. ### Intuition Let's look at the following two-dimensional feature-space. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-06-at-6.08.06-PM.png) It appears that most of the points on this scatterplot lie along the following line. This suggests some degree of correlation between $x \_1$ and $x \_2$. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-06-at-6.08.13-PM.png) As such, we could reorient the axes to be centered on the data and parallel to the line above. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-06-at-7.35.08-PM.png) Strictly speaking, we're still dealing with two dimensions. However, let's project each observation onto the primary axis. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-06-at-7.35.21-PM.png) Now, every observation lies on the primary axis. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-06-at-7.35.33-PM.png) We just compressed a two dimensional dataset into one dimension by translating and rotating our axes! After this transformation, we only really have one relevant dimension and thus we can discard the second axis. Boom. Dimensions reduced. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-06-at-6.08.45-PM.png) Comparing the original observations with our new projections, we can see that it's not an exact representation of our data. However, one could argue it does capture the essence of our data - not exact, but enough to be meaningful. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-06-at-6.09.00-PM.png) We can view the cumulative distance between our observations and projected points as a measure of information loss. Thus we'd like to orient the axes in a manner which minimizes this. How do we do this? That's where **variance** comes into play. Let's take another look at the data. Specifically, look at the spread of the data along the green and orange directions. Notice that there's a much more deviation along the green direction than there is along the orange direction. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-06-at-8.25.51-PM.png) Projecting our observations onto the orange vector requires us to move much further than we would need to for projecting onto the green vector. It turns out, the vector which is capable of capturing the maximum variance of the data minimizes the distance required to move our observations as projection onto the vector. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-06-at-8.26.01-PM.png) *Note: While this might look like [linear regression](https://www.jeremyjordan.me/linear-regression/), it is distinctly different. For one, linear regression measures error as the difference between the true output and the predicted output for a given input. This error is parallel to the y-axis. For PCA, there is no y-axis; we're doing unsupervised learning. Further, we measure error as the distance between the original observation and the projected observation along the direction of projection (orthogonal to the new axis).* More generally, we'd like to find a set of $k$ vectors on which we can project our data to reduce the feature-space from $n$-dimensions to $k$-dimensions. Further, we'd like to find these $k$ vectors such that we minimize the projection error or information loss; we earlier established that we can accomplish this by finding the direction of maximum variance within the data. ### Algorithm So far, we've established a method for reducing the dimensionality by finding the set of orthogonal vectors (through translation and rotation) which captures the maximum amount of variance within the data. Now, we'll discuss a way to achieve this algorithmically. This algorithm must contain a way to 1) find a new set of vectors to describe the data and 2) develop a way to project our original observations onto these vectors. Before we transform our feature-space into such a representation, it's important that we normalize and scale the features. This allows variance to be a homogenous measure across all features. The steps to perform PCA are as follows. **1\. Compute the covariance matrix.** Σ\=\[ cov( x 1 , x 1 ) cov( x 1 , x 2 ) ⋯ cov( x 1 , x n ) cov( x 2 , x 1 ) cov( x 2 , x 2 ) ⋯ cov( x 2 , x n ) ⋮ ⋮ ⋱ ⋮ cov( x n , x 1 ) cov( x n , x 2 ) ⋯ cov( x n , x n ) \] **2\. Find eigenvectors ($U$) and eigenvalues ($S$) of the covariance matrix using singular value decomposition.** $$ {\\left\[ {U,S,V} \\right\]} = svd(\\Sigma )$$ **3\. Select $k$ first columns from eigenvector matrix.** U\=\[ | | | | | | u 1 u 2 ⋯ u k ⋯ u n | | | | | | \] U reduce \=\[ | | | | u 1 u 2 ⋯ u k | | | | \] **4\. Compute projections of original observation onto new vector form.** $$z = U \_{reduce}^Tx$$ It's also possible to decompress $z$ and restore an approximation of the original feature-space. $${x \_{approx}} = {U \_{reduce}}z$$ We can compare this approximation to the original value in order to calculate the variance retained in our compressed feature-space. A common metric to evaluate our PCA feature compression is **explained variance**, which is another way of saying the amount of original variance by which our compressed representation is still able to retain. (Recall that we lost some variance of the data when projecting the observations onto the new vector in the 2D example.) For example, if we were to compress our feature-space in a manner that retains 95% variance of the data, we could write: $$1 - \\frac{{\\frac{1}{m}\\sum\\limits \_{i = 1}^m {{{\\left| {{x \_i} - {x \_{i,approx}}} \\right|}^2}} }}{{\\frac{1}{m}\\sum\\limits \_{i = 1}^m {{{\\left| {{x \_i}} \\right|}^2}} }} \\ge 0.95$$ ###### Choosing k One method for choosing $k$ would be to start at $k=1$, and continue increasing $k$ until we drop below some set threshold of explained variance. This would involve calculating the explained variance for each iteration and is computationally inefficient. A better approach to calculating $k$ is to leverage the eigenvalues ($S$) we found in singular value decomposition. While the eigenvectors represent the directions of maximum variance, the eigenvalues represent the magnitudes of variance in each corresponding direction. S\=\[ s 11 0 0 0 0 s 22 0 0 0 0 ⋱ 0 0 0 0 s nn \] This is quite convenient and it is much easier to use these magnitudes in calculating the explained variance for a given $k$. $$1 - \\frac{{\\frac{1}{m}\\sum\\limits \_{i = 1}^m {{{\\left| {{x \_i} - {x \_{i,approx}}} \\right|}^2}} }}{{\\frac{1}{m}\\sum\\limits \_{i = 1}^m {{{\\left| {{x \_i}} \\right|}^2}} }} = \\frac{{\\sum\\limits \_{i = 1}^k {{s \_{ii}}} }}{{\\sum\\limits \_{i = 1}^n {{s \_{ii}}} }}$$ ### Further reading - [Principal component analysis with linear algebra](http://www.math.union.edu/~jaureguj/PCA.pdf?ref=jeremyjordan.me) - [Interpretation of the Principal Components](https://onlinecourses.science.psu.edu/stat505/node/54/?ref=jeremyjordan.me) - [Linear algebra - Alternate coordinate systems](https://www.khanacademy.org/math/linear-algebra/alternate-bases/change-of-basis/v/linear-algebra-coordinates-with-respect-to-a-basis?ref=jeremyjordan.me) - [Linear algebra - Eigenvectors](https://www.khanacademy.org/math/linear-algebra/alternate-bases/eigen-everything/v/linear-algebra-introduction-to-eigenvalues-and-eigenvectors?ref=jeremyjordan.me) - [Covariance matrix](http://www.visiondummy.com/2014/04/geometric-interpretation-covariance-matrix/?ref=jeremyjordan.me) - [Singular Value Decomposition](http://andrew.gibiansky.com/blog/mathematics/cool-linear-algebra-singular-value-decomposition/?ref=jeremyjordan.me) - [More slides on PCA](http://www.cse.psu.edu/~rtc12/CSE586Spring2010/lectures/pcaLectureShort%5F6pp.pdf?ref=jeremyjordan.me) - [Making sense of principal component analysis, eigenvectors & eigenvalues](https://stats.stackexchange.com/questions/2691/making-sense-of-principal-component-analysis-eigenvectors-eigenvalues?ref=jeremyjordan.me) ### Neural networks: activation functions. URL: https://www.jeremyjordan.me/neural-networks-activation-functions/ Last updated: 2019-11-02T19:26:13.000Z Activation functions are used to determine the firing of neurons in a neural network. Given a linear combination of inputs and weights from the previous layer, the activation function controls how we'll pass that information on to the next layer. An ideal activation function is both nonlinear and differentiable. The nonlinear behavior of an activation function allows our neural network to learn nonlinear relationships in the data. Differentiability is important because it allows us to [backpropagate](https://www.jeremyjordan.me/neural-networks-training/) the model's error when training to optimize the weights. *Note: I'll continue to update this page as I get more practical experience with the activation functions.* ### Perceptron While this is the original activation first developed when neural networks were invented, it is no longer used in neural network architectures because it's incompatible with backpropagation. Backpropagation allows us to find the optimal weights for our model using a version of gradient descent; unfortunately, the derivative of a perceptron activation function cannot be used to update the weights (since it is 0). Function: f( x )\={ 0 for x<0 1 for x≥0 } Derivative: f ′ ( x )\={ 0 for x≠0 ? for x\=0 } ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/perc.png) Problems: not compatible gradient descent via backpropagation. ### Sigmoid (logistic) The sigmoid function is commonly used when teaching neural networks, however, it has fallen out of practice to use this activation function in real-world neural networks due to a problem known as the **vanishing gradient**. Recall that we included the derivative of the activation function in calculating the "error" term for each layer in the [backpropagation](https://www.jeremyjordan.me/neural-networks-training/) algorithm. The maximum value of the derivative for the sigmoid function is 0.25, thus, and you progress backwards each layer in backpropagation you're reducing the size of your "error" by at least 75% at each layer. This ends up limiting our ability to change the weights in layers close to the input layer for deep networks because so many of terms multiplied together in the derivative chain are less than or equal to 0.25. Function: $$f\\left( x \\right) = \\frac{1}{{1 + {e^{ - x}}}}$$ Derivative: $$f'\\left( x \\right) = f\\left( x \\right)\\left( {1 - f\\left( x \\right)} \\right)$$ ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/sig.png) Problems: vanishing gradient at edges, output isn't zero centered. ### Hyperbolic Tangent Function: $$f\\left( x \\right) = \\tanh \\left( x \\right) = \\frac{{{e^x} - {e^{-x}}}}{{{e^x} + {e^{-x}}}}$$ Derivative: $$f'\\left( x \\right) = 1 - f{\\left( x \\right)^2}$$ ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/tanh.png) Problems: vanishing gradient at edges. ### Inverse Tangent Function: $$f\\left( x \\right) = {\\tan ^{ - 1}}\\left( x \\right)$$ Derivative: $$f'\\left( x \\right) = \\frac{1}{{{x^2} + 1}}$$ ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/arctan.png) ### ReLU (rectified linear unit) This is one of the most popularly used activation functions of 2017\. Due to its popularity, a number of variants have been proposed that provide an incremental benefit over standard ReLUs. Function: f( x )\={ 0 for x<0 x for x≥0 } or (another way to write the ReLU function is...) f( x )\=max( x,0 ) Derivative: f ′ ( x )\={ 0 for x<0 1 for x≥0 } ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/relu.png) ### Leaky ReLU Function: f( x )\={ 0.1x for x<0 x for x≥0 } Derivative: f ′ ( x )\={ 0.1 for x<0 1 for x≥0 } ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/leaky-relu.png) ### SeLU (Scaled-exponential linear units) The SeLU activation function was [proposed](https://arxiv.org/pdf/1706.02515.pdf?ref=jeremyjordan.me) as a method to create self-normalizing networks. Function: f( α,x )\=λ{ α( e x −1 ) for x<0 x for x≥0 } Derivative: f ′ ( α,x )\=λ{ f( α,x )+α for x<0 1 for x≥0 } ### Maxout Function: $$f\\left( {\\vec x} \\right) = \\mathop {\\max }\\limits\_i {x\_i}$$ Derivative: ∂f ∂ x j \={ 1 for j\= argmax i x i 0 for j≠ argmax i x i } --- The following activation functions should only be used on the output layer. ### Identity Function: $$f\\left( x \\right) = x$$ Derivative: $$f'\\left( x \\right) = 1$$ ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/ident.png) Use: regression. ### Softmax The softmax function is commonly used as the output activation function for multi-class classification because it scales the preceding inputs from a range between 0 and 1 and normalizes the output layer so that the sum of all output neurons is equal to one. As a result, we can consider the softmax function as a categorical probability distribution. This allows you to communicate a degree of confidence in your class predictions. *Note: we use the exponential function to ensure all values in the summation are positive.* Function: $${f\_i}\\left( {\\vec x} \\right) = \\frac{{{e^{{x\_i}}}}}{{\\sum\\nolimits\_{j = 1}^J {{e^{{x\_j}}}} }} {\\rm{\\hspace{2mm} for \\hspace{2mm} i = 1,..,J}}$$ Derivative: $$\\frac{{\\partial {f\_i}\\left( {\\vec x} \\right)}}{{\\partial {x\_j}}} = {f\_i}\\left( {\\vec x} \\right)\\left( {{\\delta\_{ij}} - {f\_i}\\left( {\\vec x} \\right)} \\right)$$ Use: classification. ### Further Reading - [Stanford CS231: Activation functions](http://cs231n.github.io/neural-networks-1/?ref=jeremyjordan.me) - [The vanishing gradient problem](https://ayearofai.com/rohan-4-the-vanishing-gradient-problem-ec68f76ffb9b?ref=jeremyjordan.me) - [Which Activation Function Should I Use? - Siraj Raval](https://www.youtube.com/watch?v=-7scQpJT7uo&ref=jeremyjordan.me) - [More on rectifying functions](https://en.wikipedia.org/wiki/Rectifier%5F%28neural%5Fnetworks%29?ref=jeremyjordan.me) ### Feature selection for a machine learning model. URL: https://www.jeremyjordan.me/feature-selection/ Last updated: 2018-08-10T02:51:00.000Z Feature selection can be an important part of the machine learning process as it has the ability to greatly improve the performance of our models. While it might seem intuitive to provide your model with all of the information you have with the thinking that the more data you provide the better it can learn and generalize, it's important to also recognize that the data necessary to train a model grows exponentially with the number of features we use to train the model. This is commonly referred to as the **curse of dimensionality**. There used to be a [great interactive demonstration](https://simplystatistics.org/2014/10/24/an-interactive-visualization-to-teach-about-the-curse-of-dimensionality/?ref=jeremyjordan.me) of this concept, but it looks like the demo is no longer being hosted. Maybe at some point I'll code one up in d3.js and share it here; until then, check out [this video](https://www.youtube.com/watch?v=OyPcbeiwps8&t=13s&ref=jeremyjordan.me) if you're not familiar with the concept. Ostensibly, some of the features in our dataset may be more important than others - some features might even be entirely irrelevant. But how do we go about systematically choosing the best features to use in our model? There's two general ways one can approach the problem of selecting features. The first approach, known as **filtering**, will look at your feature space and select out the best features before we actually feed the data into our model such that the data we train on is a reduced size from our original dataset. We'll define some metric, such as information gain, to evaluate the features. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-03-at-8.53.13-PM.png) The other approach, known as **wrapping**, is to simply try out a number of reduced feature-spaces on our model and see what performs the best. In this approach, we'll use all of the features in our search for the optimal selection, combined with performance metrics from the model evaluation to select the best features. While this approach is much slower, we get to actually use the model to aid our feature selection and the result is typically better than filtering. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-03-at-8.55.40-PM.png) Ultimately, feature selection is a balance between reducing the amount of data needed to train a model without sacrificing the learning ability of a given model. If you reduce the feature-space too much, it makes it harder to learn, but if you don't reduce the feature-space at all, you may need an absurd amount of data to properly train the model. ### Filtering examples One simple filtering approach is to look at the **variance** across values for each feature. Features which have very little variance (in other words, features which are largely the same for most observations) will not prove to be helpful during learning, so we can simply ignore the features with a variance below some specified threshold. ``` from sklearn.feature_selection import VarianceThreshold selector = VarianceThreshold(threshold= 0.2) X_reduced = selector.fit_transform(X) ``` Another filtering approach is to train the dataset on a simple model, such as a decision tree, and then use the ranked feature importances to select the features you'd like to use in your desired machine learning model. You could use this technique to reduce the feature-space before training a more complex model, such as a neural network, which would take longer to train. ``` from sklearn.ensemble import ExtraTreesClassifier from sklearn.feature_selection import SelectFromModel simple_model = ExtraTreesClassifier() simple_model = simple_model.fit(X, y) selector = SelectFromModel(simple_model, prefit=True) X_reduced = selector.transform(X) ``` *Note: ExtraTreesClassifier is a randomized decision tree classifier which samples a random subset of the feature-space when deciding where to make the next split.* Lastly, you could look at each feature individually and perform statistical tests to gauge its relative importance in predicting the output. Sklearn provides selectors to do this by keeping the top $k$ features (where $k$ is defined by the user) with `SelectKBest` or keeping the top $x%$ of features (where $x$ is defined by the user) with `SelectPercentile`. ### Wrapping examples When using a wrapping technique, we'll actually use a performance metric from the desired model to aid our feature selection; for example, we could look at the error of a validation set. There's essentially two places you can start, with all of the features present or none of them. In a **forward search**, we'll essentially start with nothing and, one by one, add the feature which yields the greatest drop in validation error. We'd continue this process until the incremental gain in model performance drops below a certain threshold. In a **backward search**, we'll start with all of the features present in our model and, one by one, eliminate the feature which provides the least amount of detriment to our model performance. We'd continue this process until the detriment to model performance crosses above a certain threshold. It's also possible to perform a bidirectional search, using a combination of forward and backward search which can take into account the effect of combinations of features on model performance. We can perform a backward search in sklearn using the [recursive feature elimination](http://scikit-learn.org/stable/modules/generated/sklearn.feature%5Fselection.RFE.html?ref=jeremyjordan.me#sklearn.feature%5Fselection.RFE) method. Given a model that assigns weights to features (whether by virtue of coefficients in a linear regression model or feature importances in a decision tree model), we can iteratively prune features with low weights based on the assumption that the weights are a proxy for feature importance. This will continue until you've reached a specified number of features; alternatively, you could combine this technique with cross-validation using [RFECV](http://scikit-learn.org/stable/modules/generated/sklearn.feature%5Fselection.RFECV.html?ref=jeremyjordan.me#sklearn.feature%5Fselection.RFECV) to automatically determine the ideal number of features. *Note: we could also leverage recursive feature elimination on a simple model as a filtering technique to reduce our feature-space before training on a more complex model.* ## Further reading [Comparing Two Forward Feature Selection Algorithms](https://medium.com/square-corner-blog/comparing-two-forward-feature-selection-algorithms-c52f42868f55?ref=jeremyjordan.me) ### Soft clustering with Gaussian mixed models (EM). URL: https://www.jeremyjordan.me/gaussian-mixed-models/ Last updated: 2017-10-18T04:30:45.000Z Sometimes when we're performing clustering on a dataset, there exist points which don't belong strongly to any given cluster. If we were to use something like [k-means clustering](https://www.jeremyjordan.me/grouping-data-points-with-k-means-clustering/), we're forced to make a decision as to which group an observation belongs to. However, it would be useful to be able to quantify the likelihood of a given observation belonging to any of the clusters present in our dataset. We would then be able to see some observations are a close tie between belonging to two clusters. For example, examine the scatterplot showing two clusters (in blue) and some fringe observations (in red) that may belong to either of the two clusters. Ideally, we'd like to be as true to the data as possible when assigning observations to clusters; allowing partial assignment to multiple clusters allows us to more accurately describe the data. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/download.png) This is accomplished by **soft clustering** of the data. Formally, soft clustering (also known as fuzzy clustering) is a form clustering where observations may belong to multiple clusters. Today, I'll be writing about a soft clustering technique known as **expectation maximization** (EM) of a **Gaussian mixture model**. Essentially, the process goes as follows: 1. Identify the number of clusters you'd like to split the dataset into. 2. Define each cluster by generating a Gaussian model. 3. For every observation, calculate the probability that it belongs to each cluster (ex. Observation 23 has a 21% chance that it belongs to Cluster A, a 0.1% chance that it belongs to Cluster B, a 48% chance of Cluster C, ... and so forth). 4. Using the above probabilities, recalculate the Gaussian models. 5. Repeat until observations more or less "converge" on their assignments. Let's look at a very simple example. Suppose I have data for a set of observations with one feature, $x \_1$, that come from two distinct classes. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-01-at-6.55.05-PM.png) We can use this data to build a [Gaussian model](https://en.wikipedia.org/wiki/Normal%5Fdistribution?ref=jeremyjordan.me) for each class which would allow us to calculate the probability of a new observation belonging to each class. The class denoted by black x's would be used to build one Gaussian model and the class denoted by blue o's would be used to build a separate Gaussian model. A Gaussian model may be defined by calculating the **mean** and **variance** of a dataset; we'll do this twice, once for each class. *For those needing a refresher in statistics, we'll use the *probability distribution function* to quantify the probability of an observation belongs to a certain cluster.* ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-01-at-6.55.15-PM.png) Unfortunately, we usually perform clustering because we don't have a labeled dataset and thus don't know which class any of the observations belong to- that's what we're hoping to learn! ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-01-at-6.55.25-PM.png) Since we don't know which class each observation belongs to, we don't have an easy way to build multiple Gaussian models to partition the data. We can no longer simply calculate the mean and variance of observations belonging to each class. Let's start with a random guess of our Gaussian models and then iteratively optimize the attributes, in a similar fashion as we did for [k-mean clustering](https://www.jeremyjordan.me/grouping-data-points-with-k-means-clustering/), to find the optimal Gaussian model to express the data. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-03-at-12.09.02-PM.png) After initializing two random Gaussians, we'll compute the likelihood of each observation being expressed in both of the Gaussian models. We'll then recalculate the Gaussian models; we'll use all of the observations in calculating the mean and variance for each Gaussian, but the observations will be weighted by the likelihood of existing in the given model. As I mentioned earlier, rather than assigning observations to a class, we'll assume there's a possibility that it can belong to any of the $k$ clusters. Thus, for any given observation we'll calculate the probability of it belonging to each of the $k$ clusters. Let's attach a hidden variable to each observation which contains a vector, $z\_ i$, to store these probabilities. z i \=\[ P( x i belongs to Cluster 1 ) P( x i belongs to Cluster 2 ) ⋮ P( x i belongs to Cluster k ) \] After using $z\_i$ to calculate the weighted parameters (mean and variance) for each cluster, we update the Gaussian models. You'll see that the leftmost data (circled below) will pull both models in it's direction, but it will have a larger effect on the left Gaussian since the observations will have a higher weight for the model that it's closest to. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-03-at-12.09.17-PM.png) We'll continue this cycle of recalculating $z\_i$ and then using it to update the Gaussians until we "converge" on optimal clusters. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/07/Screen-Shot-2017-07-03-at-12.09.52-PM.png) The example above visualizes a univariate Gaussian example, we can extend this logic, however, to accomodate multivariate datasets. If you're not familiar with multivariate Gaussian distributions, I recommend you spend a little time reading about them or watching a lecture. I've found [this video](https://www.youtube.com/watch?v=TC0ZAX3DA88&ref=jeremyjordan.me) and [this video](https://www.youtube.com/watch?v=eho8xH3E6mE&ref=jeremyjordan.me) to be useful. There are subtle difference between univariate and multivariate models, namely that we now have a vector of means and a covariance matrix rather than single values for mean and variance. Now that we understand what's going on, let's dive into the specifics of the algorithm. ### Initializing the clusters Foremost, we must select how many segments we'd like to partition our data into. To do this, we can use the Bayesian information criterion (BIC). This criterion is commonly used for model selection with preference for the model possessing the lowest BIC. The [sklearn documentation](http://scikit-learn.org/stable/auto%5Fexamples/mixture/plot%5Fgmm%5Fselection.html?ref=jeremyjordan.me#sphx-glr-auto-examples-mixture-plot-gmm-selection-py) has a good example on this process. *Note: You can calculate the BIC of a given `GaussianMixture` in sklearn using the `bic()` method.* After selecting $k$ segments to partition the data into, we initialize $k$ random Gaussian models. ### Probabalistic assignment to clusters (expectation) After initializing $k$ random Gaussian models we can calculate our **expectation** of $z \_i$, a vector of probabilities that $x \_i$ belongs to the $j ^{th}$ cluster for $j=1$ to $j=k.$ As mentioned earlier, we don't know the true probabilitistic cluster assignments for $z$, so we'll start with a guess and iteratively refine it. $$ E\\left\[ {z \_{i,j}} \\right\] = \\frac{P\\left( {x = {x \_i}|\\mu = {\\mu \_j}} \\right)}{\\sum\\limits \_{m=1}^k {P\\left( {x = {x \_i}|\\mu = {\\mu \_m}} \\right)} } $$ We can calculate the probability of $x \_i$ belonging to cluster $j$ using the probability distribution function of a Gaussian distribution. \[P\\left( {x = {x \_i}|\\mu = {\\mu \_j}} \\right) = \\frac{1}{{\\sqrt {2\\pi {\\sigma \_j}^2} }}{e^{ - \\frac{{^{{{\\left( {{x \_i} - {\\mu \_j}} \\right)}^2}}}}{{2{\\sigma \_j}^2}}}}\] ### Reformulating the Gaussian models (maximization) We'll then recalculate our Gaussian models leveraging the weights we found in the expectation step. The expectation, ${E\\left\[ {{z \_{i,j}}} \\right\]}$, represents the likelihood that the $i ^{th}$ observation belongs to cluster $j$. \[{\\mu \_j} = \\frac{{\\sum\\limits \_i {E\\left\[ {{z \_{i,j}}} \\right\]{x \_i}} }}{{\\sum\\limits \_i {E\\left\[ {{z \_{i,j}}} \\right\]} }}\] \[{\\sigma \_j} = \\frac{{\\sum\\limits \_i {E\\left\[ {{z \_{i,j}}} \\right\]\\left( {{x \_i} - {\\mu \_j}} \\right)\\left( {{x \_i} - {\\mu \_k}} \\right)} }}{{\\sum\\limits \_i {E\\left\[ {{z \_{i,j}}} \\right\]} }}\] ### Defining a stopping criterion With k-means clustering, we iteratively recalculated means and reassigned observations until convergence and observations stopped moving between clusters. However, because we're now dealing with continuous probabilities, and because we never give a hard cluster assignment, we can't rely on this convergence. Instead, we'll set a stopping criterion to end the iterative cycle once the observation probabilities stop changing by above some threshold. ### Further reading - [Victor Lavrenko: Gaussian mixture models](https://www.youtube.com/watch?v=REypj2sy%5F5U&list=PLBv09BD7ez%5F4e9LtmK626Evn1ion6ynrt&ref=jeremyjordan.me) - [Ng CS229 Lecture notes: Mixtures of Gaussians and the EM algorithm](http://cs229.stanford.edu/notes/cs229-notes7b.pdf?ref=jeremyjordan.me) - [Ng CS229 Lecture notes: The EM algorithm in depth](http://cs229.stanford.edu/notes/cs229-notes8.pdf?ref=jeremyjordan.me) - [The EM Algorithm for Gaussian Mixtures](http://www.ics.uci.edu/~smyth/courses/cs274/notes/EMnotes.pdf?ref=jeremyjordan.me) ### Neural networks: representation. URL: https://www.jeremyjordan.me/intro-to-neural-networks/ Last updated: 2018-01-26T03:45:23.000Z This post aims to discuss what a neural network is and how we represent it in a machine learning model. Subsequent posts will cover more advanced topics such as training and optimizing a model, but I've found it's helpful to first have a solid understanding of what it is we're actually building and a comfort with respect to the matrix representation we'll use. **Prerequisites:** - Read my post on [logistic regression](https://www.jeremyjordan.me/logistic-regression/). - Be comfortable multiplying matrices together. ## Inspiration Neural networks are a biologically-inspired algorithm that attempt to mimic the functions of neurons in the brain. Each neuron acts as a computational unit, accepting input from the dendrites and outputting signal through the axon terminals. Actions are triggered when a specific combination of neurons are activated. ![Neuron_Hand-tuned](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/01/Neuron_Hand-tuned.png) In essence, the cell acts a **function** in which we provide input (via the dendrites) and the cell churns out an output (via the axon terminals). The whole idea behind neural networks is finding a way to 1) represent this function and 2) connect neurons together in a useful way. I found the following two graphics in a [lecture on neural networks by Andrea Palazzi](https://github.com/ndrplz/deep%5Flearning%5Flectures/blob/master/2017%5FMASTER/slides/main%5Fdnn.pdf?ref=jeremyjordan.me) that quite nicely compared biological neurons with our computational model of neurons. ![biological_neuron-1](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/01/biological_neuron-1.jpg) ![single_neuron](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/01/single_neuron.jpg) To learn more about how neurons are connected and operate together in the brain, check out this video. ## A computational model of a neuron Have you read my post on [logistic regression](https://www.jeremyjordan.me/logistic-regression/) yet? If not, go do that now; I'll wait. In logistic regression, we composed a linear model ${z\\left( x \\right)}$ with the logistic function ${g\\left( z \\right)}$ to form our predictor. This linear model was a combination of feature inputs $x\_i$ and weights $w\_i$. $$ z\\left( x \\right) = {w\_1}{x\_1} + {w\_2}{x\_2} + {w\_3}{x\_3} + {w\_4}{x\_4} + b = {w^{\\rm{T}}}x + b $$ Let's try to visualize that. ![Screen-Shot-2017-11-07-at-10.35.52-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/01/Screen-Shot-2017-11-07-at-10.35.52-AM.png) The first layer contains a node for each value in our input feature vector. These values are scaled by their corresponding weight, $w\_i$, and added together along with a bias term, $b$. The bias term allows us to build linear models that aren't fixed at the origin. The following image provides an example of why this is important. Notice how we can provide a much better decision boundary for logistic regression when our linear model isn't fixed at the origin. ![Screen-Shot-2017-06-29-at-5.34.44-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/01/Screen-Shot-2017-06-29-at-5.34.44-PM.png) The input nodes in our network visualization are all connected to a single output node, which consists of a linear combination of all of the inputs. Each connection between nodes contains a parameter, $w$, which is what we'll tune to form an optimal model (tuning these parameters will be covered in a later post). The final output is functional composition, $g\\left( {z\\left( x \\right)} \\right)$. When we pass the linear combination of inputs through the logistic (also known as sigmoid) function, the neural network community refers to this as **activation**. Namely, the sigmoid is an activation function which controls whether or not the end node "neuron" will fire. As you'll see later, there's a whole family of possible activation functions that we can use. #### Comparison to a perceptron unit Most tutorials will introduce the concept of a neural network with the perceptron, but I've found it's easier to introduce the concept of neural networks by latching onto something familiar (logistic regression). However, for the sake of completeness I'll go ahead and introduce the perceptron unit and note its similarities to the network representation of logistic regression. The perceptron is the simplest neural unit that we can build. It takes a series of inputs, $x\_i$, combined with a series of weights, $w\_i$, which are compared against a threshold value, $\\theta$. If the linear combination of inputs and weights is higher than the threshold, the neuron fires, and if the combination is less than the threshold it doesn't fire. ![Screen-Shot-2017-06-29-at-5.09.57-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/01/Screen-Shot-2017-06-29-at-5.09.57-PM.png) We can rewrite the perceptron function by moving the threshold to the left side and we end up with the same linear model used in logistic regression. The weights, $w\_i$ in the perceptron algorithm are synonomous with the weights in logistic regression and the threshold value, $\\theta$, in the perceptron algorithm is synonomous with bias $b$. f w ( x ) \= { ∑ w i x i − θ ≥ 0 → neuron fires ∑ w i x i − θ < 0 → neuron doesn't fire } At a high level, they're practically identical - the main difference being the [activation function](https://www.jeremyjordan.me/neural-networks-activation-functions/), $g\\left( z \\right)$, used to control neuron firing. The perceptron activation is a step-function from 0 (when the neuron doesn't fire) to 1 (when the neuron fires) while the logistic regression model has a smoother activation function with values ranging from 0 to 1. ## Building a network of neurons The previous model is only capable of binary classification; however, recall that we can perform multi-class classification by building a collection of logistic regression models. Let's extend our "network" to represent this. ![Screen-Shot-2017-11-07-at-12.32.19-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/01/Screen-Shot-2017-11-07-at-12.32.19-PM.png) *Note: While I didn't explicitly show the activation function here, we still use it on each linear combination of inputs. I mainly just wanted to show the connection between the visual representation and matrix form.* Here, we've built three distinct logistic regression models, each with their own set of parameters. Take a moment to make sure you understand this matrix representation. (This is why matrix multiplication is listed as a prerequisite.) It's rather convenient that we can leverage matrix operations as it allows us to perform these calculations quickly and efficiently. The above example displays the case for multi-class classification on a single example, but we can also extend our input matrix to classify a collection of examples. This is not simply useful, but necessary for our optimization algorithm (in a later post) to learn from all of the examples in an efficient manner when finding the best parameters (more commonly referred to as weights in the neural network community). ![Screen-Shot-2017-11-07-at-12.53.07-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/01/Screen-Shot-2017-11-07-at-12.53.07-PM.png) Again, go through the matrix multiplications to convince yourself of this. Although I color coded the weights here for clarity, we'll need to develop a more systematic notation. Notice how the first output neuron uses all of the blue weights, the second output neuron uses all of the green weights, and the third output neuron uses all of the orange weights. Moving forward, we'll describe our weights more succinctly as a vector $w\_i$ where the subscript $i$ now represents the neuron which uses that set of weights. ![Screen-Shot-2017-11-07-at-12.52.23-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/01/Screen-Shot-2017-11-07-at-12.52.23-PM.png) Thus, we can define a weight matrix, $W$, for a layer. Our bias may similarly be described as a vector $b$. ![Screen-Shot-2017-11-07-at-1.03.53-PM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/01/Screen-Shot-2017-11-07-at-1.03.53-PM.png) ## Hidden layers Up until now, we've been dealing solely with one-layer networks; we feed information into the input layer and observe the results in the output layer. (The input layer often isn't counted as a layer in the neural network.) The real power of neural networks emerges as we add additional layers to the network. Any layer that is between the input and output layers is known as a **hidden layer**. Thus, the following example is a neural network with an input layer, one hidden layer, and an output layer. ![Screen-Shot-2017-11-07-at-11.37.22-AM](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2018/01/Screen-Shot-2017-11-07-at-11.37.22-AM.png) I'll use the superscript $\\left\[ l \\right\]$ to refer to the ${l^{th}}$ layer of the network and the subscript $i$ to refer to the ${i^{th}}$ neuron in a layer. For example, $a\_2^{\\left\[ 1 \\right\]}$ represents the activation of the second neuron in the first hidden layer. We can calculate this value by first combining the proper weights and bias with the previous layer's values $$ z\_2^{\\left\[ 1 \\right\]} = w\_2^{\\left\[ 1 \\right\]{\\rm{T}}}{a^{\\left\[ 0 \\right\]}} + {b\_2} $$ and then passing this through our activation function, $g\\left( z \\right)$. Notice how each neuron combines *every* value from the previous layer as input. *Note: Our input vector, $x$, can also be referred to as the activations of the $0^{th}$ layer.* More generally, we can calculate the activation of neuron $i$ in layer $l$. $$ z\_i^{\\left\[ l \\right\]} = w\_i^{\\left\[ l \\right\]{\\rm{T}}}{a^{\\left\[ {l - 1} \\right\]}} + b\_i^{\\left\[ l \\right\]} $$ $$ a\_i^{\\left\[ l \\right\]} = g\\left( {z\_i^{\\left\[ l \\right\]}} \\right) $$ Similarly, we can calculate all of the activations for a given layer $l$ by using our weight matrix ${W^{\\left\[ l \\right\]}}$. $$ {Z^{\\left\[ l \\right\]}} = {W^{\\left\[ l \\right\]}}{A^{\\left\[ {l - 1} \\right\]}} + {b^{\\left\[ l \\right\]}} $$ $$ {A^{\\left\[ l \\right\]}} = g\\left( {{Z^{\\left\[ l \\right\]}}} \\right) $$ In a network, we take the output from one layer and feed it in as the input to our next layer. We can stack as many layers as we want on top of each other. The field of deep learning studies neural network architectures with **many** hidden layers. ## Matrix representation Let ${n^{\\left\[ l \\right\]}}$ represent the number of units in layer $l$. For a given layer, we'll have a weights matrix ${W^{\\left\[ l \\right\]}}$ of shape $\\left( {{n^{\\left\[ l \\right\]}},{n^{\\left\[ {l - 1} \\right\]}}} \\right)$ and a bias vector of shape $\\left( {{n^{\\left\[ l \\right\]}},1} \\right)$. The activations of a given layer will be a matrix of shape $\\left( {{n^{\\left\[ l \\right\]}},m} \\right)$ where $m$ represents the number of observations being fed through the network. Recall the earlier section where I demonstrated calculating the neural network output of multiple observations using an efficient matrix representation. When I was first learning about neural networks, the trickiest part for me was figuring out what my matrix dimensions needed to be and how to manipulate them to get them into the proper form. I'd recommend doing a couple practice problems to get more comfortable before we continue to talk about training a neural network in my next post. Feeling like you've got a grasp? Check out this [neural network cheat sheet of common architectures](http://www.asimovinstitute.org/neural-network-zoo/?ref=jeremyjordan.me). ### Support vector machines. URL: https://www.jeremyjordan.me/support-vector-machines/ Last updated: 2017-10-18T03:59:37.000Z Today we'll be talking about support vector machines (SVM); this classifier works well in complicated feature domains, albeit requiring clear separation between classes. SVMs don't work well with noisy data, and the algorithm scales roughly cubic $O(n^3)$ to input depending on your implementation (sklearn's SVM [fit time complexity is more than quadratic](http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html?ref=jeremyjordan.me), so it won't be able to train quickly on large datasets (>10,000 examples). ### Intuition Before we get into any more details, I'd like to introduce a description of SVMs I saw on [Reddit](https://www.reddit.com/r/MachineLearning/comments/15zrpp/please%5Fexplain%5Fsupport%5Fvector%5Fmachines%5Fsvm%5Flike%5Fi/?ref=jeremyjordan.me) in an attempt to explain what it is to a 5-year-old. > We have 2 colors of balls on the table that we want to separate. > ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/01/zDBbD.png) > [http://i.imgur.com/zDBbD.png](http://i.imgur.com/zDBbD.png?ref=jeremyjordan.me) > We get a stick and put it on the table, this works pretty well right? > ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/01/aLZlG.png) > [http://i.imgur.com/aLZlG.png](http://i.imgur.com/aLZlG.png?ref=jeremyjordan.me) > Some villain comes and places more balls on the table, it kind of works but one of the balls is on the wrong side and there is probably a better place to put the stick now. > ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/01/kxWgh.png) > [http://i.imgur.com/kxWgh.png](http://i.imgur.com/kxWgh.png?ref=jeremyjordan.me) > SVMs try to put the stick in the best possible place by having as big a gap on either side of the stick as possible. > ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/01/ePy4V.png) > [http://i.imgur.com/ePy4V.png](http://i.imgur.com/ePy4V.png?ref=jeremyjordan.me) > Now when the villain returns the stick is still in a pretty good spot. > ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/01/BWYYZ.png) > [http://i.imgur.com/BWYYZ.png](http://i.imgur.com/BWYYZ.png?ref=jeremyjordan.me) > There is another trick in the SVM toolbox that is even more important. Say the villain has seen how good you are with a stick so he gives you a new challenge. > ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/01/R9967.png) > [http://i.imgur.com/R9967.png](http://i.imgur.com/R9967.png?ref=jeremyjordan.me) > There’s no stick in the world that will let you split those balls well, so what do you do? You flip the table of course! Throwing the balls into the air. Then, with your pro ninja skills, you grab a sheet of paper and slip it between the balls. > ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/01/WuxyO.png) > [http://i.imgur.com/WuxyO.png](http://i.imgur.com/WuxyO.png?ref=jeremyjordan.me) > Now, looking at the balls from where the villain is standing, they balls will look split by some curvy line. > ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/01/gWdPX.png) > [http://i.imgur.com/gWdPX.png](http://i.imgur.com/gWdPX.png?ref=jeremyjordan.me) > Boring adults call the balls **data**, the stick is a **classifier**, the biggest gap trick is called **optimization**, flipping the table is called **kernelling** and the piece of paper a **hyperplane**. Sometimes it's best to keep things simple. ### The model As with logistic regression, and model-based classification in general, our goal is to determine some establishing **decision boundary** which is capable of separating the data into homogenous groups (ie. every example on the same of a decision boundary has the same label) using the features provided. In my post on logistic regression, we established that it is possible to draw a line, $z\\left( x \\right)$, so that points on one side take on positive values and points on the opposite side take on negative values. For example, in the following graph, $z = 6 - {x \_1}$ represents a decision boundary for which any values of ${x \_1} > 6$ will return a negative value for $z$ and any values of ${x \_1} < 6$ will return a positive value for $z$. For the case where we have two features, our decision boundary happens to be a line. More generally, we'll refer to this boundary as a **hyperplane**\- a subspace which consists of one less dimension than your feature space. Thus, if we were to draw a decision boundary for a dataset with three features, our boundary would be a plane. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/Screen-Shot-2017-06-16-at-3.33.09-PM.png) We further established that you can think of $z\\left( x \\right)$ as a measure of how far a certain data point exists from the decision boundary. With logistic regression, we used the notation \[z(x) = {\\theta ^T}x = {\\theta \_0}{x \_0} + {\\theta \_1}{x \_1} + {\\theta \_2}{x \_2} + ... + {\\theta \_n}{x \_n}\] where $x \_0 = 1$ by convention to represent our linear model. For support vector machines, it is more convenient to separate the intercept term, $\\theta \_0$, from the rest of the parameters ${\\theta \_{1,2,...,n}}$ due to the calculations we'll do. The standard way of writing our model for a support vector machine is \[z(x) = {w^T}x + b \] which expands to be \[z(x) = b + {w \_1}{x \_1} + {w \_2}{x \_2} + ... + {w \_n}{x \_n}\] where ${w^T}$ is equivalent to ${\\theta \_{1,2,...,n}^T}$ and $b$ is equivalent to $\\theta \_0$. This way, $x \_0$ is implicitly defined to be equal to one. More completely, we'll define our classification function, $g\\left( z \\right)$, as g( z )\={ 1 if z≥0 −1 if z<0 } which, combined with $z\\left( x \\right)$, yields our complete model for SVMs. \[{h\_{w,b}}\\left( x \\right) = g\\left( {z(x)} \\right) = g\\left( {{w^T}x + b} \\right)\] *Note: for SVMs we use class labels {-1, 1} instead of {0, 1} as we did in logistic regression.* ### Defining a margin Remember, this decision boundary is established based on what our model can learn from the training dataset. However, we'd ultimately like for the model to perform well for new data, or at the very least the data in our testing dataset. It makes somewhat intuitive sense that we can predict examples that lie far away from the decision boundary more confidently than those which lie close. For example, given two new observations, $A$ and $B$, which example would you be more confident in classifying? ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/Screen-Shot-2017-06-16-at-6.09.53-PM.png) They're technically the same class, but I'd be much more comfortable predicting the class of $B$. The optimized hyperplane will change slightly when trained on different subsets of your data, so observations that lie close to the boundary are susceptible to change in the classification output. Observations that lie far from the decision boundary are much more stable with respect predicting its class. Note how the classification for $A$ changes according to which hyperplane you select. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/Screen-Shot-2017-06-16-at-6.15.28-PM.png) In other words, we'd like to construct a hyperplane that is consistent with the data while committing the least amount to the specific training dataset - models that depend too much on the training data are prone to overfitting. We want to have the **most space possible** between the decision boundary and the data points on each side of the line in order to increase the total confidence in our predictions. In the world of SVMs, this "space" is called the margin (as visualized below). By maximizing the size of this margin, we can build an optimal classifier. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/Screen-Shot-2017-06-16-at-7.12.24-PM.png) In practice, there two main ways to measure this margin: the functional margin and the geometric margin. ###### Functional margin We can calculate the margin of our hyperplane by comparing the classifier's prediction (${{w^T}x + b}$) to the actual class ($y \_i$) since our training dataset has a labeled class for each data point. The **functional margin** is defined as: \[{{\\hat \\gamma } \_i} = {y \_i}\\left( {{w^T}x + b} \\right)\] When $y \_i =1$, we would need a large positive number for ${{w ^T}x + b}$ in order to have a large functional margin, while if $y \_i =-1$ we would need a large negative number for ${{w ^T}x + b}$ in order to have a large functional margin. Moreover, our classifier performs a correct classification for all values where ${{\\hat \\gamma } \_i}$ is positive. One way to maximize this functional margin is to arbitrarily scale $w$ and $b$, since classification with $h\_{w, b}$ only depends on the sign, not the magnitude, of the returned value. However, this provides little value and is a weakness of functional margins, as we have no useful way to maximize this expression. ###### Geometric margin The **geometric margin** is defined as the distance from a data point to the decision boundary. How might this be defined? Let's look at the data point labeled "A" in the picture below. Our margin, ${{\\hat \\gamma } \_i}$, defines the scalar length from our data point ($x \_i$) to the decision boundary while $\\frac{w}{{\\left| w \\right|}}$ represents the unit direction normal to the decision boundary (ie. the direction of the shortest path from A to B). Using vector subtraction we can define point B as: \[{\\bf{\\mathord{\\buildrel{\\lower3pt\\hbox{$\\scriptscriptstyle\\rightharpoonup$}} \\over A} }} - {\\gamma \_i}\\left( {\\frac{{\\mathord{\\buildrel{\\lower3pt\\hbox{$\\scriptscriptstyle\\rightharpoonup$}} \\over w} }}{{\\left| {\\mathord{\\buildrel{\\lower3pt\\hbox{$\\scriptscriptstyle\\rightharpoonup$}} \\over w} } \\right|}}} \\right)\] Or in other words: \[{\\bf{\\mathord{\\buildrel{\\lower3pt\\hbox{$\\scriptscriptstyle\\rightharpoonup$}} \\over A} }} - {\\rm{margin}} = {\\bf{\\mathord{\\buildrel{\\lower3pt\\hbox{$\\scriptscriptstyle\\rightharpoonup$}} \\over B} }}\] ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/Screen-Shot-2017-06-20-at-8.52.39-AM.png) Again, ${\\hat \\gamma} \_i$ is a scalar distance multiplied by a unit vector in the direction perpendicular to the decision boundary, $\\frac{w}{{\\left| w \\right|}}$. As a reminder, the decision boundary is the crossover from where points as classified as 1 or -1, so all values that lie on the decision boundary satisfy the equation ${\\omega ^T}x + b = 0$. Thus, point B in the figure above must satisfy \[{{w^T}{x \_B} + b = 0}\] since it lies on the decision boundary. Further, we know that we can calculate $x \_B$ as \[{{x \_B} = {x^i} - {\\gamma \_i}\\left( {\\frac{{\\mathord{\\buildrel{\\lower3pt\\hbox{$\\scriptscriptstyle\\rightharpoonup$}} \\over w} }}{{\\left| {\\mathord{\\buildrel{\\lower3pt\\hbox{$\\scriptscriptstyle\\rightharpoonup$}} \\over w} } \\right|}}} \\right)}\] which, combined with the first equation yields \[{{w^T}\\left( {{x^i} - {\\gamma \_i}\\left( {\\frac{w}{{\\left| w \\right|}}} \\right)} \\right) + b = 0}\] Simplifying this expression, we can solve for an equation that represents our geometric margin, ${\\gamma \_i}$. \[{{\\gamma \_i} = \\frac{{{w^T}{x^i} + b}}{{\\left| w \\right|}} = {{\\left( {\\frac{w}{{\\left| w \\right|}}} \\right)}^T}{x^i} + \\frac{b}{{\\left| w \\right|}}}\] Lastly, we make one small adjustment to keep the margin as a positive number when dealing with both classes ($y=1$ and $y=-1$). \[{\\gamma\_i} = {y\_i}\\left( {{{\\left( {\\frac{w}{{\\left| w \\right|}}} \\right)}^{\\text{T}}}{x\_i} + \\frac{b}{{\\left| w \\right|}}} \\right)\] Notice how the geometric margin is essentially the same thing as the functional margin, except $w$ and $b$ are scaled by a factor of $\\frac{1}{{\\left| w \\right|}}$. The margin of a hyperplane is formally defined as the smallest calculated margin of all the data points. \[\\gamma = \\mathop {\\min }\\limits\_{i = 1,...,m} {\\gamma\_i}\] **Summary:** The functional margin represents the lowest relative "confidence" of all classified points, while the geometric margin represents the smallest distance from $x\_i$ to the hyperplane. Functional margin: \[{{\\hat \\gamma }\_ i} = {y\_i}\\left( {{w^T}x + b} \\right)\] Geometric margin: \[{\\gamma\_i} = {y\_i}\\left( {{{\\left( {\\frac{w}{{\\left| w \\right|}}} \\right)}^{\\text{T}}}{x\_i} + \\frac{b}{{\\left| w \\right|}}} \\right)\] *Note: If ${\\left| w \\right|}=1$, then the functional margin, ${\\hat \\gamma }$, and the geometric margin, $\\gamma$, are equivalent expressions.* Now that we've sufficiently defined the "space" between our data and the hyperplane, we can now pose an optimization problem to maximize this space. ### Parameter optimizations The following section will present a path towards developing the optimization problem actually used for support vector machines. If the first two approaches are confusing, just skip them. **Approach: directly maximize geometric margin.** The straight-forward approach towards finding the parameters for a hyperplane with the most space between the data and the corresponding decision boundary is to simply maximize the geometric margin. Recall, directly maximizing the functional margin would not be useful as the parameters may be arbitrarily scaled to maximize the resulting margin. However, since the geometric margin is normalized by $\\left| w \\right|$, scaling $w$ and $b$ have no effect on the resulting margin. \[{\\max \_{\\gamma ,w,b}}\\gamma \] subject to ${y \_i}\\left( {{w^T}{x \_i} + b} \\right) \\ge \\gamma $ for $i = 1,...,m$ and $\\left| w \\right| = 1{\\rm{ }}$. The first constraint ensures that the maximized margin can be no larger than the smallest distance from an observation to the decision boundary. In essence, this enforces our definition that the geometric margin is $\\gamma = \\mathop {\\min }\\limits\_{i = 1,...,m} {\\gamma \_i}$. Moreover, this optimization favors correct predictions. Recall that any correct prediction will have a value ${y \_i}\\left( {{w^T}{x \_i} + b} \\right) > 0$. Thus, by maximizing $\\gamma$ subject to ${y \_i}\\left( {{w^T}{x \_i} + b} \\right) \\ge \\gamma$ we are also stating that the want both the **most correct predictions** and the **largest space possible** between the decision boundary and our data. Another way to think about this is that a negative margin signifies an incorrect classification and represents the distance a datapoint has "overstepped his bounds". The second constraint results in a situation where the geometric margin is equal to the functional margin. Unfortunately, this optimization problem is very difficult to solve computationally, mainly due to its non-convex space. **Approach: rewrite objective in terms of functional margin scaled by ${\\left| w \\right|}$.** We can express the geometrical margin in terms of the functional margin as $\\gamma = \\frac{{\\hat \\gamma }}{{\\left| w \\right|}}$. Thus, we can rewrite the same objective function as above in terms of the functional margin. \[{\\max \_{\\hat \\gamma ,w,b}}\\frac{{\\hat \\gamma }}{{\\left| w \\right|}}\] subject to ${y \_i}\\left( {{w^T}{x \_i} + b} \\right) \\ge \\hat \\gamma $ for $i = 1,...,m$. Unfortunately, this is still a difficult problem to solve computationally. However, it provides us a clue with what to do instead. Recall that $\\hat \\gamma$ may be arbitrarily scaled by parameters $w$ and $b$. We can introduce a scaling constraint $\\hat \\gamma = 1$ without changing the end result. Now you can see that what we are really trying to do is maximize $\\frac{1}{{\\left| w \\right|}}$. Unfortunately, this optimization problem is still difficult to solve. However, another way to maximize the fraction is to minimize the denominator, which is what we'll do now. **Approach: Focus on minimizing ${\\left| w \\right|}$.** It turns out that it is much easier to solve the following optimization problem. \[{\\min \_{\\gamma ,w,b}}\\frac{1}{2}{\\left| w \\right|^2}\] subject to ${y \_i}\\left( {{w^T}{x \_i} + b} \\right) \\ge 1 $ for $i = 1,...,m$. This is known as a **quadratic programming** problem which **is** relatively straight-forward to calculate computationally. We won't go too much into solving the optimization using quadratic programming, you can simply use a library that has this functionality already built-in, but we will examine it from a high level for a moment. For the quadratic programming problem, we represent the above optimization as \[{\\max \_{\\alpha ,x,y}}\\sum\\limits \_i {{\\alpha \_i}} - \\frac{1}{2}\\sum\\limits \_{i,j} {{\\alpha \_i}{\\alpha \_j}} {y \_i}{y \_j}x \_i^T{x \_j}\] subject to ${\\alpha \_i} \\ge 0$ and $\\sum\\limits \_i {{\\alpha \_i}{y \_i} = 0}$ where $i$ and $j$ represent a pair of observations present in the dataset. In solving this problem, we find a corresponding $\\alpha$ value for each observation, $x \_i$, in our training dataset. This $\\alpha$ value will then be used to calculate the $w$ vector using the following equation: \[w = \\sum\\limits \_i {{\\alpha \_i}{y \_i}{x \_i}} \] $b$ can also be recovered after $w$ has been solved for. It turns out that most of the values for $\\alpha \_i$ will be zero, which effectively nullifies any effect of the corresponding observation, $x \_i$, on optimizing our decision boundary. In other words, there are only a select few observations (namely, those close to the decision boundary) which are used in finding the optimal decision boundary. Remember, each observation is a $n$-dimensional vector containing the feature values associated with it. The **vectors** which correspond with non-zero $\\alpha$'s provide all of the necessary **support** in finding the optimal values for $w$. Hence, the name for our algorithm - support vector machines. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/Screen-Shot-2017-06-20-at-2.18.24-PM.png) ### Nonlinear decision boundaries - the kernel trick The other key part I want to examine in this quadratic programming problem is the term $\\sum\\limits \_{i,j} {{\\alpha \_i}{\\alpha \_j}} {y \_i}{y \_j}x \_i^T{x \_j}$. Let's break it down. - ${{\\alpha \_i}{\\alpha \_j}}$ determines whether or not this pair of observations is relevant in defining the decision boundary (ie. these values are non-zero). - ${y \_i}{y \_j}$ compares the output label of each observation, $x \_i$ and $x \_j$. This product will be positive if $x \_i$ and $x \_j$ belong to the same class and negative if $x \_i$ and $x \_j$ do not belong to the same class. - $x \_i^T{x \_j}$ is essentially a representation of how similar the two vectors $x \_i$ and $x \_j$ are to one another. In this case, we're using the inner product to judge similarity. With respect to the last term, $x \_i^T{x \_j}$, if the only thing that matters is the similarity between the two observations, then technically we could modify the actual expression as long as it's still representing "the similarity between vectors $x \_i$ and $x \_j$." Suppose instead of directly using the vectors $x \_i$ and $x \_j$, we wished to engineer additional features from the original values using $\\phi \\left( x \\right)$. ϕ( x )\=\[ x x 2 x 3 \] We could replace $x \_i^T{x \_j}$ with $\\phi {\\left( x \\right)^T}\\phi \\left( z \\right)$, which would allow us to more easily find higher-dimensional relationships in the data present. This new expression is referred to as a kernel. \[K\\left( {x,z} \\right) = \\phi {\\left( x \\right)^T}\\phi \\left( z \\right)\] However, simply mapping features into higher dimensional space isn't all that impressive. After all, it can be very computationally expensive to perform such high-dimensional calculations. Kernel functions are special in that they operate in a high-dimensional, **implicit** feature space without ever computing the coordinates of the data in that space. It can be very expensive to calculate $\\phi \\left( x \\right)$, $\\phi \\left( z \\right)$, and their corresponding dot product. However, the kernel function allows us a method for calculating $K\\left( {x,z} \\right)$ directly. By using Kernel functions to create new features, we can transform our classification problem into a higher-dimensional space, draw a linear decision boundary, and then project it back down into the real feature-space resulting in a nonlinear boundary. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/data_2d_to_3d_hyperplane.png) [*Image credit*](http://www.eric-kim.net/eric-kim-net/posts/1/kernel%5Ftrick.html?ref=jeremyjordan.me) Common kernel functions include: - Linear: $\\langle x, x'\\rangle$ - Polynomial: $(\\gamma \\langle x, x'\\rangle + r)^d$ - rbf: $\\exp(-\\gamma |x-x'|^2)$ where $\\gamma$ must be greater than 0. - Sigmoid $(\\tanh(\\gamma \\langle x,x'\\rangle + r))$ ### Regularization When building an SVM in practice, one of the most common parameters you'll tune is $C$. I'd like to take just a brief moment to examine this. We can reformulate our optimization objective to include a regularization term, a common technique to prevent overfitting. Our new optimization problem is as follows: \[{\\min \_{\\gamma ,w,b}}\\frac{1}{2}{\\left| w \\right|^2} + C\\sum\\limits \_{i = 1}^m {{\\xi \_i}} \] such that ${y \_i}\\left( {{w^T}{x \_i} + b} \\right) \\ge 1 - {\\xi \_i}$ for $i = 1,...,m$ and ${\\xi \_i} \\ge 0$ for $i = 1,...,m$. Essentially, we are now allowing the decision boundary to do a worse job classifying a few points in favor of doing a better job on the majority. Focusing on the majority allows for us to build a model which is more likely to generalize well and prevents our decision boundary from fluctuating wildly if we start to introduce outliers when training. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/Screen-Shot-2017-06-20-at-7.37.12-PM.png) Hence, $C$ controls the tradeoff between the smoothness of a decision boundary and accuracy of correct classifications. ### Further reading [Support Vector Machines - CS229 Lecture notes - Andrew Ng](http://cs229.stanford.edu/notes/cs229-notes3.pdf?ref=jeremyjordan.me) [Support Vector Machine (SVM) Tutorial](https://blog.statsbot.co/support-vector-machines-tutorial-c1618e635e93?ref=jeremyjordan.me) [Support Vector Machines Explained - Tristan Fletcher](http://cling.csd.uwo.ca/cs860/papers/SVM%5FExplained.pdf?ref=jeremyjordan.me) [Eric Kim - The kernel trick](http://www.eric-kim.net/eric-kim-net/posts/1/kernel%5Ftrick.html?ref=jeremyjordan.me) ### Boosted trees. URL: https://www.jeremyjordan.me/boosted-trees/ Last updated: 2019-09-19T22:03:03.000Z Boosting is an iterative ensembling process where models are trained in a sequential order. These models are known as "weak learners" as they are simple prediction rules which only perform slightly better than a random guess (ie. slightly better than 50% accuracy). The general concept behind boosting is to focus on the "hard" examples, or the examples that the model fails to predict correctly with confidence. These examples are given more emphasis by skewing the distribution of observations so that these examples are more likely to appear in a sample. As such, the next weak learner will focus more on getting these hard examples correct. Because your learner is always doing better than random, you'll always get some degree of information from each sequential training round. When all of the simple prediction rules are combined into one overarching model, a powerful predictor emerges. One useful characteristic of boosted ensemble learners is that, provided you use simple learners, the ensemble model is very robust against overfitting. Typically, when training a model you run the danger of overfitting if you continue optimizing for total accuracy of the training data; this is why cross validation is important as it ensures that we'll still have a model that performs well when we feed it new data. A typical validation curve (plot of training score and cross-validation score as a function of iterative trainings) will diverge at some point when you've begun to overfit your data. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/sphx_glr_plot_validation_curve_0011.png) [*Image credit*](http://scikit-learn.org/stable/modules/learning%5Fcurve.html?ref=jeremyjordan.me) However, this divergence doesn't occur in boosting models, which is quite convenient. The reason is that boosting focuses its efforts on performing well for "hard" examples. For instance, think of the case for classification; "hard" examples are those which are close to the decision boundary, so the boosting algorithm will focus its efforts on doing a better job at separating these data points. What results is an increased margin between the observations on each side of the decision boundary. As we learned with support vector machines, large margin separators tend to generalize well and are not prone to overfitting. #### AdaBoost AdaBoost (adaptive boosting) is the most popular boosting technique, developed for classification (and later extended to regression). The general steps are as follows: 1. Attach weights to each sample, initially achieving a uniform distribution. Given $m$ observations, we'll start off treating all of our data as equal. These weights are what we'll update in order to give more emphasis on the observations which perform poorly in our model. \[{w \_1}\\left( i \\right) = \\frac{1}{m}\] 2. Train a weak learner, ${{h \_t}\\left( x \\right)}$, on a random sample weighted by an initially uniform distribution, ${w \_1}\\left( i \\right)$. 3. Calculate the error associated with the learner, ${{h \_t}\\left( x \\right)}$. We'll calculate error as the probability that the model misclassifies a data point; this will implicitly take into account the weighted distribution of observations. \[{\\varepsilon \_t} = P\\left( {{h \_t}\\left( {{x \_i}} \\right) \\ne {y \_i}} \\right)\] 4. Score the model as a function of it's error. This equation is not a niave assertion, but derived from calculus in an attempt to figure a method which minimizes total error. \[{\\alpha \_t} = {\\raise0.5ex\\hbox{$\\scriptstyle 1$} \\kern-0.1em/\\kern-0.15em \\lower0.25ex\\hbox{$\\scriptstyle 2$}}\\ln \\frac{{1 - {\\varepsilon \_t}}}{{{\\varepsilon \_t}}}\] 5. Recalculate the weights for each observation to form a new distribution. More emphasis will be placed on observations which are misclassified. $z \_t$ serves as a normalization factor to ensure the weights add up to one (thus forming a distribution). \[{w \_{t + 1}}\\left( i \\right) = {w \_t}\\left( i \\right)\\frac{{{e^{ - {\\alpha \_t}{y \_i}{h \_t}\\left( {{x \_i}} \\right)}}}}{{{z \_t}}}\] \[{z \_t} = 2\\sqrt {{\\varepsilon \_t}\\left( {1 - {\\varepsilon \_t}} \\right)}\] 6. Iteratively train models and recalculate distributions in an effort to place more emphasis on the observations which are not being modeled accurately. In a sense, each sequential hypothesis will output a prediction for the class of $x \_i$, either $-1$ or $1$. Our final model will combine these predictions, weighted according to how well the individual model performs (according to ${\\alpha \_t}$). \[{H \_{final}} = {\\mathop{\\rm sgn}} \\left( {\\sum\\limits \_t {{\\alpha \_t}{h \_t}\\left( x \\right)} } \\right)\] #### Gradient Boosting (XGBoost) Gradient boosting is a generalized approach of boosting which allows for the optimization of an arbitrarily-defined loss function. I'll be revisiting this blog post in the near future to provide a more detailed description of this technique. ## Further reading [The Boosting Approach to Machine Learning: An Overview](https://www.cs.princeton.edu/courses/archive/spring07/cos424/papers/boosting-survey.pdf?ref=jeremyjordan.me) [A Kaggle Master Explains Gradient Boosting](http://blog.kaggle.com/2017/01/23/a-kaggle-master-explains-gradient-boosting/?ref=jeremyjordan.me) [Bagging and Boosting - MIT Lecture Slides](http://www.mit.edu/~9.520/spring06/Classes/class10.pdf?ref=jeremyjordan.me) [Theory and Applications of Boosting - Rob Schapire](http://media.nips.cc/Conferences/2007/Tutorials/Slides/schapire-NIPS-07-tutorial.pdf?ref=jeremyjordan.me) ### Random forests. URL: https://www.jeremyjordan.me/random-forests/ Last updated: 2019-05-20T02:21:09.000Z [Decision trees](https://www.jeremyjordan.me/decision-trees/) are desirable in that they scale well to larger datasets, they are robust against irrelevant features, and it is very easy to visualize the rationalization between a decision tree's predictions. Further, decision trees have low bias as there is minimal implicitly defined structure in the model (as opposed to linear regression, for example, which makes the assumption of linear relationships). Unfortunately, decision trees are prone to high variance and will overfit noisy data. Random forests inherit the benefits of a decision tree model whilst improving upon the performance by reducing the variance. In building a random forest, we train a collection of decision trees on random subsets of the training data. Further, on each sampling from the population, we also sample a subset of features from the overall feature space. This adds randomness (with the intent to reduce variance in the end model) to the splitting process by limiting the features available for to split on for each individual base model. ## Benefits of random sampling As mentioned previously, one of the downsides of decision trees is the fact that the model behavior varies significantly dependent on which data was used for training. This can make it difficult to train a single model which generalizes well. This is visualized below in an example where the training data is a noisy estimate of the true data generating function, $f(x)$. ![decision_tree_variance](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/05/decision_tree_variance.png) However, we can overcome this by training an ensemble of models, each trained on different subsets of our data. We'll observe this by training a collection of 50 trees, each trained on different random samples of $f(x)$ with a small amount of injected noise. Although each tree overfits to the sample on which it was trained, when you combine the predictions across trees you can build a much better estimate of the true data generating function. ![tree_ensemble](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/05/tree_ensemble.gif) This technique of randomly sampling our data with replacement, training a base estimator, and then aggregating the individual predictions is known as bootstrap aggregating, or **bagging** for short. As described earlier, when training a random forest we not only train each base estimator on a random subset of the training data, but also on a random subset of the features. This *feature bagging* ensures that each individual estimator splits on different aspects of the full observation, further reducing the variance of our predictions. > Empirical good default values are `max_features=n_features` for regression problems, and `max_features=sqrt(n_features)` for classification tasks (where `n_features` is the number of features in the data). – [scikit-learn User Guide](https://scikit-learn.org/stable/modules/ensemble.html?ref=jeremyjordan.me#parameters) ## Out of bag score When evaluating a machine learning model, we want to use independent data (not used during training) in order to form a good performance estimate. Typically, we do this by reserving a subset of our data for validation and never use it during training. However, because each individual tree only sees a random sample of data during training, we can build a performance estimate without the need for a separate hold-out set. We do this by passing the training observations through only the subset of trees that never saw that data during training. This allows us to form an **out of bag estimate** of our model's performance. ![oob](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2019/05/oob.png) ### K-nearest neighbors. URL: https://www.jeremyjordan.me/k-nearest-neighbors/ Last updated: 2017-06-23T13:14:35.000Z Many machine learning techniques involve building a model that is capable of representing the data and then finding the optimal parameters for the model to minimize error. K-nearest neighbors, however, is an example of **instance-based learning** where we instead simply store the training data and use it to make new predictions. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/Screen-Shot-2017-06-17-at-5.39.17-PM.png) In general, instance-based techniques such as k-nearest neighbors are **lazy learners**, as compared to model-based techniques which are **eager learners**. A lazy approach will only "learn" from the data (to make a prediction) when a new query is made while an eager learner will learn from the data right away and build a generalized model capable of predicting any value. Thus, lazy learners are fast to train and slower to query, while eager learners are slower to train but can make new predictions very quickly. K-nearest neighbors is based on the assumption of **locality** in your data, that nearby points have similar values. Thus, what's a good way to predict a new data point? Just look for similar observations in your training data and develop your best guess. More specifically, use the k-nearest observations to formulate your prediction. For regression, this prediction is some measure of the average value (ie. mean) of the surrounding neighbors. For classification, this prediction is a result of some voting mechanism (ie. mode) of the surrounding neighbors. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/Screen-Shot-2017-06-17-at-9.30.39-AM-1.png) In the example above, we selected $k=3$ to show an example of k-nearest neighbors for both regression and classification. ### Finding similar observations (measures of distance) In order to find the k-nearest neighbors, we must first define some measure of "closeness" (in order words, distance). Remember, our observations exist within an n-dimensional featurespace; as it turns out, there are many ways to measure the distance between two points in space. In sklearn's implementation for k-nearest neighbors, you can use any of the [available methods](http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.DistanceMetric.html?ref=jeremyjordan.me) found in the `DistanceMetric` class. Remember, we're using distance as a proxy for measuring similarity between points. One important note to make is that since we're using distance as a measure of similarity, we're implicitly constraining our model to weight all features equally; distance in the $x \_1$ dimension is on the same scale as distance in the $x \_n$ dimension. The best way to deal with data containing features of varying importance is to simply feed the KNN algorithm more data; given sufficient training data (sometimes this might mean hundreds of thousands of examples), the detrimental effect of implicitly weighting all features equally will diminish. ### Choosing k After selecting a metric to be used for finding the observations closest to the query, you must determine how many of the nearest neighbors you'd like to take into account. By default, `sklearn.neighbors.KNeighborsRegressor` and `sklearn.neighbors.KNeighborsClassifier` use 5 as the default value for `n_neighbors` (otherwise known as $k$), but this can easily be optimized using something like K-fold cross validation to try out different values for $k$ and determine the best choice. A common rule of thumb seems to be that $\\sqrt n$, where $n$ is the number of samples in your training set, seems to often perform well. Consider the case where we set $k=n$. Wherever our query point is, we'd end up using the entire dataset to predict the value. If we're using a simple average, the resulting predictor will return a constant value regardless of the input features. For classification, you'll always end up classifying new data points as whichever class is most prevalent in the training dataset. For regression, you'll always end up return the mean value of the entire training dataset. However, if we instead use a weighted average for prediction, where data points closer to the query take on higher weights, how would the results look? Let's take a dive into a Jupyter notebook and explore this. First, let's load an example dataset and prepare the data for training and testing. ``` from sklearn.datasets import load_boston from sklearn.model_selection import train_test_split boston = load_boston() features_train, features_test, labels_train, labels_test = train_test_split(boston.data, boston.target, test_size=0.4, random_state=0) import pandas as pd features_train = pd.DataFrame(features_train, columns = boston.feature_names) features_test = pd.DataFrame(features_test, columns = boston.feature_names) # Select rooms column for univariate regression example features_train = features_train.iloc[:, 5] features_test = features_test.iloc[:, 5] labels_train = pd.DataFrame(labels_train, columns = ['Price']) labels_test = pd.DataFrame(labels_test, columns = ['Price']) ``` Next, I'll just take a peek at the data to see what we're working with. ``` import matplotlib.pyplot as plt %matplotlib inline plt.scatter(features_train, labels_train) ``` Lastly, I'll train six KNN models with varying weights and values for $k$. ``` from sklearn.neighbors import KNeighborsRegressor import numpy as np n = features_train.shape[0] params = [{'weights': 'uniform', 'n_neighbors': 3}, {'weights': 'distance', 'n_neighbors': 3}, {'weights': 'uniform', 'n_neighbors': np.sqrt(n).astype(int)}, {'weights': 'distance', 'n_neighbors': np.sqrt(n).astype(int)}, {'weights': 'uniform', 'n_neighbors': n}, {'weights': 'distance', 'n_neighbors': n}] plot_range = np.arange(min(features_train), max(features_train), 0.01) for i, param in enumerate(params): model = KNeighborsRegressor(**params[i]) model.fit(features_train.values.reshape(-1, 1), labels_train) pred = model.predict(plot_range.reshape(-1, 1)) # Sort values for plotting new_X, new_y = zip(*sorted(zip(plot_range.reshape(-1, 1), pred))) plt.subplot(3, 2, i + 1) plt.scatter(features_train.values.reshape(-1, 1), labels_train, c='k', label='data') plt.plot(new_X, new_y, c='g', label='prediction') plt.legend() plt.title("KNeighborsRegressor (k = %i, weights = '%s')" % (params[i]['n_neighbors'], params[i]['weights'])) plt.ylabel('Price') plt.xlabel('Number of rooms') plt.rcParams["figure.figsize"] = [24, 14] plt.show() ``` The result is shown below. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/download--9-.png) I kind of jumped ahead of myself, as we haven't finished discussing the considerations of the KNN algorithm, but I wanted to go ahead and show you the effect of $k$. *Note: The rule of thumb $k = \\sqrt n$ seems to be our best model.* ### Tying things together (selecting neighbors) Now that we've established a metric for measuring similarity and picked how many neighbors we'd like to include in the prediction, we need a method to search through our stored data to find the k-nearest neighbors. A **brute force** search, simply calculating the distance of our query from every point in our dataset, will work fairly well with small datasets but becomes undesirably slow at larger scales. Tree-based approaches can bring greater efficiencies to the search process by inferring distances. For example, if $A$ is very far from $B$ and $C$ is very close to $B$, we can infer that $A$ is also far from $C$ without explicitly measuring the distance. The [sklearn documentation](http://scikit-learn.org/stable/modules/neighbors.html?ref=jeremyjordan.me#nearest-neighbor-algorithms) covers the **K-D Tree** and **Ball Tree** search algorithms, as well as some practical advice on choosing a specific search algorithm. ### Using stored data for predictions (measures of the average) After we've found the k-nearest neighbors, we can use their information to predict a value for the query. ###### Classification So you have a new observation with features $x \_i$ and you'd like to predict the class label, $y \_i$. You've queried the `KNeighborsClassifier` and found the $k$ nearest neighbors. Now you're tasked with using the information present among the $k$ neighbors to predict $y \_i$. The simplest method is to conduct a uniform vote where each neighbor essentially casts one vote to say "the **query** shares the same class as me". In the event that your vote ends in a tie, choose the label which is more common across the entire dataset. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/Screen-Shot-2017-06-18-at-3.15.29-PM.png) However, we can perform a more accurate vote if we decide to **weight** each vote based on how close it is to the query point. This is accomplished by weighting each vote by the inverse of its distance from the query so that closer points have stronger votes. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/Screen-Shot-2017-06-18-at-3.15.19-PM.png) ###### Regression Now let's suppose you have a new observation with features $x \_i$ and you'd like to predict a continuous target output, $y \_i$. You've queried the `KNeighborsRegressor` and found the $k$ nearest neighbors. Now you're tasked with using the information present among the $k$ neighbors to predict $y \_i$. The approach is largely the same as discussed above for classifiers, but rather than establishing a voting mechanism we'll simply calculate the mean of the neighbor's $y$ values. We can still use uniform or distance weighting. **Fun fact:** You can combine k-nearest neighbors with linear regression to build a collection of linear models as a predictor. [*Read more here.*](https://en.wikipedia.org/wiki/Local%5Fregression?ref=jeremyjordan.me) ### Summary K-nearest neighbors is an example of instance-based learning where we store the training data and use it directly to generate a prediction, rather than attempted to build a generalized model. The three main things you must define for a KNN algorithm is a way to measure **distance**, how many **neighbors ($k$)** to use in your predictions, and how to **average** the information from your neighbors to generate a prediction. ### Naive Bayes classification. URL: https://www.jeremyjordan.me/naive-bayes-classification/ Last updated: 2025-03-03T02:12:07.000Z Naive Bayes classification methods are quite simple (in terms of model complexity) and commonly used for tasks such as document classification and spam filtering. This algorithm works well for datasets with a large amount of features (ex. a body of text where every word is treated as a feature) but it is naive in the sense that it treats every feature as independent of one another. This is clearly not the case for language, where word order matters when trying to discern meaning from a statement. Nonetheless, these methods have been used quite successfully for various text classification tasks. ### Bayes' theorem In order to discuss the Naive Bayes model, we must first cover the Bayesian principles of thinking. - **We learn about the world through tests** and experiments. - **Tests are separate from events in the world.** A positive test result for a disease such as cancer is not the same as a patient having cancer. - **Tests are flawed** and can yield false positives (tests that detect things that don't exist) and false negatives (tests that miss something that does exist). - **Not all tests are equal**, so we shouldn't believe all test evidence uniformly. Bayes' theorem allows for us to calculate the **actual** probability of some event relative to the measured **test** probability. In other words, Bayes' theorem allows us to restate "the probability that you have cancer" as "the probability that you have cancer, given knowledge of some diagnostic test result." We can more accurately include evidence from a test by incorporating the **known error** of the given test. Thus, in order to include any test evidence into our probability model, we need to know the historical accuracy of the test. For the case of a cancer screening, this would be "for all of times our test yielded a positive diagnosis, what percentage of patients actually had cancer?" Additionally, we'd need to know "what percentage of patients that did not have cancer falsely received a positive test result?" More concretely, to calculate the probability of a patient having cancer, given a positive test diagnosis, we would combine 1) the probability that our test yields positive for a patient, given that they have cancer, with 2) the prior probability of a patient having cancer and divide by the total probability of a positive diagnosis. $$\\frac{{{\\rm{P}}\\left( {pos|cancer} \\right){\\rm{P}}\\left( {cancer} \\right)}}{{{\\rm{P}}\\left( {pos} \\right)}}$$ The total probability of a positive diagnosis would include both the probability of a true positive and the probability of a false positive diagnosis. $${\\rm{P}}\\left( {pos} \\right) = {\\rm{P}}\\left( {pos|cancer} \\right){\\rm{P}}\\left( {cancer} \\right) + {\\rm{P}}\\left( {pos|no \\hspace{1mm} cancer} \\right){\\rm{P}}\\left( {no \\hspace{1mm} cancer} \\right)$$ Putting it all together, we arrive at Bayes' rule. $${\\rm{P}}\\left( {cancer|pos} \\right) = \\frac{{{\\rm{P}}\\left( {pos|cancer} \\right){\\rm{P}}\\left( {cancer} \\right)}}{{{\\rm{P}}\\left( {pos|cancer} \\right){\\rm{P}}\\left( {cancer} \\right) + {\\rm{P}}\\left( {pos|no \\hspace{1mm} cancer} \\right){\\rm{P}}\\left( {no \\hspace{1mm} cancer} \\right)}}$$ Initially, we might assign the probability of cancer as frequency of the disease in the general population. However, after performing a test to screen for cancer, we have a better idea for that specific patients probability of having cancer. We could then perform a second test, using the updated probability of cancer, to incorporate additional evidence. In general, we refer to the probability of an event without considering the test evidence as the *prior probability* and the calculated probability using Bayes' rule to incorporate a test as the *posterior probability*. > Bayes’ Rule is a theorem in probability theory that answers the question, "When you encounter new information, how much should it change your confidence in a belief?" - [Source](https://io9.gizmodo.com/how-bayes-rule-can-make-you-a-better-thinker-471233405?ref=jeremyjordan.me) Bayesian thinking allows us to update our understanding of the world incrementally as new evidence becomes available. ###### Derivation of Bayes' rule It's relatively straightforward to derive the formula for Bayes' rule. First, we'll establish a defintion for the conditional probability of an event. Conditional probabilities allow us to express the probability of event $A$, given the condition that event $B$ occurs. The conditional probability of $A$ given $B$, $\\rm{P}\\left( {A|B} \\right)$ can be calculated as the probability of $A$ **and** $B$ occuring, $\\rm{P}\\left( {A \\cap B} \\right)$, divided by the probability of $B$ occuring. $$ {\\rm{P}}\\left( {A|B} \\right) = \\frac{{{\\rm{P}}\\left( {A \\cap B} \\right)}}{{{\\rm{P}}\\left( B \\right)}} $$ I like to remember this as "out of the subset of observations where $B$ occurs, how often does $A$ occur as well?" Using this definition of conditional probability, we can write the following: $${\\rm{P}}\\left( {A|B} \\right) = \\frac{{{\\rm{P}}\\left( {A \\cap B} \\right)}}{{{\\rm{P}}\\left( B \\right)}}$$ $${\\rm{P}}\\left( {B|A} \\right) = \\frac{{{\\rm{P}}\\left( {B \\cap A} \\right)}}{{{\\rm{P}}\\left( A \\right)}}$$ The probability of $A$ and $B$ occuring is the same as the probability of $B$ and $A$ occuring. $${\\rm{P}}\\left( {A \\cap B} \\right) = {\\rm{P}}\\left( {B \\cap A} \\right)$$ Solving the second equation for ${\\rm{P}}\\left( {B \\cap A} \\right)$, we find: $${\\rm{P}}\\left( {B \\cap A} \\right) = {\\rm{P}}\\left( {B|A} \\right){\\rm{P}}\\left( A \\right)$$ We can now substitute this expression for the numerator in our first equation to arrive at the final definition of Bayes' rule. $${\\rm{P}}\\left( {A|B} \\right) = \\frac{{{\\rm{P}}\\left( {B|A} \\right){\\rm{P}}\\left( A \\right)}}{{{\\rm{P}}\\left( B \\right)}}$$ ### A Naive Bayes' model We can apply Bayes' rule to the subject of classification by calculating the probability an observation belongs to some class $y$, given a feature vector $x$. $${\\rm{P}}\\left( {y|{x\_1},...,{x\_n}} \\right) = \\frac{{{\\rm{P}}\\left( y \\right){\\rm{P}}\\left( {{x\_1},...,{x\_n}|y} \\right)}}{{{\\rm{P}}\\left( {{x\_1},...,{x\_n}} \\right)}}$$ To keep things simple, we assume that each feature is independent of one another and that the probability of a feature being expressed depends only on the observation's class. This is known as the naive independence assumption. $${\\rm{P}}\\left( {{x\_i}|y,{x\_1},...,{x\_{i - 1}},{x\_{i + 1}},...{x\_n}} \\right) = {\\rm{P}}\\left( {{x\_i}|y} \\right)$$ To be clear, we are stating that the probability distribution of any feature depends **only** on the observation's class and has no dependence on the values of other features. Using this naive independence assumption we can simplify our first expression to consider the probability of features independently. $${\\rm{P}}\\left( {y|{x\_1},...,{x\_n}} \\right) = \\frac{{{\\rm{P}}\\left( y \\right)\\prod\\nolimits\_{i = 1}^n {{\\rm{P}}\\left( {{x\_i}|y} \\right)} }}{{{\\rm{P}}\\left( {{x\_1},...,{x\_n}} \\right)}}$$ This allows us to calculate the probability an observation belongs to a certain class given a feature vector describing the observation. Because the denominator is constant with respect to class, we can simply look at the numerator when determining the class of an observation. $$\\hat y = \\arg \\mathop {\\max }\\limits\_y {\\rm{P}}\\left( y \\right)\\prod\\nolimits\_{i = 1}^n {{\\rm{P}}\\left( {{x\_i}|y} \\right)} $$ Simply, we'll predict an observation's class as that which has the maximum calculated probability. In the cancer screening example I presented previously, we considered a boolean feature - the diagnosis either returns true for the disease or false. However, it's also possible to consider numerical features with a continuous range of possibilities by building a probability distribution for each feature. ### Gaussian Naive Bayes For a Gaussian Naive Bayes model, we can fit a normal distribution for each feature of all of the possible classes. $${\\rm{P}}\\left( {{x\_i}|y} \\right) = \\frac{1}{{\\sqrt {2\\pi \\sigma\_y^2} }}\\exp \\left( { - \\frac{{{{\\left( {{x\_i} - {\\mu\_y}} \\right)}2}}}{{2\\sigma\_y2}}} \\right)$$ [Victor Lavrenko](https://www.youtube.com/watch?v=r1in0YNetG8&ref=jeremyjordan.me) has a nice visualization of what these Gaussian models will look like for a dataset of two classes (adult and child) each containing two features (height and weight). The Gaussian models for child heights and child weights combine to form a conic (depicted as a contour plot) and a similar combination shown in red for the adult class feature models. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/01/gaussian-model.png) [Image credit](https://i.ytimg.com/vi/TcAQKPgymLE/maxresdefault.jpg?ref=jeremyjordan.me) I like this visualization because once we leave 2-dimensional space, it becomes rather tricky to visualize what's going on, but the idea stays the same. ### Summary In summary, Bayes' theorem provides a statistical framework for incorporating test evidence into our probabilistic viewpoint of events. We can apply Bayes' theorem to classification tasks within machine learning to calculate the probability of an observation belonging to each of the possible classes, given a feature vector that describes the observation. It's important to note that although Naive Bayes' methods can work fairly well for predicting class, the actual probabilities calculated are generally not too accurate. This is due to the fact that we often incorrectly assume independence of features for the sake of convenience. ### Further reading - [An Intuitive (and Short) Explanation of Bayes’ Theorem](https://betterexplained.com/articles/an-intuitive-and-short-explanation-of-bayes-theorem/?ref=jeremyjordan.me) - [Naive Bayes - Sklearn documentation](http://scikit-learn.org/stable/modules/naive%5Fbayes.html?ref=jeremyjordan.me) - [How Bayes’ Rule Can Make You A Better Thinker](https://io9.gizmodo.com/how-bayes-rule-can-make-you-a-better-thinker-471233405?ref=jeremyjordan.me) - [Introduction to Bayesian Inference](https://www.datascience.com/learn-data-science/fundamentals/introduction-to-bayesian-inference-data-science?ref=jeremyjordan.me) ### Logistic regression. URL: https://www.jeremyjordan.me/logistic-regression/ Last updated: 2025-03-03T01:55:44.000Z The goal of logistic regression, as with any classifier, is to figure out some way to split the data to allow for an accurate prediction of a given observation's class using the information present in the features. (For instance, if we were examining the Iris flower dataset, our classifier would figure out some method to split the data based on the following: sepal length, sepal width, petal length, petal width.) In the case of a generic two-dimensional example, the split might look something like this. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/Screen-Shot-2017-06-10-at-9.41.25-AM.png) ### The decision boundary Let's suppose we define a line that is equal to zero along this decision boundary. This way, all of the points on one side of the line take on positive values and all of the points on the other side take on negative values. For example, in the following graph, $z = 6 - {x \_1}$ represents a decision boundary for which any values of ${x \_1} > 6$ will return a negative value for $z$ and any values of ${x \_1} < 6$ will return a positive value for $z$. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/Screen-Shot-2017-06-10-at-5.48.48-PM.png) We can extend this decision boundary representation as any linear model, with or without additional polynomial features. (If this model representation doesn't look familiar, check out my post on [linear regression](https://www.jeremyjordan.me/linear-regression/)). $$z\\left( x \\right) = {\\theta \_0} + {\\theta \_1}{x \_1} + {\\theta \_2}{x \_2} + ... + {\\theta \_n}{x \_n} = {\\theta ^T}x$$ Adding [polynomial features](https://www.jeremyjordan.me/polynomial-regression/) allows us to construct nonlinear splits in the data in order to draw a more accurate decision boundary. As we continue, think of $z\\left( x \\right)$ as a measure of how far a certain data point exists from the decision boundary. ### The logistic function Next, let's take a look at the logistic function. $$g\\left( z \\right) = \\frac{1}{{1 + {e^{ - z}}}}$$ ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/download.png) As you can see, this function is asymptotically bounded between 0 and 1\. Further, for very positive inputs our output will be close to 1 and for very negative inputs our output will be close to 0\. This will essentially allow us to translate the value we obtain from $z$ into a prediction of the proper class. Inputs that are close to zero (and thus, near the decision boundary) signify that we don't have a confident prediction of 0 or 1 for the observation. At the decision boundary $z=0$, the function $g\\left( z \\right) = 0.5$. We'll use this value as a cutoff establishing the prediction criterion: $${h \_\\theta }({x \_i}) \\ge 0.5 \\to {y \_{pred}} = 1$$ $${h \_\\theta }({x \_i}) < 0.5 \\to {y \_{pred}} = 0$$ where ${y \_{pred}}$ denotes the predicted class of an observation, ${x \_i}$, and ${h \_\\theta }({x})$ represents the functional composition, ${h \_\\theta }\\left( x \\right) = g\\left( {z\\left( x \\right)} \\right)$. More concretely, we can write our model as: $${h\_\\theta }\\left( x \\right) = \\frac{1}{{1 + {e^{ - {\\theta ^T}x}}}}$$ Thus, the output of this function represents a binary prediction for the input observation's class. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/download--1--1.png) Another way to interpret this output is to view it in terms of a probabilistic prediction of the true class. In other words, ${h \_\\theta }\\left( x \\right)$ represents the estimated probability that ${y \_i} = 1$ for a given input, ${x \_i}$. $${h \_\\theta }\\left( {{x \_i}} \\right) = {\\rm{P}}\\left( {{y \_i} = 1|{x \_i};\\theta } \\right)$$ Because the class can only take on values of 0 or 1, we can also write this in terms of the probability that ${y \_i} = 0$ for a given input, ${x \_i}$. $${\\rm{P}}\\left( {{y \_i} = 0|{x \_i};\\theta } \\right) = 1 - {\\rm{P}}\\left( {{y \_i} = 1|{x \_i};\\theta } \\right)$$ For example, if ${h \_\\theta }\\left( x \\right) = 0.85$ then we can assert that there is an 85% probability that ${y \_i} = 1$ and a 15% probability that ${y \_i} = 0$. This is useful as we can not only predict the class of an observation, but we can quantify the **certainty** of such prediction. In essence, the further a point is from the decision boundary, the more certain we are about the decision. ### The cost function Next, we need to establish a cost function which can grade how well our model is performing according to the training data. This cost function, $J\\left( \\theta \\right)$, can be considered to be a summation of individual "grades" for each classification prediction in our training set, comparing ${h\_\\theta }\\left( x \\right)$ with the true class $y \_i$. We want the cost function to be large for incorrect classifications and small for correct ones so that we can minimize $J\\left( \\theta \\right)$ to find the optimal parameters. $$J\\left( \\theta \\right) = \\frac{1}{m}\\sum\\limits\_{i = 1}^m {{\\rm{Cost}}({h \_\\theta }\\left( {{x \_i}} \\right),{y \_i})}$$ In linear regression, we used the squared error as our grading mechanism. Unfortunately for logistic regression, such a cost function produces a nonconvex space that is not ideal for optimization. There will exist many local optima on which our optimization algorithm might prematurely converge before finding the true minimum. Using the Maximum Likelihood Estimator from statistics, we can obtain the following cost function which produces a convex space friendly for optimization. This function is known as the **binary cross-entropy loss**. Cost( h θ ( x ),y)\={ −log( h θ ( x ) ) if y=1 −log( 1− h θ ( x ) )if y=0 } These cost functions return high costs for incorrect predictions. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/download--2-.png) More succinctly, we can write this as $${\\rm{Cost}}({h \_\\theta }\\left( x \\right),y) = - y\\log \\left( {{h \_\\theta }\\left( x \\right)} \\right) - \\left( {1 - y} \\right)\\log \\left( {1 - {h \_\\theta }\\left( x \\right)} \\right)$$ since the second term will be zero when $y=1$ and the first term will be zero when $y=0$. Substituting this cost into our overall cost function we obtain: $$J\\left( \\theta \\right) = \\frac{1}{m}\\sum\\limits \_{i = 1}^m { - {y \_i}\\log \\left( {{h \_\\theta }\\left( {{x \_i}} \\right)} \\right) - \\left( {1 - {y \_i}} \\right)\\log \\left( {1 - {h \_\\theta }\\left( {{x \_i}} \\right)} \\right)} $$ ### Finding the best parameters ###### Gradient descent As a [reminder](https://www.jeremyjordan.me/linear-regression/), the algorithm for gradient descent is provided below. $${\\theta \_j}: = {\\theta \_j} - \\alpha \\frac{\\partial }{{\\partial {\\theta \_j}}}J\\left( \\theta \\right)$$ Taking the partial derivative of the cost function from the previous section, we find: $$\\frac{\\partial }{{\\partial {\\theta \_j}}}J\\left( \\theta \\right) = \\frac{1}{m}\\sum\\limits \_{i = 1}^m {\\left( {{h \_\\theta }\\left( {{x \_i}} \\right) - {y \_i}} \\right)} {x \_{i,j}}$$ And thus our parameter update rule for gradient descent is: $${\\theta \_j}: = {\\theta \_j} - \\frac{\\alpha }{m}\\sum\\limits \_{i = 1}^m {\\left( {{h \_\\theta }\\left( {{x \_i}} \\right) - {y \_i}} \\right)} {x \_{i,j}}$$ ###### Advanced optimizations In practice, there are more advanced optimization techniques that perform much better than gradient descent including conjugate gradient, BFGS, and L-BFGS. These advanced numerical techniques are not within the current scope of this blog post. In sklearn's implementation of logistic regression, you can define which optimization is used with the `solver` parameter. ### Multiclass classification While binary classification is useful, it'd be really nice to be able to train a classifier that can separate the data into more than two classes. For example, the Iris dataset I mentioned in the beginning of this post has three target classes: Iris Setosa, Iris Versicolour, and Iris Virginica. A simple binary classifier simply wouldn't work for this dataset! ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/Screen-Shot-2017-06-11-at-12.27.19-PM.png) However, using a one-vs-all (also known as one-vs-rest) classifier we can use a combination of binary classifiers for multiclass classification. We simply look at each class, one by one, and treat all of the other classes as the same. In a sense, you look at "Class 1" and build a classifier that can predict "Class 1" or "Not Class 1", and then do the same for each class that exists in your dataset. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/Screen-Shot-2017-06-11-at-1.52.06-PM.png) In other words, each classifier $h \_\\theta ^i\\left( x \\right)$ returns the probability that an observation belongs to class $i$. Therefore, all we have to do in order to predict the class of an observation is to select the class of whichever classifier returns the highest probability. ### Summary Each example has an $n$ dimensional feature vector, ${x \_i}$, which belongs to a class, ${y \_i}$. x∈\[ 1 x 1 ⋮ x n \], y∈{ 0,1 } Our hypothesis function predicts the likelihood that the example $i$ belongs to the class ${y \_i}=1$ $${h \_\\theta }\\left( x \\right) = \\frac{1}{{1 + {e^{ - {\\theta ^T}x}}}}$$ where $\\theta ^Tx$ represents a decision boundary. We can assign a cost function to optimize the parameters $\\theta$ to provide the best possible decision boundary. In order to build a classifier capable of distinguishing between more than two classes, we simply build a collection of classifiers which each return the probability the observation belongs to a specific class. x∈\[ 1 x 1 ⋮ x n \], y∈{ 1,2,...,k } For a dataset with $k$ classes, you'd train a collection of $k$ classifiers. $$h \_\\theta ^1\\left( x \\right),h \_\\theta ^2\\left( x \\right),...h \_\\theta ^k\\left( x \\right)$$ ### Further reading - [Logistic Regression Trained with Different Loss Functions](http://www.ccs.neu.edu/home/vip/teach/MLcourse/2%5FGD%5FREG%5Fpton%5FNN/lecture%5Fnotes/logistic%5Fregression%5Floss%5Ffunction/logistic%5Fregression%5Floss.pdf?ref=jeremyjordan.me) - [Maximum Likelihood Estimation of Logistic Regression Models: Theory and Implementation](http://www.saedsayad.com/docs/mlelr.pdf?ref=jeremyjordan.me) ### Decision trees. URL: https://www.jeremyjordan.me/decision-trees/ Last updated: 2023-03-04T23:38:18.000Z Decision trees are one of the oldest and most widely-used machine learning models, due to the fact that they work well with noisy or missing data, can easily be [ensembled](https://www.jeremyjordan.me/random-forests/) to form more robust predictors, and are incredibly fast at runtime. Moreover, you can directly visual your model's learned logic, which means that it's an incredibly popular model for domains where model interpretability is important. Decision trees are pretty easy to grasp intuitively, let's look at an example. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/03/Screen-Shot-2017-03-11-at-9.39.55-PM.png) *Note: decision trees are used by starting at the top and going down, level by level, according to the defined logic. This is known as recursive binary splitting.* For those wondering – yes, I'm sipping tea as I write this post late in the evening. Now let's dive in! ## Classification Let's look at a two-dimensional feature set and see how to construct a decision tree from data. The goal is to construct a decision boundary such that we can distinguish from the individual classes present. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/03/Screen-Shot-2017-03-11-at-9.51.20-PM.png) Any ideas on how we could make a decision tree to **classify** a new data point as "x" or "o"? Here's what I did. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/03/Screen-Shot-2017-03-11-at-10.15.37-PM.png) Run through a few scenarios and see if you agree. Look good? Cool. So now we have a decision tree for this data set; the only problem is that *I created* the splitting logic. It'd be much better if we could get a machine to do this for us. But how? If you analyze what we're doing from an abstract perspective, we're taking a subset of the data, and deciding the best manner to split the subset further. Our initial subset was the entire data set, and we split it according to the rule $x\_1 < 3.5$. Then, for each subset, we performed additional splitting until we were able to correctly classify every data point. #### Information gain and entropy How do we judge the best manner to split the data? Simply, we want to split the data in a manner which provides us the most amount of information - in other words, maximizing information gain. Going back to the previous example, we could have performed our first split at $x\_1 < 10$. However, this would essentially be a useless split and provide zero information gain. In order to mathematically quantify information gain, we introduce the concept of entropy. \[\\mathop - \\sum \\limits \_i^{} {p \_i}{\\log \_2}\\left( {{p \_i}} \\right)\] Entropy may be calculated as a summation over all classes where $p\_i$ is the fraction of data points within class $i$. This essentially represents the impurity, or noisiness, of a given subset of data. A homogenous dataset will have zero entropy, while a perfectly random dataset will yield a maximum entropy of 1\. With this knowledge, we may simply equate the information gain as a reduction in noise. information gain = entropy( parent )−\[ weighted average \]entropy( children ) Here, we're comparing the noisiness of the data before splitting the data (parent) and after the split (children). If the entropy decreases due to a split in the dataset, it will yield an information gain. A decision tree classifier will make a split according to the feature which yields the highest information gain. This is a recursive process; stopping criterion for this process include continuing to split the data until (1) the tree is capable of correctly classifying every data point, (2) the information gain from further splitting drops below a given threshold, (3) a node has fewer samples than some specified threshold, (4) the tree has reached a maximum depth, or (5) another parameter similarly calls for the end of splitting. To learn more about this process, read about the [ID3 algorithm.](https://en.wikipedia.org/wiki/ID3%5Falgorithm?ref=jeremyjordan.me) #### Techniques to avoid overfitting Often you may find that you've overfitted your model to the data, which is often detrimental to the model's performance when you introduce new data. To prevent this overfitting, one thing you could do is define some parameter which ends the recursive splitting process. As I mentioned earlier, this may be a parameter such as **maximum tree depth** or **minimum number of samples required in a split**. Controlling these model hyperparameters is the easiest way to counteract overfitting. You could also simply perform a significance test when considering a new split in the data, and if the split does not supply statistically significant information (obtained via a significance test), then you will not perform any further splits on a given node. Another technique is known as **pruning**. Here, we grow a decision tree fully on your training dataset and then go back and evaluate its performance on a new validation dataset. For each node, we evaluate whether or not it's split was useful or detrimental to the performance on the validation dataset. We then remove those nodes which caused the greatest detriment to the decision tree performance. #### Degenerate splits Evaluating a split using information gain can pose a problem at times; specifically, it has a tendency to favor features which have a high number of possible values. Say I have a data set that determines whether or not I choose to go sailing for the month of June based on features such as temperature, wind speed, cloudiness, and day of the month. If I made a decision tree with 30 child nodes (Day 1, Day 2, ..., Day 30) I could easily build a tree which accurately partitions my data. However, this is a useless feature to split based on because the second I enter the month of July (outside of my training data set), my decision tree has no idea whether or not I'm likely to go sailing. One way to circumvent this is to assign a cost function (in this case, the **gain ratio**) to prevent our algorithm from choosing attributes which provide a large number of subsets. #### Example code Here's an example implementation of a Decision Tree Classifier for classifying the flower species dataset we've studied previously. ``` import pandas as pd from sklearn import datasets from sklearn.tree import DecisionTreeClassifier from sklearn.model_selection import train_test_split iris = datasets.load_iris() # ['target_names', 'data', 'target', 'DESCR', 'feature_names'] features = pd.DataFrame(iris.data) labels = pd.DataFrame(iris.target) ### create classifier clf = DecisionTreeClassifier(criterion='entropy') ### split data into training and testing datasets features_train, features_test, labels_train, labels_test = train_test_split(features, labels, test_size=0.4, random_state=0) ### fit the classifier on the training features and labels clf.fit(features_train, labels_train) ### use the trained classifier to predict labels for the test features pred = clf.predict(features_test) ### calculate and return the accuracy on the test data from sklearn.metrics import accuracy_score accuracy = accuracy_score(labels_test, pred) ### visualize the decision tree ### you'll need to have graphviz and pydot installed on your computer from IPython.display import Image from sklearn.externals.six import StringIO import pydot from sklearn import tree dot_data = StringIO() tree.export_graphviz(clf, out_file=dot_data) graph = pydot.graph_from_dot_data(dot_data.getvalue()) Image(graph[0].create_png()) ``` Without any parameter tuning we see an accuracy of 94.9%, not too bad! The decision tree classifier in sklearn has an exhaustive set of parameters which allow for maximum control over your classifier. These parameters include: criterion for evaluating a split (this blog post talked about using **entropy** to calculate information gain, however, you can also use something known as **Gini** impurity), maximum tree depth, minimum number of samples required at a leaf node, and many more. ## Regression Regression models, in the general sense, are able to take variable inputs and predict an output from a continuous range. However, decision tree regressions are not capable of producing continuous output. Rather, these models are trained on a set of examples with outputs that lie in a continuous range. These training examples are partitioned in the decision tree and new examples that end in a given node will take on the mean of the training example values that reside in the same node. It looks something like this. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/download--1-.png) Alright, but how do we get there? As I mentioned before, the general process is very similar to a decision tree classifier with a few small changes. #### Determining the optimal split We'll still build our tree recursively, making splits on the data as we go, but we need a new method for determining the optimal split. For **classification**, we used information entropy (you can also use the Gini index or Chi-square method) to figure out which split provided the biggest gain in information about the new example's class. For **regression**, we're not trying to predict a class, but rather we're expected the generate an output given the input criterion. Thus, we'll need a new method for determining optimal fit. One way to do this is to measure whether or not a split will result in a reduction of **variance** within the data. If a split is useful, its combined weighted variance of the child nodes will be less than the original variance of the parent node. We can continue to make recursive splits on our dataset until we've effectively reduced the overall variance below a certain threshold, or upon reaching another stopping parameter (such as reaching a defined maximum depth). Notice how the mean squared error decreases as you step through the decision tree. #### Avoiding overfitting The techniques for preventing overfitting remain largely the same as for decision tree classifiers. However, it seems that not many people actually take the time to prune a decision tree for regression, but rather they elect to use a random forest regressor (a collection of decision trees) which are less prone to overfitting and perform better than a single optimized tree. The common argument for using a decision tree over a random forest is that decision trees are easier to interpret, you simply look at the decision tree logic. However, in a random forest, you're not going to want to study the decision tree logic of 500 different trees. Luckily for us, there are [still ways](https://www.researchgate.net/publication/272742180%5FRule%5FExtraction%5Ffrom%5FRandom%5FForest%5Fthe%5FRFHC%5FMethods?ref=jeremyjordan.me) to maintain interpretability within a random forest without studying each tree manually. #### Implementation Here's the code I used to generate the graphic above. ```python from sklearn import datasets # Load the diabetes dataset diabetes = datasets.load_diabetes() import pandas as pd features = pd.DataFrame(diabetes.data, columns = ["feature_1", "feature_2", "feature_3", "feature_4", "feature_5", "feature_6", "feature_7", "feature_8", "feature_9", "feature_10"]) target = pd.DataFrame(diabetes.target) from sklearn.tree import DecisionTreeRegressor regressor = DecisionTreeRegressor(max_depth = 4) regressor.fit(features, target) from IPython.display import Image from sklearn.externals.six import StringIO import pydot from sklearn import tree dot_data = StringIO() tree.export_graphviz(regressor, out_file=dot_data, feature_names=features.columns, class_names=target.columns, filled=True, rounded=True, special_characters=True) graph = pydot.graph_from_dot_data(dot_data.getvalue()) Image(graph[0].create_png()) ``` ## Further reading [Brandon Rohrer: How decision trees work](https://brohrer.github.io/how%5Fdecision%5Ftrees%5Fwork.html?ref=jeremyjordan.me) [An incredible interactive visualization of decision trees](http://www.r2d3.us/visual-intro-to-machine-learning-part-1/?ref=jeremyjordan.me) ### Decision trees for regression. URL: https://www.jeremyjordan.me/decision-trees-for-regression/ Last updated: 2017-06-24T20:13:27.000Z Before you read this post, go ahead and check out my post on [decision trees for classification](https://www.jeremyjordan.me/decision-trees-for-classification/). This post will be building on top of that, as you'll see that decision tree regressors are ***very*** similar to decision tree classifiers. Regression models, in the general sense, are able to take variable inputs and predict an output from a continuous range. However, decision tree regressions are not capable of producing continuous output. Rather, these models are trained on a set of examples with outputs that lie in a continuous range. These training examples are partitioned in the decision tree and new examples that end in a given node will take on the mean of the training example values that reside in the same node. It looks something like this. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/download--1-.png) Alright, but how do we get there? As I mentioned before, the general process is very similar to a decision tree classifier with a few small changes. ### Determining the optimal split We'll still build our tree recursively, making splits on the data as we go, but we need a new method for determining the optimal split. For **classification**, we used information entropy (you can also use the Gini index or Chi-square method) to figure out which split provided the biggest gain in information about the new example's class. For **regression**, we're not trying to predict a class, but rather we're expected the generate an output given the input criterion. Thus, we'll need a new method for determining optimal fit. One way to do this is to measure whether or not a split will result in a reduction of **variance** within the data. If a split is useful, its combined weighted variance of the child nodes will be less than the original variance of the parent node. We can continue to make recursive splits on our dataset until we've effectively reduced the overall variance below a certain threshold, or upon reaching another stopping parameter (such as reaching a defined maximum depth). Notice how the mean squared error decreases as you step through the decision tree. ### Avoiding overfitting The techniques for preventing overfitting remain largely the same as for decision tree classifiers. However, it seems that not many people actually take the time to prune a decision tree for regression, but rather they elect to use a random forest regressor (a collection of decision trees) which are less prone to overfitting and perform better than a single optimized tree. The common argument for using a decision tree over a random forest is that decision trees are easier to interpret, you simply look at the decision tree logic. However, in a random forest, you're not going to want to study the decision tree logic of 500 different trees. Luckily for us, there are [still ways](https://www.researchgate.net/publication/272742180%5FRule%5FExtraction%5Ffrom%5FRandom%5FForest%5Fthe%5FRFHC%5FMethods?ref=jeremyjordan.me) to maintain interpretability within a random forest without studying each tree manually. ### Implementation Here's the code I used to generate the graphic above. ``` from sklearn import datasets # Load the diabetes dataset diabetes = datasets.load_diabetes() import pandas as pd features = pd.DataFrame(diabetes.data, columns = ["feature_1", "feature_2", "feature_3", "feature_4", "feature_5", "feature_6", "feature_7", "feature_8", "feature_9", "feature_10"]) target = pd.DataFrame(diabetes.target) from sklearn.tree import DecisionTreeRegressor regressor = DecisionTreeRegressor(max_depth = 4) regressor.fit(features, target) from IPython.display import Image from sklearn.externals.six import StringIO import pydot from sklearn import tree dot_data = StringIO() tree.export_graphviz(regressor, out_file=dot_data, feature_names=features.columns, class_names=target.columns, filled=True, rounded=True, special_characters=True) graph = pydot.graph_from_dot_data(dot_data.getvalue()) Image(graph[0].create_png()) ``` ### Polynomial regression. URL: https://www.jeremyjordan.me/polynomial-regression/ Last updated: 2025-03-03T02:17:09.000Z Polynomial regression is very similar to [linear regression](https://www.jeremyjordan.me/linear-regression/), with a slight deviation in how we treat our feature-space. Confused? It'll make more sense in a minute, just bear with me. As a reminder, linear regression models are composed of a linear combination of inputs and weights. $${h \_\\theta }\\left( x \\right) = {\\theta \_0} + {\\theta \_1}{x \_1} + {\\theta \_2}{x \_2} + {\\theta \_3}{x \_3} + ... + {\\theta \_n}{x \_n}$$ Polynomial regression is very similar, but it allows for a linear combination of an input variable raised to varying degrees. $${h \_\\theta }\\left( x \\right) = {\\theta \_0} + {\\theta \_1}{x \_1} + {\\theta \_2}x \_1^2 + {\\theta \_3}x \_1^3 + ... + {\\theta \_n}x \_1^n$$ Notice the difference? What if I told you that those were the exact same model? Don't believe me? Well, it's not always the case but suppose $x \_2 = x \_1^2$, $x \_3 = x \_1^3$, and so forth all the way to $x \_n = x \_1^n$. Clever right? So we can actually use linear regression models to perform polynomial regression by simply creating new features. Polynomial regression is useful as it allows us to fit a model to nonlinear trends. To do this in scikit-learn is quite simple. First, let's create a fake dataset to work with. I've used sklearn's `make_regression` function and then squared the output to create a nonlinear dataset. ``` from sklearn.datasets import make_regression X, y = make_regression(n_samples = 300, n_features=1, noise=8, bias=2) y2 = y**2 ``` Next, let's fit a linear model to the data and see how it performs. ``` from sklearn.linear_model import LinearRegression model = LinearRegression() model.fit(X, y2) plt.plot(X, model.predict(X)) plt.scatter(X, y2) plt.title("Linear Model, Polynomial Degree = 1") ``` ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/download--5-.png) Not too impressive... However, if we create a couple polynomial features we can see a great improvement in our model's fit. ``` import numpy as np from sklearn.preprocessing import PolynomialFeatures poly_features = PolynomialFeatures(degree = 3) X_poly = poly_features.fit_transform(X) poly_model = LinearRegression() poly_model.fit(X_poly, y2) pred = poly_model.predict(X_poly) new_X, new_y = zip(*sorted(zip(X, pred))) # sort values for plotting plt.plot(new_X, new_y) plt.scatter(X,y2) plt.title("Polynomial Degree = 3") ``` ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/download--7-.png) A few practical notes on polynomial regression. - Be careful of blindly using high-order polynomial models. These functions are generally not well-behaved and will produce dramatic unwanted fluctuations. Remember to use model evaluation techniques such as cross-validation to make sure your model generalizes well. - Using a trick called **regularization** can prevent the aforementioned tendency for higher-order polynomial models to overfit the data in wild ways. ### Preparing data for a machine learning model. URL: https://www.jeremyjordan.me/preparing-data-for-a-machine-learning-model/ Last updated: 2019-11-02T19:30:22.000Z Before you're ready to feed a dataset into your machine learning model of choice, it's important to do some preprocessing so the data behaves nicely for our model. In an ideal world, you'll have a perfectly clean dataset with no errors or missing values present. In the real world, however, such datasets are rare and uncommon. In this post, I'll be cleaning a dataset from Kaggle on [Mental Health in Tech](https://www.kaggle.com/osmi/mental-health-in-tech-survey?ref=jeremyjordan.me). I took a look at this dataset with my friend Florian (he's awesome, check him out [here](https://www.linkedin.com/in/florian-kaiser/?ref=jeremyjordan.me)) earlier this year and decided it would serve as a good example for this post. ### Inspecting the data It's hard to know what to do if you don't know what you're working with, so let's load our dataset and take a peek. You can follow along in a Jupyter Notebook if you'd like. ``` import pandas as pd import numpy as np df = pd.read_csv("survey.csv") num_columns = len(df.columns) pd.set_option("display.max_columns", num_columns) df.head(30) ``` The pandas `head()` function returns the first 5 rows of your dataframe by default, but I wanted to see a bit more to get a better idea of the dataset. While we're at it, let's take a look at the shape of the dataframe too. Running `df.shape` will return information about the dimensionality of our dataframe (in this case it's the number of rows and columns), which will essentially tell you how many examples and features you are working with. *Pro tip: you can return the number of observations in your dataset with `df.shape[0]` and the number of features in your dataset with `df.shape[1]`. This is often helpful to do when building a model.* The first thing to understand is what features our dataset contains. You can read a description of each feature for this dataset on [Kaggle](https://www.kaggle.com/osmi/mental-health-in-tech-survey?ref=jeremyjordan.me). Rather than inspecting each feature one-by-one, I opted for the lazier route and ran the following: ``` feature_names = df.columns.tolist() feature_names.remove('Timestamp') feature_names.remove('comments') for column in feature_names: print column print df[column].value_counts(dropna=False) ``` This printed out a cell in my notebook with a ton of information about my dataset that I could easily consume by just scrolling through. As I looked through the dataset, I kept an eye out for things just as duplicate labels, erroneous or null values, and other things that just didn't quite seem right. While many columns were fine as is, a couple columns needed cleaning. Here's a list of things I found that needed attention before feeding this model into a machine learning algorithm. - The `Age` column contained people who had not been born yet (negative numbers). - The `Age` column also contained children (such as a 5 year old) that are unlikely to be taking a survey about their workplace. - The `Age` column contained someone who is 99999999999 years old, they should be in the Guiness World Records book or something. Jokes aside, this value is erroneous. - There are 49 different values listed for `Gender`. Moreso, "Male" and "male" are ostensibly the same but currently being treated as two distinct categories. - `self_employed ` and `work_interfere` contain some null values. For numerical features, it can also be helpful to quickly examine any possible correlations between features. You can use the pandas [scatter\_matrix](https://pandas.pydata.org/pandas-docs/stable/visualization.html?ref=jeremyjordan.me#scatter-matrix-plot) to easily visualize your data. ### Separate the features and labels If we're using a supervised machine learning technique, we need to make a distinction in the data between features and labels for each observation. Ultimately, this depends on what you're looking to predict or classify. In this case, I've decided that I'd like to build a classifier to predict whether or not someone will seek treatment. ``` features = df.drop('treatment', 1) labels = df['treatment'] ``` ### Cleaning null values (missing data) Let's start with figuring out what to do with the null values found in `self_employed ` and `work_interfere`. In both of these cases, the column contains categorical data. This is important to note the distinction between treating null values for categorical and numerical data, as the treatment will depend on the type of data present. There's not exactly a formulaic way to approach how to treat missing data, as the treatment largely depends on the context and the nature of the data. For example, are the data missing at random or is there a hidden relationship between missing data and some other predictor? One option in dealing with missing data is to simply **ignore or remove** the rows in which data is missing, discarding them from our analysis. However, this method is often unpopular due to the loss of information - especially if there was a pattern in your missing data. A more method for handling missing values is **imputation**, where we replace the null value with our best guess of what that value might be. Basic implementations will simply replace all missing values with the mean/median/mode of all of the values for the given feature. More advanced implementations will use other features in order to develop an estimator for your missing value. Take the following example, which is a small dataset containing three features (weather condition, temperature, and humidity) to predict whether or not I am likely to play tennis. | | Weather | Temperature | Humidity | Play tennis? | | - | ------- | ----------- | -------- | ------------ | | 1 | cloudy | 60 | NaN | yes | | 2 | rainy | 75 | 80% | NaN | | 3 | cloudy | NaN | 50% | no | | 4 | sunny | 65 | 40% | yes | If we deleted all of the rows with null values we'd be left with one row, and our predictor would always guess that I would play tennis because that's the only example it has to learn from. Let's say we instead chose to replace the null temperature value in row 3 by mean imputation. In this case, row 3 temperature would be artificially reported to be 65\. If we wanted to get fancier for replacing the null value for humidity in row 1, we could train an estimator to guess what the value might be given the weather and temperature data provided. It's important to note that with data imputation you are artifically reducing the variation in your dataset by creating new values close to (or equal to) the mean. For a more in-depth analysis of techniques for treating missing data values, check out [this chapter](http://www.stat.columbia.edu/~gelman/arm/missing.pdf?ref=jeremyjordan.me) on missing-data imputation. Scikit-learn provides an imputer implementation for dealing with missing data, as shown in the example below. ``` from sklearn.preprocessing import Imputer imputer = Imputer(missing_values = 'NaN', strategy = 'mean', axis = 0) imputer.fit(features) features = imputer.transform(features) ``` ### Finding duplicate names I mentioned earlier that there were 49 different values for 'Gender' and I suspected that some of these values should not be treated as distinct categories. Ultimately, I decided to resegment the data into 3 categories: male, female, and other (which accounts for non-binary and transgendered persons). If I wanted to build a preprocessing engine which could clean incoming data, I could have probably taken a fancier approach and used some clever regular expressions to recategorize this column. Alas, I opted for the quick and dirty approach. ``` male_terms = ["male", "m", "mal", "msle", "malr", "mail", "make", "cis male", "man", "maile", "male (cis)", "cis man"] female_terms = ["female", "f", "woman", "femake", "femaile", "femake", "cis female", "cis-female/femme", "female (cis)", "femail", "cis woman"] def clean_gender(response): if response.lower().rstrip() in male_terms: return "Male" elif response.lower().rstrip() in female_terms: return "Female" else: return "Other" df['Gender'] = df["Gender"].apply(lambda x: clean_gender(x)) ``` ### Outlier detection As I mentioned earlier, it seemed like there were some outlier values for `Age` which appear to be erroneous. Examples such as negative ages or extremely large integers could negatively affect the performance of our machine learning algorithm and we'll need to address them. I went ahead and defined an acceptable range of ages for adults in the workplace and replaced numbers outside of this range with null values. ``` df.Age.loc[(df.Age < 18) | (df.Age > 80)] = np.nan ``` These null values could then be treated using the sklearn Imputer described earlier. After defining a working range, I wanted to visualize the distribution of ages present in this dataset. ``` %matplotlib inline import seaborn as sns sns.set(color_codes=True) plot = sns.distplot(df.Age.dropna()) plot.figure.set_size_inches(6,6) ``` *Note: pandas has plotting capabilities built-in that you can use, but I'm a fan of [seaborn](https://seaborn.pydata.org/?ref=jeremyjordan.me) and wanted to show that you have options when it comes to visualizing data.* Alright, at this point is seems like we're working with a pretty clean dataset. However, we're still not ready to pass our data into a machine learning model. ### Encoding data Many machine learning algorithms expect numerical input data, so we need to figure out a way to represent our categorical data in a numerical fashion. One solution to this would be to arbitrarily assign a numerical value for each category and map the dataset from the original categories to each corresponding number. For example, let's look at the 'leave' column (How easy is it for you to take medical leave for a mental health condition?) in our dataset ``` df['leave'].value_counts(dropna=False) ``` which returns the following values. ``` Don't know 563 Somewhat easy 266 Very easy 206 Somewhat difficult 126 Very difficult 98 Name: leave, dtype: int64 ``` In order to encode this data, we could map each value to a number. ``` df['leave'] = df['leave'].map({'Very difficult': 0, 'Somewhat difficult': 1, 'Don\'t know': 2, 'Somewhat easy': 3, 'Very easy': 4}) ``` This process is known as label encoding, and sklearn conveniently will do this for you. ``` from sklearn import preprocessing label_encoder = preprocessing.LabelEncoder() label_encoder.fit(df['leave']) label_encoder.transform(df['leave']) ``` The problem with this approach is that you're introducing an order that might not be present. In this case, one could argue that the data is ordinal ("very difficult" is less than "somewhat difficult" which is less than "somewhat easy" which is less than "very easy") but many times categorical data has no order. For example, if you had a feature for animal species, it doesn't make sense to assign values such than cats are less than dogs. The danger in label encoding is that your machine learning algorithm may learn to favor dogs over cats due to artficial ordinal values you introduced during encoding. The common solution for encoding nominal data is one-hot encoding. Rather than replacing the categorical value with a numerical value (label encoding) as shown below | | species | numerical encoding | | - | ------- | ------------------ | | 1 | cat | 1 | | 2 | dog | 2 | | 3 | snake | 3 | | 4 | cat | 1 | | 5 | dog | 2 | | 6 | turtle | 4 | | 7 | dog | 2 | we instead create a column for each value and use 1's and 0's to denote expression of each value. These new columns are often referred to as dummy variables. | | species | is\_cat | is\_dog | is\_snake | is\_turtle | | - | ------- | ------- | ------- | --------- | ---------- | | 1 | cat | 1 | 0 | 0 | 0 | | 2 | dog | 0 | 1 | 0 | 0 | | 3 | snake | 0 | 0 | 1 | 0 | | 4 | cat | 1 | 0 | 0 | 0 | | 5 | dog | 0 | 1 | 0 | 0 | | 6 | turtle | 0 | 0 | 0 | 1 | | 7 | dog | 0 | 1 | 0 | 0 | You can one-hot encoding in directly in Pandas or use sklearn, although sklearn is a bit more convulted as the one-hot encoder only works for integer value inputs. For our example (where the inputs are strings), we would need to first perform label encoding and then one-hot encode the data. ``` # Using Pandas import pandas as pd pd.get_dummies(features['leave']) # Using sklearn from sklearn.preprocessing import LabelEncoder, OneHotEncoder label_encoder = LabelEncoder() ohe = OneHotEncoder(categorical_features = ['leave']) label_encoded_data = label_encoder.fit_transform(features['leave']) ohe.fit_transform(label_encoded_data.reshape(-1,1)) ``` ### Feature scaling At this point, we've successfully cleaned our data and converted it into a form which is easily consumable by machine learning algorithms. However, at this point we should consider whether or not some method of data normalization will be beneficial for our algorithm. This depends on (1) the data, (2) what we're trying to learn, and (3) the algorithm we plan to implement. Sometimes you'll need to revisit this step if you decide to change your machine learning algorithm. ###### The data Suppose you have a dataset of features with different units: temperature in Kelvin, percent relative humidity, and day of year. If this dataset is representing weather values in Raleigh we might see the following ranges for each feature. - Temperature: 270 K to 305 K - Humidity: 0 to 1 (ie. 30% humidity represented by 0.3) - Day of year: 0 to 365 When you're interpreting these values, you intuitively normalize values as you're thinking about them. For example, you recognize that an increase of 0.5 (remember: that's 50%) for humidity is much more drastic of a change than an increase of 0.5 Kelvin. However, if we don't scale these features our algorithm might learn to use temperature as the main predictor simply because it's scale is largest (and thus changes in temperature values are most significant to the algorithm). Feature scaling allows for all features to contribute equally (or more aptly, it allows for features to contribute relative to their importance rather than their scale). ###### What we're trying to learn Sometimes, we'd like to be able to interpret our coefficients and changing the scale from something real (ie. dollars) to something arbitrary makes it more difficult to interpert our results. ###### The algorithm If you're using a tool such as gradient descent to optimize your algorithm, feature scaling allows for a consistent updating of weights across all dimensions. The following sketchs are borrowed from Andrew Ng's slides on gradient descent. The first image represents two features with different scales while the latter represents a normalized feature space. The gradient descent optimization may take a long time to search for a local optimum as it goes back and forth along the contours. A normalized feature space allows for much quicker convergence. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/GeAcX.png) ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/30hnH.png) ML algorithms which require feature scaling: - Logistic regression - SVMs - Perceptrons - Neural networks - PCA ML algorithms which do not require feature scaling: - Decision trees (and random forests) - Naive Bayes *Note: the above lists are by no means exhaustive but simply serve as an example.* There's a few different methods for scaling features to a standardized range. Min-max scalers and standard scalers are two of the most commonly used approaches. Min-max scaling sets your lowest observed value to 0 and your highest observed value to 1. \[{X\_{sc}} = \\frac{{X - {X\_{\\min }}}}{{{X\_{\\max }} - {X\_{\\min }}}}\] Standard scaling will convert the data to have a standard normal distrubtion. \[z = \\frac{{X - \\mu }}{\\sigma }\] To do this (feature scaling) in practice, you can use pre-built libraries in sklearn. ``` # Feature scaling with StandardScaler from sklearn.preprocessing import StandardScaler scale_features_std = StandardScaler() features_train = scale_features_std.fit_transform(features_train) features_test = scale_features_std.transform(features_test) # Feature scaling with MinMaxScaler from sklearn.preprocessing import MinMaxScaler scale_features_mm = MinMaxScaler() features_train = scale_features_mm.fit_transform(features_train) features_test = scale_features_mm.transform(features_test) ``` A few notes on this implementation: - In practice, you may decide to only scale certain columns. For example, you don't need to scale the dummy variables from one-hot encoding. - You'll want to scale your features the same for both the training and test sets (I'm getting ahead of myself here, stay tuned for the next section), so be sure to only transform the test set and not refit the scaler. - I provided two ways to scale your feature set in the code above, choose one. ### Splitting the data for training and testing One of the last things we'll need to do in order to prepare out data for a machine learning algorithm is to split the data into training and testing subsets. Remember, machine learning is all about teaching computers to perform a task by showing it a lot of examples. We'll teach the computer using the data we have available, but ideally the algorithm will work just as well with new data. The principle behind the train/test split is to sacrifice some of the data for training in an attempt to evaluate the algorithm on "new" data it hasn't already seen. This is important because we want to ensure that our algorithm is able to generalize its learning from the given examples. A nice analogy I heard once relates the concept to studying for a test. Let's say we have two students who are preparing for the same test- one student makes an effort to understand the general concepts that will be tested while the other student simply memorizes every example from the relevant homework sets. If the test contains new problems that weren't on the homework but are still related to the concepts discussed, the former student will perform much better on the test than the student who memorized the homework answers. In machine learning, we use the term **overfitting** to describe whether or not an algorithm has read too much into the data we provided as examples, or whether it was capable of generalizing the concepts we were trying to teach it. Let's go ahead and split the data into two subsets (really it's four subsets, since we already separated features from labels). ``` from sklearn.model_selection import train_test_split features_train, features_test, labels_train, labels_test = train_test_split(features, labels, test_size=0.25, random_state = 0) ``` *Note: this will also shuffle your data. This is important if your data is ordered.* ###### Validation data When constructing a machine learning model, we often split the data into three subsets: train, validation, and test subsets. The training data is used to "teach" the model, the validation data is used to [search for the best model architecture](https://www.jeremyjordan.me/hyper-parameter-tuning/), and the test data is reserved as an unbiased evaluator of our model. When building a model, we often are presented with choices regarding the model's overall design; the validation data allows us to evaluate multiple designs in search of the best design, but in doing so we're "fitting" our model's design to this subset. Thus, the test data is still useful in determining how well our model will generalize what it learned to new data. For machine learning problems with limited data, it's desirable to maximize the amount of data used to actually train your model. However, you must still hold out sufficient data for the validation set to evaluate model performance. In such situations, K-folds cross validation may come in handy. For K-folds cross validation, we partition the dataset into $K$ bins. We will then run $K$ different learning experiments, each time choosing one bin as the evaluation subset while using the rest to train our algorithm. We can then average the results from all of the experiments to get an accurate picture of it's performance, using the entire dataset (albeit not at the same time) to both train and test the performance of our algorithm. Obviously, this approach will take more time, but it will give a more accurate evaluation of our machine learning model. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/k-fold.png) [*Image credit*](https://sebastianraschka.com/faq/docs/evaluate-a-model.html?ref=jeremyjordan.me) ###### Verifying consistent distributions across subsets You'll get the best results if you're training and evaluating your data on data that has a similar distribution. Thus, it's best to verify this when splitting your data into subsets. ### How much data is enough data? Another useful thing to do at this stage is plot **learning curves** to judge whether or not you've collected sufficient data. These learning curves plot the training and validation scores of your algorithm for varying numbers of training examples. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/sphx_glr_plot_learning_curve_0011.png) [*Image credit*](http://scikit-learn.org/stable/modules/generated/sklearn.model%5Fselection.learning%5Fcurve.html?ref=jeremyjordan.me) You will see that your model will plateau at a certain point and you'll begin to experience diminishing returns from adding additional training examples. This is primarily useful as it will tell you that you either need to go collect more data or that you've collected sufficient data and you shouldn't spend more time growing your dataset. For supervised learning, it can often be expensive to collect and label additional data for use in your model. *Note: this is where deep learning shines! Deep learning techniques are typically more capable of continuing to learn and improve the more data you feed it.* An example of sklearn's learning curve implementation can be found [here](https://github.com/scikit-learn/scikit-learn/blob/master/examples/model%5Fselection/plot%5Flearning%5Fcurve.py?ref=jeremyjordan.me). ### The curse of dimensionality Lastly, I'd like to briefly point out that simply throwing any and all information present at our machine learning algorithm generally isn't the best idea. This is commonly referred to as the curse of dimensionality, which observes that as the number of features in our dataset grow, the amount of data we need to accurately generalize grows exponentially. ### Further reading [Combining features from different sources](http://scikit-learn.org/stable/auto%5Fexamples/hetero%5Ffeature%5Funion.html?ref=jeremyjordan.me) [How to Handle Missing Data](https://towardsdatascience.com/how-to-handle-missing-data-8646b18db0d4?ref=jeremyjordan.me) [Kaggle course on feature engineering](https://www.kaggle.com/learn/feature-engineering?ref=jeremyjordan.me) ### Machine learning overview. URL: https://www.jeremyjordan.me/machine-learning-overview/ Last updated: 2017-10-18T01:45:39.000Z *Preface: this post is for the complete novice in the field of machine learning in an attempt to orient readers for my other posts on machine learning. This is a high-level view of machine learning.* ### What the heck is machine learning? The whole concept of machine learning is figuring out ways in which we can teach a computer to perform a task without needing to provide explicit instructions. Another way to think about it is that we're trying to "program" intuition in a computer. You and I can look at an email and easily discern whether or not it's spam, but how do you get a computer to do such a task? You could construct a huge convoluted logic infrastructure of "if.. then.." statements to sort out the spam emails, but it would be a pain to construct and probably wouldn't work too well. Instead, the machine learning approach is to equip the computer with skills to learn on its own and feed it a bunch of examples. Machine learning is exploding as a field right now as people are realizing a multitude of tasks that we can teach computers to perform by feeding it large datasets. ### The general machine learning framework Machine learning isn't one singular tool, it's an entire field of tools - each with their own strengths and weaknesses. The way we allow computers to learn is by providing it with a model which serves as the learning framework. The image below is a guide from scikit-learn (a collection of Python implementations of various machine learning algorithms and functions) on selecting the proper model for your machine learning application. Part of being a machine learning expert is knowing which tool to use! ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/05/ml_map.png) [*Image credit*](http://scikit-learn.org/stable/tutorial/machine%5Flearning%5Fmap/?ref=jeremyjordan.me) Once you've selected your model, you typically follow the same general procedure. 1. Preprocess your data so that it will feed properly into your model. 2. Construct your model. 3. Train your model on a dataset and tune all relevant parameters for optimal performance. 4. Evaluate your model and determine its usefulness. I'll be writing in more detail about each of these steps in following blog posts. So as a reminder, rather than explicitly programming the computer to complete a task, we're providing a ton of examples (data) for the computer to learn from with the goal of completing some given task. There are times when we tell the computer what we want it to do (by training it on a dataset with predefined outputs) and there are times when we simply hand over a dataset to the computer and ask it to discover something on its own. When we train a computer by providing it with data that has predefined outputs, we call this **supervised learning**. An example of this would be training a spam filter for your email by giving the algorithm a huge set of example emails which are labeled as "spam" or "not spam". When we simply hand over a dataset to the computer and ask it to find something interesting, this is known as **unsupervised learning**, since we're not telling the computer what output we'd like to see. Unsupervised machine learning techniques are ideal because it takes time to label datasets for training supervised models. There's also such thing as **semi-supervised learning** where we leverage a small dataset of labeled observations combined with a much larger dataset of unlabeled observations to use in training a model to perform some task. Machine learning algorithms are capable of tasks such as **classifying** objects found in photos, predicting the optimal price to sell your house with **regression**, customer and market segmentation using **clustering techniques**, and much much more. Pretty cool, huh? ### Further reading Bonus: [A Massive Google Network Learns To Identify — Cats](http://www.npr.org/2012/06/26/155792609/a-massive-google-network-learns-to-identify?ref=jeremyjordan.me) For a detailed look at the impact of machine learning on businesses, check out the following [article](https://hbr.org/cover-story/2017/07/the-business-of-artificial-intelligence?ref=jeremyjordan.me) from Harvard Business Review. [The Great A.I. Awakening](https://www.nytimes.com/2016/12/14/magazine/the-great-ai-awakening.html?ref=jeremyjordan.me): How Google used artificial intelligence to transform Google Translate, one of its more popular services — and how machine learning is poised to reinvent computing itself. [A great visual introduction to machine learning.](http://www.r2d3.us/visual-intro-to-machine-learning-part-1/?ref=jeremyjordan.me) [Machine learning glossary](https://developers.google.com/machine-learning/glossary/?ref=jeremyjordan.me) ### Linear regression. URL: https://www.jeremyjordan.me/linear-regression/ Last updated: 2025-03-03T01:50:52.000Z Linear regression is used to predict an outcome given some input value(s). While machine learning **classifiers** use ***features*** to predict a ***discrete label*** for a given instance or example, machine learning **regressors** have the ability use ***features*** to predict a ***continuous outcome*** for a given instance or example. For example, a classifier might draw a decision boundary that can tell you whether or not a house is likely to sell at a given price (when provided with features of the house) but a regressor can use those same features to predict the market value of the house. Nonetheless, regression is still a supervised learning technique and you'll still need to train your model on a set of examples with known outcomes. The basic premise behind linear regression is to provide a model which can observe linear trends in the data; you've probably encountered linear regression at some point in the past as finding the "line of best fit". ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/Screen-Shot-2017-06-04-at-1.46.51-PM.png) ### The model Our machine learning model is represented by the equation of a line. $${h\_\\theta }\\left( x \\right) = {\\theta \_0} + {\\theta \_1}x$$ In this case, we define our model as ${h \_\\theta }\\left( x \\right)$ where $h$ represents our predicted outcome as a function of our features, $x$. Typically, the model of a line is represented by $y=mx+b$. We use $h$ instead of $y$ to denote that this model is a hypothesis of the true trend. In a sense, we're attempting to provide our "best guess" for finding a relationship in the data provided. $\\theta \_0$ and $\\theta \_1$ are coefficients of our linear equation (analogous to $b$ and $m$ in the standard line equation) and will be considered parameters of the model. So far, our model has only been capable of capturing one independent variable and mapping its relationship to one dependent variable. We'll now extend our linear model to allow for $n$ different independent variables (in our case, features) to predict one dependent variable (an outcome). Thus, for each input $x$ we'll have a vector of values x\=\[ 1 x 1 x 2 ⋮ x n \] each with corresponding coefficients, $\\theta$. \[ θ 0 θ 1 θ 2 ⋮ θ n \] *Note: $x \_0 = 1$ by convention. This allows us to do matrix operations with $x$ and $\\theta$ without deviating from our defined linear model.* Extending this model to account for multiple inputs, we can write it as $${h \_\\theta }\\left( x \\right) = {\\theta \_0} + {\\theta \_1}{x \_1} + {\\theta \_2}{x \_2} + {\\theta \_3}{x \_3} + ... + {\\theta \_n}{x \_n}$$ or more succinctly as $${h\_\\theta }\\left( x \\right) = {\\theta ^T}x$$ when dealing with one example or $${h \_\\theta }\\left( x \\right) = X\\theta $$ when dealing with $i$ examples. *Note: this matrix, $X$, is also referred to as the *design matrix* and is usually capitalized by convention to denote it as a matrix as opposed to a vector.* If that doesn't quite make sense, let's take a step back and walk through the representation. For a univariate model, we could write the model in matrix form as: h θ ( x )\=\[ θ 0 θ 1 \]\[ 1 x 1 \]\= θ 0 + θ 1 x 1 Go ahead and do the matrix multiplication to convince yourself. Now that you're convinced, let's extend this to a multivariate model with $j$ features and $i$ examples. h θ ( x i )\=\[ 1 x 1,1 x 1,2 … 1 x 2,1 x 2,2 … 1 ⋮ ⋮ ⋮ 1 x i,1 x i,2 ⋯ x 1,j x 2,j ⋮ x i,j \]\[ θ 0 θ 1 θ 2 ⋮ θ j \]\=\[ y 1 y 2 y 3 ⋮ y n \] This is pretty useful because we can go ahead and calculate the prediction, ${{h \_\\theta }\\left( {{x \_i}} \\right)}$, for every example in our data set using one matrix operation. ### The cost function In order to find the optimal line to represent our data, we need to develop a cost function which favors model parameters that lead to a better fit. This cost function will approximate the error found in our model, which we can then use to optimize our parameters, $\\theta \_0$ and $\\theta \_1$, by minimizing the cost function. For linear regression, we'll use the mean squared error - the average difference between our guess, ${{h \_\\theta }\\left( {{x \_i}} \\right)}$, and the true value, ${{y \_i}}$ - to measure performance. $$\\min \\frac{1}{{2m}}\\sum\\limits\_{i = 1}^m {{{\\left( {{h \_\\theta }\\left( {{x \_i}} \\right) - {y \_i}} \\right)}^2}} $$ ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/Screen-Shot-2017-06-04-at-1.47.01-PM.png) Substituting in our definition of the model, ${{h \_\\theta }\\left( {{x \_i}} \\right)}$, we'll update our cost function as: $$J\\left( \\theta \\right) = \\frac{1}{{2m}}\\sum\\limits \_{i = 1}^m {{{\\left( {\\left( {\\sum\\limits \_{j = 0}^n {{\\theta \_j}{x \_{i,j}}} } \\right) - {y \_i}} \\right)}^2}}$$ ### Finding the best parameters There's a number of ways to find the optimal model parameters. A few common approaches are detailed below. ###### Gradient descent We use gradient descent to perform the parameter optimization in accordance with our cost function. Remember, the cost function essentially will score how well our model is fitting the present data. A smaller cost function means less error, which means we have a better model. We first start with an initial guess for what our parameter values should be. Gradient descent will then guide us through an iterative process of updating our parameter weights in search of an optimum solution. Let's first look at the simplest example, where we fix $\\theta \_0 = 0$ and find the optimal slope value. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/linreg1d.png) [*Image credit*](http://scipython.com/blog/visualizing-the-gradient-descent-method/?ref=jeremyjordan.me) Using gradient descent, the weights will continue to update until we've converged on the optimal value. For linear regression, the cost function is always convex with only one optimum, so we can be confident that our solution is the best possible one. This is not always the case, and we'll need to be weary of converging on local optima using other cost functions. As the picture depicts, gradient descent is an iterative process which guides the parameter values towards the global minimum of our cost function. $${\\theta \_j}: = {\\theta \_j} - \\eta \\frac{\\partial }{{\\partial {\\theta \_j}}}J({\\theta \_0},{\\theta \_1})$$ Gradient descent will update your parameter value by subtracting the current value by the slope of the cost function at that point multiplied by a learning rate, which basically dictates how far to step. The learning rate, $\\eta$, is always positive. Notice in the picture above how the first attempted value, $\\theta \_1 = 0$, has a negative slope on the cost function. Thus, we would update the current value by subtracting a negative number (in other words, addition) multiplied by the learning rate. In fact, every point left of the global minimum has a negative slope and would thus result in increasing our parameter weight. On the flip side, every value to the right of the global minimum has a positive slope and would thus result in decreasing our parameter weight until convergence. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/Untitled.png) One neat thing about this process is that gradient descent will automatically slow down as we're nearing the optimum. See how the initial iterations take relatively large steps in comparison to the last steps? This is because our slope begins to level off as we approach the optimum, resulting in smaller update steps. Because of this, there is no need to adjust $\\eta$ during the optimization. We can extend this idea to update both $\\theta \_0$ and $\\theta \_1$ simultaneously. In this case, our 2D curve has become a 3D contour plot. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/06/linreg2d.png) While the gradient descent algorithm remains largely the same, $${\\theta \_j}: = {\\theta \_j} - \\alpha \\frac{\\partial }{{\\partial {\\theta \_j}}}J\\left( \\theta \\right)$$ we now define the differential as $$\\frac{\\partial }{{\\partial {\\theta \_j}}}J\\left( \\theta \\right) = \\frac{1}{m}\\sum\\limits \_{i = 1}^m {\\left( {\\left( {\\sum\\limits \_{j = 0}^n {{\\theta \_j}{x \_{i,j}}} } \\right) - {y \_i}} \\right)} {x \_{i,j}}$$ using the chain rule. A few notes and practical tips on this process. - Scaling your features (as mentioned in [Preparing data for a machine learning model](https://www.jeremyjordan.me/preparing-data-for-a-machine-learning-model/)) to be on a similar scale can greatly improve the time it takes for your gradient descent algorithm to converge. - It's important that your learning rate is reasonable. If it is too large it may overstep the global optimum and never converge, and if it's too small it will take a very long time to converge. - Plot the $J\\left( \\theta \\right)$ as a function of the number of iterations you have run gradient descent. If gradient descent is working properly it will continue to decrease steadily until leveling off, hopefully, close to zero. You can also use this plot to visually determine when your cost function has converged and gradient descent may be stopped. - If $J\\left( \\theta \\right)$ begins to increase as the number of iterations increases, something has gone wrong in your optimization. Typically, if this occurs it is a sign that your learning rate is too large. Gradient descent is often the preferred method of parameter optimization for models which have a large number (think, over 10,000) of features. ###### Normal equation Remember how we used matrices to represent our model? Using normal equation for linear regression you can solve for the optimal parameters analytically in one step. $$\\theta = {\\left( {{X^T}X} \\right)^{ - 1}}{X^T}y$$ You can find a good derivation for this equation [here](https://dzone.com/articles/derivation-normal-equation?ref=jeremyjordan.me). The benefit of using the normal equation to solve for our model parameters is that feature scaling is not necessary and it can be more effective than gradient descent for small feature sets. A few notes and practical tips on this process. - This method will occasionally give you problems when dealing with noninvertible or degenerate matrices. If this is the case, it is likely that your feature set has redundant (linearly dependent) features, or simply too many features with respect to the number of training examples given. ### Implementation Linear regression is very simple to implement in sklearn. It can accommodate both univariate and multivariate models and will automatically handle one-hot encoded categorical data to avoid the dummy variable trap. ``` from sklearn.linear_model import LinearRegression model = LinearRegression() model.fit(features_train, labels_train) predictions = model.predict(features_test) import matplotlib.pyplot as plt plt.scatter(features, labels, color = 'red') plt.plot(features_test, predictions, color = 'blue') ``` ### Decision trees for classification. URL: https://www.jeremyjordan.me/decision-trees-for-classification/ Last updated: 2017-07-13T23:19:04.000Z In this third post on supervised machine learning classifiers, I'll be talking about one of the oldest and most widely used techniques - decision trees. Decision trees work well with noisy or missing data and are incredibly fast at runtime. They're additionally nice because you can visualize the decision tree that your algorithm created, so it's not as much as a black box as some of the algorithms. However, because you're making a series of splits based on a single attribute at a time, the decision boundary is only capable of making splits parallel to your feature axes. Decision trees are pretty easy to grasp intuitively, let's look at an example. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/03/Screen-Shot-2017-03-11-at-9.39.55-PM.png) *Note: decision trees are used by starting at the top and going down, level by level, according to the defined logic. This is known as recursive binary splitting.* Everybody loves a good meta reference; and yes, I'm sipping tea as I write this post. Now let's look at a two-dimensional feature set and see how to construct a decision tree from data. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/03/Screen-Shot-2017-03-11-at-9.51.20-PM.png) Any ideas on how we could make a decision tree to classify a new data point as "x" or "o"? Here's what I did. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/03/Screen-Shot-2017-03-11-at-10.15.37-PM.png) Run through a few scenarios and see if you agree. Look good? Cool. So now we have a decision tree for this data set; the only problem is that I created it. It'd be much better if we could get a machine to do this for us. But how? If you analyze what we're doing from an abstract perspective, we're taking a subset of the data, and deciding the best manner to split the subset further. Our initial subset was the entire data set, and we split it according to the rule $x\_1 < 3.5$. Then, for each subset, we performed additional splitting until we were able to correctly classify every data point. #### Information gain and entropy How do we judge the best manner to split the data? Simply, we want to split the data in a manner which provides us the most amount of information - in other words, maximizing information gain. Going back to the previous example, we could have performed our first split at $x\_1 < 10$. However, this would essentially be a useless split and provide zero information gain. In order to mathematically quantify information gain, we introduce the concept of entropy. \[\\mathop - \\sum \\limits \_i^{} {p \_i}{\\log \_2}\\left( {{p \_i}} \\right)\] Entropy may be calculated as a summation over all classes where $p\_i$ is the fraction of data points within class $i$. This essentially represents the impurity, or noisiness, of a given subset of data. A homogenous dataset will have zero entropy, while a perfectly random dataset will yield a maximum entropy of 1\. With this knowledge, we may simply equate the information gain as a reduction in noise. information gain = entropy( parent )−\[ weighted average \]entropy( children ) Here, we're comparing the noisiness of the data before splitting the data (parent) and after the split (children). If the entropy decreases due to a split in the dataset, it will yield an information gain. A decision tree classifier will make a split according to the feature which yields the highest information gain. This is a recursive process; stopping criterion for this process include continuing to split the data until (1) the tree is capable of correctly classifying every data point, (2) the information gain from further splitting drops below a given threshold, (3) a node has fewer samples than some specified threshold, (4) the tree has reached a maximum depth, or (5) another parameter similarly calls for the end of splitting. To learn more about this process, read about the [ID3 algorithm.](https://en.wikipedia.org/wiki/ID3%5Falgorithm?ref=jeremyjordan.me) #### Techniques to avoid overfitting Often you may find that you've overfitted your model to the data, which is often detrimental to the model's performance when you introduce new data. To prevent this overfitting, one thing you could do is define some parameter which ends the recursive splitting process. As I mentioned earlier, this may be a parameter such as tree depth or when the split will result in a node less than some specified number of samples. However, it may be more intuitive to simply perform a significance test when considering a new split in the data, and if the split does not supply statistically significant information (obtained via a significance test), then you will not perform any further splits on a given node. Another more common technique is known as pruning. Here, we grow a decision tree fully on your training dataset and then go back and evaluate its performance on a new validation dataset. For each node, we evaluate whether or not it's split was useful or detrimental to the performance on the validation dataset. We then remove those nodes which caused the greatest detriment to the decision tree performance. #### Degenerate splits Evaluating a split using information gain can pose a problem at times; specifically, it has a tendency to favor features which have a high number of possible values. Say I have a data set that determines whether or not I choose to go sailing for the month of June based on features such as temperature, wind speed, cloudiness, and day of the month. If I made a decision tree with 30 child nodes (Day 1, Day 2, ..., Day 30) I could easily build a tree which accurately partitions my data. However, this is a useless feature to split based on because the second I enter the month of July (outside of my training data set), my decision tree has no idea whether or not I'm likely to go sailing. One way to circumvent this is to assign a cost function (in this case, the gain ratio) to prevent our algorithm from choosing attributes which provide a large number of subsets. #### Example code Here's an example implementation of a Decision Tree Classifier for classifying the flower species dataset we've studied previously. ``` import pandas as pd from sklearn import datasets from sklearn.tree import DecisionTreeClassifier from sklearn.model_selection import train_test_split iris = datasets.load_iris() # ['target_names', 'data', 'target', 'DESCR', 'feature_names'] features = pd.DataFrame(iris.data) labels = pd.DataFrame(iris.target) ### create classifier clf = DecisionTreeClassifier(criterion='entropy') ### split data into training and testing datasets features_train, features_test, labels_train, labels_test = train_test_split(features, labels, test_size=0.4, random_state=0) ### fit the classifier on the training features and labels clf.fit(features_train, labels_train) ### use the trained classifier to predict labels for the test features pred = clf.predict(features_test) ### calculate and return the accuracy on the test data from sklearn.metrics import accuracy_score accuracy = accuracy_score(labels_test, pred) ### visualize the decision tree ### you'll need to have graphviz and pydot installed on your computer from IPython.display import Image from sklearn.externals.six import StringIO import pydot from sklearn import tree dot_data = StringIO() tree.export_graphviz(clf, out_file=dot_data) graph = pydot.graph_from_dot_data(dot_data.getvalue()) Image(graph[0].create_png()) ``` Without any parameter tuning we see an accuracy of 94.9%, not too bad! The decision tree classifier in sklearn has an exhaustive set of parameters which allow for maximum control over your classifier. These parameters include: criterion for evaluating a split (this blog post talked about using **entropy** to calculate information gain, however, you can also use something known as **Gini** impurity), maximum tree depth, minimum number of samples required at a leaf node, and many more. ### Spring Break in Silicon Valley URL: https://www.jeremyjordan.me/spring-break-in-silicon-valley/ Last updated: 2017-03-17T12:09:12.000Z This past week I was invited to spend my Spring Break in Silicon Valley meeting with startups founded or connected with NC State alumni. Here's a short description of each visit and my takeaways. #### General Remarks One of my favorite things about the Valley was a culture of people that were not afraid to take risks. Whether it's starting or investing in a company, most people have experienced the benefits of taking a big risk before and having it pay off, so they're willing to do it again. This leads to a **virtuous cycle** of companies that succeed and grow, creating vast amounts of wealth which are then reinvested in new companies. As such, there doesn't seem to be a better place to fund your startup than Silicon Valley. Another trait I found among the people I spoke with was even though some people were quite wealthy, they were more focused on starting new ventures than extravagant displays of wealth. Money out in the Valley seems to be viewed primarily as a vehicle to fund your next venture, with little intrinsic value in and of itself. #### Google At Google, we met with Thushan, an NC State graduate who started Launchpad Toys. This company created Toontastic and Telestory, products aimed at teaching kids the fundamentals of storytelling using technology. The company was acquired by Google in 2015, which was a bit of an adjustment for Thushan and his team. Going from a small team of less than 20 to an organization of close to 50,000 people is a bit of a shock. On one hand, Google has provided Thushan and his team with great resources to continue advancing his goal of empowering kids through storytelling. On the other hand, you lose some degree of autonomy. Thushan advocated for mission-driven teams, explaining that it becomes easier to settle internal arguments if you all share a common mission. In fact, he believes friction is a good thing; provided the friction doesn't excessively curtail the speed of development, the product generally comes out smoother as a result of the friction along the way. One big lesson that Thushan shared with us was the importance of good user interface and experience design. Specifically, he was designing an app for kids and had to do a lot of user testing in order to guide kids to use the app as it was intended. The biggest thing he learned is that if you are trying to guide a user through a process, the next action should be the only action on the screen. His example was showing us the many iterations in which it took to find the optimal place for the "Start" button. **Thushan's advice:** When you start to notice yourself getting comfortable with something, find a way to push yourself back out of your comfort zone - whether by a new position or a challenging goal at your current position. #### Facebook Unfortunately, our visit to Facebook wasn't much more than a tour of the campus. I have to say the campus was quite impressive, however. It felt like a small town, equipped with all sorts of perks for the employees (like free ice-cream!). #### Blue Jeans We met with Blue Jeans, a company founded by an NC State graduate, who provides video conferencing solutions for businesses. They're a later stage startup, having raised a $76.5M Series E round in September 2015\. Ted Tracy, their VP of Engineering, spoke about his experience living in Silicon Valley and some of the cultural trends which have challenged their company. Namely, he spoke about millennials' tendency to jump from company to company every two years in order to elevate their personal career. This leads to a challenge in being able to hire and retain top talent, although it seems like most of the Valley is generally accepting of this behavior. #### Pearl Automation An NC State alumnus, Saket, was on the founding team at Pearl and he met with us to share the story for how Pearl was started. If you're not familiar with Pearl, they're a consumer electronics company who designs aftermarket tech products for the automobile industry. They basically noticed that the auto industry does not follow Moore's law; the pace of change and innovation for automobiles is much slower. This is a combination of the fact that purchase cycles for a car are very long compared to other tech industries, and even new cars rolling off the production lines are equipped with three-year-old technology due to a lengthy production time. This results in simple technology features, such as backup cameras and adaptive cruise control, to be found lacking in many cars. Pearl aims to provide these new features as an aftermarket add-on to your existing car. I have to say, visiting Pearl felt very Apple-esque. The company has a strong design core and continues Apple's tradition of attention to detail in developing high-quality products. Where the company differs is its open culture, which is a stark contrast to Apple's need-to-know culture. There's a [good write-up on the cultural differences between the companies](https://www.nytimes.com/2017/01/02/technology/pearl-automation-apple-alumni.html?ref=jeremyjordan.me) in the NY Times. Pearl has been able to [raise a considerable amount of money](https://www.forbes.com/sites/aarontilley/2016/06/21/pearl-backup-camera/?ref=jeremyjordan.me#5b813aa9b74b) for a seemingly simple product; needless to say, they're skilled at VC funding. Saket shared with us a few tips for when it comes to raising money from VCs. His biggest piece of advice? Do your research. Figure out who your investor is, grab coffee and have a casual conversation about your idea first, get intelligence from peers on the investor, and when you do pitch - tailor each pitch deck to the individual. He also mentioned that it's important to consider whether the investor you're pitching to will be able to afford to continue investing in future rounds, something known as "dry powder", so their equity isn't diluted. **Saket's advice:** There's many different paths to a startup. Many people start a company right out of school. Pearl's founders worked first, at Apple, and got the opportunity to learn a lot about product development and scale-up, develop vendor relationships, and network with Apple's talented workforce before leaving to start their own company. Each path has its benefits. #### Credence Medsystems Credence was one of the smaller startups we visited, having just closed their Series A funding round in September 2016\. Their product is a preloaded syringe in which the needle retracts after use to prevent needle sticks and improper reuse. Apparently, nurses can accidentally prick themselves with a used needle when moving around in a frantic hospital environment. This is very dangerous because the nurse is not sure whether or not they might have contracted a disease from the needle stick, and so they must be taken out of circulation and spend 6 months testing for all of the potential diseases they might have possibly acquired from the needle stick; it goes without saying this is an expensive outcome for hospitals. By retracting the needle inside the syringe after use, they're also able to prevent improper reuse of needles - for example, by drug addicts who found a way into the hospital's trash and took used syringes. We met with Jeff Tillack at Credence, an alumnus from NC State and founder of Credence and a long list of other healthcare companies. He shared stories about launching healthcare products with us and the unique hurdles a healthcare company must go through in order to seek FDA approval. **Jeff's advice:** Keep your cash burn low and understand that perfection can be the enemy of progress. He also suggested to build companies for the long term, don't focus so much on your exit strategy in the early stages of the company. #### Apple We were fortunate to get an hour and a half with Apple's Chief Operating Officer, Jeff Williams - a proud NC State alumni. Jeff started his career at IBM, and only joined Apple after taking an interview with Tim Cook out of courtesy. At the time, Apple had just lost a billion dollars and Jeff did not think the company would succeed. But he knew Tim from business school (at Duke) and decided to entertain the possibility of working at Apple. After interviewing with Steve Jobs, he felt so moved by the energy and passion Jobs exuded that he decided to take the job at Apple on a whim. He said it was one of the few right-brain decisions he's made in his life and he's incredibly glad he did. Since then, Apple has grown to one of the widest-known consumer brands and one of the largest companies in the world. When asked about what is missing in today's corporate leadership, Jeff cited a trend of acceptance of mediocrity. At Apple, they are hyper-focused on developing the highest-possible quality products with great attention to detail. He explained that he could easily add $1.25 billion to their bottom line by using cheaper packaging, but they refuse to sacrifice a product's integrity in the name of profits. This said, he admitted that they are only a few products away from irrelevancy unless they continue to innovate. It is for this reason that he believes it is important, as a leader, to wake up each morning not assuming that Apple is a great company, but challenging himself and his team to continue progressing and maintain the startup culture as much as possible. As I listened to Jeff speak, I found him to be quite charismatic and down to earth. When asked about what was lacking in today's leadership he first addressed the question in a positive light and first established the qualities of a good leader (specifically, speaking of the importance of situational leadership) before addressing his concern for the acceptance of mediocracy. He concluded his response on a second positive note explaining why Apple believes so passionately in their commitment to excellence. Needless to say, I was impressed by the control he maintained in the direction of the conversation. **Jeff's advice:** Be comfortable hiring people who are better than you, as it will help you build the best possible team. #### Y Combinator Started by Paul Graham in 2005, Y Combinator is one of the nation's most prestigious startup accelerator programs. They have funded startups such as AirBnB, Dropbox, Stripe, Docker, and many more uber-successful companies. The accelerator is a combination of an investment ($120K for 7%) and a 3-month intensive program where companies are told to focus on only three things: talk to customers, write code (build your product), and exercise. It's nice to see the program helping young entrepreneurs prioritize their health alongside their career. Y Combinator has two funding cycles a year, and in each cycle they recieve roughly 7,000+ applicants. Of these applicants, they invite the top 500 for interviews, and accepts around 100 companies to the program. At the end of the three month program, all of the companies pitch at Y Combinator's Demo Day (400-500 investors come to watch this); the average company will raise another $700K as a result of Demo Day. We mainly spoke about how Y Combinator evaluates applicants for the program. Here's what we learned. - **Most companies have 2 to 3 founders.** Starting a company is hard, really hard. Y Combinator has found that it can be very valuable to launch a venture with at least one co-founder, so that you can carry each other through the valley of death. Only \~10% of the companies they accept have only one founder. - **You have to be able to describe your idea in a simple sentence.** You might be starting an incredibly complex technology company, but you still have to be able to communicate what value your company provides in simple terms. - **If you've applied before and didn't get in, be prepared to explain what's different and what you've learned.** Roughly 40% of applicants end up reapplying to the program, and it's incredibly important to make clear how the company has improved and grown in order to be seriously considered a second time. - **Have a personal tie to the problem you're trying to solve.** This helps give them confidence that you'll have the fire within you to push yourself and grow the company when things get tough. - **Build a team of people which you have already worked with previously.** Incompatible founders is actually a major reason why many startups fail. Derisk the potential for this by having a previous working relationship with your founding team. By the way, Y Combinator will be opening up a curriculum for startup founders called [Startup School](https://www.startupschool.org/?ref=jeremyjordan.me) beginning on April 5th, 2017. #### Citrine Informatics Citrine was another one of the smaller startups we visited, and they were especially interesting to me as their company sits at the intersection of materials science and data science. They maintain the world's largest database of materials properties and parameters and provide machine learning algorithms to guide researchers (both in industry and academia) in the materials selection process. For example, you might be a car manufacturer looking for the optimal metal alloy to use in your next car frame, one that is strong and lightweight, and resistant to corrosive environments. You can use Citrine's platform to search for the best possible candidate materials given the properties you are looking to optimize. The founders of Citrine (two of which are NC State alumni) came from a technical background, and they learned pretty quickly that they needed to forgo their technical critique of the product in favor of extreme confidence of their product and team when pitching to investors. They explained that when pitching to investors, you have to be able to convince the investors that not only is the problem you are working on important, but there is no better team to solve this problem and dominate the market than your own team. This isn't to say that you should forgo technical rigor and critique when developing your product, but you should be prepared to pitch to investors with an intensity and confidence that investors feel as if they will miss out if they *don't* invest in you. **Greg's advice:** When pitching to a VC, you need to be able to look them in the eye with such a firey passion that they instintively believe you when you say you're the only company in the world who can grow your startup to a successful business. #### Andreessen Horowitz We met with Scott Kupor, managing partner at Andreessen Horowitz, to learn the essentials of venture capital investing from one of the most well-respected VC firms in the nation. Compared to Y Combinator, Andreessen Horowitz is a multi-stage venture capital firm and they invest in companies ranging from seed rounds to late series rounds at well-established startups. He started with a brief primer on VC investing. For a given set of 10 companies which they invest in, they fully expect 5 or 6 to fail and do not expect to see a return on their investment. They may have 2 or 3 companies which see moderate success and provide a 2x or 3x return on their investment. But the companies which VCs are counting on are the 1 or 2 startups which see such a high success that they receive anywhere from 10x up to 50x return on their initial investment. It's these companies that cover the losses of all of their other investments and provide enough incentive to continue investing in startups. You can read more about the power law rule of investing [here.](http://reactionwheel.net/2015/06/power-laws-in-venture.html?ref=jeremyjordan.me) Because of the nature of VC investing, firms are looking for companies with large market opportunity. If they invested in great business concepts with limited market potential, they would never be able to see the explosive growth and success that provides the firm with meaningful returns. It is for this reason that **market potential** is the biggest factor when considering whether or not to invest in a company. Scott made sure to mention that for this reason, venture capital fundraising is not the right method of fundraising for all types of businesses. Following market potential, Scott argues that the **team** is the second most important factor by which VCs evaluate a company. One thing they look for in the team is founder-market fit, similar to how Y Combinator mentioned that they look for founders with a personal tie to the problem they're solving. In theory, many different people could attempt to solve the problem your team is trying to address, so you need to convince VCs that your team is uniquely equipped to tackle the problem. He also mentioned that teams must present the right level of malleability - they don't want to work with a team that is so steadfast they refuse to change their business model if it's failing, but they also don't want to work with a team that lacks the confidence in their vision and is willing to change their business model on a whim after a VC partner poses a question about their strategy. The team is often more important than the actual **product**, which is the third quality by which VCs evaluate a company. This is because the VCs know from experience that 99% of the time the final product bears little resemblance to the original product idea. While you should have a good product (or at least problem to solve), VCs would rather work with a stellar team and a bad idea than a poor team with a good idea; the former is much easier to advise than the latter. He also explained that venture capital is in the referral business. Due to the sheer number of companies that wish to present to VC firms, they use referrals as a technique to filter down the number of meetings they'll take. As a result, it's near impossible to get a meeting with a VC firm unless you have a warm introduction from someone who they have a working relationship with and respect. For those interested, Mark Suster gives a good insight into the world of VC fundraising in this [video.](http://snapstorms.com/raise-money-venture-capitalists/?ref=jeremyjordan.me) #### Flexport You probably haven't heard of this company if you're not in Silicon Valley; they provide freight forwarding services, currently disrupting the market by bringing technology to an industry which still operates over telephone calls and faxes. They've seen incredibly impressive growth over the past year and are currently trending as one of the "hot" startups in the Valley. Apparently, Silicon Valley is going through this odd trend where unsexy businesses are the "sexy" thing right now. Jindra Zitek, SVP Operations and Customer Success at Flexport, explained the challenge of maintaining their company culture while experiencing such rapid growth and hiring so many new people into the company. He mentioned the need to overcommunicate their mission internally in order to align the company behind a common goal. #### Eatsa Due to a pretty comprehensive NDA I had to sign, I can't share too much- but this company is doing some super neat work to be able to provide cheap, healthy food. Here's a video that explains the concept! #### Autodesk Although many probably wouldn't consider Autodesk to be a startup, they are an entrepreneurially-minded company which provides many resources for startups and entrepreneurs. We got to tour their impressive prototyping lab located on a pier in San Francisco; I must say, you can't beat the view. Notable, they have an [artist and innovator residency program](http://www.autodesk.com/artist-in-residence/home?ref=jeremyjordan.me) in which you can explore a new concept using Autodesk's solutions, fully funded on a stipend for a few months. However, the most interesting thing I saw at was Autodesk's generative design solution. Check it out in the video below. ### On thinking critically and skepticism. URL: https://www.jeremyjordan.me/on-thinking-critically-and-skepticism/ Last updated: 2019-02-19T03:24:18.000Z It really clicked for me sophomore year of high school. I had gained an interest in eating healthy and was doing a lot of research online to figure out the optimal healthy lifestyle. Now, the area of health can be quite a confusing one seeing as almost every food in existence has an article talking about why it's good for you and another one talking about why it's bad for you. This made it quite difficult to sift through the garbage and figure out what I should really be doing in order to live a healthy lifestyle. It was two simple words that caused a paradigm shift in the way I looked at things - Chris Kresser's self-referential description of a "healthy **skeptic**." It's a simple phrase, but it reminded me to **think critically** of everything I was reading. It brought an awareness to how I was consuming and internalizing information. I learned to consider the author's motivations, funding sources, and other implicit biases. I learned how facts can be presented in deceptive ways. Those articles which seemed to oppose each other with one saying a food is good for you and the other saying the same food is bad for you? I realized that no food is entirely good nor bad, but rather there are parts of a food that good for you and parts of food that aren't as great for you. Some people focus on the good, some people focus on the bad. From this, I learned to look at the facts and make my own conclusions, rather than automatically following the conclusion of the author. I began to not only consume information, but to **develop a system for internalizing what I was reading**. I formed a mental model of what I believed health to be, and evaluated ideas against this model. When an idea conflicted with my mental model, I was forced to dig deeper and evaluate which of the two was incorrect. Moreover, I became more selective of the sources in which I chose to consume information from - tending to read only from places which could cite studies from credible journals. Further, I was inherently motivated to search for the truth, not just what I wanted to hear. If my goal was to live a healthy lifestyle, choosing to believe an article talking about "Cookies: the new health superfood" might make me happy because I could eat more cookies, but it certainly won't benefit my health. Thus, I was inherently motivated to set aside any implicit bias in order to figure out what constitutes a healthy lifestyle. However, it is important to note that skepticism is not a habit reserved for external ideas - there were times when my internal mental model was incorrect, and I had to be willing to challenge my own ideas of what a healthy lifestyle included. Jumping forward to college, a couple of other experiences have deeply engrained in me the habit of challenging my ideas. One of my mentors told me a story about his time at $\\lbrack$company$\\rbrack$ that stood out in his memory: > One day I was walking through the cafeteria of $\\lbrack$company$\\rbrack$'s research headquarters and saw the head of research sitting at a table talking. While he spoke, all of the five other people sitting at the table nodded in unison, appearing to be blindly following the leader without question. When I saw this, I told myself I never wanted to be in that position where everyone follows along without thinking critically and challenging each other. He then went on to explain the importance of surrounding yourself with people that will challenge you. I *really* adopted this mindset after spending a summer interning at Red Hat, whose culture is based on a meritocracy. At Red Hat, the best ideas won no matter who proposed it. They've been able to cultivate a culture with a fluid exchange of ideas, where every idea is challenged, and the best ideas float to the top. I grew to **love** when people would challenge my ideas in conversation because it provided me with the chance to strengthen my thoughts. *(Shoutout to Mark DeMaria for our numerous conversations where I'd throw out a half-baked thought, and you'd challenge it and help me form a more complete idea.)* #### Fake news There's been a lot of talk about fake news lately with some claiming we've entered the "post-truth" era and studies showing that students are [often unable to separate fact from fiction](http://www.npr.org/sections/thetwo-way/2016/11/23/503129818/study-finds-students-have-dismaying-inability-to-tell-fake-news-from-real?ref=jeremyjordan.me). This "post-truth era" title sounds alarmist until you consider its merit. > Faced with the choice between changing one's mind and proving that there is no need to do so, almost everyone gets busy on the proof. - John Kenneth Galbraith Everyone keeps talking about how fake news is a problem. But why is it a problem? It's a problem not because the information is false or misleading, it's a problem because people believe it and share it with their friends. If everybody were to critically examine fake news and properly label it as trash, no one would share it and the incentive to create fake news would vanish. However, this is clearly not what has happened. Why is that so? In her talk, "Why you think you're right -- even if you're wrong," Julia Galef presents this notion of "motivated reasoning." This concept acknowledges that certain ideas which are aligned with our beliefs are "allies," and we want them to win, while other ideas which contradict our beliefs are "enemies," and we tend to shoot these ideas down. If you're watching a sports game and the referee makes a call against your team, you're motivated to find a reason why this call was poor judgment, but if the referee makes a call against the other team, you're much less motivated to find fault in his decision. In order to successfully advance society in a benign direction, we must consistently examine ourselves, our ideas, and our social norms in a critical light. We must remember that knowledge is what we know, what we consider to be true, but it is not inherently truth. Our collective knowledge evolves over time as we continue to learn about the world. We must recognize that progress in the world results from seeking the truth, not self-affirmation of one's beliefs. And above all, we must take the time and intent to apply these habits of thinking critically and skeptically in search of the truth in our daily lives. ### A guiding principle for life. URL: https://www.jeremyjordan.me/a-guiding-principle-for-life/ Last updated: 2018-08-01T15:59:36.000Z One day, when I was a freshman in college, I walked into my mentor's office and he clasped his hands together at his fingertips, moving both hands forward as one. He asked me, "what am I doing?" to which I replied, "I don't know." He proceeded to give me and hint and ask again, "This (motioning to his hands) is a ship, what is this ship seeking?" to which I replied, "I don't know." He then continued to use this image of a ship traveling through the water as an analogy for life. When a ship moves through water, what happens? ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2017/01/pexels-photo.jpg) The forward motion of the ship displaces water resulting in the creation of a wake. He then asks me, "Where is this ship heading? What determines it's orientation? What is the guiding principle for this ship?" After a pause, letting the thought sink in, he continues "This ship is seeking knowledge. All of the things people tell you to seek out in life - a college degree, a job, a paycheck, awards, research publications, critical acclaim, etc. - all of these things happen as a **result** of seeking knowledge. If you live your life by this guiding principle, you'll never be lead astray. However, if you start to chase the wake of your ship - the job, the piece of paper that says you earned a college degree, etc. - you'll end up chasing your tail and halting any sense of forward progression." For example, if you go through college seeking a high GPA and the end overall degree, you tend to miss out on the most formative aspects of higher education. However, if you approach college seeking knowledge and challenging yourself, you'll typically end up with a good GPA and graduate with a degree - the "wake" which came as a result of knowing how to orient your "ship." ![](http://stream1.gifsoup.com/view2/1977472/bailey-chasing-her-tail-o.gif) This analogy has stuck with me for quite some time and provoked a considerable amount of thought. I really liked looking at life and classifying things as "guiding principles" and "results". I have since expanded my view of this analogy from simply seeking knowledge to a number of what I call **"noble pursuits"**, including the following: seeking knowledge, sharing love, exploring foreign environments, challenging oneself, and building relationships within communities. Your noble pursuits may be different, but pursue these things and you will live a deeply satisfying life. ### New Year's Resolutions 2017 URL: https://www.jeremyjordan.me/new-years-resolutions-2017/ Last updated: 2018-01-07T19:17:55.000Z ## Be more intentional with my time. When reading On the Shortness of Life by Seneca, I came across the following passage: > It is not that we have a short time to live, but that we waste a lot of it. Life is long enough, and a sufficiently generous amount has been given to us for the highest achievements if it were all well invested. But when it is wasted in heedless luxury and spent on no good activity, we are forced at last by death’s final constraint to realize that it has passed away before we knew it was passing. So it is: we are not given a short life but we make it short, and we are not ill-supplied but wasteful of it… Life is long if you know how to use it. This passage really resonated with me and forced me to consider how I was allocating my time. I feel like I generally do a good job at choosing how to spend my time - I've focused on school, developing myself (whether in character or acquiring new skills), and supporting those around me. And yet, it's still often incredibly difficult to decide in the moment, "with all of the things I could be doing right now, which would be the best use of my time?" From my reflections of 2016, I've realized that I want to enter 2017 with a more disciplined focus. I have a variety of interests which lead to sometimes spreading my time over too many things and not getting the maximum return from any of my individual endeavors. While I still wish to engage all of my interests, I would like to do so in a manner that allows me to be more focused and intentional with my time. **End of year review:** B On a macro scale, I've been able to prioritize my time well with a strong focus of studying the field of machine learning. On a micro scale, I can still do better about elimating pesky time wasters. ## Get better sleep. I hate feeling tired in the morning; mornings are my most productive hours. I want to stick to a better, consistent sleep schedule and open my early morning hours for productive work. **End of year review:** C+ For a while, I was doing *really well* to make sure my sleep was a priority, but as the year progressed I found myself continuing to work late into the evenings (past the hour of meaningful productivity) and toward the end of the year I fell out of my nightly reading routine. ## Stay active. I enjoy being active, but I'm not the best about making time for it. I'd like to do a better job in 2017 of making sure that I stay active on a regular basis. To accomplish this, I've created two forcing functions: 1. I got a **Fitbit** for Christmas to track my activity. I especially wanted one because they have social challenges I can do with friends to see who can get the most steps over the duration of the challenge. This social aspect should help incentivize me to make time for exercise because I don't like finishing in last place! 2. I went ahead and laid out a **fitness plan** for the Spring semester. Shoutout to Paul for being crazy enough to go to 6am yoga and TRX classes with me! A healthy body yields a healthy (and thus productive) mind. **End of year review:** A On balance, I've been able to stay active through the year. Even when things were busy, I still forced myself to pull away and find time to keep my body in shape. ## Post more on here. This has been a great place for me to write about what I'm learning and store my thoughts. I'd like to develop habits of posting more frequently on here in 2017. **End of year review:** A- As I have been studying machine learning, I've generally pushed myself to write about most topics that I learn. I've found that by writing about a topic, I'm able to more effectively evaluate how well I understand the topic and by writing as I learn I can deepen my understanding. ## Do a weekly review. I'd like to establish the habit of reviewing and reflecting on my previous week every Sunday morning. I believe this regimented structure will help keep my accountable and make sure I am making progress towards my goals for the new year. **End of year review:** F To be honest, I forgot about this goal and never established the habit. ## Don't Google everything, at least not right away. It's a habit that develops gradually; I have a thought and think, "hmm, I wonder what other people have written about \_\_\_\_\_\_" and I Google it. However, instantly searching to see what other people say about a given idea inherently guides the future progression of an idea. I'd like to take more time internally to develop an idea before I search Google to see what other people have to say about the topic. **End of year review:** A I've been much better about taking the time to think things through more completely before seeing what others have to say on a subject, which has added a new (and more critical) perspective when I do read external ideas/solutions. ### Strategies for understanding a concept. URL: https://www.jeremyjordan.me/strategies-for-understanding-a-concept/ Last updated: 2016-12-17T18:30:21.000Z In my last post on learning (found [here](https://www.jeremyjordan.me/on-learning-and-intrinsic-motivation/)), I presented a different way to think about the learning process. In this post, I'll focus mainly on the struggle phase and detail strategies for reaching your "a ha" moment. ##### Think like a lightning bolt Have you ever watched a lightning bolt in slow motion? Take a minute to learn how lightning develops. Alright, wasn't that neat? Before seeing this video, I had no idea that was how lightning worked. I've realized that the birth of a lightning bolt is actually a great analogy for the birth of a eureka moment. When you're trying to grasp a subject, you're not quite sure how to approach it at first. So try different things, go different directions, look at the concept from as many different perspectives as possible. Get weird with it, be creative - that's often the way to make things stick. Be like the step leader and it's "tortuous channel that it's taking as it divides back and forth." When you've found the right path, the right intuition, your "a ha" moment happens in a flash. ##### Make sure you have the right challenge/skills ratio It doesn't make sense to try to understand multivariate calculus if you don't know how to add. In [flow state research](https://www.ted.com/talks/mihaly%5Fcsikszentmihalyi%5Fon%5Fflow?ref=jeremyjordan.me), a fine-tuned challenge/skills ratio is one of the [17 identified triggers for flow](http://www.stevenkotler.com/?ref=jeremyjordan.me), which is essentially a state of peak mental performance. > Flow appears near the emotional midpoint between boredom and anxiety, in what’s scientists call the “flow channel” — the spot where the task is hard enough to make us stretch; not hard enough to make us snap. > This sweet spot keeps attention locked in the present. When the challenge is firmly within the boundaries of known skills—meaning I’ve done it before and am fairly certain I can so again — the outcome is predetermined. We’re interested, not riveted. But when we don’t know what’s going to happen next, we pay more attention to the next. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/12/Untitled.png) ##### Focus on what you're learning, not how you're performing The good news is if you can find yourself in flow by tackling a challenge appropriate to your skill level, you'll likely do this inherently. One of the characteristics of flow is transient hypofrontality (read: your prefrontal cortex effectively shuts down) which silences your inner critic and allows you to be fully present in the moment. Now, it is important to understand how you're performing, but not while you're learning. When you're struggling to understand a new concept, it doesn't make sense to waste mental bandwidth agonizing over the fact that the concept hasn't clicked yet. Rather, you should dedicate the entirety of your mental capacity towards grasping and understanding the concept at hand. This allows your brain to take in a maximum amount of data, which increases your chances of having something click and make sense. ##### Get some sleep Speaking of which, make sure to prioritize a restful mind. Seriously, healthy sleep is incredibly beneficial for pattern recognition and learning. ### Grouping data points with k-means clustering. URL: https://www.jeremyjordan.me/grouping-data-points-with-k-means-clustering/ Last updated: 2017-10-18T04:24:15.000Z K-means clustering is a simple method for partitioning $n$ data points in $k$ groups, or clusters. Essentially, the process goes as follows: 1. Select $k$ centroids. These will be the center point for each segment. 2. Assign data points to nearest centroid. 3. Reassign centroid value to be the calculated mean value for each cluster. 4. Reassign data points to nearest centroid. 5. Repeat until data points stay in the same cluster. Here's a nice GIF to visualize the process. The $x$'s are centroids and the colored dots are data points assigned to a cluster. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/12/kmeans.gif) [*Image credit*](http://simplystatistics.org/2014/02/18/k-means-clustering-in-a-gif/?ref=jeremyjordan.me) Now let's look at each step in more detail. ### Choosing the initial centroid values Before we discuss how to initialize centroids for k-means clustering, we must first decide how many clusters to partition the data into. As a human, looking at the above example the answer is obviously three. But how could you decide this algorithmically? ###### Elbow method One method would be to try many different values of $k$ and plot the average distance of data points from their respective centroid (average within-cluster sum of squares) as a function of $k$. This will give you something along the lines of what is shown below. Notice how increasing the $k$ value from 1 to 2 drastically lowers the average distance of each data point to its respective centroid, but as you continue increasing $k$ the improvements begin to decrease asymptotically. The ideal $k$ value is found at the elbow of such a plot. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/12/BzwBY.png) [*Image credit*](http://stackoverflow.com/questions/6645895/calculating-the-percentage-of-variance-measure-for-k-means?ref=jeremyjordan.me) Unfortunately, this isn't exactly an algorithmic approach because it still depends on a manual decision as to where the elbow lies. Further, this method does not always work well when this is no clear distinction of an elbow (for example, when the curve is very smooth). ###### Silhouette analysis Another more automated approach would be to build a collection of k-means clustering models with a range of values for $k$ and then evaluate each model to determine the optimal number of clusters. We can use Silhouette analysis to evaluate each model. A Silhouette coefficient is calculated for observation, which is then averaged to determine the Silhouette score. The coefficient combines the average within-cluster distance with average nearest-cluster distance to assign a value between -1 and 1\. A value below zero denotes that the observation is probably in the wrong cluster and a value closer to 1 denotes that the observation is a great fit for the cluster and clearly separated from other clusters. This coefficient essentially measures how close an observation is to neighboring clusters, where it is desirable to be the maximum distance possible from neighboring clusters. We can automatically determine the best number of clusters, $k$, by selecting the model which yields the highest Silhouette score. ``` from sklearn.cluster import KMeans from sklearn.metrics import silhouette_score import numpy as np # Use silhouette score to find optimal number of clusters to segment the data num_clusters = np.arange(2,10) results = {} for size in num_clusters: model = KMeans(n_clusters = size).fit(data) predictions = model.predict(data) results[size] = silhouette_score(data, predictions) best_size = max(results, key=results.get) ``` You can also manually inspect the Silhouette coefficients, as discussed in [this example](http://scikit-learn.org/stable/auto%5Fexamples/cluster/plot%5Fkmeans%5Fsilhouette%5Fanalysis.html?ref=jeremyjordan.me). ###### Centroid initialization Once you've selected how many groups you'd like to partition your data into, there are a few options for picking initial centroid values. The simplest method? Select $k$ random points from your data set and call it a day. However, it's important to remember that k-means clustering results in an approximate solution converging to a local optimum - so it's possible that starting with a poor selection of centroids could mess up your clustering (ie. selecting an outlier as a centroid). The common solution to this is to run the clustering algorithm multiple times and select the initial values which end up with the best clustering performance (measured by minimum average distance to centroids - typically using within-cluster sum of squares). You can specify the number of random initializations to perform for a K-means clustering model in sci-kit learn using the `n_init` parameter. Another method would be to use another clustering technique, such as hierarchical clustering, on a sample of your data set and using the resultant cluster centroids as your initial k-means centroids. This method is typically reserved for k-means clustering applications on large datasets. You could also select your initial values in the most disperse manner possible (maximizing distance between centroids) in order to prevent unwanted localized convergence. #### Assigning data points to a cluster Once you've selected the initial centroids, the next step is to assign each data point to a cluster. Mathematically speaking, you do this: S i (t) \={ x p : ‖ x p − m i (t) ‖ 2 ≤ ‖ x p − m j (t) ‖ 2 ∀j,1≤j≤k } where clusters are represented as sets $S\_{1}$ ,..., $S\_{i}$ and the $m$ represents the centroid value (which is a mean in all subsequent steps after the first). Translating this to English, we get the following: Set $i$ contains all data points ($x$) in which the distance from the data point ($x\_{p}$) to the mean of $i$ ($m\_{i}$) is smaller than that of the distance from the data point to the mean of all other centroids ($m\_{j}$). #### Calculate new centroid values After each data point is assigned to a cluster, reassign the centroid value for each cluster to be the mean value of all the data points within the cluster. m i (t+1) \= 1 | S i (t) | ∑ x j ∈ S i (t) x j #### Reassign data points to new clusters This is where the iterative process begins. Follow the same process for initially assigning data points to clusters, this time with new centroid values. Iteratively calculate new centroid values and reassign data points until you converge on a cluster segmentation that stops changing. S i (t+1) \={ x p : ‖ x p − m i (t+1) ‖ 2 ≤ ‖ x p − m j (t+1) ‖ 2 ∀ j, 1≤j≤k } Dealing with a large data set and don't want to have to perform multiple iterations over your data? Check out the Bradley-Fayyad-Reina algorithm, which performs a similar function as k-means with only one pass through the data. #### Example Code Let's look at an implementation of k-means to group flowers in the Iris data set. First we'll import a few libraries and load the data set. ``` from sklearn import datasets from sklearn.cluster import KMeans from scipy.spatial.distance import cdist, pdist import numpy as np iris = datasets.load_iris() # attributes of iris are ['target_names', 'data', 'target', 'DESCR', 'feature_names'] ``` From this we can see that we'll be clustering the data base on the following features: - Sepal length (cm) - Sepal width (cm) - Petal length (cm) - Petal width (cm) Next, we have to choose how many groups we'd like to separate the data into. ``` # Choosing the optimal k k_range = range(1,10) # Try clustering the data for k values ranging 1 to 10 k_means_var = [KMeans(n_clusters = k).fit(iris.data) for k in k_range] centroids = [X.cluster_centers_ for X in k_means_var] k_euclid = [cdist(iris.data, cent, 'euclidean') for cent in centroids] dist = [np.min(ke, axis=1) for ke in k_euclid] # Calculate within-cluster sum of squares wcss = [sum(d**2) for d in dist] # Visualize the elbow method for determining k import matplotlib.pyplot as plt plt.plot(k_range, wcss) plt.show() ``` Inspecting this plot, k=3 seems like the best choice. ``` kmeans = KMeans(n_clusters = 3) kmeans.fit(iris.data) kmeans.labels_ ``` Let's see how we did. The labels returned by the k-means algorithm are: ``` >> kmeans.labels_ array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 2, 2, 2, 2, 0, 2, 2, 2, 2, 2, 2, 0, 0, 2, 2, 2, 2, 0, 2, 0, 2, 0, 2, 2, 0, 0, 2, 2, 2, 2, 2, 0, 2, 2, 2, 2, 0, 2, 2, 2, 0, 2, 2, 2, 0, 2, 2, 0]) ``` And the labels in the original dataset are: ``` >> iris.target array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]) ``` From this, we can see that our algorithm did a decent job clustering the first and second flower type, but it sometimes confused the third flower type for the second. Note that our algorithm was blind to any labels associated with the data; it's just putting the data points into groups. This is why the labels don't match up across our results and the provided labels. Feel like playing around more with k-means clustering? Check out this [interactive demonstration](https://www.naftaliharris.com/blog/visualizing-k-means-clustering/?ref=jeremyjordan.me). ### Further reading - [K Means - Chris Piech](http://stanford.edu/~cpiech/cs221/handouts/kmeans.html?ref=jeremyjordan.me) ### On happiness in life. URL: https://www.jeremyjordan.me/on-happiness-in-life/ Last updated: 2018-02-04T19:01:07.000Z Life isn’t about happiness. Life is about feeling the full range of emotions. Life is about exploring the human capacity. Convenient happiness is a facade. Life presents its attributes in a dualistic nature. Pushing through pain allows us to experience bliss. Challenging ourselves, feeling weak and vulnerable, allows us to grow stronger. In order to learn, you must humbly acknowledge what you do not know. Instant gratification is a convenient happiness, but it is not true happiness. Happiness is not something to be sought, but rather, a result of seeking something greater. There is so much beauty for us to discover, but we must explore. You’ll never be able to truly explore if you restrict yourself to the confines of comfort. ### Harvesting energy from vibrations for powering small electronics. URL: https://www.jeremyjordan.me/harvesting-energy-from-vibrations-for-powering-small-electronics/ Last updated: 2016-12-11T13:43:20.000Z Here's a paper I wrote examining piezoelectric vibration energy harvesters. Enjoy! **Further reading** (papers I didn't have time to read before turning this assignment in but would like to revisit): - [Power transfer of piezoelectric generated energy](https://www.google.com/patents/US5703474?ref=jeremyjordan.me) - [Strategies for increasing the operating frequency range of vibration energy harvesters: a review](http://iopscience.iop.org/article/10.1088/0957-0233/21/2/022001?ref=jeremyjordan.me) - [Energy harvesting from low frequency applications using piezoelectric materials](http://scitation.aip.org/content/aip/journal/apr2/1/4/10.1063/1.4900845?ref=jeremyjordan.me) - [The design, fabrication and evaluation of a MEMS PZT cantilever with an integrated Si proof mass for vibration energy harvesting](http://iopscience.iop.org.prox.lib.ncsu.edu/article/10.1088/0960-1317/18/5/055017/pdf?ref=jeremyjordan.me) - [A Green and Clean Alternative for Sustained Power Production](http://bst.sagepub.com.prox.lib.ncsu.edu/content/28/6/496.full.pdf+html?ref=jeremyjordan.me) - [Two dimensional woven nanogenerator](http://www.sciencedirect.com/science/article/pii/S2211285513000050?ref=jeremyjordan.me) - [Converting Biomechanical Energy into Electricity by a Muscle-Movement-Driven Nanogenerator](http://pubs.acs.org/doi/abs/10.1021/nl803904b?ref=jeremyjordan.me) Another area I'd like to read more about is triboelectric nanogenerators for wearable applications. ### Atomic force microscopy for studying surface topology. URL: https://www.jeremyjordan.me/atomic-force-microscopy-for-studying-surface-topology/ Last updated: 2016-12-17T17:45:10.000Z Ever wonder what a surface looks like at the microscale? Atomic force microscopy is commonly used in materials science to visualize surface topology. While it is often considered to be quite slow, it is often a useful materials characterization technique. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/12/002.jpg) [*Image credit*](http://staff.ustc.edu.cn/~bxlin/harvest%5Fe.htm?ref=jeremyjordan.me) \- An image of the surface of ZnO using AFM. Atomic force microscopy measures the interaction between a tip affixed on a cantilever and a sample’s surface. The cantilever acts as a spring with a known force constant, and a laser reflected off of the cantilever onto a detector is used to measure the cantilever deflection. As the tip is scanned across the sample in a raster pattern, the deflection at each position is saved and compiled into a map. Typically, a mirror is used to lengthen the laser path and allow for a higher sensitivity of deflection detection. Using the force constant and cantilever deflection, the atomic force repulsion between the sample and tip may be calculated with Hooke’s law ($F = -kx$). ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/12/AFMSchematic.jpg) Atomic force microscopy operates by scanning the sample with a physical probe and thus allows for true measurements of surface heights, as opposed to a scanning electron microscope which is only capable of producing two-dimensional images. A feedback loop is used to maintain a constant sample-tip interaction force. The changes necessary to maintain a constant force and then used to generate a topological map of the surface. For example, in the schematic above the sample is placed on a stage which can move up and down normal to the cantilever tip. When scanning across a localized peak on the material, the stage will need to move away from the cantilever in order to maintain a constant force interaction. Likewise, when scanning across a localized valley, the stage will need to be brought closer to the cantilever. It is this movement which is recorded and used to generate a topological map. The two most common AFM modes are contact and tapping, which describes the action of the tip as the sample is being scanned. Contact mode involves maintaining a constant contact force with the sample. The piezoelectric actuator adjusts the cantilever height in order to respond to changes in surface height on the sample. For the tapping mode, the piezoelectric actuator oscillates the cantilever near its resonant frequency. The feedback loop for tapping mode involves either the oscillation of amplitude or frequency. Tapping allows for scanning with extremely low friction because adhesion is a time-dependent process. Simply, the tip does not remain in contact with the material for enough time to adhere, which allows for the scanning of viscous samples with little trouble. Another benefit to tapping is the ability to measure the phase lag. This lag associated with the cantilever oscillation can be caused by changes in material properties, such as adhesion, elasticity, or viscoelasticity, and is utilized for phase imaging. Tapping is generally the preferred method for AFM due to its near frictionless interaction with the material and the additional phase lag information that can be captured. Atomic force microscopy proves advantageous over other microscopy techniques when an accurate surface height measurement is required. The true surface height can be obtained by AFM because the probe is physically interacting with the sample, and using an extremely fine tip that provides us with sub-Angstrom resolution. This is a much higher resolution than optical microscopy, which is diffraction limited to around 250 nm resolution. While scanning electron microscopy can be used for even higher resolutions than AFM, it generates a two-dimensional image; accurate surface height measurements are more easily obtained through atomic force microscopy. The images produced by AFM are limited by the nature of the tip. If a surface protrusion is narrower than the tip, the image displayed will simply be an image of the tip. Moreover, if a surface cavity is more narrow than the tip, the tip will not be able to enter and thus will not display the cavity. An ideal AFM tip would be narrower than any asperity on the surface, giving a near-true image of the material topography. Tips may be characterized by scanning over a reference sample that has many small, sharp protrusions. The repeated image displayed at each asperity represents the shape of the tip. If a probe has more than one tip, this is apparent in the image. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/12/TipQual2.png) [*Image credit*](http://www.nanophys.kth.se/nanophys/facilities/nfl/afm/icon/bruker-help/Content/SoftwareGuide/Offline/AnalysisFunct/TipQual.htm?ref=jeremyjordan.me) ### Ductile to brittle transitions in materials. URL: https://www.jeremyjordan.me/ductile-to-brittle-transitions-in-materials/ Last updated: 2016-11-11T09:53:57.000Z Understanding of fracture behavior is useful in selecting materials for product designs, and it is incredibly valuable to know if a material will experience a ductile-brittle transition and what temperature this transition occurs. Knowledge of a material's ductile-brittle transition temperature can prevent catastrophes such as the [WW2 Liberty Ship failure](http://www.brighthubengineering.com/marine-history/88389-history-of-the-liberty-ships/?ref=jeremyjordan.me). ![WW2 Liberty Ship failure](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/11/liberty-ship-failure.jpg) [*Image credit*](https://metallurgyandmaterials.files.wordpress.com/2015/12/liberty-ship-failure.jpg?w=640&ref=jeremyjordan.me) Ductile fracture is characterized by a large amount of plastic deformation which leads to slow fracture that is often detectable before the material reaches a critical point. Conversely, brittle fracture contains very little plastic deformation and often little warning before fracture occurs. The Charpy impact test, depicted below, measures the amount of energy absorbed upon impact. A weighted pendulum, which is held at some height $h$ from the sample, swings and impacts the sample at a very high strain rate. The pendulum then continues to swing until reaching a final height of $h'$. The **difference between the potential energies** at height $h$ and $h'$ is considering to be the energy that the sample absorbed upon impact. ![Impact test](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/11/impact_test.png) The sample for a Charpy impact test is notched to in order to control crack propagation and ensure consistent results across samples. The photo below displays a generic Charpy sample- note the V-shaped notch in the middle of the sample. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/11/charpysample.png) **Fracture mechanics in materials ultimately extends to ease of dislocation movement.** For samples in which dislocation motion in uninhibited, the material may accommodate an applied load through plastic deformation. For FCC metals, slip systems allow dislocations to move very easily along the close-packed planes. However, for BCC metals, there are no close-packed planes that allow for easy dislocation migration, and thus dislocation movement in **these materials (BCC metals) require a thermal activation in order to slip**. Otherwise, the material accommodates the applied stress through more drastic mechanisms, such as breaking of bonds. An FCC unit cell with the close-packed plane shown in red. ![FCC](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/11/FCC.png) A BCC unit cell with no true close-packed plane. These psuedo-"slip" systems need some degree of thermal activation before they will operate. ![BCC](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/11/BCC.png) The figure below displays the general behaviors for various materials including FCC metals, BCC metals, and high-strength materials with regard to the Charpy impact test. A ductile-brittle transition temperature is observed for metals with a body-centered cubic structure as a result of the dependence on temperature for dislocation motion. ![General behavior](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/11/general_behavior.png) I'll provide some data from an undergrad lab experiment which examines the impact behavior for annealed 1018 steel and 304 stainless steel. Annealed 1018 steel is a low-strength BCC steel, which exhibits a ductile-brittle transition as shown above in the general behavior schematic. 304 stainless steel is a low-strength FCC metal, which has close-packed planes that allow for dislocation movement at all temperatures. Thus, this metal does not exhibit a ductile-brittle transition. The ductile-brittle transition temperature, $DBTT$, may be approximated as the temperature that corresponds with the energy calculated from $DBTT = LSE + \\frac{{USE - LSE}}{2}$ where $USE$ and $LSE$ represent the upper and lower shelf energies respectively. Generally, it reflects the inflection point on the logistic-type curves. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/11/impactresults.png) ### On learning and intrinsic motivation. URL: https://www.jeremyjordan.me/on-learning-and-intrinsic-motivation/ Last updated: 2016-12-04T15:48:11.000Z During a tutoring session last week, I was working with one of my students and could tell that while she could solve most of her homework problems, she still didn't feel confident with the subject. I encouraged her to take some time to dissect the concepts and try to wrap her mind around them until she went from "I can solve this problem" to "this makes sense." During this discussion, I shared with her my view of the learning process. ***Preface:*** *This post represents my view on learning based on personal observations. However, I do make an attempt to connect this viewpoint with scientific findings at the end of the post. Moreover, this model of the learning process is based largely on gaining insights. It goes without saying that this is not an entirely holistic view of the learning process.* ##### The learning process You see, learning is often viewed as a linear activity. The more time you put in, the more progress you make. Spend twice as long studying and you'll know it twice as well! ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/10/FullSizeRender.jpg) However, this is not how I've observed learning (specifically, understanding concepts) to take place. In reality, learning is much more accurately represented by a logistic function. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/11/FullSizeRender-11.jpg) That first section? That's when you struggle. That's the frustrating part because no matter how much time you put in, it doesn't seem to pay off. Then, after struggling with understanding a concept for a period of time, you have this "a ha" moment where you look at the concept in just the right light that you can glean some intuition that puts all the pieces together and all of the sudden it makes sense - you've mastered the concept. Unfortunately, most of us (including myself) at times view learning as a linear process (as depicted by the green line). And that can be frustrating when you're in the struggle phase, looking forward linearly. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/11/FullSizeRender-12.jpg) Sadly, it's easy to give in to these frustrations. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/10/FullSizeRender-3.jpg) But wait a minute, when you look at the big picture **that's the worst possible point to walk away**! You're so close to making a great deal of progress! That eureka moment would come with just a little more struggling. I can't tell you how frustrating it is from the viewpoint of a teacher to see someone give up at that point. To say, "oh, well I guess I'm just not an (insert subject of study) person." Alas, struggling is perceived to be a negative thing. We celebrate people in school who progress effortlessly, not realizing that the "struggling" phase is where almost all of the learning takes place. Hell, I've taken classes that were so easy I've walked away not remembering a single thing we talked about. And yet I still am able to remember the intuition behind a logarithmic function because I spent a good 4 days dissecting it (read: struggling) sophomore year of high school. Moreover, the learning process isn't just one logistic curve- it's a series. After you've mastered one concept, you study it at a deeper level or continue to learn about related concepts. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/11/FullSizeRender-4.jpg) ##### Intrinsic motivation In theory, the learning process is a fine-tuned machine where the "a ha" moment from your last concept motivates you to push through the struggle phase of the next concept you're trying to learn. I urge you to place trust in the learning process, understand that it is perfectly okay to struggle, and even though it doesn't feel like you're making progress, the struggling is beneficial. When you're struggling, whether it's a concept from something you learned about in school or it's learning how to sail a boat, you're taking in a ton of information. Your brain is being inundated with data but it's not always capable of processing this data in real time. This is often why taking a break can be so beneficial. I remember when I was practicing drums in high school and I'd be struggling over a specific pattern for an hour and not be able to play it - but after a short break and five minutes of practice I could play it no problem. I hope my mental model of the learning process can provide you with the courage to push through times of struggle, and encourages you to fine-tune your own learning process so that the motivation from your "a ha" moments push you through the following period of struggle. Next time you're struggling, just remind yourself it's part of the process. It's something I frequently remind myself. ##### Yeah, but where's the science to back this up? After sharing this post on Facebook, a good friend of mine challenged me to back up this viewpoint with science. I dug up a really interesting paper, [The Aha! Moment: The Cognitive Neuroscience of Insight](https://www.researchgate.net/publication/233507940%5FThe%5FAha%5FMoment%5FThe%5FCognitive%5FNeuroscience%5Fof%5FInsight?ref=jeremyjordan.me), to compare their findings with my observations. The study compares problem-solving brain activity between an analytical-based approach and an insight-based approach towards developing a solution. The model for learning I presented in this post would be considered an **insight-based approach**. So what's going on in your brain during an "a ha!" moment? In short, your brainwaves are in an alpha-state followed by a quick burst of gamma. > "EEG showed that insight solutions were associated with a burst of high-frequency (i.e., 40-Hertz gamma-band) activity starting about 300 milliseconds before the button-press signaling that a solution was derived... Immediately prior to the burst of gamma-band EEG activity was a burst of slower, alpha-band (approximately 10 Hertz), activity measured over right occipital cortex." Alright, so we can quantify when this "a ha!" moment occurs, but what leads up to this moment? The paper then goes on to talk about "a prepared mind" for an "a ha!" moment. > "One prominent view of creativity is that it is based on the processing of remote or loose associations between ideas (Mednick, 1962). Recent research implicates the brain's right hemisphere in the **processing of remote associates** and the left hemisphere in the processing of close or tight associations (for a review, Jung-Beeman, 2005). We therefore predicted greater activity in right-hemisphere regions associated with lexical and semantic processing. Second, based on previous findings suggesting that individuals high in creativity habitually **deploy their attention in a diffuse rather than a focused manner** (Ansburg & Hill, 2003), we predicted greater diffuse activation of the visual system in high-insight participants (corresponding to less posterior alpha-band and beta-band EEG activity). Both of these predictions were supported by the data, demonstrating that task-related problem-solving strategies have their origins in individual differences in resting-state brain activity." The struggle phase is where diffuse activation and processing of remote or loose associations take place. ##### Update Since writing this post I've come across the following video related to this post, it's wonderful! ### What is x-ray diffraction? URL: https://www.jeremyjordan.me/what-is-x-ray-diffraction/ Last updated: 2016-10-28T08:55:49.000Z X-ray diffraction is a common materials characterization technique that allows for identification of crystal orientations and interatomic spacing. X-rays are used for this because the wavelength is on the same length scale as interatomic spacing and lattice parameter values. ##### What's going on? An x-ray is generated in a vacuum tube by heating up a tungsten filament past its work function to eject electrons. Vacuum conditions are necessary in order to increase the electron mean free path. The electrons are accelerated into a copper cathode with an energy on the order of 25 keV. Electron collisions with the copper cathode result in inner shell ionizations, producing x-rays as higher-energy electrons drop to fill the lower-energy vacancy. Electrons from various higher energy levels drop, producing electromagnetic waves of varying energy. Thus, x-rays of multiple wavelengths (of type K-beta and K-alpha) are generated . For x-ray diffraction, it is ideal to have monochromatic light interacting with the sample, so the K-beta x-rays are filtered out with a nickel plate. Although there are indeed two associated K-alpha energies, the energies are so similar that it does not significantly affect the overall peak profile. The x-rays pass through a collimator to adjust the beam width. ##### A little background with Bragg's law Bragg’s Law describes the relationship for constructive interference, where x-rays of wavelength λ incident on the material at angle θ are diffracted by crystal planes separated by distance d and n represents an integer. \[n\\lambda = 2d\\sin (\\theta )\] As shown see below, a phase shift in the wave causes either constructive inference, as shown on the left, or destructive interference, depicted on the right, according to the 2θ angle. ![Figure 1: Constructive Interference Depiction](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/10/Braggs_Law.svg.png) ##### Equations for interpreting XRD results The counter, seen below in a schematic of an x-ray diffractometer, is rotated over a range of 2θ values and records the x-ray signal intensity reflected from the crystal at each position. This intensity measured as a function of 2θ is used to generate a peak profile. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/10/Untitled1.png) For angles that satisfy the Bragg condition, a peak in x-ray signal will be observed due to the constructive interference. These peaks are then used to identify crystal directions and calculate lattice parameters. For cubic systems, the interplanar spacing, d, may be calculated by the following equation derived from the crystal geometry, where a is the lattice parameter and h, k, and l are the Miller indices. \[{d\_{hkl}} = \\frac{a}{{\\sqrt {{h^2} + {k^2} + {l^2}} }}\] Combining Bragg's law and the interplanar spacing yields the following equation. \[\\frac{{{\\lambda ^2}}}{{4{a^2}}} = \\frac{{{{\\sin }^2}({\\theta \_{hkl}})}}{{{h^2} + {k^2} + {l^2}}}\] Here, we've moved all of the constants to the left side of the equation, allowing us to compare ratios of sin2(θ) values and Miller indices between different peaks. When the incident electromagnetic wave impinges on the sample material, electrons residing with the material are perturbed into an oscillating pattern, which generates new x-rays and effectively scatters the incident light waves. Two types of scattering occur: coherent or elastic scattering involves tightly-bound oscillating electrons that scatter x-rays of the same wavelength, and incoherent scattering which occurs when loosely-bound electrons are set into oscillatory motion and generate x-rays of different energy. X-ray diffraction measures the intensity of coherently scattered x-rays, which peaks where waves constructively interfere. The atomic scattering factor may be defined as a ratio of the amplitude of the wave scattered by an atom divided by the amplitude of the wave scattered by a single electron. This ratio essentially describes the scattering efficiency of an atom for a given direction. ##### Example of XRD data Here's the data collected from an experiment I did during my undergraduate lab course trying to identify an unknown metal based on its crystal structure. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/10/Picture1.png) ##### How do I interpret these peaks? | Crystal Structure | Conditions | | ------------------- | --------------------------------------- | | Simple Cubic | all h, k, l values | | Body Centered Cubic | h+k+l must be even | | Face Centered Cubic | h, k, and l must be all odd or all even | Structure factors account for instances where a plane may exist halfway between two neighboring planes and causes destructive interference to occur. These factors result in a set of rules for different crystal structure geometries which determine when constructive interference may occur. The table above lists the Miller index parameters in which constructive interference will occur. A more extensive table is shown below. By comparing ratios of sin2θ (denoted as $Q^2$ in the second table) for different diffraction peaks, the crystal structure and Miller indices may be determined. | Crystal Plane (hkl) | $Q^2$ | Space lattices from which reflections are possible | | | | | ------------------- | ----- | -------------------------------------------------- | --- | --- | -- | | 100 | 1 | SC | | | | | 110 | 2 | SC | BCC | | | | 111 | 3 | SC | | FCC | DC | | 200 | 4 | SC | BCC | FCC | | | 210 | 5 | SC | | | | | 211 | 6 | SC | BCC | | | | – | 7 | | | | | | 220 | 8 | SC | BCC | FCC | DC | | 300, 221 | 9 | SC | | | | | 310 | 10 | SC | BCC | | | | 311 | 11 | SC | | FCC | DC | | 222 | 12 | SC | BCC | FCC | | | 320 | 13 | SC | | | | | 321 | 14 | SC | BCC | | | | – | 15 | | | | | | 400 | 16 | SC | BCC | FCC | DC | X-ray diffraction has a wide range of applications for materials characterization. It may be used as an experimental method to experimentally determine the volume of a unit cell for theoretical density calculations. Additionally, if the material is under residual stress the distorted lattice will exhibit a range of d-spacings that satisfy Bragg’s law, resulting in much wider diffraction peaks. The diffraction peak width may also be used in determining particle size. ### How does scanning electron microscopy work? URL: https://www.jeremyjordan.me/scanning-electron-microscopy/ Last updated: 2016-10-27T21:42:12.000Z Ever wonder how we can see things so small, the human eye doesn't even know it exists? Today I'll be writing about how scanning electron microscopes allow us to see far beyond what traditional microscopes can provide. You see, optical microscopy (which uses visible light waves for imaging) only allows us to resolve features 250 nanometers and larger due to diffraction limitations. You can read more about this diffraction barrier [here.](https://www.microscopyu.com/techniques/super-resolution/the-diffraction-barrier-in-optical-microscopy?ref=jeremyjordan.me) Scanning electron microscopy is able to achieve a much finer resolution because we're basically shooting a focused beam of electrons at a sample instead of light, and this beam of electrons has a much smaller wavelength than visisble light, allowing us to greatly reduce the diffraction limitation. #### How the machine works The picture below displays a schematic of a typical scanning electron microscope. Essentially, an electron beam is generated, focused, and then used to scan across a sample in a raster pattern. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/10/sem1.gif) [*Image credit*](https://cmrf.research.uiowa.edu/sites/cmrf.research.uiowa.edu/files/styles/large/public/sem1.gif?itok=uzYuMy9J&ref=jeremyjordan.me) The two main types of electron sources are thermionic and field emission. Thermionic sources typically use a tungsten filament which is heated to 2700 Kelvin, sufficient to overcome the filament’s work function, allowing electrons to exit the material and thus providing a source for an electron beam. Field emission sources take advantage of quantum tunneling to create an electron beam. Field emission sources are capable of producing sub-nanometer resolution images, however, this source type is much more expensive and usually cost prohibitive. In order to achieve a sufficient electron mean free path, the SEM chamber must be pumped to low vacuum conditions. Generally, the length of the chamber is one meter, which requires pressures on the order of $10^{-4}$ torr. The electron beam passes through a number of electromagnetic lenses to demagnify and focus the beam, allowing a higher resolution for more detailed imaging. An objective aperture controls the electron beam by means of limiting electrons far from the beam center. A smaller aperture will shrink the electron probe radius, but will also reduce the current as it essentially removes the electrons not near the beam center. A stigmator is used to ensure the electron beam is not elliptical in nature, but perfectly round; this allows us for consistent imaging in all axises. Lastly, an objective lens is used to focus the electron beam onto the sample. ##### What you can learn from an SEM image When the electron beam reaches the sample, a number of interactions occur. Some electrons elastically collide with and bounce back off the surface, known as backscattered electrons. This electron signal provides mainly **compositional contrast**. A grayscale image is created according to the number of backscattered electrons at each position; larger atoms will backscatter more electrons and thus be displayed brighter in the image. Secondary electrons are produced by inelastic electron collisions with the lattice. For this interaction, an incident electron ejects an electron from the sample. The electron ejected from its resident shell in the sample is considered a secondary electron, and has considerably less energy than the incident electron. A similar grayscale image may be produced by counting the number of secondary electrons ejected at each position. Secondary electrons are the most commonly used signal from imaging and provide good **topographical detail**. **Elemental makeup** of a sample can be determined by energy dispersive x-ray spectroscopy. When the high energy electron beam strikes the surface, it has the ability to eject an electron resident in the material from its shell. When an electron from a higher level energy shell drops to fill the created hole, it emits energy in the form of a light wave, usually in the x-ray range, which has an energy characteristic of the material’s discrete energy difference between levels. Thus, the emitted x-rays can be used to identify the different atoms which make up a material. X-ray signals at each position in the scanning area may be compiled to generate a compositional map of the area, which can be useful when identifying phases within a material. ### Identifying related bodies of text using TF-IDF vectorization. URL: https://www.jeremyjordan.me/identifying-related-bodies-of-text-using-tf-idf-vectorization/ Last updated: 2017-10-25T16:15:33.000Z Term frequency-inverse document frequency (TF-IDF) vectorization is a mouthful to say, but it's also a simple and convenient way to characterize bodies of text. Due to its simplicity, this method scales better than some other topic modeling techniques (latent dirichlet allocation, probabilistic latent semantic indexing) when dealing with large datasets. ### TF-IDF Vectorization Let's say we have a collection of documents: - **Document 1:** "TF-IDF vectorization is a useful tool for text analytics. It allows you to quantify the similarity of different documents." - **Document 2:** "You can use cosine similarity to analyze TF-IDF vectors and cluster text documents based on their content." - **Document 3:** "The weather today is looking quite bleak. It's a great day to write a blog post!" It's clear to us that Document 1 and 2 are related, but how can we train an algorithm to identify this? First, let's split up the documents into a collection of words. This is known as tokenization - we'll split up the sentences into a list of words, removing capitalization (so "You" and "you" are treated as the same thing) and ignoring punctuation. See the pseudo-code example below. ``` >> tokenize("TF-IDF vectorization is a useful tool for text analytics. It allows you to quantify the similarity of different documents.") "TFIDF", "vectorization", "is", "a", "useful", "tool", "for", "text", "analytics", "it", "allows", "you", "to", "quantify", "the", "similarity", "of", "different", "documents" ``` ### Term Frequency Term frequency refers to how often a term (or token) occurs within a given document. Since my example documents are quite small, we don't observe a term that appears more than once. But if I treated an entire blog post (say, the one you're currently reading) as a document, you could see how terms such as "TFIDF" and "similarity" would appear more often than terms such as "blog." **In short, term frequency describes how *popular* a term is within a single document.** ### Inverse Document Frequency Document frequency counts the number of occurrences for each term across all documents. However, if a term is popular across all documents, it doesn't provide much information relative to a single document. The token "you" might appear often and have a high term frequency for a given document, but it likely appears often across all documents so it does little to provide any information. An extreme case of this are stopwords, which are terms such as "a", "the", "you", and "like" which usually provide so little information they are discarded upon tokenization. Thus, we calculate the inverse document frequency so that the terms that appear most often are weighted down. Another way of thinking about the IDF is that it establishes a scoring mechanism for the usefulness of a given term - higher scores mean more information is present. **In short, inverse document frequency describes how *unique* a term is across the collection of documents.** Let's imagine that my document examples were each a paragraph long and the following table represents how often a few terms occur in each document. ### Term Counts | | "TFIDF" | "is" | "a" | "similarity" | "weather" | "to" | | ---------- | ------- | ---- | --- | ------------ | --------- | ---- | | Document 1 | 4 | 9 | 12 | 6 | 0 | 3 | | Document 2 | 5 | 7 | 15 | 3 | 0 | 2 | | Document 3 | 0 | 8 | 19 | 1 | 3 | 4 | | Total | 9 | 24 | 46 | 10 | 3 | 9 | The rows in these tables can be represented as N-dimensional term frequency vectors where the last row represents the document frequency. To find the TF-IDF vector for each document, we simply normalize the term frequency and inverse document frequency vectors and multiply them together. \[ 4 34 9 34 12 34 6 34 0 34 3 34 \]∘\[ 101 9 101 24 101 46 101 10 101 3 101 9 \]\= TF-IDF vector for Document 1 *Notes: Although this example uses linear frequency to help visualize how the pieces fit together, in practice you should use the log-frequency. Log-frequency lessens the difference between large term counts (ie. 1 vs 20 is treated as a larger difference than 100 vs 150). Also, this is element-wise multiplication. The TF score for "similarity" is multiplied by the IDF score for "similarity."* ### Cosine Similarity Now that we have a TF-IDF vector for each document, we can quantify the similarity between two vectors using cosine similarity. The image below is quite instructive for providing the motivation behind this technique. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/10/vector_space.png) *[Image credit](http://blog.christianperone.com/wp-content/uploads/2013/09/vector%5Fspace.png?ref=jeremyjordan.me)* Thus, if two documents contain similar language (tokens) they will be closer together in the visualized N-dimensional vector space. We can calculate the cosine of the two vectors to determine how close the vectors are with respect to their orientation. The reason we take the cosine of the angle between two vectors rather than just measuring the distance between vectors is illustrated below. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/10/Screen-Shot-2016-10-07-at-8.59.58-PM.png) Although the two red vectors are very different in magnitude, they appear much more similar than vectors 1 and 2 (which are closer in distance.) To account for varying lengths in documents, we also normalize each TF-IDF vector with respect to its length. After length-normalizing the two TF-IDF vectors you wish to compare, you can simply take the dot product to find the cosine similarity cos( q ⇀ , d ⇀ )\= q ⇀ • d ⇀ \= ∑ i\=1 n q i d i where *q* and *d* are the two documents and *n* represents the total number of terms in the global documents vocabulary. Note: these TF-IDF vectors are generally very sparse, as an individual document will only contain a small subset of the vocabulary found across the collection of documents. If you don't know how to optimize operations dealing with sparse vectors, I'd recommend using SKLearn's TF-IDF vectorizer. In fact, I'd recommend that module regardless because it works really well and has a number of features, such as tokenization, n-grams (which creates tokens for collections of words in addition to individual words), and stopword removal. ### Example Code Here's a quick example using SKLearn's TF-IDF vectorizer module. ``` from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.metrics.pairwise import cosine_similarity tfidf_vectorizer = TfidfVectorizer() print "Fitting TF-IDF matrix..." tfidf_matrix = tfidf_vectorizer.fit_transform(documents) print "Calculating cosine similarity..." dist = 1 - cosine_similarity(tfidf_matrix) ``` ### Rejection is not a setback, it's a redirection. URL: https://www.jeremyjordan.me/rejection-is-not-a-setback-its-a-redirection/ Last updated: 2016-09-20T13:28:35.000Z I decided to write this post after I found out that I was not selected to go on a trip sponsored and subsidized by my school. It was a really great opportunity to visit startups in New York that I was looking forward to, but there were only a limited number of seats available on the trip (understandably so, seeing as it was highly subsidized) and my application was not selected. This isn't the first time I've applied to something and been rejected, it's happened countless times throughout college. However, I didn't decide to write this post because I've experienced rejection. I decided to write this post because I observed how I dealt with it. See, when I found out that I wasn't chosen for this great opportunity, I had about two seconds where I was really bummed and then I immediately started brainstorming other ways to get a similar experience. Turns out, a flight from Raleigh to NYC is pretty cheap, and I have a dear friend who's studying at Columbia University who has said he'd love for me to visit. In reality, I could make this trip happen on my own accord. The way I look at it, rejection isn't a setback - sometimes it's just a way of saying "you're gonna have to be more clever than this." It's a redirection. Going back to my [river analogy](https://www.jeremyjordan.me/life-is-not-a-linear-progression), sometimes you come across resistance in your path and you have to change course and go around. Just look at the photo below. Why not just follow a path from point A to point B? Well, sometimes you can't. Each gold star along the path is a rejection, where the river was forced to go around and find another way. While the path may be arduous, rejection shall serve no more purpose than to redirect a determined mind. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/09/CO_PhotoProject2007_0531-1.jpg) *[Photo credit](https://cdn-co.milespartnership.com/sites/default/master/files/CO%5FPhotoProject2007%5F0531.jpg?ref=jeremyjordan.me)* ### The lotus leaf: how nature makes water-repellent materials. URL: https://www.jeremyjordan.me/lotus-leaf-how-nature-makes-water-repellant-materials/ Last updated: 2016-10-21T16:01:10.000Z Ever wonder what makes a material water-repellent? Hydrophobic materials can be useful in a myriad of applications, basically anywhere where you don't want your things to get wet. To figure out how to engineer materials with better hydrophobic properties, scientists have turned to nature (specifically, the lotus leaf) to study its intelligent design. Lotus leaves (pictured below) are well known for being superhydrophobic, so they serve as an excellent case study. ![Lotus Leaf](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/09/lotuseffect_feature.jpg) *Note: the concept of studying nature's design and using solutions developed by nature over thousands of years is known as *biomimicry*.* ##### Surface energy Before we dive into the "Lotus effect" (superhydrophobicity), it is important to have an understanding of a concept called **surface energy**. Imagine the animation below displays an infinite array of atoms bonded together - an infinitely large piece of material. If we were to cut a shape out of this material we'd be creating a surface, and in creating a surface we would have to break bonds between atoms (represented below as blue circles). Atoms form bonds because it allows them to exist at a lower-energy state, so it takes energy in order to break these bonds and create a surface. This energy required to break the bonds is known as surface energy. ![Surface Energy](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/09/ezgif-2440375922.gif) *Animation illustrating the energy associated with creating a surface.* Now that we have an intuitive understanding of surface energy, let's look at a water droplet. ![Water Droplet](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/09/black-and-white-waves-close-up-view-circle--1-.jpg) Any guesses why a water droplet would form a sphere when suspended in free space? *Hint: a sphere has the least possible amount of surface area for a given volume.* By assuming the shape of a sphere, a water droplet can minimize its surface energy! Another way this water droplet can lower its surface energy is through interfacial interactions (read: when two surfaces touch) with another material. This second method is known as [wetting](https://en.wikipedia.org/wiki/Wetting?ref=jeremyjordan.me), and can be represented mathematically by Youngs Equation. Notice in the image below how the water droplet on the leaf is no longer a perfect sphere? That's due to favorable interfacial interactions between the water droplet and leaf. The contact angle arises from a balance between minimizing energy through interfacial interactions and geometrical configuration. ![Wetting](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/09/drop-of-water-drip-blade-of-grass-blossom-53563.jpeg) ##### Lotus leaf Now let's take a closer look at the lotus leaf and examine what characteristics give rise to its superhydrophobic properties. To learn more, I read the following paper: [Superhydrophobicity in perfection: the outstanding properties of the lotus leaf](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3148040/?ref=jeremyjordan.me). If this post is interesting I encourage you to read this paper to learn more! The surface of the lotus leaf has a hierarchically rough structure, as shown below in (b) and (c), which discourages wetting (as explained in further detail below.) The tubules shown in (c) are made of a wax that contains non-polar methyl groups. This non-polar wax does not favor interactions with water (a polar substance.) ![Lotus leaf structure photos](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/09/Screen-Shot-2015-11-08-at-3.42.37-PM-618.png) *Image credit to paper referred to above.* To understand why a rough structure discourages wetting, recall that a material tries to minimize its surface energy through surface interactions and geometrical configurations. If we introduce a rough structure, the material has to significantly increase its surface area in order to squeeze into the gaps and interact with the surface. Thus, the material isn't as effectively able to lower its energy through interfacial interactions so it relies more on geometrical optimizations (assuming the shape of a sphere.) ![Rough structure](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/09/Screen-Shot-2016-09-06-at-12.04.00-AM.png) However, the lotus leaf takes this concept a step further because the papillae shown below exist at uneven heights. This decreases water adhesion to the leaf's surface even more, forcing the water to bead up and easily roll off. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/09/Screen-Shot-2016-09-06-at-12.58.32-AM.png) The epicuticular wax, shown above, has non-polar methyl groups (CH3) at the surface which also discourage leaf-water interactions due to the polar nature of water. (Remember, like substances dissolves like substances- which is why nonpolar oil won't mix with polar water.) These wax crystals are stabilized by hydrogen bonding among hydroxyl groups (OH), allowing the wax to have a higher than average melting temperature (90-95 °C). ![Wax structure](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/09/Beilstein_J_Nanotechnol-02-152-g014.jpg) *Image credit to paper referred to above.* ##### Summary In summary, the structure of a lotus leaf has a clever design that discourages surface wetting both through a waxy nonpolar coating and a rough structure which both make surface interactions less favorable, allowing water to bead up into spheres and easily roll off the leaf. By studying the waterproofing techniques of a lotus leaf, we can develop coatings for electronics to [waterproof your next smartphone](http://www.hzo.com/?ref=jeremyjordan.me). These same techniques can be applied for other uses as well. For example, Dr. Bharat Bhushan of Ohio State University has [developed a material](http://phys.org/news/2015-04-scientists-mesh-captures-oilbut.html?ref=jeremyjordan.me) that "if scaled up, you could potentially catch an oil spill with a net." ##### Wait, how does the oil spill-catching net actually work? Bhushan uses the same two design principles we learned about when studying why the lotus leaf has such incredible self-cleaning properties: make the surface rough to discourage "wetting", and choose a surface material based on the type of materials you wish you repel. For the lotus leaf, we wanted water to roll off the leaves, bringing any dirt along with it. For Bhushan's oil catching net, he wants to allow water to pass through while keeping oil contained by the mesh. Thus, rather than coating the surface of his net with waxy tubules (nonpolar - repels water) like the lotus leaf, Bhushan deposits a polar material on the surface to allow the passage of water but prohibit oil (nonpolar) from passing through the net. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/09/ezgif-1439904965.gif) ### Personal reflections on self-confidence. URL: https://www.jeremyjordan.me/how-to-gain-confidence-in-your-self/ Last updated: 2016-09-04T10:07:16.000Z This past Friday night I went home to cook dinner with my mom and spend some time with her as the school semester is winding down. After dinner, we were chatting and I told her that I see myself following the entrepreneurial path after graduating college. I said, "You know, somewhere along the way I gained the confidence in myself to be able to overcome any challenge I set for myself, to solve any problem I set my mind to. While I know that I could graduate and get a very well-paying job and live a comfortable life... I don't think I would be happy living a comfortable life. I've found that my happiness stems from growth, and the best way to grow is to continually challenge myself. I feel like I can accomplish some amazing things in my lifetime, and I believe entrepreneurship is the best path for me to do so - working on our world's most pressing problems." She agreed with me, but she was also curious and asked where/what I thought that "somewhere along the way" was to develop such a confidence. Well, I've been thinking on that subject and here are a few of my previous experiences that I think contributed to my bold mindset. *Keep in mind that this is not blanket advice to everyone. This is simply the things I have experienced to develop the mindset I have today.* ### Understand struggling is part of the learning process. This has been something that I've learned not because of one experience, but many. Nonetheless, it's been an incredibly beneficial realization. When I learn something new, I often try to struggle as much as possible, in as many ways as possible. What I think many people don't realize is that they are learning when they struggle- but they don't ***feel*** like they're learning because they aren't yet able to see the results. A great deal of learning takes place while you're struggling; when you do something new, your brain is taking in a lot of information that may not be immediately processed. The way I see it is that I should load up my brain with as much information as possible so there's more data to learn from. Rather than not seeing results and giving up, I place trust in the learning process and know that my efforts will be rewarded. ### Learn how to endure pain. When I look back on my childhood, there were many times when I experienced pain but didn't stop what I was doing. Why did I keep going? I mean, I don't particularly enjoy being in pain, but I was having too much fun to worry about the pain. I remember countless summer days that my friends and I would go ride on mountain biking trails for hours on end, pushing way past the point that we were tired because we were having a blast. Whenever I would crash or fall, I'd get right back on the bike and keep riding - not ready to quit. When we finally made the grueling journey back home (it stopped being fun when we got off the trails and had to bike for an additional 30 minutes to get back home), we would collapse onto my driveway and lay there because that was literally the only things our bodies would let us do. I've also enjoyed playing soccer, and there have been many times where I have been hit with the ball in painful places, stepped on with cleats, etc. But I rarely stopped after I got injured, **I kept playing because it was fun**. It's not that I don't feel the pain, I just don't choose to pay attention to it. I think there's an important lesson in that - both in developing the ability to push through the pain and in finding the motivation to keep going. ### I think everyone should jump off a cliff naked at least once in their lives. So, I provide this advice mainly in a figurative sense, but one way to gain this experience is to go out and literally jump off a cliff naked - at least that's how I did it. Back in high school, there was this rock quarry that everyone loved going to and jumping off into the water during the summer. One time I went, I decided to jump off completely in the nude and I'm glad I did. I'm not the only crazy guy to jump off a cliff naked. Shane McConkey, a world-renowned professional skier, has also been known to fly off ski jumps with his signature giant naked spread-eagle. The first time he flew off a ski jump wearing nothing more than his ski boots was immediately after attempting to do a double backflip and failing, landing on his head. However, when he hiked back up the cliff and made the jump in the nude, he landed the trick. People waste so much time and energy worrying about such insignificant things. It can be quite empowering to not only overcome a fear but to do so in your most vulnerable state. This provides a perspective on things and helps you worry about what really matters. Be bold, stop caring so much what other people may think, and test your limits. ### Establish a support network, but also occasionally interact with non-believers. I'm incredibly fortunate to have a such a supportive community of family and friends. However, every once in a while I talk with someone and I know that they think I'm crazy or are thinking, "OK, but he doesn't *really* mean that." I do not blindly dismiss people that don't believe in me - if they have a good point I'll definitely consider it, but the non-believers get to go on a list of people I'll get to prove wrong. And yeah, that can be fun too. That said, I can't express how thankful I am to have such a supportive community of friends, family, and mentors that believe in me. Having a support network is essential for when the times get tough; when you're at your weakest point, it can sometimes be a much-needed boost of confidence to hear "I don't know how you're going to do it, but I know you'll figure it out." ### Learn effective communication, because you can't do it all on your own. It's much easier to have confidence that you'll be able to accomplish your goal if you have a team, all with different backgrounds and strengths, that you can trust to work effectively. This will not happen if you cannot effectively communicate with each other. I know that I'm not an expert in a lot of areas, but to that note, I am confident in my ability to both communicate my ideas to others and understand the ideas other people attempt to communicate with me. There are a few books that I found helpful in learning effective communication techniques which I'll list below. However, these books won't help you unless you go forth and converse with others. - Crucial Conversations: Tools for Talking When Stakes Are High - Leadership and Self-Deception: Getting Out of the Box ### Travel. You'll run into problems in a completely foreign environment and be forced to come up with a solution. Better yet, travel alone. The summer after my freshman year of college I studied in Spain and spent two weeks traveling alone throughout Spain and France. After my latest trip to Europe, I remembered how beneficial it can be to just go out and explore. Half the time you don't know what you're getting yourself into, but you always make it out alive and more confident in yourself to explore new environments. ### Life is not a linear progression. URL: https://www.jeremyjordan.me/life-is-not-a-linear-progression/ Last updated: 2016-10-08T16:43:31.000Z Rather, it's a grand adventure. All too often during my college career, I find both my peers and myself under great pressure and stress trying to figure out what we want to do and how we want to live our lives. It's easy to find ourselves looking at life as if it is a linear progression, seeing as the formula for success in college is commonly presented as follows: 1. Do well on your SAT in high school so that you can get into college. 2. Keep a high GPA in college so you can be competitive for jobs or grad school. 3. Get an internship. 4. Graduate on time and get a job. 5. And the list goes on... This can lead to major breakdowns when things don't go according to plan, and things rarely go according to plan. In this article today, I'd like to propose an analogy for life, something easy to visualize, that we can look to as inspiration for when things get a little crazy: Life is a meandering river. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/09/Dep-33.jpeg) Taking a step back and looking at the bigger picture, our sometimes-linear viewpoint is only a result of a localized vision. Like a river, **our lives are a meandering path** changing by a few degrees at a time which sometimes leads to big sweeping changes. Rarely is a river linear for more than a few hundred feet. However, it's easy to feel otherwise. By extrapolating your vision based on your current status, it is easy to feel as if you're heading in a straight line. Simply, there's no way of knowing how the future unfolds so we give it our best guess based on our current situation. Nature is very efficient, so why don't rivers just go straight from point A to point B? Wouldn't this be the most efficient path, due to the fact that water would have to travel the least distance? Moreover, why shouldn't we do our best to travel straight from point A to point B, wasting the least amount of time along the way? As rivers can teach us, there must be some value, some reason, to meander. Well, for starters, where is your point B? What is your end goal? It's not always clear. Just like a river, our lives are an exploration - we live and we learn and we adjust our path as we gain new experiences. **This** is where the value lies in deviating from our "linear" path. In fact, this is what college is all about. With so many opportunities to learn about various disciplines and challenge ourselves, these years prove fruitful in discovering ourselves. Embrace this time of exploration, orient your direction, and work relentlessly towards that vision. *Keep in mind:* rivers may explore, but they also keep moving in the forward direction. Not knowing where you'll end up is no excuse for not developing a vision and working towards that general direction. So next time you start to get stressed because something didn't go according to plan, relax. Remember life is like a meandering river; deviate from the path and seek out those experiences. Always be aware and *intentional* of the direction you are heading, but be comfortable with the fact that as you gain more experiences your directional orientation may change. Live life curious, and enjoy the adventure. *Disclaimer:* this viewpoint is simply my own opinion, and I welcome a healthy discourse in the comments below. I tend to be biased towards this mindset as I am a huge proponent of lateral (read: cross-disciplinary) thinking. Thus, these explorations provide immense value in widening my perspective and connecting together seemingly unrelated ideas. ##### Update I recently came across this video (below) which talks about the same concept with a slightly different analogy - the "Tarzan Method". The analogy starts at 24:34, although the whole video is worth watching. ### My whiteboard stand-up desk. URL: https://www.jeremyjordan.me/my-whiteboard-stand-up-desk/ Last updated: 2018-07-31T17:53:34.000Z As a student who frequently spends long hours at my desk studying and doing homework, I often wish I could get up and move around or at least not be sitting all day. You've probably seen a number of articles detailing the negative health effects of sitting. So, this January I decided to treat myself. I put together a whiteboard stand-up desk and I love it so much I am writing this article so that you can enjoy it too! For a while, I didn't think it was feasible to purchase such a desk, seeing as many of them were thousands of dollars and after all, I'm a college student living on a budget. Upon researching my options, I found that I had three basic options: [hack together a permanent standing solution for $25](http://iamnotaprogrammer.com/Ikea-Standing-desk-for-22-dollars.html?ref=jeremyjordan.me#comment-689429716), [purchase an adjustable standing solution (Varidesk) for $395](https://www.amazon.com/Height-Adjustable-Standing-Desk-VARIDESK-Black/dp/B00JI6NCCK/ref=pd%5Fsim%5F229%5F2?ie=UTF8&dpID=41cHEKfeC%2BL&dpSrc=sims&preST=%5FAC%5FUL160%5FSR160%2C160%5F&refRID=12QC11G1NJNB3TP1KSK6&ref=jeremyjordan.me), or purchase an actual desk that raises and lowers. Many of these are within the $600 to $1,000 range. I was leaning towards purchasing the full desk instead of the first two options because I wanted to be able to have my notes and papers at my disposal when standing, rather than only being able to elevate my computer when standing. However, I came across [StandDesk](http://mbsy.co/standdesk/20944518?ref=jeremyjordan.me) which offers an electric-powered standing desk frame for $400\. No longer did I have to choose whether or not the benefits of having the entire desk raise and lower would be justified by the higher cost. Moreover, I was able to purchase just the frame, which let me put a custom top on. In searching, I also came across [this article](http://www.ikeahackers.net/2012/10/whiteboard-desk.html?ref=jeremyjordan.me) in which someone put together a stationary whiteboard desk, which I thought was pretty neat. Naturally, I decided my desktop had to be a whiteboard as well. And thus my remote-controlled, whiteboard stand-up desk was born. I gotta admit, it's pretty sweet. And I love working at it! See below to see how I put my desk together. ### Update After a few years with this desk, there's a few things I should warn you about. - The IKEA tabletop that I recommend will not support hanging a keyboard tray under the desk as the material is not strong enough to support it. - StandDesk customer support is **absolutely terrible**! The orders are slow to ship and their support team is helpless. One of their products offered free shipping on the product page but was charging me for shipping in the shopping cart. When I contacted support they said there was *nothing* they could do to honor the free shipping which was advertised. The customer support agent provided me with a link to complete my purchase. Because I used the link provided, the website's automated system noticed that I had an abandonded cart and offered me $30 in shipping credit to complete my order - after I had completed my order with the link provided by the unhelpful customer support agent! Oh, the painful irony. - There are much quieter standing desk options available on the market now. I'll keep this blog post so that you can see what I did as reference, but I'd suggest you shop around for different products than what I use here. --- **Materials** - [Whiteboard desk](http://www.ikea.com/us/en/catalog/products/20251139/?ref=jeremyjordan.me#/20251356): $46 - [Upper tier](http://www.ikea.com/us/en/catalog/products/30139595/?ref=jeremyjordan.me): $25 - [Legs for upper tier](http://www.ikea.com/us/en/catalog/products/10267895/?ref=jeremyjordan.me): $14 (used 6 legs) - [Desk frame](https://www.standdesk.co/products/build-your-standing-desk?ref=jeremyjordan.me): $400 The desk frame was a breeze to assemble. StandDesk emails you a video walkthrough that is very detailed. It took maybe 15 or 20 minutes to put the desk together. I was **highly** impressed with the quality of the desk. You can get a good idea for how sturdy this desk is in their [assembly video](https://drive.google.com/file/d/0B7vd5ZSXR4tzdXgweURvZXNFckE/view?ref=jeremyjordan.me). I also decided that I would like a second tier for setting my monitor and laptop on. This way, I would have more space for using the desk as a whiteboard! Check out [this article](http://www.ikeahackers.net/2015/12/10cm-lift-desk-shelf-monitor-stand.html?ref=jeremyjordan.me) for my inspiration behind the second tier. I basically just screwed the 6 legs into the shelf and that was finished. For my shelf, I positioned the brackets so that they were 1/4 inch from the front and back and 1/2 inch from the side (as pictured). I will warn you, it was a little difficult to screw into the slippery high-gloss shelf. It's possible, just proceed carefully. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/09/AAEAAQAAAAAAAAXzAAAAJGQ3ZjhjNDRjLWQwZjQtNGY1MS05OTQ3LTI0NWQ0ZmZmYTEzMA.jpg) I really wanted my desktop to double as a whiteboard, so I decided to use the IKEA tabletop mentioned above. The only downside to this desktop is that it is particle-board, which isn't the best thing to screw into. On behalf of a recommendation from my grandfather, I decided to glue three pieces of wood to the bottom of the desk and attach the whiteboard to the desk frame by screwing into the wood instead of particle board. That way, if I need to disassemble the desk at any point, I will be unscrewing and re-screwing into wood rather than particle board. I attached the wood to the bottom of the tabletop using a combination of LiquidNail and particleboard screws. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/09/AAEAAQAAAAAAAAVxAAAAJGU3YTBiNWFjLTA5ZWMtNDlkNC05MmQzLTI4NDk3YmUyODI4ZA.png) The desktop screwed into the frame easily and soon enough I had my own standing whiteboard desk! I love being able to easily jot down my thoughts while working at the computer, whether I'm writing the logic sequence for a piece of code or working through homework problems, it comes in handy quite a bit! It's been about a month since I've started using it and I've had no complaints. For those wondering how easy it is to keep the whiteboard clean - markers erase easily and leave no residue behind. ![](https://storage.ghost.io/c/38/9a/389ab7af-1195-4ce6-9b1f-d9fb39ea9790/content/images/2016/09/AAEAAQAAAAAAAAU8AAAAJDg0NjUzMTIwLWY3NzItNDI0ZC1iMjNjLTI1NWI5Y2E0Yjg1Zg.jpg)