What I learned from Andrew Ng’s Deep Learning Specialization
I recently completed Andrew Ng’s Deep Learning Specialization on Coursera and I’d like to share with you my learnings. I signed up for the 5 course program in September 2017, shortly after the announcement of the new Deep Learning courses on Coursera. Dr. Ng famously said “AI is the new electricity”, and I am very honored to be one of the first to graduate from his Deep Learning specialization that will help us “build an AI powered society”.
Here is the outline of the five courses in the Deep Learning specialization:
- Neural Networks and Deep Learning
- Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization
- Structuring Machine Learning Projects
- Convolutional Neural Networks
- Sequence Models
Each course is organized by weeks so there is clear expectation of approximately how long it would take me to complete each course. Each week follows the same format: first video lectures, then a quiz with about 10 questions. Finally there are one or a few programming assignments. At the end of each week there is an optional video with interviews with experts in the AI / deep learning field.
Before I started this program I have already self-studied AI / machine learning for a while and was familiar with Python and linear algebra. You may need a refresher if you are new to Python or rusty on your linear algebra.
Protip: I take notes while watching the video lectures so when it comes time for the quizzes, it’s fairly easy for me to answer the questions in the quiz.
There were no mentors or coaches; however, I turned to the course discussion forum for help when I get stuck with homework assignments. The courses were very well designed: challenging enough to keep me interested but not so challenging that I would feel giving up.
One of the biggest challenges I found was to get the shapes of the inputs, outputs and parameters correct. So I often use a small whiteboard, or pencil / piece of paper to draw the network diagram and write out the shapes. Towards the end of the course Keras was a bit challenging as well.
Trending AI Articles:
Course 1 — Neural Networks and Deep Learning
I learned the basics of neural networks and deep learning, such as forward and backward progradation. This course has 4 weeks of materials and all the assignments are done in NumPy, without any help of the deep learning frameworks. This way we get a solid foundation of the fundamentals of deep learning under the hood, instead of relying on libraries.
Week 1 — Intro to deep learning
Week 2 — Neural network basics. Programming assignment: build a simple image recognition classifier with logistics regression. The network recognizes whether an image is a cat.
Week 3- Shallow neural network. Programming assignment: build a shallow neural network with just one hidden layer to classify the red/blue points in a flower shaped dataset. Compare and contrast its performance with the implementation of logistics regression.
Week 4 — Deep neural network. Programming assignment: build a two layer neural network to recognize whether an image is a cat.
Course 2 — Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization
I learned various techniques to tune the neural network hyperparameters and optimize the network. There are 3 weeks in this course:
Week 1 — how to initialize complex neural networks, the difference between train/dev/test sets, bias and variance issues in network models, dropout or L2 regularization techniques, and vanishing / exploding gradients etc. We also learned a technique called gradient checking which helps debug the implementation of back prorogation. There were 3 programming assignments: 1. network initialization 2. Network regularization 3. Gradient checking
Week 2 — optimization techniques such as mini-batch gradient descent, (Stochastic) gradient descent, Momentum, RMSProp, Adam and learning rate decay etc.
Week 3 — Hyperparameter tuning, Batch Normalization and deep learning Frameworks. This is when got an introduction to the open source library TensorFlow. The programming assignment was to build a neural network with TensorSlow to classify sign languages images, with about 71% accuracy. Up until now all the assignments were done in NumPy.
Course 3 — Structuring Machine Learning Projects
This is a short two week optional course focusing on helping students how to structure a deep learning project. There was no programming assignment but it was full of expert tips on deep learning that you won’t find elsewhere, for example:
- How to decide the train/dev/test split of your dataset
- How to define human-level performance
- How to handle mis-matched training and dev/test data sets
- How to apply transfer learning, multi-task learning and end-to-end learning.
Course 4 — Convolutional Neural Networks
This is a 4-week course focusing on computer vision using Convolutional Neural Networks (CNN):
Week 1- foundation of CNN. Learned about the terminologies used in CNN such as padding, stride and filter etc, basic operations of CNN such as pooling, and how to build multi-class classification using CNN. There were two homework assignments. In the first assignment, we implemented convolutional (CONV) and pooling (POOL) layers in NumPy, including both forward propagation and backward propagation. In the second assignment, we built a TensorFlow sign language model (continuation from Course 2 week 3 assignment), this time with CNN it achieved almost 80% accuracy.
Week 2-case studies of various research papers in CNN for the classic networks: LeNet-5, AlexNet, VGG, ResNet, Inception etc. We also learned about transfer learning, and data argumentation techniques such as mirroring, random cropping, rotation, shearing, local warping, color shifting and PCA color argumentation. We were introduced to Keras via a tutorial, and then homework assignment was to use Keras to implement a neural networks for image classification using the ResNets architecture.
Week 3-apply CNNs to object detection. Programming assignment was to implement car detection for autonomous driving. It was done in Keras using the YOLO (You Only Look Once) model to draw bounding boxes for object detection.
Week 4-Face recognition and neural style transfer. The first assignment was to generate art with style transfer and the second assignment was to recognize faces.
Course 5-Sequence Models
Course 5 taught me how to build models for natural language processing, machine translation, speech recognition and synthesis words and music. It lasts 3 weeks and all the assignments were done in Keras:
Week 1-Recurrent Neural Networks. I found RNNs challenging to understand after many online tutorials and other programs but Dr. Ng was able to explain RNN in a way that I could comprehend.
There were 3 programming assignments:
- building a recurrent neural network step by step.
- use character-level language modeling to generate dinosaur names.
- generate Jazz music with LSTM.
Week 2-Natural language Processing and Word Embeddings. We learned how to use word vector and embedding for sentiment analysis, named entity recognition and machine translation. It’s interesting to learn that just like humans, machine learning has gender, ethnicity and other biases; and we learned techniques for de-biasing.
We worked on these programming assignments:
- implement a model that takes a sentence input and output an emoji based on the input.
- use word embeddings to solve word analogy problems such as Man is to Woman as King is to __,
- modify word embeddings to reduce their gender bias.
Week 3-Sequence models and attention mechanism. We learned how to augment sequence models using an attention mechanism which helps the model understand where it should focus its attention given a sequence of inputs. There were two homework assignments: one on machine translation and the other one on trigger word detection.
