Deep Learning and Neural Networks Advanced Research Seminar I/III Graduate School of Information Science Nara Institute of Science and Technology January 2014
Instructor: Kevin Duh, IS Building Room A-705 Office hours: after class, or appointment by email (
[email protected] where x=kevinduh)
Course Description Deep Learning is a family of methods that exploits using deep architectures to learn high-level feature representations from data. Recently, these methods have helped researchers achieve impressive results in various fields within Artificial Intelligence, such as speech recognition, computer vision, and natural language processing. This course provides an overview of Deep Learning and Neural Networks; the goal is to establish a foundational understanding at a level sufficient for students to start reading research papers in this exciting and growing area. Prerequisites: basic calculus, probability, linear algebra.
Course Schedule Jan 14, 16, 21, 23 (9:20-10:50am) @ IS Building Room L2 Lecture 1 (Jan 14): Machine Learning background & Neural Networks Slides (pdf), Video (HD), Video (Youtube) Recommended reading: Chapter 5 of Bishop's book Pattern Recognition and Machine Learning Lecture 2 (Jan 16): Two Types of Deep Architectures: Deep Belief Nets (DBN) and Stacked Auto-Encoders (SAE) Slides (pdf), Video (HD), Video (Youtube) Recommended reading: Sections 1 & 4 of Bengio's monograph: Learning Deep Architectures for AI Lecture 3 (Jan 21): Applications in Computer Vision, Speech Recognition, and Language Modeling Slides (pdf), Video (HD), Video (Youtube) Recommended reading: [Hinton, et. al., IEEE SPM 2012], [Le, et. al., ICML2012], [Mikolov, et. al., Interspeech 2010] Lecture 4 (Jan 23): Advanced Topics in Optimization (Hessian-free optimization, Dropout, Large-scale distributed training, Hyper-parameter search) Slides (pdf), Video (HD), Video (Youtube) Recommended reading: [Martens, ICML2010], [Hinton, et. al., 2012], [Dean, et. al., NIPS2012], [Bergstra, et. al., NIPS2011] Two video options are available: [1] Video (HD) includes slide synchronization and requires Adobe Flash Player version 10 or above. [2] Video (Youtube) may be faster to load and is recommended if you have trouble with Video (HD). If you find errors, typos, or bugs in the slides/video, please let me know.
Useful References 1. Short surveys and tutorials: Yoshua Bengio's monograph (available online): Learning Deep Architectures for AI Yann LeCun & Marc'Aurelio Ranzato's ICML2013 tutorial (computer vision perspective) Richard Socher et. al.'s NAACL2013 tutorial (natural language processing perspective) Li Deng's talk at Johns Hopkins University CSLP (speech recognition perspective) 2. In-depth lectures and books: Hugo Larochelle's lecture videos and slides at U. Sherkbrooke Geoff Hinton's Coursera course Chris Bishop's book (worth buying!): Pattern Recognition and Machine Learning 3. To go even deeper: Play with the Python Theano code samples Check out the numerous up-to-date news and references on DeepLearning.net Last modified: Thu Feb 6 18:14:26 JST 2014