Skip to content

yoninazarathy/MathematicalEngineeringDeepLearning

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

MathematicalEngineeringDeepLearning

Material for The Mathematical Engineering of Deep Learning. See the actual book content on deeplearningmath.org or (when it is out) purchase the book from CRC press.

This repository contains general supporting material for the book.

Below is a detailed list of the source code used for creating figures and tables in the book. We use Julia, Python, or R and the code is sometimes in stand alone files, sometimes in Jupyter notebooks, sometimes as R Markdown, and sometimes in Google Colab. Many of our static illustrations were created using TikZ by Ajay Hemanth and Vishnu Prasath with the source of their illustrations also available so you can adapt it for purposes.

Chapter 1

Figure Topic Source Code
1.1 Fast.ai example Python Google Colab
1.3 Architectures TikZ(a), TikZ(b), TikZ(c), TikZ(d), TikZ(e), TikZ(f), TikZ(g), TikZ(h)
1.4 Neurons TikZ(b), TikZ(d)
1.5 Data on earth Julia

Chapter 2

Figure Topic Source Code
2.1 Supervised Learning TikZ
2.2 Unsupervised Learning TikZ
2.3 Simple regression R
2.4 Breast Cancer ROC curves R
2.5 Least Squares TikZ
2.6 Loss functions Julia
Table 2.1 Linear MNIST classification Julia
2.7 Gradient Descent Learning Rate Python
2.8 Loss Landscape R
2.9 Generalization and Training TikZ or Julia
2.10 Polynomial fit R
2.11 K-fold cross validation TikZ
2.12 K-means clustering R
2.13 K-means image segmentation R
2.14 Breast Cancer PCA R
2.15 SVD Compression Julia

Chapter 3

Figure Topic Source Code
3.1 and 3.2 Logistic regression model curves and boundary R
3.3 Components of an artificial neuron TikZ
3.4 Loss landscape of MSE vs. CE on logistic regression Python
3.5 Evolution of gradient descent learning in logistic regression R(a,b) First file, R(a,b) Second file
3.6 Shallow multi-output neural network with softmax TikZ
3.7 Multinomial regression for classification R
Table 3.1 Different approaches for creating an MNIST digit classifier. Julia
3.8 Feature engineering in simple logistic regression R
3.9 Non-linear classification decision boundaries with feature engineering in logistic regression R
3.10 Non-linear classification decision boundaries with feature engineering in multinomial regression R same as 3.7
3.11 Single hidden layer autoencoder TikZ
3.12 Autoencoder projections of MNIST including using PCA R TikZ
3.13 Manifolds and autoencoders R TikZ
3.14 MNIST using autoencoders R same as 3.12
3.15 Denoising autoencoder TikZ
3.16 Interpolations with autoencoders R same as 3.12, Julia

Chapter 4

Figure Topic Source Code
4.1 Convexity and local/global extrema Python
4.2 Gradient descent with fixed or time dependent learning rate Python
4.3 Stochastic gradient descent Python
4.4 Early stopping in deep learning Julia
4.5 Non-convex loss landscapes Python
4.6 Momentum enhancing gradient descent Python
4.7 The computational graph for automatic differentiation TikZ
4.8 Line search concepts Python
4.9 The zig-zagging property of line search Python
4.10 Newton's method in one dimension Python

Chapter 5

Figure Topic Source Code
5.1 Fully Connected Feedforward Neural Networks TikZ(a), TikZ(b)
5.2 Arbitrary function approximation with neural nets TikZ(a), Julia(b,c)
5.3 Binary classification with increasing depth R
5.4 A continuous multiplication gate with 4 hidden units TikZ
5.5 A deep model with 10 layers TikZ
5.6 Several common scalar activation functions Julia(a,b)
5.7 Flow of information in general back propagation TikZ
5.8 Simple neural network hypothetical example TikZ
5.9 Flow of information in standard neural network back propagation TikZ
5.10 Computational graph for batch normalization TikZ
5.11 The effect of dropout TikZ

Chapter 6

Figure Topic Source Code
6.2 VGG19 architecture TikZ
6.3 Convolutions TikZ(a), TikZ(b)
6.6 Convolution padding TikZ
6.7 Convolution stride TikZ
6.8 Convolution dilation TikZ
6.9 Convolution input channels TikZ
6.10 Convolution output channels TikZ
6.11 Pooling TikZ(a), TikZ(b)
6.13 Inception module TikZ
6.14 Resnets TikZ
6.17 Siamese network TikZ

Chapter 7

Figure Topic Source Code
7.1 Sequence RNN tasks TikZ(a), TikZ(b), TikZ(c), TikZ(d)
7.2 Sequence RNN input output paradigms TikZ(a), TikZ(b), TikZ(c), TikZ(d)
7.3 RNN recursive graph and unfolded graph TikZ
7.4 RNN unit TikZ
7.5 RNN language prediction training TikZ
7.6 Backpropagation through time TikZ
7.7 Alternative RNN configurations TikZ(a), TikZ(b)
7.8 LSTM and GRU TikZ(a), TikZ(b)
7.9 Encoder decoder architectures TikZ(a), TikZ(b)
7.10 Encoder decoder with attention TikZ
7.11 Attention weights TikZ
7.12 Flow of information with self attention TikZ
7.13 Multi-head self attention TikZ
7.14 Positional embedding Julia(a,b)
7.15 Transformer blocks TikZ(a), TikZ(b)
7.16 Transformer encoder decoder architecture TikZ
7.17 Transfomer auto-regressive application TikZ

Chapter 8

Figure Topic Source Code
8.1 Generative modelling TikZ
8.2 Variational autoencoder TikZ
8.4 Diffusion encoder and decoder TikZ
8.6 GAN architectures TikZ
8.7 Separation of GAN distributions TikZ
8.8 Wasserstein distance TikZ
8.9 Reinforcement learning TikZ
Equation (8.72) An MDP optimal policy Julia
8.10 Applications of GNN TikZ(a), TikZ(b), TikZ(c)
8.11 Directed and undirected graphs TikZ(a), TikZ(b)
8.12 Transductive inductive learning TikZ(a), TikZ(b)
8.13 Types of GNN tasks TikZ(a), TikZ(b), TikZ(c)
8.14 Aggregation in message passing TikZ(a), TikZ(b)

About

Material for The Mathematical Engineering of Deep Learning. See https://deeplearningmath.org

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published