NLP-Models-Tensorflow, Gathers machine learning and tensorflow deep learning models for NLP problems, code simplify inside Jupyter Notebooks 100%.
- Abstractive Summarization
- Chatbot
- Dependency Parser
- Entity Tagging
- Extractive Summarization
- Generator
- Language Detection
- Neural Machine Translation
- OCR
- POS Tagging
- Question-Answers
- Sentence pairs
- Speech-to-Text
- Spelling correction
- SQUAD Question-Answers
- Stemming
- Text Augmentation
- Text Classification
- Text Similarity
- Text-to-Speech
- Topic Generator
- Topic Modeling
- Unsupervised Extractive Summarization
- Vectorizer
- Old-to-Young Vocoder
- Visualization
- Attention
Original implementations are quite complex and not really beginner friendly. So I tried to simplify most of it. Also, there are tons of not-yet release papers implementation. So feel free to use it for your own research!
I will attached github repositories for models that I not implemented from scratch, basically I copy, paste and fix those code for deprecated issues.
Tensorflow version 1.13 and above only, not included 2.X version. 1.13 < Tensorflow < 2.0
pip install -r requirements.txtTrained on India news.
Accuracy based on 10 epochs only, calculated using word positions.
Complete list (12 notebooks)
- LSTM Seq2Seq using topic modelling, test accuracy 13.22%
- LSTM Seq2Seq Luong Attention using topic modelling, test accuracy 12.39%
- LSTM Seq2Seq Beam Decoder using topic modelling, test accuracy 10.67%
- LSTM Bidirectional Luong Attention Beam Decoder using topic modelling, test accuracy 8.29%
- Pointer-Generator Bahdanau, https://github.com/xueyouluo/my_seq2seq, test accuracy 15.51%
- Copynet, test accuracy 11.15%
- Pointer-Generator Luong, https://github.com/xueyouluo/my_seq2seq, test accuracy 16.51%
- Dilated Seq2Seq, test accuracy 10.88%
- Dilated Seq2Seq Self Attention, test accuracy 11.54%
- BERT Dilated CNN Seq2seq, test accuracy 13.5%
- self-attention Pointer-Generator, test accuracy 4.34%
- Dilated-CNN Seq2seq Pointer-Generator, test accuracy 5.57%
Trained on Cornell Movie Dialog corpus, accuracy table in chatbot.
Complete list (54 notebooks)
- Basic cell Seq2Seq-manual
- LSTM Seq2Seq-manual
- GRU Seq2Seq-manual
- Basic cell Seq2Seq-API Greedy
- LSTM Seq2Seq-API Greedy
- GRU Seq2Seq-API Greedy
- Basic cell Bidirectional Seq2Seq-manual
- LSTM Bidirectional Seq2Seq-manual
- GRU Bidirectional Seq2Seq-manual
- Basic cell Bidirectional Seq2Seq-API Greedy
- LSTM Bidirectional Seq2Seq-API Greedy
- GRU Bidirectional Seq2Seq-API Greedy
- Basic cell Seq2Seq-manual Luong Attention
- LSTM Seq2Seq-manual Luong Attention
- GRU Seq2Seq-manual Luong Attention
- Basic cell Seq2Seq-manual Bahdanau Attention
- LSTM Seq2Seq-manual Bahdanau Attention
- GRU Seq2Seq-manual Bahdanau Attention
- LSTM Bidirectional Seq2Seq-manual Luong Attention
- GRU Bidirectional Seq2Seq-manual Luong Attention
- LSTM Bidirectional Seq2Seq-manual Bahdanau Attention
- GRU Bidirectional Seq2Seq-manual Bahdanau Attention
- LSTM Bidirectional Seq2Seq-manual backward Bahdanau forward Luong
- GRU Bidirectional Seq2Seq-manual backward Bahdanau forward Luong
- LSTM Seq2Seq-API Greedy Luong Attention
- GRU Seq2Seq-API Greedy Luong Attention
- LSTM Seq2Seq-API Greedy Bahdanau Attention
- GRU Seq2Seq-API Greedy Bahdanau Attention
- LSTM Seq2Seq-API Beam Decoder
- GRU Seq2Seq-API Beam Decoder
- LSTM Bidirectional Seq2Seq-API Luong Attention Beam Decoder
- GRU Bidirectional Seq2Seq-API Luong Attention Beam Decoder
- LSTM Bidirectional Seq2Seq-API backward Bahdanau forward Luong Stack Bahdanau Luong Attention Beam Decoder
- GRU Bidirectional Seq2Seq-API backward Bahdanau forward Luong Stack Bahdanau Luong Attention Beam Decoder
- Bytenet
- LSTM Seq2Seq tf.estimator
- Capsule layers LSTM Seq2Seq-API Greedy
- Capsule layers LSTM Seq2Seq-API Luong Attention Beam Decoder
- LSTM Bidirectional Seq2Seq-API backward Bahdanau forward Luong Stack Bahdanau Luong Attention Beam Decoder Dropout L2
- DNC Seq2Seq
- LSTM Bidirectional Seq2Seq-API Luong Monotic Attention Beam Decoder
- LSTM Bidirectional Seq2Seq-API Bahdanau Monotic Attention Beam Decoder
- End-to-End Memory Network Basic cell
- End-to-End Memory Network LSTM cell
- Attention is all you need
- Transformer-XL
- Attention is all you need Beam Search
- Transformer-XL LSTM
- GPT-2 LSTM
- CNN Seq2seq
- Conv-Encoder LSTM
- Tacotron Greedy decoder
- Tacotron Beam decoder
- Google NMT
Trained on CONLL English Dependency. Train set to train, dev and test sets to test.
Stackpointer and Biaffine-attention originally from https://github.com/XuezheMax/NeuroNLP2 written in Pytorch.
Accuracy based on arc, types and root accuracies after 15 epochs only.
Complete list (8 notebooks)
- Bidirectional RNN CRF Biaffine, arc accuracy 70.48%, types accuracy 65.18%, root accuracy 66.4%
- Bidirectional RNN Bahdanau CRF Biaffine, arc accuracy 70.82%, types accuracy 65.33%, root accuracy 66.77%
- Bidirectional RNN Luong CRF Biaffine, arc accuracy 71.22%, types accuracy 65.73%, root accuracy 67.23%
- BERT Base CRF Biaffine, arc accuracy 64.30%, types accuracy 62.89%, root accuracy 74.19%
- Bidirectional RNN Biaffine Attention Cross Entropy, arc accuracy 72.42%, types accuracy 63.53%, root accuracy 68.51%
- BERT Base Biaffine Attention Cross Entropy, arc accuracy 72.85%, types accuracy 67.11%, root accuracy 73.93%
- Bidirectional RNN Stackpointer, arc accuracy 61.88%, types accuracy 48.20%, root accuracy 89.39%
- XLNET Base Biaffine Attention Cross Entropy, arc accuracy 74.41%, types accuracy 71.37%, root accuracy 73.17%
Trained on CONLL NER.
Complete list (9 notebooks)
- Bidirectional RNN CRF, test accuracy 96%
- Bidirectional RNN Luong Attention CRF, test accuracy 93%
- Bidirectional RNN Bahdanau Attention CRF, test accuracy 95%
- Char Ngrams Bidirectional RNN Bahdanau Attention CRF, test accuracy 96%
- Char Ngrams Bidirectional RNN Bahdanau Attention CRF, test accuracy 96%
- Char Ngrams Residual Network Bahdanau Attention CRF, test accuracy 69%
- Char Ngrams Attention is you all Need CRF, test accuracy 90%
- BERT, test accuracy 99%
- XLNET-Base, test accuracy 99%
Trained on CNN News dataset.
Accuracy based on ROUGE-2.
Complete list (4 notebooks)
- LSTM RNN, test accuracy 16.13%
- Dilated-CNN, test accuracy 15.54%
- Multihead Attention, test accuracy 26.33%
- BERT-Base
Trained on Shakespeare dataset.
Complete list (15 notebooks)
- Character-wise RNN LSTM
- Character-wise RNN Beam search
- Character-wise RNN LSTM Embedding
- Word-wise RNN LSTM
- Word-wise RNN LSTM Embedding
- Character-wise Seq2Seq GRU
- Word-wise Seq2Seq GRU
- Character-wise RNN LSTM Bahdanau Attention
- Character-wise RNN LSTM Luong Attention
- Word-wise Seq2Seq GRU Beam
- Character-wise Seq2Seq GRU Bahdanau Attention
- Word-wise Seq2Seq GRU Bahdanau Attention
- Character-wise Dilated CNN Beam search
- Transformer Beam search
- Transformer XL Beam search
Trained on Tatoeba dataset.
Complete list (1 notebooks)
- Fast-text Char N-Grams
Trained on English-French, accuracy table in neural-machine-translation.
Complete list (53 notebooks)
1.basic-seq2seq 2.lstm-seq2seq 3.gru-seq2seq 4.basic-seq2seq-contrib-greedy 5.lstm-seq2seq-contrib-greedy 6.gru-seq2seq-contrib-greedy 7.basic-birnn-seq2seq 8.lstm-birnn-seq2seq 9.gru-birnn-seq2seq 10.basic-birnn-seq2seq-contrib-greedy 11.lstm-birnn-seq2seq-contrib-greedy 12.gru-birnn-seq2seq-contrib-greedy 13.basic-seq2seq-luong 14.lstm-seq2seq-luong 15.gru-seq2seq-luong 16.basic-seq2seq-bahdanau 17.lstm-seq2seq-bahdanau 18.gru-seq2seq-bahdanau 19.basic-birnn-seq2seq-bahdanau 20.lstm-birnn-seq2seq-bahdanau 21.gru-birnn-seq2seq-bahdanau 22.basic-birnn-seq2seq-luong 23.lstm-birnn-seq2seq-luong 24.gru-birnn-seq2seq-luong 25.lstm-seq2seq-contrib-greedy-luong 26.gru-seq2seq-contrib-greedy-luong 27.lstm-seq2seq-contrib-greedy-bahdanau 28.gru-seq2seq-contrib-greedy-bahdanau 29.lstm-seq2seq-contrib-beam-luong 30.gru-seq2seq-contrib-beam-luong 31.lstm-seq2seq-contrib-beam-bahdanau 32.gru-seq2seq-contrib-beam-bahdanau 33.lstm-birnn-seq2seq-contrib-beam-bahdanau 34.lstm-birnn-seq2seq-contrib-beam-luong 35.gru-birnn-seq2seq-contrib-beam-bahdanau 36.gru-birnn-seq2seq-contrib-beam-luong 37.lstm-birnn-seq2seq-contrib-beam-luongmonotonic 38.gru-birnn-seq2seq-contrib-beam-luongmonotic 39.lstm-birnn-seq2seq-contrib-beam-bahdanaumonotonic 40.gru-birnn-seq2seq-contrib-beam-bahdanaumonotic 41.residual-lstm-seq2seq-greedy-luong 42.residual-gru-seq2seq-greedy-luong 43.residual-lstm-seq2seq-greedy-bahdanau 44.residual-gru-seq2seq-greedy-bahdanau 45.memory-network-lstm-decoder-greedy 46.google-nmt 47.transformer-encoder-transformer-decoder 48.transformer-encoder-lstm-decoder-greedy 49.bertmultilanguage-encoder-bertmultilanguage-decoder 50.bertmultilanguage-encoder-lstm-decoder 51.bertmultilanguage-encoder-transformer-decoder 52.bertenglish-encoder-transformer-decoder 53.transformer-t2t-2gpu
Complete list (2 notebooks)
- CNN LSTM RNN, test accuracy 100%
- Im2Latex, test accuracy 100%
Trained on CONLL POS.
Complete list (8 notebooks)
- Bidirectional RNN CRF, test accuracy 92%
- Bidirectional RNN Luong Attention CRF, test accuracy 91%
- Bidirectional RNN Bahdanau Attention CRF, test accuracy 91%
- Char Ngrams Bidirectional RNN Bahdanau Attention CRF, test accuracy 91%
- Char Ngrams Bidirectional RNN Bahdanau Attention CRF, test accuracy 91%
- Char Ngrams Residual Network Bahdanau Attention CRF, test accuracy 3%
- Char Ngrams Attention is you all Need CRF, test accuracy 89%
- BERT, test accuracy 99%
Trained on bAbI Dataset.
Complete list (4 notebooks)
- End-to-End Memory Network Basic cell
- End-to-End Memory Network GRU cell
- End-to-End Memory Network LSTM cell
- Dynamic Memory
Trained on Cornell Movie--Dialogs Corpus
Complete list (1 notebooks)
- BERT
Trained on Toronto speech dataset.
Complete list (11 notebooks)
- Tacotron, https://github.com/Kyubyong/tacotron_asr, test accuracy 77.09%
- BiRNN LSTM, test accuracy 84.66%
- BiRNN Seq2Seq Luong Attention Cross Entropy, test accuracy 87.86%
- BiRNN Seq2Seq Bahdanau Attention Cross Entropy, test accuracy 89.28%
- BiRNN Seq2Seq Bahdanau Attention CTC, test accuracy 86.35%
- BiRNN Seq2Seq Luong Attention CTC, test accuracy 80.30%
- CNN RNN Bahdanau Attention, test accuracy 80.23%
- Dilated CNN RNN, test accuracy 31.60%
- Wavenet, test accuracy 75.11%
- Deep Speech 2, test accuracy 81.40%
- Wav2Vec Transfer learning BiRNN LSTM, test accuracy 83.24%
Complete list (4 notebooks)
- BERT-Base
- XLNET-Base
- BERT-Base Fast
- BERT-Base accurate
Trained on SQUAD Dataset.
Complete list (1 notebooks)
- BERT,
{"exact_match": 77.57805108798486, "f1": 86.18327335287402}Trained on English Lemmatization.
Complete list (6 notebooks)
- LSTM Seq2Seq Beam
- GRU Seq2Seq Beam
- LSTM BiRNN Seq2Seq Beam
- GRU BiRNN Seq2Seq Beam
- DNC Seq2Seq Greedy
- BiRNN Bahdanau Copynet
Complete list (8 notebooks)
- Pretrained Glove
- GRU VAE-seq2seq-beam TF-probability
- LSTM VAE-seq2seq-beam TF-probability
- GRU VAE-seq2seq-beam Bahdanau Attention TF-probability
- VAE Deterministic Bahdanau Attention, https://github.com/HareeshBahuleyan/tf-var-attention
- VAE VAE Bahdanau Attention, https://github.com/HareeshBahuleyan/tf-var-attention
- BERT-Base Nucleus Sampling
- XLNET-Base Nucleus Sampling
Trained on English sentiment dataset, accuracy table in text-classification.
Complete list (79 notebooks)
- Basic cell RNN
- Basic cell RNN Hinge
- Basic cell RNN Huber
- Basic cell Bidirectional RNN
- Basic cell Bidirectional RNN Hinge
- Basic cell Bidirectional RNN Huber
- LSTM cell RNN
- LSTM cell RNN Hinge
- LSTM cell RNN Huber
- LSTM cell Bidirectional RNN
- LSTM cell Bidirectional RNN Huber
- LSTM cell RNN Dropout L2
- GRU cell RNN
- GRU cell RNN Hinge
- GRU cell RNN Huber
- GRU cell Bidirectional RNN
- GRU cell Bidirectional RNN Hinge
- GRU cell Bidirectional RNN Huber
- LSTM RNN Conv2D
- K-max Conv1d
- LSTM RNN Conv1D Highway
- LSTM RNN Basic Attention
- LSTM Dilated RNN
- Layer-Norm LSTM cell RNN
- Only Attention Neural Network
- Multihead-Attention Neural Network
- Neural Turing Machine
- LSTM Seq2Seq
- LSTM Seq2Seq Luong Attention
- LSTM Seq2Seq Bahdanau Attention
- LSTM Seq2Seq Beam Decoder
- LSTM Bidirectional Seq2Seq
- Pointer Net
- LSTM cell RNN Bahdanau Attention
- LSTM cell RNN Luong Attention
- LSTM cell RNN Stack Bahdanau Luong Attention
- LSTM cell Bidirectional RNN backward Bahdanau forward Luong
- Bytenet
- Fast-slow LSTM
- Siamese Network
- LSTM Seq2Seq tf.estimator
- Capsule layers RNN LSTM
- Capsule layers LSTM Seq2Seq
- Capsule layers LSTM Bidirectional Seq2Seq
- Nested LSTM
- LSTM Seq2Seq Highway
- Triplet loss LSTM
- DNC (Differentiable Neural Computer)
- ConvLSTM
- Temporal Convd Net
- Batch-all Triplet-loss LSTM
- Fast-text
- Gated Convolution Network
- Simple Recurrent Unit
- LSTM Hierarchical Attention Network
- Bidirectional Transformers
- Dynamic Memory Network
- Entity Network
- End-to-End Memory Network
- BOW-Chars Deep sparse Network
- Residual Network using Atrous CNN
- Residual Network using Atrous CNN Bahdanau Attention
- Deep pyramid CNN
- Transformer-XL
- Transfer learning GPT-2 345M
- Quasi-RNN
- Tacotron
- Slice GRU
- Slice GRU Bahdanau
- Wavenet
- Transfer learning BERT Base
- Transfer learning XL-net Large
- LSTM BiRNN global Max and average pooling
- Transfer learning BERT Base drop 6 layers
- Transfer learning BERT Large drop 12 layers
- Transfer learning XL-net Base
- Transfer learning ALBERT
- Transfer learning ELECTRA Base
- Transfer learning ELECTRA Large
Trained on MNLI.
Complete list (10 notebooks)
- BiRNN Contrastive loss, test accuracy 73.032%
- BiRNN Cross entropy, test accuracy 74.265%
- BiRNN Circle loss, test accuracy 75.857%
- BiRNN Proxy loss, test accuracy 48.37%
- BERT Base Cross entropy, test accuracy 91.123%
- BERT Base Circle loss, test accuracy 89.903%
- ELECTRA Base Cross entropy, test accuracy 96.317%
- ELECTRA Base Circle loss, test accuracy 95.603%
- XLNET Base Cross entropy, test accuracy 93.998%
- XLNET Base Circle loss, test accuracy 94.033%
Trained on Toronto speech dataset.
Complete list (8 notebooks)
- Tacotron, https://github.com/Kyubyong/tacotron
- CNN Seq2seq Dilated CNN vocoder
- Seq2Seq Bahdanau Attention
- Seq2Seq Luong Attention
- Dilated CNN Monothonic Attention Dilated CNN vocoder
- Dilated CNN Self Attention Dilated CNN vocoder
- Deep CNN Monothonic Attention Dilated CNN vocoder
- Deep CNN Self Attention Dilated CNN vocoder
Trained on Malaysia news.
Complete list (4 notebooks)
- TAT-LSTM
- TAV-LSTM
- MTA-LSTM
- Dilated CNN Seq2seq
Extracted from English sentiment dataset.
Complete list (3 notebooks)
- LDA2Vec
- BERT Attention
- XLNET Attention
Trained on random books.
Complete list (3 notebooks)
- Skip-thought Vector
- Residual Network using Atrous CNN
- Residual Network using Atrous CNN Bahdanau Attention
Trained on English sentiment dataset.
Complete list (11 notebooks)
- Word Vector using CBOW sample softmax
- Word Vector using CBOW noise contrastive estimation
- Word Vector using skipgram sample softmax
- Word Vector using skipgram noise contrastive estimation
- Supervised Embedded
- Triplet-loss LSTM
- LSTM Auto-Encoder
- Batch-All Triplet-loss LSTM
- Fast-text
- ELMO (biLM)
- Triplet-loss BERT
Complete list (4 notebooks)
- Attention heatmap on Bahdanau Attention
- Attention heatmap on Luong Attention
- BERT attention, https://github.com/hsm207/bert_attn_viz
- XLNET attention
Trained on Toronto speech dataset.
Complete list (1 notebooks)
- Dilated CNN
Complete list (8 notebooks)
- Bahdanau
- Luong
- Hierarchical
- Additive
- Soft
- Attention-over-Attention
- Bahdanau API
- Luong API
- Markov chatbot
- Decomposition summarization (3 notebooks)