Edit filter log

'{{Short description|Type of neural network output and associated scoring function}} {{Use dmy dates|date=September 2017}} '''Connectionist temporal classification''' ('''CTC''') is a type of neural network output and associated scoring function, for training [[recurrent neural network]]s (RNNs) such as [[Long short-term memory|LSTM]] networks to tackle sequence problems where the timing is variable. It can be used for tasks like on-line [[handwriting recognition]]<ref>{{Cite journal|last1=Liwicki|first1=Marcus|last2=Graves|first2=Alex|author2-link=Alex Graves (computer scientist)|last3=Bunke|first3=Horst|last4=Schmidhuber|first4=Jürgen|author4-link=Juergen Schmidhuber|date=2007|title=A novel approach to on-line handwriting recognition based on bidirectional long short-term memory networks|journal=In Proceedings of the 9th International Conference on Document Analysis and Recognition, ICDAR 2007|citeseerx=10.1.1.139.5852}}</ref> or recognizing phonemes in speech audio. CTC refers to the outputs and scoring, and is independent of the underlying neural network structure. It was introduced in 2006.<ref name="graves2006">{{Cite journal|last1=Graves|first1=Alex|author1-link=Alex Graves (computer scientist) | last2=Fernández|first2=Santiago|last3=Gomez|first3=Faustino|last4=Schmidhuber|first4=Juergen|author4-link=Juergen Schmidhuber| date=2006|title=Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks|journal=Proceedings of the International Conference on Machine Learning, ICML 2006|pages=369–376|citeseerx=10.1.1.75.6306}}</ref> The input is a sequence of observations, and the outputs are a sequence of labels, which can include blank outputs. The difficulty of training comes from there being many more observations than there are labels. For example, in speech audio there can be multiple time slices which correspond to a single phoneme. Since we don't know the alignment of the observed sequence with the target labels we predict a probability distribution at each time step.<ref>{{Cite journal|last=Hannun|first=Awni|date=2017-11-27|title=Sequence Modeling with CTC|journal=Distill|language=en-US|volume=2|issue=11|doi=10.23915/distill.00008|issn=2476-0757|arxiv=1508.01211}}</ref> A CTC network has a continuous output (e.g. [[Softmax function|softmax]]), which is fitted through training to model the probability of a label. CTC does not attempt to learn boundaries and timings: Label sequences are considered equivalent if they differ only in alignment, ignoring blanks. Equivalent label sequences can occur in many ways – which makes scoring a non-trivial task, but there is an efficient [[forward–backward algorithm]] for that. CTC scores can then be used with the back-propagation algorithm to update the neural network weights. Alternative approaches to a CTC-fitted neural network include a [[hidden Markov model]] (HMM). In 2009, a [[Connectionist Temporal Classification (CTC)|Connectionist Temporal Classification]] (CTC)-trained LSTM network was the first RNN to win pattern recognition contests when it won several competitions in connected [[handwriting recognition]].<ref name="schmidhuber2015">{{Cite journal |last=Schmidhuber |first=Jürgen |date=January 2015 |title=Deep Learning in Neural Networks: An Overview |journal=Neural Networks |volume=61 |pages=85–117 |arxiv=1404.7828 |doi=10.1016/j.neunet.2014.09.003 |pmid=25462637 |s2cid=11715509}}</ref><ref name="graves20093">{{Cite conference |last1=Graves |first1=Alex |last2=Schmidhuber |first2=Jürgen |year=2009 |editor1-last=Koller |editor1-first=D. |editor2-last=Schuurmans |editor2-first=D. |editor2-link=Dale Schuurmans |editor3-last=Bengio |editor3-first=Y. |editor3-link=Yoshua Bengio |editor4-last=Bottou |editor4-first=L. |title=Offline Handwriting Recognition with Multidimensional Recurrent Neural Networks |url=https://papers.nips.cc/paper/3449-offline-handwriting-recognition-with-multidimensional-recurrent-neural-networks |publisher=Neural Information Processing Systems (NIPS) Foundation |volume=21 |pages=545–552 |book-title=Advances in Neural Information Processing Systems}}</ref> In 2014, the Chinese company [[Baidu]] used CTC-trained RNNs to break the 2S09 Switchboard Hub5'00 speech recognition dataset<ref>{{Cite web |title=2000 HUB5 English Evaluation Speech - Linguistic Data Consortium |url=https://catalog.ldc.upenn.edu/LDC2002S09 |website=catalog.ldc.upenn.edu}}</ref> benchmark without using any traditional speech processing methods.<ref name="hannun2014">{{cite arXiv |eprint=1412.5567 |class=cs.CL |first1=Awni |last1=Hannun |first2=Carl |last2=Case |title=Deep Speech: Scaling up end-to-end speech recognition |date=2014-12-17 |last3=Casper |first3=Jared |last4=Catanzaro |first4=Bryan |last5=Diamos |first5=Greg |last6=Elsen |first6=Erich |last7=Prenger |first7=Ryan |last8=Satheesh |first8=Sanjeev |last9=Sengupta |first9=Shubho}}</ref> In 2015, it was used in [[Google Voice Search|Google voice search]].<ref name="sak2015">{{Cite web |last1=Sak |first1=Haşim |last2=Senior |first2=Andrew |last3=Rao |first3=Kanishka |last4=Beaufays |first4=Françoise |last5=Schalkwyk |first5=Johan |date=September 2015 |title=Google voice search: faster and more accurate |url=http://googleresearch.blogspot.ch/2015/09/google-voice-search-faster-and-more.html}}</ref> ==References== {{reflist}} ==External links== * [https://web.stanford.edu/~jurafsky/slp3/16.pdf Section 16.4, "CTC"] in Jurafsky and Martin's [https://web.stanford.edu/~jurafsky/slp3/ ''Speech and Language Processing''], 3rd edition * {{Cite journal |last=Hannun |first=Awni |date=2017-11-27 |title=Sequence Modeling with CTC |url=https://distill.pub/2017/ctc |journal=Distill |language=en |volume=2 |issue=11 |pages=e8 |doi=10.23915/distill.00008 |issn=2476-0757}} [[Category:Artificial neural networks]]'