5 juil. 2015 Villeneuve d'Ascq (Lille) (France)
Where to Apply Dropout in Recurrent Neural Networks for Handwriting Recognition?
Théodore Bluche  1, 2@  , Jérôme Louradour  1, *  , Christopher Kermorvant  3, *@  
1 : A2iA SAS  -  Site web
privé
39 rue de la Bienfaisance, Paris -  France
2 : Laboratoire d'Informatique pour la Mécanique et les Sciences de l'Ingénieur [Orsay]  (LIMSI)  -  Site web
Université Paris XI - Paris Sud, CNRS : UPR3251
Université Paris Sud (Paris XI) Bât. 508 BP 133 91403 ORSAY CEDEX -  France
3 : Teklia
privé
Paris -  France
* : Auteur correspondant

The dropout technique is a data-driven regularization method for neural networks. It consists in randomly setting some
activations from a given hidden layer to zero during training. Repeating the procedure for each training example, it is equivalent to sample a network from an exponential number of architectures that share weights. The goal of dropout is to prevent feature detectors to rely on each other.
Dropout has successfully been applied to deep MLPs and to convolutional neural networks, for various tasks of speech recognition and computer vision. We recently proposed a way to use dropout in MDLSTM-RNNs for handwritten word and line recognition.
In this paper, we show that further improvement can be achieved by implementing dropout differently, more specifically by applying it at better positions relative to the LSTM units.


Personnes connectées : 2