Machine Learning for Natural Language Processing (ML-for-NLP)

This reading group focuses on Machine Learning techniques that may be applied to the field of Natural Language Processing. Participants are encouraged to suggest topics, papers, or tutorials (which need not involve any current application in NLP) by adding them to the lists below. Suggesting a paper does not constitute any sort of commitment to presenting that paper.

Meetings are approximately every week on Mondays. Meetings will be in 3.02 at 3pm unless otherwise stated. Announcements for this group will be made by email, and it is possible to sign up to the mailing list here.

Schedule for Spring 2020

February 3

• Neural Text Generation With Unlikelihood Training (ICLR 2020)
  Slides: https://drive.google.com/file/d/10SdZpjh50pWF-bInZdhnvjBKbCOfZyLZ/view
  Abstract: Neural text generation is a key tool in natural language applications, but it is well known there are major problems at its core. In particular, standard likelihood training and decoding leads to dull and repetitive outputs. While some post-hoc fixes have been proposed, in particular top-k and nucleus sampling, they do not address the fact that the token-level probabilities predicted by the model are poor. In this paper we show that the likelihood objective itself is at fault, resulting in a model that assigns too much probability to sequences containing repeats and frequent words, unlike those from the human training distribution. We propose a new objective, unlikelihood training, which forces unlikely generations to be assigned lower probability by the model. We show that both token and sequence level unlikelihood training give less repetitive, less dull text while maintaining perplexity, giving superior generations using standard greedy or beam search. According to human evaluations, our approach with standard beam search also outperforms the currently popular decoding methods of nucleus sampling or beam blocking, thus providing a strong alternative to existing techniques.

Schedule for Spring 2019

April 15

• Tutorial on spectral learning for NLP (Naomi)

April 8

• Tutorial on Normalizing Flows and Latent Normalizing Flows for Discrete Sequences (Nicola and Tom)

April 1

• Syntax-Directed Variational Autoencoder for Structured Data (Nicola)

March 25

• Tutorial on Variational Inference (Tom)

March 18

• Skip

March 11

• Tutorial on generative adversaria networks for text generation part II (Christos)

March 4

• Skip - ACL deadline

February 27

• Skip - ACL deadline
  

February 20

• Tutorial on generative adversaria networks for text generation part I (Christos)

February 13

• Tutorial on generative adversaria networks, adversarial training and adversarial examples (Nicola)

February 4

• Discussion/planning for future meetings

January 28

• Adversarial Sampling and Training for Semi-Supervised Information Retrieval [pdf] (Nelly)

January 21

• A Structured Variational Autoencoder for Contextual Morphological Inflection [pdf] (Caio)

January 14

• Simple and Effective Semi-Supervised Question Answering [pdf] (Nicola)

Schedule for Fall 2018

November - December 2018

• break for NAACL and winter

November 19

• Neural Language Modeling by Jointly Learning Syntax and Lexicon [pdf] (Hao)

November 12

• cancelled

November 5

• no meeting (EMNLP)

October 29

• Learning to Compose Task-Specific Tree Structures [pdf] (Rei)

October 22

• Understanding deep learning requires rethinking generalization [pdf] (Naomi)

October 15

• Semantically Equivalent Adversarial Rules for Debugging NLP Models [pdf] (Jon)

October 8

• On the Importance of Single Directions for Generalization [pdf] (Naomi)

October 1

• planning meeting

Schedule for Spring 2018

May 7

• Imagine This! Scripts to Compositions to Videos [pdf] (David)

April 30

• SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient [pdf] (Chunchuan)

April 23

• A Deep Reinforced Model for Abstractive Summarization [pdf] (Ratish)

April 16

• Counterfactual fairness [pdf] (Naomi)

April 9

• Poincaré Embeddings for Learning Hierarchical Representations [pdf] (Javad)

April 2

• Semi-supervised learning with deep generative models [pdf] (Caio)

March 26

• Tensor Fusion Network for Multimodal Sentiment Analysis [pdf] (Nikos)

February 26 - March 19

• no meetings (strike)

February 19

• Morphological Inflection Generation with Hard Monotonic Attention [pdf] (Joana)

February 12

• A hybrid approach to dialogue management based on probabilistic rules [pdf] (Craig)

February 5

• Semantic Composition via Probabilistic Model Theory [pdf] (Matt)

January 29

• no meeting

January 22

• Emergent translation in multi-agent communication [pdf] (Serhii)

January 15

• planning meeting

Schedule for Fall 2017

December 4

• Noisy Parallel Approximate Decoding for Conditional Recurrent Language Model [pdf] (Ratish)

November 27

• Non-Autoregressive Neural Machine Translation [pdf] (Nikos)

November 14

• no meeting

November 6

• Word Representation Models for Morphologically Rich Languages in Neural Machine Translation [pdf] (Sander)

October 30

• Cross-lingual, Character-Level Neural Morphological Tagging [pdf] (Clara)

October 23

• Probabilistic Models for Segmenting and Labeling Sequence Data [pdf] (Joana)

October 16

• A joint many-task model: Growing a neural network for multiple NLP tasks [pdf] (Marco)

October 9

• Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling [pdf] (Diego)

October 2

• Human Centered NLP with User-Factor Adaptation [pdf] (Pippa)

September 25

• planning meeting

Schedule for Fall 2016

December 12

• TBD (Rafael)

December 5

• TBD (Esma)

November 28

• A Decomposable Attention Model for Natural Language Inference (Imigo)

November 21

• Jointly Combining Implicit Constraints Improves Temporal Ordering (Nate)

November 14

• Policy distillation (Mihai)

November 7

• Distilling the Knowledge in a Neural Network (Naomi)

October 31

• Deep Reinforcement Learning with a Natural Language Action Space + DRL Quick Tutorial https://arxiv.org/abs/1511.04636f (Rafael)

October 24

• Learning hidden unit contributions for unsupervised acoustic model adaptation (Joachim)

October 17

• Variational inference tutorial; https://www.cs.princeton.edu/courses/archive/fall11/cos597C/lectures/variational-inference-i.pdf (Jianpeng)

October 10

• A fully bayesian approach to unsupervised POS tagging (Craig)

October 3

• Beam sampling with Infinite HMM (Joana)

September 26

• Introduction to Bayesian statistics (Shay)

September 19

• Planning meeting: planning_fall2016.pdf

Schedule for Spring 2016

May 30

• Crash-course on Clustering (Akash)

May 23

• Probabilistic program induction (Pablo)
  • "Human-level concept learning through probabilistic program induction" by Lake et al. (available here: http://web.mit.edu/cocosci/Papers/Science-2015-Lake-1332-8.pdf)

May 2

• Composing Neural Networks (Shashi)
  • Learning to compose neural networks for question answering, Jacob Andreas, Marcus Rohrbach, Trevor Darrell and Dan Klein. NAACL 2016. http://arxiv.org/pdf/1601.01705v1.pdf

Apr 25

• Bayesian optimization (Nicolas)
  • Snoek et al.: https://papers.nips.cc/paper/4522-practical-bayesian-optimization-of-machine-learning-algorithms.pdf

Apr 18

• Planning meeting (Herman and Shashi)

Apr 11

• A quick introduction to Torch (Stef)
  • Differences to Theano and Tensorflow

Apr 4

• A quick introduction to TensorFlow (Rafael and Clara)

Mar 28

• Optimization Stuffs (Michael)
  • Adalta/Adagrad/Adam/fancy SGD.
  • ADADELTA: AN ADAPTIVE LEARNING RATE METHOD (http://www.matthewzeiler.com/pubs/googleTR2012/googleTR2012.pdf) from Matthew D. Zeiler.
  • Adam: http://arxiv.org/pdf/1412.6980.pdf
  • Adagrad: http://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf
  • No more pesky learning rates: http://schaul.site44.com/publications/vsgd.pdf

Mar 21

• Generative Bayesian-ish Neural Network Stuffs (Lea)
  • "Neurons as Monte Carlo Samplers: Bayesian Inference and Learning in Spiking Networks" from Huang and Rao :link

Mar 7

• Generative Bayesian-ish Neural Network Stuffs (Jianpeng = JP)
  • "Generating Sentences from a Continuous Space" from Bowman et al. : http://arxiv.org/abs/1511.06349

Feb 29

• Generative Bayesian-ish Neural Network Stuffs (Harri)
  • Variational inference primer notes by Harri: variational_inference_primer.pdf
  • "Auto-Encoding Variational Bayes" from Kingma and Welling : http://arxiv.org/abs/1312.6114
  • "Semi-Supervised Learning with Deep Generative Models" from Kingma et al. : http://arxiv.org/abs/1406.5298

Feb 22

• Generative Bayesian-ish Neural Network Stuffs (Herman)
  • "Generative Adversarial Networks" from Goodfellow et al. : http://arxiv.org/abs/1406.2661

Feb 15

• Parsing and Generation (Joana)
  • Spectral Learning http://www.cs.columbia.edu/~scohen/naacl13tutorial/naacl13tutorial-slides.pdf (slides 22 - 27, 48 - 61 and 68 - 80) and http://www.cs.upc.edu/~bballe/slides/tutorial-emnlp14.pdf (section 2)

Feb 8

• Parsing and Generation (John)
  • Parsing Minimalist Languages, http://wwwhomes.uni-bielefeld.de/jmichaelis/esslli2009/reader_extension/harkema_2001-parsing_minimalist_languages-dissertation.pdf

Feb 1

• Parsing and Generation (Marco)
  • A Transition-based Algorithm for AMR Parsing, http://www.cs.brandeis.edu/~xuen/publications/wang-2015-naacl.pdf

Jan 25

• Parsing and Generation (Sorcha)
  • A tutorial on graph grammars, http://www8.cs.umu.se/~drewes/gratra_tutorial.pdf

Jan 18

• Parsing and Generation (Rafael)
  • Structured Generative Models of Natural Source Code: http://jmlr.org/proceedings/papers/v32/maddison14.pdf

Jan 11

• Planning meeting (Herman & Shashi)
  • Major topics discussed
    • Topics A, Topics B

Schedule for Fall 2015

Dec 7

• Neural Networks: Attention models (Liang & Xingxing)
  • Neural Machine Translation by Jointly Learning to Allogn and Translate. Dzmitry Bahdanau, KyungHyun Cho and Yoshua Bengio. http://arxiv.org/pdf/1409.0473v6.pdf
  • Additional: http://arxiv.org/pdf/1502.04623.pdf, http://arxiv.org/pdf/1511.06018v1.pdf

Nov 30

• Neural Networks: RNNs (Daniel)
  • Complex recurrent network structures such as the Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), https://wiki.inf.ed.ac.uk/twiki/pub/MLforNLP/WebHome/lstm_intro_2015.pdf
  • Blog post by Christopher Olah, http://colah.github.io/posts/2015-08-Understanding-LSTMs/

Nov 23

• Neural Networks: RNNs (Herman)
  • Deep Learning, An MIT Press book in preparation. Ian Goodfellow, Aaron Courville, and Yoshua Bengio. http://goodfeli.github.io/dlbook/ Section 10.2.2 in the RNN chapter and Section 8.2.5 in the optimization chapter.

Nov 16

• Word embeddings: Application (Rafael)
  • Translating Videos to Natural Language Using Deep Recurrent Neural Networks : http://arxiv.org/pdf/1412.4729v3.pdf

Nov 9

• Word embeddings: Application (Michael Hahn)
  • Grounded Compositional Semantics for Finding and Describing Images with Sentences, ACL 2014: http://nlp.stanford.edu/~socherr/SocherKarpathyLeManningNg_TACL2013.pdf

Nov 2

• Word embeddings (Stef)
  • Omer Levy and Yoav Goldberg, Dependency-Based Word Embeddings, http://www.aclweb.org/anthology/P14-2050.pdf

Oct 26

• Reinforcement learning and Neural Networks (Xingxing)
  • Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto. Q-learning - Sections 6.1 and 6.5.
  • Mnih et al. (2013). Playing Atari with Deep Reinforcement Learning, https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf

Oct 19

• Reinforcement Learning(Mihai)
  • Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto, Sections: 3.1 - 3.3, 3.5 - 3.8, 4.1 - 4.4.

Oct 12

• Reinforcement learning (Ben)
  • Harmon, M. E., & Harmon, S. S. (1996). Reinforcement learning: A tutorial. WL/AAFC, WPAFB Ohio, 45433. (http://www.eecs.wsu.edu/~holder/courses/cse6362/pubs/Harmon96.pdf).

Oct 5

• Planning meeting (Herman & Shashi)
  • Major topics
    • Discussed topics - A, Discussed topics - B
  • Tool/Resource sessions

Previous Meetings

Visit Previous meetings for a list of previous meetings.

Useful Information

Visit Useful Information for information about useful tools and paper recommendations from previous meetings.

Other Reading Groups

Cambridge http://www.wiki.cl.cam.ac.uk/rowiki/NaturalLanguage/ReadingGroup

Heriot-Watt https://sites.google.com/site/hwmlreadinggroup/

Toronto http://learning.cs.toronto.edu/mlreading.html

• Planning Meeting 2017: