TWiki> MLforNLP Web>WebHome (03 Feb 2020, Main.v1ndeca)EditAttach

Machine Learning for Natural Language Processing (ML-for-NLP)

This reading group focuses on Machine Learning techniques that may be applied to the field of Natural Language Processing. Participants are encouraged to suggest topics, papers, or tutorials (which need not involve any current application in NLP) by adding them to the lists below. Suggesting a paper does not constitute any sort of commitment to presenting that paper.

Meetings are approximately every week on Mondays. Meetings will be in 3.02 at 3pm unless otherwise stated. Announcements for this group will be made by email, and it is possible to sign up to the mailing list here.

Schedule for Spring 2020

February 3
  • Neural Text Generation With Unlikelihood Training (ICLR 2020)
    Abstract: Neural text generation is a key tool in natural language applications, but it is well known there are major problems at its core. In particular, standard likelihood training and decoding leads to dull and repetitive outputs. While some post-hoc fixes have been proposed, in particular top-k and nucleus sampling, they do not address the fact that the token-level probabilities predicted by the model are poor. In this paper we show that the likelihood objective itself is at fault, resulting in a model that assigns too much probability to sequences containing repeats and frequent words, unlike those from the human training distribution. We propose a new objective, unlikelihood training, which forces unlikely generations to be assigned lower probability by the model. We show that both token and sequence level unlikelihood training give less repetitive, less dull text while maintaining perplexity, giving superior generations using standard greedy or beam search. According to human evaluations, our approach with standard beam search also outperforms the currently popular decoding methods of nucleus sampling or beam blocking, thus providing a strong alternative to existing techniques.

Schedule for Spring 2019

April 15
  • Tutorial on spectral learning for NLP (Naomi)
April 8
  • Tutorial on Normalizing Flows and Latent Normalizing Flows for Discrete Sequences (Nicola and Tom)
April 1
  • Syntax-Directed Variational Autoencoder for Structured Data (Nicola)
March 25
  • Tutorial on Variational Inference (Tom)
March 18
  • Skip
March 11
  • Tutorial on generative adversaria networks for text generation part II (Christos)
March 4
  • Skip - ACL deadline
February 27
  • Skip - ACL deadline
February 20
  • Tutorial on generative adversaria networks for text generation part I (Christos)
February 13
  • Tutorial on generative adversaria networks, adversarial training and adversarial examples (Nicola)
February 4
  • Discussion/planning for future meetings
January 28
  • Adversarial Sampling and Training for Semi-Supervised Information Retrieval [pdf] (Nelly)
January 21
  • A Structured Variational Autoencoder for Contextual Morphological Inflection [pdf] (Caio)
January 14
  • Simple and Effective Semi-Supervised Question Answering [pdf] (Nicola)

Schedule for Fall 2018

November - December 2018
  • break for NAACL and winter
November 19
  • Neural Language Modeling by Jointly Learning Syntax and Lexicon [pdf] (Hao)
November 12
  • cancelled
November 5
  • no meeting (EMNLP)
October 29
  • Learning to Compose Task-Specific Tree Structures [pdf] (Rei)
October 22
  • Understanding deep learning requires rethinking generalization [pdf] (Naomi)
October 15
  • Semantically Equivalent Adversarial Rules for Debugging NLP Models [pdf] (Jon)
October 8
  • On the Importance of Single Directions for Generalization [pdf] (Naomi)
October 1
  • planning meeting

Schedule for Spring 2018

May 7

  • Imagine This! Scripts to Compositions to Videos [pdf] (David)
April 30
  • SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient [pdf] (Chunchuan)
April 23
  • A Deep Reinforced Model for Abstractive Summarization [pdf] (Ratish)
April 16
  • Counterfactual fairness [pdf] (Naomi)
April 9
  • Poincaré Embeddings for Learning Hierarchical Representations [pdf] (Javad)
April 2
  • Semi-supervised learning with deep generative models [pdf] (Caio)
March 26
  • Tensor Fusion Network for Multimodal Sentiment Analysis [pdf] (Nikos)
February 26 - March 19
  • no meetings (strike)
February 19
  • Morphological Inflection Generation with Hard Monotonic Attention [pdf] (Joana)
February 12
  • A hybrid approach to dialogue management based on probabilistic rules [pdf] (Craig)
February 5
  • Semantic Composition via Probabilistic Model Theory [pdf] (Matt)
January 29
  • no meeting
January 22
  • Emergent translation in multi-agent communication [pdf] (Serhii)
January 15
  • planning meeting

Schedule for Fall 2017

December 4
  • Noisy Parallel Approximate Decoding for Conditional Recurrent Language Model [pdf] (Ratish)
November 27
  • Non-Autoregressive Neural Machine Translation [pdf] (Nikos)
November 14
  • no meeting
November 6
  • Word Representation Models for Morphologically Rich Languages in Neural Machine Translation [pdf] (Sander)
October 30
  • Cross-lingual, Character-Level Neural Morphological Tagging [pdf] (Clara)
October 23
  • Probabilistic Models for Segmenting and Labeling Sequence Data [pdf] (Joana)
October 16
  • A joint many-task model: Growing a neural network for multiple NLP tasks [pdf] (Marco)
October 9
  • Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling [pdf] (Diego)
October 2
  • Human Centered NLP with User-Factor Adaptation [pdf] (Pippa)
September 25
  • planning meeting

Schedule for Fall 2016

December 12
  • TBD (Rafael)
December 5
  • TBD (Esma)
November 28
  • A Decomposable Attention Model for Natural Language Inference (Imigo)
November 21
  • Jointly Combining Implicit Constraints Improves Temporal Ordering (Nate)
November 14
  • Policy distillation (Mihai)
November 7
  • Distilling the Knowledge in a Neural Network (Naomi)
October 31
October 24
  • Learning hidden unit contributions for unsupervised acoustic model adaptation (Joachim)
October 17
October 10
  • A fully bayesian approach to unsupervised POS tagging (Craig)
October 3

  • Beam sampling with Infinite HMM (Joana)

September 26

  • Introduction to Bayesian statistics (Shay)

September 19

Schedule for Spring 2016

May 30

  • Crash-course on Clustering (Akash)

May 23

May 2

  • Composing Neural Networks (Shashi)
    • Learning to compose neural networks for question answering, Jacob Andreas, Marcus Rohrbach, Trevor Darrell and Dan Klein. NAACL 2016.

Apr 25

Apr 18

  • Planning meeting (Herman and Shashi)

Apr 11

  • A quick introduction to Torch (Stef)
    • Differences to Theano and Tensorflow

Apr 4
  • A quick introduction to TensorFlow (Rafael and Clara)
Mar 28

Mar 21

  • Generative Bayesian-ish Neural Network Stuffs (Lea)
    • "Neurons as Monte Carlo Samplers: Bayesian Inference and Learning in Spiking Networks" from Huang and Rao :link
Mar 7

  • Generative Bayesian-ish Neural Network Stuffs (Jianpeng = JP)
Feb 29

Feb 22

Feb 15

Feb 8

Feb 1

Jan 25

Jan 18

Jan 11

Schedule for Fall 2015

Dec 7

Nov 30

Nov 23

  • Neural Networks: RNNs (Herman)
    • Deep Learning, An MIT Press book in preparation. Ian Goodfellow, Aaron Courville, and Yoshua Bengio. Section 10.2.2 in the RNN chapter and Section 8.2.5 in the optimization chapter.

Nov 16

Nov 9

Nov 2

Oct 26

Oct 19

Oct 12

Oct 5

Previous Meetings

Visit Previous meetings for a list of previous meetings.

Useful Information

Visit Useful Information for information about useful tools and paper recommendations from previous meetings.

Other Reading Groups




  • Planning Meeting 2017:
Topic attachments
I Attachment Action Size Date Who Comment
jpgjpg 20170109_154856.jpg manage 5153.2 K 02 Feb 2017 - 08:46 Main.s1467463 Planning Meeting 2017
pdfpdf bkj-VBwalkthrough.pdf manage 3428.9 K 11 Oct 2012 - 15:56 Main.s0894398 Variational EM
jpgjpg image.jpg manage 1749.3 K 12 Oct 2015 - 12:13 Main.snaraya2 planning-15-16-1
jpgjpg image_1.jpg manage 1652.0 K 12 Oct 2015 - 12:14 Main.snaraya2 planning-15-16-2
jpgjpg jan16-1.jpg manage 1752.7 K 14 Jan 2016 - 15:53 Main.snaraya2 Jan 2016: planning meeting A
jpgjpg jan16-2.jpg manage 1760.8 K 14 Jan 2016 - 15:54 Main.snaraya2  
pdfpdf lstm_intro.pdf manage 347.8 K 24 Nov 2014 - 16:03 Main.s0830457 Slides for the LSTM intro
pdfpdf lstm_intro_2015.pdf manage 356.9 K 30 Nov 2015 - 09:21 Main.s0830457 Updated version of LSTM introduction slides
pdfpdf planning_fall2016.pdf manage 264.0 K 19 Sep 2016 - 15:26 Main.s1333293  
pdfpdf variational_inference_primer.pdf manage 137.2 K 26 Feb 2016 - 09:34 Main.s1314728  
Topic revision: r127 - 03 Feb 2020 - 13:57:10 - Main.v1ndeca
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback
This Wiki uses Cookies