PIGlets
The purpose of the session is for
PhD students interested in machine learning to get to grips with advanced material sitting in difficulty above the taught courses here and below the discussions of cutting-edge material in the usual
PIGS meetings. Anyone else who's interested in joining us for the same is welcome, but the point is to get a solid handle on the techniques rather than debate their finer points. Meetings are organised by
Gavin Gray and generally held every second Monday at 10:30am in room
2.33 in the Informatics Forum. Announcements are made through the PIGS
mailing list.
Remit from 2016
To solve problems of persistence in teaching material and to encourage group discussion, we're going to try some changes to the format. Previously, one person lead the discussion with the expectation that everyone in the room had read the paper and wanted to discuss it. Unfortunately, this means that one person is stuck with a lot of work to learn the material in order to teach it to others. Also, just reading a paper has limited scope for discussion and is not necessarily the best way to learn about every topic. Alternatives to this format may be useful, such as a tutorial format with exercises released the week before, or a practical session introducing the problem that motivated the topic of interest. Volunteers in charge of a topic are free to choose whichever format they think is best, or even outsource leading the discussion (if they can).
Another problem we're hoping to solve is not having enough teaching material, because it's a lot of work to prepare it. Making the sessions more flexible, and urging people who have volunteered for a topic to prepare as much material as they can, whenever they are studying the topic, should let us schedule greedily based on what's available. For example, if someone presents on variational inference, preparing a talk around a fundamental paper, but at the same time comes across some good exercises, these exercises could be greedily scheduled at a later date for a tutorial session. We're hoping to aggregate this material in a
Git Book. This has the added advantage of making the material persistent; future students in ANC will have access to an index of materials for advanced topics.
Proposed topics for 2017
Volunteers to own a given topic are next to each topic:
- Conditional Random Fields
- Autoencoders: Gavin Gray
- Approximate Bayesian Computation: Harri Edwards
- Dimensionality Reduction
- Probabilistic Programming: Svetlin Penkov
- Sparse Regression
- Time-series Analysis: Alina Selega
- Gaussian Processes
- Learning Theory
- Reinforcement Learning: Harri Edwards
- Bayesian Neural Networks Part 2: George Papamakarios
- Experiment Logging: Gavin Gray
Topics Covered in 2016
So far, as of 22nd of February 2016:
- Bayesian Neural Networks: George Papamakarios
- Advanced MCMC: HMC, SGLD etc.: Matt Graham
- Variational Inference: Gavin Gray
- Bayesian Optimisation: James Wilson
Proposed topics for 2014/2015
- Graphical Models Review/ Factor graphs/ EP
- Dimensionality reduction
- Probabilistic programming
- Variational inference
- Sparse regression
- Sampling methods
- Time-series analysis
- Gaussian processes
Future meetings
Previous meetings
Gaussian Processes
- Iain (16 March 2015): Introduction to GPs [ slides] [ book]
Variational bayesian inference
- Gavin (02 Feb 2015) : Variational Bayes EM (Mixture of Gaussians). Section 21.6 from Murphy or 10.2 from Bishop.
- Agamemnon (19th Jan 2015) : Variational inference introduction (Section 21.1 - 21.4 from Murphy's textbook).
- George (02 March 2015): Variational message passing
Graphical models review / Expectation propagation
Time Series / Sequential Data
Conditional Random Fields
Bayesian Non-parametrics
Variational Inference and EP
Large Data/Online and Streaming
MCMC
- Jono (8th May): An Introduction to MCMC for Machine Learning - Andrieu, C., De Freitas, N., Doucet, A., Jordan, M.I. (2003) - Pages 1-27
- Miha (22nd May): Slice Sampling - Radford Neal (2000) - Sections 1, 2, 3 and 4
- Miha (22nd May): MCMC using Hamiltonian dynamics - Radford Neal (2000) - Sections 1, 2, and 3
Boltzmann machines, MRFs etc.
Theory of kernel methods
Dynamical Models
- Simon (Jan 17th): Continuous Time Bayesian Networks, U. Nodelman, C.R. Shelton, and D. Koller (2002)
- Simon (Jan 17th): Learning continuous time Bayesian networks, Nodelman, U., Shelton, C. R., & Koller, D. (2003)
- Benigno (Dec 6th): Towards Better Understanding of the Model Implied by the use of Dynamic Features in HMMs, Bridle, J. S. (2004)
Conditional Random Fields
- Dominik (Nov 22nd): Shallow Parsing with Conditional Random Fields, Sha, F. and Pereira, F. (2003)
- Dominik (Nov 22nd): Hidden-state Conditional Random Fields, Quattoni, A., Wang, S., Morency, L.P., Collins, M. and Darrell, T. (2007)
- Ioan (Nov 8th): An Introduction to Conditional Random Fields, Sutton, C. and McCallum , A. (2010)
- Ioan (Nov 8th): Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data, Lafferty, J., McCallum , A. and Pereira, F. (2001)
Model Combination
- Krzysztof (Oct 25th): Supervised aggregation of classifiers using artificial prediction markets, Lay, N. and Barbu, A. (2010)
- Krzysztof (Oct 25th): An Introduction to Artificial Prediction Markets for Classification, Barbu, A. and Lay, N. (2011)
- Krzysztof (Oct 25th): Machine Learning Markets, Storkey, A. (2011)
- Yichuan (Oct 11th): Adaptive Mixtures of Local Experts, Jacobs, R.A., Jordan, M.I., Nowlan, S.J. and Hinton, G.E. (1991)
- Yichuan (Oct 11th): Training products of experts by minimising contrastive divergence, Hinton, G.E. (2002)
- Peter (Sept 13th): Pattern Recognition and Machine Learning - Ch 14: Combining Models, C Bishop (2006)
- Konrad (Sept 27th): Integration of Stochastic Models by Minimizing α-Divergence, Amari, S. (2007)
- Konrad (Sept 27th): Parameter Estimation for α-GMM Based on Maximum Likelihood Criterion, Wu, D. (2009)
Optimisation
Sampling
- James (Apr 26th): Adaptive Direction Sampling, Gilks, W., Roberts, G., George, E. (1994)
- James (Apr 26th): Adaptive Independence Sampler, Keith, J., Kroese, D., Sofronov, G. (2008)
- Grigoris (May 24th): MCMC Using Ensembles of States for Problems with Fast and Slow Variables such as Gaussian Process Regression, Radford, N (2010)
- Simon (Jul 18th): The Gaussian Process Density Sampler, Adams, R., Murray, I., MacKay , D. (2009)
Topic Models
- Athina (Feb 15th): Bayesian Reasoning and Machine Learning, Barber, B. (2010)
- Athina (Feb 15th): The Author-Topic Model for Authors and Documents, Rosen-Zvi M., Griffiths T., Steyvers M., Smyth P. (2004)
- Ioan (Mar 1st): Finding Scientific Topics., Thomas L. Griffiths, Mark Steyvers. (2004)
- Ioan (Mar 1st): Topic modeling: beyond bag-of-words., Hanna M. Wallach. (2006)
- Jyri (Mar 15th): Probabilistic Topic Models, David Blei, Lawrence Carin, and David Dunson (2010)
- Jyri (Mar 15th): Hierarchical Bayesian nonparametric models with applications (Sections 1-5: pages 1-25), Yee W. Teh, and Michael I. Jordan (2010)
- Ondrej (Mar 29th): Discrete Component Analysis, W. Buntine and A. Jakulin (2006)
Discriminative and generative methods
- Principled Hybrids of Generative and Discriminative models, Lasserre, J. Bishop, C. M. and Minka, T. (2006)
- Hannes (February 1st): On Discriminative vs Generative Classifiers: A comparison of logistic regression and naive Bayes, Ng, A. and Jordan, M. (2008)
- Hannes (February 1st): An Asymptotic Analysis of Generative, Discriminative, and Pseudolikelihood Estimators, Liang. and Jordan, M. (2006)
Sparse ICA
- Ali (September 14th): Independent Components Analysis: Algorithms and Applications, Aapo Hyvärinen and Erkki Oja (2000)
- Konrad (September 28th): Sparse Coding with an Overcomplete Basis Set: A Strategy Employed by V1?,, Olshausen BA, Field DJ (1997)
- Peter (October 12th): An Introduction to Compressive Sampling, Emmanuel J. Candes and Michael B. Wakin (2008)
Markov Chain Monte Carlo methods
Probabilistic inference using Markov chain Monte Carlo methods, Radford M. Neal (1993)
- Jakub (July 6th): Pages 1-59
- Ali (July 20th): Pages 59-87
- Ali (August 3rd): Pages 59-87
- Nicolas (August 17th): Pages 87-121
Slice Sampling , Radford M. Neal (2003)
- Iain (August 31st): Pages 710-720
Archives
Visit
PIGlets 2009 for a list of meetings held in 2009.