Deep Learning: Classics and Trends



Deep Learning: Classics and Trends (DL:C&T) is the hottest thing in town…JK. It is a reading group / talk series I have been running since June 2018. It started within Uber AI Labs, when we felt the need of a space to sample the overwhelmingly large amount of papers, and to hold free-form, judgemental (JK) discussions.

  • Time: every Friday, 12pm - 1pm
  • Place: Uber HQ, San Francisco
  • Format: someone presents; others listen; all eat.
  • Scope: deep learning, old (a.k.a. “let’s revisit the 2014 GAN paper”) and new (a.k.a. “look at this blog post from yesterday”).

If you are in San Francisco, definitely consider joining, either as a presenter or just an observer! Drop me an email to discuss.

Coming up:

Date Presenter Topic or Paper
2019.11.22 Rosanne Liu On the “steerability” of generative adversarial networks

Past events:

Date Presenter Topic or Paper
2019.11.15 Polina Binder Learning Deep Sigmoid Belief Networks with Data Augmentation
2019.11.08 Sanyam Kapoor Policy Search & Planning: Unifying Connections [1][2]
2019.11.01 Chris Olah Zoom in: Features and circuits as the basic unit of neural networks
2019.10.25 Renjie Liao Efficient Graph Generation with Graph Recurrent Attention Networks
2019.10.18 Nitish Shirish Keskar, Bryan McCann CTRL: A Conditional Transformer Language Model for Controllable Generation
2019.10.11 Subutai Ahmad Sparsity in the neocortex, and its implications for machine learning
2019.10.04 Eli Bingham Multiple Causes: A Causal Graphical View
2019.09.27 Xinyu Hu Learning Representations for Counterfactual Inference
2019.09.04 Jonathan Frankle The Latest Updates on the Lottery Ticket Hypothesis
2019.08.23 Ankit Jain Knowledge-aware Graph Neural Networks with Label Smoothness Regularization for Recommender Systems
2019.08.16 Jiale Zhi Meta-Learning Neural Bloom Filters
2019.08.16 Ted Moskovitz Lookahead Optimizer: k steps forward, 1 step back
2019.07.26 Rui Wang Off-Policy Evaluation for Contextual Bandits and RL [1][2][3][4]
2019.07.19 Rosanne Liu Weight Agnostic Neural Networks
2019.07.12 Joost Huizinga A Distributional Perspective on Reinforcement Learning
2019.06.28 Ashley Edwards [ICML Preview] Learning Values and Policies from Observation [1][2]
2019.06.21 Stanislav Fořt [ICML Preview] Large Scale Structure of Neural Network Loss Landscapes
2019.06.07 Joey Bose [ICML Preview] Compositional Fairness Constraints for Graph Embeddings
2019.05.31 Yulun Li IntentNet: Learning to Predict Intention from Raw Sensor Data
2019.05.24 Thomas Miconi, Rosanne Liu, Janice Lan ICLR Recap, cont.
2019.05.17 Aditya Rawal, Jason Yosinski ICLR Recap
2019.04.26 JP Chen 3D-Aware Scene Manipulation via Inverse Graphics
2019.04.19 Felipe Petroski Such Relational Deep Reinforcement Learning
2019.04.12 Piero Molino, Jason Yosinski Open mic
2019.04.05 Joel Lehman The copycat project: A model of mental fluidity and analogy-making
2019.03.29 Rosanne Liu Non-local Neural Networks
2019.03.22 Yariv Sadan Learning deep representations by mutual information estimation and maximization
2019.03.15 Chandra Khatri Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation
2019.03.01 Nikhil Dev Deshmudre BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
2019.02.22 Vashisht Madhavan Neural Turing Machines
2019.02.15 open discussion GPT-2
2019.02.08 Adrien Ecoffet HyperNetworks
2019.02.01 Jiale Zhi Non-delusional Q-learning and value iteration
2019.01.25 Yulun Li Relational Recurrent Neural Networks
2019.01.18 Rui Wang Neural Ordinary Differential Equations
2019.01.11 Jonathan Simon Generating Humorous Portmanteaus using Word Embeddings [1][2]
2018.12.21 Christian Perez Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
2018.12.14 Alexandros Papangelis Two trends in dialog [1][2]
2018.10.26 Aditya Rawal Averaging Weights Leads to Wider Optima and Better Generalization
2018.10.12 Mahdi Namazifar Troubling Trends in Machine Learning Scholarship
2018.09.28 Yariv Sadan MINE: Mutual Information Neural Estimation
2018.09.21 Jan-Matthis Lueckmann Glow and RealNVP
2018.09.14 Jane Hung The YOLO series: v1, v2, v3
2018.09.07 Rosanne Liu Pooling is Neither Necessary nor Sufficient for Appropriate Deformation Stability in CNNs
2018.08.31 Alican Bozkur Multimodal Unsupervised Image-to-Image Translation
2018.08.24 Janice Lan The Lottery Ticket Hypothesis: Finding Small, Trainable Neural Networks
2018.08.17 Yariv Sadan Opening the black box of Deep Neural Networks via Information
2018.08.10 Joost Huizinga Learning to Reinforcement Learn, and RL 2: Fast Reinforcement Learning via Slow Reinforcement Learning
2018.08.03 JP Chen Deep Convolutional Inverse Graphics Network
2018.07.27 Lei Shu Attention is all you need
2018.07.06 Neeraj Pradhan Auto encoding Variational Bayes, and ELBO
2018.06.29 Ankit Jain Dynamic Routing Between Capsules
2018.06.22 Xinyu Hu Self Normalizing Neural Networks
2018.06.15 John Sears The Decline and Fall of Adam: [1][2]
2018.06.08 Alex Gajewski GANs, etc. [1][2]
2018.06.01 Jason Yosinski Sensitivity and Generalization in Neural Networks: an Empirical Study

Congratulations! Since you have scrolled all the way down here, you get the reward of reading more texts about the scope and vision of this reading group.

  • Q: What was the initial idea of organizing a reading group like this?
    A: It started with the rather selfish idea that I want to know about papers that I don’t have time to read, and learn about topics my individual intelligence limits me from fully understanding. Besides, I enjoy being around people that are smarter than me, work out math faster than me, and value giving great talks as much as I do.

  • Q: How much work is it for you?
    A: I never travel on Fridays now.

  • Q: Where do you see it going?
    A: I want it to be a community where people work hard to tell science stories well. Each paper is a story. A great paper, apart from solid results and technical and scientific advances, stands out particularly in the way it tells the story. I hope we all value storytelling and talk giving slightly more than we do now. This ties to an eventual wish that scientific writing moves towards being lucid and understandable. This reading group is a start.

    Here is how I see different levels of storytelling, in the format of a one-hour presentation, can happen in this group.

    You can give a Level 0 talk, which is going through someone else’s paper—the storyline is already there. This is perhaps the most basic and involves the least work: you just need to understand it and retell it to others. (I assume as a researcher you already read papers yourself, and this additional work of making it into a presentation would definitely help you understand it better yourself.) And best of all, when the audience asks hard questions, you can just say “I don’t know”.

    A Level 1 talk, could mean presenting one of your own papers. The bar is higher because you are expected to know every detail of the project. And a good background introduction to lead to the exact problem always helps.

    Then we have Level 2 talks, which are usually a topic formed by understanding a field (however small it is) or making connections with a number of fields. You might be citing multiple papers, drawing connections and coming up with conclusions that are mainly your own.

  • Q: Do you have a high bar for talks given there?
    A: Yes I do. But I also know we all have to start somewhere. And I myself was a horrible presenter not too long ago (likely still am). So let’s all get better.