publications

(* denotes equal contribution)

2023

  1. Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference
    Tao Lei, Junwen Bai, Siddhartha Brahma, Joshua Ainslie, Kenton Lee, Yanqi Zhou, Nan Du, Vincent Y. Zhao, Yuexin Wu, Bo Li, Yu Zhang, and Ming-Wei Chang
    Preprint 2023
    PDF
  2. CoLT5: Faster Long-Range Transformers with Conditional Computation
    Joshua Ainslie, Tao Lei, Michiel Jong, Santiago Ontañón, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, Yun-Hsuan Sung, and Sumit Sanghai
    Preprint 2023
    PDF
  3. Rethinking the Role of Token Retrieval in Multi-Vector Retrieval
    Jinhyuk Lee, Zhuyun Dai, Sai Meher Karthik Duddu, Tao Lei, Iftekhar Naim, Ming-Wei Chang, and Vincent Y. Zhao
    Preprint 2023
    PDF

2022

  1. Training Language Models with Memory Augmentation
    Zexuan Zhong, Tao Lei, and Danqi Chen
    In EMNLP 2022
    PDF Code
  2. Mixture-of-Experts with Expert Choice Routing
    Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Y. Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon
    In NeurIPS 2022
    PDF
  3. SRU++: Pioneering Fast Recurrence with Attention for Speech Recognition
    Jing Pan, Tao Lei, Kwangyoun Kim, Kyu Han, and Shinji Watanabe
    In ICASSP 2022
    PDF

2021

  1. When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute
    Tao Lei
    In EMNLP 2021
    (Outstanding paper award)
    PDF Code

2020

  1. Structured Pruning of Large Language Models
    Ziheng Wang*, Jeremy Wohlwend*, and Tao Lei*
    In EMNLP 2020
    PDF Code
  2. Autoregressive Knowledge Distillation through Imitation Learning
    Alexander Lin, Jeremy Wohlwend, Howard Chen, and Tao Lei
    In EMNLP 2020
    PDF Code
  3. Rationalizing Text Matching: Learning Sparse Alignments via Optimal Transport
    Kyle Swanson*, Lili Yu*, and Tao Lei
    ACL 2020
    PDF Code
  4. Interactive Classification by Asking Informative Questions
    Lili Yu, Howard Chen, Sida Wang, Tao Lei, and Yoav Artzi
    In ACL 2020
    PDF Code
  5. ASAPP-ASR: Multistream CNN and Self-Attentive SRU for SOTA Speech Recognition
    Jing Pan*, Joshua Shapiro*, Jeremy Wohlwend*, Kyu J Han*, Tao Lei*, and Tao Ma
    In INTERSPEECH 2020
    PDF News

2019

  1. Building a Production Model for Retrieval-Based Chatbots
    Kyle Swanson, Lili Yu, Christopher Fox, Jeremy Wohlwend, and Tao Lei
    In 1st Workshop on NLP for Conversational AI 2019
    PDF
  2. Metric Learning for Dynamic Text Classification
    Jeremy Wohlwend, Ethan R Elenberg, Samuel Altschul, Shawn Henry, and Tao Lei
    EMNLP (DeepLo) 2019
    PDF

2018

  1. Simple Recurrent Units for Highly Parallelizable Recurrence
    Tao Lei, Yu Zhang, Sida I Wang, Hui Dai, and Yoav Artzi
    In EMNLP 2018
    PDF Code
  2. Adversarial Domain Adaptation for Duplicate Question Detection
    Darsh J Shah, Tao Lei, Alessandro Moschitti, Salvatore Romeo, and Preslav Nakov
    EMNLP 2018
    PDF Code

2017

  1. Style Transfer from Non-parallel Text by Cross-alignment
    Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola
    In NeurIPS 2017
    PDF Code Slides
  2. Interpretable Neural Models for Natural Language Processing
    Tao Lei
    Phd Thesis, MIT 2017
    PDF Slides
  3. Deriving Neural Architectures from Sequence and Graph Kernels
    Tao Lei*, Wengong Jin*, Regina Barzilay, and Tommi Jaakkola
    ICML 2017
    PDF Code

2016

  1. Rationalizing Neural Predictions
    Tao Lei, Regina Barzilay, and Tommi Jaakkola
    In EMNLP 2016
    PDF Code Slides News
  2. Semi-supervised Question Retrieval with Gated Convolutions
    Tao Lei, Hrishikesh Joshi, Regina Barzilay, Tommi Jaakkola, Katerina Tymoshenko, Alessandro Moschitti, and Lluı́s Màrquez
    In NAACL 2016
    PDF Code Slides
  3. Making Dependency Labeling Simple, Fast and Accurate
    Tianxiao Shen, Tao Lei, and Regina Barzilay
    In NAACL 2016
    PDF Code
  4. Learning to Refine Text based Recommendations
    Youyang Gu, Tao Lei, Regina Barzilay, and Tommi Jaakkola
    In EMNLP 2016
    PDF Code
  5. SLS at SemEval-2016 Task 3: Neural-based Approaches for Ranking in Community Question Answering
    Mitra Mohtarami, Yonatan Belinkov, Wei-Ning Hsu, Yu Zhang, Tao Lei, Kfir Bar, Scott Cyphers, and Jim Glass
    In SemEval 2016
    PDF

2015

  1. High-order Low-rank Tensors for Semantic Role Labeling
    Tao Lei, Yuan Zhang, Lluı́s Màrquez, Alessandro Moschitti, and Regina Barzilay
    In NAACL 2015
    PDF Code Slides
  2. Molding CNNs for Text: Non-linear, Non-consecutive Convolutions
    Tao Lei, Regina Barzilay, and Tommi Jaakkola
    In EMNLP 2015
    PDF Code Poster

2014

  1. Low-Rank Tensors for Scoring Dependency Structures
    Tao Lei, Yu Xin, Yuan Zhang, Regina Barzilay, and Tommi Jaakkola
    In ACL 2014
    (Best student paper award)
    PDF Code Slides
  2. Steps to Excellence: Simple Inference with Refined Scoring of Dependency Trees
    Yuan Zhang, Tao Lei, Regina Barzilay, Tommi Jaakkola, and Amir Globerson
    In ACL 2014
    PDF Code Slides
  3. Greed is Good if Randomized: New Inference for Dependency Parsing
    Yuan Zhang*, Tao Lei*, Regina Barzilay, and Tommi Jaakkola
    In EMNLP 2014
    PDF Supp Code
  4. Exploring Compositional Architectures and Word Vector Representations for Prepositional Phrase Attachment
    Yonatan Belinkov, Tao Lei, Regina Barzilay, and Amir Globerson
    TACL 2014
    PDF

2013

  1. From Natural Language Specifications to Program Input Parsers
    Tao Lei, Fan Long, Regina Barzilay, and Martin Rinard
    In ACL 2013
    PDF Code Slides

2012

  1. Learning High-level Planning from Text
    SRK Branavan, Nate Kushman, Tao Lei, and Regina Barzilay
    In ACL 2012
    PDF Code
  2. On Optimization of Expertise Matching with Various Constraints
    Wenbin Tang, Jie Tang, Tao Lei, Chenhao Tan, Bo Gao, and Tian Li
    Neurocomputing 2012

2010

  1. A Pattern Tree-based Approach to Learning URL Normalization Rules
    Tao Lei, Rui Cai, Jiang-Ming Yang, Yan Ke, Xiaodong Fan, and Lei Zhang
    In WWW 2010
    PDF