Simulating Hard Attention Using Soft Attention
Transactions of the Association for Computational Linguistics, 14:147–166.
Publications
My work focuses on the computational abilities and internal structure of neural sequence models, especially transformers, using tools from theoretical computer science and formal language theory.
Transformers as Transducers
Transactions of the Association for Computational Linguistics, 13:200–219.
Concise One-Layer Transformers Can Do Function Evaluation (Sometimes)
arXiv preprint.
What Formal Languages Can Transformers Express? A Survey
Transactions of the Association for Computational Linguistics, 12:543–561.
Average-Hard Attention Transformers are Constant-Depth Uniform Threshold Circuits
arXiv preprint.