Minimax and Neyman–Pearson Meta-Learning for Outlier Languages
Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review
Documents
- Minimax and Neyman–Pearson Meta-Learning for Outlier Languages
Final published version, 704 KB, PDF document
Model-agnostic meta-learning (MAML) hasbeen recently put forth as a strategy to learnresource-poor languages in a sample-efficientfashion. Nevertheless, the properties of theselanguages are often not well represented bythose available during training. Hence, weargue that the i.i.d. assumption ingrained inMAML makes it ill-suited for cross-lingualNLP. In fact, under a decision-theoretic framework, MAML can be interpreted as minimising the expected risk across training languages(with a uniform prior), which is known asBayes criterion. To increase its robustness tooutlier languages, we create two variants ofMAML based on alternative criteria: MinimaxMAML reduces the maximum risk across languages, while Neyman–Pearson MAML constrains the risk in each language to a maximum threshold. Both criteria constitute fullydifferentiable two-player games. In light ofthis, we propose a new adaptive optimiser solving for a local approximation to their Nashequilibrium. We evaluate both model variants on two popular NLP tasks, part-of-speechtagging and question answering. We reportgains for their average and minimum performance across low-resource languages in zeroand few-shot settings, compared to joint multisource transfer and vanilla MAML. The codefor our experiments is available at https://github.com/rahular/robust-maml.
Original language | English |
---|---|
Title of host publication | Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 |
Publisher | Association for Computational Linguistics |
Publication date | 2021 |
Pages | 1245-1260 |
DOIs | |
Publication status | Published - 2021 |
Event | Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 - Virtual, Online Duration: 1 Aug 2021 → 6 Aug 2021 |
Conference
Conference | Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 |
---|---|
By | Virtual, Online |
Periode | 01/08/2021 → 06/08/2021 |
ID: 300446234