BERT

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova · NAACL 2019

Read paper · arXiv:1810.04805

Implement 80/10/10 selected-token corruption, original-token labels and stable cross entropy averaged only over selected positions.

What you will implement

Complete corruption and the MLM loss; a supplied bidirectional context table trains on synthetic triples. Transformer, NSP, tokenization and language benchmarks are outside scope.

• 101. BERT: Corruption and Masked Language Modeling

  Implement 80/10/10 selected-token corruption, original-token labels and stable cross entropy averaged only over selected positions.

  Complete corruption and the MLM loss; a supplied bidirectional context table trains on synthetic triples. Transformer, NSP, tokenization and language benchmarks are outside scope.