studio-ousia/luke-large

LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention

LUKE (Language Understanding with Knowledge-based Embeddings) is a new pre-trained contextualized representation of words and entities based on transformer. LUKE treats words and entities in a given text as independent tokens, and outputs contextualized representations of them. LUKE adopts an entity-aware self-attention mechanism that is an extension of the self-attention mechanism of the transformer, and considers the types of tokens (words or entities) when computing attention scores.

LUKE achieves state-of-the-art results on five popular NLP benchmarks including SQuAD v1.1 (extractive question answering), CoNLL-2003 (named entity recognition), ReCoRD (cloze-style question answering), TACRED (relation classification), and Open Entity (entity typing).

Please check the official repository for more details and updates.

This is the LUKE large model with 24 hidden layers, 1024 hidden size. The total number of parameters in this model is 483M. It is trained using December 2018 version of Wikipedia.

Experimental results

The experimental results are provided as follows:

Task	Dataset	Metric	LUKE-large	luke-base	Previous SOTA
Extractive Question Answering	SQuAD v1.1	EM/F1	90.2/95.4	86.1/92.3	89.9/95.1 (Yang et al., 2019)
Named Entity Recognition	CoNLL-2003	F1	94.3	93.3	93.5 (Baevski et al., 2019)
Cloze-style Question Answering	ReCoRD	EM/F1	90.6/91.2	-	83.1/83.7 (Li et al., 2019)
Relation Classification	TACRED	F1	72.7	-	72.0 (Wang et al. , 2020)
Fine-grained Entity Typing	Open Entity	F1	78.2	-	77.6 (Wang et al. , 2020)

Citation

If you find LUKE useful for your work, please cite the following paper:

@inproceedings{yamada2020luke,
  title={LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention},
  author={Ikuya Yamada and Akari Asai and Hiroyuki Shindo and Hideaki Takeda and Yuji Matsumoto},
  booktitle={EMNLP},
  year={2020}
}