julien-c HF staff commited on
Commit
48e631c
1 Parent(s): 6cfbb3d

Migrate model card from transformers-repo

Browse files

Read announcement at https://discuss.huggingface.co/t/announcement-all-model-cards-will-be-migrated-to-hf-co-model-repos/2755
Original file history: https://github.com/huggingface/transformers/commits/master/model_cards/google/electra-base-discriminator/README.md

Files changed (1) hide show
  1. README.md +36 -0
README.md ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ thumbnail: https://huggingface.co/front/thumbnails/google.png
4
+
5
+ license: apache-2.0
6
+ ---
7
+
8
+ ## ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
9
+
10
+ **ELECTRA** is a new method for self-supervised language representation learning. It can be used to pre-train transformer networks using relatively little compute. ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens generated by another neural network, similar to the discriminator of a [GAN](https://arxiv.org/pdf/1406.2661.pdf). At small scale, ELECTRA achieves strong results even when trained on a single GPU. At large scale, ELECTRA achieves state-of-the-art results on the [SQuAD 2.0](https://rajpurkar.github.io/SQuAD-explorer/) dataset.
11
+
12
+ For a detailed description and experimental results, please refer to our paper [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators](https://openreview.net/pdf?id=r1xMH1BtvB).
13
+
14
+ This repository contains code to pre-train ELECTRA, including small ELECTRA models on a single GPU. It also supports fine-tuning ELECTRA on downstream tasks including classification tasks (e.g,. [GLUE](https://gluebenchmark.com/)), QA tasks (e.g., [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/)), and sequence tagging tasks (e.g., [text chunking](https://www.clips.uantwerpen.be/conll2000/chunking/)).
15
+
16
+ ## How to use the discriminator in `transformers`
17
+
18
+ ```python
19
+ from transformers import ElectraForPreTraining, ElectraTokenizerFast
20
+ import torch
21
+
22
+ discriminator = ElectraForPreTraining.from_pretrained("google/electra-base-discriminator")
23
+ tokenizer = ElectraTokenizerFast.from_pretrained("google/electra-base-discriminator")
24
+
25
+ sentence = "The quick brown fox jumps over the lazy dog"
26
+ fake_sentence = "The quick brown fox fake over the lazy dog"
27
+
28
+ fake_tokens = tokenizer.tokenize(fake_sentence)
29
+ fake_inputs = tokenizer.encode(fake_sentence, return_tensors="pt")
30
+ discriminator_outputs = discriminator(fake_inputs)
31
+ predictions = torch.round((torch.sign(discriminator_outputs[0]) + 1) / 2)
32
+
33
+ [print("%7s" % token, end="") for token in fake_tokens]
34
+
35
+ [print("%7s" % int(prediction), end="") for prediction in predictions.tolist()]
36
+ ```