Update readme.
Browse files
README.md
CHANGED
@@ -14,7 +14,8 @@ datasets:
|
|
14 |
|
15 |
## Hebrew Language Model
|
16 |
|
17 |
-
State-of-the-art language model for Hebrew.
|
|
|
18 |
|
19 |
#### How to use
|
20 |
|
@@ -29,10 +30,9 @@ alephbert.eval()
|
|
29 |
```
|
30 |
|
31 |
## Training data
|
32 |
-
|
33 |
-
|
34 |
-
|
35 |
-
- Tweets (7G text, 70M sentences)
|
36 |
|
37 |
## Training procedure
|
38 |
|
@@ -49,6 +49,4 @@ Each section was trained for 5 epochs with an initial learning rate set to 1e-4.
|
|
49 |
|
50 |
Total training time was 5 days.
|
51 |
|
52 |
-
## Eval
|
53 |
-
|
54 |
|
|
|
14 |
|
15 |
## Hebrew Language Model
|
16 |
|
17 |
+
State-of-the-art language model for Hebrew.
|
18 |
+
Based on Google's BERT architecture [(Devlin et al. 2018)](https://arxiv.org/abs/1810.04805).
|
19 |
|
20 |
#### How to use
|
21 |
|
|
|
30 |
```
|
31 |
|
32 |
## Training data
|
33 |
+
1. OSCAR [(Ortiz, 2019)](https://oscar-corpus.com/) Hebrew section (10GB text, 20M sentences).
|
34 |
+
2. Hebrew dump of [Wikipedia](https://dumps.wikimedia.org/hewiki/latest/) (650 MB text, 3.8M sentences).
|
35 |
+
3. Hebrew Tweets collected from the Twitter sample stream (7G text, 70M sentences).
|
|
|
36 |
|
37 |
## Training procedure
|
38 |
|
|
|
49 |
|
50 |
Total training time was 5 days.
|
51 |
|
|
|
|
|
52 |
|