language | tags | license | widget | ||||
---|---|---|---|---|---|---|---|
en |
|
mit |
|
transformers
library to version 4.22+ if you need to refer to the old name.Pretraining large neural language models, such as BERT, has led to impressive gains on many natural language processing (NLP) tasks. However, most pretraining efforts focus on general domain corpora, such as newswire and Web. A prevailing assumption is that even domain-specific pretraining can benefit by starting from general-domain language models. Recent work shows that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains over continual pretraining of general-domain language models. Followup work explores larger model sizes and the impact of these on performance on the BLURB benchmark.
This BiomedBERT is pretrained from scratch using abstracts from PubMed.
If you find BiomedBERT useful in your research, please cite the following paper:
@misc{https://doi.org/10.48550/arxiv.2112.07869,
doi = {10.48550/ARXIV.2112.07869},
url = {https://arxiv.org/abs/2112.07869},
author = {Tinn, Robert and Cheng, Hao and Gu, Yu and Usuyama, Naoto and Liu, Xiaodong and Naumann, Tristan and Gao, Jianfeng and Poon, Hoifung},
keywords = {Computation and Language (cs.CL), Machine Learning (cs.LG), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processing},
publisher = {arXiv},
year = {2021},
copyright = {arXiv.org perpetual, non-exclusive license}
}
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。
1. 开源生态
2. 协作、人、软件
3. 评估模型