A distantly supervised dataset for automated data extraction from diagnostic studies

Christopher Norman; Mariska Leeflang; René Spijker; Evangelos Kanoulas; Aurélie Névéol

A distantly supervised dataset for automated data extraction from diagnostic studies

Christopher Norman
, Mariska Leeflang
, René Spijker
, Evangelos Kanoulas
, Aurélie Névéol

University of Amsterdam
Université Paris-Saclay

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

Abstract

Systematic reviews are important in evidence based medicine, but are expensive to produce. Automating or semi-automating the data extraction of index test, target condition, and reference standard from articles has the potential to decrease the cost of conducting systematic reviews of diagnostic test accuracy, but relevant training data is not available. We create a distantly supervised dataset of approximately 90,000 sentences, and let two experts manually annotate a small subset of around 1,000 sentences for evaluation. We evaluate the performance of BioBERT and logistic regression for ranking the sentences, and compare the performance for distant and direct supervision. Our results suggest that distant supervision can work as well as, or better than direct supervision on this problem, and that distantly trained models can perform as well as, or better than human annotators.

Original language	English
Title of host publication	BioNLP 2019 - SIGBioMed Workshop on Biomedical Natural Language Processing, Proceedings of the 18th BioNLP Workshop and Shared Task
Publisher	Association for Computational Linguistics (ACL)
Pages	105-114
Number of pages	10
ISBN (Electronic)	9781950737284
Publication status	Published - 2019
Event	18th SIGBioMed Workshop on Biomedical Natural Language Processing, BioNLP 2019 - Florence, Italy Duration: 1 Aug 2019 → …

Publication series

Name	BioNLP 2019 - SIGBioMed Workshop on Biomedical Natural Language Processing, Proceedings of the 18th BioNLP Workshop and Shared Task

Conference

Conference	18th SIGBioMed Workshop on Biomedical Natural Language Processing, BioNLP 2019
Country/Territory	Italy
City	Florence
Period	01/08/2019 → …

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 3 Good Health and Well-being

Cite this

Norman, C., Leeflang, M., Spijker, R., Kanoulas, E., & Névéol, A. (2019). A distantly supervised dataset for automated data extraction from diagnostic studies. In BioNLP 2019 - SIGBioMed Workshop on Biomedical Natural Language Processing, Proceedings of the 18th BioNLP Workshop and Shared Task (pp. 105-114). (BioNLP 2019 - SIGBioMed Workshop on Biomedical Natural Language Processing, Proceedings of the 18th BioNLP Workshop and Shared Task). Association for Computational Linguistics (ACL).

Norman, Christopher ; Leeflang, Mariska ; Spijker, René et al. / A distantly supervised dataset for automated data extraction from diagnostic studies. BioNLP 2019 - SIGBioMed Workshop on Biomedical Natural Language Processing, Proceedings of the 18th BioNLP Workshop and Shared Task. Association for Computational Linguistics (ACL), 2019. pp. 105-114 (BioNLP 2019 - SIGBioMed Workshop on Biomedical Natural Language Processing, Proceedings of the 18th BioNLP Workshop and Shared Task).

@inproceedings{8ff546d5a9364371be45fd8a51312e09,

title = "A distantly supervised dataset for automated data extraction from diagnostic studies",

abstract = "Systematic reviews are important in evidence based medicine, but are expensive to produce. Automating or semi-automating the data extraction of index test, target condition, and reference standard from articles has the potential to decrease the cost of conducting systematic reviews of diagnostic test accuracy, but relevant training data is not available. We create a distantly supervised dataset of approximately 90,000 sentences, and let two experts manually annotate a small subset of around 1,000 sentences for evaluation. We evaluate the performance of BioBERT and logistic regression for ranking the sentences, and compare the performance for distant and direct supervision. Our results suggest that distant supervision can work as well as, or better than direct supervision on this problem, and that distantly trained models can perform as well as, or better than human annotators.",

author = "Christopher Norman and Mariska Leeflang and Ren{\'e} Spijker and Evangelos Kanoulas and Aur{\'e}lie N{\'e}v{\'e}ol",

note = "Funding Information: This project has received funding from the European Union{\textquoteright}s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 676207. Funding Information: This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 676207. Publisher Copyright: {\textcopyright} 2019 Association for Computational Linguistics; 18th SIGBioMed Workshop on Biomedical Natural Language Processing, BioNLP 2019 ; Conference date: 01-08-2019",

year = "2019",

language = "English",

series = "BioNLP 2019 - SIGBioMed Workshop on Biomedical Natural Language Processing, Proceedings of the 18th BioNLP Workshop and Shared Task",

publisher = "Association for Computational Linguistics (ACL)",

pages = "105--114",

booktitle = "BioNLP 2019 - SIGBioMed Workshop on Biomedical Natural Language Processing, Proceedings of the 18th BioNLP Workshop and Shared Task",

}

Norman, C, Leeflang, M , Spijker, R, Kanoulas, E & Névéol, A 2019, A distantly supervised dataset for automated data extraction from diagnostic studies. in BioNLP 2019 - SIGBioMed Workshop on Biomedical Natural Language Processing, Proceedings of the 18th BioNLP Workshop and Shared Task. BioNLP 2019 - SIGBioMed Workshop on Biomedical Natural Language Processing, Proceedings of the 18th BioNLP Workshop and Shared Task, Association for Computational Linguistics (ACL), pp. 105-114, 18th SIGBioMed Workshop on Biomedical Natural Language Processing, BioNLP 2019, Florence, Italy, 01/08/2019.

A distantly supervised dataset for automated data extraction from diagnostic studies. / Norman, Christopher; Leeflang, Mariska ; Spijker, René et al.
BioNLP 2019 - SIGBioMed Workshop on Biomedical Natural Language Processing, Proceedings of the 18th BioNLP Workshop and Shared Task. Association for Computational Linguistics (ACL), 2019. p. 105-114 (BioNLP 2019 - SIGBioMed Workshop on Biomedical Natural Language Processing, Proceedings of the 18th BioNLP Workshop and Shared Task).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - A distantly supervised dataset for automated data extraction from diagnostic studies

AU - Norman, Christopher

AU - Leeflang, Mariska

AU - Spijker, René

AU - Kanoulas, Evangelos

AU - Névéol, Aurélie

N1 - Funding Information: This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 676207. Funding Information: This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 676207. Publisher Copyright: © 2019 Association for Computational Linguistics

PY - 2019

Y1 - 2019

N2 - Systematic reviews are important in evidence based medicine, but are expensive to produce. Automating or semi-automating the data extraction of index test, target condition, and reference standard from articles has the potential to decrease the cost of conducting systematic reviews of diagnostic test accuracy, but relevant training data is not available. We create a distantly supervised dataset of approximately 90,000 sentences, and let two experts manually annotate a small subset of around 1,000 sentences for evaluation. We evaluate the performance of BioBERT and logistic regression for ranking the sentences, and compare the performance for distant and direct supervision. Our results suggest that distant supervision can work as well as, or better than direct supervision on this problem, and that distantly trained models can perform as well as, or better than human annotators.

AB - Systematic reviews are important in evidence based medicine, but are expensive to produce. Automating or semi-automating the data extraction of index test, target condition, and reference standard from articles has the potential to decrease the cost of conducting systematic reviews of diagnostic test accuracy, but relevant training data is not available. We create a distantly supervised dataset of approximately 90,000 sentences, and let two experts manually annotate a small subset of around 1,000 sentences for evaluation. We evaluate the performance of BioBERT and logistic regression for ranking the sentences, and compare the performance for distant and direct supervision. Our results suggest that distant supervision can work as well as, or better than direct supervision on this problem, and that distantly trained models can perform as well as, or better than human annotators.

UR - https://www.scopus.com/pages/publications/85094732207

M3 - Conference contribution

T3 - BioNLP 2019 - SIGBioMed Workshop on Biomedical Natural Language Processing, Proceedings of the 18th BioNLP Workshop and Shared Task

SP - 105

EP - 114

BT - BioNLP 2019 - SIGBioMed Workshop on Biomedical Natural Language Processing, Proceedings of the 18th BioNLP Workshop and Shared Task

PB - Association for Computational Linguistics (ACL)

T2 - 18th SIGBioMed Workshop on Biomedical Natural Language Processing, BioNLP 2019

Y2 - 1 August 2019

ER -

Norman C, Leeflang M , Spijker R, Kanoulas E, Névéol A. A distantly supervised dataset for automated data extraction from diagnostic studies. In BioNLP 2019 - SIGBioMed Workshop on Biomedical Natural Language Processing, Proceedings of the 18th BioNLP Workshop and Shared Task. Association for Computational Linguistics (ACL). 2019. p. 105-114. (BioNLP 2019 - SIGBioMed Workshop on Biomedical Natural Language Processing, Proceedings of the 18th BioNLP Workshop and Shared Task).

A distantly supervised dataset for automated data extraction from diagnostic studies

Abstract

Publication series

Conference

UN SDGs

Other files and links

Fingerprint

Cite this