Development and evaluation of a manual segmentation protocol for deep grey matter in multiple sclerosis: Towards accelerated semi-automated references

Alexandra de Sitter; Jessica Burggraaff; Fabian Bartel; Miklos Palotai; Yaou Liu; Jorge Simoes; Serena Ruggieri; Katharina Schregel; Stefan Ropele; Maria A. Rocca; Claudio Gasperini; Antonio Gallo; Menno M. Schoonheim; Michael Amann; Marios Yiannakas; Deborah Pareto; Mike P. Wattjes; Jaume Sastre-Garriga; Ludwig Kappos; Massimo Filippi; Christian Enzinger; Jette Frederiksen; Bernard Uitdehaag; Charles R. G. Guttmann; Frederik Barkhof; Hugo Vrenken

doi:10.1016/j.nicl.2021.102659

Development and evaluation of a manual segmentation protocol for deep grey matter in multiple sclerosis: Towards accelerated semi-automated references

Alexandra de Sitter
, Jessica Burggraaff^*
, Fabian Bartel
, Miklos Palotai
, Yaou Liu
, Jorge Simoes
, Serena Ruggieri
, Katharina Schregel
, Stefan Ropele
, Maria A. Rocca
, Claudio Gasperini
, Antonio Gallo
, Menno M. Schoonheim
, Michael Amann
, Marios Yiannakas
, Deborah Pareto
, Mike P. Wattjes
, Jaume Sastre-Garriga
, Ludwig Kappos
, Massimo Filippi

Christian Enzinger, Jette Frederiksen, Bernard Uitdehaag, Charles R. G. Guttmann, Frederik Barkhof, Hugo Vrenken

^*Corresponding author for this work

Vrije Universiteit (VU) Amsterdam and VU Medical Center
Harvard Medical School
University of Rome La Sapienza
San Camillo Hospital
University Medical Center Göttingen
Medical University of Graz
Neuroimaging Research Unit, Institute of Experimental Neurology, Division of Neuroscience, United States
IRCCS Ospedale San Raffaele
University of Campania Luigi Vanvitelli
Medical Image Analysis Center (MIAC), United States
University of Basel
UCL Institute of Neurology
Autonomous University of Barcelona
Hannover Medical School
Vita-Salute San Raffaele University
University of Copenhagen

Research output: Contribution to journal › Article › Academic › peer-review

4 Downloads (Pure)

Abstract

Background: Deep grey matter (dGM) structures, particularly the thalamus, are clinically relevant in multiple sclerosis (MS). However, segmentation of dGM in MS is challenging; labeled MS-specific reference sets are needed for objective evaluation and training of new methods. Objectives: This study aimed to (i) create a standardized protocol for manual delineations of dGM; (ii) evaluate the reliability of the protocol with multiple raters; and (iii) evaluate the accuracy of a fast-semi-automated segmentation approach (FASTSURF). Methods: A standardized manual segmentation protocol for caudate nucleus, putamen, and thalamus was created, and applied by three raters on multi-center 3D T1-weighted MRI scans of 23 MS patients and 12 controls. Intra- and inter-rater agreement was assessed through intra-class correlation coefficient (ICC); spatial overlap through Jaccard Index (JI) and generalized conformity index (CIgen). From sparse delineations, FASTSURF reconstructed full segmentations; accuracy was assessed both volumetrically and spatially. Results: All structures showed excellent agreement on expert manual outlines: intra-rater JI > 0.83; inter-rater ICC ≥ 0.76 and CIgen ≥ 0.74. FASTSURF reproduced manual references excellently, with ICC ≥ 0.97 and JI ≥ 0.92. Conclusions: The manual dGM segmentation protocol showed excellent reproducibility within and between raters. Moreover, combined with FASTSURF a reliable reference set of dGM segmentations can be produced with lower workload.

Original language	English
Article number	102659
Journal	NeuroImage: Clinical
Volume	30
DOIs	https://doi.org/10.1016/j.nicl.2021.102659
Publication status	Published - 1 Jan 2021

Keywords

Atrophy
Deep grey matter
MRI
Multiple Sclerosis
Reference set
Segmentation

Access to Document

10.1016/j.nicl.2021.102659

Development-and-evaluation-of-a-manual-segmentation-protocol-for-deep-grey-matter-in-multiple-sclerosis-towards-acceler.pdfFinal published version, 1.99 MBLicence: CC BY

Cite this

de Sitter, A., Burggraaff, J., Bartel, F., Palotai, M., Liu, Y., Simoes, J., Ruggieri, S., Schregel, K., Ropele, S., Rocca, M. A., Gasperini, C., Gallo, A., Schoonheim, M. M., Amann, M., Yiannakas, M., Pareto, D., Wattjes, M. P., Sastre-Garriga, J., Kappos, L., ... Vrenken, H. (2021). Development and evaluation of a manual segmentation protocol for deep grey matter in multiple sclerosis: Towards accelerated semi-automated references. NeuroImage: Clinical, 30, Article 102659. https://doi.org/10.1016/j.nicl.2021.102659

@article{5f0d8101ae5b4ae9ac529e721b1fc266,

title = "Development and evaluation of a manual segmentation protocol for deep grey matter in multiple sclerosis: Towards accelerated semi-automated references",

abstract = "Background: Deep grey matter (dGM) structures, particularly the thalamus, are clinically relevant in multiple sclerosis (MS). However, segmentation of dGM in MS is challenging; labeled MS-specific reference sets are needed for objective evaluation and training of new methods. Objectives: This study aimed to (i) create a standardized protocol for manual delineations of dGM; (ii) evaluate the reliability of the protocol with multiple raters; and (iii) evaluate the accuracy of a fast-semi-automated segmentation approach (FASTSURF). Methods: A standardized manual segmentation protocol for caudate nucleus, putamen, and thalamus was created, and applied by three raters on multi-center 3D T1-weighted MRI scans of 23 MS patients and 12 controls. Intra- and inter-rater agreement was assessed through intra-class correlation coefficient (ICC); spatial overlap through Jaccard Index (JI) and generalized conformity index (CIgen). From sparse delineations, FASTSURF reconstructed full segmentations; accuracy was assessed both volumetrically and spatially. Results: All structures showed excellent agreement on expert manual outlines: intra-rater JI > 0.83; inter-rater ICC ≥ 0.76 and CIgen ≥ 0.74. FASTSURF reproduced manual references excellently, with ICC ≥ 0.97 and JI ≥ 0.92. Conclusions: The manual dGM segmentation protocol showed excellent reproducibility within and between raters. Moreover, combined with FASTSURF a reliable reference set of dGM segmentations can be produced with lower workload.",

keywords = "Atrophy, Deep grey matter, MRI, Multiple Sclerosis, Reference set, Segmentation",

author = "\{de Sitter\}, Alexandra and Jessica Burggraaff and Fabian Bartel and Miklos Palotai and Yaou Liu and Jorge Simoes and Serena Ruggieri and Katharina Schregel and Stefan Ropele and Rocca, \{Maria A.\} and Claudio Gasperini and Antonio Gallo and Schoonheim, \{Menno M.\} and Michael Amann and Marios Yiannakas and Deborah Pareto and Wattjes, \{Mike P.\} and Jaume Sastre-Garriga and Ludwig Kappos and Massimo Filippi and Christian Enzinger and Jette Frederiksen and Bernard Uitdehaag and Guttmann, \{Charles R. G.\} and Frederik Barkhof and Hugo Vrenken",

note = "Funding Information: The members of the MAGNIMS Study Group Steering Committee are: F. Barkhof and H. Vrenken (Amsterdam UMC, location VUmc, Amsterdam, Netherlands), O. Ciccarelli and T. Yousry (Queen Square MS Centre, UCL Institute of Neurology, London, UK), N. De Stefano (University of Siena, Siena, Italy), C. Enzinger (Department of Neurology, Medical University of Graz, Graz, Austria), M. Filippi and M.A. Rocca (San Raffaele Scientific Institute, Vita-Salute San Raffaele University, Milan, Italy), C. Gasperini (San Camillo-Forlanini Hospital, Rome, Italy), L. Kappos (University of Basel, Basel, Switzerland), J. Palace (University of Oxford Hospitals Trust, Oxford, UK), A. Rovira and J. Sastre-Garriga (Hospital Universitari Vall d'Hebron, Universitat Aut?noma de Barcelona, Barcelona, Spain). The study was funded by the Nauta fonds through a travel grant. The MS Center Amsterdam is supported by the Dutch MS Research Foundation through a program grant (current grant 18-358f). D.B. is supported by project PI18/00823 from the ?Fondo de Investigaci?n Sanitaria Carlos III?. F.B. is supported by the National Institute for Health Research University College London Hospitals Biomedical Research Centre. The acquisition of data in London was funded by supported by the National Institute for Health Research University College London Hospitals Biomedical Research Centre. Funding Information: The members of the MAGNIMS Study Group Steering Committee are: F. Barkhof and H. Vrenken (Amsterdam UMC, location VUmc, Amsterdam, Netherlands), O. Ciccarelli and T. Yousry (Queen Square MS Centre, UCL Institute of Neurology, London, UK), N. De Stefano (University of Siena, Siena, Italy), C. Enzinger (Department of Neurology, Medical University of Graz, Graz, Austria), M. Filippi and M.A. Rocca (San Raffaele Scientific Institute, Vita-Salute San Raffaele University, Milan, Italy), C. Gasperini (San Camillo-Forlanini Hospital, Rome, Italy), L. Kappos (University of Basel, Basel, Switzerland), J. Palace (University of Oxford Hospitals Trust, Oxford, UK), A. Rovira and J. Sastre-Garriga (Hospital Universitari Vall d{\textquoteright}Hebron, Universitat Aut{\`o}noma de Barcelona, Barcelona, Spain). The study was funded by the Nauta fonds through a travel grant. The MS Center Amsterdam is supported by the Dutch MS Research Foundation through a program grant (current grant 18-358f). D.B. is supported by project PI18/00823 from the “Fondo de Investigaci{\'o}n Sanitaria Carlos III”. F.B. is supported by the National Institute for Health Research University College London Hospitals Biomedical Research Centre. The acquisition of data in London was funded by supported by the National Institute for Health Research University College London Hospitals Biomedical Research Centre. Publisher Copyright: {\textcopyright} 2021 The Authors Copyright: Copyright 2021 Elsevier B.V., All rights reserved.",

year = "2021",

month = jan,

day = "1",

doi = "10.1016/j.nicl.2021.102659",

language = "English",

volume = "30",

journal = "NeuroImage: Clinical",

issn = "2213-1582",

publisher = "Elsevier BV",

}

de Sitter, A, Burggraaff, J, Bartel, F, Palotai, M, Liu, Y, Simoes, J, Ruggieri, S, Schregel, K, Ropele, S, Rocca, MA, Gasperini, C, Gallo, A, Schoonheim, MM, Amann, M, Yiannakas, M, Pareto, D, Wattjes, MP, Sastre-Garriga, J, Kappos, L, Filippi, M, Enzinger, C, Frederiksen, J, Uitdehaag, B, Guttmann, CRG, Barkhof, F & Vrenken, H 2021, 'Development and evaluation of a manual segmentation protocol for deep grey matter in multiple sclerosis: Towards accelerated semi-automated references', NeuroImage: Clinical, vol. 30, 102659. https://doi.org/10.1016/j.nicl.2021.102659

TY - JOUR

T1 - Development and evaluation of a manual segmentation protocol for deep grey matter in multiple sclerosis: Towards accelerated semi-automated references

AU - de Sitter, Alexandra

AU - Burggraaff, Jessica

AU - Bartel, Fabian

AU - Palotai, Miklos

AU - Liu, Yaou

AU - Simoes, Jorge

AU - Ruggieri, Serena

AU - Schregel, Katharina

AU - Ropele, Stefan

AU - Rocca, Maria A.

AU - Gasperini, Claudio

AU - Gallo, Antonio

AU - Schoonheim, Menno M.

AU - Amann, Michael

AU - Yiannakas, Marios

AU - Pareto, Deborah

AU - Wattjes, Mike P.

AU - Sastre-Garriga, Jaume

AU - Kappos, Ludwig

AU - Filippi, Massimo

AU - Enzinger, Christian

AU - Frederiksen, Jette

AU - Uitdehaag, Bernard

AU - Guttmann, Charles R. G.

AU - Barkhof, Frederik

AU - Vrenken, Hugo

N1 - Funding Information: The members of the MAGNIMS Study Group Steering Committee are: F. Barkhof and H. Vrenken (Amsterdam UMC, location VUmc, Amsterdam, Netherlands), O. Ciccarelli and T. Yousry (Queen Square MS Centre, UCL Institute of Neurology, London, UK), N. De Stefano (University of Siena, Siena, Italy), C. Enzinger (Department of Neurology, Medical University of Graz, Graz, Austria), M. Filippi and M.A. Rocca (San Raffaele Scientific Institute, Vita-Salute San Raffaele University, Milan, Italy), C. Gasperini (San Camillo-Forlanini Hospital, Rome, Italy), L. Kappos (University of Basel, Basel, Switzerland), J. Palace (University of Oxford Hospitals Trust, Oxford, UK), A. Rovira and J. Sastre-Garriga (Hospital Universitari Vall d'Hebron, Universitat Aut?noma de Barcelona, Barcelona, Spain). The study was funded by the Nauta fonds through a travel grant. The MS Center Amsterdam is supported by the Dutch MS Research Foundation through a program grant (current grant 18-358f). D.B. is supported by project PI18/00823 from the ?Fondo de Investigaci?n Sanitaria Carlos III?. F.B. is supported by the National Institute for Health Research University College London Hospitals Biomedical Research Centre. The acquisition of data in London was funded by supported by the National Institute for Health Research University College London Hospitals Biomedical Research Centre. Funding Information: The members of the MAGNIMS Study Group Steering Committee are: F. Barkhof and H. Vrenken (Amsterdam UMC, location VUmc, Amsterdam, Netherlands), O. Ciccarelli and T. Yousry (Queen Square MS Centre, UCL Institute of Neurology, London, UK), N. De Stefano (University of Siena, Siena, Italy), C. Enzinger (Department of Neurology, Medical University of Graz, Graz, Austria), M. Filippi and M.A. Rocca (San Raffaele Scientific Institute, Vita-Salute San Raffaele University, Milan, Italy), C. Gasperini (San Camillo-Forlanini Hospital, Rome, Italy), L. Kappos (University of Basel, Basel, Switzerland), J. Palace (University of Oxford Hospitals Trust, Oxford, UK), A. Rovira and J. Sastre-Garriga (Hospital Universitari Vall d’Hebron, Universitat Autònoma de Barcelona, Barcelona, Spain). The study was funded by the Nauta fonds through a travel grant. The MS Center Amsterdam is supported by the Dutch MS Research Foundation through a program grant (current grant 18-358f). D.B. is supported by project PI18/00823 from the “Fondo de Investigación Sanitaria Carlos III”. F.B. is supported by the National Institute for Health Research University College London Hospitals Biomedical Research Centre. The acquisition of data in London was funded by supported by the National Institute for Health Research University College London Hospitals Biomedical Research Centre. Publisher Copyright: © 2021 The Authors Copyright: Copyright 2021 Elsevier B.V., All rights reserved.

PY - 2021/1/1

Y1 - 2021/1/1

N2 - Background: Deep grey matter (dGM) structures, particularly the thalamus, are clinically relevant in multiple sclerosis (MS). However, segmentation of dGM in MS is challenging; labeled MS-specific reference sets are needed for objective evaluation and training of new methods. Objectives: This study aimed to (i) create a standardized protocol for manual delineations of dGM; (ii) evaluate the reliability of the protocol with multiple raters; and (iii) evaluate the accuracy of a fast-semi-automated segmentation approach (FASTSURF). Methods: A standardized manual segmentation protocol for caudate nucleus, putamen, and thalamus was created, and applied by three raters on multi-center 3D T1-weighted MRI scans of 23 MS patients and 12 controls. Intra- and inter-rater agreement was assessed through intra-class correlation coefficient (ICC); spatial overlap through Jaccard Index (JI) and generalized conformity index (CIgen). From sparse delineations, FASTSURF reconstructed full segmentations; accuracy was assessed both volumetrically and spatially. Results: All structures showed excellent agreement on expert manual outlines: intra-rater JI > 0.83; inter-rater ICC ≥ 0.76 and CIgen ≥ 0.74. FASTSURF reproduced manual references excellently, with ICC ≥ 0.97 and JI ≥ 0.92. Conclusions: The manual dGM segmentation protocol showed excellent reproducibility within and between raters. Moreover, combined with FASTSURF a reliable reference set of dGM segmentations can be produced with lower workload.

AB - Background: Deep grey matter (dGM) structures, particularly the thalamus, are clinically relevant in multiple sclerosis (MS). However, segmentation of dGM in MS is challenging; labeled MS-specific reference sets are needed for objective evaluation and training of new methods. Objectives: This study aimed to (i) create a standardized protocol for manual delineations of dGM; (ii) evaluate the reliability of the protocol with multiple raters; and (iii) evaluate the accuracy of a fast-semi-automated segmentation approach (FASTSURF). Methods: A standardized manual segmentation protocol for caudate nucleus, putamen, and thalamus was created, and applied by three raters on multi-center 3D T1-weighted MRI scans of 23 MS patients and 12 controls. Intra- and inter-rater agreement was assessed through intra-class correlation coefficient (ICC); spatial overlap through Jaccard Index (JI) and generalized conformity index (CIgen). From sparse delineations, FASTSURF reconstructed full segmentations; accuracy was assessed both volumetrically and spatially. Results: All structures showed excellent agreement on expert manual outlines: intra-rater JI > 0.83; inter-rater ICC ≥ 0.76 and CIgen ≥ 0.74. FASTSURF reproduced manual references excellently, with ICC ≥ 0.97 and JI ≥ 0.92. Conclusions: The manual dGM segmentation protocol showed excellent reproducibility within and between raters. Moreover, combined with FASTSURF a reliable reference set of dGM segmentations can be produced with lower workload.

KW - Atrophy

KW - Deep grey matter

KW - MRI

KW - Multiple Sclerosis

KW - Reference set

KW - Segmentation

UR - https://www.scopus.com/pages/publications/85104345552

U2 - 10.1016/j.nicl.2021.102659

DO - 10.1016/j.nicl.2021.102659

M3 - Article

C2 - 33882422

SN - 2213-1582

VL - 30

JO - NeuroImage: Clinical

JF - NeuroImage: Clinical

M1 - 102659

ER -

Development and evaluation of a manual segmentation protocol for deep grey matter in multiple sclerosis: Towards accelerated semi-automated references

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this