Real-time processing of social media with SENTINEL: a syndromic surveillance system incorporating deep learning for health classification
File(s)1-s2.0-S0306457317303448-main.pdf (3.24 MB)
Published version
Author(s)
Hankin, CL
Serban, Ovidiu
Thapen, Nicholas
Maginnis, Brendan
Foot, Virginia
Type
Journal Article
Abstract
Interest in real-time syndromic surveillance based on social media data has greatly increased in recent years.
The ability to detect disease outbreaks earlier than traditional methods would be highly useful for public
health officials. This paper describes a software system which is built upon recent developments in machine
learning and data processing to achieve this goal. The system is built from reusable modules integrated into
data processing pipelines that are easily deployable and configurable. It applies deep learning to the problem
of classifying health-related tweets and is able to do so with high accuracy. It has the capability to detect
illness outbreaks from Twitter data and then to build up and display information about these outbreaks,
including relevant news articles, to provide situational awareness. It also provides nowcasting functionality
of current disease levels from previous clinical data combined with Twitter data.
The preliminary results are promising, with the system being able to detect outbreaks of influenza-like
illness symptoms which could then be confirmed by existing official sources. The Nowcasting module shows
that using social media data can improve prediction for multiple diseases over simply using traditional data
sources.
The ability to detect disease outbreaks earlier than traditional methods would be highly useful for public
health officials. This paper describes a software system which is built upon recent developments in machine
learning and data processing to achieve this goal. The system is built from reusable modules integrated into
data processing pipelines that are easily deployable and configurable. It applies deep learning to the problem
of classifying health-related tweets and is able to do so with high accuracy. It has the capability to detect
illness outbreaks from Twitter data and then to build up and display information about these outbreaks,
including relevant news articles, to provide situational awareness. It also provides nowcasting functionality
of current disease levels from previous clinical data combined with Twitter data.
The preliminary results are promising, with the system being able to detect outbreaks of influenza-like
illness symptoms which could then be confirmed by existing official sources. The Nowcasting module shows
that using social media data can improve prediction for multiple diseases over simply using traditional data
sources.
Date Issued
2019-05-01
Date Acceptance
2018-04-28
Citation
Information Processing and Management, 2019, 56 (3), pp.1166-1184
ISSN
0306-4573
Publisher
Elsevier
Start Page
1166
End Page
1184
Journal / Book Title
Information Processing and Management
Volume
56
Issue
3
Copyright Statement
© 2018 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/BY/4.0/.
Sponsor
Defence Science and Technology Laboratory (DSTL)
Grant Number
DSTL/AGR/00728/01
Subjects
Science & Technology
Technology
Computer Science, Information Systems
Information Science & Library Science
Computer Science
Real-time processing
Classification
Clustering
Event detection
EVENT DETECTION
Information & Library Sciences
0806 Information Systems
0807 Library and Information Studies
Publication Status
Published
Date Publish Online
2018-06-01