Multimodal federated learning on IoT data
File(s)
Accepted version
Author(s)
Zhao, Yuchen
Barnaghi, Payam
Haddadi, Hamed
Type
Conference Paper
Abstract
Federated learning is proposed as an alternative to centralized machine learning since its client-server structure provides better privacy protection and scalability in real-world applications. In many applications, such as smart homes with Internet-of-Things (IoT) devices, local data on clients are generated from different modalities such as sensory, visual, and audio data. Existing federated learning systems only work on local data from a single modality, which limits the scalability of the systems. In this paper, we propose a multimodal and semi-supervised federated learning framework that trains autoencoders to extract shared or correlated representations from different local data modalities on clients. In addition, we propose a multimodal FedAvg algorithm to aggregate local autoencoders trained on different data modalities. We use the learned global autoencoder for a downstream classification task with the help of auxiliary labelled data on the server. We empirically evaluate our framework on different modalities including sensory data, depth camera videos, and RGB camera videos. Our experimental results demonstrate that introducing data from multiple modalities into federated learning can improve its classification performance. In addition, we can use labelled data from only one modality for supervised learning on the server and apply the learned model to testing data from other modalities to achieve decent F1 scores (e.g., with the best performance being higher than 60%), especially when combining contributions from both unimodal clients and multimodal clients.
Date Issued
2022-06-23
Date Acceptance
2022-05-04
Citation
2022 IEEE/ACM Seventh International Conference on Internet-of-Things Design and Implementation (IoTDI), 2022
Publisher
IEEE
Journal / Book Title
2022 IEEE/ACM Seventh International Conference on Internet-of-Things Design and Implementation (IoTDI)
Copyright Statement
© 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Sponsor
Medical Research Council
Identifier
https://ieeexplore.ieee.org/document/9797401
Grant Number
UKDRI-7002
Source
2022 IEEE/ACM Seventh International Conference on Internet-of-Things Design and Implementation (IoTDI)
Subjects
cs.LG
cs.LG
Publication Status
Published
Start Date
2022-05-04
Finish Date
2022-05-06