Monday, 07 Dec 2020
Location information: Virtual Meeting.

Schedule

9:00-9:20Introduction & Motivation
9:20-10:30Neural Network Compression + Q&A
10:30-11:00Coffee Break
11:00-12:00Distributed Learning
12:00-12:30Code Demo + Q&A

Summary

Deep neural networks have recently demonstrated their incredible ability to solve complex tasks. Today's models are trained on millions of examples and are able to reliably annotate images, translate text, understand spoken language or play strategic games such as chess or Go. At the same time the number of intelligent devices on wireless networks (e.g. smartphones, IoT devices) has rapidly grown. These devices are equipped with sensors and with increasingly powerful processors that allow them to collect and process data at unprecedented scales. This development provides a unique opportunity for deep learning methods to revolutionize these applications.

However, due to limited resources (e.g., bandwidth and power), latency constraints, and data privacy concerns, centralized training schemes, which require all the data to reside at a central location and were the basis of all above-mentioned successes, are no longer available in the wireless network setting. Thus, these training schemes are increasingly substituted by distributed deep learning, which allows multiple parties to jointly train a model on their combined data, without any of the participants having to reveal their local data to other parties, or to a centralized server. This new form of collaborative training concentrates learning in locations where the models are actually used (i.e., on the network edge), and thus minimizes latency and resource consumption.

The objective of the tutorial is to introduce the most important concepts and methods in distributed deep learning, and to systematically discuss the challenges and advantages of their application in wireless networks. The tutorial will not only provide a theoretical understanding of the distributed learning problem (e.g., distributed SGD, federated averaging, convergence results) and teach the related concepts from information theory, optimization and wireless communications, but also discuss the small tricks (e.g., error accumulation, synchronization, client clustering) to make distributed learning schemes work in practice. Furthermore, we will present the recent developments and trends, in particular the applications of distributed learning to wireless networks, and give a first-hand summary of the relevant standardization activities (e.g., ITU FG ML5G, MPEG AHG CNNMCD).

Our goal is that the attendees (1) understand the methodological and theoretical concepts in distributed and federated learning, (2) have an overview of the recent developments in these fields, and (3) know how to practically apply these methods in wireless networks.

For background material on the topic, see our reading list.

Outline of the tutorial

1. Introduction
- Brief introduction to deep learning
- Challenges and limitations of current models in wireless applications
- Recent developments in research & industry

2. Distributed & Federated Learning: Concepts & Methods
- Background: Information theory, learning theory, SGD
- Basic concepts of federated and distributed learning
- Reducing communication overhead
- Clustered federated learning

3. Questions

4. Coffee Break

5. Distributed & Federated Learning in Wireless Networks
- Hierarchical federated learning over heterogenous wireless networks
- Over-the-air SGD

6. Distributed Learning & Neural Network Compression
- Conceptual similarities and differences between these two problems
- Standardization activities: ITU FG ML5G, MPEG AHG CNNMCD

7. Conclusion

8. Questions

Organizers

Wojciech Samek Deniz Gunduz
Fraunhofer Heinrich Hertz Institute Imperial Colleague London