Slot 2

Hardware Architectures for Deep Neural Networks
Joel Emer and Vivienne Sze, NVIDIA and MIT, USA


Deep neural networks (DNNs) are currently widely used for many AI applications including computer vision, speech recognition, robotics, etc. While DNNs deliver state-of-the-art accuracy on many AI tasks, it comes at the cost of high computational complexity. Accordingly, designing efficient hardware architectures for deep neural networks is an important step towards enabling the wide deployment of DNNs in AI systems.

In this class, we will provide an overview of DNNs, discuss the tradeoffs of the various architectures that support DNNs including CPU, GPU, FPGA and ASIC, and highlight important benchmarking/comparison metrics and design considerations. We will then describe recent techniques that reduce the computation cost of DNNs from both the hardware architecture and network algorithm perspective. Finally, we will discuss the different hardware requirements for inference and training.


Joel Emer is a Senior Distinguished Research Scientist at Nvidia in Westford, MA, where he is responsible for exploration of future architectures as well as modeling and analysis methodologies. He is also a Professor of the Practice in the Computer Science and Electrical Engineering department at MIT. Prior to joining NVIDIA, he worked at Intel where he was an Intel Fellow and Director of Microarchitecture Research. Previously he worked at Compaq and Digital Equipment Corporation.

He has held various research and advanced development positions investigating processor micro-architecture and developing performance modeling and evaluation techniques. He has made architectural contributions to a number of VAX, Alpha and X86 processors and is recognized as one of the developers of the widely employed quantitative approach to processor performance evaluation. He has also been recognized for his contributions in the advancement of simultaneous multi-threading technology, analysis of the architectural impact of soft errors, memory dependence prediction, pipeline and cache organization, performance modeling methodologies and spatial architectures. His current research interests include memory hierarchy design, processor reliability, spatial architectures and performance modeling.

He received a bachelor's degree with highest honors in electrical engineering in 1974, and his master's degree in 1975 -- both from Purdue University. He earned a doctorate in electrical engineering from the University of Illinois in 1979. Emer holds over 25 patents and has published more than 40 papers. He is a Fellow of both the ACM and the IEEE, and was the 2009 recipient of the Eckert-Mauchly award for lifetime contributions in computer architecture.


Vivienne Sze joined the EECS Department as an Assistant Professor in August 2013. She received the B.A.Sc. degree in Electrical Engineering from the University of Toronto in 2004, and the S.M. and Ph.D. degree in Electrical Engineering and Computer Science from MIT in 2006 and 2010, respectively. From September 2010 to July 2013, she was a Member of the Technical Staff in the Systems and Applications R&D Center at Texas Instruments.

Prof. Sze's research focuses on joint design of algorithms, architectures and circuits to build energy efficient and high performance systems. Her work on implementation-friendly video compression algorithms was used in the development of the latest video coding standard HEVC/H.265, enabling it to deliver better compression than previous standards, while still achieving high processing speeds and low hardware cost. She aims to develop energy-aware algorithms and efficient architectures for various energy-constrained applications including portable multimedia, health monitoring and distributed sensing.

She has received various awards including the Jin-Au Kong Outstanding Doctoral Thesis Prize in 2011, the 2007 DAC/ISSCC Student Design Contest Award, the 2008 A-SSCC Outstanding Design Award, the Natural Sciences and Engineering Research Council of Canada (NSERC) Julie Payette fellowship in 2004, the NSERC Postgraduate Scholarships in 2005 and 2007, and the Texas Instruments Graduate Woman's Fellowship for Leadership in Microelectronics in 2008. In 2012, she was selected by IEEE-USA as one of the "New Faces of Engineering".

  Back to course info