Skip to Main Content
Programming Machine Learning
book

Programming Machine Learning

by Paolo Perrotta
March 2020
Beginner to intermediate content levelBeginner to intermediate
342 pages
8h 38m
English
Pragmatic Bookshelf
Content preview from Programming Machine Learning

Beyond the Sigmoid

There is no such thing as a perfect replacement for the sigmoid. Different activation functions work well in different circumstances, and researchers keep coming up with brand-new ones. That being said, one activation function has proven so broadly useful that it’s become a default of sorts. Let’s talk about it.

Enter the ReLU

The go-to replacement for the sigmoid these days is the rectified linear unit, or ReLU for friends. Compared with the sigmoid, the ReLU is surprisingly simple. Here’s a Python implementation of it:

 def​ ​relu​(z):
 if​ z <= 0:
 return​ 0
 else​:
 return​ z

And the following diagram illustrates what it looks like:

The ReLU is composed of two straight segments. However, taken together they add ...

Become an O’Reilly member and get unlimited access to this title plus top books and audiobooks from O’Reilly and nearly 200 top publishers, thousands of courses curated by job role, 150+ live events each month,
and much more.
Start your free trial

You might also like

Mastering Machine Learning Algorithms - Second Edition

Mastering Machine Learning Algorithms - Second Edition

Giuseppe Bonaccorso
Practical Machine Learning for Computer Vision

Practical Machine Learning for Computer Vision

Valliappa Lakshmanan, Martin Görner, Ryan Gillard

Publisher Resources

ISBN: 9781680507706Errata Page