Table of Contents

Activation Functions

Rumman Ansari June 25, 2026 17 views Subject Details

NEURAL NETWORKS — CORE CONCEPT

Activation Functions

The decision-making power inside neural networks — helping models learn complex patterns beyond simple straight lines.

Introduction to Activation Functions

An Activation Function is a mathematical function used inside a neural network to decide whether a neuron should be activated or not. It takes the weighted sum of inputs and transforms it into an output that is passed to the next layer.

Without activation functions, a neural network would behave like a simple linear model, no matter how many layers it has. Activation functions introduce non-linearity, allowing neural networks to learn complex relationships such as image patterns, speech signals, text meaning, and real-world decision boundaries.

"Activation functions are the reason neural networks can learn complex patterns instead of acting like simple linear equations."

Why It Matters: Activation functions make deep learning powerful by helping neural networks solve non-linear problems.

Definition

DEFINITION

Activation Function = A function that transforms neuron input into neuron output

A neuron first calculates a weighted sum:

$$ z = W \cdot X + b $$

Then the activation function is applied:

$$ a = f(z) $$

X: Input values
W: Weights
b: Bias
z: Weighted sum
f(z): Activation function
a: Activated output

Simple Real-Life Analogy

Think of an Activation Function Like a Switch

A switch decides whether electricity should pass or not. Similarly, an activation function decides whether a neuron should pass useful information forward. Some activation functions work like an ON/OFF switch, while others allow partial signals to pass.

Why Do We Need Activation Functions?

Adds Non-Linearity

Allows the network to learn complex patterns instead of only straight-line relationships.

Controls Neuron Output

Decides how much information should move forward to the next layer.

Helps Learning

Supports gradient-based learning during backpropagation.

Improves Model Power

Makes deep neural networks useful for AI tasks like vision, NLP, and prediction.

What Happens Without Activation Functions?

If we do not use activation functions, every layer in a neural network will only perform linear transformations. Even if we add many layers, the final output will still behave like a single linear equation.

IMPORTANT IDEA

Linear Layer + Linear Layer + Linear Layer = Still Linear

Key Point: Activation functions allow neural networks to learn curves, boundaries, and complex relationships.

Types of Activation Functions

Step Function

The simplest activation function used in early perceptrons.

The Step Function gives output 1 if the input is greater than or equal to a threshold, otherwise it gives output 0. It is useful for simple binary decisions.

$$ f(z) = \begin{cases} 1, & \text{if } z \geq 0 \\ 0, & \text{if } z < 0 \end{cases} $$

Best For Basic perceptron and simple binary classification demonstration.

Limitation Not suitable for deep learning because it is not differentiable.

Sigmoid Function

Converts input into a value between 0 and 1.

The Sigmoid Function is commonly used when we need probability-like output. It is often used in the output layer for binary classification problems.

$$ f(z) = \frac{1}{1 + e^{-z}} $$

Output range: 0 to 1
Useful for binary classification
Output can be interpreted as probability

Best For Binary classification output layer.

Limitation Can cause vanishing gradient problem in deep networks.

Tanh Function

Converts input into a value between -1 and 1.

The Tanh Function is similar to sigmoid but its output is centered around zero. This often makes learning better than sigmoid in hidden layers.

$$ f(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}} $$

Output range: -1 to 1
Zero-centered output
Better than sigmoid for many hidden-layer use cases

Best For Hidden layers when zero-centered activation is useful.

Limitation Still suffers from vanishing gradients for very deep networks.

ReLU Function

The most commonly used activation function in deep learning.

ReLU stands for Rectified Linear Unit. It returns 0 for negative values and returns the same value for positive inputs.

$$ f(z) = \max(0, z) $$

Very simple and fast
Helps reduce vanishing gradient problem
Commonly used in hidden layers

Best For Hidden layers in most deep neural networks.

Limitation Can suffer from dying ReLU problem when neurons output 0 permanently.

Leaky ReLU Function

A modified version of ReLU that allows small negative values.

Leaky ReLU solves the dying ReLU problem by allowing a small slope for negative inputs instead of making them exactly zero.

$$ f(z) = \begin{cases} z, & \text{if } z > 0 \\ \alpha z, & \text{if } z \leq 0 \end{cases} $$

Allows small negative output
Reduces dead neuron problem
Common value of alpha is 0.01

Best For Deep networks where normal ReLU creates dead neurons.

Softmax Function

Converts outputs into probabilities for multiple classes.

The Softmax Function is used in the output layer of multi-class classification models. It converts raw scores into probabilities whose total sum is 1.

$$ Softmax(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{k} e^{z_j}} $$

Output values are probabilities
All probabilities add up to 1
Used for multi-class classification

Best For Output layer of multi-class classification models.

Comparison of Activation Functions

Function	Output Range	Common Use	Main Issue
Step	0 or 1	Basic perceptron	Not differentiable
Sigmoid	0 to 1	Binary output layer	Vanishing gradient
Tanh	-1 to 1	Hidden layers	Vanishing gradient
ReLU	0 to infinity	Hidden layers	Dying ReLU
Leaky ReLU	Negative small slope to infinity	Deep networks	Alpha needs tuning
Softmax	0 to 1, sum = 1	Multi-class output layer	Used mainly at output layer

Which Activation Function to Use?

Layer / Problem Type	Recommended Activation	Reason
Hidden Layers	ReLU	Fast, simple, and works well in most deep networks
Deep Hidden Layers with Dead Neurons	Leaky ReLU	Allows small negative values and reduces dead neurons
Binary Classification Output	Sigmoid	Produces probability between 0 and 1
Multi-Class Classification Output	Softmax	Produces probability distribution over classes
Regression Output	Linear Activation	Allows continuous numeric output

Common Problems Related to Activation Functions

Vanishing Gradient

Gradients become extremely small during backpropagation.
Early layers learn very slowly.
Common in sigmoid and tanh for deep networks.

Dying ReLU

Some ReLU neurons always output 0.
Those neurons stop learning.
Leaky ReLU can help reduce this issue.

Exploding Gradient

Gradients become extremely large.
Training becomes unstable.
Can be controlled using normalization and gradient clipping.

Wrong Output Activation

Using wrong output activation gives poor predictions.
Example: using Softmax for binary-only output unnecessarily.

Python Example: Activation Functions from Scratch

Below is a simple NumPy implementation of common activation functions.

import numpy as np

# Step Function
def step_function(z):
    return np.where(z >= 0, 1, 0)

# Sigmoid Function
def sigmoid(z):
    return 1 / (1 + np.exp(-z))

# Tanh Function
def tanh(z):
    return np.tanh(z)

# ReLU Function
def relu(z):
    return np.maximum(0, z)

# Leaky ReLU Function
def leaky_relu(z, alpha=0.01):
    return np.where(z > 0, z, alpha * z)

# Softmax Function
def softmax(z):
    exp_values = np.exp(z - np.max(z))
    return exp_values / np.sum(exp_values)

# Test values
z = np.array([-2, -1, 0, 1, 2])

print("Step:", step_function(z))
print("Sigmoid:", sigmoid(z))
print("Tanh:", tanh(z))
print("ReLU:", relu(z))
print("Leaky ReLU:", leaky_relu(z))
print("Softmax:", softmax(z))

Insight: Each activation function transforms the same input differently, which changes how a neural network learns.

Visualizing Activation Functions

The following code plots sigmoid, tanh, ReLU, and Leaky ReLU functions.

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(-5, 5, 100)

sigmoid_y = 1 / (1 + np.exp(-x))
tanh_y = np.tanh(x)
relu_y = np.maximum(0, x)
leaky_relu_y = np.where(x > 0, x, 0.01 * x)

plt.figure(figsize=(10, 6))

plt.plot(x, sigmoid_y, label="Sigmoid")
plt.plot(x, tanh_y, label="Tanh")
plt.plot(x, relu_y, label="ReLU")
plt.plot(x, leaky_relu_y, label="Leaky ReLU")

plt.title("Activation Functions")
plt.xlabel("Input")
plt.ylabel("Output")
plt.legend()
plt.grid(True)
plt.show()

Tip: Visualizing activation functions helps students clearly understand their output behavior.

Using Activation Functions in Keras

In deep learning frameworks, activation functions are usually added directly inside neural network layers.

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

model = Sequential()

# Hidden layers use ReLU
model.add(Dense(64, activation="relu", input_shape=(10,)))
model.add(Dense(32, activation="relu"))

# Binary classification output uses Sigmoid
model.add(Dense(1, activation="sigmoid"))

model.summary()

Use: ReLU is commonly used in hidden layers, while sigmoid or softmax is used in output layers depending on the problem.

Real-World Use of Activation Functions

Computer Vision

CNN models commonly use ReLU in hidden layers to detect edges, shapes, and objects.

Natural Language Processing

Neural networks use activation functions to learn relationships between words, sentences, and meanings.

Healthcare AI

Sigmoid can be used in binary disease prediction models where output represents probability.

Multi-Class Classification

Softmax is used when the model must choose one class from many possible categories.

Common Mistakes vs Best Practices

Common Mistakes	Best Practices
Using sigmoid in every hidden layer	Use ReLU or Leaky ReLU for hidden layers
Using softmax for binary classification	Use sigmoid for binary output
Ignoring vanishing gradient problem	Use ReLU-based functions in deep networks
Using wrong output activation for regression	Use linear activation for continuous numeric output
Not visualizing function behavior	Plot activation functions to understand them better

Advantages vs Limitations

Advantages

Add non-linearity to neural networks.
Help models learn complex patterns.
Support backpropagation and gradient-based learning.
Enable different outputs such as probabilities and class scores.
Improve performance in deep learning architectures.

Limitations

Some functions cause vanishing gradients.
Some functions may create dead neurons.
Wrong activation choice can reduce accuracy.
Different tasks require different output activations.
Activation function choice may need experimentation.

Prerequisites Before Learning

What You Should Know First

Basic understanding of perceptron.
Knowledge of weighted sum and bias.
Basic Python and NumPy skills.
Understanding of binary and multi-class classification.
Basic idea of neural network layers.

Pro Tips to Master Activation Functions

Smart Learning Strategy

Start with step function and sigmoid before moving to ReLU.
Visualize each activation function using graphs.
Use ReLU as the default choice for hidden layers.
Use sigmoid for binary classification output.
Use softmax for multi-class classification output.
Understand vanishing gradient before studying backpropagation deeply.

Key Formulas Cheat Sheet

NEURON OUTPUT

a = f(W · X + b)

SIGMOID

f(z) = 1 / (1 + e^-z)

TANH

f(z) = (e^z − e^-z) / (e^z + e^-z)

RELU

f(z) = max(0, z)

SOFTMAX

e^zi / Σ e^zj

Chapter Summary

Quick Recap

Activation functions transform neuron input into output.
They add non-linearity to neural networks.
Step function is used in basic perceptrons.
Sigmoid is useful for binary classification output.
Softmax is useful for multi-class classification output.
ReLU is the most common hidden-layer activation function.
Choosing the right activation function improves model performance.

Final Takeaway

Activation Functions are the decision engines of neural networks. They help neurons decide what information should move forward and allow deep learning models to learn complex, non-linear patterns. Mastering activation functions makes it easier to understand backpropagation, CNNs, RNNs, and modern deep learning models.