Neural Network
Definition
A neural network is a layered arrangement of neurons, each computing a weighted sum of its inputs, applying an activation function, and passing the result on. Networks learn by adjusting weights via gradient descent and backpropagation to minimize a cost function.
Core Ideas
Perceptron vs neuron
A perceptron is the simplest artificial neuron — a single unit that traditionally uses a step activation (output 1 above threshold, else 0), making it a binary classifier. A general neuron isn’t limited to binary output: with an appropriate activation it handles regression and multi-class problems.
Activation functions
- Sigmoid/logistic — output in (0,1), read as class probability (>0.5 → class 1); good for binary classification
- tanh, Gaussian, sine
- ReLU — common in CNNs (negatives → 0)
- Softmax — usual final layer for multi-class; converts scores to probabilities that sum to 1
Architectures
- (ML)FFNN — (multi-layer) feed-forward network; information flows forward, trained by gradient descent + backpropagation.
- MLP (multilayer perceptron) — fully connected neurons with nonlinear activations in ≥3 layers; can separate data that isn’t linearly separable.
- CNN (convolutional) — at least one convolutional layer (plus pooling and dense layers); learns features via filter/kernel optimization; excels at image recognition.
Data preparation
Normalize inputs (e.g. MinMax scaler) and encode non-numeric columns (e.g. Label Encoder) before training.
Relationships
- Machine Learning — neural networks are one ML model family
- AI & Machine Learning — deep learning underpins modern AI
- Gradient Boosting — the leading alternative for tabular data
- Data Leakage — a training pitfall that inflates apparent accuracy