Naive implementation of perceptron neural network. Under heavy development.
- Implement 3 layer MLP network with SGD back-propagation algorithm
- Test coverage at least 80%
- Allow model export/import to json
- Prepare network learning examples and analysis for:
- Classification problem
- Regression problem
- Time series problem
- Data compression
- Use MLP network for MNIST dataset
- Implement various activation functions
- Tanh
- Softmax
- Softplus
- Gaussian
- Explore back-propagation algorithms:
- SGD with momentum
- ADAM
- Levenberg-Marquardt
- Add support for more than 1 hidden layer
- Create full documentation
Major inspiration for this work comes from book Machine Learning - An Algorithmic Perspective.
git clone https://github.com/stovorov/NaiveNeurals
cd NaiveNeuralsRequires Python 3.6
source set_env.sh # sets PYTHONPATH
make venv
source venv/bin/activate
make testIf you are using Ubuntu based system you must install tkinter
$ sudo apt-get install python3.6-tkfrom NaiveNeurals.MLP.network import NeuralNetwork
from NaiveNeurals.data.dataset import DataSet
nn = NeuralNetwork()
nn.setup_network(input_data_size=2, output_data_size=1,
hidden_layer_number_of_nodes=5)
# every list in inputs represents one network input and data pushed into network
inputs = [[0, 0, 1, 1], [1, 0, 1, 0]]
targets = [[1, 0, 0, 1]]
data_set = DataSet(inputs, targets)
nn.train(data_set)If convergence is not achieved, ConvergenceError is raised.
There are 2 categories of network configuration parameters:
-
Network architecture (number of nodes, weights, activation functions etc.)
-
Learning configuration (algorithm: CG, CG_MOM, learning rate, target error rate etc.)
from NaiveNeurals.MLP.network import NeuralNetwork, LearningConfiguration
from NaiveNeurals.MLP.activation_functions import Linear, Tanh
nn = NeuralNetwork()
# with LearningConfiguration one can set multiple parameters for solver algorithm
learning_configuration = LearningConfiguration(learning_rate=0.01,
target_error=0.003,
solver='GD_MOM',
max_epochs=1000,
solver_params={'alpha': 0.95})
nn.setup_network(input_data_size=1,
output_data_size=1,
hidden_layer_number_of_nodes=25,
hidden_layer_bias=1,
output_layer_bias=-0.7,
hidden_layer_act_func=Tanh(),
output_layer_act_func=Linear())
nn.set_learning_params(learning_configuration)In most cases it is recommended to split dataset into a few smaller subsets and validation set. In batch mode network will switch training sets every 50 epochs and check error rate with validation data.
from NaiveNeurals.MLP.network import NeuralNetwork
from NaiveNeurals.utils import ConvergenceError
train_data_set1 = ...
train_data_set2 = ...
train_data_set3 = ...
validation_set = ...
nn = NeuralNetwork()
try:
nn.train_with_validation([train_data_set1, train_data_set2, train_data_set3], validation_set)
except ConvergenceError:
passOnce model is trained it can be exported to dict and stored as json file using export_model method:
from NaiveNeurals.MLP.network import NeuralNetwork
import json
nn = NeuralNetwork()
nn.setup_network(input_data_size=2, output_data_size=1,
hidden_layer_number_of_nodes=5)
# training procedure ...
with open('test.json', 'w ') as fil:
fil.writelines(json.dumps(nn.export_model()))Model can be imported using load_model method:
from NaiveNeurals.MLP.network import NeuralNetwork
model_dict = ... # loaded model - Dict
nnn = NeuralNetwork()
nnn.load_model(model_dict)If this project got your attention you can read about details below:
Classification problem example
Machine Learning - An Algorithmic Perspective (2nd edition)
Activation function in neural networks