program a neural network

Question

So I have read a little bit about neural networks, and I hear stuff having to do with input, output, weights, activation functions,hidden layers,3 layer approach, and some calculus, but what does any of this have to do with programming an actual neural network. I am taking about if statements, while loops, classes, strings, arrays, hashing, and the sort. What does input, output, weights, activation functions, and calculus have to do with actual programming. Especially the calculus part. And how do you write code that can "train" a neural network / program. I am pretty sure I feel like I am asking beginner questions.

You are asking beginner questions. Why not read up about it? — Ed Heal, Jun 03 '14 at 19:08
I tried looking it up but all I get is the gibberish about weight, input, hidden layers, and no actual program. — kkpanu9, Jun 03 '14 at 19:12
You need programming to implement a neural network. But to describe what it is and what it does (to talk about it), it is better to use something more high-level like linear algebra and calculus. When talking about web pages, I say things like HTML, CSS, database; or do you want to say `while-loop` or `if-statement` when talking about web pages, because in the end that's what you need (and has been done by someone else, so you don't need to care about). — ziggystar, Jun 03 '14 at 22:02

score 5 · Answer 1 · answered Jun 03 '14 at 19:20

Firstly, sorry for my bad english.

A neural network contains (mostly) :

Neurons
These neurons are organised as a set of neurons : input Neurons, output Neurons and "cognitive" Neurons.
These neurons have connections to each other
Activating a neuron sends informations to neurons that are linked with it

In a programmation point of view, you can view this as :

Neurons are a class
The Neuron class contains an array of "connections" (references to other Neurons, sometimes with an associated random weight).
The Neuron class has a method like "recievePulse" which calculates if the Neuron activates of not.
The Neuron class has a method like "sendPulse" which activates the "recievePulse" of the Neurons it is linked to.

recievePulse is your activation function

weight is the strength of the pulse that each Neuron sends to Neurons it is linked to

This way you have a basic neural network.

How it works :

A few defined Neurons recieve pulses with different weights. These are the senses of the network, the "input". They may be activated and pulse too, depending on the input you gave them. The "output" is a set of defined Neurons. When you activate the input Neurons, some of the output Neurons may activate, simulating a response to the input.

Then, you can deactivate all Neurons, and start again with another input.

This way, you have a basic neural network reacting with what it senses.

Input/ouput example :

input 1 : amount of blue
input 2 : amount of green
input 3 : amount of red
output 1 : "i like this color"
output 2 : "i don't like this color"
between inputs and outputs a bunch of neurons (the "cognitive part").

This network will percieve a color and decide if it likes it or not.

Training:

There are several ways to train a neural network.

For example, you can reward the neurons when their answer is good (by rising the weight of all the active connections), and give them a bad reward when their answer is bad (by reducing the weight of connections).

The different weights of connections between each Neuron can be encoded in a sort of DNA which tells the structure of Neurons etc. This way you can select "winners" and "loosers" with a fitness function and reproduce the Brains who got the best answer between them. This way, you can select the best Brains and make them evolve to become super Geniuses.

For more infos about this i recommend you to document yourself on artificial life and fitness functions.

Interesting paper about your training example: http://maircrosoft.com/maircrosoft/papers/genetic.pdf — BlackBear, Jun 03 '14 at 19:39

score 1 · Answer 2 · answered Jun 03 '14 at 20:31

In addition to everything that Heru-Luin said, I will add the following:

Neural networks can be implemented in hardware or in software. The first neural networks were "implemented" by nature in the brains of primitive organisms, and refined through evolution into the brains of modern animals today.

When implementing them as software, the "program" one writes is actually the creation of a software model of a brain, which is a collection of neurons. This is where the if-then statements, the while- or for-loops, the object-orienting, etc, come into play. If one writes an actual program in the conventional sense, in C++ or Java, this is the program that is written.

Once that model is written, though, it is still mostly useless (like an infant brain) because it has not been trained. The training process involves adjusting various parameters of the software model-- neural weights, connections, etc. Sometimes this is also referred to as programming the neural network, too. Nor is it entirely incorrect: this is just as much a "program" as the programs on the earliest digital computers, which were basically wiring changes between hardware elements. Think of it as writing a software emulator of some hardware processing device, and them implementing an emulated program on that emulated hardware.

Where the calculus comes in is in understanding how and why the training methods work, how to implement them, and how to make them computationally efficient. A full answer is beyond the scope of this answer, and is usually an early chapter in a book on neural networks. The short answer is, "Because neural networks are trying to find the lowest point on an energy surface, and the calculus of derivations is a necessary step in that procedure."