Anyone here good at neural nets? I’ve been studying some of the algorithms behind it, but seeing as I haven’t taken Calculus yet I’ve had to pick a lot of the concepts for this on the way through.
Anyway, I generate and put weights into the net, it provides an output in which I send to be backpropagated so that the weights can be corrected for maximum correctness.
Unfortunately, all it achieves is making the results worse…
Weights are generated, and 1 and 0 are inputted for an ideal output of 1. This is based on the XOR system for testing networks.
Here’s the code:
public class ANN {
private static final int HIDDEN_WEIGHT = 0;
private static final int HIDDEN_OLD_WEIGHT = 1;
private static final int HIDDEN_SUM = 2;
private static final int HIDDEN_OUT = 3;
private static final int OUTPUT_SUM = 0;
private static final int OUTPUT_OUT = 1;
float[][] weights;
float[][] weightHistory;
float[][] hlayers;
int neurons; //Rows
int hiddenLayers; //Columns
float idealOutput;
float learningRate = 0.8f;
float momentum = 0.6f;
public ANN(int neurons, int hiddenLayers, float idealOutput){
this.neurons = neurons;
this.hiddenLayers = hiddenLayers;
this.idealOutput = idealOutput;
weights = new float[neurons+1][hiddenLayers];
weightHistory = new float[neurons][hiddenLayers];
hlayers = new float[hiddenLayers+1][4];
//Randomize weights, can be overwritten
Random r = new Random();
for (int i = 0; i < neurons+1; i++){
for (int j = 0; j < hiddenLayers; j++){
weights[i][j] = r.nextFloat();
hlayers[j][HIDDEN_WEIGHT] = r.nextFloat();
}
}
hlayers[hiddenLayers][HIDDEN_WEIGHT] = r.nextFloat();
}
public void overwriteWeights(float[][] newWeights, float[] hweights){
weights = newWeights;
for (int a = 0; a < hlayers.length; a++){
hlayers[a][HIDDEN_WEIGHT] = hweights[a];
}
}
public float[] recall(float[] input){
float output[] = new float[2];
for (int j = 0; j < hiddenLayers; j++){
float sum = 0f;
//Apply weights to inputs
for (int i = 0; i < neurons; i++){
sum += weights[i][j] * input[i];
}
//Add bias
sum += weights[neurons][j];
//Send to hidden layer and apply sigmoid
hlayers[j][HIDDEN_SUM] = sum;
hlayers[j][HIDDEN_OUT] = sigmoidActivation(sum);
}
//Apply weights to outputs
for (int j = 0; j < hiddenLayers; j++){
output[OUTPUT_SUM] += hlayers[j][HIDDEN_OUT] * hlayers[j][HIDDEN_WEIGHT];
}
//Apply bias 2 and apply sigmoid
output[OUTPUT_SUM] += hlayers[hiddenLayers][OUTPUT_SUM];
output[OUTPUT_OUT] = sigmoidActivation(output[0]);
System.out.println("SUM: "+output[OUTPUT_SUM] + " OUTPUT: "+output[OUTPUT_OUT]);
backPropagate(output, input);
return output;
}
public void backPropagate(float[] output, float[] input){
float error = getError(output[OUTPUT_OUT], idealOutput);
float outDelta = getLayerDelta(output[0], error);
float[] gradient = new float[hiddenLayers];
System.out.println("Error is at "+error);
//Now we back propagate to the output
for (int j = 0; j < hiddenLayers; j++){
//float hiddenDelta = (sigmoidDerivative(hlayers[i][HIDDEN_SUM]) * hlayers[i][HIDDEN_WEIGHT]) * outDelta;
gradient[j] = outDelta * hlayers[j][HIDDEN_OUT];
float newWeight = (learningRate * gradient[j]) + (hlayers[j][HIDDEN_WEIGHT] * hlayers[j][HIDDEN_OLD_WEIGHT]);
hlayers[j][HIDDEN_OLD_WEIGHT] = hlayers[j][HIDDEN_WEIGHT];
hlayers[j][HIDDEN_WEIGHT] = newWeight;
}
for (int j = 0; j < hiddenLayers; j++){
for (int i = 0; i < neurons; i++){
float newWeight = (learningRate * gradient[j]) + (weights[i][j] * weightHistory[i][j]);
weightHistory[i][j] = weights[i][j];
weights[i][j] = newWeight;
}
}
}
public float getError(float output, float idealOutput){
return (float) output - idealOutput;
}
public float getLayerDelta(float sum, float error){
return -error * sigmoidDerivative(sum);
}
public float sigmoidActivation(float x){
return 1f / (float) (1f + Math.exp(-x));
}
public float sigmoidDerivative(float sum){
return (sigmoidActivation(sum) * (1f - sigmoidActivation(sum)));
}
}
Here’s the output when tested:
Commencing test...
Initializing network.
Overwriting weights for control variable.
Recalling [1.0, 0.0]
Test 0
SUM: 1.1265055 OUTPUT: 0.7551935
Error is at -0.24480653
Test 1
SUM: 0.7995221 OUTPUT: 0.68987226
Error is at -0.31012774
Test 2
SUM: 0.8105837 OUTPUT: 0.6922339
Error is at -0.30776608
Test 3
SUM: 0.80390143 OUTPUT: 0.6908084
Error is at -0.30919158
Test 4
SUM: 0.8040238 OUTPUT: 0.6908346
Error is at -0.30916542
Test 5
SUM: 0.803821 OUTPUT: 0.69079125
Error is at -0.30920875
Test 6
SUM: 0.80381775 OUTPUT: 0.69079053
Error is at -0.30920947
Test 7
SUM: 0.8038115 OUTPUT: 0.6907892
Error is at -0.30921078
Test 8
SUM: 0.8038112 OUTPUT: 0.6907891
Error is at -0.3092109
Test 9
SUM: 0.80381095 OUTPUT: 0.6907891
Error is at -0.3092109
Test 10
SUM: 0.80381095 OUTPUT: 0.6907891
Error is at -0.3092109
Test 11
SUM: 0.80381095 OUTPUT: 0.6907891
Error is at -0.3092109
Test 12
SUM: 0.80381095 OUTPUT: 0.6907891
Error is at -0.3092109
Test 13
SUM: 0.80381095 OUTPUT: 0.6907891
Error is at -0.3092109
Test 14
SUM: 0.80381095 OUTPUT: 0.6907891
Error is at -0.3092109
Test 15
SUM: 0.80381095 OUTPUT: 0.6907891
Error is at -0.3092109
Test 16
SUM: 0.80381095 OUTPUT: 0.6907891
Error is at -0.3092109
Test 17
SUM: 0.80381095 OUTPUT: 0.6907891
Error is at -0.3092109
Test 18
SUM: 0.80381095 OUTPUT: 0.6907891
Error is at -0.3092109
Test 19
SUM: 0.80381095 OUTPUT: 0.6907891
Error is at -0.3092109
The error is supposed to be negative in some cases, but I’m not so sure about if I did it right.
Please point out flaws, there’s bound to be a bunch because of how mathematically retarded I am when it comes to calculus.
Also, few questions,
- Can Neural Networks only learn one pattern for each network?
- What’s the point of having more than one set of hidden layer nodes?
- Where’s a good place to learn some basic Calculus fundamentals?
Thank you very much.
nobody want to read calculus books at home *.