About ANN and Levenberg Marquardt theory

7 views (last 30 days)
can anyone help me please.i am new in neural network i need to know how this Levenberg -Marquardtz parameters in matlab like this :
trainlm is a network training function that updates weight and bias values according to Levenberg-Marquardt optimization.
trainlm is often the fastest backpropagation algorithm in the toolbox, and is highly recommended as a first-choice supervised algorithm, although it does require more memory than other algorithms.
net.trainFcn = 'trainlm'
[net,tr] = train(net,...)
Training occurs according to trainlm's training parameters, shown here with their default values: net.trainParam.epochs 1000
Maximum number of epochs to train net.trainParam.goal 0
Performance goal net.trainParam.max_fail 6
Maximum validation failures net.trainParam.min_grad 1e-7
Minimum performance gradient net.trainParam.mu 0.001
Initial mu net.trainParam.mu_dec 0.1
mu decrease factor net.trainParam.mu_inc 10
mu increase factor net.trainParam.mu_max 1e10
Maximum mu net.trainParam.show 25
Epochs between displays (NaN for no displays) net.trainParam.showCommandLine 0
Generate command-line output net.trainParam.showWindow 1
Show training GUI net.trainParam.time inf
Maximum time to train in seconds
Validation vectors are used to stop training early if the network performance on the validation vectors fails to improve or remains the same for max_fail epochs in a row. Test vectors are used as a further check that the network is generalizing well, but do not have any effect on training.
trainlm is the default training function for several network creation functions including newcf, newdtdnn, newff, and newnarx.
  • are applied to Levenberg_marquardtz theory in matlab *
Like the quasi-Newton methods, the Levenberg-Marquardt algorithm was designed to approach second-order training speed without having to compute the Hessian matrix. When the performance function has the form of a sum of squares (as is typical in training feedforward networks), then the Hessian matrix can be approximated as
H = JTJ
and the gradient can be computed as
g = JTe
where J is the Jacobian matrix that contains first derivatives of the network errors with respect to the weights and biases, and e is a vector of network errors. The Jacobian matrix can be computed through a standard backpropagation technique (see [HaMe94]) that is much less complex than computing the Hessian matrix.
The Levenberg-Marquardt algorithm uses this approximation to the Hessian matrix in the following Newton-like update:
When the scalar µ is zero, this is just Newton's method, using the approximate Hessian matrix. When µ is large, this becomes gradient descent with a small step size. Newton's method is faster and more accurate near an error minimum, so the aim is to shift toward Newton's method as quickly as possible. Thus, µ is decreased after each successful step (reduction in performance function) and is increased only when a tentative step would increase the performance function. In this way, the performance function is always reduced at each iteration of the algorithm.
and where does this parameter is set up in above theroy?
sorry for my bad English. i hope Greg will understand this.

Accepted Answer

Greg Heath
Greg Heath on 17 Sep 2014
I have found that the default parameters work well about 99% of the time.
So, you really don't have to set any parameters unless your training is having a problem with the defaults.
I typically design 100 nets at a time. If training is slow (typically because of a large training set), I increase the MSEgoal and MinGrad to shorten training time
net.trainParam.goal = MSEgoal = 0.01*mean(var(target',1))
net.trainParam.min_grad = MinGrad = MSEgoal/100
When mse=MSEgoal, the net has successfully modeled 99% of the target variance. That is good enough for me.
For examples, search the NEWSGROUP and ANSWERS using
greg MSEgoal
Hope this helps.
Thank you for formally accepting my answer
Greg

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!