Adding momentum - sigmoid units, Computer Engineering

Assignment Help:

Adding Momentum - Sigmoid units:

However imagine a ball rolling down a hill as it does so then it gains momentum in which its speed increases and it becomes more difficult to stop. Alternatively as it rolls down the hill towards the valley floor as the global minimum, then it might occasionally wander into local hollows. Moreover,, there it may be that the momentum it has obtained stays it rolling up and out of the hollow and back on track to the valley floor. 

Hence the crude analogy describes one heuristic technique for avoiding local minima that called adding momentum, funnily enough. Thus the method is simple as: now for each weight remember as the previous value of Δ that was added on to the weight in the last epoch. Rather then, where updating that weight for the current epoch, add on a little of the previous Δ. Now how small to make the additional extra is controlled through a parameter α that's called the momentum which is set to a value between 0 and 1. 

Alternatively to see why this might help bypass local minima so note there that if the weight change carries on in the direction it was going in the previous epoch and then the movement will be a little much more pronounced in the current epoch. Thus this effect will be compounded as the search continues in the same direction. Where the trend finally reverses so then the search may be at the global minimum case there it is hoped that the momentum won't be enough to take it anywhere other than where it is belongs. Conversely the search may be at a fairly narrow local minimum. So next there in this case, even though the back propagation algorithm which dictates Δ will change direction then it may be that the additional extra from the previous epoch as the momentum may be enough to  counteract this effect for a few steps. Then we can saythese few steps may be all that is utilised to bypass the local minimum.


Related Discussions:- Adding momentum - sigmoid units

Recurrence relation, Take the following recurrence relation consider only f...

Take the following recurrence relation consider only for n = 2k for integers k ≥ 1: T(2) = 9, and for n ≥ 4, T(n) = n + T(n /2). Three students were working together in a stu

Recursive procedures are implemented, Recursive procedures are implemented ...

Recursive procedures are implemented by using Recursive procedures are executed by using stacks due to stacks are LIFO data structure and we require this feature to store retur

How reassembling of ip fragments is advantageous, Reassembling of IP fragme...

Reassembling of IP fragments at the ultimate destination is advantageous. Give reasons. Needing the ultimate destination to reassemble the fragments has two major advantages. F

C++, find the minimum total number of shelves including the loading process...

find the minimum total number of shelves including the loading process

Explain i/o buffer and advantage of buffering, What is an I/O buffer? What ...

What is an I/O buffer? What is the advantage of buffering? Is buffering always effective? Justify your answer with help of an example.   One type of I/O requirement arises from

And-elimination-introduction rule - artificial intelligence, And-Eliminatio...

And-Elimination-Introduction rule - Artificial intelligence: And-Elimination In English, this says that "if you know that many things are all true, then you know that any

Power and energy, Ask questiPower and EnergyQuestion 4Consider a processor ...

Ask questiPower and EnergyQuestion 4Consider a processor that runs at 2.5 GHz and 1 Volt. When running a given CPU-bound program,the processor consumes 100 W, of which 20 W is leak

Why should i use xml, ? XML can be used to explain and recognize informatio...

? XML can be used to explain and recognize information accurately and unambiguously, in a way that computers can be programmed to? Understand? (Well, at least influence as if they

The mercantile process model, The mercantile process model consists of whic...

The mercantile process model consists of which of the pahase(s): The pre-purchase phase. Purchase consummation phase. Post-purchase Interaction phase.

What is the main use of structures, What is The main use of structures ...

What is The main use of structures The main use of structures is to lump together collections of disparate variable types, so they can conveniently be treated as a unit. For ex

Write Your Message!

Captcha
Free Assignment Quote

Assured A++ Grade

Get guaranteed satisfaction & time on delivery in every assignment order you paid with us! We ensure premium quality solution document along with free turntin report!

All rights reserved! Copyrights ©2019-2020 ExpertsMind IT Educational Pvt Ltd