Adding momentum - sigmoid units, Computer Engineering

Assignment Help:

Adding Momentum - Sigmoid units:

However imagine a ball rolling down a hill as it does so then it gains momentum in which its speed increases and it becomes more difficult to stop. Alternatively as it rolls down the hill towards the valley floor as the global minimum, then it might occasionally wander into local hollows. Moreover,, there it may be that the momentum it has obtained stays it rolling up and out of the hollow and back on track to the valley floor. 

Hence the crude analogy describes one heuristic technique for avoiding local minima that called adding momentum, funnily enough. Thus the method is simple as: now for each weight remember as the previous value of Δ that was added on to the weight in the last epoch. Rather then, where updating that weight for the current epoch, add on a little of the previous Δ. Now how small to make the additional extra is controlled through a parameter α that's called the momentum which is set to a value between 0 and 1. 

Alternatively to see why this might help bypass local minima so note there that if the weight change carries on in the direction it was going in the previous epoch and then the movement will be a little much more pronounced in the current epoch. Thus this effect will be compounded as the search continues in the same direction. Where the trend finally reverses so then the search may be at the global minimum case there it is hoped that the momentum won't be enough to take it anywhere other than where it is belongs. Conversely the search may be at a fairly narrow local minimum. So next there in this case, even though the back propagation algorithm which dictates Δ will change direction then it may be that the additional extra from the previous epoch as the momentum may be enough to  counteract this effect for a few steps. Then we can saythese few steps may be all that is utilised to bypass the local minimum.


Related Discussions:- Adding momentum - sigmoid units

Some of the advantages of b2b, Some of the advantages of B2B are: i) En...

Some of the advantages of B2B are: i) Enhanced customer satisfaction Code ii) Enhanced inventory system iii) Simple and cost effective marketing iv) Coordination among

What are the different tones used in strowger telephony, What are the diffe...

What are the different tones used in strowger telephony? Explain with the help of waveforms and the timings. Dial Tone: it is used to indicate that the exchange is ready to r

Integrated services digital network service, Whole idea of ISDN is to digit...

Whole idea of ISDN is to digitize the telephone network to permit transmission of video, audio and text over existing telephone lines. The purpose of ISDN is to provide fully integ

Quantity amounts (data type quan), A field containing quantity amounts (dat...

A field containing quantity amounts (data type QUAN) must be assigned to a reference table and a reference field. Explain? As a reference table, a system table having all the v

What are the languages evolved in the html open standard, What are the lang...

What are the languages evolved in the html open standard Even the companies or developers themselves have been affected with such modifications to the open standards since they

Final implementation of object oriented modelling, Final Implementation of ...

Final Implementation of Object oriented modelling Final Implementation: At this phase, the final execution of classes and  relationships developed while object design takes

The goal of hashing, The goal of hashing is to produce a search that takes ...

The goal of hashing is to produce a search that takes   O(1) time

Computation step in time complexity of an algorithm, Q. Computation step in...

Q. Computation step in time complexity of an algorithm? So First in the computation step the local processor executes an arithmetic and logic operation. Afterwards the several

What do you mean by stack unwinding in c++, Stack unwinding in C++ is a pro...

Stack unwinding in C++ is a process during exception handling when the destructor is known for all local objects the place where the exception was thrown and where it is caught.

CPE, WHAT IS CPE

WHAT IS CPE

Write Your Message!

Captcha
Free Assignment Quote

Assured A++ Grade

Get guaranteed satisfaction & time on delivery in every assignment order you paid with us! We ensure premium quality solution document along with free turntin report!

All rights reserved! Copyrights ©2019-2020 ExpertsMind IT Educational Pvt Ltd