Example calculation of entropy, Computer Engineering

Assignment Help:

Example Calculation:

If we see an example we are working with a set of examples like S = {s1,s2,s3,s4} categorised with a binary categorisation of positives and negatives like that s1  is positive and the rest are negative. Expect further there that we want to calculate the information gain of an attribute, A, and  A can take the values {v1,v2,v3} obviously. So lat in finally assume that as: 

1745_Example Calculation of Entropy.png

Whether to work out the information gain for A relative to S but we first use to calculate the entropy of S. Means that to use our formula for binary categorisations that we use to know the proportion of positives in S and the proportion of negatives. Thus these are given such as: p+ = 1/4 and p- = 3/4. So then we can calculate as: 

Entropy(S) = -(1/4)log2(1/4) -(3/4)log2(3/4) = -(1/4)(-2) -(3/4)(-0.415) = 0.5 + 0.311

= 0.811 

Now next here instantly note that there to do this calculation into your calculator that you may need to remember that as: log2(x) = ln(x)/ln(2), when ln(2) is the natural log of 2. Next, we need to calculate the weighted Entropy(Sv) for each value v = v1, v2, v3, v4, noting that the weighting involves multiplying by (|Svi|/|S|). Remember also that Sv  is the set of examples from S which have value v for attribute A. This means that:  Sv1 = {s4}, sv2={s1, s2}, sv3 = {s3}. 

We now have need to carry out these calculations: 

(|Sv1|/|S|) * Entropy(Sv1) = (1/4) * (-(0/1)log2(0/1) - (1/1)log2(1/1)) = (1/4)(-0 -

(1)log2(1)) = (1/4)(-0 -0) = 0 

(|Sv2|/|S|) * Entropy(Sv2) = (2/4) * (-(1/2)log2(1/2) - (1/2)log2(1/2))

                                      = (1/2) * (-(1/2)*(-1) - (1/2)*(-1)) = (1/2) * (1) = 1/2 

(|Sv3|/|S|) * Entropy(Sv3) = (1/4) * (-(0/1)log2(0/1) - (1/1)log2(1/1)) = (1/4)(-0 -

(1)log2(1)) = (1/4)(-0 -0) = 0 

Note that we have taken 0 log2(0) to be zero, which is standard. In our calculation,

we only required log2(1) = 0 and log2(1/2) =  -1. We now have to add these three values together and take the result from our calculation for Entropy(S) to give us the final result: 

Gain(S,A) = 0.811 - (0 + 1/2 + 0) = 0.311 

Now we look at how information gain can be utilising in practice in an algorithm to construct decision trees.


Related Discussions:- Example calculation of entropy

The components of i-way infrastructure, What are the components of I-way In...

What are the components of I-way Infrastructure? There are three mechanism of the I-way infrastructure: Consumer access equipment Local on-Ramps Global informa

Logistics planning, During the Persian Gulf crisis of 1991, U .S forces dep...

During the Persian Gulf crisis of 1991, U .S forces deployed a Dynamic Analysis and Re planning Tool, DART ( Cross and Walker, 1994) to do automated logistics planning and schedu

Advantage of using interrupt initiated data transfer, What is the advantage...

What is the advantage of using interrupt initiated data transfer over transfer under program control without interrupt? In the interrupt initiated data transfer, the processor

Which is the slowest sorting procedure, The slowest sorting procedure is   ...

The slowest sorting procedure is    Bubble Sort is the slowest sorting process.

Define point-to-point communication, Q. Define Point-to-point Communication...

Q. Define Point-to-point Communication? The simplest form of message is a point to point communication. A message is sent from the transmitting processor to a receiving process

Example multi-layer ann with sigmoid units, Example Multi-layer ANN with Si...

Example Multi-layer ANN with Sigmoid Units: However we will concern ourselves here that with ANNs containing only one hidden layer and as this makes describing the backpropaga

An 8086 interrupt, An 8086 interrupt can take placedue to the following rea...

An 8086 interrupt can take placedue to the following reasons: 1.  Hardware interrupts caused by some external hardware device. 2.  Software interrupts that can be invoked wit

Opcode field of an instruction, For this machine there can be two more poss...

For this machine there can be two more possible addressing modes in addition to direct andimmediate.   Opcode field of an instruction is a group of bits which define various pro

Write a short note on mmx technology, Write a short note on MMX technology....

Write a short note on MMX technology. MMX (Multimedia extensions) technology adds 57 new instructions to instruction set of the Pentium - 4 microprocessors. MMX technology also

Real life business subsystems, Real Life Business Subsystems A Subsyst...

Real Life Business Subsystems A Subsystem is a component of a System even though it can also be described as a system in its own right. Consider a manufacturing firm. It compr

Write Your Message!

Captcha
Free Assignment Quote

Assured A++ Grade

Get guaranteed satisfaction & time on delivery in every assignment order you paid with us! We ensure premium quality solution document along with free turntin report!

All rights reserved! Copyrights ©2019-2020 ExpertsMind IT Educational Pvt Ltd