Example calculation of entropy, Computer Engineering

Assignment Help:

Example Calculation:

If we see an example we are working with a set of examples like S = {s1,s2,s3,s4} categorised with a binary categorisation of positives and negatives like that s1  is positive and the rest are negative. Expect further there that we want to calculate the information gain of an attribute, A, and  A can take the values {v1,v2,v3} obviously. So lat in finally assume that as: 

1745_Example Calculation of Entropy.png

Whether to work out the information gain for A relative to S but we first use to calculate the entropy of S. Means that to use our formula for binary categorisations that we use to know the proportion of positives in S and the proportion of negatives. Thus these are given such as: p+ = 1/4 and p- = 3/4. So then we can calculate as: 

Entropy(S) = -(1/4)log2(1/4) -(3/4)log2(3/4) = -(1/4)(-2) -(3/4)(-0.415) = 0.5 + 0.311

= 0.811 

Now next here instantly note that there to do this calculation into your calculator that you may need to remember that as: log2(x) = ln(x)/ln(2), when ln(2) is the natural log of 2. Next, we need to calculate the weighted Entropy(Sv) for each value v = v1, v2, v3, v4, noting that the weighting involves multiplying by (|Svi|/|S|). Remember also that Sv  is the set of examples from S which have value v for attribute A. This means that:  Sv1 = {s4}, sv2={s1, s2}, sv3 = {s3}. 

We now have need to carry out these calculations: 

(|Sv1|/|S|) * Entropy(Sv1) = (1/4) * (-(0/1)log2(0/1) - (1/1)log2(1/1)) = (1/4)(-0 -

(1)log2(1)) = (1/4)(-0 -0) = 0 

(|Sv2|/|S|) * Entropy(Sv2) = (2/4) * (-(1/2)log2(1/2) - (1/2)log2(1/2))

                                      = (1/2) * (-(1/2)*(-1) - (1/2)*(-1)) = (1/2) * (1) = 1/2 

(|Sv3|/|S|) * Entropy(Sv3) = (1/4) * (-(0/1)log2(0/1) - (1/1)log2(1/1)) = (1/4)(-0 -

(1)log2(1)) = (1/4)(-0 -0) = 0 

Note that we have taken 0 log2(0) to be zero, which is standard. In our calculation,

we only required log2(1) = 0 and log2(1/2) =  -1. We now have to add these three values together and take the result from our calculation for Entropy(S) to give us the final result: 

Gain(S,A) = 0.811 - (0 + 1/2 + 0) = 0.311 

Now we look at how information gain can be utilising in practice in an algorithm to construct decision trees.


Related Discussions:- Example calculation of entropy

What is a disk drive, What is a disk drive? The electro mechanical mech...

What is a disk drive? The electro mechanical mechanism that spins the disk and moves the read/write heads known as disk drive.

Explain ps-2 connector, PS/2 connector (PS/2 keyboards): These were int...

PS/2 connector (PS/2 keyboards): These were introduced with IBM's PS/2 computers and therefore are known as PS/2 connectors.  They have 6-pins but actually their wiring is just

Find the shortest path, The following is the required interface for the mou...

The following is the required interface for the mouse and cheese problem. Your program is required to read its input from a file named 'maze.txt' In the maze.txt

How to create a rollover image, Q. How to Create a Rollover Image? A ro...

Q. How to Create a Rollover Image? A rollover image is an image whose display changes when pointer passes ('rolls') over it. You will use Dreamweaver's Insert Rollover Image co

Highly encoded micro-instructions, Highly Encoded micro-instructions ...

Highly Encoded micro-instructions Encoded bits required in micro-instructions are small. It provided an aggregated view that is a higher view of CPU as just an encoded

Generic techniques developed - artificial intelligence, Generic Techniques ...

Generic Techniques Developed: In the pursuit of solutions to various problems in the above categories, various individual fundamental techniques have sprung up which have been

Gnome, Explain briefly about GNOME desktop

Explain briefly about GNOME desktop

Explain the working of a half subtractor, With the help of a truth table ex...

With the help of a truth table explain the working of a half subtractor. Draw the logic diagram using gates. Ans: Half Subtractor: For the subtraction of B (subtrahend) f

Inspirations of artificial intelligence, Inspirations of artificial intelli...

Inspirations of artificial intelligence: Artificial Intelligence research can be easily understood by following example in terms of how the following question has been answere

What is control store, What is control store? The microroutines for all...

What is control store? The microroutines for all the instructions in the instruction set of a computer are kept in a special memory known as the control store.

Write Your Message!

Captcha
Free Assignment Quote

Assured A++ Grade

Get guaranteed satisfaction & time on delivery in every assignment order you paid with us! We ensure premium quality solution document along with free turntin report!

All rights reserved! Copyrights ©2019-2020 ExpertsMind IT Educational Pvt Ltd