Example calculation of entropy, Computer Engineering

Assignment Help:

Example Calculation:

If we see an example we are working with a set of examples like S = {s1,s2,s3,s4} categorised with a binary categorisation of positives and negatives like that s1  is positive and the rest are negative. Expect further there that we want to calculate the information gain of an attribute, A, and  A can take the values {v1,v2,v3} obviously. So lat in finally assume that as: 

1745_Example Calculation of Entropy.png

Whether to work out the information gain for A relative to S but we first use to calculate the entropy of S. Means that to use our formula for binary categorisations that we use to know the proportion of positives in S and the proportion of negatives. Thus these are given such as: p+ = 1/4 and p- = 3/4. So then we can calculate as: 

Entropy(S) = -(1/4)log2(1/4) -(3/4)log2(3/4) = -(1/4)(-2) -(3/4)(-0.415) = 0.5 + 0.311

= 0.811 

Now next here instantly note that there to do this calculation into your calculator that you may need to remember that as: log2(x) = ln(x)/ln(2), when ln(2) is the natural log of 2. Next, we need to calculate the weighted Entropy(Sv) for each value v = v1, v2, v3, v4, noting that the weighting involves multiplying by (|Svi|/|S|). Remember also that Sv  is the set of examples from S which have value v for attribute A. This means that:  Sv1 = {s4}, sv2={s1, s2}, sv3 = {s3}. 

We now have need to carry out these calculations: 

(|Sv1|/|S|) * Entropy(Sv1) = (1/4) * (-(0/1)log2(0/1) - (1/1)log2(1/1)) = (1/4)(-0 -

(1)log2(1)) = (1/4)(-0 -0) = 0 

(|Sv2|/|S|) * Entropy(Sv2) = (2/4) * (-(1/2)log2(1/2) - (1/2)log2(1/2))

                                      = (1/2) * (-(1/2)*(-1) - (1/2)*(-1)) = (1/2) * (1) = 1/2 

(|Sv3|/|S|) * Entropy(Sv3) = (1/4) * (-(0/1)log2(0/1) - (1/1)log2(1/1)) = (1/4)(-0 -

(1)log2(1)) = (1/4)(-0 -0) = 0 

Note that we have taken 0 log2(0) to be zero, which is standard. In our calculation,

we only required log2(1) = 0 and log2(1/2) =  -1. We now have to add these three values together and take the result from our calculation for Entropy(S) to give us the final result: 

Gain(S,A) = 0.811 - (0 + 1/2 + 0) = 0.311 

Now we look at how information gain can be utilising in practice in an algorithm to construct decision trees.


Related Discussions:- Example calculation of entropy

Define bcd, Define BCD. A binary code that distinguishes between 10 ele...

Define BCD. A binary code that distinguishes between 10 elements must contain at least 4 bits, but 6 combinations will remain unassigned. Numerous dissimilar codes can be obtai

Truth tables, Any function can be expressed in a truth table.A truth table ...

Any function can be expressed in a truth table.A truth table lists all possible combinations ofinputs and gives the output produced in eachcase.Truth tables must include all combin

Define input-output interface, Define Input-Output Interface. I/O inter...

Define Input-Output Interface. I/O interface gives a method for transferring information among internal storage and external I/O devices

What is an interpreted languages, What is an interpreted languages In ...

What is an interpreted languages In interpreted languages, the instructions are implemented immediately after parsing. Both tasks are done by the interpreter. The code is sav

Virtual memory - computer architecture, Virtual memory: Virtual memory...

Virtual memory: Virtual memory is a technique which realized an application program the idea that it has contiguous working memory (an address space), whereas in fact it may b

Why is catch almost always a bad idea, Why is catch (Exception) almost alwa...

Why is catch (Exception) almost always a bad idea?  Well, if at that point you know that an error has happened, then why not write the proper code to handle that error instead

Explain the term - ancestors, Explain the term - ancestors The ancestor...

Explain the term - ancestors The ancestors of modern age computer were mechanical and electro-mechanical instruments. This ancestry can be traced as back and seventeenth centur

What is a heap, What is a heap? A complete binary tree, every of whose ...

What is a heap? A complete binary tree, every of whose elements contains a value that is  greater than or equal to the value of every of its children is known a Heap

Explain handle graphics in matlab, This is the MATLAB graphics system. It c...

This is the MATLAB graphics system. It contains high-level commands for two-dimensional and three-dimensional data visualization, image processing, animation, and presentation grap

Write Your Message!

Captcha
Free Assignment Quote

Assured A++ Grade

Get guaranteed satisfaction & time on delivery in every assignment order you paid with us! We ensure premium quality solution document along with free turntin report!

All rights reserved! Copyrights ©2019-2020 ExpertsMind IT Educational Pvt Ltd