Example calculation of entropy, Computer Engineering

Assignment Help:

Example Calculation:

If we see an example we are working with a set of examples like S = {s1,s2,s3,s4} categorised with a binary categorisation of positives and negatives like that s1  is positive and the rest are negative. Expect further there that we want to calculate the information gain of an attribute, A, and  A can take the values {v1,v2,v3} obviously. So lat in finally assume that as: 

1745_Example Calculation of Entropy.png

Whether to work out the information gain for A relative to S but we first use to calculate the entropy of S. Means that to use our formula for binary categorisations that we use to know the proportion of positives in S and the proportion of negatives. Thus these are given such as: p+ = 1/4 and p- = 3/4. So then we can calculate as: 

Entropy(S) = -(1/4)log2(1/4) -(3/4)log2(3/4) = -(1/4)(-2) -(3/4)(-0.415) = 0.5 + 0.311

= 0.811 

Now next here instantly note that there to do this calculation into your calculator that you may need to remember that as: log2(x) = ln(x)/ln(2), when ln(2) is the natural log of 2. Next, we need to calculate the weighted Entropy(Sv) for each value v = v1, v2, v3, v4, noting that the weighting involves multiplying by (|Svi|/|S|). Remember also that Sv  is the set of examples from S which have value v for attribute A. This means that:  Sv1 = {s4}, sv2={s1, s2}, sv3 = {s3}. 

We now have need to carry out these calculations: 

(|Sv1|/|S|) * Entropy(Sv1) = (1/4) * (-(0/1)log2(0/1) - (1/1)log2(1/1)) = (1/4)(-0 -

(1)log2(1)) = (1/4)(-0 -0) = 0 

(|Sv2|/|S|) * Entropy(Sv2) = (2/4) * (-(1/2)log2(1/2) - (1/2)log2(1/2))

                                      = (1/2) * (-(1/2)*(-1) - (1/2)*(-1)) = (1/2) * (1) = 1/2 

(|Sv3|/|S|) * Entropy(Sv3) = (1/4) * (-(0/1)log2(0/1) - (1/1)log2(1/1)) = (1/4)(-0 -

(1)log2(1)) = (1/4)(-0 -0) = 0 

Note that we have taken 0 log2(0) to be zero, which is standard. In our calculation,

we only required log2(1) = 0 and log2(1/2) =  -1. We now have to add these three values together and take the result from our calculation for Entropy(S) to give us the final result: 

Gain(S,A) = 0.811 - (0 + 1/2 + 0) = 0.311 

Now we look at how information gain can be utilising in practice in an algorithm to construct decision trees.


Related Discussions:- Example calculation of entropy

What is ram bus technology, What is RAM Bus technology? The key feature...

What is RAM Bus technology? The key feature of RAM bus technology is a fast signalling method used to transfer information among chips. Instead of using signals that have volta

How would you kill a process, How would you kill a process? The "kill" ...

How would you kill a process? The "kill" command takes the PID as one argument; this signifies which process to terminate. The PID of a process can be got using "ps" command

What are the main features of uml, What are the main features of UML ...

What are the main features of UML Defined system structure for the object modelling Support for all different model organization Strong modelling for behaviour an

Find out the number of control lines for 32 to 1 multiplexer, The number of...

The number of control lines for 32 to 1 multiplexer is ? Ans. For 32 (2 5 ) the number of control lines and to select one i/p between them total 5 select lines are needed.

Distinguish between http and true streaming, Question: 1. Distinguish ...

Question: 1. Distinguish between HTTP and True streaming, stating the advantages and disadvantages of both type of media streaming. 2. Name two online services which provi

Discuss the important characteristics of 8237, Question 1 Discuss the w...

Question 1 Discuss the working principle of an interrupt and hardware actions in interrupt handling 2 Discuss the important characteristics of 8237 3 Describe the format

Coding advantages of casex or casez, Coding advantages of casex or casez ...

Coding advantages of casex or casez By using casex or casez has the following coding advantages: -  It reduces number of lines, especially if the number of bits had been m

Describe the interpretation features of vba, VBA is licensed to Microsoft a...

VBA is licensed to Microsoft and this compatible with and only Microsoft products. Code written is compiled by an intermediate language known as P-code and this is stored in hostin

Implication connective - modus ponens rule, Implication connective - Modus ...

Implication connective - Modus ponens rule: We notice that this is a trivial example, so it highlights how we use truth tables: as the first line is the only one when both abo

Define the character set of c language, Define the Character Set of C Langu...

Define the Character Set of C Language? C uses the lower case letter a to z, the upper case letters A to Z, the digits 0 to 9 and certain characters as building blocks to form

Write Your Message!

Captcha
Free Assignment Quote

Assured A++ Grade

Get guaranteed satisfaction & time on delivery in every assignment order you paid with us! We ensure premium quality solution document along with free turntin report!

All rights reserved! Copyrights ©2019-2020 ExpertsMind IT Educational Pvt Ltd