Example calculation of entropy, Computer Engineering

Assignment Help:

Example Calculation:

If we see an example we are working with a set of examples like S = {s1,s2,s3,s4} categorised with a binary categorisation of positives and negatives like that s1  is positive and the rest are negative. Expect further there that we want to calculate the information gain of an attribute, A, and  A can take the values {v1,v2,v3} obviously. So lat in finally assume that as: 

1745_Example Calculation of Entropy.png

Whether to work out the information gain for A relative to S but we first use to calculate the entropy of S. Means that to use our formula for binary categorisations that we use to know the proportion of positives in S and the proportion of negatives. Thus these are given such as: p+ = 1/4 and p- = 3/4. So then we can calculate as: 

Entropy(S) = -(1/4)log2(1/4) -(3/4)log2(3/4) = -(1/4)(-2) -(3/4)(-0.415) = 0.5 + 0.311

= 0.811 

Now next here instantly note that there to do this calculation into your calculator that you may need to remember that as: log2(x) = ln(x)/ln(2), when ln(2) is the natural log of 2. Next, we need to calculate the weighted Entropy(Sv) for each value v = v1, v2, v3, v4, noting that the weighting involves multiplying by (|Svi|/|S|). Remember also that Sv  is the set of examples from S which have value v for attribute A. This means that:  Sv1 = {s4}, sv2={s1, s2}, sv3 = {s3}. 

We now have need to carry out these calculations: 

(|Sv1|/|S|) * Entropy(Sv1) = (1/4) * (-(0/1)log2(0/1) - (1/1)log2(1/1)) = (1/4)(-0 -

(1)log2(1)) = (1/4)(-0 -0) = 0 

(|Sv2|/|S|) * Entropy(Sv2) = (2/4) * (-(1/2)log2(1/2) - (1/2)log2(1/2))

                                      = (1/2) * (-(1/2)*(-1) - (1/2)*(-1)) = (1/2) * (1) = 1/2 

(|Sv3|/|S|) * Entropy(Sv3) = (1/4) * (-(0/1)log2(0/1) - (1/1)log2(1/1)) = (1/4)(-0 -

(1)log2(1)) = (1/4)(-0 -0) = 0 

Note that we have taken 0 log2(0) to be zero, which is standard. In our calculation,

we only required log2(1) = 0 and log2(1/2) =  -1. We now have to add these three values together and take the result from our calculation for Entropy(S) to give us the final result: 

Gain(S,A) = 0.811 - (0 + 1/2 + 0) = 0.311 

Now we look at how information gain can be utilising in practice in an algorithm to construct decision trees.


Related Discussions:- Example calculation of entropy

Overfitted the data, Overfitted the data: Moreover notice that as time...

Overfitted the data: Moreover notice that as time permitting it is worth giving the training algorithm the benefit of the doubt as more as possible. However that is, the error

What is the function of cu, What is the function of CU? The control uni...

What is the function of CU? The control unit works as the nerve center that coordinates all the computer operations. It issues timing signals that governs the data transfer.

Example of shared programming using library routines, Q. Example of shared ...

Q. Example of shared programming using library routines? Think about subsequent set of statements  Process A                                  Process B                  :

Problem context and specification, Problem Context and Specification : ...

Problem Context and Specification : However the development of Inductive Logic Programming has been heavily formal in mathematical in nature it means the major people in the f

Design a nand-to-and gate network, Use as few gates as possible, design a N...

Use as few gates as possible, design a NAND-to-AND gate network that realize the following Boolean algebra expression. A'BC'D + ABC'D' + A'B'CD' + AB'C'D'

Explain the logic symbols - flip flops, Explain the Logic symbols - Flip Fl...

Explain the Logic symbols - Flip Flops? These flip-flops are as well called Master-Slave flip-flops simply because their internal construction is divided into two sections. Th

First-order models, First-Order Models: Here if we proposed first-orde...

First-Order Models: Here if we proposed first-order logic as a good knowledge representation language than propositional logic is just because there is more expressive than we

Operation codes used in assembly instructions, Q. Operation codes used in a...

Q. Operation codes used in assembly instructions? Now let's describe various operation codes needed for this machine so that we can translate High level language instructions t

Micro programmed control and hardwired control, Explain micro programmed co...

Explain micro programmed control. Ans: A micro programmed control unit is made around storage till is called a control store where all the control signals are stored in a prog

Write Your Message!

Captcha
Free Assignment Quote

Assured A++ Grade

Get guaranteed satisfaction & time on delivery in every assignment order you paid with us! We ensure premium quality solution document along with free turntin report!

All rights reserved! Copyrights ©2019-2020 ExpertsMind IT Educational Pvt Ltd