Example calculation of entropy, Computer Engineering

Assignment Help:

Example Calculation:

If we see an example we are working with a set of examples like S = {s1,s2,s3,s4} categorised with a binary categorisation of positives and negatives like that s1  is positive and the rest are negative. Expect further there that we want to calculate the information gain of an attribute, A, and  A can take the values {v1,v2,v3} obviously. So lat in finally assume that as: 

1745_Example Calculation of Entropy.png

Whether to work out the information gain for A relative to S but we first use to calculate the entropy of S. Means that to use our formula for binary categorisations that we use to know the proportion of positives in S and the proportion of negatives. Thus these are given such as: p+ = 1/4 and p- = 3/4. So then we can calculate as: 

Entropy(S) = -(1/4)log2(1/4) -(3/4)log2(3/4) = -(1/4)(-2) -(3/4)(-0.415) = 0.5 + 0.311

= 0.811 

Now next here instantly note that there to do this calculation into your calculator that you may need to remember that as: log2(x) = ln(x)/ln(2), when ln(2) is the natural log of 2. Next, we need to calculate the weighted Entropy(Sv) for each value v = v1, v2, v3, v4, noting that the weighting involves multiplying by (|Svi|/|S|). Remember also that Sv  is the set of examples from S which have value v for attribute A. This means that:  Sv1 = {s4}, sv2={s1, s2}, sv3 = {s3}. 

We now have need to carry out these calculations: 

(|Sv1|/|S|) * Entropy(Sv1) = (1/4) * (-(0/1)log2(0/1) - (1/1)log2(1/1)) = (1/4)(-0 -

(1)log2(1)) = (1/4)(-0 -0) = 0 

(|Sv2|/|S|) * Entropy(Sv2) = (2/4) * (-(1/2)log2(1/2) - (1/2)log2(1/2))

                                      = (1/2) * (-(1/2)*(-1) - (1/2)*(-1)) = (1/2) * (1) = 1/2 

(|Sv3|/|S|) * Entropy(Sv3) = (1/4) * (-(0/1)log2(0/1) - (1/1)log2(1/1)) = (1/4)(-0 -

(1)log2(1)) = (1/4)(-0 -0) = 0 

Note that we have taken 0 log2(0) to be zero, which is standard. In our calculation,

we only required log2(1) = 0 and log2(1/2) =  -1. We now have to add these three values together and take the result from our calculation for Entropy(S) to give us the final result: 

Gain(S,A) = 0.811 - (0 + 1/2 + 0) = 0.311 

Now we look at how information gain can be utilising in practice in an algorithm to construct decision trees.


Related Discussions:- Example calculation of entropy

Message passing model, In the message-passing model, there exists a set of ...

In the message-passing model, there exists a set of tasks that use their own local memories during computation. Multiple tasks can reside on the similar physical machine as well ac

The concept of process-parallel computing, The Concept of Process Infor...

The Concept of Process Informally, a method is a program in execution, behind the program has been loaded in the main memory. However, a method is more than just a program code

What is mmx technology, What is MMX Technology MMX Technology: MMX (M...

What is MMX Technology MMX Technology: MMX (Multimedia extensions) technology adds 57 new instructions to instruction set of Pentium - 4 microprocessors. MMX technology also

Computer systems architecture, Assume that you are working in a software co...

Assume that you are working in a software company as a programmer and a bank is your company's client. The Bank is a most popular and one of the leading banks in Malaysia. Your

Learning algorithm for multi-layered networks, Learning algorithm for multi...

Learning algorithm for multi-layered networks: Furthermore details we see that  if S is too high, the contribution from w i * x i is reduced. It means that t(E) - o(E) is mu

Disk operating system, Q. Disk operating system? The operating system (...

Q. Disk operating system? The operating system (OS) is the first program that should be loaded into the memory of your PC before you can use it for any application. You can st

What are different types of verilog simulators, What are Different types of...

What are Different types of Verilog Simulators? There are essentially two types of simulators available. -  Event Driven -  Cycle Based

Difference between overlay graph and correlate graph, Overlay Graph: It ove...

Overlay Graph: It overlay the content of two graphs that shares an ordinary x-axis. Left Y-axis on the merged graph show's the present graph's value & Right Y-axis illustrate the v

Explain efficiency performance and issues in pipelining, Efficiency The...

Efficiency The effectiveness of pipeline can be measured the same as the ratio of busy time span to total time span counting the idle time. Let c be clock period of a pipeline

Write Your Message!

Captcha
Free Assignment Quote

Assured A++ Grade

Get guaranteed satisfaction & time on delivery in every assignment order you paid with us! We ensure premium quality solution document along with free turntin report!

All rights reserved! Copyrights ©2019-2020 ExpertsMind IT Educational Pvt Ltd