Example calculation of entropy, Computer Engineering

Assignment Help:

Example Calculation:

If we see an example we are working with a set of examples like S = {s1,s2,s3,s4} categorised with a binary categorisation of positives and negatives like that s1  is positive and the rest are negative. Expect further there that we want to calculate the information gain of an attribute, A, and  A can take the values {v1,v2,v3} obviously. So lat in finally assume that as: 

1745_Example Calculation of Entropy.png

Whether to work out the information gain for A relative to S but we first use to calculate the entropy of S. Means that to use our formula for binary categorisations that we use to know the proportion of positives in S and the proportion of negatives. Thus these are given such as: p+ = 1/4 and p- = 3/4. So then we can calculate as: 

Entropy(S) = -(1/4)log2(1/4) -(3/4)log2(3/4) = -(1/4)(-2) -(3/4)(-0.415) = 0.5 + 0.311

= 0.811 

Now next here instantly note that there to do this calculation into your calculator that you may need to remember that as: log2(x) = ln(x)/ln(2), when ln(2) is the natural log of 2. Next, we need to calculate the weighted Entropy(Sv) for each value v = v1, v2, v3, v4, noting that the weighting involves multiplying by (|Svi|/|S|). Remember also that Sv  is the set of examples from S which have value v for attribute A. This means that:  Sv1 = {s4}, sv2={s1, s2}, sv3 = {s3}. 

We now have need to carry out these calculations: 

(|Sv1|/|S|) * Entropy(Sv1) = (1/4) * (-(0/1)log2(0/1) - (1/1)log2(1/1)) = (1/4)(-0 -

(1)log2(1)) = (1/4)(-0 -0) = 0 

(|Sv2|/|S|) * Entropy(Sv2) = (2/4) * (-(1/2)log2(1/2) - (1/2)log2(1/2))

                                      = (1/2) * (-(1/2)*(-1) - (1/2)*(-1)) = (1/2) * (1) = 1/2 

(|Sv3|/|S|) * Entropy(Sv3) = (1/4) * (-(0/1)log2(0/1) - (1/1)log2(1/1)) = (1/4)(-0 -

(1)log2(1)) = (1/4)(-0 -0) = 0 

Note that we have taken 0 log2(0) to be zero, which is standard. In our calculation,

we only required log2(1) = 0 and log2(1/2) =  -1. We now have to add these three values together and take the result from our calculation for Entropy(S) to give us the final result: 

Gain(S,A) = 0.811 - (0 + 1/2 + 0) = 0.311 

Now we look at how information gain can be utilising in practice in an algorithm to construct decision trees.


Related Discussions:- Example calculation of entropy

What is "at exit-command", What is "at exit-command:? The flow logic K...

What is "at exit-command:? The flow logic Keyword at EXIT-COMMAND is a special addition to the MODULE statement in the Flow Logic .AT EXIT-COMMAND lets you call a module befor

Meaning of convergence regarding e-commerce, What does the term convergence...

What does the term convergence mean regarding E-commerce? Convergence regarding e-commerce: The capability to leverage and integrate the different data sources and proces

How address symbol table is generated, Address symbol table is generated by...

Address symbol table is generated by the  (A) memory management software.  (B) assembler.  (C) match logic of associative memory.   (D) generated by operating system

Explain keyboard input and video output, Q. Explain Keyboard Input and Vide...

Q. Explain Keyboard Input and Video output ? A Keystroke read from keyboard is termed as a console input and a character displayed on the video screen is known as a console out

Explain about the modules of magento, Magento supports installation of modu...

Magento supports installation of modules by a web-based interface accessible by the administration area of a Magento installation. Modules are hosted on the Magento eCommerce websi

How is network examined by intranets, How is network examined by intranets,...

How is network examined by intranets, extranets and Internet? When more and more businesses seek to build their mission critical business solutions onto IP networks, networking

Name the 7400 series TTL chip, Name the 7400 series TTL chip which is a pri...

Name the 7400 series TTL chip which is a priority encoder. Write its truth table.  Ans. Available IC in 74 series is 74147 that is a priority encoder. Such I

Direct mapped cache - computer architecture, Direct Mapping: In this p...

Direct Mapping: In this particular technique, block j of the primary memory maps onto block j modulo 128 of the cache. The primary memory blocks 0,128,256,...is loaded

Explain increments and skips subsequent instruction, Q. Explain Increments ...

Q. Explain Increments and skips subsequent instruction? Increments A and skips subsequent instruction if the content of A has become 0. This is a complex instruction then requi

Principle of locality, Principle of locality: Temporal  base locali...

Principle of locality: Temporal  base locality (locality in time):- If an item is referenced then it will tend to be referenced again soon. Spatial locality (local

Write Your Message!

Captcha
Free Assignment Quote

Assured A++ Grade

Get guaranteed satisfaction & time on delivery in every assignment order you paid with us! We ensure premium quality solution document along with free turntin report!

All rights reserved! Copyrights ©2019-2020 ExpertsMind IT Educational Pvt Ltd