Reference no: EM132847516

TEB2164 Introduction to Data Science

Question 1.

a. Consider the structure of training data with 32 rows as shown in TABLE Q1 for a classification problem with four possible classes.

TABLE Q1: Structure of Training Data

Fill up appropriate headers for <attr1>, <attr2>, attr3> and <attr4> according to attributes of your own training data. Headers for ID and Class have been provided. Later, fill up data corresponding to the four attributes and the four-option Class. The type of data for <attr1> is binary, <attr2> is continuous, <attr3> is nominal, and <attr4> is ordinal. The values for each attribute must be diverse.

Based on the training data that have been furnished, compute the Entropy for the overall collection of the training data, the attribute with binary data, the attribute with continuous data using multiway split, the attribute with nominal data using multiway split, and the attribute with ordinal data using multiway split. The split breakdowns for attributes requiring multiway split must be clearly indicated. Lastly, suggest with justification which attribute in the training data is the most heterogenous.

b. Suppose that you have been hired by a digital news agency to summarize top-10 daily news on a specific vertical such as computing, medicine, finance, entertainment, or law in Malaysia. As a junior data scientist, suggest a complete text mining process that you will perform to achieve the goal.

Question 2. TABLE Q2 displays an unfilled temperature readings summary from ABC weather station in East Borneo comparing October, November, and December from 1990 till 1999 to that of from 2010 till 2019. The table should display the number of months in which the average maximum daily temperature was low (< 16°C), medium, or high (> 26°C). The investigation aims to discover whether a significant difference between the two rows exists.

TABLE Q2: Temperature Readings Summary

Firstly, furnish TABLE Q2 with data in the Low, Medium, and High columns. The data for 1990-1999 must be unique from that of 2010-2019.

Assuming that the readings are independent from month to month, let unknown parameters p_d,m be the probability that a month's reading goes to bin m ∈ {Low, Medium, High} in decade d ∈ {1990 - 1999, 2010 - 2019}. As a junior data scientist, you have been requested to (i) provide expressions for the maximum likelihood estimates ^{^}P_d,m, stating what to maximize and over which variables, (ii) establish a null hypothesis H₀ such that the probabilities are identical in both 1990-1999 and 2010-2019 and these probabilities are called q_k to provide the maximum likelihood estimates ^q_k under H₀, perform a test onto H0 using the test statistics given as

t = ∑  (^{^}P_d,m - ^{^}q_m)²/^q_m
     d,m

and (iii) considering parametric sampling to compute the distribution for t under H₀. Additionally, your tasks also include (iv) explaining the relevance of one- sided test vs two-sided test for this investigation, (v) providing pseudocode to compute the p-value for the H₀ test, and finally (vi) explaining an advantage and a disadvantage of a count-based test as opposed to a linear regression-based test.