Reference no: EM132313022

Machine Learning Laboratory

Question 1: Classiftcation, Decision Trees, Na¨ive Bayes, k-NN, Weka

Consider the dataset postoperative-patient-data_simplified.arff available on moodle. This dataset contains health-status attributes of post-operative patients in a hospital, with the target class being whether the patients should be discharged (S) or remain in the hospital (A). Additional documentation regarding these attributes appears in the arff file.

1. Before you run the classifters, use the weka visualization tool to analyze the data, and report briefly on the types of the different variables and on the variables that appear to be important.

2. Run J48 (=C4.5, decision tree), Na¨ive Bayes and IBk (k-NN) to learn a model that predicts whether a patient should be discharged. Perform 10-fold cross validation, and analyze the results obtained by these algorithms as follows.

Note: Click on the "Choose" bar to select relevant parameters. Explanations of parameters you should try appear below. You should report on performance of at least two varia- tions of the operational parameters, e.g., minNumObj and unpruned for J48, and KNN and distanceWeighting for k-NN (the parameters debug and saveInstanceData are not operational). J48
• binarySplits: whether you use binary splits on nominal attributes when building the trees.
• minNumObj: the minimum number of instances per leaf.
• unpruned: whether pruning is performed (try TRUE and FALSE).
• debug: if set to TRUE, the classifier may output additional information.
• saveInstanceData: whether to save the training data for visualization.
Na¨ive Bayes (parameter variations are not relevant to this lab)
k- NN (IBk) (under lazy in weka)
• KNN: the number of neighbours to use.
• crossValidate: whether leave-one-out X-validation will be used to select the best k value between 1 and the value specified in the KNN parameter.
• distanceWeighting: specifies the distance weighting method used (when k > 1).
• debug: if set to TRUE, the classifier may output additional information. (a) J48 (=C4.5)
i. Examine the decision tree and indicate which are the main variables.
ii. What is the accuracy of the decision tree? Explain the results in the confusion matrix.

(b) Na¨ive Bayes

i. Explain the meaning of the "probability distributions" in the output, illustrating it with reference to the BP STBL attribute.

ii. Calculate (by hand) the probability that a person with the following attribute values would be discharged.
L-CORE = mid L-SURF = low L-O2 = good L-BP = high
SURF-STBL = stable CORE-STBL = stable BP-STBL = mod-stable

iii. What is the probability that a person with these attributes will remain in hospital and that s/he will be discharged? What would the Na¨ive Bayes classifier predict for this person?

iv. What is the accuracy of the Na¨ive Bayes classifier? Explain the results in the confusion matrix.

(c) k-NN
i. Find three instances in the dataset that are similar to the above patient, and use the Jaccard coefficient to calculate (by hand) the predicted outcome for this patient. Show your calculations.
ii. What is the accuracy of the k-NN classifier for different values of k? Explain the results in the confusion matrix.

3. Draw a table to compare the performance of J48, Na¨ive Bayes and IBk using the summary measures produced by weka. Which algorithm does better? Explain in terms of weka's summary measures. Can you speculate why?

Question 2: Classiftcation, Decision Trees, Na¨ive Bayes, k-NN, Weka

Consider the dataset tic-tac-toe.arff available on moodle. Each example in this dataset rep- resents a different game of tic-tac-toe, where the player writing crosses ("x") has the first move. Only those games that don't end in a draw are included, with the positive class representing the case where the first player wins and the negative class the case where the first player loses. The features encode the status of the game at the end, so each square contains a cross "x", a nought "o" or a blank "b".

1. Before you run the classifters, use the weka visualization tool to analyze the data.

(a) Which attributes seem to be the most predictive of winning or losing? (hint: if you were the "x" player, where would you put your first cross and why?)

(b) What can you infer about the advantage (or otherwise) of being the first player?

2. Run J48 (=C4.5, decision tree), Na¨ive Bayes and IBk (=k-NN) to learn a model that predicts whether the "x" player will win. Perform 10-fold cross validation, and analyze the results obtained by these algorithms as follows.

Note: When using IBk, click on the "Choose" bar to set the value of k (default is 1). Consider different values of k.

(a) J48 (=C4.5)

i. Examine the decision tree and indicate the main variables.

ii. Trace the decision tree for the following game. What would it predict?

iii. What is the first split in the decision tree? Calculate (by hand) the Information Gain obtained from the first split in the tree. Show your calculations.

iv. What is the accuracy of the decision tree? Explain the results in the confusion matrix.

(b) Na¨ive Bayes

i. Calculate (by hand) the predicted probability of a win for the following game. Show your calculations.

ii. What is the probability that a player with this configuration will win? What would the Na¨ive Bayes classifier predict for this game?

iii. What is the accuracy of the Na¨ive Bayes classifier? Explain the results in the confusion matrix.

(c) k-NN

i. Find three instances in the dataset that are similar to the following game, and use the Jaccard coefficient to calculate (by hand) the predicted outcome for this game. Show your calculations.

ii. What is the accuracy of the k-NN classifier? Explain the results in the confusion matrix.

3. Draw a table to compare the performance of J48, Na¨ive Bayes and IBk using the summary measures produced by weka. Which algorithm does better? Explain in terms of weka's summary measures. Can you speculate why?

Question 3: Regression

Consider the dataset abs.arff available on moodle. This dataset contains continuous-valued eco- nomic attributes of a country, with the target variable being the unemployment rate. Additional documentation regarding these attributes appears in the arff file.

1. Perform a linear regression (under functions in weka) to learn a linear model of the unemploy- ment rate as a function of the other variables. You can use the default parameters given in weka. What is the resultant regression function?

2. Using the resultant regression function, calculate by hand the Absolute Error for the year 1986.

3. Calculate the Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) obtained by the regression function (you can use the excel spreadsheet provided on moodle). How is MAE different from RMSE? (do these functions emphasize different aspects of performance?)