Reference no: EM132900933

Big Data Analytics Assessment

Overview

Write Python code (especially PySpark where possible) to implement the tasks below (Please explain your code in detail in your report). Where appropriate, the output of code execution should be presented that relates to your answer to the tasks into your report, as the evidence of working program (e.g., screenshots). Output included without supporting explanation or interpretation will not receive credit.

Where the tasks involve big data analytics with machine learning, you may need to formulate technical solution step by step, e.g., choose appropriate machine learning models / algorithms, justifying appropriateness of models/algorithms/techniques used, applying them in the context of the given task, and practising data visualisation techniques where appropriate. Test your solution and conduct experiments where appropriate. Evaluate the performance of implemented solution and analyse results where appropriate. Delve into deep technical explanations of the results and suggest possible improvement, etc. Make conclusions where appropriate.

You need to present your solution to the tasks into a technical report. The report should be submitted onto Blackboard.

Assignment tasks

Part
You will need to download the data file "Medical_info.csv" from the Blackboard. The dataset is about a study of factors predicting serum testosterone levels in a screened population in some country. The key to the variables in the dataset is explained in the table below. Please note there are null or missing values in some columns of the dataset.

1. Load the data file into a Spark DataFrame (1st DataFrame). Describe the structure of the DataFrame.

2. Create a new DataFrame (2nd DataFrame) by removing all the rows with null/missing values in the 1st DataFrame and calculate the number of rows removed.

3. Calculate summary statistics of the ‘age' feature in the 2nd DataFrame, including its min value, max value, mean value, median value, variance and standard deviation. Generate a histogram for the ‘age' feature and describe the distribution of the feature.

4. Display the quartile info of the ‘BMI' feature in the 2nd DataFrame. Generate a boxplot for the ‘BMI' feature and discuss the distribution of the feature based on the boxplot.

5. Use Spark DataFrame API (i.e., expression methods) to count the number of rows where ‘age' is greater than 50 and ‘BP_1' equals 1.

6. Use the ‘BP_1' feature in the 2nd DataFrame as the target label, to build two classification models based on all other columns as predictors. Conduct performance evaluation for the two models and make conclusions.

Part (II)

You will need to download the data file "Region_info.csv" from the Blackboard. The dataset provides information such as population size, average life expectancy, GDP per capita, and so on for some regions. The key to the variables in the dataset is explained in the table below.

1. Load the data file into a Spark DataFrame (1st DataFrame). Describe the structure of the created data frame.

2. Create a new DataFrame (2nd DataFrame) by removing the ‘region' column.

3. Use a graph, explore and describe the relationship between ‘fertility' feature and ‘life' feature in the 2nd DataFrame.

4. Use Spark SQL query to display the ‘fertility' and ‘life' columns in the 2nd DataFrame where ‘fertility' is great than 1.0 and ‘life' is greater than 70.

5. Build a linear regression model to predict life expectancy (the ‘life' column) in the 2nd DataFrame using the ‘fertility' column as the predictor. Conduct performance evaluation for the model and make conclusions.

6. Build a Lasso regression model to predict life expectancy (the ‘life column) in the 2nd DataFrame using all other columns as the predictor. Conduct performance evaluation for the model and make conclusions.

Attachment:- Big Data Analytics Assessment.rar