Reference no: EM132542906
ICT707 Data Science Practice - University of the Sunshine Coast
Data Science Practice Assignment -
Assignment Task - This assignment consists of two deliverables, being:
- One code implementation - The code file in Jupyter Notebook format and the relevant date set files.
- A report - The report must be uploaded as a separate file.
Part I - PySpark source code
Important Note: For code reproduction, your code must be self-contained. That is, it should not require other libraries besides PySpark environment we have used in the workshops. The data files are packaged properly with your code file.
In this component, we need to utilise Python 3 and PySpark to complete the following data analysis tasks:
1. Exploratory data analysis
2. Recommendation engine
3. Classification
You need to choose a dataset from Kaggle to complete these tasks. Remember to include the data set file in you source code submission.
Note: In your notebook, please use Heading 1 Markdown cell to separate each sub task.
Task I.1: Exploratory data analysis
This subtask requires you to explore your dataset by
- telling its number of rows and columns,
- doing the data cleaning (missing values or duplicated records) if necessary
- selecting 3 columns, and drawing 1 plot (e.g. bar chart, histogram, boxplot, etc.) for each to summarise it
Task I.2: Recommendation engine
This subtask requires you to implement a recommender system on Collaborative filtering with Alternative Least Squares Algorithm. You need to include
- Model training and predictions
- Model evaluation using MSE
Task I.3: Classification
This subtask requires you to implement a classification system on Logistic regression. You need to include
- Logistic Regression model training
- Model evaluation
Part II -Report
You are required to write a report with the following content:
• Provide a high-level survey on the advances of data science in the past 2 years.
• Explain how Spark fits into the field of data science. Compare Spark with its competitors.
• Explain your design and implementation of the machine learning parts in your code, including the following topics:
o Background of your selected data set
o For each task, which learning algorithm is used and what are its key parameters and how you set them up
o For each task, provide comments/evaluation for the model learnt
Your report should use the following template:
Table of Contents
1.0 Advancement of Data Science (550 words)
2.0 Spark in Data Science (200 words)
Machine Learning Implementation (250 words)
Data set
Collaborative filtering
Features of the model, key parameters and configuration
Evaluation
Logistic regression
Features of the model, key parameters and configuration
Evaluation
References
Attachment:- Data Science Practice.rar