Reference no: EM133137009
ITECH2302 Big Data Management
Big Data Management Report
Purpose:
The assignment helps you grasp the fundamental concepts of big data management, related knowledge and the techniques, and practical software and tools which are required for developing big data projects.
Requirements: You are required to identify a suitable dataset, provide an analysis of the data, and recommend suitable Big Data Management strategies. This will be written up as a professional report.
Details
You will use the analytical tools taught on this course (including Jupyter notebooks, pySpark, Tableau) to explore, analyse and visualise a dataset of your choosing. An important part of this work is preparing a good quality report, which details your choices, analysis, and recommendations/conclusions. Also, that it is of an appropriate style.
The dataset should be chosen from the following repository:
UC Irvine Machine Learning Repository
The aim is to use the dataset allocated to provide interesting insights, trends and patterns amongst the data. Your intended audience is the CEO and middle management of the Company for whom you are employed, and who have tasked you with this analysis.
Tasks
Data choice. Choose any dataset from the repository that has at least five attributes, and for which the default task is classification. Transform this dataset into an appropriate one to load into your chosen analytics software.
Background information. Write a description of the dataset and project. Provide an overview of what the dataset is about, including from where and how it has been gathered, and for what purpose.
Data description. Describe how many instances does the dataset contain, how many attributes there are in the dataset, their names, and include which is the class attribute. Include in your description details of any missing values, and any other relevant characteristics. Use appropriate pandas functions to initially analyse the data, for instance descriptive statistics of each attribute, including description of the range of possible values of the attributes, and visualise these in a graphical format.
Initial analysis. You will need to make decisions about which features to include in your dataframe, and how to deal with missing values (if they exist). You might need preprocess the dataset attributes. Useful techniques will include remove certain attributes, exploring different ways of discretizing continuous attributes and replacing missing values. Discretizing is the conversion of numeric attributes into "nominal" ones by binning numeric values into intervals. If you replaced missing values explain what strategy you used to select a replacement of the missing values.
GroupBy analysis. Implement various aggregate functions that will provide interesting insights into the data. Use the GroupBy function in pandas to analyse the data.
Data visualisation. Choose any data visualisation techniques that will provide helpful insights into the data. This could include plotting chosen variables against each other, and displaying them in a linechart, or binning them and using a (stacked) histogram etc. Use whichever you prefer from either matplotlib (matplotlib.pyplot.hist), pandas (pandas.DataFrame.plot), seaborn (seaborn.histplot) and/or Tableau.
Data mining. Compare and contrast at least two different data mining algorithms on your data, for instance: SVN, neural networks, k-nearest neighbour, Apriori association rules, decision tree induction etc. For each experiment you run, describe the data you used for the experiments, that is, did you use the entire dataset of just a subset of it. You must include screenshots and results from the techniques you employ.
Discussion of findings. Explain your results and include the usefulness of the approaches for the purpose of the analysis. Include any assumptions that you may have made about the analysis. In this discussion you should explain what each algorithm provides to the overall analysis task. Summarize your main findings.
Big Data Management. The data you have used will have been very small in comparison with what might be considered "big data" in this course. In this section you are to draw conclusions about how the acquisition, storage, and subsequent analysis of the data would be different if this was truly a "big data" dataset. You are to make reference to the concepts learned about the "V's" of big data (velocity, volume.. etc), data warehouses, OLAP, business intelligence, HADOOP/Spark and so on. Explain how this dataset might have links to data that could be considered be too difficult or very complex to implement in a traditional SQL database, and traditional statistical analysis, and would therefore require Big Data storage and Big Data Analytics.
Report writing. Present your work in the form of a big data management report.
Attachment:- Big Data Management Report.rar