Reference no: EM132595277

Data Application

Development Repeat Project

Task
Programmatically analyse and interrogate 2 (or more) datasets. Your data sets should fulfil the following requirements:
1. Be related in some way.
2. Complement each other such that your study (or something very similar) could not be conducted without one of your datasets.
3. Be at least moderately sized - whilst there is no upper limit on size be realistic with respect to the capabilities of your hardware.

In terms of what to do with your datasets, please observe the following minimum requirements:
1. Programmatically prepare your datasets this includes:
a. Extracting at least one from the web via an API, web crawler, HTML parser or similar (placing them potentially in DB storage or similar). It is noted that some datasets may sit behind a login (e.g. Kaggle) and can only be downloaded through a browser.
b. Clean them, deal with missing values.
c. Conform/transform and combine the datasets
d. Providing at least a cursory exploratory study to motivate your project focus and formally describe the data
e. Prepare the data for at least one MapReduce-based analysis
2. Perform analysis using MapReduce
3. Interrogate the combined dataset and MapReduce results to provide at least 3 interesting insights into the data you have chosen.

Your project MUST follow either the KDD or CRISP-DM methodology.

Structure of the Report

All deliverables should be compiled into an accompanying paper, which should be submitted along with any programming code elements.
Your project report should discuss the challenges that you encountered whilst handling your chosen datasets and the means and mechanisms you implemented to overcome these challenges.

The project report should be structured as follows:

• Abstract: a roughly 200-word executive summary of the project and the key results

• Introduction: set the scene of the project, i.e., the objectives of the project (for example what are you trying to find out)

• Literature Review: discuss how other people have used the data sets you have chosen or conducted similar topics or datasets. The literature review should be around half a page (one column in the IEEE).

• Data: present the data sets chosen, and why. Recommended here is to use tables to summarise the main characteristics of your datasets (e.g., format, size, number of instances, number of attributes and their type, etc.). Make sure to include references to the source of datasets (as proper references or footnotes not URLs in the text). Include also an exploratory analysis of the datasets.

• Methodology: essentially, provide a step-by-step description of how you have applied KDD or CRISP-DM to your project.

• Implementation and Architecture: how have you built your application workflow, how automatic it is, what technologies, components and/or forms of analytics have you used and why? Recommended here is to create a visual diagram showing the application workflow / architecture.

• Results: what did you find out about your data sets? E.g.: what was surprising? what was expected? what did you find out with respect to your motivational question that is presented in the introduction? Finally discuss any interesting aspects of your results or key challenges you solved in achieving your results.

• Conclusions and future work: what (in general) did you learn and find out? If you were to do the project again, what would you do differently? If you had more time (e.g. in your final project) what would you do next to extend your work?

• References: a complete list of academic works and/or online materials used in the project. References should include in-text citations and be properly formatted according to the IEEE referencing style. Consult the NCI Library Referencing Guide.

Attachment:- Development Repeat Project.rar