Reference no: EM132718615
CN7031 Big Data Analytics - University of East London
Big Data Analytics: Coursework
(1) Understanding Dataset: CSE-CIC-IDS20181
This dataset was originally created by the University of New Brunswick for analyzing DDoS data. You can find the full dataset and its description here. The dataset itself was based on logs of the university's servers, which found various DoS attacks throughout the publicly available period to generate totally 80 attributes with 6.40GB size. We will use about 2.6GB of the data to process it with the restricted PCs to 4GB RAM. Download it from here. When writing machine learning or statistical analysis for this data, note that the Label column is arguably the most important portion of data, as it determines if the packets sent are malicious or not.
a) The features are described in the "IDS2018_Features.xlsx" file in Moodle page.
b) The labels are as follows:
• "Label": normal traffic
• "Benign": susceptible to DoS attack
c) In this coursework, we use more than 8.2-million records with the size of 2.6GB. As a big data specialist, firstly, we should read and understand the features, then apply modeling techniques. If you want to see a few records of this dataset, you can either use [1] Hadoop HDFS and Hive, [2] Spark SQL or [3] RDD for printing a few records for your understanding.
1 Source:
(2) Big Data Query & Analysis using Spark SQL
This task is using Spark SQL for converting big sized raw data into useful information. Each member of a group should implement 2 complex SQL queries (refer to the marking scheme). Apply appropriate visualization tools to present your findings numerically and graphically. Interpret shortly your findings.
(3) Advanced Analytics using PySpark
In this section, you will conduct advanced analytics using PySpark.
3.1. Analyze and Interpret Big Data using PySpark
Every member of a group should analyze data through 3 analytical methods (e.g., advanced descriptive statistics, correlation, hypothesis testing, density estimation, etc.). You need to present your work numerically and graphically. Apply tooltip text, legend, title, X-Y labels etc. accordingly.
Note: we need a working solution without system or logical error for the good/full mark.
3.2. Design and Build a Machine Learning (ML) technique
Every member of a group should go over and apply one ML technique. You can apply one the following approaches: Classification, Regression, Clustering, Dimensionality Reduction, Feature Extraction, Frequent Pattern mining or Optimization. Explain and evaluate your model and its results into the numerical and/or graphical representations.
Attachment:- Big Data Analytics.rar