Reference no: EM133004338

COSC 2637 Big Data Processing

Overview
Write Spark programs which gives your chance to apply the essential components you learned in lectures and to understand the complexity of Spark programming.

Learning Outcome 1 - Model and implement efficient big data solutions for various application areas using appropriately selected algorithms and data structures.
Learning Outcome 2 - Analyse methods and algorithms, to compare and evaluate them with respect to time and space requirements and make appropriate design choices when solving real-world problems.
Learning Outcome 3 - Explain the Big Data Fundamentals, including the evolution of Big Data, the characteristics of Big Data and the challenges introduced.
Learning Outcome 4 - Apply non-relational databases, the techniques for storing and processing large volumes of structured and unstructured data, as well as streaming data.
Learning Outcome 5 - Apply the novel architectures and platforms introduced for Big data, i.e. Hadoop, MapReduce and Spark.

Task - Spark Streaming

Develop a spark streaming program with Scala Maven to monitor a folder in HDFS in real time such that any new file in the folder will be processed. The following three tasks are implemented in the same Scala object:

A. For each RDD of Dstream, count the word frequency and save the output in HDFS. Use regular expression to make sure that each word consists of characters only (tip: findAllIn()).

B. For each RDD of Dstream, filter out the short words (i.e., < 5characters) and then count the co-occurrence
frequency of words (the words are considered co-occurred if they are in the same line); save the output in HDFS.

C. For the Dstream, filter out the short words (i.e., < 5characters) and then count the co-occurrence frequency of words (the words are considered co-occurred if they are in the same line); save the output in HDFS. Note you are required to use updateStateByKey operation to continuously update the co- occurrence frequency of words with new information.

Format Requirements:

Failure to follow the requirements incur penalty
(a) The source codes are entailed in each Scala Maven project.
(b) You need create a single Scala Maven project for all three tasks.
(c) Submit the developed Scala Maven project in a single .zip file with a standalone jar file.
(d) The zip file should be named as sxxxxx_BDP_A3_2021.zip (replace sxxxxx bi student ID).
(e) You need include a "README" file in the zip file.
(f) In README, you must specify exactly how to run the standalone jar in AWS EMR platform to perform the tasks.
(g) Paths of input and output should not be hard-coded.
(h) Each task has its own output path.

Functional Requirements:
Failure to follow the requirements incur penalty
(i) For each task, the output file should be saved with the current time or a unique sequence number in HDFS.
(j) All three tasks are implemented in the same Scala object.

Attachment:- Big Data Processing.rar