Reference no: EM132989202
COSC 2637 Big Data Processing - RMIT University
Overview
Write advanced MapReduce programs which give your chance to develop in-depth understanding of principles when solving complex problems on Hadoop execution platform, and analyze solutions by applying the knowledge learned in this course to achieve the optimal outcome.
Learning Outcome 1: model and implement efficient big data solutions for various application areas using appropriately selected algorithms and data structures.
Learning Outcome 2: analyse methods and algorithms, to compare and evaluate them with respect to time and space requirements and make appropriate design choices when solving real-world problems.
Learning Outcome 3: motivate and explain trade-offs in big data processing technique design and analysis in written and oral form.
Learning Outcome 4: explain the Big Data Fundamentals, including the evolution of Big Data, the characteristics of Big Data and the challenges introduced.
Learning Outcome 6: apply the novel architectures and platforms introduced for Big data, i.e. Hadoop, MapReduce and Spark.
Assessment details
Task 1 - Count word co-occurrence frequency
Write a MapReduce program that uses pairs approach and outputs the frequency of word pairs.
- Given "(a, b)" and word pair "(b, a)", they are considered as different word pairs,
- Do not output count the pair of same words, e.g., "(a, a)",
- The words are considered co-occurred if they are in the same line and the number of words between them <=3.
Task 2 - Count word pair relative frequency
Write a MapReduce program that uses pairs approach and outputs the relative frequency of word pairs.
- Given "(a, b)" and word pair "(b, a)", they are considered as different word pairs,
- Do not output count the pair of same words, e.g., "(a, a)",
- The words are considered co-occurred if they are in the same line and the number of words between them <=3.
Task 3 - Implement PAM algorithm with a MapReduce Program
The most common realization of k-medoid clustering is the Partitioning Around Medoids (PAM) algorithm which is described below:
step 1. Initialize: randomly select ???? of the ???? data points as the medoids
step 2. Assignment step: Associate each data point to the closest medoid.
step 3. Update step: For each medoid m and each data point ???? associated to ???? , swap ???? and ????, and compute the total cost of the configuration (that is, the average dissimilarity of ???? to all the data points associated
to ????). Select the medoid o with the lowest cost of the configuration.
step 4. Repeat alternating steps 2 and 3 until there is no change in the assignments.
(a) Your program must correct implement PAM. In your code, provide detailed comments to specify where each step is implemented. For example
//Step 2 start.
...
Block of code;
...
//Step 2 end.
Run your PAM MapReduce program to cluster a point dataset NYTaxiLC1000 1 (with 1000 points in longitude and latitude from line 1 to line 1000) where 1 ≤ ???? ≤ 6. Note the initial medoids are always points at line 100, 200, 300, 400, 500 and 600 (i.e., ???? = 1, the initial medoid is point at line 100; ???? = 2, the initial
medoids are points at line 100, 200; and so on for k=3, 4, 5 and 6).
(b) Visualize the clustering results. The points belonging to the same cluster are with the same color. The medoid of each cluster is highlighted.
(c) Analyse what is the best setting of ???? (3 ≤ ???? ≤ 6) and explain why.
Attachment:- Big Data Processing.rar