Develop a spark streaming program with scala maven

Assignment Help JAVA Programming

Reference no: EM133019750

Task - Spark Streaming

Develop a spark streaming program with Scala Maven to monitor a folder in HDFS in real time such that any new file in the folder will be processed. The following three tasks are implemented in the same Scala object:

A. For each RDD of Dstream, count the word frequency and save the output in HDFS. Use regular expression to make sure that each word consists of characters only (tip: findAllIn()).

B. For each RDD of Dstream, filter out the short words (i.e., < 5characters) and then count the co-occurrence frequency of words (the words are considered co-occurred if they are in the same line); save the output in HDFS.

C. For the Dstream, filter out the short words (i.e., < 5characters) and then count the co-occurrence frequency of words (the words are considered co-occurred if they are in the same line); save the output in HDFS. Note you are required to use updateStateByKey operation to continuously update the cooccurrence frequency of words with new information.

You should use Scala to develop your MapReduce program over AWS EMR

Reference no: EM133019750

Questions Cloud

What is the NPV of the new product line : The product will generate free cash flow of $750,000 the first year, What is the NPV of the new product line (including any tax shields from leverage)

Prepare necessary adjusting entries on December : Office supplies on hand at the end of December 31, 2019 amounted to P 5,000. Prepare necessary adjusting entries on December

Make stockholders equity section of year-end balance sheet : Repurchased 1,700 shares of its own common stock for $19 per share. Prepare the stockholders' equity section of the year-end balance sheet

Prepare the journal entries required : During the 2019 - 2010 financial year, Canola Ltd sold inventory to Palm Ltd for $1,000,000. Prepare the journal entries required

Develop a spark streaming program with scala maven : Develop a spark streaming program with Scala Maven to monitor a folder in HDFS in real time such that any new file in the folder will be processed

Why do subordinates resist delegation : "So why do subordinates resist delegation?" Examine any FIVE reasons whysubordinates resist delegation

What is Pelamed net income : In addition, Pelamed has interest expenses of $122 million and a corporate tax rate of 21%. What is Pelamed's 2006 net income

Prepare a schedule for each month showing budgeted cash : The accounts payable balance on March 31 totals $220,000, which will be paid in April. Prepare a schedule for each month showing budgeted cash

Training and development increase employability for workers : How does workplace training and development increase employability for workers?

User Account

All Pages