Reference no: EM132218440 , Length: word count : 750

Assignment: Big data systems enterprise deployment, integration, scalability and security issues

In view of the need to handle increasingly larger amounts of data and to better cope with the ensuing big data V-characteristics (Volume, Velocity, Variety, Veracity etc.), several large organisations have begun migrating their standard legacy Enterprise Data Warehouse (EDW) systems to Big-Data-Driven-Enterprise (BDDE) schemes, which subsume the EDW.

Many of these schemes utilise the Hadoop ecosystem supported by a variety of additional data mining and business analytics sub-systems. In this Assignment, you will look at how big organisations migrate from standard EDW to BDDE schemes (that incorporate the EDW) and will investigate the key issues that often have to be dealt with in such migrations.

To complete this Assignment:

Complete out the following tasks:

1. Describe a real or an imaginary company that you want to transform to the BDDE type (you can also use your current employment company as an example, if applicable). Select the industry sector and your particular business activity. Your company's operations must include a Web presence-you must define what service this Web presence provides to its customers and additionally explain how data will be used and collected.

2. Describe data flows in your company. Identify where you will use data collected from the Web and other sources of information such as sell statistics, user activity or social media data. Describe how you plan to integrate new types of big data with current EDW data workflows that primarily use relational data. Do not forget about operational aspects such as data backups and, if applicable, long-term data storage. Data provenance may be needed in addition to regular activity and communication logs.

3. Select a suitable platform for general big data management (e.g. in-house or cloud based infrastructure) and Big Data Management System (BDMS) platform with vendor (e.g. Hadoop supplied by Cloudera CDH, AWS EMR or HortonWorks). You should provide sufficient detail about the BDMS components and tools that you intend to use within the BDDE scheme.

4. Provide suggestions for how you will address security and privacy issues when managing your customer data, and your company's data. Again, do not forget about regular backups and secure backup storage for the BDDE.

5. Define a data management policy, including data protection and access control. Briefly address a majority of the CSA top ten security and privacy challenges.

6. Suggest what big data analytics and visualization methods you will use, including specific commercial tools and platforms.

• Submit a report containing your responses for tasks 1 - 6 written up as an "EDW-to-BDDE Migration Plan" for the specified company.

• Your report should address the issues raised in a reasonably concise and practical manner.

For all Assignments (unless stated otherwise):

Your document should have 750-1,000 words (not including the list of works cited), but it is the quality of the answer that matters, not the number of words.