Total views : 175

Integrated Hadoop Cloud Framework (IHCF)

Affiliations

  • Department of CSE, NIT Patna, Ashok Rajpath, Mahendru, Patna - 800005, Bihar, India

Abstract


Objective: The paper proposes the design of an intermediate Hadoop framework which will not only make Hadoop user friendly but also it is open source and free to use. Methods: The framework has been called Integrated Hadoop Cloud Framework (IHCF) and it also supports various Hadoop based frameworks like Hive, Pig as cloud services. It can be accessed from outside the Hadoop cluster too. Various experiment results has been included which show the efficient working of IHCF. Findings: The IHCF contains modules like Setup, Client and Cloud which are working in sync with one another. The setup module controls automated creation of cluster and client module provides users access to the cluster. The cloud module handles Hadoop based frameworks and ensures that user/client can use frameworks as cloud services. Improvement: The IHCF can be customized further to ensure optimized use of clusters and prevent over/under utilization of resources in cluster.

Keywords

Big Data, Cloud, Cluster, Hadoop, Hive, Pig.

Full Text:

 |  (PDF views: 113)

References


  • Laney D. 3D data management: Controlling data volume, velocity and variety. META Group Research Note. 2001; p. 70.
  • Kirk Borne. Top 10 Big Data Challenges – A Serious Look at 10 Big Data V’s. Available from: https://www.mapr.com/ blog/top-10-big-data-challenges-serious-look-10-big-datavs.
  • Shafer J, Rixner S, Cox AL. The Hadoop distributed file system: Balancing portability and performance. 2010 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). 2010; p. 122-33. Available from: Crossref. PMid:19563422.
  • Dean J, Ghemawat S. MapReduce: simplified data processing on large clusters. Communications of the ACM. 2008 Jan; 51(1):107-13. Available from: Crossref.
  • Kallman R, Kimura H, Natkins J, Pavlo A, Rasin A, Zdonik S, Jones EP, Madden S, Stone-Braker M, Zhang Y, Hugg J. H-store: a high-performance, distributed main memory transaction processing system. Proceedings of the VLDB Endowment. 2008; 1(2):1496-99. Available from: Crossref.
  • Cohen J, Dolan B, Dunlap M, Hellerstein JM, Welton C. MAD skills: new analysis practices for big data. Proceedings of the VLDB Endowment. 2009; 2(2):1481-92. Available from: Crossref.
  • Borthakur D, Gray J, Sarma JS, Muthukkaruppan K, Spiegelberg N, Kuang H, Ranganathan K, Molkov D, Menon A, Rash S, Schmidt R. Apache Hadoop goes realtime at Facebook. Proceedings of the 2011 ACM SIGMOD International Conference on Management of data. 2011 June; p. 1071-80. Available from: Crossref.
  • What is Hadoop? Date Accessed: 12/12/2016: Available from: http://www.sas.com/en_us/insight/big-data/ hadoop.html.
  • Hadoop-Big Data Overview. Date Accessed: 12/12/2016: Available from: http://www.tutorialspoint.com/hadoop/ hadoop_big_data_overview.htm.
  • Hadoop - Big Data Solutions. Date Accessed: 12/12/2016: Available from: http://www.tutorialspoint.com/hadoop/hadoop_ big_data_solutions.htm.
  • Hadoop - Introduction to Hadoop. Date Accessed: 12/12/2016: Available from: http://www.tutorialspoint.com/hadoop/hadoop_intro-duction.htm.
  • Top Ten Benefits of Cloud Computing Security Training. Date Accessed: 12/12/2016: Available from: http://www. simplilearn.com/cloud-computing-security-training-benefitsrar412-article.
  • Hive - Introduction. Date Accessed: 12/12/2016: Available from: http://www.tutorialspoint.com/hive/hive_introduction.htm.
  • Apache Pig Overview - Hadoop Online Tutorials. Date Accessed: 12/12/2016: Available from: http://hadooptutorial.info/apache-pig-overview.
  • Lammel R. Google’s MapReduce programming model Revisited. Science of Computer Programming. 2008 Jan; 70(1):1-30. Available from: Crossref.
  • Moreira JE, Michael MM, Da Silva D, Shiloach D, Dube P, Zhang L. Scalability of the Nutch search engine. Proceedings of the 21st annual international conference on Supercomputing. 2007; p. 3-12. Available from: Crossref.
  • Scalability. Date Accessed: 12/12/2016: Available from https://en.wikipedia.org/wiki/Scalability#Horizontal_and_ vertical_scaling.

Refbacks

  • There are currently no refbacks.


Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.