Total views : 466
An Efficient Distributed Data Processing Method for Smart Environment
Background/Objectives: In current times, huge volume of data at a very high velocity gets generated through social media and various sensors in embedded systems that are connected to the Internet which causes a very Big data problem. These challenging Big data’s need to be processed and stored by traditional Relational Database Management Systems (RDBMS). Due to this reason the need for new software solutions has emerged for managing the Big data in an efficient, scalable and smart way. Methods/Statistical Analysis: In this study, an approach to combine the concept of batch processing and stream processing to an end where we can query the data set which also supports Adhoc Querying with less latency, that can be run on any Large scale Machine Learning Algorithms for recognizing any interest pattern in the streaming data set was employed. The functionalities of Hadoop ecosystem’s tool HIVE can also be used to produce the results to Adhoc queries, User Defined Functions (UDF) similar to writing a SQL Stored Procedures in the Spark System. An interface with SerDes which is Serialization and De-serialization that helps us to talk to the standard stream where we can exactly query the dataset are employed. Findings: : By proposing a new software solution AllJoyn Lambda, in which AllJoyn is integrated in the lambda architecture and the prototype implementation of the architecture is done using Apache Hadoop Yarn over Apache Spark Streaming are presented . This study illuminates the high velocity streaming data set on a database without losing any data from the streaming domain, to support Adhoc Querying from the data set and to provide a mechanism for fast data processing and analytics using Large Scale Machine Learning. This paper highlights the analysis of large scale dataset processing, handling challenges, and its comprehensive systematic review. Applications/Improvements: From this study, we conclude that, building a smart environment by using the big data setup platform improves and enhances the results for the smart environment.
AllJoyn Lambda Architecture, Big Data Analytics, Internet of Things, Smart Environment, Spark Streaming.
- Yanpei Chen, Sara Alspaugh, Randy Katz. Interactive Analytical Processing in Big Data Systems. A Cross Industry Study of MapReduce Workloads. 2010 August; 5(12):1802-1813.
- Jabez J, Muthu Kumar B. Intrusion Detection System: time probability method and hyperbolic hop field neural network. Journal of Theoretical and Applied Information Technology. 2014 September; 67(1):65-77.
- McKinsey. Big data: The next frontier for innovation, competition, and productivity. US: MGI. p. 1-156. 2011.
- Abhinandan Banik, Samir Kumar Bandyopadhyay. Big Data- A Review on Analyzing 3Vs. Journal of Scientific and Engineering Research. 2016; 3(1):21-24.
- Zaharia M, Chowdhury M, Das T, et.al. Resilient Distributed Datasets: A Fault-Tolerant Abstraction. In: Memory Cluster Computing. Proceedings of the 9th USENIX Symposium on (NSDI), USA; 2012. p. 15 -28.
- Nirmalrani V and Sakthivel P. (2015), A Hybrid Access Control Model with Multilevel Authentication and Delegation to Protect the Distributed Resources. Journal of Pure and Applied Microbiology (JPAM). 2015 November; 9(Spl. Edn. 2); 595-609. ISSN 0973-7510.
- Apache Spark SQL. http:// spark.apache.org/docs/latest/sql-programming-guide.html. Date accessed: 06/08/2015.
- Saravanan P., Sailakshmi P. Missing value imputation using fuzzy possibilistic c means optimized with support vector regression and genetic algorithm. Journal of Theoretical and Applied Information Technology (JATIT), 2015 February; 72(1):34-39. E-ISSN 1817-3195 (Online), ISSN 1992-8645 (Print)
- Aditya B. Patel, Manashvi Birla, Ushma Nair. Addressing Big Data Problem Using Hadoop and Map Reduce. Nirma university International Conference on Engineering (NUiCONE). 2012 Dec; 3(10):6-8.
- Fantacci R, Pecorella T, Viti R. and Carlini C. A network architecture solution for efficient IoT wsn backhauling: challenges and opportunities. Wireless Communications, IEEE. August 2014; 21(4): 113–119.
- Villari M, Celesti A, Fazio M, Puliafito A. Alljoyn lambda: An architecture for the management of smart environments in IoT. Proceedings of the (SMARTCOMP Workshops), Hong Kong; 2014 Nov. p. 9-14.
- Nirmalrani V, Sakthivel P. Framework for Providing Access to the Web Databases using Budget Aware Role Based Access Control. Journal of Theoretical and Applied Information Technology (JATIT). 2015 June; 76(3):296-308. E-ISSN 1817-3195 (Online), ISSN 1992-8645 (Print).
- Litwin W, Neimat M.-A, Schneider DA. LH* - A Scalable Distributed Data Structure. ACM Transactions on Data Base Systems. 1996 December; 21(4):480-52.
- There are currently no refbacks.
This work is licensed under a Creative Commons Attribution 3.0 License.