Total views : 114
Application of ETLR in Telecom Domain
Objectives: To apply ETLR (Extraction, Transformation, Loading and Retrieval) paradigm to build an efficient, effective and cost effective data warehouse for telecom industry. The focus point is to optimize every layer of telecom DWH. Methods: The data techniques used are making use of telecom infrastructure, i.e. MSC files and applying segregation logic at the source layer i.e. mediation layer. Files are pushed towards predefined separate destinations and applying multiple technology mix mainly of database inbuilt utilities and custom scripts to avoid use of commercial ETL tools; and at the same time achieving enhanced performance at every front. Technology mix includes source optimization, external table implementation and switching, DB copy utility and retrieval level optimization. We have used data loading statistics to compare the results. Findings: The ultimate result is a telecom data warehouse and the result that we have achieved using ETLR paradigm improved the data processing of the data many folds. The motive is to optimize every layer that comes in between the data warehouse building process. Source level optimization leads at the data cleaning at the source level itself, thus shifting the load at the source system and reduced the load on the DWH servers. We have supplied bunch of files to the external tables and thus utilizing the OS storage for tabular data. Transforming data using views and push them into partitioned tables using DB copy utility improved the overall performance. Using query optimization techniques and DB level tuning ensures the data availability in minimum time. The data availability of a standard DWH is sysdate-1; but in our case, we have reduced it to approx. 4 hours with indexes intact. The scalability is also a very strong point of our ETLR paradigm. Now telecom operators have a better system available for building their data warehouse without taking care of heavy license fee for commercial tools. Application/Improvements: The application of the paradigm is in mostly every sector where data processing is a big challenge and cost is a major factor. We have given its application in telecom sector in this paper. The same can be implemented in Banking Sector, Insurance Sector, social media etc. and we can put it on cloud also in case hardware is a constraint.
Big-Data, ETL, Mobile Data, Retrieval, Scripts, Telecom Sector
- Sharma S, Kumar K. ETLR - Effective DWH Design Paradigm. Springer: Springer Science Business Media Singapore: Proceedings of the International Conference on Data Engineering and Communication Technology. 2016; p. 149–57.
- Mohajir BE, Mohajir ME. Telecom data warehouse prototype for bandwidth and network throughput monitoring and analysis. 2011.
- Agrawal H, Chafle G, Goyal S, Mittal S, Mukherjea S. An enhanced extract-transform-load system for migrating data in telecom billing. Proceedings - International Conference on Data Engineering. 2008. Crossref
- Li H, Zang M, Liang C. Research and Implementation of a Universal ETL Management Platform based on Telecom Industry. Computational Intelligence and Software Engineering. 2009. Crossref
- Anand N, Kumar M. Modeling and Optimization of Extraction-Transformation-Loading processes in Data Warehouse. 2013; p. 1–5.
- Yang P, Liu Z, Ni J. Performance tuning in distributed processing of ETL. Proceedings 7th International Conference on Internet Computing for Engineering and Science. 2013 Sep. Crossref
- Li X, Mao Y. Real-Time data ETL framework for big realtime data analysis. 2015.
- Mrunalini M, Kumar TVS, Kanth KR. Simulating secure data extraction in Extraction Transformation Loading (ETL) processes. UKSim 3rd European Modelling Symposium on Computer Modelling and Simulation. 2009.
- Tu Y, Guo C. An intelligent ETL workflow framework based on data partition. Intelligent Computing and Intelligent Systems. 2010; p. 358-63. PMid:19790034
- Simitsis A, Vassiliadis P, Sellis T. Optimizing ETL processes in data warehouses. International Conference on Data Engineering. 2005; p. 564-75. Crossref
- Simitsis A, Gupta C, Wang S, Dayal U. Partitioning realtime ETL workflows. International Conference on Data Engineering. 2010. Crossref
- Bala M, Boussaid O, Alimazighi Z. P-ETL Parallel-ETL based on the MapReduce paradigm. Computer Systems and Applications. 2014. Crossref
- Hamed I, Ghozzi F. A Knowledge-based approach for qualityaware ETL process. 2015.
- Radhakrishna V, Sravankiran V, Ravikiran K. Automating ETL process with scripting technology. 3rd Nirma University International Conference on Engineering. 2012.Crossref
- Wijaya R, Pudijoatmodjo. An Overview and Implementation of Process in Data Warehouse. 2015.
- Sun K, Lan Y. SETL A scalable and high performance ETL system. 3rd International Conference on System Science Engineering Design and Manufacturing Informatization.2012; p. 6–9. Crossref
- Qin H, Jin X, Zhang X. Research on extract, transform and Load in land and resources star schema data warehouse. 5th International Symposium on Computational Intelligence and Design. 2012; p. 120–3.
- Baidya A. Global Mobile Data Traffic 2016 - 2021 Asia Pacific Heads 10x Increase. Available from: http://dazeinfo.com/2016/01/07/global-mobile-data-traffic-2016-2021-4glte5g-apac-us/. Accessed date 01/08/2016.
- There are currently no refbacks.
This work is licensed under a Creative Commons Attribution 3.0 License.