Unleashing real-time analytics: A comparative study of in-memory computing vs. traditional disk-based systems
DOI:
https://doi.org/10.14295/bjs.v3i5.553Keywords:
in-memory computing, real-time analytics, data processing efficiency, redis, postgresqlAbstract
The article presents a comprehensive study evaluating the performance differences between in-memory computing (IMC) and traditional disk-based database systems, specifically focusing on Redis and PostgreSQL. Given the escalating demands for real-time data analytics across various sectors, the research delves into the comparative efficiency of these two data management paradigms in processing large datasets. Utilizing a synthetic dataset of 23.6 million records, we orchestrated a series of data manipulation tasks, including aggregation, table joins, and filtering operations, to simulate real-world data analytics scenarios. The experiment, conducted on a high-performance computing setup, revealed that Redis significantly outperformed PostgreSQL in all tested operations, showcasing the inherent advantages of IMC in terms of speed and efficiency. Data aggregation tasks saw Redis completing the process up to ten times faster than PostgreSQL. Similarly, table joining, and data filtering tasks were executed more swiftly on Redis, emphasizing IMC's potential to facilitate instantaneous data analytics. These findings underscore the pivotal role of IMC technologies like Redis in empowering organizations to harness real-time insights from big data, a critical capability in today's fast-paced business environment. The study further discusses the implications of adopting IMC over traditional systems, considering aspects such as cost, integration challenges, and the importance of skill development for IT teams. Concluding with strategic recommendations, the article advocates for a nuanced approach to incorporating IMC technologies, highlighting their transformative potential while acknowledging the need for balanced investment and operational planning.
References
Al-Mohannadi, A., Al-Maadeed, S., Elharrouss, O., & Sadasivuni, K. K. (2021). Encoder-decoder architecture for ultrasound IMC segmentation and cIMT measurement. Sensors, 21(20), 6839. https://doi.org/10.3390/s21206839 DOI: https://doi.org/10.3390/s21206839
Amrouch, H., Du, N., Gebregiorgis, A., Hamdioui, S., & Polian, I. (2021). Towards reliable in-memory computing: From emerging devices to post-von-neumann architectures. In: 2021 IFIP/IEEE 29th International Conference on Very Large Scale Integration (VLSI-SoC), IEEE, Singapore, 1-6 p. https://doi.org/10.1109/VLSI-SoC53125.2021.9606966 DOI: https://doi.org/10.1109/VLSI-SoC53125.2021.9606966
Bach, T., Andrzejak, A., Seo, C., Bierstedt, C., Lemke, C., Ritter, D., Hwang, D. W., Sheshi, E., Schabernack, F., Renkes, F., Gaumnitz, G., Martens, J., Hoemke, L., Felderer, M., Rudolf, M., Jambigi, N., May, N., Joy, R., Scheja, R., Schwedes, S., Seibel, S., Seifert, S., Haas, S., Kraft, S., & Lehner, W. (2022). Testing very large database management systems: The case of SAP HANA. Datenbank-Spektrum, 22(3), 195-215. https://doi.org/10.1007/s13222-022-00426-x DOI: https://doi.org/10.1007/s13222-022-00426-x
Daase, B., Bollmeier, L. J., Benson, L., & Rabl, T. (2021). Maximizing persistent memory bandwidth utilization for OLAP workloads. In: Proceedings of the 2021 International Conference on Management of Data, 339-351 p. https://doi.org/10.1145/3448016.3457292 DOI: https://doi.org/10.1145/3448016.3457292
Flocchini, P., Prencipe, G., & Santoro, N. (2022). Distributed computing by oblivious mobile robots. In: Synthesis Lectures on Distributed Computing Theory, Synthesis Collection of Technology, Springer, Nature, 169 p.
Garofalo, A., Ottavi, G., Conti, F., Karunaratne, G., Boybat, I., Benini, L., & Rossi, D. (2022). A heterogeneous in-memory computing cluster for flexible end-to-end inference of real-world deep neural networks. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 12(2), 422-435. https://doi.org/10.1109/JETCAS.2022.3170152 DOI: https://doi.org/10.1109/JETCAS.2022.3170152
Guirado, R., Rahimi, A., Karunaratne, G., Alarcón, E., Sebastian, A., & Abadal, S. (2022). Wireless on-chip communications for scalable in-memory hyperdimensional computing. In: 2022 International Joint Conference on Neural Networks (IJCNN), IEEE, 1-8 p. https://doi.org/10.1109/IJCNN55064.2022.9892243 DOI: https://doi.org/10.1109/IJCNN55064.2022.9892243
Jhang, C. J., Xue, C. X., Hung, J. M., Chang, F. C., & Chang, M. F. (2021). Challenges and trends of SRAM-based computing-in-memory for AI edge devices. In: IEEE Transactions on Circuits and Systems I: Regular Papers, 68(5), 1773-1786. https://doi.org/10.1109/TCSI.2021.3064189 DOI: https://doi.org/10.1109/TCSI.2021.3064189
Kilickaya, F., & Okdem, S. (2021, November). Performance Analysis of Image Processing Techniques for Memory Usage and CPU Execution Time. In: Proceedings of the International Conference on Engineering Technologies (ICENTE’21), Konya, Turkey, 18-20 p.
Kobak, D., & Linderman, G. C. (2021). Initialization is critical for preserving global data structure in both t-SNE and UMAP. Nature Biotechnology, 39(2), 156-157. https://doi.org/10.1038/s41587-020-00809-z DOI: https://doi.org/10.1038/s41587-020-00809-z
Kumar, V., Sharma, D. K., & Mishra, V. K. (2021). Mille Cheval: a GPU-based in-memory high-performance computing framework for accelerated processing of big-data streams. The Journal of Supercomputing, 77(7), 6936-6960. https://doi.org/10.1007/s11227-020-03508-3 DOI: https://doi.org/10.1007/s11227-020-03508-3
Lersch, L. (2021). Leveraging non-volatile memory in modern storage management architectures. Technishe Universitat Dresden, Dissertation zur Erlangung des akademischen Grades Dohtoringenieur. https://core.ac.uk/reader/429344530
Naseer, H., Desouza, K., Maynard, S. B., & Ahmad, A. (2024). Enabling cybersecurity incident response agility through dynamic capabilities: the role of real-time analytics. European Journal of Information Systems, 33(2), 200-220. https://doi.org/10.1080/0960085X.2023.2257168 DOI: https://doi.org/10.1080/0960085X.2023.2257168
Patel, M., Shahroodi, T., Manglik, A., Yaglikci, A. G., Olgun, A., Luo, H., & Mutlu, O. (2022). A case for transparent reliability in DRAM systems. arXiv preprint arXiv:2204.10378.
Pedretti, G., & Ielmini, D. (2021). In-memory computing with resistive memory circuits: Status and outlook. Electronics, 10(9), 1063. https://doi.org/10.3390/electronics10091063 DOI: https://doi.org/10.3390/electronics10091063
Ranjan, J., & Foropon, C. (2021). Big data analytics in building the competitive intelligence of organizations. International Journal of Information Management, 56, 102231. https://doi.org/10.1016/j.ijinfomgt.2020.102231 DOI: https://doi.org/10.1016/j.ijinfomgt.2020.102231
Sawadogo, P., & Darmont, J. (2021). On data lake architectures and metadata management. Journal of Intelligent Information Systems, 56(1), 97-120. https://doi.org/10.1007/s10844-020-00608-7 DOI: https://doi.org/10.1007/s10844-020-00608-7
Singh, S. (2023). In-memory computation using static random access memories. Doctoral dissertation, 48 p.
Sun, Z., Kvatinsky, S., Si, X., Mehonic, A., Cai, Y., & Huang, R. (2023). A full spectrum of computing-in-memory technologies. Nature Electronics, 6(11), 823-835. https://doi.org/10.1038/s41928-023-01053-4 DOI: https://doi.org/10.1038/s41928-023-01053-4
Tuan, V. K., & Rajagopal, P. (2022). The mediating effect of the budget process on the performance of small-and medium-sized enterprises in Ho Chi Minh City, Vietnam. Journal of Entrepreneurship, Management and Innovation, 18(1), 65-92. https://www.ceeol.com/search/article-detail?id=1028436 DOI: https://doi.org/10.7341/20221813
Voss, W. G. (2021). The CCPA and the GDPR are not the same: why you should understand both. CPI antitrust chronicle, 1(1), 7-12. https://www.competitionpolicyinternational.com/the-ccpa-and-the-gdpr-are-not-the-same-why-you-should-understand-both/
Yang, J. Q., Zhou, Y., & Han, S. T. (2021). Functional applications of future data storage devices. Advanced Electronic Materials, 7(5), 2001181. https://doi.org/10.1002/aelm.202001181 DOI: https://doi.org/10.1002/aelm.202001181
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Semen M. Levin
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
1) Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
2) Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
3) Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
