Unleashing real-time analytics: A comparative study of in-memory computing vs. traditional disk-based systems

Authors

DOI:

https://doi.org/10.14295/bjs.v3i5.553

Keywords:

in-memory computing, real-time analytics, data processing efficiency, redis, postgresql

Abstract

The article presents a comprehensive study evaluating the performance differences between in-memory computing (IMC) and traditional disk-based database systems, specifically focusing on Redis and PostgreSQL. Given the escalating demands for real-time data analytics across various sectors, the research delves into the comparative efficiency of these two data management paradigms in processing large datasets. Utilizing a synthetic dataset of 23.6 million records, we orchestrated a series of data manipulation tasks, including aggregation, table joins, and filtering operations, to simulate real-world data analytics scenarios. The experiment, conducted on a high-performance computing setup, revealed that Redis significantly outperformed PostgreSQL in all tested operations, showcasing the inherent advantages of IMC in terms of speed and efficiency. Data aggregation tasks saw Redis completing the process up to ten times faster than PostgreSQL. Similarly, table joining, and data filtering tasks were executed more swiftly on Redis, emphasizing IMC's potential to facilitate instantaneous data analytics. These findings underscore the pivotal role of IMC technologies like Redis in empowering organizations to harness real-time insights from big data, a critical capability in today's fast-paced business environment. The study further discusses the implications of adopting IMC over traditional systems, considering aspects such as cost, integration challenges, and the importance of skill development for IT teams. Concluding with strategic recommendations, the article advocates for a nuanced approach to incorporating IMC technologies, highlighting their transformative potential while acknowledging the need for balanced investment and operational planning.

References

Al-Mohannadi, A., Al-Maadeed, S., Elharrouss, O., & Sadasivuni, K. K. (2021). Encoder-decoder architecture for ultrasound IMC segmentation and cIMT measurement. Sensors, 21(20), 6839. https://doi.org/10.3390/s21206839

Amrouch, H., Du, N., Gebregiorgis, A., Hamdioui, S., & Polian, I. (2021). Towards reliable in-memory computing: From emerging devices to post-von-neumann architectures. In: 2021 IFIP/IEEE 29th International Conference on Very Large Scale Integration (VLSI-SoC), IEEE, Singapore, 1-6 p. https://doi.org/10.1109/VLSI-SoC53125.2021.9606966

Bach, T., Andrzejak, A., Seo, C., Bierstedt, C., Lemke, C., Ritter, D., Hwang, D. W., Sheshi, E., Schabernack, F., Renkes, F., Gaumnitz, G., Martens, J., Hoemke, L., Felderer, M., Rudolf, M., Jambigi, N., May, N., Joy, R., Scheja, R., Schwedes, S., Seibel, S., Seifert, S., Haas, S., Kraft, S., & Lehner, W. (2022). Testing very large database management systems: The case of SAP HANA. Datenbank-Spektrum, 22(3), 195-215. https://doi.org/10.1007/s13222-022-00426-x

Daase, B., Bollmeier, L. J., Benson, L., & Rabl, T. (2021). Maximizing persistent memory bandwidth utilization for OLAP workloads. In: Proceedings of the 2021 International Conference on Management of Data, 339-351 p. https://doi.org/10.1145/3448016.3457292

Flocchini, P., Prencipe, G., & Santoro, N. (2022). Distributed computing by oblivious mobile robots. In: Synthesis Lectures on Distributed Computing Theory, Synthesis Collection of Technology, Springer, Nature, 169 p.

Garofalo, A., Ottavi, G., Conti, F., Karunaratne, G., Boybat, I., Benini, L., & Rossi, D. (2022). A heterogeneous in-memory computing cluster for flexible end-to-end inference of real-world deep neural networks. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 12(2), 422-435. https://doi.org/10.1109/JETCAS.2022.3170152

Guirado, R., Rahimi, A., Karunaratne, G., Alarcón, E., Sebastian, A., & Abadal, S. (2022). Wireless on-chip communications for scalable in-memory hyperdimensional computing. In: 2022 International Joint Conference on Neural Networks (IJCNN), IEEE, 1-8 p. https://doi.org/10.1109/IJCNN55064.2022.9892243

Jhang, C. J., Xue, C. X., Hung, J. M., Chang, F. C., & Chang, M. F. (2021). Challenges and trends of SRAM-based computing-in-memory for AI edge devices. In: IEEE Transactions on Circuits and Systems I: Regular Papers, 68(5), 1773-1786. https://doi.org/10.1109/TCSI.2021.3064189

Kilickaya, F., & Okdem, S. (2021, November). Performance Analysis of Image Processing Techniques for Memory Usage and CPU Execution Time. In: Proceedings of the International Conference on Engineering Technologies (ICENTE’21), Konya, Turkey, 18-20 p.

Kobak, D., & Linderman, G. C. (2021). Initialization is critical for preserving global data structure in both t-SNE and UMAP. Nature Biotechnology, 39(2), 156-157. https://doi.org/10.1038/s41587-020-00809-z

Kumar, V., Sharma, D. K., & Mishra, V. K. (2021). Mille Cheval: a GPU-based in-memory high-performance computing framework for accelerated processing of big-data streams. The Journal of Supercomputing, 77(7), 6936-6960. https://doi.org/10.1007/s11227-020-03508-3

Lersch, L. (2021). Leveraging non-volatile memory in modern storage management architectures. Technishe Universitat Dresden, Dissertation zur Erlangung des akademischen Grades Dohtoringenieur. https://core.ac.uk/reader/429344530

Naseer, H., Desouza, K., Maynard, S. B., & Ahmad, A. (2024). Enabling cybersecurity incident response agility through dynamic capabilities: the role of real-time analytics. European Journal of Information Systems, 33(2), 200-220. https://doi.org/10.1080/0960085X.2023.2257168

Patel, M., Shahroodi, T., Manglik, A., Yaglikci, A. G., Olgun, A., Luo, H., & Mutlu, O. (2022). A case for transparent reliability in DRAM systems. arXiv preprint arXiv:2204.10378.

Pedretti, G., & Ielmini, D. (2021). In-memory computing with resistive memory circuits: Status and outlook. Electronics, 10(9), 1063. https://doi.org/10.3390/electronics10091063

Ranjan, J., & Foropon, C. (2021). Big data analytics in building the competitive intelligence of organizations. International Journal of Information Management, 56, 102231. https://doi.org/10.1016/j.ijinfomgt.2020.102231

Sawadogo, P., & Darmont, J. (2021). On data lake architectures and metadata management. Journal of Intelligent Information Systems, 56(1), 97-120. https://doi.org/10.1007/s10844-020-00608-7

Singh, S. (2023). In-memory computation using static random access memories. Doctoral dissertation, 48 p.

Sun, Z., Kvatinsky, S., Si, X., Mehonic, A., Cai, Y., & Huang, R. (2023). A full spectrum of computing-in-memory technologies. Nature Electronics, 6(11), 823-835. https://doi.org/10.1038/s41928-023-01053-4

Tuan, V. K., & Rajagopal, P. (2022). The mediating effect of the budget process on the performance of small-and medium-sized enterprises in Ho Chi Minh City, Vietnam. Journal of Entrepreneurship, Management and Innovation, 18(1), 65-92. https://www.ceeol.com/search/article-detail?id=1028436

Voss, W. G. (2021). The CCPA and the GDPR are not the same: why you should understand both. CPI antitrust chronicle, 1(1), 7-12. https://www.competitionpolicyinternational.com/the-ccpa-and-the-gdpr-are-not-the-same-why-you-should-understand-both/

Yang, J. Q., Zhou, Y., & Han, S. T. (2021). Functional applications of future data storage devices. Advanced Electronic Materials, 7(5), 2001181. https://doi.org/10.1002/aelm.202001181

Downloads

Published

2024-04-24

How to Cite

Levin, S. M. (2024). Unleashing real-time analytics: A comparative study of in-memory computing vs. traditional disk-based systems. Brazilian Journal of Science, 3(5), 30–39. https://doi.org/10.14295/bjs.v3i5.553