Logical Data Lake

Son Le Thanh (Son)
4 min readNov 21, 2022

Introduction

In the old paradigm , consumers are served by human agent for example in the context of insurance industry. Policy holders are served by agent but that has been changed. Consumers are more and more interacting with organizations via digital channels such as mobile apps, facebook page, youtube campaign, newsletter, chatbot etc.

In that paradigm shift, consumers are expecting the digital channels must be “smart” meaning the channels must have all the information about the consumers that have been shared and stored with the enterprise on the system of records.

Business Understanding

Data movement into the lake

In the past year, I have had worked with a customer (refer as ‘The Customer’) in the insurance industry. They have various data sources — both structured and unstructured — residing in disparate systems e.g. insurance policy management system, CRM, etc.

The Customer need to bring together those data sources to have a customer 360 view so that business users can have the fully integrated and comprehensive information about the customers to conduct various analytics workloads.

That requirement has posed the challenge in moving the data across different systems into a central data mart (or data lake).

That approach works but there are a couple of challenges

  • Each data integration has to involve IT to build the ETL jobs. There could be time delay from the time the business demands for data and the data is delivered.
  • Data has to be moved across systems into the lake. That has concerning with redundant data.
  • There are potentially delay of data delivery to the business users when they need access to near real-time data.

Data Virtualization

Those limitations could be addressed by Data Virtualization component of the solution

“Data virtualization is the ability to view, access, manipulate and analyze data without the need to know or understand its physical format or location”

--

--

Son Le Thanh (Son)

I am a geek, my background was in software development but I enjoyed building community and creating useful content. I am the father of two boys.