Data Platform & Data Management

After years of experience of developing ETL applications, I can say that they are generally tested less rigorously than transaction systems.

This post deals in detail with the Protegrity platform and its interaction with Exasol. The Protegrity platform is meant to achieve extensive, end-to-end data protection over various applications and data storage devices.

Part 1: Adjusting The Validity Of The Dataset

As described in the previous blog entry, the Oracle Data Integrator (ODI) offers an integrated solution for keeping a history of data with the SCD (slowly changing dimension) methodology. Upon closer consideration and when an integration quantity is loaded practically into a target table using the integration knowledge module (IKM) SCD, it is noticeable that the ODI uses certain default values for the end of the validity period of the dataset.

In these times of digitalisation is it particularly important to be able to draw on reliable databases in order to eliminate errors at the source and facilitate a focused and precise way of working. The staging area is a solution for this type of challenge in today’s world.

Designers and architects often underestimate the need for a staging area for the database environment as they consider it a waste of space, effort and development time. Developing staging certainly requires space and effort, but this pays off over the whole life cycle of the database.

We have demonstrated in part 1 & part 2 how an AWS data lake is built from scratch and how the data is ingested in a Data Lakehouse. In this blog, we describe how to enforce GDPR law, the Right to be Forgotten (RTBF), in a Data Lakehouse. We make both the data lake and the data warehouse built in the previous blogs compliant with having a user exercise their Right to be Forgotten. Let us first understand what RTBF is.

As we have seen in the previous blog post, we should now have our transformed data in the data lake and have it available in the Glue Data Catalogue. In this blog post, we will first discuss what AWS Lake Formation is and see how we can use it to securely share access to the data.

This end-to-end and step-by-step quick demo shows you how to connect an existing on-premises data source to a modern cloud data warehouse solution such as Snowflake:

• Easily
• Fast
• Securely, and
• Without having to write a single line of code.

What Can Snowflake Do as a Cloud DB?

Because (storage) volume and execution time are paid for, the shorter times can reduce costs. A detailed online documentation is available at the following URL: https://docs.snowflake.net/manuals/index.html

Incidentally, one does not have to be an AWS customer to be able to use Snowflake. As a cloud-DB service, Snowflake itself offers no proprietary ETL tools, but leaves this to the manufacturers of ETL, reporting or self-service BI tools. These usually provide native drivers and connections to allow use of their tools with Snowflake. If no separate ETL tool is to be used at the company, there are several possibilities of loading data and realizing the ETL routes. One possibility is implementation of logic in SQL, and orchestration via Python.

Particularly in the environment of HCM and Infotype tables, full loads are often used for supplying SAP BW because clear detection of changes (CRUD - create, read, update and delete) in the source system is not possible. For example, deadlines (Infotype 0019) are not furnished with a full validity, so that a deletion does not lead to creation of a new data record which overwrites this period. In the source system, SLT/RS (SAP landscape transformation/replication server) with its trigger-based detection of changes at the database level could be used to monitor each individual data modification. Due to the relatively small amounts of data, however, use is not absolutely necessary, and change-tracking at this point would be "overkill". It should be noted that SLT/RS is sometimes included in current SAP BW license packages, and can therefore often be used, but requires an installation on the source system.