Data Platform & Data Management

The Devil in the Detail - What Details Decide on the Effectiveness of Partition Pruning?

In the previous article of this series, a practical and effective approach to using partition pruning was explained in detail. This easy-to-implement method can significantly optimize query times. However, as is often the case, some details need to be taken into account to ensure the efficient and effective use of the presented method. In this regard, we echo Theodor Fontane, who observed as far back as the 19th century that the magic always lies in the details.

After outlining the conventional methods for storing historical data in the first post of this blog series, I would like to introduce a more effective approach to partitioning a historical table in this second part.

In this article, I propose a way for physical organization of historical tables, which makes it possible to effectively use partition pruning to optimize query performance. The way is specifically designed for data warehouses, therefore it presumes relatively complicated data loads yet productive selections.

More than a year after introduction of the general data protection regulation (GDPR), many enterprises still find it hard to reconcile the topics of data warehouse (DWH) and data privacy. Customer-centric data modelling prevailing at enterprises poses a special challenge here. It leads to major conflicts with many GDPR requirements in virtually any data-driven process. But why is it so hard to unify the two topics?

Starting Point

In order to be able to use an effective customer relationship management, it is of great importance to have robust knowledge about one's own customers and customer structures - especially in the area  of campaign management. Personal information about customers, events and changes in the customer situation and behaviour in a CRM system can be used as opportunity for communication. By means of this existing data, a personalized interaction with the customer, such as offers and services, becomes possible.

Many businesses are currently starting to make initial experiences with the use of big data methods. By means of statistical methods, these methods promise to generate additional knowledge about the customers from huge amounts of mostly unstructured data in very short time. Frequently, automatically generated data such as log files or clickstreams collected from the use of the website, a web shop or data from social networks, is used for this purpose. This knowledge can be of good use for CRM activities. In this context, the question of the significance of data quality in the use of big data methods arises. Because: The analysis and thus the generated knowledge can only be as good as the quality of the data used.  

The role of the customer

The central role of the customer for the strategic alignment of businesses has been discussed in science for decades.

"It costs much more to acquire a customer than it does to keep a customer. That is why a sale to a customer is good, but a relationship with a customer is great." [1]

"Personal data are the fuel of the modern economy" [2]

"In a global information-based economy, data about customers are one of the most important sources for competitive advantage." [3]

I’d like to once again delve into the issue of memory usage. Other blogs like “SAP HANA – no more memory? Implement early unload!” have shown the importance of properly configuring SAP BW objects for the intended usage. Briefly:  anything not queried regularly and immediately should be unloaded early to storage. For this you set the “early unload priority” attribute in the HANA database table. Since this setting is not a part of the ABAP transport system (CTS), the developer or operation must always ensure that the configuration is correct. Otherwise, only data needed for staging from the data acquisition layer perform well in RAM, which burdens the system and is unnecessarily expensive.

After years of experience of developing ETL applications, I can say that they are generally tested less rigorously than transaction systems.

This post deals in detail with the Protegrity platform and its interaction with Exasol. The Protegrity platform is meant to achieve extensive, end-to-end data protection over various applications and data storage devices.