Snowflake’s Openflow makes data integration faster, simpler, and more efficient. In this article, we’ll show how these benefits play out in practice—using a real-world example to highlight strategies for handling large volumes of small incoming files with ease and performance.
High Performance Lookups in BW Transformations in Practice - Introduction
Kay Dietze
Kay Dietze
Management Consultant
Kay is a consultant, project and program manager with a focus on SAP-based business intelligence, data analytics and data management solutions. He has more than 20 years of internal and external consulting experience with customers from various industries such as aircraft and rail vehicle construction or pharmaceuticals.
Performance optimisations are not set in stone: optimisations that have worked extremely well at a company with a certain system architecture and a certain volume of data will not necessarily work equally well elsewhere. In other words, individual solutions are required. Fundamentally, however, the key point is always to find the right balance between main memory and database capacity, and between implementation complexity and serviceability. The focus is always on processing time.
Table of Contents
As we've learned in the previous article, using cache tables in SAP BW transformations is still useful even with an in-memory database such as HANA. Cache tables in this context are internal ABAP tables that are created and populated during data transformation processes (DTP). The tables are useful in particular when implementation logic needs to be applied in ABAP routines, and the transformations therefore cannot be processed in the SAP HANA database.
Often, all you need to do is duplicate or delete the data lines. This is best done with block operations. In other cases, lookups are required:
In lookups, information from one or more other database tables is added to each, hopefully relevant data record. The correct lookup structure in these cases is critical to the BW transformation performance.
Using an example that I have recently repeatedly come across in practice, this series will explore possible implementation techniques for the development of high-performance lookups.
Read our six-part series on "HIGH-PERFORMANCE LOOKUPS IN BW TRANSFORMATIONS IN PRACTICE", which will explain in detail the most common practical questions. The table below gives a summary of the areas covered:
Let’s Unlock the Full Potential of Your Data – Together!
Looking to become more data-driven, optimize processes, or leverage cutting-edge technologies? Our blog provides valuable insights – but the best way to tackle your specific challenges is through a direct conversation.
Let’s talk – our experts are just one click away!
Want To Learn More? Contact Us!
Your contact person
Helene Fuchs
Domain Lead Data Platform & Data Management
Your contact person
Pia Ehrnlechner
Domain Lead Data Platform & Data Management
Who is b.telligent?
Do you want to replace the IoT core with a multi-cloud solution and utilise the benefits of other IoT services from Azure or Amazon Web Services? Then get in touch with us and we will support you in the implementation with our expertise and the b.telligent partner network.
Snowflake’s Openflow makes data integration faster, simpler, and more efficient. In this article, we’ll show how these benefits play out in practice—using a real-world example to highlight strategies for handling large volumes of small incoming files with ease and performance.
With Openflow, Snowflake fundamentally simplifies data integration: extraction and loading happen directly as part of the Snowflake platform — no external ETL tools required. This significantly reduces integration effort and streamlines the entire pipeline management process.
Exasol is a leading manufacturer of analytical database systems. Its core product is a high-performance, in-memory, parallel processing software specifically designed for the rapid analysis of data. It normally processes SQL statements sequentially in an SQL script. But how can you execute several statements simultaneously? Using the simple script contained in this blog post, we show you how.