WhereScape 3D: Tips for dealing with model conversion rules

Model Conversion Rules are the central tool in WhereScape 3D for automated generation and adaptation of data models to individual requirements. Meant here, in particular, are data vault models due to their high potential for standardization and automation.

As requirements increase, however, adaptation and development of own model conversion rules can quickly become complex and confusing. To ensure successful handling of model conversion rules and make their use easier for beginners, we will introduce exemplary best practices, tips and tricks to you next.

Tip 1: Search for repetitive patterns (test of necessity)

A key requirement for automation using model conversion rules is the presence of repeatable patterns. This means that process execution using model conversion rules is useful only if the individual process steps contain certain patterns and regularities. Only then can the benefits of automation can be realized.

If there are no patterns, no regularity and hardly any capability of standardization, then an extremely individual process is involved. This means that the process is either executed only once, or its execution differs significantly from the previous one each time. The implementation effort required for process reproduction using model conversion rules may then exceed the benefits resulting from later automation. In this context, we therefore recommend manual implementation instead.

Tip 2: First implement model conversion manually before automating it with model conversion rules

Especially for more complex model conversion sets: Before you start developing a new model conversion set – that is, process automation – we recommend a manual implementation. Only after manual execution of the required process steps and associated documentation of your procedure should you begin implementing the model conversion rules. This has three key advantages:

1. Determination of feasibility: Only if manual handling proves feasible can the desired requirement for automation be fulfilled. Previous manual execution may therefore save time because an automated and, in many cases, more complex solution is not pursued initially.
2. Simplified selection of rules: The previously documented process execution provides you with an overview of the process steps which you have to reproduce using model conversion rules. Although not every process step always corresponds to a specific rule type, it becomes much easier to select rules required in terms of type and scope. In this context, also note that it is possible for one process step to require multiple rule types, and multiple process steps to be covered by one rule type.
3. Optimization of rule development: The knowledge acquired about individual process steps, their relationships and mutual interactions ultimately also simplifies general development (templates, matching criteria, ...).
    

Tip 3: Simple but useful: The help function

The best support during rule development is provided by WhereScape's help function, which can be accessed via the Help > Help navigation path. In particular, the sections titled "Model conversions" and "Pebble templates" provide detailed descriptions of individual rule types and the syntax of the Pebble template language.

Tip 4: Use the preview function

In contrast to WhereScape RED, 3D allows you to view the result of applying your template code already during development, thus reducing development time and susceptibility to errors of the implemented template snippets.

The preview function can be found to the right of the code editor. To use it, click the " ... " button under "Choose entity to preview template", and select an entity. The Pebble code to the left of the preview window is then applied to the selected entity, and the result displayed on the right.

Extra tip:

Enabling auto-refresh automatically updates the preview window following any change on the left side. Without auto-refresh, the update icon must be clicked to update the preview window.

Tip 5: Debug your model conversions using snapshot functionality

If a model conversion set fails to run, a corresponding error message is displayed. This can be used to obtain information on the failed rule and object to which the rule was applied in the event of the error, as well as an error description.

If this troubleshooting information is not sufficient, or if a successful but incorrect model is created, a type of debugging using snapshot functionality is recommended: A snapshot taken during rule execution allows visualization of a model after a specific rule has been executed on that model. Accordingly, snapshots prove useful, in particular, if taken before ("What does the model look like before something went wrong?") and after ("What does the model look like after something goes wrong?") execution of a faulty rule.

You can create a snapshot in just two steps:

1. The first step is to mark the rules after whose successful execution a snapshot is to be taken. To do this, navigate to the model conversion rules, click on the area to the left of one or more rules and select "Snapshot".
   
2. The second step is to execute the corresponding rules. Here you must make sure that the "Create Snapshots" field is activated during selection of the model conversion sets to be executed.
   

After execution of the rules, you can view the snapshots of the models and potentially discover incorrectly implemented aspects.

Tip 6: Define clear rules for development

To ensure consistent development and ultimately maintain consistent, clear model conversion sets, it is advisable to continue defining development guidelines. The following considerations help, for example:

1. Creation of new vs. extension of existent model conversion sets: For reasons of clarity, a new model conversion set should not be created for every new rule. Conversely, not all rules can be combined in a single model conversion set. As part of each new development, you should instead weigh between creating a new model conversion set or extending an existent model conversion set. Exemplary questions are: Can the new rules be assigned to an existent group of tasks, or is a separate issue involved? Does the complexity of the new rules together justify a custom model conversion set?
2. Avoidance of negative interactions: Often, several users develop in parallel. To avoid negative interactions due to a simultaneous modification of existing rules, you should work on copies in joint consultation, and transfer them to the original model conversion sets after successful testing.
3. Observance of a naming convention: Last but not least, you should devise a naming convention for clarity. More on this is provided in the next section.
    

Tip 7: Devise and use a naming convention

To ensure consistency and standardization, besides facilitating location of developed rules, you should design and adhere to an appropriate naming convention. Inclusion of the following components in the names of a rule set applied during transition from model category A to model category B has proven itself here:

[Individual_Abbbreviation]_[AbbreviationOfModelCategoryA][AbbreviationOfModelCategoryB] - [short description of the rule set]

[Individual_Abbreviation] can comprise, for example, an internal company abbreviation. This serves to delimit the default rules provided by WhereScape. [AbbreviationOfModelCategoryA] and [AbbreviationOfModelCategoryB] are the categories of the initial and newly generated models. This is exemplified in the rule set titled "ws3d_rvls – Create loads": This rule set (ws3d) provided by WhereScape is applied during the transition from the data vault category (RV=RawVault)  to the load and staging (ls) category, and used to generate load tables (create loads).

For the rules existent within a rule set, we here recommend that the description field be meaningful and maintained. You can also use the "Separator" rule type to define semantic sections within a model conversion set.

Tip 8: Export existent model conversion sets regularly

To minimize the risk of accidental loss and be able to load previous development versions, it is advisable to regularly export existent model conversion sets in the form of XML files.

To do this, navigate to the model conversion rules and click the disk icon displayed there. Afterwards you can determine the current repository's model conversion sets to be exported, i.e. export always takes place at the level of entire model conversion sets, and not individual rules. You can import model conversion sets using the arrow next to the disk.

As an alternative in this context, you can back up the entire repository using CLI calls, and even in automated mode using the WhereScape RED Scheduler. However, it is then no longer possible to load individual model conversion sets. Instead, the entire repository backup must always be loaded.

Extra tip:

WhereScape provides no function to combine existent model conversion sets. To avoid manually transferring the rules of one set to another, you can export both sets. Subsequently, the rules of one XML file can be copied to the other XML file, and the result imported into 3D.

Tip 9: Introduce external model conversion documentation

Often, several people work on the model conversion rules. In addition to adding new rules or deleting old rules, modification of existing rules is a critical operation here: Involved often are small changes to the template code or matching criteria, which remain undetected or are quickly forgotten by other team members without prior consultation.

To avoid future interactions and enable efficient testing, we therefore recommend introducing external model conversion documentation. The following table exemplifies the structure for a model conversion set called "Generate Data Vault", which you can also use to document other rule sets, and which ideally complements any created backups. The "Create satellites" rule in the table is first modified and later deleted, whereas the "Create hubs" rule is newly created.

Especially in large DWH projects with individual requirements, we recommend compliance with certain standards to ensure efficient and effective use of model conversion rules. In this context, this article has provided exemplary tips and tricks to help you get started.

If you too are looking for a DWH automation tool or need assistance in using WhereScape, do not hesitate to contact us, or subscribe to our newsletter to avoid missing blog posts and webinars on the topic!