Blog

After the termination of support for OWB has been officially announced, the Oracle Data Integrator (ODI) is the ETL tool of choice in the Oracle world. The development has progressed to version 12 which brought a few modifications and improvements. The GUI has continued to become more similar to OWB, although there are a few possibilities available which OWB did not offer in this way. In this blog entry, we will deal with the implementation of slowly changing dimensions in ODI.

Thanks to a compulsory interface for database connectors, the "Python Database API Specification v2.0, PEP249", all current connectors have been developed so that database connections and the SQLs for data retrieval and data transactions can be started using the same commands. Results are received in more or less the same format everywhere. It is regarding this issue that there seem to be the most severe deviations from the required standardisation.
But this should not scare anyone off from using Python scripts as a flexible method for automating database operations.

Data science has undergone an increasing professionalization and standardization during recent years. The frequently intrinsically motivated data tinkerer and diddler, who fills the niche "analysis" in his business with very high company-internal data and process know-how, is reaching his limits.

Increasing demands, especially in the course of a stronger customer focus across all industries, force businesses to professionalize the structures in the area "data science": This includes knowledge, available data sources and their preparation and data science products already used in the business.

SAS and R are topics which are very closely related: Both are popular tools for people like us who want to solve problems from the environment of statistic and machine learning on (more or less) large data volumes. Despite this apparent proximity, there are few touchpoints between the communities and only few persons work with both tools. As passionated `outside the box´ thinkers, we regret that and want to start a mini-series by means of this blog article in which we deal with topics which connect the both worlds, in loose order. For this first blog article, we will deal with the possibilities to exchange data between the systems. As there is a high number of ways, this article is limited to the transfer of SAS to R; the opposite direction will follow in a later article.

In order to successfully implement a campaign, it is important to be able to rely to a closed campaign planning process within the business. In case there is no such defined and uniform process, valuable potential is wasted and there is a risk that there will not be successes in the long term.

There are many areas in which neuroscience and machine learning overlap. One of these is the combining of learning during several learning episodes with small success in order to eventually use a merged, stronger, learned model for a particular task. In machine learning, this process is referred to as "boosting". The development of solutions of this kind is a very interesting topic, in particular in the IT industry; thus, a short introduction to machine learning is provided below which presents the basic ideas and the application of the naïve Bayes classifier in R.

Anyone who knows my former blogs knows that I am a big fan of both R and Python in daily work.

As powerful as R is in terms of functionalities for data analysis and modeling, as quickly is the motivation subdued in case of "number crunching" when RAM runs at maximum.

In this context, a nice server installation with a lot of metal (e.g. 96Gig-RAM) works wonders.

As this option is not always available, I have made a virtue of necessity and turned towards the more performant alternative, namely the Python based R alternatives, especially since I have been using Python for ETLs and data preparation for a long time.

Starting Situation

The controlling division of a telecommunications business is to be supported regarding the forecasting of the monthly development of gross adds figures. "Gross adds" is the key figure which reports the gross new customer growth within a defined period, where the number of lost customers is not taken into account. The key figure "gross adds" is primarily used in the telecommunications industry and reflects the number of newly concluded contracts (postpaid and prepaid).

‍

arcplan facilitates the creation of standardized reports, which support (and thus make more efficient) the daily work of the employees of a business. In particular, this is the case if reports present the included data in a meaningful, concise and user-friendly manner. Large volumes of information are thus often structured and presented in the form of tables. In order to illustrate relations and/or hierarchies of data within the table and avoid redundancies, it is required to choose columns and line headings which, in addition, are supposed to be placed expediently.

This article deals with the following three questions on a high level and very briefly:

• What does a data-driven person think when he hears contentions?
• Which tool is more practical for data analyses: R, Python, Java, MATLAB?
• Can sporting disciplines be the next application area for data analyses and machine learning

Ready-Made Data Sets Which Explode the Limits

I am frequently confronted with raw data that is provided to me for analysis and which, when uncompressed, can easily encompass files of half a gigabyte or more. Starting from one gigabyte and over, the desktop-supported statistics tools slowly become strained. There are, of course, tool options for only selecting part of the columns, or only loading the first 10,000 lines, etc.

But what should you do when you only want to take a random sample from the data provided? You should never rely on the file being randomly sorted. It may already have gained systematic sequence effects due to processes in the database export. It also may be the case, that you only want to analyse a tenth of a grouping, such as the purchases made by every tenth customer. To this end, the complete file has to be read as otherwise it is impossible to ensure that all of the purchases of the filtered customers are taken into account.

Uplift modelling can support campaign managers in managing and planning campaigns as it supplements the classic response model of campaign scoring.

Uplift modelling is based on the principal idea that campaign responders are grouped in two categories: those who would have reacted even without the campaign and those who would not have responded without the campaign. Unlike classic scoring, which equally aims at both groups, uplift scoring tries to exclusively isolate the second group and, wherever possible, ignore the first. For this purpose, the response information from the control group is used, which remains unused in classic campaign scoring