previous article
For years, data has been considered the resource of the future. For a correspondingly long time, many companies have diligently collected all the information which (more or less) appears useful. But how can these incredible amounts of data be stored in a meaningful way in order to generate added value? For this, the Azure cloud offers numerous possibilities on which we would like to shed light now.
Why move to the cloud?
The cloud offers some significant advantages over conventional, on-premise data centres:
- Scalability: Virtualization allows cloud-based services to quickly increase the number and performance of computing nodes ("scale out" or "scale up"). This makes it possible to react dynamically to changing requirements.
- Cost efficiency: Because selected performance levels can be dynamically adapted to requirements, there is almost no unnecessary idle time, and only the performance actually needed is paid for. Especially in scenarios involving strong chronological fluctuations, this can lead to enormous cost savings compared to conventional solutions.
- Managed services: Many cloud services are available as platform-as-a-service (PaaS) or software-as-a-service (SaaS). These managed services reduce administrative effort, which is enormously helpful given recurrent shortages of skilled IT staff.
All these points are particularly advantageous for chronologically variable requirements, e.g. posed by integration of new data sources or seasonal effects.
What can Azure do?
Azure offers a wide range of services, so here are just a few examples as a helpful, first port of call at Azure.
- SQL databases: Classic Microsoft T-SQL, available e.g. as a fully managed SQL database or highly customizable SQL server managed instance.
- Azure Cosmos DB: noSQL database, well suited for storing IoT messages. Data are queried via Mongo API or SQL API.
- Azure Data Lake Storage (ADLS) gen2: The central storage hub of many modern data platforms. Low-cost and high-performance file storage with hierarchical namespace for detailed access management.
- Azure Synapse analytics: Comprehensive analytics platform with dedicated or serverless SQL pool. Extensive ETL features, as well as Spark notebooks for advanced analytics.
- Azure Purview: This data catalogue can be used to scan and classify various data sources and sinks (especially data affected by CDPR). Data processing steps are displayed and can always be traced.
There are many more options - in total, Azure offers hundreds of different services ranging from classic virtual machines to highly abstracted no/low-code tools.
The modern cloud architecture
But how do these services work together to create a modern data platform?
- Data are loaded into the Azure data lake from various source systems (cloud or on-premise) using Azure Synapse pipelines.
- In the data lake, these data are processed and standardized to create a homogeneous interface for receiver systems. With Synapse data flows and Spark notebooks, massive amounts of data can be processed quickly.
- With Azure SQL server managed instances, data from the data lake can be optionally loaded into a data warehouse.
- With Power BI, reports can be used to generate data from SQL managed instance and the data lake to generate new insights and, thus, added value from the originally separate data sources.
Challenges of moving to the cloud
With our experience, we can help our customers handle challenges faced on the way to the cloud. These include:
- Establishing a hybrid architecture using on-premise data sources and the cloud
- Ensuring consistency of interfaces provided for receiver systems
- Ensuring secure access to public/private cloud resources
- Appropriate scaling of various services
- Handling of personal (CDPR-relevant) data in the cloud
Was this interesting? Then contact one of our many Azure experts for an initial, non-binding consultation so we can help you orientate yourself in the Azure Cloud.
