Azure Data Factory Deep Dive
With the ever-increasing volume, variety, and velocity of data, it can feel like a daunting task to create and maintain modern data integration solutions. This full-day workshop will arm you with the skills you need to build and orchestrate hybrid, complex and scalable data pipelines using Azure Data Factory (ADF).
We will start with a conceptual overview and cover the fundamentals of Azure Data Factory, including source control, security, and pricing. Then, we will build a metadata-driven solution to move data between on-premises and cloud sources. As we evaluate different design patterns and architectures for big data pipelines and modern data warehouses, we will continue to improve on our solution to make it more robust, dynamic and reusable. We will dive deep into the newly released Data Flow capabilities, and look at how to leverage SSIS lift and shift to gradually modernize existing solutions while retaining investments already made. Finally, we will configure monitoring, logging, and alerting, as well as review options for CI/CD using Azure DevOps.
If that's not enough content for one day, you will also get access to a set of hands-on labs that you can work through at your own pace. Whether you are new to Azure Data Factory or have some experience, you will leave this workshop with new skills and ideas for your projects.AlcazarMon 09:00 – 17:00
Azure Databricks: Engineering Vs Data Science
Have you looked at Azure DataBricks yet? No! Then you need to. Why you ask, there are many reasons. The number 1, knowing how to use apache Spark will earn you more money. It is that simple. Data Engineers and Data Scientists who know apache Spark are in-demand! This workshop is designed to introduce you to the skills required to do both.
In the morning we will introduce Azure DataBricks then discuss how to develop in-memory elastic scale data engineering pipelines. We will talk about shaping and cleaning data, the languages, notebooks, ways of working, design patterns and how to get the best performance. You will build an engineering pipeline with Python (Or possibly some other stuff we are not allowed to tell you about yet). The Engineering element will be delivered by UK MVP Simon Whiteley. Simon has been deploying engineering projects with Azure DataBricks since it was announced. He has real world experience in multiple environments.
Then we will shift gears, we will take the data we moved and cleansed and apply distributed machine learning at scale. We will train a model and productionise it. We will then enrich our data with our newly predicted values. The Data Science element will be led by UK MVP Terry McCann. Terry holds an MSc in Data Science and has been working with apache Spark for the last 5 years. He is dedicated to applying engineering practices to data science to make model development, training and scoring as easy an as automated as possible
By the end of the day, you will understand how Azure Databricks supports both data engineering and data science, levering apache Spark to deliver blisteringly fast data pipelines and distributed machine learning models. Bring your laptop as this will be hands on.
An understanding of ETL processing either ETL or ELT on either on-premises or in a big data environment. A basic level of Machine Learning would also be beneficial, but not critical.
Software: In the session we will be using Azure Databricks. We will have labs and demos that you can follow if you want to. If you do want to then you will need the following: - An Azure Subscription - Money on the Azure Subscription - Enough access on the subscription to make service principals. - Azure Storage explorer- PowerShell
Subscriptions: AzureHerten AasMon 09:00 – 17:00
Azure SQL Database 2.0 - the new stuff!
When Microsoft Azure SQL Database was introduced, it had only a small subset of the features available in the SQL Server database engine. With the introduction of version V12 a few years ago, Azure SQL Database is now an enterprise-class platform-as-a-service (PaaS) offering. Of course, Microsoft continued to add new features and functionalities into Azure SQL Database.
Join our pre-con to learn more about some cool new features that are available now in Azure SQL Database.
The main topics we will discuss are:
o Elastic Database Jobs
o Backup Retention Periods
o Geo-replication & Failover Groups
o Virtual Network service endpoints
o How to connect to Azure SQL Database and Managed Instance
o TDE with Bring Your Own Key
o Data Discovery and Classification
o SQL Vulnerability Assessment
Monitoring and performance tuning
o Automatic Tuning
o Intelligent Insights
o Data Migration Service
o Azure Data Sync
We are not going to start from scratch, so prior knowledge on Azure SQL Database is required.
Bring your laptop and your Azure subscription because we’re planning some exercises.DijlezaalMon 09:00 – 17:00
Power BI - From Beginning to End
Wow, Power BI is making a tidal wave in the BI industry. New features are being released regularly, blog posts are flying across twitter. How can you keep up? In this full day session, we will explain and demonstrate how to begin using Power BI effectively within your organization. This will include preparing your organization for Power BI, deciding if you need to build an SSAS Semantic Model or will the Power BI desktop work as an alternative. In addition, there will be a few hand-on labs that will help each attendee get started working with both the Power BI Desktop and the Power BI Service. Bring your laptop, grab a cup of coffee (or beverage of choice) and get ready to learn.
• Getting and shaping data
• Working with your data model
• A look at DAX
• Visuals and your report
• Tips and tricks you may not have known about
• Taking your report to the next level
• Power BI service features
• Connecting to and Refreshing Data
And more…AuditoriumMon 09:00 – 17:00
Using AKS and Azure DevOps to bring DevOps to your Database
Come and join Hamish and Rob, two MVPs from two different continents to learn practical solutions to bring DevOps to your database.
In this all day training session, based on using the Microsoft Azure platform, you will learn
• The importance of getting your database code into source control.
• How to test your database changes.
• A process for deploying database changes that will keep Change and Release Management happy through the use of branching, Pull Requests and Approval Gates.
• Some of the tools that you can use to automate build and test processes.
• How to build an automated deployment process for your database with Azure DevOps.
• Deploying your entire pipeline as and when it is needed from Dev to Prod saving your organisation money.
Database upgrades and data in general are often the most complicated part of your deployment process, so having a robust deployment path and checks before getting to production is very important. The demonstrations will showcase practical solutions that can help you and your team bring DevOps to your database. This will include using Azure DevOps, infrastructure as code, docker containers, source control, unit test frameworks and SQL Server Data Tools – all leading up to using Kubernetes in Azure.
The demonstrations will also showcase alternative methods to achieve the same result using various technology platforms and will be relevant to your workplace experiences. The training will appeal to database administrators who want to understand what DevOps is and how it can help achieve repeatable, reliable deployments to their database systems. It will also benefit developers and anyone else involved in deploying data changes who want to extend their knowledge of Continuous Delivery.
This will be a fun-filled fast paced day and you will have access to all materials to take away and will learn skills which will bring immediate benefit to your organisation.ScalaMon 09:00 – 17:00
Amplifying human ingenuity with intelligent technology
We are still in the first minutes of the first day of the intelligence revolution. Yet we already have so much powerful technology at our fingertips with the potential to transform every application, every company and every industry.
This session will discuss the disruption expected from breakthrough innovations in the field of AI, IoT and quantum computing. Using local use cases you will see the impact that these innovations already cause today and get a glimpse of what they will bring tomorrow. At Microsoft we aim to democratize these technologies and empower every person and every organization to achieve more.AuditoriumTue 08:45 — 45 min
Creating Visual Data Transformations in Azure Data Factory
Azure Data Factory v2 came with many new capabilities and improvements. One of biggest game-changers is the Mapping Data Flows feature, allowing you to transform data at scale - without having to write a single line of code!
In this session, we will first go through the capabilities and use cases for Mapping Data Flows. Then, we will explore the various transformations available, as well as the expression language and how to use the visual expression builder. Finally, we will look at how to debug, monitor, and optimize our data transformations.ScalaTue 09:45 — 60 min
DevOps for Artificial Intelligence, the road to production.
With machine learning becoming more and more an engineering problem the need to track, work together and easily deploy ML experiments with integrated CI/CD tooling is becoming more relevant than ever.
In this session we take a deep-dive into the DevOps process that comes with Azure Machine Learning service, a cloud service that you can use to track as you build, train, deploy and manage models. We zoom into how the data science process can be made traceable and deploy the model with Azure DevOps to a Kubernetes cluster.
At the end of this session you have a good grasp of the technological building blocks of Azure machine learning services and can bring a machine learning project safely into production.Herten AasTue 09:45 — 60 min
Lesser known sql server functionalities, hidden gems or false hope?
In this session we will go over some features that are lesser known in the public SQL Server eye.
The session is designed to get people a first glance at some functionalities you might not have used or even heard of. While you might not have a direct need for some or all of them, it never hurts to know!
Some examples including but not limited to:
Async stat updates,
....DijlezaalTue 09:45 — 60 min
Paginated Reports in Power BI
Abstract missing, due to last-minute changesAuditoriumTue 09:45 — 60 min
Query Folding in Power BI
Power Query allows you to extract and transform data from a variety of data sources. Have you ever experienced that the importing of data is slower than you expected? In most of those cases, query folding is not happening fully optimized.
Query folding is important because it offloads data transformations to the source, instead of performing them in Power BI.
Attend this session to learn how to ensure that you get all the performance improvements possible within your reports! As an added bonus, you will also learn in which cases query folding is not your best friend.BegijnenzolderTue 09:45 — 60 min
The Big SQL Server 2019 Big Data Cluster Show
With SQL Server 2019, Microsoft introduced a killer feature: Big Data Clusters.
They bring advanced machine learning, an in-SQL Server Data Lake and enhanced Data virtualization through PolyBase.
We will take a look at what that means, how Data Virtualization differs from Data Integration, what it takes to get a Big Data Cluster (even for small Data) deployed and how it can help you solve real time issues you may have.
To do so, we will not only guide you through the product itself but also through the technologies and tools like Spark, Notebooks and Azure Data Studio that it relies on, so you can leave with a call to action on how you can leverage these new capabilities in your organization today!AlcazarTue 09:45 — 60 min
Benchmarking in the cloud
Lifting and shifting your application to the cloud is extremely easy, on paper. The hard truth is that the only way to know for sure how it is going to perform is to test it. Benchmarking on premises is hard enough, but benchmarking in the cloud can get really hairy because of the restrictions in PaaS environments and the lack of tooling.
Join me in this session and learn how to capture a production workload, replay it to your cloud database and compare the performance. I will introduce you to the methodology and the tools to bring your database to the cloud without breaking a sweat.
With WorkloadTools, benchmarking will be as easy as pie.DijlezaalTue 11:00 — 60 min
Commonalities and differences between BI and Datascience pipelines
I have a strong background in BI (more than 30 years). While learning about Data Science and Machine Learning a few years ago, I found a lot of commonalities but I was also puzzled by some differences. Why do we have different tools (programming languages, workflow/ETL tools), even from a same tool provider such as Microsoft, although they may look so similar at a first sight? It's not just because of legacy/historical reasons. Actually, although we may and should share some data, some tools and some methodologies, needs are quite different.BegijnenzolderTue 11:00 — 60 min
Customer story: End-to-end Microsoft BI solution in Azure
How do you setup and build a complete Microsoft BI solution with only Azure Services? Lessons learned from a project with eleven different source systems in the one end and 12.000 users in seven countries in the other end. Extracting, transforming and loading with the help from Functions, Data Factory, Data Lake Store, Polybase, SQL Data Warehouse, SQL Database, Automation Runbooks, Analysis Services and Power BI Embedded.ScalaTue 11:00 — 60 min
Identity Mapping and De-Duplicating
In an enterprise, merging master data, like customer data, from multiple sources is a common problem. Typically, you do not have a single, i.e. the same key identifying a customer in different sources. You have to match data based on similarity of strings, like names and addresses. In this session, we are going to check how different algorithms for comparing strings included in SQL Server work. We are going to use four different algorithms that come with Master Data Services (Levenshtein, Jaccard, Jaro-Winkler and Ratcliff-Obershelp), and Fuzzy Lookup transformation from Integration Services. Finally, we are going to introduce how SQL Server Data Quality Services help us here.AlcazarTue 11:00 — 60 min
Power BI Performance in 6 demos
We will explore some practical and common performance problems that customers encounter when using Power BI. From report design to your data, we will cover different aspects of Power BI to get you thinking about how to use the service within your organization.AuditoriumTue 11:00 — 60 min
Using Artificial Intelligence to drive Supply Chain Planning
Meat & More is a food company specialized in meat products and meals, under the brands called Buurtslagers and Bon'ap.
Estimating daily demands of such a variety of products in many different locations is a challenging job to say the least. There is dependence on sales trends, region preferences, seasons, holidays, weather, BBQ peaks, promotions, marketing actions & decisions on the product range. By using AI techniques, Meat&More is able to estimate the expected sale per butcher shop and per product. In this session Charles Cuigniez & Bart Van Der Vurst (element61) explain how they have set up and maintain an AI-driven automated planning using SQL Server and Azure technology.
Technologies used are SQL Server, Azure Data Factory, Azure Storage, Azure Batch, Docker, Azure Container Registry, R and PowerBI. Full project is set-up using Continuous Integration & Deployment using Azure DevOps incl. automated unit testing and ARM Template deploys to the Azure Cloud.Herten AasTue 11:00 — 60 min
Analyze images in real-time with machine learning, Azure IoT Hub and Azure Stream Analytics
In this session we will look into the design and implementation of an intelligtent solution to track polar bear activity based on pictures taken on an island (simulated).
Products used in this design are Azure Storage account, IoT hub, Node.js, Azure Stream Analytics, Azure Function, Cognitive Services, Power BI and Azure SQL database.
We will discuss the role of each product during this demo heavy session.Herten AasTue 13:00 — 60 min
How can we use Azure Data Factory and Azure Databricks to train our ML models
This session will show how we can use Azure Databricks as a fully managed service by Microsoft to train and tune our machine learning models. Additionally, I will demonstrate how Data Factory can be used to automate & schedule the whole process of starting and stopping a cluster, training the model and saving it to a blob storage.
During the session I will cover the following subjects:
What is Azure Databricks?
What is Azure Data Factory?
How can we use them together to train our ML models?
The presentation will be followed by a short demo.BegijnenzolderTue 13:00 — 60 min
Power BI Report design Tips and Tricks
"A demo's filled session packed with tips and tricks to show how to transform usual Power BI reports to stunning reports
In this session you’ll learn about:
- How to use background images and useful resources to create the background templates
- Use of colours, various resources to get appealing colour pallets
- Multiple ways of using conditional formatting to highlight the specific data points
- How to create Power BI theme files
- Various DataViz resources"AuditoriumTue 13:00 — 60 min
Python Pipeline Primer: Data Engineering with Azure DataBricks
Azure DataBricks brings a Platform-as-a-Service offering of Apache Spark, which allows for blazing fast data processing, interactive querying and the hosting of machine learning models all in one place! But most of the buzz is around what it means for Data Science & AI - what about the humble data engineer who wants to harness the in-memory processing power within their ETL pipelines? How does it fit into the Modern Data Warehouse? What does data preparation look like in this new world?
This session will run through the best practices of implementing Azure DataBricks as your data ingestion, transformation and curation tool of choice. We will:
• Introduce the Azure DataBricks service
• Introduce Python and why it is the language of choice for Data Engineering on DataBricks
• Discuss the various hosting & compute options available
• Demonstrate a sample data processing task
• Compare and contrast against alternative approaches using SSIS, U-SQL and HDInsight
• Demonstrate how to manage and orchestrate your processing pipelines
• Review the wider architectures and additional extension patterns
The session is aimed at Data Engineers & BI Professionals seeking to put the Azure DataBricks technology in the right context and learn how to use the service. We will not be covering the python programming language in detail.ScalaTue 13:00 — 60 min
SQL Notebooks in Azure Data Studio for the DBA
An Azure Data Studio recent release introduced SQL Notebooks as a capability.
While you might have read about notebooks being used for Data Science, this session is for DBAs. You will see how you can use SQL Notebooks to simplify your work.
You will learn about SQL Notebooks from installing them in Azure Data Studio to creating a Notebook for Glenn Berry's Diagnostic queries as well as numerous use cases for you to use in your daily workloadDijlezaalTue 13:00 — 60 min
Start To See With tSQLt
Unit testing with databases is fairly new and is not yet embraced within database development.
Many database developers do not do any unit testing or don't do it in a automated way.
tSQLt is a unit testing framework that enables developers to create sophisticated unit tests for databases.
You can create a whole bunch of different kind of unit tests. From really simple to very complex.
In this session we'll talk about unit testing in general, unit vs integration testing, test driven development and of course tSQLt.
Come to this session if you want to learn how to implement tSQLt in your development procedures
I can assure you that it's going to make your life so much easier in the end.AlcazarTue 13:00 — 60 min
AI in Production
Abstract missing, due to last-minute changeHerten AasTue 14:15 — 60 min
Automate your Data Warehouse Development
In this session we will focus on automating your Data Warehouse development, using the Foundation Accelerator. Based on the metadata of your source systems and your input, the Foundation Accelerator will automatically create your Data Warehouse model based on Data Vault 2.0, the related DDL statements and all the T-SQL code to load your Data Warehouse.ScalaTue 14:15 — 60 min
Based on real life scenarios, in this audience interactive session we will go through some scenarios DBA's might encounter whilst dealing with SQL Server databases and you will be provided with some options about what to do. Members of the audience will then select from these options what to do and we will follow that path and see what the outcome is from there.
Each selection will have a different outcome, and along the way you will probably learn some new things.DijlezaalTue 14:15 — 60 min
Introduction to Azure Cosmos DB (demo-based)
In this session, I would like to explain the basics of the Azure Cosmos DB using the following questions:
1) What is Azure Cosmos DB?
2) What are the different APIs to query data from the Cosmos DB?
3) How to troubleshoot?
These questions will be answered through live-demos.BegijnenzolderTue 14:15 — 60 min
Monitoring Power BI
As Power BI is a self-service tool it can be hard for administrators to monitor it. Power BI is fast improving in this context but there still isn’t a consistent way of monitoring it. In this session you will hear about using the Power BI Audit log and the Power BI APIs to monitor Power BI to ensure compliance, governance, performnace and good inplementation.AuditoriumTue 14:15 — 60 min
Why my query is slow and how to fix this
Generally, there are only two options for query slowness: either the query itself is slow (high CPU time) or the query is waiting for some resource (high Wait time). All other reasons for query slowness are derived from these two options. In this session, we will learn how to determine what is the reason for the long-running query, what the request can wait for, and how it can be acceleratedAlcazarTue 14:15 — 60 min
Azure SQL Database - Lessons learned from the trenches
In this session you will learn the best practices, tips and tricks on how to successfully use Azure SQL Database on production environments. You will learn how to monitor and improve Azure SQL Database query performance. I will cover how Microsoft CSS has been using Query Store, Extended Events, DMVs to help customers monitor and improve query response times when running their databases in the Microsoft Azure cloud. These learnings are fruit of Microsoft CSS support cases, and customer field engagements. This session includes several demosDijlezaalTue 15:30 — 60 min
Microsoft Power BI Premium: Building enterprise-grade models
Power BI Premium enables you to build comprehensive, enterprise-scale analytic solutions that deliver actionable insights through familiar data visualization tools such as Microsoft Power BI and Microsoft Excel. This session will dive deep into exciting, new and upcoming features including aggregations for big data to unlock petabyte-scale datasets that was not possible before! The session will focus on performance, scalability, and DAX improvements. Learn how to use Power BI Premium to create semantic models that are reused throughout large, enterprise organizationsAuditoriumTue 15:30 — 60 min
SQL Server surprises
SQL Server contains a few surprises: transactions that don’t do what most people expects, NULL values that cause queries to spit out unexpected results, data type issues and many more. These are not bugs but features, nicely documented. But hey, who reads the manual?!
In this very interactive session you can learn some SQL Server surprises, and how to avoid them in your own T-SQL code. This 1 hour session can save you from many hours of debugging...AlcazarTue 15:30 — 60 min
The modern Cloud Data Warehouse - Snowflake on Azure
The era of cloud data warehousing has finally begun. Gone are the days where you had to provision expensive servers and spend days configuring and tweaking to get the technical details right. Using cloud infrastructure, we can skip past the technical set-up and start loading data immediately.
Snowflake is a vendor offering a native cloud data warehouse, hosted on Azure. In this session, we'll introduce you to this new technology and explain the important concepts to get you started. Walking out of this session, you'll have all the knowledge you need to embark a project with Snowflake.ScalaTue 15:30 — 60 min
Using AI to write conference session submissions
Deep learning has been used to write new Shakespearean sonnets, to imagine new delicious recipes, write hilarious Harry Potter novels and even come up with new names for beer! In this session we will understand, what is deep learning, what are neural nets, what are the steps required to build a deep learning model and look at some of the great examples mentioned.
We will then turn our new skills to the problem most speakers have! Writing session abstracts. Together we will develop a recursive neural net designed to generate new session abstracts, entirely based on previously submitted sessions to SQL Server conferences. Will we be able to produce a session you would have attended? Come along and fine out.Herten AasTue 15:30 — 60 min
Your database's journey to the cloud
Migrate your on-premise SQL Server databases to an platform hosted in Microsoft Azure.
Goal of the session is to inform/demonstrate the several database migration options when migrating your database to Microsot Azure.BegijnenzolderTue 15:30 — 60 min
3 ways to bring Power BI under Source Control
People love creating content in Power BI but how do we track changes or backup all these new reports? We’ll look at 3 different ways to bring your reports under source control and how to automate the process using Microsoft Flow, AzureDevOps and PowerShell.AuditoriumTue 16:45 — 60 min
Azure Key Vault, Azure Dev Ops and Data Factory how do these Azure Services work perfectly together!
Can we store our Connectionstrings or BlobStorageKeys or other Secretvalues somewhere else then in Azure Data Factory(ADF)? Yes you can! You can store these valuable secrets in Azure Key Vault(AKV).
But how can we achieve this in ADF? And finally how do we deploy our DataFactories in Azure Dev Ops to Test, Acceptance and Production environments with these Secrets ? Can this be setup dynamically?
During this session I will give answers on all of these questions. You will learn how to setup your Azure Key Vault, connect these secrets in ADF and finally deploy these secrets dynamically in Azure Dev Ops. As you can see a lot to talk about during this session.AlcazarTue 16:45 — 60 min
Crash Course on building and accelerating Deep Learning Solutions
Learn the end to end process of building deep learning solutions from experimentation to deployment. We will start by experimenting locally with different model architectures and hyperparameters using PyTorch. Then we'll show you how to use Azure Machine Learning service to train models at massive scale in the cloud and seamlessly deploy them into productionHerten AasTue 16:45 — 60 min
How to deploy SQL Server containers on Kubernetes in Azure
When Microsoft released SQL Server 2017 it allowed us to run SQL Server in docker containers. This radical change provides a wider and open platform that data professionals can choose how to deploy and run SQL Server. This session will demonstrate how containers are a game changer for deploying and managing SQL Server. It will also showcase how the Azure Kubernetes Service (AKS) is a scalable and highly available platform for SQL Server running in containers. Demonstrations will show how easy it is to create containers, deploy SQL Server in them and how to automate and manage your SQL Server containers using Kubernetes on the Azure platform. This session will show you the exciting future of SQL Server and you will walk away with knowledge of practical scenarios where SQL Server running in containers on Kubernetes may be the right deployment model for you.DijlezaalTue 16:45 — 60 min
Shape Maps in Power BI
A journey of a consultant answering a simple question (at least at first sight) from a client using Shape Maps. Join me on my quest to deliver a map made from scratch using various geo tools, open data and Power BI of course!
The various topics handled are:
•What are shape maps?
•How to build them from scratch (Full Demo)
•Enhance the performanceBegijnenzolderTue 16:45 — 60 min
The Battle Of Modern Data Architectures: Data Factory vs Databricks
Have you had that moment, where you arr in doubt which Azure service to use for your 'Modern Data Warehousing' solution? So many good options.. Like the Mapping/Wrangling Data Flows capabilities in Azure Data Factory, or the Delta feature in Databricks!
In this session we will take a look at the different services, compare them using real use-cases, and learn how to choose the best fit for each scenario.ScalaTue 16:45 — 60 min
Full session schedule will be released on July 3rd.