Advent of 2023, Day 13 – Creating ML Model with Microsoft Fabric

This article is originally published at https://tomaztsql.wordpress.com

In this Microsoft Fabric series:

1. Dec 01: What is Microsoft Fabric?
2. Dec 02: Getting started with Microsoft Fabric
3. Dec 03: What is lakehouse in Fabric?
4. Dec 04: Delta lake and delta tables in Microsoft Fabric
5. Dec 05: Getting data into lakehouse
6. Dec 06: SQL Analytics endpoint
7. Dec 07: SQL commands in SQL Analytics endpoint
8. Dec 08: Using Lakehouse REST API
9. Dec 09: Building custom environments
10. Dec 10: Creating Job Spark definition
11. Dec 11: Starting data science with Microsoft Fabric
12. Dec 12: Creating data science experiments with Microsoft Fabric

Yesterday, we created an MLFlow experiment and today we will look into creating an ML model.

Create a new ML model, and give it a name. You will have the option to select the following:

Since we have created an experiment yesterday, let’s open it. And you will be greeted with the familiar interface.

Of course, you can always explore the model, even download the model and all the artefacts for building a new model:

Protip: Both experiments and the ML model version look similar, and you can intuitively switch between both of them. But do not get confused, as the ML Model version applies the best-selected model from the experiment and can be used for inference.

In the view runs, you can compare different runs and check the metrics, logs and model performance.

Pretty handy tools. As said, With the model, you can also apply the model to start making the predictions. You can create a new ML Model version and you will also be able to choose the version of the model and apply the version!

And use MLFlow to read the model, make inference on delta table and store results in a new delta table.

import mlflow
from synapse.ml.predict import MLFlowTransformer
    
df = spark.read.format("delta").load("Advent2023/Tables/iris_data")
    
model = MLFlowTransformer(
    inputCols=["Sepal.Length", "Sepal.Width", "Petal.Length", "Petal.Width"], 
    outputCol="predictions", 
    modelName="Advent2023_MLmodel",
    modelVersion="Version 1")
df = model.transform(df)

This will make the model persistent and prediction available in the table.

Tomorrow we will start looking into the data warehouse.

Complete set of code, documents, notebooks, and all of the materials will be available at the Github repository: https://github.com/tomaztk/Microsoft-Fabric

Happy Advent of 2023!

Thanks for visiting r-craft.org
This article is originally published at https://tomaztsql.wordpress.com
Please visit source website for post related comments.