MLflow schema is only supported for scalar Spark types

[DCHTan_SG]

Hi all, I was following the guide:

Create models with Automated ML (preview)

When I reached the step to train a LightGBMClassifier model on the provided training data, I encountered the following warning and errors. May I know if anyone has encountered this and any way to overcome this?

"WARNING mlflow.data.spark_dataset: Failed to infer schema for Spark dataset. Exception: Unsupported Spark Type '<class 'pyspark.ml.linalg.VectorUDT'>', MLflow schema is only supported for scalar Spark types."

Cell In[104], line 9

6 with mlflow.start_run(run_name="default") as run:

7 # Create a LightGBMClassifier model with specified settings

8 model = LightGBMClassifier(objective="binary", featuresCol="features", labelCol="Exited")

----> 9 model = model.fit(train_data)

File ~/cluster-env/trident_env/lib/python3.10/site-packages/mlflow/utils/autologging_utils/safety.py:573, in safe_patch.<locals>.safe_patch_function(*args, **kwargs)

571 patch_function.call(call_original, *args, **kwargs)

572 else:

--> 573 patch_function(call_original, *args, **kwargs)

575 session.state = "succeeded"

577 try_log_autologging_event(

578 AutologgingEventLogger.get_logger().log_patch_function_success,

579 session,

(...)

583 kwargs, 584 )

[v-cboorla-msft]

Hi @DCHTan_SG 

Thanks for using Microsoft Fabric Community.

Autologging works by automatically capturing values of input parameters, output metrics, and output items of a machine learning model as it's being trained. This information is logged to your Microsoft Fabric workspace, where you can access and visualize it by using the MLflow APIs or the corresponding experiment and model items in your Microsoft Fabric workspace. This powerful feature allows you to log metrics, parameters, and models without the need for explicit log statements - all you need to do is call mlflow.autolog() before your training code. Auto-logging supports popular libraries such as Scikit-learn, XGBoost, PyTorch, Keras, Spark, and more.

For more details please refer : Autologging in Synapse Data Science - Microsoft Fabric | Microsoft Learn

Please refer this thread which might help you : How to do the Machine Model application in Fabric? - Microsoft Fabric Community 

If the issue still persists, please do let us know. Glad to help.

I hope this information helps. 

Thank you.

[DCHTan_SG]

Thank you for the prompt feedback.

I already have mlflow.autolog(disable=True) .

I am not sure how autologging is related to this error:

"WARNING mlflow.data.spark_dataset: Failed to infer schema for Spark dataset. Exception: Unsupported Spark Type '<class 'pyspark.ml.linalg.VectorUDT'>', MLflow schema is only supported for scalar Spark types."

Maybe a goood way is for you to go through the "Create AutoML trials in Fabric - Microsoft Fabric | Microsoft Learn", "How-to guides", "Run AutoML trials".

Alternatively, is there a location where I can download this notebook?

Thank you!

[v-cboorla-msft · Accepted solution]

Hi @DCHTan_SG 

Apologies for the inconvenience.

Please reach out to our support team to gain deeper insights and explore potential solutions. It's highly recommended that you reach out to our support team. Their expertise will be invaluable in suggesting the most appropriate approach.

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Thank you.