Scaling out spark applicaton

[sparkie]

Hello,

I am executing a long running Spark appliation from a notebook. In SparkUI I see that not all possible resources are used, so I would like to repartition the data to the maximum available cores. However, I don't know how to programmatically obtain the maximum available number of executors.
Please advise.

[v-shex-msft · Accepted solution]

HI @sparkie,

For use rest api to getting spark settings, you can refer to the following document:

Workspace Settings - Get Spark Settings - REST API (Spark) | Microsoft Learn

If the above not suitable for your requirement, you can also check the spark computer and pool setting at the 'Workspace settings':

Create custom Apache Spark pools in Fabric - Microsoft Fabric | Microsoft Learn

Regards,

Xiaoxin Sheng