[We wanted to inform you that Databricks Academy will be undergoing upgrade maintenance to enhance platform performance. As a result, it will be inaccessible during this period: 03:00 EST - 07:00 EST, July 13 (Saturday).] Join the Databricks Learnin...
- 37562 Views
- 77 replies
- 18 kudos
06-03-2024
We are enhancing the Community login process to provide a more secure and streamlined experience for all users. To avoid losing access to your Community account, please update your primary and secondary email addresses now (refer to “Action Required”...
- 1945 Views
- 0 replies
- 3 kudos
2 weeks ago
Your feedback is crucial to us and directly influences how we innovate and improve our customer experience. To share your success and feedback with Databricks, please visit this link. The first 20 people to submit a completed survey response will ...
- 457 Views
- 1 replies
- 4 kudos
Tuesday
Community Activity
- 762 Views
- 2 replies
- 0 kudos
Resolved! Databricks Spot Instance: Completion Guarantee
Databricks allows to use spot instances for worker nodes. I consider to use them for interactive clusters. Do I have a gurantee that code will be completed without any errors even if spot instances are evicted? I would accept execution delays but no ...
- 762 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
You could explore their "SPOT_WITH_FALLBAK" feature. If you don't want your jobs to fail because of eviction but this currently is not supported with interactive clusters. Hoping that they may extend this to all compute options soonCreate a pipeline ...
- 0 kudos
- 17 Views
- 1 replies
- 0 kudos
Serverless feature audit in data engg.
As recently announced in the summit that notebooks, jobs, workflows will run in serverless mode, how do we track/debug the compute cluster metrics in this case especially when there are performance issues while running jobs/workflows.
- 17 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Databricks is planning to enable some system tables to capture some of these metrics and same can be leveraged for troubleshooting as starting point is my view
- 0 kudos
- 8933 Views
- 9 replies
- 9 kudos
Resolved! Installing libraries on job clusters
Simple question : what is the way to go to install libraries on job clusters ? There does not seem to be a "Libraries" tab on the UI as opposed to regular clusters. Does it mean that the only option is to use init scripts ?
- 8933 Views
- 9 replies
- 9 kudos
Latest Reply
- 9 kudos
You may want to copy required libs to a volume and load it during cluster setup to avoid downloading the libs for every run.
- 9 kudos
- 3293 Views
- 4 replies
- 1 kudos
Azure Entra SSO Error: Your user has not been registered
I have set up SSO within databricks and automatic user provisioning with Azure Entra and confirmed it is working for all users. However 1 user is presented with this when signing in. The user is in the enterprise app within Azure Entra and the user i...
- 3293 Views
- 4 replies
- 1 kudos
Latest Reply
- 1 kudos
Try asking them to launch ADB workspace from Azure Portal and see if it works...
- 1 kudos
- 4 Views
- 0 replies
- 0 kudos
Denodo with Databricks
Hi Team,How can I make a SQL Endpoint web api in Databricks that can be used in denodo?Regards,Janga
- 4 Views
- 0 replies
- 0 kudos
- 66 Views
- 1 replies
- 1 kudos
Databricks data engineer associate exam Suspended
Hi Team,I recently had a disappointing experience while attempting my first Data bricks certification exam. During the exam, I was abruptly directed to Proctor Support. The proctor asked me to show my desk and the room I was in. I complied by showing...
- 66 Views
- 1 replies
- 1 kudos
Latest Reply
- 1 kudos
@Cert-Team @Certificate Please help
- 1 kudos
- 16 Views
- 0 replies
- 0 kudos
Receiving Null values from Eventhub streaming.
Hi, I am new to PySpark, and facing an issue while consuming data from the Azure eventhub. I am unable to deserialize the consumed data. I see only null values upon deserializing data using the schema. Please find the below schema, eventhub message, ...
- 16 Views
- 0 replies
- 0 kudos
- 126 Views
- 6 replies
- 1 kudos
Py4JJavaError: An error occurred while calling o465.coun
The following error occured when running .count() on a big sparkDF. Py4JJavaError: An error occurred while calling o465.count. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 6 in stage 3.0 failed 4 times, most recent failur...
- 126 Views
- 6 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @sinclair, Try finding out the null values in your data. You can use df.select([col for col in df.columns if df[col].isNull().sum() > 0]).show(). Once you've caught them, you have two options: either drop them using df.dropna(), or fill them with...
- 1 kudos
- 49 Views
- 2 replies
- 1 kudos
How to get the JSON definition - "CREATE part" for a job using JOB ID or JOB Name
I want to get the JSON definition of the "create part" of the job. I have the job id and job name. I am using databricks notebook for this. I can the get the "GET" json api definition but not able able to get the "CREATE" part json definition which I...
- 49 Views
- 2 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @InquisitiveGeek, To extract the "CREATE" part from the full JSON definition of a Databricks job, you can use the Databricks Jobs API to retrieve the job definition and then parse the relevant sections.
- 1 kudos
- 144 Views
- 3 replies
- 2 kudos
Accessing Databricks Lab Materials
Hello Databrick Support,Why is it so complex to get access to the Lab Materials after subscription. Each time I tried accessing this materials, it show a message stating "The Course has ended and no longer accessible". If this message is accurate, wh...
- 144 Views
- 3 replies
- 2 kudos
Latest Reply
- 2 kudos
Hi Databricks Support team,I am facing the same issue (This course has ended and no longer accessible) while accessing the labs " SP Lab Environment: Generative AI Solution Development LTI" for course Generative AI Engineering with Databricks. I have...
- 2 kudos
- 262 Views
- 6 replies
- 2 kudos
Unity Catalog
Hello friends! Do we have any certifications for Unity Catalog in Databricks?
- 262 Views
- 6 replies
- 2 kudos
Latest Reply
- 2 kudos
@Techelligence You can find the Platform Administrator path from here: https://customer-academy.databricks.com//lms/index.php?r=coursepath/deeplink&id_path=207&hash=1003b93351e6a5abe05aa342f12b1458f6ac2799&generated_by=714319There are many other cert...
- 2 kudos
- 12 Views
- 0 replies
- 0 kudos
Hadoop Hive migration to Databricks
Hi,Can you let me know what are the challenges and how to mitigate while migrating Hive objects to Databricks. I could not get any information on this. Can you please provide.
- 12 Views
- 0 replies
- 0 kudos
- 77 Views
- 2 replies
- 2 kudos
Resolved! How to pass a dynamic query to source server from databricks
I have this usecase wherein i am supposed to pass a dynamic query to get data from source I have tried the query option but its giving error SparkConnectGrpcException: (com.microsoft.sqlserver.jdbc.SQLServerException) Incorrect syntax near the keywor...
- 77 Views
- 2 replies
- 2 kudos
Latest Reply
- 2 kudos
Hi thanks for your reply,I have used foreign catalouge to fetch required data from information schema then i am creating the dynamic query in databricks and then passing in query this is working for me! @Kaniz_Fatma
- 2 kudos
- 60 Views
- 1 replies
- 0 kudos
How to Optimize Delta Lake Performance for Large-Scale Data Ingestion?
Hi everyone,I'm currently working on a project that involves large-scale data ingestion into Delta Lake on Databricks. While the ingestion process is functioning, I've noticed performance bottlenecks, especially with increasing data volumes. Could yo...
- 60 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Syleena23 , I believe this Comprehensive Guide to Optimize Databricks, Spark and Delta Lake Workloads provides a lot of answers to these questions and can be a great performance tuning and optimization guide in general. Please take a look. Thank ...
- 0 kudos
- 45 Views
- 2 replies
- 0 kudos
How can I increase the hard capacity of the master node?
I'm not sure if this is the right place to post my question. If not, please let me know where I should post my question. I want to download large files from the web from Databricks' master(driver) node. For example, I fetch a file over 150GB via API ...
- 45 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @himanmon,If you 100% sure that you can't download this file to storage account configured with unity catalog and you want it directly on driver node local storage, then why can't you just increase local disk space by choosing a larger instance ty...
- 0 kudos
Latest from our Blog
Enhancing the new PySpark Custom Data Sources Streaming API: Adding Progress Tracking Capability
A new PySpark Custom Data Sources API was introduced at DAIS 2024 which simplifies the integration with custom data sources in Apache Spark. Imagine seamlessly streaming incremental data from any API ...
Multi-table operations made simple with DiscoverX
Multi-Table Operations Made Simple In the ever-evolving landscape of data science and engineering, the ability to efficiently manage and manipulate data across multiple tables and databases is paramou...
Queries for Cost Attribution using System Tables
Organizations have expressed the need to see trends across their Databricks Accounts and drill down into Workspaces, SKUs, tags, and users. System Tables provide this visibility with little to no setu...
How not to build a demo
When making videos on new features announced, part of the process is researching what the feature is, thinking how best to demo it and then making the demo itself. But here’s the twist: by definition,...