Resolve Databricks Processing Crash

I’m running a real-time, production workload on Azure Databricks that ingests structured data from CSV/SQL sources and processes it in Spark notebooks. The jobs run fine until the transformation stage, where they suddenly throw runtime errors and the cluster crashes... (Budget: ₹600 - ₹1500 INR, Jobs: Azure, Cloud Computing, Data Processing, Documentation, Performance Tuning, PySpark, Scala, Spark)