Hi @AmitChandak, I have a SQL server stored procedure which updates, deletes and merges data into a table , how do I convert the stored procedure to pyspark job, is it possible to update a table in fabric using pyspark?, could you please make a video on this topic
You can update Lakehouse tables using PySpark or Spark SQL. In case you want to update warehouse tables, you can use SQL procedures. Managing Microsoft Fabric Lakehouse using Spark SQL: Data Manipulation Insert, Update, Delete, Alter: ruclips.net/video/PunKbz4iCEg/видео.html To Convert SQL procedure to PySpark code, you can take help from ChatGPT. I will try to come up video soon.
I created a Linux VM that can interact with windows and ran Spark on that and saved the Delta parquet on Lakehouse tables using OneLake explorer. But it did not work as expected. It was not able to save all log files.
@@AmitChandak I think we should try to config the spark build session to be able to connect to one lake so we can execute the spark save command to save data frame as delta directly I tried many configurations for spark sessions to connect to the one lake but did not work
Hi @AmitChandak, I have a SQL server stored procedure which updates, deletes and merges data into a table , how do I convert the stored procedure to pyspark job, is it possible to update a table in fabric using pyspark?, could you please make a video on this topic
You can update Lakehouse tables using PySpark or Spark SQL. In case you want to update warehouse tables, you can use SQL procedures.
Managing Microsoft Fabric Lakehouse using Spark SQL: Data Manipulation Insert, Update, Delete, Alter: ruclips.net/video/PunKbz4iCEg/видео.html
To Convert SQL procedure to PySpark code, you can take help from ChatGPT. I will try to come up video soon.
Hi,
Can you make a video about how to use on-premise spark (PySpark) to load a delta table on a Lakehouse ?
I created a Linux VM that can interact with windows and ran Spark on that and saved the Delta parquet on Lakehouse tables using OneLake explorer. But it did not work as expected. It was not able to save all log files.
@@AmitChandak I think we should try to config the spark build session to be able to connect to one lake so we can execute the spark save command to save data frame as delta directly
I tried many configurations for spark sessions to connect to the one lake but did not work