I have a question on threadpool in spark. When we use threadpool executor, all threads are running on same node? Like only on driver node? Or Will it utilize all the workers in the cluster? Can you please clarify ?
When you use threadpool executor, all threads are running on the same node, might run out of memory as well. o tackle your problem, can you try running each notebook as a separate process and create a Spark Context within that process. Please try using "subprocess" module in Python to spawn a new process for each notebook.
![I.N "HALLUCINATION" | [Stray Kids : SKZ-PLAYER]](http://i.ytimg.com/vi/n5B5q1Hwt_U/mqdefault.jpg)
I have a question on threadpool in spark. When we use threadpool executor, all threads are running on same node? Like only on driver node? Or Will it utilize all the workers in the cluster? Can you please clarify ?
When you use threadpool executor, all threads are running on the same node, might run out of memory as well. o tackle your problem, can you try running each notebook as a separate process and create a Spark Context within that process. Please try using "subprocess" module in Python to spawn a new process for each notebook.