Spark contexts sometimes die and don't come back

[ehiggs]

Spark contexts in the ipython notebook sometimes die. These can be restarted by closing the notebook and killing the kernel and then reopening the notebook. Sometimes, however it doesn't work. This might go away w/ spark 1.6.

[boegel]

Spark 1.6.1 has a bunch of bug fixes, maybe it could remedy this problem... http://spark.apache.org/releases/spark-release-1-6-1.html

[boegel]

Some more information I pulled out of @ehiggs:

• Spark has driver and worker processes; if the driver runs out of memory, it goes down
• issue occurs when driver is pushing the memory limits due to large random forests
• upstream issue: https://issues.apache.org/jira/browse/SPARK-13434
• see also Spark random forest issues szilard/benchm-ml#19