Tag - oozie

spark hadoop pyspark oozie hue

Run Multiple Python Scripts PySpark Application with yarn-cluster Mode

288   0   about 2 months ago

When submitting Spark applications to YARN cluster, two deploy modes can be used: client and cluster. For client mode (default), Spark driver runs on the machine that the Spark application was submitted while for cluster mode, the driver runs on a random node in a cluster. On this page, I am goin...

View detail
spark hadoop yarn oozie

Diagnostics: Container is running beyond physical memory limits

273   0   about 4 months ago

Scenario Recently I created an Oozie workflow which contains one Spark action. The Spark action master is yarn and deploy mode is cluster. Each time when the job runs about 30 minutes, the application fails with errors like the following: Application applicatio...

View detail