how to use dynamic allocation in a oozie spark action on CDH5
using spark’s dynamic allocation feature in a oozie spark action can be a tricky.
spark oozie action jobs not showing up on spark history server
If you execute spark jobs within an oozie workflow using a action node on a Cloudera CDH5 cluster, your job may not show up on your spark history server. Even if you configured all these things using the cloudera manager, your history server may only lists jobs started on the commandline using spark-submit.
fixing spark classpath issues on CDH5 accessing Accumulo 1.7.2
We experienced some strange NoSuchMethorError while migrating a Accumulo based application from 1.6.0 to 1.7.2 running on CDH5. A couple of code changes where necessary moving from 1.6.0 to 1.7.2, but these were pretty straightforward (members visibility changed, some getters were introduced). Everything compiled fine, but when we executed the spark application on the cluster we got an exception that was pointing directly to a line we changed during the migration:
Increasing mapreduce.job.counters.max on CDH5 YARN (MR2)
How to increase mapreduce.job.counters.max on YARN (MR2) for HUE / HIVE / OOZIE.