Increasing mapreduce.job.counters.max on CDH5 YARN (MR2)

by Thomas Memenga on 2014-11-14

Increasing mapreduce.job.counters.max on CDH5 YARN (MR2)

How to increase mapreduce.job.counters.max on YARN (MR2) for HUE / HIVE / OOZIE.

If your mapreduce job leverages more than 120 counters, you need to adjust

mapreduce.job.counters.max

on an MR1 cluster to match your needs.

If you want to run your job on CDH5 YARN (MR2), you will still need to set this value to a higher value. But you also need to add the following snippet into Service-Wide / Advanced - YARN Service MapReduce Advanced Configuration Snippet (Safety Valve):

<property>
 <name>mapreduce.job.counters.max</name>
 <value>yournewvaluehere</value>
</property>

Otherwise your job will finish successfully, but HUE / OOZIE will not be able to query the job’s state and will list it as failed.

Tags: