Passing many parameters from Java action to Oozie workflow

by Tabitha.Oepen on 2015-07-05

Passing many parameters from Java action to Oozie workflow

oozie’s ‘capture-output’ is a powerful method the pass dynamic configuration properties from action to action, but you may hit the maximum size limit quite fast.

When executing Java actions in an Oozie workflow, there are going to be cases where you want to have certain data to be passed from your action to the workflow itself, making it avalailable for use within the workflow. This can be easily achieved by adding “capture-output” element to your workflow action, which is an indicator for Oozie to capture the output of your action.

Here’s an example:

<action name="my-java-action">
		<java>
			<main-class>mypkg.MyJavaAction</main-class>
			<arg>yourargs</arg>
			<capture-output />
		</java>
	<ok to="mapreduce-action"/> 
    <error to="fail"/> 
</action>

The format of your Java action’s output needs to be in Java properties file format. So, in your action class, write your output to a file which path you must get from looking up oozie.action.output.properties in System.properties:

String oozieProp = System.getProperty("oozie.action.output.properties");
OutputStream os = null;
if (oozieProp != null) {
	File propFile = new File(oozieProp);
	Properties p = new Properties();
	p.setProperty("currentTimeMillis", "" + System.currentTimeMillis() );
        os = new FileOutputStream(propFile);
	p.store(os, "");
	os.close();
} else {
	throw new RuntimeException("oozie.action.output.properties System property not defined");
}

Check out an example on how to do this here.

Now do keep in mind that the property file shouldn’t exceed 2 KB in size. If your output is bigger than 2 KB, Oozie launcher will fail and you will get an exception looking (more or less) like this:

Output data exceeds its limit [2048]

Full stacktrace:

SERVER[SERVERNAME] USER[oozie] GROUP[supergroup] TOKEN[] APP[capture-output] JOB[0000009-150701190252776-oozie-oozi-W] ACTION[0000009-150701190252776-oozie-oozi-W@my-java-action] Launcher exception: Output data exceeds its limit [2048]
org.apache.oozie.action.hadoop.LauncherException: Output data exceeds its limit [2048]
    at org.apache.oozie.action.hadoop.LauncherMapper.getLocalFileContentStr(LauncherMapper.java:379)
    at org.apache.oozie.action.hadoop.LauncherMapper.handleActionData(LauncherMapper.java:355)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:267)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask
    (LocalContainerLauncher.java:370)
    at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask
    (LocalContainerLauncher.java:295)
    at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200
    (LocalContainerLauncher.java:181)
    at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run
    (LocalContainerLauncher.java:224)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

The good news is, you can easily increase the maximum size of the property file, by simply adding the below into oozie-site.xml file. You could of course adjust the value to your need.

<property>
 <name>oozie.action.max.output.data</name>
 <value>8192</value>
 </property>

Or if you’re using Cloudera Manager, you can adjust the “Oozie Server Advanced Configuration Snippet (Safety Valve) for oozie-site.xml” in Oozie configuration.

oozie-action-max-output-data-configuration-cloudera-manager-5

Make sure to restart your oozie instance after this configuration change.

Tags: