How to pass arguments to application.java class in Apache Apex?

Question

How can I pass arguments to Application.java class ? Like how we pass (String[] args) in main method in java ?

Couldn’t find any references. Can someone please share knowledge on this ?

Thanks a lot.

Answer 1

You can pass arguments as Configuration. This configuration will be passed as an argument to populateDAG() method in Application.java.

Configuration is org.apache.hadoop.conf.Configuration. You can specify it as xml. For xml syntax please refer to https://hadoop.apache.org/docs/r2.6.1/api/org/apache/hadoop/conf/Configuration.html.

There are different ways in which properties can be specified:

1. ~/.dt/dt-site.xml: By default apex cli will look for this file (~ is your home directory). You should use this file for the properties which are common to all the applications in your environment.

2. -conf option on apex cli: launch command on apex cli provides -conf option to specify properties. You need to specify the path for the configuration xml. You should use this file for the properties which are specific to a particular application or specific to this launch of the application.

3. -Dproperty-name=value: launch command on apex cli provides -D option to specify properties. You can specify multiple properties like -Dproperty-name1=value1 -Dproperty-name2=value2 etc.

Syntax for specifying operator properties is as follows:

<property>
      <name>dt.application.applicationName.operator.OperatorIdentifier.prop.property-name1</name>
      <value>value1</value>
</property>  

<property>
      <name>dt.application.applicationName.operator.OperatorIdentifier.prop.property-name2</name>
      <value>value2</value>
</property>  

OperatorIdentifier/name is the String identifier you use for dag.addOperator() in the populateDAG().

Answer 2

Since the accepted answer only describes in detail how to specify operator properties or with using ~/.dt/dt-site.xml or Apex CLI, here is one simple example how to specify properties that you can use in the Application.java (i.e. in the populateDag() Method).

Specify the properties in XML in the file: resources/META-INF/properties.xml like this:

<configuration>
  <!-- Specify some arbitrary app configs -->
  <property>
    <name>prop1</name>
    <value>val1</value>
  </property>
  <property>
    <name>prop2</name>
    <value>val2</value>
  </property>

  <!-- Specify some other configs, i.e. operator properties -->
  <property>
    <name>dt.application.applicationName.operator.OperatorIdentifier.prop.property-name1</name>
    <value>value1</value>
  </property>

</configuration>

Then in your Application.java you can get the properties like this:

public class Application implements StreamingApplication
{

  @Override
  public void populateDAG(DAG dag, Configuration conf)
  {
    String prop1 = conf.get("prop1");
    String prop2 = conf.get("prop2", "defaultValue"); // when the property is not set in XML, a default can be used

    // populate your DAG here ...
  }
}

Instead of using operator properties, this approach can be very useful if you need properties in operators before the setup() method is executed, for instance in the constructor of an operator.