How to map Google Cloud Dataflow System.err writes to warning level Google Cloud logging events?

Question

I am seeing that writes to System.err map to an Error level logging event in GCP Logging. Is there any way to map these to a different logging level, such as Warning level?

In our code we always write to a SLF4J Logger but we are using 3rd party libraries that unfortunately write to System.err as well as throw exceptions. Since we have the exception, we don’t want to log the System.err as an error.

Update: It looks like Dataflow used to have a class com.google.cloud.dataflow.sdk.runners.worker.logging.DataflowWorkerLoggingInitializer that would allow me to configure stdout and stderr how I desire. Does anyone know why it was removed?

I would have configured it using the command line option:

--workerLogLevelOverrides={\"System.err":\"WARN\",\"System.out\":\"DEBUG\"}

Or the following code:

    /**
 * Maps the "System.err" and "System.out" loggers to desired levels.
 * Note: the "System.err" and "System.out" logger are not currently part of the specification as of v1.5.1, 
 * so this is dependent on the current implementation of com.google.cloud.dataflow.sdk.runners.worker.logging.DataflowWorkerLoggingInitializer
 * which does configure those loggers.
 * @param options
 */
public static void setupSystemOutAndErrLogging(DataflowWorkerLoggingOptions options) {
    WorkerLogLevelOverrides overrides = options.getWorkerLogLevelOverrides();
    if ( overrides == null ) {
        overrides = new WorkerLogLevelOverrides();
    }
    options.setWorkerLogLevelOverrides(
            overrides.addOverrideForName("System.err", Level.WARN)
                    .addOverrideForName("System.out", Level.DEBUG));
}

The `workerLogLevelOverrides` control the filtering on a specific logger not the level that those logs are treated at. Because `System.err` is logged at `ERROR` and `OFF` is not currently an option, it is impossible to turn `System.err` off. We'll work on a fix and let you know when it should work. — Ben Chambers, May 06 '16 at 16:00
@BenChambers Oh yes, I see now where I was wrong. I was wrongly thinking workerLogLevelOverrides was mapping instead of filtering. I know better. But of course, mapping to a specific log level is what I desire, so I'll be interested in your solution. Thanks! — successhawk, May 06 '16 at 17:43

Ben Chambers · Answer 1 · 2016-05-02T17:59:53.890

1

Dataflow redirects stdout to a java.util.logging.Logger named System.out and stderr to System.err. You can configure java util logging to not log anything on these loggers, which will disable this.

The easiest way to do this is through the use of the workerLogLevelOverrides pipeline option. For example: --workerLogLevelOverrides={"System.out":"OFF", "System.err":"OFF"}.

Edit 1: Use workerLogLevelOverrides option rather than code to configure the loggers.

edited May 02 '16 at 17:59

answered Apr 27 '16 at 20:50

Ben Chambers

6,070
11
16

I tried this in just my "main" method and it didn't work. Got any ideas? I updated my question with something interesting I found. Know anything about that? – successhawk Apr 30 '16 at 19:10
Putting it in the `main` method won't work since it isn't run on the workers. [`DataflowWorkerLoggingOptions`](https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/36246223f41c3ba78a46161b27409dc75812cdf8/sdk/src/main/java/com/google/cloud/dataflow/sdk/options/DataflowWorkerLoggingOptions.java) still exists, and `--workerLogLevelOverrides` should still be available as an option. – Ben Chambers May 02 '16 at 17:35
The DataflowWorkerLoggingOptions specify the contract of what is accepted. Its currently undocumented about stderr/stdout but is accepted by Dataflow. – Lukasz Cwik May 02 '16 at 18:15
@BenChambers I don't think you read my update, because your updated answer is exactly what I put in my updated question 2 days earlier. That is what I would desire, but I tested it and it didn't work. Are you seeing different results? Also, note that I said "desire" instead of "expect", because as Lukasz pointed out, it is not part of the contract. Feature request? – successhawk May 05 '16 at 20:44

score 1 · Answer 2 · answered Jun 16 '16 at 16:57

This functionality is now available in Google Dataflow when using the Beam SDK. See commit 874e199 and DataflowWorkerLoggingOptions. With this change you have some additional control in terms of System.out and System.err logging:

Change the level of log messages emitted from System.out / System.err log, via setWorkerSystemOutMessageLevel() and setWorkerSystemErrMessageLevel().
Change the global log level filtering via setDefaultWorkerLogLevel(). This was previously available but now applies to System.out and System.err log messages.
Change the logger level overrides for System.out and System.err, via setWorkerLogLevelOverrides() with System.out and System.err logger names (as described in the original question).

Excellent! Thanks. I eyeballed the commit and it looks like exactly what I wanted. I'm sure I (and others) will benefit from this in the future. — successhawk, Jun 17 '16 at 04:12

score -1 · Answer 3 · edited May 23 '17 at 10:33

-1

Dataflow doesn't currently expose a way to control the logging level of stderr. However, I think it is possible to coax SLF4J into doing what you want; take a look at Redirect System.out and System.err to slf4j

edited May 23 '17 at 10:33

Community

1
1

answered Apr 26 '16 at 23:41

danielm

3,000
10
15

I wonder why this was downvoted with no comment. It looks like Dataflow used to support it. Know anything about that (see my updated question)? – successhawk Apr 30 '16 at 19:13
2

Downvoted because Dataflow redirects stdout, stderr and slf4j directly to `java.util.logging`, so configuring slf4j doesn't help. – Ben Chambers May 06 '16 at 05:54

How to map Google Cloud Dataflow System.err writes to warning level Google Cloud logging events?

3 Answers3