How to obtain data of filters of the convolution layers of the CNN network in DL4J to draw the activation map?

Question

How to get filters data from the layer objects for the configuration and model like this?

  ComputationGraphConfiguration config =
        new NeuralNetConfiguration.Builder()
            .seed(seed)
            .gradientNormalization(GradientNormalization.RenormalizeL2PerLayer)
            .l2(1e-3)
            .updater(new Adam(1e-3))
            .weightInit(WeightInit.XAVIER_UNIFORM)
            .graphBuilder()
            .addInputs("trainFeatures")
            .setInputTypes(InputType.convolutional(60, 200, 3))
            .setOutputs("out1", "out2", "out3", "out4", "out5", "out6")
            .addLayer(
                "cnn1",
                new ConvolutionLayer.Builder(new int[] {5, 5}, new int[] {1, 1}, new int[] {0, 0})
                    .nIn(3)
                    .nOut(48)
                    .activation(Activation.RELU)
                    .build(),
                "trainFeatures")
            .addLayer(
                "maxpool1",
                new SubsamplingLayer.Builder(
                        PoolingType.MAX, new int[] {2, 2}, new int[] {2, 2}, new int[] {0, 0})
                    .build(),
                "cnn1")
            .addLayer(
                "cnn2",
                new ConvolutionLayer.Builder(new int[] {5, 5}, new int[] {1, 1}, new int[] {0, 0})
                    .nOut(64)
                    .activation(Activation.RELU)
                    .build(),
                "maxpool1")
            .addLayer(
                "maxpool2",
                new SubsamplingLayer.Builder(
                        PoolingType.MAX, new int[] {2, 1}, new int[] {2, 1}, new int[] {0, 0})
                    .build(),
                "cnn2")
            .addLayer(
                "cnn3",
                new ConvolutionLayer.Builder(new int[] {3, 3}, new int[] {1, 1}, new int[] {0, 0})
                    .nOut(128)
                    .activation(Activation.RELU)
                    .build(),
                "maxpool2")
            .addLayer(
                "maxpool3",
                new SubsamplingLayer.Builder(
                        PoolingType.MAX, new int[] {2, 2}, new int[] {2, 2}, new int[] {0, 0})
                    .build(),
                "cnn3")
            .addLayer(
                "cnn4",
                new ConvolutionLayer.Builder(new int[] {4, 4}, new int[] {1, 1}, new int[] {0, 0})
                    .nOut(256)
                    .activation(Activation.RELU)
                    .build(),
                "maxpool3")
            .addLayer(
                "maxpool4",
                new SubsamplingLayer.Builder(
                        PoolingType.MAX, new int[] {2, 2}, new int[] {2, 2}, new int[] {0, 0})
                    .build(),
                "cnn4")
            .addLayer("ffn0", new DenseLayer.Builder().nOut(3072).build(), "maxpool4")
            .addLayer("ffn1", new DenseLayer.Builder().nOut(3072).build(), "ffn0")
            .addLayer(
                "out1",
                new OutputLayer.Builder(LossFunctions.LossFunction.NEGATIVELOGLIKELIHOOD)
                    //.nOut(36)
                        .nOut(10)
                    .activation(Activation.SOFTMAX)
                    .build(),
                "ffn1")
            .addLayer(
                "out2",
                new OutputLayer.Builder(LossFunctions.LossFunction.NEGATIVELOGLIKELIHOOD)
                    //.nOut(36)
                        .nOut(10)
                    .activation(Activation.SOFTMAX)
                    .build(),
                "ffn1")
            .addLayer(
                "out3",
                new OutputLayer.Builder(LossFunctions.LossFunction.NEGATIVELOGLIKELIHOOD)
                    //.nOut(36)
                        .nOut(10)
                    .activation(Activation.SOFTMAX)
                    .build(),
                "ffn1")
            .addLayer(
                "out4",
                new OutputLayer.Builder(LossFunctions.LossFunction.NEGATIVELOGLIKELIHOOD)
                    //.nOut(36)
                        .nOut(10)
                    .activation(Activation.SOFTMAX)
                    .build(),
                "ffn1")
            .addLayer(
                "out5",
                new OutputLayer.Builder(LossFunctions.LossFunction.NEGATIVELOGLIKELIHOOD)
                    //.nOut(36)
                        .nOut(10)
                    .activation(Activation.SOFTMAX)
                    .build(),
                "ffn1").addLayer(
                "out6",
                new OutputLayer.Builder(LossFunctions.LossFunction.NEGATIVELOGLIKELIHOOD)
                        //.nOut(36)
                        .nOut(10)
                        .activation(Activation.SOFTMAX)
                        .build(),
                "ffn1")

            //.pretrain(false)
            //.backprop(true)
            .build();

I mean the NDArray (or what?) of the convolutional layer activations after the model is got trained that is used for drawing activation maps like that:

It is not clear for me what kind of Layer's API returns the 2D data for building that.

score 3 · Accepted Answer · answered Apr 15 '20 at 19:34

3

If you are using the DL4J ui module, you can get those visualizations simply by adding the ConvolutionalIterationListener as another listener for your model.

If you don't want to use the listener, you can at least check out its code to see how you can create those visualizations on your own.

answered Apr 15 '20 at 19:34

Paul Dubs

798
4
8

thanks. Am I right it isn't mentioned somewhere in the basic documentation? Just to clarify: I need not special action for enabling this in UI Server (web app on port 9000) other than adding yet one listener to the model – Eljah Apr 15 '20 at 19:46
1

Yes, the documentation is unfortunately somewhat stale as of this moment. We are working on improving it. – Paul Dubs Apr 16 '20 at 08:12
So there is no way to get them from the `ComputationGraph` model after loading it? Only through a listener? – Matthieu Jul 22 '21 at 15:05
Those patterns depend on the input. So you need to run them through the model to get something. The listener then produces the patterns. – Paul Dubs Jul 23 '21 at 14:49
@PaulDubs I kind of found a way that seems to give me interesting data (see my [answer](https://stackoverflow.com/a/68547168/1098603) for the code). Do you think it's corresponds to those filter coefficients (I'm beginning with dl4j...)? – Matthieu Jul 27 '21 at 14:56

Matthieu · Answer 2 · 2021-07-28T14:17:12.293

You can go through each layer in your graph and vertices to isolate all Type.CONVOLUTIONAL layers, then print the filter bias and weights.

Tough, as indicated by @PaulDubs, that would give you the weights and not the activation: you'll have to apply the filters to an image to see the activation for that image specifically.

ComputationGraph model = ComputationGraph.load(new File("/data/deepl/moyens11.zip"), false);
GraphVertex[] vertices = model.getVertices();
for (Layer layer : model.getLayers()) {
    if (layer.type() != Type.CONVOLUTIONAL) {
        continue;
    }
    
    String name = Arrays.stream(vertices)
        .filter(v -> v.getLayer() == layer)
        .map(GraphVertex::getVertexName)
        .findAny().orElse("???");
    System.out.println("Layer #"+layer.getIndex()+" "+name+":");
    Map<String,INDArray> params = layer.paramTable();
    INDArray coeffsW = params.get("W"); // Weights
    INDArray coeffsB = params.get("b"); // Bias
    long[] dimsW = coeffsW.shape(); // [<nOut>, <nIn>, <convY>, <convX>] e.g. [32, 1, 3, 3]
    long[] dimsB = coeffsB.shape(); // [1, <nOut>] e.g. [1, 32]
    for (int iOut = 0; iOut < dimsW[0]; iOut++) {
        System.out.println("      Output "+iOut+": "+dimsW[2]+"x"+dimsW[3]);
        for (int iIn = 0; iIn < dimsW[1]; iIn++) {
            System.out.println("         Filter "+iIn+", bias "+coeffsB.getFloat(0, iOut));
            for (int y = 0; y < dimsW[3]; y++) {
                System.out.print("            ");
                for (int x = 0; x < dimsW[2]; x++) {
                    System.out.printf("%15g", coeffsW.getFloat(iOut, iIn, y, x));
                }
                System.out.println();
            }
        }
    }
}

Would give something like:

Layer #2 cnn1:
   Output 0: 3x3
      Filter 0, bias 0.034268413
               -0.181560      -0.107202      -0.143127
               -0.105426      0.0311019      -0.104243
               -0.114821      -0.101115      -0.105325
   Output 1: 3x3
      Filter 0, bias 0.023474146
             -0.00397262      0.0207015       0.253023
               -0.171297     -0.0244920      0.0446173
               -0.103351      -0.107709    -0.00905741
...

You are just printing the weights here. However, the question is about visualizing the feature maps, and that can by definition only happen when you have an input. — Paul Dubs, Jul 27 '21 at 19:24
@PaulDubs thanks, it makes more sense, I forgot about the activation part. — Matthieu, Jul 28 '21 at 14:15

How to obtain data of filters of the convolution layers of the CNN network in DL4J to draw the activation map?

2 Answers2