9

I got myself very surprised after I wrote this small code to compare .NET 4.5 and Java 8 performance in my computer:

class ArrayTest
{
    public int[][] jagged;

    public ArrayTest(int width, int height)
    {
        Height = height;
        Width = width;
        Random rng = new Random();
        jagged = new int[width][];
        for (int i = 0; i < height; i++)
        {
            jagged[i] = new int[width];
            for (int j = 0; j < jagged[i][j]; j++)
            {
                jagged[i][j] = rng.Next(2048);
            }
        }
    }
    public int this[int i, int j]
    {
        get
        {
            return jagged[i][j];
        }
        set
        {
            jagged[i][j] = value;
        }
    }

    public void DoMath(ArrayTest a)
    {
        for (int i = 0; i < Height; i++)
        {
            for (int j = 0; j < Width; j++)
            {
                this[i, j] *= a[i, j];
            }
        }
    }

    public int Height { get; private set; }

    public int Width { get; private set; }
}



class Program
{
    static void Main(string[] args)
    {
        Random rng = new Random();
        const int loop = 10;
        int width = 10000,
            height = 10000;

        ArrayTest a1 = new ArrayTest(width, height),
            a2 = new ArrayTest(width, height);


        Stopwatch sw = new Stopwatch();
        sw.Start();
        for (int i = 0; i < loop; i++)
        {
            a1.DoMath(a2);
        }
        sw.Stop();

        Console.WriteLine("Time taken: " + sw.ElapsedMilliseconds);

        Console.ReadKey();
    }
}

Here is the Java version:

    public class ArrayTest {
    private int width, height;
    private int[][] array;

    public ArrayTest(int width, int height) {
        this.width = width;
        this.height = height;
        array = new int[height][width];
        Random rng = new Random();
        for (int i = 0; i < height; i++) {
            for (int j = 0; j < width; j++) {
                array[i][j] = rng.nextInt(2048);
            }
        }
    }

    public int getWidth() {
        return width;
    }
    public void setWidth(int width) {
        this.width = width;
    }
    public int getHeight() {
        return height;
    }
    public void setHeight(int height) {
        this.height = height;
    }
    public int[][] getArray() {
        return array;
    }

    public void doMath(ArrayTest a) {
        for (int i = 0; i < height; i++) {
            for (int j = 0; j < width; j++) {
                array[i][j] *= a.getArray()[i][j];
            }
        }
    }

}

public class Main {

    public static void main(String[] args) {
        // TODO Auto-generated method stub
        final int loops = 10;
        int width = 10000, height = 10000;
        ArrayTest a1 = new ArrayTest(width, height),
                a2 = new ArrayTest(width, height);

        long start, end;


        start = java.lang.System.currentTimeMillis();
        for (int i = 0; i < loops; i++) {
            a1.doMath(a2);
        }
        end = java.lang.System.currentTimeMillis();
        System.out.println("Elapsed time: " + (float)(end - start));
    }

}

In my computer this C# code is taking about 5200ms to run, and the Java version is taking about 2800ms(!!!). I was actually expecting that the .NET version would run faster (or at least close to) than Java, but got very surprised with this result.

Note: I ran the .NET version compiled in release mode, outside Visual Studio.

Can somebody explain this result? Is this really right? How could I rewrite this code so that the C#.NET version gets closer to the Java one in execution speed?

[edit]

Well, I know this is not a really valid benchmarking or a fair comparison, but how could I rewrite this code so that the C#.NET version gets closer to the Java one in execution speed? Tried in any forms (manipulating the jagged array directly, via a getter, etc), but the test ran even slower.

[edit 2]

Edited the test so that I have width and height = 500, and loop = 5000. Now I get about 6300ms for .NET version, and about 3700ms for Java version. ran the test multiple times for each version, of course.

[edit 3]

Just wrote a similar test using flat arrays instead of 2D arrays, and this time the .NET version runs equally or even faster than Java. So is that it? C#.NET's jagged arrays are just slower than Java's multidimensional arrays?

Yan Paulo
  • 369
  • 1
  • 3
  • 12
  • 1
    I'm surprised, too! Q: Could you post the Java version? – FoggyDay Apr 29 '15 at 03:51
  • Could you also state how many runs you made for each version and what the average performance was? It could be that the time taken is much larger for the first run and then decreases dramatically. – Lolo Apr 29 '15 at 04:13
  • A 2x difference between OOP languages doesn't sound that weird. Can you run them in a profiler to see where the hot spots are? Might just be a difference in PRNGs; that's a good amount of random numbers to spit out. (200 million, if I'm reading this right?) Or could be use of operator overloads in .NET. Or anything, I guess. – Andrew Janke Apr 29 '15 at 04:13
  • 5
    You need to do proper benchmarking to get meaningful results, but the test isn't fair anyway. In the java version you're doing `array[i][j] *= a.getArray()[i][j];` which only accesses the array once if I understand compound assignment correctly. In the C# version you're using indexers, so you're not accessing the array directly but via the `get` and `set` methods. – Paul Boddington Apr 29 '15 at 04:15
  • 2
    There are _lots_ of reasons such a difference could exist. For sure, running just 10 loops on each, you're mostly testing the JIT compiler, not the code itself. So the test itself is not even valid. Even if a valid test showed a difference, there is still the question of writing in each language using the appropriate techniques. Apples-to-apples comparisons are very difficult to get right, and this one definitely isn't. – Peter Duniho Apr 29 '15 at 04:16
  • 1
    @pbabcdefp: there are important differences, but I think the indexer vs getArray() is of non-obvious importance. That is, both involve a single method call, and a 2d array lookup. The reason it's significant (IMHO) is that in the Java scenario, the JIT compiler has the opportunity to recognize the invariantly constrained index and so may bypass bounds-checking on the lookup. The C# code may or may not have that opportunity, depending on a) whether the indexer is inlined, and b) whether that happens before or after the bounds-checking optimization (if any) is performed. – Peter Duniho Apr 29 '15 at 04:20
  • 2
    This `jagged = new int[width][];` could have caused an `IndexOutOfRangeException` if you had not chosen `width` and `height` to be equal – Alex Apr 29 '15 at 04:22
  • 3
    AFAIK the Java compiler is more "aggressive" with unrolling loops which can make a huge difference for short loops. I have seen an example in which Java actually outperformed native code (written in C++) in a textbook implementation of the sieve of Eratosthenes. So you may want to try more iterations as well. That still won't allow a general judgement over the performance of Java vs. C#/.NET of course. – bstenzel Apr 29 '15 at 06:29
  • Thanks Alex, did't se that. – Yan Paulo Apr 29 '15 at 12:38
  • Well, about the test, I tried to rewrite it using direct access to the array instead of the property(this[i, j]), and also returning the array via a getter and then accessing it, like the Java version, but in both cases the test ran even slower. – Yan Paulo Apr 29 '15 at 12:40
  • Please use `DoMath(int[][] jagged)` instead of `doMath(ArrayTest a)`. In your case you are testing direct access in Java with pointer access in C#. I can provide more detailed information, but your question is **on hold**. In few words in order to store/load the next value of array java uses plain `mov` with pointer bump `inc`, c# uses more expensive operation `lea` which compute effective address of value on each load/store! I have changed the test to direct access and got the same results for java and C# – Ivan Mamontov Apr 29 '15 at 16:49
  • Makes sense. But I changed my method to the form you sugested, and I still obtain the same result: For a flat array, C#.NET runs way faster, and for a jagged 2d array, Java runs way faster. Could this be just in my computer? – Yan Paulo Apr 29 '15 at 23:38
  • The most obvious difference is that the .NET version is accessing the array and height/width fields using property accessors (i.e. method calls) whereas the Java version is using fields (i.e. directly accessing global variables). I would imagine this would make a big difference to the performance. – open-collar Nov 21 '17 at 08:33

2 Answers2

12

Any time you do any kind of benchmarking or performance analysis, you need to ask a lot of questions, and take any particular result with (to paraphrase Tanenbaum), "a metric ton of salt".

HOWEVER ....

I copied pasted your code, and this is what I got:

Compiler      Version   Timing
--------      -------   ------
MSVS 2012     C# 5      4448.0
Eclipse Luna  JRE 1.7    977.0

So the Java program ran 4.5x faster than the C# program.

I also ran the MSVS "Profiling Wizard":

https://msdn.microsoft.com/en-us/library/dd264959.aspx

MSVS Profiler Summary Large picture

As you might be able to see from the screenshot, the "Big Pig" in this profile was ArrayTest.get_item(int32, int32), which consumed NEARLY HALF of the execution time:

enter image description here Large picture

MyDaftQuestions
  • 4,487
  • 17
  • 63
  • 120
FoggyDay
  • 11,962
  • 4
  • 34
  • 48
  • Whoa! Can you rewrite this test using direct array access and test it in the profiler and tell me what happens? In my case, the test runs even slower, and I don't have VS Pro os any profiler to see what's going on. – Yan Paulo Apr 29 '15 at 12:42
  • 9
    We are trying to compare direct access in Java with pointer access in C#. In few words in order to store/load the next value of array java uses plain `mov` with pointer bump `inc`, c# uses more expensive operation `lea` which compute effective address of value on each load/store! I have changed the test to direct access and got the same results for java and C# – Ivan Mamontov Apr 29 '15 at 17:12
  • Well, the code got a little faster, but it wasn't enough. I had to make several changes in the way I was looping through the 2d array, so that I could get a result equal to that of Java. – Yan Paulo May 01 '15 at 17:37
1

You can't judge the performance based on single instance. You need to do it for multiple instance. If you want concrete answers on performance I would suggest you to read quora, blog or SO.

Community
  • 1
  • 1
Abhishek
  • 6,912
  • 14
  • 59
  • 85