C++ vs CSharp vs VB benchmarks

I created two benchmark applications. One does a discrete Fourier
transform on a data series and the other is an implementation of
quicksort. Before entering the main section where all the work is
getting done, I save the start time using their respective tick count
function. When the work is completed, I get a 2nd tick count and
display the difference between the two tick counts.

On both tests I am seeing Visual Basic outperforming CSharp which both
outperforms C++. All in the release version. This doesn't seem right.
Anyone have any suggestion what is going on or what I am doing wrong?

Platform is XP running under Parallels.

On Mon, 17 Dec 2007 21:29:23 -0800, <tlicious (AT) gmail (DOT) com> wrote:

I created two benchmark applications. One does a discrete Fourier
transform on a data series and the other is an implementation of
quicksort. Before entering the main section where all the work is
getting done, I save the start time using their respective tick count
function. When the work is completed, I get a 2nd tick count and
display the difference between the two tick counts.

On both tests I am seeing Visual Basic outperforming CSharp which both
outperforms C++. All in the release version. This doesn't seem right.
Anyone have any suggestion what is going on or what I am doing wrong?

Platform is XP running under Parallels.

What's the total time being measured? If it's small, you may want to use
the Stopwatch class instead. It might be a little more accurate.

Also, while I run XP/.NET etc under Parallels myself, and while I have not
noticed what I'd consider significant timing anomalies, it's worth a try
to see if you can reproduce the results without the virtual machine, on a
straight XP installation (e.g. Boot Camp, or on a completely different,
non-Mac computer).

If you do find that Parallels is somehow related to the unexpected
results, please post back here (well, post back if you learn anything new
anyway). The one thing I did notice once was that
Dictionary.TryGetValue() was closer than Dictionary.Contains() followed by
an indexed retrieval. I never thought to try that test without the VM,
but your post is making me wonder now.

Pete

On Dec 18, 1:55 am, "Peter Duniho" <NpOeStPe... (AT) nnowslpianmk (DOT) com>
wrote:

On Mon, 17 Dec 2007 21:29:23 -0800, <tlici... (AT) gmail (DOT) com> wrote:
I created two benchmark applications. One does a discrete Fourier
transform on a data series and the other is an implementation of
quicksort. Before entering the main section where all the work is
getting done, I save the start time using their respective tick count
function. When the work is completed, I get a 2nd tick count and
display the difference between the two tick counts.

On both tests I am seeing Visual Basic outperforming CSharp which both
outperforms C++. All in the release version. This doesn't seem right.
Anyone have any suggestion what is going on or what I am doing wrong?

Platform is XP running under Parallels.

What's the total time being measured? If it's small, you may want to use
the Stopwatch class instead. It might be a little more accurate.

Also, while I run XP/.NET etc under Parallels myself, and while I have not
noticed what I'd consider significant timing anomalies, it's worth a try
to see if you can reproduce the results without the virtual machine, on a
straight XP installation (e.g. Boot Camp, or on a completely different,
non-Mac computer).

If you do find that Parallels is somehow related to the unexpected
results, please post back here (well, post back if you learn anything new
anyway). The one thing I did notice once was that
Dictionary.TryGetValue() was closer than Dictionary.Contains() followed by
an indexed retrieval. I never thought to try that test without the VM,
but your post is making me wonder now.

Pete

I had someone else run it on a straight XP machine with the same
result. After some initial investigation, the cos/sin function in
cmath library seem to be the biggest usage of the time. Using a trig
table lookup instead, C++ outperforms the other two in the DFT
benchmark app. Still curious on why the quicksort is slower in VC++.
Is C++ slower at stack operations?

On Wed, 19 Dec 2007 21:10:31 -0800, <tlicious (AT) gmail (DOT) com> wrote:

I had someone else run it on a straight XP machine with the same
result. After some initial investigation, the cos/sin function in
cmath library seem to be the biggest usage of the time. Using a trig
table lookup instead, C++ outperforms the other two in the DFT
benchmark app. Still curious on why the quicksort is slower in VC++.
Is C++ slower at stack operations?

It could have something to do with memory management. Garbage collection
has a cost, but it's a very specific cost and happens at very specific
times. The flip side is that memory allocations are extremely fast, so
code that spends a relatively significant amount of time allocating new
buffers can do very well in .NET/C#. Does your quicksort implementation
use allocated buffers? Or just do everything in-place?

To know for sure why the speed different, you'd need a profiler of
course. There are any number of actual possible explanations.

Years ago, before I knew much about .NET and only shortly after .NET
actually _was_ anything (for some time, it was just a marketing term), I
had prejudices about .NET regarding it's potential for performance and
capabilities. Not only were those prejudices proved almost entirely false
as soon as I started doing any actual .NET programming, I continue to
every now and then hear about or see myself examples of things that .NET
actually does _better_.

Perhaps your tests are an example of that. Or perhaps it just means that
the code you've written is only optimal in one environment and there's a
better way to do it in some other environment. Without some detailed
analysis it would be very difficult to say for sure.

The most important thing here though, I think, is that these sorts of
benchmarks are probably not really all that meaningful anyway. It's
helpful to know some broad generalizations between environments I suppose,
but even if there was an environment that was 100% reliably
better-performing than some other environment, one could always write slow
code in the faster environment and better performing code in the slower
environment.

It is IMHO more useful to focus on picking an environment that suits the
overall needs of the applications best, and then worry about performance
issues as they come up. This doesn't mean one should write stupid code,
of course. There are some basic rules that are always helpful to follow.
But unless you're an optimization guru, it's not really even possible to
write useful comparison benchmarks for different environments. What works
best in one environment is not necessarily what will work best in another.

And even if you are an optimization guru, because in practically any
reasonable development environment it is possible to write applications
that perform well, it makes more sense to focus on the features of an
environment that help other aspects of the development process, like code
correctness, readability, built-in feature-rich support for a wide variety
of tasks, etc.

I wouldn't worry too much about apparent performance differences across
various platforms (and this is especially true if it happens that you
don't have a _lot_ of experience writing benchmarks and optimizing them
for the specific needs of each platform).

Pete

I agree. If it is purely power that you are interested in, no doubt native
C++ is going to yield the best results, performance-wise. But there are
severe trade-offs in terms of safe memory management and especially
productivity, not to mention the various advantages and potential of using
an intermediate JIT byte code language. But chiefly, productivity is the
greatest strength of the .Net platform. It was created for the same reasons
that OOP was created, which also had a small performance cost associated
with it. As software performs increasingly complex tasks, the need for
development platforms that enhance productivity increases. With Moore's law
in play, small differences in performance are vastly outweighed by the
ability to write solid code in a relatively short period of time. And the
..Net platform includes a great number of ways to improve performance,
including memory caching, the ability to use unsafe pointers when desired,
and interoperability with unmanaged libraries when necessary, to name the
most useful.

--
HTH,

Kevin Spencer
Chicken Salad Surgeon
Microsoft MVP


"Peter Duniho" <NpOeStPeAdM (AT) nnowslpianmk (DOT) com> wrote

On Wed, 19 Dec 2007 21:10:31 -0800, <tlicious (AT) gmail (DOT) com> wrote:

I had someone else run it on a straight XP machine with the same
result. After some initial investigation, the cos/sin function in
cmath library seem to be the biggest usage of the time. Using a trig
table lookup instead, C++ outperforms the other two in the DFT
benchmark app. Still curious on why the quicksort is slower in VC++.
Is C++ slower at stack operations?

It could have something to do with memory management. Garbage collection
has a cost, but it's a very specific cost and happens at very specific
times. The flip side is that memory allocations are extremely fast, so
code that spends a relatively significant amount of time allocating new
buffers can do very well in .NET/C#. Does your quicksort implementation
use allocated buffers? Or just do everything in-place?

To know for sure why the speed different, you'd need a profiler of
course. There are any number of actual possible explanations.

Years ago, before I knew much about .NET and only shortly after .NET
actually _was_ anything (for some time, it was just a marketing term), I
had prejudices about .NET regarding it's potential for performance and
capabilities. Not only were those prejudices proved almost entirely false
as soon as I started doing any actual .NET programming, I continue to
every now and then hear about or see myself examples of things that .NET
actually does _better_.

Perhaps your tests are an example of that. Or perhaps it just means that
the code you've written is only optimal in one environment and there's a
better way to do it in some other environment. Without some detailed
analysis it would be very difficult to say for sure.

The most important thing here though, I think, is that these sorts of
benchmarks are probably not really all that meaningful anyway. It's
helpful to know some broad generalizations between environments I suppose,
but even if there was an environment that was 100% reliably
better-performing than some other environment, one could always write slow
code in the faster environment and better performing code in the slower
environment.

It is IMHO more useful to focus on picking an environment that suits the
overall needs of the applications best, and then worry about performance
issues as they come up. This doesn't mean one should write stupid code,
of course. There are some basic rules that are always helpful to follow.
But unless you're an optimization guru, it's not really even possible to
write useful comparison benchmarks for different environments. What works
best in one environment is not necessarily what will work best in another.

And even if you are an optimization guru, because in practically any
reasonable development environment it is possible to write applications
that perform well, it makes more sense to focus on the features of an
environment that help other aspects of the development process, like code
correctness, readability, built-in feature-rich support for a wide variety
of tasks, etc.

I wouldn't worry too much about apparent performance differences across
various platforms (and this is especially true if it happens that you
don't have a _lot_ of experience writing benchmarks and optimizing them
for the specific needs of each platform).

Pete

tlicious (AT) gmail (DOT) com schrieb:

On Dec 18, 1:55 am, "Peter Duniho" <NpOeStPe... (AT) nnowslpianmk (DOT) com
wrote:
On Mon, 17 Dec 2007 21:29:23 -0800, <tlici... (AT) gmail (DOT) com> wrote:

On both tests I am seeing Visual Basic outperforming CSharp which both
outperforms C++. All in the release version. This doesn't seem right.
Anyone have any suggestion what is going on or what I am doing wrong?
Platform is XP running under Parallels.

I had someone else run it on a straight XP machine with the same
result. After some initial investigation, the cos/sin function in
cmath library seem to be the biggest usage of the time. Using a trig
table lookup instead, C++ outperforms the other two in the DFT
benchmark app. Still curious on why the quicksort is slower in VC++.
Is C++ slower at stack operations?

What are you sorting? An array of integers or of larger objects? Are you calling
the qsort api function std::sort, did you implement something on your own? Is an
overloaded copy operator involved in the C++ version?

Maybe your test is not fair. Can you share some code?

Norbert

Don't all three compile to msil, and therefore should run about the
same? Was the C++ compiled as native code (no CLR)? If not, try that.



Peter Duniho wrote:

On Mon, 17 Dec 2007 21:29:23 -0800, <tlicious (AT) gmail (DOT) com> wrote:

I created two benchmark applications. One does a discrete Fourier
transform on a data series and the other is an implementation of
quicksort. Before entering the main section where all the work is
getting done, I save the start time using their respective tick count
function. When the work is completed, I get a 2nd tick count and
display the difference between the two tick counts.

On both tests I am seeing Visual Basic outperforming CSharp which both
outperforms C++. All in the release version. This doesn't seem right.
Anyone have any suggestion what is going on or what I am doing wrong?

Platform is XP running under Parallels.


What's the total time being measured? If it's small, you may want to
use the Stopwatch class instead. It might be a little more accurate.

Also, while I run XP/.NET etc under Parallels myself, and while I have
not noticed what I'd consider significant timing anomalies, it's worth
a try to see if you can reproduce the results without the virtual
machine, on a straight XP installation (e.g. Boot Camp, or on a
completely different, non-Mac computer).

If you do find that Parallels is somehow related to the unexpected
results, please post back here (well, post back if you learn anything
new anyway). The one thing I did notice once was that
Dictionary.TryGetValue() was closer than Dictionary.Contains() followed
by an indexed retrieval. I never thought to try that test without the
VM, but your post is making me wonder now.

Pete