Dataset filtering performance

#1

Hi,

I was wondering what the performance of a filtering operation on a dataset
could be? For small set of data, it's ok, but what if I filter, let's say, 50
M rows?

I'm asking because I was wondering if it could be faster to filter data in
..Net that in a SQL query? In a SQL query, it could take up to 30 secondes to
execute a query on 50 M rows.

Is this a bad idea to consider import raw data from SQL and the filtering it
via .Net?

Thanks for any idea

Stephane

#2

I disagree, I have found many cases in which if a huge sample of data is
just being anaylzed and not written to in order to make a logic decision on
an application, it it more beneficial to stream in the dataset and let the
local PC do the computations. There is a very fine line that is being walked
in this case, and every scenario is different.

"Michael Nemtsev"; "MVP" <nemtsev (AT) msn (DOT) com> wrote

Quote:

Hello Stephane,

Why do u want to filter your data on your logic, not it SQL?

SQL is that app, which is aimed for such things, I see no reasons at all
to get all data and filter them on your own level,
because disadvantages are:
- it may take too much time to get all data from server
- it hinder the performance of your app to filter it
- it's not very scalable approach

---
WBR, Michael Nemtsev [.NET/C# MVP] :: blog:
http://spaces.live.com/laflour
"The greatest danger for most of us is not that our aim is too high and we
miss it, but that it is too low and we reach it" (c) Michelangelo

S> Hi,
S> S> I was wondering what the performance of a filtering operation on a
S> dataset could be? For small set of data, it's ok, but what if I
S> filter, let's say, 50 M rows?
S> S> I'm asking because I was wondering if it could be faster to filter
S> data in .Net that in a SQL query? In a SQL query, it could take up to
S> 30 secondes to execute a query on 50 M rows.
S> S> Is this a bad idea to consider import raw data from SQL and the
S> filtering it via .Net?
S> S> Thanks for any idea
S> S> Stephane
S

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Dataset filtering performance

Dotnet Framework (Performance) microsoft.public.dotnet.framework.performance

Dataset filtering performance - 08-23-2007 , 12:54 PM

Re: Dataset filtering performance - 11-16-2007 , 08:04 AM