Since strings are immutable, isn't this code:

Dim sr As StreamReader = New StreamReader("TestFile.txt")
Dim line As String

Do
line = sr.ReadLine()
Loop Until line Is Nothing

causing the string object 'line' to be constantly recreated for each line in
the file?

I have an app that reads very large (90K+lines) text files, parses the line
and inserts rows into a SQL Server database. It's running pretty well but
the memory usage is topping out over 30 MB, this is alarming to me. The old
VB6 app would run at about 7 MB.

Am I misunderstanding the whole immutability thing or obsessing over
nothing?

Paul Nations <pauln (AT) adhe (DOT) arknet.nospam.edu> wrote:
line isn't an object. It's a variable. But yes, each iteration, line
takes a different value.

It could be that you're missing something else entirely. Have you tried
using the file-reading code on its own, and seeing what the memory
usage is like there?

--
Jon Skeet - <skeet (AT) pobox (DOT) com>
http://www.pobox.com/~skeet
If replying to the group, please do not mail me too

Try using CLRProfiler and seeing where that 30 MB is coming from

--
Sriram Krishnan

http://www.dotnetjunkies.com/weblog/sriram


"Jon Skeet [C# MVP]" <skeet (AT) pobox (DOT) com> wrote

Yes, a new string is created each time.

A cursory scan of any .NET newsgroup or list will yield thousands of
panicked messages about apprarent memory-hogging events. The short answer
is that released memory is reflected in Task Manager when it's needed by the
OS, not when .NET is through with it. Also, you must allow for the runtime,
which is shared amongst any .NET apps that are running.

I have written a lot of programs of the sort you mention, and the
performance is excellent, even in debug builds. Frankly I've never even
checked memory consumption because (1) they work all day long, day after
day, unattended, for years, (2) they're plenty fast. In fact, I use a
generalized DelimitedTextFile() class that returns ArrayLists of field
values for each record in a comma-delimited file, and even with all that
parsing and assigning, and with tons of data conversion overhead
(rearranging field orders, tranforming values and the like), I can plow
through dozens of megabytes of input data in seconds.

In my experience if you encoutner actual performance problems it's a
solvable issue by refactoring your code. The only time I've had code that
ran slower than I would have liked was in a memory-intensive code section
that made extensive use of string interning, but that was a concious tradoff
of lower speed for better memory efficiency, and it's still adequately fast.

So, likely you *are* obsessing over nothing. ;-)

C# has good memory management -- you need look at memory consumption and
performance only if actual out-of-memory or slow performance issues arise,
and when they do, they will almost always be because you can improve your
code, not because you have to work around or fight some terrible deficiency
in the runtime environment.

--Bob

"Paul Nations" <pauln (AT) adhe (DOT) arknet.nospam.edu> wrote

Actually, the .Net version is faster than the old VB6 app. It will process
the 90K+ line file in about 10 min .vs. 15 for the VB6 version. And the 30
MB is easily within the specs of modern workstations. But, I have to plan
for this system to run at a few very small colleges with limited hardware.

I'm not desperate for another method, but I was just trying to find out if
there were some other method of reading a text file into a string. It
appears this is the standard method and I will run with it.

Thanks all.


"Bob Grommes" <bob (AT) bobgrommes (DOT) com> wrote

You could use the mthode ReadToEnd() which will read the entire file into
one string which should be faster but the difference is that you have all of
the 90k at once in memory.
You could also use Read or ReadBlock so you can use a char array instead of
having created temporary strings all the time.

--
cody

Freeware Tools, Games and Humour
http://www.deutronium.de.vu || http://www.deutronium.tk
"Paul Nations" <pauln (AT) adhe (DOT) arknet.nospam.edu> schrieb im Newsbeitrag
news:e1JQZt5tEHA.2804 (AT) TK2MSFTNGP14 (DOT) phx.gbl...