Hi everyone,
I've asked for your help before and you've been very welcoming and
helpful, however now I have a few further issues.

I'm developing a highly concurrent middleware server app (C#) that
takes socket connections from a frontend webpage and performs
processing and SQL queries on a back end SQL server. I'm getting
massive improvements on my predecessor's efforts - I can sustain 1000
msgs/sec in some cases as opposed to a few tens before.

However, I'm hitting a point now where latency is creeping up (over a
minute under heavy load - which is unacceptable) and I think its down
to differing lengths of time it takes the incoming messages to execute
(some are very quick, a few are several seconds - credit card auth).

I'm using async sockets to read in and transmit responses, and the
built in threadpool and where possible the new async database
operations (in .NET 2.0 beta 2) to handle actual message processing, so
IO completion ports all over the place.

Ive been using a seperate threadpool class to place longer running
(over 1 sec) processing - but i'm unsure even after some testing as to
exactly what balance I should give and how many threads to allocate.
I'm assuming all 25 of the built in ThreadPool are a gimme, but how
many is reasonable to allocate seperately? I'm beginning to think
shifting all my processing to the seperate threadpool would be
beneficial too.

Any suggestions? I'm at a bit of a loss now, this is where my expertise
on the matter ends? I'd be eternally grateful if anyone could point me
in a useful direction!

Cheers,
Kieran

you don't think that you are possibly saturating the threadpool hence
leading to latency?

ice
"Kieran Benton" <kieranbenton (AT) fastmail (DOT) fm> wrote

Yes, I think I suggested that in my original post - and why I'm moving
to a seperate threadpool for the processing. Do you have any
suggestions for the number of threads I should allow in total though?
Don't want to get to the stage where I'm using all the CPU context
switching!

what do you mean by "another threadpool"? is this basically another
process? Invariably there is only a finite number of threads (using that
term loosely) regardless of how many "threadpools" - meaning processes - you
create.

The optimum number can really only be determined by you during your own
performance testing.

ice
"Kieran Benton" <kieranbenton (AT) fastmail (DOT) fm> wrote

By "another threadpool" I mean a class that has been written to perform
the functionality of the buil-in runtime ThreadPool but allow itself to
be instanciated so that different jobs can go to different TPs
depending on their characteristics. NOT another process (I don't even
understand what you mean by this).

Performance testing is allowing me to finetune the number of threads
allocated to each pool, however I am unable to replicate the entire
expected load until final in-situ testing takes place. Is there a
guideline maximum number of active threads a process should use for
maximum scalability?

Hello,

you can check the state of the threadpool via
ThreadPool.GetAvailableThreads(). This will give you the number of available
workerthreads and completionport threads. Using GetMaxThreads() you can
check wether you are using all available threads of the threadpool.

Another thing: When your app is under full stress and latency goes up, on
which level is your cpu usage? If it is already at 100%, the only thing you
could do is an additional processor. Note that cpu usage is not the only
counter you should monitor. Check the Processor Queue Length performance
counter (under the "System" category). If this counter is often greater than
10*<processorcount>, you are maybe using to many threads.

Greetings,
Henning Krause
MVP - Exchange
http://www.infinitec.de


"Kieran Benton" <kieranbenton (AT) fastmail (DOT) fm> wrote

Thanks Henning,
Yes, I'm sure that all of the threads are being used in the threadpool,
and yes CPU is at or around 100%. I was under the impression though
that this might be down to too many context switches? When I view CPU
usage in Sysinternals Process Explorer I see a significant (15-20%)
amount of time is spent in kernal mode.

Your pointer on the Processor Queue Length is very useful, I will check
this out immediately.

Cheers,
Kieran

Hello,

the amount of time spent in the kernel might or might not be caused by heavy
context-switches. Check the performance counter for Context switches of you
process... that could give you a clue.

Greetings,
Henning Krause
MVP - Exchange
http://www.infinitec.de


"Kieran Benton" <kieranbenton (AT) fastmail (DOT) fm> wrote

i don't know of any guidelines. there are only telltale signs as to when
"too much" threading is going such as high context switching etc. i didn't
understand the fac that you had a custom threadpool in place. now that
you've explained that, it makes sense.

ice
"Kieran Benton" <kieranbenton (AT) fastmail (DOT) fm> wrote