Hi,
I'm having a serious problem with an ASP.NET web application that relies
heavily on several cache objects. One of the cache objects is a datatable
with approximately 50,000 records. From time to time, the cache object
becomes unsearchable. There are checks in the components that reference the
cache to see if it's NOTHING, if it is a DataTable and if it has rows, and
all those tests seem to say the object is ok. But when we try to execute the
..Select method on the datatable, an exception is thrown :
System.NullReferenceException: Object reference not set to an instance of an
object.
at System.Data.Select.FindClosestCandidateIndex()
at System.Data.Select.SelectRows()
at System.Data.DataTable.Select(String filterExpression, String sort)
at NCSBizComponents.ncsTill.GetStoreTillsByName_FromC ache(Int32
piStoreId, String psTillTypeDescription)

When this happens, we manually force are refresh of Cache object and we're
ok for a while.
This seems to happen more often when the app is under a heavier load.

Our web server is running Win2K3 server, running on an IBMx440 under vmware.

I've read that ADO.NET creates it's own indexes when you create a datatable.
I'm wondering if somehow these indexes are getting messed up, causing the
Select method to fail. Can we/should we create indexes manually on the
datatable in cache?

Has anyone experienced this or have any ideas how to correct it?

Thank you,
Keith F

I suppose it depends on your code to access the Cache, but ASP.NET will purce
Cache entries if the server is low on memory. Caching 50000 records feels
a bit heavy handed... so you might be seeing this purge behavior. Though
it's impossible to tell without seeing your GetStoreTillsByName_FromCache
method

-Brock
DevelopMentor
http://staff.develop.com/ballen

Our web server has 1 gig of memory and the perf logs don't usually show the
memory going about 600 MB. It's usually at around 400-500 MB.
Here's the code where we're searching the cache:

Public Function GetStoreTillsByName_FromCache(ByVal piStoreId As Integer,
ByVal psTillTypeDescription As String) As ncsStoreTills
Dim lsMethodName As String = "GetStoreTillsByName_FromCache"
Try
Dim loTransMgr As ncsTransactionManager = New
ncsTransactionManager
'Get the Tills from Cache
Dim ldtTills As DataTable =
loTransMgr.GetTillTranTypes_FromCache
Dim ldrStoreTills() As DataRow
Dim ldrRow As DataRow
Dim lsFilter, lsSort As String
Dim loTill As ncsStoreTill
Dim loTillsCollection As ncsStoreTills = New ncsStoreTills
Dim liHoldTillId As Integer = 0

If Not ldtTills Is Nothing Then

'Set filter expression to select by StoreId and Description
lsFilter = "iStoreId = " & piStoreId.ToString & " AND
TillType LIKE '%" & Replace(psTillTypeDescription, "'", "''") & "%'"
'Set sort expression to TillNbr
lsSort = "tiTillNum"
'Invoke the Select method on the Tills DataTable we got from
Cache
' to get a collection of DataRows meeting the filter
criteria
ldrStoreTills = ldtTills.Select(lsFilter, lsSort)

'Loop through the DataRows
If Not ldrStoreTills Is Nothing Then 'pjo 02/21/05

For Each ldrRow In ldrStoreTills
'Noteue to how the Cache is setup, we'll likely
have multiple records with the same till
' in our DataRows collection.
'We'll test for duplicate TillIDs and skip those.
If Not ldrRow Is Nothing Then 'pjo 02/21/05


If ldrRow("iStoreTillId") <> liHoldTillId Then
'We have a new TillId
'create a new instance of a store till
loTill = New ncsStoreTill
'set the properties from the current DataRow
With loTill
.TillDescription = ldrRow("TillType")
.TillId = ldrRow("iStoreTillId")
.TillNumber = ldrRow("tiTillNum")
.TillTypeId = ldrRow("iTillTypeId")
End With
'Add the till to the store tills collection
loTillsCollection.Add(loTill)
'Save the TillId we just added to our
collection
liHoldTillId = ldrRow("iStoreTillId")
End If
End If

Next
End If

End If



ldtTills = Nothing 'pjo 02/21/05
ldrRow = Nothing 'pjo 02/21/05
'Return the StoreTills Collection
Return loTillsCollection





Catch ex As Exception
Dim loExpMgr As ExceptionMgr = New ExceptionMgr
loExpMgr.Publish(ex, Err.Number, Err.Description, gmodName,
lsMethodName)
End Try

End Function

Thank you,
Keith


"Brock Allen" wrote:

Hmm.. well, I can't tell much from this; Sorry. Do you log the call stack
when you have unhandled exceptions? This might help in diagnosing the problem.


-Brock
DevelopMentor
http://staff.develop.com/ballen

I guess the answer to if we log the call stack for unhandled exceptions is no
(I'm not sure how to do that).
We email the exception info to system admins when we catch an error. That's
what I included in my original post.

This morning we identified the exact line of code that's throwing the
exception:
ldrStoreTills = ldtTills.Select(lsFilter, lsSort)
It's happening when we invoke the Select method of our DataTable object.
We added code to our exception handler to include the contents of the
lsFilter and lsSort varibles to make sure we're not passing invaid or empty
arguments to the Select method.
We have checks in the method that retrieves the DataTable from cache to make
sure it is a DataTable object, it actually has rows in it, and the HasErrors
property is false. I put a page out on the site that lists these properites.
I went to it when our app was throwing errors and all these properties looked
normal. However the cache wasn't searchable (apparently).

Would you have any thoughts as to why the Select method on the datatable
would be throwing an exception?

Or do you think the main problem is that a 50,000 row data table is just to
much to keep in cache?

One last question: When our code gets the DataTable object from Cache, and
puts it in a DataTable variable, is this a copy of the DataTable in Cache, or
a reference to the actual cache object?

Thank you for your thoughts.

Keith

"Brock Allen" wrote:

Hmm, yeah looking back at the callstack you provided from the first post...
the exception isn't that the DataTable is null, it's further down... so yeah,
hard to diagnose in the bowels of their code.

Well, my sense is that the DataTable wasn't designed for this; Hundreds,
maybe a couple of thousand rows, but 50K seems like you're pushing the envelope.
I've not done a ton of stress testing on the DataTable to know what it's
breaking point is though, so this is all just my sense of things not backed
by any hard data or testing. So, YMMV

It's a reference back to the same object in the cache.

NP -- sorry I've not been more helpful.

-Brock
DevelopMentor
http://staff.develop.com/ballen

Maybe try creating a new DataView off of the DataTable after you pull it out
of cache, & using that DataView to access the data? Seems like some parts of
the DataTable are getting killed off.

--
Benjamin Strackany
http://www.developmentnow.com


"Keith F." <KeithF (AT) discussions (DOT) microsoft.com> wrote