Cast fails on data from the Appdomain if anything changes in the .linq code

apbrauer73

I have written code to process log files in LINQPad. I was planning on using the AppDomain to store the data read from the files and then filter the data on the next execution of the query. Below is a snippet of saving to and reading from the AppDomain.

AppDomain.CurrentDomain.SetData("aaLogData", LogRec.ToList()); List<LogFileRecord> data = ((List<LogFileRecord>)AppDomain.CurrentDomain.GetData("aaLogData"));

This works as long as I do not edit the query after the first execution. When I say edit, it could be something as simple as adding a comment to the code or something complex like modifying the code to filter the data read from the AppDomain.

I get the following exception:

[A]System.Collections.Generic.List`1[UserQuery+LogFileRecord] cannot be cast to [B]System.Collections.Generic.List`1[UserQuery+LogFileRecord]. Type A originates from 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' in the context 'LoadNeither' at location 'C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll'. Type B originates from 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' in the context 'LoadNeither' at location 'C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll'.

I have a simplified version of the code that reproduces the same error if needed.

I think this is a bug.

LINQPad v5.02.03 on Win 7 Pro 64-bit

nescafe

As a workaround, and preferably, you could use Util.Cache() or the .Cache() extension method for exactly this purpose.

nescafe

Didn't notice it was a custom type. When using .Cache(), you should also apply the [Serializable] attribute as LINQPad informs you about:

UncacheableObjectException: Cannot load type 'UserQuery+LogFileRecord' from cache.
Add the [Serializable] attribute to this type to make it cacheable.

More info:

Go to Help | What's New and search for Cache - there are a number of examples to get you started.

(from: http://forum.linqpad.net/discussion/comment/607/#Comment_607)

apbrauer73

I did already try to use Util.Cache() but I experienced poor performance. I used it again to collect results for sharing in this post. I did make my attribute [Serializable] before testing.

Below is the code snippet:

string filePath = @"C:\Temp\LogFiles\FromField"; //Util.Cache() stores the data in the AppDomain //ReadLogFiles - reads in the file contents of all log files var data = Util.Cache(() => ReadLogFiles(filePath)); //below here is where the changes are made //any change increases execution time and memory usage string startDT = "2015-09-10 16:16:38"; string endDT = "2015-09-16 17:39:18"; data .Where(x => x.Message.Contains("@Error =")) .Where(x => x.EventTimeStamp > DateTime.Parse(startDT)) .Where(x => x.EventTimeStamp < DateTime.Parse(endDT)) .Dump("FilteredLog");

On the first run of the query it takes 22s to read and filter data from the files. This is reading in a total of 3,135,903 records. I execute the query again and it executes in 0.008 s. Great so far, but same as my previous use of the app domain. Below are my notes for 4 consecutive executions.

1 Execute the query for first time - execution time : 0min 22s
2 Execute the query again without any changes - execution time : 0min 0s great!!!
3 Execute query again but with changes to filter the data - execution time : 2min 25s huh???
4 Execute query again - execution time : 3min 8s huh???

I have also noted an increase in memory usage on each step.

Prior to step 1 my memory usage was at 3.7GB
After step 1 it was at 6.0GB
After step 2 it was at 6.0GB -- this is fine
After step 3 it was at 8.6 GB -- huh?
After step 4 it was at 9.8 GB
After closing the query the memory was back at 3.7GB

I am struggling to see the benefit of using Util.Cache(). It appears that I am just better off re-reading the files and filtering the data for a mere 22s.

I think my use of LINQpad in this scenario is not what it was intended for. I love this tool. I have learned quite a bit in using it.

Thanks for the response nescafe!

nescafe

@apbrauer73, I see what you mean and managed to create a reproduction for this.

I'm using some smaller numbers (50,000 records each containing a single string consisting of 150 random Guids).

void Main()
{
  var data = Util.Cache(() =>
    {
      "Fetching results for the first time..".Dump();
      return
        Enumerable
          .Range(0, 50000)
          .Select(i => new LogRecord { data = string.Join(", ", Enumerable.Range(0, 150).Select(j => Guid.NewGuid()).ToArray()) })
          //.Select(i => string.Join(", ", Enumerable.Range(0, 150).Select(j => Guid.NewGuid()).ToArray()))
          .ToList();
    });

  data
    .Take(10)
    //.Select(s => new LogRecord { data = s })
    .Dump();
}

// Define other methods and classes here
[Serializable]
public class LogRecord
{
  public string data { get; set; }
}

First run: 5.240s, 576MB mem usage (LINQPad.UserQuery.exe Private Bytes)
Second run: 0.001s, no change in mem usage
Third run (after code modification): 2.769s, 1,138MB mem usage

Common denominator is the caching of a custom type LogRecord which is recompiled/recreated between code changes. If you comment the first and uncomment the last two .Select()-lines, the cached object will be of type List<string> and perform better:

(after a shift-f5 to unload process and reset memory usage)

First run: 5.201s, 575MB mem usage
Second run: 0.010s, no change in mem usage
Third run (after code modification): 0.002s, no change in mem usage

So, if you can find a way to cache standard or external types (which don't change between compilations), performance will improve a lot.

JoeAlbahari

Does it make a difference if you define the custom type in 'My Extensions'?

nescafe

In the repro, it does. Makes sense since 'My Extensions' are not rebuilt upon change of the currect user query. This would likely also fix the initial problem (caching data in Appdomain).

apbrauer73

I just moved the definition of my custom type into "My Extensions". Now I am seeing a huge improvement. I can now change the filter on the data being cached and I no longer see long execution times nor any memory usage increases.

This is awesome. The more I use LINQPad the more I like it.

This would be worth mentioning for caching custom types in the "What's new in LINQPad" file for Util.Cache().

Thanks Joe

apbrauer73

I just moved the definition of my custom type into "My Extensions". Now I am seeing a huge improvement. I can now change the filter on the data being cached and I no longer see long execution times nor any memory usage increases.

This is awesome. The more I use LINQPad the more I like it.

This would be worth mentioning for caching custom types in the "What's new in LINQPad" file for Util.Cache().

Thanks Joe