BenchmarkDotNet in LinqPad (AnyCPU) throws an Exception

hopp

Recently I posted an issue with BenchmarkDotNet (https://forum.linqpad.net/discussion/1872/benchmarkdotnet-in-linqpad#latest). This turned out to be issue of completely different kind.

But during investigation I tried several LinqPad versions. Beta versions, x86 or AnyCPU I was able to run BenchmarkDotNet in LinqPad, but only in x86 build.
AnyCPU build will throw an FileLoadException.

Assembly is still being loaded. (Exception from HRESULT: 0x80131016)

 at BenchmarkDotNet.Portability.RuntimeInformation.get_IsFullFramework()
   at BenchmarkDotNet.Portability.RuntimeInformation.GetCurrentRuntime()
   at BenchmarkDotNet.Characteristics.Resolver.<>c__DisplayClass1_0`1.<Register>b__0(CharacteristicObject obj)
   at BenchmarkDotNet.Characteristics.Resolver.Resolve[T](CharacteristicObject obj, Characteristic`1 characteristic)
   at BenchmarkDotNet.Toolchains.ToolchainExtensions.GetToolchain(Job job)
   at BenchmarkDotNet.Running.BenchmarkRunnerClean.<>c__DisplayClass12_0.<GetSupportedBenchmarks>b__2(BenchmarkCase benchmark)
   at System.Linq.Enumerable.WhereArrayIterator`1.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at BenchmarkDotNet.Running.BenchmarkRunnerClean.<>c__DisplayClass12_0.<GetSupportedBenchmarks>b__0(BenchmarkRunInfo info)
   at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext()
   at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at BenchmarkDotNet.Running.BenchmarkRunnerClean.Run(BenchmarkRunInfo[] benchmarkRunInfos)
   at BenchmarkDotNet.Running.BenchmarkRunner.RunWithDirtyAssemblyResolveHelper(Type type, IConfig config)
   at BenchmarkDotNet.Running.BenchmarkRunner.RunWithExceptionHandling(Func`1 run)
   at BenchmarkDotNet.Running.BenchmarkRunner.Run[T](IConfig config)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

I don't know if this can be called a defect and to which project it actually belongs ...

JoeAlbahari

I can't run it at all. I've looked into why, and there's layer upon layer of nested problems. Some I can fix, but for some it doesn't look good. It's related to complex chains of .NET Standard dependencies, duplicated types, and type-forwards.

It might be a while before I can work further on this because the priority now is on LINQPad 6. The dependency problems are not an issue with .NET Core (and also, because LINQPad 6 has a re-written dependency resolver which looks at reference assemblies as well as runtime assemblies). There are still other unrelated barriers with Benchmark.NET in LINQPad 6 - these will either be remedied or I'll ship a built-in benchmarking tool which will be more ergonomic. I'm thinking of an interface whereby you can highlight any code and press Shift+Ctrl+B to benchmark it.

hopp

Hi Joe,
by

I can't run it at all.

you mean LINQPad 6 ?

I find your idea of running benchmark with the shortcut great !

I like benchmark dot net's output, which compares times and is also able to compare garbage collections and allocations.

|    Method |            Mean |       Error |      StdDev |    Gen 0 |    Gen 1 |    Gen 2 | Allocated |
|---------- |----------------:|------------:|------------:|---------:|---------:|---------:|----------:|
|  SortLinq |        53.03 ms |   1.0579 ms |   2.1609 ms | 222.2222 | 222.2222 | 222.2222 | 1000064 B |
| SortArray |        22.01 ms |   0.4109 ms |   0.3844 ms |        - |        - |        - |         - |
|    Bubble | 1,016,419.24 ms | 606.5232 ms | 567.3422 ms |        - |        - |        - |         - |

JoeAlbahari

Thanks - I'll look into what features I can provide. It's actually possible to write your own benchmarking right now in LINQPad 6 by #load-ing a script that has hook methods:
https://www.linqpad.net/LinqReference.aspx

Here's a more elaborate script that allows marking methods with the Benchmark attribute:
http://share.linqpad.net/lorrft.linq

To test it:

#load "Benchmark"

void Main()
{
}

[Benchmark] void Test1() => Thread.SpinWait (1000);
[Benchmark] void Test2() => Thread.SpinWait (2000);
[Benchmark] void Test3() => Thread.SpinWait (3000);