Antivirus Performance Test
Hi,
I just bought LinqPad 8 and immediately ran into something I wasn't even aware of until I started it and it showed me a warning about slow loading of assemblies.
On my new Lenovo T16 laptop that has Windows 11 23H2 I have huge antivirus test script result times (ranging from 450ms to few seconds - longest was 10 seconds). As is mentioned the cause is antivirus (built-in Windows Defender). So I added exlusions for process name, nothing changed at all. I added some folder and file exclusions nothing, as if exlusion settings are ignored completely. I checked group policy and local group policy, nothing is forced so should be fine.
Then I disabled WiFi and test times droped to 100ms almost consistently. This triggered some ideas, so I enabled back WiFi and disabled "Cloud-delivery protection", "Automatic sample submission" and this got me to 70ms. Further disabling "Real-time protection" got me to around 50ms for assembly load time. Exclusions are ignored as far as I can tell becaus enumbers don't change whether LinqPad8.exe is excluded or not.
Which is still many times longer than I get on my 8 year old laptop with Windows 10. On it and my other Windows 10 PC I get 0.1ms with exclusion to LinqPad8.exe.
In the meantime I ran into some articles that 23H2 causes CPU performance problems and Windows Defender is to blame, and there are some powershell commands to restart Windows Defender, but it didn't work for this issue.
Does anyone here have 23H2 and have this working correcly?
To me it looks like exclusions are completely ignored, and it sends every generated temp.dll file as a sample to online servers for scanning.
I have newly installed laptop and this is now getting on my nerves, especially since laptop work flawlessly and it's fast otherwise. The only solution that I can think of is to disable "Cloud-delivery protection" and "Automatic sample submission".
Comments
With realtime protection disabled, 50ms is still 100 times slower than your old laptop. Have you checked for I/O contention on your hard drive from other processes? On new laptops, there's often a lot of updates and scans going on in the background. One way to check is with Task Manager, Show All Processes, Details, select columns and choose I/O bytes read/written (there may be a better way).
Also, the fact that process-based exclusions for Defender don't appear to work could be a bug in 23H2 - maybe you should report this to Microsoft, especially if someone else can repro.
Nothing seems to be using disk (I've checked what you suggested), and everything else works much faster than old laptop. It's not that new anymore, I started setting it up 10 days ago so it had time to analyse, scan and index everything. And disk in tests shows nice speeds as modern NVMe should.
Write of that DLL file from your script is fast, but when reading starts (assembly load) then MsMpEng.exe takes over and that is very visible in ProcessMonitor. It's time is matching the delays shown in script results.
What is even stranger is that on old windows 10 laptop I have Malwarebytes installed, and with it the times are 10ms. When I exit Malwarebytes then it is briefly 0.1ms while transitioning to Windows Defender and when it takes over it is 20ms, so two times slower (but with every option turned on in Windows Defender). Also exclusions are respected and when LinqPad8.exe is ecluded it drops to 0,1ms.
On new laptop I also tried with Malwarebytes, even with ESET, but whether they work, or Windows Defender works the times are the same. Like Windows Defender works in parallel with them even when it shouldn't.
I'll do some more tests :-)
If someone reading this has 23H2 I'll be grateful if they check how it works on their machines.
Thanks!
I'll drop this info for others using Windows Defender:
I added the suggested exclusions for the LinqPad exe files. The times went up from ~400ms to ~8000ms ... super weird.
After playing around a bit I unchecked "Cloud-based protection" (don't know the exact english term for this, since I am using a german windows version). Turning this off, gives me ~20ms. Turning everything off, results in ~10ms
Using 23H2 as well
Edit:
Example, turning off the cloud-based protection mid test:
Edit 2:
Since I had other problems with my ethernet connections, I decided to install windows 10. Now I am at 1-2ms ...
It seems that perhaps Defender in 23H2 does not honor process exclusions. I've added to the thread on https://aka.ms/AAo3539 - can I encourage anyone else having this issue to do the same.
I'm on 22H2 but I'm seeing similar results.
With Windows Defender on, even with the LINQPad process added as an exclusion, I see times ranging from 400-700ms.
If I turn off "Cloud-delivered protection", the times drop down to the expected 20-40ms
I've reported it and I see you also commented on mine (the link above). Thanks.
I did a bit more tests (also added to the feedback ticket).
I've also checked on another colleague's new laptop who also installed Win 11 23H2 and he is experiencing the same issue. So it's not only mine laptop. He also gets 500ms loading times in this test, and Windows Defender exclusions don't work, Turning off "Cloud-delivery protection" and "Automatic sample submission" helps a bit (80ms) but not as when exclusion would be applied. So basically the same issue as mine.
We also tested on another colleague's laptop who has Win 11 22H2 with all updates for that version, and he doesn't experience this issue, it works the same as on my old laptop with Windows 10 22H2. When exclusions are applied loading time is less then 1ms and without exclusion (when Windows Defender does it's job in full) loading times are not that high (20ms compared to mine 500ms).
Found this page while trying to solve this exact problem on my new high-end HP ZBook laptop. Times in the 400-800ms range, exclusions have no effect. Replaced Defender with MalwareBytes, if anything, times seemed even slower.
Windows 11 23H2 Build 22631.2861
Addendum: Went back to Defender, disabled "Cloud-Delivered Protection" and can verify that times are in the 80-100ms range. Disabling "Device Security"->"Core Isolation"->"Memory integrity" reduces times to around 30ms. Still 10x slower than expected for a high-end G10 laptop.
Hi, I worked with this test as well. Such a great tool! We use Symantec. For Symantec folders (with sub-folders) can be excluded (that is a bad idea cause a virus could find out what folders are excluded, but simply guessing some candidates like). One thing that is interesting for LINQPad 8 64 bit.
If you add a line
Directory.GetCurrentDirectory().Dump();
the output surprisingly is:
C:\Users\\AppData\Local\Temp\LINQPad8_hqpnhwjq\shadow-1
instead of the folder where the query file is stored.
So all DLL files in the test will be created in the this folder - which in my case is not excluded.
So even if i stored a query in my exception folder this will not work.
I tried the same with a Visual Studio C++ project.
include
include // for file I/O
int main()
{
std::ofstream outfile("test.txt");
outfile << "my text here!" << std::endl;
outfile.close();
}
This will, as expected, create the file test.txt in the same folder as the executable.
So my question is: why is the current directory for LINQPad8 C:\Users\\AppData\Local\Temp\LINQPad8... ? Is that done by LINQPad itself or this caused by some Windows 11 virtualization feature?
It doesn't harm me - but I'm curious.
More precisely: it will create the file in the process's current directory. This is for the Windows shell the folder of the executable. From cmd is is the current directory of cmd. So it is a bad practice to use relative file path instead of fully qualifying them - but for the purpose of the test program it actually helped revealing the issue.
Today I tried this again, with all updates applied in last few weeks. And it seems that it is resolved now, at least on my PC.
And yes, like someone above mentioned, malwarebytes adds even bigger delays. This was the case even on my old PC, the test was faster with Windows Defender than with Malwarebytes.
The next LINQPad beta will support Windows Dev Drive.
Stay posted!
The new beta is out:
https://www.linqpad.net/linqpad8.aspx#beta
This will automatically detect Windows Dev Drive and use it for compilation output.
this topic is thrilling me!
I get results from 300ms (managed PC online), down to 30ms (managed offline) and below 1ms (AV excluded).
But what does that acutally mean and what is actually happening in the perf test?
I compiled and ran the code outside Linqpad and it's not slow at all, even with worst AV settings. Is it actually testing a special technique used only in Linqpad?
LINQPad just runs .NET code. You should get the same performance, unless you're running off a different hard drive with different antivirus settings. Have you tried setting up a dev drive?
managed PC is Windows 10, no dev drive. code shows 300ms when executing the embedded LinqPad8 AV test.
running the same code in a .net8 console app doesn't seem to trigger the AV and is below 1ms.
So this is why I am asking. probably that does not trigger AV but hosted by Linqpad it does?
Windows Defender uses undocumented heuristics to determine what and when to scan, and what kind of scanning to use. I also don't know how your machine has been set up. Maybe you've got exclusions that cover the output folder, or maybe Defender has just decided not to scan your console app files for some reason.
When I run the same tests in Visual Studio, I get identical results.
talking about Symantec and plus a more shitload of AV to make you feel secure.
Always looking for simple benchmarks that prove the slow down of these tools.
using on DEV drive w/ windows defender real-time protection off and cloud-delivered protection off and also the "Core Isolation"->"Memory integrity" off, results are now between 30-90
but I guess might be something related to xeon and dual cpu - but reason that I bought this machine used was to improve performance, now I'm not sure if I got a bad machine, CPU test from Intel and all DELL tests pass w/out problem.
The following Microsoft diagnostic tools will help you find out whether antivirus software is actually still running, and what else might be slowing your system:
https://learn.microsoft.com/en-us/sysinternals/downloads/procmon
https://github.com/microsoft/perfview
Let us know what you discover.
I've just got myself a new mini-pc (AMD Ryzen 7, Windows 11 Pro 23H2) and have the same kind of symptoms and performance degradation as described in this thread where anti-virus switched off is ~20ms and Cloud Delivered Protection >300ms. Tinkering with all settings here never goes below 20ms, whereas on every other computer I have used LINQPad, this antivirus test runs in <1ms. Adding exclusion folders or processes doesn't change anything for me on this new pc.
There were/are no other anti-virus tools pre or post installed. It's pretty much a vanilla new computer.
Using Dev Drive doesn't change the timings either.
I'm not sure what to put this down to. Obviously it could be anything with my setup, right? Bad Windows setup, drivers, hardware, rootkit etc.
Not the best of tests, but I just ran LINQPad inside the default Dev Home Windows Dev VM (Windows 11 Enterprise Eval 22H2) on two computers and get similarly bad results. ~20ms when AV and Cloud Delivered Protection is off. >300ms when CDP is on (default).
Obviously this is something to do with Windows (OS, Defender, or even .net core) and nothing to do with LINQPad directly! But it's a very good test and I can't "unsee" these results, knowing that "normal" should be <1ms!
Have you tried those diagnostic tools I mentioned earlier? They provide a lot of low-level information.
It was late last night when I started to look at things with procmon.
With RTP and CDP switched off, svchost.exe and MsMpEng.exe were still registering as opening LINQPad generated files... :-(
Even with Dev Drive installed?
I decided to stay in a normal drive, and was abl to pinpoint the Defender settings that I needed to disable in order to get down from >400ms to 20-30 ms.
Virus & threat protection/Cloud Delivery Protection (CDP)
Virus & threat protection/Automatic Sample Submission /ASS)
App & browser control/Smart App Control (SAC)
ASS is the biggest culprit. It brings it down from >400ms to ~150ms. CDP has little impact once ASS is disabled. But because its mosty only used by ASS, you might as well disable it.
Next was SAC. It brought the speed down to 20-30 ms. At this point i'm satisfied and I don't see a reason to pursue this further.
But on a very personal note, I do feel like saying we can't have good things. Windows Defender used to be the best of the bunch. Lightweight and darn good at its job. Windows Windows 11 turned itinto a bloated and misterious piece of software well worth removing completely if you can make your own custom windows installation disk, and then replaced by Malwarebytes.
Update. So, exclusions are actually working. What we have been doing wrong is choosing the option to exclude a file, instead of a process.
Once I added "C:\Program Files\LINQPad8\LINQPad8.exe" to my list of process exclusions, this is what I got after reenabling all the settings above:
Test 1 - 2.917 milliseconds
Test 2 - 0.327 milliseconds
Test 3 - 0.354 milliseconds
Test 4 - 0.33 milliseconds
Test 5 - 0.311 milliseconds
Test 6 - 0.364 milliseconds
Test 7 - 0.334 milliseconds
Test 8 - 0.461 milliseconds
Test 9 - 0.339 milliseconds
Test 10 - 0.296 milliseconds
According to Microsoft documentation, excluding a file, will exclude the file from scans. But when you exclude a process, it will exclude all files that process opens. And that was the problem. There's no need to disable Defender settings.
I can add all exceptions, process or files, and disable all settings and I get down to 20-40 milliseconds but no lower. At least it's something. The exceptions seem to do nothing. The only thing that does is removing the online bits.
I had similiar experiences as everyone else here. Turning off cloud based protection went from 600-4000ms to about 400ms. Then the exclusion to get to 30ms. And finally turning off Smart App Control to get to 0.3ms. Before that it was running in Evaluation-mode in my fresh-ish bare metal install. On another machine inside a virtual machine it was already off and the exclusion was all that was needed.
As of Win11 Version: 23H2 22631.4249
My exclusions are now working fine with the Dev drive and I get sub-1ms times with all Windows Defender features enabled.
I noticed this change about a week ago.
I think this might be the key, disabling Smart App Control. Like gclaugus it just randomly seemed to fix for me automatically on two computers, but I notice that this smart app control is disabled on both. I had a VM that I was having trouble with, and it had Smart App Control enabled. Once disabled the issue was gone and it seemed to honor exceptions again, though maybe exceptions were not the issue anyway..