U4-7712 - fcnMode = Single by default

Created by Shannon Deminick 13 Jan 2016, 13:29:00 Updated by Gigi2m02 12 Jul 2017, 02:21:56

Relates to: U4-9586

Relates to: U4-1047

We will make fcnMode = Single by default in the web.config, for more information on fcnMode see:

http://shazwazza.com/post/all-about-aspnet-file-change-notification-fcn/

When installing packages fcn can cause issues - even if config file are not changed, if there are enough files in the package it will overwhelm the iis buffer and cause an app pool restart.

1 Attachments

Comments

Shannon Deminick 13 Jan 2016, 13:51:20

We have this set by default on all UaaS servers - but this also requires to be set for local installs due to how packages are installed in the back office - it doesn't wait for restarts which it needs to (we also need to fix that: http://issues.umbraco.org/issue/U4-1047


Shannon Deminick 13 Jan 2016, 16:28:06

Also added nuget transforms: 5aab50ba3b68f6fc78516f53f5e881e583956c5d


Erik Thijssen 21 Mar 2016, 08:37:40

@Shandem Using Single-mode solves a lot of AppDomain issues, but there is a little downside. It doensn't work when you use virtual subdirectories. We've virtualized some cache folders (ImageProcessor) and ~/Media folder. Both folders contain a lot of subfolders (5000+) but in case of fcnmode=single, they still will receive a directory monitor object for each subdirectory. The only way to prevent this is to use fcnmode=disabled or to make your virtual directory an application in IIS (in which you place fcnmode=single).


Shannon Deminick 21 Mar 2016, 10:10:00

@e.thijssen@qvision.nl that's very interesting! Did you test how many file watchers there were with that code i mentioned on my blog post ? https://github.com/Shazwazza/UmbracoScripts/blob/master/src/Web/ASPNetFileMonitorList.cshtml


Erik Thijssen 21 Mar 2016, 10:48:09

Yes, we used your .cshtml file to check the FCN behavior. That's when we realised that all physical folders where handled by the single root directory monitor (as expected), but for virtual directories a directory monitor for each subfolder was created (which in case of ImageProcessor is a lot!). Take a look at http://referencesource.microsoft.com/#System.Web/FileChangesMonitor.cs,0820c837402228ef,references You can see that the "StartListeningToLocalResourcesDirectory" (StartListeningToVirtualSubdirectory) ONLY checks whether FCN is disabled but does nothing with the "Single" option.


Shannon Deminick 21 Mar 2016, 10:55:56

Good find, I should update my blog post with this info


Eric Schrepel 27 Oct 2016, 20:24:54

Have no idea if this is related, but we were having various "CodeDirChangeOrDirectoryRename"-related shutdowns/restarts, so we set fcnMode to Disabled for awhile to see if it would help (we're on a locally-hosted Win2008 so the Win2012 hotfix doesn't help).

But then we found that changes to templates and partial views weren't ever getting displayed unless we manually recycled the app pool, or made a change to web.config. Changed it back to Single and that's operating as intended again. But now, I presume, we'll be back to getting fairly frequent config-change shutdowns.

Is there some other way to prevent the shutdowns but still have templates/partials get updated? In IIS, we recently set the app pool parameter of "Disable Recycling for Configuration Changes" to true, so will see if that has an effect (realizing we'll need to manually recycle sometimes when making config file changes maybe). Doesn't seem to affect templates/partials anyway.


Shannon Deminick 27 Oct 2016, 22:00:58

I can't see the CodeDirChangeOrDirectoryRename being related to fcn. You cab decompile parts of aspnet to see exactly where/how that msg is triggered, it will be from within aspnet not IIS.

Make sure you have no anti virus running, that can cause havoc too for lots of things.


Gigi2m02 05 Jul 2017, 21:24:09

@Shandem Don't forget to update your blogpost with the exception for websites running in virtual directories.

We're using a website with a lot of content and even more images (+ imageResizer) in a virtual directory hosted on a 2008R2 server. Unfortunately, the clients server is configured that way.

Pulled my hair out last couple of days on combination lot of these behaviour last 2 days

  1. Cause: republish of a "node + all published child nodes" with a total of 200-300 descendants.
  2. somewhere in the republishing the app pool freezes/crashes (related to U4-6338)
  3. as the scheduled publishing couldn't complete, some parts of Umbraco are probably corrupted (examine, cache, distributed cache, ..)
  4. Lot of examine issues discovered in UmbracoTraceLog
  5. Initiated an App Pool stop + start.
  6. Wiped all Lucene Indexes short after when the write.lock file was released
  7. Examine has troubles getting its thing done. Maybe because of the huge amount of nodes. It keeps running and spitting lines in the UmbracoTracelog, as it probably has trouble getting the indexes ready before FCN kicks in. Multiple times "App is shutting down so index batch operation is ignored"
  8. Infinite loop of App Shutdown due to FCN Mode = Single. Application pool restarts. Examine tries to get its thing done again, ... App shutdown and again and again and again ..
  9. Changed FCNMode=Disabled
  10. Examine can finish tasks without issues
  11. UmbracoTraceLog is much more quiet now. Only every 2 minutes an error about Scheduled publishing failed and "ERROR Umbraco.Core.Sync.DatabaseServerMessenger - DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions".
  12. Did some research about this and found a record with the RefresherId and JSON of the NodeId's in dbo.umbracoCacheInstruction Table
  13. The file in DistCach folder contained the ID of that record too. Read somewhere that I could wipe the DistCach folder and this will make Umbraco forget about these records in UmbracoCacheInstruction and thus also the failed publish of step 1 (if I would have removed my whole TEMP instead of only ExamineIndexes in step 6 this probably wouldn't have occurred)
  14. Touched the web.config, App Pool restarts and the error never came back again.

I'll leave FCNMode=Disabled for now.


Shannon Deminick 06 Jul 2017, 00:00:05

@Gigi2m02 Few things:

Unfortunately all sorts of bad things happen when you have multiple concurrent appdomain shutdowns/restarts


Gigi2m02 06 Jul 2017, 00:46:56

@Shandem

  • We are Running 7.5.3 after an upgrade from 7.4.3 on project start. There is an upgrade to newest umbraco planned next week
  • The mentioned KB's have been installed a couple of months ago (according sys admin, I didn't manually check on him tho')

I thought I'll added my bits above for anyone landing this page on the future. Might help them :)


Shannon Deminick 06 Jul 2017, 01:04:40

Hi @Gigi2m02 IIRC the FCN fix as part of the core was 7.5.5 but might have been later, in any case you should definitely upgrade to the latest version. You can tell if the patch was installed since if it's not you'll get shutdown logs that say: "IIS configuration change" this is the only way that this exact shutdown message will get logged (that I've ever seen), the keyword here is the "IIS" part since other FCN shutdowns will just say "Configuration change"


Gigi2m02 12 Jul 2017, 02:21:56

In our case the logs were full of "CONFIG CHANGE" (see attached image), so I can state the patch was installed correctly. Since the client has to run our internal application as Virtual Directory, and always will, I'll add the move of Image Processor file outside of the webroot in our backlog. Next week we'll update to the latest Umbraco version, so I might restore "FCNMode=Single" in ACC and follow up closely in the Logs.


Priority: Normal

Type: Bug

State: Fixed

Assignee: Shannon Deminick

Difficulty: Normal

Category:

Backwards Compatible: True

Fix Submitted:

Affected versions:

Due in version: 7.3.5

Sprint: Sprint 6

Story Points:

Cycle: