U4-6338 - High IO operations on Windows 2012 R2/Windows 8, application shuts down with the ConfigurationChange reason

Could it be that you have Microsoft patch KB3000850 installed? The shutdown reason is the same as this one: http://stackoverflow.com/questions/28933078/application-domain-restarts-on-node-publish-umbraco/28943294#comment46410135_28943294

If KB3000850 is installed would it be possible for you to try and uninstall it and try to reproduce this issue again?

I have gotten similar errors as @rainbow and have the same setup on two servers in an NLB. I never thought that the errors were connected to long processing but it makes sense now that I think about it.

I don't see the KB3000850 patch installed in my version of Windows 2012 R2, nor is it in available updates.

@Matthew if you're running in a load balanced environment then this is currently the ONLY way it is supported so please make sure that you're following this documentation to the letter:https://our.umbraco.org/documentation/Installation/load-balancing

I've poured over that document countless times and am as sure as I can be that our servers are configured accordingly. The only thing I was doing that was slightly unconventional was using a Visual Studio Website Project rather than an Application ([http://www.theoutfield.net/blog/2014/07/why-we-use-website-projects-for-umbraco-development as endorsed by The Outfield]). I've since switched because [http://stackoverflow.com/questions/28707535/using-beginumbracoform-with-a-surface-controller-in-app-code Umbraco doesn't seem to fully support Website Projects].

Note that I saw these issues on a 7.2.1 installation. I haven't yet published my local version on 7.2.2 (may wait for the next update).

Update: MS confirms that it's due this patch which is included in KB3000850 - https://support.microsoft.com/en-us/kb/3007507 - so just removing KB3007507 should be enough.

The Microsoft page you link to has a hotfix available. Would removing KB3007507 be preferable over the hotfix?

This hotfix is included in KB3000850. The problem IS this hotfix so uninstalling just the KB3007507 hotfix should fix the publish problem. Removing KB3000850 will cause all included fixes to be removed which is a bit of overkill.

We were able to temporarily resolve the problem by adding more RAM. Somehow it helped.

I've just discovered this issue too. I'm using an Azure cloud service and can't find KB3007507 and it won't let me uninstall KB3000850! (plus I would just face the same problem when I redeploy my code to a fresh instance...)

Eagerly awaiting a fix

Update: MS is hoping to get a fix to us hopefully this week so we can test.

Awesome! Thanks for the update :)

Hi all, I've attached a zip containing the MS pre-release patch for testing. One for is 64bit and the other 32bit. If anyone can please install the patch on a patched KB3000850 Windows Server 2012 R2 machine where this issue exists, please let us know the results. NOTE: I could not replicate the issue on anything else but Windows Server 2012 R2, I don't actually think other environments are affected.

I've tested this on my environment and it fixes the issue, please let me know if any of you can test.

I'm having this issue again. Both hotfixes are not installed on the server. Anybody else seeing this ?

@dawoe did you test the hotfix I supplied above? If nobody tests this fix then it will take longer for MS to release it.

@Shandem I was the OP for this issue on Stackoverflow: http://stackoverflow.com/questions/28933078/application-domain-restarts-on-node-publish-umbraco

I can confirm the pre-release patch resolved the issue on a patched KB3000850. We are running Windows 2012 R2.

At the moment we can confirm that the patch resolved our issues (we had a recycle on overy publishing event). We are running Server 2012 R2 and Umbraco 7.2.4

I can confirm that the MS pre-release patch also resolved the issue for us, running Windows Server 2012 R2, IIS 8.5, Umbraco 4.7

Thanks!

Been hit by the issue (see http://www.zpqrtbnk.net/posts/iis-configuration-change-not), have installed the hotfix, so far it fixed the issue.

Hi - I was sent here from here: https://our.umbraco.org/forum/core/general/60378-Site-crashes-randomly-100-CPU-usage-Umbraco-624?p=0 - not at all sure I'm seeing the same issue - but I did get a server become unusable with 100% CPU in an Umbraco process. However, I can't install the patch, above - when I try I get:

Starting installation of hotfix KB3052480
Checking for expiration of the hotfix
ERROR: The test signed hotfix you are trying to install has expired. Please contact Microsoft Support to get a newer version.

Is this going to be an official patch soon? Or do I need to ask for the temporary fix to be updated?

Thanks!

@IanGrainger I get the same message from the patch, would like to know what to do. Getting alot of lines in the log like this : Umbraco.Core.UmbracoApplicationBase - [Thread 58] Application shutdown. Reason: ConfigurationChange

I've sent MS another message to see if they can give me an update on when this will be officially released with a Windows Update and unfortunately the reply back said that it could be as long as weeks to months. I've responded asking this to be a much higher priority and linked back to this ticket. Unfortunately I don't think they realize how many people this issue is affecting.

I'll reply here as soon as they get back to me.

I just received an update from Microsoft. The official fix will is planned to be released on 15/06/2015, they have said they will get back to me on the 16/06/2015 with any information if for some reason it didn't get released.

Wow, so glad I found this, we've been banging our heads for weeks! We're on a server 2008 cluster and seeing similar issues with the app restarting due to a ConfigurationChange. Unfortunately the restart also causes issues with some legacy apps we host within the Umbraco pool, otherwise we wouldn't really have noticed it.

Any word on whether the patch was officially released yesterday?

Also, do you know if MS is planning to release a fix for 2008R2, or if the above patches might work there?

I don't think they included it this month. This exact problem doesn't apply to 2008 so there will be no "fix" for that OS.

Looks like it wasn't included this month: https://technet.microsoft.com/en-us/security/bulletin/dn602597.aspx

The support team emailed me yesterday to investigate what is going on. Will report back when we have an answer.

As for Win 2008 R2, if this is affected as well I'll report that too them... hopefully they would have verified which OS's were actually affected. @tomnf the restart log entry is quite specific and you need to be sure it's this issue and not another one. I've attached my original bug report to Microsoft which includes diagnostic information and a way to replicate... perhaps you can double check based on this if it is in fact the same issue on Win 2008?

Ok, good news, a hotfix is available now: https://support.microsoft.com/en-us/kb/3052480

Please let me know if this resolves your issues. I'm not sure if this was released as part of normal Windows Updates, I've asked them.

If anybody is curious, I just installed this patch and I noticed it also apparently applies to Windows 8.1 (in addition to Windows Server 2012 R2). !os-versions.png!

Time will tell if this fixes the issue.

I've replied to Microsoft to re-iterate that this needs to be released as part of a scheduled windows update, it cannot solely exist as a hotfix. We'll see what they say.

Also, in my tests the issue didn't occur on Windows 8.1. You can easily test if it works, you can see my original reported issue attached previously which includes instructions and a link to a test site that will cause the problem. You can use that to test that the hotfix works.... though other people have already reported that it does resolve the issue.

Can anyone confirm if this is also related to my websites randomly shutting down with the log entry "Application shutdown. Reason: HostingEnvironment"?

The front-end umbraco sites don't show at all (white screen, no error codes), the admin still functions and lets me login, but when I go to view a content page it says something like "Node 1234 cannot be loaded" or something like that (I will write down the exact error next time). I have to restart the individual website in IIS (not the entire W3SCV) for it to function properly again...

It might be... please view the full error report attached above. If you have Windows Server 2012 R2 with KB3000850 installed then yes you are affected

BTW I don't have KB3000850 installed on my Windows Server 2012, but I DO have KB3000853 (https://support.microsoft.com/en-us/kb/3000853) which seems to be the same November rollup update but just for the non-R2 version. That might explain why some others can't see 3000850 installed but do experience the issue? However, the hotfix only seems to apply to R2 so it might not be related...

KB3000850 is actually a combination of patches, the actual patch that's problematic is KB3007507 so you could check for that one. KB3000850 comes with KB3007507 so that's why we keep referring to that one.

FYI The error I was getting in the admin is this: "Server error: Contact administrator, see log for full details. Failed to retrieve data for content id XXXX"

@Shandem Thanks Shannon! No luck reproducing with your code on a test server, but going to see if we can figure out a way to test in production where we know the issue is occurring. Will report back soon.

FWIW, we were seeing the recycles almost every day around the same times (between 12:00 - 2:00 AM), with various reasons such as ConfigurationChange and HostingEnvironment. They would also happen at other times occasionally, but these morning ones seem pretty consistent.

@tomnf IIS is by default configured to restart the application pool every day or so (29 hours, I think). Could it be that? Also, on the sites I work with, they are automatically taken offline and backed up once a day. Do you have something similar in place?

This works wonders on our solution! Thanks.

If anyone is interested, my issue had a similar symptom of randomly shutting down my Umbraco sites (all of them, in multi-site config). However the websites are hosted on Server 2012, not 2012 R2, so there was a different root cause aside from the one mentioned in this thread.

It was actually because Umbraco was failing to detect the baseURL (like this issue http://issues.umbraco.org/issue/U4-5965). But I am NOT running a load-balanced site, although I do have multiple websites on the same server (Umbraco & non-umbraco). It was trying to resolve on "localhost:80/umbraco/" which doesn't exist. So basically all of the scheduled tasks were failing, as well as the "Keep Alive" process, which meant that when the IIS process was recycled the site would be stuck in a limbo state (even though I configured the site as "always on").

So the fix for me was to edit umbracoSettings.config and adjust the scheduled task section to below (obviously check your logs to see if this is the issue for you too!):

@Shandem have you heard anything further about this being rolled into Windows Update?

Nope sorry, still waiting

On a large site writing umbraco.config can trigger this so after every publish the application restarts! On Azure Cloud Services the only solution at the moment is to switch your image to Server 2012 from Server 2012R2, the issue just disappears when you downgrade (how wonderfully bizarre) :)

Thanks for the feedback, I'll ask MS for an update tomorrow.

Hi everyone, here's the latest response from Microsoft:

The hotfix is about to be released in August on Windows Update.

It will be offered as an optional update for 3 weeks so customers will still need to select it for installation. But assuming no issues after 3 weeks it will automatically be elevated to recommended, so that all IIS customers receiving recommended updates will get it automatically.

So that is good news but everyone will still need to ensure that if you are running in the affected environment that you have the hotfix installed one way or another.

@Shandem that's great news, I wonder how long it will take to be added to their Azure images...

Have sent them that question, will let you know when they reply

I just wanted to mention that I was experiencing this same problem (or an unmistakably similar problem) on a Windows Server 2012 R2 machine with Umbraco 7.2.8 despite not having KB3000850 or KB3007507. However, the hotfix still fixed the issue.

Microsoft have released the patch as part of windows update:

The update is released as KB3052480 and can be received in Windows Update:

@Shandem sorry to dig this one up but do know (or can find out) if KB3052480 will become a mandatory patch? Azure images do not include non-MSRC mandatory patches so currently to use Umbraco on Azure we have to use Server 2012 (not R2) images. It seems this also affects Web Apps as I believe they are running on the latest family/os releases so really you can't reliably use Web Apps with Umbraco at all as a large umbraco.config can trigger this horrible issue!

@crumpled_jeavon yes that is a system update patch but as you say Azure is patched differently so whether they've patched it or not is up to them. I'd have to open up a support ticket to Azure to see since the last MS support ticket guy I asked this question said that it is a different team that would deal with that query. You can set fcnMode="Single", we ship with this by default which might help, otherwise you can try disabling it all together. Unfortunately however this bug is actually caused by IIS file watchers and not ASP.Net file watchers.

Be aware that I could not replicate this exact issue on Azure. You can test this exact issue by reading the bug report txt file attached to this issue at the top and using the website restarter project that it describes. I've just now re-rested this issue on my Azure subscription on Azure web apps and cannot reproduce it .

@Shandem it appears that currently Azure Web Apps are running unaffected family 3, IIS v8 (Windows Server 2012). I have been trying to find out if/when they are planning to update Web Apps to family 4 which would come with the issue unless it becomes mandatory. For now Family 3 is the way to go for Azure/Umbraco!

@Shandem I think I might also be experiencing this issue on my local Windows 10 laptop. I noticed that the patch was also for Windows 8.1 and it was issued a while ago, so maybe they also need to issue a Windows 10 patch?

@Chris.Houston You can check if it is the same issue by reading the "KB300850 bug report.txt" file attached. In that report, there is a simple website you can download to check if it is the exact same issue. I've ran this test numerous times on various environments and it's not the same issue - but if you can replicate this same issue on Windows 10 then let me know and I will have to get in touch with Microsoft again. However, I suspect you are suffering from a different issue. If you cannot replicate the issue with the website test then you'll need to try to provide me with something that I can try to reproduce on my local machine.

Hi all, we had this issue on a go-live and applied the following patch (same as mentioned above) and that resolved our issues.

https://support.microsoft.com/en-us/kb/3052480

Just wanted to add that.

Thanks, Nik

Just as an update for people running Win2008 R2:

• The WebsiteRestarter site that Shannon posted above doesn't generate any errors on a fully patched 2008R2 SP1 server (fully patched, uncluding all optional updates)
• The KB3052480 patch provided by Microsoft is only applicable to Win8.1 and 2012R2 and does not install on 2008R2

We're running a single 7.5.3 site on a locally-hosted fully patched Win2008R2 SP1 server and getting configuration-change shutdowns many times a day. Setting fcnMode=Disabled helps, but then changes to templates and partials don't appear until recycling or touching web.config, which makes it harder on the developer (me) though easier on the content editors to not have Umbraco restart so often. I've attached a bit of the tracelog and also screencap of our IIS settings, don't know if we have anything configured wrong.

{{ 2016-10-28 09:20:00,859 [P5884/D5/T34] INFO Umbraco.Core.Publishing.PublishingStrategy - Content 'Policy Advisory Committee' with Id '1113' has been published. 2016-10-28 09:20:05,788 [P5884/D5/T46] INFO umbraco.content - Save Xml to file... 2016-10-28 09:20:07,957 [P5884/D5/T46] INFO umbraco.content - Saved Xml to file. 2016-10-28 09:20:15,242 [P5884/D4/T25] INFO Umbraco.Core.UmbracoApplicationBase - Application shutdown. Details: ConfigurationChange

_shutDownMessage=Overwhelming Change Notification in HostingEnvironment initiated shutdown CONFIG change Overwhelming Change Notification in D:\inetpub\devclean CONFIG change Change Notification for critical directories. Overwhelming Change Notification in bin CONFIG change CONFIG change Change Notification for critical directories. Overwhelming Change Notification in App_WebReferences CONFIG change CONFIG change Change Notification for critical directories. Overwhelming Change Notification in App_Code CONFIG change CONFIG change Change Notification for critical directories. Overwhelming Change Notification in App_GlobalResources Change Notification for critical directories. Overwhelming Change Notification in App_Browsers CONFIG change CONFIG change CONFIG change CONFIG change HostingEnvironment caused shutdown

_shutDownStack= at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo) at System.Environment.get_StackTrace() at System.Web.Hosting.HostingEnvironment.InitiateShutdownInternal() at System.Web.Hosting.HostingEnvironment.InitiateShutdownWithoutDemand() at System.Web.HttpRuntime.ShutdownAppDomain(String stackTrace) at System.Web.HttpRuntime.OnAppOfflineFileChange(Object sender, FileChangeEvent e) at System.Web.DirectoryMonitor.FireNotifications() at System.Web.Util.WorkItem.CallCallbackWithAssert(WorkItemCallback callback) at System.Web.Util.WorkItem.OnQueueUserWorkItemCompletion(Object state) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() at System.Threading.ThreadPoolWorkQueue.Dispatch() 2016-10-28 09:20:31,341 [P5884/D5/T16] INFO umbraco.BusinessLogic.Log - Log scrubbed. Removed all items older than 2016-10-27 09:20:31 2016-10-28 09:21:03,243 [P5884/D2/T43] INFO Umbraco.Core.UmbracoApplicationBase - Application shutdown. Details: ConfigurationChange

_shutDownMessage=CONFIG change HostingEnvironment initiated shutdown CONFIG change CONFIG change Change Notification for critical directories. Overwhelming Change Notification in App_LocalResources CONFIG change CONFIG change CONFIG change Overwhelming Change Notification in CONFIG change CONFIG change CONFIG change Change Notification for critical directories. Overwhelming Change Notification in App_LocalResources CONFIG change Change Notification for critical directories. Overwhelming Change Notification in App_LocalResources CONFIG change CONFIG change CONFIG change CONFIG change Change Notification for critical directories. Overwhelming Change Notification in App_Code CONFIG change Change Notification for critical directories. Overwhelming Change Notification in bin CONFIG change CONFIG change Change Notification for critical directories. Overwhelming Change Notification in App_LocalResources CONFIG change CONFIG change Change Notification for critical directories. Overwhelming Change Notification in App_WebReferences CONFIG change CONFIG change Change Notification for critical directories. Overwhelming Change Notification in App_Browsers CONFIG change CONFIG change Change Notification for critical directories. Overwhelming Change Notification in App_LocalResources CONFIG change CONFIG change Change Notification for critical directories. Overwhelming Change Notification in App_LocalResources CONFIG change CONFIG change Overwhelming Change Notification in D:\inetpub\devclean CONFIG change CONFIG change CONFIG change Change Notification for critical directories. Overwhelming Change Notification in App_GlobalResources CONFIG change CONFIG change CONFIG change Change Notification for critical directories. Overwhelming Change Notification in App_LocalResources CONFIG change CONFIG change Change Notification for critical directories. Overwhelming Change Notification in App_LocalResources CONFIG change CONFIG change CONFIG change CONFIG change Change Notification for critical directories. Overwhelming Change Notification in App_LocalResources CONFIG change

CONFIG change CONFIG change CONFIG change User code called UnloadAppDomain User code called UnloadAppDomain HostingEnvironment caused shutdown

_shutDownStack= at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo) at System.Environment.get_StackTrace() at System.Web.Hosting.HostingEnvironment.InitiateShutdownInternal() at System.Web.Hosting.HostingEnvironment.InitiateShutdownWithoutDemand() at System.Web.HttpRuntime.ShutdownAppDomain(String stackTrace) at System.Web.Configuration.HttpConfigurationSystem.OnConfigurationChanged(Object sender, InternalConfigEventArgs e) at System.Configuration.Internal.InternalConfigRoot.OnConfigChanged(InternalConfigEventArgs e) at System.Configuration.BaseConfigurationRecord.OnStreamChanged(String streamname) at System.Web.DirectoryMonitor.FireNotifications() at System.Web.Util.WorkItem.CallCallbackWithAssert(WorkItemCallback callback) at System.Web.Util.WorkItem.OnQueueUserWorkItemCompletion(Object state) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() at System.Threading.ThreadPoolWorkQueue.Dispatch() 2016-10-28 09:22:11,168 [P5884/D5/T17] ERROR Umbraco.Web.Scheduling.KeepAlive - Failed (at "https://www./umbraco"). System.Threading.Tasks.TaskCanceledException: A task was canceled. at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Umbraco.Web.Scheduling.KeepAlive.d__4.MoveNext() 2016-10-28 09:25:25,089 [P5040/D2/T1] INFO Umbraco.Core.CoreBootManager - Umbraco 7.5.3 application starting on W3 }}

Could this issue be re-opened since both Win2008 users and those who have applied the recommended Win2012 patch (https://our.umbraco.org/forum/umbraco-7/using-umbraco-7/75369-application-pool-recycle-when-i-publish-content#comment-260362) are still having the problem? I don't usually follow Umbraco issue threads much, but this one is really pretty troublesome for us, as it means keeping fcnMode set to disabled, which means essentially no back-end or partials/templates development without having to recycle the app pool all the time.

Hi Eric,

We don't re open tickets, if you have a case with steps to reproduce then please open a new ticket so we can investigate. Without being able to reproduce on a different environment will make it near impossible to figure out. So first step is to determine steps to reproduce on a different server than your current one. We've seen cases where anti virus causes these issues too.

I submitted a new ticket, http://issues.umbraco.org/issue/U4-9257

Very similar to the issue I first reported last year: http://stackoverflow.com/questions/28933078/application-domain-restarts-on-node-publish-umbraco

That resulted in KB3052480 being released, which resolved the issue. Now very similar issues are occurring again after upgrading from 7.3.4 to 7.5.4.

No MS patches were applied during this time.

@Shandem @sebastiaan Just heads up in case you don't know yet. The patch KB3052480 has been included into 3 rollup patches starting from April. So in theory it should be automatically applied to all servers that gets updated regularly. http://www.catalog.update.microsoft.com/ScopedViewInline.aspx?updateid=c31a022f-eb21-454b-a08b-99e5d33070f4

@simonech This is amazing news!! Only took them a couple of years ;)

FWIW, we suddenly had the same issues (but on every request), and the suggestion of user @schlubadub on 01 Jul 2015 fixed it for us (baseUrl on scheduledTaks element). Thanks schlubadub! No idea though what triggered this...