U4-7673 - Umbraco publish fails because of Umbraco.Core.Sync.DatabaseServerMessenger - DISTRIBUTED CACHE IS NOT UPDATED

Created by Timo 08 Jan 2016, 08:41:32 Updated by Shannon Deminick 25 Sep 2017, 23:40:51

Tags: Unscheduled

Is duplicated by: U4-7428

Is duplicated by: U4-7465

Is duplicated by: U4-8589

Is duplicated by: U4-9150

Relates to: U4-9101

Relates to: U4-9120

Relates to: U4-9150

Relates to: U4-10150

Relates to: U4-7428

Relates to: U4-7465

Subtask of: U4-9085

Umbraco 7.3.1.

We've got 1 website load balanced across two servers. The website is for normal users accessible via the domainname www.domain.com (anonymized). Both servers are also accessible via a public ip adress (anonymized), so that we can test the site on both servers

  • 111.222.333.444:10020
  • 111.222.333.444:20020

Umbraco is only used on 111.222.333.444:10020 (our masterserver) for editing and publishing and is blocked on the other server (IP restriction).

When we look in the umbracoServer table the following is available (see attachment):

When publishing we find the following error in the log (frequently), which seems to block publishing as Umbraco never recovers and never actually publishes the node. Publishing in general is only possible after a restart of the application, and sometimes only allows us to publish 1 node, and sometimes even none as the error immediately pops up again.

The error is:

 2016-01-07 16:21:55,756 [P11788/D2/T30] ERROR Umbraco.Core.Sync.DatabaseServerMessenger - DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions (6538: "[a lot of json]"). Instruction is being skipped/ignored
System.Threading.ThreadAbortException: Thread was being aborted.
   at System.Threading.Monitor.ObjWait(Boolean exitContext, Int32 millisecondsTimeout, Object obj)
   at System.Threading.SemaphoreSlim.WaitUntilCountOrTimeout(Int32 millisecondsTimeout, UInt32 startTime, CancellationToken cancellationToken)
   at System.Threading.SemaphoreSlim.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at Umbraco.Core.AsyncLock.Lock()
   at umbraco.content.UpdateDocumentCache(Document d)
   at Umbraco.Web.Cache.PageCacheRefresher.Refresh(Int32 id)
   at Umbraco.Core.Sync.DatabaseServerMessenger.RefreshByIds(Guid uniqueIdentifier, String jsonIds)
   at Umbraco.Core.Sync.DatabaseServerMessenger.NotifyRefreshers(IEnumerable`1 jsonArray)
   at Umbraco.Core.Sync.DatabaseServerMessenger.ProcessDatabaseInstructions()

Also a thing of notice, the cache instruction table is at version 6580, but both servers have a pretty old lastsynced value in the App_Data\TEMP\DistCache\SERVERNAME txt file. Server 10020 has 6537, while 20020 has 6511.

Another thing of note is that the servers themselves can only can only reach each other via their local ip addresses. The public ip address is blocked in the firewall and domain.com:80/umbraco doens`t work as the website is on ssl.

And the last thing is that we couldn't reproduce this issue when the website wasn't loadbalanced. If only 1 website used the database it al worked like a charm. But in that scenario the distributed cache didn't need to be updated.

Just one thought is that this is might be related to the use of portnumbers on the servers. We also have a related issue (http://issues.umbraco.org/issue/U4-7540) with the same site.

4 Attachments

Download umbracoCacheInstruction.csv

Download trace.txt

Download umbracoSettings.config

Comments

Shannon Deminick 08 Jan 2016, 09:17:11

Are you saying that you have 2 x IIS websites pointing to the same file system location?

Sounds like you have a configuration issue with your LB setup. You will need to provide all details about your LB setup - there are so many variables and everyone seems to setup their environments differently. Please provide exact details - preferably with steps to reproduce:

  • Umbraco version
  • Traditional LB vs Flexible LB
  • IIS setup, server setup
  • File replication vs Shared file location - if shared, please specify details - remote file server, etc... ?
  • Anything specified in your server list in umbracoSettings.config

Verify that you've followed all instructions here https://our.umbraco.org/documentation/Getting-Started/Setup/Server-Setup/load-balancing/ and have double checked all settings, configurations, followed the 'common' setup information, etc...


Sebastiaan Janssen 08 Jan 2016, 09:17:55

I think you reported this before :)


Timo 08 Jan 2016, 09:38:11

@sebastiaan, true, but the situation is different as this is our production environment and the previous one was on our beta environment which was not loadbalanced!.

@Shandem I've answered your questions:

  • The umbraco version is 7.3.1
  • We're using flexible loadbalancing
  • What specifics do you need for the IIS setup / server setup. I think we have a regular setup for IIS and the servers, but everyone does it differently
  • Each server hosts copies of the load balanced website files and a file replication service is running to ensure that all files on all servers are up to date
  • The server data in the umbracoSettings file is empty ()

We've followed the instructions to setup the loadbalanced environment.

If you need anything else, please do let me know!


Shannon Deminick 08 Jan 2016, 10:00:41

Some notes:

  • For flexible LB, your servers don't need any communication between each other btw (apart from communication to the DB).
  • You should update to the latest Umbraco 7.3.4 (7.3.5 coming soon - you could try the nightly) to ensure that it's not something specific to the version you are using.
  • Ensure that distributedCall=false - if this is true, then you are not doing flexible LB
  • Your servers should absolutely stay in sync with the db, they will always attempt to sync when a request hits the website within a 5 second threshold
  • Ensure each website can speak to itself on it's DNS address (i.e. the master address)
  • The stack trace indicates that you have 2x app domains trying to write to the umbraco xml cache file at one time

More questions:


Timo 08 Jan 2016, 10:16:06

Quick update:

  • For flexible LB, your servers don't need any communication between each other btw (apart from communication to the DB). ** Check

  • You should update to the latest Umbraco 7.3.4 (7.3.5 coming soon - you could try the nightly) to ensure that it's not something specific to the version you are using. ** Might try this in the future, our previous update tryout to 7.3.3 didn't work out very well

  • Ensure that distributedCall=false - if this is true, then you are not doing flexible LB ** This is set to false

  • Your servers should absolutely stay in sync with the db, they will always attempt to sync when a request hits the website within a 5 second threshold ** This does not happen as the cache version number is outdated on both servers

  • Ensure each website can speak to itself on it's DNS address (i.e. the master address) ** The servers don't have a DNS address configured as they are only accessible via the loadbalancer and firewall via an IP address and port number (10080, 20080).

  • The stack trace indicates that you have 2x app domains trying to write to the umbraco xml cache file at one time ** Only 1 AppPool has access to the Umbraco XML, so I don't know how this could be happening?

  • Since you are using file replication, what are you including/excluding from replication? ** Only the media directory and the "\App_Plugins\UmbracoForms\Data" folder are synced one way (master to slave).

  • Did you follow these instructions? https://our.umbraco.org/documentation/Getting-Started/Setup/Server-Setup/load-balancing/files-replicated ** We did, we have a custom application that keeps the files in sync, but only for the media and the Umbraco forms data folder

  • Do you have other errors in your logs? ** Mostly the scheduled publishing which fails because it cannot reach it's own domain on www.domain.com:80/umbraco, as it is only accessible via a local ip address (which might be resolved by configuring umbracoApplicationUrl). We don't use scheduled publishing btw. In the event viewer we are seeing a lot of "The request has been aborted. (Request timed out.)" error messages

  • Do you have lots of app domain restarts occurring on either server? ** No, the web application seems to keep running throughout the day


Shannon Deminick 08 Jan 2016, 10:51:13

Your servers should absolutely stay in sync with the db, they will always attempt to sync when a request hits the website within a 5 second threshold

This does not happen as the cache version number is outdated on both servers

= this is issue #1 - you need to figure out why - something must be in the logs regarding syncing from the db.

Ensure each website can speak to itself on it's DNS address (i.e. the master address)

The servers don't have a DNS address configured as they are only accessible via the loadbalancer and firewall via an IP address and port number (10080, 20080).

= this is issue #2 - you need to allow umbraco to communicate with itself, so yes you must set umbracoApplicationUrl - regardless of whether you use scheduled publishing or not, umbraco needs to communicate with itself

It would be best to update to the latest Umbraco version.


Timo 08 Jan 2016, 12:19:07

Hi @Shandem,

Updated information, i've set the umbracoApplicationUrl to the local ip address of both servers (so for each server the local ip address). This allowed me to publish again on server 1 (master) and run an import (which published a couple of 100 nodes after each other), but it doens't sync the changes back to server 2 (slave). The lastsynced value is now updated on server 1, but stays the same on server 2. Server 2 does seem to constantly update the umbraco.config, but i can't find my last changes in there. Also the umbraco scheduled publishing is now working again (no longer appears in the log files)

The exception is a bit bigger on server 2:

2016-01-08 13:10:49,976 [P4608/D2/T8] ERROR umbraco.DataLayer.SqlHelper`1[[System.Data.SqlClient.SqlParameter, System.Data, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]] - Error executing query SELECT id, createDate, trashed, parentId, nodeObjectType, nodeUser, level, path, sortOrder, uniqueID, text FROM umbracoNode WHERE id = @id
System.Threading.ThreadAbortException: Thread was being aborted.
   at SNIReadSyncOverAsync(SNI_ConnWrapper* , SNI_Packet** , Int32 )
   at SNINativeMethodWrapper.SNIReadSyncOverAsync(SafeHandle pConn, IntPtr& packet, Int32 timeout)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
   at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
   at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
   at System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
   at Microsoft.ApplicationBlocks.Data.SqlHelper.ExecuteReader(String connectionString, CommandType commandType, String commandText, SqlParameter[] commandParameters)
   at umbraco.DataLayer.SqlHelpers.SqlServer.SqlServerHelper.ExecuteReader(String commandText, SqlParameter[] parameters)
   at umbraco.DataLayer.SqlHelper`1.ExecuteReader(String commandText, IParameter[] parameters)
 
2016-01-08 13:10:50,616 [P4608/D2/T8] ERROR Umbraco.Core.Sync.DatabaseServerMessenger - DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions (6586: "[JSON BLOB]"). Instruction is being skipped/ignored
umbraco.DataLayer.SqlHelperException: Umbraco Exception (DataLayer): SQL helper exception in ExecuteReader ---> System.Threading.ThreadAbortException: Thread was being aborted.
   at SNIReadSyncOverAsync(SNI_ConnWrapper* , SNI_Packet** , Int32 )
   at SNINativeMethodWrapper.SNIReadSyncOverAsync(SafeHandle pConn, IntPtr& packet, Int32 timeout)
   at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
   at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
   at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
   at System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
   at System.Data.SqlClient.SqlDataReader.get_MetaData()
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, SqlDataReader ds)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean asyncWrite)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)
   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
   at Microsoft.ApplicationBlocks.Data.SqlHelper.ExecuteReader(String connectionString, CommandType commandType, String commandText, SqlParameter[] commandParameters)
   at umbraco.DataLayer.SqlHelpers.SqlServer.SqlServerHelper.ExecuteReader(String commandText, SqlParameter[] parameters)
   at umbraco.DataLayer.SqlHelper`1.ExecuteReader(String commandText, IParameter[] parameters)
   --- End of inner exception stack trace ---
   at umbraco.DataLayer.SqlHelper`1.ExecuteReader(String commandText, IParameter[] parameters)
   at umbraco.cms.businesslogic.CMSNode.setupNode()
   at umbraco.cms.businesslogic.CMSNode.get_Parent()
   at umbraco.content.PublishNodeDo(Document d, XmlDocument xmlContentCopy, Boolean updateSitemapProvider)
   at umbraco.content.UpdateDocumentCache(Document d)
   at Umbraco.Web.Cache.PageCacheRefresher.Refresh(Int32 id)
   at Umbraco.Core.Sync.DatabaseServerMessenger.RefreshByIds(Guid uniqueIdentifier, String jsonIds)
   at Umbraco.Core.Sync.DatabaseServerMessenger.NotifyRefreshers(IEnumerable`1 jsonArray)
   at Umbraco.Core.Sync.DatabaseServerMessenger.ProcessDatabaseInstructions()


Shannon Deminick 08 Jan 2016, 12:24:59

What SQL server are you using is it Azure SQL ?

The way that the server's sync is based on active requests being received by them. They don't actively sync themselves in the background. If you send a request to a front-end umbraco page of server 2, you should see it sync.


Timo 08 Jan 2016, 12:29:07

@Shandem

We're using MSSQL SERVER 2014, no Azure. One of the refresh instructions is also quite big (probably because of the import action). Could it be that when the instrunction is to big it fails to execute because of it's size?:

[{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[16114]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[16113]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14514]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14522]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14523]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[15980]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14541]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14540]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14517]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14535]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14546]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14545]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[15236]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14525]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[16054]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14560]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14533]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14568]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14569]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14565]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14566]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14510]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14530]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14552]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14507]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14520]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14534]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14515]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14557]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14555]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14519]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14539]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14556]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14538]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[15261]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14536]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14567]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14509]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14511]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14521]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14524]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14531]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14547]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14548]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14549]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14550]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14529]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14551]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14563]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14559]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14553]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[15262]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14537]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14527]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14554]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14528]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14564]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14526]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14562]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14518]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14544]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14532]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14543]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14516]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14561]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14570]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14571]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14572]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14573]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14574]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14575]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14576]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14577]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14578]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14579]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14580]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14581]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14582]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14583]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14584]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14585]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14586]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14587]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14588]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14589]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14590]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14591]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14592]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14593]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14594]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14595]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14596]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14597]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14598]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14599]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14600]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14601]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14602]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14603]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14604]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14605]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14618]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14619]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14620]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14621]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14622]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14623]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14624]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14625]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14626]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14627]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14628]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14629]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14630]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14631]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14632]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14633]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14634]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14635]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14636]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14637]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14638]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14639]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14640]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14641]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14642]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14643]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14644]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14645]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14646]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[15263]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14656]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14657]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14658]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14659]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14660]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14661]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14662]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14663]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14664]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14665]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14666]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14667]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14668]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14669]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14670]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14671]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14672]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14673]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14674]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14675]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14676]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14677]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14678]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14679]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14680]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14681]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14682]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14683]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14684]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14685]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14686]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14687]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14688]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14689]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14690]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14691]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[15270]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[15271]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[15272]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14695]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14696]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14697]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14699]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14700]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14701]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14705]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14706]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14707]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14709]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14710]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14711]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14712]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14713]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14714]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14715]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14716]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14717]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14718]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14719]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14720]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14721]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14722]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14723]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14724]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14725]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14726]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14727]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14728]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14729]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14730]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14731]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14732]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14733]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14734]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14735]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14736]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14737]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14738]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14739]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14740]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14741]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14742]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14743]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14744]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14745]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14746]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14748]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14749]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14750]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14752]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14753]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14754]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14755]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14756]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14757]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14758]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14759]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14760]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14761]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14762]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14763]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14764]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14765]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14766]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14767]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14769]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14770]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14771]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14772]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14773]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14774]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14775]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14776]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14777]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14779]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14780]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14781]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14783]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14784]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14785]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14787]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14788]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14789]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14790]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14791]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14792]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14793]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14794]","JsonPayload":null},{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[14815]","JsonPayload":null


Shannon Deminick 08 Jan 2016, 12:37:02

SQL Azure requires a retry policy and the last exception you have posted can occur because it's using a legacy API call that doesn't have a retry policy. The size of the instruction shouldn't matter really since it's just performing a bulk operation.

That said, based on your stack trace I have determine that there's an optimization issue that is quite easily fixed. During publishing there are some SQL queries executing that are completely not required. I've made a note of this here: http://issues.umbraco.org/issue/U4-7681 ... which I'll update now and will be available in 7.3.5 (including the nightly).

This is an optimization though, it still seems odd that you are getting that exception as it normally indicates some sort of connectivity issue. If you Google SNIReadSyncOverAsync see if any of those results pertain to your setup.


Timo 08 Jan 2016, 12:52:55

I just updated the executionTimeout to 600 and restarted the website. Hope that that fixes the timeout issue. not seeing the error popup just yet, also no update in the lastsynced file.

I'll keep you updated.


Shannon Deminick 08 Jan 2016, 13:10:43

Ok cool. I did just realize how large that instruction is... quite large... but still, this is just a bulk operation and each item needs to be handled. The only other option we'd have for processing a ton of instructions at once is to add a threshold in the startup config to say: if there are more than X number of instructions pending, then do a 'cold boot' instead - which means that the server is treated as a new server and all caches, xml caches and indexes would be rebuilt from scratch.

The issue U4-7681 will help with bulk ops like that since there will be one less SQL query per item published.

Keep us posted on your findings.


Timo 08 Jan 2016, 13:15:46

Hi Shannon,

The executionTimeout updated to 600 allowed the website to execute the cache instruction so the second server is now also up-to-date!, which is excellent!

So it appears that when a lot of nodes are updated, the execution timeout might prevent the server from updating to the new instructions.

Thank you for your help in this, you saved my weekend :-).

Edit Indeed, a total refresh of the cache might be more efficient than updating 1 at a time with such a large instructionset.


Shannon Deminick 08 Jan 2016, 13:20:02

Good news, i'll close this issue and create a task to add that startup option.


Shannon Deminick 16 Jan 2016, 16:34:07

The task to add a startup option is here: http://issues.umbraco.org/issue/U4-7682


Timo 02 Mar 2016, 12:19:57

Hi @Shandem,

We upgraded the installation to 7.3.8 (from 7.3.1) and implemented a custom DatabaseServerMessengerOptions with a lower MaxProcessingInstructionCount (400). However it doesn`t seem to use the new setup as I still see instructions with a count over 400 (700+) failing.

Our used code is:

public class Global : UmbracoApplication
{
    protected override void OnApplicationStarted(object sender, EventArgs e)
    {
        base.OnApplicationStarted(sender, e);
        RouteConfig.RegisterRoutes(RouteTable.Routes);
        BundleConfig.RegisterBundles(BundleTable.Bundles);
    }

    public void OnApplicationInitialized(UmbracoApplicationBase umbracoApplication, ApplicationContext applicationContext)
    {
    }

    protected override void OnApplicationStarting(object sender, EventArgs e)
    {
        ServerMessengerResolver.Current.SetServerMessenger(new BatchedDatabaseServerMessenger(
            ApplicationContext.Current,
            true,
            //Default options for web including the required callbacks to build caches
            new DatabaseServerMessengerOptions
            {
                // It does a cold boot if the number exceeds the specified instruction amount
                MaxProcessingInstructionCount = 400, // TH: set this to 400
                //These callbacks will be executed if the server has not been synced
                // (i.e. it is a new server or the lastsynced.txt file has been removed)
                InitializingCallbacks = new Action[]
                {
                    //rebuild the xml cache file if the server is not synced
                    () => global::umbraco.content.Instance.RefreshContentFromDatabase(),
                    // rebuild indexes if the server is not synced
                    // NOTE: This will rebuild ALL indexes including the members, if developers want to target specific 
                    // indexes then they can adjust this logic themselves.                        
                    () =>
                    {
                        //If the developer has explicitly opted out of rebuilding indexes on startup then we 
                        // should adhere to that and not do it, this means that if they are load balancing things will be
                        // out of sync if they are auto-scaling but there's not much we can do about that.
                        if (ExamineSettings.Instance.RebuildOnAppStart == false) return;

                        foreach (var indexer in WebBootManager.GetIndexesForColdBoot())
                        {
                            indexer.RebuildIndex();
                        }
                    }
                }
            }));

        base.OnApplicationStarting(sender, e);
    }

The OnApplicationStarting method is called on startup. Did I do anything wrong with the MaxProcessingInstructionCount implementation?


Shannon Deminick 07 Mar 2016, 12:27:08

Hi (sorry been on holidays). When you say it's still 'failing' do you mean that it's not performing a Cold Boot when the count is over 400 ?


Timo 07 Mar 2016, 12:30:35

Hi @Shandem Correct, I still see the old "ERROR Umbraco.Core.Sync.DatabaseServerMessenger - DISTRIBUTED CACHE IS NOT UPDATED Thread was being aborted." exception in the logs, and the cache is outdated (the distcache id is not updated).


Shannon Deminick 07 Mar 2016, 12:36:03

OK, i'm going to have to try to replicate locally. Are you able to replicate this issue locally?


Timo 10 Mar 2016, 13:40:07

Hi @Shandem,

Not yet.


Neil Gaietto 05 May 2016, 14:38:46

We are experiencing this issue as well. Unfortunately the workaround does not solve the problem for us. This is something that randomly happens on our prod servers, consisting of 2 web and 1 admin boxes. The only way we can resolve the lock is by restarting the servers. We have spent a lot of time trying to reproduce this locally as well, but no luck.


Daniel Symonds-Lloyd 05 May 2016, 16:07:56

I have experienced this in a site hosted in Azure Websites. The site is a slave server. I worked around the issue by scaling up, hence causing the site to be hosted on a different server, clearing the queue lock.


Daniel Symonds-Lloyd 24 May 2016, 15:21:39

The way we are currently working around this is:

Commit changes to source control

Create new Azure App Service plan

Deploy to admin website

Deploy to staging production slot

Test staging slot

Swap slots

Change production site to new App Service plan

Change staging slot to new App Service plan

Delete existing App Service Plan

This ensures a new server is hosting the website, then cache is rebuilt and search index is rebuilt. There is downtime while the production site starts up on the new app service plan.


psaenz 15 Jul 2016, 21:40:08

Hi, same problem here, my configuration is: #. Umbraco 7.4.3 #. In Azure #. Load balanced using 2 instances (just for the front ends) with Flexible LB #. Backoffice is installed in a different App Service (not load balanced)

The problem I'm having is that one front end instance is in synch with the CacheInstruction table, with the instruction #48719, but the other is not, the other has the instruction #48201...'''518 instructions behind!!'''

I can see the error happening each 10 minutes in the logs, and it says:

ERROR Umbraco.Core.Sync.DatabaseServerMessenger - DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions (48202: "[{"RefreshType":3,"RefresherId":"27ab3022-3dfa-47b6-9119-5945bc88fd66","GuidId":"00000000-0000-0000-0000-000000000000","IntId":0,"JsonIds":"[381644]","JsonPayload":null}]"). Instruction is being skipped/ignored System.Threading.ThreadAbortException: Thread was being aborted. at System.Threading.Monitor.ObjWait(Boolean exitContext, Int32 millisecondsTimeout, Object obj) at System.Threading.Monitor.Wait(Object obj, Int32 millisecondsTimeout, Boolean exitContext) at System.Threading.SemaphoreSlim.WaitUntilCountOrTimeout(Int32 millisecondsTimeout, UInt32 startTime, CancellationToken cancellationToken) at System.Threading.SemaphoreSlim.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) at Umbraco.Core.AsyncLock.Lock() at umbraco.content.UpdateDocumentCache(Document d) at umbraco.content.UpdateDocumentCache(Int32 pageId) at Umbraco.Web.Cache.PageCacheRefresher.Refresh(Int32 id) at Umbraco.Core.Sync.DatabaseServerMessenger.RefreshByIds(Guid uniqueIdentifier, String jsonIds) at Umbraco.Core.Sync.DatabaseServerMessenger.NotifyRefreshers(IEnumerable`1 jsonArray) at Umbraco.Core.Sync.DatabaseServerMessenger.ProcessDatabaseInstructions()


Shannon 12 Oct 2016, 00:10:37

I am also having this problem although I do not have a load balancer set up. My problem seems to be around importing mass records. I used the CMSImport plugin and imported 1000's of records. The system then seems to get stuck with large messages in the umbracoCacheInstruction queue. I only have 27 records. I have attacked a dump of my current table.


Eric Schrepel 13 Oct 2016, 20:19:36

Same issue here, after NuGet upgrade from 7.4.3 to 7.5.3, tracelog is full of these messages followed by big JSON blobs of nodeids, then sometimes a collection of "Overwhelming change notification" messages, then IIS shuts down and restarts the site. (Sample chunk of trace log attached.) Site still runs, but restarts frequently. (Our site runs on a single self-hosted Win 2008 R2 server / SQL)


Shannon Deminick 26 Oct 2016, 09:58:04

Hi all, An update on this issue:

We have seen this issue in various installs but until now there wasn't one particular thing that would cause it. In some cases it was due to 3rd party libs, user code or the environment that was either not letting the current app domain shut down gracefully or terminating it unexpectedly or shutting down many times consecutively.

A work around is to clear the /App_Data/TEMP/DistCache files for the server you are seeing problems on and then bump the web.config, this will cause a cold-boot to occur and the site should be back up and running once caches and indexes are built.

We have fortunately this week discovered a reason why this occurs due to some core code in certain circumstances. Namely that a very large batch of instructions has been written to a single json blob (and put into a single row in the db). During startup these instructions will be processed, however the single json blob might contain thousands and we were not checking the number of parsed instructions in a single blob against the MaxProcessingInstructionCount. This means that during startup there could be tons of instructions to process which (depending on the instruction type) could take quite some time. During instruction processing there is no check to see if the app domain is shutting down again and a lock is created to prevent the app domain from shutting down until the instructions are done. This will cause problems because if the app domain does not shut down in a timely manner it will be forcefully shut off one way or another (i.e. IIS, etc...). The other reason this causes problems is because IIS will launch another app domain to run in tandem with the one shutting down which means there are now > 1 process trying to access the same resources. Lastly, the reason why the stack traces show Thread was being aborted is because the instructions are processed as part of a request thread and there's a request timeout limit that will be reached.

Some fixes for this which will be available in 7.5.5 are:

  • De-duplicating the instructions that are being processed - in many cases we don't need to process duplicate instructions on startup which will save some processing power
  • Ensuring that only the MaxProcessingInstructionCount number of instructions are ever written to a json instruction blob per row - so that there is never too many instructions being processed at once at startup
  • During syncing (instruction processing), the request used to start instruction processing will only query for the top 100 db rows instead of every row
  • During instruction writing, the maximum number of instructions written to any db row will be MaxProcessingInstructionCount
  • During instruction processing we will check if the app domain has been signaled to shut down and if so we'll exit the processing loop to allow graceful/quick shut down. This of course means the current instruction batch will need to be reprocessed, but that is ok.

In 7.6, we'll add another column to the instruction db table to indicate the number of instructions in a given json blob that the db row contains. This will provide us with some more control/granularity over the number of instructions queried/processed at a given time and also a more accurate way to detect if a cold-boot is necessary.


Shannon Deminick 26 Oct 2016, 12:26:38

PR for this is part of http://issues.umbraco.org/issue/U4-9101


Stephan 26 Oct 2016, 17:27:14

U4-9101 fixed, so marking this one fixed too


Simon Dingley 05 Nov 2016, 18:20:10

Glad that there is some understanding of this now. Is there anything we can do to prevent this in the meantime? I know you've posted a workaround for it once it's happened but a site we experience similar issues on does this in the middle of the night around 2am for some unknown reason and rebuilding indexes takes in excess of 30 minutes and the site is out of action during that time.


Laurence Gillian 18 Nov 2016, 09:20:12

I have just woken up to a website (7.5.4) down from this issue, fingers crossed the trick above solves this again!


Sebastiaan Janssen 18 Nov 2016, 10:01:36

7.5.5 is out next week. :) If you can't wait then there's nightlies available!


Simon Dingley 18 Nov 2016, 10:08:31

@Laurence.Gillian I cheated and removed the offending cache instruction then made sure the last sync id's all matched the latest in the table and rebuilt the indexes on each node behind the load balancer.


Laurence Gillian 18 Nov 2016, 10:13:48

I had, what I can only describe as a triple death rainbow of doom ;-)

  • Cache instructions => Clear Folder to Resolve
  • Hanging start due to Published Scheduling => Updated umbracoSettings.config to contain "www.domain.com/umbraco"
  • Azure CDN Toolkit was unable to access it's CDN images, as think 500 errors had been cached => Disabled in web.config

Back online! Phew!


Vilen 19 May 2017, 11:42:31

@zpqrtbnk @Shandem I still have this problem in 7.5.13, so it is actually NOT fixed

2017-05-19 14:36:42,569 [P11036/D2/T7] ERROR Umbraco.Core.Sync.DatabaseServerMessenger - DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions (id: 11294, instruction count: 576). Instruction is being skipped/ignored
The thread has been aborted, because the request has timed out.
System.Threading.ThreadAbortException: Thread was being aborted.
   at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable)
   at System.Collections.ArrayList.set_Capacity(Int32 value)
   at System.Collections.ArrayList.EnsureCapacity(Int32 min)
   at System.Collections.ArrayList.Add(Object value)
   at System.Xml.XmlNamedNodeMap.AddNode(XmlNode node)
   at System.Xml.XmlAttributeCollection.InternalAppendAttribute(XmlAttribute node)
   at System.Xml.XmlDocument.ImportAttributes(XmlNode fromElem, XmlNode toElem)
   at System.Xml.XmlDocument.ImportNodeInternal(XmlNode node, Boolean deep)
   at System.Xml.XmlDocument.ImportChildren(XmlNode fromNode, XmlNode toNode, Boolean deep)
   at System.Xml.XmlDocument.ImportNodeInternal(XmlNode node, Boolean deep)
   at System.Xml.XmlDocument.ImportChildren(XmlNode fromNode, XmlNode toNode, Boolean deep)
   at System.Xml.XmlDocument.ImportNodeInternal(XmlNode node, Boolean deep)
   at System.Xml.XmlDocument.ImportChildren(XmlNode fromNode, XmlNode toNode, Boolean deep)
   at System.Xml.XmlDocument.ImportNodeInternal(XmlNode node, Boolean deep)
   at System.Xml.XmlDocument.ImportChildren(XmlNode fromNode, XmlNode toNode, Boolean deep)
   at System.Xml.XmlDocument.ImportNodeInternal(XmlNode node, Boolean deep)
   at System.Xml.XmlDocument.ImportChildren(XmlNode fromNode, XmlNode toNode, Boolean deep)
   at System.Xml.XmlDocument.ImportNodeInternal(XmlNode node, Boolean deep)
   at System.Xml.XmlDocument.ImportChildren(XmlNode fromNode, XmlNode toNode, Boolean deep)
   at System.Xml.XmlDocument.CloneNode(Boolean deep)
   at umbraco.content.Clone(XmlDocument xmlDoc)
   at umbraco.content.SafeXmlReaderWriter..ctor(content instance, IDisposable releaser, Boolean isWriter, Boolean auto)
   at umbraco.content.UpdateDocumentCache(Document d)
   at Umbraco.Web.Cache.PageCacheRefresher.Refresh(Int32 id)
   at Umbraco.Core.Sync.DatabaseServerMessenger.RefreshByIds(Guid uniqueIdentifier, String jsonIds)
   at Umbraco.Core.Sync.DatabaseServerMessenger.NotifyRefreshers(IEnumerable`1 instructions, HashSet`1 processed)
   at Umbraco.Core.Sync.DatabaseServerMessenger.ProcessDatabaseInstructions(IReadOnlyCollection`1 instructionBatch, CacheInstructionDto dto, HashSet`1 processed, Int32& lastId)
2017-05-18 23:59:05,345 [P2500/D4/T125] ERROR Umbraco.Core.Sync.DatabaseServerMessenger - DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions (id: 11293, instruction count: 391). Instruction is being skipped/ignored
The thread has been aborted, because the request has timed out.
System.Threading.ThreadAbortException: Thread was being aborted.
   at System.Threading.Monitor.ObjWait(Boolean exitContext, Int32 millisecondsTimeout, Object obj)
   at System.Threading.SemaphoreSlim.WaitUntilCountOrTimeout(Int32 millisecondsTimeout, UInt32 startTime, CancellationToken cancellationToken)
   at System.Threading.SemaphoreSlim.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at Umbraco.Core.AsyncLock.Lock()
   at umbraco.content.UpdateDocumentCache(Document d)
   at Umbraco.Web.Cache.PageCacheRefresher.Refresh(Int32 id)
   at Umbraco.Core.Sync.DatabaseServerMessenger.RefreshByIds(Guid uniqueIdentifier, String jsonIds)
   at Umbraco.Core.Sync.DatabaseServerMessenger.NotifyRefreshers(IEnumerable`1 instructions, HashSet`1 processed)
   at Umbraco.Core.Sync.DatabaseServerMessenger.ProcessDatabaseInstructions(IReadOnlyCollection`1 instructionBatch, CacheInstructionDto dto, HashSet`1 processed, Int32& lastId)


Shannon Deminick 22 May 2017, 04:28:01

@xumix You'll need to provide more information about this:

  • You 'still' have this problem - but what have you done to reproduce it? ** Have you previously tried cold booting this instance? (i.e. delete the App_Data/TEMP/Distcache folder and bump the web config), this will make the site update all it's caches from the db and update it's latest instruction read ID. ** Having 576 being processed at once is quite large, what are you doing to get that many instructions registered? and did this occur even after you've tried cold booting?

The thread is aborting because instruction processing occurs in a web request so your web request is timing out before it completes. You can change your timeout settings in see https://msdn.microsoft.com/en-us/library/e1f13641%28v=vs.71%29.aspx?f=255&MSPPError=-2147217396


Vilen 22 May 2017, 08:03:28

@Shandem 1) I've tried cold boot and it helped, but this is not a fix . For example I add some prop and set it on a whole site catalog. 2) I've just republished one of my site's catalogs

executionTimeout is just the same on the master node and slaves, but master performs publish operations without problems. Btw publish lasts for 5 min or so and does not timeout, at least when I execute it from localhost


Shannon Deminick 22 May 2017, 08:40:52

@xumix the master server processes instructions totally differently. Are you on Azure and if so what SQL server type are you using ?


Vilen 22 May 2017, 08:44:59

@Shandem I'm not on Azure, Win2008 R2, Sql server 2012 SP3, executionTimeout =120 Why is background worker timeout tied to executionTimeout at all?


Shannon Deminick 22 May 2017, 08:48:51

As noted above:

The thread is aborting because instruction processing occurs in a web request so your web request is timing out before it completes. You can change your timeout settings in see https://msdn.microsoft.com/en-us/library/e1f13641%28v=vs.71%29.aspx?f=255&MSPPError=-2147217396

That's a long time for instruction processing, do you have events tied to cache refreshing updates? Is this a site that was upgraded from a very old site?


Jeroen Breuer 22 May 2017, 08:54:59

@xumix Please read this topic starting with this post: https://our.umbraco.org/forum/umbraco-7/using-umbraco-7/73020-flexible-load-balancing-vs-traditional-load-balancing#comment-262732

I had the same issue you reported, but it was a problem with my code. In my situation I'd try to rebuild Examine with each instruction. That took way too much time and that's what caused the DISTRIBUTED CACHE IS NOT UPDATED error in my situation.


Shannon Deminick 22 May 2017, 09:26:58

@jbreuer yes rebuilding indexes is very expensive and should be avoided entirely :) https://our.umbraco.org/documentation/reference/Common-Pitfalls/#rebuilding-indexes


Jeroen Breuer 22 May 2017, 09:49:49

@Shandem That's what we also discussed in that topic ;-). Just wanted to mention it as an example that if you have a very expensive task as part of your instruction that could also cause the DISTRIBUTED CACHE IS NOT UPDATED error.


Shannon Deminick 22 May 2017, 09:53:04

yup :)


Jeroen Breuer 04 Jul 2017, 13:54:28

@xumix Do you still have this error? I've just encountered the same error on Umbraco 7.5.13:

2017-07-04 12:00:52,336 [29] ERROR Umbraco.Core.Sync.DatabaseServerMessenger - DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions (id:11173, instruction count: 1). Instruction is being skipped/ignored The thread has been aborted, because the request has timed out. System.Threading.ThreadAbortException: Thread was being aborted. at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable) at System.Collections.ArrayList.set_Capacity(Int32 value) at System.Collections.ArrayList.Add(Object value) at System.Xml.XmlNamedNodeMap.AddNode(XmlNode node) at System.Xml.XmlAttributeCollection.SetNamedItem(XmlNode node) at System.Xml.XmlDocument.ImportAttributes(XmlNode fromElem, XmlNode toElem) at System.Xml.XmlDocument.ImportNodeInternal(XmlNode node, Boolean deep) at System.Xml.XmlDocument.ImportChildren(XmlNode fromNode, XmlNode toNode, Boolean deep) at System.Xml.XmlDocument.ImportNodeInternal(XmlNode node, Boolean deep) at System.Xml.XmlDocument.ImportChildren(XmlNode fromNode, XmlNode toNode, Boolean deep) at System.Xml.XmlDocument.ImportNodeInternal(XmlNode node, Boolean deep) at System.Xml.XmlDocument.ImportChildren(XmlNode fromNode, XmlNode toNode, Boolean deep) at System.Xml.XmlDocument.ImportNodeInternal(XmlNode node, Boolean deep) at System.Xml.XmlDocument.ImportChildren(XmlNode fromNode, XmlNode toNode, Boolean deep) at System.Xml.XmlDocument.ImportNodeInternal(XmlNode node, Boolean deep) at System.Xml.XmlDocument.ImportChildren(XmlNode fromNode, XmlNode toNode, Boolean deep) at System.Xml.XmlDocument.ImportNodeInternal(XmlNode node, Boolean deep) at System.Xml.XmlDocument.ImportChildren(XmlNode fromNode, XmlNode toNode, Boolean deep) at System.Xml.XmlDocument.ImportNodeInternal(XmlNode node, Boolean deep) at System.Xml.XmlDocument.ImportChildren(XmlNode fromNode, XmlNode toNode, Boolean deep) at System.Xml.XmlDocument.CloneNode(Boolean deep) at umbraco.content.SafeXmlReaderWriter..ctor(content instance, IDisposable releaser, Boolean isWriter, Boolean auto) at umbraco.content.UpdateDocumentCache(Document d) at Umbraco.Web.Cache.PageCacheRefresher.Refresh(Int32 id) at Umbraco.Core.Sync.DatabaseServerMessenger.RefreshByIds(Guid uniqueIdentifier, String jsonIds) at Umbraco.Core.Sync.DatabaseServerMessenger.NotifyRefreshers(IEnumerable1 instructions, HashSet1 processed) at Umbraco.Core.Sync.DatabaseServerMessenger.ProcessDatabaseInstructions(IReadOnlyCollection1 instructionBatch, CacheInstructionDto dto, HashSet1 processed, Int32& lastId)

2017-07-04 12:00:52,929 [8] ERROR Umbraco.Core.Sync.DatabaseServerMessenger - DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions (id:11173, instruction count: 1). Instruction is being skipped/ignored The thread has been aborted, because the request has timed out. System.Threading.ThreadAbortException: Thread was being aborted. at System.Threading.Monitor.ObjWait(Boolean exitContext, Int32 millisecondsTimeout, Object obj) at System.Threading.SemaphoreSlim.WaitUntilCountOrTimeout(Int32 millisecondsTimeout, UInt32 startTime, CancellationToken cancellationToken) at System.Threading.SemaphoreSlim.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) at Umbraco.Core.AsyncLock.Lock() at umbraco.content.UpdateDocumentCache(Document d) at Umbraco.Web.Cache.PageCacheRefresher.Refresh(Int32 id) at Umbraco.Core.Sync.DatabaseServerMessenger.RefreshByIds(Guid uniqueIdentifier, String jsonIds) at Umbraco.Core.Sync.DatabaseServerMessenger.NotifyRefreshers(IEnumerable1 instructions, HashSet1 processed) at Umbraco.Core.Sync.DatabaseServerMessenger.ProcessDatabaseInstructions(IReadOnlyCollection1 instructionBatch, CacheInstructionDto dto, HashSet1 processed, Int32& lastId)

We've cleared the umbracoCacheInstruction table, but that didn't help. Last time we've fixed the issue by making sure the cache instruction wasn't doing any heavy work. Nothing has changed since than and suddenly we got these errors again.


Vilen 04 Jul 2017, 13:56:56

@jbreuer yup, 7.6.3 still the same


Vilen 04 Jul 2017, 14:01:11

@Shandem why do you run background task in http thread for the first place? It must be untied from the http runtime at all.

 2017-06-30 11:03:39,357 [P12504/D2/T35] ERROR Umbraco.Core.Sync.DatabaseServerMessenger - DISTRIBUTED CACHE IS NOT UPDATED. Failed to execute instructions (id: 26108, instruction count: 2). Instruction is being skipped/ignored
The thread has been aborted, because the request has timed out.
System.Threading.ThreadAbortException: Thread was being aborted.

This bug is definitely NOT fixed


Simon Dingley 04 Jul 2017, 14:03:04

Following large bulk operations? I still think the size of an instruction is an issue in the version my customer is running but can't say for certain in the versions you mentioned. If I delete the offending instruction and remove the lastsync file I think it sorted it out.


Vilen 04 Jul 2017, 14:05:23

@ProNotion no large operation at all: instruction count = 2(!)


Jeroen Breuer 04 Jul 2017, 14:09:24

Even if the instruction count is 2 it could still timeout if the instruction takes a very long time. In my case the instruction should be pretty simple. I'll add some logging to see if I can find the issue.


Shannon Deminick 04 Jul 2017, 14:09:27

One row can contain the Max count of instructions. I'd be interested to see what is in these 2 Rows if you can post that information here.

Also, we dont run a background task on the http request, the instructions are processed at the beginning of the http request, the ThreadAbortException is the http request timing out.

We could process instructions in the background. At the time of creating this flexible Load Balancing functionality it didn't seem make sense. It means that your site could be replying to more requests than it currently does with stale data ... but I think that is not important anyways since there's a 5 second threshold (which is configurable) in which a server node will still be sending out stale data. In reality a server node cannot become instantly in sync with the master, there will always be some sort of delay so I think it would be worth visiting processing all instructions in the background on a timer instead of in the request to avoid this issue.


Shannon Deminick 05 Jul 2017, 04:30:16

Also remember you can delete the App_Data/TEMP/DistCache folder which will cause a cold boot and the server you do that on will rebuild it's own caches from scratch which will get it back in sync without reading instructions - thus 'fixing' this issue


Jeroen Breuer 05 Jul 2017, 08:19:38

After clearing the umbracoCacheInstruction table and doing a full cycle of app on both frontend servers the error was gone. Now all servers are up to date again.


Shannon Deminick 05 Jul 2017, 08:26:23

You shouldn't need to clear the table, doing a cold boot (i.e. deleting the /app_data/temp/distcache folder) will accomplish the same thing. I'd still be interested to see what is in the outstanding instruction rows that cause this.


piero ciula 22 Sep 2017, 13:23:35

Hi Shannon , we have updated umbraco from 7.5.2 to v. 7.7.1 but we still have the problem: application freeze. The log include the message “DISTRIBUTED CACHE IS NOT UPDATED”. the workaround to bring the application back to work is to delete the "APP DATA" folder on the frontoffice instance.

The application architecture consisting in one frontoffice instance and one backoffice instance that communicating through the same DB , without a load balancing configuration. It’s an Azure Cloud based Application. Is the flexible load balancing the right way to resolve the problem ( https://our.umbraco.org/documentation/getting-started/setup/server-setup/load-balancing/flexible )? Thanks in advanced, Piero


Shannon Deminick 25 Sep 2017, 12:40:09

@Piero This means

The application architecture consisting in one frontoffice instance and one backoffice instance that communicating through the same DB

that you are already load balancing so i'm not sure what you mean.

Deleting the entire App_Data folder is also not a correct work around for the problem, see above http://issues.umbraco.org/issue/U4-7673#comment=67-39893

It would be far more helpful for any bug report to include full details of the issue and how to replicate it. Surely something is happening to start causing this issue (i.e. bulk importing of data)


piero ciula 25 Sep 2017, 14:03:51

Hi Shannon,

that you are already load balancing so i'm not sure what you mean. We have not load balancing /flexible load balancing configuration (umbraco.setting.config in attachment).


Shannon Deminick 25 Sep 2017, 23:40:51

@Piero You said you have 2x web applications "consisting in one frontoffice instance and one backoffice instance" -> this means you are load balancing. Anytime you have more than one Umbraco instance referencing a single database you are load balancing and you are doing it via Flexible Load Balancing. Therefore you must follow the flexible load balancing instructions.


Priority: Major

Type: Bug

State: Fixed

Assignee:

Difficulty: Normal

Category: Architecture

Backwards Compatible: True

Fix Submitted:

Affected versions: 7.5.13

Due in version: 7.5.5

Sprint: Sprint 45

Story Points:

Cycle: