U4-10150 - Process load balancing instructions on a background thread instead of a request thread

Created by Shannon Deminick 13 Jul 2017, 03:10:39 Updated by Shannon Deminick 05 Apr 2018, 05:06:53

Relates to: U4-7673

Relates to: U4-9120

Subtask of: U4-9432

Currently we process load balancing instructions on a request thread but we could probably process these on a background thread instead and on a timer so it doesn't interfere with any front-end requests and is not susceptible to request timeouts.

There's a lot of information here: http://issues.umbraco.org/issue/U4-7673

This could be an option in the DatabaseServerMessengerOptions so that (for whatever reason) if we need to maintain compatibility we can have an option to continue to process them in the request thread.

NOTE: This is listed as a breaking change because if you are using an UmbracoContext.Current reference in a cache refresher event, it will now be null because cache refreshing is not happening in a request thread.


Shannon Deminick 31 Oct 2017, 04:21:04

PR https://github.com/umbraco/Umbraco-CMS/pull/2274

What is done:

Load balancing instructions used to be read on a request thread which causes all sorts of problems when there are too many instructions to process, so the BatchedDatabaseServerMessenger no longer binds to UmbracoModule.RouteAttempt in order to process instructions, instead it binds to a new event Scheduler.Initializing which is raised when the other scheduled tasks are initialized which is once the first real request for the website is made. In this event we check if the background task should be created and if so it creates the task to run every 5 seconds (or or whatever is defined in the DatabaseServerMessengerOptions.ThrottleSeconds). This means that even if there are tons of instructions to process we won't get the dreaded "DISTRIBUTED CACHE NOT UPDATED" problem (http://issues.umbraco.org/issue/U4-7673) which is due to the request timing out. The other benefit is that it doesn't take an active request to refresh the server's cache, instead it's just done on a schedule.

For testing:

  • probably easiest to add breakpoints in code when it's run to ensure that the new InstructionProcessing.PerformRun is being executed on the schedule
  • have a look at the logs and you'll see that InstructionProcessing task is being added, started and shutdown properly
  • set up a load balanced instance - basically create another site with the same version that you are running by running build.bat and using VSCode on the \build.tmp\WebApp folder to run the site, point the site to the same DB as the one referenced in VS = you are now load balancing. Now create, update, delete, content in the 'master' site - you can choose which one will be master for this test but i would suggest testing the site launched from VS as the Slave server so you can breakpoint in to InstructionProcessing.PerformRun to see that it's being executed and that it is processing instructions created from the VSCode (master) site

Robert Copilau 03 Nov 2017, 10:42:24

Test results:

*Logs look good, tasks added, syncing from db and completes every 5 seconds *In a load balanced scenario everything works, I can C.R.U.D content and the slave site gets updated


Jeavon Leopold 04 Apr 2018, 10:05:58

FYI, this issue should be listed as a breaking change as UmbracoContext within a cache event is now null whereas previously it was available

Shannon Deminick 05 Apr 2018, 05:06:02

OK, ill update.

Priority: Normal

Type: Feature (request)

State: Fixed


Difficulty: Normal


Backwards Compatible: False

Fix Submitted:

Affected versions:

Due in version: 7.8.0

Sprint: Sprint 71

Story Points: 3

Cycle: 5