U4-6503 - Adding items to Runtime Cache slows down application

Created by Lucas Wolkowski 06 Apr 2015, 14:36:33 Updated by Shannon Deminick 21 Jan 2016, 16:30:27

Duplicates: U4-7558

Background

We have big Umbraco installation that is caching lots of values for performance / loopup reasons. After performance testing and hitting lots of sites we have noticed big performance degradation when opening nodes for edit and publishing. (note: there might be lots of other cases that is affected by the same issue)

After further investigation cause of the issues was tracked down to ASP.NET cache and Umbraco enumerating entire cache while searching for the key in '''GetCacheItemsByKeySearch''' method.

Adding items to ASP.NET runtime cache slows down back-end editing. Front-end can be slowed down as well if any methods that are using '''GetCacheItemsByKeySearch''' are called (e.g. if you are getting DataType PreValues)

Problem

Umbraco shouldn't be enumarating all cache keys in '''GetCacheItemsByKeySearch'''. This leads to big performance degradation if implementation is using ASP.NET runtime cache.

Repro steps:

  1. Install Umbraco via NuGet

  2. Host in IIS

  3. Open backend, open Home>Explore node. (see screens: ''CleanInstall-EditingNode.png'' & ''CleanInstall-EditingNode-Perf.png'')

  4. Add SurfaceController that will add items to the cache public class CacheTestController : SurfaceController

    { public ActionResult AddUsingUmbAppCache() { for (int i = 0; i < 3000000; i++) { ApplicationContext.ApplicationCache.RuntimeCache.GetCacheItem("CustomKey-" + i, () => true); }

         return Content("OK - AddUsingUmbAppCache");
     }
    
     public ActionResult AddUsingHttpContex()
     {
         for (int i = 0; i < 3000000; i++)
         {
             HttpContext.Cache.Insert("CustomKey-" + i, true);
         }
    
         return Content("OK - AddUsingHttpContex");
     }
    

    }

  5. Open URL that will add items using HttpContext: http://localhost/umbraco/surface/CacheTest/AddUsingHttpContex

  6. Open Home>Explore -> this is opening now much longer than in point 3. (see screens: ''CleanInstall-EditingNode-HttpContextInsert.png'' & ''CleanInstall-EditingNode-HttpContextInsert-Perf.png'')

  7. Restart IIS.

  8. Open site to start IIS.

  9. Open URL that will add items using Umbraco CacheHelper: http://localhost/umbraco/surface/CacheTest/AddUsingUmbAppCache

  10. Open Home>Explore -> this is opening now much longer than in point 3. (see screens: ''CleanInstall-EditingNode-AppContextGet.png'' & ''CleanInstall-EditingNode-AppContextGet-Perf.png'')

Using UmbracoApplicationCache is even slower because all items inserted into the cache will be prefixed with ''"umbrtmche-"'' and all the items will be evaluated twice (since 1st time all will be returned by Umbraco.Core.Cache.HttpRuntimeCacheProvider GetDictionaryEntries.

See Umbraco.Core.Cache.HttpRuntimeCacheProvider :: GetDictionaryEntries, Umbraco.Core.Cache.CacheProviderExtensions :: GetCacheItemsByKeySearch and Umbraco.Core.Cache.DictionaryCacheProviderBase :: GetCacheItemsByKeySearch

Please see screenshots of page timings and performance test screens showing affected methods, call stack, timings and number of calls. ''Please check affected versions, I've only set affected versions I'm aware of!''

Update: Publishing is even more affected (see screen: ''CleanInstall-Publishing-Perf.png'' ). Looks like all usages of GetCacheItemsByKeySearch, GetCacheItemsByKeyExpression, ClearCacheByKeySearch, ClearCacheByKeyExpression and ClearCacheObjectTypes are affected by implementations adding items to ASP.NET runtime cache.

''Note'': My example of 3millions bool values is the cache is just to show the issue of application performance being affected by cache key search.

Update 2: One of the possible solution would be for umbraco to use System.Runtime.Caching.MemoryCache if the intention is to store Umbraco values in memory (on each server). That way developers who are writing into HttpContext.Current.Cache are not affected by Key-Search on Cache dictionary.

After quick search via source code I found Umbraco.Core.Cache.ObjectCacheRuntimeCacheProvider that implements MemoryCache and is used in CoreBootManager! Unfortunately this is overriden and replaced with HttpRuntimeCacheProvider in WebBootManager. Perhaps WebBootManager class should initialise cache this way: var cacheHelper = new CacheHelper( new ObjectCacheRuntimeCacheProvider(), new StaticCacheProvider(), new HttpRequestCacheProvider()); or are there some disadvantages or this approach and some reason to use HttpRuntimeCache for Umbraco cache?

''Note'': these classes are internal so custom BootManager with this implementation cannot be created without some hacks + some obsolete ''(but used by Umbraco source)'' methods of CacheHelper are checking whether cache object is of HttpRuntimeCacheProvider type;

''Note'':This doesn't really resolves underlying issue - on big installations if there is lots of items in Cache in which Umbraco searches by enumerating all the keys then system slows down.

7 Attachments

Comments

Martin Griffiths 06 Nov 2015, 11:55:49

adding to this...

Should the Cache & Distributed Cache (Umbraco.Core.CacheHelper) via HttpRuntime.Cache be updated to the newer .net 4.5 System.Runtime.Caching API?


Shannon Deminick 21 Jan 2016, 16:30:12

This is solved in 7.3.5: http://issues.umbraco.org/issue/U4-7558


Priority: Critical

Type: Bug

State: Duplicate

Assignee:

Difficulty: Normal

Category: Architecture

Backwards Compatible: True

Fix Submitted:

Affected versions: 7.2.0, 7.1.8, 7.1.9, 7.2.1, 7.2.2, 7.2.3, 7.2.4

Due in version:

Sprint:

Story Points:

Cycle: