U4-7865 - CPU Leak- 100% Usage

Created by DotNetAshish 30 Jan 2016, 19:52:17 Updated by Sebastiaan Janssen 10 May 2016, 19:01:12

Relates to: U4-8429

Relates to: U4-2037

Hi, I am experiencing 100% CPU usage in my Umbraco site. During initial investigation I found below hint from the Windbg tool which is pointing to AsynchronousRollingFileAppender.cs

OS Thread Id: 0x34b8 (27) Child SP IP Call Site 000000dfe466e3b8 00007ffc5ea4111a [HelperMethodFrame: 000000dfe466e3b8] System.Threading.Thread.SleepInternal(Int32) 000000dfe466e4b0 00007ffc53b9f399 System.Threading.Thread.Sleep(Int32) 000000dfe466e4e0 00007ffbf5ceee3a *** WARNING: Unable to verify checksum for Umbraco.Core.dll Umbraco.Core.Logging.AsynchronousRollingFileAppender.AppendLoggingEvents() [d:\Dev\Umbraco7\src\Umbraco.Core\Logging\AsynchronousRollingFileAppender.cs @ 143] 000000dfe466e540 00007ffc53bb30ee System.Threading.Tasks.Task.Execute() 000000dfe466e5b0 00007ffc53b18355 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) 000000dfe466e710 00007ffc53b180c9 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) 000000dfe466e740 00007ffc53bb33c5 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef) 000000dfe466e820 00007ffc53bb2a65 System.Threading.Tasks.Task.ExecuteEntry(Boolean) 000000dfe466e860 00007ffc53b18355 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) 000000dfe466e9c0 00007ffc53b180c9 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) 000000dfe466e9f0 00007ffc53b180a7 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) 000000dfe466ea40 00007ffc543e673d System.Threading.ThreadHelper.ThreadStart(System.Object) 000000dfe466ed48 00007ffc54d40453 [GCFrame: 000000dfe466ed48] 000000dfe466f098 00007ffc54d40453 [DebuggerU2MCatchHandlerFrame: 000000dfe466f098] 000000dfe466f238 00007ffc54d40453 [ContextTransitionFrame: 000000dfe466f238] 000000dfe466f458 00007ffc54d40453 [DebuggerU2MCatchHandlerFrame: 000000dfe466f458] .

Can you please confirm if there is no issue with the logging and this is not the real cause of CPU peak ?

Comments

Nicholas Westby 30 Jan 2016, 22:13:30

Which version of Umbraco are you using? There have been some recent versions that did have CPU issues, but those issues have been fixed as far as I'm aware.

Also, here's a related issue that was already reported and fixed: http://issues.umbraco.org/issue/U4-2037


DotNetAshish 31 Jan 2016, 07:11:25

@Knickerbocker , Thanks for prompt response . I am using Umbraco version 7.1.8 and another website with version 7.3.0


Shannon Deminick 31 Jan 2016, 11:48:34

This is a false positive, the reason it is saying that the logger is the culprit is because this particular logger uses it's own standalone thread in a loop. Version 7.3+ uses a different logger (see the log4net config that 7.3.6 is released with). It basically performs the same function on an async background thread instead of a standalone thread so you probably won't get a false positive with that.

I'd suggest changing to the new logger, or simply turn off logging if you want and then re-run your profiler so you can determine what the actual cause is.


DotNetAshish 31 Jan 2016, 19:09:28

Hi @Shandem : Thanks for suggestion. Would it be possible to use this newly released logger with the old Umbraco versions i.e. 7.1.8 or 7.3.0 as due to extensive customize code it won't be possible for us to upgrade to 7.3.6, Please advise ?


Shannon Deminick 01 Feb 2016, 07:17:29

It isn't going to make any difference, as I said this is a false positive and is not your issue.

If you want to determine your real issue using the profiler you are using just disable logging and run your profiler again.

You could use the new logger if you built it from source, otherwise no, since its built into the core libraries.

Also, you should upgrade to the latest Umbraco versions.... We've fixed hundreds of bugs since the release you are using.


Sebastiaan Janssen 01 Feb 2016, 07:40:35

Just wanted to chip in: the only way to know for sure why your CPU usage is 100% is by creating a memory dump and analyzing what's going on there. :)

I'll close this issue, not because I don't believe you but because it's not a common issue that other people have. If you can analyze a memory dump on the latest version of Umbraco (currently 7.3.6) and point to something that Umbraco is doing wrong then make sure to re-open.


DotNetAshish 03 Feb 2016, 05:07:49

Hi @sebastiaan , The above stack trace is taken from memory dump using Windbg which is pointing to AsynchronousRollingFileAppender.


Shannon Deminick 03 Feb 2016, 07:55:16

Hi, please see my previous notes. The logger runs on its own thread in an infinite loop, so yes it will show up as being active always, but it is not causing hightened cpu cycles, otherwise we would see massive issues tens of thousands of websites.

If you want to determine your real issue using the profiler you are using just disable logging and run your profiler again.

In the meantime I'm happy to look at your mem dump to see what else is in there if you attach it. Also be sure your testing with the latest Umbraco version as Sebastiaan mentioned.


DotNetAshish 03 Feb 2016, 08:36:42

@Shandem Can you please share your email to share dump ?


Shannon Deminick 03 Feb 2016, 08:51:48

Yes, it is shannon@umbraco.com, or you can attach to this issue with Visible to HQ only.


Shannon Deminick 03 Feb 2016, 08:55:22

But please, re-run your tests/profilers/etc... with a different logger for log4net so you don't see this false positive in your reports. If you want to use the 7.3 logger in an older version, you're more than welcome to copy the code into your local project:


DotNetAshish 03 Feb 2016, 17:05:45

Hi @Shandem Shannon : Due to heavy size I have shared you dump through google drive. Please acknowledge if you got it.


Shannon Deminick 04 Feb 2016, 10:05:09

Hi, i have received it, we are currently quite busy with a release schedule so i will have a look when i have time. In the meantime, I urge you to test your solution without this logger enabled so you can identity what you real issue is.


DotNetAshish 08 Feb 2016, 19:54:48

got 100% CPU and few suspected debugging below :sent dumps as well

OS Thread Id: 0x37f0 (28) Child SP IP Call Site 00000088eecee988 00007ff84c57111a [HelperMethodFrame: 00000088eecee988] System.Threading.Thread.SleepInternal(Int32) 00000088eeceea80 00007ff84193f399 System.Threading.Thread.Sleep(Int32) 00000088eeceeab0 00007ff7e3abf64a SYMSRV: C:\ProgramData\dbg\sym\Umbraco.Core.pdb\FD757B64E1D5473DA1DCBC06F3E9A26A1\Umbraco.Core.pdb not found

SYMSRV: http://msdl.microsoft.com/download/symbols/Umbraco.Core.pdb/FD757B64E1D5473DA1DCBC06F3E9A26A1/Umbraco.Core.pdb not found SYMSRV: C:\ProgramData\dbg\sym\Umbraco.Core.pdb\FD757B64E1D5473DA1DCBC06F3E9A26A1\Umbraco.Core.pdb not found DBGHELP: e:\eos_feb08\binaries\Umbraco.Core.pdb - file not found DBGHELP: e:\eos_feb08\binaries\dll\Umbraco.Core.pdb - file not found DBGHELP: e:\eos_feb08\binaries\symbols\dll\Umbraco.Core.pdb - file not found DBGHELP: c:\users\ashish\documents\eos_umbraco\umbracocms.allbinaries.7.1.8\Umbraco.Core.pdb cached to C:\ProgramData\dbg\sym\Umbraco.Core.pdb\FD757B64E1D5473DA1DCBC06F3E9A26A1\Umbraco.Core.pdb DBGHELP: c:\users\ashish\documents\eos_umbraco\umbracocms.allbinaries.7.1.8\Umbraco.Core.pdb cached to C:\ProgramData\dbg\sym\Umbraco.Core.pdb\FD757B64E1D5473DA1DCBC06F3E9A26A1\Umbraco.Core.pdb *** WARNING: Unable to verify checksum for Umbraco.Core.dll DBGHELP: Umbraco_Core - private symbols & lines C:\ProgramData\dbg\sym\Umbraco.Core.pdb\FD757B64E1D5473DA1DCBC06F3E9A26A1\Umbraco.Core.pdb Umbraco.Core.Logging.AsynchronousRollingFileAppender.AppendLoggingEvents() [d:\Dev\Umbraco7\src\Umbraco.Core\Logging\AsynchronousRollingFileAppender.cs @ 143] 00000088eeceeb10 00007ff8419530ee System.Threading.Tasks.Task.Execute() 00000088eeceeb80 00007ff8418b8355 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) 00000088eeceece0 00007ff8418b80c9 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) 00000088eeceed10 00007ff8419533c5 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef) 00000088eeceedf0 00007ff841952a65 System.Threading.Tasks.Task.ExecuteEntry(Boolean) 00000088eeceee30 00007ff8418b8355 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) 00000088eeceef90 00007ff8418b80c9 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) 00000088eeceefc0 00007ff8418b80a7 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) 00000088eecef010 00007ff84218673d System.Threading.ThreadHelper.ThreadStart(System.Object) 00000088eecef318 00007ff842af0453 [GCFrame: 00000088eecef318] 00000088eecef668 00007ff842af0453 [DebuggerU2MCatchHandlerFrame: 00000088eecef668] 00000088eecef808 00007ff842af0453 [ContextTransitionFrame: 00000088eecef808] 00000088eecefa28 00007ff842af0453 [DebuggerU2MCatchHandlerFrame: 00000088eecefa28]

OS Thread Id: 0x15b64 (58) Child SP IP Call Site 00000089094cba50 00007ff84196022b System.Collections.Generic.Dictionary2[[System.__Canon, mscorlib],[System.Boolean, mscorlib]].Insert(System.__Canon, Boolean, Boolean) 00000089094cbae0 00007ff7e56d6b1e Umbraco.Core.Models.EntityBase.TracksChangesEntityBase.OnPropertyChanged(System.Reflection.PropertyInfo) [d:\Dev\Umbraco7\src\Umbraco.Core\Models\EntityBase\TracksChangesEntityBase.cs @ 40] 00000089094cbb30 00007ff7e56da10c Umbraco.Core.Models.EntityBase.TracksChangesEntityBase.SetPropertyValueAndDetectChanges[[System.__Canon, mscorlib]](System.Func2<System.__Canon,System.__Canon>, System.__Canon, System.Reflection.PropertyInfo) [d:\Dev\Umbraco7\src\Umbraco.Core\Models\EntityBase\TracksChangesEntityBase.cs @ 151] 00000089094cbb70 00007ff7e56ea1fd Umbraco.Core.Models.Membership.UserType.set_Permissions(System.Collections.Generic.IEnumerable1<System.String>) [d:\Dev\Umbraco7\src\Umbraco.Core\Models\Membership\UserType.cs @ 66] 00000089094cbe08 00007ff842af0453 [DebuggerU2MCatchHandlerFrame: 00000089094cbe08] 00000089094cc108 00007ff842af0453 [HelperMethodFrame_PROTECTOBJ: 00000089094cc108] System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean) 00000089094cc280 00007ff84188c42c System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[]) 00000089094cc2f0 00007ff841940603 System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo) 00000089094cc370 00007ff841961adc System.Reflection.RuntimePropertyInfo.SetValue(System.Object, System.Object, System.Object[]) 00000089094cc3c0 00007ff7e56ebe0d Umbraco.Core.Models.DeepCloneHelper.DeepCloneRefProperties(Umbraco.Core.Models.IDeepCloneable, Umbraco.Core.Models.IDeepCloneable) [d:\Dev\Umbraco7\src\Umbraco.Core\Models\DeepCloneHelper.cs @ 166] 00000089094cc470 00007ff7e56eafdb Umbraco.Core.Models.EntityBase.Entity.DeepClone() [d:\Dev\Umbraco7\src\Umbraco.Core\Models\EntityBase\Entity.cs @ 241] 00000089094cc4b0 00007ff7e56eb946 Umbraco.Core.Models.DeepCloneHelper.DeepCloneRefProperties(Umbraco.Core.Models.IDeepCloneable, Umbraco.Core.Models.IDeepCloneable) [d:\Dev\Umbraco7\src\Umbraco.Core\Models\DeepCloneHelper.cs @ 75] 00000089094cc560 00007ff7e579119f Umbraco.Core.Models.Membership.User.DeepClone() [d:\Dev\Umbraco7\src\Umbraco.Core\Models\Membership\User.cs @ 450] 00000089094cc5b0 00007ff7e56ccf8a Umbraco.Core.Persistence.Caching.RuntimeCacheProvider.GetById(System.Type, System.Guid) [d:\Dev\Umbraco7\src\Umbraco.Core\Persistence\Caching\RuntimeCacheProvider.cs @ 75] 00000089094cc610 00007ff7e56cc89b Umbraco.Core.Persistence.Repositories.RepositoryBase2[[System.Int32, mscorlib],[System.__Canon, mscorlib]].TryGetFromCache(Int32) [d:\Dev\Umbraco7\src\Umbraco.Core\Persistence\Repositories\RepositoryBase.cs @ 120] 00000089094cc6b0 00007ff7e56cc70c Umbraco.Core.Persistence.Repositories.RepositoryBase2[[System.Int32, mscorlib],[System.__Canon, mscorlib]].Get(Int32) [d:\Dev\Umbraco7\src\Umbraco.Core\Persistence\Repositories\RepositoryBase.cs @ 91] 00000089094cc730 00007ff7e61472fc Umbraco.Core.Persistence.Repositories.UserRepository+<PerformGetByQuery>d__9.MoveNext() [d:\Dev\Umbraco7\src\Umbraco.Core\Persistence\Repositories\UserRepository.cs @ 93] 00000089094cc7b0 00007ff83e701004 System.Linq.Enumerable+WhereEnumerableIterator1[[System.__Canon, mscorlib]].MoveNext() 00000089094cc800 00007ff83e6d53a7 System.Linq.Enumerable.FirstOrDefault[System.__Canon, mscorlib] 00000089094cc860 00007ff7e6145075 Umbraco.Core.Services.UserService.GetByUsername(System.String) [d:\Dev\Umbraco7\src\Umbraco.Core\Services\UserService.cs @ 220] 00000089094cc900 00007ff7e6c7e606 Umbraco.Web.Security.Providers.UmbracoMembershipProvider2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].ValidateUser(System.String, System.String) 00000089094cc970 00007ff7e6c7e241 Umbraco.Web.Editors.AuthenticationController.PostLogin(Umbraco.Web.Models.LoginModel) 00000089094cc9f0 00007ff7e6d2201e DynamicClass.lambda_method(System.Runtime.CompilerServices.Closure, System.Object, System.Object[]) 00000089094cca30 00007ff7e58a1715 System.Web.Http.Controllers.ReflectedHttpActionDescriptor+ActionExecutor+<>c__DisplayClass13.<GetExecutor>b__c(System.Object, System.Object[]) 00000089094cca70 00007ff7e58a168f System.Web.Http.Controllers.ReflectedHttpActionDescriptor+ActionExecutor.Execute(System.Object, System.Object[]) 00000089094ccaa0 00007ff7e57cbcc8 System.Threading.Tasks.TaskHelpers.RunSynchronously[[System.__Canon, mscorlib]](System.Func1<System.Threading.Tasks.Task1<System.__Canon>>, System.Threading.CancellationToken) 00000089094ccaf0 00007ff7e57cc089 System.Web.Http.Controllers.ApiControllerActionInvoker+<>c__DisplayClass3.<InvokeActionAsync>b__0() 00000089094ccb40 00007ff7e57cbcc8 System.Threading.Tasks.TaskHelpers.RunSynchronously[[System.__Canon, mscorlib]](System.Func1<System.Threading.Tasks.Task1<System.__Canon>>, System.Threading.CancellationToken) 00000089094ccb90 00007ff7e57c9558 System.Web.Http.Controllers.ApiControllerActionInvoker.InvokeActionAsync(System.Web.Http.Controllers.HttpActionContext, System.Threading.CancellationToken) 00000089094ccbe0 00007ff7e57c9423 System.Web.Http.ApiController+<>c__DisplayClass3.<ExecuteAsync>b__2() 00000089094ccc10 00007ff7e57a60e1 System.Web.Http.Filters.ActionFilterAttribute.System.Web.Http.Filters.IActionFilter.ExecuteActionFilterAsync(System.Web.Http.Controllers.HttpActionContext, System.Threading.CancellationToken, System.Func1<System.Threading.Tasks.Task1<System.Net.Http.HttpResponseMessage>>) 00000089094ccca0 00007ff7e57a5b6b System.Web.Http.ApiController+<>c__DisplayClass14+<>c__DisplayClass16.<InvokeActionWithActionFilters>b__13() 00000089094cccd0 00007ff7e57a60e1 System.Web.Http.Filters.ActionFilterAttribute.System.Web.Http.Filters.IActionFilter.ExecuteActionFilterAsync(System.Web.Http.Controllers.HttpActionContext, System.Threading.CancellationToken, System.Func1<System.Threading.Tasks.Task1<System.Net.Http.HttpResponseMessage>>) 00000089094ccd60 00007ff7e57a5b6b System.Web.Http.ApiController+<>c__DisplayClass14+<>c__DisplayClass16.<InvokeActionWithActionFilters>b__13() 00000089094ccd90 00007ff7e57a5785 System.Web.Http.ApiController+<>c__DisplayClass3.<ExecuteAsync>b__1() 00000089094ccde0 00007ff7e57a569f System.Threading.Tasks.TaskHelpersExtensions+<>c__DisplayClass321[[System.__Canon, mscorlib]].b__31(System.Threading.Tasks.Task) 00000089094cce10 00007ff7e57a55b4 System.Threading.Tasks.TaskHelpersExtensions.ThenImpl[[System.__Canon, mscorlib],[System.__Canon, mscorlib]](System.__Canon, System.Func2<System.__Canon,System.Threading.Tasks.Task1<System.__Canon>>, System.Threading.CancellationToken, Boolean) 00000089094cce80 00007ff7e57a42f4 System.Web.Http.ApiController+<>c__DisplayClass3.b__0() 00000089094ccee0 00007ff7e57923af System.Web.Http.ApiController.ExecuteAsync(System.Web.Http.Controllers.HttpControllerContext, System.Threading.CancellationToken) 00000089094ccf50 00007ff7e56ded56 System.Web.Http.Dispatcher.HttpControllerDispatcher.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken) 00000089094ccfe0 00007ff83727d4af System.Net.Http.HttpMessageInvoker.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken) 00000089094cd030 00007ff7e56de3cd System.Web.Http.Dispatcher.HttpRoutingDispatcher.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken) 00000089094cd090 00007ff7e56dd23f System.Web.Http.HttpServer.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken) 00000089094cd0e0 00007ff83727d4af System.Net.Http.HttpMessageInvoker.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken) 00000089094cd130 00007ff7e56da2cf System.Web.Http.WebHost.HttpControllerHandler.BeginProcessRequest(System.Web.HttpContextBase, System.AsyncCallback, System.Object) 00000089094cd1b0 00007ff83d536072 System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() 00000089094cd290 00007ff83d4fd510 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef) 00000089094cd330 00007ff83d51ab5a System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception) 00000089094cd480 00007ff83d4fd6a3 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback) 00000089094cd4d0 00007ff83d4f75de System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext) 00000089094cd570 00007ff83d500561 System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32) 00000089094cd780 00007ff83d4fff92 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32) 00000089094cd7d0 00007ff83dc55a41 DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32) 00000089094cdfe8 00007ff842b4736e [InlinedCallFrame: 00000089094cdfe8] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatus ByRef) 00000089094cdfe8 00007ff83d5a956b [InlinedCallFrame: 00000089094cdfe8] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatus ByRef) 00000089094cdfc0 00007ff83d5a956b DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef) 00000089094ce090 00007ff83d50074f System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32) 00000089094ce2a0 00007ff83d4fff92 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32) 00000089094ce2f0 00007ff83dc55a41 DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32) 00000089094ce4f8 00007ff842b475c3 [ContextTransitionFrame: 00000089094ce4f8] OS Thread Id: 0x1fe34 (59) Child SP IP Call Site 000000890911ba20 00007ff841960214 System.Collections.Generic.Dictionary2[[System.__Canon, mscorlib],[System.Boolean, mscorlib]].Insert(System.__Canon, Boolean, Boolean) 000000890911bab0 00007ff7e56d6b1e Umbraco.Core.Models.EntityBase.TracksChangesEntityBase.OnPropertyChanged(System.Reflection.PropertyInfo) [d:\Dev\Umbraco7\src\Umbraco.Core\Models\EntityBase\TracksChangesEntityBase.cs @ 40] 000000890911bb00 00007ff7e56da10c Umbraco.Core.Models.EntityBase.TracksChangesEntityBase.SetPropertyValueAndDetectChanges[[System.__Canon, mscorlib]](System.Func2<System.__Canon,System.__Canon>, System.__Canon, System.Reflection.PropertyInfo) [d:\Dev\Umbraco7\src\Umbraco.Core\Models\EntityBase\TracksChangesEntityBase.cs @ 151] 000000890911bb40 00007ff7e56ea1fd Umbraco.Core.Models.Membership.UserType.set_Permissions(System.Collections.Generic.IEnumerable1<System.String>) [d:\Dev\Umbraco7\src\Umbraco.Core\Models\Membership\UserType.cs @ 66] 000000890911bdd8 00007ff842af0453 [DebuggerU2MCatchHandlerFrame: 000000890911bdd8] 000000890911c0d8 00007ff842af0453 [HelperMethodFrame_PROTECTOBJ: 000000890911c0d8] System.RuntimeMethodHandle.InvokeMethod(System.Object, System.Object[], System.Signature, Boolean) 000000890911c250 00007ff84188c42c System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(System.Object, System.Object[], System.Object[]) 000000890911c2c0 00007ff841940603 System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo) 000000890911c340 00007ff841961adc System.Reflection.RuntimePropertyInfo.SetValue(System.Object, System.Object, System.Object[]) 000000890911c390 00007ff7e56ebe0d Umbraco.Core.Models.DeepCloneHelper.DeepCloneRefProperties(Umbraco.Core.Models.IDeepCloneable, Umbraco.Core.Models.IDeepCloneable) [d:\Dev\Umbraco7\src\Umbraco.Core\Models\DeepCloneHelper.cs @ 166] 000000890911c440 00007ff7e56eafdb Umbraco.Core.Models.EntityBase.Entity.DeepClone() [d:\Dev\Umbraco7\src\Umbraco.Core\Models\EntityBase\Entity.cs @ 241] 000000890911c480 00007ff7e56eb946 Umbraco.Core.Models.DeepCloneHelper.DeepCloneRefProperties(Umbraco.Core.Models.IDeepCloneable, Umbraco.Core.Models.IDeepCloneable) [d:\Dev\Umbraco7\src\Umbraco.Core\Models\DeepCloneHelper.cs @ 75] 000000890911c530 00007ff7e579119f Umbraco.Core.Models.Membership.User.DeepClone() [d:\Dev\Umbraco7\src\Umbraco.Core\Models\Membership\User.cs @ 450] 000000890911c580 00007ff7e56ccf8a Umbraco.Core.Persistence.Caching.RuntimeCacheProvider.GetById(System.Type, System.Guid) [d:\Dev\Umbraco7\src\Umbraco.Core\Persistence\Caching\RuntimeCacheProvider.cs @ 75] 000000890911c5e0 00007ff7e56cc89b Umbraco.Core.Persistence.Repositories.RepositoryBase2[[System.Int32, mscorlib],[System.__Canon, mscorlib]].TryGetFromCache(Int32) [d:\Dev\Umbraco7\src\Umbraco.Core\Persistence\Repositories\RepositoryBase.cs @ 120] 000000890911c680 00007ff7e56cc70c Umbraco.Core.Persistence.Repositories.RepositoryBase2[[System.Int32, mscorlib],[System.__Canon, mscorlib]].Get(Int32) [d:\Dev\Umbraco7\src\Umbraco.Core\Persistence\Repositories\RepositoryBase.cs @ 91] 000000890911c700 00007ff7e61472fc Umbraco.Core.Persistence.Repositories.UserRepository+<PerformGetByQuery>d__9.MoveNext() [d:\Dev\Umbraco7\src\Umbraco.Core\Persistence\Repositories\UserRepository.cs @ 93] 000000890911c780 00007ff83e701004 System.Linq.Enumerable+WhereEnumerableIterator1[[System.__Canon, mscorlib]].MoveNext() 000000890911c7d0 00007ff83e6d53a7 System.Linq.Enumerable.FirstOrDefault[System.__Canon, mscorlib] 000000890911c830 00007ff7e6145075 Umbraco.Core.Services.UserService.GetByUsername(System.String) [d:\Dev\Umbraco7\src\Umbraco.Core\Services\UserService.cs @ 220] 000000890911c8d0 00007ff7e6c7e606 Umbraco.Web.Security.Providers.UmbracoMembershipProvider2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].ValidateUser(System.String, System.String) 000000890911c940 00007ff7e6c7e241 Umbraco.Web.Editors.AuthenticationController.PostLogin(Umbraco.Web.Models.LoginModel) 000000890911c9c0 00007ff7e6d2201e DynamicClass.lambda_method(System.Runtime.CompilerServices.Closure, System.Object, System.Object[]) 000000890911ca00 00007ff7e58a1715 System.Web.Http.Controllers.ReflectedHttpActionDescriptor+ActionExecutor+<>c__DisplayClass13.<GetExecutor>b__c(System.Object, System.Object[]) 000000890911ca40 00007ff7e58a168f System.Web.Http.Controllers.ReflectedHttpActionDescriptor+ActionExecutor.Execute(System.Object, System.Object[]) 000000890911ca70 00007ff7e57cbcc8 System.Threading.Tasks.TaskHelpers.RunSynchronously[[System.__Canon, mscorlib]](System.Func1<System.Threading.Tasks.Task1<System.__Canon>>, System.Threading.CancellationToken) 000000890911cac0 00007ff7e57cc089 System.Web.Http.Controllers.ApiControllerActionInvoker+<>c__DisplayClass3.<InvokeActionAsync>b__0() 000000890911cb10 00007ff7e57cbcc8 System.Threading.Tasks.TaskHelpers.RunSynchronously[[System.__Canon, mscorlib]](System.Func1<System.Threading.Tasks.Task1<System.__Canon>>, System.Threading.CancellationToken) 000000890911cb60 00007ff7e57c9558 System.Web.Http.Controllers.ApiControllerActionInvoker.InvokeActionAsync(System.Web.Http.Controllers.HttpActionContext, System.Threading.CancellationToken) 000000890911cbb0 00007ff7e57c9423 System.Web.Http.ApiController+<>c__DisplayClass3.<ExecuteAsync>b__2() 000000890911cbe0 00007ff7e57a60e1 System.Web.Http.Filters.ActionFilterAttribute.System.Web.Http.Filters.IActionFilter.ExecuteActionFilterAsync(System.Web.Http.Controllers.HttpActionContext, System.Threading.CancellationToken, System.Func1<System.Threading.Tasks.Task1<System.Net.Http.HttpResponseMessage>>) 000000890911cc70 00007ff7e57a5b6b System.Web.Http.ApiController+<>c__DisplayClass14+<>c__DisplayClass16.<InvokeActionWithActionFilters>b__13() 000000890911cca0 00007ff7e57a60e1 System.Web.Http.Filters.ActionFilterAttribute.System.Web.Http.Filters.IActionFilter.ExecuteActionFilterAsync(System.Web.Http.Controllers.HttpActionContext, System.Threading.CancellationToken, System.Func1<System.Threading.Tasks.Task1<System.Net.Http.HttpResponseMessage>>) 000000890911cd30 00007ff7e57a5b6b System.Web.Http.ApiController+<>c__DisplayClass14+<>c__DisplayClass16.<InvokeActionWithActionFilters>b__13() 000000890911cd60 00007ff7e57a5785 System.Web.Http.ApiController+<>c__DisplayClass3.<ExecuteAsync>b__1() 000000890911cdb0 00007ff7e57a569f System.Threading.Tasks.TaskHelpersExtensions+<>c__DisplayClass321[[System.__Canon, mscorlib]].b__31(System.Threading.Tasks.Task) 000000890911cde0 00007ff7e57a55b4 System.Threading.Tasks.TaskHelpersExtensions.ThenImpl[[System.__Canon, mscorlib],[System.__Canon, mscorlib]](System.__Canon, System.Func2<System.__Canon,System.Threading.Tasks.Task1<System.__Canon>>, System.Threading.CancellationToken, Boolean) 000000890911ce50 00007ff7e57a42f4 System.Web.Http.ApiController+<>c__DisplayClass3.b__0() 000000890911ceb0 00007ff7e57923af System.Web.Http.ApiController.ExecuteAsync(System.Web.Http.Controllers.HttpControllerContext, System.Threading.CancellationToken) 000000890911cf20 00007ff7e56ded56 System.Web.Http.Dispatcher.HttpControllerDispatcher.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken) 000000890911cfb0 00007ff83727d4af System.Net.Http.HttpMessageInvoker.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken) 000000890911d000 00007ff7e56de3cd System.Web.Http.Dispatcher.HttpRoutingDispatcher.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken) 000000890911d060 00007ff7e56dd23f System.Web.Http.HttpServer.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken) 000000890911d0b0 00007ff83727d4af System.Net.Http.HttpMessageInvoker.SendAsync(System.Net.Http.HttpRequestMessage, System.Threading.CancellationToken) 000000890911d100 00007ff7e56da2cf System.Web.Http.WebHost.HttpControllerHandler.BeginProcessRequest(System.Web.HttpContextBase, System.AsyncCallback, System.Object) 000000890911d180 00007ff83d536072 System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() 000000890911d260 00007ff83d4fd510 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef) 000000890911d300 00007ff83d51ab5a System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception) 000000890911d450 00007ff83d4fd6a3 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback) 000000890911d4a0 00007ff83d4f75de System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext) 000000890911d540 00007ff83d500561 System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32) 000000890911d750 00007ff83d4fff92 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32) 000000890911d7a0 00007ff83dc55a41 DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32) 000000890911dfb8 00007ff842b4736e [InlinedCallFrame: 000000890911dfb8] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatus ByRef) 000000890911dfb8 00007ff83d5a956b [InlinedCallFrame: 000000890911dfb8] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatus ByRef) 000000890911df90 00007ff83d5a956b DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef) 000000890911e060 00007ff83d50074f System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32) 000000890911e270 00007ff83d4fff92 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32) 000000890911e2c0 00007ff83dc55a41 DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32) 000000890911e4c8 00007ff842b475c3 [ContextTransitionFrame: 000000890911e4c8]

==========================================================================================================


DotNetAshish 08 Feb 2016, 20:49:22

@Shannon HI Shannon other than above debugging output I found few other, Please have a look Thread 47 Current frame: (MethodDesc 00007ff841433f58 +0x1c4 System.Collections.Generic.Dictionary2[[System.__Canon, mscorlib],[System.Boolean, mscorlib]].Insert(System.__Canon, Boolean, Boolean)) Thread 48 Current frame: (MethodDesc 00007ff841433f58 +0x1ec System.Collections.Generic.Dictionary2[[System.__Canon, mscorlib],[System.Boolean, mscorlib]].Insert(System.__Canon, Boolean, Boolean)) Thread 49 Current frame: (MethodDesc 00007ff841433f58 +0x1db System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.Boolean, mscorlib]].Insert(System.__Canon, Boolean, Boolean))

Thread 50 Current frame: (MethodDesc 00007ff841433f58 +0x1ec System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib],[System.Boolean, mscorlib]].Insert(System.__Canon, Boolean, Boolean))

=================================

Thread 28 Current frame: ntdll!NtDelayExecution+0xa Child-SP RetAddr Caller, Callee 00000088eecee7e0 00007ff84997124a KERNELBASE!SleepEx+0xa2, calling ntdll!NtDelayExecution 00000088eecee850 00007ff8499744b3 KERNELBASE!SleepEx+0x113, calling ntdll!RtlActivateActivationContextUnsafeFast 00000088eecee880 00007ff842af1c68 clr!CExecutionEngineClrSleepEx+0x29, calling kernel32!SleepEx 00000088eecee8b0 00007ff842c63597 clr!ThreadUserSleep+0x7c, calling clr!ClrSleepEx 00000088eecee910 00007ff842c63493 clr!ThreadNativeSleep+0xb7, calling clr!ThreadUserSleep 00000088eecee930 00007ff84188805a (MethodDesc 00007ff841505c70 +0x2a System.IO.Stream.Close()), calling 00007ff842b073d8 (stub for System.GC._SuppressFinalize(System.Object)) 00000088eecee970 00007ff7e3ac0d01 (MethodDesc 00007ff7e3aa9ed0 +0x41 log4net.Appender.FileAppender+LockingModelBase.CloseStream(System.IO.Stream)) 00000088eecee9b0 00007ff84193f399 (MethodDesc 00007ff841431c90 +0x9 System.Threading.Thread.Sleep(Int32)), calling 00007ff842c633dc (stub for System.Threading.Thread.SleepInternal(Int32)) 00000088eeceea48 00007ff842c6343b clr!ThreadNativeSleep+0x5f, calling clr!LazyMachStateCaptureState 00000088eeceea70 00007ff84193f399 (MethodDesc 00007ff841431c90 +0x9 System.Threading.Thread.Sleep(Int32)), calling 00007ff842c633dc (stub for System.Threading.Thread.SleepInternal(Int32)) 00000088eeceeaa0 00007ff7e3abf64a (MethodDesc 00007ff7e3aab410 +0x8a Umbraco.Core.Logging.AsynchronousRollingFileAppender.AppendLoggingEvents()), calling (MethodDesc 00007ff841431c90 +0 System.Threading.Thread.Sleep(Int32)) 00000088eeceeb00 00007ff8419530ee (MethodDesc 00007ff84150bf30 +0x6e System.Threading.Tasks.Task.Execute()) 00000088eeceeb70 00007ff8418b8355 (MethodDesc 00007ff84150faf0 +0x285 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)) 00000088eeceeb90 00007ff842b95f7b clr!ThreadLocalBlockAllocateStaticFieldObjRefPtrs+0x3b, calling clr!ThreadStaticHandleTableAllocateHandles 00000088eeceebd0 00007ff842c76d8a clr!EEClassGetPackableField+0x11b, calling clr!PackedDWORDFields<11>BitVectorGet 00000088eeceec10 00007ff842b95d82 clr!MethodTableGetGCThreadStaticsBasePointer+0x4e, calling clr!ThreadLocalModuleGetGCStaticsBasePointer 00000088eeceecd0 00007ff8418b80c9 (MethodDesc 00007ff84150fae0 +0x9 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)), calling (MethodDesc 00007ff84150faf0 +0 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)) 00000088eeceed00 00007ff8419533c5 (MethodDesc 00007ff84150bf50 +0x2c5 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)), calling (MethodDesc 00007ff84150fae0 +0 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)) 00000088eeceed30 00007ff842c75d4b clr!LookupMap<MethodTable * __ptr64>GetElement+0x67, calling clr!LookupMapBaseFindHotItemValuePtr 00000088eeceed60 00007ff842af312e clr!ModuleLookupTypeDef+0x53, calling clr!LookupMap<MethodTable * __ptr64>GetElement 00000088eeceeda0 00007ff842af0519 clr!JIT_GetSharedGCThreadStaticBase+0x99, calling clr!_security_check_cookie 00000088eeceede0 00007ff841952a65 (MethodDesc 00007ff84150b760 +0x85 System.Threading.Tasks.Task.ExecuteEntry(Boolean)), calling (MethodDesc 00007ff84150bf50 +0 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)) 00000088eeceee20 00007ff8418b8355 (MethodDesc 00007ff84150faf0 +0x285 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)) 00000088eeceef80 00007ff8418b80c9 (MethodDesc 00007ff84150fae0 +0x9 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)), calling (MethodDesc 00007ff84150faf0 +0 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)) 00000088eeceefb0 00007ff8418b80a7 (MethodDesc 00007ff84150fad0 +0x57 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)), calling (MethodDesc 00007ff84150fae0 +0 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)) 00000088eecef000 00007ff84218673d (MethodDesc 00007ff841616d20 +0x5d System.Threading.ThreadHelper.ThreadStart(System.Object)), calling (MethodDesc 00007ff84150fad0 +0 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)) 00000088eecef050 00007ff842af0453 clr!CallDescrWorkerInternal+0x83 00000088eecef090 00007ff842af031e clr!CallDescrWorkerWithHandler+0x4a, calling clr!CallDescrWorkerInternal 00000088eecef0a0 00007ff842b956d6 clr!MetaSigSkipArg+0x2e, calling clr!SigParserSkipExactlyOne 00000088eecef0d0 00007ff842b0422a clr!MethodDescCallSiteCallTargetWorker+0x380, calling clr!CallDescrWorkerWithHandler 00000088eecef140 00007ff842b956d6 clr!MetaSigSkipArg+0x2e, calling clr!SigParserSkipExactlyOne 00000088eecef160 00007ff842b0a69f clr!MethodDescCallSiteCallTargetWorker+0xc8, calling clr!_chkstk 00000088eecef170 00007ff842cacd88 clr!ArgIteratorTemplateForceSigWalk+0x2e7, calling clr!MetaSigGetElemSize 00000088eecef1b0 00007ff842b044c0 clr!MethodDescGetSingleCallableAddrOfVirtualizedCode+0x50, calling clr!MethodTableGetRestoredSlot 00000088eecef270 00007ff842bc4f11 clr!ThreadNativeKickOffThread_Worker+0x105, calling clr!MethodDescCallSite::CallTargetWorker 00000088eecef2e0 00007ff84c50cdd5 ntdll!RtlpLowFragHeapAllocFromContext+0x355, calling ntdll!memset 00000088eecef340 00007ff84c50c937 ntdll!RtlAllocateHeap+0xd7, calling ntdll!RtlpLowFragHeapAllocFromContext 00000088eecef450 00007ff84355a89a S


Shannon Deminick 09 Feb 2016, 09:34:19

Hi, I will advise again:

  • Disable the logger that ships with Umbraco core so that it does not show up as a false positive in these reports
  • Upgrade to the latest Umbraco version and verify that the problem is or is not solved - there is 660 issues fixed between your version and 7.3.7 including this one: http://issues.umbraco.org/issue/U4-6292


DotNetAshish 09 Feb 2016, 10:39:19

Hi @Shannon I will do that but I would request you to please go through the dump once for better suggestion as I am experiencing similar issue with one of our simple Umbraco website which is developed in a higher version 7.3.0


Shannon Deminick 09 Feb 2016, 13:36:40

I've had a quick look at your mem dumps. The first one (EOS_208) doesn't seem to contain much information and all threads listed there apart from a couple are non-managed threads. The stack trace of one of the threads (i.e. the logger thread) shows there's a logger, but a mem dump will always show a thread using the logger because it is always operating on a standalone thread.

The 2nd mem dump you sent has far more information but unfortunately it doesn't contain thread timings and therefore it's difficult to tell which thread is actually using up the CPU. That said there are a few stack traces in there that are processing inside of TracksChangesEntityBase. Since you are using an old version with a known problem, I'd suggest that for that particular site, upgrading would probably solve the issue. As previously mentioned, this is the issue I'm referring to: http://issues.umbraco.org/issue/U4-6292

If you have a recent site that has a CPU issue, you will need to ensure to first upgrade to the latest version and test again. If you can then re-produce the issue with that version, please provide steps to reproduce and attach memory dumps/logs files (basically any information that helps to reproduce the problem)


DotNetAshish 20 Feb 2016, 09:51:25

Hi @Shannon I recently learned about serious vulnerabilities allowing to perform SSRF attack, CSRF Bypass attack, and persistent XSS. Reference ticket U4-7459. Can you please suggest which version of Umbraco we should use ? Earlier suggested version by you was 7.3.6.


Sebastiaan Janssen 20 Feb 2016, 09:55:46

You need to use 7.3.2 or higher. Recommend version is 7.3.8. 7.3.6 is very much not recommended any more due to a bug discovered recently which is fixed in 7.38.


DotNetAshish 20 Feb 2016, 09:59:17

@sebastiaan what about 7.4 ?


Sebastiaan Janssen 20 Feb 2016, 10:01:23

Sure but your upgrade path will be more difficult due to small breaking changes


Sebastiaan Janssen 20 Feb 2016, 10:10:32

Yes, this is correct. The challenges depend on your app and it may be not noticeable all info is available on https://our.umbraco.org


DotNetAshish 20 Feb 2016, 10:36:42

@sebastiaan Also U4-7459 issue is fixed in 7.3.8 not in 7.4 ?


Sebastiaan Janssen 20 Feb 2016, 10:46:19

Yes, this is correct. The issue is fixed in 7.3.2 and all versions afterwards.


DotNetAshish 20 Feb 2016, 11:08:57

@sebastiaan : Now Version 7.3.2 is confusing me as in my last conversation with Shannon he sugested 7.3.6 version and I recently learned about U4-7459 bug. If U4-759 bug was fixed in 7.3.2 then which bug was introduced in 7.3.6 and fixed in 7.3.8 ? Can you please give me details as I would need to convince my team that old applications upgrade would be 7.3.8 and we would start using 7.4 for our new application.


Sebastiaan Janssen 20 Feb 2016, 11:13:34

From https://our.umbraco.org/contribute/releases/737 you can read: "This release fixes a high priority bug introduced in 7.3.6: some larger sites had significant extra load times in 7.3.6 due to a faulty update to our caching layer, this is now fixed."

All of the release notes are available;

https://our.umbraco.org/contribute/releases/732 https://our.umbraco.org/contribute/releases/733 https://our.umbraco.org/contribute/releases/734 https://our.umbraco.org/contribute/releases/735 https://our.umbraco.org/contribute/releases/736 https://our.umbraco.org/contribute/releases/737 https://our.umbraco.org/contribute/releases/738 https://our.umbraco.org/contribute/releases/740 https://our.umbraco.org/contribute/releases/741


Priority: Normal

Type: Bug

State: Closed

Assignee:

Difficulty: Normal

Category:

Backwards Compatible: True

Fix Submitted:

Affected versions:

Due in version:

Sprint:

Story Points:

Cycle: