U4-7046 - Cannot create content - due to transaction changes/master/slave/scheduled tasks

Created by Shannon Deminick 31 Aug 2015, 16:02:30 Updated by Stephan 04 Sep 2015, 07:33:07

Relates to: U4-6992

Try to create any content in the back office and you will get the following error. We need to test, test, test and re-test that this transaction update doesn't also affect everything else especially since we are in RC.

Received an error from the server An error occurred

Already in a transaction with a lower isolation level.

EXCEPTION DETAILS:

System.Exception: Already in a transaction with a lower isolation level. STACKTRACE:

at Umbraco.Core.Persistence.Database.BeginTransaction(IsolationLevel isolationLevel) in X:\Projects\Umbraco\Umbraco_7\src\Umbraco.Core\Persistence\PetaPoco.cs:line 333 at Umbraco.Core.Persistence.Transaction..ctor(Database db, IsolationLevel isolationLevel) in X:\Projects\Umbraco\Umbraco_7\src\Umbraco.Core\Persistence\PetaPoco.cs:line 2336 at Umbraco.Core.Persistence.Database.GetTransaction(IsolationLevel isolationLevel) in X:\Projects\Umbraco\Umbraco_7\src\Umbraco.Core\Persistence\PetaPoco.cs:line 301 at Umbraco.Core.Persistence.LockingRepository1.WithReadLocked[TResult](Func2 func, Boolean autoCommit) in X:\Projects\Umbraco\Umbraco_7\src\Umbraco.Core\Persistence\LockingRepository.cs:line 50 at Umbraco.Core.Services.ServerRegistrationService.GetActiveServers() in X:\Projects\Umbraco\Umbraco_7\src\Umbraco.Core\Services\ServerRegistrationService.cs:line 122 at Umbraco.Core.Sync.DatabaseServerRegistrar.get_Registrations() in X:\Projects\Umbraco\Umbraco_7\src\Umbraco.Core\Sync\DatabaseServerRegistrar.cs:line 38 at Umbraco.Web.Cache.DistributedCache.Refresh(Guid factoryGuid, Int32 id) in X:\Projects\Umbraco\Umbraco_7\src\Umbraco.Web\Cache\DistributedCache.cs:line 127 at Umbraco.Web.Cache.DistributedCacheExtensions.RefreshUserPermissionsCache(DistributedCache dc, Int32 userId) in X:\Projects\Umbraco\Umbraco_7\src\Umbraco.Web\Cache\DistributedCacheExtensions.cs:line 90 at Umbraco.Web.Cache.CacheRefresherEventHandler.<>c.<CacheRefresherEventHandler_AssignedPermissions>b__36_1(Int32 x) in X:\Projects\Umbraco\Umbraco_7\src\Umbraco.Web\Cache\CacheRefresherEventHandler.cs:line 507 at Umbraco.Core.EnumerableExtensions.ForEach[TItem](IEnumerable1 items, Action1 action) in X:\Projects\Umbraco\Umbraco_7\src\Umbraco.Core\EnumerableExtensions.cs:line 92 at Umbraco.Web.Cache.CacheRefresherEventHandler.CacheRefresherEventHandler_AssignedPermissions(PermissionRepository1 sender, SaveEventArgs1 e) in X:\Projects\Umbraco\Umbraco_7\src\Umbraco.Web\Cache\CacheRefresherEventHandler.cs:line 507 at Umbraco.Core.Events.EventExtensions.RaiseEvent[TSender,TArgs](TypedEventHandler2 eventHandler, TArgs args, TSender sender) in X:\Projects\Umbraco\Umbraco_7\src\Umbraco.Core\Events\EventExtensions.cs:line 47 at Umbraco.Core.Persistence.Repositories.PermissionRepository1.ReplaceEntityPermissions(EntityPermissionSet permissionSet) in X:\Projects\Umbraco\Umbraco_7\src\Umbraco.Core\Persistence\Repositories\PermissionRepository.cs:line 260 at Umbraco.Core.Persistence.Repositories.ContentRepository.PersistNewItem(IContent entity) in X:\Projects\Umbraco\Umbraco_7\src\Umbraco.Core\Persistence\Repositories\ContentRepository.cs:line 397 at Umbraco.Core.Persistence.Repositories.RepositoryBase2.PersistNewItem(IEntity entity) in X:\Projects\Umbraco\Umbraco_7\src\Umbraco.Core\Persistence\Repositories\RepositoryBase.cs:line 307 at Umbraco.Core.Persistence.UnitOfWork.PetaPocoUnitOfWork.Commit(Action1 transactionCompleting) in X:\Projects\Umbraco\Umbraco_7\src\Umbraco.Core\Persistence\UnitOfWork\PetaPocoUnitOfWork.cs:line 112 at Umbraco.Core.Persistence.UnitOfWork.PetaPocoUnitOfWork.Commit() in X:\Projects\Umbraco\Umbraco_7\src\Umbraco.Core\Persistence\UnitOfWork\PetaPocoUnitOfWork.cs:line 92 at Umbraco.Core.Services.ContentService.SaveAndPublishDo(IContent content, Int32 userId, Boolean raiseEvents) in X:\Projects\Umbraco\Umbraco_7\src\Umbraco.Core\Services\ContentService.cs:line 1967 at Umbraco.Core.Services.ContentService.Umbraco.Core.Services.IContentServiceOperations.SaveAndPublish(IContent content, Int32 userId, Boolean raiseEvents) in X:\Projects\Umbraco\Umbraco_7\src\Umbraco.Core\Services\ContentService.cs:line 881 at Umbraco.Core.Services.ContentService.SaveAndPublishWithStatus(IContent content, Int32 userId, Boolean raiseEvents) in X:\Projects\Umbraco\Umbraco_7\src\Umbraco.Core\Services\ContentService.cs:line 1040 at Umbraco.Web.Editors.ContentController.PostSave(ContentItemSave contentItem) in X:\Projects\Umbraco\Umbraco_7\src\Umbraco.Web\Editors\ContentController.cs:line 272 at lambda_method(Closure , Object , Object[] ) at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass10.b__9(Object instance, Object[] methodParameters) at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.Execute(Object instance, Object[] arguments) at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ExecuteAsync(HttpControllerContext controllerContext, IDictionary`2 arguments, CancellationToken cancellationToken) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Controllers.ApiControllerActionInvoker.d__0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.ActionFilterAttribute.d__5.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Web.Http.Filters.ActionFilterAttribute.d__5.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.ActionFilterAttribute.d__0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.ActionFilterAttribute.d__5.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Web.Http.Filters.ActionFilterAttribute.d__5.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.ActionFilterAttribute.d__0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.ActionFilterAttribute.d__5.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Web.Http.Filters.ActionFilterAttribute.d__5.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.ActionFilterAttribute.d__0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.ActionFilterAttribute.d__5.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Web.Http.Filters.ActionFilterAttribute.d__5.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.ActionFilterAttribute.d__0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.ActionFilterAttribute.d__5.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Web.Http.Filters.ActionFilterAttribute.d__5.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.ActionFilterAttribute.d__0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Controllers.ActionFilterResult.d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.AuthorizationFilterAttribute.d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Filters.AuthorizationFilterAttribute.d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Dispatcher.HttpControllerDispatcher.d__1.MoveNext()

Comments

Stephan 31 Aug 2015, 16:12:07

Arh... so ''within'' a database transaction we suddenly decide we need to trigger some distributed cache notifications? While at the same time holding the content service write lock? Oh my... I never imagined we would be doing server-related stuff as part of another transaction. Need to think about it.


Stephan 01 Sep 2015, 11:33:51

Fixed by having the repository cache the list of server registrations. The cache is NOT populated "on demand" because that would mean reading the registrations in any "default" transaction, ie ReadCommited transactions, and that is not safe. So the cache is populated whenever registrations are written to (within a proper write-lock & transaction) ie at least each time the server is touched.

Some additional notes: we ''need'' to cache server registrations somehow, since anytime events trigger the DistributedCache may want to get them--and we cannot hit the database all the time. However... not sure at all that the best place to cache is within the repository. Caching in a repository means that the repository can pretend that some element exists, when it actually has changed or been removed. There ''will'' be a lapse of time before our distributed mechanism updates the caches.

It's OK here, because we have stuff in place to ensure it works.

But... caching in a repository for eg content or media, means that we cannot reliably update anything on multiple nodes simultaneously. And the database-level lock mechanisms cannot work if we do not hit the database. See NHibernate: level-1 cache is for the current session, and level-2 cache needs to be either centralized (memcached) or have a ''pretty good'' sync mechanism to ensure consistency.


Stephan 04 Sep 2015, 07:33:05

This precise issue (content creation) is fixed.


Priority: Show-stopper

Type: Bug

State: Fixed

Assignee: Stephan

Difficulty: Normal

Category:

Backwards Compatible: True

Fix Submitted:

Affected versions:

Due in version: 7.3.0

Sprint:

Story Points:

Cycle: