U4-11202 - v8 / Database Issues (1)

Created by Stephan 11 Apr 2018, 10:00:13 Updated by Claus Jensen 31 May 2018, 08:22:20

Subtask of: U4-11279

Seeing some weird exceptions w/ database: open data reader on connection, closed transations...

Something's got to be wrong in the scope / database code.

Comments

Stephan 19 Apr 2018, 14:17:21

Also reported by @Shandem :

UPDATE [umbracoUserLogin] SET [userId] = @0, [loggedInUtc] = @1, [lastValidatedUtc] = @2, [loggedOutUtc] = @3, [ipAddress] = @4 WHERE [sessionId] = @5 -- @0:-1 @1:19/04/2018 8:19:20 AM @2:19/04/2018 10:46:49 AM @3: @4:::1 @5:cbe51cb9-9b18-48f8-a34c-af48a83c6787 2018-04-19 20:46:50,724 [P15064/D3/T104] ERROR Umbraco.Core.UmbracoApplicationBase - An unhandled exception occurred. System.InvalidOperationException: This SqlTransaction has completed; it is no longer usable. at System.Data.SqlClient.SqlTransaction.ZombieCheck() at System.Data.SqlClient.SqlTransaction.Commit()


Shannon Deminick 01 May 2018, 23:05:05

Another error seen:

Server Error in '/' Application.
The handle is invalid. (Exception from HRESULT: 0x80070006 (E_HANDLE))
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. 
Exception Details: System.Runtime.InteropServices.COMException: The handle is invalid. (Exception from HRESULT: 0x80070006 (E_HANDLE))
Source Error: 
Line 248:                    _database.Dispose();
Line 249:                    _database = null;
Line 250:                    throw;
Line 251:                }
Line 252:            }
Source File: C:\Users\Shannon\Documents\_Projects\Umbraco\Umbraco_8.0\src\Umbraco.Core\Scoping\Scope.cs    Line: 250 
Stack Trace: 
[COMException (0x80070006): The handle is invalid. (Exception from HRESULT: 0x80070006 (E_HANDLE))]
   System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo) +0
   System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection) +890
   System.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection) +159
   System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection) +1079
   System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions) +309
   System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry) +198
   System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry) +422
   System.Data.SqlClient.SqlConnection.Open() +199
   NPoco.Database.OpenSharedConnectionImp(Boolean isInternal) +223
   NPoco.Database.BeginTransaction(IsolationLevel isolationLevel) +36
   Umbraco.Core.Scoping.Scope.get_Database() in C:\Users\Shannon\Documents\_Projects\Umbraco\Umbraco_8.0\src\Umbraco.Core\Scoping\Scope.cs:250
   Umbraco.Core.Persistence.Repositories.Implement.NPocoRepositoryBase`2.get_Database() in C:\Users\Shannon\Documents\_Projects\Umbraco\Umbraco_8.0\src\Umbraco.Core\Persistence\Repositories\Implement\NPocoRepositoryBase.cs:31
   Umbraco.Core.Persistence.Repositories.Implement.UserRepository.PerformGet(Int32 id) in C:\Users\Shannon\Documents\_Projects\Umbraco\Umbraco_8.0\src\Umbraco.Core\Persistence\Repositories\Implement\UserRepository.cs:91
   Umbraco.Core.Cache.DefaultRepositoryCachePolicy`2.Get(TId id, Func`2 performGet, Func`2 performGetAll) in C:\Users\Shannon\Documents\_Projects\Umbraco\Umbraco_8.0\src\Umbraco.Core\Cache\DefaultRepositoryCachePolicy.cs:160
   Umbraco.Core.Persistence.Repositories.Implement.RepositoryBase`2.Get(TId id) in C:\Users\Shannon\Documents\_Projects\Umbraco\Umbraco_8.0\src\Umbraco.Core\Persistence\Repositories\Implement\RepositoryBaseOfTIdTEntity.cs:197
   Umbraco.Core.Services.Implement.UserService.GetUserById(Int32 id) in C:\Users\Shannon\Documents\_Projects\Umbraco\Umbraco_8.0\src\Umbraco.Core\Services\Implement\UserService.cs:728
   Umbraco.Core.Security.<FindByIdAsync>d__11.MoveNext() in C:\Users\Shannon\Documents\_Projects\Umbraco\Umbraco_8.0\src\Umbraco.Core\Security\BackOfficeUserStore.cs:183
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() +31
   Umbraco.Core.Security.<ValidateSessionAsync>d__2.MoveNext() in C:\Users\Shannon\Documents\_Projects\Umbraco\Umbraco_8.0\src\Umbraco.Core\Security\SessionIdValidator.cs:95
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() +31
   Umbraco.Core.Security.<ValidateSessionAsync>d__1.MoveNext() in C:\Users\Shannon\Documents\_Projects\Umbraco\Umbraco_8.0\src\Umbraco.Core\Security\SessionIdValidator.cs:36
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   System.Runtime.CompilerServices.TaskAwaiter.GetResult() +28
   Umbraco.Core.Security.<EnsureValidSessionId>d__7.MoveNext() in C:\Users\Shannon\Documents\_Projects\Umbraco\Umbraco_8.0\src\Umbraco.Core\Security\BackOfficeCookieAuthenticationProvider.cs:109
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   System.Runtime.CompilerServices.TaskAwaiter.GetResult() +28
   Umbraco.Core.Security.<ValidateIdentity>d__6.MoveNext() in C:\Users\Shannon\Documents\_Projects\Umbraco\Umbraco_8.0\src\Umbraco.Core\Security\BackOfficeCookieAuthenticationProvider.cs:95
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   Microsoft.Owin.Security.Cookies.<AuthenticateCoreAsync>d__2.MoveNext() +2532
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   Microsoft.Owin.Security.Infrastructure.<BaseInitializeAsync>d__0.MoveNext() +817
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   Microsoft.Owin.Security.Infrastructure.<Invoke>d__0.MoveNext() +329
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   Microsoft.AspNet.Identity.Owin.<Invoke>d__0.MoveNext() +448
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   Microsoft.AspNet.Identity.Owin.<Invoke>d__0.MoveNext() +448
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.<RunApp>d__5.MoveNext() +193
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.<DoFinalWork>d__2.MoveNext() +184
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.StageAsyncResult.End(IAsyncResult ar) +117
   System.Web.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +511
   System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step) +213
   System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +131


Claus Jensen 18 May 2018, 09:45:30

Changing this to a spike instead. It's something we need to investigate a bit for and the outcome will be new tasks for fixing the issues we locate.


Stephan 18 May 2018, 12:12:31

Hint: have reproduced - it ''could'' be this: when a Sql statement causes a hard error, Sql aborts the transaction (at server level) and so should we (in our code) - but, if we, like, try-catch and retry, or just try to properly rollback the transaction - well it can fail because the transaction is gone already.

Not sure how to deal with it, not sure what's the impact WRT our retry strategy... but just adding this note to remember.

See also: SET XACT_ABORT (Transact-SQL)


Stephan 18 May 2018, 12:47:49

Hint: also seems to apply if one transaction is deadlocked by another one, and that ''can'' happen while debugging... need to investigate


Stephan 29 May 2018, 09:16:21

Also:

Failed to build PureLive models. There is already an open DataReader associated with this Command which must be closed first.

at Umbraco.Core.Scoping.Scope.ReadLock(Int32[] lockIds) at Umbraco.Core.Services.Implement.ContentTypeServiceBase``3.GetAll(Int32[] ids) at Umbraco.ModelsBuilder.Umbraco.Application.GetAllTypes() at Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory.GetModelsAssembly(Boolean forceRebuild) at Umbraco.ModelsBuilder.Umbraco.PureLiveModelFactory.EnsureModels()


Stephan 29 May 2018, 10:48:27

update: identified a deadlock scenario in our code, that explains the first problem seen by Shannon ''and'' might also explain others - investigating


Stephan 30 May 2018, 08:46:38

happens in branch temp8-U4-11202 with PR https://github.com/umbraco/Umbraco-CMS/pull/2652

status: deadlock in UserRepository identified and fixed, looking at other potential issues


Stephan 31 May 2018, 06:06:47

moving to review because the branch contains a fix that should deal with the most common exception - BUT - it does not fix everything, so after review & merge, PLEASE reopen the issue or something - not done yet


Stephan 31 May 2018, 06:12:50

moving to review - the branch contains a fix that should deal with the most common exception

we are not done yet, so the rest of the task is now U4-11392.


Priority: Major

Type: Task

State: Fixed

Assignee:

Difficulty: Normal

Category:

Backwards Compatible: True

Fix Submitted:

Affected versions: 8.0.0

Due in version: 8.0.0

Sprint: Sprint 86

Story Points: 3

Cycle: