U4-5055 - Umbraco update fails with large set of media content due to timeout in RebuildXmlStructures

Created by Bradley Kronson 05 Jun 2014, 14:40:56 Updated by wtct 05 May 2016, 17:01:58

Relates to: U4-5427

Subtask of: U4-5411

I have an Umbraco install with over 300 000 media nodes (yes)

I'm trying to upgrade from 6.1.6 to 6.20 or 6.21. The upgrade runs for several hours and fails.

It is failing in the routine called RebuildXmlStructures. Which I see rebuilds the XML for the media content. I see this routine is not that optimally built - it seems to load ALL media items in to memory and then rebuild the XML content for them. Which is probably why it is failing.

My question is this, is it necessary for the RebuildXmlStructures to be called for an upgrade from 6.1.6 to 6.20 or 6.21 - what would happen if I commented out this line and ran the upgrade without this.

I was also thinking of manually changing the data type GUID for all media items in the database to something else and the running the upgrade so they will be skipped. And then changing it back afterwards. Is this a bad idea ?

Any solution for making the upgrade happen would be appreciated. I need the new version, it offers some bug fixes I need and I also want to use memberservice API

Comments

Shannon Deminick 29 Sep 2014, 07:15:56

Hi @abacus ,

You could use your workaround by changing the data type GUID for media items and then changing them back.

IIRC the reason for the xml structure rebuild for media was that there was a previous bug that was not keeping those xml structures updated in a consistent format. If you aren't having a problem with your media (sounds like you are not) then you might not have to rebuild your structures. From memory, i think the issue related to the xml structures for media not being kept up to date properly which would then cause problems with Examine since it was indexing incorrect media values.

We've made lots of performance improvements with the DAL in 7.2, i'll have a look to see if this process can be improved as well. (I'm sure it can! but not sure if there'll be enough time to fix it before the next release).


Bradley Kronson 29 Sep 2014, 07:37:53

Thanks for the feedback @Shandem - yup not experiencing any issues with the media

Will the performance improvements be fed downstream to version 6 ?


Shannon Deminick 29 Sep 2014, 07:40:14

Unfortunately not, there's quite a lot of changes that have been made that will be available only in 7.2+


Shannon Deminick 23 Oct 2014, 07:31:48

I've updated rebuilding of xml structures here: 44a39e7ca6765bbf2f35c2f62d741447b49a2efa It will no longer put everything into memory and will do it in batches but still in a single db transaction.


Torben Rahbek Koch 10 Apr 2015, 10:18:53

The problem still remains, though. Just upgradede a 6.2.5 (with around 45000 media items) to 7.2.4 and experienced the timeout. The workaround with changing the nodeObjectType Guid works.

Seems the command timeout is set to low. Maybe you could look up how many media items there are and adjust the timeout accordingly?

Just an idea.


Shannon Deminick 13 May 2015, 22:53:56

The problem is the http timeout of the request for the upgrade process. We'd need to make it an async request with a large timeout


wtct 05 May 2016, 17:01:58

I have similar problem when trying to upgrade from 4.7.2 to 6.2.6. We have more than 225 000 media nodes.

I have just joined to thread:

https://our.umbraco.org/forum/core/general/55426-v41110-Upgrade-to-v621-Hangs-on-Database-Upgrade-Screen-at-90

@Shannon could you wonder which version has problems with xml of medias?


Priority: Major

Type: Performance Problem

State: Fixed

Assignee: Shannon Deminick

Difficulty: Normal

Category:

Backwards Compatible: True

Fix Submitted:

Affected versions:

Due in version: 7.2.0

Sprint:

Story Points:

Cycle: