We have moved to GitHub Issues
Created by Shannon Deminick 17 Jan 2017, 06:18:00 Updated by Matt Harding 28 Mar 2017, 11:33:06Tags: Unscheduled
Relates to: U4-9371
Relates to: U4-9393
We already do this for media since it is much much faster so we should be able to do this for both content and members since both published content and member data exist serialized in the cmsContentXml table.
initial benchmarks of this show that with 8500 items in the index, the current way using the normal IContentService lookups takes: 6014ms, when changed to use the cmsContentXml table it takes 516ms which is 92% faster.
Similarly, for 10001 members, using the fast lookup takes 559ms and the current slow lookup takes 9773ms = 94% faster
For testing follow these instructions: http://issues.umbraco.org/issue/U4-9371#comment=67-33971
Then you can test the old way by adding this attribute to your indexer:
disableXmlDocLookup=true, verify that the amount of data that is in the index is the same the new way and the old way. The default is
disableXmlDocLookup=false which will be the case when this attribute is not specified.
That.. is SO much faster!
I went down from a site using 8 minutes to build a member index (80k members) to 1.5 minutes.
YES!!!! I know you've prob checked but want to make sure that only published content ends up in the external index, it defo should work but want to be sure i didn't re revert the unfix that i did last time with descendants :P
This release is going to be so much better for examine and azure and we can remove that item from the wall :)
I found while re-indexing the Our Umbraco data that we needed to do some extra checks, so the PR has been re-opened with these changes: https://github.com/umbraco/Umbraco-CMS/pull/1694/commits/931db6f0b177cd381d78dd9005b13c971dffc2a3
@Shandem can you check these additional commits?
I learned that for the 120k members in my local Our copy that it took about 14 minutes for the full reindex to run 2017-01-17 15:18:47,243 [P10440/D12/T15] INFO UmbracoExamine.DataServices.UmbracoLogService - PerformIndexAll - Start data queries - member, Provider=InternalMemberIndexer, NodeId=-1 2017-01-17 15:32:16,706 [P10440/D12/T15] INFO UmbracoExamine.DataServices.UmbracoLogService - PerformIndexAll - End data queries - member, took 809463ms, Provider=InternalMemberIndexer, NodeId=-1
Our has some more custom properties than the other test site I used, so it's not as fast as that one and I haven't tried running the reindex before the upgrade (I'll do that to see how fast/slow it goes).
@sebastiaan I've re-opend for your review. I've committed to the dev-v7 branch here https://github.com/umbraco/Umbraco-CMS/commit/db414e8045d0a455dae568a4f67675eab6c3ccef
If you can please have a look and then test the scenarios:
All checked, all good now!
Just for info, this change also resolves http://issues.umbraco.org/issue/U4-9677 which caused external index rebuilds not to include published items that had a newer unpublished version.
Backwards Compatible: True
Due in version: 7.5.8
Sprint: Sprint 50