U4-10587 - Email field in InternalMemberIndex seems to be attempted to being added to the index twice

Created by Claus Jensen 24 Oct 2017, 21:54:32 Updated by Robert Copilau 12 Dec 2017, 10:31:16

Subtask of: U4-9609

Getting this in the logs when saving a member:

UmbracoExamine.DataServices.UmbracoLogService - Field "email" is listed multiple times in the index set "InternalMemberIndexSet". Please ensure all names are unique, Provider=InternalMemberIndexer, NodeId=1145

Confirmed in vanilla 7.6.11 and 7.7.3.

  • Create a member
  • Check logs

Removing the email entry from ExamineIndex.config will remove the error - but will also cause the email field to not be indexed at all. I thought it may have been added from somewhere else apart from the config file (causing the double indexing), but it seems it is actually this entry somehow being added twice?

1 Attachments


Robert Copilau 01 Nov 2017, 16:54:09

I think problem lies with the User and System having the same field. I am not quite sure how to fix as it might be and examine issue. @Shandem do you have any pointers that I could follow to fix this? Is there a way to exclude the email field from the user fields? I know that I can apply a filter and just add the fields that I want to the indexer. Should I even do that?

Shannon Deminick 02 Nov 2017, 05:39:28

@robertcopilau yes so if there are 2 (or more) fields with the same name, regardless if they are from user or system fields you will get this warning. By default this is the examine config we ship with:

  <!-- The internal index set used by Umbraco back-office for indexing members - DO NOT REMOVE -->
  <IndexSet SetName="InternalMemberIndexSet" IndexPath="~/App_Data/TEMP/ExamineIndexes/InternalMember/">
      <add Name="id" />
      <add Name="nodeName"/>
      <add Name="updateDate" />
      <add Name="writerName" />
      <add Name="loginName" />
      <add Name="email" />
      <add Name="nodeTypeAlias" /> 

And the attributes are based on the serialized member which is done here: https://github.com/umbraco/Umbraco-CMS/blob/dev-v7/src/Umbraco.Core/Services/EntityXmlSerializer.cs#L104 which is what is adding the email 'system' field.

The 'user' field gets in there because on a default installation a starter kit is installed and there is a Person document type that contains the email property type. Examine will just get a collection of all Property Types in the system and will index them if content is found.

So this won't cause problems having this warning, but it's annoying. So the 'fix' is to sub class the UmbracoDataService https://github.com/umbraco/Umbraco-CMS/blob/dev-v7/src/UmbracoExamine/DataServices/UmbracoDataService.cs for the member indexer. This is actually a service that can be set via configuration (which most people don't do). This service is constructed here by default for all indexers if not set via config: https://github.com/umbraco/Umbraco-CMS/blob/dev-v7/src/UmbracoExamine/BaseUmbracoIndexer.cs#L153 So you could create a virtual method on the BaseUmbracoIndexer, something like IDataService CreateDefaultUmbracoDataService() which by default returns new UmbracoDataService(); and the BaseUmbracoIndexer.DataService is assigned this value. Then in the UmbracoMemberIndexer you can override the new CreateDefaultUmbracoDataService and return a sub class of UmbracoDataService called something like UmbracoMemberDataService and in that class you can override ContentService to return a sub class of UmbracoContentService called something like UmbracoMemberContentService and override GetAllUserPropertyNames. Instead of doing the default lookup which is

var result = _applicationContext.DatabaseContext.Database.Fetch<string>("select distinct alias from cmsPropertyType order by alias");

you can lookup only property types that exist for member objects based on the member node object type: Constants.ObjectTypes.MemberType (9B5416FB-E72F-45A9-A07B-5A9A2709CE43) so the lookup could be:

var result = _applicationContext.DatabaseContext.Database.Fetch<string>("select distinct cmsPropertyType.alias from cmsPropertyType 
inner join cmsContentType on cmsContentType.nodeId = cmsPropertyType.contentTypeId
inner join umbracoNode on umbracoNode.id = cmsContentType.nodeId
where umbracoNode.nodeObjectType = @nodeObjectType
order by alias", new {nodeObjectType = Constants.ObjectTypes.MemberType});

clear as mud ? ;)

Robert Copilau 02 Nov 2017, 08:18:27

Not at all, now I have something to work with, thanks :D.

Shannon Deminick 11 Dec 2017, 14:51:45

PR: https://github.com/umbraco/Umbraco-CMS/pull/2347

  • Review code, ensure it makes sense
  • Test rebuilding indexes, ensure it works and that there are no errors in the error log
  • Ensure that the warning is no longer listed in the log when the starter kit is installed

Robert Copilau 12 Dec 2017, 10:31:06

All indexes rebuilds with no errors and there is no more warnings listed in the logs regarding the email field, merging.

Priority: Normal

Type: Bug

State: Fixed


Difficulty: Normal


Backwards Compatible: True

Fix Submitted:

Affected versions: 7.7.3, 7.6.11

Due in version: 7.7.8

Sprint: Sprint 74

Story Points: 0.5

Cycle: 5