U4-11387 - Date-properties are xml-serialized with wrong format and will cause exception in examine.

Created by Dennis Öhman 30 May 2018, 08:27:25 Updated by Dennis Öhman 18 Jun 2018, 07:14:03

Tags: Consider for sprint

If you set these properties umbracoMemberLastLockoutDate, umbracoMemberLastLockoutDate etc. to DateTime in InternalMemberIndexSet

  1. Install Umbraco with the default starterkit
  2. Set default language to nb-NO or something else
  3. Add "umbracoMemberLastLockoutDate" as DateTime in the InternalMemberIndexSet
  4. Add a login-partialview from the partialview snippets and add it to the front page
  5. Create a member in the backoffice
  6. Use the member username and generate some failed logins on the frontpage
  7. Rebuild InternalMemberIndexer.

Now you will get the exception: 2018-05-31 11:47:57,929 [P34100/D29/T117] ERROR UmbracoExamine.DataServices.UmbracoLogService - Provider=InternalMemberIndexer, NodeId=-1

System.Exception: Error indexing queue items,System.FormatException: 31.05.2018 11:47:27 is not a valid value for DateTime. ---> System.FormatException: String was not recognized as a valid DateTime. at System.DateTimeParse.Parse(String s, DateTimeFormatInfo dtfi, DateTimeStyles styles) at System.ComponentModel.DateTimeConverter.ConvertFrom(ITypeDescriptorContext context, CultureInfo culture, Object value) --- End --- at System.ComponentModel.DateTimeConverter.ConvertFrom(ITypeDescriptorContext context, CultureInfo culture, Object value) at Examine.LuceneEngine.Providers.LuceneIndexer.TryConvert[T](String val, Object& parsedVal) at Examine.LuceneEngine.Providers.LuceneIndexer.SetDateTimeField(String fieldName, String valueToParse, Resolution resolution, Index lucenePolicy, Fieldable& field, Fieldable& sortedField) at Examine.LuceneEngine.Providers.LuceneIndexer.AddDocument(Dictionary2 fields, IndexWriter writer, Int32 nodeId, String type) at Examine.LuceneEngine.Providers.LuceneIndexer.ProcessIndexQueueItem(IndexOperation op, IndexWriter writer) at Examine.LuceneEngine.Providers.LuceneIndexer.ProcessQueueItem(IndexOperation item, ICollection1 indexedNodes, IndexWriter writer) at Examine.LuceneEngine.Providers.LuceneIndexer.ForceProcessQueueItems(Boolean block) , IndexSet: InternalMemberIndexSet

Comments

Sebastiaan Janssen 30 May 2018, 11:47:50

I am not familiar with any of this, so let me play devil's advocate for a second:

What happens if you force this date format to German on a multilingual site that supports contains both German and US dates?

Also, what happens to backwards compatibility for existing sites? Does this new fix only kick in if someone writes an eventhandler to do indexing? I don't really see who this could help apart from a very small subset of people who know about this option. :-)

Again, I have no idea about how any of this works, so I'm just throwing ideas out there.


Dennis Öhman 30 May 2018, 13:38:01

Thats understandable! I also have no idea. Probably I'm wrong about all of this =)

An example with a member on a norwegian site: 1 - A member updates their profile (events happens...) 2 - A "member-cache-refresher-refresh" is added to DatabaseServerMessenger. 3 - MemberCacheRefresher in ExamineEvents triggers on all servers. 4 - The member is collected from the database and is converted to Xml and added to Examine Que. 5 - Examine running Async starts the task with InvariatCulture and fails to parse the dates in the xml.

This also happens with the startup-rebuildindex of the members if the culture is "wrong". So the "solution" wont fix this. An image of the exception: https://pbs.twimg.com/media/Dd-Lfb-U0AgnqRT.jpg:large


Dennis Öhman 30 May 2018, 13:38:09

Maybe I should set DefaultThreadCurrentCulture :P


Sebastiaan Janssen 30 May 2018, 13:43:55

Interesting, it's an error I've seen before as well. Maybe @Shandem can comment on this with a more definitive answer?


Shannon Deminick 31 May 2018, 03:28:04

There's some things I need clarification on:

*Is this a problem with Examine in particular or something else? Examine doesn't use BackgroundTaskRunner, it runs it's own threading operation *Why/how are dates being formatted and sent to Examine and not formatted as the ISO standard: yyyy-MM-dd HH:mm:ss? Are these custom date fields or Umbraco dates (i.e. created date)? *I don't think DatabaseServerMessenger needs to change at all, this just writes rows to the DB


Dennis Öhman 31 May 2018, 07:15:56

@Shandem

  • Well it sort of Examine and Umbraco (but probably me). When Umbraco converts e.g a member to xml and add the document to the Examine que, the date is converted to one format. Then when Examine starts processing the que async with it's own thread set to InvariantCulture, it fails to parse/convert the dates from the xml to DateTime objects.

  • Actually I don't know =/. I have the same problem in a custom indexes, even if I add dates to the examine xml with the format to standard yyyy-MM-dd HH:mm:ss.

  • You are right. In my defense. It was the only way I could figure out, to be able to define a culture and pass it down to BatchedDatabaseServerMessenger and forward to it's BackgroundTaskRunner and then forwarded to the background task 😆. Sorry...

Right now I have removed all the datetime from indexing.


Shannon Deminick 31 May 2018, 07:30:38

@madsoulswe can you provide simple steps to replicate from a vanilla umbraco install (latest version)? Also, what .NET Framework version are you running this site with?


Dennis Öhman 31 May 2018, 10:04:23

@Shandem Sorry I think everything I just said was unrelated to the real "issue".

If you set these properties umbracoMemberLastLockoutDate, umbracoMemberLastLockoutDate etc. to DateTime in InternalMemberIndexSet

  • Install Umbraco with the default starterkit
  • Set default language to nb-NO or something else
  • Add "umbracoMemberLastLockoutDate" as DateTime in the InternalMemberIndexSet
  • Add a login-partialview from the partialview snippets and add it to the front page
  • Create a member in the backoffice
  • Use the member username and generate some failed logins on the frontpage
  • Rebuild InternalMemberIndexer.

Now you will get the exception: 2018-05-31 11:47:57,929 [P34100/D29/T117] ERROR UmbracoExamine.DataServices.UmbracoLogService - Provider=InternalMemberIndexer, NodeId=-1

System.Exception: Error indexing queue items,System.FormatException: 31.05.2018 11:47:27 is not a valid value for DateTime. ---> System.FormatException: String was not recognized as a valid DateTime. at System.DateTimeParse.Parse(String s, DateTimeFormatInfo dtfi, DateTimeStyles styles) at System.ComponentModel.DateTimeConverter.ConvertFrom(ITypeDescriptorContext context, CultureInfo culture, Object value) --- End of inner exception stack trace --- at System.ComponentModel.DateTimeConverter.ConvertFrom(ITypeDescriptorContext context, CultureInfo culture, Object value) at Examine.LuceneEngine.Providers.LuceneIndexer.TryConvert[T](String val, Object& parsedVal) at Examine.LuceneEngine.Providers.LuceneIndexer.SetDateTimeField(String fieldName, String valueToParse, Resolution resolution, Index lucenePolicy, Fieldable& field, Fieldable& sortedField) at Examine.LuceneEngine.Providers.LuceneIndexer.AddDocument(Dictionary2 fields, IndexWriter writer, Int32 nodeId, String type) at Examine.LuceneEngine.Providers.LuceneIndexer.ProcessIndexQueueItem(IndexOperation op, IndexWriter writer) at Examine.LuceneEngine.Providers.LuceneIndexer.ProcessQueueItem(IndexOperation item, ICollection1 indexedNodes, IndexWriter writer) at Examine.LuceneEngine.Providers.LuceneIndexer.ForceProcessQueueItems(Boolean block), IndexSet: InternalMemberIndexSet


Shannon Deminick 31 May 2018, 10:54:43

great, that's what i need, now we can look at a fix


Dennis Öhman 18 Jun 2018, 07:14:03

I renamed the issue and changed the description.


Priority: Normal

Type: Bug

State: Open

Assignee:

Difficulty:

Category:

Backwards Compatible: False

Fix Submitted:

Affected versions:

Due in version:

Sprint:

Story Points:

Cycle: