U4-5789 - Add native option to UmbracoExamine library to run the indexes locally

Created by Shannon Deminick 11 Nov 2014, 23:45:04 Updated by Shannon Deminick 05 Feb 2015, 23:24:49

Based on this project:

https://github.com/Shandem/UmbracoExamine.TempStorage

We can move this functionality inside of Umbraco core and have it as a config option. When this config option is set, we'll need to fix the 'syncing' part of this library so that the local storage is always synced back to the main storage when writing. There's been notices that on app startup the index might be locked when copying over, we can solve that by storing locally in a GUID folder and removing other GUID folders in a try / catch.

Comments

Shannon Deminick 13 Nov 2014, 04:48:33

I've added this to the core now. So for each indexer/searcher in ExamineSettings.config you can specify one option:

useTempStorage="true"

If you specify these for the Indexer, you also must specify it for the associated searcher!

Here's what that does:

This enables temp/codegen folder storage for the index. The index will be stored in local temp storage and mirrored from the main storage on startup. Any indexing/writing operation will be written to both indexes (local + remote file system storage). Any search operation will only use the local index. During app startup, it will sync the remove index with the local one. If for whatever reason, the index is locked and cannot be synced then the system will just use purely the normal file system storage (on the next restart it will try again to sync locally ... so far i cannot get it to not sync in my tests)


Matthew 05 Feb 2015, 21:16:24

Is this applicable to version 7.2.1?

I've set useTempStorage to true on all providers and I'm getting many of the following error: Could not load data from Examine index for media System.IO.FileNotFoundException: Could not find file 'C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\0a0f48d7\16dfc951\App_Data\TEMP\ExamineIndexes\Internal_1l.cfs'. File name: 'C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files\root\0a0f48d7\16dfc951\App_Data\TEMP\ExamineIndexes\Internal_1l.cfs' at Lucene.Net.Index.SegmentInfos.FindSegmentsFile.Run(IndexCommit commit) at Lucene.Net.Index.DirectoryReader.DoReopenNoWriter(Boolean openReadOnly, IndexCommit commit) at Examine.LuceneEngine.Providers.LuceneSearcher.ValidateSearcher(Boolean forceReopen) at UmbracoExamine.UmbracoExamineSearcher.GetSearchFields() at UmbracoExamine.UmbracoExamineSearcher.CreateSearchCriteria(String type, BooleanOperation defaultOperation) at Umbraco.Web.PublishedCache.XmlPublishedCache.PublishedMediaCache.GetUmbracoMedia(Int32 id)


Matthew 05 Feb 2015, 22:12:45

Deleting the /TEMP/ExamineIndexes folder and then resetting the app pools clears any existing server errors on the front end, but I get this warning a few times in the log: Cannot sync index files from main storage, the index is currently locked.


Shannon Deminick 05 Feb 2015, 22:41:30

@kirschner@magnet.fsu.edu this Umbraco implementation will not work for load balanced setups and there's no option to turn off sync storage. The implementation in 7.2.2 is better and has this option but sync storage will never work in LB env (if that is what you are running) unless you use this option in 7.2.2 http://issues.umbraco.org/issue/U4-5995

If you want to test this stuff it would be better if you use a 7.2.2 nightly build with the options listed here: http://issues.umbraco.org/issue/U4-5993

since this 7.1.9 implementation is based off of Umbraco.TempStorage project with sync storage turned on ... which clearly has issues.


Matthew 05 Feb 2015, 23:14:21

Yes, we're running two nodes together on an NLB with a central file system. I've been following the [https://our.umbraco.org/documentation/installation/load-balancing load balancing documentation] pretty closely, but ran into some confusion when I got to the [https://our.umbraco.org/documentation/installation/load-balancing#LuceneExamineconfiguration Examine/Lucene configuration section]. I saw the link to the [https://github.com/Shazwazza/UmbracoExamine.TempStorage UmbracoExamine.TempStorage project] but wasn't sure what to do with it. I did some more searching and found this thread leading me to think that TempStorage was no longer necessary as of 7.1.9.

It would seem that this is not the case, and my solution lies in one of two options:

  • adding TempStorage (along with editing ExamineSettings.config) to my solution
  • or updating the site to an Umbraco 7.2.2 nightly build.

Is this correct?


Shannon Deminick 05 Feb 2015, 23:24:49

If you are load balancing you can:

  • Use Umbraco.TempStorage - you will need to build the source and follow install instructions and ensure the sync option is NOT turned on
  • Use the 7.2.2 nightly (use at your own risk but should be stable) and follow instructions here: http://issues.umbraco.org/issue/U4-5993 but ensure that the LocalOnly is set, otherwise you can experiment with the Sync option set but ensure that your indexes are stored in machine specific folders: http://issues.umbraco.org/issue/U4-5995


Priority: Normal

Type: Task

State: Fixed

Assignee:

Difficulty: Normal

Category:

Backwards Compatible: True

Fix Submitted:

Affected versions:

Due in version: 7.1.9

Sprint:

Story Points:

Cycle: