4.0 (December 16 2015)

Upgrading

4.x represents a major upgrade from the 3.x series. This release replaces our custom in-memory indices with Apache Lucene.

One result of this change is that the engine’s memory footprint is much smaller. Datasets that previously required gigabytes of RAM can now be handled by an engine configured to use 256MB or 512MB. Indices are accessed via memory mapped files on disk, which means that the engine relies on your OS disk cache more. It is important that you monitor the size of your indices, and ensure that you have sufficient RAM for the OS disk cache to be effective.

Because the indices now reside on disk, waiting minutes for the engine to startup is a thing of the past. The engine now typically boots in just a few seconds.

Be aware that posting a new dimension file will now cause the engine to reindex the entire dataset. The engine can no longer reindex just the additional or modified dimensions. A new index will be created in the background, and will be made available for search as soon as it has caught up with the changes applied to the current index. This way, the engines data will never revert back to a previous state when the new index comes online.

If you are using Java 6 you will need to upgrade to the latest Oracle Java 7 or 8, as the Lucene library requires a minimum of Java 7.

Directory Layout

In order to allow for a quicker downgrade process, the engine leaves 3.x specific files on disk when performing an in-place upgrade from 3.x to 4.x

3.x and 4.x engines use the following common files for data:

db/dimensions.xml
db/changesets/

3.x engines use:

db/items/
db/text/

4.x engines use:

db/indices/gen-{number}
db/indices/index-{number}/config.json
db/indices/index-{number}/dimensions.xml
db/indices/index-{number}/index/

When performing an in-place upgrade from 3.x to 4.x, the db/items and db/text directories will be left intact, and a new db/indices directory will be created. Once you are sure you no longer want to perform an in-place downgrade to 3.x, then you can remove the db/items and db/text directories.

When performing an in-place downgrade from 4.x to 3.x, the db/indices directory will be left intact, and the existing db/items and db/text directories will be used. You can remove the db/indices directory if you do not plan to upgrade again soon.

Compatibility Changes

ENGINE-939 Replaces explain API with debug API

The explain API has been replaced by the new debug API.

For more information, see Debug Request and Debug Response.

ENGINE-833 Remove LRU Item Cache

The item cache used by the properties pipeline of queries has been removed.

ENGINE-838 Removes reindexMethod=createDrop|warmSwap option from dimensions

Dimensions no longer support an explicit reindexMethod, instead the entire dataset is reindexed for every dimensions file change.

There is a new option in discovery.properties called index-max-catchup-on-swap that helps to control the way the new index is made available when applying dimensions file changes. The option defaults to 5 which causes the engine to try and catch up with changesets in the new index before making it current (warm swap). If you want to emulate a cold swap (immediately making the new index available) then set this value to 0.

ENGINE-1027 Replace engine startup options for 4.x

The two startup options on the settings tab have been removed.

  • Engine responds while indexing at startup (default of true)
  • Engine rebuilds indexes from scratch at startup (default of false)

The engine will respond while indexing at startup by default. If you want to disable this (for load balancing purposes) add a line to discovery.properties like so:

index-max-ready-catchup-on-startup =  5

The default value for this property is 0, which causes the engine to be queryable as soon as possible. Any larger value causes it to wait until changesets have been applied. The limit exists to avoid saturation events and 5 is a good default value to emulate the older behavior.

If you wish to rebuild indices on startup, just remove the db/indices directory manually when the engine is stopped.

Improvements

ENGINE-1016 Adds an ordinal map cache for groupBy searches

Introduces an ordinal map cache to speed up groupBy searches.

ENGINE-1002 Upgrades Apache Lucene from 4.10.4 to 5.4.0

The Apache Lucene library has been upgraded from 4.10.4 to 5.4.0.