Web Services¶

Listing Stored Changesets¶

List stored changesets. URL Parameters: full - returns extended history past the last snapshot.

http://example.com:8090/ws/changeset [GET or HEAD]

GET /ws/changeset HTTP/1.1

HTTP/1.1 200 OK
Content-Type: text/xml; charset=utf-8

<changesets total="1">
    <changeset id="a873e97062faebe9a4eb4402e6952188" snapshot="true"
        date="Wed, 13 Oct 2010 20:36:31 GMT" md5="ed37baa943969c6fadabed9806f58fb9"
        href="http://localhost:8090/ws/changeset/a873e97062faebe9a4eb4402e6952188"
        length="67291575" rawLength="14785239" />
</changesets>

Exporting a Stored Changeset¶

Retrieve the contents of a changeset by identifier id.

http://example.com:8090/ws/changeset/[id] [GET or HEAD]

GET /ws/changeset/123 HTTP/1.1

HTTP/1.1 200 OK
Content-Type: text/xml

[Changeset follows in response body]

Apply/Store Changeset¶

Apply and store a new changeset. The input is a changeset. In the response, the Location header is set to the URL of the changeset, and the body lists the changeset identifier.

http://example.com:8090/ws/changeset/[id] [POST]

POST /ws/changeset HTTP/1.1
Content-Type: text/xml

[Changeset follows in response body]

HTTP/1.1 201 Created
Content-Type: text/plain

3b9897903f544040ba40e48b82d01310

Export the Dimensions Document¶

Retrieve the current defined dimensions. The output is a dimension definition document.

http://example.com:8090/ws/dimensions [POST]

curl http://example.com:8090/ws/dimensions

Post a new Dimensions Document¶

Set the current defined dimensions. The input is a dimensions specification document. The content type is text/xml.

http://example.com:8090/ws/dimensions [POST]

curl --header 'Content-Type: text/xml' --data-binary @dimensions.xml \
    http://example.com:8090/ws/dimensions

Parameters¶

• id - (Required.) Identifier of item to retrieve

Example Request¶

GET /ws/item/123 HTTP/1.1

Example Response¶

HTTP/1.1 200 OK
Content-Type: application/json

{
    "_id": "123",
    "genres": "Science Fiction,Action,Adventure",
    "release_year": "2009",
    "MPAA_rating": "PG-13",
    "duration": "127",
    "title": "Star Trek",
    "synopsis": "Boldly going where no one has gone before."
}

Parameters (GET)¶

• items - (Required.) Delimited list of identifiers of items to retrieve.
• delimiter - The delimited used to separate the ids in the request. Default is a comma (,).
• properties - Changeset property ids (names) for which data should be returned. Default is all properties.

Example Request (GET)¶

GET /ws/items?items=123,missing_id&delimiter=, HTTP/1.1

Example Request (POST)¶

The POST method takes a JSON object as its input. The input defines the identifiers and (optionally) properties to return.

curl --header 'Content-Type: application/json' \
    http://example.com:8090/ws/items

The format of the JSON request object is

{
    "ids": [
            "123",
            "missing_id"
           ],
    "properties":
           [
             "_id",
             "genres",
             "release_year",
             "MPAA_rating",
             "duration",
             "title",
             "synopsis"
           ]
}

Example Response (POST and GET)¶

If any requested id is not found, then the response includes a key “_exists” that has the value of “false”.

HTTP/1.1 200 OK
Content-Type: application/json

[
    {
        "_id": "123",
        "genres": "Science Fiction,Action,Adventure",
        "release_year": "2009",
        "MPAA_rating": "PG-13",
        "duration": "127",
        "title": "Star Trek",
        "synopsis": "Boldly going where no one has gone before."
    },
    {
        "_id": "missing_id",
        "_exists": false
    }
]

Request¶

The POST endpoint expects a payload of type application/json.

A GET is equivalent to a POST with a request of:

{
  "sources": true
}

The payload can have the following fields:

scores

Whether to include scores in the metadata. Boolean that defaults to false.

sources

Whether to include sources (item properties) in the response. Boolean that defaults to false.

criteria

Optional search criteria, follows the same rules as the criteria option of a query request. Defaults to no value and thus returns all items.

exactMatchesOnly

Whether to include just exact matches from the query. Boolean that defaults to false.

• When true you will get just exact matches and their metadata will not include an exact field.
• When false you will get exact and fuzzy matches and their metadata will inclue an exact field.

values

Optional list of dimension ids whose indexed values should be included in the response. An array of string, defaults to no value.

Response¶

The response is a stream of JSON documents terminated with a newline and a final newline at the end of the stream. The declared content type is application/x-ndjson.

{header}\n
{a}\n
{b}\n
{c}\n
{d}\n
\n

The first object in the stream is a header object that describes the layout of the rest of the stream. Subsequent objects come in groups of one or more that represent a single item.

The format attribute of the header object declares the contents of all the groups in the stream. It is an array of strings which describe the contents and order of each object in a group. Valid values are any of meta, dimension, and source.

Valid object formats are:

meta

This is always present and contains metadata about the item. Valid attributes of the metadata object are id, score, and exact.

dimension

This is only present when the request contains values. The dimension object contains the indexed values of the item keyed by the dimension id.

source

This is only present when the request has sources set to true. The source object is the original JSON document that was indexed.

More simply put, each group starts with a metadata object and then optionally follows with dimension and/or source objects.

Example Request for sources¶

{
    "criteria": [
        {"dimension": "example", "value": "example"}
    ],
    "exactMatchesOnly": true,
    "source": true
}

Example Response for sources¶

{"format":["meta","source"]}\n
{"id":"1001"}\n
{"color":"red"}\n
{"id":"1002"}\n
{"color":"green"}\n
\n

Example Request for indexed values¶

{
    "criteria": [
        {"dimension": "example", "value": "example"}
    ],
    "exactMatchesOnly": true,
    "values": ["location"]
}

Example Response for indexed values¶

{"format":["meta","dimension"]}\n
{"id":"1001"}\n
{"location":[{"latitude":40.1,"longitude":-72.3}]}\n
{"id":"1002"}\n
{"location":[{"latitude":41.0,"longitude":-73.0}]}\n
\n

Parameters¶

• query - (Required.) Query string to process for suggestions
• maxSuggestions - Maximum number of suggestions to return/ integer
• distanceAlgorithm - Word distance algorithm to use. string
• morePopular - Uses indexed term relevance to identify best suggestions. boolean
• highlighting - Enabled or disables highlighting of the suggested queries
• highlighting.preTemplate - If highlighting is enabled, specifies the string to place before of the replaced query term.
• highlighting.postTemplate - If highlighting is enabled, specifies the string to place after of the replaced query term.
• escapeHtml - Sanitizes any HTML in the original query and suggestions. boolean

Example Request¶

GET /ws/didyoumean/freetext?query=st+marys&distanceAlgorithm=levenshtein
    &highlighting.preTemplate=%3Ci%3E&highlighting.postTemplate=%3C/i%3E HTTP/1.1

Example Response¶

See: DidYouMean for a description of the Did You Mean? response JSON format.

HTTP/1.1 200 OK
Content-Type: application/json

{
  "tokenCount": 2,
  "uncertainCount": 1,
  "query": {
    "value": "st marys",
    "label": "<i>st</i> <i>marys</i>"
  },
  "suggestions": [
    {
      "value": "st mary",
      "label": "st <i>mary</i>"
    },
    {
      "value": "st maria",
      "label": "st <i>maria</i>"
    },
    {
      "value": "st mark",
      "label": "st <i>mark</i>"
    },
    {
      "value": "st mar",
      "label": "st <i>mar</i>"
    }
  ]
}

Example Usage¶

To see the available data points:

$ curl http://localhost:8090/ws/statistics
http
changeset
item
checkpoint
changeset.apply
index
query
json
xmlrpc

To get the data itself GET from the URL with /fetch/${datapoint}. Where datapoint is a space delimited list of options as output from /ws/statistics.

Get the HTTP statistics:

$ curl http://localhost:8090/ws/statistics/fetch/http
http.time.count: 2663
http.time.mean: 88.51032669921146
http.time.min: 0
http.time.max: 34629
http.time.variance: 480464.4911592846
http.time.stddev: 693.1554595898994
http.time.sum: 235703
http.uncaught.io: 0
http.uncaught.runtime: 0
http.uncaught.error: 0

Get the item statistics:

$ curl http://localhost:8090/ws/statistics/fetch/item
item.count: 2381
item.errors: 0
item.disk: 8518018

Get both the HTTP and item statistics:

$ curl http://localhost:8090/ws/statistics/fetch/item+http
item.count: 2381
item.errors: 0
item.disk: 8518018
http.time.count: 2668
http.time.mean: 88.34632683658175
http.time.min: 0
http.time.max: 34629
http.time.variance: 479578.0629898773
http.time.stddev: 692.515749272085
http.time.sum: 235708
http.uncaught.io: 0
http.uncaught.runtime: 0
http.uncaught.error: 0

The returned data expands as needed, so if an engine has only serviced one query it will look like this:

$ curl http://localhost:8090/ws/statistics/fetch/query
query.regular.count: 1
query.regular.size.mean: 22555
query.regular.size.sum: 22555
query.regular.time.mean: 91
query.regular.time.sum: 91

If it has serviced two queries then you’ll get more information:

$ curl http://localhost:8090/ws/statistics/fetch/query
query.regular.count: 2
query.regular.size.mean: 22556.0
query.regular.size.min: 22555
query.regular.size.max: 22557
query.regular.size.variance: 2.0
query.regular.size.stddev: 1.4142135623730951
query.regular.size.sum: 45112
query.regular.time.mean: 70.0
query.regular.time.min: 49
query.regular.time.max: 91
query.regular.time.variance: 882.0
query.regular.time.stddev: 29.698484809834994
query.regular.time.sum: 140

Field details¶

The most interesting fields are described here.

Total time taken for all queries returning non empty results

• query.regular.time.sum

Total time taken for queries returning empty results

• query.empty.time.sum

Total number of invalid/failed/successful XMLRPC

• xmlrpc.invalid.count
• xmlrpc.failed.count
• xmlrpc.success.count

Total time taken for invalid/failed/successful XMLRPC

• xmlrpc.invalid.sum
• xmlrpc.failed.sum
• xmlrpc.success.sum

Total number of invalid/failed/successful JSON calls

• json.invalid.count
• json.failed.count
• json.success.count

Total time taken for invalid/failed/successful JSON calls

• json.invalid.sum
• json.failed.sum
• json.success.sum

Number of items in the current partition

• index.items

Number of indexes

• index.count

Number of items in the dataset

• item.count

Size of the dataset on disk (db/items directory)

• item.disk

Number of created changesets by type:

• changeset.reset.size.count
• changeset.delta.size.count
• changeset.snapshot.size.count
• changeset.bulk.size.count
• changeset.checkpoint.size.count

Total uncompressed size:

• changeset.reset.size.sum
• changeset.delta.size.sum
• changeset.snapshot.size.sum
• changeset.bulk.size.sum
• changeset.checkpoint.size.sum

Total compressed size:

• changeset.reset.compressed.sum
• changeset.delta.compressed.sum
• changeset.snapshot.compressed.sum
• changeset.bulk.compressed.sum
• changeset.checkpoint.compressed.sum

Total number of applied changesets (those that are written to the DB file)

• changeset.apply.count

Break down of item actions across the changeset applications

• changeset.apply.created.sum
• changeset.apply.modified.sum
• changeset.apply.deleted.sum

Total time taken generating checkpoints:

• checkpoint.time.sum

Total number of checkpoints generated;

• checkpoint.time.count

Total number of HTTP requests served;

• http.time.count

Total time to server them:

• http.time.sum

Example Usage¶

To create a checkpoint:

$ curl -X POST http://example.com:8090/ws/checkpoint

Optional URL Parameters¶

The following options can be specified as query parameters on the URL

• success - An integer status code value that determines the status code returned if the engine is “queryable.” For example, if using an Amazon elastic load balancer that does not support status code 204, success=200 might be an appropriate option.

  Default: 204 (No Content)

• error - An integer status code value that determines the status code returned if the engine is not “queryable”.

  Default: 503 (Service Unavailable)

  New in version 3.8.

• retryAfter - Optional value for the response header Retry-After for when the engine is not “queryable”.

  Default: 30

  New in version 3.8.

http://example.com:8090/ws/queryable [GET]

Example Usage¶

To check if the engine can respond to queries.

$ curl --head http://example.com:8090/ws/queryable

Example Response¶

<system-state>
    <running>
        <true />
    </running>
    <queryable>
        <true />
    </queryable>
</system-state>