Script Fields & Debugging
As mentioned in the chapter Aggregation Data Tables , each individual hit returned from a _search request can contain custom attributes generated on-the-fly through a script.
These custom attributes (script fields) are typically either:
- modifications of the original values (e.g. price conversions, differently sorted arrays, …)
- or newly computed values (e.g. sums, boosts, exponentials, distance measurements, etc.)
A _search request can specify more than one script field at once:
POST myindex/_search
{
"script_fields": {
"using_doc_values": {
"script": "doc['price'].value * 42"
},
"using_source": {
"script": "params['_source']['price'] * 42"
}
}
} Unlike in script queries where you can only access doc_values , script fields also allow for accessing the original document _source.
Quoting the docs : the doc[...] notation will cause the terms for that field to be loaded to memory (cached), which will result in faster execution. But keep in mind that this notation only allows for accessing simple-valued fields (you can't get a JSON object from it — only from the _source). You can, however, target array fields too, as I've shown here .
Despite the fact that accessing _source directly will be slower that accessing the doc-values, scripts in script fields are only executed for the top N documents. As such, they aren't really a big performance consideration — particularly when you imagine that your request's size would typically be in the lower double digits.
I discuss script speed in more detail in the parent chapter .
Script fields can be used in Kibana charts as a means of data cleansing but also as part of regular _search requests.
Given an index containing smartphones:
POST smartphones/_doc
{
"name": "iPhone 12 Pro Max",
"color": "gold",
"last_modified_utc": 1609521634371,
"price": {
"amount": 1600000,
"currency": "usd"
},
"availability_per_gb": [
{
"gigabytes": 128, "units": 58
},
{
"gigabytes": 256, "units": 32
},
{
"gigabytes": 512, "units": 0
}
],
"warehouse_location": [
-97.71703, 30.32035
]
}PUT smartphones
{
"mappings": {
"properties": {
"name": {
"type": "text",
"fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } },
"color": {
"type": "keyword"
},
"last_modified_utc": {
"type": "date", "format": "epoch_millis"
},
"price": {
"properties": {
"amount": { "type": "long" },
"currency": { "type": "keyword" }
}
},
"availability_per_gb": {
"type": "nested",
"properties": {
"gigabytes": {
"type": "integer"
},
"units": {
"type": "integer"
}
}
},
"warehouse_location": {
"type": "geo_point"
}
}
}
}You want to retrieve:
- the
colorin upper case - the
last_modified_utcdate in the formatyyyy/MM/dd HH:mm:ss& in PST instead of UTC - the geo distance in miles from the
warehouse_locationto a customer in Phoenix, AZ - the number of all available
unitsinside the fieldavailability_per_gb.
After familiarizing yourself with the available Painless methods on doc_values , you'll discover that points 1, 2, and 3 can all be achieved via doc-values, and point 4 requires that you access the document _source.
1. Converting a keyword field to upper case is straightforward:
POST smartphones/_search
{
"script_fields": {
"uppercase_name": {
"script": "doc['color'].value.toUpperCase()"
}
}
} 2. Converting a millisecond epoch timestamp to a date string in Painless can be done by first transforming the timestamp into a java.time.Instant object and then formatting a DateTimeFormatter with the desired time zone — America/Los_Angeles (PST):
POST smartphones/_search
{
"script_fields": {
"parsed_last_modified": {
"script": """
DateTimeFormatter
.ofPattern('yyyy/MM/dd HH:mm:ss')
.withZone(ZoneId.of('America/Los_Angeles'))
.format(Instant.ofEpochMilli(doc['last_modified_utc'].value.millis));
"""
}
}
} 3. The geo point doc-values support the arcDistance method which expects lat & lon as arguments (in that order). The coordinates of Phoenix, AZ, are 33.4484, -112.0740 so calculating the respective geo distances to the city of Phoenix can be executed through: