The Journey to Support Nanosecond Timestamps in Elasticsearch

The ability to store dates in nanosecond resolution required a significant refactoring within the Elasticsearch code base. Read this blog post for the why and how on our journey to be able to store dates in nanosecond resolution from Elasticsearch 7.0 onwards. Elasticsearch supports a date mapping type that parses the string representation of a date in a variety of configurable formats, converts this date into milliseconds since the epoch and then stores it as a long value in Lucene.

This simple yet powerful approach has brought us a long way. However, there are many use cases which require dates to be stored at a higher resolution than just milliseconds. It’s not only the common log file case, which most of the time is fine with milliseconds, but think of the manufacturing industry, where many sensors trigger or actions can take place within a single millisecond.

This is where you need to be able to store these kind of events at a higher resolution. There was a problem with this though. The existing way of parsing and storing dates simply did not support storing with any resolution higher than milliseconds.

The library Elasticsearch uses is called Joda-Time, and it used to be the de-facto standard for handling time in Java. However, this library is in maintenance mode and will not be extended further, even though it still receives bug fixes and especially time zone database updates. The reason for this is simple: with the release of Java SE 8 onwards, the Date and Time API (JSR 310) is part of JDK.

Source: elastic.co