Insertion & Ingestion
You'll hear everywhere that once your documents have been indexed, transforming/adjusting them becomes difficult. This is mostly true but keep in mind that:
Getting to the "right" configuration will inevitably take a few iterations so take your time and don't be afraid to reindex often when you're just starting with your implementation. Later, though, the aim is indeed to minimize production downtime and fully reindex as seldom as possible.
A good amount of time should be spent on the index & doc structure design but more often than not, there'll be situations in production which'll require:
- adding new fields,
- dropping (empty-ish) fields,
- parsing / decoding / converting / combining fields.
In this section we'll discuss ways to:
- make use of pre-processors to handle data inconsistencies and cleansing → Pre-Indexing: Pipelines
- ingest lots of documents simultaneously → Indexing in Bulk
- handle updates and adjustments → Post-Indexing: Updates
If you're looking for ways to simplify your ingestion process, you may find Logstash useful. Don't let the name confuse you — while it was initially designed for log collection, it outgrew its original purpose and is now widely used for collecting and transforming (unstructured) data for later use in Elasticsearch.