Skip to content

Elasticsearch Integration

Infrastructure Integration

Instructions

Follow the installation guide below for your given collector environment.

During installation, use the configuration section below as reference.

After installation, the infrastructure datasources in the table below will be available in the AOC.

Installation Guide

Installing this integration consists of creating a yaml file in the filesystem of your collectors. Click below for instructions on how to do so for your given collectors environment.

Docker

Kubernetes

Mesos-Marathon

Debian

Ubuntu

RHEL/CentOS

SUSE

Configuration

Edit elastic.yaml to configure the agent. Please refer elastic.yaml for all available configuration options.

    init_config:
    instances:
      - url: http://localhost:9200

Infrastructure Datasources

Datasource Available Aggregations Unit Description
elasticsearch.active_primary_shards avg max min sum shard The number of active primary shards in the cluster.
elasticsearch.active_shards avg max min sum shard The number of active shards in the cluster.
elasticsearch.breakers.fielddata.estimated_size_in_bytes avg max min sum byte The estimated size in bytes of the field data circuit breaker.
elasticsearch.breakers.fielddata.overhead avg max min sum The constant multiplier for byte estimations of the field data circuit breaker.
elasticsearch.breakers.fielddata.tripped avg max min sum The number of times the field data circuit breaker has tripped.
elasticsearch.breakers.parent.estimated_size_in_bytes avg max min sum byte The estimated size in bytes of the parent circuit breaker.
elasticsearch.breakers.parent.overhead avg max min sum The constant multiplier for byte estimations of the parent circuit breaker.
elasticsearch.breakers.parent.tripped avg max min sum The number of times the parent circuit breaker has tripped.
elasticsearch.breakers.request.estimated_size_in_bytes avg max min sum byte The estimated size in bytes of the request circuit breaker.
elasticsearch.breakers.request.overhead avg max min sum The constant multiplier for byte estimations of the request circuit breaker.
elasticsearch.breakers.request.tripped avg max min sum The number of times the request circuit breaker has tripped.
elasticsearch.cache.field.evictions avg max min sum eviction The total number of evictions from the field data cache.
elasticsearch.cache.field.size avg max min sum byte The size of the field cache.
elasticsearch.cache.filter.count avg max min sum item The number of items in the filter cache.
elasticsearch.cache.filter.evictions avg max min sum eviction The total number of evictions from the filter cache.
elasticsearch.cache.filter.size avg max min sum byte The size of the filter cache.
elasticsearch.cluster_status avg max min sum The elasticsearch cluster health as a number: red = 0, yellow = 1, green = 2
elasticsearch.docs.count avg max min sum document The total number of documents in the cluster across all shards.
elasticsearch.docs.deleted avg max min sum document The total number of documents deleted from the cluster across all shards.
elasticsearch.fielddata.evictions avg max min sum eviction The total number of evictions from the fielddata cache.
elasticsearch.fielddata.size avg max min sum byte The size of the fielddata cache.
elasticsearch.flush.total avg max min sum flush The total number of index flushes to disk since start.
elasticsearch.flush.total.time avg max min sum second The total time spent flushing the index to disk.
elasticsearch.fs.total.available_in_bytes avg max min sum byte The total number of bytes available to this Java virtual machine on this file store.
elasticsearch.fs.total.disk_io_op avg max min sum operation The total I/O operations on the file store.
elasticsearch.fs.total.disk_io_size_in_bytes avg max min sum byte Total bytes used for all I/O operations on the file store.
elasticsearch.fs.total.disk_read_size_in_bytes avg max min sum byte The total bytes read from the file store.
elasticsearch.fs.total.disk_reads avg max min sum read The total number of reads from the file store.
elasticsearch.fs.total.disk_write_size_in_bytes avg max min sum byte The total bytes written to the file store.
elasticsearch.fs.total.disk_writes avg max min sum write The total number of writes to the file store.
elasticsearch.fs.total.free_in_bytes avg max min sum byte The total number of unallocated bytes in the file store.
elasticsearch.fs.total.total_in_bytes avg max min sum byte The total size in bytes of the file store.
elasticsearch.get.current avg max min sum request The number of get requests currently running.
elasticsearch.get.exists.time avg max min sum second The total time spent on get requests where the document existed.
elasticsearch.get.exists.total avg max min sum request The total number of get requests where the document existed.
elasticsearch.get.missing.time avg max min sum second The total time spent on get requests where the document was missing.
elasticsearch.get.missing.total avg max min sum request The total number of get requests where the document was missing.
elasticsearch.get.time avg max min sum second The total time spent on get requests.
elasticsearch.get.total avg max min sum request The total number of get requests.
elasticsearch.http.current_open avg max min sum connection The number of current open HTTP connections.
elasticsearch.http.total_opened avg max min sum connection The total number of opened HTTP connections.
elasticsearch.id_cache.size avg max min sum byte The size of the id cache
elasticsearch.indexing.delete.current avg max min sum document The number of documents currently being deleted from an index.
elasticsearch.indexing.delete.time avg max min sum second The total time spent deleting documents from an index.
elasticsearch.indexing.delete.total avg max min sum document The total number of documents deleted from an index.
elasticsearch.indexing.index.current avg max min sum document The number of documents currently being indexed to an index.
elasticsearch.indexing.index.time avg max min sum second The total time spent indexing documents to an index.
elasticsearch.indexing.index.total avg max min sum document The total number of documents indexed to an index.
elasticsearch.indices.indexing.index_failed avg max min sum The number of failed indexing operations.
elasticsearch.indices.indexing.throttle_time avg max min sum millisecond The total time indexing waited due to throttling.
elasticsearch.indices.query_cache.evictions avg max min sum eviction The number of query cache evictions.
elasticsearch.indices.query_cache.hit_count avg max min sum hit The number of query cache hits.
elasticsearch.indices.query_cache.memory_size_in_bytes avg max min sum byte The memory used by the query cache.
elasticsearch.indices.query_cache.miss_count avg max min sum miss The number of query cache misses.
elasticsearch.indices.recovery.current_as_source avg max min sum The number of ongoing recoveries for which a shard serves as a source.
elasticsearch.indices.recovery.current_as_target avg max min sum The number of ongoing recoveries for which a shard serves as a target.
elasticsearch.indices.recovery.throttle_time avg max min sum millisecond The total time recoveries waited due to throttling.
elasticsearch.indices.request_cache.evictions avg max min sum eviction The number of request cache evictions.
elasticsearch.indices.request_cache.hit_count avg max min sum hit The number of request cache hits.
elasticsearch.indices.request_cache.memory_size_in_bytes avg max min sum byte The memory used by the request cache.
elasticsearch.indices.request_cache.miss_count avg max min sum miss The number of request cache misses.
elasticsearch.indices.segments.count avg max min sum segment The number of segments in an index shard.
elasticsearch.indices.segments.doc_values_memory_in_bytes avg max min sum byte The memory used by doc values.
elasticsearch.indices.segments.fixed_bit_set_memory_in_bytes avg max min sum byte The memory used by fixed bit set.
elasticsearch.indices.segments.index_writer_max_memory_in_bytes avg max min sum byte The maximum memory used by the index writer.
elasticsearch.indices.segments.index_writer_memory_in_bytes avg max min sum byte The memory used by the index writer.
elasticsearch.indices.segments.memory_in_bytes avg max min sum byte The memory used by index segments.
elasticsearch.indices.segments.norms_memory_in_bytes avg max min sum byte The memory used by norms.
elasticsearch.indices.segments.stored_fields_memory_in_bytes avg max min sum byte The memory used by stored fields.
elasticsearch.indices.segments.term_vectors_memory_in_bytes avg max min sum byte The memory used by term vectors.
elasticsearch.indices.segments.terms_memory_in_bytes avg max min sum byte The memory used by terms.
elasticsearch.indices.segments.version_map_memory_in_bytes avg max min sum byte The memory used by the segment version map.
elasticsearch.indices.translog.operations avg max min sum operation The number of operations in the transaction log.
elasticsearch.indices.translog.size_in_bytes avg max min sum byte The size of the transaction log.
elasticsearch.initializing_shards avg max min sum shard The number of shards that are currently initializing.
elasticsearch.merges.current avg max min sum merge The number of currently active segment merges.
elasticsearch.merges.current.docs avg max min sum document The number of documents across segments currently being merged.
elasticsearch.merges.current.size avg max min sum byte The size of the segments currently being merged.
elasticsearch.merges.total avg max min sum merge The total number of segment merges.
elasticsearch.merges.total.docs avg max min sum document The total number of documents across all merged segments.
elasticsearch.merges.total.size avg max min sum byte The total size of all merged segments.
elasticsearch.merges.total.time avg max min sum second The total time spent on segment merging.
elasticsearch.number_of_data_nodes avg max min sum node The number of data nodes in the cluster.
elasticsearch.number_of_nodes avg max min sum node The total number of nodes in the cluster.
elasticsearch.pending_tasks_priority_high avg max min sum task The number of high priority pending tasks.
elasticsearch.pending_tasks_priority_urgent avg max min sum task The number of urgent priority pending tasks.
elasticsearch.pending_tasks_total avg max min sum task The total number of pending tasks.
elasticsearch.primaries.docs.count avg max min sum document The total number of documents in the primary shards.
elasticsearch.primaries.docs.deleted avg max min sum document The total number of documents deleted from the primary shards.
elasticsearch.primaries.flush.total avg max min sum flush The total number of index flushes to disk from the primary shards since start.
elasticsearch.primaries.flush.total.time avg max min sum second The total time spent flushing the index to disk from the primary shards.
elasticsearch.primaries.get.current avg max min sum request The number of get requests currently running on the primary shards.
elasticsearch.primaries.get.exists.time avg max min sum request The total time spent on get requests from the primary shards where the document existed.
elasticsearch.primaries.get.exists.total avg max min sum request The total number of get requests on primary shards where the document existed.
elasticsearch.primaries.get.missing.time avg max min sum second The total time spent on get requests from the primary shards where the document was missing.
elasticsearch.primaries.get.missing.total avg max min sum request The total number of get requests from the primary shards where the document was missing.
elasticsearch.primaries.get.time avg max min sum second The total time spent on get requests from the primary shards.
elasticsearch.primaries.get.total avg max min sum request The total number of get requests from the primary shards.
elasticsearch.primaries.indexing.delete.current avg max min sum document The number of documents currently being deleted from an index on the primary shards.
elasticsearch.primaries.indexing.delete.time avg max min sum second The total time spent deleting documents from an index on the primary shards.
elasticsearch.primaries.indexing.delete.total avg max min sum document The total number of documents deleted from an index on the primary shards.
elasticsearch.primaries.indexing.index.current avg max min sum document The number of documents currently being indexed to an index on the primary shards.
elasticsearch.primaries.indexing.index.time avg max min sum second The total time spent indexing documents to an index on the primary shards.
elasticsearch.primaries.indexing.index.total avg max min sum document The total number of documents indexed to an index on the primary shards.
elasticsearch.primaries.merges.current avg max min sum merge The number of currently active segment merges on the primary shards.
elasticsearch.primaries.merges.current.docs avg max min sum document The number of documents across segments currently being merged on the primary shards.
elasticsearch.primaries.merges.current.size avg max min sum byte The size of the segments currently being merged on the primary shards.
elasticsearch.primaries.merges.total avg max min sum merge The total number of segment merges on the primary shards.
elasticsearch.primaries.merges.total.docs avg max min sum document The total number of documents across all merged segments on the primary shards.
elasticsearch.primaries.merges.total.size avg max min sum byte The total size of all merged segments on the primary shards.
elasticsearch.primaries.merges.total.time avg max min sum second The total time spent on segment merging on the primary shards.
elasticsearch.primaries.refresh.total avg max min sum refresh The total number of index refreshes on the primary shards.
elasticsearch.primaries.refresh.total.time avg max min sum second The total time spent on index refreshes on the primary shards.
elasticsearch.primaries.search.fetch.current avg max min sum fetch The number of query fetches currently running on the primary shards.
elasticsearch.primaries.search.fetch.time avg max min sum second The total time spent on query fetches on the primary shards.
elasticsearch.primaries.search.fetch.total avg max min sum fetch The total number of query fetches on the primary shards.
elasticsearch.primaries.search.query.current avg max min sum query The number of currently active queries on the primary shards.
elasticsearch.primaries.search.query.time avg max min sum second The total time spent querying on the primary shards.
elasticsearch.primaries.search.query.total avg max min sum query The total number of queries to the primary shards.
elasticsearch.primaries.store.size avg max min sum byte The total size of all the primary shards.
elasticsearch.process.open_fd avg max min sum file The number of opened file descriptors associated with the current process, or -1 if not supported.
elasticsearch.refresh.total avg max min sum refresh The total number of index refreshes.
elasticsearch.refresh.total.time avg max min sum second The total time spent on index refreshes.
elasticsearch.relocating_shards avg max min sum shard The number of shards that are reloacting from one node to another.
elasticsearch.search.fetch.current avg max min sum fetch The number of search fetches currently running.
elasticsearch.search.fetch.open_contexts avg max min sum query The number of active searches.
elasticsearch.search.fetch.time avg max min sum second The total time spent on the search fetch.
elasticsearch.search.fetch.total avg max min sum fetch The total number of search fetches.
elasticsearch.search.query.current avg max min sum query The number of currently active queries.
elasticsearch.search.query.time avg max min sum second The total time spent on queries.
elasticsearch.search.query.total avg max min sum query The total number of queries.
elasticsearch.store.size avg max min sum byte The total size in bytes of the store.
elasticsearch.thread_pool.bulk.active avg max min sum thread The number of active threads in the bulk pool.
elasticsearch.thread_pool.bulk.queue avg max min sum thread The number of queued threads in the bulk pool.
elasticsearch.thread_pool.bulk.threads avg max min sum thread The total number of threads in the bulk pool.
elasticsearch.thread_pool.bulk.rejected avg max min sum thread The number of rejected threads in the bulk pool.
elasticsearch.thread_pool.fetch_shard_started.active avg max min sum thread The number of active threads in the fetch shard started pool.
elasticsearch.thread_pool.fetch_shard_started.threads avg max min sum thread The total number of threads in the fetch shard started pool.
elasticsearch.thread_pool.fetch_shard_started.queue avg max min sum thread The number of queued threads in the fetch shard started pool.
elasticsearch.thread_pool.fetch_shard_started.rejected avg max min sum thread The number of rejected threads in the fetch shard started pool.
elasticsearch.thread_pool.fetch_shard_store.active avg max min sum thread The number of active threads in the fetch shard store pool.
elasticsearch.thread_pool.fetch_shard_store.threads avg max min sum thread The total number of threads in the fetch shard store pool.
elasticsearch.thread_pool.fetch_shard_store.queue avg max min sum thread The number of queued threads in the fetch shard store pool.
elasticsearch.thread_pool.fetch_shard_store.rejected avg max min sum thread The number of rejected threads in the fetch shard store pool.
elasticsearch.thread_pool.flush.active avg max min sum thread The number of active threads in the flush queue.
elasticsearch.thread_pool.flush.queue avg max min sum thread The number of queued threads in the flush pool.
elasticsearch.thread_pool.flush.threads avg max min sum thread The total number of threads in the flush pool.
elasticsearch.thread_pool.flush.rejected avg max min sum thread The number of rejected threads in the flush pool.
elasticsearch.thread_pool.force_merge.active avg max min sum thread The number of active threads for force merge operations.
elasticsearch.thread_pool.force_merge.threads avg max min sum thread The total number of threads for force merge operations.
elasticsearch.thread_pool.force_merge.queue avg max min sum thread The number of queued threads for force merge operations.
elasticsearch.thread_pool.force_merge.rejected avg max min sum thread The number of rejected threads for force merge operations.
elasticsearch.thread_pool.generic.active avg max min sum thread The number of active threads in the generic pool.
elasticsearch.thread_pool.generic.queue avg max min sum thread The number of queued threads in the generic pool.
elasticsearch.thread_pool.generic.threads avg max min sum thread The total number of threads in the generic pool.
elasticsearch.thread_pool.generic.rejected avg max min sum thread The number of rejected threads in the generic pool.
elasticsearch.thread_pool.get.active avg max min sum thread The number of active threads in the get pool.
elasticsearch.thread_pool.get.queue avg max min sum thread The number of queued threads in the get pool.
elasticsearch.thread_pool.get.threads avg max min sum thread The total number of threads in the get pool.
elasticsearch.thread_pool.get.rejected avg max min sum thread The number of rejected threads in the get pool.
elasticsearch.thread_pool.index.active avg max min sum thread The number of active threads in the index pool.
elasticsearch.thread_pool.index.queue avg max min sum thread The number of queued threads in the index pool.
elasticsearch.thread_pool.index.threads avg max min sum thread The total number of threads in the index pool.
elasticsearch.thread_pool.index.rejected avg max min sum thread The number of rejected threads in the index pool.
elasticsearch.thread_pool.listener.active avg max min sum thread The number of active threads in the listener pool.
elasticsearch.thread_pool.listener.queue avg max min sum thread The number of queued threads in the listener pool.
elasticsearch.thread_pool.listener.threads avg max min sum thread The total number of threads in the listener pool.
elasticsearch.thread_pool.listener.rejected avg max min sum thread The number of rejected threads in the listener pool.
elasticsearch.thread_pool.management.active avg max min sum thread The number of active threads in the management pool.
elasticsearch.thread_pool.management.queue avg max min sum thread The number of queued threads in the management pool.
elasticsearch.thread_pool.management.threads avg max min sum thread The total number of threads in the management pool.
elasticsearch.thread_pool.management.rejected avg max min sum thread The number of rejected threads in the management pool.
elasticsearch.thread_pool.merge.active avg max min sum thread The number of active threads in the merge pool.
elasticsearch.thread_pool.merge.queue avg max min sum thread The number of queued threads in the merge pool.
elasticsearch.thread_pool.merge.threads avg max min sum thread The total number of threads in the merge pool.
elasticsearch.thread_pool.merge.rejected avg max min sum thread The number of rejected threads in the merge pool.
elasticsearch.thread_pool.percolate.active avg max min sum thread The number of active threads in the percolate pool.
elasticsearch.thread_pool.percolate.queue avg max min sum thread The number of queued threads in the percolate pool.
elasticsearch.thread_pool.percolate.threads avg max min sum thread The total number of threads in the percolate pool.
elasticsearch.thread_pool.percolate.rejected avg max min sum thread The number of rejected threads in the percolate pool.
elasticsearch.thread_pool.refresh.active avg max min sum thread The number of active threads in the refresh pool.
elasticsearch.thread_pool.refresh.queue avg max min sum thread The number of queued threads in the refresh pool.
elasticsearch.thread_pool.refresh.threads avg max min sum thread The total number of threads in the refresh pool.
elasticsearch.thread_pool.refresh.rejected avg max min sum thread The number of rejected threads in the refresh pool.
elasticsearch.thread_pool.search.active avg max min sum thread The number of active threads in the search pool.
elasticsearch.thread_pool.search.queue avg max min sum thread The number of queued threads in the search pool.
elasticsearch.thread_pool.search.threads avg max min sum thread The total number of threads in the search pool.
elasticsearch.thread_pool.search.rejected avg max min sum thread The number of rejected threads in the search pool.
elasticsearch.thread_pool.snapshot.active avg max min sum thread The number of active threads in the snapshot pool.
elasticsearch.thread_pool.snapshot.queue avg max min sum thread The number of queued threads in the snapshot pool.
elasticsearch.thread_pool.snapshot.threads avg max min sum thread The total number of threads in the snapshot pool.
elasticsearch.thread_pool.snapshot.rejected avg max min sum thread The number of rejected threads in the snapshot pool.
elasticsearch.transport.rx_count avg max min sum packet The total number of packets received in cluster communication.
elasticsearch.transport.rx_size avg max min sum byte The total size of data received in cluster communication.
elasticsearch.transport.server_open avg max min sum connection The number of connections opened for cluster communication.
elasticsearch.transport.tx_count avg max min sum packet The total number of packets sent in cluster communication.
elasticsearch.transport.tx_size avg max min sum byte The total size of data sent in cluster communication.
elasticsearch.unassigned_shards avg max min sum shard The number of shards that are unassigned to a node.
jvm.gc.collection_count avg max min sum garbage collection The total number of garbage collections run by the JVM.
jvm.gc.collection_time avg max min sum second The total time spent on garbage collection in the JVM.
jvm.gc.collectors.old.collection_time avg max min sum second The total time spent in major GCs in the JVM that collect old generation objects.
jvm.gc.collectors.old.count avg max min sum garbage collection The total count of major GCs in the JVM that collect old generation objects.
jvm.gc.collectors.young.collection_time avg max min sum second The total time spent in minor GCs in the JVM that collects young generation objects.
jvm.gc.collectors.young.count avg max min sum garbage collection The total count of minor GCs in the JVM that collects young generation objects.
jvm.gc.concurrent_mark_sweep.collection_time avg max min sum second The total time spent on "concurrent mark & sweep" GCs in the JVM.
jvm.gc.concurrent_mark_sweep.count avg max min sum garbage collection The total count of "concurrent mark & sweep" GCs in the JVM.
jvm.gc.par_new.collection_time avg max min sum second The total time spent on "parallel new" GCs in the JVM.
jvm.gc.par_new.count avg max min sum garbage collection The total count of "parallel new" GCs in the JVM.
jvm.mem.heap_committed avg max min sum byte The amount of memory guaranteed to be available to the JVM heap.
jvm.mem.heap_in_use avg max min sum The amount of memory currently used by the JVM heap as a value between 0 and 1.
jvm.mem.heap_max avg max min sum byte The maximum amount of memory that can be used by the JVM heap.
jvm.mem.heap_used avg max min sum byte The amount of memory in bytes currently used by the JVM heap.
jvm.mem.non_heap_committed avg max min sum byte The amount of memory guaranteed to be available to JVM non-heap.
jvm.mem.non_heap_used avg max min sum byte The amount of memory in bytes currently used by the JVM non-heap.
jvm.threads.count avg max min sum thread The number of active threads in the JVM.
jvm.threads.peak_count avg max min sum thread The peak number of threads used by the JVM.