Kafka Integration¶

Infrastructure Integration¶

Instructions¶

Follow the installation guide below for your given collector environment.

During installation, use the configuration section below as reference.

After installation, the infrastructure datasources in the table below will be available in the AOC.

Installation Guide¶

Installing this integration consists of creating a yaml file in the filesystem of your collectors. Click below for instructions on how to do so for your given collectors environment.

Configuration¶

Ensure that JMX Remote is enabled on Kafka brokers.

Edit kafka.yaml to configure the agent. Please refer kafka.yaml for all available configuration options.

instances:
  - host: localhost
    port: 9999 # This is the JMX port on which Kafka exposes its metrics (usually 9999)
    tags:
      kafka: broker

init_config:
  is_jmx: true

  conf:
    # Producers (v0.9.0.x to v0.10.2.x)
    - include:
        domain: 'kafka.producer'
        bean_regex: 'kafka\.producer:type=producer-metrics,client-id=.*'
        attribute:
          response-rate:
            metric_type: gauge
            alias: kafka.producer.response_rate
    - include:
        domain: 'kafka.producer'
        bean_regex: 'kafka\.producer:type=producer-metrics,client-id=.*'
        attribute:
          request-rate:
            metric_type: gauge

    # Consumers (v0.9.0.x to v0.10.2.x)
    - include:
        domain: 'kafka.consumer'
        bean_regex: 'kafka\.consumer:type=consumer-fetch-manager-metrics,client-id=.*'
        attribute:
          bytes-consumed-rate:
            metric_type: gauge
            alias: kafka.consumer.bytes_in
    - include:
        domain: 'kafka.consumer'
        bean_regex: 'kafka\.consumer:type=consumer-fetch-manager-metrics,client-id=.*'
        attribute:
          records-consumed-rate:
            metric_type: gauge
            alias: kafka.consumer.messages_in

In addition, you can edit kafka_consumer.yaml as well.

Infrastructure Datasources¶

Datasource	Available Aggregations	Unit	Description
kafka.net.bytes_out	`avg` `max` `min` `sum`	byte/second	Outgoing byte rate.
kafka.net.bytes_in	`avg` `max` `min` `sum`	byte/second	Incoming byte rate.
kafka.net.bytes_rejected	`avg` `max` `min` `sum`	byte/second	Rejected byte rate.
kafka.messages_in	`avg` `max` `min` `sum`	message	Incoming message rate.
kafka.request.fetch.failed	`avg` `max` `min` `sum`	request	Number of client fetch request failures.
kafka.request.fetch.failed_per_second	`avg` `max` `min` `sum`	request/second	Rate of client fetch request failures per second.
kafka.request.produce.time.avg	`avg` `max` `min` `sum`	request/second	Average time for a produce request.
kafka.request.produce.time.99percentile	`avg` `max` `min` `sum`	request/second	Time for produce requests for 99th percentile.
kafka.request.produce.failed_per_second	`avg` `max` `min` `sum`	request/second	Rate of failed produce requests per second.
kafka.request.produce.failed	`avg` `max` `min` `sum`	request	Number of failed produce requests.
kafka.request.fetch.time.avg	`avg` `max` `min` `sum`	request/second	Average time per fetch request.
kafka.request.fetch.time.99percentile	`avg` `max` `min` `sum`	request/second	Time for fetch requests for 99th percentile.
kafka.request.update_metadata.time.avg	`avg` `max` `min` `sum`	millisecond	Average time for a request to update metadata.
kafka.request.update_metadata.time.99percentile	`avg` `max` `min` `sum`	millisecond	Time for update metadata requests for 99th percentile.
kafka.request.metadata.time.avg	`avg` `max` `min` `sum`	millisecond	Average time for metadata request.
kafka.request.metadata.time.99percentile	`avg` `max` `min` `sum`	millisecond	Time for metadata requests for 99th percentile.
kafka.request.offsets.time.avg	`avg` `max` `min` `sum`	millisecond	Average time for an offset request.
kafka.request.offsets.time.99percentile	`avg` `max` `min` `sum`	millisecond	Time for offset requests for 99th percentile.
kafka.request.handler.avg.idle.pct	`avg` `max` `min` `sum`	fraction	Average fraction of time the request handler threads are idle.
kafka.replication.isr_shrinks	`avg` `max` `min` `sum`	node/second	Rate of replicas leaving the ISR pool.
kafka.replication.isr_expands	`avg` `max` `min` `sum`	node/second	Rate of replicas joining the ISR pool.
kafka.replication.leader_elections	`avg` `max` `min` `sum`	event/second	Leader election rate.
kafka.replication.unclean_leader_elections	`avg` `max` `min` `sum`	event/second	Unclean leader election rate.
kafka.replication.under_replicated_partitions	`avg` `max` `min` `sum`		Number of unreplicated partitions.
kafka.log.flush_rate	`avg` `max` `min` `sum`	flush/second	Log flush rate.
kafka.consumer.delayed_requests	`avg` `max` `min` `sum`	request	Number of delayed consumer requests.
kafka.consumer.expires_per_second	`avg` `max` `min` `sum`	eviction/second	Rate of delayed consumer request expiration.
kafka.expires_sec	`avg` `max` `min` `sum`	eviction/second	Rate of delayed producer request expiration.
kafka.follower.expires_per_second	`avg` `max` `min` `sum`	eviction/second	Rate of request expiration on followers.
kafka.producer.delayed_requests	`avg` `max` `min` `sum`	request	Number of producer requests delayed.
kafka.producer.expires_per_seconds	`avg` `max` `min` `sum`	eviction/second	Rate of producer request expiration.
kafka.producer.request_rate	`avg` `max` `min` `sum`	request/second	Number of producer requests per second.
kafka.producer.response_rate	`avg` `max` `min` `sum`	response/second	Number of producer responses per second.
kafka.producer.request_latency_avg	`avg` `max` `min` `sum`	millisecond	Producer average request latency.
kafka.producer.bytes_out	`avg` `max` `min` `sum`	byte/second	Producer bytes out rate.
kafka.producer.message_rate	`avg` `max` `min` `sum`	message/second	Producer message rate.
kafka.producer.io_wait	`avg` `max` `min` `sum`	nanosecond	Producer I/O wait time.
kafka.consumer.max_lag	`avg` `max` `min` `sum`	offset	Maximum consumer lag.
kafka.consumer.fetch_rate	`avg` `max` `min` `sum`	request	The minimum rate at which the consumer sends fetch requests to a broker.
kafka.consumer.bytes_in	`avg` `max` `min` `sum`	byte/second	Consumer bytes in rate.
kafka.consumer.messages_in	`avg` `max` `min` `sum`	message/second	Rate of consumer message consumption.
kafka.consumer.zookeeper_commits	`avg` `max` `min` `sum`	write/second	Rate of offset commits to ZooKeeper.
kafka.consumer.kafka_commits	`avg` `max` `min` `sum`	write/second	Rate of offset commits to Kafka.
kafka.broker_offset	`avg` `max` `min` `sum`	offset	Current message offset on broker.
kafka.consumer_lag	`avg` `max` `min` `sum`	offset	Lag in messages between consumer and broker.
kafka.consumer_offset	`avg` `max` `min` `sum`	offset	Current message offset on consumer.