payload: {"action":"created","issue":{"url":"https://api.github.com/repos/influxdata/telegraf/issues/5421","repository_url":"https://api.github.com/repos/influxdata/telegraf","labels_url":"https://api.github.com/repos/influxdata/telegraf/issues/5421/labels{/name}","comments_url":"https://api.github.com/repos/influxdata/telegraf/issues/5421/comments","events_url":"https://api.github.com/repos/influxdata/telegraf/issues/5421/events","html_url":"https://github.com/influxdata/telegraf/issues/5421","id":409557933,"node_id":"MDU6SXNzdWU0MDk1NTc5MzM=","number":5421,"title":"vSphere Input Plugin intermittent silent collection gaps for some hosts & virtual machines","user":{"login":"bdeam","id":40452846,"node_id":"MDQ6VXNlcjQwNDUyODQ2","avatar_url":"https://avatars2.githubusercontent.com/u/40452846?v=4","gravatar_id":"","url":"https://api.github.com/users/bdeam","html_url":"https://github.com/bdeam","followers_url":"https://api.github.com/users/bdeam/followers","following_url":"https://api.github.com/users/bdeam/following{/other_user}","gists_url":"https://api.github.com/users/bdeam/gists{/gist_id}","starred_url":"https://api.github.com/users/bdeam/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bdeam/subscriptions","organizations_url":"https://api.github.com/users/bdeam/orgs","repos_url":"https://api.github.com/users/bdeam/repos","events_url":"https://api.github.com/users/bdeam/events{/privacy}","received_events_url":"https://api.github.com/users/bdeam/received_events","type":"User","site_admin":false},"labels":[{"id":1078341799,"node_id":"MDU6TGFiZWwxMDc4MzQxNzk5","url":"https://api.github.com/repos/influxdata/telegraf/labels/area/vsphere","name":"area/vsphere","color":"f48d38","default":false},{"id":194687826,"node_id":"MDU6TGFiZWwxOTQ2ODc4MjY=","url":"https://api.github.com/repos/influxdata/telegraf/labels/bug","name":"bug","color":"ff8564","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":27,"created_at":"2019-02-12T23:58:08Z","updated_at":"2019-04-15T13:32:22Z","closed_at":null,"author_association":"NONE","body":"### Relevant telegraf.conf:\r\n\r\n### System info:\r\n\r\n[Include Telegraf version, operating system name, and other relevant details]\r\nTelegraf 1.9.0-1 (RPM package installed)\r\nRHEL 7.5 Maipo (32GB RAM, 8x 2.2GHz CPU)\r\n### Steps to reproduce:\r\n\r\n1. ... Enable telegraf\r\n2. ... 'journalctl -f -t telegraf' shows no instances of '[agent] input \"inputs.vsphere\" did not complete within its interval'\r\n3. ... Missing/Gaps in metrics for some ESX Hosts and Virtual Machines\r\n\r\n### Expected behavior:\r\n\r\nCollection of all listed metrics for Hosts & VMs is successful, and where not a failure message is logged\r\n\r\n### Actual behavior:\r\n\r\nIntermittent gaps in metrics for some hosts & VMs is observed as below:\r\n![image](https://user-images.githubusercontent.com/40452846/52676487-37313c80-2f7a-11e9-8d81-6aa102c65e97.png)\r\n\r\n\r\n\r\n### Additional info:\r\n\r\nI have tried increasing & decreasing the go routines, increased intervals to 2mins+ with no success. I have left the collector to run for over 12 hours with no change in behavior. \r\n[Include gist of relevant config, logs, etc.]\r\n# Configuration for telegraf agent\r\n```\r\n[agent]\r\n  ## Default data collection interval for all inputs\r\n  interval = \"10s\"\r\n  ## Rounds collection interval to 'interval'\r\n  ## ie, if interval=\"10s\" then always collect on :00, :10, :20, etc.\r\n  round_interval = true\r\n\r\n  ## Telegraf will send metrics to outputs in batches of at most\r\n  ## metric_batch_size metrics.\r\n  ## This controls the size of writes that Telegraf sends to output plugins.\r\n  metric_batch_size = 1000\r\n\r\n  ## For failed writes, telegraf will cache metric_buffer_limit metrics for each\r\n  ## output, and will flush this buffer on a successful write. Oldest metrics\r\n  ## are dropped first when this buffer fills.\r\n  ## This buffer only fills when writes fail to output plugin(s).\r\n  metric_buffer_limit = 10000\r\n\r\n  ## Collection jitter is used to jitter the collection by a random amount.\r\n  ## Each plugin will sleep for a random time within jitter before collecting.\r\n  ## This can be used to avoid many plugins querying things like sysfs at the\r\n  ## same time, which can have a measurable effect on the system.\r\n  collection_jitter = \"0s\"\r\n\r\n  ## Default flushing interval for all outputs. Maximum flush_interval will be\r\n  ## flush_interval + flush_jitter\r\n  flush_interval = \"10s\"\r\n  ## Jitter the flush interval by a random amount. This is primarily to avoid\r\n  ## large write spikes for users running a large number of telegraf instances.\r\n  ## ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s\r\n  flush_jitter = \"0s\"\r\n\r\n  ## By default or when set to \"0s\", precision will be set to the same\r\n  ## timestamp order as the collection interval, with the maximum being 1s.\r\n  ##   ie, when interval = \"10s\", precision will be \"1s\"\r\n  ##       when interval = \"250ms\", precision will be \"1ms\"\r\n  ## Precision will NOT be used for service inputs. It is up to each individual\r\n  ## service input to set the timestamp at the appropriate precision.\r\n  ## Valid time units are \"ns\", \"us\" (or \"µs\"), \"ms\", \"s\".\r\n  precision = \"\"\r\n\r\n  ## Logging configuration:\r\n  ## Run telegraf with debug log messages.\r\n  debug = true\r\n  ## Run telegraf in quiet mode (error log messages only).\r\n  quiet = false\r\n  ## Specify the log file name. The empty string means to log to stderr.\r\n  logfile = \"\"\r\n\r\n  ## Override default hostname, if empty use os.Hostname()\r\n  hostname = \"\"\r\n  ## If set to true, do no set the \"host\" tag in the telegraf agent.\r\n  omit_hostname = false\r\n\r\n# # Read metrics from VMware vCenter\r\n [[inputs.vsphere]]\r\n   ## List of vCenter URLs to be monitored. These three lines must be uncommented\r\n   ## and edited for the plugin to work.\r\n   vcenters = [\"https://london22s304.generic.com/sdk\", \"https://test04.generic.com/sdk\", \"https://test06.generic.com/sdk\", \"https://test101.generic.com/sdk\" ]\r\n   username = \"acc@generic.com\"\r\n   password = \"password\"\r\n   interval = \"60s\"\r\n\r\n   ## VMs\r\n   ## Typical VM metrics (if omitted or empty, all metrics are collected)\r\n   vm_metric_include = [\r\n     \"cpu.demand.average\",\r\n     \"cpu.idle.summation\",\r\n     \"cpu.latency.average\",\r\n     \"cpu.readiness.average\",\r\n     \"cpu.ready.summation\",\r\n     \"cpu.run.summation\",\r\n     \"cpu.usagemhz.average\",\r\n     \"cpu.usage.average\",\r\n     \"cpu.used.summation\",\r\n     \"cpu.wait.summation\",\r\n     \"mem.active.average\",\r\n     \"mem.granted.average\",\r\n     \"mem.latency.average\",\r\n     \"mem.swapin.average\",\r\n     \"mem.swapinRate.average\",\r\n     \"mem.swapout.average\",\r\n     \"mem.swapoutRate.average\",\r\n     \"mem.usage.average\",\r\n     \"mem.vmmemctl.average\",\r\n     \"net.bytesRx.average\",\r\n     \"net.bytesTx.average\",\r\n     \"net.droppedRx.summation\",\r\n     \"net.droppedTx.summation\",\r\n     \"net.usage.average\",\r\n     \"power.power.average\",\r\n     \"virtualDisk.numberReadAveraged.average\",\r\n     \"virtualDisk.numberWriteAveraged.average\",\r\n     \"virtualDisk.read.average\",\r\n     \"virtualDisk.readOIO.latest\",\r\n     \"virtualDisk.throughput.usage.average\",\r\n     \"virtualDisk.totalReadLatency.average\",\r\n     \"virtualDisk.totalWriteLatency.average\",\r\n     \"virtualDisk.write.average\",\r\n     \"virtualDisk.writeOIO.latest\",\r\n     \"sys.uptime.latest\",\r\n   ]\r\n  # vm_metric_exclude = [] ## Nothing is excluded by default\r\n  # vm_instances = true ## true by default\r\n\r\n   ## Hosts\r\n   ## Typical host metrics (if omitted or empty, all metrics are collected)\r\n      host_metric_include = [\r\n     \"cpu.coreUtilization.average\",\r\n     \"cpu.costop.summation\",\r\n     \"cpu.demand.average\",\r\n     \"cpu.idle.summation\",\r\n     \"cpu.latency.average\",\r\n     \"cpu.readiness.average\",\r\n     \"cpu.ready.summation\",\r\n     \"cpu.swapwait.summation\",\r\n     \"cpu.usage.average\",\r\n     \"cpu.usagemhz.average\",\r\n     \"cpu.used.summation\",\r\n     \"cpu.utilization.average\",\r\n     \"cpu.wait.summation\",\r\n     \"disk.deviceReadLatency.average\",\r\n     \"disk.deviceWriteLatency.average\",\r\n     \"disk.kernelReadLatency.average\",\r\n     \"disk.kernelWriteLatency.average\",\r\n     \"disk.numberReadAveraged.average\",\r\n     \"disk.numberWriteAveraged.average\",\r\n     \"disk.read.average\",\r\n     \"disk.totalReadLatency.average\",\r\n     \"disk.totalWriteLatency.average\",\r\n     \"disk.write.average\",\r\n     \"mem.active.average\",\r\n     \"mem.latency.average\",\r\n     \"mem.state.latest\",\r\n     \"mem.swapin.average\",\r\n     \"mem.swapinRate.average\",\r\n     \"mem.swapout.average\",\r\n     \"mem.swapoutRate.average\",\r\n     \"mem.totalCapacity.average\",\r\n     \"mem.usage.average\",\r\n     \"mem.vmmemctl.average\",\r\n     \"net.bytesRx.average\",\r\n     \"net.bytesTx.average\",\r\n     \"net.droppedRx.summation\",\r\n     \"net.droppedTx.summation\",\r\n     \"net.errorsRx.summation\",\r\n     \"net.errorsTx.summation\",\r\n     \"net.usage.average\",\r\n     \"power.power.average\",\r\n     \"storageAdapter.numberReadAveraged.average\",\r\n     \"storageAdapter.numberWriteAveraged.average\",\r\n     \"storageAdapter.read.average\",\r\n     \"storageAdapter.write.average\",\r\n     \"sys.uptime.latest\",\r\n   ]\r\n  # host_metric_exclude = [] ## Nothing excluded by default\r\n  # host_instances = true ## true by default\r\n   ## Clusters\r\n   # cluster_metric_include = [] ## if omitted or empty, all metrics are collected\r\n   cluster_metric_exclude = [\"*\"] ## Nothing excluded by default\r\n   # cluster_instances = true ## true by default\r\n\r\n   ## Datastores\r\n   # datastore_metric_include = [] ## if omitted or empty, all metrics are collected\r\n   datastore_metric_exclude = [\"*\"] ## Nothing excluded by default\r\n   # datastore_instances = false ## false by default for Datastores only\r\n\r\n   ## Datacenters\r\n   # datacenter_metric_include = [] ## if omitted or empty, all metrics are collected\r\n   datacenter_metric_exclude = [ \"*\" ] ## Datacenters are not collected by default.\r\n   # datacenter_instances = false ## false by default for Datastores only\r\n\r\n   ## Plugin Settings\r\n   ## separator character to use for measurement and field names (default: \"_\") \r\n     # separator = \"_\"\r\n\r\n   ## number of objects to retreive per query for realtime resources (vms and hosts)\r\n   ## set to 64 for vCenter 5.5 and 6.0 (default: 256)\r\n   max_query_objects = 64\r\n\r\n   ## number of metrics to retreive per query for non-realtime resources (clusters and datastores)\r\n   ## set to 64 for vCenter 5.5 and 6.0 (default: 256)\r\n   max_query_metrics = 64\r\n\r\n   ## number of go routines to use for collection and discovery of objects and metrics\r\n   collect_concurrency = 5\r\n   discover_concurrency = 5\r\n\r\n   ## whether or not to force discovery of new objects on initial gather call before collecting metrics\r\n   ## when true for large environments this may cause errors for time elapsed while collecting metrics\r\n   ## when false (default) the first collection cycle may result in no or limited metrics while objects are discovered\r\n   #   force_discover_on_init = false\r\n\r\n   ## the interval before (re)discovering objects subject to metrics collection (default: 300s)\r\n   # object_discovery_interval = \"300s\"\r\n\r\n   ## timeout applies to any of the api request made to vcenter\r\n   timeout = \"60s\"\r\n\r\n   ## Optional SSL Config\r\n   # ssl_ca = \"/path/to/cafile\"\r\n   # ssl_cert = \"/path/to/certfile\"\r\n   # ssl_key = \"/path/to/keyfile\"\r\n   ## Use SSL but skip chain & host verification\r\n   insecure_skip_verify = true\r\n\r\n   \r\n  ## This is is a Windows Server vCenter, not an appliance.\r\n [[inputs.vsphere]]\r\n   ## List of vCenter URLs to be monitored. These three lines must be uncommented\r\n   ## and edited for the plugin to work.\r\n   vcenters = [\"https://london22s302.generic.com/sdk\" ]\r\n   username = \"acc@generic.com\"\r\n   password = \"password\"\r\n   interval = \"60s\"\r\n\r\n   ## VMs\r\n   ## Typical VM metrics (if omitted or empty, all metrics are collected)\r\n   vm_metric_include = [\r\n     \"cpu.demand.average\",\r\n     \"cpu.idle.summation\",\r\n     \"cpu.latency.average\",\r\n     \"cpu.readiness.average\",\r\n     \"cpu.ready.summation\",\r\n     \"cpu.run.summation\",\r\n     \"cpu.usagemhz.average\",\r\n     \"cpu.usage.average\",\r\n     \"cpu.used.summation\",\r\n     \"cpu.wait.summation\",\r\n     \"mem.active.average\",\r\n     \"mem.granted.average\",\r\n     \"mem.latency.average\",\r\n     \"mem.swapin.average\",\r\n     \"mem.swapinRate.average\",\r\n     \"mem.swapout.average\",\r\n     \"mem.swapoutRate.average\",\r\n     \"mem.usage.average\",\r\n     \"mem.vmmemctl.average\",\r\n     \"net.bytesRx.average\",\r\n     \"net.bytesTx.average\",\r\n     \"net.droppedRx.summation\",\r\n     \"net.droppedTx.summation\",\r\n     \"net.usage.average\",\r\n     \"power.power.average\",\r\n     \"virtualDisk.numberReadAveraged.average\",\r\n     \"virtualDisk.numberWriteAveraged.average\",\r\n     \"virtualDisk.read.average\",\r\n     \"virtualDisk.readOIO.latest\",\r\n     \"virtualDisk.throughput.usage.average\",\r\n     \"virtualDisk.totalReadLatency.average\",\r\n     \"virtualDisk.totalWriteLatency.average\",\r\n     \"virtualDisk.write.average\",\r\n     \"virtualDisk.writeOIO.latest\",\r\n     \"sys.uptime.latest\",\r\n   ]\r\n  # vm_metric_exclude = [] ## Nothing is excluded by default\r\n  # vm_instances = true ## true by default\r\n\r\n     ## Hosts\r\n   ## Typical host metrics (if omitted or empty, all metrics are collected)\r\n   host_metric_include = [\r\n     \"cpu.coreUtilization.average\",\r\n     \"cpu.costop.summation\",\r\n     \"cpu.demand.average\",\r\n     \"cpu.idle.summation\",\r\n     \"cpu.latency.average\",\r\n     \"cpu.readiness.average\",\r\n     \"cpu.ready.summation\",\r\n     \"cpu.swapwait.summation\",\r\n     \"cpu.usage.average\",\r\n     \"cpu.usagemhz.average\",\r\n     \"cpu.used.summation\",\r\n     \"cpu.utilization.average\",\r\n     \"cpu.wait.summation\",\r\n     \"disk.deviceReadLatency.average\",\r\n     \"disk.deviceWriteLatency.average\",\r\n     \"disk.kernelReadLatency.average\",\r\n     \"disk.kernelWriteLatency.average\",\r\n     \"disk.numberReadAveraged.average\",\r\n     \"disk.numberWriteAveraged.average\",\r\n     \"disk.read.average\",\r\n     \"disk.totalReadLatency.average\",\r\n     \"disk.totalWriteLatency.average\",\r\n     \"disk.write.average\",\r\n     \"mem.active.average\",\r\n     \"mem.latency.average\",\r\n     \"mem.state.latest\",\r\n     \"mem.swapin.average\",\r\n     \"mem.swapinRate.average\",\r\n     \"mem.swapout.average\",\r\n     \"mem.swapoutRate.average\",\r\n     \"mem.totalCapacity.average\",\r\n     \"mem.usage.average\",\r\n     \"mem.vmmemctl.average\",\r\n     \"net.bytesRx.average\",\r\n     \"net.bytesTx.average\",\r\n     \"net.droppedRx.summation\",\r\n     \"net.droppedTx.summation\",\r\n     \"net.errorsRx.summation\",\r\n     \"net.errorsTx.summation\",\r\n     \"net.usage.average\",\r\n     \"power.power.average\",\r\n     \"storageAdapter.numberReadAveraged.average\",\r\n     \"storageAdapter.numberWriteAveraged.average\",\r\n     \"storageAdapter.read.average\",\r\n     \"storageAdapter.write.average\",\r\n     \"sys.uptime.latest\",\r\n   ]\r\n  # host_metric_exclude = [] ## Nothing excluded by default\r\n  # host_instances = true ## true by default\r\n   ## Clusters\r\n   # cluster_metric_include = [] ## if omitted or empty, all metrics are collected\r\n   cluster_metric_exclude = [\"*\"] ## Nothing excluded by default\r\n   # cluster_instances = true ## true by default\r\n\r\n   ## Datastores\r\n   # datastore_metric_include = [] ## if omitted or empty, all metrics are collected\r\n   datastore_metric_exclude = [\"*\"] ## Nothing excluded by default\r\n   # datastore_instances = false ## false by default for Datastores only\r\n\r\n   ## Datacenters\r\n   # datacenter_metric_include = [] ## if omitted or empty, all metrics are collected\r\n   datacenter_metric_exclude = [ \"*\" ] ## Datacenters are not collected by default.\r\n   # datacenter_instances = false ## false by default for Datastores only\r\n\r\n   ## Plugin Settings\r\n   ## separator character to use for measurement and field names (default: \"_\")\r\n   # separator = \"_\"\r\n  ## Plugin Settings\r\n   ## separator character to use for measurement and field names (default: \"_\")\r\n   # separator = \"_\"\r\n\r\n   ## number of objects to retreive per query for realtime resources (vms and hosts)\r\n   ## set to 64 for vCenter 5.5 and 6.0 (default: 256)\r\n   max_query_objects = 64\r\n\r\n   ## number of metrics to retreive per query for non-realtime resources (clusters and datastores)\r\n   ## set to 64 for vCenter 5.5 and 6.0 (default: 256)\r\n   max_query_metrics = 64\r\n\r\n   ## number of go routines to use for collection and discovery of objects and metrics\r\n   collect_concurrency = 5\r\n   discover_concurrency = 5\r\n\r\n   ## whether or not to force discovery of new objects on initial gather call before collecting metrics\r\n   ## when true for large environments this may cause errors for time elapsed while collecting metrics\r\n   ## when false (default) the first collection cycle may result in no or limited metrics while objects are discovered\r\n   # force_discover_on_init = false\r\n\r\n   ## the interval before (re)discovering objects subject to metrics collection (default: 300s)\r\n   # object_discovery_interval = \"300s\"\r\n\r\n   ## timeout applies to any of the api request made to vcenter\r\n   timeout = \"60s\"\r\n\r\n   ## Optional SSL Config\r\n   # ssl_ca = \"/path/to/cafile\"\r\n   # ssl_cert = \"/path/to/certfile\"\r\n   # ssl_key = \"/path/to/keyfile\"\r\n   ## Use SSL but skip chain & host verification\r\n   insecure_skip_verify = true\r\n\r\n```"},"comment":{"url":"https://api.github.com/repos/influxdata/telegraf/issues/comments/483252806","html_url":"https://github.com/influxdata/telegraf/issues/5421#issuecomment-483252806","issue_url":"https://api.github.com/repos/influxdata/telegraf/issues/5421","id":483252806,"node_id":"MDEyOklzc3VlQ29tbWVudDQ4MzI1MjgwNg==","user":{"login":"prydin","id":5239535,"node_id":"MDQ6VXNlcjUyMzk1MzU=","avatar_url":"https://avatars3.githubusercontent.com/u/5239535?v=4","gravatar_id":"","url":"https://api.github.com/users/prydin","html_url":"https://github.com/prydin","followers_url":"https://api.github.com/users/prydin/followers","following_url":"https://api.github.com/users/prydin/following{/other_user}","gists_url":"https://api.github.com/users/prydin/gists{/gist_id}","starred_url":"https://api.github.com/users/prydin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prydin/subscriptions","organizations_url":"https://api.github.com/users/prydin/orgs","repos_url":"https://api.github.com/users/prydin/repos","events_url":"https://api.github.com/users/prydin/events{/privacy}","received_events_url":"https://api.github.com/users/prydin/received_events","type":"User","site_admin":false},"created_at":"2019-04-15T13:32:22Z","updated_at":"2019-04-15T13:32:22Z","author_association":"CONTRIBUTOR","body":"Debug logs would be helpful if you can provide them!"}}
other: {"actor":{"display_login":"prydin"}}