monitoring - grafana 和 thanos 数据源中的重复时间戳值显示单个时间戳值

Question

Grafana 显示重复的时间戳值，Thanos 显示正确的单个时间戳值。我正在为应用程序 API Thanos 发出一个显示正确值的 curl 请求，但是当我在 grafana 中运行相同的查询时，它显示了两个计数值。我正在使用电报代理来收集普罗米修斯中的指标。

我的整个设置都在 Kubernetes 中运行，并且我正在使用 telegraf statsd 进行应用程序监控。

电报会议 >>

[agent]
  interval = "15s"
  round_interval = true


  metric_batch_size = 1000

  metric_buffer_limit = 10000
  flush_buffer_when_full = true

  collection_jitter = "0s"

  flush_interval = "1s"
  flush_jitter = "0s"
  quiet = false


  debug = false

   logfile = "/var/log/telegraf/telegraf.log"

   logfile_rotation_max_size = "10MB"

   logfile_rotation_max_archives = 5

   hostname = "${HOSTNAME}"


   [global_tags]
    dc = "${datacenter}"
    component = "k8s"
    role = "node"
    job  = "${job}"
    service = "containerorchestration"
    subcomponent = "worker"
    organization = "${organization}"
    environment = "${environment}"
    environmentversion = "${environmentversion}"
    infraversion = "${infraversion}"



  [[inputs.cpu]]
    percpu = false
    totalcpu = true
    fielddrop = ["time_*"]

  [[inputs.disk]]
    ignore_fs = ["tmpfs", "devtmpfs", "devfs", "overlay", "aufs", "squashfs"]

  [[inputs.diskio]]

  [[inputs.kernel]]

  [[inputs.mem]]

  [[inputs.processes]]

  [[inputs.swap]]

  [[inputs.system]]

  [[inputs.net]]
    fielddrop = ["icmp_*", "icmpmsg_*", "tcp_*", "udp_*", "udplite_*", "ip_*"]

  [[inputs.netstat]]

  [[inputs.linux_sysctl_fs]]




 [[outputs.prometheus_client]]
   listen = ":9273"
   metric_version = 2
   path = "/metrics"
   expiration_interval = "16s"
   export_timestamp = true

电报统计配置 >>

[[inputs.statsd]]
  protocol = "udp"
  max_tcp_connections = 250
  tcp_keep_alive = false

  service_address = ":8130"

  delete_gauges = true
  delete_counters = true
  delete_sets = true
  delete_timings = true
  parse_data_dog_tags = true

  percentiles = [90.0, 95.0, 99.0]

  metric_separator = "_"

  datadog_extensions = true

  allowed_pending_messages = 10000

  percentile_limit = 1000

和普罗米修斯工作会议>

- job_name: 'ec2-telegraf'
  sample_limit: 4000
  metrics_path: '/metrics'
  scrape_interval: '15s'
  ec2_sd_configs:
    - region: "XXXXXXX"
      profile: "XXXXXXXXXX"
      role_arn: XXXXXXXXXXXXXXX
      refresh_interval: 100s
      port: 9273
      filters:
        - name: instance-state-name
          values:
            - running
        - name: tag:Environment
          values:
            - performance
        
  relabel_configs:
    - source_labels: [__meta_ec2_tag_Businessunit]
      target_label: businessunit
    - source_labels: [__meta_ec2_tag_Environment]
      target_label: environment
    - source_labels: [__meta_ec2_tag_Techteam]
      target_label: techteam
    - source_labels: [__meta_ec2_tag_component]
      target_label: component
    - source_labels: [__meta_ec2_tag_subcomponent]
      target_label: subcomponent
    - source_labels: [__meta_ec2_tag_role]
      target_label: role
    - source_labels: [__meta_ec2_tag_aws_autoscaling_groupName]
      target_label: asgname
    - source_labels: [__meta_ec2_tag_Service]
      target_label: service

需要帮助，请分享建议。谢谢

标签： monitoringprometheusgrafanatelegrafthanos