django - APM Django - 如何将来自 Filebeat 的容器日志与来自 Kibana 中 ElasticAPM 的数据相关联?
问题描述
更新和
Kibana 版本:
7.14.0
Elasticsearch 版本:
7.14.0
APM 服务器版本:
7.14.0
Filebeat 版本:
7.14.0
APM 代理语言和版本:
Python Django - elastic-apm 6.3.3
说明:
APM 服务器和 Python 代理与 Filebeat 一样按预期工作。他们正在收集日志和指标,但我不知道如何在 Kibana 中关联它们:
来自日志部分的实时流式传输正在运行,我可以使用container.id
.
1. APM、Elasticsearch、Kibana 配置
码头工人-compose.yml:
version: '3.3'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.14.0
hostname: elasticsearch
environment:
- ES_JAVA_OPTS=-Xms512m -Xmx512m
- ELASTIC_PASSWORD=password
ports:
- 192.168.100.100:9200:9200
volumes:
- ./data:/usr/share/elasticsearch/data
- ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
networks:
- elk
kibana:
image: docker.elastic.co/kibana/kibana:7.14.0
hostname: kibana
restart: always
ports:
- 192.168.100.100:5601:5601
volumes:
- ./kibana.yml:/usr/share/kibana/config/kibana.yml:ro
networks:
- elk
apm:
image: docker.elastic.co/apm/apm-server:7.14.0
hostname: apm
command: --strict.perms=false
depends_on:
- elasticsearch
cap_add: ["CHOWN", "DAC_OVERRIDE", "SETGID", "SETUID"]
cap_drop: ["ALL"]
volumes:
- ./apm-server.yml:/usr/share/apm-server/apm-server.yml
ports:
- 192.168.100.100:8200:8200
networks:
- elk
networks:
elk:
driver: bridge
apm-server.yml:
apm-server:
host: "apm:8200"
secret_token: token
rum:
enabled: true
kibana:
enabled: true
host: "kibana:5601"
protocol: "http"
username: "elastic"
password: "password"
setup.template.enabled: true
setup.template.name: "apm-%{[observer.version]}"
setup.template.pattern: "apm-%{[observer.version]}-*"
setup.template.fields: "${path.config}/fields.yml"
setup.template.overwrite: false
setup.template.settings:
index:
number_of_shards: 1
number_of_replicas: 0
codec: best_compression
number_of_routing_shards: 30
mapping.total_fields.limit: 2000
output.elasticsearch:
hosts: ["elasticsearch:9200"]
username: elastic
password: password
index: "apm-%{[observer.version]}-%{+yyyy.MM.dd}"
indices:
- index: "apm-%{[observer.version]}-sourcemap"
when.contains:
processor.event: "sourcemap"
- index: "apm-%{[observer.version]}-error-%{+yyyy.MM.dd}"
when.contains:
processor.event: "error"
- index: "apm-%{[observer.version]}-transaction-%{+yyyy.MM.dd}"
when.contains:
processor.event: "transaction"
- index: "apm-%{[observer.version]}-span-%{+yyyy.MM.dd}"
when.contains:
processor.event: "span"
- index: "apm-%{[observer.version]}-metric-%{+yyyy.MM.dd}"
when.contains:
processor.event: "metric"
- index: "apm-%{[observer.version]}-onboarding-%{+yyyy.MM.dd}"
when.contains:
processor.event: "onboarding"
monitoring.enabled: true
monitoring.elasticsearch:
username: "elastic"
password: "password"
hosts: ["elasticsearch:9200"]
弹性搜索.yml:
---
cluster.name: "docker-cluster"
network.host: 0.0.0.0
discovery.type: single-node
xpack.license.self_generated.type: basic
xpack.security.enabled: true
xpack.monitoring.collection.enabled: true
kibana.yml:
---
server.name: kibana
server.host: "0"
server.publicBaseUrl: https://kibana.domain.com
elasticsearch.hosts: [ "http://elasticsearch:9200" ]
monitoring.ui.container.elasticsearch.enabled: true
elasticsearch.username: elastic
elasticsearch.password: password
2. Python 代理配置:
基础.py:
(...)
import elasticapm
import logging
from elasticapm.handlers.logging import Formatter
from elasticapm.handlers.logging import LoggingFilter
console = logging.StreamHandler()
console.addFilter(LoggingFilter())
logging.getLogger("").addHandler(console)
fh = logging.FileHandler("spam.log")
formatter = Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
fh.setFormatter(formatter)
formatstring = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
formatstring = (
formatstring + " | elasticapm "
"transaction.id=%(elasticapm_transaction_id)s "
"trace.id=%(elasticapm_trace_id)s "
"span.id=%(elasticapm_span_id)s"
)
ELASTIC_APM = {
"SERVICE_NAME": "app",
"SECRET_TOKEN": "secret",
"SERVER_URL": "http://192.168.100.100:8200",
"ENVIRONMENT": "test",
}
(...)
MIDDLEWARE = [
"elasticapm.contrib.django.middleware.TracingMiddleware",
"elasticapm.contrib.django.middleware.Catch404Middleware",
(...)
]
INSTALLED_APPS = [
"elasticapm.contrib.django",
(...)
]
(...)
3. Filebeat 代理配置
/etc/filebeat/filebeat.yml:
filebeat.inputs:
- type: container
paths:
- '/var/lib/docker/containers/*/*.log'
processors:
- add_docker_metadata:
host: "unix:///var/run/docker.sock"
- decode_json_fields:
fields: ["message"]
target: "json"
overwrite_keys: true
output.elasticsearch:
hosts: ["192.168.100.100:9200"]
indices:
- index: "docker-%{[agent.version]}-%{+yyyy.MM.dd}"
username: "elastic"
password: "password"
logging.json: true
logging.metrics.enabled: false
/usr/share/filebeat/module/elasticsearch/server/ingest/pipeline-json.yml:
(...)
- grok:
field: message
patterns: %{GREEDYDATA:msg} | elasticapm transaction.id=%{DATA:transaction.id} trace.id=%{DATA:trace.id} span.id=%{DATA:span.id}
(...)
如何让它发挥作用?我究竟做错了什么?
更新#1
好的,我意识到没有transaction.id
,在过去的几天里,我试图让它出现。
现在我只是将日志保存到日志文件中,如果它可以工作,我将使用 FIlebeat 将日志发送到 Kibana。
我尝试使用ecs
( docs ) 重新格式化日志,但在transaction.id
.
整个配置在这里:
设置/记录器/__init__.py:
import os
import ecs_logging
LOG_LEVEL = os.environ.get("LOG_LEVEL") or "INFO"
BASIC_LOGGER = {
"handlers": ["console"],
"level": LOG_LEVEL,
"propagate": False,
}
LOGGING = {
"version": 1,
"disable_existing_loggers": False,
"formatters": {
"ecs_logging": {"()": ecs_logging.StdlibFormatter}
},
"handlers": {
"file": {
'level': LOG_LEVEL,
'class': 'logging.FileHandler',
'filename': 'file.log',
"formatter": "ecs_logging"
},
},
"loggers": {
**{logger: BASIC_LOGGER for logger in LOGGERS_LIST},
"django": {"handlers": ["file"], 'level': LOG_LEVEL, "propagate": False}
},
}
基础.py:
(...)
import elasticapm
import logging
import ecs_logging
ELASTIC_APM = {
"SERVICE_NAME": "app",
"SECRET_TOKEN": "secret",
"SERVER_URL": "http://192.168.100.100:8200",
"ENVIRONMENT": "test",
"LOG_ECS_REFORMATTING": "override", #tried with off - no difference
}
logger = logging.getLogger("django")
(...)
MIDDLEWARE = [
"elasticapm.contrib.django.middleware.TracingMiddleware",
"elasticapm.contrib.django.middleware.Catch404Middleware",
(...)
]
INSTALLED_APPS = [
"elasticapm.contrib.django",
(...)
]
(...)
文件日志:
{ "@timestamp":"2021-09-06T13:19:20.773Z",
"log.level":"error",
"message":"\"GET / HTTP/1.1\" 500 222222",
"ecs":{
"version":"1.6.0" },
"event":{
"dataset":"app.log" },
"log":{
"logger":"django.server",
"origin":{
"file":{
"line":161,
"name":"whttp.py" },
"function":"log_message" },
"original":"\"GET / HTTP/1.1\" 500 222222" },
"process":{
"name":"MainProcess",
"pid":256,
"thread":{
"id":845453411344867,
"name":"Thread-45" } },
"request":"<socket.socket fd=3, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('172.0.0.4', 9000), raddr=('172.0.0.88', 51198)>",
"server_time":"06/Sep/2021 13:19:20",
"service":{
"name":"app" },
"status_code":500 }
{ "@timestamp":"2021-09-06T13:19:21.088Z",
"log.level":"error",
"message":"Internal Server Error: /favicon.ico",
"ecs":{
"version":"1.6.0" },
"error":{
"message":"Error -3 connecting to redis:6379. Temporary failure in name resolution.",
"stack_trace":" File \"/usr/local/lib/python/site-packages/django/core/handlers/exception.py\", (...whole trace, but no transaction or span id...)",
"type":"ConnectionError" },
"event":{
"dataset":"app.log" },
"log":{
"logger":"django.request",
"origin":{
"file":{
"line":224,
"name":"log.py" },
"function":"log_response" },
"original":"Internal Server Error: /favicon.ico" },
"process":{
"name":"MainProcess",
"pid":222,
"thread":{
"id":948373627243366,
"name":"Thread-46" } },
"request":"<WSGIRequest: GET '/favicon.ico'>",
"service":{
"name":"app" },
"status_code":500 }
{ "@timestamp":"2021-09-06T13:21:32.883Z",
"log.level":"info",
"message":"Watching for file changes with StatReloader",
"ecs":{
"version":"1.6.0" },
"event":{
"dataset":"app.log" },
"log":{
"logger":"django.utils.autoreload",
"origin":{
"file":{
"line":555,
"name":"autoreload.py" },
"function":"run_with_reloader" },
"original":"Watching for file changes with StatReloader" },
"process":{
"name":"MainProcess",
"pid":223,
"thread":{
"id":576784135494040,
"name":"MainThread" } },
"service":{
"name":"app" } }
正如你所看到的ecs
,它正在工作,但为什么它不显示transaction.id
和span.id
?
解决方案
推荐阅读
- angular - 如果@Input 正在接收对象,ngOnChanges 不会触发
- javascript - 尝试在类中创建属性然后设置它们的值
- python - 如何从具有动态列的数据框中获取值
- python - JSON 未正确保存
- configuration - 部分或覆盖 asp.net 核心 appsettings 中的配置选项?
- c# - 循环在word文档中添加表格
- java - 当我尝试使用新的地方时出现问题-compat
- java - 如何将复选框的信息从视图发送到 Thymeleaf 中的控制器?
- reactjs - 使用 Next.js 8 和 styled-jsx 时的强烈 FOUC
- machine-learning - 部署具有一种热编码特征的机器学习模型