Filebeat收集tomcat日志

首先tomcat对我们有用的日志文件主要是catalina和localhost_access_log,下面主要介绍如何收集这2种日志

tomcat使用log4j日志输出

先去这个地址下载对应tomcat版本的相关jar包：http://archive.apache.org/dist/tomcat/

就是我下面图片红框标注的2个jar

我们进入tomcat的lib目录下，将tomcat-juli-adapters.jar放入到里面

然后在lib目录下执行

wget https://archive.apache.org/dist/logging/log4j/1.2.17/log4j-1.2.17.jar

接着进入tomcat的bin目录,先将原本里面的tomcat-juli.jar重命名备份，然后再将我们刚刚下载的tomcat-juli.jar放入到里面

mv tomcat-juli.jar tomcat-juli.jar.bak
#将刚刚下载的tomcat-juli.jar放进bin目录

进入tomcat的conf目录

mv logging.properties  logging.properties.bak

# 在<Context>标签中添加 swallowOutput 属性 ，即 <Context swallowOutput="true">
vim context.xml

进入tomcat的lib目录

vim log4j.properties

log4j.rootLogger = INFO, CATALINA

# Define all the appenders
log4j.appender.CATALINA=org.apache.log4j.RollingFileAppender
log4j.appender.CATALINA.File=${catalina.base}/logs/catalina
log4j.appender.CATALINA.layout=org.apache.log4j.PatternLayout
log4j.appender.CATALINA.layout.ConversionPattern={"time":"%d{yyyy-MM-dd HH:mm:ss,SSS}","logtype":"%p","loginfo":"%c:%m"}%n
log4j.appender.CATALINA.MaxFileSize=2MB
log4j.appender.CATALINA.MaxBackupIndex=10

log4j.appender.LOCALHOST = org.apache.log4j.DailyRollingFileAppender
log4j.appender.LOCALHOST.File = ${catalina.base}/logs/localhost
log4j.appender.LOCALHOST.Append = true
log4j.appender.LOCALHOST.Encoding = UTF-8
log4j.appender.LOCALHOST.DatePattern = '.'yyyy-MM-dd'.log'
log4j.appender.LOCALHOST.layout = org.apache.log4j.PatternLayout
log4j.appender.LOCALHOST.layout.ConversionPattern = %d [%t] %-5p %c- %m%n

log4j.appender.MANAGER = org.apache.log4j.DailyRollingFileAppender
log4j.appender.MANAGER.File = ${catalina.base}/logs/manager
log4j.appender.MANAGER.Append = true
log4j.appender.MANAGER.Encoding = UTF-8
log4j.appender.MANAGER.DatePattern = '.'yyyy-MM-dd'.log'
log4j.appender.MANAGER.layout = org.apache.log4j.PatternLayout
log4j.appender.MANAGER.layout.ConversionPattern = %d [%t] %-5p %c- %m%n

log4j.appender.HOST-MANAGER = org.apache.log4j.DailyRollingFileAppender
log4j.appender.HOST-MANAGER.File = ${catalina.base}/logs/host-manager
log4j.appender.HOST-MANAGER.Append = true
log4j.appender.HOST-MANAGER.Encoding = UTF-8
log4j.appender.HOST-MANAGER.DatePattern = '.'yyyy-MM-dd'.log'
log4j.appender.HOST-MANAGER.layout = org.apache.log4j.PatternLayout
log4j.appender.HOST-MANAGER.layout.ConversionPattern = %d [%t] %-5p %c- %m%n

log4j.appender.CONSOLE = org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.Encoding = UTF-8
log4j.appender.CONSOLE.layout = org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern = %d [%t] %-5p %c- %m%n

# Configure which loggers log to which appenders
log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost] = INFO, LOCALHOST
log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager] =\
  INFO, MANAGER
log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager] =\
  INFO, HOST-MANAGER

到此为止，我们已经成功使用log4j来输出tomcat日志了，并且是json格式的，方便filebeat收集

下面我们来将localhost_access_log来转成json格式的日志输出

通过vim打开conf/server.xml文件，拉到文件最后面

#修改此处内容
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log" suffix=".log"
               pattern="{"client":"%h",  "client user":"%l",   "authenticated":"%u",   "access time":"%t",     "method":"%r",   "status":"%s",  "send bytes":"%b",  "Query?string":"%q",  "partner":"%{Referer}i",  "Agent version":"%{User-Agent}i"}" />

启动tomcat，可以发现我们成功的将catalina和localhost_access_log以json格式输出日志了

filebeat收集日志

- input_type: log
  enabled: true
  paths:
    - /usr/local/apache-tomcat-9.0.36/logs/localhost_access_log*.log
  fields:
    source: localhost_access_log
- input_type: log
  enabled: true
  paths:
    - /usr/local/apache-tomcat-9.0.36/logs/catalina
  fields:
    source: catalina

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["192.168.203.133:9200"]
  indices:
    - index: "localhost_access_log-%{+yyyy.MM.dd}"
      when.contains:
        source: "localhost_access_log"
    - index: "catalina-%{+yyyy.MM.dd}"
      when.contains:
        source: "catalina"

启动filebeat

./filebeat -e -c filebeat.yml

配置好index后，可以在discover看到收集的日志