0%

ELK搭建

到新公司一直都没很忙,没有太多时间,这周本来是上五天班,结果只上三天,有点像大学了,有天使周五放假,所以连着周末,就一样有三天了,在朋友下载了很多电影,很久没有去过电影院了,都不知道有哪些电影上映了,或者下映了,还下载了<<老友记>>,英语这硬伤,也是个时候正个八经的学下了.

虽然这个搭建起来了,可是…哈哈哈哈,我也不知道该怎么用

一:大概

  • 安装的三个软件都是默认设置
  • 使用的都是默认端口
  • 只是跑起来,看下效果
    image
  • 基本的意思是:logstash读取日志,放在es里,再用kibana读出来

二:安装运行ElasticSearch

  1. 下载tar.gz包并解压(6.3版本)
  2. Run bin/elasticsearch (or bin\elasticsearch.bat on Windows)
  3. 浏览器访问http://localhost:9200/
  4. 出现这样的启动完成了
    1
    {
    2
      "name" : "8V2Xmlw",
    3
      "cluster_name" : "elasticsearch",
    4
      "cluster_uuid" : "fnWvBRwlRoSDJY-BOpYzkg",
    5
      "version" : {
    6
        "number" : "6.3.0",
    7
        "build_flavor" : "default",
    8
        "build_type" : "zip",
    9
        "build_hash" : "424e937",
    10
        "build_date" : "2018-06-11T23:38:03.357887Z",
    11
        "build_snapshot" : false,
    12
        "lucene_version" : "7.3.1",
    13
        "minimum_wire_compatibility_version" : "5.6.0",
    14
        "minimum_index_compatibility_version" : "5.0.0"
    15
      },
    16
      "tagline" : "You Know, for Search"
    17
    }

三:安装运行Logstash

  1. 下载tar.gz包并解压(6.2.2版本,用6.3的没有启动成功)
  2. cd logstash-6.2.2
  3. bin/logstash(先启动试试,能不能正常启动,可以不用这一步)
  4. 在根目录创建logstash-simple.conf
    1
    # 配置输入为 beats
    2
    input {
    3
        beats {
    4
                port => "5044"
    5
        }
    6
    7
    }
    8
    # 数据过滤
    9
    filter {
    10
        grok {
    11
                match => { "message" => "%{COMBINEDAPACHELOG}" }
    12
        }
    13
        geoip {
    14
                source => "clientip"
    15
        }
    16
    17
    }
    18
    # 输出到本机的 ES
    19
    output {
    20
        elasticsearch {
    21
                hosts => [ "localhost:9200"  ]
    22
        }
    23
    }
  5. 启动bin/logstash -f logstash-simple.conf --config.reload.automatic,要等一会才有反应
  6. 出现这样的基本就对了
    1
    [2018-06-16T22:40:40,039][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/home/anthony/下载/logstash-6.2.2/modules/fb_apache/configuration"}
    2
    [2018-06-16T22:40:40,106][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/home/anthony/下载/logstash-6.2.2/modules/netflow/configuration"}
    3
    [2018-06-16T22:40:41,926][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
    4
    [2018-06-16T22:40:43,839][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.2.2"}
    5
    [2018-06-16T22:40:45,772][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
    6
    [2018-06-16T22:40:52,182][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
    7
    [2018-06-16T22:40:53,492][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
    8
    [2018-06-16T22:40:53,511][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
    9
    [2018-06-16T22:40:54,259][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
    10
    [2018-06-16T22:40:54,442][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>nil}
    11
    [2018-06-16T22:40:54,450][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}

四:安装运行FileBeats

通过 FileBeats 收集目标日志,然后统一输出到 LogStash 做进一步的过滤,在由 LogStash 输出到 ES 中进行存储。

  1. 下载地址https://www.elastic.co/downloads/beats/filebeat(6.3版本)

  2. 编辑filebeat.yml
    找到,按照格式编辑,paths代表的是日志路径有两个output,注释掉elasticsearch output,打开logstash output

    1
    - type: log
    2
      # Change to true to enable this input configuration.
    3
      enabled: True
    4
    5
      # Paths that should be crawled and fetched. Glob based paths.
    6
      paths:
    7
        - /home/anthony/桌面/mylog/*.log
    8
    9
    10
    #-------------------------- Elasticsearch output ------------------------------
    11
    #output.elasticsearch:
    12
      # Array of hosts to connect to. 注释掉这个
    13
      #hosts: ["localhost:9200"]
    14
    15
      # Optional protocol and basic auth credentials.
    16
      #protocol: "https"
    17
      #username: "elastic"
    18
      #password: "changeme"
    19
    20
    #----------------------------- Logstash output --------------------------------
    21
    output.logstash:
    22
      # The Logstash hosts,打开这个注释
    23
      hosts: ["localhost:5044"]
  3. 运行

    1
    # FileBeat 需要以 root 身份启动,因此先更改配置文件的权限
    2
    sudo chown root filebeat.yml
    3
    sudo ./filebeat -e -c filebeat.yml -d "publish"
  4. 结果是看到打印出来一堆东西,最后是以complete结尾,就是成功了

五:安装运行kinbana

  1. 下载解压(6.3版本)
  2. 运行bin/kibana
  3. 选左边的导航栏,Discover,(创建索引)

第二步的时候选择@timestamp

  1. 再点Discover

  2. 就可以查看日志了