名称 最后更新
.github/workflows 正在载入提交数据...
doc 正在载入提交数据...
kq 正在载入提交数据...
stash 正在载入提交数据...
.dockerignore 正在载入提交数据...
.gitattributes 正在载入提交数据...
.gitignore 正在载入提交数据...
.golangci.yml 正在载入提交数据...
LICENSE 正在载入提交数据...
go.mod 正在载入提交数据...
go.sum 正在载入提交数据...
readme.md 正在载入提交数据...

go-stash简介

go-stash是一个高效的从Kafka获取,根据配置的规则进行处理,然后发送到ElasticSearch集群的工具。

go-stash有大概logstash 5倍的吞吐性能,并且部署简单,一个可执行文件即可。

go-stash

Quick Start

gostash -f etc/config.json

config.json示例如下:

{
    "Input": {
        "Kafka": {
            "Name": "gostash",
            "Brokers": [
                "172.16.186.16:19092",
                "172.16.186.17:19092"
            ],
            "Topic": "k8slog",
            "Group": "pro",
            "NumProducers": 16
        }
    },
    "Filters": [
        {
            "Action": "drop",
            "Conditions": [
                {
                    "Key": "k8s_container_name",
                    "Value": "-rpc",
                    "Type": "contains"
                },
                {
                    "Key": "level",
                    "Value": "info",
                    "Type": "match",
                    "Op": "and"
                }
            ]
        },
        {
            "Action": "remove_field",
            "Fields": [
                "message",
                "_source",
                "_type",
                "_score",
                "_id",
                "@version",
                "topic",
                "index",
                "beat",
                "docker_container",
                "offset",
                "prospector",
                "source",
                "stream"
            ]
        }
    ],
    "Output": {
        "ElasticSearch": {
            "Hosts": [
                "172.16.141.4:9200",
                "172.16.141.5:9200"
            ],
            "DailyIndexPrefix": "k8s_pro-"
        }
    }
}

微信交流群

添加我的微信:kevwan,请注明go-stash,我拉进go-stash社区群🤝