Elasticsearch & Mysql 配置搜索

Elasticsearch下载，解压即可
https://www.elastic.co/downloads/elasticsearch

运行
su – es -c “elasticsearch -d”

不允许以root用户启动elasticsearch

错误：
max file descriptors [65535] for elasticsearch process is too low, increase to at least [65536]
解决：
vim /etc/security/limits.conf
* soft nproc 65536
* hard nproc 65536
* soft nofile 65536
* hard nofile 65536

错误：
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
解决：
vim /etc/sysctl.conf
添加
vm.max_map_count=262144

应用修改
sysctl -p

错误：
the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured


vim elasticsearch.yml

cluster.initial_master_nodes: [“node-1″,”node-2”]，把node-2去掉

以守护进程启动
./bin/elasticsearch -d

浏览器访问
http://127.0.0.1:9200/

{
"name" : "vK_-Q-1",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "2H9ai9YlQniKTkhknmIMtg",
"version" : {
"number" : "6.3.0",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "424e937",
"build_date" : "2018-06-11T23:38:03.357887Z",
"build_snapshot" : false,
"lucene_version" : "7.3.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}

// 创建index t1
curl -XPUT http://vm:9200/t1?pretty

// 查询全部docs
curl http://vm:9200/_search?pretty

// 查询 index=t1 的docs
curl http://vm:9200/t1/_search?pretty

// 查询 index=t1 && type=doc 的docs
curl http://vm:9200/t1/doc/_search?pretty

// 查询 index=t1 && type=doc && id=3 doc
curl http://vm:9200/t1/doc/3?pretty

// 条件查询，字段content里面包含hello的记录
curl http://vm:9200/t1/doc/_search?q=content:hello&pretty

// 删除doc
curl -XPOST -H “Content-Type: application/json” http://vm:9200/t1/doc/_delete_by_query?conflicts=proceed -d ‘{“query”: {“match_all”: {}}}’

// 中文分词插件elasticsearch-analysis-ik
https://github.com/medcl/elasticsearch-analysis-ik

下载后解压到 es_dir/plugs/elasticsearch-analysis-ik 目录

或者执行命令
./bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v6.3.0/elasticsearch-analysis-ik-6.3.0.zip

重启Elasticsearch

下面是官方提供的一个演示流程

// 创建index t3
curl -XPUT http://vm:9200/t3

// 创建mapping
curl -XPOST http://vm:9200/t4/doc/_mapping -H ‘Content-Type:application/json’ -d'{“properties”: {“content”: {“type”: “text”,”analyzer”: “ik_max_word”,”search_analyzer”: “ik_max_word”}}}’

ik_max_word: 最大粒度分词，词多
ik_smart : 较小粒度，词少

// 给index t3 添加 docs
curl -XPOST http://vm:9200/t3/doc/1 -H ‘Content-Type:application/json’ -d'{“content”:”中国是人口最多的国家”}’
curl -XPOST http://vm:9200/t3/doc/2 -H ‘Content-Type:application/json’ -d'{“content”:”美国是最富有的国家”}’
curl -XPOST http://vm:9200/t3/doc/3 -H ‘Content-Type:application/json’ -d'{“content”:”日本是一个岛国”}’

// 搜索
curl -XGET -H ‘Content-Type:application/json’ http://vm:9200/t4/doc/_search?pretty -d ‘{“query” : { “match” : { “content” : “日本” }},”highlight” : {“pre_tags” : [“<tag1>”, “<tag2>”],”post_tags” : [“</tag1>”, “</tag2>”],”fields” : {“content” : {}}}}’

Logstash配置

Logstash 是开源的服务器端数据处理管道，能够同时 从多个来源采集数据、转换数据，然后将数据发送到您最喜欢的 “存储库” 中。

Logstash下载，解压即可
https://www.elastic.co/downloads/logstash

安装 jdbc 和 elasticsearch 插件
bin/logstash-plugin install logstash-input-jdbc
bin/logstash-plugin install logstash-output-elasticsearch

下载mysql-java连接驱动并解压
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz

创建io目录，在下面新建mysql.conf文件

input {
jdbc {
jdbc_driver_library => "/data/apps/logstash-6.3.0/mysql-connector-java-5.1.46-bin.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://localhost:3306/test"
jdbc_user => "user"
jdbc_password => "password"
schedule => "* * * * *"
statement => "SELECT * FROM documents"
#use_column_value => true
#tracking_column_type => "timestamp"
#tracking_column => "update_time"
#last_run_metadata_path => "syncpoint_table"
}
}

filter {}

output {
elasticsearch {
hosts => "127.0.0.1:9200"
#user => "<user>"
#password => "<password>"
index => "t5"
document_id => "%{id}"
}
}

 

开始导入到Elasticsearch
./bin/logstash -f ./io/mysql.conf