it happens time out per 1minute when use etcd v3 watch method

[bingchengxiaobing117]

problem

when I use etcd v3 watch method(default params),The following questions occurs:

1. it happens time out per 1minute .
2. There is a lot of connection building with etcd server in per worker progress

log info:

2020/04/15 15:05:34 [info] 9020#9020: *20628390 [lua] v3.lua:502: request_chunk(): http request method: POST path: /v3beta/watch body: {"create_request":{"range_end":"*********","start_revision":6647555,"key":"L2V0Y2QvY2ZnL2dsb2JhbA=="}} query: nil, context: ngx.timer
2020/04/15 15:06:34 [info] 9020#9020: *20628390 [lua] v3.lua:502: request_chunk(): http request method: POST path: /v3beta/watch body: {"create_request":{"range_end":"*********","start_revision":6647555,"key":"L2V0Y2QvY2ZnL2dsb2JhbA=="}} query: nil, context: ngx.timer
2020/04/15 15:07:34 [info] 9020#9020: *20628390 [lua] v3.lua:502: request_chunk(): http request method: POST path: /v3beta/watch body: {"create_request":{"range_end":"*********","start_revision":6647555,"key":"L2V0Y2QvY2ZnL2dsb2JhbA=="}} query: nil, context: ngx.timer
2020/04/15 15:08:34 [info] 9020#9020: *20628390 [lua] v3.lua:502: request_chunk(): http request method: POST path: /v3beta/watch body: {"create_request":{"range_end":"*********","start_revision":6647555,"key":"L2V0Y2QvY2ZnL2dsb2JhbA=="}} query: nil, context: ngx.timer
.......

other info:

etcd endpoints has three ip list such as (1.1.1.1:2379;1.1.1.2:2379;1.1.1.3:2379;)

connection info:
such as worker id: 9020 exits 58 establish connection

root@****:~# netstat -apn|grep 2379 |grep 'ESTABLISHED 9020' |wc -l
58

@YuanSheng Wang Have you met this problem? How to solve

[moonming]

@membphis

[membphis]

@nic-chen do you have time to confirm this problem?

[membphis]

@YuanSheng Wang Have you met this problem? How to solve

I do not know how to fix it now. we mainly use etcd v2 now.

[bingchengxiaobing117]

I has used etcd v2 don't has this problem, I think the reason is etcd v3 should be http response with no buffering( cause by the timing of buffer flush ) , you can see etcd office api guide such as watch

curl -N http://localhost:2379/v3/watch \
  -X POST -d '{"create_request": {"key":"Zm9v"} }' &

so how to set the timing of buffer flush in lua-rest-http module

[Yiyiyimu]

Hi @bingchengxiaobing117 are you still facing the same problem. It seems I could not reproduce it.

Test code:

local etcd, err = require("resty.etcd").new({protocol = "v3"})
check_res(etcd, err)

local res, err = etcd:set("/test", "abc")
check_res(res, err)

ngx.timer.at(200, function ()
    etcd:set("/test", "bcd3")
end)

local cur_time = ngx.now()
local body_chunk_fun, err = etcd:watch("/test", {timeout = 300})
if not body_chunk_fun then
    ngx.say("failed to watch: ", err)
end

Log:

2020/09/06 14:47:21 [info] 28274#28274: *1 [lua] v3.lua:284: set(): v3 set body: {"header":{"raft_term":"2","cluster_id":"8925027824743593106","member_id":"13803658152347727308","revision":"7"}}, client: 127.0.0.1, server: localhost, request: "GET /t HTTP/1.1", host: "localhost"
2020/09/06 14:47:21 [info] 28274#28274: *1 [lua] v3.lua:501: request_chunk(): http request method: POST path: /v3/watch body: {"create_request":{"key":"L3Rlc3Q="}} query: nil, client: 127.0.0.1, server: localhost, request: "GET /t HTTP/1.1", host: "localhost"
2020/09/06 14:50:41 [info] 28274#28274: *3 [lua] v3.lua:284: set(): v3 set body: {"header":{"raft_term":"2","cluster_id":"8925027824743593106","member_id":"13803658152347727308","revision":"8"}}, context: ngx.timer, client: 127.0.0.1, server: 0.0.0.0:1984
2020/09/06 14:52:21 [notice] 28274#28274: signal 15 (SIGTERM) received from 28253, exiting
2020/09/06 14:52:21 [notice] 28274#28274: signal 3 (SIGQUIT) received from 28253, shutting down
2020/09/06 14:52:21 [info] 28274#28274: epoll_wait() failed (4: Interrupted system call)
2020/09/06 14:52:21 [notice] 28274#28274: exiting
2020/09/06 14:52:21 [notice] 28274#28274: exit