python - Why uWSGI workers stop responding SIGHUP? -

python source:

import time import os import signal import threading import datetime import uwsgi  to_be_killed = {}   def print_still_alive():     = time.time()     still_alive_pids = set(w['pid'] w in uwsgi.workers() if w['pid'] > 0)     print datetime.datetime.now(), ('still alive: %r' % {k: int(now - v) k, v in                                     to_be_killed.iteritems() if k in still_alive_pids})   class periodictimer(threading._timer):     def run(self):         while not self.finished.is_set():             self.finished.wait(self.interval)             if not self.finished.is_set():                 self.function(*self.args, **self.kwargs)   def cycle():     print_still_alive()      w in uwsgi.workers():         if w['pid'] <= 0:             continue          pid = w['pid']         to_be_killed.setdefault(pid, time.time())         os.kill(pid, signal.sighup)         print datetime.datetime.now(), 'sent hup %s' % pid   if uwsgi.masterpid() == os.getpid():     timer = periodictimer(5, cycle, ())     timer.setdaemon(true)     timer.start()   def application(env, start_response):     start_response('200 ok', [('content-type', 'text/html')])     return ['hello world %s: %s' % (os.getpid(), time.time())] 

run with:

uwsgi --http :8003 --wsgi simple_th:application --honour-stdin --master --workers 5 --py-auto-reload=1 --enable-threads 

uwsgi version:

$ uwsgi --version 2.0.8 

linux:

$ uname -a linux xxx 3.13.0-24-generic #47-ubuntu smp fri may 2 23:30:00 utc 2014 x86_64 x86_64 x86_64 gnu/linux 

the problem after time workers stop responding hup signal. looks in log:

$ uwsgi --http :8003 --wsgi simple_th:application --honour-stdin --master --workers 5 --py-auto-reload=1 --enable-threads *** starting uwsgi 2.0.8 (64bit) on [wed nov 19 15:43:35 2014] *** compiled version: 4.8.2 on 14 november 2014 19:44:37 os: linux-3.13.0-24-generic #47-ubuntu smp fri may 2 23:30:00 utc 2014 nodename: dmugtasimov-thinkpad-t520 machine: x86_64 clock source: unix detected number of cpu cores: 4 current working directory: /home/dmugtasimov/gitrep/apigateway/tmp detected binary path: /home/dmugtasimov/gitrep/apigateway/env/bin/uwsgi !!! no internal routing support, rebuild pcre support !!! processes number limit 63268 memory page size 4096 bytes detected max file descriptor number: 1024 lock engine: pthread robust mutexes thunder lock: disabled (you can enable --thunder-lock) uwsgi http bound on :8003 fd 4 uwsgi socket 0 bound tcp address 127.0.0.1:36063 (port auto-assigned) fd 3 python version: 2.7.6 (default, mar 22 2014, 23:03:41)  [gcc 4.8.2] python main interpreter initialized @ 0x1f0f350 python threads support enabled server socket listen backlog limited 100 connections mercy graceful operations on workers 60 seconds mapped 436608 bytes (426 kb) 5 cores *** operational mode: preforking *** wsgi app 0 (mountpoint='') ready in 0 seconds on interpreter 0x1f0f350 pid: 21265 (default app) *** uwsgi running in multiple interpreter mode *** spawned uwsgi master process (pid: 21265) spawned uwsgi worker 1 (pid: 21267, cores: 1) spawned uwsgi worker 2 (pid: 21268, cores: 1) spawned uwsgi worker 3 (pid: 21269, cores: 1) spawned uwsgi worker 4 (pid: 21270, cores: 1) spawned uwsgi worker 5 (pid: 21271, cores: 1) spawned uwsgi http 1 (pid: 21272) python auto-reloader enabled 2014-11-19 15:43:40.553574 still alive: {} 2014-11-19 15:43:40.554045 sent hup 21267 gracefully killing worker 1 (pid: 21267)... 2014-11-19 15:43:40.554127 sent hup 21268 gracefully killing worker 2 (pid: 21268)... gracefully killing worker 3 (pid: 21269)... 2014-11-19 15:43:40.555841 sent hup 21269 2014-11-19 15:43:40.555936 sent hup 21270 gracefully killing worker 5 (pid: 21271)... gracefully killing worker 4 (pid: 21270)... 2014-11-19 15:43:40.560523 sent hup 21271 respawned uwsgi worker 1 (new pid: 21281) respawned uwsgi worker 2 (new pid: 21282) respawned uwsgi worker 3 (new pid: 21283) respawned uwsgi worker 4 (new pid: 21284) respawned uwsgi worker 5 (new pid: 21285) python auto-reloader enabled 2014-11-19 15:43:45.561159 still alive: {} gracefully killing worker 1 (pid: 21281)... 2014-11-19 15:43:45.561491 sent hup 21281 gracefully killing worker 2 (pid: 21282)... 2014-11-19 15:43:45.562548 sent hup 21282 gracefully killing worker 3 (pid: 21283)... 2014-11-19 15:43:45.562621 sent hup 21283 2014-11-19 15:43:45.562665 sent hup 21284 gracefully killing worker 5 (pid: 21285)... gracefully killing worker 4 (pid: 21284)... 2014-11-19 15:43:45.569720 sent hup 21285 respawned uwsgi worker 1 (new pid: 21292) respawned uwsgi worker 2 (new pid: 21293) respawned uwsgi worker 3 (new pid: 21294) respawned uwsgi worker 4 (new pid: 21295) respawned uwsgi worker 5 (new pid: 21296) python auto-reloader enabled 2014-11-19 15:43:50.570273 still alive: {} 2014-11-19 15:43:50.570476 sent hup 21292 gracefully killing worker 1 (pid: 21292)... 2014-11-19 15:43:50.570516 sent hup 21293 gracefully killing worker 2 (pid: 21293)... 2014-11-19 15:43:50.570572 sent hup 21294 2014-11-19 15:43:50.570622 sent hup 21295 gracefully killing worker 4 (pid: 21295)... 2014-11-19 15:43:50.570673 sent hup 21296 gracefully killing worker 5 (pid: 21296)... gracefully killing worker 3 (pid: 21294)... respawned uwsgi worker 1 (new pid: 21304) respawned uwsgi worker 2 (new pid: 21305) respawned uwsgi worker 3 (new pid: 21306) respawned uwsgi worker 4 (new pid: 21307) respawned uwsgi worker 5 (new pid: 21308) python auto-reloader enabled 2014-11-19 15:43:55.571036 still alive: {} gracefully killing worker 1 (pid: 21304)... 2014-11-19 15:43:55.571167 sent hup 21304 gracefully killing worker 2 (pid: 21305)... 2014-11-19 15:43:55.571200 sent hup 21305 2014-11-19 15:43:55.571232 sent hup 21306 2014-11-19 15:43:55.571260 sent hup 21307 2014-11-19 15:43:55.571290 sent hup 21308 gracefully killing worker 3 (pid: 21306)... gracefully killing worker 4 (pid: 21307)... gracefully killing worker 5 (pid: 21308)... respawned uwsgi worker 1 (new pid: 21317) python auto-reloader enabled respawned uwsgi worker 2 (new pid: 21322) respawned uwsgi worker 3 (new pid: 21323) respawned uwsgi worker 4 (new pid: 21324) respawned uwsgi worker 5 (new pid: 21325) 2014-11-19 15:44:00.571999 still alive: {} gracefully killing worker 1 (pid: 21317)... 2014-11-19 15:44:00.572323 sent hup 21317 2014-11-19 15:44:00.572408 sent hup 21322 gracefully killing worker 2 (pid: 21322)... 2014-11-19 15:44:00.572461 sent hup 21323 2014-11-19 15:44:00.572516 sent hup 21324 gracefully killing worker 5 (pid: 21325)... 2014-11-19 15:44:00.572568 sent hup 21325 gracefully killing worker 3 (pid: 21323)... gracefully killing worker 4 (pid: 21324)... respawned uwsgi worker 2 (new pid: 21331) respawned uwsgi worker 3 (new pid: 21332) respawned uwsgi worker 5 (new pid: 21333) respawned uwsgi worker 1 (new pid: 21337) respawned uwsgi worker 4 (new pid: 21338) python auto-reloader enabled 2014-11-19 15:44:05.572873 still alive: {} gracefully killing worker 1 (pid: 21337)... 2014-11-19 15:44:05.575816 sent hup 21337 2014-11-19 15:44:05.575888 sent hup 21331 gracefully killing worker 3 (pid: 21332)... 2014-11-19 15:44:05.579080 sent hup 21332 2014-11-19 15:44:05.579159 sent hup 21338 2014-11-19 15:44:05.579189 sent hup 21333 gracefully killing worker 5 (pid: 21333)... gracefully killing worker 2 (pid: 21331)... gracefully killing worker 4 (pid: 21338)... respawned uwsgi worker 2 (new pid: 21342) respawned uwsgi worker 3 (new pid: 21343) respawned uwsgi worker 5 (new pid: 21344) respawned uwsgi worker 1 (new pid: 21345) respawned uwsgi worker 4 (new pid: 21346) python auto-reloader enabled 2014-11-19 15:44:10.579536 still alive: {} gracefully killing worker 1 (pid: 21345)... 2014-11-19 15:44:10.582839 sent hup 21345 2014-11-19 15:44:10.583022 sent hup 21342 2014-11-19 15:44:10.583111 sent hup 21343 2014-11-19 15:44:10.583188 sent hup 21346 gracefully killing worker 4 (pid: 21346)... gracefully killing worker 5 (pid: 21344)... 2014-11-19 15:44:10.586615 sent hup 21344 gracefully killing worker 3 (pid: 21343)... gracefully killing worker 2 (pid: 21342)... respawned uwsgi worker 2 (new pid: 21353) respawned uwsgi worker 3 (new pid: 21354) respawned uwsgi worker 5 (new pid: 21355) respawned uwsgi worker 1 (new pid: 21356) respawned uwsgi worker 4 (new pid: 21357) python auto-reloader enabled 2014-11-19 15:44:15.587064 still alive: {} gracefully killing worker 1 (pid: 21356)... 2014-11-19 15:44:15.587251 sent hup 21356 2014-11-19 15:44:15.587290 sent hup 21353 2014-11-19 15:44:15.587324 sent hup 21354 gracefully killing worker 3 (pid: 21354)... 2014-11-19 15:44:15.587356 sent hup 21357 2014-11-19 15:44:15.587390 sent hup 21355 gracefully killing worker 4 (pid: 21357)... gracefully killing worker 2 (pid: 21353)... gracefully killing worker 5 (pid: 21355)... respawned uwsgi worker 2 (new pid: 21367) respawned uwsgi worker 3 (new pid: 21368) respawned uwsgi worker 5 (new pid: 21369) respawned uwsgi worker 1 (new pid: 21370) respawned uwsgi worker 4 (new pid: 21371) python auto-reloader enabled 2014-11-19 15:44:20.588180 still alive: {} gracefully killing worker 1 (pid: 21370)... 2014-11-19 15:44:20.588577 sent hup 21370 gracefully killing worker 2 (pid: 21367)... 2014-11-19 15:44:20.588658 sent hup 21367 2014-11-19 15:44:20.588704 sent hup 21368 gracefully killing worker 3 (pid: 21368)... 2014-11-19 15:44:20.588749 sent hup 21371 gracefully killing worker 5 (pid: 21369)... 2014-11-19 15:44:20.588797 sent hup 21369 gateway "uwsgi http 1" has been buried (pid: 21272) ...gracefully killing workers... respawned uwsgi http 1 (pid: 21380) worker 2 killed (pid: 21367) respawned uwsgi worker 2 (new pid: 21381) worker 3 killed (pid: 21368) respawned uwsgi worker 3 (new pid: 21382) worker 5 killed (pid: 21369) respawned uwsgi worker 5 (new pid: 21383) worker 1 killed (pid: 21370) respawned uwsgi worker 1 (new pid: 21384) python auto-reloader enabled 2014-11-19 15:44:25.589164 still alive: {21371: 5} 2014-11-19 15:44:25.589327 sent hup 21384 gracefully killing worker 1 (pid: 21384)... 2014-11-19 15:44:25.589356 sent hup 21381 2014-11-19 15:44:25.589381 sent hup 21382 2014-11-19 15:44:25.589421 sent hup 21371 2014-11-19 15:44:25.589450 sent hup 21383 gracefully killing worker 2 (pid: 21381)... gracefully killing worker 3 (pid: 21382)... gracefully killing worker 5 (pid: 21383)... respawned uwsgi worker 2 (new pid: 21390) respawned uwsgi worker 3 (new pid: 21391) respawned uwsgi worker 5 (new pid: 21392) respawned uwsgi worker 1 (new pid: 21393) python auto-reloader enabled 2014-11-19 15:44:30.589732 still alive: {21371: 10} 

i did 5 trials. time required reproduce varies, in seconds: 5, 60, 20, 50, 35. takes time see problem.

why happens?

p.s. problem, maybe related. workers try handle sigint/sigquit signal (ctrl+c in linux terminal)

^cthe process id 21265 sigint/sigquit received...killing workers... process id 21371 sigint/sigquit received...killing workers... gateway "uwsgi http 1" has been buried (pid: 21380) gateway "uwsgi http 1" has been buried (pid: 21380) worker 2 buried after 1 seconds worker 3 buried after 1 seconds worker 5 buried after 1 seconds worker 1 buried after 1 seconds 2014-11-19 15:45:15.629073 still alive: {21371: 55} 2014-11-19 15:45:15.629531 sent hup 21371 

note: inserted printf "the process id %d" before "sigint/sigquit received...killing workers..." see process gets signal

21371 pid of worker, stopped respond sighup long before ctrl+c

sometime workers try handle sigint/sigquit without becoming not responsive sighup, think problems partly related.