php5.2of curl-Bug Server Stuck on php Process Troubleshooting

A few days ago, Dongzheng classmate reported that the Linode server was almost frozen, so I took some time today to investigate the specific cause, and the ultimate cause was somewhat heroic: file_get_contents did not set a timeout, plus the php5.2There is a bug in the curl code, which led to the PHP process entering a deadlock loop.

This afternoon, it was found that the system load was very high, so I went to take a look and found a large group of PHP processes had not exited, occupying a lot of CPU, as shown in the figure:

Problem process:

The script running later is my RSS scheduled update task, it seems there is a problem with somewhere in the PHP code, so strace -p 14043Looked at it:

select(5, [4], [4], [], {15, 0}) = 1 (out [4], left {14, 999996)
poll([{fd=4, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
clock_gettime(CLOCK_MONOTONIC, {4582888, 760370017}) = 0
clock_gettime(CLOCK_MONOTONIC, {4582888, 760468615}) = 0
clock_gettime(CLOCK_MONOTONIC, {4582888, 760565053}) = 0
select(5, [4], [4], [], {15, 0}) = 1 (out [4], left {14, 999997)

There was a dead loop on fd 4, so let's see what FD is: ll /proc/14043/fd

lrwx—— 1 wuhaiwen wuhaiwen 64　 7Month 21 11:00 4 -> socket:[53176380]

Looking again, it turned out that there was a dead loop when requesting a webpage from CSDN, but did not know where the request was made. Thought of using GDB to check the php process, bt shows:

(gdb) bt
#0 0x00007f6721f8f013 in __select_nocancel () at ../sysdeps/unix/syscall-template.S:82
#1 0×0000000000481952 in php_curl_stream_read (stream=0×2280650,
buf=0x22ea5d0 “2Fwww.laruence.com%2Ftag%2F%25e6%25ad%25a3%25e5%2588%2599%27+class%3D%27tag-link-191%27+title%3D%273+topics%27+style%3D%27font-size%3A+9.0243902439pt%3B%27%3E%E6%AD%A3%E5%88%99%3C%2Fa%3E%3C%2Ftags%3E\”"…, count=8192) at /home/wuhaiwen/install/php-env/src/php/php-5.2.8/ext/curl/streams.c:169
#2 0x00000000006738f9 in php_stream_fill_read_buffer (stream=0×2280650, size=4283) at /home/wuhaiwen/install/php-env/src/php/php-5.2.8/main/streams/streams.c:554
#3 0x0000000000673c39 in _php_stream_read (stream=0×2280650,
buf=0x2301fd5 “f='http://www.laruence.com/tag/json' class='tag-link-79′ title='3 topics' style='font-size: 9.0243902439pt;'>json</a>\n<a href='http://www.laruence.com/tag/module' class='tag-link-43′ title='2 topics' “…, size=4283) at /home/wuhaiwen/install/php-env/src/php/php-5.2.8/main/streams/streams.c:600
#4 0x0000000000674c51 in _php_stream_copy_to_mem (src=0×2280650, buf=0x7fff376ed898, maxlen=<optimized out>, persistent=0)
at /home/wuhaiwen/install/php-env/src/php/php-5.2.8/main/streams/streams.c:1267
#5 0x00000000005fdb85 in zif_file_get_contents (ht=<optimized out>, return_value=0x2223da0, return_value_ptr=<optimized out>, this_ptr=<optimized out>, return_value_used=<optimized out>)
at /home/wuhaiwen/install/php-env/src/php/php-5.2.8/ext/standard/file.c:565
#6 0x00000000006c2a59 in zend_do_fcall_common_helper_SPEC (execute_data=0x7fff376edc60) at /home/wuhaiwen/install/php-env/src/php/php-5.2.8/Zend/zend_vm_execute.h:200
#7 0x00000000006c239f in execute (op_array=0x1f26730) at /home/wuhaiwen/install/php-env/src/php/php-5.2.8/Zend/zend_vm_execute.h:92
·············
#16 0x0000000000730d8e in main (argc=4, argv=0x7fff376f2468) at /home/wuhaiwen/install/php-env/src/php/php-5.2.8/sapi/cli/php_cli.c:1133

Look at what PHP script is currently being executed:

(gdb) p *op_array
$4 = {type = 2 '\002', function_name = 0x1e54278 "getContent", scope = 0x1f8e850, fn_flags = 257, prototype = 0x0, num_args = 2, required_num_args = 1, arg_info = 0x1fd5e20,
pass_rest_by_reference = 0 '\000', return_reference = 0 '\000', refcount = 0x1fd3ab8, opcodes = 0x1fddcc8, last = 28, size = 28, vars = 0x1fd3cc0, last_var = 6, size_var = 16, T = 15,
brk_cont_array = 0x0, last_brk_cont = 0, current_brk_cont = 4294967295, try_catch_array = 0x0, last_try_catch = 0, static_variables = 0x0, start_op = 0x0, backpatch_count = 0,
done_pass_two = 1 '\001', uses_this = 0 '\000', filename = 0x1fd3b58 "/home/wuhaiwen/webroot/kulvrss/libs/Myrss/Model/UrlContenter.php", line_start = 9, line_end = 30, doc_comment = 0x0,
doc_comment_len = 0, reserved = {0x0, 0x0, 0x0, 0x0}}

Found the location of the problematic code, it turns out to be a call to file_get_contents($url) without setting a timeout time, so PHP was stuck on the network request. Therefore, stream_context_create was used to set the timeout time to solve the problem.

Here, it seems that the problem is solved, but why did the php process consume so much CPU and the system load was so high without setting a timeout time? In theory, it should wait for I/O is it? Looking at the CPU situation above, it seems to be in a dead loop rhythm.

Based on the above bt stack, let's first look at the call to the second-to-last function:

#1 0×0000000000481952 in php_curl_stream_read (stream=0×2280650,
buf=0x22ea5d0 “2Fwww.laruence.com%2Ftag%2F%25e6%25ad%25a3%25e5%2588%2599%27+class%3D%27tag-link-191%27+title%3D%273+topics%27+style%3D%27font-size%3A+9.0243902439pt%3B%27%3E%E6%AD%A3%E5%88%99%3C%2Fa%3E%3C%2Ftags%3E\”"…, count=8192) at /home/wuhaiwen/install/php-env/src/php/php-5.2.8/ext/curl/streams.c:169

Let's take a look at the code, I used:5.2.8The version of PHP is relatively old. The code is as follows:

static size_t php_curl_stream_read(php_stream *stream, char *buf, size_t count TSRMLS_DC)
{
    php_curl_stream *curlstream = (php_curl_stream *) stream->abstract;
    size_t didread = 0;
    if (curlstream->readbuffer.readpos >= curlstream->readbuffer.writepos && curlstream->pending) {
//········
        do {
            /* get the descriptors from curl */
            curl_multi_fdset(curlstream->multi, &curlstream->readfds, &curlstream->writefds, &curlstream->excfds, &curlstream->maxfd);
            /* if we are in blocking mode, set a timeout */
            tv.tv_usec = 0;
            tv.tv_sec = 15; /* TODO: allow this to be configured from the script */
            /* wait for data */
            switch (select(curlstream->maxfd + 1, &curlstream->readfds, &curlstream->writefds, &curlstream->excfds, &tv)) {
                case -1:
                    /* error */
                    return 0;
                case 0:
                    /* no data yet: timed-out */
                    return 0;
                default:
                    /* fetch the data */
                    do {
                        curlstream->mcode = curl_multi_perform(curlstream->multi, &curlstream->pending);
                    } while (curlstream->mcode == CURLM_CALL_MULTI_PERFORM);
            }
        } while (curlstream->readbuffer.readpos >= curlstream->readbuffer.writepos && curlstream->pending > 0);
    }
//··········
    return didread;
}

GDB进去发现，代码一直在里面的do-while循环里面！心想curl_multi_fdset怎么不用先FD_ZERO清空FD呢？一般做法都是会先清空的。

莫非是PHP的bug，于是网上找了一下发现了这个Pierrick-Charron的提交确实是一个bug，其实curl_multi_fdset的文档开头写了：

此功能从给定的多句柄中提取文件描述符信息。libcurl返回其fd_set集合。应用程序可以使用这些集合在select()上操作，但请在调用此函数之前确保FD_ZERO它们，因为curl_multi_fdset(3) only adds its own descriptors,

Alright, finally verify with GDB, I manually cleared the fd below the do above, before the curl_multi_fdset call, to see if it can exit the loop:

(gdb) print FD_ZERO(&curlstream->readfds)
No symbol 'FD_ZERO' in current context.

FD_ZERO is not available, never mind, it's originally a macro definition, just expand it: #define FD_ZERO(p) bzero((char *)(p), sizeof(*(p)))

Directly modify the three parameter arrays of curl_muti_fdset with call as follows:

(gdb) call bzero((char *)(&curlstream->readfds), sizeof(*(&curlstream->readfds)))
$5 = 17055392
(gdb) call bzero((char *)(&curlstream->writefds), sizeof(*(&curlstream->writefds)))
$6 = 17055520
(gdb) call bzero((char *)(&curlstream->excfds), sizeof(*(&curlstream->excfds)))
$7 = 17055648

Then execute GDB step by step, as expected due to curlstream-pending changes to 0, thus exiting the loop, back to the large function php_stream_fill_read_buffer

This is basically over. The PHP version with the problem should be5.2Specifically, it was not carefully reviewed, readers can refer to the above submission changes or directly check their own version code for any issues.