In the last post, we walked through how to install HipHop PHP on a Ubuntu 13.04 EC2. Well thats great but it now leads us to the question of how fast HipHop PHP actually is. The problem with “toy benchmarks” is they tend to not really capture the real performance characteristics of whatever you’re benchmarking. This is why comparing the performance of a “Hello World” app across various languages and frameworks is generally a waste of time, since its not capturing a real world scenario. Luckily, I actually have some “real world”’ish benchmarks from my PHP: Does “big-o” complexity really matter? post a couple of months ago.
Ok so great, lets checkout the repository, run the benchmark with HipHop and Zend PHP, and then marvel at how HipHop blows Zend PHP out of the water.
$ git clone git@github.com:Setfive/setfive.github.com.git
$ cd setfive.github.com/autocomplete_test/
$ ~/dev/hiphop-php/hphp/hhvm/hhvm --hphp -thhbc -o some_dir autocompleteBenchmark.php algorithms.php
$ ~/dev/hiphop-php/hphp/hhvm/hhvm -vRepo.Authoritative=true -vRepo.Central.Path=some_dir/hhvm.hhbc -vEval.Jit=1 autocompleteBenchmark.php
test_name, min, max, average
readMemoryScan,9.4527468681335, 9.7483551502228, 9.6144312620163
unsortedTableScan,9.2616581916809, 9.5636370182037, 9.3831548213959
sortedTableScan,4.0028560161591, 4.2003200054169, 4.1263886928558
serializednFileScan,0.12966299057007, 0.1957049369812, 0.14499731063843
$ php autocompleteBenchmark.php
test_name, min, max, average
readMemoryScan,1.1715579032898, 1.3546500205994, 1.2789875268936
unsortedTableScan,0.98337507247925, 1.0579788684845, 1.0203387260437
sortedTableScan,0.41418695449829, 0.54504704475403, 0.47947511672974
serializednFileScan,0.25355696678162, 0.31970500946045, 0.27470426559448
Wtf?
Well so that is weird, in 3 out of the 4 tests HipHop is an order of magnitude slower than Zend PHP. Clearly, something is definitely not right. I double checked the commands and everything is being run correctly. I started debugging the readMemoryScan function on HipHop specifically and it turns out that the problem function is actually str_getcsv. I decided to remove that function as well as the array_maps() since I wasn’t sure if HipHop would be able to optimize given the anonymous function being passed in. The new algorithms file is algorithms_hiphop.php which has str_getcsv replaced with an explode and array_map replaced with a loop.
Running the same benchmarks again except with the new algorithms file gives you:
$ ~/dev/hiphop-php/hphp/hhvm/hhvm -vRepo.Authoritative=true -vRepo.Central.Path=some_dir/hhvm.hhbc -vEval.Jit=1 autocompleteBenchmark.php
test_name, min, max, average
readMemoryScan,0.35798001289368, 0.43651509284973, 0.37085726261139
unsortedTableScan,0.25601601600647, 0.31864595413208, 0.27621953487396
sortedTableScan,0.12703394889832, 0.18750810623169, 0.13350143432617
serializednFileScan,0.13122606277466, 0.19621300697327, 0.15197308063507
$ php autocompleteBenchmark.php
test_name, min, max, average
readMemoryScan,0.52727389335632, 0.70799994468689, 0.64662590026855
unsortedTableScan,0.41211080551147, 0.48418807983398, 0.46322596073151
sortedTableScan,0.20118999481201, 0.2757511138916, 0.22925112247467
serializednFileScan,0.2424430847168, 0.36282706260681, 0.30896728038788
Wow. So the HipHop implementation is clearly faster but what’s even more surprising is that the Zend PHP implementation gains a significant speedup just by removing str_getcsv and array_map.
Anyway, as expected, HipHop is a faster implementation most likely due to its JIT compilation and additional optimizations that it’s able to add along the way.
Despite the speedup though, Facebook has made it clear that HipHop will only support a subset of the PHP language, notably that the dynamic features will never be implemented. At the end of the day, its not clear if HipHop will gain any mainstream penetration but hopefully it’ll push Zend to keep improving their interpreter and potentially incorporate some of HipHop’s JIT features.
Per Dan’s comment below, HHVM currently supports almost all of PHP 5.5s features.
Why is str_getcsv so slow?
Well benchmarks are all fine and well but I was curious why str_getcsv was so slow on both Zend and HipHop. Digging around, the HipHop implementation looks like:
<?php
function str_getcsv(
$input,
$delimiter = ',',
$enclosure = '"',
$escape = '\\',
) {
$args = array($delimiter, $enclosure, $escape);
$defaults = array(',', '"', '\\');
$args_len = count($args);
for ($i = 0; $i < $args_len; ++$i) {
$arg_len = strlen($args[$i]);
// fgetcsv returns false if passed anything but a single char for its
// last three args. str_getcsv's behavior is to truncate strings down
// to their first char, and to turn invalid args into the default args.
if ($arg_len === 0) {
$args[$i] = $defaults[$i];
} else if ($arg_len > 1) {
$args[$i] = substr($args[$i], 0, 1);
}
}
$temp = tmpfile();
fwrite($temp, $input);
fseek($temp, 0);
$ret = fgetcsv($temp, 0, $args[0], $args[1], $args[2]);
fclose($temp);
return $ret !== false ? $ret : array(null);
}
So basically just a wrapper around fgetcsv that works by writing the string to a temporary file. I’d expect file operations to be slow but I’m still surprised they’re that slow.
Anyway, looking at the Zend implementation it’s a native C function that calls into php_fgetcsv but doesn’t use temporary files.
/* {{{ proto array str_getcsv(string input[, string delimiter[, string enclosure[, string escape]]])
Parse a CSV string into an array */
PHP_FUNCTION(str_getcsv)
{
char *str, delim = ',', enc = '"', esc = '\\';
char *delim_str = NULL, *enc_str = NULL, *esc_str = NULL;
int str_len = 0, delim_len = 0, enc_len = 0, esc_len = 0;
if (zend_parse_parameters(ZEND_NUM_ARGS() TSRMLS_CC, "s|sss", &str, &str_len, &delim_str, &delim_len,
&enc_str, &enc_len, &esc_str, &esc_len) == FAILURE) {
return;
}
delim = delim_len ? delim_str[0] : delim;
enc = enc_len ? enc_str[0] : enc;
esc = esc_len ? esc_str[0] : esc;
php_fgetcsv(NULL, delim, enc, esc, str_len, str, return_value TSRMLS_CC);
}
/* }}} */
PHPAPI void php_fgetcsv(php_stream *stream, char delimiter, char enclosure, char escape_char, size_t buf_len, char *buf, zval *return_value TSRMLS_DC) /* {{{ */
{
char *temp, *tptr, *bptr, *line_end, *limit;
size_t temp_len, line_end_len;
int inc_len;
zend_bool first_field = 1;
/* initialize internal state */
php_mblen(NULL, 0);
/* Now into new section that parses buf for delimiter/enclosure fields */
/* Strip trailing space from buf, saving end of line in case required for enclosure field */
bptr = buf;
tptr = (char *)php_fgetcsv_lookup_trailing_spaces(buf, buf_len, delimiter TSRMLS_CC);
line_end_len = buf_len - (size_t)(tptr - buf);
line_end = limit = tptr;
/* reserve workspace for building each individual field */
temp_len = buf_len;
temp = emalloc(temp_len + line_end_len + 1);
/* Initialize return array */
array_init(return_value);
/* Main loop to read CSV fields */
/* NB this routine will return a single null entry for a blank line */
do {
char *comp_end, *hunk_begin;
tptr = temp;
inc_len = (bptr < limit ? (*bptr == '\0' ? 1: php_mblen(bptr, limit - bptr)): 0);
if (inc_len == 1) {
char *tmp = bptr;
while (isspace((int)*(unsigned char *)tmp)) {
tmp++;
}
if (*tmp == enclosure) {
bptr = tmp;
}
}
if (first_field && bptr == line_end) {
add_next_index_null(return_value);
break;
}
first_field = 0;
/* 2. Read field, leaving bptr pointing at start of next field */
if (inc_len != 0 && *bptr == enclosure) {
int state = 0;
bptr++; /* move on to first character in field */
hunk_begin = bptr;
/* 2A. handle enclosure delimited field */
for (;;) {
switch (inc_len) {
case 0:
switch (state) {
case 2:
memcpy(tptr, hunk_begin, bptr - hunk_begin - 1);
tptr += (bptr - hunk_begin - 1);
hunk_begin = bptr;
goto quit_loop_2;
case 1:
memcpy(tptr, hunk_begin, bptr - hunk_begin);
tptr += (bptr - hunk_begin);
hunk_begin = bptr;
/* break is omitted intentionally */
case 0: {
char *new_buf;
size_t new_len;
char *new_temp;
if (hunk_begin != line_end) {
memcpy(tptr, hunk_begin, bptr - hunk_begin);
tptr += (bptr - hunk_begin);
hunk_begin = bptr;
}
/* add the embedded line end to the field */
memcpy(tptr, line_end, line_end_len);
tptr += line_end_len;
if (stream == NULL) {
goto quit_loop_2;
} else if ((new_buf = php_stream_get_line(stream, NULL, 0, &new_len)) == NULL) {
/* we've got an unterminated enclosure,
* assign all the data from the start of
* the enclosure to end of data to the
* last element */
if ((size_t)temp_len > (size_t)(limit - buf)) {
goto quit_loop_2;
}
zval_dtor(return_value);
RETVAL_FALSE;
goto out;
}
temp_len += new_len;
new_temp = erealloc(temp, temp_len);
tptr = new_temp + (size_t)(tptr - temp);
temp = new_temp;
efree(buf);
buf_len = new_len;
bptr = buf = new_buf;
hunk_begin = buf;
line_end = limit = (char *)php_fgetcsv_lookup_trailing_spaces(buf, buf_len, delimiter TSRMLS_CC);
line_end_len = buf_len - (size_t)(limit - buf);
state = 0;
} break;
}
break;
case -2:
case -1:
php_mblen(NULL, 0);
/* break is omitted intentionally */
case 1:
/* we need to determine if the enclosure is
* 'real' or is it escaped */
switch (state) {
case 1: /* escaped */
bptr++;
state = 0;
break;
case 2: /* embedded enclosure ? let's check it */
if (*bptr != enclosure) {
/* real enclosure */
memcpy(tptr, hunk_begin, bptr - hunk_begin - 1);
tptr += (bptr - hunk_begin - 1);
hunk_begin = bptr;
goto quit_loop_2;
}
memcpy(tptr, hunk_begin, bptr - hunk_begin);
tptr += (bptr - hunk_begin);
bptr++;
hunk_begin = bptr;
state = 0;
break;
default:
if (*bptr == enclosure) {
state = 2;
} else if (*bptr == escape_char) {
state = 1;
}
bptr++;
break;
}
break;
default:
switch (state) {
case 2:
/* real enclosure */
memcpy(tptr, hunk_begin, bptr - hunk_begin - 1);
tptr += (bptr - hunk_begin - 1);
hunk_begin = bptr;
goto quit_loop_2;
case 1:
bptr += inc_len;
memcpy(tptr, hunk_begin, bptr - hunk_begin);
tptr += (bptr - hunk_begin);
hunk_begin = bptr;
break;
default:
bptr += inc_len;
break;
}
break;
}
inc_len = (bptr < limit ? (*bptr == '\0' ? 1: php_mblen(bptr, limit - bptr)): 0);
}
quit_loop_2:
/* look up for a delimiter */
for (;;) {
switch (inc_len) {
case 0:
goto quit_loop_3;
case -2:
case -1:
inc_len = 1;
php_mblen(NULL, 0);
/* break is omitted intentionally */
case 1:
if (*bptr == delimiter) {
goto quit_loop_3;
}
break;
default:
break;
}
bptr += inc_len;
inc_len = (bptr < limit ? (*bptr == '\0' ? 1: php_mblen(bptr, limit - bptr)): 0);
}
quit_loop_3:
memcpy(tptr, hunk_begin, bptr - hunk_begin);
tptr += (bptr - hunk_begin);
bptr += inc_len;
comp_end = tptr;
} else {
/* 2B. Handle non-enclosure field */
hunk_begin = bptr;
for (;;) {
switch (inc_len) {
case 0:
goto quit_loop_4;
case -2:
case -1:
inc_len = 1;
php_mblen(NULL, 0);
/* break is omitted intentionally */
case 1:
if (*bptr == delimiter) {
goto quit_loop_4;
}
break;
default:
break;
}
bptr += inc_len;
inc_len = (bptr < limit ? (*bptr == '\0' ? 1: php_mblen(bptr, limit - bptr)): 0);
}
quit_loop_4:
memcpy(tptr, hunk_begin, bptr - hunk_begin);
tptr += (bptr - hunk_begin);
comp_end = (char *)php_fgetcsv_lookup_trailing_spaces(temp, tptr - temp, delimiter TSRMLS_CC);
if (*bptr == delimiter) {
bptr++;
}
}
/* 3. Now pass our field back to php */
*comp_end = '\0';
add_next_index_stringl(return_value, temp, comp_end - temp, 1);
} while (inc_len > 0);
out:
efree(temp);
if (stream) {
efree(buf);
}
}
/* }}} */
Looking at the actual implementation of php_fgetcsv though its not surprising its significantly slower compared to explode().
