search.conf paramter max_indexing_buffer_size doesn't seem to work

edited March 19 in General Support

We are using CA 1.7.9. We've got a big collection of 20.000 objects and are using sql search for it.

By rebuilding the search index with CaUtils we experienced an error "PHP Fatal error: Allowed memory size of XXXXXX bytes exhausted". We tried to extend the memory by ini_set('memory_limit', '8000M'), and that worked well for a while, but our collection has grown and still is.

It seems that the max_indexing_buffer_size parameter in search.conf is not taken. We can't see any difference while rebuilding the search index.

Rebuilding the search index of the tables one by one helped. But some strange stuff is going on after doing that and we had major issues with searching.

Is there a better way to safe memory while rebuilding the search index?

Comments

  • All max_indexing_buffer_size does is control the size of the in-memory buffer used to store indexing information before it is written out to the database. The lower it is, the more frequent writes to the database will be.

    I suspect your memory issue is due to something else. Does this happen on the test systems as well as production? If so, I can go in and take a look.


    seth

  • I have resolved memory issues in the past with reindexing my modifying /app/lib/Search/SearchIndexer.php

    in public function reindex($pa_table_names=null, $pa_options=null) {

    ~ line 247-248

    change:

    if (!($vn_i % 500)) { // Pre-load attribute values for next 500 items to index; improves index performance

    $va_id_slice = array_slice($va_ids, $vn_i, 500);

    to a lower number such as 200:

    if (!($vn_i % 200)) { // Pre-load attribute values for next 500 items to index; improves index performance

    $va_id_slice = array_slice($va_ids, $vn_i, 200);

  • @brucek Changing the number of records used for preloading would only help, maybe, on a server with very limited memory. This is not the problem here.

    The problem here is pretty strange. One one machine, running PHP 7.4.16, the indexing quickly eats all available memory. The identical configuration on the test server works fine. After looking at the running instance, it turns out that for some reason PHP is not reclaiming memory used by internal settings objects generated by part of CA that handle list items. No idea why this is happening only on one machine and not the other, given that they appear identical in all respects. Perhaps some of the underlying libraries are different ... both PHP 7.4.16 are from the remi repository, so PHP at least should be literally identical.

    I have a work around for this issue, which changes how settings object are created, which is being tested now. I'm not sure how prevalent this issue is, given that we've only had one report so far (this one), but it seems like a good idea to patch for it just in case.


    seth

  • edited May 18

    Seth, I am having a similar issue (MySql server has gone away) with 1.7.11 (PHP 7.4.3) - see attached image.

    Previous version (providence 1.7.8 with PHP 7.1.27-1) is working fine. More than a Providence error, it seems it is MySql memory that get consumed until service crashes and restarts automatically.

    Tried to reconfigure MySql memory parameters, but nothing changes.

    Giovanni

    1208 x 295 - 76K
  • This isn't a PHP memory error, it's an issue with MySQL. Take a look in the mysql log and see if there's a clue as to why it crashed.

  • Done. Did not find anything particular. It crashes.

    I tried several times modifying mysql parameters.

    Now I have found a clue.

    It seems it could be that php app does not close used connections, so increasing memory usage over time up to saturate it. I now set wait_timeout to 900 instead 28800 (seconds) hoping that any outstanding connection will be closed by mysql itself.

    The point is that I get the error 12 hour after the beginning of reindexing process (it takes more than 1 day to complete), so I can make only one test per day.

    Tomorrow we will see...

  • Nothing new. Still got an error.

    Trying a migration to Mysql 8

  • Fixed using MySQL 8.

Sign In or Register to comment.