Core

I found a memory consumption problem in

TYPO3\CMS\Core\DataHandling\DataHandler::processClearCacheQueue()

due to this dubious construct, which is called for each and every record added:

foreach (static::$recordsToClearCacheFor as $table => $uids) {
  foreach (array_unique($uids) as $uid) {
    ...

    foreach ($pageIdsThatNeedCacheFlush as $pageId) {
      if ($pageId >= 0) {
        $tagsToClear[] = 'pageId_' . $pageId;
      }
    }
    // Queue delete cache for current table and record
    $tagsToClear[] = $table;
    $tagsToClear[] = $table . '_' . $uid;
    $tagsToClear = array_unique($tagsToClear);
  }
}

I am adding 6500 records, so FOR EACH AND EVERY record 'pageId_' . $pageId, $table and $table . '_' . $uid are added to $tagsToClear and afterwards 'pageId_' . $pageId and $table are removed again for 6499 times. There is no reason to do it like this. array_unique() is meant here to reduce memory consumption in $tagsToClear but for big arrays with array_unique() this is terrible - xhprof showed 2,020,470,496 bytes (~2 gigabyte !!!) of used memory for array_unique() in processClearCacheQueue()

It is possible to set

clearCache_disable=1

to disable clearing of the cache but there is also a simple code solution to solve this since entries in this array are meant to be unique anyway:

foreach (static::$recordsToClearCacheFor as $table => $uids) {
  foreach (array_unique($uids) as $uid) {
    ...

    foreach ($pageIdsThatNeedCacheFlush as $pageId) {
      if ($pageId >= 0) {
        $tagsToClear['pageId_' . $pageId] = 'pageId_' . $pageId;
      }
    }
    // Queue delete cache for current table and record
    $tagsToClear[$table] = $table;
    $tagsToClear[$table . '_' . $uid] = $table . '_' . $uid;
  }
}

array_unique memory consumption drops from

IMemUse%: 13064.1%
Incl. MemUse (bytes): 2,020,470,496 bytes (~2 gigabyte !!!)

to

IMemUse%: 4.1%
Incl. MemUse (bytes): 629,464 bytes

There may be optimizations possible like

foreach (static::$recordsToClearCacheFor as $table => $uids) {
  foreach (array_unique($uids) as $uid) {
    ...

    foreach ($pageIdsThatNeedCacheFlush as $pageId) {
      $pageKey = 'pageId_' . $pageId;
      if ($pageId >= 0 && !isset($tagsToClear[$pageKey])) {
        $tagsToClear[$pageKey] = $pageKey;
      }
    }
    // Queue delete cache for current table and record
    if (!isset($tagsToClear[$table])) {
      $tagsToClear[$table] = $table;
    }
    $tagsToClear[$table . '_' . $uid] = $table . '_' . $uid;
  }
}

but even without those its SOOOO much better...

Regards,
Stephan