Bug #63668

Bug #63692: Memory consumption while bulk inserting

High memory consumption in DataHandler->fillInFieldArray while bulk inserting

Added by Stephan Großberndt 8 months ago. Updated 8 months ago.

Status:Resolved Start date:2014-12-08
Priority:Must have Due date:
Assigned To:- % Done:

100%

Category:Performance Spent time: -
Target version:next-patchlevel
TYPO3 Version:6.2 Is Regression:No
PHP Version: Sprint Focus:
Complexity:easy

Description

DataHandler->fillInFieldArray calls

$eFile = \TYPO3\CMS\Core\Html\RteHtmlParser::evalWriteFile($vconf['spec']['static_write'], array_merge($currentRecord, $fieldArray));

for each field of each record to insert. In each call array_merge($currentRecord, $fieldArray) is done only to throw it away again immediately and do nothing in most cases - does anyone use staticFileEditPath? I dont't think so: http://forum.typo3.org/index.php/t/195196/

$types_fieldConfig = BackendUtility::getTCAtypes($table, $currentRecord);
$theTypeString = BackendUtility::getTCAtypeValue($table, $currentRecord);
if (is_array($types_fieldConfig)) {
    foreach ($types_fieldConfig as $vconf) {
        // Write file configuration:
        // inserted array_merge($currentRecord,$fieldArray) 170502
        $eFile = \TYPO3\CMS\Core\Html\RteHtmlParser::evalWriteFile($vconf['spec']['static_write'], array_merge($currentRecord, $fieldArray));
        // RTE transformations:
        if (!$this->dontProcessTransformations) {
}

Instead of unconditionally merging and throwing away again only merge if static_write is enabled

$types_fieldConfig = BackendUtility::getTCAtypes($table, $currentRecord);
$theTypeString = BackendUtility::getTCAtypeValue($table, $currentRecord);
if (is_array($types_fieldConfig)) {
    foreach ($types_fieldConfig as $vconf) {
        if (is_array($vconf['spec']['static_write']) && $GLOBALS['TYPO3_CONF_VARS']['BE']['staticFileEditPath']) {
            // only do array_merge if static_write is enabled
            $eFile = \TYPO3\CMS\Core\Html\RteHtmlParser::evalWriteFile($vconf['spec']['static_write'], array_merge($currentRecord, $fieldArray));
        } else {
            $eFile = null;
        }
        // RTE transformations:
        if (!$this->dontProcessTransformations) {

I think even more easy optimizations are possible here (keep array_merge and do not create again in $mixedRec = array_merge($currentRecord, $fieldArray);) but this is only executed if static_write is enabled anyway.

For bulk insert of 6500 records:

  • 98000 calls less
  • 0.5 seconds faster
  • memory usage of array_merge()-Function down from 128MB to 6MB

Related issues

related to Core - Task #63818: Remove staticFileEditPath magic Resolved 2014-12-12

Associated revisions

Revision c1965e21
Added by Stephan Großberndt 8 months ago

[BUGFIX] Better performance for bulk inserts without static_write

In DataHandler->fillInFieldArray for every record to insert or update
current and new data are merged to be handed over to the static_write
functionality. If this functionality is disabled (which is the
default) the merged data is discarded again.

Checks if the static_write functionality is enabled and merges data
only if it is enabled. This drastically lowers memory usage for bulk
operations.

In master, the static_write functionality was removed completely,
for 6.2 the performance of this code section is improved.

Resolves: #63668
Releases: 6.2
Change-Id: I625a771a98a5f79bab3a8d22464895fcbdcf4068
Reviewed-on: http://review.typo3.org/35370
Reviewed-by: Christian Kuhn <>
Tested-by: Christian Kuhn <>
Reviewed-by: Anja Leichsenring <>
Tested-by: Anja Leichsenring <>

Revision 4dd99fee
Added by Christian Kuhn 8 months ago

[!!!][TASK] Remove static file edit magic

Configuring RTE fields TCA in defaultExtras section with static_write
enabled the DataHandler to write content to specific sections of files
in a specific directory.

This feature is documented and existed for a long time, but is not used
in the wild. Searching for keywords just revealed a very outdated
demo page, the according core source and documentation. No extension
or real live use case was found and even a call for feedback on this
by Benni half a year ago got no answers.

The patch removes this feature without substitution and improves
DataHandler performance as a nice side effect.

Resolves: #63818
Resolves: #63668
Releases: master
Change-Id: I5532ba1565370baeb03ce074c8ab8036a0936d5b
Reviewed-on: http://review.typo3.org/35369
Reviewed-by: Anja Leichsenring <>
Tested-by: Anja Leichsenring <>
Reviewed-by: Markus Klein <>
Reviewed-by: Helmut Hummel <>
Reviewed-by: Wouter Wolters <>
Tested-by: Benjamin Mack <>
Reviewed-by: Christian Kuhn <>
Tested-by: Christian Kuhn <>

History

#1 Updated by Philipp Gampe 8 months ago

  • Parent task set to #63692

#2 Updated by Gerrit Code Review 8 months ago

  • Status changed from New to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35192

#3 Updated by Gerrit Code Review 8 months ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35192

#4 Updated by Gerrit Code Review 8 months ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35369

#5 Updated by Gerrit Code Review 8 months ago

Patch set 1 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35370

#6 Updated by Gerrit Code Review 8 months ago

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35369

#7 Updated by Christian Kuhn 8 months ago

the patch for master was abandoned, the issue is solved with #63818
for 6.2, the patch was re-pushed.

#8 Updated by Gerrit Code Review 8 months ago

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35369

#9 Updated by Stephan Großberndt 8 months ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100

#10 Updated by Gerrit Code Review 8 months ago

  • Status changed from Resolved to Under Review

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35192

#11 Updated by Christian Kuhn 8 months ago

  • Status changed from Under Review to Resolved

Also available in: Atom PDF