Bug #63666

Bug #63692: Memory consumption while bulk inserting

High memory consumption in BackendUtility->getTCAtypes while bulk inserting

Added by Stephan Großberndt 8 months ago. Updated 8 months ago.

Status:Resolved Start date:2014-12-08
Priority:Must have Due date:
Assigned To:- % Done:

100%

Category:Performance Spent time: -
Target version:next-patchlevel
TYPO3 Version:6.2 Is Regression:No
PHP Version: Sprint Focus:
Complexity:easy

Description

DataHandler->fillInFieldArray calls

$types_fieldConfig = BackendUtility::getTCAtypes($table, $currentRecord);
for each record to insert. In each call the type of the current record is checked and the complete types configuration is parsed again.

static public function getTCAtypes($table, $rec, $useFieldNameAsKey = 0) {
    if ($GLOBALS['TCA'][$table]) {
    // Get type value:
    $fieldValue = self::getTCAtypeValue($table, $rec);
    // Get typesConf
    $typesConf = $GLOBALS['TCA'][$table]['types'][$fieldValue];
    // Get fields list and traverse it
    $fieldList = explode(',', $typesConf['showitem']);
    $altFieldList = array();
    // Traverse fields in types config and parse the configuration into a nice array:
    foreach ($fieldList as $k => $v) {
        [...]
    }
    if ($useFieldNameAsKey) {
        $fieldList = $altFieldList;
    }
    // Return array:
    return $fieldList;
}

A new static cache using tablename, type and parameter as key improves performance and memory consumption:

static public function getTCAtypes($table, $rec, $useFieldNameAsKey = 0) {
    static $tcaTableTypeConf_cache = array();

    if ($GLOBALS['TCA'][$table]) {
    // Get type value:
    $fieldValue = self::getTCAtypeValue($table, $rec);

    if ($tcaTableTypeConf_cache[$table . 'type' . $fieldValue . 'fnk' . $useFieldNameAsKey]) {
        return $tcaTableTypeConf_cache[$table . 'type' . $fieldValue . 'fnk' . $useFieldNameAsKey];
    }

    // Get typesConf
    $typesConf = $GLOBALS['TCA'][$table]['types'][$fieldValue];
    // Get fields list and traverse it
    $fieldList = explode(',', $typesConf['showitem']);
    $altFieldList = array();
    // Traverse fields in types config and parse the configuration into a nice array:
    foreach ($fieldList as $k => $v) {
        [...]
    }
    if ($useFieldNameAsKey) {
        $fieldList = $altFieldList;
    }

    $tcaTableTypeConf_cache[$table . 'type' . $fieldValue . 'fnk' . $useFieldNameAsKey] = $fieldList;
    // Return array:
    return $fieldList;
}
For bulk insert of 6500 typeless (only type '1') records:
  • 465000 calls less
  • 3.3 seconds faster
  • memory usage of explode()-Function down from 128MB to 58MB

Associated revisions

Revision e0467960
Added by Stephan Großberndt 8 months ago

[BUGFIX] Better caching of TCA for bulk inserts

Introduces a runtime cache for the TCA types configuration. This
improves performance for bulk inserts since getTCAtypes is called for
each record of each table and the available types during a call do not
change.

Resolves: #63666
Releases: master, 6.2
Change-Id: Ia58352f1bae1e1eb57b87e5db0f1ad84e34ae37e
Reviewed-on: http://review.typo3.org/35185
Reviewed-by: Alexander Opitz <>
Tested-by: Alexander Opitz <>
Reviewed-by: Georg Ringer <>
Tested-by: Georg Ringer <>
Reviewed-by: Christian Kuhn <>
Tested-by: Christian Kuhn <>

Revision 87ec9a59
Added by Stephan Großberndt 8 months ago

[BUGFIX] Better caching of TCA for bulk inserts

Introduces a runtime cache for the TCA types configuration. This
improves performance for bulk inserts since getTCAtypes is called for
each record of each table and the available types during a call do not
change.

Resolves: #63666
Releases: master, 6.2
Change-Id: Ia58352f1bae1e1eb57b87e5db0f1ad84e34ae37e
Reviewed-on: http://review.typo3.org/35521
Reviewed-by: Christian Kuhn <>
Tested-by: Christian Kuhn <>

History

#1 Updated by Philipp Gampe 8 months ago

  • Parent task set to #63692

#2 Updated by Gerrit Code Review 8 months ago

  • Status changed from New to Under Review

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35185

#3 Updated by Gerrit Code Review 8 months ago

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35185

#4 Updated by Gerrit Code Review 8 months ago

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35185

#5 Updated by Gerrit Code Review 8 months ago

Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35185

#6 Updated by Gerrit Code Review 8 months ago

Patch set 7 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35185

#7 Updated by Gerrit Code Review 8 months ago

Patch set 8 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35185

#8 Updated by Gerrit Code Review 8 months ago

Patch set 9 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35185

#9 Updated by Gerrit Code Review 8 months ago

Patch set 1 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35521

#10 Updated by Stephan Großberndt 8 months ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100

Also available in: Atom PDF