Bug #63666
Bug #63692: Memory consumption while bulk inserting
High memory consumption in BackendUtility->getTCAtypes while bulk inserting
Status: | Resolved | Start date: | 2014-12-08 | |
---|---|---|---|---|
Priority: | Must have | Due date: | ||
Assigned To: | - | % Done: | 100% |
|
Category: | Performance | Spent time: | - | |
Target version: | next-patchlevel | |||
TYPO3 Version: | 6.2 | Is Regression: | No | |
PHP Version: | Sprint Focus: | |||
Complexity: | easy |
Description
DataHandler->fillInFieldArray calls
$types_fieldConfig = BackendUtility::getTCAtypes($table, $currentRecord);for each record to insert. In each call the type of the current record is checked and the complete types configuration is parsed again.
static public function getTCAtypes($table, $rec, $useFieldNameAsKey = 0) { if ($GLOBALS['TCA'][$table]) { // Get type value: $fieldValue = self::getTCAtypeValue($table, $rec); // Get typesConf $typesConf = $GLOBALS['TCA'][$table]['types'][$fieldValue]; // Get fields list and traverse it $fieldList = explode(',', $typesConf['showitem']); $altFieldList = array(); // Traverse fields in types config and parse the configuration into a nice array: foreach ($fieldList as $k => $v) { [...] } if ($useFieldNameAsKey) { $fieldList = $altFieldList; } // Return array: return $fieldList; }
A new static cache using tablename, type and parameter as key improves performance and memory consumption:
static public function getTCAtypes($table, $rec, $useFieldNameAsKey = 0) { static $tcaTableTypeConf_cache = array(); if ($GLOBALS['TCA'][$table]) { // Get type value: $fieldValue = self::getTCAtypeValue($table, $rec); if ($tcaTableTypeConf_cache[$table . 'type' . $fieldValue . 'fnk' . $useFieldNameAsKey]) { return $tcaTableTypeConf_cache[$table . 'type' . $fieldValue . 'fnk' . $useFieldNameAsKey]; } // Get typesConf $typesConf = $GLOBALS['TCA'][$table]['types'][$fieldValue]; // Get fields list and traverse it $fieldList = explode(',', $typesConf['showitem']); $altFieldList = array(); // Traverse fields in types config and parse the configuration into a nice array: foreach ($fieldList as $k => $v) { [...] } if ($useFieldNameAsKey) { $fieldList = $altFieldList; } $tcaTableTypeConf_cache[$table . 'type' . $fieldValue . 'fnk' . $useFieldNameAsKey] = $fieldList; // Return array: return $fieldList; }For bulk insert of 6500 typeless (only type '1') records:
- 465000 calls less
- 3.3 seconds faster
- memory usage of explode()-Function down from 128MB to 58MB
Associated revisions
[BUGFIX] Better caching of TCA for bulk inserts
Introduces a runtime cache for the TCA types configuration. This
improves performance for bulk inserts since getTCAtypes is called for
each record of each table and the available types during a call do not
change.
Resolves: #63666
Releases: master, 6.2
Change-Id: Ia58352f1bae1e1eb57b87e5db0f1ad84e34ae37e
Reviewed-on: http://review.typo3.org/35185
Reviewed-by: Alexander Opitz <opitz.alexander@googlemail.com>
Tested-by: Alexander Opitz <opitz.alexander@googlemail.com>
Reviewed-by: Georg Ringer <georg.ringer@gmail.com>
Tested-by: Georg Ringer <georg.ringer@gmail.com>
Reviewed-by: Christian Kuhn <lolli@schwarzbu.ch>
Tested-by: Christian Kuhn <lolli@schwarzbu.ch>
[BUGFIX] Better caching of TCA for bulk inserts
Introduces a runtime cache for the TCA types configuration. This
improves performance for bulk inserts since getTCAtypes is called for
each record of each table and the available types during a call do not
change.
Resolves: #63666
Releases: master, 6.2
Change-Id: Ia58352f1bae1e1eb57b87e5db0f1ad84e34ae37e
Reviewed-on: http://review.typo3.org/35521
Reviewed-by: Christian Kuhn <lolli@schwarzbu.ch>
Tested-by: Christian Kuhn <lolli@schwarzbu.ch>
History
#1 Updated by Philipp Gampe 8 months ago
- Parent task set to #63692
#2 Updated by Gerrit Code Review 8 months ago
- Status changed from New to Under Review
Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35185
#3 Updated by Gerrit Code Review 8 months ago
Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35185
#4 Updated by Gerrit Code Review 8 months ago
Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35185
#5 Updated by Gerrit Code Review 8 months ago
Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35185
#6 Updated by Gerrit Code Review 8 months ago
Patch set 7 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35185
#7 Updated by Gerrit Code Review 8 months ago
Patch set 8 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35185
#8 Updated by Gerrit Code Review 8 months ago
Patch set 9 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35185
#9 Updated by Gerrit Code Review 8 months ago
Patch set 1 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35521
#10 Updated by Stephan Großberndt 8 months ago
- Status changed from Under Review to Resolved
- % Done changed from 0 to 100
Applied in changeset e0467960abe3e269c8bbc38135ff0c7245c2b5b3.