Bug #63670

Bug #63692: Memory consumption while bulk inserting

High memory consumption in DataHandler->checkValue_input while bulk inserting

Added by Stephan Großberndt 8 months ago. Updated 4 months ago.

Status:Resolved Start date:2014-12-08
Priority:Must have Due date:
Assigned To:- % Done:

100%
Category:Performance Spent time: -
Target version:next-patchlevel
TYPO3 Version:6.2 Is Regression:No
PHP Version: Sprint Focus:
Complexity:

Description

In DataHandler->checkValue_input for each input value of each record to insert is executed.

// Process evaluation settings:
$evalCodesArray = GeneralUtility::trimExplode(',', $tcaFieldConf['eval'], TRUE);

While this may seem minor introducing a cache here helps:

public function checkValue_input($res, $value, $tcaFieldConf, $PP, $field = '') {
    static $evalCodesCache = array();

    [...]
    // Process evaluation settings:
    if ($evalCodesCache[$tcaFieldConf['eval']]) {
        $evalCodesArray = $evalCodesCache[$tcaFieldConf['eval']];
    } else {
        $evalCodesArray = GeneralUtility::trimExplode(',', $tcaFieldConf['eval'], TRUE);
        $evalCodesCache[$tcaFieldConf['eval']] = $evalCodesArray;
    }
For bulk inserting of 6500 records:
  • 2 calls instead of 44646 to trimExplode from this function
  • 0.7 sec faster
  • memory_usage of trimExplode down by 17MB

eval.txt Magnifier - Ocurrences of reading 'eval' (7.7 kB) Stephan Großberndt, 2015-01-05 16:39

Associated revisions

Revision 5acee1d3
Added by Stephan Großberndt 5 months ago

[BUGFIX] DataHandler: Cache TCA field eval array

Introduce a runtime cache for the eval values of TCA input and text
fields. This improves performance for bulk inserts since
DataHandler->checkValue_input and checkValue_text are called for each
according field of each record and the available evaluations do not
change during a call. If no eval is set, do not execute
checkValue_*_Eval functions at all.

Additionally align usages of getMemoryCache() to runtimeCache and
keys of nestedElementCalls and elementsToBeDeleted in cache_runtime.

Resolves: #63670
Releases: master, 6.2
Change-Id: I3d2122a61e3194402743fa0b364d6bd80d4e72dc
Reviewed-on: http://review.typo3.org/35186
Reviewed-by: Nicole Cordes <>
Reviewed-by: Markus Klein <>
Tested-by: Markus Klein <>
Reviewed-by: Jigal van Hemert <>
Tested-by: Jigal van Hemert <>

Revision 4a9e5fdb
Added by Nicole Cordes 5 months ago

[FOLLOWUP][BUGFIX] DataHandler: Cache TCA field eval array

This fixes failing functional tests.

Resolves: #63670
Releases: master
Change-Id: Ief6ec0d6a1d2f45a7b8cb512741af722807aad9d
Reviewed-on: http://review.typo3.org/37937
Reviewed-by: Stephan Großberndt <>
Reviewed-by: Markus Klein <>
Tested-by: Markus Klein <>

Revision c1bb6311
Added by Stephan Großberndt 4 months ago

[BUGFIX] DataHandler: Cache TCA field eval array

Introduce a runtime cache for the eval values of TCA input and text
fields. This improves performance for bulk inserts since
DataHandler->checkValue_input and checkValue_text are called for each
according field of each record and the available evaluations do not
change during a call. If no eval is set, do not execute
checkValue_*_Eval functions at all.

Additionally align usages of getMemoryCache() to runtimeCache and
keys of nestedElementCalls and elementsToBeDeleted in cache_runtime.

Resolves: #63670
Releases: master, 6.2
Change-Id: I3d2122a61e3194402743fa0b364d6bd80d4e72dc
Reviewed-on: http://review.typo3.org/37963
Reviewed-by: Markus Klein <>
Tested-by: Markus Klein <>

History

#1 Updated by Philipp Gampe 8 months ago

  • Parent task set to #63692

#2 Updated by Gerrit Code Review 8 months ago

  • Status changed from New to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35186

#3 Updated by Gerrit Code Review 8 months ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35186

#4 Updated by Gerrit Code Review 8 months ago

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35186

#5 Updated by Gerrit Code Review 8 months ago

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35186

#6 Updated by Gerrit Code Review 8 months ago

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35186

#7 Updated by Gerrit Code Review 8 months ago

Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35186

#8 Updated by Gerrit Code Review 8 months ago

Patch set 7 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35186

#9 Updated by Gerrit Code Review 7 months ago

Patch set 8 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35186

#10 Updated by Gerrit Code Review 7 months ago

Patch set 9 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35186

#11 Updated by Stephan Großberndt 7 months ago

#12 Updated by Gerrit Code Review 7 months ago

Patch set 10 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35186

#13 Updated by Gerrit Code Review 7 months ago

Patch set 11 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35186

#14 Updated by Gerrit Code Review 7 months ago

Patch set 12 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35186

#15 Updated by Gerrit Code Review 6 months ago

Patch set 13 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35186

#16 Updated by Gerrit Code Review 6 months ago

Patch set 14 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35186

#17 Updated by Gerrit Code Review 5 months ago

Patch set 15 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35186

#18 Updated by Stephan Großberndt 5 months ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100

#19 Updated by Gerrit Code Review 5 months ago

  • Status changed from Resolved to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/37937

#20 Updated by Gerrit Code Review 5 months ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/37937

#21 Updated by Gerrit Code Review 5 months ago

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/37937

#22 Updated by Nicole Cordes 5 months ago

  • Status changed from Under Review to Resolved

#23 Updated by Gerrit Code Review 5 months ago

  • Status changed from Resolved to Under Review

Patch set 1 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/37963

#24 Updated by Stephan Großberndt 4 months ago

  • Status changed from Under Review to Resolved

Also available in: Atom PDF