Bug #63675

Bug #63692: Memory consumption while bulk inserting

High memory consumption for softRefs in ReferenceIndex->getRelations  while bulk inserting

Added by Stephan Großberndt 8 months ago. Updated 4 months ago.

Status:Closed Start date:2014-12-08
Priority:Should have Due date:
Assigned To:- % Done:

80%
Category:Performance Spent time: -
Target version:6.2.12
TYPO3 Version:6.2 Is Regression:No
PHP Version: Sprint Focus:Stabilization Sprint
Complexity:

Description

For every field of every record ReferenceIndex->getRelations executes:

                // Soft References:
                if (strlen($value) && ($softRefs = BackendUtility::explodeSoftRefParserList($conf['softref']))) {

The return value of explodeSoftRefParserList is not dependent on the record itself, but only table and field. Introducing a cache for softRefs

public function getRelations($table, $row, $onlyField = '') {
    static $softRef_cache = array();

    [...]

                // Soft References:
                if (isset($softRef_cache[$table . 'f' . $field])) {
                    $softRefs = $softRef_cache[$table . 'f' . $field];
                } else {
                    $softRefs = BackendUtility::explodeSoftRefParserList($conf['softref']);
                    $softRef_cache[$table . 'f' . $field] = $softRefs;
                }
                if (strlen($value) && $softRefs) {

reduces when inserting 6500 records:

  • 8 instead of 51,024 calls to explodeSoftRefParserList
  • 0.52 seconds faster
  • 4kb used instead of 2,5MB

Associated revisions

Revision 72c3dec5
Added by Stephan Großberndt 4 months ago

[BUGFIX] Cache results of explodeSoftRefParserList()

BackendUtility::explodeSoftRefParserList() is called with a very
limited set of input parameters, but processing is rather expensive.

Introduce a runtime cache to avoid processing the softrefs over
and over.
This especially speeds up calls from ReferenceIndex::getRelations(),
which is called many times.

Resolves: #63675
Releases: master, 6.2
Change-Id: I34a4392533828667c1b63b1ecb4ca12580e05e06
Reviewed-on: http://review.typo3.org/35190
Reviewed-by: Nicole Cordes <>
Reviewed-by: Markus Klein <>
Tested-by: Markus Klein <>
Reviewed-by: Andreas Fernandez <>
Reviewed-by: Xavier Perseguers <>
Tested-by: Xavier Perseguers <>

Revision 5a15c87f
Added by Stephan Großberndt 4 months ago

[BUGFIX] Cache results of explodeSoftRefParserList()

BackendUtility::explodeSoftRefParserList() is called with a very
limited set of input parameters, but processing is rather expensive.

Introduce a runtime cache to avoid processing the softrefs over
and over.
This especially speeds up calls from ReferenceIndex::getRelations(),
which is called many times.

Resolves: #63675
Releases: master, 6.2
Change-Id: I34a4392533828667c1b63b1ecb4ca12580e05e06
Reviewed-on: http://review.typo3.org/38662
Reviewed-by: Xavier Perseguers <>
Tested-by: Xavier Perseguers <>
Reviewed-by: Andreas Fernandez <>
Tested-by: Andreas Fernandez <>

History

#1 Updated by Philipp Gampe 8 months ago

  • Parent task set to #63692

#2 Updated by Gerrit Code Review 8 months ago

  • Status changed from New to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35190

#3 Updated by Philipp Gampe 8 months ago

  • Subject changed from High memory consumption in ReferenceIndex->getRelations while bulk inserting to High memory consumption for softRefs in ReferenceIndex->getRelations  while bulk inserting

#4 Updated by Gerrit Code Review 8 months ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35190

#5 Updated by Gerrit Code Review 7 months ago

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35190

#6 Updated by Gerrit Code Review 7 months ago

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35190

#7 Updated by Gerrit Code Review 7 months ago

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35190

#8 Updated by Gerrit Code Review 7 months ago

Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35190

#9 Updated by Gerrit Code Review 7 months ago

Patch set 7 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35190

#10 Updated by Gerrit Code Review 7 months ago

Patch set 8 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35190

#11 Updated by Gerrit Code Review 7 months ago

Patch set 9 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35190

#12 Updated by Gerrit Code Review 6 months ago

Patch set 10 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35190

#13 Updated by Gerrit Code Review 5 months ago

Patch set 11 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35190

#14 Updated by Stephan Großberndt 4 months ago

  • Target version changed from next-patchlevel to 6.2.12
  • Sprint Focus set to Stabilization Sprint

#15 Updated by Gerrit Code Review 4 months ago

Patch set 1 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/38662

#16 Updated by Stephan Großberndt 4 months ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100

#17 Updated by Xavier Perseguers 4 months ago

  • Status changed from Resolved to Under Review
  • % Done changed from 100 to 80

Still not ready for 6.2

#18 Updated by Gerrit Code Review 4 months ago

Patch set 2 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/38662

#19 Updated by Andreas Fernandez 4 months ago

  • Status changed from Under Review to Closed

Merged

Also available in: Atom PDF