Bug #63676: High memory consumption for nonFields in ReferenceIndex->getRelations while bulk inserting - Core - TYPO3 Forge

Core Indexed Search Linkvalidator Release Cycles Team Resources Workspaces & Versioning TYPO3 CMS Usability Team Community Extensions Distributions Feature-Requests TYPO3 6.2 Projects (+)(Archived Projects)

Bug #63676

Bug #63692: Memory consumption while bulk inserting

High memory consumption for nonFields in ReferenceIndex->getRelations while bulk inserting

Added by Stephan Großberndt 8 months ago. Updated 5 months ago.

Status:

Resolved

Start date:

2014-12-08

Priority:

Must have

Due date:

Assigned To:

% Done:

100%

Category:

Performance

Spent time:

Target version:

next-patchlevel

TYPO3 Version:

6.2

Is Regression:

PHP Version:

Sprint Focus:

Complexity:

no-brainer

Description

For each call of every record to get Relations from with ReferenceIndex->getRelations

public function getRelations($table, $row, $onlyField = '') {
    // Initialize:
    $uid = $row['uid'];
    $nonFields = explode(',', 'uid,perms_userid,perms_groupid,perms_user,perms_group,perms_everybody,pid');

a static list of nonFields is initialized by exploding a string. Using a property instead

protected $nonRelationFields = array(
    'uid', 'perms_userid', 'perms_groupid', 'perms_user', 'perms_group', 'perms_everybody', 'pid'
);

[...]
if (!in_array($field, $this->nonRelationFields) && is_array($GLOBALS['TCA'][$table]['columns'][$field]) && (!$onlyField || $onlyField === $field)) {

reduced for bulk inserting of 6500 records

10MB less ram used
6500 calls less - obvious :-)

Associated revisions

Revision 4eccaf60
Added by Stephan Großberndt 5 months ago

[BUGFIX] Do not fetch records from tables without references

If no field of a table can contain a reference by its TCA definition
do not fetch record from this table at all. For tables with fields
which can possibly contain references fetch only these fields instead
of all.

Cache list of fields which don't have to be checked in the local member
'nonRelationFields' instead of recreating the list for each and every
record.

Don't try to update references for tables sys_log, sys_history and
tx_extensionmanager_domain_model_extension since they cannot contain
references and usually have a big amount of records.

Introduce some early returns to make code more readable.

Improves performance for both bulk inserts and updating the reference
index from the command line.

Resolves: #63676
Resolves: #63782
Releases: master, 6.2
Change-Id: Ibc6f988bfde6042339595bf7b3e18125c64ae72d
Reviewed-on: http://review.typo3.org/35191
Reviewed-by: Frank Nägler <typo3@naegler.net>
Tested-by: Frank Nägler <typo3@naegler.net>
Reviewed-by: Anja Leichsenring <aleichsenring@ab-softlab.de>
Tested-by: Anja Leichsenring <aleichsenring@ab-softlab.de>

Revision 223e0876
Added by Stephan Großberndt 5 months ago

[BUGFIX] Do not fetch records from tables without references

Cache list of fields which don't have to be checked in the local member
'nonRelationFields' instead of recreating the list for each and every
record.

Don't try to update references for tables sys_log, sys_history and
tx_extensionmanager_domain_model_extension since they cannot contain
references and usually have a big amount of records.

Introduce some early returns to make code more readable.

Improves performance for both bulk inserts and updating the reference
index from the command line.

Resolves: #63676
Resolves: #63782
Releases: master, 6.2
Change-Id: Ibc6f988bfde6042339595bf7b3e18125c64ae72d
Reviewed-on: http://review.typo3.org/37478
Reviewed-by: Nicole Cordes <typo3@cordes.co>
Tested-by: Nicole Cordes <typo3@cordes.co>

History

#1 Updated by Philipp Gampe 8 months ago

Parent task set to #63692

#2 Updated by Gerrit Code Review 8 months ago

Status changed from New to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#3 Updated by Gerrit Code Review 8 months ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#4 Updated by Gerrit Code Review 8 months ago

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#5 Updated by Gerrit Code Review 8 months ago

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#6 Updated by Gerrit Code Review 8 months ago

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#7 Updated by Gerrit Code Review 8 months ago

Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#8 Updated by Gerrit Code Review 7 months ago

Patch set 7 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#9 Updated by Gerrit Code Review 7 months ago

Patch set 8 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#10 Updated by Gerrit Code Review 7 months ago

Patch set 9 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#11 Updated by Gerrit Code Review 7 months ago

Patch set 10 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#12 Updated by Gerrit Code Review 7 months ago

Patch set 11 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#13 Updated by Gerrit Code Review 6 months ago

Patch set 12 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#14 Updated by Gerrit Code Review 6 months ago

Patch set 13 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#15 Updated by Gerrit Code Review 6 months ago

Patch set 14 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#16 Updated by Gerrit Code Review 6 months ago

Patch set 15 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#17 Updated by Gerrit Code Review 6 months ago

Patch set 16 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#18 Updated by Gerrit Code Review 6 months ago

Patch set 17 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#19 Updated by Gerrit Code Review 6 months ago

Patch set 18 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#20 Updated by Gerrit Code Review 6 months ago

Patch set 19 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#21 Updated by Gerrit Code Review 5 months ago

Patch set 20 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#22 Updated by Gerrit Code Review 5 months ago

Patch set 21 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#23 Updated by Gerrit Code Review 5 months ago

Patch set 22 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#24 Updated by Gerrit Code Review 5 months ago

Patch set 23 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#25 Updated by Gerrit Code Review 5 months ago

Patch set 24 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#26 Updated by Gerrit Code Review 5 months ago

Patch set 25 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#27 Updated by Gerrit Code Review 5 months ago

Patch set 26 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#28 Updated by Gerrit Code Review 5 months ago

Patch set 27 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#29 Updated by Stephan Großberndt 5 months ago

Status changed from Under Review to Resolved
% Done changed from 0 to 100

Applied in changeset 4eccaf60b7102a9300674834a8feee3803b85f9c.

#30 Updated by Gerrit Code Review 5 months ago

Status changed from Resolved to Under Review

Patch set 1 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/37478

#31 Updated by Gerrit Code Review 5 months ago

Patch set 2 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/37478

#32 Updated by Stephan Großberndt 5 months ago

Status changed from Under Review to Resolved

Applied in changeset 223e0876e1fc81e896b032f9b526c86b961c017d.

Also available in: Atom PDF

Core

Issues

Custom queries