Bug #63782: Do not fetch all columns in ReferenceIndex->generateRefIndexData - Core - TYPO3 Forge

Core Indexed Search Linkvalidator Release Cycles Team Resources Workspaces & Versioning TYPO3 CMS Usability Team Community Extensions Distributions Feature-Requests TYPO3 6.2 Projects (+)(Archived Projects)

Bug #63782

Bug #63692: Memory consumption while bulk inserting

Do not fetch all columns in ReferenceIndex->generateRefIndexData

Added by Stephan Großberndt 8 months ago. Updated 5 months ago.

Status:

Resolved

Start date:

2014-12-11

Priority:

Should have

Due date:

Assigned To:

Stephan Großberndt

% Done:

100%

Category:

Performance

Spent time:

Target version:

next-patchlevel

TYPO3 Version:

6.2

Is Regression:

PHP Version:

Sprint Focus:

Complexity:

medium

Description

Currently ReferenceIndex->generateRefIndexData fetches all data from each table and selects afterwards which fields and their contents are looked at searching for relations.

public function generateRefIndexData($table, $uid) {
    if (isset($GLOBALS['TCA'][$table])) {
        // Get raw record from DB:
        $record = $GLOBALS['TYPO3_DB']->exec_SELECTgetSingleRow('*', $table, 'uid=' . (int)$uid);

If the fields are selected beforehand using the information from the TCA a lot less data is fetched. This improves the speed of building the reference index as standalone task as well as when bulk inserting many records.

Associated revisions

Revision 4eccaf60
Added by Stephan Großberndt 5 months ago

[BUGFIX] Do not fetch records from tables without references

If no field of a table can contain a reference by its TCA definition
do not fetch record from this table at all. For tables with fields
which can possibly contain references fetch only these fields instead
of all.

Cache list of fields which don't have to be checked in the local member
'nonRelationFields' instead of recreating the list for each and every
record.

Don't try to update references for tables sys_log, sys_history and
tx_extensionmanager_domain_model_extension since they cannot contain
references and usually have a big amount of records.

Introduce some early returns to make code more readable.

Improves performance for both bulk inserts and updating the reference
index from the command line.

Resolves: #63676
Resolves: #63782
Releases: master, 6.2
Change-Id: Ibc6f988bfde6042339595bf7b3e18125c64ae72d
Reviewed-on: http://review.typo3.org/35191
Reviewed-by: Frank Nägler <typo3@naegler.net>
Tested-by: Frank Nägler <typo3@naegler.net>
Reviewed-by: Anja Leichsenring <aleichsenring@ab-softlab.de>
Tested-by: Anja Leichsenring <aleichsenring@ab-softlab.de>

Revision 223e0876
Added by Stephan Großberndt 5 months ago

[BUGFIX] Do not fetch records from tables without references

Cache list of fields which don't have to be checked in the local member
'nonRelationFields' instead of recreating the list for each and every
record.

Don't try to update references for tables sys_log, sys_history and
tx_extensionmanager_domain_model_extension since they cannot contain
references and usually have a big amount of records.

Introduce some early returns to make code more readable.

Improves performance for both bulk inserts and updating the reference
index from the command line.

Resolves: #63676
Resolves: #63782
Releases: master, 6.2
Change-Id: Ibc6f988bfde6042339595bf7b3e18125c64ae72d
Reviewed-on: http://review.typo3.org/37478
Reviewed-by: Nicole Cordes <typo3@cordes.co>
Tested-by: Nicole Cordes <typo3@cordes.co>

History

#1 Updated by Mathias Schreiber 8 months ago

Parent task set to #63692

#2 Updated by Gerrit Code Review 8 months ago

Status changed from New to Under Review

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#3 Updated by Gerrit Code Review 8 months ago

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#4 Updated by Gerrit Code Review 8 months ago

Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#5 Updated by Gerrit Code Review 7 months ago

Patch set 7 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#6 Updated by Gerrit Code Review 7 months ago

Patch set 8 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#7 Updated by Gerrit Code Review 7 months ago

Patch set 9 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#8 Updated by Gerrit Code Review 7 months ago

Patch set 10 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#9 Updated by Gerrit Code Review 7 months ago

Patch set 11 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#10 Updated by Gerrit Code Review 6 months ago

Patch set 12 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#11 Updated by Gerrit Code Review 6 months ago

Patch set 13 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#12 Updated by Gerrit Code Review 6 months ago

Patch set 14 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#13 Updated by Gerrit Code Review 6 months ago

Patch set 15 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#14 Updated by Gerrit Code Review 6 months ago

Patch set 16 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#15 Updated by Gerrit Code Review 6 months ago

Patch set 17 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#16 Updated by Gerrit Code Review 6 months ago

Patch set 18 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#17 Updated by Gerrit Code Review 6 months ago

Patch set 19 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#18 Updated by Gerrit Code Review 5 months ago

Patch set 20 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#19 Updated by Gerrit Code Review 5 months ago

Patch set 21 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#20 Updated by Gerrit Code Review 5 months ago

Patch set 22 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#21 Updated by Gerrit Code Review 5 months ago

Patch set 23 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#22 Updated by Gerrit Code Review 5 months ago

Patch set 24 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#23 Updated by Gerrit Code Review 5 months ago

Patch set 25 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#24 Updated by Gerrit Code Review 5 months ago

Patch set 26 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#25 Updated by Gerrit Code Review 5 months ago

Patch set 27 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35191

#26 Updated by Stephan Großberndt 5 months ago

Status changed from Under Review to Resolved
% Done changed from 0 to 100

Applied in changeset 4eccaf60b7102a9300674834a8feee3803b85f9c.

#27 Updated by Gerrit Code Review 5 months ago

Status changed from Resolved to Under Review

Patch set 1 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/37478

#28 Updated by Gerrit Code Review 5 months ago

Patch set 2 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/37478

#29 Updated by Stephan Großberndt 5 months ago

Status changed from Under Review to Resolved

Applied in changeset 223e0876e1fc81e896b032f9b526c86b961c017d.

Also available in: Atom PDF

Core

Issues

Custom queries