Bug #60494

editing gridelement in foreign language exhausts memory on large sites

Added by Martin Schnabel about 1 year ago. Updated about 1 year ago.

Status:Resolved Start date:2014-07-23
Priority:Should have Due date:
Assigned To:- % Done:

100%

Category:- Spent time: -
Target version:-
TYPO3 Version:6.2 Is Regression:
PHP Version:5.3 Sprint Focus:
Complexity:

Description

I have a site with very many pages. tt_content has about 14000 rows. When editing a gridelement in a foreign language typo3 takes too much memory. I had to set memory_limit to 512M.

With some debugging I found out that the problem is inside TYPO3\CMS\Backend\Utility\BackendUtility::getProcessedValue when it is called for $col = 'tx_gridelements_children'. The array $rParts gets very large because it is filled with all uids of tt_content where tx_gridelements_children is 0. For my site that means about 10000 elements. In the later processing of the uids php runs out of memory.

As a workaground I used the hook inside BackendUtility::getProcessedValue by setting

1$GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS']['t3lib/class.t3lib_befunc.php']['preProcessValue'][] = '&\WV\ImgReloaded\Hooks\GridElementsFix->preProcessValue';

inside ext_tables.php (inside an extension of mine)

The hook sets the field type to flex that no processing by BackendUtility::getProcessedValue will be done:

 1<?php
 2
 3namespace WV\ImgReloaded\Hooks;
 4
 5// Hook done in ext_tables.php
 6
 7class GridElementsFix {
 8
 9    public function preProcessValue(&$params, &$ref) {
10        if (is_array($params) && array_key_exists('foreign_field', $params) && $params['foreign_field'] == 'tx_gridelements_container') {
11            $params['type'] = 'flex';
12        }
13    }
14
15}

I know the workaround is ugly. I hope someone can find a better fix.

(I had gridelements rev 1824f01414736595970f12f239fa76f4850288b3)


Related issues

related to Core - Bug #59671: IRRE related child records are not listed (comma separate... Resolved 2014-06-19

Associated revisions

Revision 169f8b91
Added by Jo Hasenau about 1 year ago

[BUGFIX] Fetch related records for existing uids only

Before fetching related records from a foreign table, we must ensure
that there is a uid to be considered, since otherwise we will fetch
any record having a 0 value for that field.

There are method calls in the core, that don't provide a uid, so the
default value 0 is used, i.e. when fetching children of translated
gridelements, causing memory outage when tt_content is > 10000 records.

Resolves: #60494
Releases: 6.3, 6.2
Change-Id: I1105964f98f79074bb37dc1921180b493fac4bc6
Reviewed-on: http://review.typo3.org/31781
Reviewed-by: Sascha Egerer <>
Tested-by: Sascha Egerer <>
Reviewed-by: Markus Klein <>
Tested-by: Markus Klein <>

Revision 14fc7504
Added by Jo Hasenau about 1 year ago

[BUGFIX] Fetch related records for existing uids only

Before fetching related records from a foreign table, we must ensure
that there is a uid to be considered, since otherwise we will fetch
any record having a 0 value for that field.

There are method calls in the core, that don't provide a uid, so the
default value 0 is used, i.e. when fetching children of translated
gridelements, causing memory outage when tt_content is > 10000 records.

Resolves: #60494
Releases: 6.3, 6.2
Change-Id: I1105964f98f79074bb37dc1921180b493fac4bc6
Reviewed-on: http://review.typo3.org/31947
Reviewed-by: Markus Klein <>
Tested-by: Markus Klein <>

History

#1 Updated by Jo Hasenau about 1 year ago

  • Status changed from New to Needs Feedback

IMHO this is a core bug, since the original type is "inline" and I don't see a good reason, why getProcessedValue should consider each element having a value of 0 here.

#2 Updated by Jo Hasenau about 1 year ago

The problem seems to be that $uid is 0 in this case and therefor

$records = self::getRecordsByField($theColConf['foreign_table'], $theColConf['foreign_field'], $uid);

fetches each record that is NOT child of a container.
So the question is, why is getProcessedValue called with a 0 value here.

#3 Updated by Jo Hasenau about 1 year ago

This is the definition of the method

/**
     * Returns a human readable output of a value from a record
     * For instance a database record relation would be looked up to display the title-value of that record. A checkbox with a "1" value would be "Yes", etc.
     * $table/$col is tablename and fieldname
     * REMEMBER to pass the output through htmlspecialchars() if you output it to the browser! (To protect it from XSS attacks and be XHTML compliant)
     *
     * @param string $table Table name, present in TCA
     * @param string $col Field name, present in TCA
     * @param string $value The value of that field from a selected record
     * @param integer $fixed_lgd_chars The max amount of characters the value may occupy
     * @param boolean $defaultPassthrough Flag means that values for columns that has no conversion will just be pass through directly (otherwise cropped to 200 chars or returned as "N/A")
     * @param boolean $noRecordLookup If set, no records will be looked up, UIDs are just shown.
     * @param integer $uid Uid of the current record
     * @param boolean $forceResult If BackendUtility::getRecordTitle is used to process the value, this parameter is forwarded.
     * @return string
     */
    static public function getProcessedValue($table, $col, $value, $fixed_lgd_chars = 0, $defaultPassthrough = 0, $noRecordLookup = FALSE, $uid = 0, $forceResult = TRUE) {

And these are different use cases and how it is called:

$defaultLanguageValue = BackendUtility::getProcessedValue($table, $field, $this->defaultLanguageData[$table . ':' . $row['uid']][$field], 0, 1);

$defaultLanguageValue = BackendUtility::getProcessedValue($table, $field, $this->additionalPreviewLanguageData[$table . ':' . $row['uid']][$previewLanguage['uid']][$field], 0, 1);

$diffres = $t3lib_diff_Obj->makeDiffDisplay(
    BackendUtility::getProcessedValue($table, $field, $dLVal['old'][$field], 0, 1),
    BackendUtility::getProcessedValue($table, $field, $dLVal['new'][$field], 0, 1)
);

As you can see they are using just 5 out of the 8 possible parameters, so $uid will always be 0, which is just plain wrong for the case of inline elements pointing to a foreign table.

I will move this to the core tracker.

#4 Updated by Jo Hasenau about 1 year ago

  • Project changed from Grid Elements to Core

#5 Updated by Jo Hasenau about 1 year ago

This is not just about Gridelements, but AFAICS about any element using fields of type inline.

#6 Updated by Jo Hasenau about 1 year ago

IMHO it should be enough to introduce a check for $uid being available in line 2080 of typo3/sysext/backend/Classes/Utility/BackendUtility.php, since we need to check that for existing uids only.

#7 Updated by Gerrit Code Review about 1 year ago

  • Status changed from Needs Feedback to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/31781

#8 Updated by Gerrit Code Review about 1 year ago

Patch set 1 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/31947

#9 Updated by Jo Hasenau about 1 year ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100

Also available in: Atom PDF