Bug #66614

FAL processed files are deleted

Added by Markus Klein 3 months ago. Updated about 1 month ago.

Status:Resolved Start date:2015-04-27
Priority:Must have Due date:
Assigned To:Markus Klein % Done:

100%
Category:File Abstraction Layer (FAL) Spent time: -
Target version:7.3 (Packages)
TYPO3 Version:6.2 Is Regression:No
PHP Version: Sprint Focus:Stabilization Sprint
Complexity:hard

Description

How to reproduce

  • set `config.noScaleUp = 0` in TS
  • add a magic image to RTE
  • preview the page in FE

Result: Image broken

Root cause

Setting config.noScaleUp causes this line to be executed in typo3/sysext/frontend/Classes/Page/PageGenerator.php:88

$GLOBALS['TYPO3_CONF_VARS']['GFX']['im_noScaleUp'] = isset($GLOBALS['TSFE']->config['config']['noScaleUp'])
   ? '' . $GLOBALS['TSFE']->config['config']['noScaleUp']
   : $GLOBALS['TYPO3_CONF_VARS']['GFX']['im_noScaleUp'];

This makes ['GFX']['im_noScaleUp'] a string, while in BE (when the preview image is generated in RTE) this is only loaded from install tool, where this is a boolean value.

This tiny difference has the effect that the calculated checksums used to determine whether a processed file already exists mismatch and the already generated processed file is deleted from disk.

Related issues

related to Core - Task #67071: Add cleanup tool for processed files to Install Tool Resolved 2015-05-21
related to Core - Bug #67385: SQL parser does not support string as join condition Resolved 2015-06-10

Associated revisions

Revision ab13da3d
Added by Markus Klein about 1 month ago

[BUGFIX] Avoid unnecessary generation of processed files

Removing [GFX] off the checksum data used to identify a processed file
ensures that data type inconsistencies within [GFX] do not cause a
regeneration of the processed file.

A new upgrade wizard for the Install Tool is added to migrate existing
processed files without the need to regenerate those.

Resolves: #66614
Releases: master, 6.2
Change-Id: Id90ce923b26ee726d15cf3d0cfcafc4ec4d7405d
Reviewed-on: http://review.typo3.org/39644
Reviewed-by: Andreas Fernandez <>
Reviewed-by: Xavier Perseguers <>
Tested-by: Xavier Perseguers <>
Reviewed-by: Stefan Neufeind <>
Reviewed-by: Alexander Opitz <>
Tested-by: Alexander Opitz <>

Revision 9067813e
Added by Xavier Perseguers about 1 month ago

[BUGFIX] SQL parser does not support string as join condition

The SQL parser is capable of parsing

JOIN T1 ON T1.some_field = T2.some_other_field

and

JOIN T1 ON T1.some_field = 4

but not

JOIN T1 ON T1.some_field = 'some string'

Change-Id: I7806699ea44685ab717cc0e8a75f7b53a1b2b1cc
Resolves: #67385
Related: #66614
Releases: master, 6.2
Reviewed-on: http://review.typo3.org/40144
Reviewed-by: Andreas Fernandez <>
Tested-by: Andreas Fernandez <>
Reviewed-by: Alexander Opitz <>
Reviewed-by: Xavier Perseguers <>
Tested-by: Xavier Perseguers <>

Revision 6d44ed99
Added by Xavier Perseguers about 1 month ago

[BUGFIX] SQL parser does not support string as join condition

The SQL parser is capable of parsing

JOIN T1 ON T1.some_field = T2.some_other_field

and

JOIN T1 ON T1.some_field = 4

but not

JOIN T1 ON T1.some_field = 'some string'

Change-Id: I7806699ea44685ab717cc0e8a75f7b53a1b2b1cc
Resolves: #67385
Related: #66614
Releases: master, 6.2
Reviewed-on: http://review.typo3.org/40145
Reviewed-by: Xavier Perseguers <>
Tested-by: Xavier Perseguers <>

Revision 66ab2d68
Added by Alexander Opitz about 1 month ago

[BUGFIX] Avoid unnecessary generation of processed files

Removing [GFX] off the checksum data used to identify a processed file
ensures that data type inconsistencies within [GFX] do not cause a
regeneration of the processed file.

A new upgrade wizard for the Install Tool is added to migrate existing
processed files without the need to regenerate those.

Resolves: #66614
Releases: master, 6.2
Change-Id: Id90ce923b26ee726d15cf3d0cfcafc4ec4d7405d
Reviewed-on: http://review.typo3.org/40143
Reviewed-by: Xavier Perseguers <>
Tested-by: Xavier Perseguers <>

Revision acdca0a6
Added by Morton Jonuschat 14 days ago

[BUGFIX] Make ProcessedFileChecksumUpdate compatible with DBAL

Use explicit CAST statements in the SQL queries performed by the
updater to allow other DBMS to compare a VARCHAR with an INT field.

The SQLparser has been extended to support CAST statements as part
of a JOIN clause.

Resolves: #67708
Related: #66614
Releases: master
Change-Id: I4a13eae92290b9ce8149aa5ef93df1305c02715a
Reviewed-on: http://review.typo3.org/40576
Reviewed-by: Markus Klein <>
Reviewed-by: Andreas Fernandez <>
Tested-by: Andreas Fernandez <>
Reviewed-by: Xavier Perseguers <>
Reviewed-by: Christian Kuhn <>
Tested-by: Christian Kuhn <>

History

#1 Updated by Gerrit Code Review 3 months ago

  • Status changed from Accepted to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/38972

#2 Updated by Markus Klein 3 months ago

  • Description updated (diff)

#3 Updated by Markus Klein 3 months ago

  • Status changed from Under Review to Accepted
  • Priority changed from Must have to Should have
  • Target version deleted (next-patchlevel)
  • Complexity changed from no-brainer to hard

Abandoned patch.

The issue is way more dramatic.

We have the situation that user experience will differ for BE and FE, if GFX|im_noScaleUp does not match config.noScaleUp.
BE and FE would behave differently.

#4 Updated by Markus Klein 3 months ago

  • Priority changed from Should have to Must have
  • Target version set to next-patchlevel

#56147 deals with the issue that processed files are not regenerated. This is related here, since with this bug you can trigger the other bug easily.
Actually, if #56147 would be working, I would have never noticed this bug here.

Regarding this issue: We decided to remove the [GFX] part from the checksum and add a cleanup-util to the Install Tool.

#5 Updated by Markus Klein 2 months ago

  • Target version changed from next-patchlevel to 6.2.13
  • Sprint Focus set to Stabilization Sprint

I figured out the whole process now.

  1. After inserting the magic file into RTE
  2. a ProcessedFile is generated for the thumb
  3. When calling FE, it tries to render the file but (due to the data type issue) a new thumb (with new filename) is generated (so a new ProcessedFile)
  4. Having the click-enlarge static template loaded of RTE, causes it to render the name of the original processed file (from >cObj>parameters, which contains the parsed RTE content), which has been deleted by the last step, when the new processed file was generated.
  5. In FE it looks like the thumb-regeneration has failed, which is not true. Just the filename of the original thumb is used, which does not exist anymore.

Ultima ratio:
Fixing the configuration change issue (removing GFX from the checksum) fixes the issue. The remaining FAL handling is totally fine in this case.

Implementation steps are:
  1. Remove [GFX] from the checksum generated in \TYPO3\CMS\Core\Resource\Processing\AbstractGraphicalTask::getChecksumData()
  2. Add a cleanup utility to Install-Tool, which allows to remove all processed files (#67071)

#6 Updated by Gerrit Code Review 2 months ago

  • Status changed from Accepted to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#7 Updated by Gerrit Code Review 2 months ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#8 Updated by Gerrit Code Review 2 months ago

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#9 Updated by Gerrit Code Review 2 months ago

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#10 Updated by Gerrit Code Review 2 months ago

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#11 Updated by Gerrit Code Review 2 months ago

Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#12 Updated by Gerrit Code Review 2 months ago

Patch set 7 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#13 Updated by Gerrit Code Review 2 months ago

Patch set 8 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#14 Updated by Gerrit Code Review 2 months ago

Patch set 9 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#15 Updated by Gerrit Code Review about 1 month ago

Patch set 10 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#16 Updated by Gerrit Code Review about 1 month ago

Patch set 11 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#17 Updated by Gerrit Code Review about 1 month ago

Patch set 12 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#18 Updated by Mathias Schreiber about 1 month ago

  • Target version changed from 6.2.13 to 7.3 (Packages)

#19 Updated by Gerrit Code Review about 1 month ago

Patch set 13 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/39644

#20 Updated by Gerrit Code Review about 1 month ago

Patch set 1 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/40143

#21 Updated by Markus Klein about 1 month ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100

#22 Updated by Gerrit Code Review about 1 month ago

  • Status changed from Resolved to Under Review

Patch set 2 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/40143

#23 Updated by Alexander Opitz about 1 month ago

  • Status changed from Under Review to Resolved

Also available in: Atom PDF