Bug #57393

Task #57578: TceformsUpdateWizard improvements (Migrate all file relations from tt_content.image and pages.media)

Endless Loop in TceformsUpdateWizard

Added by Benjamin Pucher over 1 year ago. Updated 5 months ago.

Status:Resolved Start date:2014-03-27
Priority:Must have Due date:
Assigned To:- % Done:

100%
Category:Install Tool Spent time: -
Target version:next-patchlevel Estimated time:0.25 hour
TYPO3 Version:6.2 Is Regression:No
PHP Version:5.3 Sprint Focus:
Complexity:medium

Description

I'm currently updating a large TYPO3 Website from 4.5 to 6.2 and have found a bug in the Upgrade Wizard.
The migration with title 'Migrate all file relations from tt_content.image and pages.media' (typo3/sysext/install/Classes/Updates/TceformsUpdateWizard.php) ends up in an endless loop.
The problem exists in the do/while loop starting at line 188 respectively in the getRecordsFromTable method in typo3/sysext/install/Classes/Updates/TceformsUpdateWizard.php.

188 do {
189     $records = $this->getRecordsFromTable($table, $fieldToMigrate, $fieldsToGet, self::RECORDS_PER_QUERY);
190     foreach ($records as $record) {
191         $this->migrateField($table, $record, $fieldToMigrate, $fieldConfiguration, $customMessages);
192     }
193 } while (count($records) === self::RECORDS_PER_QUERY);

self::RECORDS_PER_QUERY is set to 1000. Every time a query like 'select ... LIMIT 1000' is created. Because there's no starting point defined after the LIMIT statement (which should be 0 on first loop and then += self::RECORDS_PER_QUERY) the query will always fetch the first self::RECORDS_PER_QUERY entries. Thus the while condition is always true.
To fix this the LIMIT statement should have the starting point included like LIMIT <entries_fetched_so_far>, self::RECORDS_PER_QUERY.

Please fix this asap since many people will update their TYPO3 LTS Websites shortly.

Related issues

related to Core - Bug #43725: TceformsUpdateWizard exceeds memory_limit/max_execution_time Accepted 2012-12-07
related to Core - Task #57572: TceformsUpdateWizard has to handle SQL errors Resolved 2014-04-02

Associated revisions

Revision 57f749d1
Added by Nicole Cordes 5 months ago

[BUGFIX] Prevent infinite loop in TceformsUpdateWizard

If you have more than 1000 missing files within file relations in
tt_content.image and pages.media an infinite loop pops up, because
always the same records are fetched from database. This patch stores
the uids of failed records for each table to prevent the infinite loop
and restart processing after timeout.

Releases: 6.2
Resolves: #57393
Change-Id: I559be13efe88ae3111321dafb8d6a79886f903ae
Reviewed-on: http://review.typo3.org/37032
Reviewed-by: Stefan Neufeind <>
Reviewed-by: Wouter Wolters <>
Reviewed-by: Göran Bodenschatz <>
Reviewed-by: Christian Kuhn <>
Tested-by: Christian Kuhn <>
Reviewed-by: Markus Klein <>
Tested-by: Markus Klein <>

History

#1 Updated by Steffen Krüchten over 1 year ago

I can confirm this Bug, also trying to update a small Installation to 6.2.0 from 4.5.32 with PHP 5.3.27. Everything else went smooth, Frontend is up and running but no Content images at all.

Upgrade Wizard throws a "Uncaught TYPO3 Exception" in Debug-Mode when trying to execute "Migrate all file relations from tt_content.image and pages.media":
PHP Warning: Invalid argument supplied for foreach() in typo3/sysext/install/Classes/Updates/TceformsUpdateWizard.php line 190.

#2 Updated by Markus Klein over 1 year ago

  • Status changed from New to Needs Feedback
  • Target version deleted (next-patchlevel)

These are two different issues.

The issue reported here is IMO not an issue. The limit is ok, you can always start from 0, because the select query only looks at records not yet migrated!

Regarding the foreach() thing. There seems to be a DB issue, the select seems to fail.
I'll create a new ticket for this, which introduces a better check and issues an exception with the SQL error.

#3 Updated by Markus Klein over 1 year ago

The new ticket for the foreach() issue is #57572

#4 Updated by Markus Klein over 1 year ago

  • Parent task set to #57578

#5 Updated by Tobias Braumann over 1 year ago

I can confirm this Bug, too. In our case the Problem was that we have a lot of pictures in tt_content,
which no longer exist and the script don't set the image field to 1 or delete the entries.
So in the foreach() there are still the same 1000 entries (infinityloop).

We fix that problem by increasing the value self::RECORDS_PER_QUERY to the number of entries from tt_content.

#6 Updated by Alexander Opitz 11 months ago

  • Status changed from Needs Feedback to New

#7 Updated by Sebastian Helzle 9 months ago

We also had this issue today with a big site (~15000 pages) and more than 1000 missing media files in the page properties because we tried the upgrade on a lightweight copy without the assets.

#8 Updated by Gerrit Code Review 5 months ago

  • Status changed from New to Under Review

Patch set 1 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/37032

#9 Updated by Gerrit Code Review 5 months ago

Patch set 2 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/37032

#10 Updated by Gerrit Code Review 5 months ago

Patch set 3 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/37032

#11 Updated by Markus Klein 5 months ago

We kindly ask you to please test the patch linked above with your databases.
If it works for you, please vote accordingly.

Thanks a lot!

#12 Updated by Markus Klein 5 months ago

  • Target version set to next-patchlevel
  • Complexity changed from easy to medium

#13 Updated by Gerrit Code Review 5 months ago

Patch set 4 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/37032

#14 Updated by Gerrit Code Review 5 months ago

Patch set 5 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/37032

#15 Updated by shanmugarajan k 5 months ago

It seems,once click Execute, i never see this "Migrate all file relations from tt_content.image and pages.media" upgrade wizard.

I have around 10000+ records.

I re-imported the db, but still i could not see the wizard again. can you please let me know if i doing something wrong?

#16 Updated by shanmugarajan k 5 months ago

i just saw in the LocalConfiguration.php

'INSTALL' => array() having the 'wizardDone' wizard is entered. Hence its not visible again.

so it seems, this is added as soon as first run completed.

#17 Updated by Markus Klein 5 months ago

@shanmugarajan

Which TYPO3 version are you upgrading from?
Are you sure you actually need this patch?

It can well be that the wizard manages to complete the migration in one go, so it won't show up again.

#18 Updated by shanmugarajan k 5 months ago

Hello Markus,

Yes. i upgrading from 4.5 to 6.2.

Yes. needed. thanks.

#19 Updated by Nicole Cordes 5 months ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100

Also available in: Atom PDF