Checksum mismatch in db migration

[eikek]

Spin off from issue #1488 it's better to have a separate issue. It seems to be a longer journey sadly. Context: #1488 (comment) and following comments.

@enkol Can you post me the full logs (from the start)? What is irritating is that 1.34.0 is not part of the fixup changesets. So changing the table flyway_fixup_history can only have an effect on the single fixup migration that is included.

[enkol]

Coming from a running v0.34.0 and changing to v0.35.0 results in:

Apr 16, 2022 3:26:36 PM org.jline.utils.Log logr
WARNING: Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information)
2022.04.16 15:26:35:988 io-compute-1 INFO docspell.config.ConfigFactory.default:36
    Using config file: /opt/docspell.conf
2022.04.16 15:26:37:0000 [io-comp...] [INFO ] docspell.restserver.Main.run:34 - 
***>  ______                          _ _
***>  |  _  \                        | | |
***>  | | | |___   ___ ___ _ __   ___| | |
***>  | | | / _ \ / __/ __| '_ \ / _ \ | |
***>  | |/ / (_) | (__\__ \ |_) |  __/ | |
***>  |___/ \___/ \___|___/ .__/ \___|_|_|
***>                      | |
***>                      |_| v0.35.0 (#f5958b3a)
***>  << REST Server >>
***>  Id:       rest1
***>  Base-Url: https://docspell.home
***>  Database: jdbc:postgresql://postgres.local:5432/docspell
***>  Fts:      PostgreSQL(default)
***>  Config:   /opt/docspell.conf
***>  FileRepo: DefaultDatabase(true)
***>  
2022.04.16 15:26:37:0001 [io-comp...] [INFO ] docspell.restserver.RestServer.serve:46 - Starting server with options ServerOptions(Duration(45000ms),false,1024)
2022.04.16 15:26:38:0000 [io-comp...] [INFO ] docspell.store.migrate.FlywayMigrate.createFlyway:38 - Creating Flyway for: List(classpath:db/fixups/postgresql, classpath:db/fixups/common)
2022.04.16 15:26:38:0001 [io-comp...] [INFO ] docspell.store.migrate.FlywayMigrate.runFixups:69 - !!! Running fixup migrations
2022.04.16 15:26:38:0002 [io-comp...] [INFO ] org.flywaydb.core.internal.license.VersionPrinter - Flyway Community Edition 8.5.8 by Redgate
2022.04.16 15:26:38:0003 [io-comp...] [INFO ] org.flywaydb.core.internal.license.VersionPrinter - See what's new here: https://flywaydb.org/documentation/learnmore/releaseNotes#8.5.8
2022.04.16 15:26:38:0004 [io-comp...] [INFO ] org.flywaydb.core.internal.license.VersionPrinter - 
2022.04.16 15:26:38:0005 [io-comp...] [INFO ] org.flywaydb.core.internal.database.base.BaseDatabaseType - Database: jdbc:postgresql://postgres.local:5432/docspell (PostgreSQL 14.2)
2022.04.16 15:26:38:0000 [io-comp...] [INFO ] org.http4s.blaze.client.PoolManager - Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
org.flywaydb.core.api.exception.FlywayValidateException: Validate failed: Migrations have failed validation
Migration checksum mismatch for migration version 1.33.0
-> Applied to database : 1347412019
-> Resolved locally    : 1776159438
Either revert the changes to the migration, or run repair to update the schema history.
Need more flexibility with validation rules? Learn more: https://rd.gt/3AbJUZE
        at org.flywaydb.core.Flyway$1.execute(Flyway.java:130)
        at org.flywaydb.core.Flyway$1.execute(Flyway.java:124)
        at org.flywaydb.core.FlywayExecutor.execute(FlywayExecutor.java:205)
        at org.flywaydb.core.Flyway.migrate(Flyway.java:124)
        at docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$8(FlywayMigrate.scala:70)
        at blocking @ fs2.io.file.FilesCompanionPlatform$AsyncFiles.exists(FilesPlatform.scala:204)
        at liftF @ docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$4(FlywayMigrate.scala:68)
        at map @ docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$7(FlywayMigrate.scala:70)
        at delay @ docspell.logging.impl.ScribeWrapper$Impl.log(ScribeWrapper.scala:28)
        at liftF @ docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$4(FlywayMigrate.scala:68)
        at flatMap @ docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$1(FlywayMigrate.scala:64)
        at delay @ docspell.logging.impl.ScribeWrapper$Impl.log(ScribeWrapper.scala:28)
        at map @ docspell.store.migrate.FlywayMigrate.$anonfun$createFlyway$1(FlywayMigrate.scala:38)
        at map @ docspell.store.migrate.FlywayMigrate.$anonfun$createFlyway$1(FlywayMigrate.scala:38)
        at flatMap @ docspell.store.migrate.FlywayMigrate.createFlyway(FlywayMigrate.scala:37)
        at liftF @ docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$4(FlywayMigrate.scala:68)
        at flatMap @ docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$1(FlywayMigrate.scala:64)
        at filter @ docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$2(FlywayMigrate.scala:67)
        at flatMap @ docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$1(FlywayMigrate.scala:64)
        at blocking @ fs2.io.file.FilesCompanionPlatform$AsyncFiles.exists(FilesPlatform.scala:204)
        at as @ docspell.restserver.RestServer$.$anonfun$serve$4(RestServer.scala:70)

flyway_fixup_history:

installed_rank	version	description	type	script	checksum	installed_by	installed_on	execution_time	success
1	1	<< Flyway Baseline >>	BASELINE	<< Flyway Baseline >>	;	null	2022-03-22 14:36:31.618387	"0"	1
2	"1.33.0"	fix reorganize files	SQL	V1.33.0__fix_reorganize_files.sql	1347412019	docspell	2022-04-16 14:25:40.151521	68	1

Deleting the 1.33.0 entry results in:

Apr 16, 2022 3:35:35 PM org.jline.utils.Log logr
WARNING: Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information)
2022.04.16 15:35:34:826 io-compute-2 INFO docspell.config.ConfigFactory.default:36
    Using config file: /opt/docspell.conf
2022.04.16 15:35:35:0000 [io-comp...] [INFO ] docspell.restserver.Main.run:34 - 
***>  ______                          _ _
***>  |  _  \                        | | |
***>  | | | |___   ___ ___ _ __   ___| | |
***>  | | | / _ \ / __/ __| '_ \ / _ \ | |
***>  | |/ / (_) | (__\__ \ |_) |  __/ | |
***>  |___/ \___/ \___|___/ .__/ \___|_|_|
***>                      | |
***>                      |_| v0.35.0 (#f5958b3a)
***>  << REST Server >>
***>  Id:       rest1
***>  Base-Url: https://docspell.home
***>  Database: jdbc:postgresql://postgres.local:5432/docspell
***>  Fts:      PostgreSQL(default)
***>  Config:   /opt/docspell.conf
***>  FileRepo: DefaultDatabase(true)
***>  
2022.04.16 15:35:35:0001 [io-comp...] [INFO ] docspell.restserver.RestServer.serve:46 - Starting server with options ServerOptions(Duration(45000ms),false,1024)
2022.04.16 15:35:37:0000 [io-comp...] [INFO ] docspell.store.migrate.FlywayMigrate.createFlyway:38 - Creating Flyway for: List(classpath:db/fixups/postgresql, classpath:db/fixups/common)
2022.04.16 15:35:37:0001 [io-comp...] [INFO ] docspell.store.migrate.FlywayMigrate.runFixups:69 - !!! Running fixup migrations
2022.04.16 15:35:37:0002 [io-comp...] [INFO ] org.flywaydb.core.internal.license.VersionPrinter - Flyway Community Edition 8.5.8 by Redgate
2022.04.16 15:35:37:0003 [io-comp...] [INFO ] org.flywaydb.core.internal.license.VersionPrinter - See what's new here: https://flywaydb.org/documentation/learnmore/releaseNotes#8.5.8
2022.04.16 15:35:37:0004 [io-comp...] [INFO ] org.flywaydb.core.internal.license.VersionPrinter - 
2022.04.16 15:35:37:0005 [io-comp...] [INFO ] org.flywaydb.core.internal.database.base.BaseDatabaseType - Database: jdbc:postgresql://postgres.local:5432/docspell (PostgreSQL 14.2)
2022.04.16 15:35:37:0006 [io-comp...] [INFO ] org.flywaydb.core.internal.command.DbValidate - Successfully validated 2 migrations (execution time 00:00.030s)
2022.04.16 15:35:37:0007 [io-comp...] [INFO ] org.flywaydb.core.internal.command.DbMigrate - Current version of schema "public": 1
2022.04.16 15:35:37:0008 [io-comp...] [INFO ] org.flywaydb.core.internal.command.DbMigrate - Migrating schema "public" to version "1.33.0 - fix reorganize files"
2022.04.16 15:35:37:0009 [io-comp...] [INFO ] org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor - DB: table "valid_file_ids" does not exist, skipping
2022.04.16 15:35:37:0010 [io-comp...] [INFO ] org.flywaydb.core.internal.sqlscript.DefaultSqlScriptExecutor - DB: table "obsolete_files" does not exist, skipping
2022.04.16 15:35:37:0011 [io-comp...] [INFO ] org.flywaydb.core.internal.command.DbMigrate - Successfully applied 1 migration to schema "public", now at version v1.33.0 (execution time 00:00.133s)
2022.04.16 15:35:37:0012 [io-comp...] [INFO ] docspell.store.migrate.FlywayMigrate.createFlyway:38 - Creating Flyway for: List(classpath:db/migration/postgresql, classpath:db/migration/common)
2022.04.16 15:35:37:0013 [io-comp...] [INFO ] docspell.store.migrate.FlywayMigrate.run:53 - !!! Running main migrations
2022.04.16 15:35:37:0014 [io-comp...] [INFO ] org.flywaydb.core.internal.license.VersionPrinter - Flyway Community Edition 8.5.8 by Redgate
2022.04.16 15:35:37:0015 [io-comp...] [INFO ] org.flywaydb.core.internal.license.VersionPrinter - See what's new here: https://flywaydb.org/documentation/learnmore/releaseNotes#8.5.8
2022.04.16 15:35:37:0016 [io-comp...] [INFO ] org.flywaydb.core.internal.license.VersionPrinter - 
2022.04.16 15:35:37:0017 [io-comp...] [INFO ] org.flywaydb.core.internal.database.base.BaseDatabaseType - Database: jdbc:postgresql://postgres.local:5432/docspell (PostgreSQL 14.2)
2022.04.16 15:35:37:0000 [io-comp...] [INFO ] org.http4s.blaze.client.PoolManager - Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
org.flywaydb.core.api.exception.FlywayValidateException: Validate failed: Migrations have failed validation
Migration checksum mismatch for migration version 1.0.0
-> Applied to database : 317337288
-> Resolved locally    : -118882639
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.1.0
-> Applied to database : 655930044
-> Resolved locally    : -1102699734
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.2.0
-> Applied to database : 811358974
-> Resolved locally    : 1092778025
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.3.0
-> Applied to database : -1440002901
-> Resolved locally    : -186517118
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.4.0
-> Applied to database : 857738097
-> Resolved locally    : 290692248
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.5.0
-> Applied to database : 405891450
-> Resolved locally    : -1626360500
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.6.0
-> Applied to database : -1400448292
-> Resolved locally    : 1094333603
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.7.0
-> Applied to database : 945262447
-> Resolved locally    : 1006625579
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.8.0
-> Applied to database : 1074998524
-> Resolved locally    : -1218545371
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.9.0
-> Applied to database : -922383879
-> Resolved locally    : -837034213
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.9.1
-> Applied to database : -920209386
-> Resolved locally    : 1434270743
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.9.3
-> Applied to database : -807943034
-> Resolved locally    : 710044106
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.9.4
-> Applied to database : 770293754
-> Resolved locally    : 136218986
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.10.0
-> Applied to database : 175251272
-> Resolved locally    : -649911439
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.12.0
-> Applied to database : -1136859296
-> Resolved locally    : -1608113072
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.13.0
-> Applied to database : 1090781225
-> Resolved locally    : -1680178524
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.14.0
-> Applied to database : -1121419990
-> Resolved locally    : 916718441
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.15.0
-> Applied to database : -456374843
-> Resolved locally    : 18578961
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.16.0
-> Applied to database : 1011393315
-> Resolved locally    : -252417141
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.17.0
-> Applied to database : -491746438
-> Resolved locally    : -198654654
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.18.0
-> Applied to database : -828258448
-> Resolved locally    : 683920003
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.19.0
-> Applied to database : -1255719254
-> Resolved locally    : 943341824
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.20.0
-> Applied to database : 1870008223
-> Resolved locally    : -779274773
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.20.3
-> Applied to database : 763931359
-> Resolved locally    : 907574747
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.21.0
-> Applied to database : 1165916844
-> Resolved locally    : -1714022554
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.21.1
-> Applied to database : 352694567
-> Resolved locally    : -1005379095
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.23.0
-> Applied to database : 1152128793
-> Resolved locally    : 636649519
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.25.0
-> Applied to database : -943147294
-> Resolved locally    : 943175394
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.25.1
-> Applied to database : 1797712347
-> Resolved locally    : -2048732059
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.25.2
-> Applied to database : 396118090
-> Resolved locally    : 1542597368
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.25.3
-> Applied to database : -902371387
-> Resolved locally    : -603580747
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.26.2
-> Applied to database : 944511097
-> Resolved locally    : -1897063579
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.26.3
-> Applied to database : 1476446475
-> Resolved locally    : 684359551
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.27.1
-> Applied to database : -1354199549
-> Resolved locally    : 644010924
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.28.0
-> Applied to database : -1745510720
-> Resolved locally    : -400090870
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.28.1
-> Applied to database : -537116935
-> Resolved locally    : 553402105
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.29.0
-> Applied to database : -1762725568
-> Resolved locally    : 1802387995
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.29.1
-> Applied to database : 1582374027
-> Resolved locally    : 1814856936
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.30.0
-> Applied to database : 1647444395
-> Resolved locally    : 936029495
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.31.0
-> Applied to database : 1464170959
-> Resolved locally    : 525900721
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.32.0
-> Applied to database : 1586845926
-> Resolved locally    : 1750332662
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.32.1
-> Applied to database : -741341079
-> Resolved locally    : -722702183
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.33.0
-> Applied to database : 1583732153
-> Resolved locally    : 924153984
Either revert the changes to the migration, or run repair to update the schema history.
Migration checksum mismatch for migration version 1.34.0
-> Applied to database : -146289187
-> Resolved locally    : -750119324
Either revert the changes to the migration, or run repair to update the schema history.
Need more flexibility with validation rules? Learn more: https://rd.gt/3AbJUZE
        at org.flywaydb.core.Flyway$1.execute(Flyway.java:130)
        at org.flywaydb.core.Flyway$1.execute(Flyway.java:124)
        at org.flywaydb.core.FlywayExecutor.execute(FlywayExecutor.java:205)
        at org.flywaydb.core.Flyway.migrate(Flyway.java:124)
        at docspell.store.migrate.FlywayMigrate.$anonfun$run$5(FlywayMigrate.scala:54)
        at blocking @ fs2.io.file.FilesCompanionPlatform$AsyncFiles.exists(FilesPlatform.scala:204)
        at map @ docspell.store.migrate.FlywayMigrate.$anonfun$run$4(FlywayMigrate.scala:54)
        at delay @ docspell.logging.impl.ScribeWrapper$Impl.log(ScribeWrapper.scala:28)
        at flatMap @ docspell.store.migrate.FlywayMigrate.$anonfun$run$2(FlywayMigrate.scala:53)
        at delay @ docspell.logging.impl.ScribeWrapper$Impl.log(ScribeWrapper.scala:28)
        at map @ docspell.store.migrate.FlywayMigrate.$anonfun$createFlyway$1(FlywayMigrate.scala:38)
        at map @ docspell.store.migrate.FlywayMigrate.$anonfun$createFlyway$1(FlywayMigrate.scala:38)
        at flatMap @ docspell.store.migrate.FlywayMigrate.createFlyway(FlywayMigrate.scala:37)
        at flatMap @ docspell.store.migrate.FlywayMigrate.$anonfun$run$1(FlywayMigrate.scala:52)
        at blocking @ fs2.io.file.FilesCompanionPlatform$AsyncFiles.exists(FilesPlatform.scala:204)
        at liftF @ docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$4(FlywayMigrate.scala:68)
        at map @ docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$7(FlywayMigrate.scala:70)
        at delay @ docspell.logging.impl.ScribeWrapper$Impl.log(ScribeWrapper.scala:28)
        at liftF @ docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$4(FlywayMigrate.scala:68)
        at flatMap @ docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$1(FlywayMigrate.scala:64)
        at delay @ docspell.logging.impl.ScribeWrapper$Impl.log(ScribeWrapper.scala:28)

flyway_fixup_history:

installed_rank	version	description	type	script	checksum	installed_by	installed_on	execution_time	success
1	1	<< Flyway Baseline >>	BASELINE	<< Flyway Baseline >>	;	null	2022-03-22 14:36:31.618387	"0"	1
2	"1.33.0"	fix reorganize files	SQL	V1.33.0__fix_reorganize_files.sql	1776159438	docspell	2022-04-16 15:35:37.391163	70	1

For reference, latest five entries in flyway_schema_history:

installed_rank	version	description	type	script	checksum	installed_by	installed_on	execution_time	success
54	"1.32.0"	notification channel name	SQL	V1.32.0__notification_channel_name.sql	1586845926	docspell	2022-02-11 14:49:04.22228	1	1
55	1.32.1	notification hook multi channel	SQL	V1.32.1__notification_hook_multi_channel.sql	-741341079	docspell	2022-02-11 14:49:04.228573	25	1
56	1.32.2	MigrateChannels	JDBC	db.migration.postgresql.V1_32_2__MigrateChannels		docspell	2022-02-11 14:49:04.257781	39	1
57	"1.33.0"	reorganize file ids	SQL	V1.33.0__reorganize_file_ids.sql	1583732153	docspell	2022-03-06 19:19:58.924624	131	1
58	"1.34.0"	item relation	SQL	V1.34.0__item_relation.sql	-146289187	docspell	2022-03-22 14:36:31.955354	26	1

Goging back to v.0.34.0 results in:

Apr 16, 2022 3:42:16 PM org.jline.utils.Log logr
WARNING: Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information)
2022.04.16 15:42:15:842 io-compute-2 INFO docspell.config.ConfigFactory.default:36
    Using config file: /opt/docspell.conf
2022.04.16 15:42:16:0000 [io-comp...] [INFO ] docspell.restserver.Main.run:34 - 
***>  ______                          _ _
***>  |  _  \                        | | |
***>  | | | |___   ___ ___ _ __   ___| | |
***>  | | | / _ \ / __/ __| '_ \ / _ \ | |
***>  | |/ / (_) | (__\__ \ |_) |  __/ | |
***>  |___/ \___/ \___|___/ .__/ \___|_|_|
***>                      | |
***>                      |_| v0.34.0 (#c6da3eb0)
***>  << REST Server >>
***>  Id:       rest1
***>  Base-Url: https://docspell.home
***>  Database: jdbc:postgresql://postgres.local:5432/docspell
***>  Fts:      PostgreSQL(default)
***>  Config:   /opt/docspell.conf
***>  FileRepo: DefaultDatabase(true)
***>  
2022.04.16 15:42:18:0000 [io-comp...] [INFO ] docspell.store.migrate.FlywayMigrate.createFlyway:37 - Creating Flyway for: List(classpath:db/fixups/postgresql, classpath:db/fixups/common)
2022.04.16 15:42:18:0001 [io-comp...] [INFO ] docspell.store.migrate.FlywayMigrate.runFixups:64 - !!! Running fixup migrations
2022.04.16 15:42:18:0002 [io-comp...] [INFO ] org.flywaydb.core.internal.license.VersionPrinter - Flyway Community Edition 8.5.4 by Redgate
2022.04.16 15:42:18:0003 [io-comp...] [INFO ] org.flywaydb.core.internal.license.VersionPrinter - See what's new here: https://flywaydb.org/documentation/learnmore/releaseNotes#8.5.4
2022.04.16 15:42:18:0004 [io-comp...] [INFO ] org.flywaydb.core.internal.license.VersionPrinter - 
2022.04.16 15:42:18:0005 [io-comp...] [INFO ] org.flywaydb.core.internal.database.base.BaseDatabaseType - Database: jdbc:postgresql://postgres.local:5432/docspell (PostgreSQL 14.2)
2022.04.16 15:42:18:0000 [io-comp...] [INFO ] org.http4s.blaze.client.PoolManager - Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
org.flywaydb.core.api.exception.FlywayValidateException: Validate failed: Migrations have failed validation
Migration checksum mismatch for migration version 1.33.0
-> Applied to database : 1776159438
-> Resolved locally    : 1347412019
Either revert the changes to the migration, or run repair to update the schema history.
Need more flexibility with validation rules? Learn more: https://rd.gt/3AbJUZE
        at org.flywaydb.core.Flyway$1.execute(Flyway.java:130)
        at org.flywaydb.core.Flyway$1.execute(Flyway.java:124)
        at org.flywaydb.core.FlywayExecutor.execute(FlywayExecutor.java:205)
        at org.flywaydb.core.Flyway.migrate(Flyway.java:124)
        at docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$5(FlywayMigrate.scala:65)
        at blocking @ fs2.io.file.FilesCompanionPlatform$AsyncFiles.exists(FilesPlatform.scala:204)
        at map @ docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$4(FlywayMigrate.scala:65)
        at delay @ docspell.logging.impl.ScribeWrapper$Impl.log(ScribeWrapper.scala:28)
        at flatMap @ docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$2(FlywayMigrate.scala:64)
        at delay @ docspell.logging.impl.ScribeWrapper$Impl.log(ScribeWrapper.scala:28)
        at map @ docspell.store.migrate.FlywayMigrate.$anonfun$createFlyway$1(FlywayMigrate.scala:37)
        at map @ docspell.store.migrate.FlywayMigrate.$anonfun$createFlyway$1(FlywayMigrate.scala:37)
        at flatMap @ docspell.store.migrate.FlywayMigrate.createFlyway(FlywayMigrate.scala:36)
        at flatMap @ docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$1(FlywayMigrate.scala:63)
        at blocking @ fs2.io.file.FilesCompanionPlatform$AsyncFiles.exists(FilesPlatform.scala:204)
        at as @ docspell.restserver.RestServer$.$anonfun$serve$1(RestServer.scala:67)
        at $anonfun$apply$10 @ fs2.Stream$.$anonfun$resourceWeak$3(Stream.scala:3624)
        at $anonfun$apply$10 @ fs2.Stream$.$anonfun$resourceWeak$3(Stream.scala:3624)
        at blocking @ fs2.io.file.FilesCompanionPlatform$AsyncFiles.exists(FilesPlatform.scala:204)
        at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:162)
        at $anonfun$apply$10 @ fs2.Stream$.$anonfun$resourceWeak$3(Stream.scala:3624)

Deleting the 1.33.0 entry from flyway_fixup_history results in a new entry after startup, now without any errors and Docspell works fien as before with v0.34.0.

flyway_fixup_history:

installed_rank	version	description	type	script	checksum	installed_by	installed_on	execution_time	success
1	1	<< Flyway Baseline >>	BASELINE	<< Flyway Baseline >>	;	null	2022-03-22 14:36:31.618387	"0"	1
2	"1.33.0"	fix reorganize files	SQL	V1.33.0__fix_reorganize_files.sql	1347412019	docspell	2022-04-16 15:44:10.942675	61	1

[eikek]

Thank you very much for these good details! I'll take a deeper look , currently I admit I'm quite puzzled. It seems that the fix for the flyway_fixup_migration succeeded and then the same error happens to the main migration. That is very weird! No migration was changed, so checksums should not change as well.

I found one issue flyway/flyway#3431 that seems to describe this problem.

[enkol]

This also happens with an "empty" database i used goging from postgres 13 to 14. If needed i can provide a dump of it alongside my docker-compose configuration.

[enkol]

Thanks for all your efforts! For me this is far from urgent, as i was able to go back to v0.34.0 without issues. So I'm fine for now.

[eikek]

Thanks @enkol ! I'll come back if I need more data, thanks for the proposal. If that happens even to an empty database, there is something very spooky going on :)! One more thing: How/which system are you running this on? Is it the docker arm images?

[enkol]

I'm running with Docker in an Ubuntu 20.04 VM on a Intel i3 box with Proxmox 7. Postgres is also with Docker postgres:14-bullseye in the same VM.

[enkol]

If that happens even to an empty database, there is something very spooky going on :)!

Just to clarify, i meant 'empty' in terms of no documents uploaded. The database was used to test setting up Docspell with Postgres 14 in Februar. I can reproduce the checksum issues with this db switching between 0.34.0 and 0.35.0.

[Skyr]

Same symptoms here. The flyway ticket mentioned shows the problem appears after moving from JDK 11 to JDK 17... did you change the JRE in the containers? If yes, then reverting to the old version might be a temproary fix.

My wild guess would be some strange encoding behaviour when reading text files (like changing the line endings)...

[eikek]

I didn't change anything in that regard. Could be related to a new flyway version in combination with the alpine jdk. But have to reproduce this first. I don't use docker, but I tested the docker image and that worked here 🤷🏼

[eikek]

So, unfortunately I can't reproduce it :( I even installed an Ubuntu 20.04 VM (via virtualbox) to run the docker containers (using docker-compose up -- db restserver - to only start the restserver). I first started with version 0.34.0 a few times and then moved to 0.35.0. I also tried the upgrade on a RPI4 (docker) and with OpenJDK17 on my amd notebook (no docker). The error doesn't occur.

From what I can see, flyway uses a CRC32 checksum and I don't see why this would change when upgrading java or anything else in the environment. It should only depend on its input, no? Their code didn't change as well. The docker images didn't change, besides from upgraded packages due to new image creation.

Not sure how to proceed :/ Maybe we can try to find common characteristics of the systems where this happens? Are you all on a VM, for example? or all using proxmox? Maybe a different openjdk is also worth a try, there could be a problem with openjdk11 and alpine… (but not for me :)). But since it happens for you only with 0.35.0 a change in the image that is triggered in your environment seems most likely to me for now.

Flyway just had another patch release, it could also be worth a try - you could run the nightly docker images that have this version.

[enkol]

Thanks for having a deeper look into this and spending so much effort to reproduce it yourself!
I'll try a nightly image this weekend, to see if it changes anything. If not, I can see if I'm able to provide a docker-compose and pg-dump to reproduce the issue.

[eikek]

Another option is what @Skyr said: charset/encoding changes. This would affect the input to the checksum algo, because flyway converts the bytes to characters and then converts characters back to bytes (line by line) using UTF-8. That means if the encoding to read the migration somehow changed, it could produce a different checksum. From what I saw, flyway is by default using UTF-8 (unless it is set explicitely). You could try to change the default encoding for java using the system property file.encoding. You can inject it by setting a special environment variable:

JAVA_TOOL_OPTIONS="-Dfile.encoding=UTF-8"

The migrations are in ascii, it doesn't matter so much (except for some esoteric and non-8-bit charsets) how they are read. But worth a try.

[enkol]

Nightly build produces the same error, i'll try the encoding thingy next.
Edit: Changing the encoding didn't help either.

Apr 23, 2022 4:09:13 PM org.jline.utils.Log logr
WARNING: Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information)
2022.04.23 16:09:12:881 io-compute-1 INFO docspell.config.ConfigFactory.default:36
    Using config file: /opt/docspell.conf
2022.04.23 16:09:14:0000 [io-comp...] [INFO ] docspell.restserver.Main.run:34 - 
***>  ______                          _ _
***>  |  _  \                        | | |
***>  | | | |___   ___ ___ _ __   ___| | |
***>  | | | / _ \ / __/ __| '_ \ / _ \ | |
***>  | |/ / (_) | (__\__ \ |_) |  __/ | |
***>  |___/ \___/ \___|___/ .__/ \___|_|_|
***>                      | |
***>                      |_| v0.36.0-SNAPSHOT (#d8727ef6)
***>  << REST Server >>
***>  Id:       rest1
***>  Base-Url: https://docspell.home
***>  Database: jdbc:postgresql://postgres.local:5432/docspell
***>  Fts:      PostgreSQL(default)
***>  Config:   /opt/docspell.conf
***>  FileRepo: DefaultDatabase(true)
***>  
2022.04.23 16:09:14:0001 [io-comp...] [INFO ] docspell.restserver.RestServer.serve:46 - Starting server with options ServerOptions(Duration(45000ms),false,1024)
2022.04.23 16:09:15:0000 [io-comp...] [INFO ] docspell.store.migrate.FlywayMigrate.createFlyway:38 - Creating Flyway for: List(classpath:db/fixups/postgresql, classpath:db/fixups/common)
2022.04.23 16:09:15:0001 [io-comp...] [INFO ] docspell.store.migrate.FlywayMigrate.runFixups:69 - !!! Running fixup migrations
2022.04.23 16:09:15:0002 [io-comp...] [INFO ] org.flywaydb.core.internal.license.VersionPrinter - Flyway Community Edition 8.5.9 by Redgate
2022.04.23 16:09:15:0003 [io-comp...] [INFO ] org.flywaydb.core.internal.license.VersionPrinter - See what's new here: https://flywaydb.org/documentation/learnmore/releaseNotes#8.5.9
2022.04.23 16:09:15:0004 [io-comp...] [INFO ] org.flywaydb.core.internal.license.VersionPrinter - 
2022.04.23 16:09:15:0005 [io-comp...] [INFO ] org.flywaydb.core.internal.database.base.BaseDatabaseType - Database: jdbc:postgresql://postgres.local:5432/docspell (PostgreSQL 14.2)
2022.04.23 16:09:15:0000 [io-comp...] [INFO ] org.http4s.blaze.client.PoolManager - Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
org.flywaydb.core.api.exception.FlywayValidateException: Validate failed: Migrations have failed validation
Migration checksum mismatch for migration version 1.33.0
-> Applied to database : 1347412019
-> Resolved locally    : 1776159438
Either revert the changes to the migration, or run repair to update the schema history.
Need more flexibility with validation rules? Learn more: https://rd.gt/3AbJUZE
        at org.flywaydb.core.Flyway$1.execute(Flyway.java:130)
        at org.flywaydb.core.Flyway$1.execute(Flyway.java:124)
        at org.flywaydb.core.FlywayExecutor.execute(FlywayExecutor.java:205)
        at org.flywaydb.core.Flyway.migrate(Flyway.java:124)
        at docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$8(FlywayMigrate.scala:70)
        at blocking @ fs2.io.file.FilesCompanionPlatform$AsyncFiles.exists(FilesPlatform.scala:204)
        at liftF @ docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$4(FlywayMigrate.scala:68)
        at map @ docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$7(FlywayMigrate.scala:70)
        at delay @ docspell.logging.impl.ScribeWrapper$Impl.log(ScribeWrapper.scala:28)
        at liftF @ docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$4(FlywayMigrate.scala:68)
        at flatMap @ docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$1(FlywayMigrate.scala:64)
        at delay @ docspell.logging.impl.ScribeWrapper$Impl.log(ScribeWrapper.scala:28)
        at map @ docspell.store.migrate.FlywayMigrate.$anonfun$createFlyway$1(FlywayMigrate.scala:38)
        at map @ docspell.store.migrate.FlywayMigrate.$anonfun$createFlyway$1(FlywayMigrate.scala:38)
        at flatMap @ docspell.store.migrate.FlywayMigrate.createFlyway(FlywayMigrate.scala:37)
        at liftF @ docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$4(FlywayMigrate.scala:68)
        at flatMap @ docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$1(FlywayMigrate.scala:64)
        at filter @ docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$2(FlywayMigrate.scala:67)
        at flatMap @ docspell.store.migrate.FlywayMigrate.$anonfun$runFixups$1(FlywayMigrate.scala:64)
        at blocking @ fs2.io.file.FilesCompanionPlatform$AsyncFiles.exists(FilesPlatform.scala:204)
        at as @ docspell.restserver.RestServer$.$anonfun$serve$4(RestServer.scala:70)

[enkol]

I've prepared a minimal docker-compose to reproduce the issue:
https://1drv.ms/u/s!Ahri17Z7_U8VxU22A3LFjAIrp2tj?e=vFqbkA