Option to validate only if all migration are applied

[ysfaran]

As far as I can tell this library supports running migrations, but is there also an option to just check if all migration where applied?

I am asking because I want to integrate a step into my CI/CD pipeline, where I block merges if not all migration where applied to the search database. This step would ensure that you don't accidentally delete a migration script or add new ones without applying them and so on.

Ideally the java program should return a non-zero error code, when not all migrations where applied, which is equal to just throwing an error.

[ysfaran]

I just had rough look and it seems like this would be quite simple to implement:

You basically need to create a new validate function in the MigrationServiceImpl that mimics this code, but throws an error (validation failure case) or nothing (validation success case):

elasticsearch-evolution/elasticsearch-evolution-core/src/main/java/com/senacor/elasticsearch/evolution/core/internal/migration/execution/MigrationServiceImpl.java

Lines 196 to 216 in f57ee90

	List<MigrationScriptProtocol> history = new ArrayList<>(historyRepository.findAll());
	List<ParsedMigrationScript> res = new ArrayList<>(scriptsInFilesystemMap.values());
	if (outOfOrder) {
	for (MigrationScriptProtocol protocol : history) {
	final ParsedMigrationScript parsedMigrationScript = scriptsInFilesystemMap.get(protocol.getVersion());
	if (null == parsedMigrationScript) {
	logger.warn("""
	there are less migration scripts than already executed history entries! \
	You should never delete migration scripts you have already executed. \
	Or maybe you have to cleanup the Elasticsearch-Evolution history index manually! \
	Already executed history version {} is not present in migration files\
	""", protocol.getVersion());
	} else {
	validateOnMigrateIfEnabled(protocol, parsedMigrationScript);
	if (protocol.isSuccess()) {
	res.remove(parsedMigrationScript);
	}
	}
	}
	} else {

[ysfaran]

Until this is implemented officially, I created a really dirty workaround with hard coded config values:

public void runValidate() throws IOException {
        LOG.info("Running validation..");

        final var openSearchClient = getOpenSearchClient(domainEndpointUrl);

        SearchRequest request = new SearchRequest.Builder()
            .index("es_evolution")
            .build();

        SearchResponse<EvolutionHistoryItem> response = openSearchClient.search(request, EvolutionHistoryItem.class);
        List<EvolutionHistoryItem> historyItems = response.hits().hits().stream()
                                                          .map(Hit::source)
                                                          .toList();

        final var resourcePatternResolver = ResourcePatternUtils.getResourcePatternResolver(resourceLoader);
        final var resources = Arrays.asList(resourcePatternResolver.getResources("classpath:" + Path.of("es", "migration", "*.http")));

        if (resources.size() != historyItems.size()) {
            throw new MigrationValidationError("Migration scripts count mismatch: " + resources.size() + " (local) != " + historyItems.size() + " (OpenSearch)");
        }

        for (var resource : resources) {
            final var matchedHistoryItem = historyItems.stream()
                                                       .filter(item -> item.getScriptName().equals(resource.getFilename()))
                                                       .findFirst();

            if (matchedHistoryItem.isEmpty()) {
                throw new MigrationValidationError("Migration script not found in history: " + resource.getFilename());
            }

            if (matchedHistoryItem.get().getChecksum() != getResourceContent(resource).hashCode()) {
                throw new MigrationValidationError("Migration script checksum mismatch: " + resource.getFilename());
            }
            LOG.info("Successfully checked migration file: {}", resource.getFilename());
        }

        LOG.info("Validation finished successfully.");
    }

    public String getResourceContent(Resource resource) throws IOException {
        try (Reader reader = new InputStreamReader(resource.getInputStream(), StandardCharsets.UTF_8)) {
            return FileCopyUtils.copyToString(reader);
        }
    }

    public OpenSearchClient getOpenSearchClient(final String domainEndpointUrl) {

        SdkAsyncHttpClient httpClient = NettyNioAsyncHttpClient.builder()
                                                               .sslProvider(SslProvider.JDK).build();

        OpenSearchTransport transport = new AwsSdk2Transport(
            httpClient,
            domainEndpointUrl.split("https://")[1],
            "es",
            Region.of(awsRegion),
            AwsSdk2TransportOptions.builder().setCredentials(DefaultCredentialsProvider.create()).build()
        );

        return new OpenSearchClient(transport);
    }

This solution uses OpenSearchClient and it also doesn't cover all corner cases. It basically just checks for filename match and content match based on the existing checksum. Use it with caution.