indexParallel exceptions should bubble up

[vladak]

A failure in indexParallel such as this one:

2024-11-19 08:07:02.605+0000 SEVERE t117 IndexDatabase.indexParallel: 674 successes (98.8%) after aborting parallel-indexing
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
        at org.opengrok.indexer.index.IndexDatabase.indexParallel(IndexDatabase.java:2004)
        at org.opengrok.indexer.index.IndexDatabase.update(IndexDatabase.java:765)
        at org.opengrok.indexer.index.Indexer.lambda$doIndexerExecution$59(Indexer.java:1212)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.OutOfMemoryError: Java heap space

will more often than not have grave consequences on the index, eventually leading to index corruption. Currently such errors are logged only:

opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/index/IndexDatabase.java

Lines 2007 to 2014 in 41b2c23

	} catch (InterruptedException | ExecutionException e) {
	interrupted = true;
	int successCount = successCounter.intValue();
	double successPct = 100.0 * successCount / worksCount;
	String exmsg = String.format("%d successes (%.1f%%) after aborting parallel-indexing",
	successCount, successPct);
	LOGGER.log(Level.SEVERE, exmsg, e);
	}

Instead, these should be reported in the overall index exit code on

opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/index/Indexer.java

Lines 446 to 455 in 41b2c23

	} finally {
	env.shutdownSearchExecutor();
	/*
	* Normally the IndexParallelizer is bounced (i.e. thread pools within are terminated)
	* via auto-closed in doIndexerExecution(), however there are cases (--noIndex) that
	* avoid that path, yet use the IndexParallelizer. So, bounce it here for a good measure.
	*/
	env.getIndexerParallelizer().bounce();
	stats.report(LOGGER, "Indexer finished", "indexer.total");
	}

so they can be reflected e.g. in the Docker container program which already consults the exit code. Obviously this will be more handy in case of per project reindex as opposed to reindex of many projects at once.

[vladak]

This was a case of per project (linux Git repo) reindex with the indexer running with max 32 GB heap space and annotation + history cache enabled. It seems to me this should not cause heap shortage so should be further investigated.