HDDS-14020. Fix Deadlock in Background service shutdown

[swamirishi]

What changes were proposed in this pull request?

Currently both background task and shutdown acquire a synchronized lock which can lead to a deadlock on shutdown which means the threadPool shutdown takes time unnecessarily. The shutdown thread is waiting for the task to finish while acquiring a synchronized lock and task requires a lock to finish which is not possible.
The synchronized block on task was taken to ensure there are no parallel tasks that run together. Introducing a semaphore with 1 would solve this issue.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-14020

How was this patch tested?

Unit test in Defrag service finishes faster after this which takes a lot of time to shutdown otherwise

[sumitagrawl]

background task are added to async executor, and comes out of loop. it will be fraction of second to be in this block, and task count will be max 10s in number. So this is not a problem.

[swamirishi]

No this would be a problem because the shutdown thread would wait for the threads to finish for the tasks already present in the queue of the threadPoolExecutor. Those have to finish first. Right now the shutdown function would always end up doing a force shutdown if the interval is small enough.
This wait would happen unnecessarily everytime.

ozone/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BackgroundService.java

Lines 198 to 200 in c42fc3a

	if (!exec.awaitTermination(60, TimeUnit.SECONDS)) {
	exec.shutdownNow();
	}

[github-actions]

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

[github-actions]

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

[smengcl]

Closing this one in favor of #9390