Add leading silence detection and removal logic (#17648)

Closes #17614

Summary of the issue:
Some voices output a leading silence part before the actual speech voice. By removing the silence part, the delay between keypress and user hearing the audio will be shorter, therefore make the voices more responsive.

Description of user facing changes
Users may find the voices more responsive. All voices using NVDA's WavePlayer will be affected, including eSpeak-NG, OneCore, SAPI5, and some third-party voice add-ons.

This should only affect the leading silence parts. Silence between sentences or at punctuation marks are not changed, but this may depend on how the voice uses WavePlayer.

Description of development approach
I wrote a header-only library silenceDetect.h in nvdaHelper/local. It supports most wave formats (8/16/24/32-bit integer and 32/64-bit float wave), and uses a simple algorithm: check each sample to see if it's outside threshold range (currently hard-coded to +/- 1/2^10 or 0.0009765625). It uses template-related code and requires C++ 20 standard.

The WasapiPlayer in wasapi.cpp is updated to handle silence. A new member function, startTrimmingLeadingSilence, and the exported version wasPlay_startTrimmingLeadingSilence, is added, to set or clear the isTrimmingLeadingSilence flag. If isTrimmingLeadingSilence is true, the next chunk fed in will have its leading silence removed. When non-silence is detected, isTrimmingLeadingSilence will be reset to false. So every time a new utterance is about to be spoken, startTrimmingLeadingSilence should be called.

In nvwave.py, startTrimmingLeadingSilence() will be called when:

the player is initialized;
the player is stopped;
idle is called;
_idleCheck determines that the player is idle.
Usually voices will call idle when an utterance is completed, so that audio ducking can work correctly, so here idle is used to mark the starting point of the next utterance. If a voice doesn't use idle this way, then this logic might be messed up.

As long as the synthesizer uses idle as intended, the synthesizer's code doesn't need to be modified to benefit from this feature.

As leading silence can also be introduced by a BreakCommand at the beginning of the speech sequence, WavePlayer will check the speech sequence first; if there's a BreakCommand at the beginning, the leading silence will not be trimmed for the current utterance. To check the exact speech sequence that is about to be spoken, a new extension point, pre_synthSpeak, is added in synthDriverHandler, which will be invoked just before SpeechManager calls getSynth().speak(). The existing pre_speech is called before SpeechManager processes and queues the speech sequence, so pre_synthSpeak is needed to provide a more accurate sequence.

When the purpose of a WavePlayer is not SPEECH, it does not trim the leading silence by default, because of the way playWaveFile works (it calls idle after every chunk). Users of WavePlayer will still be able to enable/disable automatic trimming by calling enableTrimmingLeadingSilence, or to initiate trimming manually for the next audio section by calling startTrimmingLeadingSilence.

Other possible ways/things that may worth considering (but hasn't been implemented):

Put silence detection/removal logic in a separate module instead of in WavePlayer. The drawback is that every voice synthesizer module needs to be modified to utilize a separate module.
Use another audio library, such as PyDub, to detect/remove silence.
Add a setting item to turn this feature on or off.
Add a public API function to allow a synthesizer to opt out of this feature.

Author: gexgd0419

nvdaHelper/local/nvdaHelperLocal.def

@@ -82,6 +82,7 @@ EXPORTS
 	wasPlay_pause
 	wasPlay_resume
 	wasPlay_setChannelVolume
+	wasPlay_startTrimmingLeadingSilence
 	wasPlay_startup
 	wasSilence_init
 	wasSilence_playFor

nvdaHelper/local/silenceDetect.h

@@ -0,0 +1,219 @@
+// A part of NonVisual Desktop Access (NVDA)
+// This file is covered by the GNU General Public License.
+// See the file COPYING for more details.
+// Copyright (C) 2025 NV Access Limited, gexgd0419
+
+#ifndef SILENCEDETECT_H
+#define SILENCEDETECT_H
+
+#include <windows.h>
+#include <mmreg.h>
+#include <stdint.h>
+#include <type_traits>
+#include <limits>
+
+namespace SilenceDetect {
+
+/**
+ * Compile-time wave format tag.
+ * Supports integer and floating-point formats.
+ * `SampleType` should be the smallest numeric type that can hold a sample, for example, 32-bit int for 24-bit format.
+ * Signedness of `SampleType` matters. For unsigned types, the zero point is at middle, e.g. 128 for 8-bit unsigned.
+ * `bytesPerSample` should be <= `sizeof(SampleType)` for integer formats,
+ * and == `sizeof(SampleType)` for floating-point formats.
+ * Assumes C++20 standard.
+ */
+template <typename SampleType, size_t bytesPerSample = sizeof(SampleType)>
+struct WaveFormat {
+	static_assert(std::is_arithmetic_v<SampleType>, "SampleType should be an integer or floating-point type");
+	static_assert(!(std::is_floating_point_v<SampleType> && bytesPerSample != sizeof(SampleType)),
+		"When SampleType is a floating-point type, bytesPerSample should be equal to sizeof(SampleType)");
+	static_assert(!(std::is_integral_v<SampleType> && !(bytesPerSample <= sizeof(SampleType) && bytesPerSample > 0)),
+		"When SampleType is an integer type, bytesPerSample should be less than or equal to sizeof(SampleType) and greater than 0");
+
+	typedef SampleType SampleType;
+	static constexpr size_t bytesPerSample = bytesPerSample;
+
+	static constexpr SampleType zeroPoint() {
+		// for unsigned types, zero point is at middle
+		// for signed types, zero is zero
+		if constexpr (std::is_unsigned_v<SampleType>)
+			return SampleType(1) << (bytesPerSample * 8 - 1);
+		else
+			return SampleType();
+	}
+
+	static constexpr SampleType (max)() {
+		if constexpr (std::is_floating_point_v<SampleType>) {
+			// For floating-point samples, maximum value is 1.0
+			return SampleType(1);
+		} else {
+			// Trim the maximum value to `bytesPerSample` bytes
+			return (std::numeric_limits<SampleType>::max)() >> ((sizeof(SampleType) - bytesPerSample) * 8);
+		}
+	}
+
+	static constexpr SampleType (min)() {
+		if constexpr (std::is_floating_point_v<SampleType>) {
+			// For floating-point samples, minimum value is -1.0
+			return SampleType(-1);
+		} else {
+			// Trim the minimum value to `bytesPerSample` bytes
+			return (std::numeric_limits<SampleType>::min)() >> ((sizeof(SampleType) - bytesPerSample) * 8);
+		}
+	}
+
+	static constexpr SampleType defaultThreshold() {
+		// Default threshold: 1 / 2^10 or 0.0009765625
+		if constexpr (std::is_floating_point_v<SampleType>)
+			return SampleType(1) / (1 << 10);
+		else if constexpr (bytesPerSample * 8 > 10)
+			return SampleType(1) << (bytesPerSample * 8 - 10);
+		else
+			return SampleType();
+	}
+
+	static constexpr auto toSigned(SampleType smp) {
+		if constexpr (std::is_integral_v<SampleType>) {
+			// In C++20, signed integer types must use two's complement,
+			// so the following conversion is well-defined.
+			using SignedType = std::make_signed_t<SampleType>;
+			return SignedType(smp - zeroPoint());
+		} else {
+			return smp;
+		}
+	}
+
+	static constexpr SampleType fromSigned(SampleType smp) {
+		if constexpr (std::is_integral_v<SampleType>) {
+			// Signed overflow is undefined behavior,
+			// so convert to unsigned first.
+			using UnsignedType = std::make_unsigned_t<SampleType>;
+			return SampleType(UnsignedType(smp) + zeroPoint());
+		} else {
+			return smp;
+		}
+	}
+
+	static constexpr SampleType signExtend(SampleType smp) {
+		if constexpr (std::is_unsigned_v<SampleType> || bytesPerSample == sizeof(SampleType)) {
+			return smp;
+		} else {
+			constexpr auto shift = (sizeof(SampleType) - bytesPerSample) * 8;
+			// Convert to unsigned first to prevent left-shifting negative numbers
+			using UnsignedType = std::make_unsigned_t<SampleType>;
+			return SampleType(UnsignedType(smp) << shift) >> shift;
+		}
+	}
+};
+
+inline WORD getFormatTag(const WAVEFORMATEX* wfx) {
+	if (wfx->wFormatTag == WAVE_FORMAT_EXTENSIBLE) {
+		auto wfext = reinterpret_cast<const WAVEFORMATEXTENSIBLE*>(wfx);
+		if (IS_VALID_WAVEFORMATEX_GUID(&wfext->SubFormat))
+			return EXTRACT_WAVEFORMATEX_ID(&wfext->SubFormat);
+	}
+	return wfx->wFormatTag;
+}
+
+/**
+ * Return the leading silence wave data length, in bytes.
+ * Assumes the wave data to be of one channel (mono).
+ * Uses a `WaveFormat` type (`Fmt`) to determine the wave format.
+ */
+template <class Fmt>
+size_t getLeadingSilenceSizeMono(
+	const unsigned char* waveData,
+	size_t size,
+	typename Fmt::SampleType threshold
+) {
+	using SampleType = Fmt::SampleType;
+	constexpr size_t bytesPerSample = Fmt::bytesPerSample;
+
+	if (size < bytesPerSample)
+		return 0;
+
+	constexpr SampleType zeroPoint = Fmt::zeroPoint();
+	const SampleType minValue = zeroPoint - threshold, maxValue = zeroPoint + threshold;
+
+	// Check each sample
+	SampleType smp = SampleType();
+	const unsigned char* const pEnd = waveData + (size - (size % bytesPerSample));
+	for (const unsigned char* p = waveData; p < pEnd; p += bytesPerSample) {
+		memcpy(&smp, p, bytesPerSample);
+		smp = Fmt::signExtend(smp);
+		// this sample is out of range, so the previous sample is the final sample of leading silence.
+		if (smp < minValue || smp > maxValue)
+			return p - waveData;
+	}
+
+	// The whole data block is silence
+	return size;
+}
+
+/**
+ * Invoke a functor with an argument of a WaveFormat type that corresponds to the specified WAVEFORMATEX.
+ * Return false if the WAVEFORMATEX is unknown.
+ */
+template <class Func>
+bool callByWaveFormat(const WAVEFORMATEX* wfx, Func&& func) {
+	switch (getFormatTag(wfx)) {
+	case WAVE_FORMAT_PCM:
+		switch (wfx->wBitsPerSample) {
+		case 8:  // 8-bits are unsigned, others are signed
+			func(WaveFormat<uint8_t>());
+			break;
+		case 16:
+			func(WaveFormat<int16_t>());
+			break;
+		case 24:
+			func(WaveFormat<int32_t, 3>());
+			break;
+		case 32:
+			func(WaveFormat<int32_t>());
+			break;
+		default:
+			return false;
+		}
+		break;
+	case WAVE_FORMAT_IEEE_FLOAT:
+		switch (wfx->wBitsPerSample) {
+		case 32:
+			func(WaveFormat<float>());
+			break;
+		case 64:
+			func(WaveFormat<double>());
+			break;
+		default:
+			return false;
+		}
+		break;
+	default:
+		return false;
+	}
+	return true;
+}
+
+/**
+ * Return the leading silence wave data length, in bytes.
+ * Uses a `WAVEFORMATEX` to determine the wave format.
+ */
+inline size_t getLeadingSilenceSize(
+	const WAVEFORMATEX* wfx,
+	const unsigned char* waveData,
+	size_t size
+) {
+	size_t len;
+	if (!callByWaveFormat(wfx, [=, &len](auto fmtTag) {
+			using Fmt = decltype(fmtTag);
+			len = getLeadingSilenceSizeMono<Fmt>(
+				waveData, size, Fmt::defaultThreshold());
+		}))
+		return 0;
+
+	return len - len % wfx->nBlockAlign;  // round down to block (channel) boundaries
+}
+
+}  // namespace SilenceDetect
+
+#endif  // SILENCEDETECT_H

nvdaHelper/local/wasapi.cpp

@@ -24,6 +24,7 @@ This license can be found at:
 #include <mmdeviceapi.h>
 #include <common/log.h>
 #include <random>
+#include "silenceDetect.h"
 
 /**
  * Support for audio playback using WASAPI.
@@ -194,6 +195,8 @@ class WasapiPlayer {
 	HRESULT resume();
 	HRESULT setChannelVolume(unsigned int channel, float level);
 
+	void startTrimmingLeadingSilence(bool start);
+
 	private:
 	void maybeFireCallback();
 
@@ -245,6 +248,7 @@ class WasapiPlayer {
 	unsigned int defaultDeviceChangeCount;
 	unsigned int deviceStateChangeCount;
 	bool isUsingPreferredDevice = false;
+	bool isTrimmingLeadingSilence = false;
 };
 
 WasapiPlayer::WasapiPlayer(wchar_t* endpointId, WAVEFORMATEX format,
@@ -342,6 +346,19 @@ HRESULT WasapiPlayer::feed(unsigned char* data, unsigned int size,
 		return true;
 	};
 
+	if (isTrimmingLeadingSilence) {
+		size_t silenceSize = SilenceDetect::getLeadingSilenceSize(&format, data, size);
+		if (silenceSize >= size) {
+			// The whole chunk is silence. Continue checking for silence in the next chunk.
+			remainingFrames = 0;
+		} else {
+			// Silence ends in this chunk. Skip the silence and continue.
+			data += silenceSize;
+			remainingFrames = (size - silenceSize) / format.nBlockAlign;
+			isTrimmingLeadingSilence = false;  // Stop checking for silence
+		}
+	}
+
 	while (remainingFrames > 0) {
 		UINT32 paddingFrames;
 
@@ -643,6 +660,10 @@ HRESULT WasapiPlayer::setChannelVolume(unsigned int channel, float level) {
 	return volume->SetChannelVolume(channel, level);
 }
 
+void WasapiPlayer::startTrimmingLeadingSilence(bool start) {
+	isTrimmingLeadingSilence = start;
+}
+
 HRESULT WasapiPlayer::disableCommunicationDucking(IMMDevice* device) {
 	// Disable the default ducking experience used when a communication audio
 	// session is active, as we never want NVDA's audio to be ducked.
@@ -839,6 +860,10 @@ HRESULT wasPlay_setChannelVolume(
 	return player->setChannelVolume(channel, level);
 }
 
+void wasPlay_startTrimmingLeadingSilence(WasapiPlayer* player, bool start) {
+	player->startTrimmingLeadingSilence(start);
+}
+
 /**
  * This must be called once per session at startup before wasPlay_create is
  * called.

projectDocs/dev/developerGuide/developerGuide.md

@@ -1393,6 +1393,7 @@ For examples of how to define and use new extension points, please see the code
 |`Action` |`synthIndexReached` |Notifies when a synthesizer reaches an index during speech.|
 |`Action` |`synthDoneSpeaking` |Notifies when a synthesizer finishes speaking.|
 |`Action` |`synthChanged` |Notifies of synthesizer changes.|
+|`Action` |`pre_synthSpeak` |Notifies when the current synthesizer is about to speak something.|
 
 ### tones {#tonesExtPts}
 

source/config/configSpec.py

@@ -45,6 +45,7 @@
 	autoDialectSwitching = boolean(default=false)
 	delayedCharacterDescriptions = boolean(default=false)
 	excludedSpeechModes = int_list(default=list())
+	trimLeadingSilence = boolean(default=true)
 
 	[[__many__]]
 		capPitchChange = integer(default=30,min=-100,max=100)

source/gui/settingsDialogs.py

@@ -3737,6 +3737,7 @@ def __init__(self, parent):
 		#  Advanced settings panel
 		label = _("Speech")
 		speechSizer = wx.StaticBoxSizer(wx.VERTICAL, self, label=label)
+		speechBox = speechSizer.GetStaticBox()
 		speechGroup = guiHelper.BoxSizerHelper(speechSizer, sizer=speechSizer)
 		sHelper.addItem(speechGroup)
 
@@ -3767,6 +3768,14 @@ def __init__(self, parent):
 			["featureFlag", "cancelExpiredFocusSpeech"],
 		)
 
+		# Translators: This is the label for a checkbox control in the
+		#  Advanced settings panel.
+		label = _("Trim leading silence in speech audio")
+		self.trimLeadingSilenceCheckBox = speechGroup.addItem(wx.CheckBox(speechBox, label=label))
+		self.bindHelpEvent("TrimLeadingSilenceSpeech", self.trimLeadingSilenceCheckBox)
+		self.trimLeadingSilenceCheckBox.SetValue(config.conf["speech"]["trimLeadingSilence"])
+		self.trimLeadingSilenceCheckBox.defaultValue = self._getDefaultValue(["speech", "trimLeadingSilence"])
+
 		# Translators: This is the label for a group of advanced options in the
 		#  Advanced settings panel
 		label = _("Virtual Buffers")
@@ -3951,6 +3960,7 @@ def haveConfigDefaultsBeenRestored(self):
 			and self.wtStrategyCombo.isValueConfigSpecDefault()
 			and self.cancelExpiredFocusSpeechCombo.GetSelection()
 			== self.cancelExpiredFocusSpeechCombo.defaultValue
+			and self.trimLeadingSilenceCheckBox.IsChecked() == self.trimLeadingSilenceCheckBox.defaultValue
 			and self.loadChromeVBufWhenBusyCombo.isValueConfigSpecDefault()
 			and self.caretMoveTimeoutSpinControl.GetValue() == self.caretMoveTimeoutSpinControl.defaultValue
 			and self.reportTransparentColorCheckBox.GetValue()
@@ -3979,6 +3989,7 @@ def restoreToDefaults(self):
 		self.diffAlgoCombo.SetSelection(self.diffAlgoCombo.defaultValue)
 		self.wtStrategyCombo.resetToConfigSpecDefault()
 		self.cancelExpiredFocusSpeechCombo.SetSelection(self.cancelExpiredFocusSpeechCombo.defaultValue)
+		self.trimLeadingSilenceCheckBox.SetValue(self.trimLeadingSilenceCheckBox.defaultValue)
 		self.loadChromeVBufWhenBusyCombo.resetToConfigSpecDefault()
 		self.caretMoveTimeoutSpinControl.SetValue(self.caretMoveTimeoutSpinControl.defaultValue)
 		self.reportTransparentColorCheckBox.SetValue(self.reportTransparentColorCheckBox.defaultValue)
@@ -3989,6 +4000,14 @@ def restoreToDefaults(self):
 
 	def onSave(self):
 		log.debug("Saving advanced config")
+
+		if config.conf["speech"]["trimLeadingSilence"] != self.trimLeadingSilenceCheckBox.IsChecked():
+			# Reload the synthesizer if "trimLeadingSilence" changes
+			config.conf["speech"]["trimLeadingSilence"] = self.trimLeadingSilenceCheckBox.IsChecked()
+			currentSynth = getSynth()
+			if not setSynth(currentSynth.name):
+				_synthWarningDialog(currentSynth.name)
+
 		config.conf["development"]["enableScratchpadDir"] = self.scratchpadCheckBox.IsChecked()
 		selectiveUIAEventRegistrationChoice = self.selectiveUIAEventRegistrationCombo.GetSelection()
 		config.conf["UIA"]["eventRegistration"] = self.selectiveUIAEventRegistrationVals[