Hi Johan,
To me this looks like a clear case of your Realtec ALC883 HD Audio (sound chip) driver being "out of order".
I know this has been your own assumption, and I know you already spent much time on it. But let me to explain what happens, so we maybe can solve this problem :
Note : "hanging of the sound" in this case means "the last part of the buffer sent to tha DAC keeps on repeating". tadadadadadadada
When you say "when I activate a button", in the case that the sound starts hanging you will have used a button that just stops the Engine. This can be seen by the XXEngine3.exe which would disappear from the processes in Task Manager. For instance, using the Playlist Load button, or the Clear button will not cause this to happen (this just does not stop the sound).
When Engine#3 stops feeding the soundcard (which comes first in your SPDIF case opposed to the DAC), it disconnects from the soundcard (sound chip), and it must be released for the matter of "giving it back to the system" and freeing the Exclusive Mode which is at order here (Engine#3). Now, here it goes wrong, and the soundcard does not stop.
The sole FACT that XXEngine3.exe disappears from the process list (hence it really has gone), just *proves* that it is not any software re-feeding this (last) buffer over and over again. What would happen though, is that the soundcard keeps on cycling through its own buffer, over and over releasing the sound data. Together with the process it's still in use exclusively, and there is no way you can stop it until a next reboot, UNLESS (I think) you tell it that other sound processes can interrupt it (second checkbox (uncheck) in the Advanced tab of the Device's properties). BUT, you really shouldn't do this, because then an "you've got mail" beep, will interrupt (abort) music playing. So I'm just explaining things here.
Possibly there are two solutions :
1. Try to find an external soundcard which really is better in about all cases.
2. I could try to interrupt / end / kill / abort / whatever the soundcard when Engine# is ordered to quit.
The last option obviously is the most frienddly for you, but possibly not the best assuming that you want the best soundquality which from theory can't come from the internal sound chip. Even when it's only passing through SPDIF !!!
If - and only if we choose for this second option, it can be a longthreaded process, because it would be me trying somthing in the program, you trying whether it helped or not. For this matter, note that all what should be done to kill the soundcard's process is already in there, but I wouldn't be me to find another way. Those ways just would be, say, illegal and without real logic, and that is exactly why it might take long to get there. And, even without guarantee obviously.
At writing this, I just thought of another option (in the "illegal" area
) : you could try what happens if you remove the power from the DAC. You can try this when it's looping, and you can try it during normal playback. Possibly none of them helps, and probably when done during playback you receive the same thing : it starts to loop (but how to hear that ? switch on the DAC again, hehe).
The results from the last suggestion above migh bring me what can be done about it (for insiders : like throwing the same exception you receive at shutting of the power of the DAC, *if* that helps of course).
When you are going to try it, do not forget to post the error message you'll receive, with which I can only hope it is not "XXHighEnd does not work anymore". But if so, I can do something about *that*.
Lastly, when you have laying around another soundcard (and the cheapest might be better than the soundchip you're using now), maybe try that first as a solution. Take care -when you are going to buy a 10 euro sound card- that it comes with Vista drivers.
I really hope you can make something of this.
Peter