RAM size, SFS and out of memory exception ?

[Suteetat]

I have been away for awhile after I built my new computer. One thing I just noticed.
With my old computer with 4GB RAM with 1GB Ramdisk, I could play 16/44 file with 4x oversampling
and SFS of 200 but needed to reduced SFS to <100 for 24/192 file to avoid
out of memory exception.
My new computer now has 12GB RAM runnng 4GB Ramdisk. Unfortunately, I still have exactly the same problem.
I still get out of memory exception if I try to increase SFS to > 100 with 24/192 files. This is with
attended mode and only one 5 minutes track loaded in the playlist. 
I still prefer the sound of 16/44 file with larger SFS so I wonder if there is anyway to increase SFS
when playing high resolution file as well? 

[PeterSt]

There has been one other post about the same, and I don't recall it was worked out.

I have 3GB in my system, and 1GB of it is RAMDisk. Whether it be a native 24/192 (see below) or upsampled to that, it doesn't matter (uses the same amount of memory). My SFS is 240 here ... (mind you 0.9z-3 ... with 0.9z-2 and earlier that would be impossible).
Even the 192 upsampled to 384 still plays with 240.

It must be something about contiguous memory not being available.

I could play 16/44 file with 4x oversampling and SFS of 200 but needed to reduced SFS to <100 for 24/192 file to avoid out of memory exception.

This is interesting though, and something we may continue on. In by book this really wouldn't matter, apart from the small difference of 4 x 44.1 being 176.4 and not 192 (so the former uses somewhat less memory, but never twice as less of course).

Somehow I suspect the RAMDisk to interfere here, already because before (the RAMDIsk adventures) these inconsistensies were never reported. IOW, you may try what happens with the RAMDisk shut off ? (notice it may be a waste of your time, but still it's good to eliminate it as a reason).

And do you have switched On your Virtual Memory ? you should !

Peter



PS: Please put your system in your signature, which would avoid a couple of questions at times.

[Suteetat]

Peter, I tried installing XXHighend on my SSD C: drive. Unfortunately the same problem with high rez file still persist with larger SFS
I have virtual memory allocated with maximum of 20GB on my SSD C: drive.

[PeterSt]

I am sorry, but I meant to uninstall the RAMDisk, hence not have it there at all.

I don't think I made that clear enough ...

[Suteetat]

Hi Peter,
I removed RAMDisk and have all 12GB RAM available but still the same problem unfortunately. I am wonderng if anyone else has the same problem or not?

[PeterSt]

Here is the only one that *I* know of : http://www.phasure.com/index.php?topic=1440.msg14027#msg14027 (this post and down).

Whether there was something solved or not I can't tell.

Notice that the track length doesn't matter because it is the SDS determining how much of it goes into memory.

And if you didn't discover it already ... this is not a thing I can do something about. It must me something in the PC (OS), like I said ... not contiguous memory (parts of around 2 times the SFS size in MB).

Actually I think there *are* more people with this problem, which will be all who had their SFS at some level before 0.9z-3, then upgraded to 0.9z-3 and next had to lower their old setting in order to be able to play. This, while 0.9z-3 allows for a twice  or so *higher* setting compared to before versions.


Peter

[Suteetat]

I did a little experiment and it was a bit unexpected. I was using Dataram Ramdisk initially. So after I uninstalled dataram, I downloaded Vsuit Ramdisk professional traial version. I made Ramdisk with 60000 mb (5.8 GB). Now I can play all the high rez file with SFS upto at least 180 but not yet 200.
I am not sure why that should be the case but whatever works, I suppose!
 

[PeterSt]

Hmm ... that doesn't seem to make sense to me ... (except for that I think the RAMDisk may interfere, as told. But you tried completely without ...)

To hopefully help you on the track further (remember, this is really out of my control, nor could I debug things etc. *unless* I had it myself) :

(but also take care that you didn't denote the page file (swa file) for virtual memory on the RAMDisk ! because that would give the same Out of Memory when full)


The memory we are talking about (and which needs to be there in 3 contigious parts for twice the SFS size and another contigious part for the SFS size itself) is just normal memory. This is determined when XXEngine3.exe starts hence the first notes start to play. It only happens once per playback session (click Play, no matter Attended or Unattended (possibly Alt-P afterwards with Unattended)), and once it goes, it goes and it is impossible to later (!) run out of memory *if*

a. Your Virtual Memory is On (paging file);
b. The second digit in SFS.ini is 1.

This is independent of track size or anything, but of course depends on the SFS size itself.

One additional thing : XXEngine3.exe can not use more than 2GB of memory ever.


Now, during writing this, a most stupid long shot sprung to my mind, and never mind I'm programming for over 35 years now I really don't know what actually happens with an application (like XX) written for a 32 bits OS, running in "both 32 bit and 64 bit" mode, and whether it may be possible that twice the memory is consumed because of this. Yea, I may be thinking like a baby now at suggesting that a 64 bit OS may use twice the memory for actual 32 bit memory parts. I guess this won't be the case, but still.
It also may be related to the trick I apply to "reserve" that memory in one go, which is not how it's suppose to be done in dot-net because it will take care of it herself (with the result of much less available, as told before). So, I kind of force a. the memory to be really in use by me when I want it and b. it to be contigious. Both -at normal use- won't be the case meaning that dot-net may be nicely allocating the small parts piece by piece, while now I'm "doing" it, it may do it in twice as large parts (but leaving half unused).

If this is not the case, it must be possible by means of some tool to "see" the blocks of contiguous free memory - before XXEngine3.exe starts (which may tell nothing much) and after it has started (and then proove the free contiguous blocks indeed are not there after the Out of memory shows up). If they *are* there, try to proove that if multiplied by 2 there are not there (my 64 bit story).

Peter

[Josef]

I really don't know what actually happens with an application (like XX) written for a 32 bits OS, running in "both 32 bit and 64 bit" mode, and whether it may be possible that twice the memory is consumed because of this.

No, it does not work like that.
For native application (e.g. written in C++) you would get twice the memory allocated only if you are e.g. allocating an array of integers instead of bytes AND you recompile with 64-bit compiler.
In .Net however, integers (and other data types) stay the same size. Only pointers are expanded to 64 bits (for obvious reasons). You can also force 32-bit pointers to be used always (look in compilation flags and utilities e.g. corflags.exe)

The memory we are talking about (and which needs to be there in 3 contigious parts for twice the SFS size and another contigious part for the SFS size itself)

I noticed that z3 uses twice as much memory as z2: z2 used 3xSFS size and z3 uses 3x2xSFS - I wanted to ask you if this was by design and by looking at above it seems it is.

But I guess you wanted to say that last part should be 1/2 SFS size and not 1 SFS? (because that's what I see: e.g. for 100MB SFS there are 3 blocks of 200MB and 1 of 50MB)

I am not sure what you mean by 'contigious' though: Looking at memory it can be seen that SFS blocks are NOT contiguous - maybe that's another idea to implement?

[PeterSt]

Josef, your help is much appreciated. Thank you very much (and I was actually hoping from some response from you).

About the SFS sizes ... you are completely correct, and while I wrote some things by heart I completely forgot how I changed things. And worse, forgot to tell about it. And more worse : the implications of it regarding "old" possibilities of SFS and mem useage vs. the "new". And as a matter of fact, I'm a bit confused now. I will work it out later (with the Release Notes of 0.9z-4), but the main point is :
Where I can use twice the SFS size from before (0.9z-2) and *think* I can use 240 while before it was 120, this actually has to be remapped to your own findings about the sizes. But it is hard to compare, because initial array sizes were limited to 120MB when implied larger by the SFS setting, while now this limit is not there anymore. So, the larger size would be used allright (old), but only when a track (size) demanded it. With new not anymore. There the whole "SFS" memory is allocated at once, regardless.
Now, my 240 will actually be 480 for those three "x 2" arrays, and so I don't get much how I suddenly can use 4 times more than before. Must be over-smart somehow.
But as said, I will work it out.

And yes, it was my idea from the start (0.9z-3) to have that memory contiguous. So, the damn thing is not eh ? outsmarted by dot-net again.
Which the more makes me not understand what is happening to those who can't get the high SFS number ... But I must say Sorry for the confusement, and not checking it. But also :

Those who could get, say 200 with "old", must realize that "new" 200 will be 400 or so (which again, I forgot to tell, or better : forgot about myself, including the "unworkable" relation between old and new, because both can be set in 0.9z-3 and things will become very vague. Anyway, if we again do the math : 200 * 2 plus 200 * 2 plus 200 * 2 plus 200 / 2 = 1300MB which is less than the 2GB limit for 32 bit applications (for a thread). But note that an SFS of 300 already adds up to 1950. So, more than 300 doesn't seem possible at all for "new" (which is SFS.ini second parameter = 1 or 2.).

I now wonder ... maybe the non-contiguous memory emerges from some Second Generation Garbage prepairing (although I'd think that only happens with obsolete memory).

Peter

PS:

Only pointers are expanded to 64 bits (for obvious reasons).

Now let's hope that if I add to a pointer, it doesn't create a new one internally. Haha. But I guess it's really explained by the "* 2" for "new". One more note on this : this now explicitly anticipates on going from 16 to 32 bits which is not true for a. those having a 16 bit DAC only and b. when you play native 24 bit material. So this creates another vagueness but indirectly it always has been there with the difference of it being allocated now at once, and not on the fly. In the end the memory is not wasted though, if you only play large enough tracks to consume it (which is harder for 16/44.1 of course).

[Josef]

And yes, it was my idea from the start (0.9z-3) to have that memory contiguous. So, the damn thing is not eh ? outsmarted by dot-net again.

Don't feel bad Actually, I suspect .Net is _not_ your problem here - rather, it is Windows itself. I'm afraid the only way to get all SFS memory contiguous would be by writing a kernel device driver: drivers can allocate contiguously, user applications cannot....

Think the most you can achieve in user space is have 2MB contiguous blocks by using large pages feature (I'm pretty sure that's not what you're doing at the moment). It's still a lot better then default 4KB pages and also has added benefit of having everything locked in RAM (it can't get swapped out). Additionally, CPU caching would be improved even further so all in all it should have a rather positive effect on SQ.

However, if you decide to pursue this then I'm afraid you will have to first convert XXEngine3.exe process into an auto-start service (which should be relatively easy). The way it works now (memory being released after playback stops and then allocated again when it starts) would probably result in Out Of Memory exceptions very, very quickly....

this now explicitly anticipates on going from 16 to 32 bits which is not true for a. those having a 16 bit DAC only

If you decide to pursue above strategy then memory will become scarce quickly and maybe you'll need to 'optimize' by not expanding to 32 bits if DAC is selected as 16/44.....(would reduce SFS RAM needs by 50% and thus make it much easier for Windows to find contiguous blocks)


 

[PeterSt]

About the latter : that is what you logically will think (memory getting scarce etc.). But the only thing what happens is that a same SFS size will allow for more minutes of 16/44.1 compared to 24/192. So, memory isn't wasted assuming you want to play 10 minute 16/44.1 at some stage, and might that not happen ... the memory wasn't needed for anything else. Of course this depends on the SFS setting itself, and if you want to start some other stuff in between, just lower it the amount needed (empirically found).

It would be true though that making it dependend on what the DAC can do seems more understandable. But in the end it is not again, because those with a 16 bit DAC will never know the difference. Also, just in case there is a mix is in the playlist and the DAC is 24 bits, everything is remapped to 24 bits anyway (but resulting in 32 bits for most DACs), because of the Volume (possibility). There was a time that it was dependend on the Volume being -0 or not, but I removed that because of unnecessary complexity and code.

Btw, your mentioned 4KB will be in order which IIRC I found at testing the various cluster sizes of RAMDisks. There too I noticed that it's really all 4KB instead of the denoted cluster size (which in the end thus does nothing).
I will look into the 2MB feature which indeed I don't use. Not for contiguousy (?) but for speed perhaps.

Thanks as always - and the device driver is still in my mind ...
Peter

[Josef]

So, memory isn't wasted assuming you want to play 10 minute 16/44.1 at some stage, and might that not happen ...

This really is not about wasting memory - rather, it's about memory being as continuous as possible!

For example: if you implement Windows Large Page feature, for e.g. 100MB SFS (=setting of 50 in z3, 100 in z2) you will have 50 pages.

Suppose average size of WAVs is 5 minutes i.e. ~50MB. Now that you convert it to 32 bits you need 100MB i.e. all 50 pages. But if you keep SFS at 16 bits (if DAC==16/44) then you only need half RAM i.e. 25 pages: In other words, on average, you increase the chances of having the complete WAV fully continuous in RAM by 100%.... (likewise, for 24 bit DACs you'll increase the chances by 50%.) Why? Because while each 2MB block is guaranteed to be continuous, individual blocks can (and, chances are, will be) scattered around! So, less blocks = better chances of having more of WAV continuous in RAM

After all, it seems it's only when using your DAC that 32 bits make sense as AFAIK you use extra bits for digital volume control so you really do need them....(but even that is debatable if people opt to use pre-amps instead )

Btw, your mentioned 4KB will be in order which IIRC I found at testing the various cluster sizes of RAMDisks. There too I noticed that it's really all 4KB instead of the denoted cluster size (which in the end thus does nothing).

Yes - Agree it does not make a difference but there's more to it as you will get the same thing with HD or SDD (nothing special about RAMDisk). Ultimate rule master here is Windows' virtual memory manager which _always_ works with 4KB pages.

Cluster sizes do have a role though: they are meant to enable fine-tuning of disk access times for particular loads. For example, both music & video files are rather large. Having larger cluster sizes helps improve disk read speeds on HDDs by reducing fragmentation. In theory this is also beneficial for RAMDisks as for, e.g. 64KB cluster you need less RAM accesses then for 4KB (1 in best-case vs up to 16 RAM accesses in worst-case!)
In practice, however, if you defragment your HDD you get all the benefits even with default 4KB clusters and RAM access is comparatively so fast that it does not matter what cluster size you use....
It might, however, matter if we were playing directly from RAMDisk but we don't do that so it's a moot point

[PeterSt]

Thanks again Josef.

About the Cluster sizes ... yes I know of course. But my point was that with a RAMDisk this really is faked, because there's no way I could proove any ms difference for throughput times at writing nor reading (small pieces). You may try it yourself. Or at least not with the RAMDisk most are using. Not even with the FAT16 (IIRC) which allows 1MB cluster sizes. I don't know how I did it anymore, but I recall that it all came down to 4KB clusters (ah, I think I did that by the wasted space of small files -> should be rounded to the cluster size, but they are not).

To be continued ...