Random out of memory exceptions
Posted: Sat Aug 08, 2020 9:36 am
We have/had random out of memory exceptions when handling with bigger RTF-documents of about 30 MB in our program together with the TEControl for formatted text
One of the places for the exceptions was the transferring of the textcontrol contents to a string. Here the X#-function Mem2String is used which internally calls the function Memalloc for unmanaged memory. The reason may be a fragmentation of the memory or temporary limits of the 2 GB for 32-bit programs.
Based on the idea from Chris I have made the function mem2string_static as attached, that allocates only memory once and does not always allocate and de-allocate memory.
There is another function in our programs with random exceptions: a complex function RTFToText with many strtrans and string decompositions/compositions. Here I had also random exceptions in the Dotnet String-replace-functions called by strtran, although it is managed memory.
I think that the reasons are similar to the above errors. I believe that Dotnet does not directly call the GC if needed, because it runs as a separate thread, whereas in VO the GC may be called in every memory allocating function. The VO GC makes the program slow, but may save the program before reaching the limits. Here I have added an explicit calling of the garbage collector in case of big documents:
if slen(cRes) > 10_000_000
System.GC.Collect()
System.GC.WaitForPendingFinalizers()
end if
I tried also using the 3GB Large address aware changing of the exe file Wolfgang proposed for Win32 programs. But I had errors in the memory handling when exceeding 2 GB when using with Dotnet in contrast to Win32 where it works very well. On the other hand the X#-programs do not use so much space for the dlls/program code, so that the Dotnet programs in general have more reserves compared to Win32 programs.
One of the places for the exceptions was the transferring of the textcontrol contents to a string. Here the X#-function Mem2String is used which internally calls the function Memalloc for unmanaged memory. The reason may be a fragmentation of the memory or temporary limits of the 2 GB for 32-bit programs.
Based on the idea from Chris I have made the function mem2string_static as attached, that allocates only memory once and does not always allocate and de-allocate memory.
There is another function in our programs with random exceptions: a complex function RTFToText with many strtrans and string decompositions/compositions. Here I had also random exceptions in the Dotnet String-replace-functions called by strtran, although it is managed memory.
I think that the reasons are similar to the above errors. I believe that Dotnet does not directly call the GC if needed, because it runs as a separate thread, whereas in VO the GC may be called in every memory allocating function. The VO GC makes the program slow, but may save the program before reaching the limits. Here I have added an explicit calling of the garbage collector in case of big documents:
if slen(cRes) > 10_000_000
System.GC.Collect()
System.GC.WaitForPendingFinalizers()
end if
I tried also using the 3GB Large address aware changing of the exe file Wolfgang proposed for Win32 programs. But I had errors in the memory handling when exceeding 2 GB when using with Dotnet in contrast to Win32 where it works very well. On the other hand the X#-programs do not use so much space for the dlls/program code, so that the Dotnet programs in general have more reserves compared to Win32 programs.