VO2740: conversion utf8 to Ansii: MemFree error

[kitz]

Hello,
I try to convert strings which may contain UTF-8 characters to Windows 1252 (German) encoding with this function:

Code: Select all

FUNCTION ConvertFromCodePageToCodePage(cString AS STRING, dwFrom AS DWORD, dwTo AS DWORD) AS STRING
	LOCAL pUStr                  AS PTR
	LOCAL pBuffer, pBuf1, pBuf2        AS PTR
	LOCAL nLen, nLenRes, nULen, nULenRes     AS LONG
	// Convert VO string from dynamic memory to fixed memory
	nLen     := LONG(SLen(cString))
	pBuffer  := MemAlloc(DWORD(nLen))
	pBuf1 := pBuffer
	IF pBuffer == NULL_PTR
		BREAK "MemAlloc error"
	ENDIF
	MemCopyString(pBuffer,cString,DWORD(nLen))
	
	// Determine length of Unicode string
	// And allocate enough space to hold it
	nULen    := MultiByteToWideChar(dwFrom,0,pBuffer,nLen,NULL_PTR, 0)
	IF nULen == 0
		BREAK "MultiByteToWideChar GetLen error"
	ENDIF
	IF pBuffer <> pBuf1
		BREAK "pBuffer modified"
	ENDIF
	pUStr    := SysAllocStringLen(NULL_PTR,DWORD(nULen))
	IF pUStr == NULL_PTR
		BREAK "SysAllocStringLen error"
	ENDIF
	// Convert Fixed memory Ansi string to Fixed memory Unicode string
	nULenRes := MultiByteToWideChar(dwFrom,0,pBuffer,nLen,pUStr,nULen)
	IF nULen <> nULenRes
		BREAK "MultiByteToWideChar Copy Len<>"
	ENDIF
	// Now determine size needed for ANSI string
	nLen :=      WideCharToMultiByte(dwTo,0,pUStr,nULen,NULL_PTR,0, NULL,NULL)
	
	// Allocate Fixed memory buffer to hold the UTF8 string
	pBuffer  := MemRealloc(pBuffer, DWORD(nLen+1))
	IF pBuffer == NULL_PTR
		BREAK "MemRealloc error"
	ENDIF
	pBuf2 := pBuffer
	// Convert Unicode to Ansi
	nLenRes  := WideCharToMultiByte(dwTo,0,pUStr,nULen,pBuffer,nLen ,NULL,NULL)
	IF nLen <> nLenRes
		BREAK "WideCharToMultiByte Len<>"
	ENDIF
	
	// Convert fixed memory buffer to dynamic memory string
	cString  := Mem2String(pBuffer,DWORD(nLen))
	IF pBuffer <> pBuf2
		BREAK "pBuffer 2 modified"
	ENDIF
	// Release fixed memory buffer
	IF MemFree(pBuffer) <> 0
		BREAK "MemFree error: " + cString     // <------ here it breaks
	ENDIF
	// Release the Unicode String
	SysFreeString(pUStr)
	
   RETURN cString

The MemFree() function near the end doesn't return 0 so according to the vo27 doc it fails!
Without all the test IF's it failed after some calls with an 5333 error. It works for some calls and fails with others. The string which creates the error is about 400 characters long and contains a handful of UTF-8 characters.
I absolutely don't understand what's going on. pBuffer is saved and compared before freeing, so it's the same.
Debugging works regarding breakpoints, but I am not able to look at any variable, neither by hovering nor by ctrl-x. Mostly it shows empty, sometimes a result from a previous assignment to some other variable in the code, so unusable.
VO2740, Win10 Enterprise 20H2, Windows Information Protection enabled (Company policy)
Any ideas?
BR Kurt

[robert]

Kurt,
I am not sure why you are checking pBuffer against pBuf1 and pBuf2. There is no way that the call to WideCharToMultiByte() can change the pointer which is a local variable.
If MemFree() fails then this is usually an indication that some process has written outside of the boundaries of the allocated memory.
VO allocates a small extra buffer before and after the memory block that you allocate with MemAlloc().
MemFree() inspects these extra buffers to see if the are (accidentally) overwritten.
I suspect that one of the parameters to WideCharToMultiByte() is incorrect and that this function therefore overwrites the extra block of memory at the end of the pbuffer block.

Robert

Edit:
PS Maybe we can create a (new) separate method in the SqlSelect class in which the oStmt is allocated. That would allow you to override that method in your subclass and return an instance of a subclass of SqlStatement.
If you like this idea, then please create an issue in our GitHub repo, so we will not forget it.

[kitz]

Robert,
thanks for the explanation. I compared everything to be sure it was not overwritten (I got 5333 errors sometimes) as I can't really debug and watch.
I now avoided this situation by creating a large global byte buffer with MemAlloc() in the start() routine as well as a large global character buffer with SysAllocStringLen() the same way. Then I check the sizes against the buffer sizes and reuse the global buffers because this conversion is called 12 times per record with 100's of records. This seems to work fine.
If I have time I will check the buffer contents to see what happens.
I assume, the EDIT part belongs to my other question regarding SQL Classes
BR Kurt