Corruption detected on an index ... but NOT if index created within XSharp

[RolWil]

With the link script ( maybe not the right we orde) I meant the libraries and files that you referring to in your harbour program.

Robert

Hi Robert,

Thanks, I have no 3rd party libraries/files. My "includes" and "References" are as follows:

[robert]

I was asking for your HARBOUR code, not the X# code.

Robert

[RolWil]

Well OK then. Detecting what the cause of this issues is took some doing. Eliminating code line by line didn’t directly highlight the issue and if anything, I ended up following red herrings. HOWEVER, it did appear to be “caused’ by a particular file/table name “BFMB.DBF” in my application. Since the data (.DBF) is the same between the Harbour and XSharp, the problem had to be with the indexes. Eliminating indexes one-by-one (this DBF has 8 indexes) led me to one that contains the soundex() function.

However, a quick test showed that soundex() resulted in the exact same returned value from Harbour as XSharp – NOT what I had expected. At this point, I was tearing what hair I have left, out. Finally, I thought I would look at the content of the NTX files and sure enough, that’s where the problem lies as follows.

Here’s a side-by-side comparison of NTX files for the same DBF and that worked in both Harbour and XSharp.

The index quite simple and only involves fields of the table: “BFMB->FC_MBCLIE+BFMB->FC_MBDIVI … “
Running a binary compare (using ‘fc’ CMD) on the two NTX iles raises two areas of difference: Bytes 002-003 and 021A-021D. The two bytes 002-003 are the compiler/index version; 021A-021D is officially unused but XSharp inserts the name of the physical NTX file: “BFMB”.

Not surprisingly then, using either of these index files causes no problem in XSharp since, for all intents and purposes, they are identical.

Not so when the soundex() function forms part of the index key! Performing a binary compare of two .NTX files with identical key (which includes use of soundex()) yields issues from bytes 0400 on right to the end of the file. Looking at the two files side-by-side, we note a shift in the contents between the two .NTX files

[RolWil]

Hi All,

For now, I've simply closed the index at fault to be able to continue - it's not needed for that particular function.

Does anyone have any idea of what's gone haywire with indexes that use the "soundex()" function as I outlined in my Nov 30 post?

Thanks.
Roland

[Chris]

Hi Roland,

Can't think of something that might be causing it. Can you please provide a sample showing the problem (code + dbf/index files) so we can have a close look?

[robert]

Guys,
Most likely there is a (small) difference between the Soundex() function in Harbour and VO/X#.

Robert

[RolWil]

Hi Gentlemen,

I had run a test comparing the return values of Soundex() from both XSharp and Harbour – they return identical values.

Re-Creating the problem

I can’t give the data file to a third party – even anonymized. So instead, I created a DBF (IDXISSUE.DBF) with a few records and columns FIRSTNAME and LASTNAME and then created an index on Harbour and XSharp using soundex().

The code in Harbour is:

Code: Select all

use IDXISSUE
index on SOUNDEX(IDXISSUE->LASTNAME) to IDXISSUE_HC
return

The code in XSharp:

Code: Select all

SetAnsi(.t.)
SetCollation("Clipper")           
SetNatDLL("ENGLISH")
RDDSetDefault("DBFNTX")              
USE IDXISSUE
INDEX ON SOUNDEX(IDXISSUE->LASTNAME) TO IDXISSUE_XS

I then opened the two NTX up as binary file and you can see where the differences between the two:

[robert]

Roland,
Can we get the IDXISSUE.DBF so we can test it ourselves?

Robert

[RolWil]

Hi Robert,

I attached a ZIP with the DBF and the NTX's.

Cheers
Roland

[robert]

Roland,
When I create the index with VO then it is 100% identical to the index created with X#.
VO and X# both store the tag name in the NTX file. Harbour does not, but that is not the most important difference.
The difference seems to be that the algorithm inside Harbour that determines the maximum number of keys that fit on a page.
For X# and VO that number is 70 and harbour calculates 72.

Because these numbers are different then the distribution inside the index differs as well.

The formula in the Harbour source code is

pTag->MaxKeys = ( NTXBLOCKSIZE - 2 ) / ( uiKeyLen + 10 ) - 1;

NTXBLOCKSIZE = 1024
The Key length = 4
So this formula results in 72

VO and X# use this formula
see https://github.com/X-Sharp/XSharpPublic ... Create.prg

num := ( ( BUFF_SIZE - 4) / (SELF:_keySize + 10))
_halfPage := ((num - 1) / 2)
_MaxEntry := (SELF:_halfPage * 2)

num = 72
halfPage = 35
MaxEntry = 70

I have no idea why Harbour uses a different formula.
I am pretty sure that the formula in VO and X# was derived from the original formula in Clipper.


Robert