Corruption detected on an index ... but NOT if index created within XSharp

This forum is meant for questions and discussions about the X# language and tools
RolWil
Posts: 68
Joined: Mon Jul 18, 2022 3:16 am

Re: Corruption detected on an index ... but NOT if index created within XSharp

Post by RolWil »

robert wrote: Mon Nov 27, 2023 4:07 pm With the link script ( maybe not the right we orde) I meant the libraries and files that you referring to in your harbour program.

Robert
Hi Robert,

Thanks, I have no 3rd party libraries/files. My "includes" and "References" are as follows:
Attachments
XSharp_Settings2.png
User avatar
robert
Posts: 4554
Joined: Fri Aug 21, 2015 10:57 am
Location: Netherlands

Re: Corruption detected on an index ... but NOT if index created within XSharp

Post by robert »

I was asking for your HARBOUR code, not the X# code.

Robert
XSharp Development Team
The Netherlands
robert@xsharp.eu
RolWil
Posts: 68
Joined: Mon Jul 18, 2022 3:16 am

Re: Corruption detected on an index ... but NOT if index created within XSharp

Post by RolWil »

Well OK then. Detecting what the cause of this issues is took some doing. Eliminating code line by line didn’t directly highlight the issue and if anything, I ended up following red herrings. HOWEVER, it did appear to be “caused’ by a particular file/table name “BFMB.DBF” in my application. Since the data (.DBF) is the same between the Harbour and XSharp, the problem had to be with the indexes. Eliminating indexes one-by-one (this DBF has 8 indexes) led me to one that contains the soundex() function.

However, a quick test showed that soundex() resulted in the exact same returned value from Harbour as XSharp – NOT what I had expected. At this point, I was tearing what hair I have left, out. Finally, I thought I would look at the content of the NTX files and sure enough, that’s where the problem lies as follows.

Here’s a side-by-side comparison of NTX files for the same DBF and that worked in both Harbour and XSharp.

The index quite simple and only involves fields of the table: “BFMB->FC_MBCLIE+BFMB->FC_MBDIVI … “
Running a binary compare (using ‘fc’ CMD) on the two NTX iles raises two areas of difference: Bytes 002-003 and 021A-021D. The two bytes 002-003 are the compiler/index version; 021A-021D is officially unused but XSharp inserts the name of the physical NTX file: “BFMB”.

Not surprisingly then, using either of these index files causes no problem in XSharp since, for all intents and purposes, they are identical.
XSharp_NTX1.png
Not so when the soundex() function forms part of the index key! Performing a binary compare of two .NTX files with identical key (which includes use of soundex()) yields issues from bytes 0400 on right to the end of the file. Looking at the two files side-by-side, we note a shift in the contents between the two .NTX files
XSharp_NTX2.png
RolWil
Posts: 68
Joined: Mon Jul 18, 2022 3:16 am

Re: Corruption detected on an index ... but NOT if index created within XSharp

Post by RolWil »

Hi All,

For now, I've simply closed the index at fault to be able to continue - it's not needed for that particular function.

Does anyone have any idea of what's gone haywire with indexes that use the "soundex()" function as I outlined in my Nov 30 post?

Thanks.
Roland
User avatar
Chris
Posts: 4953
Joined: Thu Oct 08, 2015 7:48 am
Location: Greece

Re: Corruption detected on an index ... but NOT if index created within XSharp

Post by Chris »

Hi Roland,

Can't think of something that might be causing it. Can you please provide a sample showing the problem (code + dbf/index files) so we can have a close look?
Chris Pyrgas

XSharp Development Team
chris(at)xsharp.eu
User avatar
robert
Posts: 4554
Joined: Fri Aug 21, 2015 10:57 am
Location: Netherlands

Re: Corruption detected on an index ... but NOT if index created within XSharp

Post by robert »

Guys,
Most likely there is a (small) difference between the Soundex() function in Harbour and VO/X#.

Robert
XSharp Development Team
The Netherlands
robert@xsharp.eu
RolWil
Posts: 68
Joined: Mon Jul 18, 2022 3:16 am

Re: Corruption detected on an index ... but NOT if index created within XSharp

Post by RolWil »

Hi Gentlemen,

I had run a test comparing the return values of Soundex() from both XSharp and Harbour – they return identical values.

Re-Creating the problem

I can’t give the data file to a third party – even anonymized. So instead, I created a DBF (IDXISSUE.DBF) with a few records and columns FIRSTNAME and LASTNAME and then created an index on Harbour and XSharp using soundex().

The code in Harbour is:

Code: Select all

use IDXISSUE
index on SOUNDEX(IDXISSUE->LASTNAME) to IDXISSUE_HC
return
The code in XSharp:

Code: Select all

SetAnsi(.t.)
SetCollation("Clipper")           
SetNatDLL("ENGLISH")
RDDSetDefault("DBFNTX")              
USE IDXISSUE
INDEX ON SOUNDEX(IDXISSUE->LASTNAME) TO IDXISSUE_XS
I then opened the two NTX up as binary file and you can see where the differences between the two:
IndexIssue.png
User avatar
robert
Posts: 4554
Joined: Fri Aug 21, 2015 10:57 am
Location: Netherlands

Re: Corruption detected on an index ... but NOT if index created within XSharp

Post by robert »

Roland,
Can we get the IDXISSUE.DBF so we can test it ourselves?

Robert
XSharp Development Team
The Netherlands
robert@xsharp.eu
RolWil
Posts: 68
Joined: Mon Jul 18, 2022 3:16 am

Re: Corruption detected on an index ... but NOT if index created within XSharp

Post by RolWil »

Hi Robert,

I attached a ZIP with the DBF and the NTX's.

Cheers
Roland
Attachments
IndexIssue.zip
(1.05 KiB) Downloaded 167 times
User avatar
robert
Posts: 4554
Joined: Fri Aug 21, 2015 10:57 am
Location: Netherlands

Re: Corruption detected on an index ... but NOT if index created within XSharp

Post by robert »

Roland,
When I create the index with VO then it is 100% identical to the index created with X#.
VO and X# both store the tag name in the NTX file. Harbour does not, but that is not the most important difference.
The difference seems to be that the algorithm inside Harbour that determines the maximum number of keys that fit on a page.
For X# and VO that number is 70 and harbour calculates 72.

Because these numbers are different then the distribution inside the index differs as well.

The formula in the Harbour source code is

pTag->MaxKeys = ( NTXBLOCKSIZE - 2 ) / ( uiKeyLen + 10 ) - 1;

NTXBLOCKSIZE = 1024
The Key length = 4
So this formula results in 72

VO and X# use this formula
see https://github.com/X-Sharp/XSharpPublic ... Create.prg

num := ( ( BUFF_SIZE - 4) / (SELF:_keySize + 10))
_halfPage := ((num - 1) / 2)
_MaxEntry := (SELF:_halfPage * 2)


num = 72
halfPage = 35
MaxEntry = 70

I have no idea why Harbour uses a different formula.
I am pretty sure that the formula in VO and X# was derived from the original formula in Clipper.


Robert
XSharp Development Team
The Netherlands
robert@xsharp.eu
Post Reply