Is this possible on CDX ?

[DexterZ]

Hello X#Devs,

I'm just wondering if this is possible, using DBFCDX functionality but the operation is on CDX file only. I just wanted to use the CDX as my key value pair file storage with insert,update and deletion operation on the CDX file only, not on DBF.

If not possible, links tuts on how CDX was structured and how it's B+Tree works internally.

Many thanks ^_^y

Dex

[wriedmann]

Hi Dex,
the CDX file contains the order expression and the relative record number in the DBF file.
Therefore you cannot use the standard CDX file as key-value storage - you need also the DBF file.
CDX supports also user defined orders. This is an order where the programmer is responsible to maintain the order expressions - and with this it is possible to have more than one reference in one order to one record in the DBF file.
AFAIK the X# RDD currently does not supports this, but it will be added because it is possible with both Clipper and VO (don't know about FoxPro, unfortunately).
Wolfgang

[DexterZ]

Hello Sir Wolfgang,

Thanks for the reply and sad at the same time because it's not possible T_T does VO or X# RDD source code on CDX manipulation open too ?

I just want to study it or most likely convert it to C#, I know xHarbour is open source with it's CDX functionally but I'm not really sure if it's like X# which I already tested and works side by side with VFP and .NET on CDX file handling and manipulation.

Many thanks ^_^Y
Dex

[wriedmann]

Hi Dexter,

the X# RDD is on Github like the entire X# project sources.
I don't think it will be possible to convert it so easily to C# as you need the macrocompiler or at least a macro evaluation engine together with the used xBase function to have automatically maintained indexes.
If you don't, you have to maintain the indexes yourself like Sequiters CodeBase.

I can only warn you: it is a very complex piece of code.

But if you need it from C#: this is possible using the X# runtime.

For a simple key-value store you should use another database on the market - there are some open source products that are very well tested. You can look at Redis for example.

Wolfgang

[robert]

Dex,
Why bother using the index without DBF. You can also simply create a DBF with one field and index that field. Appending records to the DBF will automatically add them to the index as well.
And if you really want it you could try to use a CDX without RDD. The source is on Github, like Wolfgang said.
But we never designed the CDX driver to be used without DBF. There are references to the RDD object in the CDX code everywhere.

Robert

[DexterZ]

Thanks guyz, everything is taking noted ^_^y

[mainhatten]

the CDX file contains the order expression and the relative record number in the DBF file.

Hi Wolfgang,
speaking about the cdx implementation in vfp your above description is correct, but not complete.
Yes, the index expression is contained in the cdx header, but for each tag record not only the record #, but the corresponding value entry as computed by the index expression the is stored as well.Try in vfp

Code: Select all

CREATE TABLE cdxtest (pk int, cf chr(25))
INDEX on "My_cdx" + PADR(ALLTRIM(cf),20, "$") + "value" TAG keyval
INSERT INTO cdxtest VALUES (1, "Who's on First")
INSERT INTO cdxtest VALUES (2, "wHat's on Second")
INSERT INTO cdxtest VALUES (3, "I don't know")
USE

and check the resulting .cdx. You will see entries corresponding to cf field "mangled" by the index expression of Tag(1). As .idx/cdx are compressing indeces, sometimes leading char expression will ***seem*** to be missing (here countermeasured by "Who" and "wHat" to have different cases in first letters).
Insofar the OP probably searched for

I just wanted to use the CDX as my key value pair file storage with insert,update and deletion operation on the CDX file only

a very efficient key-value retrieval method like specialised Key-Value DBs offer, your description

Therefore you cannot use the standard CDX file as key-value storage - you need also the DBF file.

is 100% correct, as it is not doable via cdx alone, as the .cdx pairing is record number <> calculated value and a key/value search operation looks in the cdx for the record sporting the key and reads out corresponding value field.
Storing a "calculated value" in a DB of course goes against normalization rules, and hiding such columns in cdx offers little benefit aka "purloined letter" obfuscation - but you can do stupid pet tricks like the following:

Code: Select all

USE d:tvdbcdxtest.DBF
INDEX on pk TAG pk candidate
SET ORDER TO PK   && PK
SELECT pk, UPPER(cf) as ucf FROM cdxtest INTO TABLE pet_trick
SELECT pet_trick
SET RELATION TO pk INTO cdxtest
INDEX on "smartass "+STR(pk) + PADR(ALLTRIM(LEFT(cdxtest.cf, 7) + SUBSTR(pet_trick.ucf,8)), 35) + " Ahem" TAG stupid
use

which mangles the lower case info not included in the pet_trick dbf record into the cdx (check with any file editor) - and this is not expected if you do not mention the info of the value being recorded in the b-tree. Such tags are clearly not best practice and frail, as exact relations are needed, but such obfuscations can be enhanced for each record by speccing new alias/relation pair before saving.
Not to be compared with real encryption - but if quantum computing really breaks encryption considered safe when using traditional computers, such stupid pet tricks might become useful again.

(Yes, it has been mentioned that my mind sometimes follows strange and weird paths...)

regards
thomas

[wriedmann]

Hi Thomas,

sorry, my message seems to have explained things not clearly enough:
of course the current value of the index expression is stored in the index file - it needs that, otherwise it could not be found. Only the b-tree alone would not be enough....

But your CDX tricks are very interesting, and worth to check out....

A few weeks ago I searched for a possibility to read only the CDX, and not the DBF, to build fast filters, on index expressions only, but unfortunately this is not possible (AFAIK the Rushmore tecnology uses something).
But maybe Robert adds something to the RDD (or maybe it is already in the X# RDD and I have not found it yet).

Wolfgang

[FFF]

(AFAIK the Rushmore tecnology uses something).
But maybe Robert adds something to the RDD (or maybe it is already in the X# RDD and I have not found it yet).

FWIW, maaaany moons ago there was an article by Uwe Holz in SDT about Rushmore with Vo/CDX. Unfortunately i seem to have lost the issue ;(, but i remember this, because my first data-centric contact used FoxPro, when "Rushmore" was "the" buzzword

[mainhatten]

sorry, my message seems to have explained things not clearly enough:
of course the current value of the index expression is stored in the index file - it needs that, otherwise it could not be found. Only the b-tree alone would not be enough....

Hi Wolfgang,
on proof-reading I hesitated/wondered if your "order expression" should be interpreted as "values ordering records" or "expression building the order", but went with the second as in vfp expression ("the thing that gets EVAL()ed") is better defined as a concept plus the "index expression", which is returned by Key(), indeed is included once and decided to post...

But your CDX tricks are very interesting, and worth to check out....

thx

A few weeks ago I searched for a possibility to read only the CDX, and not the DBF, to build fast filters, on index expressions only, but unfortunately this is not possible (AFAIK the Rushmore tecnology uses something).
But maybe Robert adds something to the RDD (or maybe it is already in the X# RDD and I have not found it yet).

Hmmm, I never looked on the vfp C API for that. Rushmore uses bitmaps to filter the result set if one or more indeces can be used to describe the result set of a query (independant if xBase "for", SQL "where" or SQL "join on" (second only syntactic sugar for SQL where)) and traverses each tag helpful, then combines the resulting bitmaps, which is a fast operation.
I was under the impression that by now every xBase dialect used a variant.
Even worse: if Bitmap filtering is not supported in Clipper/VO/xSharp current RDD, chances are that bitmap indeces (in vfp keyword "binary") are not supported as well?
When you have to use low-selectivity filtering on several attributes, such bitmap indices can lessen the hit ISAM style access over network boundries creates compared to C/S systems, but they speed up local access as well.