DBF Append Speed in X# vs VO - Huge Difference

This forum is meant for questions and discussions about the X# language and tools
User avatar
Chris
Posts: 4953
Joined: Thu Oct 08, 2015 7:48 am
Location: Greece

Re: DBF Append Speed in X# vs VO - Huge Difference

Post by Chris »

Hi Horst,

There have been a lot of improvements and fixes in the RDDs since 2020, and lately we haven't gotten any report about index corruption anymore, but of course something might have still slipped, a bug that we are not aware of. Did you make sure you had the index file properly opened when you edited the dbf from the tool?
Chris Pyrgas

XSharp Development Team
chris(at)xsharp.eu
User avatar
Chris
Posts: 4953
Joined: Thu Oct 08, 2015 7:48 am
Location: Greece

Re: DBF Append Speed in X# vs VO - Huge Difference

Post by Chris »

Guys,

About dbf speed issues, there have been many improvements in the past months also in this area, but not all cases are improved. If you would like us to (try to) improve performance in your particular scenario, please send us a full sample showing the speed issue and we'll have a look.
Chris Pyrgas

XSharp Development Team
chris(at)xsharp.eu
User avatar
Horst
Posts: 337
Joined: Tue Oct 13, 2015 3:27 pm

Re: DBF Append Speed in X# vs VO - Huge Difference

Post by Horst »

Hello Chris

I was seeking with the tool, so i think the index was open.
If it will happend again, i try to reproduce the error and send you more informations.
On the files i am working with the x# app, i cant see troubles with the indexes.

Horst
User avatar
robert
Posts: 4554
Joined: Fri Aug 21, 2015 10:57 am
Location: Netherlands

Re: DBF Append Speed in X# vs VO - Huge Difference

Post by robert »

Horst,

If you're accessing the DBF/CDX with an external tool, make sure that the tool uses the same collation settings that your app uses.
Otherwise, corruption of the CDX can occur

Robert
XSharp Development Team
The Netherlands
robert@xsharp.eu
User avatar
wriedmann
Posts: 3773
Joined: Mon Nov 02, 2015 5:07 pm
Location: Italy

Re: DBF Append Speed in X# vs VO - Huge Difference

Post by wriedmann »

Hi to all,
we have several projects were we are running both X# and VO applications on the same database, and we don't have registered index corruptions the last months.
About performance we cannot say a lot because in the network both VO and X# applications are running much slower than in the past, maybe because Microsoft make some changes in the network stack.
I will do some tests in the weekend.
Wolfgang
P.S. for the people that may interest that: all USB-C and Thunderbolt docking stations (both no name and large producers like HP, Dell and Lenovo) have really slow network interfaces, between 30 and 50% of the speed of either integrated network cards or dedicated USB-C network adapters
Wolfgang Riedmann
Meran, South Tyrol, Italy
wolfgang@riedmann.it
https://www.riedmann.it - https://docs.xsharp.it
Jamal
Posts: 322
Joined: Mon Jul 03, 2017 7:02 pm

Re: DBF Append Speed in X# vs VO - Huge Difference

Post by Jamal »

Hi Chris,

For an example of the speed issue, please see the following post which is still relevant:

https://www.xsharp.eu/forum/topic?p=17140#p17140

Jamal
Jamal
Posts: 322
Joined: Mon Jul 03, 2017 7:02 pm

Re: DBF Append Speed in X# vs VO - Huge Difference

Post by Jamal »

Hey Guys,

I am still lurking around!

The speed issue is still in X# 2.21.

It took 22.51 minutes this time because I have a faster SSD on a new PC, but it is a long time. The dbf size after append is 20 MB.

The issue seems to occur when a DBFCDX order is active. When an order is not used, the loop is very fast and took only about 7 seconds.

Code:

Code: Select all

FUNCTION Start() AS VOID STRICT
    LOCAL cDBF AS STRING
    LOCAL aFields AS ARRAY
    LOCAL i AS DWORD

    LOCAL nStart, nElapsed AS REAL8

    RddSetDefault ( "DBFCDX" )

    cDBF := "C:\test\mytest"
    FErase ( cDBF + ".cdx" )

    aFields := { { "LAST" , "C" , 20 , 0 } }


    DbCreate( cDBF , aFields)
    DbUseArea( ,,cDBF )

    DBCreateOrder("last", "C:\test\mytest.cdx", "last")   // comment this out and it will be faster


   // wait

    nStart := Seconds()

    ? "start : " + ntrim(nStart)

    FOR i := 1 UPTO 1000000
        DbAppend()
        fieldput(1, asstring(i))
        // ? i
    NEXT

    nElapsed := Seconds() - nStart

    ? "Elapsed: " + ntrim(nElapsed) + " seconds" + " - Minutes: " + ntrim(nElapsed/60.00)

    wait


   Console.ReadKey()

Jamal
User avatar
Horst
Posts: 337
Joined: Tue Oct 13, 2015 3:27 pm

Re: DBF Append Speed in X# vs VO - Huge Difference

Post by Horst »

Hi Jamal

I tried it also with your sample. On my machine it took 50 min. !
And its the Index.
When i open the file shared and create the index at the end its 54s . In exclusive mode 19s

You can't make the index after the append ?

Horst
Jamal
Posts: 322
Joined: Mon Jul 03, 2017 7:02 pm

Re: DBF Append Speed in X# vs VO - Huge Difference

Post by Jamal »

Hi Horst,

Actually, that was a workaround which I am using now, but I wanted to mention that the issue still exists when an index file is active.

Hopefully, sometime in the future a fix is found.

Jamal
User avatar
Chris
Posts: 4953
Joined: Thu Oct 08, 2015 7:48 am
Location: Greece

Re: DBF Append Speed in X# vs VO - Huge Difference

Post by Chris »

Guys,

The fundamental problem with this seems to be that in X# the time needed to complete the test grows exponentially with the number of records, while in VO it grows linearly. Will log this, maybe something can be changed to the index algorithm in X# to speed it up. Without an index, it grow linearly also in X#, which is good.

But is it a realistic real life scenario needing to add millions (or 100,000s) of records at once with the index activated?
Chris Pyrgas

XSharp Development Team
chris(at)xsharp.eu
Post Reply