Chances for some small optimizations of vfp runtime?

[mainhatten]

Hi Robert, Chris & Fabrice,
I read a bit in the sources, trying to get my footing back in Dotnet and xSharp.

a) If I am not mistaken, the code for the vfp "evaluate()" is in
/Runtime/XSharp.RT/Functions/Macro.Prg
as it has the NeedsAccessToLocals(TRUE) qualifier Robert mentioned when discussing non-function variable access seen also in Type().

Code: Select all

/// <include file="VoFunctionDocs.xml" path="Runtimefunctions/evaluate/*" />
/// <seealso cref="NeedsAccessToLocalsAttribute" />
[NeedsAccessToLocals(TRUE)];
FUNCTION Evaluate(cString AS STRING) AS USUAL
    RETURN Evaluate(cString, TRUE)

/// <include file="VoFunctionDocs.xml" path="Runtimefunctions/evaluate/*" />
/// <param name="lAllowSingleQuotes">Should single quotes be allowed as string delimiters.</param>
/// <seealso cref="NeedsAccessToLocalsAttribute" />
[NeedsAccessToLocals(TRUE)];
FUNCTION Evaluate(cString AS STRING, lAllowSingleQuotes AS LOGIC) AS USUAL
    LOCAL oMacro AS XSharp._Codeblock
    LOCAL uRes   AS USUAL
    oMacro := MCompile(cString, lAllowSingleQuotes)
    IF oMacro != NULL_OBJECT .AND. ! oMacro:IsBlock
        uRes := oMacro:EvalBlock()
    ELSE
        // strange but evaluate on a codeblock returns the block in stead of evaluating it
        uRes := oMacro
    ENDIF
    RETURN uRes

Robert mentioned the drawback of NeedsAccessToLocals(TRUE), as performance is hit every time collecting all visible locals at start and checking if changes need to be persisted at function end. The implementation hits each vfp Evaluate() twice, as vfp only knows single paramter.
C# compiler is capable to work with optional / default parameters, IIRC xSharp can do the same?
If the function signature defines 2nd Parameter as optional, defaulting to TRUE,
the single parameter evaluate() is not needed for code from vfp sources and does not cause NeedsAccessToLocals(TRUE) to fire twice on each call. Should result in better performance?

b) [The Vfp compatibility list](https://www.xsharp.eu/itm-help/foxpro-c ... ility-list) from 9/2023 lists GetPEM() under "Functions that won't be supported". Please move into "partially supported (more or less)" and support GetPEM only for "P", as this is the best way to access a property in vfp if object handle is present and (list of) name(s) need to be accessed.

Attached is micro-benchmarking vfp code which compares the runtimes of GetPEM() to 2 versions of Evaluate() and Macroexpansion when the always time for scaffolding the test is eliminated.

This function also is a real function, no need for NeedsAccessToLocals(TRUE).
Untested:

Code: Select all

FUNCTION GetPEM(oObj as USUAL, cString AS STRING) AS USUAL
    ///try
 		RETURN oObj.&cString
   /// catch 
                /// throw Exception ("Property " + m.cString + " not found")
   ///endtry

I have not added xSharp benchmarking, as I am uncertain if bare &cString Macro is the most fitting. When scanning the sources for EvalBlock I found several places, so perhaps there is a better way plus the Evaluate() times of xSharp will be better if my gut feeling from a) is correct. No need to create a storm in a teacup posting preliminary bad results for xSharp.

c) really no change in vfp compatibility list from 9/23? Sounds sad...

regards
thomas

[Chris]

Hi Thomas,

Yes indeed, this is the source code location of Evaluate()!

Please correct me if I am wrong, but I think you believe that using the NeedsAccessToLocals attribute on a method/function, causes a performance hit on that particular function itself, am I right? If that's the case, this isn't how it works, the performance hit occurs only on the code that calls such a function, not in the function itself.

When you call for example Evaluate() from a function named Test(), then Evaluate() needs access to the locals of this function, so if Test() defines a number of locals, then those locals are collected before the call to Evaluate() and updated after it, if necessary. If Test() has no local variables, then there's no related performance hit involved at all. Btw, in most cases the delay of handling the locals should be very small, compared to the time needed by Evaluate() to macro compile an expression, so probably is not something to really worry about.

About GetPEM(), some of its functionalities cannot be implemented in .Net, but I think some others indeed can, unless I'm not thinking of something that prevents this. Will let Robert comment on that (he marked it as obsolete), just give him a little time as he's currently on vacation.

About VFP updates, it's just this particular text in that page that hasn't been updated, otherwise there have been plenty updates since then, you can see this if you view the history of the \Runtime\XSharp.VFP folder. But, in any case, how much focus we put in each dialect is dictated by our customers, who help us keep X# going and unfortunately we haven't had much financial support from VFP developers so far. But still, we are still working on VFP compatibility at reasonable pace I think.

[robert]

Thomas,
The [NeedsAccessToLocals(TRUE)] attribute is only effective when compiling in the VFP dialect.
The function Evaluate(cString) is in the XSharp.RT assembly and not compiled in the VFP dialect. So its single local cString is not pushed to the list of locals. You can see that if you decompile the XSharp.RT assembly:

Code: Select all

[NeedsAccessToLocals(true)]
public static __Usual Evaluate(string cString)
{
	return Evaluate(cString, lAllowSingleQuotes: true);
}

Assume this code

Code: Select all

Function Start() As Void Strict
    LOCAL u AS USUAL
    try
        u = "Hello World"
        ? Evaluate("u+= ' from VFP dialect'")  // assign a new value to u
        ? u                                    // show that u was updated
    catch e as exception
        ? e:ToString()
    end try
    wait

This gets compiled to:

Code: Select all

public static void Start()
	{
		try
		{
			<Module>.$AppInit();
			int nLevel;
			nLevel = Functions.__MemVarInit();
			try
			{
				try
				{
					__Usual u;
					u = "Hello World";
					Functions.__LocalPut("u", u);
					__Usual uValueList;
					uValueList = Functions.Evaluate("u+= ' from VFP dialect'");
					if (Functions.__LocalsUpdated())
					{
						u = Functions.__LocalGet("u");
					}
					Functions.__LocalsClear();
					Functions.QOut(uValueList);
					Functions.QOut(u);
				}
				catch (Exception e)
				{
					Functions.QOut(e.ToString());
				}
				Functions._wait();
			}
			finally
			{
				Functions.__MemVarRelease(nLevel);
			}
		}
		finally
		{
			<Module>.$AppExit();
			GC.Collect();
			GC.WaitForPendingFinalizers();
		}
		

The macro compiler does not "know" that u is a local. It calls "__MemVarPut" to update the variable with the name "u".
The XSharp.Memvar class "knows" that there is a local with the name "u".
Therefore, the "access to locals" will only work if you are compiling with /memvar (Enable Memvar Support), since the references to the locals are stored in the "dynamic memory variable stack".

Robert

[mainhatten]

Hi Robert,
thx for the detailed explanation - esp. last paragraph - makes clear, how xSharp handles 1st parameter array modification or creation in the cases like aDir, aFields or aMembers, where an existing array is modified (even when called without @ on calling source line) or created inside the calling functions stack - if there is no name defined in local, normal private array will be created.

Your example would not work in vfp:
a) "+=" is not a supported syntax in the compiler, will only be sendkeys/keyboard replaced after entering
b) assignment operation (":=" of xSharp Core) is not allowed inside the evaluated string, has to be done as Macro (even in form of the store command). Evaluate in vfp will interpret "=" after "u" as comparison operator and will return the result of the comparison expression.
But of course you are correct, Evaluate() not only can access local of calling function / method, but can modify the stack, not only in aDir(), but in explicitly coded "DoSomething(@RefParameter)" if RefParameter is modified inside DoSomething.
Performance benefit of simplifying Evaluate() signatures into single function with optional 2nd parameter

Code: Select all

Evaluate(cString AS STRING, lAllowSingleQuotes = TRUE AS LOGIC) AS USUAL

(hope I remembered syntax and order correctly) would be minimal, as method calls are much faster in Dotnet compared to vfp.
GetPEM(obj, cString) IMO still the cleanest way to "get" a property if all you have is an object handle and the name as string and should be "partially supported" for the property case, stating that Events and Methods use cases are not supported, as such calls will be found in applications - developers keen on having optimized source would hate to introduce #xSharp for such places as long as code needs to support both platforms.

regards
thomas

[mainhatten]

Hi Chris,

but I think you believe that using the NeedsAccessToLocals ... so probably is not something to really worry about.

Mostly correct, I thought that before the call local would be collected and then injected - but viewed that as "time between calling line, wanted Macro expansion and again copy back before next source line. Yes, not something to worry about, but still: best optimization is not running such code at all

About GetPEM(), some of its functionalities cannot be implemented in .Net, but I think some others indeed can, unless I'm not thinking of something that prevents this. Will let Robert comment on that (he marked it as obsolete), just give him a little time as he's currently on vacation.

That's why I specifically carved out to support "getP(roperties)", not events or methods. The micro-benchmark shows why great programmers use it in vfp directly & often (I usually "optimize" getting the object from long chain into local, but in fast coding first write

Code: Select all

Eval("loMyCachedObj." + m.lcMyKnownPropertyname), 

only to ritually facepalm myself on 3rd read of my own code just before check-in... Not obsolete at all (see runtimes in benchmark), and really no effort to add.

About VFP updates, it's just this particular text in that page that hasn't been updated, otherwise there have been plenty updates since then, you can see this if you view the history of the \Runtime\XSharp.VFP folder. But, in any case, how much focus we put in each dialect is dictated by our customers, who help us keep X# going and unfortunately we haven't had much financial support from VFP developers so far. But still, we are still working on VFP compatibility at reasonable pace I think.

Uuuumpfff - that page is - at least for me - the best way to get an idea, what to expect when trying to implement or port something to xSharp. I don't want to go for example to
https://www.xsharp.eu/runtimehelp/html/ ... embers.htm
only to read that it still falls under "todo".
I have read that SQL support has grown - from SP support and "in language", which would be necessary on mobile / ARM, as ADS has no driver there and vfpOLEDB is x86 only. Mobile options under X#3 have lured me here again, as that is an area I cannot work for either with vfp, vfpA or vfp compiler, while Dotnet can target those devices. And this is weird me, who reads source code, not comics in leisure time... How can any new vfp dev interested in xSharp get a fast overview, whether the effort to learn & try out ist not wasted?
If you put a link to a tiny code snippet to each entry there and update that page with new testcode based on further implementation, entry might be easier (or might get error reports, if implementation has bugs not found when implementing, but vfp head tries special feature...) Will perhaps write a bit more from vfp POV over the weekend.

regards
thomas

[wriedmann]

Hi Thomas,
PMFJI, but VFP features seem to me really a hen - egg problem.
Investing time & money implementing VFP features when there is no interest and no people willing to help is very hard, and without that VFP people will not be attracted.
VO support in X# is so great because first of all the devteam has a VO background, and most people using it have also VO background, and therefore not only most of the money comes from people with VO background, but also support issues and enhancement requests.
IMHO to have really great VFP support there would be a need for a few VFP companies engaging the devteam for a migration to X#.
A similar thing was also done for at least one Harbour project.
Wolfgang

[kevclark64]

As far as I am aware, the project converter for VFP to X# is not completed. That would have to be done before a company could contemplate a migration from VFP to X#. If that were available and could get you 90% of the way toward a working X# project then there might be interest in the FoxPro community. But whether it's worth putting the effort into VFP conversion just for the chance of possibly getting some interest is certainly a good question.

[mainhatten]

Kevin,

my guess is xSharp implementation was started without detailed analysis which vfp features would be needed for how large a percentage of current vfp apps. More a kind of "Which syntax do we need to support, so that VO programs can be built with Vfp syntax" and not "How and why are vfp programs architected in what way?"

Add the typical programmer optimism on the topics of needed effort and time
MS had larger groups for last FoxPro integration, Rushmore analysis and optimization, Windows integration plus First 2 vfp versions. Versions 6, 7, 8, 9 and Sedna were done by smaller teams, but probably still >5, with internal test teams and larger beta test groups drawn from (back then) huge dev group.

Most Vfp apps I know of use several of the following:

• 1) working on local data (subsets) for speed, Rushmore needed for DBF subsets
  2) direct data binding to with buffered cursors, views or tables, persisting via TableUpdate(), no scatter
  3) validation of DBF fields/records via DBC constraints and checks, sometimes read from data dictionary
  4) Saves with Transactions or Rollback in case of error, even with DBF
  5) Option to switch backends with relative ease, depending on chosen tech (SPT, View, Cursoradapter)
  6) SQL capabilities enhanced in each version
  7) Often some kind of ActiveX usage, from Controls to Office integration

currently not documented or not implemented is:

• 1) Rushmore, which was a large reason of pre-Vfp Foxpro to outrun dBase, Clipper, dbMan, Arago.
  Should be available via ADS or Vfp OleDB drivers, but is not (documented or (better) demonstrated).
  2) Tableupdate() and supporting functions are not implemented (worse: seem to be low in priority)
  3) Some DBC functionality documented and supported, have not tested things I saw no documentation for
  4) Transactions not implemented for DBF, probably workable via ADS or Vfp ODBC or OleDB drivers (Intel only)
  5) Newest and best tech (Cursoradapter) in name only implemented, 97% of functionality is missing.
  SPT via ODBC is restricted to vfp6 functionality and basically is a FoxPro DOS technology...
  6) Implementation scope at least unclear
  7) No direct way to implement, wording in doc is too negative, as Dotnet supports ActiveX

Usually 4 or 5 of those 7 points are used in a Vfp app which edits data. Currently xSharp supports some enhancements over FoxPro DOS/Windows, misses some great functionality (Rushmore, some functions) of FoxPro DOS/Windows.

The developers / customers still working with vfp stay for the data handling, which touches first 6 points.

Building a GUI converter without a critical working path of 2,3,4,5&6 is wasted effort, as data cannot be persisted.
Aiming for accuracy in converter has high risk of wasted effort as target GUI of customer might be different.

From Vfp vantage xSharp offers ability to run some Foxpro DOS style programs, with long field names&types, but missing Rushmore (late 80s invention) for "driverless code". The Vfp DNA of disconnected local data, buffered and able to employ SQL backend functionality like SP, validation, constraints even on DBF seems to be lost in xSharp dev perspective. Missing documentation of "vfp milestones reached, current roadmap" 3 years after most functionality was promised to be working creates impression of another failed attempt in new visitors.

regards
thomas

[kevclark64]

Thomas, I agree with you that much work needs to be done on the back end for real VFP compatibility. But all those things will need a lot of testing and iterating, and if there isn't a project converter then there won't be a base of people doing the testing. If there were a working project converter then some people (like me) would do the conversion and start working through the issues. It may be a waste of time to perfect a project converter without having things like cursor adapters, tableupdate, rollback, rushmore, etc., but it's definitely a waste of time to try to implement those things without a project converter since there will be no VFP-oriented user base to use them.