Input needed on FoxPro local support

[robert]

Foxpro implements local variables a bit different from other development languages.
Take this code:

Code: Select all

LOCAL myVar
PRIVATE myPrivate
myVar = 42
myPrivate = DATE()
? Type("myVar")      && "N"
? Type("myPrivate")  && "D"
MyFunction("myVar")

FUNCTION MyFunction (cVar)
? "Inside MyFunction"
? TYPE(cVar)  && U
? Type("MyVar") && U
? TYPE("MyPrivate") && "D"
ENDFUNC

The Type() function apparently has access to the names and values of the local variables of the code that call is, where 'our' function MyFunction has not.
In most other development languages local variable names are "thrown away" at compile time and even runtime functions like Type() do not have access to these names.
In the other XBase dialects you would write

Code: Select all

? ValType(myVar) to see "N"

and of course inside MyFunction this would not work, since myVar is not declared at that level, so ValType(myVar) would not compile at all.

At this moment we are emulating this "visibility" of local variables when you compile Foxpro code with /fox2 compiler option.
Local variables are implemented as "true local variables", but we are also adding the name of the local to a special table in the runtime (the same table where privates are stored) with a code block that allows the runtime to read and write the variables from the locals stack of the function or procedure where the local is declared.
This works, but adding this information adds some overhead to the compiled code and it also produces problems for local parameters declared with the REF modifiers.

We are not sure how many of you are depending on this behavior (that locals are visible by name).
We are considering the following option:

When the /fox2 compiler option is enabled then locals are compiled just like privates. No extra get/set codeblocks are needed to read/write them, and they become untyped (just like in FoxPro)
We may be able to still parse the AS <type> clause and use it for intellisense, and we can probably also use the type to validate assignments at compile time, but the variable themselves will be untyped. That costs some runtime performance, but a lot less than having to add all the get/set codeblocks that are generated at this moment.
The only disadvantage of this is that (at least without extra work) in the example above the call to

Code: Select all

? Type("MyVar")

inside MyFunction
would return "N" and not "U".

Please let us know how important the existing FoxPro behavior for you is and what you think of our proposed solution. Or would you like us to help you clean up your code?

Robert

[kevclark64]

The way Foxpro makes variables visible in called functions and procedures has always seemed more than a little crazy to me since it is so easy to introduce program errors by modifying variables accidentally. That behavior is a big reason why I started declaring all my variables a long time ago. So I would never rely on that behavior; if I want a particular variable to be available within a function I always pass it as a parameter. I personally can't think of a scenario in which relying on this behavior of Foxpro would be either necessary or a good idea, but others may have a different opinion.

[Loy2888]

Hi Robert,

IMO, the existing FoxPro behavior for this is important especially when you have recursive functions - you may want to make those variables local for each recursion.. So, for me, the FoxPro behavior is right on these situations.

[Eric Selje]

It's important to note that the PRIVATE keyword in FoxPro does not actually denote the variable's scope at all. What it actually is doing is saying "If I have any variables in scope with this name at this point, shield it from getting changed while this function is running because I'm about to declare another variable with the same name and I don't want to overwrite the previously-named variable." Is that weird? Yes. Is it important that this behavior remains? Sadly, yes.

The only scope keywords are PUBLIC and LOCAL. A variable that's instantiated without one of those two keywords is in scope as long as the function that it's initialized in is in scope. It's very bad practice to not declare scope, and also to assume a variable is in scope, but that's the kind of stuff that allows bad programmers to be productive in VFP.

Hope this helps,

Eric

[JanX]

Robert,

I do not work with FoxPro. But another Clipper descendant. And I think the problem ist your line MyFunction("myVar"). You pass a text "MyVar" to the function and not the var with Value 42. Should be MyFunction(myVar)

Just a guess of a non FoxPro developer ...

[robert]

Jan,
I am sorry but this was done on purpose:
The Type() function in FoxPro accepts a string and looks for a local variable with that name. I wanted to demonstrate that (without extra work or special permissions for an X# version of the Type() function) this is not possible in a function in most other compiler languages.

With the /fox2 compiler option we have added support for this behavior, but that has a disadvantage that it adds quite some overhead to the code. At this moment the /fox2 compiler option "registers" the local variables to the runtime together with a Get/Set code block that allows to read and write its current value.
There are probably only a hand full of functions in the runtime that need to be able to read locals like this:
- Type()
- Evaluate()
- & operand for macro substitution.

My current thinking is that we can detect that a block of code calls any of these. And then only when these are called then we will actually "expose" the locals to the runtime.

Robert
PS: You can safely say over here that your "descendant" is Xbase++.

[robert]

Hello to all.
This is what we plan to do for the next build:

- The current Locals support for macro compiler and other runtime functions that need access to locals will be removed. This means that your code in the FoxPro dialect will have to be recompiled !

- We will add an attribute to the runtime that can be used to mark "Special" functions in the runtime that need access to locals (NeedsAccessToLocalsAttribute)

- Function in the runtime (but you could use that in your code too) that need access to locals by name will be marked with this attribute:

Code: Select all

[NeedsAccessToLocals];
FUNCTION Type(cString AS STRING) AS STRING

- When the compiler detects that you are compiling with the /fox2 (Expose Locals) commandline option AND when a call to a function/method is detected marked with this special attribute then the compiler will generate some extra code before and after the function call
to :
- Before the call: add name/value pairs to a special table in the runtime for each local
- After the call: call a runtime function to detect if any locals were changed by the special function. When yes, then simply update all the locals.
- After the call: regardless if the locals were changed or not, call a runtime function to clear the locals table.

This has the advantage:
- even when you compile with /fox2, when your code does NOT use one of the special functions like Type(), Eval() etc then the compiler will not "pollute" the generated code to register locals
- we are no longer using Get/Set codeblocks, so this should also work with REF parameters and OUT parameters
- the visibility of locals to the runtime is restricted to functions that need it. Other functions will not be able to "touch" your locals.


Robert

[mainhatten]

Hi Robert,
[reordered a bit to better answer ]

This has the advantage:
- even when you compile with /fox2, when your code does NOT use one of the special functions like Type(), Eval() etc then the compiler will not "pollute" the generated code to register locals
- we are no longer using Get/Set codeblocks, so this should also work with REF parameters and OUT parameters
- the visibility of locals to the runtime is restricted to functions that need it. Other functions will not be able to "touch" your locals.

I thought a general solution to a local problem was a bad idea (could not resist), so this is good news.
Also it opens up options for those who try to fix their code for better speed in xSharp while staying "runnable" in vfp and keeps runtime / performance between xSharp dialects at a level easy to compare and tweak if something puts breaks on code speed.

- When the compiler detects that you are compiling with the /fox2 (Expose Locals) commandline option AND when a call to a function/method is detected marked with this special attribute then the compiler will generate some extra code before and after the function call
to :
- Before the call: add name/value pairs to a special table in the runtime for each local

Here I did not envision a table, but a specific object / collection mirroring "local" to be added as a last parameter to the xSharp implementation of the functions - why a table ? Objects are easier to to handle in memory ?
As that should be enough for Type() end eval(), which probably will have most # in sources, the next step

- After the call: call a runtime function to detect if any locals were changed by the special function. When yes, then simply update all the locals.

could be avoided, if there were 2 special attributes, signifying need to read or read/write, and the check plus update can be sidestepped

Code: Select all

(NeedsRAccessToLocalsAttribute)
function Type(tcStr)
function Eval()

(NeedsWAccessToLocalsAttribute)
function someOtherStuff()

- After the call: regardless if the locals were changed or not, call a runtime function to clear the locals table.

Whatever is faster - might be needed again, and just checking if key is already in a hash table might be faster
You were certainly correct that a few of vfp type() and eval() calls could be written in a way much better at least for xSharp, but sometimes it makes a lot of sense in vfp to work via strings - and not going against strict functon definition, for instance.

Code: Select all

function DoWithTable(tcAlias)
local lcType, loRef
lcType = Type(m.tcAlias + ".WhatFieldINeedToCheck") 
loRef = Eval("m.goApp."+m.lcSpecialObjPath)   && goApp as public, of which a handful can be beneficial..

For such code parts it would be great to have "pure function" versions available as well, again sidestepping the analysis of locals, as a coder interested in higher speed in his xSharp version can identify those in his code, and change to

Code: Select all

function DoWithTable(tcAlias)
local lcType, loRef
lcType = Type_Pure(m.tcAlias + ".WhatFieldINeedToCheck") 
loRef = Eval_Pure("m.goApp."+m.lcSpecialObjPath)

and keep his ability to run identical code in vfp and xSharp by adding #defines mapping the xxxx_pure function names back to xxxx for vfp compilation. Benefit for those making additional effort and avoiding a speed penalty in vfp code circumvented with totallay different code only.

- The current Locals support for macro compiler and other runtime functions that need access to locals will be removed. This means that your code in the FoxPro dialect will have to be recompiled !

Can you get more specific on embolded part ? Macroexpanding a local variable should be better than having to use a private (if that was intended). Or is it that the compilation is done into a new function, isolating all previous locals (Have not watched your video on codeblocks, might help clear up my picture)? I'd have to think about that a bit, but SWAG is it might be necessary to have the functionality of (NeedsRAccessToLocalsAttribute) available for the macrocompiler generated inline "function" as well.

Hope I was clear & regards
thomas

[robert]

Thomas,

Hi Robert,

- Before the call: add name/value pairs to a special table in the runtime for each local

Here I did not envision a table, but a specific object / collection mirroring "local" to be added as a last parameter to the xSharp implementation of the functions - why a table ? Objects are easier to to handle in memory ?
As that should be enough for Type() end eval(), which probably will have most # in sources, the next step

- After the call: call a runtime function to detect if any locals were changed by the special function. When yes, then simply update all the locals.

I wrote table but it is actually a dictionary.

could be avoided, if there were 2 special attributes, signifying need to read or read/write, and the check plus update can be sidestepped

Code: Select all

(NeedsRAccessToLocalsAttribute)
function Type(tcStr)
function Eval()

You are probably thinking of the use of type in expressions such as

Code: Select all

Type("MyLocal")

However a function such as Type or Eval could also receive a string in the form of:

Code: Select all

Type ("MyLocal = 10")

which is an assignment. It should still return "N" because the value in MyLocal is numeric after the assignment.
So it is too easy to assume that Type() and Eval() will not assign values.
I have changed Type() to check to see if the string it received is "just" an identifier. When it is then it will look in the current visual locals, privates and publics and find the variable (and will avoid the overhead of calling the macro compiler).

When the string is not a simple identifier then the string is sent to the macro compiler and the resulting macro is evaluated (like in the assignment above).
You can see the new code for Type() here:
https://github.com/X-Sharp/XSharpPublic ... o.prg#L136

You may wonder why the Identifier check is so complicated. Well, we allow identifiers with not only ascii characters but also accented or even Chinese or Japanese characters. The same rules exist in the compiler and macro compiler.

The implementation of the Memvars and the locals dictionary is in:
https://github.com/X-Sharp/XSharpPublic ... MemVar.prg

This code is a bit tricky because for speed reasons we want to avoid having to allocate MemVarLevel objects for functions/methods that do not declare new privates. And this code also is written to work in a multi threaded program where each thread has its own "privates stack".

The functions that the compiler calls (such as __LocalPut() and __LocalsUpdated() ) are in:
https://github.com/X-Sharp/XSharpPublic ... upport.prg

Robert

[mainhatten]

Hi Robert,

I wrote table but it is actually a dictionary.

sometimes my reading is too literal, as in trying to envision specs...

could be avoided, if there were 2 special attributes, signifying need to read or read/write, and the check plus update can be sidestepped

Code: Select all

(NeedsRAccessToLocalsAttribute)
function Type(tcStr)
function Eval()

You are probably thinking of the use of type in expressions such as

Code: Select all

Type("MyLocal")

However a function such as Type or Eval could also receive a string in the form of:

Code: Select all

Type ("MyLocal = 10")

which is an assignment. It should still return "N" because the value in MyLocal is numeric after the assignment.
So it is too easy to assume that Type() and Eval() will not assign values.

I'd agree hands down if vfp supported prefix assignment - but the old fox does not support Type("++MyLocal"). Coming from Roslyn C#, thinking about pre/postfix operators of course is something to keep in mind. Your example works not as you described above in old vfp, it does not assign, but returns comparison result in type(), but changes locals from other stack level if call to eval() is done with string filled with function sporting ref-parameter, see:

Code: Select all

CLEAR
LOCAL lnRef
? "errorfree unassigned compare", TYPE("lnRef = 10")
? "lnRef", lnRef
lnRef = -5
? "compare", TYPE("lnRef = 10"), eval("lnRef = 10"), TYPE("lnRef + 10"), eval("lnRef + 10")
? "Before Call", lnRef
? inc_l(@lnRef)
? "Before Type1", lnRef
? "result Type1", TYPE("inc_l(@lnRef)")
? "After Type1", lnRef
? "result Type2", TYPE("10 = inc_l(@lnRef)")
? "After Type2", lnRef
? EVALUATE("inc_l(@lnRef)")
? "After EVALUATE()", lnRef, " changed local in calling using vfp @ ref parameter, which ***kinda*** makes sense"

FUNCTION inc_l(tnRef)
tnRef = IIF(VARTYPE(tnref)="N", tnRef, 0) + 1
return m.tnRef

Have not really tried hard to force something changing the locals in the old fox,except the code above - could also be possible with macros, but would be the exception found in code, not the rule. And a less often used mantra is "code to exceptions", so perhaps for such special cases a "eval_writeLocal" function could be added to xSharp, same as eval_Pure() I hinted at, which need to switch to in most cases could be detected by compiler at encountering ref parameter in the function call inside the eval-string. Perhaps Antonio, Matt, Dragan and other fox heads drawn to funny areas can post some code variations

I have changed Type() to check to see if the string it received is "just" an identifier. When it is then it will look in the current visual locals, privates and publics and find the variable (and will avoid the overhead of calling the macro compiler).

Probably implemented already, but first in search order should be alias.fieldname in vfp, which throws almost everybody used to other languages also the reason to m.Dot your read-access code besides minimal speed benefit.

When the string is not a simple identifier then the string is sent to the macro compiler and the resulting macro is evaluated (like in the assignment above).
You can see the new code for Type() here:
https://github.com/X-Sharp/XSharpPublic ... o.prg#L136

You may wonder why the Identifier check is so complicated. Well, we allow identifiers with not only ascii characters but also accented or even Chinese or Japanese characters. The same rules exist in the compiler and macro compiler.

The implementation of the Memvars and the locals dictionary is in:
https://github.com/X-Sharp/XSharpPublic ... MemVar.prg

This code is a bit tricky because for speed reasons we want to avoid having to allocate MemVarLevel objects for functions/methods that do not declare new privates. And this code also is written to work in a multi threaded program where each thread has its own "privates stack".

The functions that the compiler calls (such as __LocalPut() and __LocalsUpdated() ) are in:
https://github.com/X-Sharp/XSharpPublic ... upport.prg

Will have to look at the code with time to think, will perhaps comment then

regards
thomas