Fox developper "UNBOXING XSHARPDEV " experience

[jpmoschi]

I want to share an experience with the Xsharp Forum.

A bug in the latest revision v2.11.0.1 made me wonder why what happened? (a confusion between variables, free procedures, class methods and arrays that at first I reported as a warning and then I realized that it was a very important error that triggered my alerts. For more details follow the following link Warning, not error: incorrect confuse a method with a field or memvar. · Issue #988 · X-Sharp/XSharpPublic (github.com)

As to my modest understanding the problem was in the compiler, I downloaded XSharpDev from GitHub and began to try to understand it since it contains a gigantic accumulation of culture poured out by Robert (no one is surprised at this point if I say he is a genius, right?)

I began to absorb culture via the XSharpDev project and from there the next title emerges. If you continue reading keep in mind that there may be gigantic mistakes because I am trying to tell the experience of learning, know how to excuse them and please do not hesitate to observe them to continue learning

Unboxing XsharpDev. How is the XSharp language built?

Physically the main project of XSHARPDEV is XSharp.CodeAnalysis.dll (there is a similar one for c# developed by MS), this is the "Heracles" of this story, the program written by Robert in C# where all own and third-party components are integrated. Robert uses resources from the Open Source community and if they are not enough he simply creates his own.

Particularly for the bug I mentioned at the beginning Robert modified XSharp/src/Compiler/XSharpCodeAnalysis/Binder/Binder_Invocation.cs, This is the link for the most curious, although at this point it is almost an anecdote: .

What are the main components?
I reiterate my apologies if I'm wrong, I continue to order parts according to the following criteria: what I think is most important first:

• Obviously ROSLYN or actually "SDK Compiler Platform" (by MS).
  o The official repository is here dotnet/roslyn: The Roslyn .NET compiler provides C# and Visual Basic languages with rich code analysis APIs. (github.com)
  o Released by Microsoft a few years ago for the community
  o Xsharp uses a static image of a historical version of Roslyn, it's important to say because I don't know how far away from the latest version of the platform it is and perhaps this comment is of little importance at the moment. Robert, neatly, made changes in 198 of the 10297 files that make up the Open-Source project by putting the compilation policy #if XSHARP before.
• MSBuild, the MS build engine. I will not comment, but if you want to understand at least a little the details and why there are many things within XSharpDev it is essential to know it specially to understand the strategy of continuous improvement and how is the work done by Robert in search of support in our community
• ANTLR, a tool for defining languages, has a strange name "ANother Tool for Language Recognition"
  o The version in use is almost the latest ANTLR4 https://github.com/antlr
  o Strategically Robert chooses this successful open-source tool for the manufacture
  of languages that standardizes the way to assimilate old XBase languages with the same origin, but with small or huge differences generated by the evolution of each one for more than 30 years. At the same time guaranteeing the accompaniment of the ANTLR community, much larger and more diversified than ours (XSHARP) (at least for now)
  o In the Xsharp blog I found a historical document where Robert explains a little why he joins the project, you can find it in https://www.xsharp.eu/articles/blog/hav ... t-our-mind
  o It is a gigantic universe, also open source, invented by "another Robert". There is a book written by its Author and the idea is that the grammatical rules for each language are defined (from a simple calculator to languages as diverse as SQL, JavaScript and others), which are then parsed to generate source code c# or others, which then translates Robert to Types recognized by Roslyn. Then the result is the DLL that understands our PRGs.
  o Those rules are defined in a 1500-line file (yes, it's very powerful!) and include all the syntax of the language. If you are curious, the file is this XSharpDev/XSharp.g4 at main · X-Sharp/XSharpDev (github.com)
  o Someone can answer: is it necessary to buy and read the whole book or is there a shorter way?
• Dot Net Arcade. Sharing Infrastructure Across .NET Core (by MS)
  I do not want to go into detail, it makes our work easier, coordinates the distribution of all the components, projects of tens of thousands of files are automatically in “My Computer” and in that of all the participants of this world network and we share them collaboratively and everything works respecting the indispensable seniority of Robert, the one who knows.
  You can find it in github.com/dotnet/arcade
  Basically, this allowed me to test the fix that I myself reported almost immediately after Robert made the adjustment in version 2.11a generating the entire project and signing it digitally with the same public signature without waiting for the release of the new version of XSharp 2.12 soon to be published.
• XUNIT Unit Test projects (Xunit.Net) a free, open source, community-focused unit testing tool for the .NET Framework.
  Another strategic choice of Robert, without a mechanism for unit testing, it is impossible to validate the development of a programming language. This is where I believe lies the secret to XSharp's success as a stable and continuously evolving finished product firmly conducted.
• SandCastle Documentation tool and Help File Builder (SHFB) , inside <https://github.com/EWSoftware>
  This powerful and popular tool even adapted by Robert a few years ago to the needs of Xsharp both for the construction of the online help that is distracted by the application and to feed the online help on the web
  The source code for these plugins can be found in our Github repo: <https://github.com/X-Sharp/XSharpPublic ... Tools/SHFB>
• Inno Setup 5: A free and popular installer for Windows programs https://jrsoftware.org (by Jordan Russell) The version used is 5, the last one is 6.2.
• I leave the list open, surely there is some component that I do not know enough to understand if it is more or less important. For me, it is important to finish understanding the resolution of the powerful dynamic code so exploited by FoxPro programmers because FoxPro is also an interpreter. I searched X# Online Help on Scripting and I need to understand: How does all this happen thinking of Roslyn+ANTLR+Xsharp.Team.Magic?

VFP programmers never need anything other than FoxPro to develop systems of any size, so we were forced to learn Dot Net quite some time ago which is not little but the XSharp project forces us to overcome several challenges:

• Recognize the other dialects because it is essential. All dialects are important, it is clear that some are practically finished assimilating by XSharp and others like FoxPro are halfway there. It is much more difficult simply because, unlike the others, it became a giant, it grew too much.
• To know worthily the components that I mention to be able to understand the challenge of XSharp, to be able to put yourself in Robert's shoes and try to actively accompany him with his team in the feat to cover us all, particularly FoxPro. This doesn't have to scare anyone who just wants to simply program in XSharp as we do in FoxPro. But I must be honest, some of us have and must accept the challenge in order to collaborate.

Now I wonder and this is the concrete purpose of this publication:
A. Is there anyone who has traveled the same path I am traveling and documenting the experience?
B. Is it possible to shorten our learning curve with community guidance?
C. How many people we are? I'm trying to acknowledge the FoxPro community, looking for a feedback.
D. Is there a way to do step-by-step execution of language creation?
E. How to debug Roslyn Analyzer project? I could not find within XSharpDev any project template like Analizer*vsix.
F. How to debug ANTLR component?

Best regards
Juan

[wriedmann]

Hi Juan,
thank you very much for sharing!
I have to admit that even with my programming experience I'm far, far away from being able to understand how the compiler works.
But the work on the compiler is not done only by Robert alone, AFAIK most work on the compiler and on the macrocompiler is done by Nikos Kokkalis, a greek guy (and IMHO also a programming genius like Robert).
IMHO there is a lot of work to do for Fox people not on the compiler, but on the runtime libraries....
Wolfgang
P.S. It may interest you: nearly all of my new X# development is done using the Core dialect, practically a C# with Xbase syntax

[robert]

Wolfgang, Juan.
First I would like to thank Wolfgang for stepping in. Indeed Nikos has done a LOT of the work in the compiler. The first release of X# in Januari 2016 was 100% his work, as well as almost everything in the macro compiler.
Since then I have also started to work on the compiler, mostly to implement the various Xbase dialects with their "quirks" and to add new C# features to the language, by inventing a syntax, adding keywords to the lexer definition, adding a statement rule to to the parser definition and by transforming the resulting X# parse tree into a C# parse tree.
But when things get really complicated I have to ask Nikos for help. For example to find an efficient way to implement parameters by reference when using the Clipper calling convention. Nikos implemented most of that.
One of the things Nikos also did was replacing the generated Lexer (we used to use a Antlr generated lexer) with a handwritten lexer, which dramatically improved the performance of the lexer.
We use that same lexer inside Visual Studio to find tokens and to colorize them.
The VS integration contains a DLL called XSharp.VSParser that contains the lexer and parser and you can use that to parse your own source code as well. I know of at least one customer that is doing that.

I have no idea to be honest how many people understand how we are building the compiler. Inside our team that's Nikos and me.
What we did do is to create a GitHub action that builds the compiler source every night into a set of binaries, as well as when a pull request is merged into the main branch.

I also want to give credit to the other 2 team members. Chris is mostly doing support, and almost all of the compiler tests and runtime tests are written by him. He is our QA guy. No build gets released until Chris thinks it is ready. Chris also worked on several parts of the runtime, the VOXporter, the VO compatible editors inside VS (even if he does not like VS) and more.

And Fabrice has worked on many things. He did our first VS integration and is still adding features to our VS integration such as the Lightbulb tooltips. Fabrice has also written several parts of the runtime (Mostly in the RDD system), he is working in our FoxPro Exporter and he is doing R&D for our Xanthi project (Client Server XBase RDD).
In other words: this is a team effort. X# would have never been a success if it was a one man project.

W.r.t. analyzers: X# also supports writing analyzers and running them at compile time. There are some samples on the web that show how to do this for C#.
If you want to do this for X# then you should realize that you do not need to add a reference to the Microsoft.Codeanalysis assemblies but to the XSharp.CodeAnalysis assembly that comes with our compiler.
Also some of the namespace have been changed by us to avoid namespace conflicts if you would want to create a project that references both our compiler DLL and the C# compiler DLL.

If you guys think it is interesting we could do one or more sessions about this (and maybe show how to create and use an analyzer).
I do not think there will be a big audience for this but it might still be an interesting.

Robert
From the "FabToys" wifi network in the house of Fabrice.

[wriedmann]

Hi Robert,
I have only mentioned Nikos because Juan mentioned the compiler.
Of course, I know also the invaluable work that Chris is doing (he has helped me a lot of times, not speaking about XIDE - my interface to X#, and the piece of software I use most of my time), and I also know the important work of Fabrice on the VS integration and on the VFP tools.
Thank you to your "gang of four" for your great work on the X# project!
Wolfgang

[VR]

Hi Robert,

a Session on how to build an X# analyzer or an example project with a simple analyzers could be interesting. Especially if it also shows how to implement a simple light bulb code fix (if i recall, this is connected to analyzers in VS).

The two interns, that last year created the VSCode Plugin and the X# Power Tools, will come back this summer. Creating some x# analyzers could be an interesting task and challenge for them.

[robert]

Volkmar,
we are not running analyzers or the Roslyn code inside the editor. We have tried this but found that too slow, especially with incomplete code. We have therefore written a special parser to find the various classes, methods etc, that is much more forgiving than the compiler parser.

So running an analyzer in the editor will not work (for now at least).
But you can run an analyzer in the compiler and check for lots of things, such as naming conventions, check for code that uses the clipper calling convention etc.
I will prepare an example on how to do this and will schedule an online session about this.

Robert

[VR]

Thanks. It will be interesting to see, how this works. Related question. Will the SourceGenerators (which are something similar to Analyzers, that run at build time) also work with x#?

[robert]

Volkmar,
Yes, they should.
However Meinhard detected a problem with them that we're looking into.

Robert