SocrateX & Application checker

It’s been a while since I first wrote about the Application Checker in 2019, and here I am again. In this blog post, I’ll talk about SocrateX and XQuery too, and I’ll also show how to generate the files and databases used to analyze the code.

If you want to know more about App Checker or SocrateX, you can read these resources in addition to the post I’ve linked above:

• AppChecker on Github
• Peter Villadsen’s blog posts: Part 1 and Part 2
• SocrateX on GitHub

What’s new?

What has changed that has prompted me to write about it again? Two things:

First of all, Peter Villadsen has started blogging at Peter’s X++ developer Blog, and he’s started with two blog posts about SocrateX.

These two blog posts are the source and inspiration of what I’m writing here, you can read many of the things I’m explaining here in his two blog posts, so I recommend that you go to his blog and read it.

The second reason is that the build options were updated in version 10.0.17, and now we can run App checker rules without having to install BaseX:

There are three new options at the bottom:

• “Run Socratex Checks“: if we want to use this option, we still need to install BaseX. This option will generate the ASTs (we’ll see later) and create a BaseX database for the selected package/models. This will also run the App Checker rules.
• “Run Appchecker Rules“: this option will run the checks of the queries contained in the K:\AosService\PackagesLocalDirectory\bin\AppCheckerRules\X++ folder. If you add your queries to that folder, they’ll run too. This option won’t generate any ASTs.
• “Log Appchecker diagnostics as errors“: the previous option shows only warnings, but if we select this one too, Appchecker will convert the warnings into errors, and our build will fail. Even though this option can be selected alone, it won’t run the Appchecker Rules, and you need to select both to run the rules and show them as errors.

What’s SocrateX?

SocrateX is a tool developed by Microsoft that helps us understand and analyze our X++ code which is stored in a BaseX XML database.

Socratex: A tool for reasoning over source code.

This tool allows you to reason over source code as if it were data stored in a database. It works by submitting queries in the XQuery language towards a server where a dedicated XML database handles the queries against a repository of XML documents that describe the source code corpus in a specific language.

https://github.com/microsoft/Dynamics365FO-AppChecker/tree/master/tools

When we tell the compiler to generate the Abstract Syntax Trees (ASTs) from our code… wait, what are Abstract Syntax Trees? It’s a way to represent the structure of our code, in this case, translated into XML. This XML will be stored in a BaseX database to navigate using XQuery (more on this later).

And what’s XQuery?

XQuery is a language that we’ll use to navigate the generated ASTs in our BaseX databases. All the AppChecker queries used to check our code are done using it, and if we open one of the queries, for example, the ExtensionsWithoutPrefix.xq one, we see this:

<Diagnostics Category='Mandatory' href='docs.microsoft.com/Socratex/ExtensionsWithoutPrefix' Version='1.0'>
{
  for $a in /Class
  where $a/AttributeList/Attribute[@Name = "ExtensionOf"]
  let $targetClass := $a/AttributeList/Attribute[@Name = "ExtensionOf"]/AttributeExpression/IntrinsicAttributeLiteral[@FunctionName = "classStr"]/string(@Arg1)
  where $a/@Name = fn:concat($targetClass, "_Extension")
  return
    <Diagnostic>
      <Moniker>ExtensionsWithoutPrefix</Moniker>
      <Severity>Error</Severity>
      <Path>{string($a/@PathPrefix)}</Path>
      <Message>Extension class name should include a prefix</Message>
      <DiagnosticType>AppChecker</DiagnosticType>
      <Line>{string($a/@StartLine)}</Line>
      <Column>{string($a/@StartCol)}</Column>
      <EndLine>{string($a/@EndLine)}</EndLine>
      <EndColumn>{string($a/@EndCol)}</EndColumn>
    </Diagnostic>
}
</Diagnostics>

I’m not used to working with XQuery at all, I’ve just tried to create some rules, but I’ll try to explain what this one does.

The first line, “for $a in /Class”, is getting the classes in the variable $a, and the second line, “where $a/AttributeList/Attribute[@Name = “ExtensionOf”]” is filtering on the Classes node to only select the one that are decorated with the [ExtensionOf()] attribute.

The third line, starting with the keyword let, is used to store in the variable $targetClass the value of the root class that we’re extending, contained in the IntrinsicAttributeLiteral node and the attribute FunctionName, and that is extending a class, hence the [@FunctionName = “classStr”] filter.

The fourth line filters the first variable, $a, which contains all our classes, whose name is like the base class found in the third line, concatenated with _Extension.

And finally, we’re returning those classes decorated with AttributeOf that are equal to the ExtendedClassName_Extension pattern, without any prefix at the beginning, or suffix at the end.

For example, suppose I’m extending the CustPostInvoice class, and I create an extension class named CustPostInvoice_Extension, and decorate it with the ExtensionOf attribute, it will appear in the warning list because I’m using the base class name without anything else. It should be named like CustPostInvoiceAAS_Extension.

Creating a query

So thanks to XQuery you can generate your own rules. For example, continuing with the extensions, I’m creating a query that will show me all the classes that end with _Extension, but don’t have the ExtensionOf attribute:

for $c in /Class
let $className := $c/@FullName
where matches($className, '_Extension$')
    and not($c//AttributeList//Attribute/@Name = "ExtensionOf")    
return $className

It’s a stupid example, but it’s an example. We get the class nodes into $c, then get the name of the class into $className and filter the classes to show the ones that end in “_Extension” but aren’t decorated with the ExtensionOf attribute.

If you need more examples, you can check the GitHub project where there’s a Samples folder with far better examples than what I’ve done.

If you want to learn more about XQuery you can check the W3C tutorial or, as Peter Villadsen recommends, Priscilla Walmsley’s book XQuery (ISBN-13: 978-0596006341, ISBN-10: 0596006349).

Some XQuery syntax

As we’ve seen above, the XQuery syntax resembles a bit that of T-SQL. I’ve used the for, let, where and return keywords, which along the order by make the five basic keywords.

• for: like in for $c in /Class (or /Table or /Form), will loop through all the /Class (or /Table or /Form). Then we can query the elements of classes using $c instead of /Class.
• let: this keyword is used to save the value of an element, attribute, etc. in a variable. In the example above it’s getting the FullName attribute for each element in $c, and for each value of the $c variable read in the loop, we will get the value of that element in the $className variable.
• order by: same as SQL, it will order data depending on the condition we use.
• where: as in SQL it’s used to filter data.
• return: will return data. In the example above it returns the $className variable, which contains the FullName attribute.

This keywords are known as FLOWR (flower).

Variables are declared using the dollar ($) sign as you might’ve noticed.

You can access the attributes of an XML element using the NodeName/@Attribute pattern. And you can access subnodes using the NodeName/ChildNode, as many times as nodes there is.

There a bunch more things you can do, if you check the examples you can learn more about the syntax, I’m just explaining the simpliest of all things we can do with XQuery.

Generate ASTs

Now we need to generate the ASTs to run queries on our code!

How do we generate the ASTs? There are several ways of doing this, and all of them require BaseX to be installed because all the options will generate the ASTs and generate a BaseX database.

First option: you can do a full build and select the “Run Socratex Checks” options:

As we’ve seen before, this option will generate the ASTs and create a BaseX database.

Second option: in Visual Studio go to the Dynamics 365 menu, Deploy and Create Deployment Package. In the new dialog, select which packages you want to generate the ASTs for and select the “Run Appchecker” option.

This option will also generate the ASTs and create the BaseX database.

Third option: as Peter explains in his blog posts (please, if you still are reading this and haven’t read his blog posts, go and do it), the last option is using the xppc.exe (the compiler) from the command line. xppc.exe is located in K:\AosService\PackagesLocalDirectory\bin folder, or C if you’re in a VHD VM. Call it like this:

xppc.exe  -metadata="K:\AosService\PackagesLocalDirectory" -compilermetadata="K:\AosService\PackagesLocalDirectory" -output="K:\AosService\PackagesLocalDirectory\YOUR_PACKAGE\bin" -modelmodule="YOUR_PACKAGE" -xmllog="K:\AosService\PackagesLocalDirectory\YOUR_PACKAGE\BuildModelResult.xml" -log="K:\AosService\PackagesLocalDirectory\YOUR_PACKAGE\BuildModelResult.log" -appBase="K:\AosService\PackagesLocalDirectory\bin" -refPath="K:\AosService\PackagesLocalDirectory\YOUR_PACKAGE\bin" -referenceFolder="K:\AosService\PackagesLocalDirectory" -writeAsts -astOutputPath="K:\YOUR_PACKAGE"

You need to change the YOUR_PACKAGE parts for your package name.

This command will only generate the ASTs. To create the database we need to run another command:

java.exe -cp 'C:\Program Files (x86)\BaseX\BaseX.jar' org.basex.BaseX -c "CREATE DB YOUR_DB K:\YOUR_PACKAGE"

If you’ve installed BaseX in a different location, change the .jar route.

Why use the command line? Because we can, and because I asked Mötz Jensen if somebody had tried to add this to d365fo.tools, and then offered to do it 😅 And I will!

If you go to the folder you’ve selected to save the ASTs you’ll see folders for Classes, Entities, Forms, or Tables, but not for extensions of objects. I guess that it is this way because Form/Table extensions only have pieces of code or fields (what we extend), and it’s not possible to analyze it.

Use SocrateX explorer

Once we have the ASTs generated and in the BaseX database, we can start using SocrateX (check Peter’s posts to download it). Run SocrateX, and use the same password as the username to log in:

If you get the error “Unable to connect, or bad credentials provided.” it’s because BaseX server is not running, go to your start menu and start it:

When SocrateX starts, we can see three windows. The top one shows the source code, the bottom left one is where we write the queries, and the bottom right one is where we see the queries’ results.

And we are done. We have a way of analyzing our code that may help us write better code and follow not only X++ best practices but coding best practices. Go and try running the queries against your codebase and see what you can make better!