Manually deploy Retail packages for Microsoft Dynamics 365 for Finance and Operations

First Microsoft Dynamics 365 for Finance and Operations Retail post! I hope more will come.

As you might know, one of the setbacks of the database refresh from production in LCS is that some data doesn’t get copied. This is a safety feature that prevents, among others, that emails are sent or batches run accidentally after a DB restore.

Continue reading “Manually deploy Retail packages for Microsoft Dynamics 365 for Finance and Operations”

Parse XML and JSON easily in MSDyn365FO

Some time ago I had to create an interface between MSDyn365FO and a web service that returned data as XML. I decided to use X++’s XML classes (XmlDocument,  XmlNodeList, XmlElement, etc…) to parse the XML and get the data. These classes are terrible. You get the job done but in an ugly way. There’s a better method to quickly parse XML or JSON in MSDyn365FO.

Continue reading “Parse XML and JSON easily in MSDyn365FO”

Feature management: create a custom feature

Feature management has been around in Microsoft Dynamics 365 for Finance and Operations for some time now. Before that features were enabled through flighting running a SQL query on dev and UAT boxes (and the DSE team would do it on production).

Now we have a nice workspace showing all the available features and flighting is still around too. The main difference between flighting and features is that flighting is enabled to a selected group of customers, like a preview of a feature.

Continue reading “Feature management: create a custom feature”

Set up the new Azure DevOps tasks for Packaging and Model Versioning

You can read my complete guide on Microsoft Dynamics 365 for Finance & Operations and Azure DevOps.

During this past night (at least it was night for me :P) the new tasks for Azure DevOps to deploy the packages and update model versions have been published:

There’s an announcement in the Community blogs too with extended details on setting it up. Let´s see the new tasks and how to set them up.

Continue reading “Set up the new Azure DevOps tasks for Packaging and Model Versioning”

Application Checker: enforcing better coding practices?

Unless you’ve been working for an ISV there’s a high percentage of probabilities that you’ve never cared about Dynamics Best Practices (BP), or maybe you have. I haven’t worked for an ISV myself but back when I started working with AX I was handed the development BP document and I’ve tried to follow most of them when writing code.

But BPs could be ignored and not implemented without any issue. This is why Microsoft will publish…

Application Checker

Application Checker is a tool that will change that. It will force some rules that our code will have to meet, otherwise the code won’t compile (and maybe won’t even deploy to the environments).

We got an advance of it during last MBAS session “X++ programming with quality” by Dave Froslie and Peter Villadsen. Unfortunately the session wasn’t recorded.

Application Checker: enforcing better coding practices? 2

Application Checker: enforcing better coding practices? 3

App checker is built on BaseX, an XML analysis tool, and powers Socratex which Microsoft uses to track code quality. I don’t know if Socratex will be publicly released and I don’t remember if this was clarified during the session.

The set of rules can be found in Application Checker’s GitHub project and it’s still WIP. I think there’s loooots of things to decide before this goes GA, and I’m a bit worried and afraid of some of the rules 😛

Rule types

There’s different types of rules, some will become errors and other warnings. For example:

ExtensionsWithoutPrefix.xq: this rule will throw an error avoiding your code to compile. It checks if the extension class has a name ending with _Extension and an attribute ExtensionOf. If it has it must have a prefix. E.g.: if we extend the class CustPostInvoice it can’t be named CustPostInvoice_Extension, it needs a prefix like CustPostInvoiceAAS_Extension.

Application Checker: enforcing better coding practices? 4

SelectForUpdateAbsent.xq: this rule will throw a warning. When there’s a forUpdate clause in a select statement and no doUpdate, update, delete, doDelete or write is called later it will let us know.

Application Checker: enforcing better coding practices? 5

As of today, there’s 21 rules in the GitHub project. You can contribute to the project, and you could enforce your own rules without sending them to the project on your dev boxes, just add them to the local rules folder. I’d create a rule that makes the space after an if/while/for/switch mandatory and throws an error otherwise, but that’s only a bit of my OCD when writing/reading code.

Try it on your code

We can already use Application Checker on our development environments since PU26, I think. We just need to install JRE and BaseX in the dev box and select the check when doing a full build.

Application Checker: enforcing better coding practices? 6

Some examples

ComplexityIndentationCombined.xq

This query checks the (wait for it…) cyclomatic complexity of the methods. I’ll try to explain it… Cyclomatic complexity is a metric for software quality, and is the number of independent paths in the code. Depending on the number of ifs, whiles, sitches, etc… the code can have different outcomes through different paths, that’s what complexity calculates.

Taking this as an example, a dumb one but ignore it, just look at the amount of different paths that could happen:

In App checker the error appears when the complexity is over 30. I’ve used Lizard code complexity analyzer to calculate the complexity of the method below and I’m getting a 49.

The rule also checks for the indentation depth, failing if it’s greater than 2. In the end the purpose of both rules is to try to cut up long/large methods, which will also help in enabling more extension points in different places of our logic, like Microsoft did with Data Provider classes for reports.

Application Checker: enforcing better coding practices? 7

BalancedTtsStatement.xq

This one gives me mixed feelings. The rule checks that the ttsbegin and the ttscommit of a method are in the same scope. So the following is not possible:

Application Checker: enforcing better coding practices? 8

Imagine you’ve developed an integration with an external application that writes data to an intermediate table in MSDyn365FO and you process all pending data sequentially. You don’t want to throw an error if something goes wrong because you need the process to continue with the following record, so you ttsabort the wrong line, store the error and continue. If this is not possible… how should we do this? Create a batch that creates a task for each line to process?

Plus, the standard models have plenty of ttscommit inside if statements.

RecursiveMethods.xq

Application Checker: enforcing better coding practices? 9

This rule will block the use of recursion on static methods. I don’t get why. Application checker should be a way to better coding practices, not forbidding some patterns. If somebody gets a recursive method to prod and the exit condition isn’t met… hello testing?

Some final thoughts

Will this force developers to code better? I don’t think so, but that’s probably not Application checker’s purpose. For centuries humans have found ways to bypass rules, laws and all kinds of restrictions and this won’t be an exception.

Will it help? Hell yes! But the best way to ensure code quality is promoting the best practices in your team, through internal trainings or code reviews. And even then if someone doesn’t care about clean code will keep on writing terrible code, which might work but won’t be beautiful at all.

Finally, I’m not sure about some rules, like avoiding recursion on static methods or the tts thing. We’ll just have to wait and see which rules make it to the final release and how will Application checker be finally implemented in the MSDyn365FO application lifecycle by blocking (or not) the deployments of code which doesn’t pass all the checks or if it will be included into the build process.

Self-service deployments: the future is here

Right now Microsoft Dynamics 365 for Finance and Operations has an old style monolithic architecture, even it’s now in Azure’s cloud, what we really have is a single (or multiple for Tier 2+ environments) VM that runs everything: the AOS/IIS, Azure SQL Server, the Batch service, MR, etc. Exactly the same as AX 2009/2012.

This is going to change in the coming months with the self-service deployments. We’ll move from the monolithic architecture to microservices that will run all the needed components with the help of Azure’s Service Fabric. MSDyn365FO will be on a real SAAS model.

Before starting let me clarify that all these changes will only apply to Microsoft-managed Tier 2+ environments: sandbox and production environments. The build environment (until it’s made obsolete) and the cloud-hosted environments on the customer or partner subscription will still be single VMs.

What’s new?

Faster deployments

When you deploy a new environment it will start deploying without waiting for Microsoft to do it (it’s self-service!). Additionally, thanks to the new microservices architecture, it will be ready to use in under 30 minutes compared to 6-8 hours of regular environments. The first time feels like…

Self-service deployments: the future is here 10

Subscription estimator

We still need to fill out the subscription estimator for licensing purposes and for MS to estimate the size of the production environment. The self-service environments can be escalated more flexibly and quickly.

No RDP access

The access to the VM desktop has been removed because… well, I guess it’s because there’s no VM anymore. All the operations that could need us to access the RDP can be done from LCS.

No SQL Server access

Yes, no RDP access means no RDP access to the SQL box either. We still have access to the Azure SQL DB, we just need to ask for it from LCS and it’s granted in seconds:

Self-service deployments: the future is here 11

Additionally you must whitelist your IP (or the one you’ll access SQL from) from the Maintain – Enable access button on LCS to be able to connect to the Azure SQL Server. The access to the DB and the firewall rule will be enabled for 8 hours.

As usual, there’s no access to the production DB.

One deployable package to rule them all

If you’ve recently tried to deploy a deployable package (DP) without all the packages the environment has (basically generating the DP for a single model/package from Visual Studio) you must’ve noticed the warning about the difference in the packages from the DP and the environment.

With the self-service deployments you must include all models/packages AND!! ISVs in one single deployable package.

Production updates

First, we can start the deployment to production without the 5 hours in notice we need to schedule now. We still can schedule the deployment but we can also start it instantly.

Next, the way the production environment is updated changes a bit from what we’re used to. With the new deployments we will update the sandbox environment as we do now, once it’s done we’ll select a sandbox environment to be promoted into production. This is probably another benefit of the architecture changes.

In the future the deployment downtime will also be reduced to zero for the service updates as long as you’re on the latest update. This won’t be available for custom DPs.

How do I get this?

At the moment this is only available for some new customers. Current customers will be migrated during the coming months, MS will contact the customers to schedule a maintenance window to apply the changes.

For more information check the session Microsoft Dynamics 365 for Finance and Operations: Strategic Lifecycle Services Investments from last June’s MBAS.

Our experience with it

We got into the private deployment preview program almost a year ago with one of Axazure’s customers. The customer is now live with the self-service environments and everything has been fine so far.

But the beginning was a bit hard. Some of the functionalities were still not available at the moment, like DB refresh or… package deployment. Yes, we needed to ask MS to deploy our DPs each time. We couldn’t even put the environments in maintenance mode! In the first months of 2019 a lot of functionality was added to LCS and in June we finally got the production self-service update functionality available. The help we’ve gotten from Microsoft’s product team has been very valuable and they have unlocked some issues that were stopping the progress of the project.

Slow set-based operations?

In Microsoft Dynamics 365 for Finance and Operations we can execute the CRUD operations from code in two different ways, record-per-record or set-based.

Microsoft’s recommendation is to always use set-based operations, if possible, as you can check on the Implementation Best Practices for Dynamics 365: Performance best practices for a successful Dynamics 365 Finance and Operations implementation session from last June’s Business Applications Summit.

Why?

Set-based Vs. Record-per-record

When we run a query in MSDyn365FO we’re using its data access layer which will later be translated into real SQL. We can see the differences using xRecord’s getSQLStatement with generateonly on the query (and forceliterals to show the parameter’s values) to get the SQL query. For example if we run the following code:

Slow set-based operations? 12

We’ll get this SQL statement:

 

We can see all the fields are being selected, and the where clause contains the account number we selected (plus DataAreaId and Partition).

When a while select is run on MSDyn365FO a select SQL statement is executed on SQL Server for each loop of the while. The same happens if an update or delete is executed inside the loop. This is know as record-per-record operation.

Imagine you need to update all the customers with the customer group 10 to update their note. We could do this with a while select, like this:

Slow set-based operations? 13

This would make as many calls as customers from the group 10 existed to SQL Server, one for each loop. Or we could use set-based operations:

Slow set-based operations? 14

This will execute a single SQL statement on SQL Server that will update all the customers with the customer group 10 instead of a query for each customer:

There’s three set-based operations in MSDyn365FO, update_recordset to update records, insert_recordset to create records and delete_from to delete the records. Plus we can make massive inserts using RecordSortedList and RecordInsertList.

Running this methods instead of while selects should obviously be faster as it’s only executing a single SQL query. But…

Why could my set-based operations be running slow?

There’s some well-documented scenarios in which set-based operations fall back to record-per-record operations as we can see in the following table:

DELETE_FROM UPDATE_RECORDSET INSERT_RECORDSET ARRAY_INSERT Use … to override
Non-SQL tables Yes Yes Yes Yes Not applicable
Delete actions Yes No No No skipDeleteActions
Database log enabled Yes Yes Yes No skipDatabaseLog
Overridden method Yes Yes Yes Yes skipDataMethods
Alerts set up for table Yes Yes Yes No skipEvents
ValidTimeStateFieldType property not equal to None on a table Yes Yes Yes Yes Not applicable

In the example, if the update method of the CustTable is overriden (which it is) the operation from the update_recordset piece will be run like a while select that updates each record.

In the case of the update_recordset this can be solved calling the skipDataMethods method before running the update:

Slow set-based operations? 15

This will avoid calling the update method (or insert in case of the insert_recordset), more or less like calling doUpdate in a loop. The rest of the methods can be overriden with the corresponding method on the last column.

So, for bulk updates I’d always use set-based operations and enable this on data entities too with the EnableSetBasedSqlOperations property.

And now another but is coming.

Should I always use set-based operations when updating large sets of data?

Well it depends on which data you’re working with. There’s a wonderful blog post from Denis Trunin called “Blocking in D365FO(and why you shouldn’t always follow MS recommendations)” that shows a perfect example where set-based operations would be counterproductive.

As always, developing an ERP is quite sensitive, and similar scenarios can have different solutions. Analyze the requirements and decide which one to use.

Update to Visual Studio 2019 for #MSDyn365FO

Tired of developing in Visual Studio 2015? You feel you’ve been left and forgotten in the past? Worry no more, you can use Visual Studio 2017/2019 to develop Microsoft Dynamics 365 for Finance & Operations!

What are the advantages?

Absolutely none at all! Visual Studio will still go non-responding whatever the version is because it’s the dev tools extension what’s causing the issues.

Of course we get the option to use Live Share, and for screen sharing sessions that’s way better than teams. Hey, and we’ll be using the latest VS version!

Is it difficult?

No, zero mysteries. The first thing we need to do is downloading Visual Studio 2019 Professional (or Enterprise but it won’t make such a difference for D365 development) and install it:

Update to Visual Studio 2019 for #MSDyn365FO 16

Select the .NET desktop development option and press install. When the installation is finished we log in with our account.

The next step is installing the Dynamics developer tools extension for VS. Go to drive K and in the DeployablePackages you’ll find some ZIP files that have the extension in the DevToolsService/Scripts folder:

Update to Visual Studio 2019 for #MSDyn365FO 17

Update to Visual Studio 2019 for #MSDyn365FO 18

An alternative is, for example, downloading a Platform Update package which also has the dev tools extensions, and maybe with some update to them.

Install the extension and the VS2019 option is already there:

Update to Visual Studio 2019 for #MSDyn365FO 19

Once installed open VS as the admin and…

Update to Visual Studio 2019 for #MSDyn365FO 20

And also…

Update to Visual Studio 2019 for #MSDyn365FO 21

Don’t panic! The extension was made for VS2015 and using it in a newer version can cause some warnings, but it’s just that, the tools are installed and ready to use:

Update to Visual Studio 2019 for #MSDyn365FO 22Update to Visual Studio 2019 for #MSDyn365FO 23

As I said in the beginning, the dev tools extension is the one causing the unresponsiveness or blocks in VS, and Visual Studio 2019 is letting us know:

Update to Visual Studio 2019 for #MSDyn365FO 24Update to Visual Studio 2019 for #MSDyn365FO 25

But regardless of the warnings working with Visual Studio 2019 is possible. I’ve been doing so for a week and I still haven’t found a blocking issue that makes me go back to VS2015.

Update: it looks like opening a report design will only display its XML instead of the designer. Thanks to David Murray for warning me about it!

Dev tools preview

In October 2019 the dev tools’ preview version will be published, as we could see in the MBAS in Atlanta. Let’s see which new features this will bring us both in a possible VS version upgrade or performance.