Setup Entity Store’s export to Azure Data Lake storage

It’s easy to start this post, because many people can ask:

What’s a Data Lake?

Fishing in a Data Lake. By cazapelusas.

A Data Lake is not an Azure product but a term referring to a place where data is stored, regardless of whether it’s structured or unstructured. Its only purpose is storing the data ready to be consumed by other systems. It’s like a lake that stores the water of its tributaries, but instead of water with data.

In Azure the Data Lake is a Blob storage which holds the data. And this data can come from Microsoft Dynamics 365 for Finance or Supply Chain Management (I’ll go crazy with the name changes of Axapta 7) or from other sources.

Currently, and since PU23, #MSDyn365FO (#MSDyn365F ? or #MSDyn365SCM ?) officially supports exporting the Entity Store to Azure Data Lake storage Gen1, but compatibility with Data Lake Storage Gen2 is on the works in a private program with Data Feeds that will allow us to export entities and tables (YES!) in near real time. If you want to know more check the Data Management, Data Entities, OData and Integrations Yammer group in the Insider Program (if you still haven’t joined, you should).

Comparison vs. BYOD

The first thing we must notice is the price. Storage is cheaper than a database, even if it’s a single SaaS DB on Azure SQL. For example, a 1GB Blob storage account on Azure costs $21.6/month.

And the simplest Gen 4 with 1 vCore Azure SQL database costs $190.36/month. Almost 10 times more.

And what about performance? This comes from observation, not a real performance test, but data is transferred real fast. And it’s fast because in a Data Lake data is sent raw, there’s no data transformation until it’s consumed (ETL for a DB, ELT for a Data Lake) so there’s less time spent until data reaches its destiny. This doesn’t have a real impact for small sets of data but it does for large ones.

Setup

The process to export the Entity Store to a Data Lake is pretty simple and it’s well documented (but not updated) on the docs. I’ll explain step by step.

Create a storage account on Azure

On Azure go to or search in the top bar for Storage accounts and add a new one with a setup like the one in the pics below:

Make sure to disable Gen2 storage:

And you can go to review & create. When the account is ready go to Access Keys and copy the connection string:

Azure Key Vault

The next step is creating a Key Vault. For this step you need to select the same region as your Dynamics 365 instance:

When the Key Vault is ready go to the resource and create a new secret. Paste the connection string from the storage account into the value and press create:

Create an AAD App Registration

Now we’ll create an AAD App. Give it a name, select the supported account types you need and fill the URL with the base URL of your #MSDyn365FO instance:

Click register and now we must add the Azure Key Vault API to the app as in the image below:

Select the API and add the delegated user_impersonation permission:

Don’t forget to press the button you can see above to grant privileges (must be done by an Azure admin). Now go to secrets and create a new one, give it a name and copy the secret value. When you close the tab you will not be able to recover that secret anymore so copy it and save it somewhere until we need it.

Setup the Key Vault

Go back to the Key Vault we created in the second step and go to Access policies. Add a new one:

You have to select Get and List for Key and Secret permissions:

Now press Select principal and here add the AAD App created in the third step:

Add it and don’t forget to save in the access policies screen!!

Set up MSDyn365F… and O or and SCM or whatever its name is this month

Navigate to System Administration -> Setup -> System parameters and go to the Data Connections tab. Here there’s 4 fields for the key vault. The Application ID field corresponds to the Application ID of the AAD App (pretty obvious) and the Application Secret is the secret from the AAD App. This part is easy and clear.

The DNS name is the url on your Key Vault and the Secret name field is the name of your Key Vault’s secret where you pasted the storage account connection string.

Once all these fields are complete you can press Test Azure Key Vault and Test Azure Storage and, if you followed all steps correctly, you should see the following messages:

If any of the validations don’t succeed I’d just delete all resources and start from scratch, probably a secret mismatch.
Now, the two buttons you see next to the setup fields:
  • Enable Data Lake integration: will enable the full push of the entity store to the storage account you have just created and which is the main purpose of this post.
  • Trickle update Data Lake: will make updates after data is changed (Trickle Feed).

Setup Entity Store

Now we just need to go to the Entity Store (under System Administration -> Setup -> Entity Store) and enable the refresh of the entities we’d like to hydrate the Data Lake (I love this, it looks like it’s the correct technical word to use when feeding the Data Lake):

And done, our data is now being pushed to an Azure Blob:

The entities are saved each in a folder, and inside each folder there another folder for each measure of that entity and a CSV file with the data in it.

Now this can be consumed in Power BI with the blob connector, or feed Azure Data Factory or whatever you can think about, because that’s the purpose of the Data Lake.

 

Manually deploy Retail packages for Microsoft Dynamics 365 for Finance and Operations

First Microsoft Dynamics 365 for Finance and Operations Retail post! I hope more will come.

As you might know, one of the setbacks of the database refresh from production in LCS is that some data doesn’t get copied. This is a safety feature that prevents, among others, that emails are sent or batches run accidentally after a DB restore.

Remember that it’s a good idea to have a SQL query/script that changes all endpoints, passwords, enables users, etc. that you can run after a prod DB refresh, just like it was done with AX2009/2012. Just F5 it in SSMS and the environment will be ready to use and to export to your dev boxes.

Another thing that doesn’t get moved after a DB refresh are storage specific files, ER XLSX, DocuValue files and the self-service Retail installers.

Retail packages

Retail packages are the executable files used to install MPOS on the… well, on the points of sale (POS). These files are stored in an Azure blob storage which is specific to each environment, so after the DB refresh there’s no self service packages in the target environment because the reference was to the production blob:

Microsoft’s official fix for this is applying a binary package that will recreate the EXE files in the VM’s storage where the Deployable Package is run. And as you all know this is time consuming and while you can run it outside working hours you can fix it in less than 10 minutes.

The workaround

Ahhh “workaround”… it’s such a beautiful word with so many different meanings… And this workaround has a restriction: it only applies to dev boxes and Tier 2+ regular environments, this can’t be done on self-service environments as we don’t have access to the AOS VM.

What we need to do is log into the AOS VM using the RDP and go to the service volume (usually K on dev, G on Tier 2+). There should be a folder called DeployablePackages, if you have applied any, otherwise just go with the official fix. However, if the folder doesn’t exist this can probably be done in another way, which is using the files from the install drive, but I haven’t tried.

Sort the files by date modified (newer first) and inside the first folder you should see another folder called RetailSelfService:

And inside this folder you’ll see 3 more folders, Packages, Scripts and ServiceModel. Inside the Packages folders there’s the EXE files and inside the Scripts folder the scripts (obviously Mr. Obvious), open it and the open the Upgrade folder and you’ll find a PowerShell script called UpdateRetailSelfService. You need to run this script in PowerShell as an administrator. It will take between 3 and 5 minutes and when it’s done the packages will be uploaded to the environment’s storage and appear in the Retail Parameters form.

That doesn’t work for me!

There’s a case in which the installers will not be restored: if you have no setup done for Retail. Why? The PowerShell script runs a SQL query that will check for the following:

  • Has Channel data in Channel DB
  • Has any Channel data in AOS
  • Has transaction data in AOS
  • Has transaction data in Channel DB
  • Has Channel DB extensions

If none of the above conditions is met the script will not upload the installers to the blob. But we can do something! Yes, you can go and configure a Channel DB for instance. But what if you don’t feel like doing it?

Remember the UpdateRetailSelfService script I talked about before? Edit it and comment the following lines:

This will make the script skip the check and will deploy the installers.

That’s pretty dirty, right? Yes.

What about self-service environments?

I’m sure this can be also done modifying a Deployable Package that contains the Retail packages (the one for a monthly version update), leaving Retail only in the DefaultTopologyData.xml file, and even editing the script if needed. But I haven’t tried. Any volunteers?

Parse XML and JSON easily in MSDyn365FO

Some time ago I had to create an interface between MSDyn365FO and a web service that returned data as XML. I decided to use X++’s XML classes (XmlDocument,  XmlNodeList, XmlElement, etc…) to parse the XML and get the data. These classes are terrible. You get the job done but in an ugly way. There’s a better method to quickly parse XML or JSON in MSDyn365FO.

.NET to the rescue

There’s a feature in Visual Studio that will help us with this but it’s not available in Unified Operations projects. Open Visual Studio and create a new .NET project. Now you just need to copy a sample of the XML text you want to parse, go to the Edit menu, Paste Special, Paste XML As Classes:

And we’ll have a data contract with the needed elements to access all the element nodes using classes and dot notation to access data! For example, for this sample XML file we will get the following:

namespace AASXMLHelper
{

    // NOTE: Generated code may require at least .NET Framework 4.5 or .NET Core/Standard 2.0.
    /// <remarks/>
    [System.SerializableAttribute()]
    [System.ComponentModel.DesignerCategoryAttribute("code")]
    [System.Xml.Serialization.XmlTypeAttribute(AnonymousType = true)]
    [System.Xml.Serialization.XmlRootAttribute(Namespace = "", IsNullable = false)]
    public partial class catalog
    {

        private catalogBook[] bookField;

        /// <remarks/>
        [System.Xml.Serialization.XmlElementAttribute("book")]
        public catalogBook[] book
        {
            get
            {
                return this.bookField;
            }
            set
            {
                this.bookField = value;
            }
        }
    }

    /// <remarks/>
    [System.SerializableAttribute()]
    [System.ComponentModel.DesignerCategoryAttribute("code")]
    [System.Xml.Serialization.XmlTypeAttribute(AnonymousType = true)]
    public partial class catalogBook
    {

        private string authorField;

        private string titleField;

        private string genreField;

        private decimal priceField;

        private System.DateTime publish_dateField;

        private string descriptionField;

        private string idField;

        /// <remarks/>
        public string author
        {
            get
            {
                return this.authorField;
            }
            set
            {
                this.authorField = value;
            }
        }

        /// <remarks/>
        public string title
        {
            get
            {
                return this.titleField;
            }
            set
            {
                this.titleField = value;
            }
        }

        /// <remarks/>
        public string genre
        {
            get
            {
                return this.genreField;
            }
            set
            {
                this.genreField = value;
            }
        }

        /// <remarks/>
        public decimal price
        {
            get
            {
                return this.priceField;
            }
            set
            {
                this.priceField = value;
            }
        }

        /// <remarks/>
        [System.Xml.Serialization.XmlElementAttribute(DataType = "date")]
        public System.DateTime publish_date
        {
            get
            {
                return this.publish_dateField;
            }
            set
            {
                this.publish_dateField = value;
            }
        }

        /// <remarks/>
        public string description
        {
            get
            {
                return this.descriptionField;
            }
            set
            {
                this.descriptionField = value;
            }
        }

        /// <remarks/>
        [System.Xml.Serialization.XmlAttributeAttribute()]
        public string id
        {
            get
            {
                return this.idField;
            }
            set
            {
                this.idField = value;
            }
        }
    }


}

You can create this in a .NET Class library and consume it from Finance and Operations. This is the fastest way to use all the classes and members of the classes. Maybe all this can be implemented as Dynamics 365 FnO classes, but you’d have to create as many classes as different types of nodes exist in the XML. And the original purpose of this was being able to parse an XML file faster. I’d just stick with the .NET library.

All these steps are also valid for a JSON file, copy the sample JSON text, paste special and you’ll get all the classes needed to access the data.

Use it in MSDyn365FO

Once you have your library or you’ve created all the classes in FnO (c’mon don’t do this) add the reference to your project and (following the example above) you just need to do the following:

catalog			catalog		= new catalog();
XmlSerializer	        serializer	= new XmlSerializer(catalog.GetType());
TextReader		sr	        = new StringReader(xmlSample);
        
catalog = serializer.Deserialize(sr);

catalogBook[]   books   = catalog.book;
catalogBook     book    = books.GetValue(0);

Declare a variable of the same type as the main node in the XML file, catalog in the example. Then we will create a new XmlSerializer using our type and create a TextReader from the XML as a string. Finally we need to deserialize the XML and assign the result to the catalog and…

As you can see the data is accessible using dot notation and the classes that were created using the paste special feature.

With the help of tools that are not specific from X++ programming experience we can achieve this, and it is definitely faster than having to parse the XML file using the Xml* classes from Dynamics.

Feature management: create a custom feature

Feature management has been around in Microsoft Dynamics 365 for Finance and Operations for some time now. Before that features were enabled through flighting running a SQL query on dev and UAT boxes (and the DSE team would do it on production).

Now we have a nice workspace showing all the available features and flighting is still around too. The main difference between flighting and features is that flighting is enabled to a selected group of customers, like a preview of a feature.

In each new PU there’s new functionalities added to MsDyn365FO, and in PU30, released recently under the PEAP (Preview Early Access Program), we got new enhancements to the Feature management, this time adding a new property to Menu items and Menus called “Feature class“:

This is not enabled yet, and if you try adding a class to a menu item you’ll get a warning and no functionality.

If you read the docs you’ll see that creating new features is not enabled yet, but if you search in the metadata for feature classes…

Creating a custom feature

We’ll take the TaxSetupValidationFeature class as an example. This class implements the interface IFeatureMetadata, and all feature classes use a Singleton pattern to get the feature instance! (It’s exciting because is the first time I see it being used in MSDyn365FO).

The methods to be implemented include the feature’s name and description, the model and some setup. Just copy all the methods and the member of the class and paste it into one you create.

using Composition = System.ComponentModel.Composition;

[Composition.ExportAttribute(identifierstr(Dynamics.AX.Application.IFeatureMetadataV0))]
public final class AASTestFeature implements IFeatureMetadataV0
{
    private static AASTestFeature instance;


    private void new()
    {
    }

    private static void TypeNew()
    {
        instance = new AASTestFeature();
    }

    [Hookable(false)]
    public FeatureModuleV0 module()
    {
        return FeatureModuleV0::AccountsReceivable;
    }

    /// <summary>
    /// Obtains the singleton object instance.
    /// </summary>
    /// <returns>The <c>AASTestFeature</c> instance.</returns>
    [Hookable(false)]
    public static AASTestFeature instance()
    {
        return AASTestFeature::instance;
    }

    [Hookable(false)]
    public LabelId label()
    {
        return literalStr("Enable Test Feature");
    }

    [Hookable(false)]
    public LabelId summary()
    {
        return literalStr("Enables 'Say hello' button");
    }

    [Hookable(false)]
    public WebSiteURL learnMoreUrl()
    {
        return "";
    }

    [Hookable(false)]
    public boolean isEnabledByDefault()
    {
        return false;
    }

    [Hookable(false)]
    public boolean canDisable()
    {
        return true;
    }

    public static boolean isEnable()
    {
        return FeatureStateProviderV0::isFeatureEnabled(AASTestFeature::instance());
    }

}

Now build your solution and go to the feature management workspace, click the check for updates button and your feature should appear in the list:

Let’s use the feature (in quite a stupid way). Create an extension of a form and on its init method check if the feature is enabled, if it is display a message:

[ExtensionOf(formStr(CustTable))]
final class CustTableAASFeatureDemo_Extension
{
  void init()
    {
        next init();

        if (FeatureStateProviderV0::isFeatureEnabled(AASTestFeature::instance()))
        {
            info('Hello! The feature is enabled!');
        }
    }

}

Before enabling the feature go to the form to check there’s no message is being displayed:

No message there, OK.

Go back to feature management and enable your feature.

Go back to the form (CustTable in my example) and…

There’s the message!

Custom features are working as of PU30, at least on dev boxes, and maybe Tier 2+ sandbox environments too. Don’t try it on a production environment until it’s officially released (but that’s not possible as it’s a PEAP release).

This is just a small test of the classes that are available now, we’ll see new features in PU31 when the Feature Class property will work and, as I read on Twitter:

Set up the new Azure DevOps tasks for Packaging and Model Versioning

During this past night (at least it was night for me :P) the new tasks for Azure DevOps to deploy the packages and update model versions have been published:

There’s an announcement in the Community blogs too with extended details on setting it up. Let´s see the new tasks and how to set them up.

Update Model Version task

This one is the easiest, just add it to your build definition under the current model versioning task, disable the original one and you’re done. If you have any filters in your current task, like excluding any model, you must add the filter in the Descriptor Search Pattern field using Azure DevOps pattern syntax.

Create Deployable Package task

This task will replace the Generate packages from the current build definitions. To set it up we just need to do a pair of changes to the default values:

X++ Tools Path

This is your build VM’s physical bin folder, the AosService folder is usually on the unit K for cloud-hosted VMs. I guess this will change when we go VM-less to do the builds.

Update!: the route to the unit can be changed for $(ServiceDrive), getting a path like $(ServiceDrive)\AOSService\PackagesLocalDirectory\bin.

Location of the X++ binaries to package

The task comes with this field filled in as $(Build.BinariesDirectory) but this didn’t work out for our build definitions, maybe the variable isn’t set up on the proj file. After changing this to $(Agent.BuildDirectory)\Bin the package is generated.

Filename and path for the deployable package

The path on the image should be changed to $(Build.ArtifactStagingDirectory)\Packages\AXDeployableRuntime_$(Build.BuildNumber).zip. You can leave it without the Packages folder in the path, but if you do that you will need to change the Path to Publish field in the Publish Artifact: Package step of the definition.

Add Licenses to Deployable Package task

This task will add the license files to an existing Deployable Package. Remember that the path of the deployable package must be the same as the one in the Create Deployable Package task.

And you’re done! A step closer from getting rid of the build VM.

If you need help setting up the release pipeline you can check this post I wrote.

Application Checker: enforcing better coding practices?

Unless you’ve been working for an ISV there’s a high percentage of probabilities that you’ve never cared about Dynamics Best Practices (BP), or maybe you have. I haven’t worked for an ISV myself but back when I started working with AX I was handed the development BP document and I’ve tried to follow most of them when writing code.

But BPs could be ignored and not implemented without any issue. This is why Microsoft will publish…

Application Checker

Application Checker is a tool that will change that. It will force some rules that our code will have to meet, otherwise the code won’t compile (and maybe won’t even deploy to the environments).

We got an advance of it during last MBAS session “X++ programming with quality” by Dave Froslie and Peter Villadsen. Unfortunately the session wasn’t recorded.

App checker is built on BaseX, an XML analysis tool, and powers Socratex which Microsoft uses to track code quality. I don’t know if Socratex will be publicly released and I don’t remember if this was clarified during the session.

The set of rules can be found in Application Checker’s GitHub project and it’s still WIP. I think there’s loooots of things to decide before this goes GA, and I’m a bit worried and afraid of some of the rules 😛

Rule types

There’s different types of rules, some will become errors and other warnings. For example:

ExtensionsWithoutPrefix.xq: this rule will throw an error avoiding your code to compile. It checks if the extension class has a name ending with _Extension and an attribute ExtensionOf. If it has it must have a prefix. E.g.: if we extend the class CustPostInvoice it can’t be named CustPostInvoice_Extension, it needs a prefix like CustPostInvoiceAAS_Extension.

SelectForUpdateAbsent.xq: this rule will throw a warning. When there’s a forUpdate clause in a select statement and no doUpdate, update, delete, doDelete or write is called later it will let us know.

As of today, there’s 21 rules in the GitHub project. You can contribute to the project, and you could enforce your own rules without sending them to the project on your dev boxes, just add them to the local rules folder. I’d create a rule that makes the space after an if/while/for/switch mandatory and throws an error otherwise, but that’s only a bit of my OCD when writing/reading code.

Try it on your code

We can already use Application Checker on our development environments since PU26, I think. We just need to install JRE and BaseX in the dev box and select the check when doing a full build.

Some examples

ComplexityIndentationCombined.xq

This query checks the (wait for it…) cyclomatic complexity of the methods. I’ll try to explain it… Cyclomatic complexity is a metric for software quality, and is the number of independent paths in the code. Depending on the number of ifs, whiles, sitches, etc… the code can have different outcomes through different paths, that’s what complexity calculates.

Taking this as an example, a dumb one but ignore it, just look at the amount of different paths that could happen:

class AASAppCheckerDemo
{            
    public void complexMethod()
    {
        int a, b, c, d, e, f, g, h, i;

        if (a)
        {
            if (d)
            {				
                if (d)
                {
                }
                else if (e)
                {
                }
                else if(f)
                {
                }                
            }
            else if (e)
            {
                if (d)
                {
                }
                else if (e)
                {
                }
                else if(f)
                {
                }                
            }
            else if(f)
            {                
                if (d)
                {
                }
                else if (e)
                {
                }
                else if(f)
                {
                }
            }
        }
        else if (b)
        {
            if (d)
            {
                if (d)
                {
                }
                else if (e)
                {
                }
                else if(f)
                {
                }
            }
            else if (e)
            {
                if (d)
                {
                }
                else if (e)
                {
                }
                else if(f)
                {
                }                
            }
            else if(f)
            {               
                if (d)
                {
                }
                else if (e)
                {
                }
                else if(f)
                {
                }                
            }
        }
        else if(c)
        {
            if (d)
            {                
                if (d)
                {
                }
                else if (e)
                {
                }
                else if(f)
                {
                }                
            }
            else if (e)
            {                
                if (d)
                {
                }
                else if (e)
                {
                }
                else if(f)
                {
                }                
            }
            else if(f)
            {
                if (d)
                {
                }
                else if (e)
                {
                }
                else if(f)
                {
                }
            }
        }        
    }

}

In App checker the error appears when the complexity is over 30. I’ve used Lizard code complexity analyzer to calculate the complexity of the method below and I’m getting a 49.

The rule also checks for the indentation depth, failing if it’s greater than 2. In the end the purpose of both rules is to try to cut up long/large methods, which will also help in enabling more extension points in different places of our logic, like Microsoft did with Data Provider classes for reports.

BalancedTtsStatement.xq

This one gives me mixed feelings. The rule checks that the ttsbegin and the ttscommit of a method are in the same scope. So the following is not possible:

public void ttsCheck()
{
    StagingTable stagingTable;

    try
    {
        while select forupdate stagingTable
            where !stagingTable.Processed
        {
            ttsbegin;                
            boolean ret = this.doThings();

            if (ret)
            {
                stagingTable.Processed = true;
                stagingTable.update();
                ttscommit;
            }
            else
            {
                ttsabort;

                ttsbegin;
                stagingTable.Processed	= false;
                stagingTable.ErrorMsg	= 'An error ocurred, see log.';
                stagingTable.update();
                ttscommit;
            }
        }
    }
    catch (Exception::Error)
    {
        ttsabort;

        ttsbegin;
        stagingTable.Processed = false;
        stagingTable.update();
        ttscommit;
    }
    catch
    {
        ttsabort;

        ttsbegin;
        stagingTable.Processed = false;
        stagingTable.update();
        ttscommit;
    }
}

private boolean doThings()
{
    return true;
}

Imagine you’ve developed an integration with an external application that writes data to an intermediate table in MSDyn365FO and you process all pending data sequentially. You don’t want to throw an error if something goes wrong because you need the process to continue with the following record, so you ttsabort the wrong line, store the error and continue. If this is not possible… how should we do this? Create a batch that creates a task for each line to process?

Plus, the standard models have plenty of ttscommit inside if statements.

RecursiveMethods.xq

This rule will block the use of recursion on static methods. I don’t get why. Application checker should be a way to better coding practices, not forbidding some patterns. If somebody gets a recursive method to prod and the exit condition isn’t met… hello testing?

Some final thoughts

Will this force developers to code better? I don’t think so, but that’s probably not Application checker’s purpose. For centuries humans have found ways to bypass rules, laws and all kinds of restrictions and this won’t be an exception.

Will it help? Hell yes! But the best way to ensure code quality is promoting the best practices in your team, through internal trainings or code reviews. And even then if someone doesn’t care about clean code will keep on writing terrible code, which might work but won’t be beautiful at all.

Finally, I’m not sure about some rules, like avoiding recursion on static methods or the tts thing. We’ll just have to wait and see which rules make it to the final release and how will Application checker be finally implemented in the MSDyn365FO application lifecycle by blocking (or not) the deployments of code which doesn’t pass all the checks or if it will be included into the build process.

Self-service deployments: the future is here

Right now Microsoft Dynamics 365 for Finance and Operations has an old style monolithic architecture, even it’s now in Azure’s cloud, what we really have is a single (or multiple for Tier 2+ environments) VM that runs everything: the AOS/IIS, Azure SQL Server, the Batch service, MR, etc. Exactly the same as AX 2009/2012.

This is going to change in the coming months with the self-service deployments. We’ll move from the monolithic architecture to microservices that will run all the needed components with the help of Azure’s Service Fabric. MSDyn365FO will be on a real SAAS model.

Before starting let me clarify that all these changes will only apply to Microsoft-managed Tier 2+ environments: sandbox and production environments. The build environment (until it’s made obsolete) and the cloud-hosted environments on the customer or partner subscription will still be single VMs.

What’s new?

Faster deployments

When you deploy a new environment it will start deploying without waiting for Microsoft to do it (it’s self-service!). Additionally, thanks to the new microservices architecture, it will be ready to use in under 30 minutes compared to 6-8 hours of regular environments. The first time feels like…

Subscription estimator

We still need to fill out the subscription estimator for licensing purposes and for MS to estimate the size of the production environment. The self-service environments can be escalated more flexibly and quickly.

No RDP access

The access to the VM desktop has been removed because… well, I guess it’s because there’s no VM anymore. All the operations that could need us to access the RDP can be done from LCS.

No SQL Server access

Yes, no RDP access means no RDP access to the SQL box either. We still have access to the Azure SQL DB, we just need to ask for it from LCS and it’s granted in seconds:

Additionally you must whitelist your IP (or the one you’ll access SQL from) from the Maintain – Enable access button on LCS to be able to connect to the Azure SQL Server. The access to the DB and the firewall rule will be enabled for 8 hours.

As usual, there’s no access to the production DB.

One deployable package to rule them all

If you’ve recently tried to deploy a deployable package (DP) without all the packages the environment has (basically generating the DP for a single model/package from Visual Studio) you must’ve noticed the warning about the difference in the packages from the DP and the environment.

With the self-service deployments you must include all models/packages AND!! ISVs in one single deployable package.

Production updates

First, we can start the deployment to production without the 5 hours in notice we need to schedule now. We still can schedule the deployment but we can also start it instantly.

Next, the way the production environment is updated changes a bit from what we’re used to. With the new deployments we will update the sandbox environment as we do now, once it’s done we’ll select a sandbox environment to be promoted into production. This is probably another benefit of the architecture changes.

In the future the deployment downtime will also be reduced to zero for the service updates as long as you’re on the latest update. This won’t be available for custom DPs.

How do I get this?

At the moment this is only available for some new customers. Current customers will be migrated during the coming months, MS will contact the customers to schedule a maintenance window to apply the changes.

For more information check the session Microsoft Dynamics 365 for Finance and Operations: Strategic Lifecycle Services Investments from last June’s MBAS.

Our experience with it

We got into the private deployment preview program almost a year ago with one of Axazure’s customers. The customer is now live with the self-service environments and everything has been fine so far.

But the beginning was a bit hard. Some of the functionalities were still not available at the moment, like DB refresh or… package deployment. Yes, we needed to ask MS to deploy our DPs each time. We couldn’t even put the environments in maintenance mode! In the first months of 2019 a lot of functionality was added to LCS and in June we finally got the production self-service update functionality available. The help we’ve gotten from Microsoft’s product team has been very valuable and they have unlocked some issues that were stopping the progress of the project.

Slow set-based operations?

In Microsoft Dynamics 365 for Finance and Operations we can execute the CRUD operations from code in two different ways, record-per-record or set-based.

Microsoft’s recommendation is to always use set-based operations, if possible, as you can check on the Implementation Best Practices for Dynamics 365: Performance best practices for a successful Dynamics 365 Finance and Operations implementation session from last June’s Business Applications Summit.

Why?

Set-based Vs. Record-per-record

When we run a query in MSDyn365FO we’re using its data access layer which will later be translated into real SQL. We can see the differences using xRecord’s getSQLStatement with generateonly on the query (and forceliterals to show the parameter’s values) to get the SQL query. For example if we run the following code:

We’ll get this SQL statement:

SELECT TOP 1 T1.PAYMTERMID,T1.LINEDISC,T1.TAXWITHHOLDGROUP_TH,T1.PARTYCOUNTRY,T1.ACCOUNTNUM,T1.ACCOUNTSTATEMENT,T1.AFFILIATED_RU,T1.AGENCYLOCATIONCODE,T1.BANKACCOUNT,T1.BANKCENTRALBANKPURPOSECODE,T1.BANKCENTRALBANKPURPOSETEXT,T1.BANKCUSTPAYMIDTABLE,T1.BIRTHCOUNTYCODE_IT,T1.BIRTHPLACE_IT,T1.BLOCKED,T1.CASHDISC,T1.CASHDISCBASEDAYS,T1.CCMNUM_BR,T1.CLEARINGPERIOD,T1.CNAE_BR,T1.CNPJCPFNUM_BR,T1.COMMERCIALREGISTER,T1.COMMERCIALREGISTERINSETNUMBER,T1.COMMERCIALREGISTERSECTION,T1.COMMISSIONGROUP,T1.COMPANYCHAINID,T1.COMPANYIDSIRET,T1.COMPANYNAFCODE,T1.COMPANYTYPE_MX,T1.CONSDAY_JP,T1.CONTACTPERSONID,T1.CREDITCARDADDRESSVERIFICATION,T1.CREDITCARDADDRESSVERIFICATIONLEVEL,T1.CREDITCARDADDRESSVERIFICATIONVOID,T1.CREDITCARDCVC,T1.CREDITMAX,T1.CREDITRATING,T1.CURP_MX,T1.CURRENCY,T1.CUSTCLASSIFICATIONID,T1.CUSTEXCLUDECOLLECTIONFEE,T1.CUSTEXCLUDEINTERESTCHARGES,T1.CUSTFINALUSER_BR,T1.CUSTGROUP,T1.CUSTITEMGROUPID,T1.CUSTTRADINGPARTNERCODE,T1.CUSTWHTCONTRIBUTIONTYPE_BR,T1.DEFAULTDIMENSION,T1.DEFAULTDIRECTDEBITMANDATE,T1.DEFAULTINVENTSTATUSID,T1.DESTINATIONCODEID,T1.DLVMODE,T1.DLVREASON,T1.DLVTERM,T1.EINVOICE,T1.EINVOICEATTACHMENT,T1.EINVOICEEANNUM,T1.ENDDISC,T1.ENTRYCERTIFICATEREQUIRED_W,T1.EXPORTSALES_PL,T1.EXPRESSBILLOFLADING,T1.FACTORINGACCOUNT,T1.FEDERALCOMMENTS,T1.FEDNONFEDINDICATOR,T1.FINECODE_BR,T1.FISCALCODE,T1.FISCALDOCTYPE_PL,T1.FORECASTDMPINCLUDE,T1.FOREIGNRESIDENT_RU,T1.FREIGHTZONE,T1.GENERATEINCOMINGFISCALDOCUMENT_BR,T1.GIROTYPE,T1.GIROTYPEACCOUNTSTATEMENT,T1.GIROTYPECOLLECTIONLETTER,T1.GIROTYPEFREETEXTINVOICE,T1.GIROTYPEINTERESTNOTE,T1.GIROTYPEPROJINVOICE,T1.ICMSCONTRIBUTOR_BR,T1.IDENTIFICATIONNUMBER,T1.IENUM_BR,T1.INCLTAX,T1.INSSCEI_BR,T1.INTBANK_LV,T1.INTERCOMPANYALLOWINDIRECTCREATION,T1.INTERCOMPANYAUTOCREATEORDERS,T1.INTERCOMPANYDIRECTDELIVERY,T1.INTERESTCODE_BR,T1.INVENTLOCATION,T1.INVENTPROFILEID_RU,T1.INVENTPROFILETYPE_RU,T1.INVENTSITEID,T1.INVOICEACCOUNT,T1.INVOICEADDRESS,T1.INVOICEPOSTINGTYPE_RU,T1.IRS1099CINDICATOR,T1.ISRESIDENT_LV,T1.ISSUEOWNENTRYCERTIFICATE_W,T1.ISSUERCOUNTRY_HU,T1.LINEOFBUSINESSID,T1.LVPAYMTRANSCODES,T1.MAINCONTACTWORKER,T1.MANDATORYCREDITLIMIT,T1.MANDATORYVATDATE_PL,T1.MARKUPGROUP,T1.MCRMERGEDPARENT,T1.MCRMERGEDROOT,T1.MULTILINEDISC,T1.NIT_BR,T1.NUMBERSEQUENCEGROUP,T1.ONETIMECUSTOMER,T1.ORDERENTRYDEADLINEGROUPID,T1.ORGID,T1.OURACCOUNTNUM,T1.PACKAGEDEPOSITEXCEMPT_PL,T1.PACKMATERIALFEELICENSENUM,T1.PARTY,T1.PARTYSTATE,T1.PASSPORTNO_HU,T1.PAYMDAYID,T1.PAYMENTREFERENCE_EE,T1.PAYMIDTYPE,T1.PAYMMODE,T1.PAYMSCHED,T1.PAYMSPEC,T1.PDSCUSTREBATEGROUPID,T1.PDSFREIGHTACCRUED,T1.PDSREBATETMAGROUP,T1.PRICEGROUP,T1.RESIDENCEFOREIGNCOUNTRYREGIONID_IT,T1.RFC_MX,T1.SALESCALENDARID,T1.SALESDISTRICTID,T1.SALESGROUP,T1.SALESPOOLID,T1.SEGMENTID,T1.SERVICECODEONDLVADDRESS_BR,T1.STATEINSCRIPTION_MX,T1.STATISTICSGROUP,T1.SUBSEGMENTID,T1.SUFRAMA_BR,T1.SUFRAMANUMBER_BR,T1.SUFRAMAPISCOFINS_BR,T1.SUPPITEMGROUPID,T1.TAXGROUP,T1.TAXLICENSENUM,T1.TAXPERIODPAYMENTCODE_PL,T1.TAXWITHHOLDCALCULATE_IN,T1.TAXWITHHOLDCALCULATE_TH,T1.UNITEDVATINVOICE_LT,T1.USECASHDISC,T1.USEPURCHREQUEST,T1.VATNUM,T1.VENDACCOUNT,T1.WEBSALESORDERDISPLAY,T1.AUTHORITYOFFICE_IT,T1.EINVOICEREGISTER_IT,T1.FOREIGNERID_BR,T1.PRESENCETYPE_BR,T1.TAXGSTRELIEFGROUPHEADING_MY,T1.FOREIGNTAXREGISTRATION_MX,T1.CUSTWRITEOFFREFRECID,T1.ISEXTERNALLYMAINTAINED,T1.SATPAYMMETHOD_MX,T1.SATPURPOSE_MX,T1.CFDIENABLED_MX,T1.FOREIGNTRADE_MX,T1.WORKFLOWSTATE,T1.USEORIGINALDOCUMENTASFACTURE_RU,T1.COLLECTIONLETTERCODE,T1.BLOCKFLOORLIMITUSEINCHANNEL,T1.AXZMODEL182LEGALNATURE,T1.AXZCRMGUID,T1.MODIFIEDDATETIME,T1.MODIFIEDBY,T1.CREATEDDATETIME,T1.RECVERSION,T1.PARTITION,T1.RECID,T1.MEMO 

FROM CUSTTABLE T1 

WHERE (((PARTITION=5637144576) AND (DATAAREAID=N'usmf')) AND (ACCOUNTNUM=N'0001'))

We can see all the fields are being selected, and the where clause contains the account number we selected (plus DataAreaId and Partition).

When a while select is run on MSDyn365FO a select SQL statement is executed on SQL Server for each loop of the while. The same happens if an update or delete is executed inside the loop. This is know as record-per-record operation.

Imagine you need to update all the customers with the customer group 10 to update their note. We could do this with a while select, like this:

This would make as many calls as customers from the group 10 existed to SQL Server, one for each loop. Or we could use set-based operations:

This will execute a single SQL statement on SQL Server that will update all the customers with the customer group 10 instead of a query for each customer:

UPDATE CUSTTABLE
SET MEMO = 'Special customer'
WHERE (((PARTITION=5637144576) 
    AND (DATAAREAID=N'usmf')) 
    AND (CUSTGROUP=N'10'))

There’s three set-based operations in MSDyn365FO, update_recordset to update records, insert_recordset to create records and delete_from to delete the records. Plus we can make massive inserts using RecordSortedList and RecordInsertList.

Running this methods instead of while selects should obviously be faster as it’s only executing a single SQL query. But…

Why could my set-based operations be running slow?

There’s some well-documented scenarios in which set-based operations fall back to record-per-record operations as we can see in the following table:

DELETE_FROMUPDATE_RECORDSETINSERT_RECORDSETARRAY_INSERTUse … to override
Non-SQL tablesYesYesYesYesNot applicable
Delete actionsYesNoNoNoskipDeleteActions
Database log enabledYesYesYesNoskipDatabaseLog
Overridden methodYesYesYesYesskipDataMethods
Alerts set up for tableYesYesYesNoskipEvents
ValidTimeStateFieldType property not equal to None on a tableYesYesYesYesNot applicable

In the example, if the update method of the CustTable is overriden (which it is) the operation from the update_recordset piece will be run like a while select that updates each record.

In the case of the update_recordset this can be solved calling the skipDataMethods method before running the update:

This will avoid calling the update method (or insert in case of the insert_recordset), more or less like calling doUpdate in a loop. The rest of the methods can be overriden with the corresponding method on the last column.

So, for bulk updates I’d always use set-based operations and enable this on data entities too with the EnableSetBasedSqlOperations property.

And now another but is coming.

Should I always use set-based operations when updating large sets of data?

Well it depends on which data you’re working with. There’s a wonderful blog post from Denis Trunin called “Blocking in D365FO(and why you shouldn’t always follow MS recommendations)” that shows a perfect example where set-based operations would be counterproductive.

As always, developing an ERP is quite sensitive, and similar scenarios can have different solutions. Analyze the requirements and decide which one to use.

Update to Visual Studio 2019 for #MSDyn365FO

Tired of developing in Visual Studio 2015? You feel you’ve been left and forgotten in the past? Worry no more, you can use Visual Studio 2017/2019 to develop Microsoft Dynamics 365 for Finance & Operations!

What are the advantages?

Absolutely none at all! Visual Studio will still go non-responding whatever the version is because it’s the dev tools extension what’s causing the issues.

Of course we get the option to use Live Share, and for screen sharing sessions that’s way better than teams. Hey, and we’ll be using the latest VS version!

Is it difficult?

No, zero mysteries. The first thing we need to do is downloading Visual Studio 2019 Professional (or Enterprise but it won’t make such a difference for D365 development) and install it:

Select the .NET desktop development option and press install. When the installation is finished we log in with our account.

The next step is installing the Dynamics developer tools extension for VS. Go to drive K and in the DeployablePackages you’ll find some ZIP files that have the extension in the DevToolsService/Scripts folder:

An alternativa is, for example, downloading a Platform Update package which also has the dev tools extensions, and maybe with some update to them.

Install the extension and the VS2019 option is already there:

Once installed open VS as the admin and…

And also…

Don’t panic! The extension was made for VS2015 and using it in a newer version can cause some warnings, but it’s just that, the tools are installed and ready to use:

As I said in the beggining, the dev tools extension is the one causing the unresponsiveness or blocks in VS, and Visual Studio 2019 is letting us know:

But regardless of the warnings working with Visual Studio 2019 is possible. I’ve been doing so for a week and I still haven’t found a blocking issue that makes me go back to VS2015.

Dev tools preview

In October 2019 the dev tools’ preview version will be published, as we could see in the MBAS in Atlanta. Let’s see which new features this will bring us both in a possible VS version upgrade or performance.

Consume a SOAP web service in Dynamics 365 for Finance and Operations using ChannelFactory

If you ever need to consume a SOAP web service from Dynamics 365 for Finance and Operations, the first step you should take is asking the people responsible for that web service to create a REST version. If that’s not possible this post is for you.

I’ll use this SOAP web service I found online at http://www.dneonline.com/calculator.asmx for the example, it’s a simple service-calculator with four methods to add, substract, multiply or divide two integers.

Consuming a SOAP service in .NET

Let’s start with the basics. How do we consume a SOAP web service in Visual Studio? Easy peasy. Just add a service reference to your project:

And point it to the web service of your choice:

This will add the reference to the project:

With that done we can create an instance of the web service’s client and call one of it’s methods:

3 + 6 = 9, it looks like it’s working.

Consuming a SOAP service in Dynamics 365 for Finance and Operations

To consume the web service on FnO create a new project in Visual Studio, right click on References and add the service reference:

Hmmm… nope, it can’t be done, no service reference option.

Consuming a SOAP service in Dynamics 365 for Finance and Operations (I hope…)

The problem is that we cannot add a service reference in Visual Studio on 365 dev boxes.

What do the docs say about this? Well, like in AX2012 we need to create a .NET class library that will consume that web service, then add the reference to our DLL on 365 and call the service methods from a client object. All right!

There it is. A reference to our class library and a runnable class that will do the job:

Let’s run it!

What?

An exception of type ‘System.InvalidOperationException’ occurred in System.ServiceModel.dll but was not handled in user code

Additional information: Could not find default endpoint element that references contract ‘AASSOAPCalculatorService.CalculatorSoap’ in the ServiceModel client configuration section. This might be because no configuration file was found for your application, or because no endpoint element matching this contract could be found in the client element.

Contract? What contract? I know nothing about a contract. Nobody told me about any contract! What does the Wikipedia say about SOAP?

Soap is the term for a salt of a fatty acid or for a variety of cleansing and lubricating products produced from such a substance.

Oops wrong soap…

SOAP provides the Messaging Protocol layer of a web services protocol stack for web services. It is an XML-based protocol consisting of three parts:

  • an envelope, which defines the message structure and how to process it

  • a set of encoding rules for expressing instances of application-defined datatypes

  • a convention for representing procedure calls and responses

The envelope is the contract. A data contract is an agreement between a service and a client that abstractly describes the data to be exchanged. That contract.

Consuming a SOAP service in Dynamics 365 for Finance and Operations (I promise this is the good one)

If we check the class library there’s a file called app.config:

In this file we can see the endpoint the DLL is using. This is fixed (hardcoded) and in case there’s a test endpoint and a production endpoint we should change the address accordingly and have two different DLLs, one for each endpoint. We can also see the data contract being used by the service, the one called AASSOAPCalculatorService.CalculatorSoap. Because #MSDyn365FO is a web-based ERP we could solve this by adding the system.serviceModel node in the web.config file of the server, right? (app.config for desktop apps, web.config for web apps). Yes, but this would be useless as we have no access to the production environment to do this, and it will be impossible to do in the sandbox Tier-2+ environments when the self-service environments start to roll out.

So, what do we do? Easy, ChannelFactory<T> to the rescue! The ChannelFactory<T> allows us to create an instance of the factory for our service contract and then creates a channel between the client and the service. The client being our class in D365and the service the endpoint (obviously).

Then we do the following:

The BasicHttpBinding object can be a BasicHttpsBinding if the web service is running on HTTPS. The endpoint is the URL of the web service. Then we instantiate a service contract from our class with the binding and enpoint and create the channel. Now we can call the web service methods and…

It’s working! And it’s even better, if there’s different endpoints for a test and prod web service we just have to parametrize them!

But really, don’t use SOAP services, go with the REST.