Self-service deployments: the future is here

Right now Microsoft Dynamics 365 for Finance and Operations has an old style monolithic architecture, even it’s now in Azure’s cloud, what we really have is a single (or multiple for Tier 2+ environments) VM that runs everything: the AOS/IIS, Azure SQL Server, the Batch service, MR, etc. Exactly the same as AX 2009/2012.

This is going to change in the coming months with the self-service deployments. We’ll move from the monolithic architecture to microservices that will run all the needed components with the help of Azure’s Service Fabric. MSDyn365FO will be on a real SAAS model.

Before starting let me clarify that all these changes will only apply to Microsoft-managed Tier 2+ environments: sandbox and production environments. The build environment (until it’s made obsolete) and the cloud-hosted environments on the customer or partner subscription will still be single VMs.

What’s new?

Faster deployments

When you deploy a new environment it will start deploying without waiting for Microsoft to do it (it’s self-service!). Additionally, thanks to the new microservices architecture, it will be ready to use in under 30 minutes compared to 6-8 hours of regular environments. The first time feels like…

Subscription estimator

We still need to fill out the subscription estimator for licensing purposes and for MS to estimate the size of the production environment. The self-service environments can be escalated more flexibly and quickly.

No RDP access

The access to the VM desktop has been removed because… well, I guess it’s because there’s no VM anymore. All the operations that could need us to access the RDP can be done from LCS.

No SQL Server access

Yes, no RDP access means no RDP access to the SQL box either. We still have access to the Azure SQL DB, we just need to ask for it from LCS and it’s granted in seconds:

Additionally you must whitelist your IP (or the one you’ll access SQL from) from the Maintain – Enable access button on LCS to be able to connect to the Azure SQL Server. The access to the DB and the firewall rule will be enabled for 8 hours.

As usual, there’s no access to the production DB.

One deployable package to rule them all

If you’ve recently tried to deploy a deployable package (DP) without all the packages the environment has (basically generating the DP for a single model/package from Visual Studio) you must’ve noticed the warning about the difference in the packages from the DP and the environment.

With the self-service deployments you must include all models/packages AND!! ISVs in one single deployable package.

Production updates

First, we can start the deployment to production without the 5 hours in notice we need to schedule now. We still can schedule the deployment but we can also start it instantly.

Next, the way the production environment is updated changes a bit from what we’re used to. With the new deployments we will update the sandbox environment as we do now, once it’s done we’ll select a sandbox environment to be promoted into production. This is probably another benefit of the architecture changes.

In the future the deployment downtime will also be reduced to zero for the service updates as long as you’re on the latest update. This won’t be available for custom DPs.

How do I get this?

At the moment this is only available for some new customers. Current customers will be migrated during the coming months, MS will contact the customers to schedule a maintenance window to apply the changes.

For more information check the session Microsoft Dynamics 365 for Finance and Operations: Strategic Lifecycle Services Investments from last June’s MBAS.

Our experience with it

We got into the private deployment preview program almost a year ago with one of Axazure’s customers. The customer is now live with the self-service environments and everything has been fine so far.

But the beginning was a bit hard. Some of the functionalities were still not available at the moment, like DB refresh or… package deployment. Yes, we needed to ask MS to deploy our DPs each time. We couldn’t even put the environments in maintenance mode! In the first months of 2019 a lot of functionality was added to LCS and in June we finally got the production self-service update functionality available. The help we’ve gotten from Microsoft’s product team has been very valuable and they have unlocked some issues that were stopping the progress of the project.

Slow set-based operations?

In Microsoft Dynamics 365 for Finance and Operations we can execute the CRUD operations from code in two different ways, record-per-record or set-based.

Microsoft’s recommendation is to always use set-based operations, if possible, as you can check on the Implementation Best Practices for Dynamics 365: Performance best practices for a successful Dynamics 365 Finance and Operations implementation session from last June’s Business Applications Summit.

Why?

Set-based Vs. Record-per-record

When we run a query in MSDyn365FO we’re using its data access layer which will later be translated into real SQL. We can see the differences using xRecord’s getSQLStatement with generateonly on the query (and forceliterals to show the parameter’s values) to get the SQL query. For example if we run the following code:

We’ll get this SQL statement:

SELECT TOP 1 T1.PAYMTERMID,T1.LINEDISC,T1.TAXWITHHOLDGROUP_TH,T1.PARTYCOUNTRY,T1.ACCOUNTNUM,T1.ACCOUNTSTATEMENT,T1.AFFILIATED_RU,T1.AGENCYLOCATIONCODE,T1.BANKACCOUNT,T1.BANKCENTRALBANKPURPOSECODE,T1.BANKCENTRALBANKPURPOSETEXT,T1.BANKCUSTPAYMIDTABLE,T1.BIRTHCOUNTYCODE_IT,T1.BIRTHPLACE_IT,T1.BLOCKED,T1.CASHDISC,T1.CASHDISCBASEDAYS,T1.CCMNUM_BR,T1.CLEARINGPERIOD,T1.CNAE_BR,T1.CNPJCPFNUM_BR,T1.COMMERCIALREGISTER,T1.COMMERCIALREGISTERINSETNUMBER,T1.COMMERCIALREGISTERSECTION,T1.COMMISSIONGROUP,T1.COMPANYCHAINID,T1.COMPANYIDSIRET,T1.COMPANYNAFCODE,T1.COMPANYTYPE_MX,T1.CONSDAY_JP,T1.CONTACTPERSONID,T1.CREDITCARDADDRESSVERIFICATION,T1.CREDITCARDADDRESSVERIFICATIONLEVEL,T1.CREDITCARDADDRESSVERIFICATIONVOID,T1.CREDITCARDCVC,T1.CREDITMAX,T1.CREDITRATING,T1.CURP_MX,T1.CURRENCY,T1.CUSTCLASSIFICATIONID,T1.CUSTEXCLUDECOLLECTIONFEE,T1.CUSTEXCLUDEINTERESTCHARGES,T1.CUSTFINALUSER_BR,T1.CUSTGROUP,T1.CUSTITEMGROUPID,T1.CUSTTRADINGPARTNERCODE,T1.CUSTWHTCONTRIBUTIONTYPE_BR,T1.DEFAULTDIMENSION,T1.DEFAULTDIRECTDEBITMANDATE,T1.DEFAULTINVENTSTATUSID,T1.DESTINATIONCODEID,T1.DLVMODE,T1.DLVREASON,T1.DLVTERM,T1.EINVOICE,T1.EINVOICEATTACHMENT,T1.EINVOICEEANNUM,T1.ENDDISC,T1.ENTRYCERTIFICATEREQUIRED_W,T1.EXPORTSALES_PL,T1.EXPRESSBILLOFLADING,T1.FACTORINGACCOUNT,T1.FEDERALCOMMENTS,T1.FEDNONFEDINDICATOR,T1.FINECODE_BR,T1.FISCALCODE,T1.FISCALDOCTYPE_PL,T1.FORECASTDMPINCLUDE,T1.FOREIGNRESIDENT_RU,T1.FREIGHTZONE,T1.GENERATEINCOMINGFISCALDOCUMENT_BR,T1.GIROTYPE,T1.GIROTYPEACCOUNTSTATEMENT,T1.GIROTYPECOLLECTIONLETTER,T1.GIROTYPEFREETEXTINVOICE,T1.GIROTYPEINTERESTNOTE,T1.GIROTYPEPROJINVOICE,T1.ICMSCONTRIBUTOR_BR,T1.IDENTIFICATIONNUMBER,T1.IENUM_BR,T1.INCLTAX,T1.INSSCEI_BR,T1.INTBANK_LV,T1.INTERCOMPANYALLOWINDIRECTCREATION,T1.INTERCOMPANYAUTOCREATEORDERS,T1.INTERCOMPANYDIRECTDELIVERY,T1.INTERESTCODE_BR,T1.INVENTLOCATION,T1.INVENTPROFILEID_RU,T1.INVENTPROFILETYPE_RU,T1.INVENTSITEID,T1.INVOICEACCOUNT,T1.INVOICEADDRESS,T1.INVOICEPOSTINGTYPE_RU,T1.IRS1099CINDICATOR,T1.ISRESIDENT_LV,T1.ISSUEOWNENTRYCERTIFICATE_W,T1.ISSUERCOUNTRY_HU,T1.LINEOFBUSINESSID,T1.LVPAYMTRANSCODES,T1.MAINCONTACTWORKER,T1.MANDATORYCREDITLIMIT,T1.MANDATORYVATDATE_PL,T1.MARKUPGROUP,T1.MCRMERGEDPARENT,T1.MCRMERGEDROOT,T1.MULTILINEDISC,T1.NIT_BR,T1.NUMBERSEQUENCEGROUP,T1.ONETIMECUSTOMER,T1.ORDERENTRYDEADLINEGROUPID,T1.ORGID,T1.OURACCOUNTNUM,T1.PACKAGEDEPOSITEXCEMPT_PL,T1.PACKMATERIALFEELICENSENUM,T1.PARTY,T1.PARTYSTATE,T1.PASSPORTNO_HU,T1.PAYMDAYID,T1.PAYMENTREFERENCE_EE,T1.PAYMIDTYPE,T1.PAYMMODE,T1.PAYMSCHED,T1.PAYMSPEC,T1.PDSCUSTREBATEGROUPID,T1.PDSFREIGHTACCRUED,T1.PDSREBATETMAGROUP,T1.PRICEGROUP,T1.RESIDENCEFOREIGNCOUNTRYREGIONID_IT,T1.RFC_MX,T1.SALESCALENDARID,T1.SALESDISTRICTID,T1.SALESGROUP,T1.SALESPOOLID,T1.SEGMENTID,T1.SERVICECODEONDLVADDRESS_BR,T1.STATEINSCRIPTION_MX,T1.STATISTICSGROUP,T1.SUBSEGMENTID,T1.SUFRAMA_BR,T1.SUFRAMANUMBER_BR,T1.SUFRAMAPISCOFINS_BR,T1.SUPPITEMGROUPID,T1.TAXGROUP,T1.TAXLICENSENUM,T1.TAXPERIODPAYMENTCODE_PL,T1.TAXWITHHOLDCALCULATE_IN,T1.TAXWITHHOLDCALCULATE_TH,T1.UNITEDVATINVOICE_LT,T1.USECASHDISC,T1.USEPURCHREQUEST,T1.VATNUM,T1.VENDACCOUNT,T1.WEBSALESORDERDISPLAY,T1.AUTHORITYOFFICE_IT,T1.EINVOICEREGISTER_IT,T1.FOREIGNERID_BR,T1.PRESENCETYPE_BR,T1.TAXGSTRELIEFGROUPHEADING_MY,T1.FOREIGNTAXREGISTRATION_MX,T1.CUSTWRITEOFFREFRECID,T1.ISEXTERNALLYMAINTAINED,T1.SATPAYMMETHOD_MX,T1.SATPURPOSE_MX,T1.CFDIENABLED_MX,T1.FOREIGNTRADE_MX,T1.WORKFLOWSTATE,T1.USEORIGINALDOCUMENTASFACTURE_RU,T1.COLLECTIONLETTERCODE,T1.BLOCKFLOORLIMITUSEINCHANNEL,T1.AXZMODEL182LEGALNATURE,T1.AXZCRMGUID,T1.MODIFIEDDATETIME,T1.MODIFIEDBY,T1.CREATEDDATETIME,T1.RECVERSION,T1.PARTITION,T1.RECID,T1.MEMO 

FROM CUSTTABLE T1 

WHERE (((PARTITION=5637144576) AND (DATAAREAID=N'usmf')) AND (ACCOUNTNUM=N'0001'))

We can see all the fields are being selected, and the where clause contains the account number we selected (plus DataAreaId and Partition).

When a while select is run on MSDyn365FO a select SQL statement is executed on SQL Server for each loop of the while. The same happens if an update or delete is executed inside the loop. This is know as record-per-record operation.

Imagine you need to update all the customers with the customer group 10 to update their note. We could do this with a while select, like this:

This would make as many calls as customers from the group 10 existed to SQL Server, one for each loop. Or we could use set-based operations:

This will execute a single SQL statement on SQL Server that will update all the customers with the customer group 10 instead of a query for each customer:

UPDATE CUSTTABLE
SET MEMO = 'Special customer'
WHERE (((PARTITION=5637144576) 
    AND (DATAAREAID=N'usmf')) 
    AND (CUSTGROUP=N'10'))

There’s three set-based operations in MSDyn365FO, update_recordset to update records, insert_recordset to create records and delete_from to delete the records. Plus we can make massive inserts using RecordSortedList and RecordInsertList.

Running this methods instead of while selects should obviously be faster as it’s only executing a single SQL query. But…

Why could my set-based operations be running slow?

There’s some well-documented scenarios in which set-based operations fall back to record-per-record operations as we can see in the following table:

DELETE_FROMUPDATE_RECORDSETINSERT_RECORDSETARRAY_INSERTUse … to override
Non-SQL tablesYesYesYesYesNot applicable
Delete actionsYesNoNoNoskipDeleteActions
Database log enabledYesYesYesNoskipDatabaseLog
Overridden methodYesYesYesYesskipDataMethods
Alerts set up for tableYesYesYesNoskipEvents
ValidTimeStateFieldType property not equal to None on a tableYesYesYesYesNot applicable

In the example, if the update method of the CustTable is overriden (which it is) the operation from the update_recordset piece will be run like a while select that updates each record.

In the case of the update_recordset this can be solved calling the skipDataMethods method before running the update:

This will avoid calling the update method (or insert in case of the insert_recordset), more or less like calling doUpdate in a loop. The rest of the methods can be overriden with the corresponding method on the last column.

So, for bulk updates I’d always use set-based operations and enable this on data entities too with the EnableSetBasedSqlOperations property.

And now another but is coming.

Should I always use set-based operations when updating large sets of data?

Well it depends on which data you’re working with. There’s a wonderful blog post from Denis Trunin called “Blocking in D365FO(and why you shouldn’t always follow MS recommendations)” that shows a perfect example where set-based operations would be counterproductive.

As always, developing an ERP is quite sensitive, and similar scenarios can have different solutions. Analyze the requirements and decide which one to use.