Slow set-based operations?

In Microsoft Dynamics 365 for Finance and Operations we can execute the CRUD operations from code in two different ways, record-per-record or set-based.

Microsoft’s recommendation is to always use set-based operations, if possible, as you can check on the Implementation Best Practices for Dynamics 365: Performance best practices for a successful Dynamics 365 Finance and Operations implementation session from last June’s Business Applications Summit.

Why?

Set-based Vs. Record-per-record

When we run a query in MSDyn365FO we’re using its data access layer which will later be translated into real SQL. We can see the differences using xRecord’s getSQLStatement with generateonly on the query (and forceliterals to show the parameter’s values) to get the SQL query. For example if we run the following code:

We’ll get this SQL statement:

 

We can see all the fields are being selected, and the where clause contains the account number we selected (plus DataAreaId and Partition).

When a while select is run on MSDyn365FO a select SQL statement is executed on SQL Server for each loop of the while. The same happens if an update or delete is executed inside the loop. This is know as record-per-record operation.

Imagine you need to update all the customers with the customer group 10 to update their note. We could do this with a while select, like this:

This would make as many calls as customers from the group 10 existed to SQL Server, one for each loop. Or we could use set-based operations:

This will execute a single SQL statement on SQL Server that will update all the customers with the customer group 10 instead of a query for each customer:

There’s three set-based operations in MSDyn365FO, update_recordset to update records, insert_recordset to create records and delete_from to delete the records. Plus we can make massive inserts using RecordSortedList and RecordInsertList.

Running this methods instead of while selects should obviously be faster as it’s only executing a single SQL query. But…

Why could my set-based operations be running slow?

There’s some well-documented scenarios in which set-based operations fall back to record-per-record operations as we can see in the following table:

DELETE_FROMUPDATE_RECORDSETINSERT_RECORDSETARRAY_INSERTUse … to override
Non-SQL tablesYesYesYesYesNot applicable
Delete actionsYesNoNoNoskipDeleteActions
Database log enabledYesYesYesNoskipDatabaseLog
Overridden methodYesYesYesYesskipDataMethods
Alerts set up for tableYesYesYesNoskipEvents
ValidTimeStateFieldType property not equal to None on a tableYesYesYesYesNot applicable

In the example, if the update method of the CustTable is overriden (which it is) the operation from the update_recordset piece will be run like a while select that updates each record.

In the case of the update_recordset this can be solved calling the skipDataMethods method before running the update:

This will avoid calling the update method (or insert in case of the insert_recordset), more or less like calling doUpdate in a loop. The rest of the methods can be overriden with the corresponding method on the last column.

So, for bulk updates I’d always use set-based operations and enable this on data entities too with the EnableSetBasedSqlOperations property.

And now another but is coming.

Should I always use set-based operations when updating large sets of data?

Well it depends on which data you’re working with. There’s a wonderful blog post from Denis Trunin called “Blocking in D365FO(and why you shouldn’t always follow MS recommendations)” that shows a perfect example where set-based operations would be counterproductive.

As always, developing an ERP is quite sensitive, and similar scenarios can have different solutions. Analyze the requirements and decide which one to use.

The mystery of the non-filtering query

There’s no mystery here but a misperception.

Recently, a colleague found a little issue when using an AOT query to feed a view with a range dynamically filtered using a SysQueryRangeUtil method.

Recreating the issue

The query is pretty simple, only showing ledger transaction data from the GeneralJournalEntry and GeneralJournalAccountEntry tables. A range in the Ledger field from the current company was added as you can see in the pic below:

Query en Visual Studio

We created a new range method by extending the SysQueryRangeUtil class. Using the Ledger::current() to filter the active company.

Extensión en visual studio

The we used the query to feed data to the view and added two fields just for testing purposes:

Vista en Visual Studio

Everything quite straightforward. Let’s check the view in the table browser…

Explorador de tablas

No data! And I can tell there’s data in here:

Registros en SSMS

What’s going on in here? If we use the query in a job (yeah, I know, Runnable Class…) the range is filtering the data as expected.

Psyduck is confused
Me in a tribute to “Psyduck is confused” by cazapelusas.com

So… let’s see the view design in SSMS:

Diseño de la vista en SSMS

Well, it definitely looks like something’s being filtered in here. The range is working! Is it? Sure? Which company does that Ledger table RecId corresponds to?

Registro de DAT

Qué haces besando a la lisiada!?
Why are you quering the damned DAT? (Sorry this was funnier in Spanish)

What’s going on?

There’s an easy and clear explanation but one doesn’t think of it until he faces this specific issue. While the view* is a Data Dictionary object, and when the project is synchronized the view is created in SQL Server, the query* is a X++ object and only exists within the application. The view is created in SQL and we can see and query it in SSMS. The AOT query doesn’t. It feeds the view and provides a data back end, but all X++ added functionality stays in 365, including the SysQueryRangeUtil filters.

The solution is an easy one. Removing the range in the query and adding it in the form data source will do the trick (if this can be considered a trick…).

(*) Note: the links to the docs point to AX 2012 docs but should be valid.