APIOps: ALM for Azure API Management

In this post, I’m going to talk about APIOps, the DevOps techniques applied to Azure API Management, that allow us to automate the promotion of changes between API Management instances.

It would be possible to attain a certain degree of automation using Bicep, but it wouldn’t apply to many elements of the APIM, and that’s why we need and like APIOps!

If you want to learn more about Azure API Management, you can read my previous posts and video:

• Azure API Management for Dynamics 365 F&O
• Azure API Management for your integrations
• IaC with Bicep: deploy Azure API Management Architecture
• Empower Your F&O Integrations with Azure API Management: Unlocking Superpowers!

Why use APIOps?

The biggest benefit from using APIOps is automation. With DevOps we can automate the deployments of updated code to sandbox environments thanks to CD pipelines, and we can have CI pipelines to make sure our code builds correctly.

Also, automation brings us many other benefits. By applying the principles of Application Lifecycle Management (ALM) to APIs, APIOps makes the entire APIM lifecycle much easier on us. This automation reduces the risk of errors when moving changes between development, UAT or production APIMs. It also accelerates the deployment process thanks to the publishing pipeline that moves the changes between environments/instances.

How does APIOps work?

Later we’ll learn how it works in more detail, but now let’s see a high-level solution of how APIOps works. The current solution, which is provided by Microsoft, is based on two pipelines and some extra tools hosted on their GitHub repo.

There is an extractor pipeline that will “read” your source APIM instance and generate several files used to define the features and characteristics of your APIM instance.

The extractor pipeline creates a Pull Request in your Azure DevOps repo. When we merge this PR into the main branch, the publisher pipeline runs.

The publisher pipeline takes the setup of your source APIM instance from the files created by the extractor pipeline.

First, the publisher pipeline will publish the changes to the same source APIM instance so you can do some testing.

Then the changes are deployed to the UAT pipeline (if you have one), but only after manual approval. And finally, and also after manual approval, they move to the production APIM instance.

Two supported scenarios

APIOps supports two ways of working: users that build their APIs from the Azure portal and users that build their APIs from an IDE (like VS). In this post, I’ll be showing the first way, from the Azure portal.

In the first scenario (from Azure portal), the workflow looks like this:

1. Make all changes to the APIM in the Azure portal.
2. Manually run the extractor pipeline, which will automatically create a PR with all the changes.
3. Approve and merge the PR into the main branch.
4. The publisher pipeline will be triggered and publish the changes to the current “dev” environment.
5. Once the changes are reviewed and validated, there are manual approvals to promote the changes to higher environments like UAT or production.

In the second scenario (from an IDE), the workflow is slightly different:

1. Do the changes in the IDE and commit them to a dev branch.
2. Manually create a PR to the main branch.
3. Approve the PR and merge it to the main branch
4. The publisher pipeline will be triggered and publish the changes to the current “dev” environment.
5. Once the changes are reviewed and validated, there are manual approvals to promote the changes to higher environments like UAT or production.

You can see the difference between both scenarios is in the first steps, and also that when using an IDE the extractor pipeline isn’t used because we’re doing the changes in code that’s checked-in to the dev branch.

Configuring APIOps

Now that we know how APIOps works, it’s time to configure everything. But first, here are some links that contain all the information and we’ll need:

• APIOps GitHub project
• APIOps documentation

The scenario I’ll describe only has a development and production APIM instance, but I’ll explain how to also add a UAT instance between dev and prod.

Starting scenario

For this example, we’ll have a “source” APIM instance with a single API called customers that has an update method:

The APIM is using a policy to authenticate to Dynamics 365 F&O. We want all of these to move into the production APIM!

Upload base files

The first step will be preparing the Git repo in Azure DevOps, so head to the releases page of APIOps’ GitHub project and look for the latest version. Beware of not using one of the many alpha versions! For this post, I’m using v5.1.4, which is the latest non-alpha version.

Download the Azure_DevOps.zip file from there, unzip it and upload the contents to your Git repo in Azure DevOps. Once uploaded, your repo should look like this:

This ZIP file contains the YAML files for the extractor and publisher pipelines.

Create a service connection

We need to create a service connection in Azure DevOps, pointing to the Azure that contains the APIM resources. Go to your project settings and select “Service connections” in the left menu under pipelines.

Create a new one, select “Azure Resource Manager”, then “Workload Identity federation (automatic)”, and in the next step you must choose your subscription, resource group where the resources are and give the connection a name. You can use the subscription name plus resource group to identify it easily.

Create variable group

Go to the Pipelines menu, then Library and create a new variable group called apim-automation and the following variables: SERVICE_CONNECTION_NAME, APIM_NAME, RESOURCE_GROUP_NAME, RESOURCE_GROUP_NAME_Prod and apiops_release_version.

Now for the values use:

• SERVICE_CONNECTION_NAME: the name of the connection you created in the previous step.
• APIM_NAME: the name of your dev APIM.
• RESOURCE_GROUP_NAME: the azure resource group name where your dev APIM is.
• RESOURCE_GROUP_NAME_Prod: the azure resource group name where your production APIM is.
• apiops_release_version: the version of the release from the APIOps you’re using. In my case, it is v6.0.1.1. You need to specify it with the points, so it would be “v6.0.1.1”.

Create environment

Now, in the left menu, still in the Pipelines section, go to “Environments”, and create a new one called “Prod”. We will use this in the approval steps of the pipelines. Change to the “Approvals and checks” tab on top, and add a new check of the type “Approvals”:

Add someone as the approver, I’ll add myself for the example.

Create config file

Now go back to your repo and create a new file in the root. Call it configuration.prod.yaml with this contents:

apimServiceName: apim-aristeinfo
namedValues:
loggers:
diagnostics:
backends:
apis: customers

In the apimServiceName element add your production’s APIM name. And to the apis element your API name.

The config file is used to replace the values of the dev APIM for the ones that the prod instance will have. like named values that use Key Vaults, loggers etc. You can learn all from the config file here!

Now, also in the root of the repo, create a folder called artifacts. This folder will be used to store the extracted configuration of your dev APIM instance.

Create the extractor pipeline

Now it’s time to create two pipelines from the YAML files we uploaded in the beginning, an extractor and a publisher pipeline. Go to the pipelines menu and create a new one.

We start with the extractor pipeline. Select “Azure Repos Git”:

Then select your repo, and in the next step “Existing Azure Pipelines YAML file”:

And select the run-extractor.yml file:

Save it, don’t run it yet, and then go to pipelines, select it and click “Run pipeline”. We will get some fields to fill in:

We must fill in these fields:

• APIM instance name: the name of your dev APIM.
• APIM instance resource group name: the resource group name where your dev APIM is.
• APIM repository for pull request: the name of your Azure DevOps repo.
• Folder where you want to extract the artifacts: the folder we created in the root of our repo, which we named artifacts.

Now just click run… and there’s a chance your pipeline won’t start running! Click the first stage in the API and check whether you’re getting this message: “This pipeline needs permission to access 2 resources before this run can continue to Create artifacts from portal”.

Click view and “Permit” if you have anything to allow. If you didn’t select the “Grant access permission to all pipelines” checkbox when you created the service connection you will get these two items to permit:

If you did select the checkbox, you will only get the variable group permit. Now let’s see if the pipeline run works…

Most likely you’ll see a successful (with a warning) first stage, and a failed second stage, the one called “Create artifacts pull request”. This is due to missing permissions. Click the second stage, and go to the step with the error. You will see a line that says: “You need the Git ‘GenericContribute’ permission to perform this action.”, followd bu an identity like Build\3d382728- and more numbers 🙂

Copy that GUID, go to your project settings, then repositories, the security tab on top, and paste it in the search box:

And set to “Allow” the following permissions: Contribute, Contribute to pull requests, Create branch and Create tag.

Now go back to the pipelines, enter the failed run, and rerun the failed job in the second step:

Now hopefully the pipeline will create a pull request. So go to Repos, Pull Requests, switch to the active tab and yours should be there. But leave it there for now.

Create the publisher pipeline

First, install the “Replace Tokens” extension for Azure DevOps to your organization. It’s required and the publisher pipeline will fail otherwise.

Now we have to create the publisher pipeline. We’ll do the same as with the extractor one, create a new one, from an existing YAML file, but this time select the “/tools/pipelines/run-publisher.yaml” file.

This one is a CI pipeline, that will run after every commit or merge to the main branch.

We’re done here. Yes, that was all. Now back to the Pull Request.

Merge the Pull Request

Open the pull request and go to the “Files” tab. You will see that it has created a structure of folders and files with the definition of your dev APIM.

Also, if you go to the files in your repo, and click the branches, you’ll see that the pull request has been created in a different branch:

And that’s why we must merge it to main. Go to the PR again and approve it, or just complete it, it’s your choice 🤪

Once the PR is completed, the publisher pipeline will start… to ask for more permissions! Like the extractor one, we have to permit changes to the variable group and service connection:

Now the first stage will publish the changes to the dev APIM instance. At this point, you can go to your dev APIM to validate and test the changes.

Remember we created an environment with an approval before? It will be used now before publishing the changes to the prod APIM instance:

Click Review, and approve it, and the publisher pipeline will apply the changes to the production pipeline.

And that’s all, I hope this post has been useful to start “playing” with the APIOps concept and automate you APIs lifecycle even more! This is quite “basic” and I’d like to write some more content about the config file and how to replace some values in policies using named values for example. I’ll leave that for another post!