Auto-Merging APIs and Microservices Specifications to Ease API Integration

In the API world, the vast majority of external services will be composed of numerous microservices internally. While trying to hide this complexity from API consumers, we must deal with combining the various microservices’ specifications into a single unified interface.

In this how-to, I will illustrate how easy it is to automatically merge API specification documents of each microservice to create one composite API definition which will allow auto-generation of developer experience components like API portal/documentation, SDKs, etc. I will also mention some approaches available to automatically filter out endpoints and their related data from the documents before they are merged.

Why Would Businesses Need to Merge Their APIs?

Businesses use the microservices architectural pattern to split their core services into smaller, independent units. This increases efficiency and reduces development and maintenance costs. However, from a consumer point of view, these microservices semantically represent a single service. Providing unified API documentation with a single SDK per language and only one machine-readable API specification will significantly reduce the time and effort it takes for them to integrate with their service as compared to integrating with multiple APIs.

Therefore, how a service is structured internally in a business should be abstracted away from the end-users to ease API integration and consequently increase API adoption rates:

Combining microservices is a tricky challenge. Businesses try to tackle this by either configuring an API gateway or by manually combining each API specification document associated with the microservice. The gateway approach is overkill for services that require simple concatenation without requiring any additional configurations. On the other hand, the manual approach is error-prone, time-consuming and managing updates is a headache.

Merging API Specifications of Microservices Using APIMatic

APIMatic lets you automatically merge ‘N’ specifications into a single specification. The input specifications can be any combination of specifications supported by APIMatic such as RAML, OpenAPI, Postman Collections, etc. You can then use the merged API definition to generate an API developer portal bundled with client SDKs, language-specific documentation, live code samples, and more. 

Characteristics of Specification Documents of the Same Business

Before diving headfirst into merging, here are some high-level characteristics that specification documents of microservices (e.g. OpenAPI) belonging to the same service share:

Different Base Server URLs

Since each microservice targets a different business capability or resource, their base server URLs will likely share some initial common segments but can be overall unique. Upon merge, therefore, this server configuration should remain intact.

Unique Endpoints

The endpoints/operations of each microservice will likely work with different resources and so are expected to be unique. Merging them would most probably just be a simple concatenation.

Overlapping Schemas

Because the services belong to the same business, there can be a large number of overlapping schema definitions in the documents. Upon merge, these definitions should not be duplicated.

Equal or Varying Authentication Schemes

The authentication schemes used by the services may or may not vary, therefore, all schemes present in the documents should be preserved upon merge.

Step-by-Step Walkthrough of Merging Two API Specifications Using APIMatic

To illustrate merging in a better way, I provide an example of a hypothetical company “MusicPlay” that offers a music application to its users where they can browse and listen to tracks of their choice and even create personalized playlists. Their database holds a large number of artists, albums, and tracks that any user can listen from.

MusicPlay internally has two microservices: a User API service that solely manages user accounts and their playlists, and a Library API service that sits on top of the database and helps users query for artists, albums, or tracks as well as lets artists add/remove albums and tracks, etc.

I will now take you through the steps to merge the two API definitions User API and Library API using APIMatic in order to create unified API documentation and SDKs. The OpenAPI specification documents I use for this purpose can be found here and here respectively. 

Step 1: Create a Local Directory Structure for Specifications

Merging requires me to place the specifications in an appropriate directory structure which I create as discussed below. If you want, you can check out what the final structure looks like here. 

Create a Root Directory

In my local file system, I create a root directory named MusicPlay Merge Example. This directory will hold the specification documents as well as the configuration file required to enable merging.

/MusicPlay Merge Example

Create a Subdirectory for User API

Each API specification that needs to be added should have its own dedicated subdirectory inside the root directory. Accordingly, I create a subdirectory named User API in the directory MusicPlay Merge Example and place the User API’s OpenAPI specification file in it as shown below:

/MusicPlay Merge Example
   /User API
       MusicPlay User API.yaml

Create a Subdirectory for Library API

Similar to the previous step, I then create another subdirectory in the root directory named Library API in which I place the OpenAPI specification file for the Library API/microservice:

/MusicPlay Merge Example
   /Library API
       MusicPlay Library API.yaml
   /User API
       MusicPlay User API.yaml

Step 2: Add Configuration Metadata Files

Metadata File to Enable Merging

I then create an empty JSON file in the root directory and name it APIMATIC-META.json. The name is important as it ensures that the file is auto-detected as the APIMatic’s metadata file during import:

/MusicPlay Merge Example
   /Library API
       MusicPlay Library API.yaml
   /User API
       MusicPlay User API.yaml
   APIMATIC-META.json

I add the following configurations in the metadata JSON file:

{
  "MergeConfiguration": {
      "MergeApis": true,
      "MergeOrderOfDirectories": ["User API", "Library API"],
      "MergedApiName": "MusicPlay API",
      "MergeSettings": {
          "ConflictStrategy": "KeepLeft",
          "DescriptionConflictStrategy": "KeepBoth"
      }
  }
}

Some details about the configurations and their effect on the two APIs being merged are given below:

• MergeConfiguration defines the overall merging process for your API specifications.
• Setting MergeApis to true enables merging. Without this, the first API description detected in the directory will be picked up and any other API descriptions will be ignored.
• MergeOrderOfDirectories will control in which order the two API descriptions will be merged. The order matters as the items from the first (left) API are picked up first and the items from the second (right) API are added after those of the first. The order also plays an important role when resolving merge conflicts as will be explained under the ConflictStrategy merge setting. For the current example, although alphabetically Library API the first API directory and User API the second API directory, User API will be treated as the first API because of the order specified in MergeOrderOfDirectories.
• MergedApiName helps assign the name “MusicPlay API” to the merged API definition. If this was not provided, then by default the name of either the left API or right API would have been picked up depending on the order and conflict strategy used for merging the two APIs.
• MergeSettings controls the process that merges User API and Library API definitions.
• KeepLeft conflict strategy in the ConflictStrategy merge setting ensures that if a conflict occurs, information of the left User API will remain intact while that of the right Library API will be updated or discarded as applicable.
• DescriptionConflictStrategy is an alternate conflict strategy for descriptions. Its KeepBoth value ensures that if two conflicting descriptions are found in the User API and Library API, then both will be preserved.

Metadata File to Enable Endpoint Filter for Subdirectory User API

The User API contains an operation getAllUsers which is tagged as internal in the OpenAPI file as it is not meant for public use. Therefore, the merged API definition should not contain this endpoint. To filter it out, I add a configuration metadata JSON file APIMATIC-META.json in the subdirectory User API as shown below:

/MusicPlay Merge Example
   /Library API
       MusicPlay Library API.yaml
   /User API
       MusicPlay User API.yaml
      APIMATIC-META.json
   APIMATIC-META.json

I then add the following to the metadata file to enable filtering:

{
  "RemoveEndpointsWithTags": ["internal"]
}

RemoveEndpointsWithTags will ensure that during merging, the endpoints that contain the tag internal and their related data will be removed.

Step 3: Zip the Directory

The contents of the directory MusicPlay Merge Example are now ready for merging:

/MusicPlay Merge Example
  /Library API
      MusicPlay Library API.yaml
  /User API
      MusicPlay User API.yaml
      APIMATIC-META.json
  APIMATIC-META.json

I zip them into a final MusicPlay.zip file. Take note that it is important to ZIP the contents of the root directory and not include the root directory itself.

Step 4: Upload ZIP File into APIMatic

Next, I login into APIMatic. 

Logging into APIMatic will open the APIMatic Dashboard for me. I click on the Import API option as shown below:

Here, I browse and upload my MusicPlay.zip file and then click on Import:

During import, the ZIP file I uploaded undergoes merging behind the scenes. As part of this, the individual API specifications as well as the merged API specification are validated. The merging completes successfully and I’m shown some informational validation messages linked to the individual as well as merged API definitions:

Since the validation messages are not errors/warnings, the import was successful. So, I go ahead and click on Proceed. The merged API definition will then become visible in my Dashboard as shown below:

Notice how it has the name MusicPlay API which I provided earlier in the configuration file.

Step 5: Preview Developer Experience Portal

From the Dashboard entity of the merged API definition, I click on the Generate button and then click on Preview API Portal:

I will be taken to a preview mode of the unified API documentation. Here, I can choose to open up the HTTP documentation or even any language-specific documentation. I can also generate SDKs or publish and customize the portal to suit my needs better.

Analysis of the Merged Documentation

To illustrate what the merged output for my MusicPlay APIs looks like, I decided to compare a few components below. For each component, I show what the input data from the APIs was and what the same information looks like in the autogenerated HTTP documentation after merging.

Basic API Information

MusicPlay User API basic API information:

openapi: "3.0.0"
info:
  version: 1.0.0
  title: MusicPlay User API
  description: Helps manage user accounts on the MusicPlay application.
servers:
  - url: https://musicplay.io/api/v1/users
security:
  - oauth2: []

MusicPlay Library API basic API information:

openapi: "3.0.0"
info:
  version: 1.0.0
  title: MusicPlay Library API
  description: Provides data related to the MusicPlay library including artists, tracks, etc.
servers:
  - url: https://musicplay.io/api/v1/library
security:
  - oauth2: []

Merged output:

API Descriptions

As can be seen in the Introduction section, descriptions of both the User and Library APIs were combined because of the KeepBoth description conflict strategy configured earlier. Notice that the description from the User API has been added first as it was chosen to be the left API in the merge order.

API Servers

Production servers https://musicplay.io/api/v1/users and https://musicplay.io/api/v1/library from both APIs are added to the final server configuration.

API Authentication

MusicPlay User API authentication configuration:

components:  
  securitySchemes:
    oauth2:
      type: oauth2
      flows:
        authorizationCode:
          authorizationUrl: https://musicplay.io/api/oauth/authorize
          tokenUrl: https://musicplay.io/api/oauth/token
          scopes:
            write:user: modify user profile in your account
            read:user: read user profile
            write:playlist: modify user playlist
            read:playlist: read user playlist
            write:playlist-track: modify user playlist track
            read:playlist-track: read user playlist track

MusicPlay Library API authentication configuration:

components:
  securitySchemes:
    oauth2:
      type: oauth2
      flows:
        authorizationCode:
          authorizationUrl: https://musicplay.io/api/oauth/authorize
          tokenUrl: https://musicplay.io/api/oauth/token
          scopes:
            write:artist: modify artist profile
            read:artist: read artist profile
            write:album: modify artist album   
            read:album: read artist album
            write:track: modify track details
            read:track: read track details

Merged output:

Since both APIs used the same OAuth 2 grant flow, the scopes of both APIs were merged together. Scopes from the left User API are shown first and then the scopes of the right Library API are shown:

Models

MusicPlay User API model definitions (simplified):

components:
  schemas:
    UserProfile:
      type: object
      properties:        
        id:
          type: string
        name:
          type: string
        joined:
          type: string
          format: date-time
        bio:
          type: string
        profilePictureUrl:
          type: string
        playlists:
          type: array
          items:
            $ref: "#/components/schemas/UserPlaylist"
    UserPlaylist:
      type: object
      properties:      
        id:
          type: string
        name:
          type: string
        description:
          type: string
        createdBy:
          type: string
        createdOn:
          type: string
        tracks:
          type: array
          items:
            $ref: "#/components/schemas/UserPlaylistTrack"
        isPublic:
          type: boolean
    UserPlaylistTrack:
      type: object
      properties:      
        id:
          type: string 
        playlistId:
          type: string
        title:
          type: string
        addedBy:
          type: string
        addedOn:
          type: string
          format: date-time
        track:
          $ref: "#/components/schemas/Track"
    Artist:
      type: object
      properties:
        id:
          type: string
        name:
          type: string
        bio:
          type: string
        albums:
          type: array
          items:
            $ref: "#/components/schemas/Album"
        profilePictureUrl:
          type: string
    Album:
      type: object
      properties:
        id:
          type: string
        name:
          type: string
        artistId:
          type: string
        createdOn:
          type: string
          format: date-time
        tracks:
          type: array
          items:
            $ref: "#/components/schemas/AlbumTrack"
    AlbumTrack:
      type: object
      properties:      
        id:
          type: string 
        albumId:
          type: string
        title:
          type: string
        addedBy:
          type: string
        addedOn:
          type: string
          format: date-time
        track:
          $ref: "#/components/schemas/Track"
    Track:
      type: object
      properties:
        id:
          type: string
        title:
          type: string
        duration:
          type: number
          format: int64 
        artist:
          $ref: "#/components/schemas/Artist"                      
    Error:
      type: object
      required:
        - code
        - message
      properties:
        code:
          type: integer
          format: int32
        message:
          type: string

MusicPlay Library API model definitions (simplified):

components:
  schemas:
    SearchResult:
      type: object
      properties:
        title:
          type: string
        id:
          type: string
        type:
          $ref: "#/components/schemas/LibraryComponentType"
        album:
          $ref: "#/components/schemas/Album"
        artist:
          $ref: "#/components/schemas/Artist"
        track:
          $ref: "#/components/schemas/Track"
    LibraryComponentType:
      type: string
      enum:
        - Artist
        - Album
        - Track
    Artist:
      type: object
      properties:
        id:
          type: string
        name:
          type: string
        bio:
          type: string
        albums:
          type: array
          items:
            $ref: "#/components/schemas/Album"
        profilePictureUrl:
          type: string
    Album:
      type: object
      properties:
        id:
          type: string
        name:
          type: string
        artistId:
          type: string
        createdOn:
          type: string
          format: date-time
        tracks:
          type: array
          items:
            $ref: "#/components/schemas/AlbumTrack"        
    AlbumTrack:
      type: object
      properties:      
        id:
          type: string 
        albumId:
          type: string
        title:
          type: string
        addedBy:
          type: string
        addedOn:
          type: string
          format: date-time
        track:
          $ref: "#/components/schemas/Track"
    Track:
      type: object
      properties:
        id:
          type: string
        title:
          type: string
        duration:
          type: number
          format: int64 
        artist:
          $ref: "#/components/schemas/Artist"
    Error:
      type: object
      required:
        - code
        - message
      properties:
        code:
          type: integer
          format: int32
        message:
          type: string

Merged output:

Overlapping models like Artist , Album, Track, etc. were not duplicated in the merged API definition. Just like with other components, models from the left User API are listed first and then the models from the Library API are shown.

API Endpoints

MusicPlay User API endpoints (simplified):

paths:
  /:
    get:
      operationId: getAllUsers
      security:
        - oauth2: ["read:user"]
      tags:
        - users
        - internal
      responses:
        '200':
          description: User profile object.
          content:
            application/json:    
              schema:
                type: array
                items:
                  $ref: "#/components/schemas/UserProfile"
        default:
          description: Unexpected error.
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"   
  /{user-id}:
    parameters:
    - name: user-id
      in: path
      required: true
      schema:
        type: string
    get:
      operationId: getUserProfile
      security:
        - oauth2: ["read:user"]
      tags:
        - user profile
      responses:
        '200':
          description: User profile object.
          content:
            application/json:    
              schema:
                $ref: "#/components/schemas/UserProfile"
        default:
          description: Unexpected error.
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"

MusicPlay Library API endpoints (simplified):

paths:
  /search:
    get:
      operationId: searchLibrary
      parameters:
      - name: searchQuery
        in: query
        required: true
        schema:
          type: string
      - name: types
        in: query
        schema:
          type: string
      tags:
        - library
      responses:
        '200':
          description: Search results.
          content:
            application/json:    
              schema:
                type: array
                items:
                  $ref: "#/components/schemas/SearchResult"
        default:
          description: Unexpected error.
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"

Merged output:

The endpoints are simply concatenated. Endpoint groups (or tags) from both APIs are listed in the API Endpoints section on the left. Endpoints from the User API are listed first since it was chosen to be the first API to be merged, in the merge order. Note also, that the tag internal is not part of the final tags/groups as it was filtered out during the merge process.

Useful Links

The full details of how merging works in APIMatic with sample walkthroughs and available configurations are documented here.

API descriptions for the two APIs “User API” and “Library API” (complete + simplified versions) as well as ready-to-merge directory structures for them can be found in my GitHub repo here. 

Conclusion

To ease API integration and maximize API adoption rates, bring your services together under a single Developer Experience portal where your consumers will have direct access to language-specific documentation, live code samples, and SDKs per language. Reduce any manual efforts to get there by automatically merging specification documents that accompany your microservices.