API Integration 101

Over the past month, in my current role as Head of Platforms and Integrations, I have had the opportunity to interview over 50 candidates for API Integration roles. Through these interviews, I’ve noticed a knowledge gap in certain areas which include system design patterns, API user management, integration concepts, and API specifications.

While it’s not possible to cover all topics pertaining to APIs and integration in a single article, I’d like to focus on a few key concepts that could pique the interest of individuals who are eager to learn and grow in this field.

A multi tenant system which enables third parties as well as internal systems use the same services.

OSI model – The foundation of all computer communication

The OSI (Open Systems Interconnection) model can be best described as the universal language for computer networking and it is a theoretical way of describing how different layers of a network communicate with each other. There are seven (7) layers in the OSI model which have their specific functions and protocols however within this article, we will focus on two layers i.e the application & network layer.

Understanding API documentation

APIs (Application Programmable Interface) are a set of clearly defined methods of communication between two pieces of software. The most important step of software integration is understanding the API user guide, which is often embedded within the API documentation. Read more on API and its use cases.

The API documentation usually defines the following :

1. API Specification
2. Schemas (Data format)
3. Security & Connection layer (application layer (API key, OAuth ), network layer (IP address & port))

API SPECIFICATION

The API specification describes the functionality and expected behaviour of an API. It standardises the data exchange between web services, examples of these specifications include GraphQL, REST, SOAP etc. Let’s use REST APIs as a case study for expatiating on API .

REST APIs

REST stands for Representational state transfer. In this specification, clients and servers exchange data using HTTP.

HTTP

The hypertext transfer protocol is the foundation of the World Wide Web, and it is used to load webpages using hypertext links. The HTTP is an application layer protocol designed to transfer information between networked devices and runs on top of other layers of the network protocol stack.

A simple flow over HTTP occurs by a client(user) making a request to a server which then responds with message.

HTTP protocol requests are made up of a URL, method, headers and body.

• Methods can sometimes be referred to as verbs which describes the action to be taken by that HTTP request. These include POST, GET, PUT, PATCH and DELETE but also known as CRUD operations: create, read, update, delete.
• Headers represent additional information being requested (meta-information) along to the server. E.g Content-type= application/json, authentication token.
• A Body contains the data a client sends to the server as a request, in order to retrieve information in the form of a response.

SCHEMAS

This how the data in your request and response is defined. Examples include JSON, XML, with the most common one being JSON — JavaScript object notation.

SECURITY – API USER MANAGEMENT

This is an important element of API management: managing the users of API resources, in terms of what and how much access each user has to the server.

Two most common types are: API Keys and OAuth (client ID & secret key).

OAuth is an open protocol for authorisation that allows users to share their private resources to a third party. There are two versions: OAuth 1.0 (deprecated) and OAuth 2.0 (active) with 2.1 in draft. OAuth provides a client with a username and password (aka client ID & secret key) to access server resources in a process called the authentication and authorization. OAuth is provisioned with different environments namely sandbox and production. Users can test all API features using the client ID and secret key provided for the sandbox environment to login and complete testing. Once all is complete in test, the client ID and secret key must be generated for the next environment which is Production. The API gateway also serves as a protection layer against DDOS, by rate limiting API calls per set time limit.

An API key is simply a string of alphanumeric code used to identify & authorize an application user. It is often passed in the request header and can be extracted or intercepted by bad actors. To protect your users against this loophole, it is advisable to adopt a second layer called IP whitelisting on the network layer of this connection.

IP Whitelisting

Creating an IP whitelist for your API key ensures the key is only valid when the API is being accessed from a registered and pre-approved IP address.

IP connection whitelisting occurs at two layers of the OSI model i.e the network and application layer. On the network layer, it is important to whitelist the client side IP and port addresses on the server side. Whilst on the application layer, the API user will store & send each request with the API key in the header .

As an Integration PM, you are expected to understand all of the above and then analyse each endpoint, their functionalities, and how it will all fit your business requirements. Once the alignment between business requirement and API feature is completed, then an Integration document should be created and must include certain key elements which are defined under the concepts of integration.

Concepts in API integration

1. User Flow i.e UML, unified modeling Language
2. Frontend development
3. Backend development
4. Reconciliation and settlement
5. Commission configuration

To break these concepts down, let’s define a use case.

An example could be an online order management system that requires payment acceptance. In this use case, there is a requirement for the integration of a payment service provider. The requirement here is clear and the order management system needs a flow whereby it’s users can complete a purchase online and receive value.

1. User flow: conceptually this can be defined using UML’s, to visualise the flow between the user, actions taken, the request and the response involved. I often use flowcharts to communicate user flows to my development team.
2. Frontend development: In this use case, the first step is to conceptually define how this payment system will flow into your order management user experience. The next step will be to build the UI which essentially defines the scope of the integration. The UX designer must have built a low or high level flow to incorporate both pieces of software to enable a user seamlessly complete a purchase. A front end engineer will bring the design to life by hardcoding and applying each step to its backend endpoints.
3. Backend development: There is a bit of system design pattern consideration here. Assumption, where the order management system leverages a micro service architecture, the decision to be made will be to develop a payment micro service with a server-less DB. This Server less DB will store all attributes of the payment service provider and connect the order check out directly to the third party service provider (Payment service). This backend integration process is not a one size fits all but is highly adaptable. There are many ways to go about the backend integration depending on the system architecture, dependencies, patterns and other such factors.
4. Reconciliation & settlement: This essentially describes the checks and balances between two systems. In our use case, payment occurs on the order management system however it is processed by the payment service provider. There are going to be successful, unsuccessful and pending transactions between these two systems and this process has to be defined clearly to enable both parties to understand how to handle these cases efficiently. An example of settlement processes include T + 1 which means settlement happens after a day on all accounts.
5. Commission configuration: This is not a consideration for all integrations however for our use case it is popular. Automating this process also ties back into the reconciliation and settlement process. Developing the commissions flow into the reconciliation process is key and very important in the case of charge backs, refunds etc.

A few companies have innovated around integrations for some use cases (ERP Integration, data loading and flow building) thereby birthing the concept of an Integrated platform as a service (IPAAS). The IPAAS enables low code integrations between two pieces of software i.e backend services or server side applications.

Key elements of the IPAAS

1. Connectors: These are pre-built application instances whereby users can enter a username and password (client ID and secret key) and complete an integration.
2. Fields and mapping: This enables each endpoint to be mapped one to one i.e orders will be mapped to inventory between two systems.
3. Universal connectors: These are HTTP, RestAPI, GraphQL, webhooks etc
4. API gateways: It acts as an entry point for all client requests to your systems APIs. But also protects your resources against compromise.
5. White labeling: The IPAAS serves as an integration layer which can be used to expose an instance of your software in a multi-tenant type of system.

Technology and innovation are like two peas in a pod. It’s best to always keep your ears peeled and constantly rethink the art of software development.

Technology development is 60% art and 40% engineering. The art is in imagining and defining how the solution will work, the engineering is in bringing that solution to life.