Blog

A Survey of API Management Platforms

In my previous blog, I discussed how I landed up interning at Dentsu. I also discussed that I worked on scouting and building a POC for a cloud agnostic, open source API management tool/platform which could help in setting up API design, gateway, store, and analytics. In this blog, I will be jotting down my work in much more detail.

We will be exploring four API Management platforms, namely:

Our main requirements which will help us in evaluating an API Management tool are:

  1. Documenting our APIs 
  2. Building an API Gateway 
  3. Securing our APIs  
  4. Versioning APIs

I explored three platforms which can serve the above-mentioned use cases and we will discuss below our findings for the same. With documentation being our foremost use case, we will focus on the platform which can solve the same, in the most efficient way. 

APIGEE 

Apigee API Management

It’s an API management platform which gives developers access to manage API calls, throttle traffic, control error handling, cache, build developer portal, etc. 

API Documentation 

The platform uses SmartDocs feature to document the APIs. To use Smartdocs, one must build a Developer Services Portal (DSP). One of the most recommended option (by apigee) to build a DSP is to use Drupal. Drupal brings along its Content Management Capabilities and apigee plugs its own batteries like API Products (more on that later), Stats, Developer and Company Apps, etc. to give the DSP a rich set of functionalities.   

Having said all of that, there are quite a few caveats to it. The first and foremost being, building a Drupal based portal. Besides being a tedious task to set it up which also requires a full-fledged hosting service to host the developer portal, it adds a thick layer of friction to our relatively simple use case (documentation of APIs). Another requirement in the developer portal is to build API Products. API Products are nothing but bundled APIs for different services that solve specific use cases. API products deliver API bundles with customizable scopes, environments, authentication, authorization, etc.  

Hands-on Experience

Managed to setup the developer portal. Very non intuitive interface to add APIs on their developer portal. Let’s take a simple case: I have written a function/class based API and defined its route as well in my code base. I couldn’t find anyway to add that endpoint to the API product. Now, it’s hard to understand that if I cannot add anything in my API product how will it help me to attach the API product to the developer portal. 

Later, I got a hang of the entire work flow. However there are two caveats (will list them down after writing the workflow).

  • Write Specs (using OpenApi Specification). For this, the API developer must know all the parameters, response codes, and other metadata. If the API changes later on, the spec needs to be changed (manually).
  • Define API proxy for the spec defined above.
  • Build an API Product and attach the API Proxy to the product.
  • Build the Developer Portal and attach the API Product to it. There could be multiple products attached to a portal.
  • Each stage gives you its own settings to tinker around. For example – you can set the environment (test or prod), throttling limit, developer portal’s access limitation (private, internal, public) etc.

The two caveats:

  • The evaluation plan (free plan) gives access to only 1 user and supposedly 1 environment. Hence, only 1 user can build or operate the APIs on the platform. The provision of only 1 environment is also a tad bit dicey. I got two options (prod and test) but not entirely sure about this.
  • The developer portal is the integrated developer portal given by apigee platform. This means that it comes with a lot less flexibility and fewer features. Foremost being, the URL of our portal will be hosted on apigee and won’t be customizable (for ex- https://shubhanshugupta-eval-peopleportal.apigee.io/). 

POSTMAN

Postman API Management

Postman is a popular API development environment. It provides an easy interface to design, debug, do automated testing, document, monitor and publish the APIs.  

API Documentation 

Just like API Products discussed above, Postman has ‘collections’. A postman collection, allows you to save your requests in such a way that they can be reused, shared and bundled. In order to generate API documentation, one only needs to add some specific (and optional) comments. Postman automatically generates 

  • Sample requests, headers and other metadata 
  • Descriptions associated with requests, folders and collections 
  • Generated code snippets in several programming languages 

Postman provides workspaces in which teams can share and collaborate. The documentation of an API collection can be shared within workspaces and can be made open for public consumption or can be limited to internal team.  

APIARY

Apiary helps in creating a mock API framework wherein we can first design an API and then implement it as well. With Apiary, API design can be mocked – with mock input and output. This mocked API, can then be integrated and tested to meet business use case.  

API Documentation 

Apiary gives a provision of API Blueprint. It’s a High-level API description language which helps in its design and documentation. After generating a blueprint of an API, Apiary also gives a provision of a tool called ‘dredd’ which is used to run local tests for validating the blueprint. Like Postman, documentation of APIs in Apiary can be achieved by writing the API description and rest of the details (headers, requests, metadata, etc.) get automatically populated by Apiary. 

WSO2 API Manager 

Image result for wso2
WSO2 API Management

The platform is similar to apigee with lot less burden of understanding the heavy documentation and is quite intuitive to setup as well. Everything good about it except that its trial (or free) plan lasts for only 2 weeks post which we will need to pay (starting from $129/month).

Survey Analysis and Results 

Among these three platforms, I found that Apigee provides an entire spectrum of design, development, gateway, and security for APIs. However, all of this comes with a heavy on-boarding cost and friction, especially when the team has progressed many levels of a software development lifecycle. Also, when the business is not entirely based on selling a product which the customers can access only via APIs, Apigee might be a little more resource intensive especially when the requirement is relatively simpler.  

Apiary and Postman are great platforms, exclusively for API design, development and testing. They provide features like API gateway, and security in collaboration with platforms like AWS. Even with regards to API documentation, Postman provides a more seamless and smooth experience in comparison to Apiary. Developers need to write API blueprints in the case of Apiary in a pre-defined format prescribed by the platform. Whereas, if Postman is already a preferred tool for API development and testing, the documentation process becomes a breeze.  


Thanks for reading and I have written other posts related to software engineering and data science as well. You might want to check them out here. You can also subscribe to my blog to receive updates straight in your inbox.

What am I doing right now? Internship.. Studies.. or Both?!

My experience of hunting for and landing an internship in Singapore.

Bhagavad Gita

सुखदु:खे समे कृत्वा लाभालाभौ जयाजयौ |

ततो युद्धाय युज्यस्व नैवं पापमवाप्स्यसि ||

Chapter 2 Verse 38, Bhagavad Gita

Shree Krishna says Fight for the sake of duty, treating alike happiness and distress, loss and gain, victory and defeat. Fulfilling your duty and responsibility in this way, you will never incur sin.

Arjuna’s was apprehensive that by killing his enemies, he would incur sin. Shree Krishna addresses his apprehension and he advises him to do his duty (dharma), without attachment to the fruits of his action. Such an attitude will release him from any sinful reactions.

To be honest, I was/am not planning to kill anybody as in Mahabharat. However, as soon as my semester at NUS started in August, I knew that getting into a comfort zone is going to be my biggest enemy and if I don’t kill it, I will become complacent. Therefore, I began my hunt for a good internship opportunity early in the semester. Over a period of time, when you just apply and apply, and don’t hear from anyone, it’s a little disheartening. Some wanted immediate joining, some were looking for summer interns, etc. However, luck turned its tide and soon, I started getting online test invites. I knew that good performance in online coding tests usually translates to face-to-face (f2f) interviews and I was fortunate enough to be able to crack almost all the coding tests.

Coding Test Format

Most of the firms had two coding questions: a general algorithmic question (I used to code in Python) and a SQL coding question. Some of the firms also gave MCQs related to product analysis, data analysis, probability, etc along with the two coding questions.

Interview Experience and Format

The f2f interviews, majorly test whether the candidate is genuine in portraying his/her persona and knowledge by walking through the resume and the coding test. Below, I will give an account of 2 of my 3 interesting interview experiences of which I accepted one.

All of my f2f interviews were heavily concentrated on my past experience. All except Grab, wherein I had two f2f interviews. The interviewer in my first f2f interview dug into my past Software Development experience and very subtly steered the conversation to a made up scenario based problem solving session. I was asked to do a storyboarding concept of how can I visualize customers pain points on the app, given I have unlimited access to all kind of data. This was an open ended question which had numerous possibilities and not one correct approach. I was literally drained of energy in my second f2f experience, which I took remotely from my home (in India). The interviewer was a entrepreneur himself, and he was pretty interested in my past entrepreneurial journey and learnings.

I also appeared for a coding test (on codility) and an f2f interview for Global Data Innovation Center (GDIC) at Dentsu Aegis Network, Singapore. My f2f interview was taken by two senior members of the GDIC and they grilled me quite well on my past experience. I guess, with good scores in coding tests, and a good past experience of doing substantial work back in India helped me crack almost all the f2f interviews. I subsequently received offer from 2 firms, and finalized my internship offer at GDIC, Denstu Aegis Network.

Internship Experience Preview

With a long commitment (full time during December and part-time from January to April) in sight, I began my internship in December 2018. I initially worked on scouting and building a POC for a cloud agnostic, open source API management tool/platform which could help in setting up API design, gateway, store, and analytics. Since it’s all Open Source, I will probably compile all my findings in a separate blog. This project helped me in getting acquainted with the code base and I could leverage domain knowledge of APIs and API management from my previous Software Development experience.

Sharing the knowledge that I gained through my first project at GDIC

After completing the first assignment and getting acquainted to the projects in GDIC, I was asked to contribute in one of the GDIC’s ambitious, research oriented project which was quite unprecedented as per the competitive landscape. Therefore, I began my journey in Project Pearl. What’s Project Pearl? Well..I promise that it’s going to be an interesting read as well and I will continue that in my next blog. So stay tuned!

Thanks for reading and I have written other posts related to software engineering and data science as well. You might want to check them out here. You can also subscribe to my blog to receive updates straight in your inbox.

Multi Class Classification in Text using R: Predicting Ted Talk Ratings

Multi Class Classification in Text

This blog is in continuation to my NLP blog series. In the previous blogs, I discussed data pre-processing steps in R and recognizing emotions present in ted talks. In this blog, I am going to predict the ratings of the ted talks given by viewers. This would require Multi Class Classification and quite a bit of data cleaning and preprocessing. We will discuss each step in detail below. So, let’s dive in.

Continue reading “Multi Class Classification in Text using R: Predicting Ted Talk Ratings”

Emotions in Ted Talks: Text Analytics in R

Image result for emotions nlp

This post is in continuation with my NLP blog series. You might want to checkout my previous blog in which I discussed data pre-processing in R. In this blog, I will determine the emotions in the Ted Talks. At the end, I will compute a HeatMap of emotions and talks to aid in our visualization.

So, without further ado, let’s dive in!

Continue reading “Emotions in Ted Talks: Text Analytics in R”

Data Preprocessing in R

Image result for nlp

I have recently got my hands dirty with Natural Language Processing (NLP). I know, it’s a little late to the party but I am at least in the party!

To start with a general overview, I implemented quite a few tasks related to NLP including Text Classification, Document Similarity, Part-of-Speech (POS) Tagging, Emotion Recognition, etc. These tasks were made possible by implementing text pre-processing (noise removal, stemming) and text to features (TF-IDF, N-Grams, Topic Modeling, etc). I implemented these in both R and Python. So, I will try to jot down my experiences in both of these environments. Therefore, I will write this as a blog series, wherein each blog will discuss only one particular thing implemented in one particular environment.

Continue reading “Data Preprocessing in R”

DataKind Singapore

I have recently shifted gears in my life. A shift to academics after spending quite sometime in the industry has been equally exciting and challenging at the same time. Even with all the diversity in the cohort of my own program (in terms of work experience and country of origin) along with a campus hustling with activities all day, I wanted to explore the communities ingrained deep within the culture of Singapore. A natural choice for me, was to look for meetups.

Continue reading “DataKind Singapore”

My learning experience with Google’s Machine Learning Crash Course

Google’s Machine Learning Crash Course (MLCC)

I came to know about Google’s Machine Learning Crash Course (MLCC) from Sundar Pichai’s tweet. I then enquired about it with some close acquaintances working in Google. I was soon pretty convinced of pursuing this course, after their good words about it and my own research on the course content. This post is going to be an account of my learnings from MLCC. I will structure the learnings in such a way that it will look more like a review. I will also include what I really liked about the course and things which I think they can possibly improve, if the creators are planning to update the course content. So, let’s get started!

Continue reading “My learning experience with Google’s Machine Learning Crash Course”

Engineering challenges of streaming a million concurrent JSON data streams from product to CRM

At Truebil, I was fortunate enough to be given an opportunity to solve a unique engineering problem. We had already outsourced CRM development to a third party but I had to integrate the data flow to CRM from our product and back. I mentioned earlier that I was given a unique engineering problem because of the challenges it posed. The challenge didn’t lie in the CRM integration alone, but the fact that Truebil has umpteen number of in-house products which spawn data close to about 1 Million data streams per hour. Besides building a bastion of such magnitude, I knew that it would be equally challenging to work with three different verticals and stakeholders. This post is going to be an account of my experiences dealing with two things. First, how Truebil catered to the transfer of a million data stream to and from CRM without hampering its operations and customer support. The second experience deals in the art of working with different verticals. Not that I have mastered the art or something, but I will share my own experiences and learnings in this post.

  

 

Continue reading “Engineering challenges of streaming a million concurrent JSON data streams from product to CRM”