Around two weeks back, NITI Aayog launched the National Data and Analytics Platform (NDAP) for open public use. It said that this platform will host foundational datasets from various government agencies and provide tools for analytics and visualisation. A beta release was done last year in August, which provided access to a limited number of users for testing and feedback. NDAP seeks to “democratise” data delivery by resolving many of the challenges that currently exist in the open data universe.

Object Technology Solutions India (OTSI) has developed NDAP.  In an exclusive interaction with Analytics India Magazine, Chandra Talluri, CEO, OTSI, talks more about the tech powering NDAP.

OTSI provides its expertise in four major practise areas of technology:

• Digital transformation
• Data and analytics – data insights, visualisation
• Automation – test automation and RPA
• Disruptive tech (more focus on R&D)

Sole tech partner for NDAP

NITI Aayog had floated an RFP for hiring the consultancy firm which can provide tech solution architecture for this project. “It’s a more outcome-based project. OTSI worked with the NITI Aayog team, and understood various aspects of data in the public sector and open data sources. We proposed the entire solution – right from the design to solution architecture, the approach that has to be taken, and how to approach the milestones. Overall, we are the sole technology partner involved in this project,” said Talluri.

Tech stack – AWS, Python and more

Though the entire tech stack can’t be revealed at the moment, Talluri informs that the overall solution is hosted on AWS. OTSI used a lot of components in the sourcing related to scraping, mostly done on Python-specific algorithms. On the database side, it used multiple databases. There have been applications of vector databases as well. The AWS RedShift database has also been used for specific purposes. On the front end, it is a ReactJS based application.

Big names participated, OTSI got an edge because of past records

“To be honest, this is one of the prestigious projects. Many big names have participated in the RFP. It was primarily targeted at Tier 1 partners. OTSI has a good track record of successfully delivering analytics projects in various public sectors and ministries as well. We have a track record of working with public sector and ministries, our understanding of working with public datasets, adds Talluri.

Ministry of Corporate Affairs, Ministry of Home Affairs are past clientele

OTSI worked with the Ministry of Corporate Affairs (MCA) to set up a business intelligence/data mining and analytics unit in the Ministry and generate customised statistical reports using business intelligence, data mining, and analytics tools to be used by the government departments for policymaking and regulatory purposes. The main priority of this project is to demonstrate the potential of data mining, visualisation, and interactive reporting on the company filing forms submitted by companies to the MCA21 system. OTSI has also worked with the Ministry of Home Affairs and PM division (MHA) for ICJS – Interoperable Criminal Justice System. It has partnered with state governments as well.

Similar platforms have limited datasets

Similar platforms have existed before, then what is so special about NDAP? AIM asked Talluri.

Talluri informs that in many other portals, very limited datasets are available. They are also not regularly updated. “They have taken some of the open data sources and integrated them into the platform. There is no mechanism for taking all this data and getting updates periodically in terms of sourcing. The number of data sources we have touched on NDAP is over 30,000 (more to come) scraping those websites. The source is quite broad here,” he states.

Image: Home – NDAP (niti.gov.in)

Don’t think “Merging” two different datasets is available in other similar government platforms

In NDAP, there exists a feature called “merging” of datasets. Algorithms that OTSI has built to make the merging possible is one of the major challenges that OTSI had to overcome, informs Talluri.

Talluri gives an example: 

Say you wanted to pick up datasets on education and census data and merge both. India is going through lots of changes, including geographical changes with new states forming. Telangana was not there before 2014. How do you merge these datasets when you want to study districts of Telangana and educational KPIs before 2014?

Talluri says, “We have come up with methods, including time-series and geographical based algorithms, and we are able to split this data at the district level based on the geographic region to give insights prior to 2014 as well. We plan to drive it down to village levels too.”

More datasets will be added

It’s an ongoing process. Talluri informs, “Right now, we have 47 ministries, and 14 sectors have been covered. We will add more datasets from each ministry. It’s all about getting access as well. Right now, we are crawling and getting the data. Some ministries have exposed the APIs while others have not. Some ministries have published the data in PDFs;  we are reading those files as well. Some are standalone PDF docs. Some are published in the dashboards, and we are crawling from dashboards too.”

Is NITI Aayog late to the game?

The US, UK and many other western countries have these kinds of portals available that the Government has taken and tried in various ways. The Indian government, too, has tried similar things in the past but may not have received the desired results. Now, NITI has taken responsibility of given a more holistic view through NDAP.

This portal is targeted not just at regular citizens but will largely help the policymakers too. Research institutes like IITs and IIMs will be largely benefited from helping them. Instead of researchers struggling to find the data, it will be available to them, right at their disposal. The NDAP is aimed to give the ease of use for users who are probably not so conversant with the tech.”
Check out how the NDAP portal uses the Merge feature.