Building a data mining solution

Article by Narender Kumar, Big Data & Cloud Lead Developer

Overview

In this blog, we will learn how we can build a scalable solution with a simplified GUI that can be used to dig into huge data stored in the data lake and provide us some specific data that we need.

We can develop the GUI as a webpage that takes inputs such as product type, sales limit, date, etc., and provide us a URL link of the data that we can click and download related data.

We have implemented this solution using Azure services but we can use similar services by any cloud provider/open-source as well.

The Architecture

Process

1. Metadata

2. Indexes of metadata

We can push our data to the Azure SQL server and create indexes on top of it using Azure Search. Azure Search provides a REST API that can be used by web services to query the metadata.

3. GUI and web jobs

4. Usage Insights

Technologies used

1. Azure Data lake

2. Azure SQL Database

3. Azure Search

4. Azure Web Apps

We used it to deploy the front end of the web application.

5. Azure AD

6. Azure Blob

7. Azure Application Insights

8. Azure Log Analytics

We used Azure services for this requirement. We have similar services available on other cloud services also as below :

Conclusion

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
BigData & Cloud Practice

Abzooba is an AI and Data Company. BD&C Practice is one of the fastest growing groups in Abzooba helping several fortune 500 clients in there cognitive journey