Apache Nifi configurable ETL tool

What is Apache Nifi

Apache Nifi is an open source, powerful, easy to use, scalable, reliable and secure framework. Nifi is a developer friendly platform which has many built-in processors to connect to a wide range of technologies including cloud. It is a great and flexible tool to collect, process, transform and store to destination location with no extra coding effort. With great flexibility, It also provides a feature to customize or extend their processor based on organization needs.

You don’t need to know any programming language (except if your requirement doesn’t match with the incredible processors nifi offers). All you have to do is drag the required processors between which you want to automate the data flow, and connect them with the arrow in the required direction. And voila! You can achieve your target. Isn’t it amazing..!

Nifi as an ETL tool

Apache Nifi is a flexible ETL tool with a wide variety of in-built processor integration with Kafka, AWS S3, Spark, Hbase or any nosql database. Not just nifi allows one to Extract data from the source, and Load it to some other location, it allows one to perform the various other tasks such as

• Enrichment and Preparation of data:- We can add, enhance and improve the existing content of the data. This way, it can help in bringing game changing transformations in making our data more valuable.
• Data Cleaning:- When we extract raw data from some source let’s say, hbase. Chances are, that it has some missing, corrupt or unwanted data. So, in order to get rid of such noisy data, nifi provides some amazing processors to deal with it. With the help of them, you can apply multiple cleaning operations to bring the desired result.
• Conversion between formats:- From making invalid json into valid json, from changing json data into csv, etc. In short, To transform data(csv, avro, xml, etc) as per the need, Nifi has processors to handle them all.
• Routing decision:- every coin has two sides, head and tail. And so does every decision. Decisions can be accepted by some and rejected by others. Same goes with nifi. It too provides us with matched and failure functionalities. We can define the constraints or conditions. If our data or attributes matches with the conditions applied, it is routed to the next desired processor thereafter, and if the input doesn’t match with the imposed condition, it is routed to “failure queue” from where it can again be routed as per the need or can even be terminated.

This is just a basic overview of nifi. I hope it has helped you in knowing the power of nifi. Want to know more? Start using nifi to unravel more about it.In the next blog, I will provide more insight related to Nifi Features.