{"id":6225,"date":"2017-07-27T10:34:31","date_gmt":"2017-07-27T10:34:31","guid":{"rendered":"http:\/\/stratio.com\/?p=6225"},"modified":"2023-09-20T13:29:57","modified_gmt":"2023-09-20T13:29:57","slug":"data-ingestion-kafka-connector","status":"publish","type":"post","link":"https:\/\/www.stratio.com\/blog\/data-ingestion-kafka-connector\/","title":{"rendered":"One File System to ingest them all (and in Kafka bind them)"},"content":{"rendered":"

Sounds epic, doesn\u2019t it? Actually, it\u2019s not that epic!<\/p>\n

It could be interesting (or very geeky) to talk about how to ingest data in Middle-earth (and what for). However,\u00a0I guess it would be out of the scope of this blog, so I\u2019m afraid this post has nothing to do with that. This post is about how to ingest data from different kinds of file systems by means of Kafka-Connect using a connector I\u2019ve forged <\/i>recently.<\/p>\n

<\/p>\n

From the legacy systems -in terms of hardware and software- for managing huge amounts of data to Big Data systems, a lot has happened; the way we might know an architecture for processing data is pretty different from how we did in the “old days”. I can think of some approaches we are using or hearing a lot about such as Microservices, Lambda or Kappa architectures in which we find a wide spectrum of amazing technologies, thanks especially to the open source community. These approaches are focused on how to process data with different sorts of objectives for addressing mostly business needs for data intensive applications.
\n<\/a>
\n\"Lord<\/p>\n

How to deal with legacy systems<\/h2>\n

Even though new Big Data systems have emerged with a completely different paradigm to the “legacies” -solving some issues that organizations were complaining about- and that their popularity is growing, most businesses see the prospect of replacing them as too risky and feel locked into their “legacy systems” that are often critical to their day-to-day jobs. \u00a0Dealing with these systems can sometimes be a bit awkward; they impose strong requirements on us to integrate our apps with their ecosystem, breaking some rules we wouldn\u2019t normally like to but, at least for the time being, we have to\u00a0manage this and do our best. For this reason, Big Data techs cannot look away and must face these limitations on a daily basis.<\/p>\n

One interesting thing users\/organizations are realizing, is the importance of streaming data and the “goldmine” it represents. I\u2019m not going to talk about how important (near) real-time processing is becoming or what a Kappa architecture is.\u00a0I will talk about how data, stored in a \u2018mainframe\u2019 or something similar, could be fed into a streaming platform.<\/p>\n

I wouldn\u2019t say it\u2019s rare that when trying to integrate\u00a0your application straight into one of these \u2018legacy systems\u2019, you\u00a0come across someone who tells you: “No, you can\u2019t access it in that way”. What now,\u00a0then? One alternative is to export all the data you need to files in a shared file system and process them later. If we\u2019re talking about ingesting this data into a streaming platform, one of the core pieces we use intensively in our Big Data platform is Apache Kafka<\/a>.<\/p>\n

We can consider Kafka as a de-facto platform for streaming architectures. It\u2019s very active, continuously improving and its ecosystem is very rich and growing. One of the components closely related to Kafka is Kafka-Connect: a framework used for interacting from\/to Kafka with external systems. Currently, there are dozens of connectors for Kafka-Connect available which allow us to ingest or bulk data from\/to several kind of systems, but in this post I\u2019m focusing on a connector I\u2019m actually working on: kafka-connect-fs<\/a>. So let\u2019s get into the nitty-gritty!<\/p>\n

What is this connector all about?<\/h2>\n

In a simple way: it\u2019s a Kafka-Connect source connector (for the moment) for ingesting data from files with different sorts of formats persisted in a file system and loading them into Kafka.<\/p>\n

\"Data\"<\/p>\n

\u00a0Why this connector?<\/h2>\n

Well, as I\u2019ve told you, there are a lot of connectors out there and some of them even ingest data from files, but the idea behind this one was to provide an abstraction layer to the native file system itself and, thus, allowing you to connect to \u201cany\u201d file system you like.<\/p>\n

Thanks to hadoop-common<\/i> lib, we have this abstraction layer using the following abstract class\u00a0\u2018org.apache.hadoop.fs.FileSystem\u2019\u00a0which includes a generic file system implementation, and allows\u00a0custom implementation. Out of the box, there are several file system types you can use, but in case your file system is not included, you can develop your own one! Some of them you can use are:<\/p>\n