KSQL or not KSQL? Apache Kafka’s Latest Release May Answer the Question!

Confluent’s October 17th showcase of Apache Kafka’s New Release to Dublin’s Open Source Community at a Hadoop User Group Ireland Meetup

Many ask about “Big Data” in terms of “ohhhh… can you do things with big data that nobody else can do?” Now thats definitely a question for Superman and probably Elon Musk! As one comes from another planet and the other wants to go to another planet, I bet they are using big data like no other!! As for the rest of us here on terra firma, big data has evolved to a point where large data pipelines have presented big data professionals with a substantial challenge! How to successfully manage data in an effective manner without having to be a genius level coder?

Enter Apache Kafka, its latest release and some fascinating talks at a Hadoop User Group Ireland meetup, which was hosted by Bank of Ireland at their start up branch in Trinity College, Dublin. The event was well attended by HUG Ireland members who are big data professionals in numerous areas including Travel, Consultancy, Information Technology and Financial Services. The speakers at the event were Technical Evangelists from Confluent, who released the latest version of Kafka along with a new query language called KSQL.

Up first was Robin Moffatt Partner Technical Evangelist with Confluent who brought us through the Kafka ecosystem where streamlined data pipelines based on Kafka Connect can use the highly developed messenger to transform often convoluted ETL (Extract/Transform/Load) processes into a much more streamlined version of itself. The result for investing in the Kafka set up is a big data ecosystem that saves money, time and patience in any big data processing job.

Sitting on top of this is the real time updating of data via Kafka Streams and KSQL as the new query language, which gives you real time data transparency in a manner somewhat similar to Apache Spark. The new advantage Kafka’s ecosystem brings is KSQL, where ‘SQL like’ querying of Kafka brings real value in terms of fast (a.k.a. low latency) and structured results without the need to code like a pro! This open source platform is definitely one to watch and whilst KSQL is relatively new, adoption seems to be good with many phasing it in by up-to c.70% of their data been streamed up to the Analysis platform in a type safe manner using Kafka Streams and KSQL. Also, are you using KSQL and want to embed KSQL queries into your Java code? No problem, it’s a very simple task with KSQL, which produces embeddable Java code as part of its output. Good user experiences are thus reported from the open source community using Kafka.

So, with the new Kafka centric data pipeline set up, you can configure it for good interoperability with higher-level layers like the Machine Learning Platform H20, which Kai Waehner Technical Evangelist from Confluent ably demonstrated in the second part of this event. He went into the structure of the streamlined Kafka pipeline and how one can in real time use advanced analytics layers like H20 to perform deep learning jobs on streaming data via Kafka Streams. The need for training your model for accuracy is a standard in any machine-learning model (including deep learning and neural networks). He demonstrated this noting the performance characteristics of the Kafka platform feeding a future of process efficiencies for the big data world based on a data pipeline that is Kafka centric! In Kai’s words, Kafka’s streaming platform is an “all rounder” for your predictive and data analysis needs.

The meetup event was hugely successful given the fascinating topic and range of possibilities these software innovations are bringing to the table! I was in contact with Robin and Kai after the event as they shared my enthusiasm for the evening. Kai on Kafka’s new release… “As we have seen in the two talks, the Apache Kafka open source ecosystem evolved from a scalable, distributed messaging pipeline to a powerful streaming platform with integration, stream processing and other enhancements.” Here here Kai! Well said sir!! Robin left me with a clear positive for a bright future with “We always enjoy supporting the community in Ireland with meet ups such as this and are so excited to see all the enthusiasm and interest.” Irish Tech News is right there with Confluent’s Open Source Ambassadors in that sentiment and hope to see more from them and the Apache Kafka journey into the future!