Think about you’re getting groceries delivered, or searching for a advice on what to look at subsequent on TV, or utilizing a bank card with out worrying an excessive amount of about fraud. The functions that energy these interactions all depend on knowledge in movement, and there’s a good probability Apache Kafka powers the functions.
Greater than 80% of the Fortune 100 use Kafka because the occasion streaming substrate to energy real-time, user-facing functions and software-driven again ends. Kafka has turn into the go-to for any group trying to combine more and more numerous portfolios of functions and microservices via immutable occasion logs reasonably than mutable knowledge shops. The advantages are manifold, however recall that Kafka is a distributed system, and volunteering to function a distributed system your self is an more and more controversial alternative.
That is why the cloud exists. Via absolutely managed cloud companies, distributors bear the capital bills and accumulate the operational experience essential to run infrastructure properly. Confluent, the primary absolutely managed Kafka service available on the market, helps you to give attention to constructing functions and including worth to the enterprise reasonably than turning dials on operationally complicated infrastructure layers. I’d prefer to stroll you thru how Confluent can deliver peace and ease to the lives of the individuals who work with Kafka.
Cloud-native is the way forward for infrastructure
There may be at all times a better demand for software performance than there may be the capability to ship it. This means that software groups ought to give attention to the actions that create probably the most worth that they probably can. Usually, this implies offering new options that instantly give a aggressive edge to prospects and customers.
In fact, all functions require storage and compute infrastructure to operate with ongoing growth and upkeep, distracting from value-creating function growth. That is very true for Kafka, as a result of distributed knowledge infrastructure imposes a major alternative value on groups deciding to function it themselves. Put merely: Your job is in the end to deal with your prospects. Whereas working Kafka could also be a way to that finish, it’s possible not probably the most sensible approach to get the job performed. This problem is one in every of many causes that led to the rise of managed cloud companies.
Elastic scaling for reals this time
Elastic scalability has at all times been an inherent a part of the cloud’s mythology however has been gradual in coming to actuality. Early on within the cloud’s historical past, database innovators utilized new approaches to horizontal elastic scalability of huge datasets. Extra lately, microservices and container orchestration have helped deliver software scalability to the lots. Nonetheless, knowledge infrastructure typically has remained notoriously proof against simple scalability.
Kafka has a superb horizontal scale story: matters are partitioned, particular person partition logs are assigned to totally different brokers, then consumed by scalable clusters of shopper functions. There are some scriptable instruments to manage these scale-oriented capabilities, however self-managed clusters nonetheless require important operational and technical experience. For instance, partition logs don’t stay evenly distributed on brokers as a cluster adjustments over time. Additional, new matters are added, and partitions obtain probably uneven learn and write site visitors, as enterprise situations evolve. That’s only one instance of one thing cluster directors should attend to over time.
Confluent has built-in elastic scalability. Clusters scale from 0 to 100MBps throughput with no intervention and as much as 11GBps (the present document as of this writing) via a easy net UI—no transferring partitions round, no rebalancing brokers. Because the world slowly catches as much as the cloud’s unique guarantees of elastic scale, Confluent brings scale to knowledge infrastructure in a really cloud-native method.
Connecting your data everywhere
Your life will be multicloud anyway, so data infrastructure layers need to be multicloud-capable to be serious contenders. Confluent is multicloud, natively supporting AWS, Microsoft Azure, and Google Cloud. This flexibility is essential when you need to run on more than one cloud, or at least be able to threaten to. Confluent makes this easy by using a single management web UI and a unified control plane abstracted from the particular cloud infrastructure.
But multicloud isn’t always enough! Sometimes you don’t want to move everything to the cloud. Many companies want to maintain a mix of on-prem, private cloud, or public cloud services. This hybrid cloud experience is top of mind for Confluent, making it possible to maintain backup services, segregate products, and manage a sophisticated Plan B through the Confluent UI.
Don’t get lost in the ecosystem, get the whole package
As the Kafka community has discovered in the 10 years since its birth, you need more than distributed logs to build a successful event-driven system. You also need reliable and secure connections between all your systems and streams, which is no mean feat. Then you can start to extract value from the whole system with real-time stream processing.
Several components have emerged around core Kafka functionality to help deliver on those needs, both from the open source ecosystem and from Confluent:
- Kafka Connect: The universal data integration framework that provides an ecosystem of connectors. It removes the need to re-write connectors for every new data source.
- Kafka Streams: A stream processing framework that enriches the existing Kafka client framework with sophisticated stream processing functionality, rather than offloading stream processing to another distributed system.
- Confluent Schema Registry: Helps maintain compatibility between evolving applications as message formats change over time.
- ksqlDB: The event streaming database for Kafka using SQL to build stream processing applications you might otherwise have built with Kafka Streams.
- Confluent Metrics API: Unifies many of the individual metrics you could collect through the JMX interface on various system components into a single, queryable stream of JSON data.
The fact of components like these is that teams will eventually need them. They have emerged from the Kafka community and from Confluent for that very reason. It is nearly impossible to be proficient enough in each of these areas to build a solution that does not require continual attention for smooth operation.
With Confluent, you have all the tools you need to be successful with Kafka at your fingertips. You can use one platform, and everything you need is there in a seamless, integrated way, including hundreds of connectors to popular data sources.
Data security at scale is a must
Kafka has a minimum viable security story: It offers robust encryption of data in flight and ACL-based authentication and authorization as options. Confluent expands on these features in the ways enterprises expect.
For example, all data in Confluent is encrypted at rest as well as in flight. On top of that, applications also require authentication with every call, removing the threat of accidentally having “wide open” ports.
A variety of other Confluent features help keep security simple, including SAML-based single sign-on and secure access to other cloud resources in your VPCs.
As proof of these secure capabilities, Confluent meets many industry standards and certification achievements, meeting requirements for PCI, HIPAA, and GDPR as well as SOC1, SOC2, SOC3, and ISO 27001 certifications.
It is a challenge to achieve all of these certifications while also providing many other secure and convenient features out of the box. Developers can build with confidence while leaving the heavy security lift to the managed platform.
But don’t just take my word for it. You can try our fully managed Kafka service for free through Confluent or your cloud provider of choice.
Tim Berglund is senior director of developer advocacy at Confluent.
New Tech Forum provides a venue to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to [email protected]
Copyright © 2021 IDG Communications, Inc.