Apache Kafka API Code Sample in Java

In my earlier blog post, I already have explained what is Apache Kafka and how to install and test it with Apache zookeeper using out of box Apache Kafka commands. If you have missed my earlier article then please refer here: https://javainsider.wordpress.com/2017/05/12/apache-kafka-introduction-installation/

A simple producer/consumer application:

You’ve seen how Kafka works out of the box. Next, let’s develop a custom producer/consumer test client java program. The test producer will send 50  new messages to Kafka server from the sample standalone program. The test consumer will retrieve messages for a given topic and print them to the console in our standalone java application. The producer and consumer components in this case are your own implementations of kafka-console-producer.bat / .sh and kafka-console-consumer.bat/ .sh.

Let’s start by creating a TestProducer.java class. This client class contains logic to create 50 test messages and send them as a message to the Kafka server.

package com.javainsider.kafkaapi;
import java.util.Properties;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;

public class TestProducer {

 @SuppressWarnings("unchecked")
 public static void main(String[] args) {

 Properties props = new Properties();
 props.put("bootstrap.servers", "localhost:9092");
 props.put("acks", "all");
 props.put("retries", 0);
 props.put("batch.size", 16384);
 props.put("linger.ms", 1);
 props.put("buffer.memory", 33554432);
 props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
 props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

 @SuppressWarnings("rawtypes")
 KafkaProducer producer = new KafkaProducer(props);
 for (int key = 0; key < 50; key++) {
 producer.send(new ProducerRecord<String, String>("javainsider", Integer.toString(key), "My test message for key:"+key));
 System.out.println("Kafka topic Key=" + key +", Kafka Message:"+"My test message for key:"+key);
 }

 producer.close();
 }

}

Be sure configure the properties properly as per your kafka server host and port configured in your environment. You should also be sure you have the right topic name configured.

Let’s start by creating a TestConsumer.java class. This client class contains logic to read all messages from kafka server and print them into the console.

package com.javainsider.kafkaapi;

import java.util.Arrays;
import java.util.Properties;

import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;

public class TestConsumer {

 public static void main(String[] args) {
 String topic = "javainsider";
 Properties props = new Properties();
 props.put("bootstrap.servers", "localhost:9092");
 props.put("group.id", "test");
 props.put("enable.auto.commit", "true");
 props.put("auto.commit.interval.ms", "1000");
 props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
 props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
 @SuppressWarnings("resource")
 KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);

 consumer.subscribe(Arrays.asList(topic));
 System.out.println("Subscribed to topic " + topic);

 while (true) {
 ConsumerRecords<String, String> records = consumer.poll(100);
 for (ConsumerRecord<String, String> record : records)
 System.out.printf("Kafka offset = %d, Kafka topic Key = %s, Kafka Message: = %s\n", record.offset(), record.key(), record.value());
 }
 }
}

Take care of your Kafka system configurations properly with topic name.

You can download the complete code from GIT:  https://github.com/javainsider/kafkaapi

Happy coding…..

Proxy Design Pattern (GOF)

When to use this pattern?
Proxy pattern is used when we need to create a wrapper to cover the main object’s complexity from the client.

• Provide a surrogate or placeholder for another object to control access to it.

• Use an extra level of indirection to support distributed, controlled, or intelligent access.

• Add a wrapper and delegation to protect the real component from undue complexity.

Problem:
You need to support resource-hungry objects, and you do not want to instantiate such objects unless and until they are actually requested by the client.

What are the usage scenarios?

1. Virtual Proxy – Imagine a situation where there is multiple database call to extract huge size image. Since this is an expensive operation we can possibly use the proxy pattern which would create multiple proxies and point to the huge size memory consuming object for further processing. The real object gets created only when a client first requests/accesses the object and after that we can just refer to the proxy to reuse the object. This avoids duplication of the object and hence saving memory.
2. Remote Proxy – A remote proxy can be thought about the stub in the RPC call. The remote proxy provides a local representation of the object which is present in the different address location. Another example can be providing interface for remote resources such as web service or REST resources.
3. Protective Proxy – The protective proxy acts as an authorisation layer to verify if the actual user has access to appropriate content. An example can be thought about the proxy server which provides restrictive internet access in office. Only the websites and contents which are valid will be allowed and the remaining ones will be blocked.
4. Smart Proxy – A smart proxy provides additional layer of security by interposing specific actions when the object is accessed. An example can be to check if the real object is locked before it is accessed to ensure that no other object can change it.

Participants:

1. Subject – This object defines the common interface for RealSubject and Proxy so that a Proxy can be used anywhere a RealSubject is expected.
2. Proxy – It maintains a reference to the RealSubject so that Proxy can access it. It also implements the same interface as the RealSubject so that Proxy can be used in place of RealSubject. Proxy also controls the access to the RealSubject and can create or delete this object.
3. RealSubject – This refers the main object which the proxy represents.

 

Code Example: –  Virtual Proxy Example:

As mentioned earlier virtual proxy is useful to save expensive memory resources. Let’s take a scenario where the real image contains a huge size data which clients needs to access. To save our resources and memory the implementation will be as below:

– Create an interface which will be accessed by the client. All its methods will be implemented by the ProxyImage class and RealImage class.
– RealImage runs on the different system and contains the image information is accessed from the database.
– The ProxyImage which is running on a different system can represent the RealImage in the new system. Using the proxy we can avoid multiple loading of the image.

Image.java:

public interface Image {
public void showImage();
}

 

RealImage.java:

public class RealImage implements Image {

private String fileName = null;
public RealImage(String strFileName){
this.fileName = strFileName;
}
@Override
public void showImage() {
System.out.println(“Show Image:” +fileName);

}
}

 

 

ProxyImage.java:

public class ProxyImage implements Image {
private RealImage img= null;
private String fileName = null;

public ProxyImage(String strFileName) {
this.fileName = strFileName;
}
/*
* (non-Javadoc)
* @see com.proxy.virtualproxy.Image#showImage()
*/
@Override
public void showImage() {
if(img == null){
img = new RealImage(fileName);
}
img.showImage();
}
}

Client.java:

public class Client {
public static void main(String[] args) {
final Image img1 = new ProxyImage(“Image***1”);
final Image img2 = new ProxyImage(“Image***2”);
img1.showImage();
img2.showImage();
}
}

Benefits:

•  One of the advantages of Proxy pattern as you have seen in the above example is about security.

•  This pattern avoids duplication of objects which might be huge size and memory intensive. This in turn increases the performance of the application.

•  The remote proxy also ensures about security by installing the local code proxy (stub) in the client machine and then accessing the server with help of the remote code.

 

Drawbacks/Consequences:

This pattern introduces another layer of abstraction which sometimes may be an issue if the RealSubject code is accessed by some of the clients directly and some of them might access the Proxy classes. This might cause disparate behaviour.

Rules of thumb:
– There are few differences between the related patterns. Like Adapter pattern gives a different interface to its subject, while Proxy patterns provides the same interface from the original object but the decorator provides an enhanced interface. Decorator pattern adds additional behaviour at runtime.

– Proxy used in Java API:  java.rmi.*;

 

 

Solutions Architect ~ An Overview

The Solutions Architect is a very experienced architect with cross-domain, cross-functional and cross-industry expertise. He/she outlines solution architecture descriptions, then monitors and governs their implementation.

The Solutions Architect is responsible for the development of the overall vision that underlies the projected solution and transforms that vision through execution into the solution.

To be effective the Solutions Architect must have experience on multiple Hardware and Software Environments and be comfortable with complex heterogeneous systems environments.

 

 

 

Involvement and Responsibility:

1. Involved with a project at the time of inception.
2. Involved in the Functional analysis (FA) of developing the initial requirements.
3. Remain involved throughout the balance of the project.
4. Should have hands-on experience in multiple industries and across several disciplines
5. They can master a variety of hardware platforms including distributed platforms, desktops, and mobile devices.
6. A broad and deep understanding of Databases is also required.
7. Akin to that they should also possess skill and understanding of a variety of Operating Systems
8. Solutions Architects decide which technologies to use.
9. They work very closely with developers to ensure proper implementation.
10. They are the link between the needs of the organization and the developers.
11. A solutions architect must have a balanced mix of technical and business skills.

Solution Architecture in larger organization:

Solution Architects in large organizations often act as the bridge between Enterprise Architects and Application Architects.

• An enterprise architect’s deliverable s are usually more abstract than a solution architect’s deliverable s. But that is not always the case, and the main distinction between enterprise architect and solution architect lies in their different motivations.

• The enterprise architect is primarily employed in the design, planning and governance of strategic and cross-organizational rationalization or optimization of an enterprise’s services, processes or components.

• The solution architect is primarily employed to help programmer and project managers in the design, planning and governance of implementation projects of any kind.

When the solution architect starts and stops depends on the funding model for the process of solution identification and delivery.
E.g. An enterprise may employ a solution architect on a feasibility study, or to prepare a solution vision or solution outline for an Invitation to Tender. A supplier may employ a solution architect at “bid time”, before any implementation project is costed or resourced. Both may employ a solution architect to govern an implementation project, or play a leading role within it.

Next I will be posting what exactly the role and responsibility of a Java Solution Architect and what are the skills he/she should.