Creating Twitter Producer
In this section, we will learn to create a twitter producer.
There are basically three steps to create a twitter producer:
Step1: Create a new java package, following the package naming convention rules. Then, create a java class within it, say 'tweetproducer.java.'
Step2: Create a twitter client by creating a method for it. Now, copy the Quickstart code from the 'github twitter java' to the twitter client method, as shown below:
Paste it in the newly created method. This code will create a connection between the client and the hbc host. The BlockingQueue will stop the client to dequeue or enqueue the messages when the queue is empty or already full. As we are using hbc-core, we only require the msgQueue. Also, we will follow the terms, not the people. Therefore, copy the highlighted code only.
Now, copy the 'Creating a client' code given below the connection code as:
Paste the code below the connection code. This code will create a twitter client through the client builder. As we are using msgQueue, do not copy the red highlighted code, which is for the eventMessageQueue. It is not required.
Step3: Create the producer in a similar way we learned in the previous sections with a bootstrap server connection.
Step4: After creating the Kafka producer, its time to send tweets to Kafka. Copy the while loop code from the 'github twitter java', given below the 'Creating a client' code. Paste below the producer code.
Now, we are ready to read tweets from Twitter. Although, a Kafka producer read messages from a topic. So, create the specified topic using the '-create' command on the CLI. Also, specify the partition value and the replication factor.
Here, the topic 'twitter_topic' has been created with partition value 6 and replication-factor 1. Finally, execute the code and experience Kafka in the real-world application.
The complete code for creating the Twitter Client is given below:
In the above code, the user will specify the consumerKey, consumerSecret key, token key as well as the secret key. As it is sensitive information, therefore it cannot be displayed. Copy the key from the 'developer.twitter.com' and paste at their respective positions.
Copy the keys from 'Keys and Tokens' and paste in the code.
The output of the above code will be displayed as:
The client establishes a connection with the Hosebird. After this, we can see too many tweets produced on 'India'. Post some tweets on any specified topic and try out.
Try out the 'kafka-console-consumer -bootstrap-server 127.0.0.1:9092 -topic twitter_topic' command on the CLI. The output will be the same as on the IntelliJ IDEA terminal:
In this way, we can create a real Twitter-Kafka-Producer and send tweets to Kafka.