Javatpoint Logo
Javatpoint Logo

Spark Word Count Example

In Spark word count example, we find out the frequency of each word exists in a particular file. Here, we use Scala language to perform Spark operations.

Steps to execute Spark word count example

In this example, we find and display the number of occurrences of each word.

  • Create a text file in your local machine and write some text into it.
Spark Word Count Example
  • Check the text written in the sparkdata.txt file.
Spark Word Count Example
  • Create a directory in HDFS, where to kept text file.
  • Upload the sparkdata.txt file on HDFS in the specific directory.
Spark Word Count Example
  • Now, follow the below command to open the spark in Scala mode.
Spark Word Count Example
  • Let's create an RDD by using the following command.

Here, pass any file name that contains the data.

  • Now, we can read the generated result by using the following command.
Spark Word Count Example
  • Here, we split the existing data in the form of individual words by using the following command.
  • Now, we can read the generated result by using the following command.
Spark Word Count Example
  • Now, perform the map operation.

Here, we are assigning a value 1 to each word.

  • Now, we can read the generated result by using the following command.
Spark Word Count Example
  • Now, perform the reduce operation

Here, we are summarizing the generated data.

  • Now, we can read the generated result by using the following command.
Spark Word Count Example

Here, we got the desired output.






Please Share

facebook twitter google plus pinterest

Learn Latest Tutorials


Preparation


Trending Technologies


B.Tech / MCA