Apache Pig TOKENIZE Function

The Apache Pig TOKENIZE function is used to splits the existing string and generates a bag of words in a result.

Example of TOKENIZE Function

In this example, we split the string in the tokens.

Steps to execute TOKENIZE Function

• Create a text file in your local machine and insert the list of tuples.

• Check the tuples inserted in the text files.

• Upload the text files on HDFS in the specific directory.

• Open the pig MapReduce run mode.

• Load the file that contains the data.

• Now, execute and verify the data.

• Let's return the tokenized form of the string.

Here, we got the desired output.