PySpark SparkFiles

PySpark provides the facility to upload your files using sc.addFile. We can also get the path of working directory using SparkFiles.get. Moreover, to resolve the path to the files added through SparkContext.addFile(), the following types of class method are available in SparkFiles, such as:

get(filename)
getrootdirectory()

Note: SparkFiles only contain the class method that can be used as required. Users should not create SparkFiles instances.

Let's learn about the class method in detail.

Class Methods of PySpark SparkFiles

get(filename)

The get(filename) specifies the path of the file which is added through SparkContext.addFile().

import os
class SparkFiles(object):
     """
     Resolves paths to files added through
     L{SparkContext.addFile()<pyspark.context.SparkContext.addFile>}.
     SparkFiles consist of only class methods; users should not create SparkFiles
     instances.
     """
      root_directory = None
      is_running_on_worker = False
      sc = None
      def __init__(self):
         raise NotImplementedError("Do not construct SparkFiles objects")
     @classmethod
       def get(cls, filename):
         """
         Get the absolute path of a file added through C{SparkContext.addFile()}.
         """
         path = os.path.join(SparkFiles.getRootDirectory(), filename)
         return os.path.abspath(path)
        @classmethod
        def getRootDirectory(cls):

getrootdirectory()

This class method specifies the path to the root directory. Basically, it contains the whole file which is added through the SparkContext.addFile().

import os
  class SparkFiles(object):
     """
     Resolves paths to files added through
     L{SparkContext.addFile()<pyspark.context.SparkContext.addFile>}.
     SparkFiles contains only classmethods; users should not create SparkFiles
     instances.
     """
      root_directory = None
      is_running_on_worker = False
      sc = None
      def __init__(self):
         raise NotImplementedError("Do not construct SparkFiles objects")
     @classmethod
      def get(cls, filename):
        ...
     @classmethod
      def getRootDirectory(cls):
         """
         Than Get the root directory which contains files added through
         C{SparkContext.addFile()}.
         """
 if cls._is_running_on_worker:
     return cls._root_directory
 else:
     # This will have to change if we support multiple SparkContexts:
     return cls._sc._jvm.spark.SparkFiles.getRootDirectory()

Next TopicPySpark StorageLevel

← prev next →