Class

org.apache.spark.sql.execution.datasources.xskipper

PrunedInMemoryDataSkippingIndex

Related Doc: package xskipper

Permalink

class PrunedInMemoryDataSkippingIndex extends InMemoryDataSkippingIndex

used to support PrunedInMemoryFileIndex which might result from running PruneFileSourcePartitions rule, since the PrunedInMemoryFileIndex specify its own partitionSpec we can not use an InMemoryFileIndex

Linear Supertypes
InMemoryDataSkippingIndex, InMemoryFileIndex, PartitioningAwareFileIndex, Logging, FileIndex, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. PrunedInMemoryDataSkippingIndex
  2. InMemoryDataSkippingIndex
  3. InMemoryFileIndex
  4. PartitioningAwareFileIndex
  5. Logging
  6. FileIndex
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new PrunedInMemoryDataSkippingIndex(sparkSession: SparkSession, tableBasePath: Path, fileStatusCache: FileStatusCache, partitionSpec: PartitionSpec, metadataOpsTimeNs: Option[Long], tableIdentifiers: Seq[String], fileFilters: Seq[DataSkippingFileFilter], metadataFilterFactories: Seq[MetadataFilterFactory], clauseTranslators: Seq[ClauseTranslator], backend: MetadataStoreManagerType)

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. def allFiles(): Seq[FileStatus]

    Permalink
    Definition Classes
    PartitioningAwareFileIndex
  5. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  6. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  7. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  8. def equals(other: Any): Boolean

    Permalink
    Definition Classes
    InMemoryFileIndex → AnyRef → Any
  9. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  11. val hadoopConf: Configuration

    Permalink
    Attributes
    protected
    Definition Classes
    PartitioningAwareFileIndex
  12. def hashCode(): Int

    Permalink
    Definition Classes
    InMemoryFileIndex → AnyRef → Any
  13. def inferPartitioning(): PartitionSpec

    Permalink
    Attributes
    protected
    Definition Classes
    PartitioningAwareFileIndex
  14. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  15. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  16. def inputFiles: Array[String]

    Permalink
    Definition Classes
    PartitioningAwareFileIndex → FileIndex
  17. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  18. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  19. def leafDirToChildrenFiles: Map[Path, Array[FileStatus]]

    Permalink
    Attributes
    protected
    Definition Classes
    InMemoryFileIndex → PartitioningAwareFileIndex
  20. def leafFiles: LinkedHashMap[Path, FileStatus]

    Permalink
    Attributes
    protected
    Definition Classes
    InMemoryFileIndex → PartitioningAwareFileIndex
  21. def listFiles(partitionFilters: Seq[Expression], dataFilters: Seq[Expression]): Seq[PartitionDirectory]

    Permalink
    Definition Classes
    InMemoryDataSkippingIndex → PartitioningAwareFileIndex → FileIndex
  22. def listLeafFiles(paths: Seq[Path]): LinkedHashSet[FileStatus]

    Permalink
    Definition Classes
    InMemoryFileIndex
  23. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  24. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  25. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  26. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  27. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  28. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  29. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  30. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  31. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  32. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  33. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  34. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  35. val metadataOpsTimeNs: Option[Long]

    Permalink
    Definition Classes
    PrunedInMemoryDataSkippingIndex → FileIndex
  36. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  37. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  38. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  39. def partitionSchema: StructType

    Permalink
    Definition Classes
    PartitioningAwareFileIndex → FileIndex
  40. val partitionSpec: PartitionSpec

    Permalink
    Definition Classes
    PrunedInMemoryDataSkippingIndex → InMemoryFileIndex → PartitioningAwareFileIndex
  41. def refresh(): Unit

    Permalink
    Definition Classes
    InMemoryFileIndex → FileIndex
  42. val rootPaths: Seq[Path]

    Permalink
    Definition Classes
    InMemoryFileIndex → FileIndex
  43. def sizeInBytes: Long

    Permalink
    Definition Classes
    PartitioningAwareFileIndex → FileIndex
  44. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  45. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  46. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  47. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  48. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from InMemoryDataSkippingIndex

Inherited from InMemoryFileIndex

Inherited from PartitioningAwareFileIndex

Inherited from Logging

Inherited from FileIndex

Inherited from AnyRef

Inherited from Any

Ungrouped