Class

org.apache.spark.sql.execution.datasources.xskipper

PrunedInMemoryDataSkippingIndex

Related Doc: package xskipper

Permalink

class PrunedInMemoryDataSkippingIndex extends InMemoryDataSkippingIndex

used to support PrunedInMemoryFileIndex which might result from running PruneFileSourcePartitions rule, since the PrunedInMemoryFileIndex specify its own partitionSpec we can not use an InMemoryFileIndex

Linear Supertypes
InMemoryDataSkippingIndex, InMemoryFileIndex, PartitioningAwareFileIndex, Logging, FileIndex, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. PrunedInMemoryDataSkippingIndex
  2. InMemoryDataSkippingIndex
  3. InMemoryFileIndex
  4. PartitioningAwareFileIndex
  5. Logging
  6. FileIndex
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new PrunedInMemoryDataSkippingIndex(sparkSession: SparkSession, tableBasePath: Path, fileStatusCache: FileStatusCache, partitionSpec: PartitionSpec, metadataOpsTimeNs: Option[Long], tableIdentifiers: Seq[String], fileFilters: Seq[DataSkippingFileFilter], metadataFilterFactories: Seq[MetadataFilterFactory], clauseTranslators: Seq[ClauseTranslator], backend: MetadataStoreManagerType)

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. def allFiles(): Seq[FileStatus]

    Permalink
    Definition Classes
    PartitioningAwareFileIndex

  5. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  6. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  7. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  8. def equals(other: Any): Boolean

    Permalink
    Definition Classes
    InMemoryFileIndex → AnyRef → Any

  9. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  10. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  11. val hadoopConf: Configuration

    Permalink
    Attributes
    protected
    Definition Classes
    PartitioningAwareFileIndex

  12. def hashCode(): Int

    Permalink
    Definition Classes
    InMemoryFileIndex → AnyRef → Any

  13. def inferPartitioning(): PartitionSpec

    Permalink
    Attributes
    protected
    Definition Classes
    PartitioningAwareFileIndex

  14. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  15. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  16. def inputFiles: Array[String]

    Permalink
    Definition Classes
    PartitioningAwareFileIndex → FileIndex

  17. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  18. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  19. def leafDirToChildrenFiles: Map[Path, Array[FileStatus]]

    Permalink
    Attributes
    protected
    Definition Classes
    InMemoryFileIndex → PartitioningAwareFileIndex

  20. def leafFiles: LinkedHashMap[Path, FileStatus]

    Permalink
    Attributes
    protected
    Definition Classes
    InMemoryFileIndex → PartitioningAwareFileIndex

  21. def listFiles(partitionFilters: Seq[Expression], dataFilters: Seq[Expression]): Seq[PartitionDirectory]

    Permalink
    Definition Classes
    InMemoryDataSkippingIndex → PartitioningAwareFileIndex → FileIndex

  22. def listLeafFiles(paths: Seq[Path]): LinkedHashSet[FileStatus]

    Permalink
    Definition Classes
    InMemoryFileIndex

  23. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  24. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  25. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  26. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  27. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  28. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  29. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  30. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  31. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  32. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  33. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  34. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  35. val metadataOpsTimeNs: Option[Long]

    Permalink
    Definition Classes
    PrunedInMemoryDataSkippingIndex → FileIndex

  36. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  37. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  38. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  39. def partitionSchema: StructType

    Permalink
    Definition Classes
    PartitioningAwareFileIndex → FileIndex

  40. val partitionSpec: PartitionSpec

    Permalink
    Definition Classes
    PrunedInMemoryDataSkippingIndex → InMemoryFileIndex → PartitioningAwareFileIndex

  41. def refresh(): Unit

    Permalink
    Definition Classes
    InMemoryFileIndex → FileIndex

  42. val rootPaths: Seq[Path]

    Permalink
    Definition Classes
    InMemoryFileIndex → FileIndex

  43. def sizeInBytes: Long

    Permalink
    Definition Classes
    PartitioningAwareFileIndex → FileIndex

  44. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  45. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any

  46. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  47. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  48. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from InMemoryDataSkippingIndex

Inherited from InMemoryFileIndex

Inherited from PartitioningAwareFileIndex

Inherited from Logging

Inherited from FileIndex

Inherited from AnyRef

Inherited from Any

Ungrouped