o

io.xskipper.metadatastore.parquet

ParquetMetadataStoreManager

object ParquetMetadataStoreManager extends MetadataStoreManager with Logging

A MetadataBackend backed by Parquet objects

Linear Supertypes
Logging, MetadataStoreManager, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ParquetMetadataStoreManager
  2. Logging
  3. MetadataStoreManager
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clearActiveMetadataHandles(): Unit

    Clear all active MetadataHandles managed by this MetadataStoreManager

    Clear all active MetadataHandles managed by this MetadataStoreManager

    Definition Classes
    MetadataStoreManager
  6. def clearStats(): Unit

    Clears the stats for all active MetadataHandle instances.

    Clears the stats for all active MetadataHandle instances. Should be called before each query (or call clearActiveMetadataHandles) to make sure the aggregated stats are cleared

    Definition Classes
    MetadataStoreManager
  7. def clone(): AnyRef
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @native() @throws( ... )
  8. def createMetadataHandle(sparkSession: SparkSession, tableIdentifier: String): MetadataHandle

    Creates a new instance MetadataHandle Used by getOrCreateMetadataHandle in order to create new instances

    Creates a new instance MetadataHandle Used by getOrCreateMetadataHandle in order to create new instances

    Definition Classes
    ParquetMetadataStoreManager → MetadataStoreManager
  9. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  10. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  11. def finalize(): Unit
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  12. def getActiveMetadataHandles(): Map[String, MetadataHandle]

    Returns all active MetadataHandles managed by this MetadataStoreManager

    Returns all active MetadataHandles managed by this MetadataStoreManager

    Definition Classes
    MetadataStoreManager
  13. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  14. def getDataSkippingFileFilterParams(tid: String, sparkSession: SparkSession, fileIndex: FileIndex): Map[String, String]

    Returns a map of parameters to be set by the io.xskipper.search.DataSkippingFileFilter on the MetadataHandle when using this MetadataStoreManager.

    Returns a map of parameters to be set by the io.xskipper.search.DataSkippingFileFilter on the MetadataHandle when using this MetadataStoreManager. This map can be used to pass specific parameters depending on the type of FileIndex that is being replaced. For example - in ParquetMetadataStoreManager we can specify that the metadata location should be inferred from the table/default db properties

    tid

    the table identifier for which the DataSkippingFileFilter will be created

    sparkSession

    the sparks session

    fileIndex

    the fileIndex that will be replaced by io.xskipper.search.DataSkippingFileFilter

    returns

    a map of parameters to be set on the MetadataStore when the DataSkippingFileFilter is created

    Definition Classes
    ParquetMetadataStoreManager → MetadataStoreManager
  15. def getLatestQueryAggregatedStats(): QueryIndexStatsResult

    Gets the aggregated latest query skipping stats for all active MetadataHandle instances.

    Gets the aggregated latest query skipping stats for all active MetadataHandle instances. In order to get reliable results it is assumed that either clearStats or clearActiveMetadataHandles was called before running the query.

    This is needed since the way we aggregate the skipping stats is by going over all active MetadataHandles of the MetadataStoreManager and aggregating their stats. When running multiple queries there could be a scenario in which the first query used dataset a and the second query didn't use it, therefore, when calling aggregatedStats for the second query the MetadataHandle for dataset a will be present as an active MetadataHandle therefore we need its stats to be cleared.

    returns

    QueryIndexStatsResult instance with the latest query aggregated stats

    Definition Classes
    MetadataStoreManager
  16. def getOrCreateMetadataHandle(sparkSession: SparkSession, tableIdentifier: String): MetadataHandle

    Gets an existing MetadataHandle or, if there is no existing one, creates a new one.

    Gets an existing MetadataHandle or, if there is no existing one, creates a new one.

    sparkSession

    the sparks session

    tableIdentifier

    the table identifier of the dataset for which the MetadataHandle instance is created

    returns

    a new instance of the MetadataHandle associated with this backend

    Definition Classes
    MetadataStoreManager
  17. def getParams(sparkSession: SparkSession): Map[String, String]

    returns

    a map representing relevant MetadataStoreManager parameters

    Definition Classes
    ParquetMetadataStoreManager → MetadataStoreManager
  18. def getType: MetadataStoreManagerType

    returns

    the MetadataStoreManagerType associated with this backend

    Definition Classes
    ParquetMetadataStoreManager → MetadataStoreManager
  19. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  20. def init(): Unit

    A unit function for setting up the MetadataStoreManager.

    A unit function for setting up the MetadataStoreManager. The setup is called once when the MetadataStoreManager is registered

    Definition Classes
    ParquetMetadataStoreManager → MetadataStoreManager
  21. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  22. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  23. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  24. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  25. def listIndexedDatasets(sparkSession: SparkSession): Map[String, (MetadataVersionStatus.MetadataVersionStatus, Seq[Index])]

    returns

    Map of indexed datasets to their respective md version status and indexes

    Definition Classes
    ParquetMetadataStoreManager → MetadataStoreManager
  26. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  27. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  28. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  29. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  30. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  31. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  32. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  33. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  34. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  35. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  36. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  37. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  38. val metadataHandlesInstanceMap: ConcurrentHashMap[String, MetadataHandle]
    Definition Classes
    MetadataStoreManager
  39. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  40. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  41. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  42. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  43. def toString(): String
    Definition Classes
    AnyRef → Any
  44. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  45. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  46. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @throws( ... )

Inherited from Logging

Inherited from MetadataStoreManager

Inherited from AnyRef

Inherited from Any

Ungrouped