t

io.xskipper.metadatastore

MetadataStoreManager

trait MetadataStoreManager extends AnyRef

A trait each MetadataStoreManager should implement

Linear Supertypes
AnyRef, Any
Known Subclasses
ParquetMetadataStoreManager
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. MetadataStoreManager
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def createMetadataHandle(sparkSession: SparkSession, tableIdentifier: String): MetadataHandle

    Creates a new instance MetadataHandle Used by getOrCreateMetadataHandle in order to create new instances

  2. abstract def getDataSkippingFileFilterParams(tid: String, sparkSession: SparkSession, fileIndex: FileIndex): Map[String, String]

    Returns a map of parameters to be set by the DataSkippingFileFilter on the MetadataHandle when using this MetadataStoreManager.

    Returns a map of parameters to be set by the DataSkippingFileFilter on the MetadataHandle when using this MetadataStoreManager. This map can be used to pass specific parameters depending on the type of FileIndex that is being replaced. For example - in parquet.ParquetMetadataStoreManager we can specify that the metadata location should be inferred from the table/default db properties

    tid

    the table identifier for which the DataSkippingFileFilter will be created

    sparkSession

    the sparks session

    fileIndex

    the fileIndex that will be replaced by DataSkippingFileFilter

    returns

    a map of parameters to be set on the MetadataStore when the DataSkippingFileFilter is created

  3. abstract def getParams(sparkSession: SparkSession): Map[String, String]

    returns

    a map representing relevant MetadataStoreManager parameters

  4. abstract def getType: MetadataStoreManagerType

    returns

    the MetadataStoreManagerType associated with this MetadataStoreManager

  5. abstract def init(): Unit

    A unit function which enables to do setup for the MetadataStoreManager.

    A unit function which enables to do setup for the MetadataStoreManager. The setup is called once when the MetadataStoreManager is registered

  6. abstract def listIndexedDatasets(sparkSession: SparkSession): Map[String, (MetadataVersionStatus.MetadataVersionStatus, Seq[Index])]

    returns

    Map of indexed datasets to their respective md version status and indexes

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clearActiveMetadataHandles(): Unit

    Clear all active MetadataHandles managed by this MetadataStoreManager

  6. def clearStats(): Unit

    Clears the stats for all active MetadataHandle instances.

    Clears the stats for all active MetadataHandle instances. Should be called before each query (or call clearActiveMetadataHandles) to make sure the aggregated stats are cleared

  7. def clone(): AnyRef
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @native() @throws( ... )
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  10. def finalize(): Unit
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  11. def getActiveMetadataHandles(): Map[String, MetadataHandle]

    Returns all active MetadataHandles managed by this MetadataStoreManager

  12. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  13. def getLatestQueryAggregatedStats(): QueryIndexStatsResult

    Gets the aggregated latest query skipping stats for all active MetadataHandle instances.

    Gets the aggregated latest query skipping stats for all active MetadataHandle instances. In order to get reliable results it is assumed that either clearStats or clearActiveMetadataHandles was called before running the query.

    This is needed since the way we aggregate the skipping stats is by going over all active MetadataHandles of the MetadataStoreManager and aggregating their stats. When running multiple queries there could be a scenario in which the first query used dataset a and the second query didn't use it, therefore, when calling aggregatedStats for the second query the MetadataHandle for dataset a will be present as an active MetadataHandle therefore we need its stats to be cleared.

    returns

    QueryIndexStatsResult instance with the latest query aggregated stats

  14. def getOrCreateMetadataHandle(sparkSession: SparkSession, tableIdentifier: String): MetadataHandle

    Gets an existing MetadataHandle or, if there is no existing one, creates a new one.

    Gets an existing MetadataHandle or, if there is no existing one, creates a new one.

    sparkSession

    the sparks session

    tableIdentifier

    the table identifier of the dataset for which the MetadataHandle instance is created

    returns

    a new instance of the MetadataHandle associated with this backend

  15. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  16. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  17. val metadataHandlesInstanceMap: ConcurrentHashMap[String, MetadataHandle]
  18. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  19. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  20. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  21. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  22. def toString(): String
    Definition Classes
    AnyRef → Any
  23. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  25. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped