MetadataStoreManager

Abstract Value Members

abstract def createMetadataHandle(sparkSession: SparkSession, tableIdentifier: String): MetadataHandle

Creates a new instance MetadataHandle Used by getOrCreateMetadataHandle in order to create new instances
abstract def getDataSkippingFileFilterParams(tid: String, sparkSession: SparkSession, fileIndex: FileIndex): Map[String, String]

Returns a map of parameters to be set by the DataSkippingFileFilter on the MetadataHandle when using this MetadataStoreManager.
Returns a map of parameters to be set by the DataSkippingFileFilter on the MetadataHandle when using this MetadataStoreManager. This map can be used to pass specific parameters depending on the type of FileIndex that is being replaced. For example - in parquet.ParquetMetadataStoreManager we can specify that the metadata location should be inferred from the table/default db properties
tid
the table identifier for which the DataSkippingFileFilter will be created
sparkSession
the sparks session
fileIndex
the fileIndex that will be replaced by DataSkippingFileFilter
returns
a map of parameters to be set on the MetadataStore when the DataSkippingFileFilter is created
abstract def getParams(sparkSession: SparkSession): Map[String, String]

returns
a map representing relevant MetadataStoreManager parameters
abstract def getType: MetadataStoreManagerType

returns
the MetadataStoreManagerType associated with this MetadataStoreManager
abstract def init(): Unit

A unit function which enables to do setup for the MetadataStoreManager.
A unit function which enables to do setup for the MetadataStoreManager. The setup is called once when the MetadataStoreManager is registered
abstract def listIndexedDatasets(sparkSession: SparkSession): Map[String, (MetadataVersionStatus, Seq[Index])]

returns
Map of indexed datasets to their respective md version status and indexes

Concrete Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clearActiveMetadataHandles(): Unit

Clear all active MetadataHandles managed by this MetadataStoreManager
def clearStats(): Unit

Clears the stats for all active MetadataHandle instances.
Clears the stats for all active MetadataHandle instances. Should be called before each query (or call clearActiveMetadataHandles) to make sure the aggregated stats are cleared
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
def getActiveMetadataHandles(): Map[String, MetadataHandle]

Returns all active MetadataHandles managed by this MetadataStoreManager
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def getLatestQueryAggregatedStats(): QueryIndexStatsResult

Gets the aggregated latest query skipping stats for all active MetadataHandle instances.
Gets the aggregated latest query skipping stats for all active MetadataHandle instances. In order to get reliable results it is assumed that either clearStats or clearActiveMetadataHandles was called before running the query.
This is needed since the way we aggregate the skipping stats is by going over all active MetadataHandles of the MetadataStoreManager and aggregating their stats. When running multiple queries there could be a scenario in which the first query used dataset a and the second query didn't use it, therefore, when calling aggregatedStats for the second query the MetadataHandle for dataset a will be present as an active MetadataHandle therefore we need its stats to be cleared.
returns
QueryIndexStatsResult instance with the latest query aggregated stats
def getOrCreateMetadataHandle(sparkSession: SparkSession, tableIdentifier: String): MetadataHandle

Gets an existing MetadataHandle or, if there is no existing one, creates a new one.
Gets an existing MetadataHandle or, if there is no existing one, creates a new one.
sparkSession
the sparks session
tableIdentifier
the table identifier of the dataset for which the MetadataHandle instance is created
returns
a new instance of the MetadataHandle associated with this backend
def hashCode(): Int

Definition Classes
AnyRef → Any
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
val metadataHandlesInstanceMap: ConcurrentHashMap[String, MetadataHandle]
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
AnyRef → Any
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Doc: package metadatastore

trait MetadataStoreManager extends AnyRef

Abstract Value Members

abstract def createMetadataHandle(sparkSession: SparkSession, tableIdentifier: String): MetadataHandle

abstract def getDataSkippingFileFilterParams(tid: String, sparkSession: SparkSession, fileIndex: FileIndex): Map[String, String]

abstract def getParams(sparkSession: SparkSession): Map[String, String]

abstract def getType: MetadataStoreManagerType

abstract def init(): Unit

abstract def listIndexedDatasets(sparkSession: SparkSession): Map[String, (MetadataVersionStatus, Seq[Index])]

Concrete Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

def clearActiveMetadataHandles(): Unit

def clearStats(): Unit

def clone(): AnyRef

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def finalize(): Unit

def getActiveMetadataHandles(): Map[String, MetadataHandle]

final def getClass(): Class[_]

def getLatestQueryAggregatedStats(): QueryIndexStatsResult

def getOrCreateMetadataHandle(sparkSession: SparkSession, tableIdentifier: String): MetadataHandle

def hashCode(): Int

final def isInstanceOf[T0]: Boolean

val metadataHandlesInstanceMap: ConcurrentHashMap[String, MetadataHandle]

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from AnyRef

Inherited from Any

Ungrouped