MetadataHandle

Abstract Value Members

abstract def dropAllMetadata(): Unit

Drops all of the metadata associated with the given index
abstract def finalizeMetadataUpload(): Unit

Finalize metadata creation in the metadatastore (implementation specific)
abstract def getAllIndexedFiles(): Future[Set[String]]

Returns a set of all indexed files (async)
abstract def getDeletionChunkSize(): Int

returns
Maximum number of objects to delete in one chunk
abstract def getIndexStatus(): IndexStatusResult

Returns index statistics
abstract def getIndexes(): Seq[Index]

returns the sequence of indexes that exist in the metadatastore for the tableIdentifier
abstract def getMdVersionStatus(): MetadataVersionStatus

returns the version status of the metadata.
returns the version status of the metadata. we do not have a strict requirement for the metadatastore to use metadata from a version different than its current version for filtering / refresh but we do expect it to be able to tell the version status, and whether or not it can be upgraded to comply with the current version.
returns
the version of the metadata
abstract def getRequiredObjects(query: Any): Future[Set[String]]

Returns the required file ids for the given query (async)
Returns the required file ids for the given query (async)
query
the query to be used in order to get the relevant files (this query is of type Any and it is the responsibility of the metadatastore implementation to cast it to as instance which matches the translation for this MetaDataStore)
returns
the set of fileids required for this query
abstract def getUploadChunkSize(): Int

returns
Maximum number of objects to index in one chunk
abstract def indexExists(): Boolean

Returns true if an index exists for the tableIdentifier
Returns true if an index exists for the tableIdentifier
returns
true if the an index exists for the tableIdentifier
abstract def isMetadataUpgradePossible(): Boolean

returns whether or not the metadata can be upgraded to comply with the current version
returns whether or not the metadata can be upgraded to comply with the current version
returns
true if the metadata can be upgraded, false otherwise
abstract def refresh(): Unit

Refreshes the MetadataHandle by re-syncing with the metadatastore (implementation specific)
abstract def removeMetaDataForFiles(files: Seq[String]): Unit

Removes the metadata for a sequence of files.
Removes the metadata for a sequence of files. This method may assume that the Metadta version status is MetadataVersionStatus.CURRENT
files
a sequence of files for which the metadata will be removed
abstract def setParams(params: Map[String, String]): Unit

Set metadataStore specific parameters such as user, password, host Note: the implementation should not count on the Map not changing during the run therefore, it needs to save its parameters locally
Set metadataStore specific parameters such as user, password, host Note: the implementation should not count on the Map not changing during the run therefore, it needs to save its parameters locally
params
a map of parameters (each metadata store expects certain input)
abstract def upgradeMetadata(indexes: Seq[Index]): Unit

Upgrades the metadata to comply with the current version
Upgrades the metadata to comply with the current version
indexes
- the indexes stored in the metadataStore.
abstract def uploadMetadata(metaData: RDD[Row], indexes: Seq[Index], isRefresh: Boolean): Unit

Uploads the metadata to the metadatastore This method may assume that the metadata version status is MetadataVersionStatus.CURRENT
Uploads the metadata to the metadatastore This method may assume that the metadata version status is MetadataVersionStatus.CURRENT
metaData
RDD that contains for each file a list of abstract metaData types to be uploaded
indexes
a sequence of indexes that created the metadata
isRefresh
indicates whether the operation is a refresh operation

Concrete Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clean(): Unit

Cleans the MetadataHandle instance (implementation specific)
def clearStats(): Unit

Clears the latest stats
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def getStats(): Option[QueryIndexStatsResult]
def hashCode(): Int

Definition Classes
AnyRef → Any
def initMetadataUpload(indexes: Seq[Index]): Unit

Initialize metadata for a dataset in the metadatastore (implementation specific)
Initialize metadata for a dataset in the metadatastore (implementation specific)
indexes
a sequence of metadata indexes
def isEncryptionSupported(): Boolean

Returns whether or not this MetadataHandle supports encryption
Returns whether or not this MetadataHandle supports encryption
returns
true if the MetadataHandle supports encryption
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
final def setParams(params: Map[String, String]): Unit
var stats: Option[QueryIndexStatsResult]

Attributes
protected
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
AnyRef → Any
def updateQueryStats(currentFilterStatistics: QueryIndexStatsResult): Unit

Updates the MetadataHandle stats with the given query stats called from the dataSkippingFileIndex rule in the catalyst to update the stats during the query tun time
Updates the MetadataHandle stats with the given query stats called from the dataSkippingFileIndex rule in the catalyst to update the stats during the query tun time
currentFilterStatistics
the updated stats
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Doc: package metadatastore

trait MetadataHandle extends AnyRef

Abstract Value Members

abstract def dropAllMetadata(): Unit

abstract def finalizeMetadataUpload(): Unit

abstract def getAllIndexedFiles(): Future[Set[String]]

abstract def getDeletionChunkSize(): Int

abstract def getIndexStatus(): IndexStatusResult

abstract def getIndexes(): Seq[Index]

abstract def getMdVersionStatus(): MetadataVersionStatus

abstract def getRequiredObjects(query: Any): Future[Set[String]]

abstract def getUploadChunkSize(): Int

abstract def indexExists(): Boolean

abstract def isMetadataUpgradePossible(): Boolean

abstract def refresh(): Unit

abstract def removeMetaDataForFiles(files: Seq[String]): Unit

abstract def setParams(params: Map[String, String]): Unit

abstract def upgradeMetadata(indexes: Seq[Index]): Unit

abstract def uploadMetadata(metaData: RDD[Row], indexes: Seq[Index], isRefresh: Boolean): Unit

Concrete Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

def clean(): Unit

def clearStats(): Unit

def clone(): AnyRef

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def finalize(): Unit

final def getClass(): Class[_]

def getStats(): Option[QueryIndexStatsResult]

def hashCode(): Int

def initMetadataUpload(indexes: Seq[Index]): Unit

def isEncryptionSupported(): Boolean

final def isInstanceOf[T0]: Boolean

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

final def setParams(params: Map[String, String]): Unit

var stats: Option[QueryIndexStatsResult]

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

def updateQueryStats(currentFilterStatistics: QueryIndexStatsResult): Unit

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from AnyRef

Inherited from Any

Ungrouped