t

io.xskipper.metadatastore

MetadataHandle

trait MetadataHandle extends AnyRef

a trait each MetadataHandle should implement note that the tableIdentifier is not in the signature since it is likely that it will be passed in the constructor.

Linear Supertypes
AnyRef, Any
Known Subclasses
ParquetMetadataHandle
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. MetadataHandle
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def dropAllMetadata(): Unit

    Drops all of the metadata associated with the given index

  2. abstract def finalizeMetadataUpload(): Unit

    Finalize metadata creation in the metadatastore (implementation specific)

  3. abstract def getAllIndexedFiles(): Future[Set[String]]

    Returns a set of all indexed files (async)

  4. abstract def getDeletionChunkSize(): Int

    returns

    Maximum number of objects to delete in one chunk

  5. abstract def getIndexStatus(): IndexStatusResult

    Returns index statistics

  6. abstract def getIndexes(): Seq[Index]

    returns the sequence of indexes that exist in the metadatastore for the tableIdentifier

  7. abstract def getMdVersionStatus(): MetadataVersionStatus.MetadataVersionStatus

    returns the version status of the metadata.

    returns the version status of the metadata. we do not have a strict requirement for the metadatastore to use metadata from a version different than its current version for filtering / refresh but we do expect it to be able to tell the version status, and whether or not it can be upgraded to comply with the current version.

    returns

    the version of the metadata

  8. abstract def getRequiredObjects(query: Any): Future[Set[String]]

    Returns the required file ids for the given query (async)

    Returns the required file ids for the given query (async)

    query

    the query to be used in order to get the relevant files (this query is of type Any and it is the responsibility of the metadatastore implementation to cast it to as instance which matches the translation for this MetaDataStore)

    returns

    the set of fileids required for this query

  9. abstract def getUploadChunkSize(): Int

    returns

    Maximum number of objects to index in one chunk

  10. abstract def indexExists(): Boolean

    Returns true if an index exists for the tableIdentifier

    Returns true if an index exists for the tableIdentifier

    returns

    true if the an index exists for the tableIdentifier

  11. abstract def isMetadataUpgradePossible(): Boolean

    returns whether or not the metadata can be upgraded to comply with the current version

    returns whether or not the metadata can be upgraded to comply with the current version

    returns

    true if the metadata can be upgraded, false otherwise

  12. abstract def refresh(): Unit

    Refreshes the MetadataHandle by re-syncing with the metadatastore (implementation specific)

  13. abstract def removeMetaDataForFiles(files: Seq[String]): Unit

    Removes the metadata for a sequence of files.

    Removes the metadata for a sequence of files. This method may assume that the Metadta version status is MetadataVersionStatus.CURRENT

    files

    a sequence of files for which the metadata will be removed

  14. abstract def setParams(params: Map[String, String]): Unit

    Set metadataStore specific parameters such as user, password, host Note: the implementation should not count on the Map not changing during the run therefore, it needs to save its parameters locally

    Set metadataStore specific parameters such as user, password, host Note: the implementation should not count on the Map not changing during the run therefore, it needs to save its parameters locally

    params

    a map of parameters (each metadata store expects certain input)

  15. abstract def upgradeMetadata(indexes: Seq[Index]): Unit

    Upgrades the metadata to comply with the current version

    Upgrades the metadata to comply with the current version

    indexes

    - the indexes stored in the metadataStore.

  16. abstract def uploadMetadata(metaData: RDD[Row], indexes: Seq[Index], isRefresh: Boolean): Unit

    Uploads the metadata to the metadatastore This method may assume that the metadata version status is MetadataVersionStatus.CURRENT

    Uploads the metadata to the metadatastore This method may assume that the metadata version status is MetadataVersionStatus.CURRENT

    metaData

    RDD that contains for each file a list of abstract metaData types to be uploaded

    indexes

    a sequence of indexes that created the metadata

    isRefresh

    indicates whether the operation is a refresh operation

Concrete Value Members

  1. def clean(): Unit

    Cleans the MetadataHandle instance (implementation specific)

  2. def clearStats(): Unit

    Clears the latest stats

  3. def getStats(): Option[QueryIndexStatsResult]
  4. def initMetadataUpload(indexes: Seq[Index]): Unit

    Initialize metadata for a dataset in the metadatastore (implementation specific)

    Initialize metadata for a dataset in the metadatastore (implementation specific)

    indexes

    a sequence of metadata indexes

  5. def isEncryptionSupported(): Boolean

    Returns whether or not this MetadataHandle supports encryption

    Returns whether or not this MetadataHandle supports encryption

    returns

    true if the MetadataHandle supports encryption

  6. final def setParams(params: Map[String, String]): Unit
  7. def updateQueryStats(currentFilterStatistics: QueryIndexStatsResult): Unit

    Updates the MetadataHandle stats with the given query stats called from the dataSkippingFileIndex rule in the catalyst to update the stats during the query tun time

    Updates the MetadataHandle stats with the given query stats called from the dataSkippingFileIndex rule in the catalyst to update the stats during the query tun time

    currentFilterStatistics

    the updated stats