t

io.xskipper.metadatastore

MetadataHandle

trait MetadataHandle extends AnyRef

a trait each MetadataHandle should implement note that the tableIdentifier is not in the signature since it is likely that it will be passed in the constructor.

Linear Supertypes
AnyRef, Any
Known Subclasses
ParquetMetadataHandle
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. MetadataHandle
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def dropAllMetadata(): Unit

    Drops all of the metadata associated with the given index

  2. abstract def finalizeMetadataUpload(): Unit

    Finalize metadata creation in the metadatastore (implementation specific)

  3. abstract def getAllIndexedFiles(): Future[Set[String]]

    Returns a set of all indexed files (async)

  4. abstract def getDeletionChunkSize(): Int

    returns

    Maximum number of objects to delete in one chunk

  5. abstract def getIndexStatus(): IndexStatusResult

    Returns index statistics

  6. abstract def getIndexes(): Seq[Index]

    returns the sequence of indexes that exist in the metadatastore for the tableIdentifier

  7. abstract def getMdVersionStatus(): MetadataVersionStatus.MetadataVersionStatus

    returns the version status of the metadata.

    returns the version status of the metadata. we do not have a strict requirement for the metadatastore to use metadata from a version different than its current version for filtering / refresh but we do expect it to be able to tell the version status, and whether or not it can be upgraded to comply with the current version.

    returns

    the version of the metadata

  8. abstract def getRequiredObjects(query: Any): Future[Set[String]]

    Returns the required file ids for the given query (async)

    Returns the required file ids for the given query (async)

    query

    the query to be used in order to get the relevant files (this query is of type Any and it is the responsibility of the metadatastore implementation to cast it to as instance which matches the translation for this MetaDataStore)

    returns

    the set of fileids required for this query

  9. abstract def getUploadChunkSize(): Int

    returns

    Maximum number of objects to index in one chunk

  10. abstract def indexExists(): Boolean

    Returns true if an index exists for the tableIdentifier

    Returns true if an index exists for the tableIdentifier

    returns

    true if the an index exists for the tableIdentifier

  11. abstract def isMetadataUpgradePossible(): Boolean

    returns whether or not the metadata can be upgraded to comply with the current version

    returns whether or not the metadata can be upgraded to comply with the current version

    returns

    true if the metadata can be upgraded, false otherwise

  12. abstract def refresh(): Unit

    Refreshes the MetadataHandle by re-syncing with the metadatastore (implementation specific)

  13. abstract def removeMetaDataForFiles(files: Seq[String]): Unit

    Removes the metadata for a sequence of files.

    Removes the metadata for a sequence of files. This method may assume that the Metadta version status is MetadataVersionStatus.CURRENT

    files

    a sequence of files for which the metadata will be removed

  14. abstract def setParams(params: Map[String, String]): Unit

    Set metadataStore specific parameters such as user, password, host Note: the implementation should not count on the Map not changing during the run therefore, it needs to save its parameters locally

    Set metadataStore specific parameters such as user, password, host Note: the implementation should not count on the Map not changing during the run therefore, it needs to save its parameters locally

    params

    a map of parameters (each metadata store expects certain input)

  15. abstract def upgradeMetadata(indexes: Seq[Index]): Unit

    Upgrades the metadata to comply with the current version

    Upgrades the metadata to comply with the current version

    indexes

    - the indexes stored in the metadataStore.

  16. abstract def uploadMetadata(metaData: RDD[Row], indexes: Seq[Index], isRefresh: Boolean): Unit

    Uploads the metadata to the metadatastore This method may assume that the metadata version status is MetadataVersionStatus.CURRENT

    Uploads the metadata to the metadatastore This method may assume that the metadata version status is MetadataVersionStatus.CURRENT

    metaData

    RDD that contains for each file a list of abstract metaData types to be uploaded

    indexes

    a sequence of indexes that created the metadata

    isRefresh

    indicates whether the operation is a refresh operation

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clean(): Unit

    Cleans the MetadataHandle instance (implementation specific)

  6. def clearStats(): Unit

    Clears the latest stats

  7. def clone(): AnyRef
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @native() @throws( ... )
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  10. def finalize(): Unit
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  11. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def getStats(): Option[QueryIndexStatsResult]
  13. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  14. def initMetadataUpload(indexes: Seq[Index]): Unit

    Initialize metadata for a dataset in the metadatastore (implementation specific)

    Initialize metadata for a dataset in the metadatastore (implementation specific)

    indexes

    a sequence of metadata indexes

  15. def isEncryptionSupported(): Boolean

    Returns whether or not this MetadataHandle supports encryption

    Returns whether or not this MetadataHandle supports encryption

    returns

    true if the MetadataHandle supports encryption

  16. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  17. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  18. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  19. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  20. final def setParams(params: Map[String, String]): Unit
  21. val stats: Option[QueryIndexStatsResult]
    Attributes
    protected
  22. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  23. def toString(): String
    Definition Classes
    AnyRef → Any
  24. def updateQueryStats(currentFilterStatistics: QueryIndexStatsResult): Unit

    Updates the MetadataHandle stats with the given query stats called from the dataSkippingFileIndex rule in the catalyst to update the stats during the query tun time

    Updates the MetadataHandle stats with the given query stats called from the dataSkippingFileIndex rule in the catalyst to update the stats during the query tun time

    currentFilterStatistics

    the updated stats

  25. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  26. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  27. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped