o

io.xskipper.metadatastore.parquet

ParquetMetadataStoreManager

object ParquetMetadataStoreManager extends MetadataStoreManager with Logging

A MetadataBackend backed by Parquet objects

Linear Supertypes
Logging, MetadataStoreManager, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ParquetMetadataStoreManager
  2. Logging
  3. MetadataStoreManager
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clearActiveMetadataHandles(): Unit

    Clear all active MetadataHandles managed by this MetadataStoreManager

    Clear all active MetadataHandles managed by this MetadataStoreManager

    Definition Classes
    MetadataStoreManager
  6. def clearStats(): Unit

    Clears the stats for all active MetadataHandle instances.

    Clears the stats for all active MetadataHandle instances. Should be called before each query (or call clearActiveMetadataHandles) to make sure the aggregated stats are cleared

    Definition Classes
    MetadataStoreManager
  7. def clone(): AnyRef
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @native() @throws( ... )
  8. def createMetadataHandle(sparkSession: SparkSession, tableIdentifier: String): MetadataHandle

    Creates a new instance MetadataHandle Used by getOrCreateMetadataHandle in order to create new instances

    Creates a new instance MetadataHandle Used by getOrCreateMetadataHandle in order to create new instances

    Definition Classes
    ParquetMetadataStoreManager → MetadataStoreManager
  9. def disableSkippedObjectStats(): Unit

    Disables the Skipped Object stats, other fields are still maintained.

    Disables the Skipped Object stats, other fields are still maintained.

    Definition Classes
    MetadataStoreManager
  10. def enableSkippedObjectStats(): Unit

    Enables the skipped Objects stats

    Enables the skipped Objects stats

    Definition Classes
    MetadataStoreManager
  11. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  12. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  13. def finalize(): Unit
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  14. def getActiveMetadataHandles(): Map[String, MetadataHandle]

    Returns all active MetadataHandles managed by this MetadataStoreManager

    Returns all active MetadataHandles managed by this MetadataStoreManager

    Definition Classes
    MetadataStoreManager
  15. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  16. def getDataSkippingFileFilterParams(tid: String, sparkSession: SparkSession, fileIndex: FileIndex): Map[String, String]

    Returns a map of parameters to be set by the io.xskipper.search.DataSkippingFileFilter on the MetadataHandle when using this MetadataStoreManager.

    Returns a map of parameters to be set by the io.xskipper.search.DataSkippingFileFilter on the MetadataHandle when using this MetadataStoreManager. This map can be used to pass specific parameters depending on the type of FileIndex that is being replaced. For example - in ParquetMetadataStoreManager we can specify that the metadata location should be inferred from the table/default db properties

    tid

    the table identifier for which the DataSkippingFileFilter will be created

    sparkSession

    the sparks session

    fileIndex

    the fileIndex that will be replaced by io.xskipper.search.DataSkippingFileFilter

    returns

    a map of parameters to be set on the MetadataStore when the DataSkippingFileFilter is created

    Definition Classes
    ParquetMetadataStoreManager → MetadataStoreManager
  17. def getLatestQueryAggregatedStats(): QueryIndexStatsResult

    Gets the aggregated latest query skipping stats for all active MetadataHandle instances.

    Gets the aggregated latest query skipping stats for all active MetadataHandle instances. In order to get reliable results it is assumed that either clearStats or clearActiveMetadataHandles was called before running the query.

    This is needed since the way we aggregate the skipping stats is by going over all active MetadataHandles of the MetadataStoreManager and aggregating their stats. When running multiple queries there could be a scenario in which the first query used dataset a and the second query didn't use it, therefore, when calling aggregatedStats for the second query the MetadataHandle for dataset a will be present as an active MetadataHandle therefore we need its stats to be cleared.

    returns

    QueryIndexStatsResult instance with the latest query aggregated stats

    Definition Classes
    MetadataStoreManager
  18. def getOrCreateMetadataHandle(sparkSession: SparkSession, tableIdentifier: String): MetadataHandle

    Gets an existing MetadataHandle or, if there is no existing one, creates a new one.

    Gets an existing MetadataHandle or, if there is no existing one, creates a new one.

    sparkSession

    the sparks session

    tableIdentifier

    the table identifier of the dataset for which the MetadataHandle instance is created

    returns

    a new instance of the MetadataHandle associated with this backend

    Definition Classes
    MetadataStoreManager
  19. def getParams(sparkSession: SparkSession): Map[String, String]

    returns

    a map representing relevant MetadataStoreManager parameters

    Definition Classes
    ParquetMetadataStoreManager → MetadataStoreManager
  20. def getType: MetadataStoreManagerType

    returns

    the MetadataStoreManagerType associated with this backend

    Definition Classes
    ParquetMetadataStoreManager → MetadataStoreManager
  21. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  22. def init(): Unit

    A unit function for setting up the MetadataStoreManager.

    A unit function for setting up the MetadataStoreManager. The setup is called once when the MetadataStoreManager is registered

    Definition Classes
    ParquetMetadataStoreManager → MetadataStoreManager
  23. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  24. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  25. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  26. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  27. def listIndexedDatasets(sparkSession: SparkSession): Map[String, (MetadataVersionStatus, Seq[Index])]

    returns

    Map of indexed datasets to their respective md version status and indexes

    Definition Classes
    ParquetMetadataStoreManager → MetadataStoreManager
  28. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  29. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  30. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  31. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  32. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  33. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  34. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  35. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  36. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  37. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  38. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  39. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  40. val metadataHandlesInstanceMap: ConcurrentHashMap[String, MetadataHandle]
    Definition Classes
    MetadataStoreManager
  41. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  42. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  43. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  44. val objectStatsEnabled: Boolean
    Attributes
    protected
    Definition Classes
    MetadataStoreManager
  45. def skippedObjectStatsEnabled: Boolean

    whether the Skipped Object stats are enabled

    whether the Skipped Object stats are enabled

    Definition Classes
    MetadataStoreManager
  46. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  47. def toString(): String
    Definition Classes
    AnyRef → Any
  48. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  49. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  50. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @throws( ... )

Inherited from Logging

Inherited from MetadataStoreManager

Inherited from AnyRef

Inherited from Any

Ungrouped