Value Members

1.  final def !=(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

2.  final def ##(): Int

   Definition Classes

   AnyRef → Any

3.  final def ==(arg0: Any): Boolean

   Definition Classes

   AnyRef → Any

4.  final def asInstanceOf[T0]: T0

   Definition Classes

   Any

5.  def clean(): Unit

   Cleans the MetadataHandle instance (implementation specific)

   Cleans the MetadataHandle instance (implementation specific)

   Definition Classes

   MetadataHandle

6.  def clearStats(): Unit

   Clears the latest stats

   Clears the latest stats

   Definition Classes

   MetadataHandle

7.  def clone(): AnyRef

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @native() @throws( ... )

8.  def dropAllMetadata(): Unit

   Drops all of the metadata associated with the given index

   Drops all of the metadata associated with the given index

   Definition Classes

   ParquetMetadataHandle → MetadataHandle

9.  final def eq(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

10.  def equals(arg0: Any): Boolean

    Definition Classes

    AnyRef → Any

11.  def finalize(): Unit

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

12.  def finalizeMetadataUpload(): Unit

    Finalize metadata creation in the metadatastore (implementation specific)

    Finalize metadata creation in the metadatastore (implementation specific)

    Definition Classes

    ParquetMetadataHandle → MetadataHandle

13.  def getAllIndexedFiles(filter: Option[Any]): Future[Set[String]]

    Returns a set of all indexed files (async)

    Returns a set of all indexed files (async)

    filter

    optional filter to apply (can be used to get all indexed file for a given partition)

    returns

    a set of all indexed files ids

    Definition Classes

    ParquetMetadataHandle → MetadataHandle

14.  final def getClass(): Class[_]

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

15.  def getDeletionChunkSize(): Int
    returns

    Maximum number of objects to delete in one chunk

    Definition Classes

    ParquetMetadataHandle → MetadataHandle

16.  def getIndexStatus(): IndexStatusResult

    Returns index statistics

    Returns index statistics

    Definition Classes

    ParquetMetadataHandle → MetadataHandle

17.  def getIndexes(): Seq[Index]

    returns the sequence of indexes that exist in the metadatastore for the tableIdentifier

    returns the sequence of indexes that exist in the metadatastore for the tableIdentifier

    Definition Classes

    ParquetMetadataHandle → MetadataHandle

18.  def getMDPath(): ParquetMetadataPath
    returns

    the metadata path of the current instance

19.  def getMdVersionStatus(): MetadataVersionStatus

    returns the version status of the metadata.

    returns the version status of the metadata. we do not have a strict requirement for the metadatastore to use metadata from a version different than its current version for filtering / refresh but we do expect it to be able to tell the version status, and whether or not it can be upgraded to comply with the current version.

    Definition Classes

    ParquetMetadataHandle → MetadataHandle

20.  def getNumberOfIndexedObjects(): Long

    Gets the number of indexed objects

    Gets the number of indexed objects

    returns

    the number of indexed objects

21.  def getRequiredObjects(query: Any, filter: Option[Any]): Future[Set[String]]

    Returns the required file ids for the given query (async)

    Returns the required file ids for the given query (async)

    query

    the query to be used in order to get the relevant files (this query is of type Any and it is the responsibility of the metadatastore implementation to cast it to as instance which matches the translation for this MetaDataStore)

    filter

    an optional filter to apply (can be used to get all indexed file for a given partition)

    returns

    the set of fileids required for this query

    Definition Classes

    ParquetMetadataHandle → MetadataHandle

22.  def getStats(): Option[QueryIndexStatsResult]

    Definition Classes

    MetadataHandle

23.  def getUploadChunkSize(): Int
    returns

    Maximum number of objects to index in one chunk

    Definition Classes

    ParquetMetadataHandle → MetadataHandle

24.  def hashCode(): Int

    Definition Classes

    AnyRef → Any

    Annotations

    @native()

25.  def indexExists(): Boolean

    Returns true if an index exists for the tableIdentifier

    Returns true if an index exists for the tableIdentifier

    returns

    true if the an index exists for the tableIdentifier

    Definition Classes

    ParquetMetadataHandle → MetadataHandle

26.  def initMetadataUpload(indexes: Seq[Index]): Unit

    Initialize metadata for a dataset in the metadatastore (implementation specific)

    Initialize metadata for a dataset in the metadatastore (implementation specific)

    indexes

    a sequence of metadata indexes

    Definition Classes

    MetadataHandle

27.  def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

    Attributes

    protected

    Definition Classes

    Logging

28.  def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Attributes

    protected

    Definition Classes

    Logging

29.  def isEncryptionSupported(): Boolean

    Returns whether or not this MetadataHandle supports encryption

    Returns whether or not this MetadataHandle supports encryption

    returns

    true if the MetadataHandle supports encryption

    Definition Classes

    ParquetMetadataHandle → MetadataHandle

30.  final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

31.  def isMetadataUpgradePossible(): Boolean

    returns whether or not the metadata can be upgraded to comply with the current version

    returns whether or not the metadata can be upgraded to comply with the current version

    returns

    true if the metadata can be upgraded, false otherwise

    Definition Classes

    ParquetMetadataHandle → MetadataHandle

32.  def isTraceEnabled(): Boolean

    Attributes

    protected

    Definition Classes

    Logging

33.  def log: Logger

    Attributes

    protected

    Definition Classes

    Logging

34.  def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

35.  def logDebug(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

36.  def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

37.  def logError(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

38.  def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

39.  def logInfo(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

40.  def logName: String

    Attributes

    protected

    Definition Classes

    Logging

41.  def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

42.  def logTrace(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

43.  def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes

    protected

    Definition Classes

    Logging

44.  def logWarning(msg: ⇒ String): Unit

    Attributes

    protected

    Definition Classes

    Logging

45.  final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

46.  final def notify(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

47.  final def notifyAll(): Unit

    Definition Classes

    AnyRef

    Annotations

    @native()

48.  def refresh(): Unit

    Refreshes the MetadataHandle by re-syncing with the metadatastore (implementation specific)

    Refreshes the MetadataHandle by re-syncing with the metadatastore (implementation specific)

    Definition Classes

    ParquetMetadataHandle → MetadataHandle

49.  def removeMetaDataForFiles(files: Seq[String]): Unit

    Removes the metadata for a sequence of files.

    Removes the metadata for a sequence of files. This method may assume that the Metadta version status is MetadataVersionStatus.CURRENT

    files

    a sequence of files for which the metadata will be removed

    Definition Classes

    ParquetMetadataHandle → MetadataHandle

50.  val session: SparkSession
51.  def setMDPath(md: String, type: String): ParquetMetadataHandle
52.  def setParams(params: Map[String, String]): Unit

    Set metadataStore specific parameters such as user, password, host Note: the implementation should not count on the Map not changing during the run therefore, it needs to save its parameters locally

    Set metadataStore specific parameters such as user, password, host Note: the implementation should not count on the Map not changing during the run therefore, it needs to save its parameters locally

    params

    a map of parameters (each metadata store expects certain input)

    Definition Classes

    ParquetMetadataHandle → MetadataHandle

53.  final def setParams(params: Map[String, String]): Unit

    Definition Classes

    MetadataHandle

54.  val stats: Option[QueryIndexStatsResult]

    Attributes

    protected

    Definition Classes

    MetadataHandle

55.  final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes

    AnyRef

56.  def toString(): String

    Definition Classes

    AnyRef → Any

57.  def updateQueryStats(currentFilterStatistics: QueryIndexStatsResult): Unit

    Updates the MetadataHandle stats with the given query stats called from the dataSkippingFileIndex rule in the catalyst to update the stats during the query tun time

    Updates the MetadataHandle stats with the given query stats called from the dataSkippingFileIndex rule in the catalyst to update the stats during the query tun time

    currentFilterStatistics

    the updated stats

    Definition Classes

    MetadataHandle

58.  def upgradeMetadata(indexes: Seq[Index], fileIndex: FileIndex): Unit

    Upgrades the metadata to comply with the current version

    Upgrades the metadata to comply with the current version

    indexes

    - the indexes stored in the metadataStore.

    fileIndex

    the file index of the indexed dataset or table

    Definition Classes

    ParquetMetadataHandle → MetadataHandle

59.  def uploadMetadata(metaData: RDD[Row], partitionSchema: Option[StructType], indexes: Seq[Index], isRefresh: Boolean): Unit

    Uploads the metadata to the metadatastore This method may assume that the metadata version status is MetadataVersionStatus.CURRENT

    Uploads the metadata to the metadatastore This method may assume that the metadata version status is MetadataVersionStatus.CURRENT

    metaData

    RDD that contains for each file a list of abstract metaData types to be uploaded

    indexes

    a sequence of indexes that created the metadata

    isRefresh

    indicates whether the operation is a refresh operation

    Definition Classes

    ParquetMetadataHandle → MetadataHandle

60.  final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

61.  final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

62.  final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @native() @throws( ... )