Class

io.xskipper.index

Index

Related Doc: package index

Permalink

abstract class Index extends Serializable with Logging

Represents an abstract index for a metadata on a file.

Linear Supertypes
Logging, Serializable, Serializable, AnyRef, Any
Known Subclasses
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Index
  2. Logging
  3. Serializable
  4. Serializable
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new Index(params: Map[String, String], keyMetadata: Option[String], cols: String*)

    Permalink

    params

    a map containing the index parameters if exists

    keyMetadata

    optional key metadata for encryption

    cols

    a sequence of columns associated with the index

Abstract Value Members

  1. abstract def getMetaDataTypeClassName(): String

    Permalink

    returns

    the (full) name of the MetaDataType class used by this index

  2. abstract def getName: String

    Permalink

    returns

    the name of the index

  3. abstract def getRowMetadata(row: Row): Any

    Permalink

    Gets a DataFrame row and extract the raw metadata needed by the index

    Gets a DataFrame row and extract the raw metadata needed by the index

    row

    Row a row to be indexed

    returns

    raw metadata needed by the index or null if the row contain null value

  4. abstract def isValid(df: DataFrame, schemaMap: Map[String, (String, DataType)]): Unit

    Permalink

    Gets a DataFrame and checks whether it is valid for the index No need to check column existence as it is checked by the index builder

    Gets a DataFrame and checks whether it is valid for the index No need to check column existence as it is checked by the index builder

    df

    the DataFrame to be checked

    schemaMap

    a map containing column names (as appear in the object) and their data types the key is the column name in lower case

    Exceptions thrown

    [[XskipperException]] with the reason if invalid

  5. abstract def reduce(md1: MetadataType, md2: MetadataType): MetadataType

    Permalink

    Same as above reduce given two accumulated metadata

    Same as above reduce given two accumulated metadata

    returns

    updated metadata for the index

  6. abstract def reduce(accuMetadata: MetadataType, curr: Any): MetadataType

    Permalink

    Given an accumulated metadata and new value - process the new value and returns an updated accumulated metadata

    Given an accumulated metadata and new value - process the new value and returns an updated accumulated metadata

    accuMetadata

    accumulated metadata created by processing all values until curr

    curr

    new value to be processed

    returns

    updated metadata for the index

Concrete Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  6. var colsMap: Map[String, IndexField]

    Permalink
  7. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  8. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  9. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. def generateBaseMetadata(): MetadataType

    Permalink

    returns

    "zero" value of the index - will be used for the first comparison to the object's rows data (by default this is null)

  11. def generateColsMap(schemaMap: Map[String, (String, StructField)]): Unit

    Permalink

    Generate the column map according to a given schema

    Generate the column map according to a given schema

    schemaMap

    a map containing column names (as appear in the object) and their data types the key is the column name in lower case

  12. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  13. def getCols: Seq[String]

    Permalink

    returns

    the index columns (in lower case)

  14. def getIndexCols: Iterable[IndexField]

    Permalink

    returns

    the columns which the indexed is defined on

  15. def getKeyMetadata(): Option[String]

    Permalink
  16. def getParams: Map[String, String]

    Permalink

    returns

    the index params map

  17. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  18. val indexCols: Seq[String]

    Permalink
  19. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  20. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  21. def isEncrypted(): Boolean

    Permalink
  22. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  23. var isOptimized: Boolean

    Permalink
  24. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  25. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  26. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  27. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  28. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  29. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  30. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  31. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  32. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  33. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  34. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  35. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  36. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  37. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  38. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  39. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  40. def optCollectMetaData(filePath: String, df: DataFrame, format: String, options: Map[String, String]): MetadataType

    Permalink

    For some formats we might have an optimized way for collecting the metadata This function enables this by receiving the entire file DataFrame instead of processing it row by row (For example in Parquet we can read the min/max from the footer)

    For some formats we might have an optimized way for collecting the metadata This function enables this by receiving the entire file DataFrame instead of processing it row by row (For example in Parquet we can read the min/max from the footer)

    filePath

    the path of the file that is being processed

    df

    a DataFrame with the file data

    format

    the format of the file

    options

    the options that were used to read the file

    returns

    the collected MetadataType or null if no metadata was collected

  41. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  42. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  43. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  44. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  45. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Logging

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped