MinMaxIndex

Instance Constructors

new MinMaxIndex(col: String, keyMetadata: Option[String] = None)

col
the column on which the index is applied

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
val col: String

the column on which the index is applied
var colsMap: Map[String, IndexField]

Definition Classes
Index
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
def generateBaseMetadata(): MetadataType

returns
"zero" value of the index - will be used for the first comparison to the object's rows data (by default this is null)

Definition Classes
Index
def generateColsMap(schemaMap: Map[String, (String, StructField)]): Unit

Generate the column map according to a given schema
Generate the column map according to a given schema
schemaMap
a map containing column names (as appear in the object) and their data types the key is the column name in lower case

Definition Classes
Index
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def getCols: Seq[String]

returns
the index columns (in lower case)

Definition Classes
Index
def getIndexCols: Iterable[IndexField]

returns
the columns which the indexed is defined on

Definition Classes
Index
def getKeyMetadata(): Option[String]

Definition Classes
Index
def getMetaDataTypeClassName(): String

returns
the (full) name of the MetaDataType class used by this index

Definition Classes
MinMaxIndex → Index
def getName: String

returns
the name of the index

Definition Classes
MinMaxIndex → Index
def getParams: Map[String, String]

returns
the index params map

Definition Classes
Index
def getRowMetadata(row: Row): Any

Gets a DataFrame row and extract the raw metadata needed by the index
Gets a DataFrame row and extract the raw metadata needed by the index
row
Row a row to be indexed
returns
raw metadata needed by the index or null if the row contain null value

Definition Classes
MinMaxIndex → Index
val indexCols: Seq[String]

Definition Classes
Index
def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

Attributes
protected
Definition Classes
Logging
def initializeLogIfNecessary(isInterpreter: Boolean): Unit

Attributes
protected
Definition Classes
Logging
def isEncrypted(): Boolean

Definition Classes
Index
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
var isOptimized: Boolean

Definition Classes
Index
def isTraceEnabled(): Boolean

Attributes
protected
Definition Classes
Logging
def isValid(df: DataFrame, schemaMap: Map[String, (String, DataType)]): Unit

Gets a DataFrame and checks whether it is valid for the index
Gets a DataFrame and checks whether it is valid for the index
df
the DataFrame to be checked
schemaMap
a map containing column names (as appear in the object) and their data types the key is the column name in lower case

Definition Classes
MinMaxIndex → Index
Exceptions thrown
[[XskipperException]] if invalid index
val keyMetadata: Option[String]
def log: Logger

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logName: String

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def optCollectMetaData(filePath: String, df: DataFrame, format: String, options: Map[String, String]): MetadataType

For some formats we might have an optimized way for collecting the metadata This function enables this by receiving the entire file DataFrame instead of processing it row by row (For example in Parquet we can read the min/max from the footer)
For some formats we might have an optimized way for collecting the metadata This function enables this by receiving the entire file DataFrame instead of processing it row by row (For example in Parquet we can read the min/max from the footer)
filePath
the path of the file that is being processed
df
a DataFrame with the file data
format
the format of the file
options
the options that were used to read the file
returns
the collected MetadataType or null if no metadata was collected

Definition Classes
Index
def reduce(accuMetadata: MetadataType, curr: Any): MetadataType

Given an accumulated metadata and new value - process the new value and returns an updated accumulated metadata
Given an accumulated metadata and new value - process the new value and returns an updated accumulated metadata
accuMetadata
accumulated metadata created by processing all values until curr
curr
new value to be processed
returns
updated metadata for the index

Definition Classes
MinMaxIndex → Index
def reduce(md1: MetadataType, md2: MetadataType): MetadataType

Same as above reduce given two accumulated metadata
Same as above reduce given two accumulated metadata
returns
updated metadata for the index

Definition Classes
MinMaxIndex → Index
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Docs: object MinMaxIndex | package index

case class MinMaxIndex(col: String, keyMetadata: Option[String] = None) extends Index with Product with Serializable

Instance Constructors

new MinMaxIndex(col: String, keyMetadata: Option[String] = None)

Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

final def asInstanceOf[T0]: T0

def clone(): AnyRef

val col: String

var colsMap: Map[String, IndexField]

final def eq(arg0: AnyRef): Boolean

def finalize(): Unit

def generateBaseMetadata(): MetadataType

def generateColsMap(schemaMap: Map[String, (String, StructField)]): Unit

final def getClass(): Class[_]

def getCols: Seq[String]

def getIndexCols: Iterable[IndexField]

def getKeyMetadata(): Option[String]

def getMetaDataTypeClassName(): String

def getName: String

def getParams: Map[String, String]

def getRowMetadata(row: Row): Any

val indexCols: Seq[String]

def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

def initializeLogIfNecessary(isInterpreter: Boolean): Unit

def isEncrypted(): Boolean

final def isInstanceOf[T0]: Boolean

var isOptimized: Boolean

def isTraceEnabled(): Boolean

def isValid(df: DataFrame, schemaMap: Map[String, (String, DataType)]): Unit

val keyMetadata: Option[String]

def log: Logger

def logDebug(msg: ⇒ String, throwable: Throwable): Unit

def logDebug(msg: ⇒ String): Unit

def logError(msg: ⇒ String, throwable: Throwable): Unit

def logError(msg: ⇒ String): Unit

def logInfo(msg: ⇒ String, throwable: Throwable): Unit

def logInfo(msg: ⇒ String): Unit

def logName: String

def logTrace(msg: ⇒ String, throwable: Throwable): Unit

def logTrace(msg: ⇒ String): Unit

def logWarning(msg: ⇒ String, throwable: Throwable): Unit

def logWarning(msg: ⇒ String): Unit

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def optCollectMetaData(filePath: String, df: DataFrame, format: String, options: Map[String, String]): MetadataType

def reduce(accuMetadata: MetadataType, curr: Any): MetadataType

def reduce(md1: MetadataType, md2: MetadataType): MetadataType

final def synchronized[T0](arg0: ⇒ T0): T0

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from Product

Inherited from Equals

Inherited from Index

Inherited from Logging

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped