org.apache.calcite.adapter.druid

Class DruidQuery

java.lang.Object

org.apache.calcite.rel.AbstractRelNode

org.apache.calcite.adapter.druid.DruidQuery

All Implemented Interfaces:

Cloneable, BindableRel, InterpretableRel, RelOptNode, RelNode, ArrayBindable, Bindable<Object[]>, Typed

public class DruidQuery
extends AbstractRelNode
implements BindableRel

Relational expression representing a scan of a Druid data set.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
private static class	DruidQuery.DruidQueryNode Interpreter node that executes a Druid query and sends the results to a Sink.
private static class	DruidQuery.JsonAggregation Aggregation element of a Druid "groupBy" or "topN" query.
private static class	DruidQuery.JsonCardinalityAggregation Aggregation element that calls the "cardinality" function.
private static class	DruidQuery.JsonCollation Collation element of a Druid "groupBy" query.
private static class	DruidQuery.JsonExpressionPostAgg Druid Json Expression post aggregate.
private static class	DruidQuery.JsonFilteredAggregation Aggregation element that contains a filter
private static class	DruidQuery.JsonLimit Collation element of a Druid "groupBy" query.
protected static class	DruidQuery.JsonPostAggregation Post-Aggregator Post aggregator abstract writer
static class	DruidQuery.QuerySpec Druid query specification.
private static class	DruidQuery.ScanQuery Druid Scan Query Body

Nested classes/interfaces inherited from interface org.apache.calcite.interpreter.InterpretableRel

InterpretableRel.InterpreterImplementor

Nested classes/interfaces inherited from interface org.apache.calcite.rel.RelNode

RelNode.Context

Field Summary

Fields

Modifier and Type	Field and Description
(package private) Map<SqlOperator,DruidSqlOperatorConverter>	converterOperatorMap This operator map provides DruidSqlOperatorConverter instance to convert a Calcite RexNode to Druid Expression when possible.
private static int	DAYS_IN_TEN_YEARS
static List<DruidSqlOperatorConverter>	DEFAULT_OPERATORS_LIST Provides a standard list of supported Calcite operators that can be converted to Druid Expressions.
protected static String	DRUID_QUERY_FETCH
(package private) DruidTable	druidTable
private static String	EXTRACT_COLUMN_NAME_PREFIX
private static String	FLOOR_COLUMN_NAME_PREFIX
(package private) com.google.common.collect.ImmutableList<org.joda.time.Interval>	intervals
protected DruidQuery.QuerySpec	querySpec
(package private) com.google.common.collect.ImmutableList<RelNode>	rels
(package private) RelOptTable	table
private static Pattern	VALID_SIG

Fields inherited from class org.apache.calcite.rel.AbstractRelNode

digest, id, rowType, traitSet

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	DruidQuery(RelOptCluster cluster, RelTraitSet traitSet, RelOptTable table, DruidTable druidTable, List<org.joda.time.Interval> intervals, List<RelNode> rels, Map<SqlOperator,DruidSqlOperatorConverter> converterOperatorMap) Creates a DruidQuery.

Method Summary

Modifier and Type	Method and Description
Enumerable<Object[]>	bind(DataContext dataContext)
protected static List<DruidQuery.JsonAggregation>	computeDruidJsonAgg(List<AggregateCall> aggCalls, List<String> aggNames, Project project, DruidQuery druidQuery) Translates aggregate calls to Druid DruidQuery.JsonAggregations when possible.
private static DruidJsonFilter	computeFilter(Filter filterRel, DruidQuery druidQuery) Translates Filter rel to Druid Filter Json object if possible.
protected static Pair<List<String>,List<VirtualColumn>>	computeProjectAsScan(Project projectRel, RelDataType inputRowType, DruidQuery druidQuery) Translates list of projects to Druid Column names and Virtual Columns if any We can not use Pair.zip(Object[], Object[]), since size can be different
protected static Pair<List<DimensionSpec>,List<VirtualColumn>>	computeProjectGroupSet(Project projectNode, ImmutableBitSet groupSet, RelDataType inputRowType, DruidQuery druidQuery) Computes the project group set.
RelOptCost	computeSelfCost(RelOptPlanner planner, RelMetadataQuery mq)
private DruidQuery.JsonLimit	computeSort(Integer fetch, List<Integer> collationIndexes, List<RelFieldCollation.Direction> collationDirections, ImmutableBitSet numericCollationIndexes, List<String> queryOutputFieldNames)
RelNode	copy(RelTraitSet traitSet, List<RelNode> inputs)
private static DruidQuery	create(RelOptCluster cluster, RelTraitSet traitSet, RelOptTable table, DruidTable druidTable, List<org.joda.time.Interval> intervals, List<RelNode> rels, Map<SqlOperator,DruidSqlOperatorConverter> converterOperatorMap) Creates a DruidQuery.
static DruidQuery	create(RelOptCluster cluster, RelTraitSet traitSet, RelOptTable table, DruidTable druidTable, List<RelNode> rels) Creates a DruidQuery.
static DruidQuery	create(RelOptCluster cluster, RelTraitSet traitSet, RelOptTable table, DruidTable druidTable, List<RelNode> rels, Map<SqlOperator,DruidSqlOperatorConverter> converterOperatorMap) Creates a DruidQuery.
protected DruidQuery.QuerySpec	deriveQuerySpec()
RelDataType	deriveRowType()
RelWriter	explainTerms(RelWriter pw)
static DruidQuery	extendQuery(DruidQuery query, List<org.joda.time.Interval> intervals) Extends a DruidQuery.
static DruidQuery	extendQuery(DruidQuery query, RelNode r) Extends a DruidQuery.
protected static String	extractColumnName(RexNode rexNode, RelDataType rowType, DruidQuery query)
static String	format(String message, Object... formatArgs) Equivalent of String.format(Locale.ENGLISH, message, formatArgs).
protected CalciteConnectionConfig	getConnectionConfig()
DruidTable	getDruidTable()
Class<Object[]>	getElementType()
private double	getIntervalCostMultiplier()
private static DruidQuery.JsonAggregation	getJsonAggregation(String name, AggregateCall aggCall, RexNode filterNode, String fieldName, String aggExpression, DruidQuery druidQuery)
protected Map<SqlOperator,DruidSqlOperatorConverter>	getOperatorConversionMap()
protected DruidQuery.QuerySpec	getQuery(RelDataType rowType, Filter filter, Project project, ImmutableBitSet groupSet, List<AggregateCall> aggCalls, List<String> aggNames, List<Integer> collationIndexes, List<RelFieldCollation.Direction> collationDirections, ImmutableBitSet numericCollationIndexes, Integer fetch, Project postProject, Filter havingFilter)
DruidQuery.QuerySpec	getQuerySpec()
String	getQueryString()
QueryType	getQueryType()
private double	getQueryTypeCostMultiplier()
RelOptTable	getTable()
TableScan	getTableScan()
protected int	getTimestampFieldIndex()
RelNode	getTopNode()
Node	implement(InterpretableRel.InterpreterImplementor implementor)
boolean	isValid(Litmus litmus, RelNode.Context context)
private static boolean	isValidLeafCast(RexNode rexNode)
(package private) static boolean	isValidSignature(String signature) Returns whether a signature represents an sequence of relational operators that can be translated into a valid Druid query.
(package private) static String	metadataQuery(String dataSourceName, List<org.joda.time.Interval> intervals) Generates a JSON string to query metadata about a data source.
private String	planAsGroupBy(List<DimensionSpec> groupByKeyDims, DruidJsonFilter jsonFilter, List<VirtualColumn> virtualColumnList, List<DruidQuery.JsonAggregation> aggregations, List<DruidQuery.JsonExpressionPostAgg> postAggregations, DruidQuery.JsonLimit limit, DruidJsonFilter havingFilter)
private String	planAsTimeSeries(List<DimensionSpec> groupByKeyDims, DruidJsonFilter jsonFilter, List<VirtualColumn> virtualColumnList, List<DruidQuery.JsonAggregation> aggregations, List<DruidQuery.JsonExpressionPostAgg> postAggregations, DruidQuery.JsonLimit limit, DruidJsonFilter havingFilter)
private String	planAsTopN(List<DimensionSpec> groupByKeyDims, DruidJsonFilter jsonFilter, List<VirtualColumn> virtualColumnList, List<DruidQuery.JsonAggregation> aggregations, List<DruidQuery.JsonExpressionPostAgg> postAggregations, DruidQuery.JsonLimit limit, DruidJsonFilter havingFilter)
void	register(RelOptPlanner planner)
(package private) String	signature() Returns a string describing the operations inside this query.
protected static Pair<String,ExtractionFunction>	toDruidColumn(RexNode rexNode, RelDataType rowType, DruidQuery druidQuery)
protected static void	writeArray(com.fasterxml.jackson.core.JsonGenerator generator, List<?> elements)
protected static void	writeField(com.fasterxml.jackson.core.JsonGenerator generator, String fieldName, Object o)
protected static void	writeFieldIf(com.fasterxml.jackson.core.JsonGenerator generator, String fieldName, Object o)
protected static void	writeObject(com.fasterxml.jackson.core.JsonGenerator generator, Object o)

Methods inherited from class org.apache.calcite.rel.AbstractRelNode

accept, accept, childrenAccept, collectVariablesSet, collectVariablesUsed, computeDigest, computeSelfCost, estimateRowCount, explain, getChildExps, getCluster, getCollationList, getConvention, getCorrelVariable, getDescription, getDigest, getExpectedInputRowType, getId, getInput, getInputs, getQuery, getRelTypeName, getRows, getRowType, getTraitSet, getVariablesSet, getVariablesStopped, isDistinct, isKey, isValid, metadata, onRegister, recomputeDigest, replaceInput, sole, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.calcite.rel.RelNode

accept, accept, childrenAccept, collectVariablesSet, collectVariablesUsed, computeSelfCost, estimateRowCount, explain, getChildExps, getCollationList, getConvention, getCorrelVariable, getExpectedInputRowType, getInput, getInputs, getQuery, getRelTypeName, getRows, getRowType, getVariablesSet, getVariablesStopped, isDistinct, isKey, isValid, metadata, onRegister, recomputeDigest, replaceInput

Methods inherited from interface org.apache.calcite.plan.RelOptNode

getCluster, getDescription, getDigest, getId, getTraitSet

Field Detail

DEFAULT_OPERATORS_LIST

public static final List<DruidSqlOperatorConverter> DEFAULT_OPERATORS_LIST

Provides a standard list of supported Calcite operators that can be converted to Druid Expressions. This can be used as is or re-adapted based on underline engine operator syntax.

querySpec

protected DruidQuery.QuerySpec querySpec

table

final RelOptTable table

druidTable

final DruidTable druidTable

intervals

final com.google.common.collect.ImmutableList<org.joda.time.Interval> intervals

rels

final com.google.common.collect.ImmutableList<RelNode> rels

converterOperatorMap

final Map<SqlOperator,DruidSqlOperatorConverter> converterOperatorMap

This operator map provides DruidSqlOperatorConverter instance to convert a Calcite RexNode to Druid Expression when possible.

VALID_SIG

private static final Pattern VALID_SIG

EXTRACT_COLUMN_NAME_PREFIX

private static final String EXTRACT_COLUMN_NAME_PREFIX

See Also:

Constant Field Values

FLOOR_COLUMN_NAME_PREFIX

private static final String FLOOR_COLUMN_NAME_PREFIX

See Also:

Constant Field Values

DRUID_QUERY_FETCH

protected static final String DRUID_QUERY_FETCH

See Also:

Constant Field Values

DAYS_IN_TEN_YEARS

private static final int DAYS_IN_TEN_YEARS

See Also:

Constant Field Values

Constructor Detail

DruidQuery

protected DruidQuery(RelOptCluster cluster,
                     RelTraitSet traitSet,
                     RelOptTable table,
                     DruidTable druidTable,
                     List<org.joda.time.Interval> intervals,
                     List<RelNode> rels,
                     Map<SqlOperator,DruidSqlOperatorConverter> converterOperatorMap)

Creates a DruidQuery.

Parameters:

cluster - Cluster

traitSet - Traits

table - Table

druidTable - Druid table

intervals - Intervals for the query

rels - Internal relational expressions

converterOperatorMap - mapping of Calcite Sql Operator to Druid Expression API.

Method Detail

isValidSignature

static boolean isValidSignature(String signature)

Returns whether a signature represents an sequence of relational operators that can be translated into a valid Druid query.

create

public static DruidQuery create(RelOptCluster cluster,
                                RelTraitSet traitSet,
                                RelOptTable table,
                                DruidTable druidTable,
                                List<RelNode> rels)

Creates a DruidQuery.

create

public static DruidQuery create(RelOptCluster cluster,
                                RelTraitSet traitSet,
                                RelOptTable table,
                                DruidTable druidTable,
                                List<RelNode> rels,
                                Map<SqlOperator,DruidSqlOperatorConverter> converterOperatorMap)

Creates a DruidQuery.

create

private static DruidQuery create(RelOptCluster cluster,
                                 RelTraitSet traitSet,
                                 RelOptTable table,
                                 DruidTable druidTable,
                                 List<org.joda.time.Interval> intervals,
                                 List<RelNode> rels,
                                 Map<SqlOperator,DruidSqlOperatorConverter> converterOperatorMap)

Creates a DruidQuery.

extendQuery

public static DruidQuery extendQuery(DruidQuery query,
                                     RelNode r)

Extends a DruidQuery.

extendQuery

public static DruidQuery extendQuery(DruidQuery query,
                                     List<org.joda.time.Interval> intervals)

Extends a DruidQuery.

toDruidColumn

protected static Pair<String,ExtractionFunction> toDruidColumn(RexNode rexNode,
                                                               RelDataType rowType,
                                                               DruidQuery druidQuery)

Parameters:

rexNode - leaf Input Ref to Druid Column

rowType - row type

druidQuery - druid query

Returns:

Pair of Column name and Extraction Function on the top of the input ref or of(null, null) when can not translate to valid Druid column

isValidLeafCast

private static boolean isValidLeafCast(RexNode rexNode)

Parameters:

rexNode - rexNode

Returns:

true if the operand is an inputRef and it is a valid Druid Cast operation

extractColumnName

@Nullable
protected static String extractColumnName(RexNode rexNode,
                                                    RelDataType rowType,
                                                    DruidQuery query)

Parameters:

rexNode - Druid input ref node

rowType - rowType

query - Druid Query

Returns:

Druid column name or null when not possible to translate.

format

public static String format(String message,
                            Object... formatArgs)

Equivalent of String.format(Locale.ENGLISH, message, formatArgs).

signature

String signature()

Returns a string describing the operations inside this query.

For example, "sfpahol" means TableScan (s) followed by Filter (f) followed by Project (p) followed by Aggregate (a) followed by Filter (h) followed by Project (o) followed by Sort (l).

See Also:

isValidSignature(String)

isValid

public boolean isValid(Litmus litmus,
                       RelNode.Context context)

Specified by:

isValid in interface RelNode

Overrides:

isValid in class AbstractRelNode

getOperatorConversionMap

protected Map<SqlOperator,DruidSqlOperatorConverter> getOperatorConversionMap()

copy

public RelNode copy(RelTraitSet traitSet,
                    List<RelNode> inputs)

Specified by:

copy in interface RelNode

Overrides:

copy in class AbstractRelNode

deriveRowType

public RelDataType deriveRowType()

Overrides:

deriveRowType in class AbstractRelNode

getTableScan

public TableScan getTableScan()

getTopNode

public RelNode getTopNode()

getTable

public RelOptTable getTable()

Specified by:

getTable in interface RelNode

Overrides:

getTable in class AbstractRelNode

getDruidTable

public DruidTable getDruidTable()

explainTerms

public RelWriter explainTerms(RelWriter pw)

Overrides:

explainTerms in class AbstractRelNode

computeSelfCost

public RelOptCost computeSelfCost(RelOptPlanner planner,
                                  RelMetadataQuery mq)

Specified by:

computeSelfCost in interface RelNode

Overrides:

computeSelfCost in class AbstractRelNode

getIntervalCostMultiplier

private double getIntervalCostMultiplier()

getQueryTypeCostMultiplier

private double getQueryTypeCostMultiplier()

register

public void register(RelOptPlanner planner)

Specified by:

register in interface RelNode

Overrides:

register in class AbstractRelNode

getElementType

public Class<Object[]> getElementType()

Specified by:

getElementType in interface ArrayBindable

Specified by:

getElementType in interface Typed

bind

public Enumerable<Object[]> bind(DataContext dataContext)

Specified by:

bind in interface Bindable<Object[]>

implement

public Node implement(InterpretableRel.InterpreterImplementor implementor)

Specified by:

implement in interface InterpretableRel

getQuerySpec

public DruidQuery.QuerySpec getQuerySpec()

deriveQuerySpec

protected DruidQuery.QuerySpec deriveQuerySpec()

getQueryType

public QueryType getQueryType()

getQueryString

public String getQueryString()

getConnectionConfig

protected CalciteConnectionConfig getConnectionConfig()

computeFilter

@Nullable
private static DruidJsonFilter computeFilter(@Nullable
                                                       Filter filterRel,
                                                       DruidQuery druidQuery)

Translates Filter rel to Druid Filter Json object if possible. Currently Filter rel input has to be Druid Table scan

Parameters:

filterRel - input filter rel

druidQuery - Druid query

Returns:

DruidJson Filter or null if can not translate one of filters

computeProjectAsScan

@Nullable
protected static Pair<List<String>,List<VirtualColumn>> computeProjectAsScan(@Nullable
                                                                                       Project projectRel,
                                                                                       RelDataType inputRowType,
                                                                                       DruidQuery druidQuery)

Translates list of projects to Druid Column names and Virtual Columns if any We can not use Pair.zip(Object[], Object[]), since size can be different

Parameters:

projectRel - Project Rel

druidQuery - Druid query

Returns:

Pair of list of Druid Columns and Expression Virtual Columns or null when can not translate one of the projects.

computeProjectGroupSet

@Nullable
protected static Pair<List<DimensionSpec>,List<VirtualColumn>> computeProjectGroupSet(@Nullable
                                                                                                Project projectNode,
                                                                                                ImmutableBitSet groupSet,
                                                                                                RelDataType inputRowType,
                                                                                                DruidQuery druidQuery)

Computes the project group set.

Parameters:

projectNode - Project under the Aggregates if any

groupSet - Ids of grouping keys as they are listed in projects list

inputRowType - Input row type under the project

druidQuery - Druid Query

Returns:

A list of DimensionSpec containing the group by dimensions, and a list of VirtualColumn containing Druid virtual column projections; or null, if translation is not possible. Note that the size of lists can be different.

computeDruidJsonAgg

@Nullable
protected static List<DruidQuery.JsonAggregation> computeDruidJsonAgg(List<AggregateCall> aggCalls,
                                                                                List<String> aggNames,
                                                                                @Nullable
                                                                                Project project,
                                                                                DruidQuery druidQuery)

Translates aggregate calls to Druid DruidQuery.JsonAggregations when possible.

Parameters:

aggCalls - List of AggregateCalls to translate

aggNames - List of aggregate names

project - Input project under the aggregate calls, or null if we have TableScan immediately under the Aggregate

druidQuery - Druid Query Rel

Returns:

List of valid Druid DruidQuery.JsonAggregations, or null if any of the aggregates is not supported

getQuery

protected DruidQuery.QuerySpec getQuery(RelDataType rowType,
                                        Filter filter,
                                        Project project,
                                        ImmutableBitSet groupSet,
                                        List<AggregateCall> aggCalls,
                                        List<String> aggNames,
                                        List<Integer> collationIndexes,
                                        List<RelFieldCollation.Direction> collationDirections,
                                        ImmutableBitSet numericCollationIndexes,
                                        Integer fetch,
                                        Project postProject,
                                        Filter havingFilter)

computeSort

private DruidQuery.JsonLimit computeSort(@Nullable
                                         Integer fetch,
                                         List<Integer> collationIndexes,
                                         List<RelFieldCollation.Direction> collationDirections,
                                         ImmutableBitSet numericCollationIndexes,
                                         List<String> queryOutputFieldNames)

Parameters:

fetch - limit to fetch

collationIndexes - index of fields as listed in query row output

collationDirections - direction of sort

numericCollationIndexes - flag of to determine sort comparator

queryOutputFieldNames - query output fields

Returns:

always an non null Json Limit object

planAsTimeSeries

@Nullable
private String planAsTimeSeries(List<DimensionSpec> groupByKeyDims,
                                          DruidJsonFilter jsonFilter,
                                          List<VirtualColumn> virtualColumnList,
                                          List<DruidQuery.JsonAggregation> aggregations,
                                          List<DruidQuery.JsonExpressionPostAgg> postAggregations,
                                          DruidQuery.JsonLimit limit,
                                          DruidJsonFilter havingFilter)

planAsTopN

@Nullable
private String planAsTopN(List<DimensionSpec> groupByKeyDims,
                                    DruidJsonFilter jsonFilter,
                                    List<VirtualColumn> virtualColumnList,
                                    List<DruidQuery.JsonAggregation> aggregations,
                                    List<DruidQuery.JsonExpressionPostAgg> postAggregations,
                                    DruidQuery.JsonLimit limit,
                                    DruidJsonFilter havingFilter)

planAsGroupBy

@Nullable
private String planAsGroupBy(List<DimensionSpec> groupByKeyDims,
                                       DruidJsonFilter jsonFilter,
                                       List<VirtualColumn> virtualColumnList,
                                       List<DruidQuery.JsonAggregation> aggregations,
                                       List<DruidQuery.JsonExpressionPostAgg> postAggregations,
                                       DruidQuery.JsonLimit limit,
                                       DruidJsonFilter havingFilter)

getJsonAggregation

@Nullable
private static DruidQuery.JsonAggregation getJsonAggregation(String name,
                                                                       AggregateCall aggCall,
                                                                       RexNode filterNode,
                                                                       String fieldName,
                                                                       String aggExpression,
                                                                       DruidQuery druidQuery)

writeField

protected static void writeField(com.fasterxml.jackson.core.JsonGenerator generator,
                                 String fieldName,
                                 Object o)
                          throws IOException

Throws:

IOException

writeFieldIf

protected static void writeFieldIf(com.fasterxml.jackson.core.JsonGenerator generator,
                                   String fieldName,
                                   Object o)
                            throws IOException

Throws:

IOException

writeArray

protected static void writeArray(com.fasterxml.jackson.core.JsonGenerator generator,
                                 List<?> elements)
                          throws IOException

Throws:

IOException

writeObject

protected static void writeObject(com.fasterxml.jackson.core.JsonGenerator generator,
                                  Object o)
                           throws IOException

Throws:

IOException

metadataQuery

static String metadataQuery(String dataSourceName,
                            List<org.joda.time.Interval> intervals)

Generates a JSON string to query metadata about a data source.

getTimestampFieldIndex

protected int getTimestampFieldIndex()

Returns:

index of the timestamp ref or -1 if not present