…rate AQE tests for backends (apache#4428)

)

Co-authored-by: kyligence-git <gluten@kyligence.io>

if we build debug velox version, TestRuntime.GetResultIterator could fail since it passes an empty columns with an int schema(arrow has a debug check here):

DCHECK_EQ(schema->num_fields(), static_cast<int>(columns.size()));

We can simply call gluten::createZeroColumnBatch(1) to avoid it.

…ed (apache#4177)

…he#4447)

…s updated (apache#4444)

…artitioned by grouping keys (apache#4443)

[VL] support spark nanvl function.

…ction (apache#4439)

The SparkResourceUtil#getTaskSlots function in the codebase has a FIXME comment, expressing concern about its performance when dynamic allocation is enabled. After further analysis and understanding of Spark's dynamic allocation implementation, it has been determined that the number of executors may be changed, but the cores of each executor remains constant with dynamic allocation.

)

* [GLUTEN-1632][CH]Daily Update Clickhouse Version (20240120)

* fix build due to ClickHouse/ClickHouse#58902

---------

Co-authored-by: kyligence-git <gluten@kyligence.io>
Co-authored-by: Chang Chen <baibaichen@gmail.com>

de2f01 Fix aggregation over all-null keys with ignoreNullKeys = true (8422)
ec3bb6 Use same max-output-buffer-size config for all types of output buffers (8426)
a0dc74 Presto serializer code cleanup (8420)
f6a14d Document setNoCopy and setEmpty Simple Function APIs (8382)
0a2267 Clear hash table for each spilled partition in RowNumber (8413)
f62ac7 Add kPartitionPathAsLowerCase to control whether make the partition path as lower case (8409)
3486c7 Add hour Spark Function (8237)
d05045 Add new config API in AssertQueryBuilder (8431)
a33890 Extract the task utilities to ArbitratorTestUtil (8417)
e09a92 Treat timestamp partition value as local time (8432)
44cdc9 Fix simd::gatherBits for Mac M1 or when AVX2 is disabled (8415)
ccd54d Remove unused variables in velox/functions/lib/DateTimeFormatter.cpp (8436)
0a4eae Remove unused function from velox/functions/FunctionRegistry.cpp (8444)
afda02 Remove unused function from velox/parse/Expressions.cpp (8442)
e65967 Remove unused function from velox/tpch/gen/dbgen/text.cpp (8441)
000899 Remove unused variables in velox/common/memory/MemoryArbitrator.cpp (8440)
03af39 Remove unused variables in velox/tpch/gen/dbgen/speed_seed.cpp (8435)
46331f Remove unused variables in velox/common/base/SpillStats.cpp (8439)
88b81e Remove unused variables in velox/common/caching/SsdFile.cpp (8437)
f24fea Remove unused function from velox/serializers/PrestoSerializer.cpp (8443)
8ed27b Remove unused function from velox/dwio/dwrf/common/Common.cpp (8156)
f68bd2 Add support for arrays of complex types to array_min/max Presto functions (8430)
2e081d Add reverse(varbinary) Presto function (8429)
1ab046 Add hex Spark function (8202)
3154f7 Mismatched tags (8445)
264561 Scatter struct nulls when deserializing Presto wire format (8318)
f0ac5f Add support for UNKNOWN key to map_agg Presto aggregate function (8452)
3cf1f3 Remove deprecated configs kCastToIntByTruncate and `kCastStringToDateIsIso8601` (8352)
e4d05f Fix bucket counts in histogram metrics (8458)

…like joins

…tic-build-centos7-test (apache#4427)

063f2b Mismatch tags take 2 (8461)
5db9d8 Preserve recursive encodings while serializing PrestoPage (8249)
063f2b Refactor struct Converter and reduce template parameters (7915)

Currently when there is a filter on short column it doesn't get pushed to Velox and the whole scan is executed in vanilla spark.
Ex:
create table abc (a short, b int) using parquet
select * from abc where a = 1


This PR address this issue so we can have scan now in Velox.
Needed additional changes as there was bug with edge case on short filter like a > -32768 which could not filter the value -32768
(Fixes: apache#4222)
Prev plan for above query:

VeloxColumnarToRowExec
+- ^(1) FilterExecTransformer (isnotnull(a#120) AND (a#120 = 1)), false
   +- RowToVeloxColumnar
      +- *(1) ColumnarToRow
         +- FileScan parquet default.abc[a#120,b#121] Batched: true, DataFilters: [isnotnull(a#120), (a#120 = 1)], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/home/kapil/dev/src/Gluten/gluten-ut/spark33/target/scala-2.12/te..., PartitionFilters: [], PushedFilters: [IsNotNull(a), EqualTo(a,1)], ReadSchema: struct<a:smallint,b:int>
New Plan

VeloxColumnarToRowExec
+- ^(1) FilterExecTransformer (isnotnull(a#120) AND (a#120 = 1)), false
   +- ^(1) NativeFileNativeScan parquet default.abc[a#120,b#121] Batched: true, DataFilters: [isnotnull(a#120), (a#120 = 1)],

…age index. (apache#4476)

* update parquet file with page index.

* Adjust the file size metric as the parquet file has been modified.

* Add rule to rewrite multi-chilren count

* add comments

* nit

---------

Co-authored-by: Kent Yao <yao@apache.org>

…#4459)

## What changes were proposed in this pull request?

1. We should move `aggregateFunction` validation to `doValidateInternal`.
2. For `getAttrForAggregateExprs`, we can use simpler implementation to get the correct attributes from `aggregateExpressions`.
3. For `ConverterUtils.getAttrFromExpr`, we can use the `Expression.references` method to obtain the desired Attribute. In the original implementation, when iterating over `BinaryLike` types, the left child is used for iteration. We can use `reference.head` to get the same Attribute.

Due to the inclusion of custom agg in ClickHouse, the original logic has been retained, and the `getAttrForAggregateExprs` has been abstracted as an interface to accommodate different logics implemented by Velox and ClickHouse.

## How was this patch tested?

Exists CI.

…e#4489)

## What changes were proposed in this pull request?

This pr does two things

- propagate fallback reason to union and shuffle exchange during validation
- rename UnionExecTransformer to ColumnarUnionExec since we do not transform union in fact 

## How was this patch tested?

Pass CI

…rator stage types (apache#4478)

[VL] Refactor 'getLiteralValue' logic.

…se_ch/20240120 (apache#4496)

Upgrade clickhouse version to the commit 377f89518a6 of the rebase_ch/20240120

d9f0f17 Remove static from anonymous namespace in HiveDataSink.cpp (8466)
228745f Collect the information about long operator calls on task (8474)
f98fc99 Back out "Scatter struct nulls when deserializing Presto wire format" (8463)
43bad4e Change spill to use iobuf API of WriteFile (8473)
f0be156 Rename max_arbitrary_buffer_size (part 1) (8472)
3f617ee Add TableScan::curStatus_ to track down where TableScan is stuck. (8457)
d747ac3 Fix min/max(x, n) (8311)
ce18fe4 Test side-effect-free of Aggregate::extractXxx (8399)
9397755 Use Status as return type for rescaleWithRoundUp (8465)
0b2b073 Fix json_parse for large number (8455)
6f6c1a8 Skip empty row groups in parquet reader (8000)
713a66a Fix flaky test `DecimalUtilTest.divideWithRoundUp` (8380)
5e56843 Remove gtest dependency for velox_dwio_common_compression (8433)
3777dfc Enhance empty file handling (8459)
0d71afa Remove unused TTo template parameter from CastBaseTest::testThrow (8464)
e38603a Add aggregations over distinct inputs to AggregationFuzzer (8328)

a0fc422 Fix documentation for HiveConfig hive.s3.use-instance-credentials (8309)
fce32e4 Make nulls initialValue to false in EvalCtx#ensureErrorsVectorSize (8479)
8923756 Fix invalid write in ensureWritable (8450)
8c74d3e Add find_in_set Spark function (8358)
38f0b56 Fix int64::max() to double conversion issue (8177)
8bfebb3 Add support for tinyint/smallint inputs for date_add/sub Spark functions (8379)
3949a3b Remove Wshadow warning for velox/type/Filter.h (8267)
a94ed0e Add SingleThreadedTaskCursor (8406)

)

* [GLUTEN-1632][CH]Daily Update Clickhouse Version (20240124)

* fix build due to ClickHouse/ClickHouse#58886

* workaround BoundedReadBuffer's issue due to ClickHouse/ClickHouse#58886

---------

Co-authored-by: kyligence-git <gluten@kyligence.io>
Co-authored-by: Chang Chen <baibaichen@gmail.com>

…he#4507)

…e#4210)

What changes were proposed in this pull request?
(Fixes: apache#4147)

Notice: current pr must be tested and merged after Kyligence/ClickHouse#477 is merged

…#4457)

* [GLUTEN-4454][CH] Support write mergetree data for CH backend

Support write mergetree data for CH backend:

1. create table for the mergetree data, and also support to config the orderByKey, primaryKey for CH;
2. now support to write to partition table and bucket table;
3. insert overwrite doesn't support now

Example:

```
CREATE TABLE IF NOT EXISTS lineitem_mergetree_partition
(
 l_orderkey      bigint,
 l_partkey       bigint,
 l_suppkey       bigint,
 l_linenumber    bigint,
 l_quantity      double,
 l_extendedprice double,
 l_discount      double,
 l_tax           double,
 l_returnflag    string,
 l_linestatus    string,
 l_commitdate    date,
 l_receiptdate   date,
 l_shipinstruct  string,
 l_shipmode      string,
 l_comment       string,
 l_shipdate      date
)
USING clickhouse
PARTITIONED BY (l_shipdate, l_returnflag)
TBLPROPERTIES (orderByKey='l_orderkey',
               primaryKey='l_orderkey')
LOCATION '$basePath/lineitem_mergetree_partition'
```

```
insert into table lineitem_mergetree_partition
 select
 l_orderkey      ,
 l_partkey       ,
 l_suppkey       ,
 l_linenumber    ,
 l_quantity      ,
 l_extendedprice ,
 l_discount      ,
 l_tax           ,
 l_returnflag    ,
 l_linestatus    ,
 l_commitdate    ,
 l_receiptdate   ,
 l_shipinstruct  ,
 l_shipmode      ,
 l_comment,
 l_shipdate
 from lineitem
 where l_shipdate BETWEEN date'1992-01-01' AND date'1992-06-10'
```

Co-authored-by: liuneng <1398775315@qq.com>
Co-authored-by: Shuai li <loneylee@live.cn>

* add a todo

---------

Co-authored-by: liuneng <1398775315@qq.com>
Co-authored-by: Shuai li <loneylee@live.cn>

71bb6d0 Fix hash join spill issue caused by new internal state (#8520)
147ff30 Signal timeout when locking task in getLongRunningOpCalls (#8511)
54e87d8 Refactor TableWriter arbitration test (#8504)
077fd73 Add CAST(varchar as decimal) (apache#5307)
4cd32e3 Fix ExchangeBenchmark and PrefixSortAlgorithmBenchmark (#8493)
5fa6497 Refactor ParquetReader header to prevent including thrift headers (#8484)
b03643e Add from_unixtime Spark function (apache#7532)
ef8783f Avoid creating BlockingState for a yield driver thread (#8490)
1deb036 Upgrade fmt to 10.1.1 (from 8.0.1) (#7941)
aa6bc30 Add memory arbitration time runtime stats for operator (#8482)

…p column in input (child node) (apache#4441)

[VL] Fix Filter node falling back to Spark whenever there is Timestamp column in input (child node).

…angeExec node fallback info (apache#4527)

… makes data lost apache#4525

What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)

Fixes: apache#4521

How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)

unit tests

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

…#4520)

[VL] fix delta column mapping for struct type columns.

…4546)

* fix build due to apache#4457

(cherry picked from commit c91c18e)

* Rename Logger.h to LoggerExtend.h due to this PR(ClickHouse/ClickHouse#58831). It adds a Logger.h in the Common directory, which conflicts with our extern-local-engine/Common/Logger.h

* style

)

* [GLUTEN-1632][CH]Daily Update Clickhouse Version (20240127)

* fix build due to ClickHouse/ClickHouse#59228

---------

Co-authored-by: kyligence-git <gluten@kyligence.io>
Co-authored-by: Chang Chen <baibaichen@gmail.com>

Added support for CartesianProductExec

CartesianProductExec in Spark will get converted to NestedLoopJoin in Velox.

…ache#4555)

…pache#4553)

…nction (apache#4541)

…erating substrait plan (apache#4481)

…er (apache#4497)

Dataproc now uses Hadoop 3.3.6

This change adds support for Ubuntu 18.04 builds by adding the necessary setup steps in dev/vcpkg/setup-build-depends.sh

…pache#4550)

Co-authored-by: Kent Yao <yao@apache.org>

…pache#4563)

Support Gluten Build in Debian 12 environment

…apache#4573)

…nsformer to cpp-ch/local-engine/Parser to prevent duplicated expression evaluation (apache#4562)

* rename  files in scalar function parser

* move some backend specific logic from scala to cpp

* finish sha2

* finish sha1

* finish equalNullSafe

* finish size

* fix bugs

* finish position/locate

* finish explode/posexplode

* add and remove files

* revert files

…33 & Spark34 (apache#4580)

The patch removes the restriction that broadcast hash join and broadcast exchange should be enabled and validated at the same time. spark.gluten.sql.columnar.broadcastExchange spark.gluten.sql.columnar.broadcastJoin can be then turned on/off individually.

C2Rs / R2Cs will work as expected to convert between vanilla Spark broadcast relation and Gluten (Velox, as of now)'s broadcast relation.

By doing this, the related broadcast exchange+join coherent validation rules will be removed from core module. They will continue existing in backends-clickhouse module until we implement this feature for CH backend.

7bca88a  Fix GCC OOM when building by reducing parallelism. (#8592)
25e2a7e  Fix CI MacOS build (#8601)
8bbec59  Remove unnecessary check in BooleanMix#getFlatBool (#8595)
e68d502  Seperate prefix-sort encoder testing methods into test utils (#8494)
5b8b4bd  Optimize count(distinct <complex type>) (#8560)
a113acf  Fix flaky test reclaimFromNonReclaimableSortTableWriter (#8591)
7eeb28a  Parquet writer tests for supported compression kinds (apache#7332)
64ed660  Fix arrow include guards (#8217)

* Add postProcessPushDownFilter in SparkPlanExecApi

* fix IcebergScanTransformer build

* Fix UT for TPCH 22, since we didn't push down subquery for parquet

…ec (apache#4589)

…e#4590)

92f380b  Eliminate redundant Expr::addNulls calls (#8616)
09d07d3  Fix doc-gen-job so that it has 3 retries before failing. (#7706)
f4c326d  Add flag for registering Presto compatible signatures for Bitwise_*_agg functions (#8491)
eb6b585  Cleanup circleci script (#8618)
37ba7cb  Break up AllTypes.nonSortedSpillFunctions test (#8621)
d66b18f  Change cast(double as json) to pad 0 and use scientific notation (#8498)
a945094  Make DuckDbQueryRunner.createTable flush once after all rows are written instead of after every row (#8611)
3beff35  Optimize copying primitive type vectors (#8617)
fbbeb56  PyVelox implementation for Array Vector (apache#6100)
280fc86  Add fmt::format_as for SpillStats and MemoryArbitrator::Stats (#8608)
a65afdb  Make LocalExchangeSource.stop_ atomic (#8609)

)

* [GLUTEN-1632][CH]Daily Update Clickhouse Version (20240202)

* fix build due to ClickHouse/ClickHouse#59399

---------

Co-authored-by: kyligence-git <gluten@kyligence.io>
Co-authored-by: Chang Chen <baibaichen@gmail.com>