Apache Arrow Gandiva项目解析

最初知道Apache Arrow Gandiva是无意间看Arrow项目的时候看到的，冲着项目主页上的LLVM，JIT的字样，我还实际尝试在Ubuntu安装和运行了下，但最后因为实在想不清楚，在什么场景下能用上，就弃坑了😂

直到前几天，我读完NoisePage的论文和部分源码，总感觉Arrow和LLVM的结合在哪里见到过——就是Apache Arrow Gandiva，那干脆这回一并把源码看了，搞清楚这东西到底是什么

项目历史&现状简述

该项目由Dremio在2018年捐给Apache Arrow，现作为Apache Arrow的子项目之一（信息来源：Gandiva: A LLVM-based Analytical Expression Compiler for Apache Arrow）如果你再进一步深究的话，会发现Arrow当中有不少人现在就在Dremio中工作，而Dremio项目也使用Apache Arrow，而Gandiva则宣称为Dremio执行引擎的一部分

Gandiva最大的亮点是使用LLVM的自动向量化完成Arrow的向量化处理，而在LLVM部分当中，还实现了Project和Filter——这里如果加上Join和Aggregation操作，很多SQL操作就齐活了，如果你再把NoisePage算上的话，甚至能完成整套纯LLVM的Arrow CURD处理机制

虽然网传这个项目烂尾（根本就没这回事好吧😅），但事实是Gandiva一直都有commit进行维护，今年LLVM20出来以后也很快做了跟进

目前Gandiva有C和C++的相关库，但对于Rust版本的Arrow似乎就不提供相关支持了：Interfaces for gandiva bindings.

源码解析

代码下载于2025.6.24，所有代码均平铺在单层目录上

Gandiva源码的地址：https://github.com/apache/arrow/tree/main/cpp/src/gandiva

1
|-- CMakeLists.txt
2
|-- GandivaConfig.cmake.in
3
|-- annotator.cc
4
|-- annotator.h
5
|-- annotator_test.cc
6
|-- arrow.h
7
|-- basic_decimal_scalar.h
8
|-- bitmap_accumulator.cc
9
|-- bitmap_accumulator.h
10
|-- bitmap_accumulator_test.cc
11
|-- cache.cc
12
|-- cache.h
13
|-- cache_test.cc
14
|-- cast_time.cc
15
|-- compiled_expr.h
16
|-- condition.h
17
|-- configuration.cc
18
|-- configuration.h
19
|-- context_helper.cc
20
|-- date_utils.cc
21
|-- date_utils.h
22
|-- decimal_ir.cc
23
|-- decimal_ir.h
24
|-- decimal_scalar.h
25
|-- decimal_type_util.cc
26
|-- decimal_type_util.h
27
|-- decimal_type_util_test.cc
28
|-- decimal_xlarge.cc
29
|-- decimal_xlarge.h
30
|-- dex.h
31
|-- dex_visitor.h
32
|-- encrypt_utils.cc
33
|-- encrypt_utils.h
34
|-- encrypt_utils_test.cc
35
|-- engine.cc
36
|-- engine.h
37
|-- engine_llvm_test.cc
38
|-- eval_batch.h
39
|-- execution_context.h
40
|-- exported_funcs.cc
41
|-- exported_funcs.h
42
|-- exported_funcs_registry.cc
43
|-- exported_funcs_registry.h
44
|-- exported_funcs_registry_test.cc
45
|-- expr_decomposer.cc
46
|-- expr_decomposer.h
47
|-- expr_decomposer_test.cc
48
|-- expr_validator.cc
49
|-- expr_validator.h
50
|-- expression.cc
51
|-- expression.h
52
|-- expression_cache_key.h
53
|-- expression_registry.cc
54
|-- expression_registry.h
55
|-- expression_registry_test.cc
56
|-- external_c_functions.cc
57
|-- field_descriptor.h
58
|-- filter.cc
59
|-- filter.h
60
|-- formatting_utils.h
61
|-- func_descriptor.h
62
|-- function_holder.h
63
|-- function_holder_maker_registry.cc
64
|-- function_holder_maker_registry.h
65
|-- function_ir_builder.cc
66
|-- function_ir_builder.h
67
|-- function_registry.cc
68
|-- function_registry.h
69
|-- function_registry_arithmetic.cc
70
|-- function_registry_arithmetic.h
71
|-- function_registry_common.h
72
|-- function_registry_datetime.cc
73
|-- function_registry_datetime.h
74
|-- function_registry_hash.cc
75
|-- function_registry_hash.h
76
|-- function_registry_math_ops.cc
77
|-- function_registry_math_ops.h
78
|-- function_registry_string.cc
79
|-- function_registry_string.h
80
|-- function_registry_test.cc
81
|-- function_registry_timestamp_arithmetic.cc
82
|-- function_registry_timestamp_arithmetic.h
83
|-- function_signature.cc
84
|-- function_signature.h
85
|-- function_signature_test.cc
86
|-- gandiva.pc.in
87
|-- gandiva_aliases.h
88
|-- gandiva_object_cache.cc
89
|-- gandiva_object_cache.h
90
|-- gdv_function_stubs.cc
91
|-- gdv_function_stubs.h
92
|-- gdv_function_stubs_test.cc
93
|-- gdv_hash_function_stubs.cc
94
|-- gdv_string_function_stubs.cc
95
|-- hash_utils.cc
96
|-- hash_utils.h
97
|-- hash_utils_test.cc
98
|-- in_holder.h
99
|-- interval_holder.cc
100
|-- interval_holder.h
101
|-- interval_holder_test.cc
102
|-- literal_holder.cc
103
|-- literal_holder.h
104
|-- llvm_generator.cc
105
|-- llvm_generator.h
106
|-- llvm_generator_test.cc
107
|-- llvm_includes.h
108
|-- llvm_types.cc
109
|-- llvm_types.h
110
|-- llvm_types_test.cc
111
|-- local_bitmaps_holder.h
112
|-- lru_cache.h
113
|-- lru_cache_test.cc
114
|-- lvalue.h
115
|-- make_precompiled_bitcode.py
116
|-- native_function.h
117
|-- node.h
118
|-- node_visitor.h
119
|-- precompiled
120
|   |-- CMakeLists.txt
121
|   |-- arithmetic_ops.cc
122
|   |-- arithmetic_ops_test.cc
123
|   |-- bitmap.cc
124
|   |-- bitmap_test.cc
125
|   |-- decimal_ops.cc
126
|   |-- decimal_ops.h
127
|   |-- decimal_ops_test.cc
128
|   |-- decimal_wrapper.cc
129
|   |-- epoch_time_point.h
130
|   |-- epoch_time_point_test.cc
131
|   |-- extended_math_ops.cc
132
|   |-- extended_math_ops_test.cc
133
|   |-- hash.cc
134
|   |-- hash_test.cc
135
|   |-- print.cc
136
|   |-- string_ops.cc
137
|   |-- string_ops_test.cc
138
|   |-- testing.h
139
|   |-- time.cc
140
|   |-- time_constants.h
141
|   |-- time_fields.h
142
|   |-- time_test.cc
143
|   |-- timestamp_arithmetic.cc
144
|   `-- types.h
145
|-- precompiled_bitcode.cc.in
146
|-- projector.cc
147
|-- projector.h
148
|-- random_generator_holder.cc
149
|-- random_generator_holder.h
150
|-- random_generator_holder_test.cc
151
|-- regex_functions_holder.cc
152
|-- regex_functions_holder.h
153
|-- regex_functions_holder_test.cc
154
|-- regex_util.cc
155
|-- regex_util.h
156
|-- selection_vector.cc
157
|-- selection_vector.h
158
|-- selection_vector_impl.h
159
|-- selection_vector_test.cc
160
|-- simple_arena.h
161
|-- simple_arena_test.cc
162
|-- symbols.map
163
|-- tests
164
|   |-- CMakeLists.txt
165
|   |-- binary_test.cc
166
|   |-- boolean_expr_test.cc
167
|   |-- date_time_test.cc
168
|   |-- decimal_single_test.cc
169
|   |-- decimal_test.cc
170
|   |-- external_functions
171
|   |   |-- CMakeLists.txt
172
|   |   |-- multiply_by_two.cc
173
|   |   `-- multiply_by_two.h
174
|   |-- filter_project_test.cc
175
|   |-- filter_test.cc
176
|   |-- generate_data.h
177
|   |-- hash_test.cc
178
|   |-- huge_table_test.cc
179
|   |-- if_expr_test.cc
180
|   |-- in_expr_test.cc
181
|   |-- literal_test.cc
182
|   |-- micro_benchmarks.cc
183
|   |-- null_validity_test.cc
184
|   |-- projector_build_validation_test.cc
185
|   |-- projector_test.cc
186
|   |-- test_util.cc
187
|   |-- test_util.h
188
|   |-- timed_evaluate.h
189
|   |-- to_string_test.cc
190
|   `-- utf8_test.cc
191
|-- to_date_holder.cc
192
|-- to_date_holder.h
193
|-- to_date_holder_test.cc
194
|-- tree_expr_builder.cc
195
|-- tree_expr_builder.h
196
|-- tree_expr_test.cc
197
|-- value_validity_pair.h
198
`-- visibility.h

由于代码量极大，只选取部分进行分析

node

关于Tree的Node的定义

1
namespace gandiva {
2

3
class FieldNode;
4
class FunctionNode;
5
class IfNode;
6
class LiteralNode;
7
class BooleanNode;
8
template <typename Type>
9
class InExpressionNode;
10

11
/// \brief Visitor for nodes in the expression tree.
12
class GANDIVA_EXPORT NodeVisitor {
13
 public:
14
  virtual ~NodeVisitor() = default;
15

16
  virtual Status Visit(const FieldNode& node) = 0;
17
  virtual Status Visit(const FunctionNode& node) = 0;
18
  virtual Status Visit(const IfNode& node) = 0;
19
  virtual Status Visit(const LiteralNode& node) = 0;
20
  virtual Status Visit(const BooleanNode& node) = 0;
21
  virtual Status Visit(const InExpressionNode<int32_t>& node) = 0;
22
  virtual Status Visit(const InExpressionNode<int64_t>& node) = 0;
23
  virtual Status Visit(const InExpressionNode<float>& node) = 0;
24
  virtual Status Visit(const InExpressionNode<double>& node) = 0;
25
  virtual Status Visit(const InExpressionNode<gandiva::DecimalScalar128>& node) = 0;
26
  virtual Status Visit(const InExpressionNode<std::string>& node) = 0;
27
};
28

29
}  // namespace gandiva

tree_expr

1
tree_expr_test.cc
2
tree_expr_builder.cc
3
tree_expr_builder.h

用于解析计算树，比如4*5+3这种，通过TreeExprBuilder完成树的构建

1
TEST_F(TestExprTree, TestField) {
2
  Annotator annotator;
3

4
  auto n0 = TreeExprBuilder::MakeField(i0_);
5
  EXPECT_EQ(n0->return_type(), int32());
6

7
  auto n1 = TreeExprBuilder::MakeField(b0_);
8
  EXPECT_EQ(n1->return_type(), boolean());
9

10
  ExprDecomposer decomposer(*registry_, annotator);
11
  ValueValidityPairPtr pair;
12
  auto status = decomposer.Decompose(*n1, &pair);
13
  DCHECK_EQ(status.ok(), true) << status.message();
14

15
  auto value = pair->value_expr();
16
  auto value_dex = std::dynamic_pointer_cast<VectorReadFixedLenValueDex>(value);
17
  EXPECT_EQ(value_dex->FieldType(), boolean());
18

19
  EXPECT_EQ(pair->validity_exprs().size(), 1);
20
  auto validity = pair->validity_exprs().at(0);
21
  auto validity_dex = std::dynamic_pointer_cast<VectorReadValidityDex>(validity);
22
  EXPECT_NE(validity_dex->ValidityIdx(), value_dex->DataIdx());
23
}

借助函数重载,使用访问者模式，实现树的遍历与转换

1
class GANDIVA_EXPORT TreeExprBuilder {
2
 public:
3
  /// \brief create a node on a literal.
4
  static NodePtr MakeLiteral(bool value);
5
  static NodePtr MakeLiteral(uint8_t value);
6
  static NodePtr MakeLiteral(uint16_t value);
7
  static NodePtr MakeLiteral(uint32_t value);
8
  static NodePtr MakeLiteral(uint64_t value);
9
  static NodePtr MakeLiteral(int8_t value);
10
  static NodePtr MakeLiteral(int16_t value);
11
  static NodePtr MakeLiteral(int32_t value);
12
  static NodePtr MakeLiteral(int64_t value);
13
  static NodePtr MakeLiteral(float value);
14
  static NodePtr MakeLiteral(double value);
15
  static NodePtr MakeStringLiteral(const std::string& value);
16
  static NodePtr MakeBinaryLiteral(const std::string& value);
17
  static NodePtr MakeDecimalLiteral(const DecimalScalar128& value);

to_date_holder

完成字符串往时间的转化

1
EST_F(TestToDateHolder, TestSimpleDateTime) {
2
  EXPECT_OK_AND_ASSIGN(auto to_date_holder, ToDateHolder::Make("YYYY-MM-DD HH:MI:SS", 1));
3

4
  auto& to_date = *to_date_holder;
5
  bool out_valid;
6
  std::string s("1986-12-01 01:01:01");
7
  int64_t millis_since_epoch =
8
      to_date(&execution_context_, s.data(), (int)s.length(), true, &out_valid);
9
  EXPECT_EQ(millis_since_epoch, 533779200000);
10

11
  s = std::string("1986-12-01 01:01:01.11");
12
  millis_since_epoch =
13
      to_date(&execution_context_, s.data(), (int)s.length(), true, &out_valid);
14
  EXPECT_EQ(millis_since_epoch, 533779200000);
15

16
  s = std::string("1986-12-01 01:01:01 +0800");
17
  millis_since_epoch =
18
      to_date(&execution_context_, s.data(), (int)s.length(), true, &out_valid);
19
  EXPECT_EQ(millis_since_epoch, 533779200000);
20

21
#if 0
22
  // TODO : this fails parsing with date::parse and strptime on linux
23
  s = std::string("1886-12-01 00:00:00");
24
  millis_since_epoch =
25
      to_date(&execution_context_, s.data(), (int) s.length(), true, &out_valid);
26
  EXPECT_EQ(out_valid, true);
27
  EXPECT_EQ(millis_since_epoch, -2621894400000);
28
#endif
29

30
  s = std::string("1886-12-01 01:01:01");
31
  millis_since_epoch =
32
      to_date(&execution_context_, s.data(), (int)s.length(), true, &out_valid);
33
  EXPECT_EQ(millis_since_epoch, -2621894400000);
34

35
  s = std::string("1986-12-11 01:30:00");
36
  millis_since_epoch =
37
      to_date(&execution_context_, s.data(), (int)s.length(), true, &out_valid);
38
  EXPECT_EQ(millis_since_epoch, 534643200000);
39
}

simple_arena

没太理解内容，似乎是关于内存分配处理的内容，实现以Trunk为单位的内存分配

1
TEST_F(TestSimpleArena, TestAlloc) {
2
  int64_t chunk_size = 4096;
3
  SimpleArena arena(arrow::default_memory_pool(), chunk_size);
4

5
  // Small allocations should come from the same chunk.
6
  int64_t small_size = 100;
7
  for (int64_t i = 0; i < 20; ++i) {
8
    auto p = arena.Allocate(small_size);
9
    EXPECT_NE(p, nullptr);
10

11
    EXPECT_EQ(arena.total_bytes(), chunk_size);
12
    EXPECT_EQ(arena.avail_bytes(), chunk_size - (i + 1) * small_size);
13
  }
14

15
  // large allocations require separate chunks
16
  int64_t large_size = 100 * chunk_size;
17
  auto p = arena.Allocate(large_size);
18
  EXPECT_NE(p, nullptr);
19
  EXPECT_EQ(arena.total_bytes(), chunk_size + large_size);
20
  EXPECT_EQ(arena.avail_bytes(), 0);
21
}

selection_vector

实现对于Arrow格式存储的选择向量（Selection Vector）

这里需要补充下关于选择向量的相关知识

Selection Vector 是一种在数据处理系统中使用的技术，用来表示一批数据中哪些行被选中（有效），从而避免对不相关的数据行进行操作。它常见于列式数据库、矢量化执行引擎（如 Apache Arrow、Dremio、Gandiva）中，用于提升性能。

Selection Vector（选择向量）本质上是一个索引数组，存储的是被选中行在原始数据批中的下标。

避免复制数据：只需操作向量而不移动原始数据。

高效过滤：可以快速跳过不符合条件的行。

矢量化执行支持：配合批处理（batch processing），提升 SIMD 性能。

落到具体选择上，可能就是bitmap或是个Set

1
TEST_F(TestSelectionVector, TestInt16Set) {
2
  int max_slots = 10;
3

4
  std::shared_ptr<SelectionVector> selection;
5
  auto status = SelectionVector::MakeInt16(max_slots, pool_, &selection);
6
  EXPECT_EQ(status.ok(), true) << status.message();
7

8
  selection->SetIndex(0, 100);
9
  EXPECT_EQ(selection->GetIndex(0), 100);
10

11
  selection->SetIndex(1, 200);
12
  EXPECT_EQ(selection->GetIndex(1), 200);
13

14
  selection->SetNumSlots(2);
15
  EXPECT_EQ(selection->GetNumSlots(), 2);
16

17
  // TopArray() should return an array with 100,200
18
  auto array_raw = selection->ToArray();
19
  const auto& array = dynamic_cast<const arrow::UInt16Array&>(*array_raw);
20
  EXPECT_EQ(array.length(), 2) << array_raw->ToString();
21
  EXPECT_EQ(array.Value(0), 100) << array_raw->ToString();
22
  EXPECT_EQ(array.Value(1), 200) << array_raw->ToString();
23
}

也可以通过Bitmap实现向量选择

1
TEST_F(TestSelectionVector, TestInt64PopulateFromBitMap) {
2
  int max_slots = 200;
3

4
  std::shared_ptr<SelectionVector> selection;
5
  auto status = SelectionVector::MakeInt64(max_slots, pool_, &selection);
6
  EXPECT_EQ(status.ok(), true) << status.message();
7

8
  int bitmap_size = RoundUpNumi64(max_slots) * 8;
9
  std::vector<uint8_t> bitmap(bitmap_size);
10

11
  arrow::bit_util::SetBit(&bitmap[0], 0);
12
  arrow::bit_util::SetBit(&bitmap[0], 5);
13
  arrow::bit_util::SetBit(&bitmap[0], 121);
14
  arrow::bit_util::SetBit(&bitmap[0], 220);
15

16
  status = selection->PopulateFromBitMap(&bitmap[0], bitmap_size, max_slots - 1);
17
  EXPECT_EQ(status.ok(), true) << status.message();
18

19
  EXPECT_EQ(selection->GetNumSlots(), 3);
20
  EXPECT_EQ(selection->GetIndex(0), 0);
21
  EXPECT_EQ(selection->GetIndex(1), 5);
22
  EXPECT_EQ(selection->GetIndex(2), 121);
23
}

regex_functions/util

正则表达式相关，似乎能检测SQL相关的符号，这部分使用了Google的re2库，参考PCRE（Perl Compatible Regular Expressions）实现标准

1
const std::set<char> RegexUtil::pcre_regex_specials_ = {
2
    '[', ']', '(', ')', '|', '^', '-', '+', '*', '?', '{', '}', '$', '\\', '.'};

而测试也基本围绕些简易字符串展开

你甚至能看到关于中文字符的检测，这可太稀罕了，C++的UTF-8识别这块我一直摸不着头脑😂

1
  input_string = "路%c$大";
2
  extract_index = 2;  // Retrieve all matched string
3

4
  ret = extract_numbers(&execution_context_, input_string.c_str(),
5
                        static_cast<int32_t>(input_string.length()), extract_index,
6
                        &out_length);
7
  ret_as_str = std::string(ret, out_length);
8
  EXPECT_EQ(out_length, 1);
9
  EXPECT_EQ(ret_as_str, "c");

random_generator

随机数生成器，里面包含了随机种子信息

1
namespace gandiva {
2

3
/// Function Holder for 'random'
4
class GANDIVA_EXPORT RandomGeneratorHolder : public FunctionHolder {
5
 public:
6
  ~RandomGeneratorHolder() override = default;
7

8
  static Result<std::shared_ptr<RandomGeneratorHolder>> Make(const FunctionNode& node);
9

10
  double operator()() { return distribution_(generator_); }
11

12
 private:
13
  explicit RandomGeneratorHolder(int seed) : distribution_(0, 1) {
14
    int64_t seed64 = static_cast<int64_t>(seed);
15
    seed64 = (seed64 ^ 0x00000005DEECE66D) & 0x0000ffffffffffff;
16
    generator_.seed(static_cast<uint64_t>(seed64));
17
  }
18

19
  RandomGeneratorHolder() : distribution_(0, 1) {
20
    generator_.seed(::arrow::internal::GetRandomSeed());
21
  }
22

23
  std::mt19937_64 generator_;
24
  std::uniform_real_distribution<> distribution_;
25
};
26

27
}  // namespace gandiva

project

关于Gandiva如何处理Apache Arrow的Project的代码了，

/// \brief projection using expressions.

///

/// A projector is built for a specific schema and vector of expressions.

/// Once the projector is built, it can be used to evaluate many row batches.

看以看到实现中LLVM Generator，output_fields，是否使用已有的缓存，以及代码生成设置相关属性

1
  std::unique_ptr<LLVMGenerator> llvm_generator_;
2
  SchemaPtr schema_;
3
  FieldVector output_fields_;
4
  std::shared_ptr<Configuration> configuration_;
5
  bool built_from_cache_;
6
};

这里面还涉及了关于数据缓冲区的代码

1
Status Projector::AllocArrayData(const DataTypePtr& type, int64_t num_records,
2
                                 arrow::MemoryPool* pool,
3
                                 ArrayDataPtr* array_data) const {
4
  arrow::Status astatus;
5
  std::vector<std::shared_ptr<arrow::Buffer>> buffers;
6

7
  // The output vector always has a null bitmap.
8
  int64_t size = arrow::bit_util::BytesForBits(num_records);
9
  ARROW_ASSIGN_OR_RAISE(auto bitmap_buffer, arrow::AllocateBuffer(size, pool));
10
  buffers.push_back(std::move(bitmap_buffer));
11

12
  // String/Binary vectors have an offsets array.
13
  auto type_id = type->id();
14
  if (arrow::is_binary_like(type_id)) {
15
    auto offsets_len = arrow::bit_util::BytesForBits((num_records + 1) * 32);
16

17
    ARROW_ASSIGN_OR_RAISE(auto offsets_buffer, arrow::AllocateBuffer(offsets_len, pool));
18
    buffers.push_back(std::move(offsets_buffer));
19
  }
20

21
  // The output vector always has a data array.
22
  int64_t data_len;
23
  if (arrow::is_primitive(type_id) || type_id == arrow::Type::DECIMAL) {
24
    const auto& fw_type = static_cast<const arrow::FixedWidthType&>(*type);
25
    data_len = arrow::bit_util::BytesForBits(num_records * fw_type.bit_width());
26
  } else if (arrow::is_binary_like(type_id)) {
27
    // we don't know the expected size for varlen output vectors.
28
    data_len = 0;
29
  } else {
30
    return Status::Invalid("Unsupported output data type " + type->ToString());
31
  }
32
  ARROW_ASSIGN_OR_RAISE(auto data_buffer, arrow::AllocateResizableBuffer(data_len, pool));
33

34
  // This is not strictly required but valgrind gets confused and detects this
35
  // as uninitialized memory access. See arrow::util::SetBitTo().
36
  if (type->id() == arrow::Type::BOOL) {
37
    memset(data_buffer->mutable_data(), 0, data_len);
38
  }
39
  buffers.push_back(std::move(data_buffer));
40

41
  *array_data = arrow::ArrayData::Make(type, num_records, std::move(buffers));
42
  return Status::OK();
43
}

有点奇怪的是这部分内容没有没有配备test

lru_cache

从Boost库修改的LRU Cache，因为代码使用了模板，所以这里看不出来是存了什么

1
// modified from boost LRU cache -> the boost cache supported only an
2
// ordered map.
3
namespace gandiva {
4
// a cache which evicts the least recently used item when it is full
5
template <class Key, class Value>
6
class LruCache {
7
 public:
8
  using key_type = Key;
9
  using value_type = Value;
10
  using list_type = std::list<key_type>;

测试代码是直接使用string

1
TEST_F(TestLruCache, TestLruBehavior) {
2
  cache_.insert(TestCacheKey(1), "hello");
3
  cache_.insert(TestCacheKey(2), "hello");
4
  cache_.get(TestCacheKey(1));
5
  cache_.insert(TestCacheKey(3), "hello");
6
  // should have evicted key 2.
7
  ASSERT_EQ(*cache_.get(TestCacheKey(1)), "hello");
8
}

llvm_types

有一个llvm_types用于全局的types生成管理，用于映射Arrow的类型，这样的代码也能在NoisePage里面找到

1
class GANDIVA_EXPORT LLVMTypes {
2
 public:
3
  explicit LLVMTypes(llvm::LLVMContext& context);
4

5
  llvm::Type* void_type() { return llvm::Type::getVoidTy(context_); }
6

7
  llvm::Type* i1_type() { return llvm::Type::getInt1Ty(context_); }
8

9
  llvm::Type* i8_type() { return llvm::Type::getInt8Ty(context_); }
10

11
  llvm::Type* i16_type() { return llvm::Type::getInt16Ty(context_); }
12

13
  llvm::Type* i32_type() { return llvm::Type::getInt32Ty(context_); }
14

15
  llvm::Type* i64_type() { return llvm::Type::getInt64Ty(context_); }
16

17
  llvm::Type* i128_type() { return llvm::Type::getInt128Ty(context_); }
18

19
  llvm::StructType* i128_split_type() {
20
    // struct with high/low bits (see decimal_ops.cc:DecimalSplit)
21
    return llvm::StructType::get(context_, {i64_type(), i64_type()}, false);
22
  }

以及一些简单的内容初始化

1
  llvm::Constant* i128_zero() { return i128_constant(0); }
2
  llvm::Constant* i128_one() { return i128_constant(1); }

llvm_includes

开头的关闭MSVC的警告可以记录以下，这是我头一回遇到，看以看出Gandiva是能在Windows上面运行的

1
#if defined(_MSC_VER)
2
#  pragma warning(push)
3
#  pragma warning(disable : 4141)
4
#  pragma warning(disable : 4146)
5
#  pragma warning(disable : 4244)
6
#  pragma warning(disable : 4267)
7
#  pragma warning(disable : 4291)
8
#  pragma warning(disable : 4624)
9
#endif

甚至还考虑到了不同LLVM版本的情况

1
#if LLVM_VERSION_MAJOR >= 10
2
#  define LLVM_ALIGN(alignment) (llvm::Align((alignment)))
3
#else
4
#  define LLVM_ALIGN(alignment) (alignment)
5
#endif

llvm_generator

最为核心的LLVM代码生成

生成器似乎可以对缓存有效利用

1
class GANDIVA_EXPORT LLVMGenerator {
2
 public:
3
  /// \brief Factory method to initialize the generator.
4
  static Result<std::unique_ptr<LLVMGenerator>> Make(
5
      const std::shared_ptr<Configuration>& config, bool cached,
6
      std::optional<std::reference_wrapper<GandivaObjectCache>> object_cache =
7
          std::nullopt);
8

9
  /// \brief Get the cache to be used for LLVM ObjectCache.
10
  static std::shared_ptr<Cache<ExpressionCacheKey, std::shared_ptr<llvm::MemoryBuffer>>>
11
  GetCache();

存储关于SelectionVector：：Mode的信息

1
SelectionVector::Mode selection_vector_mode() { return selection_vector_mode_; }

build将表达式输入生成代码

1
  /// \brief Build the code for the expression trees for default mode with a LLVM
2
  /// ObjectCache. Each element in the vector represents an expression tree
3
  Status Build(const ExpressionVector& exprs, SelectionVector::Mode mode);
4

5
  /// \brief Build the code for the expression trees for default mode. Each
6
  /// element in the vector represents an expression tree
7
  Status Build(const ExpressionVector& exprs);

execute将Arrow量输入LLVM IR函数

1
  /// \brief Execute the built expression against the provided arguments for
2
  /// default mode.
3
  Status Execute(const arrow::RecordBatch& record_batch,
4
                 const ArrayDataVector& output_vector) const;
5

6
  /// \brief Execute the built expression against the provided arguments for
7
  /// all modes. Only works on the records specified in the selection_vector.
8
  Status Execute(const arrow::RecordBatch& record_batch,
9
                 const SelectionVector* selection_vector,
10
                 const ArrayDataVector& output_vector) const;

基本LLVMContext和IRbuilder自然是少不了，但这里的创建Global String居然不用检查重复，不知道是疏忽，还是因为前边有检查😂

1
  llvm::LLVMContext* context() { return engine_->context(); }
2
  llvm::IRBuilder<>* ir_builder() { return engine_->ir_builder(); }
3
  llvm::Constant* CreateGlobalStringPtr(const std::string& string) {
4
    return engine_->CreateGlobalStringPtr(string);
5
  }

然后Vistor模式重新过一遍解析树

1
 class Visitor : public DexVisitor {
2
   public:
3
    Visitor(LLVMGenerator* generator, llvm::Function* function,
4
            llvm::BasicBlock* entry_block, llvm::Value* arg_addrs,
5
            llvm::Value* arg_local_bitmaps, llvm::Value* arg_holder_ptrs,
6
            std::vector<llvm::Value*> slice_offsets, llvm::Value* arg_context_ptr,
7
            llvm::Value* loop_var);
8

9
    void Visit(const VectorReadValidityDex& dex) override;
10
    void Visit(const VectorReadFixedLenValueDex& dex) override;
11
    void Visit(const VectorReadVarLenValueDex& dex) override;
12
    void Visit(const LocalBitMapValidityDex& dex) override;
13
    void Visit(const TrueDex& dex) override;
14
    void Visit(const FalseDex& dex) override;
15
    void Visit(const LiteralDex& dex) override;
16
    void Visit(const NonNullableFuncDex& dex) override;
17
    void Visit(const NullableNeverFuncDex& dex) override;
18
    void Visit(const NullableInternalFuncDex& dex) override;
19
    void Visit(const IfDex& dex) override;
20
    void Visit(const BooleanAndDex& dex) override;
21
    void Visit(const BooleanOrDex& dex) override;
22
    void Visit(const InExprDexBase<int32_t>& dex) override;
23
    void Visit(const InExprDexBase<int64_t>& dex) override;
24
    void Visit(const InExprDexBase<float>& dex) override;
25
    void Visit(const InExprDexBase<double>& dex) override;
26
    void Visit(const InExprDexBase<gandiva::DecimalScalar128>& dex) override;
27
    void Visit(const InExprDexBase<std::string>& dex) override;
28
    template <typename Type>
29
    void VisitInExpression(const InExprDexBase<Type>& dex);
30

31
    LValuePtr result() { return result_; }
32

33
    bool has_arena_allocs() { return has_arena_allocs_; }

还有专门关于LLVM函数生成与函数调用的函数

1
    std::vector<llvm::Value*> BuildParams(int holder_idx,
2
                                          const ValueValidityPairVector& args,
3
                                          bool with_validity, bool with_context);
4

5
    // Generate code to invoke a function call.
6
    LValuePtr BuildFunctionCall(const NativeFunction* func, DataTypePtr arrow_return_type,
7
                                std::vector<llvm::Value*>* params);
8

9
    // Generate code for an if-else condition.
10
    LValuePtr BuildIfElse(llvm::Value* condition, std::function<LValuePtr()> then_func,
11
                          std::function<LValuePtr()> else_func,
12
                          DataTypePtr arrow_return_type);

通过接口添加预定义的LLVM IR函数

1
  /// Generate code to make a function call (to a pre-compiled IR function) which takes
2
  /// 'args' and has a return type 'ret_type'.
3
  llvm::Value* AddFunctionCall(const std::string& full_name, llvm::Type* ret_type,
4
                               const std::vector<llvm::Value*>& args);

关于Cache的详细实现

1
std::shared_ptr<Cache<ExpressionCacheKey, std::shared_ptr<llvm::MemoryBuffer>>>
2
LLVMGenerator::GetCache() {
3
  static std::shared_ptr<Cache<ExpressionCacheKey, std::shared_ptr<llvm::MemoryBuffer>>>
4
      shared_cache = std::make_shared<
5
          Cache<ExpressionCacheKey, std::shared_ptr<llvm::MemoryBuffer>>>();
6

7
  return shared_cache;
8
}
9

10
Status LLVMGenerator::SetLLVMObjectCache(GandivaObjectCache& object_cache) {
11
  return engine_->SetLLVMObjectCache(object_cache);
12
}

build的部分实现

1
Status LLVMGenerator::Build(const ExpressionVector& exprs, SelectionVector::Mode mode) {
2
  selection_vector_mode_ = mode;
3

4
  for (auto& expr : exprs) {
5
    auto output = annotator_.AddOutputFieldDescriptor(expr->result());
6
    ARROW_RETURN_NOT_OK(Add(expr, output));
7
  }
8

9
  // Compile and inject into the process' memory the generated function.
10
  ARROW_RETURN_NOT_OK(engine_->FinalizeModule());
11

12
  // setup the jit functions for each expression.
13
  for (auto& compiled_expr : compiled_exprs_) {
14
    auto fn_name = compiled_expr->GetFunctionName(mode);
15
    ARROW_ASSIGN_OR_RAISE(auto fn_ptr, engine_->CompiledFunction(fn_name));
16
    auto jit_fn = reinterpret_cast<EvalFunc>(fn_ptr);
17
    compiled_expr->SetJITFunction(selection_vector_mode_, jit_fn);
18
  }
19

20
  return Status::OK();
21
}

这部分的详细内容有空的话值得细看，而关于Test的话，这边给的示范样例是LLVM自动向量化向量加

1
TEST_F(TestLLVMGenerator, TestAdd) {
2
  // Setup LLVM generator to do an arithmetic add of two vectors
3
  ASSERT_OK_AND_ASSIGN(auto generator,
4
                       LLVMGenerator::Make(TestConfigWithIrDumping(), false));
5
  Annotator annotator;
6

7
  auto field0 = std::make_shared<arrow::Field>("f0", arrow::int32());
8
  auto desc0 = annotator.CheckAndAddInputFieldDescriptor(field0);
9
  auto validity_dex0 = std::make_shared<VectorReadValidityDex>(desc0);
10
  auto value_dex0 = std::make_shared<VectorReadFixedLenValueDex>(desc0);
11
  auto pair0 = std::make_shared<ValueValidityPair>(validity_dex0, value_dex0);
12

13
  auto field1 = std::make_shared<arrow::Field>("f1", arrow::int32());
14
  auto desc1 = annotator.CheckAndAddInputFieldDescriptor(field1);
15
  auto validity_dex1 = std::make_shared<VectorReadValidityDex>(desc1);
16
  auto value_dex1 = std::make_shared<VectorReadFixedLenValueDex>(desc1);
17
  auto pair1 = std::make_shared<ValueValidityPair>(validity_dex1, value_dex1);
18

19
  DataTypeVector params{arrow::int32(), arrow::int32()};
20
  auto func_desc = std::make_shared<FuncDescriptor>("add", params, arrow::int32());
21
  FunctionSignature signature(func_desc->name(), func_desc->params(),
22
                              func_desc->return_type());
23
  const NativeFunction* native_func =
24
      generator->function_registry_->LookupSignature(signature);
25

26
  std::vector<ValueValidityPairPtr> pairs{pair0, pair1};
27
  auto func_dex = std::make_shared<NonNullableFuncDex>(
28
      func_desc, native_func, FunctionHolderPtr(nullptr), -1, pairs);
29

30
  auto field_sum = std::make_shared<arrow::Field>("out", arrow::int32());
31
  auto desc_sum = annotator.CheckAndAddInputFieldDescriptor(field_sum);
32

33
  // LLVM 10 doesn't like the expr function name to be the same as the module name when
34
  // LLJIT is used
35
  std::string fn_name = "llvm_gen_test_add_expr";
36

37
  ASSERT_OK(generator->engine_->LoadFunctionIRs());
38
  ASSERT_OK(generator->CodeGenExprValue(func_dex, 4, desc_sum, 0, fn_name,
39
                                        SelectionVector::MODE_NONE));
40

41
  ASSERT_OK(generator->engine_->FinalizeModule());
42
  auto const& ir = generator->engine_->ir();
43
  EXPECT_THAT(ir, testing::HasSubstr("vector.body"));
44

45
  ASSERT_OK_AND_ASSIGN(auto fn_ptr, generator->engine_->CompiledFunction(fn_name));
46
  ASSERT_TRUE(fn_ptr);
47

48
  auto eval_func = reinterpret_cast<EvalFunc>(fn_ptr);
49
  constexpr size_t kNumRecords = 4;
50
  std::array<uint32_t, kNumRecords> a0{1, 2, 3, 4};
51
  std::array<uint32_t, kNumRecords> a1{5, 6, 7, 8};
52
  uint64_t in_bitmap = 0xffffffffffffffffull;
53

54
  std::array<uint32_t, kNumRecords> out{0, 0, 0, 0};
55
  uint64_t out_bitmap = 0;
56

57
  std::array<uint8_t*, 6> addrs{
58
      reinterpret_cast<uint8_t*>(a0.data()),  reinterpret_cast<uint8_t*>(&in_bitmap),
59
      reinterpret_cast<uint8_t*>(a1.data()),  reinterpret_cast<uint8_t*>(&in_bitmap),
60
      reinterpret_cast<uint8_t*>(out.data()), reinterpret_cast<uint8_t*>(&out_bitmap),
61
  };
62
  std::array<int64_t, 6> addr_offsets{0, 0, 0, 0, 0, 0};
63

64
  eval_func(addrs.data(), addr_offsets.data(), nullptr, nullptr, nullptr,
65
            0 /* dummy context ptr */, kNumRecords);
66

67
  EXPECT_THAT(out, testing::ElementsAre(6, 8, 10, 12));
68
  EXPECT_EQ(out_bitmap, 0ULL);
69
}

这一块可以看下GPT的解析

这个 TestAdd 测试用例是在测试 LLVM 代码生成器生成向量加法运算的完整流程。让我详细解释：

测试目标

验证 Gandiva 能否正确生成并执行两个 int32 向量相加的 LLVM 代码。

测试流程分析

1. 初始化 LLVM 生成器
1
ASSERT_OK_AND_ASSIGN(auto generator, LLVMGenerator::Make(TestConfigWithIrDumping(), false));
创建支持 IR 转储的 LLVM 代码生成器

2. 设置输入向量描述符
1
// 第一个输入向量 f0 (int32)
2
auto field0 = std::make_shared<arrow::Field>("f0", arrow::int32());
3
auto desc0 = annotator.CheckAndAddInputFieldDescriptor(field0);
4
auto validity_dex0 = std::make_shared<VectorReadValidityDex>(desc0);  // 有效性位图
5
auto value_dex0 = std::make_shared<VectorReadFixedLenValueDex>(desc0); // 数据值
6
auto pair0 = std::make_shared<ValueValidityPair>(validity_dex0, value_dex0);
7

8
// 第二个输入向量 f1 (int32) - 类似的设置
3. 定义加法函数
1
DataTypeVector params{arrow::int32(), arrow::int32()};
2
auto func_desc = std::make_shared<FuncDescriptor>("add", params, arrow::int32());
定义一个接受两个 int32 参数、返回 int32 的加法函数

4. 创建函数表达式
1
auto func_dex = std::make_shared<NonNullableFuncDex>(
2
    func_desc, native_func, FunctionHolderPtr(nullptr), -1, pairs);
将输入向量对与加法函数绑定

5. 生成 LLVM 代码
1
ASSERT_OK(generator->CodeGenExprValue(func_dex, 4, desc_sum, 0, fn_name,
2
                                      SelectionVector::MODE_NONE));
为表达式生成 LLVM IR 代码

处理 4 条记录

6. 编译并获取函数指针
1
ASSERT_OK_AND_ASSIGN(auto fn_ptr, generator->engine_->CompiledFunction(fn_name));
2
auto eval_func = reinterpret_cast<EvalFunc>(fn_ptr);
将 LLVM IR 编译为机器码

获取可执行的函数指针

7. 准备测试数据
1
std::array<uint32_t, kNumRecords> a0{1, 2, 3, 4};  // 第一个向量
2
std::array<uint32_t, kNumRecords> a1{5, 6, 7, 8};  // 第二个向量
3
uint64_t in_bitmap = 0xffffffffffffffffull;        // 全部有效
4
std::array<uint32_t, kNumRecords> out{0, 0, 0, 0}; // 输出向量
8. 执行生成的函数
1
eval_func(addrs.data(), addr_offsets.data(), nullptr, nullptr, nullptr,
2
          0 /* dummy context ptr */, kNumRecords);
调用编译后的函数执行向量加法

9. 验证结果
1
EXPECT_THAT(out, testing::ElementsAre(6, 8, 10, 12));  // 1+5, 2+6, 3+7, 4+8
2
EXPECT_EQ(out_bitmap, 0ULL);  // 输出位图应为0（全部有效）
关键技术点

1. 向量化优化验证
1
EXPECT_THAT(ir, testing::HasSubstr("vector.body"));
检查生成的 LLVM IR 中包含向量化循环体

确保 LLVM 进行了向量化优化

2. 内存布局
1
std::array<uint8_t*, 6> addrs{
2
    reinterpret_cast<uint8_t*>(a0.data()),  // 第一个向量数据
3
    reinterpret_cast<uint8_t*>(&in_bitmap), // 第一个向量有效性位图
4
    reinterpret_cast<uint8_t*>(a1.data()),  // 第二个向量数据
5
    reinterpret_cast<uint8_t*>(&in_bitmap), // 第二个向量有效性位图
6
    reinterpret_cast<uint8_t*>(out.data()), // 输出向量数据
7
    reinterpret_cast<uint8_t*>(&out_bitmap) // 输出向量有效性位图
8
};
测试意义

这个测试验证了 Gandiva 的核心功能：

表达式解析：将高级表达式转换为内部表示

代码生成：生成高效的 LLVM IR 代码

编译执行：将 IR 编译为机器码并执行

向量化优化：确保生成的代码利用了 SIMD 指令

正确性：验证计算结果的准确性

这是一个端到端的集成测试，确保整个代码生成和执行流水线工作正常。

这里的testing::HasSubstr是GMock的组件

这里可以看到，C语言函数是可以直接register上去的

1
TEST_F(TestLLVMGenerator, VerifyExtendedCFunctions) {
2
  VerifyFunctionMapping("multiply_by_three_int32", [](auto registry) {
3
    return TestConfigWithCFunction(std::move(registry));
4
  });
5

6

7
//test_util.cc
8
std::shared_ptr<Configuration> TestConfigWithCFunction(
9
    std::shared_ptr<FunctionRegistry> registry) {
10
  return BuildConfigurationWithRegistry(std::move(registry), [](auto reg) {
11
    return reg->Register(GetTestExternalCFunction(),
12
                         reinterpret_cast<void*>(multiply_by_three));
13
  });
14
}
15

16
static int64_t multiply_by_three(int32_t value) { return value * 3; }

literal_holder

Gandiva 中统一表示和处理各种类型的常量值

1
namespace gandiva {
2

3
using LiteralHolder =
4
    std::variant<bool, float, double, int8_t, int16_t, int32_t, int64_t, uint8_t,
5
                 uint16_t, uint32_t, uint64_t, std::string, DecimalScalar128>;
6

7
GANDIVA_EXPORT std::string ToString(const LiteralHolder& holder);
8

9
}  // namespace gandiva

std::variant 是 C++17 引入的一个类型安全的联合体（type-safe union），它可以在运行时保存一个多个预设类型中的一个值，但不会像传统的 union 那样不安全。

Rust 的 enum 枚举类型是 std::variant 的更强版本

Interval_holder

处理各类时间间隔

1
  // Pass only years and days to cast
2
  data = "P12Y15D";
3
  response = cast_interval_day(&execution_context_, data.data(), 7, true, &out_valid);
4
  qty_days_in_response = 15;
5
  qty_millis_in_response = 0;
6
  EXPECT_TRUE(out_valid);
7
  EXPECT_FALSE(execution_context_.has_error());
8
  EXPECT_EQ(response, (qty_millis_in_response << 32) | qty_days_in_response);

hash_utils

hash组件用的是OpenSSL，主要是关于Sha类，Md5l类函数

1
GANDIVA_EXPORT
2
const char* gdv_sha512_hash(int64_t context, const void* message, size_t message_length,
3
                            int32_t* out_length) {
4
  constexpr int sha512_result_length = 128;
5
  return gdv_hash_using_openssl(context, message, message_length, EVP_sha512(),
6
                                sha512_result_length, out_length);
7
}
8

9
/// Hashes a generic message using the SHA256 algorithm
10
GANDIVA_EXPORT
11
const char* gdv_sha256_hash(int64_t context, const void* message, size_t message_length,
12
                            int32_t* out_length) {
13
  constexpr int sha256_result_length = 64;
14
  return gdv_hash_using_openssl(context, message, message_length, EVP_sha256(),
15
                                sha256_result_length, out_length);
16
}
17

18
/// Hashes a generic message using the SHA1 algorithm
19
GANDIVA_EXPORT
20
const char* gdv_sha1_hash(int64_t context, const void* message, size_t message_length,
21
                          int32_t* out_length) {
22
  constexpr int sha1_result_length = 40;
23
  return gdv_hash_using_openssl(context, message, message_length, EVP_sha1(),
24
                                sha1_result_length, out_length);
25
}
26

27
GANDIVA_EXPORT
28
const char* gdv_md5_hash(int64_t context, const void* message, size_t message_length,
29
                         int32_t* out_length) {
30
  constexpr int md5_result_length = 32;
31
  return gdv_hash_using_openssl(context, message, message_length, EVP_md5(),
32
                                md5_result_length, out_length);
33
}

gandiva_object_cache

直接对result1 = evaluate("column1 + column2 * 3");这类操作的结果进行缓存，相关操作继承自llvm::ObjectCache，使用llvm::memorybuffer缓存相关代码

1
class GandivaObjectCache : public llvm::ObjectCache {
2
 public:
3
  explicit GandivaObjectCache(
4
      std::shared_ptr<Cache<ExpressionCacheKey, std::shared_ptr<llvm::MemoryBuffer>>>&
5
          cache,
6
      ExpressionCacheKey key);
7

8
  ~GandivaObjectCache() {}
9

10
  void notifyObjectCompiled(const llvm::Module* M, llvm::MemoryBufferRef Obj);
11

12
  std::unique_ptr<llvm::MemoryBuffer> getObject(const llvm::Module* M);
13

14
 private:
15
  ExpressionCacheKey cache_key_;
16
  std::shared_ptr<Cache<ExpressionCacheKey, std::shared_ptr<llvm::MemoryBuffer>>> cache_;
17
};

function_signature

给函数上Hash，我猜应该是缓存记录

1
  EXPECT_EQ(FunctionSignature("extract_month", {arrow::date32()}, arrow::int64()),
2
            FunctionSignature("extract_month", {local_date32_type_}, local_i64_type_));
3

4
TEST_F(TestFunctionSignature, TestHash) {
5
  FunctionSignature f1("add", {arrow::int32(), arrow::int32()}, arrow::int64());
6
  FunctionSignature f2("add", {local_i32_type_, local_i32_type_}, local_i64_type_);
7
  EXPECT_EQ(f1.Hash(), f2.Hash());
8

9
  FunctionSignature f3("extractDay", {arrow::int64()}, arrow::int64());
10
  FunctionSignature f4("extractday", {arrow::int64()}, arrow::int64());
11
  EXPECT_EQ(f3.Hash(), f4.Hash());
12
}

function_register

1
class GANDIVA_EXPORT FunctionRegistry {
2
 public:
3
  using iterator = const NativeFunction*;
4
  using FunctionHolderMaker =
5
      std::function<arrow::Result<std::shared_ptr<FunctionHolder>>(
6
          const FunctionNode& function_node)>;
7

8
  FunctionRegistry();
9
  FunctionRegistry(const FunctionRegistry&) = delete;
10
  FunctionRegistry& operator=(const FunctionRegistry&) = delete;
11

12
  /// Lookup a pre-compiled function by its signature.
13
  const NativeFunction* LookupSignature(const FunctionSignature& signature) const;
14

15
  /// \brief register a set of functions into the function registry from a given bitcode
16
  /// file
17
  arrow::Status Register(const std::vector<NativeFunction>& funcs,
18
                         const std::string& bitcode_path);
19

20
  /// \brief register a set of functions into the function registry from a given bitcode
21
  /// buffer
22
  arrow::Status Register(const std::vector<NativeFunction>& funcs,
23
                         std::shared_ptr<arrow::Buffer> bitcode_buffer);
24

25
  /// \brief register a C function into the function registry
26
  /// @param func the registered function's metadata
27
  /// @param c_function_ptr the function pointer to the
28
  /// registered function's implementation
29
  /// @param function_holder_maker this will be used as the function holder if the
30
  /// function requires a function holder
31
  arrow::Status Register(
32
      NativeFunction func, void* c_function_ptr,
33
      std::optional<FunctionHolderMaker> function_holder_maker = std::nullopt);
34

35
  /// \brief get a list of bitcode memory buffers saved in the registry
36
  const std::vector<std::shared_ptr<arrow::Buffer>>& GetBitcodeBuffers() const;
37

38
  /// \brief get a list of C functions saved in the registry
39
  const std::vector<std::pair<NativeFunction, void*>>& GetCFunctions() const;
40

41
  const FunctionHolderMakerRegistry& GetFunctionHolderMakerRegistry() const;
42

43
  iterator begin() const;
44
  iterator end() const;
45
  iterator back() const;
46

47
  friend arrow::Result<std::shared_ptr<FunctionRegistry>> MakeDefaultFunctionRegistry();
48

49
 private:
50
  std::vector<NativeFunction> pc_registry_;
51
  SignatureMap pc_registry_map_;
52
  std::vector<std::shared_ptr<arrow::Buffer>> bitcode_memory_buffers_;
53
  std::vector<std::pair<NativeFunction, void*>> c_functions_;
54
  FunctionHolderMakerRegistry holder_maker_registry_;
55

56
  Status Add(NativeFunction func);
57
};
58

59
/// \brief get the default function registry
60
GANDIVA_EXPORT std::shared_ptr<FunctionRegistry> default_function_registry();
61

62
}  // namespace gandiva

function_ir_builder

一个十分通用的IR生成器（这玩意我怎么之前没想到过呢.jpg)，甚至能实现If-else的block块跳转

1
class FunctionIRBuilder {
2
 public:
3
  explicit FunctionIRBuilder(Engine* engine) : engine_(engine) {}
4
  virtual ~FunctionIRBuilder() = default;
5

6
 protected:
7
  LLVMTypes* types() { return engine_->types(); }
8
  llvm::Module* module() { return engine_->module(); }
9
  llvm::LLVMContext* context() { return engine_->context(); }
10
  llvm::IRBuilder<>* ir_builder() { return engine_->ir_builder(); }
11
  llvm::Constant* CreateGlobalStringPtr(const std::string& string) {
12
    return engine_->CreateGlobalStringPtr(string);
13
  }
14

15
  /// Build an if-else block.
16
  llvm::Value* BuildIfElse(llvm::Value* condition, llvm::Type* return_type,
17
                           std::function<llvm::Value*()> then_func,
18
                           std::function<llvm::Value*()> else_func);
19

20
  struct NamedArg {
21
    std::string name;
22
    llvm::Type* type;
23
  };
24

25
  /// Build llvm fn.
26
  llvm::Function* BuildFunction(const std::string& function_name, llvm::Type* return_type,
27
                                std::vector<NamedArg> in_args);
28

29
 private:
30
  Engine* engine_;
31
};

filter

这部分也是在LLVM中实现，看起来和Project差不多

1
 private:
2
  std::unique_ptr<LLVMGenerator> llvm_generator_;
3
  SchemaPtr schema_;
4
  std::shared_ptr<Configuration> configuration_;
5
  bool built_from_cache_;

如果想要添加缓存，直接SetLLVMObjectCache即可

1
Status Engine::SetLLVMObjectCache(GandivaObjectCache& object_cache) {
2
  auto cached_buffer = object_cache.getObject(nullptr);
3
  if (cached_buffer) {
4
    auto error = lljit_->addObjectFile(std::move(cached_buffer));
5
    if (error) {
6
      return Status::CodeGenError("Failed to add cached object file to LLJIT: ",
7
                                  llvm::toString(std::move(error)));
8
    }
9
  }
10
  return Status::OK();
11
}

在PassManager里面可以挂上Optimize

1
static void OptimizeModuleWithNewPassManager(llvm::Module& module,
2
                                             llvm::TargetIRAnalysis target_analysis) {
3
  // Setup an optimiser pipeline
4
  llvm::PassBuilder pass_builder;
5
  llvm::LoopAnalysisManager loop_am;
6
  llvm::FunctionAnalysisManager function_am;
7
  llvm::CGSCCAnalysisManager cgscc_am;
8
  llvm::ModuleAnalysisManager module_am;
9

10
  function_am.registerPass([&] { return target_analysis; });
11

12
  // Register required analysis managers
13
  pass_builder.registerModuleAnalyses(module_am);
14
  pass_builder.registerCGSCCAnalyses(cgscc_am);
15
  pass_builder.registerFunctionAnalyses(function_am);
16
  pass_builder.registerLoopAnalyses(loop_am);
17
  pass_builder.crossRegisterProxies(loop_am, function_am, cgscc_am, module_am);
18

19
  pass_builder.registerPipelineStartEPCallback([&](llvm::ModulePassManager& module_pm,
20
                                                   llvm::OptimizationLevel Level) {
21
    module_pm.addPass(llvm::ModuleInlinerPass());
22

23
    llvm::FunctionPassManager function_pm;
24
    function_pm.addPass(llvm::InstCombinePass());
25
    function_pm.addPass(llvm::PromotePass());
26
    function_pm.addPass(llvm::GVNPass());
27
    function_pm.addPass(llvm::NewGVNPass());
28
    function_pm.addPass(llvm::SimplifyCFGPass());
29
    function_pm.addPass(llvm::LoopVectorizePass());
30
    function_pm.addPass(llvm::SLPVectorizerPass());
31
    module_pm.addPass(llvm::createModuleToFunctionPassAdaptor(std::move(function_pm)));
32

33
    module_pm.addPass(llvm::GlobalOptPass());
34
  });

engine

关于LLVM Engine的配置基本都在engine.h，engine.cc，engine_llvm_test.cc里面，还可以加载预编译好LLVM IR

1
 /// load pre-compiled IR modules from precompiled_bitcode.cc and merge them into
2
  /// the main module.
3
  Status LoadPreCompiledIR();
4

5
  // load external pre-compiled bitcodes into module
6
  Status LoadExternalPreCompiledIR();
7

8
  // Create and add mappings for cpp functions that can be accessed from LLVM.
9
  arrow::Status AddGlobalMappings();
10

11
  // Remove unused functions to reduce compile time.
12
  Status RemoveUnusedFunctions();
13

14
  std::unique_ptr<llvm::LLVMContext> context_;
15
  std::unique_ptr<llvm::orc::LLJIT> lljit_;
16
  std::unique_ptr<llvm::IRBuilder<>> ir_builder_;
17
  std::unique_ptr<llvm::Module> module_;
18
  LLVMTypes types_;
19

20
  std::vector<std::string> functions_to_compile_;
21

22
  bool optimize_ = true;
23
  bool module_finalized_ = false;
24
  bool cached_;
25
  bool functions_loaded_ = false;
26
  std::shared_ptr<FunctionRegistry> function_registry_;
27
  std::string module_ir_;
28
  std::unique_ptr<llvm::TargetMachine> target_machine_;
29
  const std::shared_ptr<Configuration> conf_;
30
};

encrypt

Gandiva里面有加密套件的相关设置（但是却没看到文档关于如何使用的），其使用的AES加密也来自OpenSSL组件

1
GANDIVA_EXPORT
2
int32_t aes_encrypt(const char* plaintext, int32_t plaintext_len, const char* key,
3
                    int32_t key_len, unsigned char* cipher);
4

5
/**
6
 * Decrypt data using aes algorithm
7
 **/
8
GANDIVA_EXPORT
9
int32_t aes_decrypt(const char* ciphertext, int32_t ciphertext_len, const char* key,
10
                    int32_t key_len, unsigned char* plaintext);

具体的Test

1
TEST(TestShaEncryptUtils, TestAesEncryptDecrypt) {
2
  // 16 bytes key
3
  auto* key = "12345678abcdefgh";
4
  auto* to_encrypt = "some test string";
5

6
  auto key_len = static_cast<int32_t>(strlen(reinterpret_cast<const char*>(key)));
7
  auto to_encrypt_len =
8
      static_cast<int32_t>(strlen(reinterpret_cast<const char*>(to_encrypt)));
9
  unsigned char cipher_1[64];
10

11
  int32_t cipher_1_len =
12
      gandiva::aes_encrypt(to_encrypt, to_encrypt_len, key, key_len, cipher_1);
13

14
  unsigned char decrypted_1[64];
15
  int32_t decrypted_1_len = gandiva::aes_decrypt(reinterpret_cast<const char*>(cipher_1),
16
                                                 cipher_1_len, key, key_len, decrypted_1);
17

18
  EXPECT_EQ(std::string(reinterpret_cast<const char*>(to_encrypt), to_encrypt_len),
19
            std::string(reinterpret_cast<const char*>(decrypted_1), decrypted_1_len));

decimal_ir

对于浮点数代码的生成进行了特别的处理，看来这里面坑不小😂

1
class DecimalIR : public FunctionIRBuilder {
2
 public:
3
  explicit DecimalIR(Engine* engine)
4
      : FunctionIRBuilder(engine), enable_ir_traces_(false) {}
5

6
  /// Build decimal IR functions and add them to the engine.
7
  static Status AddFunctions(Engine* engine);
8

9
  void EnableTraces() { enable_ir_traces_ = true; }
10

11
  llvm::Value* CallDecimalFunction(const std::string& function_name,
12
                                   llvm::Type* return_type,
13
                                   const std::vector<llvm::Value*>& args);
14

15
 private:
16
  /// The intrinsic fn for divide with small divisors is about 10x slower, so not
17
  /// using these.
18
  static const bool kUseOverflowIntrinsics = false;
19

20
  // Holder for an i128 value, along with its with scale and precision.
21
  class ValueFull {
22
   public:
23
    ValueFull(llvm::Value* value, llvm::Value* precision, llvm::Value* scale)
24
        : value_(value), precision_(precision), scale_(scale) {}
25

26
    llvm::Value* value() const { return value_; }
27
    llvm::Value* precision() const { return precision_; }
28
    llvm::Value* scale() const { return scale_; }
29

30
   private:
31
    llvm::Value* value_;
32
    llvm::Value* precision_;
33
    llvm::Value* scale_;
34
  };
35

36
  // Holder for an i128 value, and a boolean indicating overflow.
37
  class ValueWithOverflow {
38
   public:
39
    ValueWithOverflow(llvm::Value* value, llvm::Value* overflow)
40
        : value_(value), overflow_(overflow) {}
41

42
    // Make from IR struct
43
    static ValueWithOverflow MakeFromStruct(DecimalIR* decimal_ir, llvm::Value* dstruct);
44

45
    // Build a corresponding IR struct
46
    llvm::Value* AsStruct(DecimalIR* decimal_ir) const;
47

48
    llvm::Value* value() const { return value_; }
49
    llvm::Value* overflow() const { return overflow_; }
50

51
   private:
52
    llvm::Value* value_;
53
    llvm::Value* overflow_;
54
  };

附录：Arrow类型与LLVM类型的映射

Gandiva 类型（arrow 数据类型）	C 函数类型
int8	int8_t
int16	int16_t
int32	int32_t
int64	int64_t
uint8	uint8_t
uint16	uint16_t
uint32	uint32_t
uint64	uint64_t
float32	float
float64	double
boolean	bool
date32	int32_t
date64	int64_t
timestamp	int64_t
time32	int32_t
time64	int64_t
interval_month	int32_t
interval_day_time	int64_t
utf8（作为参数类型）	const char*、uint32_t
utf8（作为返回类型）	int64_t context、const char、uint32_t
binary（作为参数类型）	const char*、uint32_t
utf8（作为返回类型）	int64_t context、const char、uint32_t

总结

抛开不知道为什么项目文件不区分文件夹的问题，项目代码质量很高，关键点的注释和测试样例可以让人理解Gandiva做的事情，很有意思。

虽然有提供Gandiva外部函数的相关手册，但具体怎么用的话，还是要是要看测试样例

相关资料

中文文档

Dremio提供的资料

大家写的Blog

项目历史&现状简述

源码解析

node

tree_expr

to_date_holder

simple_arena

selection_vector

regex_functions/util

random_generator

project

lru_cache

llvm_types

llvm_includes

llvm_generator

测试目标

测试流程分析

1. 初始化 LLVM 生成器

2. 设置输入向量描述符

3. 定义加法函数

4. 创建函数表达式

5. 生成 LLVM 代码

6. 编译并获取函数指针

7. 准备测试数据

8. 执行生成的函数

9. 验证结果

关键技术点

1. 向量化优化验证

2. 内存布局

测试意义