[Enhancement] add support for restore to ccr

.github/workflows/go.yml

@@ -0,0 +1,29 @@
+# This workflow will build a golang project
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-go
+
+name: Go
+
+on:
+  push:
+    branches: [ "dev", "branch-2.0", "branch-2.1", "branch-3.0" ]
+  pull_request:
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v3
+
+    - name: Set up Go
+      uses: actions/setup-go@v4
+      with:
+        go-version: '1.20'
+
+    - name: Format
+      run: if [ "$(gofmt -s -l . | wc -l)" -gt 0 ]; then exit 1; fi
+
+    - name: Build
+      run: make
+
+    - name: Test
+      run: make test

.github/workflows/golangci-lint.yml

@@ -0,0 +1,29 @@
+name: golangci-lint
+on:
+  push:
+    branches:
+      - main
+      - dev
+      - branch-3.0
+      - branch-2.1
+      - branch-2.0
+  pull_request:
+
+permissions:
+  contents: read
+  # Optional: allow read access to pull request. Use with `only-new-issues` option.
+  # pull-requests: read
+
+jobs:
+  golangci:
+    name: lint
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-go@v5
+        with:
+          go-version: '1.20'
+      - name: golangci-lint
+        uses: golangci/golangci-lint-action@v6
+        with:
+          version: v1.60

.gitignore

@@ -4,4 +4,4 @@
 bin
 output
 ccr.db
-backup
+tarball

CHANGELOG.md

@@ -1,16 +1,169 @@
 # 更新日志
 
-## v 0.5
+### Fix
+
+## 3.0.4/2.1.8
+
+注意：从这个版本开始 doris 和 ccr-syncer 的 2.0 版本将不再更新，需要使用 ccr-syncer 的需要先升级到 2.1 及以上版本。
+
+### Fix
+
+- 修复 table name 中带 `-` 无法同步的问题 (selectdb/ccr-syncer#168)
+- 修复部分同步下可能同步多次增量数据的问题 (selectdb/ccr-syncer#186)
+- 修复 create 又立即 drop 的情况下无法找到 table 的问题 (selectdb/ccr-syncer#188)
+- 跳过不支持的 table 类型，比如 ES TABLE
+- 避免在同步快照、binlog 期间对上游 name 产生依赖 (selectdb/ccr-syncer#205, selectdb/ccr-syncer#239)
+- 修复全量同步期间 view 的别名问题 (selectdb/ccr-syncer#207)
+- 修复 add partition with keyword name 的问题 (selectdb/ccr-syncer#212)
+- 跳过 drop tmp partition (selectdb/ccr-syncer#214)
+- 修复快照过期的问题，过期后会重做 (selectdb/ccr-syncer#229)
+- 修复 rename 导致的上下游 index name 无法匹配的问题 (selectdb/ccr-syncer#235)
+- 修复并行创建 table/backup 时 table 丢失的问题 (selectdb/ccr-syncer#237)
+- 修复 partial snapshot 期间，上游 table/partition 已经被删除/重命名/替换的问题 (selectdb/ccr-syncer#240, selectdb/ccr-syncer#241, selectdb/ccr-syncer#249, selectdb/ccr-syncer#255)
+- 检查 database connection 错误 (selectdb/ccr-syncer#247)
+- 过滤已经被删除的 table (selectdb/ccr-syncer#248)
+- 修复 create table 时下游 table 已经存在的问题 (selectdb/ccr-syncer#161)
+
+### Feature
+
+- 支持 atomic restore，全量同步期间下游仍然可读 (selectdb/ccr-syncer#166)
+- 支持处理包装在 barrier log 中的其他 binlog （主要用于在 2.0/2.1 上增加新增的 binlog 类型）(selectdb/ccr-syncer#208)
+- 支持 rename table (2.1) (selectdb/ccr-syncer#209)
+- 跳过 modify partition binlog (selectdb/ccr-syncer#213)
+- 支持 modify comment binlog (selectdb/ccr-syncer#140)
+- 支持 replace table binlog (selectdb/ccr-syncer#245)
+- 支持 drop view binlog (selectdb/ccr-syncer#138)
+- 支持 modify view def binlog (selectdb/ccr-syncer#184)
+- 支持 inverted index 相关 binlog (selectdb/ccr-syncer#252)
+- 支持 table sync 下的 txn insert (WIP) (selectdb/ccr-syncer#234, selectdb/ccr-syncer#259)
+- 支持 rename partition/rollup binlogs (selectdb/ccr-syncer#268)
+- 支持 add/drop rollup binlogs (selectdb/ccr-syncer#269)
+- 支持 modify view/comment in 2.1 (selectdb/ccr-syncer#270, selectdb/ccr-syncer#273)
+- 支持 table sync 下的 replace table (selectdb/ccr-syncer#279)
+
+### Improve
+
+- 支持同步 rename column，需要 doris xxxx (selectdb/ccr-syncer#139)
+- 支持在全量同步过程中，遇到 table signature 不匹配时，使用 alias 替代 drop (selectdb/ccr-syncer#179)
+- 增加 monitor，在日志中 dump 内存使用率 (selectdb/ccr-syncer#181)
+- 过滤 schema change 删除的 indexes，避免全量同步 (selectdb/ccr-syncer#185)
+- 过滤 schema change 创建的 shadow indexes 的更新，避免全量同步 (selectdb/ccr-syncer#187)
+- 增加 `mysql_max_allowed_packet` 参数，控制 mysql sdk 允许发送的 packet 大小 (selectdb/ccr-syncer#196)
+- 限制一个 JOB 中单个 BE 的 ingest 并发数，减少对 BE 的连接数和文件描述符消耗 (selectdb/ccr-syncer#195)
+- 避免在获取 job status 等待锁 (selectdb/ccr-syncer#198)
+- 避免 backup/restore 任务阻塞查询 ccr job progress (selectdb/ccr-syncer#201, selectdb/ccr-syncer#206)
+- 避免将 snapshot job info 和 meta （这两个数据可能非常大）持久化到 mysql 中 (selectdb/ccr-syncer#204)
+- 上游 db 中没有 table 时，打印 info 而不是 error (selectdb/ccr-syncer#211)
+- 在 ccr syncer 重启后，复用由当前 job 发起的 backup/restore job (selectdb/ccr-syncer#218, selectdb/ccr-syncer#224, selectdb/ccr-syncer#226)
+- 支持读取压缩后的快照/恢复快照时压缩，避免碰到 thrift max message size 限制 (selectdb/ccr-syncer#223)
+- API job_progress 避免返回 persist data (selectdb/ccr-syncer#271)
+
+## 2.0.15/2.1.6
+
+### Fix
+
+- 修复 `REPLACE_IF_NOT_NULL` 语句的默认值语法不兼容问题 (selectdb/ccr-syncer#180)
+- 修复 table sync 下 partial snapshot 没有更新 dest table id 的问题 (selectdb/ccr-syncer#178)
+- **修复 table sync with alias 时，lightning schema change 找不到 table 的问题** (selectdb/ccr-syncer#176)
+- 修复 db sync 下 partial snapshot table 为空的问题 (selectdb/ccr-syncer#173)
+- 修复 create table 时下游 view 已经存在的问题（先删除 view），feature gate: `feature_create_view_drop_exists` (selectdb/ccr-syncer#170,selectdb/ccr-syncer#171)
+- 修复 table not found 时没有 rollback binlog 的问题
+- **修复下游删表后重做 snapshot 是 table mapping 过期的问题 (selectdb/ccr-syncer#162,selectdb/ccr-syncer#163,selectdb/ccr-syncer#164)**
+- 修复 full sync 期间 view already exists 的问题，如果 signature 不匹配会先删除 (selectdb/ccr-syncer#152)
+- 修复 2.0 中 get view 逻辑，兼容 default_cluster 语法 (selectdb/ccr-syncer#149)
+- 修复 job state 变化时仍然更新了 job progress 的问题，对之前的逻辑无影响，主要用于支持 partial sync (selectdb/ccr-syncer#124)
+- 修复 get_lag 接口中不含 lag 的问题 (selectdb/ccr-syncer#126)
+- 修复下游 restore 时未清理 orphan tables/partitions 的问题 (selectdb/ccr-syncer#128)
+    - 备注： 暂时禁用，因为 doris 侧发现了 bug (selectdb/ccr-syncer#153,selectdb/ccr-syncer#161)
+- **修复下游删表后重做 snapshot 时 dest meta cache 过期的问题 (selectdb/ccr-syncer#132)**
+
+### Feature
+
+- 增加 `/force_fullsync` 用于强制触发 fullsync (selectdb/ccr-syncer#167)
+- 增加 `/features` 接口，用于列出当前有哪些 feature 以及是否打开 (selectdb/ccr-syncer#175)
+- 支持同步 drop view（drop table 失败后使用 drop view 重试）(selectdb/ccr-syncer#169)
+- 支持同步 rename 操作 (selectdb/ccr-syncer#147)
+- schema change 使用 partial sync 而不是 fullsync (selectdb/ccr-syncer#151)
+- partial sync 使用 rename 而不是直接修改 table，因此表的读写在同步过程中不受影响 (selectdb/ccr-syncer#148)
+- 支持 partial sync，减少需要同步的数据量 (selectdb/ccr-syncer#125)
+- 添加参数 `allowTableExists`，允许在下游 table 存在时，仍然创建 ccr job（如果 schema 不一致，会自动删表重建）(selectdb/ccr-syncer#136)
+
+### Improve
+
+- 日志输出 milliseconds (selectdb/ccr-syncer#182)
+- 如果下游表的 schema 不一致，则将表移动到 RecycleBin 中（之前是强制删除）(selectdb/ccr-syncer#137)
+
+## 2.0.14/2.1.5
+
+### Fix
+
+- 过滤已经删除的 partitions，避免 full sync，需要 doris 2.0.14/2.1.5 (selectdb/ccr-syncer#117)
+- 过滤已经删除的 tables，避免 full sync (selectdb/ccr-syncer#123)
+- 兼容 doris 3.0 alternative json name，doris 3.0 必须使用该版本的 CCR syncer (selectdb/ccr-syncer#121)
+- 修复 list jobs 接口在高可用环境下不可用的问题 (selectdb/ccr-syncer#120)
+
+## 2.0.11
+
+对应 doris 2.0.11。
+
+### Feature
+
+- 支持以 postgresql 作为 ccr-syncer 的元数据库 (selectdb/ccr-syncer#77)
+- 支持 insert overwrite 相关操作 (selectdb/ccr-syncer#97,selectdb/ccr-syncer#99)
+
+### Fix
+
+- 修复 drop partition 后因找不到 partition id 而无法继续同步的问题 (selectdb/ccr-syncer#82)
+- 修复高可用模式下接口无法 redirect 的问题 (selectdb/ccr-syncer#81)
+- 修复 binlog 可能因同步失败而丢失的问题 (selectdb/ccr-syncer#86,selectdb/ccr-syncer#91)
+- 修改 connect 和 rpc 超时时间默认值，connect 默认 10s，rpc 默认 30s (selectdb/ccr-syncer#94,selectdb/ccr-syncer#95)
+- 修复 view 和 materialized view 使用造成空指针问题 (selectdb/ccr-syncer#100)
+- 修复 add partition sql 错误的问题 (selectdb/ccr-syncer#99)
+
+
+## 2.1.3/2.0.3.10
+
+### Fix
+
+- 修复因与上下游 FE 网络中断而触发 full sync 的问题
+
+### Feature
+
+- 增加 `/job_progress` 接口用于获取 JOB 进度
+- 增加 `/job_details` 接口用于获取 JOB 信息
+- 保留 job 状态变更的各个时间点，并在 `/job_progress` 接口中展示
+
+### Fix
+
+- 修复若干 keywords 没有 escape 的问题
+
+## 2.0.3.9
+
+配合 doris 2.0.9 版本
+
+### Feature
+
+- 添加选项以启动 pprof server
+- 允许配置 rpc 合 connection 超时
+
+### Fix
+
+- restore 每次重试时使用不同的 label 名
+- update table 失败时（目标表不存在）会触发快照同步
+- 修复同步 sql 中包含关键字的问题
+- 如果恢复时碰到表 schema 发生变化，会先删表再重试恢复
+
+## 0.5
 
 ### 支持高可用
 - 现在可以部署多个Syncer节点来保证CCR功能的高可用。
 - db是Syncer集群划分的依据，同一个集群下的Syncer共用一个db。
 - Syncer集群采用对称设计，每个Syncer都会相对独立的执行被分配到的job。在某个Syncer节点down掉后，它的jobs会依据负载均衡算法被分给其他Syncer节点。
 
-## v 0.4
+## 0.4
 * 增加 enable_db_binlog.sh 方便用户对整库开启binlog
 
-## v 0.3
+## 0.3
 
 ### LOG
 

LICENSE

@@ -0,0 +1,20 @@
+This free license agreement is made between SelectDB Inc. (hereinafter referred to as "SelectDB", "we", "our" or "us") and the users ("user", "your" or "you") of the SelectDB products. SelectDB products refer to the software and services provided by SelectDB, including any updates, error fixes, and documentation. You acknowledge that you have fully read, understood and accepted this agreement in its entirety before you begin a trial or purchase of SelectDB products or services. You agree that by clicking the "agree" box or similar or using our product and services, you are agreeing to enter to this agreement, which is a legally binding contract between you and SelectDB. If you do not agree with any provision of this agreement, you must not purchase or use any of our services.
+SelectDB reserves the right to change our products and services in accordance with applicable laws and our corporate policies without prior notice. We will post the changes on SelectDB.io. You agree that by continuing to use our services after the announcement of any changes to this agreement, you acknowledge that you have fully read, understood and accepted the modified products and services and will use our products and services in accordance with the modified agreement. If you disagree with any changes, you should no longer use our products and services.
+License Rights and Limitations
+SelectDB grants you a free, non-exclusive, non-transferable, limited license to use the SelectDB products.
+- You have the right to install and use the SelectDB product or service on multiple computers, and run it for your personal use or internal business operations, subject to the terms of this agreement.
+- You may redistribute the unmodified software and product documentation in accordance with the terms of this agreement, provided that you do not charge any fees related to such distribution or use of the software.
+The effectiveness of your license is subject to the following conditions:
+- You shall not remove any proprietary notices or markings of SelectDB or the licensor from the software or documentation.
+- You shall not modify, reverse engineer, decompile, or attempt to extract the source code of the software.
+- You shall not use the software for illegal purposes or violate any applicable laws or regulations.
+Intellectual Property Rights
+- All intellectual property rights of SelectDB, including but not limited to copyrights, patents, and trademarks, are owned by the licensor.
+- This agreement does not grant the user any intellectual property rights.
+Disclaimer
+- The software is provided "as is" without any warranties, representations, conditions, or guarantees of any kind.
+- The licensor does not provide any warranties regarding the suitability, merchantability, accuracy, reliability, or any other aspect of the software.
+- To the maximum extent permitted by applicable law, the licensor shall not be liable for any direct, indirect, incidental, special, or consequential damages arising from the use of the software.
+Miscellaneous
+- This agreement constitutes the entire agreement between the licensor and the user regarding the use of the software and supersedes any prior oral or written agreements.
+- This agreement shall be governed by the laws of Singapore in terms of interpretation, validity, and performance.

Makefile

@@ -9,6 +9,31 @@ tag := $(shell git describe --abbrev=0 --always --dirty --tags)
 sha := $(shell git rev-parse --short HEAD)
 git_tag_sha := $(tag):$(sha)
 
+ifeq ($(shell uname -i),x86_64)
+	# Make them happy
+	platform := x64
+else
+	platform := arm64
+endif
+tarball_suffix := $(tag)-$(platform)
+
+LDFLAGS="-X 'github.com/selectdb/ccr_syncer/pkg/version.GitTagSha=$(git_tag_sha)'"
+GOFLAGS=
+
+GOFORMAT := gofmt -l -d -w
+
+# COVERAGE=ON make
+ifeq ($(COVERAGE),ON)
+    GOFLAGS += -cover
+endif
+
+.PHONY: flag_coverage
+## COVERAGE=ON : Set coverage flag
+
+.PHONY: default
+## default: Build ccr_syncer
+default: ccr_syncer
+
 .PHONY: build
 ## build : Build binary
 build: ccr_syncer get_binlog ingest_binlog get_meta snapshot_op get_master_token spec_checker rows_parse
@@ -27,12 +52,12 @@ lint:
 .PHONY: fmt
 ## fmt : Format all code
 fmt:
-	$(V)go fmt ./...
+	$(V)$(GOFORMAT) .
 
 .PHONY: test
 ## test : Run test
 test:
-	$(V)go test $(shell go list ./... | grep -v github.com/selectdb/ccr_syncer/cmd | grep -v github.com/selectdb/ccr_syncer/pkg/rpc/kitex_gen/) | grep -F -v '[no test files]'
+	$(V)go test $(shell go list ./... | grep -v github.com/selectdb/ccr_syncer/cmd | grep -v github.com/selectdb/ccr_syncer/pkg/rpc/kitex_gen/)
 
 .PHONY: help
 ## help : Print help message
@@ -42,11 +67,24 @@ help: Makefile
 
 # --------------- ------------------ ---------------
 # --------------- User Defined Tasks ---------------
-.PHONY: cmd/ccr_syncer
+
+.PHONY: cloc
+## cloc : Count lines of code
+cloc:
+	$(V)tokei -C . -e pkg/rpc/kitex_gen -e pkg/rpc/thrift
+
+.PHONY: gen_mock
+## gen_mock : Generate mock
+gen_mock:
+	$(V)mockgen -source=pkg/rpc/fe.go -destination=pkg/ccr/fe_mock.go -package=ccr
+	$(V)mockgen -source=pkg/ccr/metaer.go -destination=pkg/ccr/metaer_mock.go -package=ccr
+	$(V)mockgen -source=pkg/ccr/metaer_factory.go -destination=pkg/ccr/metaer_factory_mock.go -package=ccr
+	$(V)mockgen -source=pkg/rpc/rpc_factory.go -destination=pkg/ccr/rpc_factory_mock.go -package=ccr
+
 .PHONY: ccr_syncer
 ## ccr_syncer : Build ccr_syncer binary
 ccr_syncer: bin
-	$(V)go build -ldflags "-X github.com/selectdb/ccr_syncer/pkg/version.GitTagSha=$(git_tag_sha)" -o bin/ccr_syncer ./cmd/ccr_syncer
+	$(V)go build ${GOFLAGS} -ldflags ${LDFLAGS} -o bin/ccr_syncer ./cmd/ccr_syncer
 
 .PHONY: get_binlog
 ## get_binlog : Build get_binlog binary
@@ -59,9 +97,9 @@ run_get_binlog: get_binlog
 
 .PHONY: sync_thrift
 ## sync_thrift : Sync thrift
-# TODO(Drogon): Add build thrift
 sync_thrift:
-	$(V)rsync -avc $(THRIFT_DIR)/ rpc/thrift/
+	$(V)rsync -avc $(THRIFT_DIR)/ pkg/rpc/thrift/
+	$(V)$(MAKE) -C pkg/rpc/ gen_thrift
 
 .PHONY: ingest_binlog
 ## ingest_binlog : Build ingest_binlog binary
@@ -98,7 +136,29 @@ get_lag: bin
 rows_parse: bin
 	$(V)go build -o bin/rows_parse ./cmd/rows_parse
 
+.PHONY: thrift_get_meta
+## thrift_get_meta : Build thrift_get_meta binary
+thrift_get_meta: bin
+	$(V)go build -o bin/thrift_get_meta ./cmd/thrift_get_meta
+
+.PHONY: metrics
+## metrics : Build metrics binary
+metrics: bin
+	$(V)go build -o bin/metrics ./cmd/metrics
+
 .PHONY: todos
 ## todos : Print all todos
 todos:
-	$(V)grep -rnw . -e "TODO" | grep -v '^./rpc/thrift' | grep -v '^./.git'
+	$(V)grep -rnw . -e "TODO" | grep -v '^./pkg/rpc/thrift' | grep -v '^./.git'
+
+.PHONY: tarball
+## tarball : Archive files and release ccr-syncer-$(version)-$(platform).tar.xz
+tarball: default
+	$(V)mkdir -p tarball/ccr-syncer-$(tarball_suffix)/{bin,db,doc,log}
+	$(V)cp CHANGELOG.md README.md LICENSE tarball/ccr-syncer-$(tarball_suffix)/
+	$(V)cp bin/ccr_syncer tarball/ccr-syncer-$(tarball_suffix)/bin/
+	$(V)cp shell/{enable_db_binlog.sh,start_syncer.sh,stop_syncer.sh} tarball/ccr-syncer-$(tarball_suffix)/bin/
+	$(V)cp -r doc/* tarball/ccr-syncer-$(tarball_suffix)/doc/
+	$(V)cd tarball/ && tar cfJ ccr-syncer-$(tarball_suffix).tar.xz ccr-syncer-$(tarball_suffix)
+	$(V)echo archive: tarball/ccr-syncer-$(tarball_suffix).tar.xz
+