Engine abstraction tracking issue #6402

brson · 2020-01-03T07:07:11Z

This issue is to organize and track progress on abstracting TiKV over generic storage
engines (previous issue: #4184). There are many tasks to be done and contributions are welcome.

In scope: the process of abstracting the entire RocksDB engine API into the engine_traits and engine_rocks crates, decoupling TiKV and its test suite from RocksDB., and prototyping new engine bindings. Out of scope: refactoring engine_traits to better support with non-RocksDB engines.

The engine_traits crate docs describe the design and
migration process.

The issue is updated to contain a checklist of tasks that definitely need to
happen, along with name of who is working on it. When you decide to take a task,
say so on the issue before attempting it so that someone else doesn't duplicate
your work.

It's hard to know in what order these need to happen, nor exactly how to
complete each of them, as each tends to present new challenges. Instead I have
documented here, in the engine_traits crate docs, and in a status report, my experiences so far and suggestions for how to proceed.

The most important thing to keep in mind is:

Make small changes: small commits, small PRs. When you find that something else
needs to change first, stop what you are doing, and make that change, and submit
that PR. Don't let your patches grow out of control. Don't be afraid to throw away
your work and try something else.

Task list (updated 2021/01/10)

The text was updated successfully, but these errors were encountered:

brson · 2020-01-03T07:21:44Z

It's kinda buried in the op, but for those wanting to contribute, please read my experience report about what we've done so far and what I've learned in the process: https://gist.github.com/f6aa58bebc7de16ea17741bce31f6080.

brson · 2020-01-22T06:51:24Z

FWIW I've been working on-and-off on abstracting TablePropertiesCollection and its associated types for about a month, trying different techniques. I'm now writing a completely new set of bindings within rust-rocksdb that use reference counting.

Rustin170506 · 2020-02-20T10:19:21Z

I am working on rocks/util module.

brson · 2020-02-21T03:56:43Z

Thanks @Rustin-Liu !

brson · 2020-02-21T03:58:40Z

Per my comment on #6661 (comment), it looks like after @Rustin-Liu's migration of the MetricsFlusher there will still be more work to be done on it.

MetricsFlusher needs to be implemented in the engine_traits crate so that it can be instantiated with two different engines (vs. being a trait that is implemented in each engine). Doing so will require refactoring the implementation and abstracting out some of the calls it makes, as well as the lists of metrics it uses.

brson · 2020-02-21T04:31:41Z

IOLimiter no longer exists, replaced by async-speed-limit in 965e8bf

brson · 2020-02-26T17:30:07Z

Now that raftstore is its own crate (thanks @overvenus), we have a nice intermediate goal: remove the engine dependency from raftstore. That simplifies the problem quite a bit.

brson · 2020-02-26T18:15:19Z

@Rustin-Liu a good next util to abstract would be util::compact_range. This is something preventing us from fully abstracting raftstore over engines.

It looks like that will require some new abstractions in engine_traits, perhaps a new file with a CompactExt trait that contains compact_range_cf_opt. I have not looked closely though.

brson · 2020-03-18T20:10:51Z

I've updated op with current progress, and added a task to document engine_traits cc @hicqu

brson · 2021-03-18T20:50:30Z

I guess I should point out that we are far enough along that anybody that wants to can begin prototyping engine_traits implementations for alternate engines, now that we have engine_traits_tests merged. I myself have begun two out of tree.

Getting a new engine to pass engine_traits_tests is the first step to adding a new engine.

Rough instructions are in this comment, but it mostly involves copying the engine_panic crate, replacing "Panic" in the source with "MyEngine", touching up the manifests of any crates that depend on concrete engines, and implementing constructors in engine_test.

An engine can be tested against the engine tests with e.g.:

cargo test -p engine_traits_tests --no-default-features --features=protobuf-codec,test-engines-agatedb

It will still be many months before tikv can fully run on other engines. There are lots of tiny problems to solve yet, and some of the abstractions exported by engine_traits are too rocks-specific to remain.

brson · 2021-04-22T20:02:33Z

I've posted a new issue to remove the table properties abstractions from engine traits: #10068

brson · 2021-07-06T20:04:23Z

This PR finally makes it possible to launch tikv-server with a non-rocks engine: #10529

ref tikv#6402 Signed-off-by: Brian Anderson <[email protected]>

* server: Allow flow_info_receiver to be None Signed-off-by: Brian Anderson <[email protected]> * server: Move init_flow_receiver call into init_raw_engines Signed-off-by: Brian Anderson <[email protected]> * server: Extract debug service creation to CreateKvEngine::start_debug_service Signed-off-by: Brian Anderson <[email protected]> * server: Extract backup service creation to CreateKvEngine::start_backup_service Signed-off-by: Brian Anderson <[email protected]> * server: Extract rocks cfg controller registration to CreateKvEngine::register_kv_config Signed-off-by: Brian Anderson <[email protected]> * server: Extract create_raftstore_compaction_listener to free function Signed-off-by: Brian Anderson <[email protected]> * server: Extract create_kv_engine to CreateKvEngine trait Signed-off-by: Brian Anderson <[email protected]> * server: Parameterize EngineResourceInfo over EK Signed-off-by: Brian Anderson <[email protected]> * server: Parameterize TikvServer over EK Signed-off-by: Brian Anderson <[email protected]> * server: Document CreateKvEngine trait Signed-off-by: Brian Anderson <[email protected]> * clippy Signed-off-by: Brian Anderson <[email protected]> * server: Extract raft engine creation to CreateRaftEngine::create_raft_engine Signed-off-by: Brian Anderson <[email protected]> * server: Extract CreateRaftEngine::register_raft_config method Signed-off-by: Brian Anderson <[email protected]> * server: Extract CreateRaftEngine::create_engine_infos method Signed-off-by: Brian Anderson <[email protected]> * server: Unspecialize init_raw_engines method Signed-off-by: Brian Anderson <[email protected]> * server: Extract setup code to generic run_server function Signed-off-by: Brian Anderson <[email protected]> * clippy ref #6402 Signed-off-by: Brian Anderson <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

brson · 2021-12-01T04:00:12Z

Status update:

TiKV is almost able to build and even run with engines other than rocks, but the final PR to enable other engines, #10945, was reverted due to hitting an unwrap on some tests that I am not familiar with. It should be relatively easy to revive that branch and fix the unwrap issue, but I have not investigated. Once that branch is merged, #10528 can be closed.

Building on that patch, I have already proceeded to demonstrate tikv running another engine, the most recent attempt being my engine-simple-try-2 branch, https://github.com/brson/tikv/tree/engine-simple-try-2. This branch was at the time able to launch tikv with my own engine built specifically for testing the engine traits and reach a steady state where it could do simple writes and reads.

The engine I developed for this purpose is at https://github.com/brson/blocksy3. This engine is designed for easy readability, and to support exactly the features required by TiKV, but not for performance.

TiKV contains a set of tests specifically designed for bootstrapping new engines, engine_traits_tests, the usage of which is documented in that crate. An engine that passes that test suite should more-or-less be usable within TiKV for the simplest use cases, at which point further work can be put into filling out test cases for less-used parts of the engine traits API.

TiKV is not fully abstracted over engines - there are a few subsystems which are dynamically disabled for non-rocks engines, but which should be relatively easy to rediscover once somebody digs in, and which are partially described here #10529

An issue for creating agatedb bindings is open here #9844

I expect the design of the engines traits to change significantly as support for other engines is added. The current design is just a thin layer over rocks, with only the most obviously unimplementable parts of rocks papered-over with very high level abstractions.

skyzh · 2021-12-01T04:04:04Z

Good to know that! Hope that we can integrate more engines into TiKV.

# This is the 1st commit message: Add latency stats for kv get and kv batch get Signed-off-by: mornyx <[email protected]> # This is the commit message #2: tikv_util: upgrade `gag` from 0.1 to 1.0 (tikv#11277) ref tikv#11276 Signed-off-by: kennytm <[email protected]> # This is the commit message tikv#3: codec: add compile time assert to guarantee safety (tikv#11357) * codec: add compile time assert to guarantee safety close tikv#11356. Signed-off-by: Jay Lee <[email protected]> * add more explanation. Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#4: rafstore: fix commit state check in apply thread (tikv#11401) * fix apply check & close tikv#11396 Signed-off-by: gengliqi <[email protected]> * make format Signed-off-by: gengliqi <[email protected]> * format Signed-off-by: gengliqi <[email protected]> # This is the commit message tikv#5: backup: create external storage per-backup reqeust. (tikv#11113) * backup: create external storage per-backup reqeust. fixed tikv#11112 Signed-off-by: yujuncen <[email protected]> * backup: make clippy happy Signed-off-by: yujuncen <[email protected]> * backup: fix build Signed-off-by: yujuncen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#6: Remove duplicate enum and macro about numberic_enum_mod (tikv#11347) * remove duplicated and useless `remove duplicated and useless` close tikv#11297 Signed-off-by: yuqi1129 <[email protected]> * remove duplicate macro rocksdb_enum_mod Signed-off-by: yuqi1129 <[email protected]> * fix format problem Signed-off-by: yuqi1129 <[email protected]> * resolve discussion Signed-off-by: yuqi1129 <[email protected]> * fix test error Signed-off-by: yuqi1129 <[email protected]> * move macro `numeric_enum_serializing_mod` to a proper location Signed-off-by: yuqi1129 <[email protected]> * move `perf_level_serde` of mod config in engine trait to mod perf_context Signed-off-by: yuqi1129 <[email protected]> * Add test about `rate-limiter-mode` and fix compile error in integration test Signed-off-by: yuqi1129 <[email protected]> Co-authored-by: Xinye Tao <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#7: ctl: detect raft db correctly (tikv#11395) Close tikv#11393. Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#8: Support to iterate table properties for KvEngine (tikv#11388) * support to iterate table properties for KvEngine * close tikv#11387 Signed-off-by: qupeng <[email protected]> # This is the commit message tikv#9: gc: Fix GC scan effectiveness to avoid OOM (tikv#11416) * close tikv#11410 limit the pending task for gc worker Signed-off-by: Connor1996 <[email protected]> * avoid using prefix seek Signed-off-by: Connor1996 <[email protected]> * rename test Signed-off-by: Connor1996 <[email protected]> * address comment Signed-off-by: Connor1996 <[email protected]> * fix build Signed-off-by: Connor1996 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#10: fix a panic about parsing memory.limit_in_bytes (tikv#11354) * handle int overflow for memory.limit_in_bytes clsoe tikv#11353. Signed-off-by: qupeng <[email protected]> * close tikv#11353 Signed-off-by: qupeng <[email protected]> * address comments Signed-off-by: qupeng <[email protected]> # This is the commit message tikv#11: build: fix make docker on Linux (tikv#11349) * fix make docker fix tikv#11312 Signed-off-by: tabokie <[email protected]> * fix cargo Signed-off-by: tabokie <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#12: engine_rocks: ignore NoSpace error during flush or compaction (tikv#11225) * ignore NoSpace error during flush or compaction fix tikv#11224 Signed-off-by: tabokie <[email protected]> * use constant Signed-off-by: tabokie <[email protected]> * fix typo Signed-off-by: tabokie <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#13: gc: set pending capacity for gc worker (tikv#11429) Signed-off-by: Connor1996 <[email protected]> # This is the commit message tikv#14: raftstore: support synchronous write (tikv#11418) * support sync write & ref tikv#11165 Signed-off-by: gengliqi <[email protected]> * add tests Signed-off-by: gengliqi <[email protected]> * change default store-io-pool-size to 0 Signed-off-by: gengliqi <[email protected]> * remove unnecessary test Signed-off-by: gengliqi <[email protected]> * merge raftstore: separate read write ready(tikv#10592) Signed-off-by: gengliqi <[email protected]> * address comments Signed-off-by: gengliqi <[email protected]> * fix test Signed-off-by: gengliqi <[email protected]> * make clippy happy Signed-off-by: gengliqi <[email protected]> * address comment Signed-off-by: gengliqi <[email protected]> * remove offset in NormalFsm Signed-off-by: gengliqi <[email protected]> * make format Signed-off-by: gengliqi <[email protected]> # This is the commit message tikv#15: metrics: add Raft Engine panel (tikv#11415) Signed-off-by: tabokie <[email protected]> # This is the commit message tikv#16: resource-metering: add some metrics (tikv#11414) * resource-metering: add some metrics Signed-off-by: Zhenchi <[email protected]> * more log Signed-off-by: Zhenchi <[email protected]> * polish description, ref tikv#11436 Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#17: metrics: add asynchronous write io panels (tikv#11408) * update metrics name Signed-off-by: gengliqi <[email protected]> * update tikv-details & close tikv#10540 Signed-off-by: gengliqi <[email protected]> * rename Raft GC to Raft Log GC Signed-off-by: gengliqi <[email protected]> * address comment Signed-off-by: gengliqi <[email protected]> * fix datasource Signed-off-by: gengliqi <[email protected]> * add more metrics Signed-off-by: gengliqi <[email protected]> * update metrics Signed-off-by: gengliqi <[email protected]> * update Signed-off-by: gengliqi <[email protected]> # This is the commit message tikv#18: storage: fix reverse scan check memory locks (tikv#11447) close tikv#11440 Signed-off-by: youjiali1995 <[email protected]> # This is the commit message tikv#19: test: make test_sending_fail_with_net_error more statable (tikv#11366) * sending count should deregister on time Signed-off-by: bufferflies <[email protected]> * make test statable Signed-off-by: bufferflies <[email protected]> * remove unused change Signed-off-by: bufferflies <[email protected]> * remove unused change Signed-off-by: bufferflies <[email protected]> * close tikv#11365 Signed-off-by: bufferflies <[email protected]> * reformat annotation Signed-off-by: bufferflies <[email protected]> * add send once filter Signed-off-by: bufferflies <[email protected]> * lint pass Signed-off-by: bufferflies <[email protected]> * lint pass Signed-off-by: bufferflies <[email protected]> * replace sendOnceMsgFilter Signed-off-by: bufferflies <[email protected]> * remove unnecessary change Signed-off-by: bufferflies <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#20: raftstore: increase batch for raftlog-gc-worker (tikv#11405) * increase batch Signed-off-by: Little-Wallace <[email protected]> * do not change seek Signed-off-by: Little-Wallace <[email protected]> * fix fmt Signed-off-by: Little-Wallace <[email protected]> * fix test Signed-off-by: Little-Wallace <[email protected]> * fix number overflow Signed-off-by: Little-Wallace <[email protected]> * increase batch Signed-off-by: Little-Wallace <[email protected]> * revert time Signed-off-by: Little-Wallace <[email protected]> * set last_compacted_idx for split region Signed-off-by: Little-Wallace <[email protected]> * log large range gc task Signed-off-by: Little-Wallace <[email protected]> * address comment Signed-off-by: Little-Wallace <[email protected]> * close tikv#11404 Signed-off-by: Little-Wallace <[email protected]> Co-authored-by: 5kbpers <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#21: Make tikv-server fully generic over kv engines (tikv#10945) * server: Allow flow_info_receiver to be None Signed-off-by: Brian Anderson <[email protected]> * server: Move init_flow_receiver call into init_raw_engines Signed-off-by: Brian Anderson <[email protected]> * server: Extract debug service creation to CreateKvEngine::start_debug_service Signed-off-by: Brian Anderson <[email protected]> * server: Extract backup service creation to CreateKvEngine::start_backup_service Signed-off-by: Brian Anderson <[email protected]> * server: Extract rocks cfg controller registration to CreateKvEngine::register_kv_config Signed-off-by: Brian Anderson <[email protected]> * server: Extract create_raftstore_compaction_listener to free function Signed-off-by: Brian Anderson <[email protected]> * server: Extract create_kv_engine to CreateKvEngine trait Signed-off-by: Brian Anderson <[email protected]> * server: Parameterize EngineResourceInfo over EK Signed-off-by: Brian Anderson <[email protected]> * server: Parameterize TikvServer over EK Signed-off-by: Brian Anderson <[email protected]> * server: Document CreateKvEngine trait Signed-off-by: Brian Anderson <[email protected]> * clippy Signed-off-by: Brian Anderson <[email protected]> * server: Extract raft engine creation to CreateRaftEngine::create_raft_engine Signed-off-by: Brian Anderson <[email protected]> * server: Extract CreateRaftEngine::register_raft_config method Signed-off-by: Brian Anderson <[email protected]> * server: Extract CreateRaftEngine::create_engine_infos method Signed-off-by: Brian Anderson <[email protected]> * server: Unspecialize init_raw_engines method Signed-off-by: Brian Anderson <[email protected]> * server: Extract setup code to generic run_server function Signed-off-by: Brian Anderson <[email protected]> * clippy ref tikv#6402 Signed-off-by: Brian Anderson <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#22: Revert "Make tikv-server fully generic over kv engines (tikv#10945)" (tikv#11462) This reverts commit df2e921. Ref tikv#11461. Signed-off-by: Jay Lee <[email protected]> # This is the commit message tikv#23: raftstore: flush waterfall metrics when using synchronous write (tikv#11451) Signed-off-by: gengliqi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#24: fix one disk full unstable test case on assert condition. (tikv#11391) close tikv#11389 Signed-off-by: tier-cap <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#25: resolved_ts: return early after regions already had quorum (tikv#11352) * close tikv#11351 Signed-off-by: 5kbpers <[email protected]> * fix tests Signed-off-by: 5kbpers <[email protected]> * fix tests Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Signed-off-by: 5kbpers <[email protected]> Co-authored-by: Zixiong Liu <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * make clippy happy Signed-off-by: 5kbpers <[email protected]> * close tikv#11400 Signed-off-by: 5kbpers <[email protected]> Co-authored-by: qupeng <[email protected]> Co-authored-by: Zixiong Liu <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#26: support to get approximate keys and size for a given range from UserCollectedProperties (tikv#11465) Signed-off-by: qupeng <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#27: copr: add paging API support for streaming-like processling (tikv#11300) * copr response with range Signed-off-by: you06 <[email protected]> * switch to new kvproto Signed-off-by: you06 <[email protected]> * fix bugs... Signed-off-by: you06 <[email protected]> * format Signed-off-by: you06 <[email protected]> * remove unsed field Signed-off-by: you06 <[email protected]> * add comments Signed-off-by: you06 <[email protected]> * update kvproto Signed-off-by: you06 <[email protected]> * fix bench Signed-off-by: you06 <[email protected]> * update cargo.lock Signed-off-by: you06 <[email protected]> * update comment Signed-off-by: you06 <[email protected]> * add test for copr paging Signed-off-by: you06 <[email protected]> * close tikv#11448 Signed-off-by: you06 <[email protected]> * Update components/tidb_query_executors/src/runner.rs Co-authored-by: Lei Zhao <[email protected]> Signed-off-by: you06 <[email protected]> * check range in response Signed-off-by: you06 <[email protected]> * format code Signed-off-by: you06 <[email protected]> * add test for reverse scan Signed-off-by: you06 <[email protected]> * add test for multi ranges Signed-off-by: you06 <[email protected]> * test drained with multi ranges Signed-off-by: you06 <[email protected]> * fix clippy Signed-off-by: you06 <[email protected]> * fix lint Signed-off-by: you06 <[email protected]> * paging by all record count Signed-off-by: you06 <[email protected]> * format Signed-off-by: you06 <[email protected]> Co-authored-by: Lei Zhao <[email protected]> # This is the commit message tikv#28: encryption: support seek for encryption writer (tikv#10929) * support seek for encryption writer Signed-off-by: tabokie <[email protected]> * minor fix, ref: tikv#11119 Signed-off-by: tabokie <[email protected]> * fix error from merge Signed-off-by: tabokie <[email protected]> * fight format checker, ref tikv#11119 Signed-off-by: tabokie <[email protected]> * add wrappers Signed-off-by: tabokie <[email protected]> * remove pub quantifiers and fix lint Signed-off-by: tabokie <[email protected]> * fix name export Signed-off-by: tabokie <[email protected]> * revert naming changes Signed-off-by: tabokie <[email protected]> * address comment on unit test Signed-off-by: tabokie <[email protected]> * fix test and inline wrapper calls Signed-off-by: tabokie <[email protected]> * fix comments Signed-off-by: tabokie <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#29: server: remove register PD process (tikv#10985) * remove register PD process Signed-off-by: Ryan Leung <[email protected]> * resolve conflicts close tikv#11472 Signed-off-by: Ryan Leung <[email protected]> # This is the commit message tikv#30: engine: provide customized file builder to raft engine (tikv#10937) * open raft engine with customized file builder ref tikv#11119 Signed-off-by: tabokie <[email protected]> * overhaul configurations Signed-off-by: tabokie <[email protected]> * update default config and address comment Signed-off-by: tabokie <[email protected]> Co-authored-by: qupeng <[email protected]> # This is the commit message tikv#31: *: update some dependencies (tikv#11458) close tikv#10997, ref tikv#11276 Signed-off-by: tabokie <[email protected]> # This is the commit message tikv#32: coprocessor: add `to_binary` and `from_binary` function (tikv#11397) Signed-off-by: xiongjiwei <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#33: server: fix IOType configure for recving snapshot (tikv#11122) * fix IOType for recving snapshot Signed-off-by: MuZhou233 <[email protected]> * Set io_type before Write::write_all. close tikv#10659 Signed-off-by: MuZhou233 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#34: raftstore: add pessimistic lock table to peer (tikv#11459) ref tikv#11452 Signed-off-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#35: engine: fix backward compatibility issue of raft-engine.recovery-mode (tikv#11490) * add test case Signed-off-by: tabokie <[email protected]> * update raft-engine, fix tikv#11489 Signed-off-by: tabokie <[email protected]> * handle timeout error Signed-off-by: tabokie <[email protected]> Co-authored-by: qupeng <[email protected]> # This is the commit message tikv#36: metrics: add duration composition panel (tikv#11491) * metrics: add duration composition pannel Signed-off-by: zhangjinpeng1987 <[email protected]> * close tikv#11492 Signed-off-by: zhangjinpeng1987 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#37: *: support read-through lock (tikv#11238) * storage: get/batch_get support read-through lock Signed-off-by: youjiali1995 <[email protected]> * storage: scan support read-through lock Signed-off-by: youjiali1995 <[email protected]> * coprocessor: support read-through lock Signed-off-by: youjiali1995 <[email protected]> * don't check access_locks when checking memory locks Signed-off-by: youjiali1995 <[email protected]> * fix clippy close tikv#11402 Signed-off-by: youjiali1995 <[email protected]> * update kvproto and add some comments Signed-off-by: youjiali1995 <[email protected]> * test desc scan Signed-off-by: youjiali1995 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#38: storage: support raw value encode for API V2 (tikv#11139) * implement api v2 raw value Signed-off-by: Andy Lok <[email protected]> * storage: support raw value encode for API V2 ref tikv#10938 close tikv#11041 Signed-off-by: Andy Lok <[email protected]> * fix clippy Signed-off-by: Andy Lok <[email protected]> * fix test Signed-off-by: Andy Lok <[email protected]> * fix test Signed-off-by: Andy Lok <[email protected]> * improve Signed-off-by: Andy Lok <[email protected]> * fix clippy Signed-off-by: Andy Lok <[email protected]> * remove println Signed-off-by: Andy Lok <[email protected]> * improve test Signed-off-by: Andy Lok <[email protected]> * fix test Signed-off-by: Andy Lok <[email protected]> * add test Signed-off-by: Andy Lok <[email protected]> * fix Signed-off-by: Andy Lok <[email protected]> * fix Signed-off-by: Andy Lok <[email protected]> * disable test Signed-off-by: Andy Lok <[email protected]> * fix Signed-off-by: Andy Lok <[email protected]> * fix Signed-off-by: Andy Lok <[email protected]> * fix Signed-off-by: Andy Lok <[email protected]> * fix test Signed-off-by: Andy Lok <[email protected]> * fix test Signed-off-by: Andy Lok <[email protected]> * fix test Signed-off-by: Andy Lok <[email protected]> * fix test again Signed-off-by: Andy Lok <[email protected]> * fix test again Signed-off-by: Andy Lok <[email protected]> * fix test again Signed-off-by: Andy Lok <[email protected]> * fix test Signed-off-by: Andy Lok <[email protected]> * fix overflow Signed-off-by: Andy Lok <[email protected]> * fix overflow Signed-off-by: Andy Lok <[email protected]> * improve Signed-off-by: Andy Lok <[email protected]> * improve comment Signed-off-by: Andy Lok <[email protected]> * add case Signed-off-by: Andy Lok <[email protected]> * improve comment Signed-off-by: Andy Lok <[email protected]> * fix clippy Signed-off-by: Andy Lok <[email protected]> * improve Signed-off-by: Andy Lok <[email protected]> * fix test Signed-off-by: Andy Lok <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#39: *: replace snapshot `CbContext` with `SnapshotExt` (tikv#11471) ref tikv#11452 Signed-off-by: Yilin Chen <[email protected]> # This is the commit message tikv#40: Add comments Signed-off-by: mornyx <[email protected]> # This is the commit message tikv#41: Extract one stats struct for KvGet/KvBatchGet Signed-off-by: mornyx <[email protected]> # This is the commit message tikv#42: raftstore: relax merge result check (tikv#11478) * add test case Signed-off-by: Jay Lee <[email protected]> * raftstore: relax merge result check Close tikv#11475. Signed-off-by: Jay Lee <[email protected]> * fix unstable transfer leader case Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Liqi Geng <[email protected]> # This is the commit message tikv#43: engine: move raft engine purge job to a dedicated thread (tikv#11497) * update raft engine and implement batch_gc, ref tikv#11119 Signed-off-by: tabokie <[email protected]> * introduce purge-worker Signed-off-by: tabokie <[email protected]> * clean up old worker Signed-off-by: tabokie <[email protected]> * update raft engine Signed-off-by: tabokie <[email protected]> # This is the commit message tikv#44: Update max_ts for Prewrite and AcquirePessimisticLock with should_not_exist flag. (tikv#11488) * update max_ts for prewrite with should_not_exist flag Signed-off-by: dwangxxx <[email protected]> * close tikv#11315 Signed-off-by: dwangxxx <[email protected]> * close tikv#11315 Signed-off-by: dwangxxx <[email protected]> * fix unit test Signed-off-by: dwangxxx <[email protected]> * fix format Signed-off-by: dwangxxx <[email protected]> * fix test Signed-off-by: dwangxxx <[email protected]> * add test. Signed-off-by: dwangxxx <[email protected]> * fix clippy error Signed-off-by: dwangxxx <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#45: Rename KvGetLatencyStats Signed-off-by: mornyx <[email protected]> # This is the commit message tikv#46: Rename Signed-off-by: mornyx <[email protected]> # This is the commit message tikv#47: *: simplify pull request template (tikv#11504) We only needs to explain what's change. Problem description should be traced as issue. This PR also polishes contributing guide to explain the new flow. Close tikv#11503. Signed-off-by: Jay Lee <[email protected]> Co-authored-by: qupeng <[email protected]> # This is the commit message tikv#48: cdc: introduce TsFilter into incremental scan (tikv#11385) * close tikv#11384 Signed-off-by: qupeng <[email protected]> # This is the commit message tikv#49: metric: add stale read count metric (tikv#11051) Signed-off-by: linning <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#50: storage:change configuration storage.max-key-size default value from 4096 to 8192 (tikv#11331) close tikv#11241 Signed-off-by: TonsnakeLin <[email protected]> # This is the commit message tikv#51: storage: make it cheaper to clone Scheduler (tikv#11501) closes tikv#11500 Signed-off-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#52: raftclient: count term and index in estimated size (tikv#11493) * add test case Signed-off-by: Jay Lee <[email protected]> * count term and index tag If there are many entries in a message, the estimated size of message can be way smaller than the actual size. This PR fixes the error by also counting index and term in estimation. It also remove the hard limit as the estimation is closed enough. Close tikv#9714. Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#53: scanner/forward: move to next key if accesses a lock with delete type (tikv#11544) close tikv#11541 Signed-off-by: youjiali1995 <[email protected]> # This is the commit message tikv#54: *: add configuration for in-memory pessimistic lock (tikv#11481) ref tikv#11452 Signed-off-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> # This is the commit message tikv#55: backup: fix background panic in `endpoint::tests::test_adjust_thread_pool_size` (tikv#11556) * backup: fix tikv#11548 Signed-off-by: yujuncen <[email protected]> # This is the commit message tikv#56: tikv_util: reimplement poll_future_notify (tikv#11550) The current implementation is very easy to get deadlock if the future is woken immediately during polling. This PR resolves the issue by maintaining states instead. Note, if the future is implemented in the wrong way that always notifying the future without checking if it has readiness, the new implementation can run into dead loop. Close tikv#11549. Co-authored-by: qupeng <[email protected]> # This is the commit message tikv#57: fix taking rootfs as filesystem by mistake (tikv#11554) * fix adding rootfs as filesystem by mistake Signed-off-by: onlyacat <[email protected]> * reformat by lints Signed-off-by: onlyacat <[email protected]> * close tikv#11552 Signed-off-by: onlyacat <[email protected]> # This is the commit message tikv#58: raftstore: set term when catch up logs for merge (tikv#11545) * add test case Signed-off-by: Jay Lee <[email protected]> * raftstore: set term when catch up logs for merge Prepare merge may includes logs from different terms, which may be even larger than the term of a follower that is lag behind. So when catching up logs by commit merge, its term should also be set to get a correct metadata. Close tikv#11526. Signed-off-by: Jay Lee <[email protected]> * add log for becoming follower Signed-off-by: Jay Lee <[email protected]> * correct commit log check Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]>

Signed-off-by: mornyx <[email protected]> close tikv#8942 tikv_util: upgrade `gag` from 0.1 to 1.0 (tikv#11277) ref tikv#11276 Signed-off-by: kennytm <[email protected]> codec: add compile time assert to guarantee safety (tikv#11357) * codec: add compile time assert to guarantee safety close tikv#11356. Signed-off-by: Jay Lee <[email protected]> * add more explanation. Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> rafstore: fix commit state check in apply thread (tikv#11401) * fix apply check & close tikv#11396 Signed-off-by: gengliqi <[email protected]> * make format Signed-off-by: gengliqi <[email protected]> * format Signed-off-by: gengliqi <[email protected]> backup: create external storage per-backup reqeust. (tikv#11113) * backup: create external storage per-backup reqeust. fixed tikv#11112 Signed-off-by: yujuncen <[email protected]> * backup: make clippy happy Signed-off-by: yujuncen <[email protected]> * backup: fix build Signed-off-by: yujuncen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Remove duplicate enum and macro about numberic_enum_mod (tikv#11347) * remove duplicated and useless `remove duplicated and useless` close tikv#11297 Signed-off-by: yuqi1129 <[email protected]> * remove duplicate macro rocksdb_enum_mod Signed-off-by: yuqi1129 <[email protected]> * fix format problem Signed-off-by: yuqi1129 <[email protected]> * resolve discussion Signed-off-by: yuqi1129 <[email protected]> * fix test error Signed-off-by: yuqi1129 <[email protected]> * move macro `numeric_enum_serializing_mod` to a proper location Signed-off-by: yuqi1129 <[email protected]> * move `perf_level_serde` of mod config in engine trait to mod perf_context Signed-off-by: yuqi1129 <[email protected]> * Add test about `rate-limiter-mode` and fix compile error in integration test Signed-off-by: yuqi1129 <[email protected]> Co-authored-by: Xinye Tao <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> ctl: detect raft db correctly (tikv#11395) Close tikv#11393. Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Support to iterate table properties for KvEngine (tikv#11388) * support to iterate table properties for KvEngine * close tikv#11387 Signed-off-by: qupeng <[email protected]> gc: Fix GC scan effectiveness to avoid OOM (tikv#11416) * close tikv#11410 limit the pending task for gc worker Signed-off-by: Connor1996 <[email protected]> * avoid using prefix seek Signed-off-by: Connor1996 <[email protected]> * rename test Signed-off-by: Connor1996 <[email protected]> * address comment Signed-off-by: Connor1996 <[email protected]> * fix build Signed-off-by: Connor1996 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> fix a panic about parsing memory.limit_in_bytes (tikv#11354) * handle int overflow for memory.limit_in_bytes clsoe tikv#11353. Signed-off-by: qupeng <[email protected]> * close tikv#11353 Signed-off-by: qupeng <[email protected]> * address comments Signed-off-by: qupeng <[email protected]> build: fix make docker on Linux (tikv#11349) * fix make docker fix tikv#11312 Signed-off-by: tabokie <[email protected]> * fix cargo Signed-off-by: tabokie <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> engine_rocks: ignore NoSpace error during flush or compaction (tikv#11225) * ignore NoSpace error during flush or compaction fix tikv#11224 Signed-off-by: tabokie <[email protected]> * use constant Signed-off-by: tabokie <[email protected]> * fix typo Signed-off-by: tabokie <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> gc: set pending capacity for gc worker (tikv#11429) Signed-off-by: Connor1996 <[email protected]> raftstore: support synchronous write (tikv#11418) * support sync write & ref tikv#11165 Signed-off-by: gengliqi <[email protected]> * add tests Signed-off-by: gengliqi <[email protected]> * change default store-io-pool-size to 0 Signed-off-by: gengliqi <[email protected]> * remove unnecessary test Signed-off-by: gengliqi <[email protected]> * merge raftstore: separate read write ready(tikv#10592) Signed-off-by: gengliqi <[email protected]> * address comments Signed-off-by: gengliqi <[email protected]> * fix test Signed-off-by: gengliqi <[email protected]> * make clippy happy Signed-off-by: gengliqi <[email protected]> * address comment Signed-off-by: gengliqi <[email protected]> * remove offset in NormalFsm Signed-off-by: gengliqi <[email protected]> * make format Signed-off-by: gengliqi <[email protected]> metrics: add Raft Engine panel (tikv#11415) Signed-off-by: tabokie <[email protected]> resource-metering: add some metrics (tikv#11414) * resource-metering: add some metrics Signed-off-by: Zhenchi <[email protected]> * more log Signed-off-by: Zhenchi <[email protected]> * polish description, ref tikv#11436 Signed-off-by: Zhenchi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> metrics: add asynchronous write io panels (tikv#11408) * update metrics name Signed-off-by: gengliqi <[email protected]> * update tikv-details & close tikv#10540 Signed-off-by: gengliqi <[email protected]> * rename Raft GC to Raft Log GC Signed-off-by: gengliqi <[email protected]> * address comment Signed-off-by: gengliqi <[email protected]> * fix datasource Signed-off-by: gengliqi <[email protected]> * add more metrics Signed-off-by: gengliqi <[email protected]> * update metrics Signed-off-by: gengliqi <[email protected]> * update Signed-off-by: gengliqi <[email protected]> storage: fix reverse scan check memory locks (tikv#11447) close tikv#11440 Signed-off-by: youjiali1995 <[email protected]> test: make test_sending_fail_with_net_error more statable (tikv#11366) * sending count should deregister on time Signed-off-by: bufferflies <[email protected]> * make test statable Signed-off-by: bufferflies <[email protected]> * remove unused change Signed-off-by: bufferflies <[email protected]> * remove unused change Signed-off-by: bufferflies <[email protected]> * close tikv#11365 Signed-off-by: bufferflies <[email protected]> * reformat annotation Signed-off-by: bufferflies <[email protected]> * add send once filter Signed-off-by: bufferflies <[email protected]> * lint pass Signed-off-by: bufferflies <[email protected]> * lint pass Signed-off-by: bufferflies <[email protected]> * replace sendOnceMsgFilter Signed-off-by: bufferflies <[email protected]> * remove unnecessary change Signed-off-by: bufferflies <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> raftstore: increase batch for raftlog-gc-worker (tikv#11405) * increase batch Signed-off-by: Little-Wallace <[email protected]> * do not change seek Signed-off-by: Little-Wallace <[email protected]> * fix fmt Signed-off-by: Little-Wallace <[email protected]> * fix test Signed-off-by: Little-Wallace <[email protected]> * fix number overflow Signed-off-by: Little-Wallace <[email protected]> * increase batch Signed-off-by: Little-Wallace <[email protected]> * revert time Signed-off-by: Little-Wallace <[email protected]> * set last_compacted_idx for split region Signed-off-by: Little-Wallace <[email protected]> * log large range gc task Signed-off-by: Little-Wallace <[email protected]> * address comment Signed-off-by: Little-Wallace <[email protected]> * close tikv#11404 Signed-off-by: Little-Wallace <[email protected]> Co-authored-by: 5kbpers <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Make tikv-server fully generic over kv engines (tikv#10945) * server: Allow flow_info_receiver to be None Signed-off-by: Brian Anderson <[email protected]> * server: Move init_flow_receiver call into init_raw_engines Signed-off-by: Brian Anderson <[email protected]> * server: Extract debug service creation to CreateKvEngine::start_debug_service Signed-off-by: Brian Anderson <[email protected]> * server: Extract backup service creation to CreateKvEngine::start_backup_service Signed-off-by: Brian Anderson <[email protected]> * server: Extract rocks cfg controller registration to CreateKvEngine::register_kv_config Signed-off-by: Brian Anderson <[email protected]> * server: Extract create_raftstore_compaction_listener to free function Signed-off-by: Brian Anderson <[email protected]> * server: Extract create_kv_engine to CreateKvEngine trait Signed-off-by: Brian Anderson <[email protected]> * server: Parameterize EngineResourceInfo over EK Signed-off-by: Brian Anderson <[email protected]> * server: Parameterize TikvServer over EK Signed-off-by: Brian Anderson <[email protected]> * server: Document CreateKvEngine trait Signed-off-by: Brian Anderson <[email protected]> * clippy Signed-off-by: Brian Anderson <[email protected]> * server: Extract raft engine creation to CreateRaftEngine::create_raft_engine Signed-off-by: Brian Anderson <[email protected]> * server: Extract CreateRaftEngine::register_raft_config method Signed-off-by: Brian Anderson <[email protected]> * server: Extract CreateRaftEngine::create_engine_infos method Signed-off-by: Brian Anderson <[email protected]> * server: Unspecialize init_raw_engines method Signed-off-by: Brian Anderson <[email protected]> * server: Extract setup code to generic run_server function Signed-off-by: Brian Anderson <[email protected]> * clippy ref tikv#6402 Signed-off-by: Brian Anderson <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Revert "Make tikv-server fully generic over kv engines (tikv#10945)" (tikv#11462) This reverts commit df2e921. Ref tikv#11461. Signed-off-by: Jay Lee <[email protected]> raftstore: flush waterfall metrics when using synchronous write (tikv#11451) Signed-off-by: gengliqi <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> fix one disk full unstable test case on assert condition. (tikv#11391) close tikv#11389 Signed-off-by: tier-cap <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> resolved_ts: return early after regions already had quorum (tikv#11352) * close tikv#11351 Signed-off-by: 5kbpers <[email protected]> * fix tests Signed-off-by: 5kbpers <[email protected]> * fix tests Signed-off-by: 5kbpers <[email protected]> * Apply suggestions from code review Signed-off-by: 5kbpers <[email protected]> Co-authored-by: Zixiong Liu <[email protected]> * address comments Signed-off-by: 5kbpers <[email protected]> * make clippy happy Signed-off-by: 5kbpers <[email protected]> * close tikv#11400 Signed-off-by: 5kbpers <[email protected]> Co-authored-by: qupeng <[email protected]> Co-authored-by: Zixiong Liu <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> support to get approximate keys and size for a given range from UserCollectedProperties (tikv#11465) Signed-off-by: qupeng <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> copr: add paging API support for streaming-like processling (tikv#11300) * copr response with range Signed-off-by: you06 <[email protected]> * switch to new kvproto Signed-off-by: you06 <[email protected]> * fix bugs... Signed-off-by: you06 <[email protected]> * format Signed-off-by: you06 <[email protected]> * remove unsed field Signed-off-by: you06 <[email protected]> * add comments Signed-off-by: you06 <[email protected]> * update kvproto Signed-off-by: you06 <[email protected]> * fix bench Signed-off-by: you06 <[email protected]> * update cargo.lock Signed-off-by: you06 <[email protected]> * update comment Signed-off-by: you06 <[email protected]> * add test for copr paging Signed-off-by: you06 <[email protected]> * close tikv#11448 Signed-off-by: you06 <[email protected]> * Update components/tidb_query_executors/src/runner.rs Co-authored-by: Lei Zhao <[email protected]> Signed-off-by: you06 <[email protected]> * check range in response Signed-off-by: you06 <[email protected]> * format code Signed-off-by: you06 <[email protected]> * add test for reverse scan Signed-off-by: you06 <[email protected]> * add test for multi ranges Signed-off-by: you06 <[email protected]> * test drained with multi ranges Signed-off-by: you06 <[email protected]> * fix clippy Signed-off-by: you06 <[email protected]> * fix lint Signed-off-by: you06 <[email protected]> * paging by all record count Signed-off-by: you06 <[email protected]> * format Signed-off-by: you06 <[email protected]> Co-authored-by: Lei Zhao <[email protected]> encryption: support seek for encryption writer (tikv#10929) * support seek for encryption writer Signed-off-by: tabokie <[email protected]> * minor fix, ref: tikv#11119 Signed-off-by: tabokie <[email protected]> * fix error from merge Signed-off-by: tabokie <[email protected]> * fight format checker, ref tikv#11119 Signed-off-by: tabokie <[email protected]> * add wrappers Signed-off-by: tabokie <[email protected]> * remove pub quantifiers and fix lint Signed-off-by: tabokie <[email protected]> * fix name export Signed-off-by: tabokie <[email protected]> * revert naming changes Signed-off-by: tabokie <[email protected]> * address comment on unit test Signed-off-by: tabokie <[email protected]> * fix test and inline wrapper calls Signed-off-by: tabokie <[email protected]> * fix comments Signed-off-by: tabokie <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> server: remove register PD process (tikv#10985) * remove register PD process Signed-off-by: Ryan Leung <[email protected]> * resolve conflicts close tikv#11472 Signed-off-by: Ryan Leung <[email protected]> engine: provide customized file builder to raft engine (tikv#10937) * open raft engine with customized file builder ref tikv#11119 Signed-off-by: tabokie <[email protected]> * overhaul configurations Signed-off-by: tabokie <[email protected]> * update default config and address comment Signed-off-by: tabokie <[email protected]> Co-authored-by: qupeng <[email protected]> *: update some dependencies (tikv#11458) close tikv#10997, ref tikv#11276 Signed-off-by: tabokie <[email protected]> coprocessor: add `to_binary` and `from_binary` function (tikv#11397) Signed-off-by: xiongjiwei <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> server: fix IOType configure for recving snapshot (tikv#11122) * fix IOType for recving snapshot Signed-off-by: MuZhou233 <[email protected]> * Set io_type before Write::write_all. close tikv#10659 Signed-off-by: MuZhou233 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> raftstore: add pessimistic lock table to peer (tikv#11459) ref tikv#11452 Signed-off-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> engine: fix backward compatibility issue of raft-engine.recovery-mode (tikv#11490) * add test case Signed-off-by: tabokie <[email protected]> * update raft-engine, fix tikv#11489 Signed-off-by: tabokie <[email protected]> * handle timeout error Signed-off-by: tabokie <[email protected]> Co-authored-by: qupeng <[email protected]> metrics: add duration composition panel (tikv#11491) * metrics: add duration composition pannel Signed-off-by: zhangjinpeng1987 <[email protected]> * close tikv#11492 Signed-off-by: zhangjinpeng1987 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> *: support read-through lock (tikv#11238) * storage: get/batch_get support read-through lock Signed-off-by: youjiali1995 <[email protected]> * storage: scan support read-through lock Signed-off-by: youjiali1995 <[email protected]> * coprocessor: support read-through lock Signed-off-by: youjiali1995 <[email protected]> * don't check access_locks when checking memory locks Signed-off-by: youjiali1995 <[email protected]> * fix clippy close tikv#11402 Signed-off-by: youjiali1995 <[email protected]> * update kvproto and add some comments Signed-off-by: youjiali1995 <[email protected]> * test desc scan Signed-off-by: youjiali1995 <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> storage: support raw value encode for API V2 (tikv#11139) * implement api v2 raw value Signed-off-by: Andy Lok <[email protected]> * storage: support raw value encode for API V2 ref tikv#10938 close tikv#11041 Signed-off-by: Andy Lok <[email protected]> * fix clippy Signed-off-by: Andy Lok <[email protected]> * fix test Signed-off-by: Andy Lok <[email protected]> * fix test Signed-off-by: Andy Lok <[email protected]> * improve Signed-off-by: Andy Lok <[email protected]> * fix clippy Signed-off-by: Andy Lok <[email protected]> * remove println Signed-off-by: Andy Lok <[email protected]> * improve test Signed-off-by: Andy Lok <[email protected]> * fix test Signed-off-by: Andy Lok <[email protected]> * add test Signed-off-by: Andy Lok <[email protected]> * fix Signed-off-by: Andy Lok <[email protected]> * fix Signed-off-by: Andy Lok <[email protected]> * disable test Signed-off-by: Andy Lok <[email protected]> * fix Signed-off-by: Andy Lok <[email protected]> * fix Signed-off-by: Andy Lok <[email protected]> * fix Signed-off-by: Andy Lok <[email protected]> * fix test Signed-off-by: Andy Lok <[email protected]> * fix test Signed-off-by: Andy Lok <[email protected]> * fix test Signed-off-by: Andy Lok <[email protected]> * fix test again Signed-off-by: Andy Lok <[email protected]> * fix test again Signed-off-by: Andy Lok <[email protected]> * fix test again Signed-off-by: Andy Lok <[email protected]> * fix test Signed-off-by: Andy Lok <[email protected]> * fix overflow Signed-off-by: Andy Lok <[email protected]> * fix overflow Signed-off-by: Andy Lok <[email protected]> * improve Signed-off-by: Andy Lok <[email protected]> * improve comment Signed-off-by: Andy Lok <[email protected]> * add case Signed-off-by: Andy Lok <[email protected]> * improve comment Signed-off-by: Andy Lok <[email protected]> * fix clippy Signed-off-by: Andy Lok <[email protected]> * improve Signed-off-by: Andy Lok <[email protected]> * fix test Signed-off-by: Andy Lok <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> *: replace snapshot `CbContext` with `SnapshotExt` (tikv#11471) ref tikv#11452 Signed-off-by: Yilin Chen <[email protected]> Add comments Signed-off-by: mornyx <[email protected]> Extract one stats struct for KvGet/KvBatchGet Signed-off-by: mornyx <[email protected]> raftstore: relax merge result check (tikv#11478) * add test case Signed-off-by: Jay Lee <[email protected]> * raftstore: relax merge result check Close tikv#11475. Signed-off-by: Jay Lee <[email protected]> * fix unstable transfer leader case Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Liqi Geng <[email protected]> engine: move raft engine purge job to a dedicated thread (tikv#11497) * update raft engine and implement batch_gc, ref tikv#11119 Signed-off-by: tabokie <[email protected]> * introduce purge-worker Signed-off-by: tabokie <[email protected]> * clean up old worker Signed-off-by: tabokie <[email protected]> * update raft engine Signed-off-by: tabokie <[email protected]> Update max_ts for Prewrite and AcquirePessimisticLock with should_not_exist flag. (tikv#11488) * update max_ts for prewrite with should_not_exist flag Signed-off-by: dwangxxx <[email protected]> * close tikv#11315 Signed-off-by: dwangxxx <[email protected]> * close tikv#11315 Signed-off-by: dwangxxx <[email protected]> * fix unit test Signed-off-by: dwangxxx <[email protected]> * fix format Signed-off-by: dwangxxx <[email protected]> * fix test Signed-off-by: dwangxxx <[email protected]> * add test. Signed-off-by: dwangxxx <[email protected]> * fix clippy error Signed-off-by: dwangxxx <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> Rename KvGetLatencyStats Signed-off-by: mornyx <[email protected]> Rename Signed-off-by: mornyx <[email protected]> *: simplify pull request template (tikv#11504) We only needs to explain what's change. Problem description should be traced as issue. This PR also polishes contributing guide to explain the new flow. Close tikv#11503. Signed-off-by: Jay Lee <[email protected]> Co-authored-by: qupeng <[email protected]> cdc: introduce TsFilter into incremental scan (tikv#11385) * close tikv#11384 Signed-off-by: qupeng <[email protected]> metric: add stale read count metric (tikv#11051) Signed-off-by: linning <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> storage:change configuration storage.max-key-size default value from 4096 to 8192 (tikv#11331) close tikv#11241 Signed-off-by: TonsnakeLin <[email protected]> storage: make it cheaper to clone Scheduler (tikv#11501) closes tikv#11500 Signed-off-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> raftclient: count term and index in estimated size (tikv#11493) * add test case Signed-off-by: Jay Lee <[email protected]> * count term and index tag If there are many entries in a message, the estimated size of message can be way smaller than the actual size. This PR fixes the error by also counting index and term in estimation. It also remove the hard limit as the estimation is closed enough. Close tikv#9714. Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> scanner/forward: move to next key if accesses a lock with delete type (tikv#11544) close tikv#11541 Signed-off-by: youjiali1995 <[email protected]> *: add configuration for in-memory pessimistic lock (tikv#11481) ref tikv#11452 Signed-off-by: Yilin Chen <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> backup: fix background panic in `endpoint::tests::test_adjust_thread_pool_size` (tikv#11556) * backup: fix tikv#11548 Signed-off-by: yujuncen <[email protected]> tikv_util: reimplement poll_future_notify (tikv#11550) The current implementation is very easy to get deadlock if the future is woken immediately during polling. This PR resolves the issue by maintaining states instead. Note, if the future is implemented in the wrong way that always notifying the future without checking if it has readiness, the new implementation can run into dead loop. Close tikv#11549. Co-authored-by: qupeng <[email protected]> fix taking rootfs as filesystem by mistake (tikv#11554) * fix adding rootfs as filesystem by mistake Signed-off-by: onlyacat <[email protected]> * reformat by lints Signed-off-by: onlyacat <[email protected]> * close tikv#11552 Signed-off-by: onlyacat <[email protected]> raftstore: set term when catch up logs for merge (tikv#11545) * add test case Signed-off-by: Jay Lee <[email protected]> * raftstore: set term when catch up logs for merge Prepare merge may includes logs from different terms, which may be even larger than the term of a follower that is lag behind. So when catching up logs by commit merge, its term should also be set to get a correct metadata. Close tikv#11526. Signed-off-by: Jay Lee <[email protected]> * add log for becoming follower Signed-off-by: Jay Lee <[email protected]> * correct commit log check Signed-off-by: Jay Lee <[email protected]> Co-authored-by: Ti Chi Robot <[email protected]> storage: Add API V2 check for RawKV and TxnKV requests (part 2) (tikv#11228) * wip Signed-off-by: pingyu <[email protected]> * wip Signed-off-by: pingyu <[email protected]> * wip Signed-off-by: pingyu <[email protected]> * wip Signed-off-by: pingyu <[email protected]> * wip Signed-off-by: pingyu <[email protected]> * wip Signed-off-by: pingyu <[email protected]> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2) Signed-off-by: pingyu <[email protected]> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2) Signed-off-by: pingyu <[email protected]> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2) Signed-off-by: pingyu <[email protected]> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2) Signed-off-by: pingyu <[email protected]> * resolve conflict Signed-off-by: pingyu <[email protected]> * add raw_key_maybe_unbounded_into_option Signed-off-by: pingyu <[email protected]> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2) Signed-off-by: pingyu <[email protected]> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2) Signed-off-by: pingyu <[email protected]> * ref tikv#10974: address comments Signed-off-by: pingyu <[email protected]> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2) Signed-off-by: pingyu <[email protected]> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2) Signed-off-by: pingyu <[email protected]> * remove make_invalid_key_prefix_err Signed-off-by: andylokandy <[email protected]> Co-authored-by: andylokandy <[email protected]> storage: read and write in-memory pessimistic locks in the scheduler (tikv#11529) ref tikv#11452 Signed-off-by: Yilin Chen <[email protected]> tolerate unmounted cgroup controller (tikv#11570) * tolerate unmounted cgroup controller, fix tikv#11569 Signed-off-by: tabokie <[email protected]> * add test Signed-off-by: tabokie <[email protected]> * address comment Signed-off-by: tabokie <[email protected]> cdc: stabilize some cdc cases (tikv#11586) * close tikv#11560 Signed-off-by: qupeng <[email protected]>

cscetbon · 2023-02-26T03:09:08Z

Hey @brson any news on this ?

brson added sig/engine SIG: Engine status/mentor This issue is currently mentored help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. labels Jan 3, 2020

brson mentioned this issue Jan 3, 2020

Abstract storage engine API #4184

Closed

brson changed the title ~~Engine abstraction~~ Engine abstraction tracking issue Jan 3, 2020

brson mentioned this issue Jan 28, 2020

Add reference-counted TablePropertiesCollection bindings tikv/rust-rocksdb#425

Merged

brson mentioned this issue Feb 16, 2020

Migrate to abstract TablePropertiesCollection types #6630

Merged

Rustin170506 mentioned this issue Feb 20, 2020

engine_rocks,engine_traits: abstract rocks util #6661

Merged

brson mentioned this issue Feb 26, 2020

Move WriteBatch from engine to engine_traits and engine_rocks #6697

Merged

brson mentioned this issue Feb 26, 2020

Parameterize much of raftstore over engines #6707

Merged

This was referenced Feb 29, 2020

*: move event listener into engine_rocks #6739

Merged

*: remove CompactionJobInfo re-exports #6989

Merged

brson mentioned this issue Mar 6, 2020

Remove engine::util #7029

Merged

Rustin170506 mentioned this issue Mar 7, 2020

engine: clean up rocks util mod #7031

Closed

This was referenced Mar 7, 2020

Abstract more raftstore over engine_traits #7032

Merged

Delete several parts of engine crate #7033

Merged

brson mentioned this issue Mar 20, 2020

Make engine only a dev-dependency of raftstore #7164

Merged

This was referenced Mar 31, 2020

Remove protobuf dependencies from engine crate #7320

Merged

Remove iterator code from engine crate #7321

Merged

Replace uses of DB with RocksEngine in tikv #7368

Merged

raftstore: Convert RocksEngine to type parameters #7371

Merged

This was referenced Mar 18, 2021

server: Use KvTestEngine instead of RocksEngine in test #9846

Merged

Add a few constructor tests to engine_traits_tests #9848

Closed

This was referenced Apr 13, 2021

tikv: Make RaftKv generic over engines #10013

Merged

Make the server::snap module generic over engines #10014

Merged

server: Add KvEngine typarams to resolve #10026

Merged

brson mentioned this issue Apr 21, 2021

Add a WriteBatch test to engine_traits_tests #10058

Closed

This was referenced Apr 22, 2021

Add KvTestEngineIterator export to engine_test and use it in one tikv test #10069

Merged

Replace some RocksEngine with KvEngine type parameters in tikv::server #10096

Merged

brson mentioned this issue Apr 29, 2021

Replace RocksEngine with KvEngine in lock manager #10105

Merged

This was referenced May 28, 2021

server: Replace RocksEngine with KvEngine typaram #10260

Merged

engine_traits: Remove TableProperties / UserCollectedProperties abstractions #10300

Closed

This was referenced Jun 10, 2021

Abstract cdc crate over engines #10344

Merged

engine_traits: Remove TableProperties abstractions #10375

Merged

This was referenced Jul 6, 2021

Add more tests to engine_traits_tests #10527

Merged

Make it possible to launch tikv-server with non-rocks KV engines #10529

Closed

morigs mentioned this issue Jul 8, 2021

In-mem database backend etcd-io/etcd#13140

Closed

skyzh mentioned this issue Jul 27, 2021

whether support Rust to use? wann to merge to tikv bytedance/terarkdb#54

Closed

brson mentioned this issue Aug 19, 2021

Convert more of tikv-server to generic kv engines #10785

Merged

brson mentioned this issue Sep 13, 2021

Make tikv-server fully generic over kv engines #10945

Merged

brson added a commit to brson/tikv that referenced this issue Nov 22, 2021

clippy

2d60fb5

ref tikv#6402 Signed-off-by: Brian Anderson <[email protected]>

tejasvi mentioned this issue Mar 11, 2022

Make storage pluggable #1693

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Engine abstraction tracking issue #6402

Engine abstraction tracking issue #6402

brson commented Jan 3, 2020 •

edited

Loading

brson commented Jan 3, 2020 •

edited

Loading

brson commented Jan 22, 2020

Rustin170506 commented Feb 20, 2020

brson commented Feb 21, 2020

brson commented Feb 21, 2020

brson commented Feb 21, 2020

brson commented Feb 26, 2020

brson commented Feb 26, 2020

brson commented Mar 18, 2020

brson commented Mar 18, 2021 •

edited

Loading

brson commented Apr 22, 2021

brson commented Jul 6, 2021

brson commented Dec 1, 2021

skyzh commented Dec 1, 2021

cscetbon commented Feb 26, 2023

Engine abstraction tracking issue #6402

Engine abstraction tracking issue #6402

Comments

brson commented Jan 3, 2020 • edited Loading

Task list (updated 2021/01/10)

brson commented Jan 3, 2020 • edited Loading

brson commented Jan 22, 2020

Rustin170506 commented Feb 20, 2020

brson commented Feb 21, 2020

brson commented Feb 21, 2020

brson commented Feb 21, 2020

brson commented Feb 26, 2020

brson commented Feb 26, 2020

brson commented Mar 18, 2020

brson commented Mar 18, 2021 • edited Loading

brson commented Apr 22, 2021

brson commented Jul 6, 2021

brson commented Dec 1, 2021

skyzh commented Dec 1, 2021

cscetbon commented Feb 26, 2023

brson commented Jan 3, 2020 •

edited

Loading

brson commented Jan 3, 2020 •

edited

Loading

brson commented Mar 18, 2021 •

edited

Loading