Release date: December 21, 2024

👍 Highlights​

• Supports label_join() and label_replace() for PromQL.
• Supports pushing down IN filters.
• Updates dashboard to v0.7.3

🚀 Features​

• feat: adjust WAL purge default configurations by @killme2008 in #5107
• feat(vector): add scalar add function by @zhongzc in #5119
• feat: update dashboard to v0.7.2 by @ZonaHex in #5141
• feat: implement v1/sql/parse endpoint to parse GreptimeDB's SQL dialect by @waynexia in #5144
• feat: support push down IN filter by @NiwakaDev in #5129
• feat: add prefetch support to PuffinFileFooterReader for reduced I/O time by @WenyXu in #5145
• feat: add prefetch support to InvertedIndexFooterReader for reduced I/O time by @WenyXu in #5146
• feat: introduce PuffinMetadataCache by @WenyXu in #5148
• feat(fuzz): add alter table options for alter fuzzer by @CookiePieWw in #5074
• feat(index): add file_size_hint for remote blob reader by @WenyXu in #5147
• feat: collect reader metrics from prune reader by @evenyag in #5152
• feat: Add vector_scalar_mul function. by @linyihai in #5166
• feat: introduce SKIPPING index (part 1) by @waynexia in #5155
• feat: update dashboard to v0.7.3 by @ZonaHex in #5172
• feat(bloom-filter): add basic bloom filter creator (Part 1) by @zhongzc in #5177
• feat: introduce Buffer for non-continuous bytes by @CookiePieWw in #5164
• feat: impl label_join and label_replace for promql by @killme2008 in #5153
• feat: do not keep MemtableRefs in ScanInput by @evenyag in #5184
• feat: do not remove time filters in ScanRegion by @evenyag in #5180
• feat: extract hints from http header by @fengjiachun in #5128
• feat: show create postgresql foreign table by @sunng87 in #5143
• feat: show flow's mem usage in INFORMATION_SCHEMA.FLOWS by @discord9 in #4890

🐛 Bug Fixes​

• fix: loki write row len error by @paomian in #5161
• fix: support alter table ~ add ~ custom_type by @NiwakaDev in #5165
• fix: correct set_region_role_state_gracefully behaviors by @WenyXu in #5171
• fix: deletion between two put may not work in last_non_null mode by @evenyag in #5168
• fix: display inverted and fulltext index in show index by @lyang24 in #5169
• fix(sqlness): enforce order in union tests by @v0y4g3r in #5190
• fix: validate matcher op for name in promql by @waynexia in #5191
• fix: ensure table route metadata is eventually rolled back on failure by @WenyXu in #5174
• fix(flow): batch builder with type by @discord9 in #5195
• fix: auto created table ttl check by @discord9 in #5203

🚜 Refactor​

• refactor: cache inverted index with fixed-size page by @CookiePieWw in #5114
• refactor: produce BatchBuilder from a Batch to modify it again by @MichaelScofield in #5186
• refactor: remove unused symbols by @waynexia in #5193

📚 Documentation​

• docs: Add index panels to standalone grafana dashboard by @evenyag in #5140
• docs: fix grafana dashboard row by @evenyag in #5192

⚡ Performance​

• perf: avoid cache during compaction by @v0y4g3r in #5135
• perf: avoid holding memtable during compaction by @v0y4g3r in #5157

🧪 Testing​

• test: part of parser test migrated from duckdb by @CookiePieWw in #5125
• test: flow rebuild by @discord9 in #5162
• test: sqlness upgrade compatibility tests by @discord9 in #5126

⚙️ Miscellaneous Tasks​

• chore: bump main branch version to 0.12 by @evenyag in #5133
• chore: add /ready api for health checking by @shuiyisong in #5124
• chore: fix aws_lc not in depend tree check in CI by @discord9 in #5121
• chore: set store_key_prefix for all kvbackend by @fengjiachun in #5132
• chore: decide tag column in log api follow table schema if table exists by @paomian in #5138
• chore: pipeline dryrun api can currently receives pipeline raw content by @paomian in #5142
• ci: use 4xlarge for nightly build by @evenyag in #5158
• chore: remove unused dep by @shuiyisong in #5163
• chore: gauge for flush compaction by @v0y4g3r in #5156
• chore: add nix-shell configuration for a minimal environment for development by @sunng87 in #5175
• chore: add aquamarine to dep lists by @evenyag in #5181
• chore: make nix compilation environment config more robust by @sunng87 in #5183
• chore: update PR template by @killme2008 in #5199
• ci: install latest protobuf in dev-builder image by @MichaelScofield in #5196

All Contributors​

We would like to thank the following contributors from the GreptimeDB community:

@CookiePieWw, @MichaelScofield, @NiwakaDev, @WenyXu, @ZonaHex, @discord9, @evenyag, @fengjiachun, @killme2008, @linyihai, @lyang24, @paomian, @shuiyisong, @sunng87, @v0y4g3r, @waynexia, @zhongzc

Release date: December 10, 2024

👍 Highlights​

• Dashboard supports using a log view to explore logs.
• Fixes performance regressions in v0.10.
• Supports tables with TTL=instant as source tables for flow tasks.
• Supports postgres_fdw to integrate GreptimeDB with existing PostgreSQL databases.

Breaking changes​

• feat!: remove GET method in /debug path by @waynexia in #5102
• fix!: fix regression caused by unbalanced partitions and splitting ranges by @evenyag in #5090
• feat!: enable read cache and write cache when using remote object stores by @killme2008 in #5093

🚀 Features​

• feat: use cache kv manager for SchemaMetadataManager by @v0y4g3r in #5053
• feat: add decolorize processor by @waynexia in #5065
• feat: enable compression for metasrv client by @WenyXu in #5078
• feat: recover file cache index asynchronously by @WenyXu in #5087
• feat: define basic structures and implement TimeFilter by @waynexia in #5086
• feat: update dashboard to v0.7.0 by @ZonaHex in #5100
• feat: add more transaction related statement for postgres interface by @sunng87 in #5081
• feat: ttl=0/instant/forever/humantime&ttl refactor by @discord9 in #5089
• feat: add cursor statements by @sunng87 in #5094
• feat: update pgwire to 0.28 by @sunng87 in #5113
• feat: update dashboard to v0.7.1 by @ZonaHex in #5123

🐛 Bug Fixes​

• fix(metric-engine): set ttl also on opening metadata regions by @v0y4g3r in #5051
• fix: pass series row selector to file range reader by @evenyag in #5054
• fix: allow physical region alter region options by @lyang24 in #5046
• fix(flow): minor fix about count(*)&sink keyword by @discord9 in #5061
• fix: correct is_exceeded_size_limit behavior for in-memory store by @WenyXu in #5082
• fix: schema cache invalidation by @v0y4g3r in #5067
• fix: put PipelineChecker at the end by @waynexia in #5092
• fix: show create table doesn't quote option keys which contains dot by @killme2008 in #5108

🚜 Refactor​

• refactor: expose configs for http clients used in object store by @MichaelScofield in #5041
• refactor: remove built-in apidocs and schemars by @sunng87 in #5068
• refactor: replace LogHandler with PipelineHandler by @waynexia in #5096
• refactor: relocate CLI to a dedicated directory by @WenyXu in #5101
• refactor: extract implicit conversion helper functions of vector type by @zhongzc in #5118

📚 Documentation​

• docs: tweak readme and AUTHOR by @killme2008 in #5069
• docs: remove lg_prof_interval from env by @evenyag in #5103

⚡ Performance​

• perf: take a new batch to reduce last row cache usage by @evenyag in #5095

🧪 Testing​

• test: adds sqlness test for TTL by @killme2008 in #5063

⚙️ Miscellaneous Tasks​

• chore: bump version of main branch to v0.11.0 by @zhongzc in #5057
• chore: remove openssl deps by @sunng87 in #5079
• chore: correct example config file by @WenyXu in #5105
• chore: Add timeout setting for find_ttl. by @linyihai in #5088
• ci: set meta replicas to 1 by @WenyXu in #5111
• chore: Reduce FETCH_OPTION_TIMEOUT from 10 to 3 seconds in config.rs by @v0y4g3r in #5117

All Contributors​

We would like to thank the following contributors from the GreptimeDB community:

@MichaelScofield, @WenyXu, @ZonaHex, @discord9, @evenyag, @killme2008, @linyihai, @lyang24, @sunng87, @v0y4g3r, @waynexia, @zhongzc

Release date: November 26, 2024

👍 Highlights​

• Vector type: GreptimeDB supports vector data types to optimize edge scenarios, such as IoT in vehicles, enabling efficient storage and computation of vector data for real-time perception and AI applications in smart driving.
• Free index: Inverted indexes are no longer tied to primary keys, allowing users to create them on any column, enhancing query flexibility and efficiency across different scenarios.
• Alter table options: Enhancements include setting TTL for databases and tables, modifying compaction parameters, and enabling/disabling full-text indexes on columns.
• Loki remote write: GreptimeDB now supports the Loki remote write protocol, allowing users to log data in Loki format through Grafana tools.
• Performance Optimization: Up to 10x improvement for queries fetching the latest N records by timestamp (ORDER BY timestamp DESC LIMIT N).
• Remove the option: region_engine.mito.max_background_jobs and add three new options: region_engine.mito.max_background_flushes, region_engine.mito.max_background_compactions, and region_engine.mito.max_background_purges for more detailed configuration.

Breaking changes​

• fix!: replace timeout_millis and connect_timeout_millis with Duration in DatanodeClientOptions by @WenyXu in #4867
• feat!: Divide flush and compaction job pool by @evenyag in #4871

🚀 Features​

• feat: yields empty batch after reading a range by @evenyag in #4845
• feat: update dashboard to v0.6.0 by @ZonaHex in #4861
• feat: Sort within each PartitionRange by @discord9 in #4847
• feat: Add functionality to the Opentelemetry write interface to extract fields from attr to top-level data. by @paomian in #4859
• feat: Limit CPU in runtime (#3685) by @ActivePeter in #4782
• feat: introduce the PluginOptions by @WenyXu in #4835
• feat: add json_path_match udf by @Kev1n8 in #4864
• feat: optimizer rule for windowed sort by @waynexia in #4874
• feat(index): support building inverted index for the field column on Mito by @zhongzc in #4887
• feat: add json datatype for grpc protocol by @WenyXu in #4897
• feat: Support altering table TTL by @v0y4g3r in #4848
• feat: adds the number of rows and index files size to region_statistics table by @killme2008 in #4909
• feat: implement parse_query api by @sunng87 in #4860
• feat: enhance windowed-sort optimizer rule by @waynexia in #4910
• feat: get row group time range from cached metadata by @evenyag in #4869
• feat: simple limit impl in PartSort by @waynexia in #4922
• feat: heartbeat_flush_threshold option by @fengjiachun in #4924
• feat: support to insert json data via grpc protocol by @WenyXu in #4908
• feat: add more geo functions by @sunng87 in #4888
• feat: support filter with windowed sort by @waynexia in #4960
• feat(index): support SQL to specify inverted index columns by @zhongzc in #4929
• feat(puffin): apply range reader by @zhongzc in #4928
• feat: alter fulltext options by @CookiePieWw in #4952
• feat: introduce vector type by @zhongzc in #4964
• feat: refine region state checks and handle stalled requests by @WenyXu in #4971
• feat: support alter twcs compaction options by @lyang24 in #4965
• feat: add distance functions by @zhongzc in #4987
• feat: implement statement/execution timeout session variable by @lyang24 in #4792
• feat: introduce DynamicTimeoutLayer by @WenyXu in #5006
• feat: Loki remote write by @shuiyisong in #4941
• feat: make greatest supports timestamp and datetime types by @killme2008 in #5005
• feat: update dashboard to v0.6.1 by @ZonaHex in #5017
• feat: CREATE OR REPLACE FLOW by @discord9 in #5001
• feat: also shutdown gracefully on sigterm on unix by @discord9 in #5023
• feat(vector): remove simsimd and use nalgebra instead by @zhongzc in #5027
• feat: reimplement limit in PartSort to reduce memory footprint by @waynexia in #5018
• feat(vector): add conversion between vector and string by @zhongzc in #5029
• feat: add unset table options support by @WenyXu in #5034
• feat: alter database ttl by @CookiePieWw in #5035
• feat: use cache kv manager for SchemaMetadataManager by @v0y4g3r in #5053

🐛 Bug Fixes​

• fix: fix broken import by @WenyXu in #4880
• fix: pyo3 ut by @v0y4g3r in #4894
• fix(config): update tracing section headers in example TOML files by @waynexia in #4898
• fix: set transaction variables not working in mysql protocol by @killme2008 in #4912
• fix: prune batches from memtable by time range by @evenyag in #4913
• fix: typo by @killme2008 in #4931
• fix: panic when jsonb corrupted by @CookiePieWw in #4919
• fix: data_length, index_length, table_rows in tables by @killme2008 in #4927
• fix: violations of elided_named_lifetimes by @WenyXu in #4936
• fix: database base ttl by @v0y4g3r in #4926
• fix: pprof by @discord9 in #4938
• fix: the region_stats API will return an error in instance test by @linyihai in #4951
• fix: bugs introduced by alter table options by @killme2008 in #4953
• fix: do not pick compacting/expired files by @evenyag in #4955
• fix: round euclidean result in sqlness by @v0y4g3r in #4956
• fix: column already exists by @waynexia in #4961
• fix: json_path_exists null results by @Kev1n8 in #4881
• fix(otlp): replace otlp trace attr type from string to jsonb by @paomian in #4918
• fix: alter table add column id alloc mismatch by @discord9 in #4972
• fix: physical table statistics info by @killme2008 in #4975
• fix: run install.sh error by @zyy17 in #4989
• fix: obsolete wal entries while opening a migrated region by @WenyXu in #4993
• fix: ensure Create Or Replace and If Not Exist cannot coexist in create view by @lyang24 in #5003
• fix: correct unset_maintenance_mode behavior by @WenyXu in #5009
• fix: distinct respect in range by @discord9 in #5015
• fix: inverted index constraint to be case-insensitive by @zhongzc in #5020
• fix: android build failed due to simsimd by @zhongzc in #5019
• fix: prune memtable/files range independently in each partition by @evenyag in #4998
• fix: find latest window by @v0y4g3r in #5037
• fix: prepare param mismatch by @CookiePieWw in #5025
• fix: prevent metadata region from inheriting database ttl by @waynexia in #5044
• fix(metric-engine): set ttl also on opening metadata regions by @v0y4g3r in #5051
• fix: pass series row selector to file range reader by @evenyag in #5054

🚜 Refactor​

• refactor: json conversion by @CookiePieWw in #4893
• refactor: make use of the "pre_execute" in sql execution interceptor by @MichaelScofield in #4875
• refactor: simplify WeightedChoose by @WenyXu in #4916
• refactor: refactor alter parser by @CookiePieWw in #4933
• refactor: pass LogicalPlan to promql execution interceptor by @MichaelScofield in #4937
• refactor: consolidate DatanodeClientOptions by @linyihai in #4966
• refactor: support distinct JSON format and improve type conversions by @WenyXu in #4979
• refactor(mito): tidy memtable stats by @v0y4g3r in #4982
• refactor: use UNSET instead of enable by @CookiePieWw in #4983
• refactor(grafana): update cluster dashboard by @zyy17 in #4980
• refactor: Avoid wrapping Option for CacheManagerRef by @linyihai in #4996
• refactor: introduce MaintenanceModeManager by @WenyXu in #4994
• refactor: split up different stmts by @CookiePieWw in #4997
• refactor: unify mysql execute through cli and protocol by @CookiePieWw in #5038
• refactor: expose configs for http clients used in object store by @MichaelScofield in #5041

🧪 Testing​

• test: add fuzz test for metric region migration by @WenyXu in #4862
• test: more sqlness tests for flow by @discord9 in #4988
• test: subquery test migrated from duckdb by @CookiePieWw in #4985
• test: reduce round precision to avoid platform diff by @zhongzc in #5013

⚙️ Miscellaneous Tasks​

• chore: bump greptime-meter by @v0y4g3r in #4858
• chore: make pusher log easy to understand by @fengjiachun in #4841
• chore: better column schema check for flow by @discord9 in #4855
• chore: udapte Rust toolchain to 2024-10-19 by @v0y4g3r in #4857
• chore: add schema urls to otlp logs by @shuiyisong in #4876
• chore: graceful exit on bind fail by @discord9 in #4882
• chore: remove struct size assertion by @v0y4g3r in #4885
• ci: install numpy in CI by @discord9 in #4895
• chore: update proto depend by @discord9 in #4899
• chore: provide more info in check batch message by @evenyag in #4906
• chore: short desc markdown about change log level by @discord9 in #4921
• chore: update default cache size to 1Gib by @killme2008 in #4923
• chore: minor refactor for weighted choose by @fengjiachun in #4917
• chore: fix typos in change log level doc by @WenyXu in #4948
• chore: add json path for pipeline by @paomian in #4925
• chore: paginated query region stats by @fengjiachun in #4942
• chore: update cluster dashboard by @zyy17 in #4995
• chore: update greptime-proto to e1070a by @discord9 in #4992
• chore: rename change to modify by @CookiePieWw in #5000
• chore(cli): set default timeout for cli commands by @WenyXu in #5021
• chore: bump version to 0.10.0 by @zhongzc in #5040
• chore: bump version to v0.10.1 by @zhongzc in #5048
• chore: bump version to v0.10.2 by @zhongzc in #5056

Build​

• build(deps): switch to upstream jsonb by @CookiePieWw in #4986

New Contributors​

• @linyihai made their first contribution in #4996
• @ActivePeter made their first contribution in #4782

All Contributors​

We would like to thank the following contributors from the GreptimeDB community:

@ActivePeter, @CookiePieWw, @Kev1n8, @MichaelScofield, @WenyXu, @ZonaHex, @discord9, @evenyag, @fengjiachun, @killme2008, @linyihai, @lyang24, @paomian, @shuiyisong, @sunng87, @v0y4g3r, @waynexia, @zhongzc, @zyy17

Release date: November 25, 2024

👍 Highlights​

• Vector type: GreptimeDB supports vector data types to optimize edge scenarios, such as IoT in vehicles, enabling efficient storage and computation of vector data for real-time perception and AI applications in smart driving.
• Free index: Inverted indexes are no longer tied to primary keys, allowing users to create them on any column, enhancing query flexibility and efficiency across different scenarios.
• Alter table options: Enhancements include setting TTL for databases and tables, modifying compaction parameters, and enabling/disabling full-text indexes on columns.
• Loki remote write: GreptimeDB now supports the Loki remote write protocol, allowing users to log data in Loki format through Grafana tools.
• Performance Optimization: Up to 10x improvement for queries fetching the latest N records by timestamp (ORDER BY timestamp DESC LIMIT N).
• Remove the option: region_engine.mito.max_background_jobs and add three new options: region_engine.mito.max_background_flushes, region_engine.mito.max_background_compactions, and region_engine.mito.max_background_purges for more detailed configuration.

Breaking changes​

• fix!: replace timeout_millis and connect_timeout_millis with Duration in DatanodeClientOptions by @WenyXu in #4867
• feat!: Divide flush and compaction job pool by @evenyag in #4871

🚀 Features​

• feat: yields empty batch after reading a range by @evenyag in #4845
• feat: update dashboard to v0.6.0 by @ZonaHex in #4861
• feat: Sort within each PartitionRange by @discord9 in #4847
• feat: Add functionality to the Opentelemetry write interface to extract fields from attr to top-level data. by @paomian in #4859
• feat: Limit CPU in runtime (#3685) by @ActivePeter in #4782
• feat: introduce the PluginOptions by @WenyXu in #4835
• feat: add json_path_match udf by @Kev1n8 in #4864
• feat: optimizer rule for windowed sort by @waynexia in #4874
• feat(index): support building inverted index for the field column on Mito by @zhongzc in #4887
• feat: add json datatype for grpc protocol by @WenyXu in #4897
• feat: Support altering table TTL by @v0y4g3r in #4848
• feat: adds the number of rows and index files size to region_statistics table by @killme2008 in #4909
• feat: implement parse_query api by @sunng87 in #4860
• feat: enhance windowed-sort optimizer rule by @waynexia in #4910
• feat: get row group time range from cached metadata by @evenyag in #4869
• feat: simple limit impl in PartSort by @waynexia in #4922
• feat: heartbeat_flush_threshold option by @fengjiachun in #4924
• feat: support to insert json data via grpc protocol by @WenyXu in #4908
• feat: add more geo functions by @sunng87 in #4888
• feat: support filter with windowed sort by @waynexia in #4960
• feat(index): support SQL to specify inverted index columns by @zhongzc in #4929
• feat(puffin): apply range reader by @zhongzc in #4928
• feat: alter fulltext options by @CookiePieWw in #4952
• feat: introduce vector type by @zhongzc in #4964
• feat: refine region state checks and handle stalled requests by @WenyXu in #4971
• feat: support alter twcs compaction options by @lyang24 in #4965
• feat: add distance functions by @zhongzc in #4987
• feat: implement statement/execution timeout session variable by @lyang24 in #4792
• feat: introduce DynamicTimeoutLayer by @WenyXu in #5006
• feat: Loki remote write by @shuiyisong in #4941
• feat: make greatest supports timestamp and datetime types by @killme2008 in #5005
• feat: update dashboard to v0.6.1 by @ZonaHex in #5017
• feat: CREATE OR REPLACE FLOW by @discord9 in #5001
• feat: also shutdown gracefully on sigterm on unix by @discord9 in #5023
• feat(vector): remove simsimd and use nalgebra instead by @zhongzc in #5027
• feat: reimplement limit in PartSort to reduce memory footprint by @waynexia in #5018
• feat(vector): add conversion between vector and string by @zhongzc in #5029
• feat: add unset table options support by @WenyXu in #5034
• feat: alter database ttl by @CookiePieWw in #5035

🐛 Bug Fixes​

• fix: fix broken import by @WenyXu in #4880
• fix: pyo3 ut by @v0y4g3r in #4894
• fix(config): update tracing section headers in example TOML files by @waynexia in #4898
• fix: set transaction variables not working in mysql protocol by @killme2008 in #4912
• fix: prune batches from memtable by time range by @evenyag in #4913
• fix: typo by @killme2008 in #4931
• fix: panic when jsonb corrupted by @CookiePieWw in #4919
• fix: data_length, index_length, table_rows in tables by @killme2008 in #4927
• fix: violations of elided_named_lifetimes by @WenyXu in #4936
• fix: database base ttl by @v0y4g3r in #4926
• fix: pprof by @discord9 in #4938
• fix: the region_stats API will return an error in instance test by @linyihai in #4951
• fix: bugs introduced by alter table options by @killme2008 in #4953
• fix: do not pick compacting/expired files by @evenyag in #4955
• fix: round euclidean result in sqlness by @v0y4g3r in #4956
• fix: column already exists by @waynexia in #4961
• fix: json_path_exists null results by @Kev1n8 in #4881
• fix(otlp): replace otlp trace attr type from string to jsonb by @paomian in #4918
• fix: alter table add column id alloc mismatch by @discord9 in #4972
• fix: physical table statistics info by @killme2008 in #4975
• fix: run install.sh error by @zyy17 in #4989
• fix: obsolete wal entries while opening a migrated region by @WenyXu in #4993
• fix: ensure Create Or Replace and If Not Exist cannot coexist in create view by @lyang24 in #5003
• fix: correct unset_maintenance_mode behavior by @WenyXu in #5009
• fix: distinct respect in range by @discord9 in #5015
• fix: inverted index constraint to be case-insensitive by @zhongzc in #5020
• fix: android build failed due to simsimd by @zhongzc in #5019
• fix: prune memtable/files range independently in each partition by @evenyag in #4998
• fix: find latest window by @v0y4g3r in #5037
• fix: prepare param mismatch by @CookiePieWw in #5025
• fix: prevent metadata region from inheriting database ttl by @waynexia in #5044

🚜 Refactor​

• refactor: json conversion by @CookiePieWw in #4893
• refactor: make use of the "pre_execute" in sql execution interceptor by @MichaelScofield in #4875
• refactor: simplify WeightedChoose by @WenyXu in #4916
• refactor: refactor alter parser by @CookiePieWw in #4933
• refactor: pass LogicalPlan to promql execution interceptor by @MichaelScofield in #4937
• refactor: consolidate DatanodeClientOptions by @linyihai in #4966
• refactor: support distinct JSON format and improve type conversions by @WenyXu in #4979
• refactor(mito): tidy memtable stats by @v0y4g3r in #4982
• refactor: use UNSET instead of enable by @CookiePieWw in #4983
• refactor(grafana): update cluster dashboard by @zyy17 in #4980
• refactor: Avoid wrapping Option for CacheManagerRef by @linyihai in #4996
• refactor: introduce MaintenanceModeManager by @WenyXu in #4994
• refactor: split up different stmts by @CookiePieWw in #4997
• refactor: unify mysql execute through cli and protocol by @CookiePieWw in #5038

📚 Documentation​

• docs: change cpu/mem panel to time-series by @evenyag in #4844
• docs: add TOC to readme by @killme2008 in #4949

🧪 Testing​

• test: add fuzz test for metric region migration by @WenyXu in #4862
• test: more sqlness tests for flow by @discord9 in #4988
• test: subquery test migrated from duckdb by @CookiePieWw in #4985
• test: reduce round precision to avoid platform diff by @zhongzc in #5013

⚙️ Miscellaneous Tasks​

• chore: bump greptime-meter by @v0y4g3r in #4858
• chore: make pusher log easy to understand by @fengjiachun in #4841
• chore: better column schema check for flow by @discord9 in #4855
• chore: udapte Rust toolchain to 2024-10-19 by @v0y4g3r in #4857
• chore: add schema urls to otlp logs by @shuiyisong in #4876
• chore: graceful exit on bind fail by @discord9 in #4882
• chore: remove struct size assertion by @v0y4g3r in #4885
• ci: install numpy in CI by @discord9 in #4895
• chore: update proto depend by @discord9 in #4899
• chore: provide more info in check batch message by @evenyag in #4906
• chore: short desc markdown about change log level by @discord9 in #4921
• chore: update default cache size to 1Gib by @killme2008 in #4923
• chore: minor refactor for weighted choose by @fengjiachun in #4917
• chore: fix typos in change log level doc by @WenyXu in #4948
• chore: add json path for pipeline by @paomian in #4925
• chore: paginated query region stats by @fengjiachun in #4942
• chore: update cluster dashboard by @zyy17 in #4995
• chore: update greptime-proto to e1070a by @discord9 in #4992
• chore: rename change to modify by @CookiePieWw in #5000
• chore(cli): set default timeout for cli commands by @WenyXu in #5021
• chore: bump version to 0.10.0 by @zhongzc in #5040
• chore: bump version to v0.10.1 by @zhongzc in #5048

Build​

• build(deps): switch to upstream jsonb by @CookiePieWw in #4986

New Contributors​

• @linyihai made their first contribution in #4996
• @ActivePeter made their first contribution in #4782

All Contributors​

We would like to thank the following contributors from the GreptimeDB community:

@ActivePeter, @CookiePieWw, @Kev1n8, @MichaelScofield, @WenyXu, @ZonaHex, @discord9, @evenyag, @fengjiachun, @killme2008, @linyihai, @lyang24, @paomian, @shuiyisong, @sunng87, @v0y4g3r, @waynexia, @zhongzc, @zyy17

Release date: November 22, 2024

👍 Highlights​

• Vector type: GreptimeDB supports vector data types to optimize edge scenarios, such as IoT in vehicles, enabling efficient storage and computation of vector data for real-time perception and AI applications in smart driving.
• Free index: Inverted indexes are no longer tied to primary keys, allowing users to create them on any column, enhancing query flexibility and efficiency across different scenarios.
• Alter table options: Enhancements include setting TTL for databases and tables, modifying compaction parameters, and enabling/disabling full-text indexes on columns.
• Loki remote write: GreptimeDB now supports the Loki remote write protocol, allowing users to log data in Loki format through Grafana tools.
• Performance Optimization: Up to 10x improvement for queries fetching the latest N records by timestamp (ORDER BY timestamp DESC LIMIT N).

Breaking changes​

• fix!: replace timeout_millis and connect_timeout_millis with Duration in DatanodeClientOptions by @WenyXu in #4867
• feat!: Divide flush and compaction job pool by @evenyag in #4871

🚀 Features​

• feat: yields empty batch after reading a range by @evenyag in #4845
• feat: update dashboard to v0.6.0 by @ZonaHex in #4861
• feat: Sort within each PartitionRange by @discord9 in #4847
• feat: Add functionality to the Opentelemetry write interface to extract fields from attr to top-level data. by @paomian in #4859
• feat: Limit CPU in runtime (#3685) by @ActivePeter in #4782
• feat: introduce the PluginOptions by @WenyXu in #4835
• feat: add json_path_match udf by @Kev1n8 in #4864
• feat: optimizer rule for windowed sort by @waynexia in #4874
• feat(index): support building inverted index for the field column on Mito by @zhongzc in #4887
• feat: add json datatype for grpc protocol by @WenyXu in #4897
• feat: Support altering table TTL by @v0y4g3r in #4848
• feat: adds the number of rows and index files size to region_statistics table by @killme2008 in #4909
• feat: implement parse_query api by @sunng87 in #4860
• feat: enhance windowed-sort optimizer rule by @waynexia in #4910
• feat: get row group time range from cached metadata by @evenyag in #4869
• feat: simple limit impl in PartSort by @waynexia in #4922
• feat: heartbeat_flush_threshold option by @fengjiachun in #4924
• feat: support to insert json data via grpc protocol by @WenyXu in #4908
• feat: add more geo functions by @sunng87 in #4888
• feat: support filter with windowed sort by @waynexia in #4960
• feat(index): support SQL to specify inverted index columns by @zhongzc in #4929
• feat(puffin): apply range reader by @zhongzc in #4928
• feat: alter fulltext options by @CookiePieWw in #4952
• feat: introduce vector type by @zhongzc in #4964
• feat: refine region state checks and handle stalled requests by @WenyXu in #4971
• feat: support alter twcs compaction options by @lyang24 in #4965
• feat: add distance functions by @zhongzc in #4987
• feat: implement statement/execution timeout session variable by @lyang24 in #4792
• feat: introduce DynamicTimeoutLayer by @WenyXu in #5006
• feat: Loki remote write by @shuiyisong in #4941
• feat: make greatest supports timestamp and datetime types by @killme2008 in #5005
• feat: update dashboard to v0.6.1 by @ZonaHex in #5017
• feat: CREATE OR REPLACE FLOW by @discord9 in #5001
• feat: also shutdown gracefully on sigterm on unix by @discord9 in #5023
• feat(vector): remove simsimd and use nalgebra instead by @zhongzc in #5027
• feat: reimplement limit in PartSort to reduce memory footprint by @waynexia in #5018
• feat(vector): add conversion between vector and string by @zhongzc in #5029
• feat: add unset table options support by @WenyXu in #5034
• feat: alter database ttl by @CookiePieWw in #5035

🐛 Bug Fixes​

• fix: fix broken import by @WenyXu in #4880
• fix: pyo3 ut by @v0y4g3r in #4894
• fix(config): update tracing section headers in example TOML files by @waynexia in #4898
• fix: set transaction variables not working in mysql protocol by @killme2008 in #4912
• fix: prune batches from memtable by time range by @evenyag in #4913
• fix: typo by @killme2008 in #4931
• fix: panic when jsonb corrupted by @CookiePieWw in #4919
• fix: data_length, index_length, table_rows in tables by @killme2008 in #4927
• fix: violations of elided_named_lifetimes by @WenyXu in #4936
• fix: database base ttl by @v0y4g3r in #4926
• fix: pprof by @discord9 in #4938
• fix: the region_stats API will return an error in instance test by @linyihai in #4951
• fix: bugs introduced by alter table options by @killme2008 in #4953
• fix: do not pick compacting/expired files by @evenyag in #4955
• fix: round euclidean result in sqlness by @v0y4g3r in #4956
• fix: column already exists by @waynexia in #4961
• fix: json_path_exists null results by @Kev1n8 in #4881
• fix(otlp): replace otlp trace attr type from string to jsonb by @paomian in #4918
• fix: alter table add column id alloc mismatch by @discord9 in #4972
• fix: physical table statistics info by @killme2008 in #4975
• fix: run install.sh error by @zyy17 in #4989
• fix: obsolete wal entries while opening a migrated region by @WenyXu in #4993
• fix: ensure Create Or Replace and If Not Exist cannot coexist in create view by @lyang24 in #5003
• fix: correct unset_maintenance_mode behavior by @WenyXu in #5009
• fix: distinct respect in range by @discord9 in #5015
• fix: inverted index constraint to be case-insensitive by @zhongzc in #5020
• fix: android build failed due to simsimd by @zhongzc in #5019
• fix: prune memtable/files range independently in each partition by @evenyag in #4998
• fix: find latest window by @v0y4g3r in #5037
• fix: prepare param mismatch by @CookiePieWw in #5025

🚜 Refactor​

• refactor: json conversion by @CookiePieWw in #4893
• refactor: make use of the "pre_execute" in sql execution interceptor by @MichaelScofield in #4875
• refactor: simplify WeightedChoose by @WenyXu in #4916
• refactor: refactor alter parser by @CookiePieWw in #4933
• refactor: pass LogicalPlan to promql execution interceptor by @MichaelScofield in #4937
• refactor: consolidate DatanodeClientOptions by @linyihai in #4966
• refactor: support distinct JSON format and improve type conversions by @WenyXu in #4979
• refactor(mito): tidy memtable stats by @v0y4g3r in #4982
• refactor: use UNSET instead of enable by @CookiePieWw in #4983
• refactor(grafana): update cluster dashboard by @zyy17 in #4980
• refactor: Avoid wrapping Option for CacheManagerRef by @linyihai in #4996
• refactor: introduce MaintenanceModeManager by @WenyXu in #4994
• refactor: split up different stmts by @CookiePieWw in #4997
• refactor: unify mysql execute through cli and protocol by @CookiePieWw in #5038

📚 Documentation​

• docs: change cpu/mem panel to time-series by @evenyag in #4844
• docs: add TOC to readme by @killme2008 in #4949

🧪 Testing​

• test: add fuzz test for metric region migration by @WenyXu in #4862
• test: more sqlness tests for flow by @discord9 in #4988
• test: subquery test migrated from duckdb by @CookiePieWw in #4985
• test: reduce round precision to avoid platform diff by @zhongzc in #5013

⚙️ Miscellaneous Tasks​

• chore: bump greptime-meter by @v0y4g3r in #4858
• chore: make pusher log easy to understand by @fengjiachun in #4841
• chore: better column schema check for flow by @discord9 in #4855
• chore: udapte Rust toolchain to 2024-10-19 by @v0y4g3r in #4857
• chore: add schema urls to otlp logs by @shuiyisong in #4876
• chore: graceful exit on bind fail by @discord9 in #4882
• chore: remove struct size assertion by @v0y4g3r in #4885
• ci: install numpy in CI by @discord9 in #4895
• chore: update proto depend by @discord9 in #4899
• chore: provide more info in check batch message by @evenyag in #4906
• chore: short desc markdown about change log level by @discord9 in #4921
• chore: update default cache size to 1Gib by @killme2008 in #4923
• chore: minor refactor for weighted choose by @fengjiachun in #4917
• chore: fix typos in change log level doc by @WenyXu in #4948
• chore: add json path for pipeline by @paomian in #4925
• chore: paginated query region stats by @fengjiachun in #4942
• chore: update cluster dashboard by @zyy17 in #4995
• chore: update greptime-proto to e1070a by @discord9 in #4992
• chore: rename change to modify by @CookiePieWw in #5000
• chore(cli): set default timeout for cli commands by @WenyXu in #5021
• chore: bump version to 0.10.0 by @zhongzc in #5040

Build​

• build(deps): switch to upstream jsonb by @CookiePieWw in #4986

New Contributors​

• @linyihai made their first contribution in #4996
• @ActivePeter made their first contribution in #4782

All Contributors​

We would like to thank the following contributors from the GreptimeDB community:

@ActivePeter, @CookiePieWw, @Kev1n8, @MichaelScofield, @WenyXu, @ZonaHex, @discord9, @evenyag, @fengjiachun, @killme2008, @linyihai, @lyang24, @paomian, @shuiyisong, @sunng87, @v0y4g3r, @waynexia, @zhongzc, @zyy17

Release date: October 18, 2024

Highlights​

• Enhanced support for geo-related functions: #4770, #4823, #4798
• Expanded support for JSON functions: #4807
• Fixed panic issue in union all queries: #4796

Breaking changes​

• feat!: move v1/prof API to debug/prof by @evenyag in #4810
• feat!: implement interval type by multiple structs by @evenyag in #4772

🚀 Features​

• feat: protect datanode with concurrency limit. by @lyang24 in #4699
• feat: add region_statistics table by @WenyXu in #4771
• feat(mito): limit compaction output file size by @v0y4g3r in #4754
• feat: add a new status code for "external" errors by @MichaelScofield in #4775
• feat: unordered scanner scans data by time ranges by @evenyag in #4757
• feat: add StatementStatistics for slow query logging implementation by @zyy17 in #4719
• feat: support to reject write after flushing by @WenyXu in #4759
• feat: add add_handler_after, add_handler_before, replace_handler by @WenyXu in #4788
• feat: customize channel information for sqlness tests by @J0HN50N133 in #4729
• feat: set max log files to 720 by default, info log only by @Kev1n8 in #4787
• feat: introduce HeartbeatHandlerGroupBuilderCustomizer by @WenyXu in #4803
• feat: add function to aggregate path into a geojson path by @sunng87 in #4798
• feat: add json format output for http interface by @Kev1n8 in #4797
• feat: Merge sort Logical plan by @discord9 in #4768
• feat: json_path_exists udf by @CookiePieWw in #4807
• feat: expose RegionMigrationManagerRef by @WenyXu in #4812
• feat: information extension by @fengjiachun in #4811
• feat: introduce LeadershipChangeNotifier and LeadershipChangeListener by @WenyXu in #4817
• feat: add more h3 functions by @sunng87 in #4770
• feat: remove the distributed lock by @fengjiachun in #4825
• feat: enable prof features by default by @evenyag in #4815
• feat: cache logical region's metadata by @waynexia in #4827
• feat: defer HeartbeatHandlerGroup construction and enhance LeadershipChangeNotifier by @WenyXu in #4826
• feat: add some s2 geo functions by @sunng87 in #4823
• feat: introduce default customizers by @WenyXu in #4831

🐛 Bug Fixes​

• fix: use information_schema returns Unknown database by @J0HN50N133 in #4774
• fix: dead loop on detecting postgres ssl handshake by @sunng87 in #4778
• fix: update pgwire for potential issue with connection establish by @sunng87 in #4783
• fix(sqlness): sqlness isolation by @v0y4g3r in #4780
• fix: ts conversion during transform phase by @shuiyisong in #4790
• fix: Panic in UNION ALL queries by @v0y4g3r in #4796
• fix: error in admin function is not formatted properly by @waynexia in #4820
• fix: correct table name formatting by @WenyXu in #4819
• fix: case sensitive for field matcher by @waynexia in #4822

🚜 Refactor​

• refactor: Change the error type in the pipeline crate from String to Error by @paomian in #4763
• refactor: introduce HeartbeatHandlerGroupBuilder by @WenyXu in #4785
• refactor: change sqlness ports to avoid conflict with local instance by @sunng87 in #4794
• refactor: replace info logs with debug logs in region server by @waynexia in #4829

📚 Documentation​

• docs: add TM to logos by @evenyag in #4789

⚙️ Miscellaneous Tasks​

• chore: make sure aws-lc-sys wouldn't be built by @discord9 in #4767
• chore: replace anymap with anymap2 by @v0y4g3r in #4781
• chore: add json write by @paomian in #4744
• chore: bump promql-parser to v0.4.1 and use to_string() for EvalStmt by @zyy17 in #4832
• chore: bump version v0.9.4 by @discord9 in #4833

New Contributors​

• @Kev1n8 made their first contribution in #4797

All Contributors​

We would like to thank the following contributors from the GreptimeDB community:

@CookiePieWw, @J0HN50N133, @Kev1n8, @MichaelScofield, @WenyXu, @discord9, @evenyag, @fengjiachun, @lyang24, @paomian, @shuiyisong, @sunng87, @v0y4g3r, @waynexia, @zyy17

Release date: October 15, 2024

Highlights​

• Enhanced support for geo-related functions: #4770, #4823, #4798
• Expanded support for JSON functions: #4807
• Fixed panic issue in union all queries: #4796

Breaking changes​

• feat!: move v1/prof API to debug/prof by @evenyag in #4810
• feat!: implement interval type by multiple structs by @evenyag in #4772

🚀 Features​

• feat: protect datanode with concurrency limit. by @lyang24 in #4699
• feat: add region_statistics table by @WenyXu in #4771
• feat(mito): limit compaction output file size by @v0y4g3r in #4754
• feat: add a new status code for "external" errors by @MichaelScofield in #4775
• feat: unordered scanner scans data by time ranges by @evenyag in #4757
• feat: add StatementStatistics for slow query logging implementation by @zyy17 in #4719
• feat: support to reject write after flushing by @WenyXu in #4759
• feat: add add_handler_after, add_handler_before, replace_handler by @WenyXu in #4788
• feat: customize channel information for sqlness tests by @J0HN50N133 in #4729
• feat: set max log files to 720 by default, info log only by @Kev1n8 in #4787
• feat: introduce HeartbeatHandlerGroupBuilderCustomizer by @WenyXu in #4803
• feat: add function to aggregate path into a geojson path by @sunng87 in #4798
• feat: add json format output for http interface by @Kev1n8 in #4797
• feat: Merge sort Logical plan by @discord9 in #4768
• feat: json_path_exists udf by @CookiePieWw in #4807
• feat: expose RegionMigrationManagerRef by @WenyXu in #4812
• feat: information extension by @fengjiachun in #4811
• feat: introduce LeadershipChangeNotifier and LeadershipChangeListener by @WenyXu in #4817
• feat: add more h3 functions by @sunng87 in #4770
• feat: remove the distributed lock by @fengjiachun in #4825
• feat: enable prof features by default by @evenyag in #4815
• feat: cache logical region's metadata by @waynexia in #4827
• feat: defer HeartbeatHandlerGroup construction and enhance LeadershipChangeNotifier by @WenyXu in #4826
• feat: add some s2 geo functions by @sunng87 in #4823
• feat: introduce default customizers by @WenyXu in #4831

🐛 Bug Fixes​

• fix: use information_schema returns Unknown database by @J0HN50N133 in #4774
• fix: dead loop on detecting postgres ssl handshake by @sunng87 in #4778
• fix: update pgwire for potential issue with connection establish by @sunng87 in #4783
• fix(sqlness): sqlness isolation by @v0y4g3r in #4780
• fix: ts conversion during transform phase by @shuiyisong in #4790
• fix: Panic in UNION ALL queries by @v0y4g3r in #4796
• fix: error in admin function is not formatted properly by @waynexia in #4820
• fix: correct table name formatting by @WenyXu in #4819
• fix: case sensitive for field matcher by @waynexia in #4822

🚜 Refactor​

• refactor: Change the error type in the pipeline crate from String to Error by @paomian in #4763
• refactor: introduce HeartbeatHandlerGroupBuilder by @WenyXu in #4785
• refactor: change sqlness ports to avoid conflict with local instance by @sunng87 in #4794
• refactor: replace info logs with debug logs in region server by @waynexia in #4829

📚 Documentation​

• docs: add TM to logos by @evenyag in #4789

⚙️ Miscellaneous Tasks​

• chore: make sure aws-lc-sys wouldn't be built by @discord9 in #4767
• chore: replace anymap with anymap2 by @v0y4g3r in #4781
• chore: add json write by @paomian in #4744
• chore: bump promql-parser to v0.4.1 and use to_string() for EvalStmt by @zyy17 in #4832
• chore: bump version v0.9.4 by @discord9 in #4833

New Contributors​

• @Kev1n8 made their first contribution in #4797

All Contributors​

We would like to thank the following contributors from the GreptimeDB community:

@CookiePieWw, @J0HN50N133, @Kev1n8, @MichaelScofield, @WenyXu, @discord9, @evenyag, @fengjiachun, @lyang24, @paomian, @shuiyisong, @sunng87, @v0y4g3r, @waynexia, @zyy17

Release date: September 06, 2024

Highlights​

• fixes the following issues
  • last_value function may return empty results due to cache #4652
  • query timestamp column in append mode misses some data #4669
  • crash while query tables with last_non_null merge mode #4687

Breaking changes​

• feat!: impl admin command by @killme2008 in #4600

🚀 Features​

• feat: Implement the Buf to avoid extra memory allocation by @ozewr in #4585
• feat: remove sql in error desc by @waynexia in #4589
• feat: collect filters metrics for scanners by @evenyag in #4591
• feat: refactoring LruCacheLayer with list_with_metakey and concurrent_stat_in_list by @ozewr in #4596
• feat: add postgres response for transaction related statements by @sunng87 in #4562
• feat: allow skipping topic creation by @WenyXu in #4616
• feat: remove some redundant clone/conversion on constructing MergeScan stream by @waynexia in #4632
• feat: replay WAL entries respect index by @WenyXu in #4565
• feat(flow): use DataFusion's optimizer by @discord9 in #4489
• feat: copy database ignores view and temporary tables by @fengjiachun in #4640
• feat: pre-download the ingested sst by @MichaelScofield in #4636
• feat: import cli tool by @fengjiachun in #4639
• feat: show create database by @fengjiachun in #4642
• feat: initialize partition range from ScanInput by @waynexia in #4635
• feat: remove the requirement that partition column must be PK by @waynexia in #4647
• feat: remove files from the write cache in purger by @evenyag in #4655
• feat: grpc writing supports TTL hint by @killme2008 in #4651
• feat: export import database by @fengjiachun in #4654
• feat: supports name in object storage config by @killme2008 in #4630
• feat: change log level dynamically by @discord9 in #4653
• feat: schedule compaction when adding sst files by editing region by @MichaelScofield in #4648
• feat: add geohash and h3 as built-in functions by @sunng87 in #4656
• feat: add more spans to mito engine by @lyang24 in #4643
• feat: invalidate cache via invalidator on region migration by @fengjiachun in #4682

🐛 Bug Fixes​

• fix: pipeline prepare loop break detects a conditional error by @paomian in #4593
• fix: incremental compilation always compile the common-version crate by @fengys1996 in #4605
• fix: failed to get github sha by @daviderli614 in #4620
• fix: change toolchain file name by @daviderli614 in #4621
• fix: failed to get version by @daviderli614 in #4622
• fix: fallback to window size in manifest by @evenyag in #4629
• fix: update properties on updating partitions by @waynexia in #4627
• fix: set selector_result_cache_size in unit test by @WenyXu in #4631
• fix: config api and export metrics default database by @killme2008 in #4633
• fix: set selector_result_cache_size in unit test again by @WenyXu in #4641
• fix(mito): avoid caching empty batches in row group by @v0y4g3r in #4652
• fix: correct otlp endpoint formatting by @WenyXu in #4646
• fix: expose missing options for initializing regions by @WenyXu in #4660
• fix: use 'target' for 'actions-rust-lang/setup-rust-toolchain' to fix cross build failed by @zyy17 in #4661
• fix: use number of partitions as parallelism in region scanner by @waynexia in #4669
• fix: ref to auth err by @shuiyisong in #4681
• fix: last non null iter not init by @evenyag in #4687

🚜 Refactor​

• refactor(mito2): reduce duplicate IndexOutput struct by @leaf-potato in #4592
• refactor: skip checking the existence of the SST files by @MichaelScofield in #4602
• refactor: add fallback_to_local region option by @zyy17 in #4578
• refactor: add app in greptime_app_version metric by @zyy17 in #4626
• refactor: reduce a object store "stat" call by @MichaelScofield in #4645
• refactor: remove unused error variants by @waynexia in #4666
• refactor: make init_global_logging() clean and add log_format by @zyy17 in #4657

📚 Documentation​

• docs: log benchmark by @shuiyisong in #4597
• docs: move v0.9.1 benchmark report to tsbs dir by @evenyag in #4598
• docs: add example configs introduced by pg_kvbackend by @lyang24 in #4573

⚡ Performance​

• perf: set simple filter on primary key columns to exact filter by @waynexia in #4564
• perf: optimize series divide algo by @waynexia in #4603
• perf: accelerate scatter query by @waynexia in #4607
• perf(flow): Map&Reduce Operator use batch to reduce alloc by @discord9 in #4567

⚙️ Miscellaneous Tasks​

• chore: keep symbol table in nightly profile by @fengys1996 in #4588
• chore: bump tikv-jemalloc* to "0.6" by @fengys1996 in #4590
• chore: disable ttl for write cache by default by @evenyag in #4595
• chore: bump opendal version to 0.49 by @WenyXu in #4587
• chore: upgrade toolchain to nightly-2024-08-07 by @WenyXu in #4549
• chore: add commerial support section by @sunng87 in #4601
• chore: bump rust toolchain to 2024-06-06 by @waynexia in #4606
• chore: bump rskafka to 75535b by @WenyXu in #4608
• chore: add stats feature for jemalloc-ctl by @discord9 in #4610
• chore: optimize common_version build by @fengys1996 in #4611
• ci: add push dev-build images to aws ecr by @daviderli614 in #4618
• chore: setting docker authentication in dev-build image by @daviderli614 in #4623
• ci: improve toolchain resolution in ci by @sunng87 in #4614
• ci: add check-builder-rust-version job in release and change release-dev-builder-images trigger condition by @zyy17 in #4615
• chore: modify grafana config to accord with version 9 by @shuiyisong in #4634
• chore: adding heartbeat sent/recv counts in greptimedb nodes by @lyang24 in #4624
• ci: disable macos integration test and some minor refactoring by @zyy17 in #4658
• chore: refactor metadata key value trait by @fengjiachun in #4664
• chore(ci): set etcd resources limits by @WenyXu in #4665
• chore(dockerfile): remove mysql and postgresql clients in greptimedb image by @daviderli614 in #4685
• chore: bump version v0.9.3 by @v0y4g3r in #4684

Build​

• build(deps): bump backon to 1.0 by @waynexia in #4625
• build: add mysql and postgresql clients to greptimedb image by @MichaelScofield in #4677

New Contributors​

• @ozewr made their first contribution in #4596

All Contributors​

We would like to thank the following contributors from the GreptimeDB community:

@MichaelScofield, @WenyXu, @daviderli614, @discord9, @evenyag, @fengjiachun, @fengys1996, @killme2008, @leaf-potato, @lyang24, @ozewr, @paomian, @shuiyisong, @sunng87, @v0y4g3r, @waynexia, @zyy17

Release date: August 19, 2024

👍 Highlights​

• #4545: Improve the performance of count(*) for append-only tables.
• #4552: Allow more than one segment for full-text index to handle a large number of rows.

🚀 Features​

• feat: introduce new kafka topic consumer respecting WAL index by @WenyXu in #4424
• feat: flow recreate on reboot by @discord9 in #4509
• feat: change the default selector to RoundRobin by @fengjiachun in #4528
• feat: add SASL and TLS config for Kafka client by @WenyXu in #4536
• feat(log_store): introduce the IndexCollector by @WenyXu in #4461
• feat(flow): add some metrics by @discord9 in #4539
• feat(flow): add eval_batch for ScalarExpr by @discord9 in #4551
• feat: implement postgres kvbackend by @lyang24 in #4421
• feat: add GcsConfig credential field by @daviderli614 in #4568
• feat: able to handle concurrent region edit requests by @MichaelScofield in #4569
• feat(log_store): introduce the CollectionTask by @WenyXu in #4530

🐛 Bug Fixes​

• fix(tql): avoid unwrap on parsing tql query by @waynexia in #4502
• fix: too large shadow-rs consts by @MichaelScofield in #4506
• fix: fix incorrect result of topk with cte by @waynexia in #4523
• fix: install script by @v0y4g3r in #4527
• fix: larger stack size in debug mode by @discord9 in #4521
• fix: rollback only if dropping the metric physical table fails by @WenyXu in #4525
• fix: configuration example for selector by @sunng87 in #4532
• fix: change the type of oid in pg_namespace to u32 by @J0HN50N133 in #4541
• fix: append table stats by @v0y4g3r in #4561
• fix(fulltext-index): single segment is not sufficient for >50M rows SST by @zhongzc in #4552
• fix(common_version): short_version with empty branch by @leaf-potato in #4572
• fix(sqlness): redact all volatile text by @v0y4g3r in #4583

🚜 Refactor​

• refactor: reuse aligned ts array in range manipulate exec by @waynexia in #4535
• refactor(plugin): add SetupPlugin and StartPlugin error by @zyy17 in #4554

📚 Documentation​

• docs: add v0.9.1 bench result by @evenyag in #4511
• docs: Adds more panels to grafana dashboards by @evenyag in #4540
• docs: update grafana readme by @evenyag in #4550

⚡ Performance​

• perf: merge small byte ranges for optimized fetching by @WenyXu in #4520
• perf: count(*) for append-only tables by @v0y4g3r in #4545
• perf: Optimizing pipeline performance by @paomian in #4390

🧪 Testing​

• test: more on processors by @shuiyisong in #4493

⚙️ Miscellaneous Tasks​

• chore: bump opendal version to 0.48 by @WenyXu in #4499
• chore: bump shadow-rs version to set the path to find the correct git repo by @MichaelScofield in #4494
• chore: bump rust-postgres to 0.7.11 by @WenyXu in #4504
• chore: reduce fuzz tests in CI by @WenyXu in #4505
• chore: set default otlp_endpoint by @WenyXu in #4508
• chore: support swcs as the short name for strict window compaction by @v0y4g3r in #4517
• ci: squeeze some disk space for complex fuzz tests by @MichaelScofield in #4519
• ci: fix windows temp path by @evenyag in #4518
• chore: use configData by @WenyXu in #4522
• chore: update snafu to make clippy happy by @MichaelScofield in #4507
• ci: download the latest stable released version by default and do some small refactoring by @zyy17 in #4529
• chore: make mysql server version changeable by @MichaelScofield in #4531
• chore: upload kind logs by @WenyXu in #4544
• chore: update validator signature by @shuiyisong in #4548
• chore: Helper function to convert Vec<Value> to VectorRef by @discord9 in #4546
• chore: set topic to 3 for sqlness test by @discord9 in #4560
• chore: remove unused code by @WenyXu in #4559
• chore(log_store): remove redundant metrics by @WenyXu in #4570
• chore: bump version to v0.9.2 by @zhongzc in #4581
• chore: setup kafka before downloading binary step by @WenyXu in #4582

Build​

• build(deps): bump zerovec-derive from 0.10.2 to 0.10.3 by @dependabot[bot] in #4346
• build(deps): bump zerovec from 0.10.2 to 0.10.4 by @dependabot[bot] in #4335

All Contributors​

We would like to thank the following contributors from the GreptimeDB community:

@J0HN50N133, @MichaelScofield, @WenyXu, @daviderli614, @discord9, @evenyag, @fengjiachun, @leaf-potato, @lyang24, @paomian, @shuiyisong, @sunng87, @v0y4g3r, @waynexia, @zhongzc, @zyy17

Release date: August 02, 2024

This is a patch release, containing some important bug fixes:

• #4447: Full-text search may miss some rows when a full-text index is applied.
• #4476: Data ingestion can stall due to missing notifications.

It's highly recommended to upgrade to this version if you're using v0.9.0.

Breaking changes​

• refactor!: Remove Mode from FrontendOptions by @zyy17 in #4401

🚀 Features​

• feat: FLOWS table in information_schema&SHOW FLOWS by @discord9 in #4386
• feat: export database data by @fengjiachun in #4382
• feat(flow): flush_flow function by @discord9 in #4416
• feat: support pg_namespace, pg_class and related psql command by @J0HN50N133 in #4428
• feat: show root cause and db name on the error line by @sunng87 in #4442
• feat: track prometheus HTTP API's query latency by @waynexia in #4458
• feat: support setting time range in Copy From statement by @realtaobo in #4405
• feat: remove dedicated runtime for grpc, mysql and pg protocols by @waynexia in #4436
• feat: default export catalog name by @fengjiachun in #4464
• feat: track channels with query context and w/rcu by @sunng87 in #4448
• feat: hint options for gRPC insert by @fengjiachun in #4454
• feat: improve extract hints by @fengjiachun in #4479
• feat: export all schemas and data at once in export tool by @fengjiachun in #4478
• feat: time poll elapsed for RegionScan plan by @waynexia in #4482
• feat: update dashboard to v0.5.4 by @ZonaHex in #4483
• feat(compaction): add file number limits to TWCS compaction by @v0y4g3r in #4481
• feat: enlarge default page cache size by @evenyag in #4490

🐛 Bug Fixes​

• fix: ensure keep alive is completed in time by @WenyXu in #4349
• fix: user provider can't be configured by config file or environment variables by @zyy17 in #4398
• fix: dictionary key type use u32 by @evenyag in #4396
• fix(wal): handle WAL deletion on region drop by @v0y4g3r in #4400
• fix: add back AuthBackend which is required by custom auth backend by @sunng87 in #4409
• fix(fulltext-index): clean up 0-value timer by @zhongzc in #4423
• fix(metrics): RowGroupLastRowCachedReader metrics by @v0y4g3r in #4418
• fix: information_schema tables and views column value by @killme2008 in #4438
• fix: remove to_timezone function by @killme2008 in #4439
• fix: use heartbeat runtime instead of background runtime by @WenyXu in #4445
• fix(fulltext-search): prune rows in row group forget to take remainder by @zhongzc in #4447
• fix: use status code to http status mapping in error IntoResponse by @shuiyisong in #4455
• fix: missing pre_write check on prometheus remote write by @shuiyisong in #4460
• fix: check_partition uses unqualified name by @waynexia in #4452
• fix: overflow when parsing default value with negative numbers by @killme2008 in #4459
• fix: prometheus api only returns 200 by @shuiyisong in #4471
• fix: generate unique timestamp for inserting tests by @WenyXu in #4472
• fix: notify flush receiver after write buffer is released by @evenyag in #4476
• fix: avoid total size overflow by @evenyag in #4487

🚜 Refactor​

• refactor: add &mut Plugins argument in plugins setup api and remove unnecessary mut by @zyy17 in #4389
• refactor: add get_storage_path() and get_catalog_and_schema() by @zyy17 in #4397
• refactor(query): Remove PhysicalPlanner trait by @leaf-potato in #4412
• refactor: Remove PhysicalOptimizer and LogicalOptimizer trait by @leaf-potato in #4426
• refactor: add RetryInterceptor to print detailed error by @zyy17 in #4434
• refactor(servers): improve postgres error message by @leaf-potato in #4463

📚 Documentation​

• docs(common_error): format enum StatusCode docs by @leaf-potato in #4427
• docs: update readme by @killme2008 in #4430
• docs: update readme by @killme2008 in #4431
• docs: update project status by @killme2008 in #4440
• docs: tweak readme by @killme2008 in #4465
• docs(contributing): replace expired links by @leaf-potato in #4468

⚡ Performance​

• perf: reduce lock scope and improve log by @evenyag in #4453

⚙️ Miscellaneous Tasks​

• chore: support pattern as pipeline key name by @shuiyisong in #4368
• ci: disable auto review by @evenyag in #4387
• chore: add metrics for log ingestion by @shuiyisong in #4411
• chore: add a compile cfg for python in cmd package by @discord9 in #4406
• chore: update grafana dashboard to reflect recent metric changes by @waynexia in #4417
• chore: add docs for config file by @WenyXu in #4432
• chore: add dynamic cache size adjustment for InvertedIndexConfig by @v0y4g3r in #4433
• chore: temporarily disable fuzz chaos tests by @WenyXu in #4457
• ci: keep sqlness log by default by @evenyag in #4449
• chore: add more metrics about parquet and cache by @MichaelScofield in #4410
• chore(ci): bring back chaos tests by @WenyXu in #4456
• ci: make docker image args configurable from env vars by @zyy17 in #4484
• chore: bump version v0.9.1 by @v0y4g3r in #4486

New Contributors​

• @leaf-potato made their first contribution in #4468

All Contributors​

We would like to thank the following contributors from the GreptimeDB community:

@J0HN50N133, @MichaelScofield, @WenyXu, @ZonaHex, @discord9, @evenyag, @fengjiachun, @killme2008, @leaf-potato, @realtaobo, @shuiyisong, @sunng87, @v0y4g3r, @waynexia, @zhongzc, @zyy17