版本发布日志 #
这里是极限网关历史版本发布的相关说明。
1.26.1 (2024-08-13) #
Bug fix #
- 修复部分特殊场景下数据迁移异常问题
- 同步更新 Framework 修复的已知问题
1.26.0 (2024-06-07) #
Improvements #
- feat: add wildcard_domain filter
- chore: remove security filter and translog_viewer
1.25.0 (2024-04-30) #
Improvements #
- add push err and log pop timeout err
Bug fix #
- Fix date_range_precision_tuning work with filter query
1.24.0 (2024-04-15) #
Improvements #
- Refactoring http client tls config
- Write field routing to bulk metadata when field _routing exists in scrolled doc
Bug fix #
- Fix(reshuffle filter): make sure queue config always have label
type
- Fix rotate config usage
1.23.0 (2024-03-01) #
Bug fix #
- Fix consumer offsets were not reset after deleting instance queues.
1.22.0 (2024-01-26) #
Bug fix #
- Fix update default EOF max retry times
Improvements #
- Unified version number with INFINI Console
- Optimize bytes pool
- Limit inflight request size
- Add limit to entry
1.21.0 (2023-12-29) #
Bug fix #
- Fix log error when temp file was missing
1.20.0 (2023-12-01) #
Bug fix #
- Fix the number of connections not being released and abnormal memory growth caused by Framework Bug
Improvements #
- Add configuration to allow setting fasthttp client parameters
1.19.0 (2023-11-16) #
Features #
- Add
http
processor - Add basic-auth based security to API module
- Allow to register self to config manager
- Allow to panic on config error or not
Bug fix #
- Fix
rewrite_to_bulk
the issue with_type
was missing in newer version - Fix
rewrite_to_bulk
, support none-index doc operations
Improvements #
- Update test, assert parse result
1.18.0 (2023-09-21) #
Breaking changes #
- Finally removed
request_body_truncate
andresponse_body_truncate
filter
Features #
- Support kafka based replication
- Add
_util.generate_uuid
to request context - Add
_util.increment_id.BUCKET_NAME
to request context - Add
singleton
to pipeline config, prevent multiple pipelines running at the same time - Add a pluggable distributed locker implementation
- Add a general
preference
config for application - Generalize abstraction for queue, refactoring disk_queue, complete kafka implementation
- Add
merge_to_bulk
processor, retiredindexing_merge
processor - Add
flow_replay
processor, retiredflow_runner
processor - Add
replication_correlation
for replication use case - Add
hash_mod
filter - Add new parameters to
bulk_response_process
filter - Add
request_reshuffle
filter - Add resource limit, allow to set max number of cpus or binding affinity
- Support nested variables in template
- Add
rewrite_to_bulk
filter
Bug fix #
- Fix retry delay was not working in pipeline
- Fix number was not supported in template
- Fix queue selector by labels, if more than one labels specified, they should all match together neither any match should be found
Improvements #
- Lowercase all module names
- Prefetch elasticsearch metadata during start
- Adding a shutdown signal to application scope
- Refactoring queue api, support kafka management
- Add
enabled
to badger module - Allow to register module/plugin with priority
- Unified queue usage and initialization
- Optimize
bulk_reshuffle
filter’s performance, add headerX-Bulk-Reshuffled
to response - Support to use variables in
queue
filter, allow to output last produced message offset
1.17.0 (2023-07-21) #
Features #
- Add filter
consumer
to subscribe message from queue. - Add filter
stmp
to send email messages.
Improvements #
- Auto skip corrupted disk_queue file
1.16.0 (2023-06-30) #
Features #
- Add filter
security
to support unified identity management through the Console, and supports LDAP authentication login.
1.15.0 (2023-06-08) #
Breaking changes #
Features #
- Add filter
auto_generate_doc_id
to fill UUID during document creation
Bug fix #
- Fix
floating_ip
freqently switched between active and standby - Fix
elasticsearch
overwriting upstreamx-forwarded-for
header. - Fix
queue_consumer
high CPU usage when queue is empty.
Improvements #
1.14.0 (2023-05-25) #
Breaking changes #
Features #
- Support customize service name
metrics
addeduser_in_ms
andsys_in_ms
to track CPU utilization.elasticsearch
addeddial_timeout
option.
Bug fix #
- Fix missing stdout outputs after real-time logging push turned on.
logging
fixed brokenmin_elapsed_time_in_ms
option.- Fix a issue that consumes idle queues causing high CPU usage.
Improvements #
1.13.0 (2023-05-11) #
Breaking changes #
Features #
- Allow to toggle each rules in router, disabled by default
- Added support for
loong64
architecture. - Added support for
riscv64
architecture. elasticsearch
addeddial_timeout
option.
Bug fix #
- Fix http response header from upstream was not well returned to client
- Fix duplicated running pipelines when hot-reloading pipeline configruations.
bulk_indexing
fix leaking goroutines when pipeline stopped or released.
Improvements #
- Prefer to set headers instead of add headers, avoid generate duplicated headers
- Improve the responsiveness of stopping pipeline.
pipeline
addedenabled
option to toggle the pipelines quickly.
1.12.1 (2023-04-20) #
Bug fix #
elasticsearch
fix potential http connection close.elasticsearch
return proper error info when upstream timed out.
1.12.0 (2023-04-06) #
Breaking changes #
bulk_indexing
renamebulk.response_handle.retry_exception
tobulk.response_handle.retry_rules
Improvements #
bulk_indexing
improve logging and default retry times.
Bug fix #
bulk_indexing
fix potential data loss when 429 status encountered.bulk_indeixng
fix disk queue file retention rules not working properly.metrics
fix prometheus metrics exportingpipeline
fix auto-reloadingbadger
fix potential file corruption.bulk_reshuffle
fix potential memory leaking.
1.11.0 #
Breaking changes #
- Rename condition
has_fields
toexists
Features #
echo
added more options:status
,continue
,response
,logging
,logging_level
.switch
addedcontinue
andunescape
options.- Add max memory setting to system config
- Add flags to set max memory soft limit
Bug fix #
- Fix path init when install service
Improvements #
unescape
path in switch filter by default- Update badger to v4
- Add more memory info to stats
- Escape metric name for prometheus
1.10.0 #
Breaking changes #
bulk_indexing
move bulk response handling controls frombulk
tobulk.response_handle
.
Features #
- Support environment variables in config
- Add option
action
toratio
filter, allow drop request immediately - Add
context_parser
filter - Add
context_switch
filter - Add
_sys.*
to request context
Bug fix #
Improvements #
1.9.0 #
Breaking changes #
- Refactoring config for ip access control
- Disable elasticsearch metadata refresh by default
- Update default config path from
configs
toconfig
- Remove
sample-configs
, moved to dedicated integrated-testing project - Remove field
conntime
, update field@timestamp
totimestamp
inlogging
filter - Rename
disorder_
tofast_
Features #
- Support listen on IPv6 address
- Add general health api
- Add
request_ip
to context - Add badger filter plugin
- Allow to split produce and consume messages from s3
- Add
bulk_request_throttle
filter - Support access request context and more output options in
echo
filter - Add
body_json
to response context - Add cert config to API module, support mTLS
- Add api to clear scroll context
- Floating_ip support stick by
priority
- Add keystore util
- Allow to save success bulk results in
bulk_indexing
processor - Enable watch and reload the major config file
- Support run background job in one goroutine
- Allow to handle async_bulk request logging
- Add config to control cluster health check while cluster not available, set default to false
- Allow to follow redirects in http filter, set default read and write timeout to 30s
- Support collect instance metrics to monitoring gateway
- Add json log format
Bug fix #
- Fix user was removed in logging filter
- Fix incorrect message size issue, reload when files changed in disk_queue
- Fix issue that
index_diff
could not finished automatically - Fix hostname was not well updated in filter
set_request_header
orset_hostname
- Fix to check consumer’s lag instead of queue’s lag in
flow_runner
processor - Fix file not found error for disk_queue
- Fix the delete requests was not proper handled in filter
bulk_reshuffle
,bulk_request_mutate
andbulk_indexing
processor - Fix memory leak caused by misuse of bytes buffer
- Fix to handle the last request in replay processor
- Fix url args was not updated after change
- Fix memory leak when serving high-concurrent requests
- Fix nil id caused error when using sliced workers in
bulk_indexing
processor - Fix index name with dot
- Refactoring time fields for orm, skip empty time
- Refactoring stats, allow to register extended stats
- Fix to restart gateway entrypoint on flow change
- Update ratio filter, fix random number, add header to ratio filter
- Fix query parameter
no_cache
was not well respected inget_cache
filter - Fix single delete request was ignored in bulk requests
- Fix request mutate filter
Improvements #
- Remove newline in indexing_merge and json_indexing processor
- Improve instance check, add config to disable
- Add option
skip_insecure_verify
to s3 module - Improve instance check, enable config to disable
- Update the way to get ctx process info, optimize memory usage
- Improve indexing performance for
bulk_indexing
processor - Refactoring disk_queue, speedup message consumption
- Enable segment compress for disk_queue by default
- Skip download s3 files when s3 was not enabled
- Add option to log warning messages for throttle filters
- Optimize hash performance for getting primary shardID and partitionID
- Add cache for get index routing table
- Optimize performance for bulk response processing
- Refactoring bulk_processor, pass meta info to payload func
- Don’t call payload func for delete action
- Improve queue consumer’s lag check
- Enable prepare flat files ahead for read by default, skip unnecessary file
- Add object pool for xxhash
- Refactoring disk_queue, handle consumer in-flight segments in memory
- Add config to remove duplicated newline for bulk_processor
- Add metric timestamp in stats api
- Improve error on routing table missing
- Refactoring bytes buffer and object pool, expose metrics via API
- Refactoring tasks pooling, support throttle and unified control
- Optimize badger file size and memory usage
- Refactoring time fields for orm, skip empty time
- Refactoring stats, allow to register extended stats
- Refactoring to handle bulk response results
- Add client_session_cache_size to tls setting
- Safety add newline to each bytes when handle bulk requests
1.8.1 #
Bug fix #
- Remove newline in document for processor
es_scroll
anddump_hash
1.8.0 #
Breaking changes #
- Remove config
compress_on_message_payload
from disk_queue - Rename parameter
consumer
toname
inconsumer_has_lag
condition - Remove redundancy prefix name of the disk_queue files
Features #
- Add segment level based disk_queue file compression
Bug fix #
- Fix nil host in
bulk_indexing
processor - Fix nil body in
bulk_response_process
filter - Fix sliced consume in
bulk_indexing
processor
Improvements #
- Handle bulk stats to
bulk_response_process
and used inlogging
filter
1.7.0 #
Breaking changes #
Features #
- Add prometheus format to
stats
API - Add
redirect
filter - Add
context_flow
toflow
filter - Add
permitted_client_ip_list
torouter
- Add Centos based docker image
Bug fix #
- Fix
date_range_precision
filter failed to parse on specify field - Fix disk usage status in windows platform
Improvements #
- Merge events during config change, prevent unnecessary reload
- Handle templates when loading config
- Add cache to
ldap_auth
filter
1.6.0 #
Breaking changes #
- Update disk_queue folder structure, use UUID as folder name instead of the queue name
- Parameter
mode
was removed frombulk_reshuffle
filter, onlyasync
was supported - Rename filter
bulk_response_validate
tobulk_response_process
Features #
- Add metadata to queue
- Support subscribe queue by specify labels
- Support concurrent worker control for
bulk_indexing
processor - Auto detect new queues for
bulk_indexing
processor - Allow to consume queue messages over disk queue
- Auto sync disk_queue files to remote s3 in background
- Add api to operate gateway entry
- Support plugin auto discovery
- Add API to operate gateway entities
- Filter
bulk_request_mutate
support remove_type
in bulk requests for es v8.0+ - Add elasticsearch adapter for version 8.0+
- Add
http
filter for general reverse proxy usage, like proxy Kibana - Add
consumer_has_lag
condition to check queue status - Add
record
filter to play requests easier - Add zstd compress to disk_queue, disabled by default
- Add
disorder_bulk_indexing
processor - Add
javascript
filter - Add
prefix
andsuffix
to when conditions - Add
indexing_merge
processor
Bug fix #
- Fix
date_range_precision_tuning
filter for complex range query - Fix node availability initially check
- Fix
basic_auth
filter not asking user to input auth info in browser - Fix
null
id not fixed in filterbulk_request_mutate
andbulk_reshuffle
- Fix
switch
filter not forwarding whenremove_prefix
was disabled - Fix buffer was not proper reset in
flow_runner
processor - Fix entry not loading the pre-defined TLS certificates
- Fix
set_basic_auth
not proper reset previous user information - Fix
elapsed
in request logging not correct - Fix
switch
filter, usestrings.TrimPrefix
instead ofstrings.TrimLeft
- Fix the last query_string args can’t be deleted, parameter
no_cache
inget_cache
filter fixed - Fix s3 downloaded file corrupted
Improvements #
- Handle http public address, remove prefix if that exists
- Refactor
bulk_reshuffle
filter andbulk_indexing
processor - Should not fetch nodes info when elasticsearch discovery disabled
- Seamless consume queue message across files
- Persist consumer offset to local store
- Add API to reset consumer offset
- Refactoring ORM framework
- Expose error of mapping put
- Refactoring pipeline framework
- Improve multi-instance check, multi-instance disabled by default
- Add CPU and memory metrics to stats api
- Seamless fetch queue files from s3 server
- Proper handle 409 version conflicts in bulk requests
- Allow memory queue to retry 1s when it is full
- Proper handle the cluster available check
- Proper handle the cluster partial failure check
- Exit bulk worker while no new messages returned
- Optimize bytes buffer usage, reduced memory usage
1.5.0 #
Breaking changes #
Features #
- Add API to scroll messages from disk queue
- Prevent out of space, disk usage reserved for disk_queue
- Add
context_filter
andcontext_limiter
for general purpose - Add
bulk_request_mutate
filter - Add
basic_auth
filter - Add
set_context
filter - Add
context_regex_replace
filter - Add
to_string
property torequest
andresponse
context
Bug fix #
- Fix bulk response validate incorrectly caused by jsonParser
- Fix nil exception in
request_path_limiter
caused by refactoring - Fix big size document out of order caused by bulk buffer
Improvements #
- Fix TCP not keepalived in some case
- Add closing progress bar to pipeline module
- Add
retry_delay_in_ms
config to pipeline module - Handle partial failure in bulk requests
- Optimize scroll performance of
dump_hash
processor - Improve API directory
1.4.0 #
Breaking changes #
- Rename flow config
filter_v2
tofilter
, only support new syntax - Rename pipeline config
pipelines_v2
topipeline
,processors
toprocessor
, only support new syntax - Rename filter
request_logging
tologging
- Merge dump filters to
dump
filter - Response headers renamed, dashboard may broken
- Remove filter
request_body_truncate
andresponse_body_truncate
Features #
- Add option to disable file logging output
- Add option
compress
toqueue_consumer
processor
Bug fix #
- Fix invalid host header setting in elasticsearch reverse proxy
- Fix cluster available health check
- Fix gzip encoding issue for requests forwarding
Improvements #
- Support string type in
in
condition
1.3.0 #
Breaking changes #
- Switch to use
pipelines_v2
syntax only - Rename filter
disk_enqueue
toqueue
- Rename processor
disk_queue_consumer
toqueue_consumer
- Rename filter
redis
toredis_pubsub
Features #
- Refactoring pipeline framework, support DAG based task schedule
- Add
dump_hash
andindex_diffs
processor - Add
redis
output andredis
queue adapter - Add
set_request_query_args
filter - Add
ldap_auth
filter - Add
retry_limiter
filter - Add
request_body_json_set
andrequest_body_json_del
filter - Add
stats
filter - Add
health_check
config toelastic
module - Add API to pipeline framework, support
_start
and_stop
pipelines
Bug fix #
- Fix data race issue in bulk_reshuffle
- Fix
fix_null_id
always executed in bulk_reshuffle - Auto handle big sized documents in bulk requests
Improvements #
- Refactoring flow runner to service pipeline
- Optimize CPU and Memory usage
- Optimize index diff service, speedup and cross version compatibility
- Set the default max file size of queue files to 1 GB
- Proper handle elasticsearch failure during startup
- Support custom depth check to
queue_has_lag
condition - Support multi hosts for elasticsearch configuration
- Add parameter
auto_start
to prevent pipeline running on start - Add
keep_running
parameter to pipeline config - Safety shutdown pipeline and entry service
- Support more complex routing pattern rules
1.2.0 #
Features #
- Support alias in bulk_reshuffle filter.
- Support truncate in request_logging filter.
- Handle 429 retry in json_indexing service.
- Add forcemerge service.
- Add
response_body_regex_replace
filter. - Add
request_body_regex_replace
filter. - Add
sleep
filter. - Add option to log slow requests only.
- Add cluster and bulk status to request logging.
- Add
filter_v2
and support_ctx
to access request context. - Add
dump_context
filter. - Add
translog
filter, support rotation and compression. - Add
set_response
filter. - Add
set_request_header
filter. - Add
set_hostname
filter. - Add
set_basic_auth
filter. - Add
set_response_header
filter. - Add
elasticsearch_health_check
filter. - Add
drop
filter.
Bug fix #
- Fix truncate body filter, correctly resize the body bytes.
- Fix cache filter.
- Fix floating_ip module.
- Fix dirty write in diskqueue.
- Fix compression enabled requests.
- Fix date_range_precision_tuning filter.
- Fix invalid indices status on closed indices #23.
- Fix document hash for elasticsearch 6.x.
- Fix floating_ip feature run with daemon mode.
- Fix async bulk to work with beats.
Improvements #
- Optimize memory usage, fix memory leak.
Acknowledgement #
Thanks to the following enterprises and teams #
- China Everbright Bank, China Citic Bank, BSG, Yogoo
Thanks to the following individual contributors #
- MaQianghua, YangFan, Tanzi, FangLi
1.1.0 #
- Request Logging and Dashboard.
- Support ARM Platform [armv5\v6\v7\v8(arm64)].
- Fix Elasticsearch Nodes Auto Discovery.
- Add Request Header Filter.
- Add Request Method Filter.
- Add Sample Filter.
- Request Logging Performance Optimized (100x speedup).
- Add Request Path Filter.
- Add Debug Filter.
- Add User Info to Logging Message.
- Support Routing Partial Traffic to Specify Processing Flow (by Ratio).
- Support Traffic Clone, Support Dual-Write or 1:N Write.
- Elasticsearch topology auto discovery, support filter by nodes,tags,roles.
- Backend failure auto detection, auto retry and select another available endpoint.
- Floating IP feature ready to use.
- Add bulk_reshuffle filter.
1.0.0 #
- Rewritten for performance
- Index level request throttle
- Request caching
- Kibana MAGIC speedup
- Upstream auto discovery
- Weighted upstream selections
- Max connection limit per upstream