Skip to content
This repository has been archived by the owner on Jun 16, 2023. It is now read-only.

Publish 2.1.1 to Maven Center repository
Compare
Choose a tag to compare
@longdafeng longdafeng released this 09 Mar 01:55
· 167 commits to master since this release
2.1.1
b515985

Publish 2.1.1 to Maven Center repository

For Chinese release notes, please refer to https://github.com/alibaba/jstorm/blob/master/history_cn.md

New features

  1. 1.5~6x performance boost from worst to best scenarios compared to JStorm-2.1.0
  2. Add application-level auto-batch
  3. Add independent control channel to separate control msgs from biz msgs to guarantee high priority for control msgs
  4. Dramatic performance boost in metrics, see "Improvements" section
  5. Support jdk1.8
  6. Add Nimbus hook and topology hook
  7. Metrics system:
    1. Support disable/enable metrics on the fly
    2. Add jstorm metrics design docs, see JSTORM-METRICS.md
  8. JStorm web UI:
    1. Add zookeeper viewer in web UI, thanks to @dingjun84
    2. Add log search and deep log search, support both backward search and forward search
    3. Support log file download
  9. Support changing log level on the fly
  10. Change error structure in zk, add errorLevel, errorCode and duration.
  11. Add supervisor health check
  12. Add -Dexclude.jars option to enable filtering jars manually

Improvements

  1. Metrics:
    1. use JHistogram/JMeter instead of Histogram/Meter, change internal Clock.tick to System.currentTimeMillis to improve performance (50+% boost in Meter and 25%+ boost in Histogram)
    2. add TupleLifeCycle metric
    3. add supervisor metrics: total_cpu_usage, total_mem_usage, disk_usage
    4. remove some unnecessary metrics like emitTime, etc.
    5. Use HeapByteBuffer instead of List to transmit metric data points, reduce 60+% metrics memory usage
    6. Change sample rate from 10% to 5% by default
    7. Remove AsmTimer and related code
  2. Log related:
    1. Use logback by default instead of log4j, exclude slf4j-log4j12 dependency
    2. Use jstorm.log.dir property instead of ${jstorm.home}/logs, see jstorm.logback.xml
    3. Change all log4j Logger's to slf4j Logger's
    4. Set default log page size(log.page.size) in defaults.yaml to 128KB (web UI)
    5. Change topology log structure, add ${topology.name} directory, see jstorm.logback.xml
    6. Add timestamp in supervisor/nimbus gc log files; backup worker gc log before launching a new worker;
    7. Set logback/log4j file encoding to UTF-8
  3. Refine backpressure stragety to avoid over-backpressure
  4. Change acker pending rotating map to single thread to improve performance
  5. Update RefreshConnections to avoid downloading assignments from zk frequently
  6. Change default memory of Supervisor to 1G (previous 512MB)
  7. Use ProcessLauncher to launch processes
  8. Add DefaultUncaughtExceptionHandler for supervisor and nimbus
  9. Change local ports to be different from 0.9.x versions (supervisor.slots.ports.base, nimbus.thrift.port,
    nimbus.deamon.logview.port, supervisor.deamon.logview.port)
  10. Change highcharts to echarts to avoid potential license violation
  11. Dependency upgrades:
    1. Upgrade kryo to 2.23.0
    2. Upgrade disruptor to 3.2.2

Bug fix

  1. Fix deadlock when starting workers
  2. Fix the bug that when localstate file is empty, supervisor can't start
  3. Fix kryo serialization for HeapByteBuffer in metrics
  4. Fix total memory usage calculation
  5. Fix the bug that empty worker is assigned when configured worker number is bigger than the actual number for user defined scheduler
  6. Fix UI log home directory
  7. Fix XSS security bug in web UI
  8. Don't start TopologyMetricsRunnable thread in local mode, thanks to @L-Donne
  9. Fix JSTORM-141, JSTORM-188 that TopologyMetricsRunnable consumes too much CPU
  10. Remove MaxTenuringThreshold JVM option support jdk1.8, thanks to @249550148
  11. Fix possible NPE in MkLocalShuffer

Deploy and scripts

  1. Add cleanup for core dumps
  2. Add supervisor health check in healthCheck.sh
  3. Change jstorm.py to terminate the original python process when starting nimbus/supervisor

Upgrade guide

  1. JStorm 2.1.1 is mostly compatible with 2.1.0, but it's better to restart your topologies to finish the upgrade.
  2. If you're using log4j, be cautious that we have switched default logging system to logback, if you still want to use log4j, please add "user.defined.log4j.conf: jstorm.log4j.properties" to your conf/storm.yaml.
  3. If you're using slf4j-api + log4j, please add slf4j-log4j12 dependency in your pom config.
Assets 4
Loading