1.4.0

This is a substantial release. Although there are no known 'showstopper' bugs as of release, we do recommend you test any planned upgrade to your application prior to running in production.

Some of the major changes include:

• We have officially dropped python 2.6 support
• The KafkaConsumer now includes a background thread to handle coordinator heartbeats
• API protocol handling has been separated from networking code into a new class, KafkaProtocol
• Added support for kafka message format v2
• Refactored DNS lookups during kafka broker connections
• SASL authentication is working (we think)
• Removed several circular references to improve gc on close()

Thanks to all contributors -- the state of the kafka-python community is strong!

Detailed changelog are listed below:

Client

• Fixes for SASL support
  • Refactor SASL/gssapi support (dpkp #1248 #1249 #1257 #1262 #1280)
  • Add security layer negotiation to the GSSAPI authentication (asdaraujo #1283)
  • Fix overriding sasl_kerberos_service_name in KafkaConsumer / KafkaProducer (natedogs911 #1264)
  • Fix typo in _try_authenticate_plain (everpcpc #1333)
  • Fix for Python 3 byte string handling in SASL auth (christophelec #1353)
• Move callback processing from BrokerConnection to KafkaClient (dpkp #1258)
• Use socket timeout of request_timeout_ms to prevent blocking forever on send (dpkp #1281)
• Refactor dns lookup in BrokerConnection (dpkp #1312)
• Read all available socket bytes (dpkp #1332)
• Honor reconnect_backoff in conn.connect() (dpkp #1342)

Consumer

• KAFKA-3977: Defer fetch parsing for space efficiency, and to raise exceptions to user (dpkp #1245)
• KAFKA-4034: Avoid unnecessary consumer coordinator lookup (dpkp #1254)
• Handle lookup_coordinator send failures (dpkp #1279)
• KAFKA-3888 Use background thread to process consumer heartbeats (dpkp #1266)
• Improve KafkaConsumer cleanup (dpkp #1339)
• Fix coordinator join_future race condition (dpkp #1338)
• Avoid KeyError when filtering fetchable partitions (dpkp #1344)
• Name heartbeat thread with group_id; use backoff when polling (dpkp #1345)
• KAFKA-3949: Avoid race condition when subscription changes during rebalance (dpkp #1364)
• Fix #1239 regression to avoid consuming duplicate compressed messages from mid-batch (dpkp #1367)

Producer

• Fix timestamp not passed to RecordMetadata (tvoinarovskyi #1273)
• Raise non-API exceptions (jeffwidman #1316)
• Fix reconnect_backoff_max_ms default config bug in KafkaProducer (YaoC #1352)

Core / Protocol

• Add kafka.protocol.parser.KafkaProtocol w/ receive and send (dpkp #1230)
• Refactor MessageSet and Message into LegacyRecordBatch to later support v2 message format (tvoinarovskyi #1252)
• Add DefaultRecordBatch implementation aka V2 message format parser/builder. (tvoinarovskyi #1185)
• optimize util.crc32 (ofek #1304)
• Raise better struct pack/unpack errors (jeffwidman #1320)
• Add Request/Response structs for kafka broker 1.0.0 (dpkp #1368)

Bugfixes

• use python standard max value (lukekingbru #1303)
• changed for to use enumerate() (TheAtomicOption #1301)
• Explicitly check for None rather than falsey (jeffwidman #1269)
• Minor Exception cleanup (jeffwidman #1317)
• Use non-deprecated exception handling (jeffwidman a699f6a)
• Remove assertion with side effect in client.wakeup() (bgedik #1348)
• use absolute imports everywhere (kevinkjt2000 #1362)

Test Infrastructure

• Use 0.11.0.2 kafka broker for integration testing (dpkp #1357 #1244)
• Add a Makefile to help build the project, generate docs, and run tests (tvoinarovskyi #1247)
• Add fixture support for 1.0.0 broker (dpkp #1275)
• Add kafka 1.0.0 to travis integration tests (dpkp #1365)
• Change fixture default host to localhost (asdaraujo #1305)
• Minor test cleanups (dpkp #1343)
• Use latest pytest 3.4.0, but drop pytest-sugar due to incompatibility (dpkp #1361)

Documentation

• Expand metrics docs (jeffwidman #1243)
• Fix docstring (jeffwidman #1261)
• Added controlled thread shutdown to example.py (TheAtomicOption #1268)
• Add license to wheel (jeffwidman #1286)
• Use correct casing for MB (jeffwidman #1298)

Logging / Error Messages

• Fix two bugs in printing bytes instance (jeffwidman #1296)

1.3.5

Bugfixes

• Fix partition assignment race condition (jeffwidman #1240)
• Fix consumer bug when seeking / resetting to the middle of a compressed messageset (dpkp #1239)
• Fix traceback sent to stderr not logging (dbgasaway #1221)
• Stop using mutable types for default arg values (jeffwidman #1213)
• Remove a few unused imports (jameslamb #1188)

Client

• Refactor BrokerConnection to use asynchronous receive_bytes pipe (dpkp #1032)

Consumer

• Drop unused sleep kwarg to poll (dpkp #1177)
• Enable KafkaConsumer beginning_offsets() and end_offsets() with older broker versions (buptljy #1200)
• Validate consumer subscription topic strings (nikeee #1238)

Documentation

• Small fixes to SASL documentation and logging; validate security_protocol (dpkp #1231)
• Various typo and grammar fixes (jeffwidman)

1.3.4

Bugfixes

• Avoid multiple connection attempts when refreshing metadata (dpkp #1067)
• Catch socket.errors when sending / recving bytes on wake socketpair (dpkp #1069)
• Deal with brokers that reappear with different IP address (originsmike #1085)
• Fix join-time-max and sync-time-max metrics to use Max() measure function (billyevans #1146)
• Raise AssertionError when decompression unsupported (bts-webber #1159)
• Catch ssl.EOFErrors on Python3.3 so we close the failing conn (Ormod #1162)
• Select on sockets to avoid busy polling during bootstrap (dpkp #1175)
• Initialize metadata_snapshot in group coordinator to avoid unnecessary rebalance (dpkp #1174)

Client

• Timeout idle connections via connections_max_idle_ms (dpkp #1068)
• Warn, dont raise, on DNS lookup failures (dpkp #1091)
• Support exponential backoff for broker reconnections -- KIP-144 (dpkp #1124)
• Add gssapi support (Kerberos) for SASL (Harald-Berghoff #1152)
• Add private map of api key -> min/max versions to BrokerConnection (dpkp #1169)

Consumer

• Backoff on unavailable group coordinator retry (dpkp #1125)
• Only change_subscription on pattern subscription when topics change (Artimi #1132)
• Add offsets_for_times, beginning_offsets and end_offsets APIs (tvoinarovskyi #1161)

Producer

• Raise KafkaTimeoutError when flush times out (infecto)
• Set producer atexit timeout to 0 to match del (Ormod #1126)

Core / Protocol

• 0.11.0.0 protocol updates (only - no client support yet) (dpkp #1127)
• Make UnknownTopicOrPartitionError retriable error (tvoinarovskyi)

Test Infrastructure

• pylint 1.7.0+ supports python 3.6 and merge py36 into common testenv (jianbin-wei #1095)
• Add kafka 0.10.2.1 into integration testing version (jianbin-wei #1096)
• Disable automated tests for python 2.6 and kafka 0.8.0 and 0.8.1.1 (jianbin-wei #1096)
• Support manual py26 testing; dont advertise 3.3 support (dpkp)
• Add 0.11.0.0 server resources, fix tests for 0.11 brokers (dpkp)
• Use fixture hostname, dont assume localhost (dpkp)
• Add 0.11.0.0 to travis test matrix, remove 0.10.1.1; use scala 2.11 artifacts (dpkp #1176)

Logging / Error Messages

• Improve error message when expiring batches in KafkaProducer (dpkp #1077)
• Update producer.send docstring -- raises KafkaTimeoutError (infecto)
• Use logging's built-in string interpolation (jeffwidman)
• Fix produce timeout message (melor #1151)
• Fix producer batch expiry messages to use seconds (dnwe)

Documentation

• Fix typo in KafkaClient docstring (jeffwidman #1054)
• Update README: Prefer python-lz4 over lz4tools (kiri11 #1057)
• Fix poll() hyperlink in KafkaClient (jeffwidman)
• Update RTD links with https / .io (jeffwidman #1074)
• Describe consumer thread-safety (ecksun)
• Fix typo in consumer integration test (jeffwidman)
• Note max_in_flight_requests_per_connection > 1 may change order of messages (tvoinarovskyi #1149)

1.3.3

Release 1.3.3

1.3.2

Patch Release 1.3.2

1.3.1

Bugfixes

• Fix AttributeError in BrokerConnectionMetrics after reconnecting

1.3.0

Incompatible Changes

• Delete KafkaConnection class (dpkp 769)
• Rename partition_assignment -> assignment in MemberMetadata for consistency
• Move selectors34 and socketpair to kafka.vendor (dpkp 785)
• Change api_version config to tuple; deprecate str with warning (dpkp 761)
• Rename _DEFAULT_CONFIG -> DEFAULT_CONFIG in KafkaProducer (dpkp 788)

Improvements

• Vendor six 1.10.0 to eliminate runtime dependency (dpkp 785)
• Add KafkaProducer and KafkaConsumer.metrics() with instrumentation similar to java client (dpkp 754 / 772 / 794)
• Support Sasl PLAIN authentication (larsjsol PR 779)
• Add checksum and size to RecordMetadata and ConsumerRecord (KAFKA-3196 / 770 / 594)
• Use MetadataRequest v1 for 0.10+ api_version (dpkp 762)
• Fix KafkaConsumer autocommit for 0.8 brokers (dpkp 756 / 706)
• Improve error logging (dpkp 760 / 759)
• Adapt benchmark scripts from https://github.com/mrafayaleem/kafka-jython (dpkp 754)
• Add api_version config to KafkaClient (dpkp 761)
• New Metadata method with_partitions() (dpkp 787)
• Use socket_options configuration to setsockopts(). Default TCP_NODELAY (dpkp 783)
• Expose selector type as config option (dpkp 764)
• Drain pending requests to the coordinator before initiating group rejoin (dpkp 798)
• Send combined size and payload bytes to socket to avoid potentially split packets with TCP_NODELAY (dpkp 797)

Bugfixes

• Ignore socket.error when checking for protocol out of sync prior to socket close (dpkp 792)
• Fix offset fetch when partitions are manually assigned (KAFKA-3960 / 786)
• Change pickle_method to use python3 special attributes (jpaulodit 777)
• Fix ProduceResponse v2 throttle_time_ms
• Always encode size with MessageSet (#771)
• Avoid buffer overread when compressing messageset in KafkaProducer
• Explicit format string argument indices for python 2.6 compatibility
• Simplify RecordMetadata; short circuit callbacks (#768)
• Fix autocommit when partitions assigned manually (KAFKA-3486 / #767 / #626)
• Handle metadata updates during consumer rebalance (KAFKA-3117 / #766 / #701)
• Add a consumer config option to exclude internal topics (KAFKA-2832 / #765)
• Protect writes to wakeup socket with threading lock (#763 / #709)
• Fetcher spending unnecessary time during metrics recording (KAFKA-3785)
• Always use absolute_import (dpkp)

Test / Fixtures

• Catch select errors while capturing test fixture logs
• Fix consumer group test race condition (dpkp 795)
• Retry fixture failures on a different port (dpkp 796)
• Dump fixture logs on failure

Documentation

• Fix misspelling of password (ssaamm 793)
• Document the ssl_password config option (ssaamm 780)
• Fix typo in KafkaConsumer documentation (ssaamm 775)
• Expand consumer.fetcher inline comments
• Update kafka configuration links -> 0.10.0.0 docs
• Fixup metrics_sample_window_ms docstring in consumer

1.2.5

Bugfixes

• Fix bug causing KafkaProducer to double-compress message batches on retry
• Check for double-compressed messages in KafkaConsumer, log warning and optionally skip
• Drop recursion in _unpack_message_set; only decompress once

1.2.4

Bugfixes

• Update consumer_timeout_ms docstring - KafkaConsumer raises StopIteration, no longer ConsumerTimeout
• Use explicit subscription state flag to handle seek() during message iteration
• Fix consumer iteration on compacted topics (dpkp PR 752)
• Support ssl_password config when loading cert chains (amckemie PR 750)

1.2.3

Patch Improvements

• Fix gc error log: avoid AttributeError in _unregister_cleanup (dpkp PR 747)
• Wakeup socket optimizations (dpkp PR 740)
• Assert will be disabled by "python -O" (tyronecai PR 736)
• Randomize order of topics/partitions processed by fetcher to improve balance (dpkp PR 732)
• Allow client.check_version timeout to be set in Producer and Consumer constructors (eastlondoner PR 647)