serde: add Buf Schema Registry (BSR) protobuf support #2014
Conversation
This commit adds support for deserializing protobuf messages that use plain protobuf payloads but use Buf Schema Registry (BSR). Unlike Confluent Schema Registry which embeds schema information in the message payload, BSR stores schema metadata in Kafka record headers. The BSR wire format uses two headers to identify the schema: - buf.registry.value.schema.message: The fully-qualified message type - buf.registry.value.schema.commit: The schema version/commit hash The implementation includes: Backend Changes: - New BSRClient interface and HTTP client for fetching schemas from BSR - ProtobufBSRSerde for deserializing BSR-encoded messages - Configuration support for BSR endpoint URL and authentication token - Integration with existing serde service to automatically detect BSR messages - Comprehensive unit and integration tests Frontend Changes: - Added PAYLOAD_ENCODING_PROTOBUF_BSR enum value - Updated message preview components to display BSR encoding - Extended proto definitions and TypeScript interfaces The BSR serde is prioritized before other protobuf serdes in the deserialization pipeline since BSR messages are identifiable by their specific headers, allowing for efficient format detection. All existing tests pass and new tests verify BSR deserialization works correctly with dynamic protobuf messages.
weeco
left a comment
weeco
left a comment
There was a problem hiding this comment.
Hey @sbhuiyan-buf ,
i haven't completed the review on the serde and the client itself, but I was wondering whether it would make sense to use https://github.com/bufbuild/bsr-kafka-serde-go/tree/main to reduce the maintenance burden.
The redpanda console is a bit different in that it is needing to use the BSR client to get the filedescriptor and use it to then deserialize to JSON to display on a UI. The bsr-kafka-serde-go is hiding that client for its own use to deserialize to the strict application types that application developers are using themselves |
2 participants
This commit adds support for deserializing protobuf messages that use plain protobuf payloads but use Buf Schema Registry (BSR). Unlike Confluent Schema Registry which embeds schema information in the message payload, BSR stores schema metadata in Kafka record headers.
The BSR wire format uses two headers to identify the schema:
The implementation includes:
Backend Changes:
Frontend Changes:
The BSR serde is prioritized before other protobuf serdes in the deserialization pipeline since BSR messages are identifiable by their specific headers, allowing for efficient format detection.
All existing tests pass and new tests verify BSR deserialization works correctly with dynamic protobuf messages.