Data Serialization

Phase 1: JSON Serialization (Current)

Configuration

spring.kafka.producer.value-serializer=JsonSerializer
spring.kafka.consumer.value-deserializer=JsonDeserializer

Benefits

• Human-Readable: Easy to debug and inspect messages
• Simple Setup: No additional infrastructure required
• Flexibility: Dynamic schema, no code generation

Limitations

• No Schema Enforcement: Producers and consumers must agree on structure manually
• Larger Message Size: JSON is verbose compared to binary formats
• Schema Evolution Challenges: Breaking changes can cause consumer failures

Phase 2: Avro Serialization (Future)

What is Avro?

Apache Avro is a data serialization system that provides:

• Compact binary format
• Schema defined in JSON
• Code generation for type safety
• Schema evolution with backward/forward compatibility

Confluent Schema Registry

Centralized repository for managing Avro schemas:

• Store and version schemas
• Enforce compatibility rules
• Automatic schema validation
• Schema evolution support

Configuration

spring.kafka.producer.value-serializer=KafkaAvroSerializer
spring.kafka.consumer.value-deserializer=KafkaAvroDeserializer
spring.kafka.properties.schema.registry.url=http://localhost:8081

Schema Evolution Strategies

Backward Compatibility

New schema can read data written with old schema. Add optional fields only.

Forward Compatibility

Old schema can read data written with new schema. Remove optional fields only.

Full Compatibility

Both backward and forward compatible. Most restrictive but safest.

Migration Strategy

1. Setup Schema Registry: Deploy Confluent Schema Registry
2. Define Avro Schemas: Create .avsc files for DTOs
3. Dual Write: Produce to both JSON and Avro topics temporarily
4. Consumer Migration: Update consumers to read from Avro topic
5. Producer Migration: Switch producers to Avro-only
6. Cleanup: Remove JSON topic after validation