Roadmap

Remove required dependency on Confluent Schema Registry

Is your feature request related to a problem? Please describe.

Confluent Schema Registry is a required component for DataHub. This makes adoption of DataHub harder for companies that don’t prefer the licensing.

Describe the solution you'd like

DataHub uses the Schema Registry more as a convenience than a necessity. The Kafka Topic schemas are built along-side the code, and with the MCP/MCL pathway are no longer going to change as often.

We could remove the dependency on Schema Registry as a required component, and default to in-memory (compiled) schemas on the classpath, if a schema registry is not found to be configured on startup.

This will simplify deployment for certain environments and companies.

Describe alternatives you've considered

We could also use another schema registry like Karapace for this, but since our requirements are so simple, we might be able to just remove this requirement altogether.

Additional context

https://datahubspace.slack.com/archives/CV2KB471C/p1643033131052800