• Introduction

  • This document is a comprehensive architectural design for applying near-real-time ingestion & consumption on metadata in the Atlan platform.

• Solution Overview

  • The solution uses Apache Kafka for brokering messages flow at near real-time speed. It consists of metadata ingestion, transformation, storage, and consumption components. Inbound (external & internal) and Outbound (internal and externally sourced).

• Architecture

  • Components

    • Kafka Broker - manages to broker of messages.
    • Producers: Any service producing metadata to Kafka topics.
    • Consumers: Simple subscribers to Kafka topics
    • Store: A database (likely NoSQL Database) that can save the inserted data in a scalable way.
    • Flink or Spark - In order to generate sense from big data and effectively send outbound communication on any PI data detection or any alerting event.

  • Data Flow

    • Inbound Metadata Ingestion:
      • Producers extract metadata from sources (e.g., Monte Carlo) and publish to Kafka.
      • Kafka Brokers guarantee durable delivery to consumers using optional acknowledgements.
      • Subscribe to the Metadata, so that it can be consumed as outbound metadata.
      • Consumers read Kafka metadata for processing and forwarding data to downstream systems (like enabling access-control on top of sql queries). For allowing downstream systems to access data at an sql query level, consumers read from kafka meta data.

• Implementation Details

  • Producer Implementation
    • The implementation of the producer.
    • Producer Configuration: Connecting to Kafka, performing retries and guaranteeing delivery of messages.
  • Consumer Implementation
    • The consumer’s implementation.
    • Consumer Configuration: Subscribing to topics in Kafka and handling messaging.
    • Processing Logic: Decoding messages and applying business logic.
  • HTTP Server for Metadata Ingestion
    • How the server is set up to receive metadata via HTTP requests and starts Kafka Producers.

• Multi-Tenancy Support

  • Data Segregation: Tenant data isolation with Kafka topic partitioning and tenant schema in metadata store.
  • Access Control: This includes role-based access controls for RBAC operations on tenants’ data processing in the system.

• Deployment and Scalability

  • Containerization– Dockerizing components.
  • Orchestration- using Kubernetes for deployment as well as scaling activities in containers-orchestration of containerized applications;
  • Monitoring and Logging – Prometheus monitoring tool & Grafana logging tool integration into the system for visibility purposes/improved transparency/monitoring/logging support.

• Cost Management

  • Infrastructure Optimization - cloud-native resources; on-demand resource scaling mechanisms utilization;
  • Open Source Tools – cost reduction strategies through open-source technology adoption.

Note - Prototype has basic implementation of the kafka based ingester and producer.

Github -

https://github.com/TheShubham99/kafka-metadata-processor