Webapp CVE Processor

This Go application is designed to process cves and push them to kafka topic.

Downloading Data

This application downloads CVE or CVE Deltas data to downloads directory.

Extracting Data

Unzips the downloaded file while skipping specific JSON files using goroutines.

Pushing messages to Kafka

Configured a synchronous Kafka producer for idempotent message production, retries, and acknowledgment requirements using the Sarama library (Go library for Apache Kafka) and establishes a connection to the Kafka broker.

Determines the number of concurrent goroutines to use based on environment variable and splits the extracted file into groups and processes each group concurrently using goroutines.

You can adjust go_routines environment variable based on the resource limits of a pod to which this container belongs. For example, on a machine with more CPU cores and memory, you can increase the resource limits of a pod and increase the number of goroutines to speed up processing.

By using go_routines as an environment variable, you can easily configure the application for different resource limits without changing the code and leverage more CPU cores, especially on high-performance machines.

Environment Variables

The application uses environment variables that are retrieved using os.Getenv() and used to establish connections to Kafka broker.

Setting Environment Variables

Once the Docker image is built, appropriate environment variables like kafka_host, kafka_port, kafka_topic and go_routines need to be supplied to the container to ensure that the pod does not fail.

Building the Docker Image

1. Build and push Docker Image using buildx:

   docker buildx build --platform linux/amd64,linux/arm64 -t DOCKER_USERNAME/cve-processor:latest --push .