{"id":35,"date":"2024-08-31T13:55:53","date_gmt":"2024-08-31T13:55:53","guid":{"rendered":"https:\/\/www.intverse.io\/blog\/?p=35"},"modified":"2024-09-01T11:49:24","modified_gmt":"2024-09-01T11:49:24","slug":"documentation-deploying-redpanda-connect-on-eks-using-argocd","status":"publish","type":"post","link":"https:\/\/www.intverse.io\/blog\/documentation-deploying-redpanda-connect-on-eks-using-argocd\/","title":{"rendered":"Doc: Deploying Redpanda Connect on EKS using ArgoCD"},"content":{"rendered":"\n

Overview<\/strong><\/h4>\n\n\n\n

Redpanda Connect<\/a>, formerly known as Benthos (acquired by Redpanda), is a declarative data streaming service that solves a wide range of data engineering problems with simple, chained, stateless processing steps<\/a>. It implements transaction-based resiliency with back pressure, so when connecting to at-least-once sources and sinks it\u2019s able to guarantee at-least-once delivery without needing to persist messages during transit.<\/p>\n\n\n\n

It\u2019s simple to deploy<\/a>, comes with a wide range of connectors<\/a>, and is totally data agnostic, making it easy to drop into your existing infrastructure. Connect has functionality that overlaps with integration frameworks, log aggregators, and ETL workflow engines, and can therefore be used to complement these traditional data engineering tools or act as a simpler alternative.<\/p>\n\n\n\n

Argo CD<\/a> is a declarative, GitOps continuous delivery tool for Kubernetes. It pairs very well with Redpanda Connect and we will see how good the experience is in deploying and maintaining the Redpanda Connect components in this context.<\/p>\n\n\n\n

In this document, we will walk through the steps of deploying Redpanda Connect on an Amazon EKS cluster using ArgoCD.  The process involves fetching the Benthos Helm chart, uploading Helm chart manifests to GitHub, modifying the ConfigMap in Bitbucket, and deploying through ArgoCD.<\/p>\n\n\n\n

Prerequisites<\/strong><\/h4>\n\n\n\n

1. Amazon EKS Cluster<\/strong>: Ensure you have an EKS cluster running.<\/p>\n\n\n\n

2. ArgoCD<\/strong>: Installed and configured on your EKS cluster.<\/p>\n\n\n\n

3. Kubectl<\/strong>: Installed and configured to interact with your EKS cluster.<\/p>\n\n\n\n

4. Helm<\/strong>: Installed for fetching and managing Helm charts.<\/p>\n\n\n\n

5. GitHub Account<\/strong>: For storing Helm chart manifests.<\/p>\n\n\n\n

6. Bitbucket Account<\/strong>: For managing ConfigMap modifications.<\/p>\n\n\n\n

Introduction to Redpanda Connect<\/strong><\/h4>\n\n\n\n

Connect is a declarative data streaming service that solves a wide range of data engineering problems with simple, chained, stateless processing steps<\/a>. It implements transaction-based resiliency with back pressure, so when connecting to at-least-once sources and sinks it\u2019s able to guarantee at-least-once delivery without needing to persist messages during transit.<\/p>\n\n\n\n

It\u2019s simple to deploy<\/a>, comes with a wide range of connectors<\/a>, and is totally data agnostic, making it easy to drop into your existing infrastructure. Connect has functionality that overlaps with integration frameworks, log aggregators and ETL workflow engines, and can therefore be used to complement these traditional data engineering tools or act as a simpler alternative.<\/p>\n\n\n\n

Refer: https:\/\/docs.redpanda.com\/redpanda-connect\/about\/<\/a><\/p>\n\n\n\n

\n

NOTE: At the time of this writing (late August 2024), Redpanda has not yet converted all references to Redpanda Connect from Benthos and many updates are happening during this product integration phase. As a close partner of Redpanda, IntVerse regularly keeps up with changes and will be regularly updating this document – i.e., eventually this note will disappear! <\/em><\/p>\n<\/blockquote>\n\n\n\n

Step-by-Step Deployment<\/strong><\/h4>\n\n\n\n
1. Fetch the Helm Chart<\/strong><\/h5>\n\n\n\n

Use Helm to fetch the Benthos chart and untar it:<\/p>\n\n\n\n

helm fetch benthos\/benthos --untar<\/code><\/pre>\n\n\n\n
2. Review the Chart<\/strong>: <\/h5>\n\n\n\n
Navigate to the directory where the Helm chart is untarred. This directory will contain the Chart.yaml, values.yaml, and templates.<\/p>\n\n\n\n
cd benthos<\/p>\n\n\n\n
Upload Helm Chart Manifests to GitHub<\/strong><\/h4>\n\n\n\n
1. Create a Git Repository<\/strong>: <\/h5>\n\n\n\n
Set up a Git repository on GitHub if you don\u2019t have one.<\/p>\n\n\n\n
2. Add the Helm Chart<\/strong>: <\/h5>\n\n\n\n
Upload the Benthos directory (or relevant Helm chart files) to a new directory in your GitHub repository.<\/p>\n\n\n\n
git init<\/code><\/pre>\n\n\n\n
git add benthos<\/code><\/pre>\n\n\n\n
git commit -m \"Add Benthos Helm chart\"<\/code><\/pre>\n\n\n\n
git remote add origin https:\/\/github.com\/your-username\/your-repo.git<\/code><\/pre>\n\n\n\n
git push -u origin main<\/code><\/pre>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Modify ConfigMap on Bitbucket<\/strong><\/h4>\n\n\n\n
1. Access Bitbucket<\/strong>: <\/h5>\n\n\n\n
Go to your Bitbucket repository where the ConfigMap is stored.<\/p>\n\n\n\n
2. Edit ConfigMap<\/strong>: <\/h5>\n\n\n\n
Modify the ConfigMap as needed. For example, update benthos-config.yaml in your Bitbucket repository. Once we deploy the Benthos Application it will take input and output from the configmap we provided here. The config parameter should contain the configuration as it would be parsed by the Benthos binary.<\/p>\n\n\n\n
Integrating Benthos and Redpanda BYOC using ConfigMap:<\/strong><\/h4>\n\n\n\n
Modify Benthos ConfigMap to match our example topology. In this use case, we are using Kafka (Redpanda) as an \u201cinput\u201d and File as an \u201cOutput\u201d.<\/p>\n\n\n\n
Reference:
https:\/\/docs.redpanda.com\/redpanda-connect\/components\/inputs\/about\/<\/a><\/p>\n\n\n\n
\nhttps:\/\/docs.redpanda.com\/redpanda-connect\/components\/processors\/about\n<\/div><\/figure>\n\n\n\n
\nhttps:\/\/docs.redpanda.com\/redpanda-connect\/components\/outputs\/about\n<\/div><\/figure>\n\n\n\n
NOTE on this step:<\/strong><\/h4>\n\n\n\n
If you\u2019re seeing issues writing to or reading from Kafka with this component then it\u2019s worth trying out the newer kafka_franz input<\/a>. <\/p>\n\n\n\n
I\u2019m seeing logs that report\u00a0Failed to connect to Kafka: Kafka: the client has run out of available brokers to talk to (Is your cluster reachable?), but the brokers are reachable.<\/p>\n\n\n\n
Unfortunately, this error message will appear for a wide range of connection problems even when the broker endpoint can be reached. Double-check your authentication configuration and also ensure that you have enabled TLS<\/a> if applicable.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Commit Changes<\/strong>: <\/h4>\n\n\n\n
Commit the changes to Bitbucket.<\/p>\n\n\n\n
git add benthos-config.yaml<\/code><\/pre>\n\n\n\n
git commit -m \"Update Benthos ConfigMap\"<\/code><\/pre>\n\n\n\n
git push origin master<\/code><\/pre>\n\n\n\n
Create an ArgoCD Application for Benthos<\/strong><\/h4>\n\n\n\n
Create Application<\/strong>: <\/h5>\n\n\n\n
In the ArgoCD UI, click on +New APP. Choose EDIT AS YAML and use the following template:<\/p>\n\n\n\n
apiVersion: argoproj.io\/v1alpha1<\/code><\/pre>\n\n\n\n
kind: Application<\/code><\/pre>\n\n\n\n
metadata:<\/p>\n\n\n\n
  name: benthos<\/code><\/pre>\n\n\n\n
spec:<\/p>\n\n\n\n
  destination:<\/code><\/pre>\n\n\n\n
    namespace: benthos<\/code><\/pre>\n\n\n\n
    server: https:\/\/kubernetes.default.svc<\/code><\/pre>\n\n\n\n
  source:<\/p>\n\n\n\n
    path: redpandaconnect\/benthos<\/code><\/pre>\n\n\n\n
    repoURL: https:\/\/github.com\/Intverse\/benthos.git<\/code><\/pre>\n\n\n\n
    targetRevision: main<\/code><\/pre>\n\n\n\n
  sources: []<\/p>\n\n\n\n
  project: default<\/code><\/pre>\n\n\n\n
  syncPolicy:<\/p>\n\n\n\n
    automated:<\/code><\/pre>\n\n\n\n
      prune: true<\/code><\/pre>\n\n\n\n
      selfHeal: true<\/code><\/pre>\n\n\n\n
      allowEmpty: true<\/code><\/pre>\n\n\n\n
    syncOptions:<\/p>\n\n\n\n
      - CreateNamespace=true<\/code><\/pre>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Save and Create: <\/strong>Save and create the application.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Verify Deployment<\/strong><\/h4>\n\n\n\n
Check Logs<\/strong>: Monitor the logs of the Benthos pods to ensure that they are processing iterations correctly and that there are no issues with the deployment<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Check logs of pod\u00a0<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Streams mode<\/strong><\/h4>\n\n\n\n
Prepare ConfigMap<\/strong>: When running Benthos in streams mode, combine individual stream configuration files into a single Kubernetes ConfigMap. Ensure this ConfigMap is applied before deploying the Helm chart.<\/p>\n\n\n\n
Update values.yaml<\/strong>: Enable streams mode in values.yaml file:<\/p>\n\n\n\n
# values.yaml<\/code><\/pre>\n\n\n\n
streams:<\/code><\/pre>\n\n\n\n
 enabled: true<\/code><\/pre>\n\n\n\n
 streamsConfigMap: \"benthos-streams\"<\/code><\/pre>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Currently, the streams mode ConfigMap should be applied separately from and before installation of<\/strong> the helm chart; support for deploying additional ConfigMap’s within the chart may be implemented later.<\/p>\n\n\n\n
Created config.yaml file on bitbucket so we can explicitly able to give multiple inputoutput iterations for benthos.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Create an ArgoCD application for benthos config<\/strong>:<\/h4>\n\n\n\n
\n
Click on +NewAPP Then click on EDIT AS YAML, save and create the application<\/p>\n<\/blockquote>\n\n\n\n
apiVersion: argoproj.io\/v1alpha1<\/code><\/pre>\n\n\n\n
kind: Application<\/code><\/pre>\n\n\n\n
metadata:<\/code><\/pre>\n\n\n\n
 name: bent<\/code><\/pre>\n\n\n\n
spec:<\/code><\/pre>\n\n\n\n
 destination:<\/code><\/pre>\n\n\n\n
   namespace: rbenthos<\/code><\/pre>\n\n\n\n
   server: https:\/\/kubernetes.default.svc<\/code><\/pre>\n\n\n\n
 source:<\/code><\/pre>\n\n\n\n
   path: benthosconfig<\/code><\/pre>\n\n\n\n
   repoURL: https:\/\/github.com\/Intverse\/benthos.git<\/code><\/pre>\n\n\n\n
   targetRevision: main<\/code><\/pre>\n\n\n\n
 sources: []<\/code><\/pre>\n\n\n\n
 project: default<\/code><\/pre>\n\n\n\n
 syncPolicy:<\/code><\/pre>\n\n\n\n
   automated:<\/code><\/pre>\n\n\n\n
     prune: true<\/code><\/pre>\n\n\n\n
     selfHeal: true<\/code><\/pre>\n\n\n\n
     allowEmpty: true<\/code><\/pre>\n\n\n\n
   syncOptions:<\/code><\/pre>\n\n\n\n
     - CreateNamespace=true<\/code><\/pre>\n\n\n\n
Save and Create<\/strong>: Save and create the application.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Deploy Benthos Application<\/strong>:<\/h4>\n\n\n\n
Deploy Application<\/strong>: In ArgoCD, configure the repository URL to point to the Benthos Helm chart manifest file and ensure values.yaml has streams enabled.<\/p>\n\n\n\n
Create Application<\/strong>: Click on +New APP, choose EDIT AS YAML, and use the following template:<\/p>\n\n\n\n
apiVersion: argoproj.io\/v1alpha1<\/code><\/pre>\n\n\n\n
kind: Application<\/code><\/pre>\n\n\n\n
metadata:<\/code><\/pre>\n\n\n\n
  name: benthos<\/code><\/pre>\n\n\n\n
spec:<\/code><\/pre>\n\n\n\n
  destination:<\/code><\/pre>\n\n\n\n
    namespace: rbenthos<\/code><\/pre>\n\n\n\n
    server: https:\/\/kubernetes.default.svc<\/code><\/pre>\n\n\n\n
  source:<\/code><\/pre>\n\n\n\n
    path: redpandaconnect\/benthos<\/code><\/pre>\n\n\n\n
    repoURL: https:\/\/github.com\/Intverse\/benthos.git<\/code><\/pre>\n\n\n\n
    targetRevision: main<\/code><\/pre>\n\n\n\n
  sources: []<\/code><\/pre>\n\n\n\n
  project: default<\/code><\/pre>\n\n\n\n
  syncPolicy:<\/code><\/pre>\n\n\n\n
    automated:<\/code><\/pre>\n\n\n\n
      prune: true<\/code><\/pre>\n\n\n\n
      selfHeal: true<\/code><\/pre>\n\n\n\n
      allowEmpty: true<\/code><\/pre>\n\n\n\n
    syncOptions:<\/code><\/pre>\n\n\n\n
      - CreateNamespace=true<\/code><\/pre>\n\n\n\n
Save and Create<\/strong>: Save and create the application.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Verify Deployment<\/strong><\/h4>\n\n\n\n
Check Logs<\/strong>: Monitor the logs of the Benthos pods to ensure that they are processing iterations correctly and that there are no issues with the deployment.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Check logs of pod if it is taking multiple iterations or not:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Conclusion:<\/strong><\/h4>\n\n\n\n
By following this guide, you\u2019ve successfully deployed Redpanda Connect on an Amazon EKS cluster using ArgoCD, demonstrating the power and flexibility of combining these robust tools. Redpanda Connect\u2019s ability to handle complex data streaming tasks with ease, coupled with ArgoCD\u2019s seamless continuous delivery capabilities, makes for an efficient and scalable solution for modern data engineering challenges.<\/p>\n\n\n\n
This deployment process not only simplifies the management of your streaming data infrastructure but also ensures resilience and scalability, allowing your team to focus on building and optimizing data pipelines without the overhead of manual operations. With Redpanda Connect and ArgoCD, you\u2019re well-equipped to handle the demands of today\u2019s data-driven environments, ensuring reliable, real-time data processing across your organization.<\/p>\n\n\n\n
To further enhance your capabilities and overcome any data streaming challenges, consider leveraging the expertise of the IntVerse team. Their services can provide tailored solutions, ensuring that your Redpanda Connect deployment is optimized for your specific needs. With IntVerse by your side, you can confidently tackle complex data engineering problems, knowing you have the right support to maximize the potential of your data streaming infrastructure.<\/p>\n\n\n\n
Now that your setup is complete, you can explore further customization and optimization of your streaming data pipelines, backed by the expertise and support of the IntVerse team.<\/p>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
Overview Redpanda Connect, formerly known as Benthos (acquired by Redpanda), is a declarative data streaming service that solves a wide range of data engineering problems with simple, chained, stateless processing steps. It implements transaction-based resiliency with back pressure, so when connecting to at-least-once sources and sinks it\u2019s able to guarantee at-least-once delivery without needing to persist messages […]<\/p>\n","protected":false},"author":1,"featured_media":145,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"saved_in_kubio":false,"footnotes":""},"categories":[8],"tags":[11,10,9,12],"class_list":["post-35","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-redpanda-intverse","tag-eks-using-argocd","tag-intverse","tag-redpanda","tag-redpanda-connect"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.intverse.io\/blog\/wp-json\/wp\/v2\/posts\/35","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.intverse.io\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.intverse.io\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.intverse.io\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.intverse.io\/blog\/wp-json\/wp\/v2\/comments?post=35"}],"version-history":[{"count":8,"href":"https:\/\/www.intverse.io\/blog\/wp-json\/wp\/v2\/posts\/35\/revisions"}],"predecessor-version":[{"id":162,"href":"https:\/\/www.intverse.io\/blog\/wp-json\/wp\/v2\/posts\/35\/revisions\/162"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.intverse.io\/blog\/wp-json\/wp\/v2\/media\/145"}],"wp:attachment":[{"href":"https:\/\/www.intverse.io\/blog\/wp-json\/wp\/v2\/media?parent=35"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.intverse.io\/blog\/wp-json\/wp\/v2\/categories?post=35"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.intverse.io\/blog\/wp-json\/wp\/v2\/tags?post=35"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}