Skip to content
This repository was archived by the owner on Nov 16, 2023. It is now read-only.

Commit a001e98

Browse files
mtarngandrebriggs
mtarng
authored and
andrebriggs
committed
Migrating sayar/fabrikate-kafka-sample into a samples directory (#29)
* Initial commit * Initial fab components for kafka and schema registry. Added manifests for kafka cluster and schema registry user. Added Kafka namespace yaml. Added schema registry configuration using plaintext. Added network policies. Added azure-pipeline.yml for fab build and deploy. * Fix typo in README.md. * Using strimzi's network policies and cleaned up network policies. * Updated fabrikate component type for kafka operator. * Update README.md Closes #2 * Added Prometheus and Grafana for Observability (#4) * Added fabrikate component for prometheus and grafana * Added metrics configuration * added network policy for prometheus. * Added sample dashboards for kafka and zookeeper for grafana. * Added additional prometheus scrape config. * Updated the README * Use inline method. (#5) * Setup kafka connect * Initial support for portworx * Add manifests for kafka-connect and kafka-connect-user * Add sample mongoDB connector and a README file * Refactored and renamed in prep for @cnadolny's PR. (#7) * Refactored and renamed in prep for @cnadolny's PR. * Moved client helpers to a different folder to distinguish them from local tests scripts. * Change acl for the kafka connect user to a different topic * Adding manual instructions around portworx setup * Added mirror maker and documentation (#6) * Enable prometheus metrics for kafka connect (#10) * Commenting out portworx namespace and adding prometheus annotations * Unix base * Add setting for logs * Strimzi Kafka Cluster deployment liveness test * Refactoring portworx for easier navigation to deployment instructions * Adding additional details for the Portworx Documentation * DRYing livetest * portworx snapshot and backup guidance init * added zookeeper to allow full perf test testing * Clarifying comments in test * Working failover test, init snapshot test * Adding Loadbalancer without TLS for external accessibilty * External test running with kafkacat * Renaming livetest to internaltest * Refactored portworx, stable failover test, unstable backup test * Update strimzi-px-install.sh how to use env vars for secrets * typo 💬 * adding test color output. Also using external bootstrap for test * Modified tls ports for px-kafka brokers and docs correction * Change config for logs * First commit * Readme updates * Adding namespace * Set up CI with Azure Pipelines [skip ci] * Set up CI with Azure Pipelines [skip ci] * updated tests and added timeoout in between each * Add helm chart for component and config for kafka server logs * Add helm chart files * Add description to the fabrikate-kafka helm chart * Updating image * updating tests to use topic subdirectories. Also adding SSL support to external test * Cleaned up and added optional flag for TLS for externaltest * Adding optional config for tls enabled loadbalancer. Refactoring test README * Adding support for portworx volume encryption * replaced zookeeper with broker-list in perf tests * Adding Kafka Connect Test (#21) * Updating README to reflect mTLS support * adding kafkaconnect test components * fixing bug in internaltest * removing extra logs from test * Update README.md * Update README.md * Delete TODO.md * Update README.md * fixing broken link * Revised portworx failover test * Refactoring portworx-manual, updating docs for failover description * removed kubectl apply for unneeded yaml file * Updating portworx docs * Update README.md * Nesting fabrikate-kafka-sample code and documentation into samples/kafka-strimzi-operator * renaming to kafka-strimzi-portworx * adding newline at end of files * Fixing portworx config values
1 parent 8722a9a commit a001e98

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

48 files changed

+11129
-0
lines changed

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
12
# Binaries for programs and plugins
23
*.exe
34
*.exe~

‎samples/kafka-strimzi-portworx/.gitignore‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
test/temp
2+
generated/
3+
components/
4+
helm_repos/
5+
.DS_Store

‎samples/kafka-strimzi-portworx/LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2019 Rami Sayar
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎samples/kafka-strimzi-portworx/README.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# fabrikate-kafka-sample
2+
3+
This goal of this Fabrikate defintion is to provide an end to end example of running a secure stateful workload with backed by virtual storage on a single Kubernetes cluster.
4+
5+
## Overview
6+
![](images/Kafka-Sample.png)
7+
8+
The following components are deployed on to your AKS cluster:
9+
* Strimzi Kafka Operator (including Kafka Connect)
10+
* Confluent Schema Registry
11+
* Prometheus
12+
* Grafana
13+
* Stork & Portworx (Disabled by Default)
14+
15+
16+
## Installation Instructions
17+
1. Follow [instructions](/portworx-manual/README.md#setting-up-portworx-manually) to install Portworx manually on your Kubernetes cluster
18+
2. Use [Fabrikate](https://github.com/Microsoft/fabrikate) to install and generate this _high level definition_ to produce manifest yaml
19+
3. Install [Grafana dashboards](#setting-up-grafana-dashboards) to build your "pane of glass" to enable observability.
20+
4. Run [tests](/test/README.md) to validate your installation
21+
22+
## Validations
23+
24+
The repository provides multiple tests to help prove the end to end goal. We have tests that execute from within the cluster and from outside the cluster. Additionally load tests targeted at the virtual storage layer and end to end layer using TLS mutual authentication.
25+
26+
## Caveats
27+
28+
Currently Portworx must be installed manually due to the lack of a Helm chart. Please refer to [instructions](#installation-instructions) on how to successfuly install this sample.
29+
30+
## Future Work
31+
32+
- [ ] Work with Portworx on Helm Chart
33+
- [ ] Use a secret store type target of Key Vault for Portworx Volume Secret
34+
- [ ] Confifgure TLS & SSL with PX-Secure for Portworx Pods
35+
- [ ] User authentication for accessing Portworx volumes
36+
- [ ] Get past possible VNET issues in Terraform template
37+
38+
## Additional Notes
39+
40+
This sample also sets up a default scalable Kafka Cluster configuration (see [manifests/kafka-cluster.yaml](./manifests/kafka-cluster.yaml)) with persistent volumes, three replicas and TLS mutual authentication.
41+
42+
It also includes Kubernetes network policies to restrict traffic to the Kafka cluster (see [manifests/kafka-networkpolicy.yaml](./manifests/kafka-networkpolicies.yaml)). Only TLS traffic is permitted to the Kafka cluster. Plaintext is only permitted for the Confluent Schema Registry app.
43+
44+
## Setting Up Grafana Dashboards
45+
46+
The sample configuration provided does not expose grafana and prometheus metrics through an externally accessible IP. You may choose to create an external IP. Alternatively, you can connect to your cluster and port forward the grafana dashboard.
47+
48+
`kubectl port-forward [POD NAME HERE grafana] -n grafana 3000`
49+
50+
Browse to `localhost:3000/dashboard/import`.
51+
52+
There are six dashboards [Kafka, Kafka Connect, Zookeeper Metrics, Portworx Node, Portworx Volume, and Portworx Cluster] included in the `dashboards/` folder. You can either copy paste the content of the json files or use the "Upload .json File".

‎samples/kafka-strimzi-portworx/azure-pipeline.yml‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
trigger:
2+
- master
3+
4+
pool:
5+
vmImage: 'Ubuntu-16.04'
6+
7+
steps:
8+
- checkout: self
9+
persistCredentials: true
10+
clean: true
11+
12+
- bash: |
13+
curl $BEDROCK_BUILD_SCRIPT > build.sh
14+
chmod +x ./build.sh
15+
displayName: Download Bedrock orchestration script
16+
env:
17+
BEDROCK_BUILD_SCRIPT: https://raw.githubusercontent.com/Microsoft/bedrock/master/gitops/azure-devops/build.sh
18+
19+
- task: ShellScript@2
20+
displayName: Validate fabrikate definitions
21+
inputs:
22+
scriptPath: build.sh
23+
condition: eq(variables['Build.Reason'], 'PullRequest')
24+
env:
25+
VERIFY_ONLY: 1
26+
27+
- task: ShellScript@2
28+
displayName: Transform fabrikate definitions and publish to YAML manifests to repo
29+
inputs:
30+
scriptPath: build.sh
31+
condition: ne(variables['Build.Reason'], 'PullRequest')
32+
env:
33+
ACCESS_TOKEN_SECRET: $(ACCESS_TOKEN)
34+
COMMIT_MESSAGE: $(Build.SourceVersionMessage)
35+
REPO: $(MANIFEST_REPO)
36+
BRANCH_NAME: $(Build.SourceBranchName)

‎samples/kafka-strimzi-portworx/azure-pipelines.yml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Starter pipeline
2+
# Start with a minimal pipeline that you can customize to build and deploy your code.
3+
# Add steps that build, run tests, deploy, and more:
4+
# https://aka.ms/yaml
5+
6+
trigger:
7+
- master
8+
9+
pool:
10+
vmImage: 'ubuntu-latest'
11+
12+
steps:
13+
- script: echo Hello, world!
14+
displayName: 'Run a one-line script'
15+
16+
- script: |
17+
echo Add other tasks to build, test, and deploy your project.
18+
echo See https://aka.ms/yaml
19+
displayName: 'Run a multi-line script'

‎samples/kafka-strimzi-portworx/component.yaml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
name: "kafka"
2+
generator: "static"
3+
path: "./manifests"
4+
subcomponents:
5+
- name: "strimzi-kafka-operator"
6+
type: "helm"
7+
path: "strimzi-kafka-operator"
8+
method: "helm"
9+
source: "http://strimzi.io/charts/"
10+
- name: "confluent-schema-registry"
11+
type: "helm"
12+
source: "https://kubernetes-charts-incubator.storage.googleapis.com/"
13+
method: "helm"
14+
path: "schema-registry"
15+
- name: "prometheus-grafana"
16+
source: "https://github.com/microsoft/fabrikate-definitions.git"
17+
path: "definitions/fabrikate-prometheus-grafana"
18+
method: "git"
19+
- name: "kafka-cluster"
20+
type: "helm"
21+
source: ""
22+
path: "/helm-chart/kafka-cluster"
23+
method: "local"
24+
# - name: "portworx"
25+
# type: helm
26+
# source: https://github.com/portworx/helm
27+
# method: git
28+
# path: charts/portworx

0 commit comments

Comments
 (0)