feat: end to end testing with local provider implementation

[Raffo]

What does it do ?

This pull request implements end to end testing by using a new custom local provider that saves records in memory. The idea behind this is that the new local provider, configured as a webhook, will create records in memory based on the usual Kubernetes resources. The same code exposes a minimal implementation of a DNS server. We can then query directly this minimal DNS server to verify that external DNS did its job. This idea allows us to implement end-to-end testing without having to issue certificates or creating real domains. The script can be launched with a bash script, and are wrapped in GitHub actions to be executed on GitHub.

Motivation

We have custom end-to-end testing infrastructure since almost the beginning of this project that I've been maintaining myself. That infrastructure has relied on an AWS account, and a private zone. Unfortunately, due to security requirements, I cannot make that infrastructure available to the bigger project, and that meant that I was always the bottleneck to run end to end tests. The motivation behind this work is to stop using that infrastructure and relying only on kind and local tests.

#5601

More

I went through a lot of ideas also using the /etc/hosts file which is not possible for a number of reasons in Kubernetes and other possible alternatives. This is what I landed to and the code is probably not perfect. I would love anyway for feedback and I will keep this and as draft until I will have received this feedback.

• Yes, this PR title follows Conventional Commits
• Yes, I added unit tests
• Yes, I updated end user documentation accordingly

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign szuecs for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coveralls]

Pull Request Test Coverage Report for Build 19195986294

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 69 unchanged lines in 5 files lost coverage.
• Overall coverage decreased (-0.05%) to 78.615%

---

Files with Coverage Reduction	New Missed Lines	%
openshift_route.go	1	79.49%
apis/externaldns/types.go	1	99.71%
source.go	3	83.33%
istio_gateway.go	8	90.1%
execute.go	56	64.72%

Totals
Change from base Build 18968339179:	-0.05%
Covered Lines:	16054
Relevant Lines:	20421

---

💛 - Coveralls

[mloiseleur]

For the feature, it would really help and improve quality to have CI like this 👍 .

For the implementation choice, I'm still unsure 🤔.

✔️ On one hand, having a small custom DNS implementation allows behavior flexibility and speed
❌ on the other hand, it will require us to implement every new feature we want to add in e2e testing and we may miss some specific details that will be in standard implementation.

💡 It remembers me that I've opened a PR about documenting usage of coredns for local dev testing (#5887).

@Raffo Did you consider using coredns as an option for the implementation ?

[Raffo]

@mloiseleur Yes, I considered it. Unfortunately coreDNS doesn't really work. It re needs a whole etcd server up and running. And that sounded more complicated than what we wanted to do. Remind that we're not testing a provider, any provider, not the coredns provider, not this fake provider, but we're testing the core of the project like the image can start and the plan works and and so on. Happy to change, but currently a bit unsure of the benefits as well 🤔