Unsure about why migrating to terraform-plugin-testing break our tests

[remyleone]

Could you tell us a little bit more about the difference between what happens in the SDKv2 and what happen in the plugin-testing? I'm not sure about the difference in behaviour and what to watch out for.

I'm trying to migrate here: scaleway/terraform-provider-scaleway#3104

Because of the sweep redefinition problem, I think that I cannot migrate only a part of the code
Therefore I think we are stuck in a complete atomic migration that have to be done in one go.

terraform-plugin-testing version

	github.com/hashicorp/terraform-plugin-testing v1.13.2

Relevant provider source code

https://github.com/scaleway/terraform-provider-scaleway/pull/3104/files#diff-17c7c02df009425b1fe3afbb517e9cb2f40510a62d26c1455c6d80a0a57356f9

Terraform Configuration Files

All the one I had before

Expected Behavior

The CI should not break as terraform-plugin-testing should be totally retrocompatible with the code I already got. It should not affect the CI in any way to migrate to it.

Actual Behavior

Many of my jobs are broken. I'm not sure why changing the import path would trigger a fail in my test suite. Is there a change about how the testing it done between sdkv2 and plugin-testing ?

Steps to Reproduce

References

[austinvalle]

Hey there @remyleone 👋🏻 , apologies for the delayed response here.

You're correct that with that redefined flag problem you'll need to update all the import references to terraform-plugin-testing, along with the non-testing code references in SDKv2@v2.26.0 in the helper/retry and helper/id, in one go.

The only functional change made in terraform-plugin-testing since it was created was a bug fix made in v1.6.0 that removed an additional refresh, which with some provider implementations w/ TypeSet, would show up as a "plan not empty" error, see: #237 and #327.

If you think that might be a potential cause in your case, you could always try and install terraform-plugin-testing@v1.5.1, which was prior to that change.

---

I took a look at a couple of your CI logs in scaleway/terraform-provider-scaleway#3104 and I wasn't really able to pinpoint the set problem mentioned above or any other known SDKv2/Plugin-testing bugs, although the logs were very verbose so I was just pouring through the raw log file 😆.

Would you be able to isolate a couple of the tests that are failing in your CI and give us the specific errors produced by them so we can see if there are any other bugs we don't know about yet 🙂

[remyleone]

It seems to be working with the version you mentioned scaleway/terraform-provider-scaleway#3263