I'm part of a 20 odd group of developers who maintain around 7 components (websites and microservices) for a single domain in our company (e.g. delivery tracking domain).

In order to ensure quality, we have a "domain scoped end to end tests". Our E2E testing however has one problem: our E2E environment can have a component that's on a version that's different from production. The reason for this disparity is the components (e.g. microservice) have their own release schedules.

How do you handle this problem? My question is almost similar to this

Do you enforce that E2E must be run on the correct microservice versions? If so how?

Or do you accepted the fact that CI and production will never have identical microservice versions and just rely on monitoring production?

Lastly, how important is E2E in your CI/CD pipeline?

Note that I'm only interested in E2E testing of our microservices since it's impossible for me to control when a 3rd party service gets released.

Thanks

CodePudding user response：

Consider testing in general as a net that catches bugs, and e2e as another part of that net. One cannot expect to build an infinitely dense net that will catch all the bugs, and the marginal value decreases as we invest more effort in improving that net.

With these terms, what bugs are we trying to catch with the e2e part of the net, specifically with testing multiple microservices in different versions? We are probably focused on a class of bugs that emerge when two services fail to properly integrate after changes, or worse, some special edge case that was until now hidden but will be revealed by new behavior.

In any case, since the effort in e2e test is large, specifically around bringing up the environment properly, than its preferred to cover these bugs in different ways - API tests, integration tests between two services.

So actually, where are e2e tests useful? In my opinion, there strength is finding not subtle but major failures that can affect the quality of the entire system. I tend to focus my e2e efforts on covering as much of the architecture, such that each microservice, database, communication path, etc, is covered. Usually there is good correlation with feature groups in this aspect so there is also good coverage as far as product is concerned.

To summarize - should I care that I merged code that wasn't tested exactly how the production will be deployed? I don't think so - if there was a major failure, it would have been noticed with a different set of versions. If there was a subtle bug, it would have been caught by an integration or API test. If there's a really hard to catch bug, then I'll pay the price seeing it in production - which is the case sometimes.