Skip to content

Auto-Testing: Good Thing or Not So Much of a Good Thing?


In the Microsoft World

In theory, I completely agree with auto-testing. In the past, I've been on development teams that heavily leaned on unit and system testing. However, at least in NAV practice, I can see a huge imbalance between test maintenance and benefit.

Microsoft runs tests on all their supported NAV versions twice a day. With monthly cumulative updates and multiple language versions, there are hundreds of public branches of NAV actively being developed, built and delivered on a monthly basis at Microsoft; the main branch with roughly a million lines of code. Microsoft supports hundreds of databases used by over 100,000 clients. So, for Microsoft, a test plan is written once for each version, and, the benefit is enormous.

In a “Normal” World

A typical client’s database represents a single version being used by a single client. Test plans and scripts take the same amount of time to write whether it’s supported by one client or 200,000. For ArcherPoint clients, each test plan would take 10 to 100 times longer to create than manually testing the same functionality. The payoff would not come soon enough to realize any gains at all.

Compounding the uphill battle for auto-testing gains in a single user system is frequent customizations. Every time a customization is introduced, new tests will need to be added and old tests will break. I can't imagine re-writing test plans during an upgrade. During the upgrade, the test phase is traditionally a manual process performed entirely by the clients’ key users. Theoretically, if we were to write auto tests for all of their functional processes, we could eliminate some of this test phase. However, at what cost? Certainly it would add cost to the upgrade project.

Auto-testing adds cost to an upgrade. It adds cost period. What is the gain and when is that gain realized?

Don’t get me wrong, I love writing auto test scripts. And there’s no better feeling than seeing the green light after running an auto-test that just posted 15,000 sales orders. But, is it worth the cost for our clients? Maybe in multi-tenant environments? Can we automate some, without adding too much cost? Can anyone see any low hanging fruit in auto-testing? Is auto-testing a good thing or not so much of a good thing? We’d love to hear your thoughts on this - please comment on this post below.

For more on testing in NAV (auto and manual), and NAV in general, please visit our resource center.




After further research into Auto Testing, the answer to my questions I asked in this blog "Is auto-testing a good thing or not so much of a good thing?" is: It's a good thing.

In fact, the cost of not having Auto Testing far out weighs the Cost of having Auto Testing. However, it is not just that easy. To fully benefit from Auto Testing you need to start practicing Test Driven developement, i.e., you start out creating tests. You integrate your code only if the test succeeds. So, there are some paradigm shifts in thinking that will be difficult to adopt in a team of developers that are used to doing it the wrong way. And by "the wrong way", I mean the way most developers have been developing for decades. The wrong way results in sprinkling bad code everywhere there is a customization to base NAV code, without any thought for upgradeability and testing.

The well-meaning developer has been trained over the years to follow MicroSoft's "Patterns" that they have presented to us, some decades old, in the form of well indented IF statements, long running Functions (CodeUnit 80) and ill-conceived, non-self-documenting Function names. Following these patterns by ultimately modifying them and conforming the modification to those patterns is an anti-pattern that I will call LemmingOffACliff Anti-Pattern. You should replace this bad modified code with Patterns that actually may at first seem as Anti-Patterns. But, because we are trying to make code more upgradeable and ultimately cheaper to maintain and scale into the future, we need to break the habbit of the LemmingOffACliff Anti-Pattern and start using what I call Ant-Patterns. These are Patterns that tread lightly on base NAV code. I will have a blog soon that will hopefully help all developers write good code.

Read ArcherPoint's Blog Follow us on Twitter Follow us on Facebook Follow us on LinkedIn Link to our RSS feed Join us on Google+ Watch us on YouTube