Auto-Testing in ERP
In the Microsoft World
In theory, I completely agree with auto-testing. In the past, I’ve been on development teams that heavily leaned on unit and system testing. However, at least in NAV practice, I can see a huge imbalance between test maintenance and benefit.
Microsoft runs tests on all their supported NAV versions twice a day. With monthly cumulative updates and multiple language versions, there are hundreds of public branches of NAV actively being developed, built and delivered on a monthly basis at Microsoft; the main branch with roughly a million lines of code. Microsoft supports hundreds of databases used by over 100,000 clients. So, for Microsoft, a test plan is written once for each version, and, the benefit is enormous.
In a “Normal” World
A typical client’s database represents a single version being used by a single client. Test plans and scripts take the same amount of time to write whether it’s supported by one client or 200,000. For ArcherPoint clients, each test plan would take 10 to 100 times longer to create than manually testing the same functionality. The payoff would not come soon enough to realize any gains at all.
Compounding the uphill battle for auto-testing gains in a single user system is frequent customizations. Every time a customization is introduced, new tests will need to be added and old tests will break. I can’t imagine re-writing test plans during an upgrade. During the upgrade, the test phase is traditionally a manual process performed entirely by the clients’ key users. Theoretically, if we were to write auto tests for all of their functional processes, we could eliminate some of this test phase. However, at what cost? Certainly it would add cost to the upgrade project.
Auto-testing adds cost to an upgrade. It adds cost period. What is the gain and when is that gain realized?
Don’t get me wrong, I love writing auto test scripts. And there’s no better feeling than seeing the green light after running an auto-test that just posted 15,000 sales orders. But, is it worth the cost for our clients? Maybe in multi-tenant environments? Can we automate some, without adding too much cost? Can anyone see any low hanging fruit in auto-testing? Is auto-testing a good thing or not so much of a good thing? We’d love to hear your thoughts on this – please comment on this post below.
For more on testing in NAV (auto and manual), and NAV in general, please visit our resource center.