It is hard for me to think of a project which does not start out with requirements saying things like “The error messages produced by the system must be comprehensible to the user (unlike the existing system.)” And, given that system acceptance testing is usually based on making sure that the system works on systematic positive data but not complete negative data, this requirement falls through the cracks. So you can bet that in a few years time, when the system needs to be re-jigged or replaced, the customers will deliver the same requirements.
I am increasingly thinking that this kind of requirement sets the project up for a fall, because if the error messages are in fact vital to success, then they need to be provided by the business as an input for developers to use, rather than being something that developers make up. If it becomes the developers job to extract requirements, that significantly changes their role and the kind of stakeholder engagement they need. If the error messages are not inputs provided and signed-off by the stakeholders, then that must mean the developer has authority to make up their own minds what a “comprehensible” message is, and they only need to justify their approach not the individual messages.
From an Agile Scrum view, I am saying that error messages and diagnostics that users or operators will see should be managed by the Product Owner, and constitute part of what a “done” task requires, or be their own user story. The Product Owner needs to supply the details and liaise between the Development Team (who may ask the Product Owner “When this certain technical problem happens, how do we describe it to the user?” and the Product Owner may discuss it to get sign-off from the stakeholders before clarifying the user story (and tasks) with the new details.
Anyone who has written any size application knows that part of the architecture needs to be the status/notification/logging/dashboard/metrics component. But the failure to involve the stakeholder, to put make it their responsibility, is the root of the problem.
So I think Schematron’s aproach, where in a sense the assertion text comes ahead of the assertion’s XPath test, is an example of what is needed. At the least, the error strings should be exposed to be editable as a resource by non-developers. If you are using a Waterfall approach, then the Business Analyst should at least include a list of terminology and phrases for developers to use. If you are using an Agile approach then Developers should make sure that the specific errors and diagnostics are treated as the Product Owner’s responsibilities.
So a requirement like “The error messages produced by the system must be comprehensible to the user” should trigger a process that ends up with all the error messages and diagnostics specifically signed off by the stakeholder. Not a process where the acceptance testers form an opinion based on the errors and diagnostic messages that happened to be triggered during their testing. (Which is not to say that the error text is set in stone, of course.)