Evaluation Hints and Tips
Finally we come to the evaluation event itself. A lot of preparation work has gone into it. If you have pitched your scripts correctly and you have a good supplier demonstrating a good product things should run very smoothly. Things become more difficult where the supplier hasn't prepared well, hasn't understood your requirements or is trying to sell you something that isn't a good match to your requirements. This is when your assessment starts to move into the grey areas. There are some key signs and phrases to watch out for:
"I'll need to switch to a different version to show you that." Always ensure you know what version of the product they are demonstrating to you and what version they intend to sell you. As a cautionary tale I once saw a demonstration of a product version that existed only on the salesman's laptop. He had undertaken a number of non-supported modifications in order to show that the product could meet our requirements.
"That will be in the next version." Don't take their word for it - ask to see the evidence. If they are clear enough about the next release to be able to say definitively what is in it then they will already have detailed design documentation and confirmed release dates that they can show you. "That functionality is currently under development; we don't yet have a release date for it" means nothing - treat it as vapourware.
"You can configure it to do that." You need to be clear about what is standard end-user configuration of the system, what requires a technical specialist and what is bespoke development. Don't go in for bespoke development unless you absolutely have to. It will cost you more, it won't be part of the standard support agreement and it will cause you ongoing maintenance headaches (see below).
"It's totally upgrade proof." Be especially cautious of this one if it follows the one about configuration. As a general rule bespoke work is never upgrade proof. Be wary about changes to screens or forms. End user acceptance of screen layout and terminology is a major issue for most organisations. Many suppliers will tell you how easy it is to rename something that doesn't suit your own terminology or change the layout of a screen but you could be creating a huge maintenance overhead for yourself.
"This is an optional feature." This one should ring a big warning bell. Firstly is it part of their product at all? Many suppliers will demonstrate third party tools which bolt on to their product without making this clear to you. Third party tools mean additional licence costs, possible differences in the technology stack and possible contractual and implementation complications if the supplier doesn't want to act as a "one-stop-shop" in providing the solution to you. If the optional feature is part of their own suite make sure they are quoting you the full cost of everything they are demonstrating.
"I'll show you what X customer has done." Integration with web technology allows many of these systems to offer great benefits but it also affords quite a few opportunities to pull the wool over unwary eyes. If they show you someone else's flash website or portal be sure to establish what you are seeing of the core product and what you are seeing of another customer's development efforts.
"Yes - it does that." This warning may sound overly cynical but if the phrase isn't accompanied by a demonstration it isn't worth much.
