Now that I got your attention with the previous post, I will encourage anyone interacting with a vendor (my company included) to hold them accountable to deliver over the long haul. As an industry both from vendor and consumer, we have to get better and here are a few thoughts that I encourage you to consider:
1 – Don’t let your vendor present to you. Make them whiteboard the solution.
In my previous life I was a manager for an SE team and I invited one of my security “experts” into an important customer meeting. He had a set of slides that he had accumulated from corporate. He walked through the slides talking about the solution and then the customer asked a question, to which he said, “I don’t know.” The customer was polite enough to point out that his slide had the very answer, but they were looking for clarification. You see slides are the crutch of any organization. Slides are the marketing department run wild to make some of the most ugly solutions look like a work of art. They make the lipstick on the pig actually look attractive. They have animations and custom graphics that lure you into believing that everything works. After all if a company spent that much time and money on building these works of art, then the solution has to work equally as great – right? Nothing could be further from the truth. PowerPoint engineering is not a substitute for a workable solution – don’t fall for that trap. If the vendor really knows what they are doing, make them whiteboard the entire solution and talk to you along the way. Take away the crutch and see if they can run.
2 – Test the product at scale
Quite frankly anyone can make a four switch solution look to work well – but when you fully load it, things can go terribly wrong. Case and point, I was told about some testing the other day on another solution, where when the customer loaded up about one hundred mac addresses on a switch, periodically the switch would flood all traffic on all links. It was what appeared to be a lookup miss in the mac table lookup process. While you may not notice this with 10 connected hosts – I guarantee you will with 10,000 hosts. Decide just how big your network will become in terms of hosts, VLANs, VNIs, L3VPNs, VRF, routes, racks, BGP peers, MLAG’ed hosts, etc. Tell the vendor that you want to see this at simulated scale in their lab. Do a proof of concept. Most vendors even offer this in a remote fashion where they build it and you can test/break it remotely. Make changes to the environment and see how it reacts. I encourage you to pull linecards, reboot switches, do upgrades, etc.
3 – Call their tech support with a problem
Have you ever been working late in the evening and hit a problem – so you call tech support because your change window ends in 2 hours and you have to get this working? You pick up the phone and call – you are subsequently put on hold waiting for the next available representative. Once you get to a live person, they are the equivalent of “reboot the box and let me know if the problem goes away.” 90 minutes into the call and 30 minutes prior to your change window expiring, you get to someone. By this time, you have to revert your change, mark it bad, and start to collect documentation to go before your change review board as to why the change was bad. To add insult to injury, three days later you get an email back from tech support as to the exact configuration change that is necessary to work around a bug you are hitting. HOW FRUSTRATING! I encourage you to call any vendor’s tech support and give them a question. Is the person answering the phone the one to answer your problem, or are they a glorified metric keeper – you know the ones that can say they answered your call within x number of minutes. I don’t need you to answer my call, I need you to fix my problem.
4 – See it, feel it, experience it
I once had a very smart executive tell me that “if I can’t see it, feel it, experience it, then it didn’t happen.” In other words, get the equipment in-house. Do things to it without the vendor around. Upgrade it and see how long it takes. Pull cables and see what happens. Change the link speeds, do all ports flap? Patch the operating system, extend the operating system, kill things, etc. In addition to the product, I encourage you to talk to the leaders in the company. Do they take time to speak with you or are you not important enough to them? Find the people that build the product and software and ask them about it. I am always thrilled when I get to watch a vendor talk about their product like they are talking about their own child. Do their eyes light up when they discuss the greatness? Experience the solution in all aspects. It is only at this point that you can truly assess the product you are about to buy.
Yes, I know this is long and yes I know you may be tired of hearing from me by now – but we can be better. Yes I work for a vendor, something that I am proud of. No we aren’t the right product for everyone and I am glad to tell you that if it applies. However if my organization is not living up to this standard, we want to know because I can assure you that it is not what we are about, just go watch Ken Duda’s video on quality to see how we operate.
In the end, if you are unwilling to invoke any of the aforementioned suggestions, you can ride off into the sunset and be perfectly content with mediocrity. After all, my dog was perfectly fine with eating her own vomit and lived a long life that way. I did find that when we watched her closely we could dramatically short cut her illness if we simply removed her from the situation when it happened – are you willing to remove yourself from your current vendor when bad things repeatedly happen or will you continue to choke down the unpleasant result?