Skip navigation

Category Archives: Career

Now that I got your attention with the previous post, I will encourage anyone interacting with a vendor (my company included) to hold them accountable to deliver over the long haul.  As an industry both from vendor and consumer, we have to get better and here are a few thoughts that I encourage you to consider:

1 – Don’t let your vendor present to you. Make them whiteboard the solution.Scratch Pad P34

In my previous life I was a manager for an SE team and I invited one of my security “experts” into an important customer meeting. He had a set of slides that he had accumulated from corporate. He walked through the slides talking about the solution and then the customer asked a question, to which he said, “I don’t know.” The customer was polite enough to point out that his slide had the very answer, but they were looking for clarification. You see slides are the crutch of any organization. Slides are the marketing department run wild to make some of the most ugly solutions look like a work of art. They make the lipstick on the pig actually look attractive. They have animations and custom graphics that lure you into believing that everything works. After all if a company spent that much time and money on building these works of art, then the solution has to work equally as great – right? Nothing could be further from the truth. PowerPoint engineering is not a substitute for a workable solution – don’t fall for that trap. If the vendor really knows what they are doing, make them whiteboard the entire solution and talk to you along the way. Take away the crutch and see if they can run.

2 – Test the product at scale

Quite frankly anyone can make a four switch solution look to work well – but when you fully load it, things can go terribly wrong. Case and point, I was told about some testing the other day on another solution, where when the customer loaded up about one hundred mac addresses on a switch, periodically the switch would flood all traffic on all links. It was what appeared to be a lookup miss in the mac table lookup process. While you may not notice this with 10 connected hosts – I guarantee you will with 10,000 hosts. Decide just how big your network will become in terms of hosts, VLANs, VNIs, L3VPNs, VRF, routes, racks, BGP peers, MLAG’ed hosts, etc. Tell the vendor that you want to see this at simulated scale in their lab.   Do a proof of concept. Most vendors even offer this in a remote fashion where they build it and you can test/break it remotely. Make changes to the environment and see how it reacts. I encourage you to pull linecards, reboot switches, do upgrades, etc.

3 – Call their tech support with a problem

Have you ever been working late in the evening and hit a problem – so you call tech support because your change window ends in 2 hours and you have to get this working? You pick up the phone and call – you are subsequently put on hold waiting for the next available representative. Once you get to a live person, they are the equivalent of “reboot the box and let me know if the problem goes away.” 90 minutes into the call and 30 minutes prior to your change window expiring, you get to someone. By this time, you have to revert your change, mark it bad, and start to collect documentation to go before your change review board as to why the change was bad. To add insult to injury, three days later you get an email back from tech support as to the exact configuration change that is necessary to work around a bug you are hitting. HOW FRUSTRATING! I encourage you to call any vendor’s tech support and give them a question. Is the person answering the phone the one to answer your problem, or are they a glorified metric keeper – you know the ones that can say they answered your call within x number of minutes. I don’t need you to answer my call, I need you to fix my problem.

4 – See it, feel it, experience it

I once had a very smart executive tell me that “if I can’t see it, feel it, experience it, then it didn’t happen.” In other words, get the equipment in-house. Do things to it without the vendor around. Upgrade it and see how long it takes. Pull cables and see what happens. Change the link speeds, do all ports flap? Patch the operating system, extend the operating system, kill things, etc.   In addition to the product, I encourage you to talk to the leaders in the company. Do they take time to speak with you or are you not important enough to them? Find the people that build the product and software and ask them about it. I am always thrilled when I get to watch a vendor talk about their product like they are talking about their own child. Do their eyes light up when they discuss the greatness? Experience the solution in all aspects. It is only at this point that you can truly assess the product you are about to buy.

Yes, I know this is long and yes I know you may be tired of hearing from me by now – but we can be better. Yes I work for a vendor, something that I am proud of. No we aren’t the right product for everyone and I am glad to tell you that if it applies. However if my organization is not living up to this standard, we want to know because I can assure you that it is not what we are about, just go watch Ken Duda’s video on quality to see how we operate.

In the end, if you are unwilling to invoke any of the aforementioned suggestions, you can ride off into the sunset and be perfectly content with mediocrity. After all, my dog was perfectly fine with eating her own vomit and lived a long life that way. I did find that when we watched her closely we could dramatically short cut her illness if we simply removed her from the situation when it happened – are you willing to remove yourself from your current vendor when bad things repeatedly happen or will you continue to choke down the unpleasant result?

I had this dog. She was a good dog, but she did some things like all dogs do that disgust me. Every time she would get sick, we would have to run to find her and quickly pull her away from the vomit so she wouldn’t immediately lap it up again or the cycle would continue. Now that you are sufficiently disgusted with this behavior…This is the way that I feel when customers tell me that they continue to believe the promises of vendors that don’t deliver. They believe the PowerPoint engineering that are beautiful works of art – but nothing more. They believe the executive promises that they will get it right this time. When are we as an industry going to quit blindly believing what we are told and demand to see it, experience it, feel it, touch it, and break it?

I was presenting at a customer the other day and a short way into the presentation we got to talking about the problems that the customer had in their network. They were most interested in how our architecture and design specifically was going to solve their problems.  This particular customer was interesting as we talked through some of the challenges. You see it appears that they had been sold the proverbial “we will get it right eventually” tactic. I can’t tell you how frustrating it is to hear customers go through years of pain and agony. Years of changing architectures each with the promise of fixing all the things that the previous design couldn’t. Years of forklifting one platform for another because that one has a bug in the hardware or there is a new “Special” ASIC that they need to take advantage of. I even had a customer suggest that they were going to replace a vendor’s custom ASIC solution that had a HW bug with a brand new version of a different custom ASIC versus pushing towards an off the shelf merchant silicon solution. AHHHH – holding my tongue at this point was tough!

The few, the brave innovators of our industry, are willing to take the risk. Unfortunately they are only willing to do that after they have been burnt so many times that they have a learned response to their current vendor. When I walk into a conference room, I look for the most bleary eyed person in the room. That is the poor guy or gal that has to spend weekend after weekend troubleshooting problems, testing code version only to find YAB (Yet Another Bug), and doing painful upgrades in the hope that the next version will be better. They are the soldiers that get burnt out – but they are the ones that often hide the fact that the organization is eating its own vomit. Sure, just like with my dog, I sometimes make it in time to pull it away, while other times I am too late. These guys and gals are the equivalent, they hear that fateful sound of the network having problems and they devote their personal time and energy to fix the problem and limp along another day.  They are the heroes of the business and they deserve better!

In my next post, I will suggest ways that we can all avoid these pitfalls and make our networks better.  Stay tuned for Thursday’s post 🙂

I have had the opportunity over the last couple of weeks to deliver several SDN presentations to different audiences.  As I prepared for the presentation I kept coming back to one truth – the network has always been software defined.  What changed is that certain vendors decided to take that software and put a configuration interface on top of it that gave you limited access to the functions and capabilities of the devices.  In other words you could say that networking has predominantly been CLI defined or vendor defined based upon the features and functions that are exposed to the customer.

As I thought back to my career, the majority being a network engineer, I was struck by this construct.  When I was doing development every new problem could generally be satisfied by code…software…back then in an editor and compiler.  So why shouldn’t I be able to solve my networking problems the same way?  I understand that there are standards and protocols that help protect us from bad code by an end-user (because application developers always write perfect code…right?).  However if I know how I want to manipulate the flow or control plane – shouldn’t I be able to do it given the risks?  The answer previously has been “NO” – but that is changing.  With the advent of EOS and extensibility the ability to manipulate SysDB and the control plane in general has opened the possibilities.  I understand that other companies may be running to the “bolt-on” SDN approach where they build APIs that will interface with the hardware, but these will ultimately be flawed unless you truly have an architected operating system that interfaces in an OPEN way with the underlying hardware and software.

So I am not an SDN purist that thinks that Openflow=SDN.  In fact I am much more in the camp that if you have a highly programmable platform the agent that you use to push control plane information to the hardware doesn’t matter.  You could use Openflow, Closed Flow, Darrin Fl0w – it doesn’t matter.  What does matter is that you can translate the flow or forwarding entries into information that the hardware can then populate its various elements to switch the packet efficiently through the device.

In short – I came back to the networking industry to combine all the great things that I had learned in the application world over the last 5 years.  The intersection of SDN or Programmability with networking provides for endless opportunities as to where we can take application control of the environment.  You see somewhere networking lost its way and became CLI or Vendor defined networking…not any more.  The new revolution is to open this programmability for those that want it through the use of OPEN APIs and interfaces.  Now I can have a truly business defined network or customer defined network…all due to enabling the programmatic inclusion of the underlying hardware and software.

That’s all for now – off to Cleveland for two more fun days of customer meetings!

As I sit here on Spring Break in Florida, I am struck by what I was thinking last year about this time in terms of my career.  You see just about this time every year, my family with my wife and two daughters head south.   Last year (2012) was really no different, except that I have recently been assigned a new role in my job.

I had spent the previous 14 months working with Service Management and ITIL adoption throughout an organization of 12K plus employees and contractors.  I would routinely wear a bright orange sweater to work and as I walked into a room and someone would comment about how bright it was – I would say “I am from Service Management – don’t shoot!”  As I sat in my office one day near the end of the year, my boss walks in and says “we have a new opportunity for you,” which is always interesting.  I was being moved to a more business facing application area.  The area had a great leader named Jim that had stood up the function from the beginning.  It was known for its technical depth and the architecture team had two very seasoned individuals – of which I was to become their new leader.

As I went home that evening I felt every emotion that you could feel – from frustration to excitement.  Ultimately I would be leaving Service Management, which wasn’t all bad and moving to an application area to lead a team that knew much more than I ever did about developing application frameworks.

So back to my vacation in 2012, which was about 3 months after I had gotten this new role and I was enjoying the team, learning a lot, and was spending a lot of time leading, but not spending a lot of time exercising my creative ambitions.  So I have a constant search on a few job sites where I had query setup to match my ideal job.  As I was on vacation, one came across at a company I have long respected.  VMWare had a Cloud Solution Architect position available, where I could work from home and become an evangelist for cloud computing (something I love) and also work for a great organization.

I sat on the beach contemplating whether I should apply.  I have this philosophy that once I put my resume out – I am leaving, maybe not today, but at sometime in the near future.  I knew there would be no going back…so I sat in the sun for a week thinking and thinking and thinking…

On the way home I called a friend who worked for VMWare and he agreed to pass my name to those in the hiring position.  But if I was going to jump into this pond, I should look at the rest of the industry and at the time Software Defined Networking was all the buzz.  So I reached out to an old friend at Arista, who was making great waves with SDN – she just happened to be the CEO and someone else that I respect greatly.

Fast forward and suffice it to say that I decided to put my resume out…but nothing really materialized for some time.  This was unchartered territory for me.  Just as I recommitted to staying put, something came up.  Arista was growing like crazy and they were looking for more of an enterprise focused Solutions Architect (again my ideal job).  We talked and within a week I was on a plane for a quick overnight stay and interviews.  I got to meet with a lot of old friends and some new faces.  In total it was obvious that Arista was a special place, made up of people that wanted to make a difference in the networking world.

Fast forward to November in 2012 and I was in a place where it was time to resign from my position at State Farm.  It was possibly one of the hardest things I have ever done.  I would be leaving some great people that had spent a lot of time helping me grow.  Many of those people became great friends over the previous 5 years. Many were shocked, many were upset with me, and most were very supportive.  Opportunities of a lifetime don’t come every day and I was worried that if I didn’t sieze the opportunity – I would never have it again.  So as I end this post and contemplate the next one (which will be why Arista is such a great place) – I just want to encourage anyone out there that is considering making a move…don’t wait.  Pursue your dreams, pursue your innovative spirit, pursue excellence…you will never be disappointed for it is the chances that we don’t take that we ultimately regret!  One year after considering it – I am doing what I love with great people.  The choice is yours to do the same!

BTW – we are looking for great SEs in Minneapolis, Kansas and Ohio 🙂