Skip navigation

Leadership Lessons from Vacation

Every year I try to take an extended vacation of more than one week.  I do this as I find that it takes me more than a week to really get into “the groove” of vacation.  I set out with the normal family vacation goals, but I want to take some time in each location and look  for leadership lessons, to think about my leadership, and to put together a strawman plan for the coming year.  So in June we set out to visit Hawaii – particularly a week in Honolulu, Oahu, and a week in Maui.  Here are the lessons I learned from my time away in 2018:

1 – If you have never thanked someone who has served or prayed with/for someone who has given the greatest sacrifice for our country – you should!

I love history, but to see Pearl Harbor and her stories and even to set foot on the ships that fought to preserve our freedom – a huge thanks is in order.  We were out gunned, surprised, and by most accounts shouldn’t have recovered from that fateful day in December 7th, 1941.  Instead our service men and women fought with valor, honor and heroism.  I do love what our military does for us all. But for some reason this trip where you could be in the submarine that men served, on the bow of the battleship, and even seeing some of the ordnance that was used – made it so much more real. Inside the USS Missouri, it still smells of a school lunch room where men would go and eat a quick meal before returning to their posts.  So my first leadership lesson on this trip – is to remember to thank those that have had a profound impact on your life, maybe directly, maybe indirectly.  Even better yet – sit down and hear their stories…you won’t be disappointed.

2 – Time with our family is priceless

I spend a lot of my time on the road, in hotels, in airports…you get the picture.  This is a lot of time away from my wife and my two children.  Though they are beginning to understand the mutual sacrifice, the two weeks away with them was a time to reconnect.  My oldest daughter and I got to go surf together (she was much better at it than I).  My wife and I got to walk hand in hand down a beautiful beach remembering what it was like to be young and in love.  My youngest and I got to explore new places like “Dragon Teeth” and she got to explore and just be a kid.  Never forget the reason that you do a job…and they were all sitting next to me on the flight home!

3 – Never stop trying new things

We got to body surf in the surf on DT Flemming beach. We got to surf the waves.  We got to snorkel a beautiful reef and see fish that we could only see in pictures before.  We got to try all sorts of new food (though I missed the Huli Huli Chicken). All of it was amazing.  Live with the wonder of a child, never give up on trying new things – whether in life or in leadership of the team.

4 – Everyone should walk on the clouds at sunset once in your life

We went driving like crazy people up the side of the Haleakala Crater summit at sunset.  The switchbacks were many and the altitude was high.  We stopped eventually around 9500 feet to get some pictures of the sun setting and the cloud ceiling looking like you could walk out on it to touch the sun.

There is great beauty in so many things in life so drive like a madman to your own 9500 feet…but then stop and experience the beauty and take time to enjoy the view!

5 – Life will throw you many twists and turns

We drove the road to Hana and finished it up by coming around on the backside, which essentially takes you all the way around the mountain.  It was a harrowing journey of turns, switchbacks, one lane bridges, and some rude people along the way.

Much like our everyday lives, we will encounter stress so slow down and make the right decision as you go around the curve.  We sometimes will have an impasse, where someone will need to back up and show some grace as the other passes by on the one lane bridge.  We will encounter jerks that pass you in the most dangerous spots because they are in a hurry to get to the destination and you seemingly are in their way…just let them go by as they take their own lives in their hands.  But through all these challenges, you will experience breathtaking views.  Just remember as in leadership that it isn’t always about the destination – it is about the journey and what you see along the way!

6 – Work will survive without you

We all want to be indispensable.  But guess what?  The company will survive without you.  Yes, they may suffer a little while you are out, but if your company can’t survive without you for two weeks – you are no leader.  Never mind the time away will allow you to reflect on your own leadership and how you can be better – which will only make the company better.

7 – Always be planning your next move

I love what I get the privilege to do every day. That doesn’t mean that I can be on auto-pilot forever.  I need to think about my career.  I need to think about what I want to do or become in the future.  I need to set goals for my learning, advancement, and leadership.  I have made some of my most impactful career decisions while on vacation.  There comes a clarity while on vacation as to what you want to become – whether that involves your current role or a future one that you aspire to be in.

8 – Read…Read…Read…

I don’t get to do this as much as I used to. Previously I would pick up a book at the airport bookstore and read it on every plane ride…now I sleep on most plane rides.  For this trip, I recently picked a copy of one of my favorite magazines – Harvard Business Review.  I updated my kindle with a few books that looked interesting.  Vacation is a time while traveling to read…learn…refresh your knowledge…do this more often!

9 – Everything is better with friends

 The last two years we have traveled and spent some of our time away with friends, whether they be local to where we are traveling or if they are traveling with us. I can’t explain the difference that this makes. It turns an already good vacation in to a great vacation. Much like those that we work with – it is better to be with great friends rather than a bunch of strangers.  So pick your team with the same rigor – looking for people that you could hope to call friends one day…because everything is better when shared with friends!

10 – All things must come to an end

It sounds a bit morbid but let me explain.  My kids asked repeatedly if they could stay in Maui. I responded with “unless you want to become farmers and your mother and I become full time Uber drivers” we need to go back.  All things come to an end.  Your vacation comes to an end, your job will eventually come to an end, and yes even your life (barring any cryogenic freezing of our brains) will come to an end – that is a constant that we can plan on.  What you can control is how you spend your time here.  Love deeper, experience the unknown, try things and fail, try things and succeed, and drive the winding road…but take in the views. We were all brought into this world without any of our own doing and we will likely leave it the same way.  The story between the bookends is ours to decide…what will yours say?

The Networking Hierarchy of Needs with Intent-Based Networking

Given all the information out there this week about Intent-Based Networking Systems (IBNS), I always think it is good to separate the “martketecture” from the reality. We are embarking on a new era that could rival that of the days of SDN as the “future of networking.” We will see more “PowerPoint engineering” in the coming weeks with the associated tweets and blog posts that will make even the most skeptical network engineers question their protocol knowledge and existence.

Let’s start with what is “Intent-Based Networking?” I actually love famed Gartner Analyst, Andrew Lerner’s definition of what is an Intent-Based Networking System:

  1. Translation and Validation– The system takes a higher-level business policy (what) as input from end users and converts it to the necessary network configuration (how). The system then generates and validates the resulting design and configuration for correctness.
  2. Automated Implementation– The system can configure the appropriate network changes (how) across existing network infrastructure. This is typically done via network automation and/or network orchestration.
  3. Awareness of Network State– The system ingests real-time network status for systems under its administrative control, and is protocol- and transport-agnostic.
  4. Assurance and Dynamic Optimization/Remediation– The system continuously validates (in real time) that the original business intent of the system is being met, and can take corrective actions (such as blocking traffic, modifying network capacity or notifying) when desired intent is not met.

Even in a recent article Lerner goes on to describe a vendor’s recent IBNS launch:

“It’s a platform that should enable intent driven network management in the future,” he says. “Except for some discrete, tight use cases around configuration, it’s not quite completely glued all together yet.”

So there you have it, vendors already claiming that they have an IBNS solution – which is really a bunch of disjointed parts – sound familiar? Aren’t we really jumping the gun? The same thing happened with Software Defined Networking – everyone claimed to have an SDN solution. In fact a common question I would get is “what is your SDN solution?” Ugh, let’s play buzzword bingo and see who is left standing.

As a kid one of my favorite shows was Looney Tunes where Wile E. Coyote would chase the Road Runner endlessly utilizing every method that he could to catch that bird.

In the picture above, this was the standard approach – he would lite the fuse on the rocket to go fast only to blow up when that approach reached its conclusion. You see Wile E. had a great intent – to catch the Road Runner. However he was operating from a flawed platform and approaches. You know how the story goes – again and again he tries and fails, at times blowing himself up, other times he would fall off a cliff. The moral of this story today is that his intent was great and maybe even the method seemed like a good idea, but he lacked the basic understanding of how things worked in his endeavor.

Another popular perspective is Maslow’s Hierarchy of Needs that was updated around 1954. Maslow’s theory suggests that the most basic level of needs must be met before the individual will strongly desire the secondary or higher level needs

Being a bit of a psychology buff, this is fascinating to me because it very simply describes human behavior. I began to wonder if in fact that we could apply the “Hierarchy of Needs” to other parts of life.

So after much thought in the Intent-Based Networking space combined with my history of Service Management, Application Development, and 20 years in Networking, I decided to propose “DT’s Hierarchy of Networking Needs”

Let me explain each of these layers below:

Quality System

You have to start with Quality! Recently I had a customer comment to me that it was beautiful that when things broke, they broke and recovered the way we expected. You see things are going to break, bugs are going to happen, and even cosmic rays are going to cause SEUs. If a vendor tells you that things like this won’t happen they are lying to you. It all starts with having a good quality system that you ideally can update and patch without taking down the entire system. It begins with confidence that when you do an upgrade that you aren’t going to step into the quagmire of new bugs that lie in waiting for you to trip on them. As Ken Duda once said “If the network isn’t working then nobody is working!” So the basic principle that aligns to that of Maslow is you have to have the physiological needs met – in this case, high confidence that things work as expected with high quality.

Basic Features/Capabilities

Anyone can build a switch that can pass packets between two interfaces – if you can’t you don’t deserve to be in networking. After you can pass packets, you have to be able to do it under stress and scale. Then you start to organize things into a full system where devices talk to other devices. In order for this to work, you need to have some level of standards based protocol support such that everyone plays nice in the sandbox. Let’s face it, if you need STP support and one vendor doesn’t have it, you are up the proverbial “creek without a paddle.” Likewise if you are routing and you need BGP to handle the scale of routing – then that becomes a basic functionality. You must have a basic set of features and capabilities to make a system interoperate correctly.

Telemetry/Visibility/Management

Next after building a network where everything works together, you need to be able to manage the system for when things do happen. Those things could be misbehaving applications, cables going bad, NICs acting up, and even the occasional bug. Being able to see what is going on in the network is essential. We are moving in a new direction with things like streaming telemetry, which will be the death of SNMP (finally). This telemetry will also be a basis for us to engage in machine learning in the future as a critical data feed. You also need to be able to extract not just control plane information but you must be able to mirror ports and aggregate these mirrored ports for visibility for security, inspection, and troubleshooting. So the third need becomes the Telemetry/Visibility/Management of the network.

Programmability

The era of SDN ushered in new forms of programmability and automation that had been unheard of previously in the networking arena. You should be able to run your network with similar systems to managing your servers – in reality it is Linux under the cover so why wouldn’t you? We need to be able to inject our programmatic will into the control plane to steer packets in ways that are not in the natural order, we must be able to query the systems and leverage this information to provide proactive notifications of problems before they ever happen. Programmability goes way beyond that of a CLI – in fact I would say that the CLI will slowly age out in favor of programmatic/API integration into the network. The network is yours, so shouldn’t you be able to do with it what you want, versus a vendor telling you “only do it this way?”

Intent-Based Networking

We arrive at the top of the pyramid where we have reached the main goal in our networking lives – that of signaling the intent to the network and having it “auto-magically” instantiate this optimized network that has the security policy that you intend and it is automated in every way thinkable. Yes, I agree that it sounds like a utopia that we all would like to live in…

BUT – can you image trying to instantiate a network configuration when you have no confidence that the next bug you are going to trip over will loop your overlay and crush your users? Can you imagine trying to instantiate this network without well-formed APIs that cross vendor boundaries to make sure one vendor interoperates the same way with another? How about instantiating a network that is opaque in its monitoring and management capabilities? The reality here is that we have a lot of work as an industry to get the first 4 layers of needs satisfied. Yes I have an opinion as to what my current employer is doing and I do believe it is spectacular to get those pieces in order in a way that doesn’t lock a customer in. Customer’s should want to stay with a vendor, not because of some Vendor Defined Networking model but because they do have the highest quality in the industry, with the necessary features needed, combined with great telemetry and visibility, AND for the rainy day where I want to do something spectacular – I can program my way to the finish. This in my mind is where we as an industry must focus. It seems easy to create PowerPoint slides and videos of how this IBNS world will solve everything, but I ask you first to train and run the race – don’t just step over the finish line and declare victory!

 

Stop the dog from…

Now that I got your attention with the previous post, I will encourage anyone interacting with a vendor (my company included) to hold them accountable to deliver over the long haul.  As an industry both from vendor and consumer, we have to get better and here are a few thoughts that I encourage you to consider:

1 – Don’t let your vendor present to you. Make them whiteboard the solution.Scratch Pad P34

In my previous life I was a manager for an SE team and I invited one of my security “experts” into an important customer meeting. He had a set of slides that he had accumulated from corporate. He walked through the slides talking about the solution and then the customer asked a question, to which he said, “I don’t know.” The customer was polite enough to point out that his slide had the very answer, but they were looking for clarification. You see slides are the crutch of any organization. Slides are the marketing department run wild to make some of the most ugly solutions look like a work of art. They make the lipstick on the pig actually look attractive. They have animations and custom graphics that lure you into believing that everything works. After all if a company spent that much time and money on building these works of art, then the solution has to work equally as great – right? Nothing could be further from the truth. PowerPoint engineering is not a substitute for a workable solution – don’t fall for that trap. If the vendor really knows what they are doing, make them whiteboard the entire solution and talk to you along the way. Take away the crutch and see if they can run.

2 – Test the product at scale

Quite frankly anyone can make a four switch solution look to work well – but when you fully load it, things can go terribly wrong. Case and point, I was told about some testing the other day on another solution, where when the customer loaded up about one hundred mac addresses on a switch, periodically the switch would flood all traffic on all links. It was what appeared to be a lookup miss in the mac table lookup process. While you may not notice this with 10 connected hosts – I guarantee you will with 10,000 hosts. Decide just how big your network will become in terms of hosts, VLANs, VNIs, L3VPNs, VRF, routes, racks, BGP peers, MLAG’ed hosts, etc. Tell the vendor that you want to see this at simulated scale in their lab.   Do a proof of concept. Most vendors even offer this in a remote fashion where they build it and you can test/break it remotely. Make changes to the environment and see how it reacts. I encourage you to pull linecards, reboot switches, do upgrades, etc.

3 – Call their tech support with a problem

Have you ever been working late in the evening and hit a problem – so you call tech support because your change window ends in 2 hours and you have to get this working? You pick up the phone and call – you are subsequently put on hold waiting for the next available representative. Once you get to a live person, they are the equivalent of “reboot the box and let me know if the problem goes away.” 90 minutes into the call and 30 minutes prior to your change window expiring, you get to someone. By this time, you have to revert your change, mark it bad, and start to collect documentation to go before your change review board as to why the change was bad. To add insult to injury, three days later you get an email back from tech support as to the exact configuration change that is necessary to work around a bug you are hitting. HOW FRUSTRATING! I encourage you to call any vendor’s tech support and give them a question. Is the person answering the phone the one to answer your problem, or are they a glorified metric keeper – you know the ones that can say they answered your call within x number of minutes. I don’t need you to answer my call, I need you to fix my problem.

4 – See it, feel it, experience it

I once had a very smart executive tell me that “if I can’t see it, feel it, experience it, then it didn’t happen.” In other words, get the equipment in-house. Do things to it without the vendor around. Upgrade it and see how long it takes. Pull cables and see what happens. Change the link speeds, do all ports flap? Patch the operating system, extend the operating system, kill things, etc.   In addition to the product, I encourage you to talk to the leaders in the company. Do they take time to speak with you or are you not important enough to them? Find the people that build the product and software and ask them about it. I am always thrilled when I get to watch a vendor talk about their product like they are talking about their own child. Do their eyes light up when they discuss the greatness? Experience the solution in all aspects. It is only at this point that you can truly assess the product you are about to buy.

Yes, I know this is long and yes I know you may be tired of hearing from me by now – but we can be better. Yes I work for a vendor, something that I am proud of. No we aren’t the right product for everyone and I am glad to tell you that if it applies. However if my organization is not living up to this standard, we want to know because I can assure you that it is not what we are about, just go watch Ken Duda’s video on quality to see how we operate.

In the end, if you are unwilling to invoke any of the aforementioned suggestions, you can ride off into the sunset and be perfectly content with mediocrity. After all, my dog was perfectly fine with eating her own vomit and lived a long life that way. I did find that when we watched her closely we could dramatically short cut her illness if we simply removed her from the situation when it happened – are you willing to remove yourself from your current vendor when bad things repeatedly happen or will you continue to choke down the unpleasant result?

Like a dog eating it’s own vomit!

I had this dog. She was a good dog, but she did some things like all dogs do that disgust me. Every time she would get sick, we would have to run to find her and quickly pull her away from the vomit so she wouldn’t immediately lap it up again or the cycle would continue. Now that you are sufficiently disgusted with this behavior…This is the way that I feel when customers tell me that they continue to believe the promises of vendors that don’t deliver. They believe the PowerPoint engineering that are beautiful works of art – but nothing more. They believe the executive promises that they will get it right this time. When are we as an industry going to quit blindly believing what we are told and demand to see it, experience it, feel it, touch it, and break it?

I was presenting at a customer the other day and a short way into the presentation we got to talking about the problems that the customer had in their network. They were most interested in how our architecture and design specifically was going to solve their problems.  This particular customer was interesting as we talked through some of the challenges. You see it appears that they had been sold the proverbial “we will get it right eventually” tactic. I can’t tell you how frustrating it is to hear customers go through years of pain and agony. Years of changing architectures each with the promise of fixing all the things that the previous design couldn’t. Years of forklifting one platform for another because that one has a bug in the hardware or there is a new “Special” ASIC that they need to take advantage of. I even had a customer suggest that they were going to replace a vendor’s custom ASIC solution that had a HW bug with a brand new version of a different custom ASIC versus pushing towards an off the shelf merchant silicon solution. AHHHH – holding my tongue at this point was tough!

The few, the brave innovators of our industry, are willing to take the risk. Unfortunately they are only willing to do that after they have been burnt so many times that they have a learned response to their current vendor. When I walk into a conference room, I look for the most bleary eyed person in the room. That is the poor guy or gal that has to spend weekend after weekend troubleshooting problems, testing code version only to find YAB (Yet Another Bug), and doing painful upgrades in the hope that the next version will be better. They are the soldiers that get burnt out – but they are the ones that often hide the fact that the organization is eating its own vomit. Sure, just like with my dog, I sometimes make it in time to pull it away, while other times I am too late. These guys and gals are the equivalent, they hear that fateful sound of the network having problems and they devote their personal time and energy to fix the problem and limp along another day.  They are the heroes of the business and they deserve better!

In my next post, I will suggest ways that we can all avoid these pitfalls and make our networks better.  Stay tuned for Thursday’s post 🙂

You get what you measure…even if it is the wrong thing!

You may be reading this post thinking it is about some management philosophy – sorry to disappoint you!  This post has a two-fold purpose 1 – as a response to an article I read last week  (https://www.linkedin.com/pulse/why-network-industry-has-been-stuck-1980s-ciscos-embrace-joe-howard) and 2 – my new role that I picked up in the last month.

For those that don’t know, I devoted a lot of my pre-children days to certification (which I highly encourage anyone getting into IT to do).  I posted previously that I got all the Microsoft and Novell Certifications that I could and I wanted a new challenge, so I undertook the R&S CCIE.  This was back in the days of the 2 day lab and scenarios that at that time were highly applicable to what I was doing day in and day out.  By the time I decided to take the Security IE, it had switched to a one day exam that was more of a “stump the chump” mentality in that what was tested was very rarely implemented in networks but was a lot of required learning nonetheless.

As I read the above post from Joe Howard (which is a very good post) – I was struck with what I get to experience many days in my current job.  I get to go visit a lot of customers and no less than once a week I hear about some crazy design that makes no sense and in reality reminds me of a CCIE lab – overly complex to solve a relatively simple problem.  What I believe happened is that customer networks have slowly become the CCIE’s practice lab – not from solving problems but from creating unnecessary complexity to ensure job security and/or prepare the individual to pass the lab.  At a minimum the art of simplicity was thrown out as the first step in solving a problem.  This in turn creates a patchwork of design modifications and point in time fixes that over the course of time becomes a big ball of mud.

I was at a large customer the other day that I have worked with over the years.  They had the simple edict – no proprietary protocols EVER.  This adherence to discipline in itself helped keep the network clean and sleek.  As people have moved on, proprietary things have moved in and complexity ensued locking in yet another customer.  I talked with another customer the who had a full MPLS network inside of their relatively small datacenter.  I was shocked.  When I asked why – there was no valid reason that VLANs couldn’t have solved his problem.  Combine this with yet another example of a customer that implemented hundreds of static routes throughout the network and complained that he could never go on vacation in fear that something would go wrong and he wouldn’t be there to fix it.  We have got to stop building complex networks in the name of job security and training for our IE lab. I believe that the IE has made networks a work of art versus solving the business demands.  The industry is moving on and some of the most beautiful designs I see are those of the cloud providers who are incredibly large, but incredibly simple.  This equals more stability and greater ability to orchestrate everything they do.  Several that I spoke to don’t ever plug a laptop into the devices, everything is automated.  If it can’t be automated it doesn’t go in.  Complexity is very very hard to automate.  To drive my point home about vendor introduced unnecessary complexity, I went to a few different vendor’s sites the other day and one had over 100 data center design guides in some flavor – how in the heck do you decide how to put that all together? No wonder it takes an IE level person to make this stuff work.  The industry can do better here!

Now to my second quest.  I recently took over the leadership for training and certification at Arista.  Working with some great people like Gary Donahue who is a legend in the networking industry with his Network Warrior and Arista Warrior series of books.  Gary and I were talking about launching the next line of Arista professional training and exams – one which will be an “Expert” level lab exam.  What I don’t want the exam to be is a reason for someone to introduce unnecessary complexity into a production network to show how smart they are. We are thinking that half of the exam will be about deep knowledge of protocols and building data center networks, but the other half will be about automating and orchestrating capabilities thus forcing simplicity in design. We plan to include items like leveraging the unmodified Linux capabilities to solve real problems, extensibility in fixing and introducing new capabilities, and then of course troubleshooting, and last but not least, using DevOps to detect and fix the problems before they ever cause a problem.  Yes we want to unleash the proverbial “Chaos Monkey” in the lab and see how well the network stands up.

Network

We are early in deciding what the lab will become.  We are getting together some of the foremost experts in the company to help us build scenarios and ultimately the lab.  I won’t apologize for it being brutal.  However my hope is that it is a useful brutal to drive simplicity versus a reason to implement unnecessary complexity in real networks with little return.

Yes the journey for both of my CCIE’s was worth it.  I learned a ton and implemented much of my routing and switching IE in real life.  It is time for the next generation of learning and development where we encourage creativity – but creativity starting with simplicity to solve the problem.  In the end if you can’t orchestrated it and maintain it – you might want to think about a different approach to solving the problem.

Why Compromise?

I have spent nearly 20 years in networking. In fact as I was cleaning out my store room last night I found a reminder of one of the early certifications…my Novell CNE 5 (I have a 4.11 cert somewhere).

Novell

This is where I cut my teeth – building networks large and small. Building servers that at the time were the best in the business (Compaq Proliant).  At that time 10-BaseT networking was new and I even did some 100VG-AnyLAN.  Early in my career I was known for cleaning up the messes. I used to joke that I was going to buy a fireman suit because I was always putting out the fires. The reason I was always called in to fix the mess, was that I was unwilling to compromise. I was unwilling to do half the job that needed to be done. I was unwilling to walk away with something broken. My parent’s called it stubbornness, I called it character 🙂

Fast forward almost 20 years and a few management jobs later including a stint at a 90 year old company and I found myself asking why? Why was I willing to compromise to get something lesser than what I know was the right answer. Somewhere along the way we let mediocre seep into what we do and then mediocre becomes the norm.

One of the early requirements of my networking career was “no proprietary protocols.”  As much as I would like to say this was my idea, it was actually the idea of one of my customers who at the time had the largest private WAN in the world. A WAN that I was fortunate enough to design and build twice in my tenure (over 9 years)…all with no proprietary protocols. We even went so far as to implement RIP on a large scale because we needed something lightweight for dial-backup and EIGRP would have been easier. I have had other customers over the course of time that this requirement was softened and then they built something that only one vendor can provide…and then they are stuck. I was at a different customer this last week and they told me that they had a vendor that convinced them that EIGRP was the best thing…so they listened. Two years later they were basically held hostage in pricing because they had implemented something that only one vendor could do.

So back to my why compromise? The pressures being placed on the network today are bigger than ever before. The people requirements are getting even worse. Forbes estimates that “75% of IT time and activity are spent keeping the lights on.” An Avaya survey reveals 80% percent of companies lose revenue when the network goes down (pretty sure it is closer to 100% today). They also found that 1 in 5 companies fired an IT employee as a result of network downtime. So we are asking people to do more with less. The network being down in the financial area can cost an average of $540,358 per incident…but yet we ask employees to compromise. Would you be willing to choose a mediocre solution if your job depended upon it? Would you want options to hold your vendors accountable to one another? Would you throw a networking company out for poor code or product quality? If you choose a technology that is not 100% interchangeable – you will.

Dilbert

If you look at network silicon pre-2006 – speeds weren’t increasing with Moore’s law? This is because custom silicon (that built by the networking vendors) was focused on features. Feature keep you locked in…speed is fleeting. Fast forward to today and with the inclusion of merchant silicon we can now see the speed increases (chart below):

10GChart       Mooreslaw

10G density is on the same trajectory as Moore’s and the gaps in features used are getting less and less. So the silicon is the same, the features are similar, how do you differentiate? The answer in my mind is quality. Quality of the software, quality of the support, quality of the hardware, quality of the company. Don’t get me wrong, some vendors have the ability to get more out of the same chip than others – and you should evaluate that…but if you are 1 in 5 employees that is about to lose your job, shouldn’t you care about quality of what you are building?

If I am building a Data Center today, here are the criteria:

  • Highly Redundant and Available with 6x9s uptime (most apps want 5x9s)
  • Highly Automated and orchestrated from day 1 with minimal human CLI interaction
  • Vendor agnostic – where I can swap any spine group or leaf group out for another vendor without losing functionality
  • FAST – I want to upgrade in place when new speeds are available without a new architecture
  • Extensible to solve my business problems. As a vendor you build the network, as a consumer I know what I need it to do, so don’t limit me from doing it
  • Cost effective – I am willing to pay more for these things, but there should be competition to keep the price competitive
  • Simple – I shouldn’t have to retrain my people to make a change

I am sure there are others that you all would include.  This is a compilation of what I see in the Cloud providers and the most critical enterprises. However, if you are putting a bullet in the chamber and spinning it roulette style, then as networking professionals we should be unwilling to compromise from building what is the best.

I tweeted out a few months ago that “when politics and PowerPoint win – network engineers lose.” Nothing could be more true!  I encourage you to listen to networking vendors, but also encourage you to test it for yourself.  It is only then that you can tell the difference between PowerPoint engineering and true quality engineering and thus decide if your are compromising or not.

Have a happy day!

 

Software Defined Everything

I was recently asked by a customer to give my perspective on a number of topics including Software Defined Everything.  I will be making a set of blog posts on this and other industry perspectives in the coming weeks.  For now – let’s talk Software Defined Everything!

For those running in IT circles there are many different categories of “Software Defined.” Today there is software-defined networking (SDN), software-defined storage (SDS), and software-defined data center (SDDC) to name a few.

One of the best lines that I have ever heard was from Carl Eschenbach – VMWare’s President and Chief Operating Officer – “Software doesn’t run on software the last time that I checked!”  This is very true.  However software does give us a degree of nimbleness that can ultimately help with solving the business problems.  In fact if you look at the Cloud Computing models that are out there today – they are largely due to the agility that comes with software.  At a recent event I saw a slide from IDC that accentuated the value of software and openness (recreated below)

StartingTech

The reality is that software coupled with Cloud Computing has enabled new business models like no other.  As I visit with many different cloud providers even their own models focus on the ability to capitalize on agility that they have and their competitors don’t.  Specifically customers tell me that Software Defined Networking is about “better orchestration and operational efficiency.”  This is also synonymous to openness to choose the best solution in the network.  Let’s face it – no single vendor is capable of being #1 in all areas.

The other idea around SDE is that you can completely abstract the software from the hardware.  While this has been tried with OpenFlow, we haven’t seen it take off in the general enterprise.  We also see this with the idea of “white box” switches that run a version of a Network OS.  The challenge I see here is how do you handle problems.  For instance –

Quanta LY6 switch has memory parity error _soc_mem_array_sbusdma_read: L2_ENTRY.ipipe0 failed(ERR)

Is this a hardware error or a software error?  Obviously this is on a Quanta white box switch.  How should the software react to this parity error?  Is it a single event upset or a more troubling error?  In the end parity errors suck!  Yes I know that cosmic radiation taking your network down sounds like a story in a science fiction novel, but they do happen.  In fact in networks with 15-20K devices they are likely to happen to the tune of 1-2 a week and that is assuming that you have built the device to be as protected as possible.  The reality is that software needs to account for these errors and minimize their impact.  Otherwise your switch is rebooting or worse yet – hung!  So as much as we would like to completely have the hardware abstracted from the software in network switches, the blast radius is often too large – where if you lose a TOR switch at least 48 physical hosts are affected.  Worse case the switch having a parity error causes instability elsewhere.  Think for a second how much of an impact a chassis that experiences a SEU in the control plane would have.

Lastly a recent article in The Register declared that “software-defined networking is making users grumpy rather than delivering promised benefits.”  I have my opinions as to why this is the case and I will share some of that insight with some facts from a  recent Gartner poll in the future.

I came back to networking after a hiatus where I spent time in  Service Management and Software Architectures at a customer.  SDN to me was the perfect storm where a highly programmable network could be achieved without the archaic ways of old with “screen scraping” and antiquated SNMP.  I once had a problem in my Cisco days where we wanted to have two links out of a remote office.  One link was a DSL link and the other a T1 link.  We wanted the delay sensitive traffic to traverse the T1 and backup/bulk traffic on the DSL link provided everything was up.  We wanted the mission critical traffic to switch to the DSL link if the T1 was down.   In order to achieve this with the CLI, I needed to do GRE tunnels with IPSEC tunnels on a secure T1 and the DSL circuit.  Sure we made it work, but troubleshooting it was somewhere next to impossible.  The real solution would have been a simple “If Then Else” statement in the logic of the device – but I couldn’t do this because of the CLI.  Now as we approach 2016 we are seeing SD-WAN solutions solve this problem and more.

To me Software Defined – is all about programmatic access to be able to extend and grow a solution based upon your needs.  Yes this means APIs and even SDKs for every solution.  At the end of the day, the customer knows how they want to use a solution.  You never know, sometimes customers actually do some pretty cool things that you never expected.

Have a Happy Day!

Brazil – this time without the police!

I recently had the opportunity to travel to Sao Paulo Brazil.  For those of you that know me will remember my last trip to Brazil.  The short story includes these things:

  • Stolen Laptop, Passport, IPad, and laptop bag
  • 10 hours in a Brazilian police station
  • Visit to the US Consulate
  • Walking down a scary street to get a picture
  • Finally getting out of the country

So I decided to make the trip back to FutureCom again this year.  Problem is that the Brazilian economy is in the toilet and for the most part the show was a bit more subdued.

Meanwhile I spent some time taking in the Brazilian culture and here are my random musings.

1 – The country/city is actually very artistic.  I enjoyed the fact that they paint a lot of the concrete structures which cuts down on the graffiti.

2 – Traffic Sucks!  I decided that the reason that the economy is in the toilet is because it takes people FOREVER to get anywhere.  Really – it could take an hour to go 5 miles. Productivity is poor because you are always in a CAR!   I swear a guy in a rascal could make it faster than riding in a car, which leads me to point 3

3 – Motorcycle riders are absolutely nuts.  I watched them run 40 to 50 MPH through traffic between cars.  They even have the audacity to honk at you if two cars are too close together.

4 – The people are amazing.  Even though I speak no Portuguese, people were amazingly kind.  Probably much more kind that we Americans are to those non-english speaking that travel to the US.

5 – The food…well they really like their beef.  While visiting the report comes out from the WHO about beef and how it is going to kill us all.  To celebrate I had not one but two fillets at dinner (that is the only way they served them). Take that WHO!

While visiting I got to experience a real Brazilian apartment only to hear what sounded like gunfire outside…which was really fireworks when they scored a recent football goal.  I got to meet one of my co-workers family and his wife prepared a meal for us – it was mucho bom – which is Brazilian for very good.  In total, it was a good trip, but I am happy to head back home to the family a day earlier than expected.

Good times and great friends.  I am blessed in many ways and this week reminded me again of just how lucky I am.

Have a happy day!

 

Vendor Defined Networking (VDN) no more…

I have had the opportunity over the last couple of weeks to deliver several SDN presentations to different audiences.  As I prepared for the presentation I kept coming back to one truth – the network has always been software defined.  What changed is that certain vendors decided to take that software and put a configuration interface on top of it that gave you limited access to the functions and capabilities of the devices.  In other words you could say that networking has predominantly been CLI defined or vendor defined based upon the features and functions that are exposed to the customer.

As I thought back to my career, the majority being a network engineer, I was struck by this construct.  When I was doing development every new problem could generally be satisfied by code…software…back then in an editor and compiler.  So why shouldn’t I be able to solve my networking problems the same way?  I understand that there are standards and protocols that help protect us from bad code by an end-user (because application developers always write perfect code…right?).  However if I know how I want to manipulate the flow or control plane – shouldn’t I be able to do it given the risks?  The answer previously has been “NO” – but that is changing.  With the advent of EOS and extensibility the ability to manipulate SysDB and the control plane in general has opened the possibilities.  I understand that other companies may be running to the “bolt-on” SDN approach where they build APIs that will interface with the hardware, but these will ultimately be flawed unless you truly have an architected operating system that interfaces in an OPEN way with the underlying hardware and software.

So I am not an SDN purist that thinks that Openflow=SDN.  In fact I am much more in the camp that if you have a highly programmable platform the agent that you use to push control plane information to the hardware doesn’t matter.  You could use Openflow, Closed Flow, Darrin Fl0w – it doesn’t matter.  What does matter is that you can translate the flow or forwarding entries into information that the hardware can then populate its various elements to switch the packet efficiently through the device.

In short – I came back to the networking industry to combine all the great things that I had learned in the application world over the last 5 years.  The intersection of SDN or Programmability with networking provides for endless opportunities as to where we can take application control of the environment.  You see somewhere networking lost its way and became CLI or Vendor defined networking…not any more.  The new revolution is to open this programmability for those that want it through the use of OPEN APIs and interfaces.  Now I can have a truly business defined network or customer defined network…all due to enabling the programmatic inclusion of the underlying hardware and software.

That’s all for now – off to Cleveland for two more fun days of customer meetings!

Just over a year since considering a career move…

As I sit here on Spring Break in Florida, I am struck by what I was thinking last year about this time in terms of my career.  You see just about this time every year, my family with my wife and two daughters head south.   Last year (2012) was really no different, except that I have recently been assigned a new role in my job.

I had spent the previous 14 months working with Service Management and ITIL adoption throughout an organization of 12K plus employees and contractors.  I would routinely wear a bright orange sweater to work and as I walked into a room and someone would comment about how bright it was – I would say “I am from Service Management – don’t shoot!”  As I sat in my office one day near the end of the year, my boss walks in and says “we have a new opportunity for you,” which is always interesting.  I was being moved to a more business facing application area.  The area had a great leader named Jim that had stood up the function from the beginning.  It was known for its technical depth and the architecture team had two very seasoned individuals – of which I was to become their new leader.

As I went home that evening I felt every emotion that you could feel – from frustration to excitement.  Ultimately I would be leaving Service Management, which wasn’t all bad and moving to an application area to lead a team that knew much more than I ever did about developing application frameworks.

So back to my vacation in 2012, which was about 3 months after I had gotten this new role and I was enjoying the team, learning a lot, and was spending a lot of time leading, but not spending a lot of time exercising my creative ambitions.  So I have a constant search on a few job sites where I had query setup to match my ideal job.  As I was on vacation, one came across at a company I have long respected.  VMWare had a Cloud Solution Architect position available, where I could work from home and become an evangelist for cloud computing (something I love) and also work for a great organization.

I sat on the beach contemplating whether I should apply.  I have this philosophy that once I put my resume out – I am leaving, maybe not today, but at sometime in the near future.  I knew there would be no going back…so I sat in the sun for a week thinking and thinking and thinking…

On the way home I called a friend who worked for VMWare and he agreed to pass my name to those in the hiring position.  But if I was going to jump into this pond, I should look at the rest of the industry and at the time Software Defined Networking was all the buzz.  So I reached out to an old friend at Arista, who was making great waves with SDN – she just happened to be the CEO and someone else that I respect greatly.

Fast forward and suffice it to say that I decided to put my resume out…but nothing really materialized for some time.  This was unchartered territory for me.  Just as I recommitted to staying put, something came up.  Arista was growing like crazy and they were looking for more of an enterprise focused Solutions Architect (again my ideal job).  We talked and within a week I was on a plane for a quick overnight stay and interviews.  I got to meet with a lot of old friends and some new faces.  In total it was obvious that Arista was a special place, made up of people that wanted to make a difference in the networking world.

Fast forward to November in 2012 and I was in a place where it was time to resign from my position at State Farm.  It was possibly one of the hardest things I have ever done.  I would be leaving some great people that had spent a lot of time helping me grow.  Many of those people became great friends over the previous 5 years. Many were shocked, many were upset with me, and most were very supportive.  Opportunities of a lifetime don’t come every day and I was worried that if I didn’t sieze the opportunity – I would never have it again.  So as I end this post and contemplate the next one (which will be why Arista is such a great place) – I just want to encourage anyone out there that is considering making a move…don’t wait.  Pursue your dreams, pursue your innovative spirit, pursue excellence…you will never be disappointed for it is the chances that we don’t take that we ultimately regret!  One year after considering it – I am doing what I love with great people.  The choice is yours to do the same!

BTW – we are looking for great SEs in Minneapolis, Kansas and Ohio 🙂