Defining the Role of Distinguished Engineers

Clarke Rodgers (00:11):
Paul, thank you so much for joining me today.

Paul Vixie (00:13):
I'm glad to be here.

Clarke Rodgers (00:15):
If you'd be so kind, if you could please cover some of your background. It's the illustrious innovator and thought leader who helped invent the Internet to some degree in DNS. Please walk us through that.

Paul Vixie (00:30):
I did have an early role, but I’m not Al Gore, I did not invent it. And with DNS, I started out just kind of trying to fix it for myself. I worked at a minicomputer company back in the late eighties and the software was terrible and the protocol was not very well understood. So, it was self-defense and then later it stuck and I started a company to maintain that. And then when I left that company, I did a startup in the security space.

Somewhere during all of that I was inducted into the Internet Hall of Fame and I completed a Ph. D. in Keio, at Keio University in Japan, which is a strange way to wind up since it all started by dropping out of high school in 1980.

Clarke Rodgers (01:18):
Oh wow. Let's go into that a little bit. What caused the dropping out of high school and then sort of the change in movement into the technology space?

Paul Vixie (01:28):
I had been working sort of part-time, small jobs in the minicomputer field at the time. So, when my guidance counselor said, “Yeah, you're going to be a junior again next year,” it was because I had spent all of my time in the computer lab.

Clarke Rodgers (01:43):
Oh wow.

Paul Vixie (01:44):
So, I realized this is probably not going to get better and I should just get out there. Later, I moved on from that job as we inevitably do at that age. But yeah, it was all from being a poor student because I was spending too much time programming.

Clarke Rodgers (02:00):
Well, I think it worked out for you in the end.

Paul Vixie (02:03):
I am happy with all of the weird decisions that were made.

Clarke Rodgers (02:08):
If you’d be so kind, please tell me a little bit about your role here at AWS.

Paul Vixie (02:13):
So I have two roles, perhaps three. I was hired as a Vice President, Distinguished Engineer — that is a very small group of very experienced people. So that role is very much self-directed. To be a Vice President, Distinguished Engineer, your job description is to find trouble and get into it. It's very rare that anybody says, “Paul, we really need you to do a certain thing.” And so that's a huge honor in a company of this size to be set loose, especially coming in twenty-five years after the fact and not knowing how we got here. So that's been huge.

Now, over the summer, I was appointed the deputy CISO for AWS and coincidentally asked to lead the Office of the CISO. You can think of the Office of the CISO as C-level solution architects. If there is a conversation with the customer where somebody from their C-level is going to be present, normally we want to put our CISO in that room. The trouble is we only have one CISO and we have a lot of rooms, and so we are kind of the backup band for the CISO and that's half dozen people that are absolutely at the top of the field, top of their game. And I had been working with them, in fact, I embedded with them in order to get contact with customers. And that's I think why I was tapped to lead that team.

Clarke Rodgers (03:44):
Was there anything that really drew you to AWS?

Paul Vixie (03:47):
It was the description of the job. In other words, if I had been asked to come here and be the VP of one thing or another and be in charge of a certain function, I would've said, “Yeah, I've been there, done that.” But this is primarily a technology position, at least the distinguished engineer part of my job is basically an executive-level individual contributor. And to understand why that was exciting, the last time I had the word “engineer” in my title was also the last time someone other than me wrote me an offer letter — and that ended in 1993. So I've been just a CEO of startups and yeah, I was always a CEO who coded, but still not really a professional engineer. That wasn't my primary job at any of my companies.

So, to be able to get back on the original track that I would've been on without all those startups and to just be an individual contributing engineer was flattering. I didn't think I could still do it. They had to talk me into it, kind of. But once they described the position, it sat in the back of my brain and chittered at me and said, “Yeah, you really want to get back to that.” So, I didn't know. I had no idea that I was employable because I'm a minor public figure in the Internet and security industries and I'm known as a bit of a bomb-tosser. And so, I would've considered me too controversial for this job. They thought otherwise. So far, they've been right.

Clarke Rodgers (05:22):
So, you've been here for a while now. You've been able to sort of poke around. What have been your impressions of specifically the security culture at AWS?

Paul Vixie (05:32):
So, in my various startup companies, I sold to companies like this one. Corporate governance is pretty much the same everywhere except that here, the security team has been here so long that we make a justifiable claim that security is job zero — and that's not just a claim. It's reflected all the way up and down the org chart, up and down the operating plans, up and down the budget priorities.

In other words, of course, we're here to deliver for customers, like hopefully any company is there for their customers. The difference is we will not do something for our customers that also leads them to be insecure. We never leave the security out. That is what's different here.

Clarke Rodgers (06:20):
So, when you're diving deep with a service team, for example, in this Distinguished Engineer role, can you give us a peek behind the scenes of, is there conflict when there's a discussion around feature release versus security fix? Or is it just, “Nope, there's a security fix that has to happen.” What do those discussions look like?

Paul Vixie (06:42):
We don't get complaints of the form, “Gee, if you block this launch on the grounds of some application security testing result and so forth, it's going to make us too late. We will miss our window, and that's already in progress.” That doesn't happen. That's deliberately and explicitly not allowed to happen. What we do get is sometimes when it's a really “ought to” but not “absolutely got to do,” where I or somebody else who's representing AWS Security will say, “This is going to cause problems and is going to be a lot more expensive to fix later.” The service teams here have autonomy at that level.

They can decide what's important, but they know that if there is a security problem, then the fact that we talk to them about it is going to be a matter of record. They're going to have some explaining to do. And so yeah, there is sometimes tension because obviously everybody's got a boss and if the boss says, we got to have this by date certain, then you march to that and we're sensitive to that. We don't want to sort of blow anything up. But we understand that the people we're working with have the prime objective of meeting their metrics and we're the second prime directive.

Clarke Rodgers (08:09):
So, let's switch to your other day job, which, running the Office of the CISO, you have a lot of exposure to our customers and you interact with them throughout the world, either in person or in virtual. One of the vehicles that you've participated in at AWS is the AWS CISO Circles. Could you share a little bit about that and your experience with it?

Paul Vixie (08:30):
My introduction to CISO Circles was, “Gee, Paul, could you go to…” — it was Madrid in this case — “and participate in a CISO Circle?” So I had to say, “What is that, and what would my participation look like?” I'd only been here a year at that time. It may be that this calls for somebody with greater expertise about sort of who we are and how we got here than I currently have. And I went and it was terrific because what we had was a bunch of CISOs from a particular industry who had come together. And the whole thing is under a bit of an NDA, you're expected to be able to speak freely because the other people in the room are not going to repeat what you've said.

Clarke Rodgers (09:20):
Chatham House rule.

Paul Vixie (09:21):
Chatham House rules, and these are, in some cases, competitors. They're CISOs of companies that are in the same industry maybe or in the same region, and they wouldn't normally share ideas or share experiences. But because they were in our house and we were there and we were kind of moderating the discussion and seeding the discussion with some presentations about various bits of technology and then encouraging discussion, encouraging challenges, encouraging, ultimately, what turned out to be an argument among themselves.

It was beautiful to see, because that's one of the things that any mid to large company has got to be able to do, is to benefit from the experiences of their competitors. And we should not be trying to compete based on who is more secure, because if you and I are in the same industry and you get popped, that still hurts me and it probably means I'm next. So in this way...

Clarke Rodgers (10:24):
So sharing information and best practices...

Paul Vixie (10:26):
Yeah, we need to stand shoulder to shoulder in some areas, even while competing in others. And they ended up getting to know each other, figuring out how much trust they could extend without risking their company's secrets and so forth. And the intent is that they stay in touch with each other after the event. And of course, inevitably that means somebody will say, “Oh, well, I had this terrible experience with some API in the cloud that day,” and somebody else will say, “Well, it works for me,” and if they end up complaining to each other about us, well, if we have it coming, then we should embrace that too.

We're not going to be able to get better at serving our customers if we don't know what's going wrong. So, it turns out to be a huge source of business intelligence for us. “Oh, I wish I had…” and sometimes you get to say, “Actually, that exists.” We're a very big company and we innovate a lot, and we do create new features or new technologies at a rate that no customer could keep up with. I mean, it's my job to do so, and I have a hard time keeping up with it.

So, we have to have that customer interface where somebody knows what the customers’ problems are and what the industry's problems are and what they're doing and how they're doing it, what their pain points are. Turns out we have people that are doing that that the customers don't know they should be making time to meet with on a regular basis. And if the CISO Circle helps that happen, onesie-twosie, I hope it then spreads by word of mouth and becomes a movement.

Clarke Rodgers (11:59):
Fantastic. Well, Paul, thank you so much for joining me today.

Paul Vixie (12:02):
It has been great. Thanks again for having me.