For Everyone (Part III)

Introduction#

001. "Security Training for Everyone Part III: Rise of the Machines".

Welcome to another round of "Security Training for Everyone"! The most fun, informative, well-delivered, humble, and non-optional training you'll attend within the next hour.

As I'm sure you're aware by now (since you're here), all PagerDuty employees, contractors, and interns are required to undergo annual security training in order to understand the threats we face and how to protect us from them. Doing the same training each year would quickly get tedious (as those of you who went through our old training a few times will attest to), so you'll be pleased to know that this is not a repeat of last year's session, or even the year before's session. While we will be going over some of the same topics in order to refresh your knowledge, and some of the takeaways will be the same, the content is all new. Topics that we paid a lot of attention to in the last few years will only be skimmed over this time around, and we're going to dig a bit deeper on the topics we only briefly touched on in the past.

So this is part three of "Security Training for Everyone", which I'm calling "Part III: Rise of the Machines".

Just like last year, I came up with that title entirely on my own. It's definitely not stolen from something else.

Classifications#

002. Slide classifications.

We'll start with a quick reminder of our slide classifications. We have three main classifications, Public, Restricted, and Internal Only. Every one of the slides in this talk will have one of these classifications shown in the bottom left. Please make sure to follow the rules for each classification type:

• If it's public, you can take a photo, tweet it out, do whatever you want with it.

• It it's restricted, you can only share it with customers under an NDA, or with folks internally.

• If it's internal only, then please keep it internal to PagerDuty and not share it anywhere else.

Similarly to previous years, I'll be producing three versions of the training slides. For the restricted and public versions I will be redacting certain information on various slides so that we can lower the classification rather than redacting the slides entirely. That way it's more useful when sharing.

Goal#

003. Goal of the training.

The goal of the training hasn't changed from previous years. It's still that we want to give you insight and visibility in to the threats we face as a company, and to teach you how to protect us against those threats. It's not just to give you a bunch of rules to follow, the intent here is to arm you with knowledge so that you understand the reasons behind those rules as well.

Agenda#

004. Agenda.

So we have quite a jam-packed agenda today, and there's a lot of topics we'd like to cover. I'm going to deep-dive on some and just touch on others. We tend to pick one or two topics each year that we deep dive on and then the next year they don't get quite as much visibility.

But hopefully you'll get some useful information you can all take away.

Colours and Animals#

005. Colours and animals.

A note on colours and animals. For those who don't know about this, in previous years the way we track attendance to this is at the start there will be a slide with a colour. At the end there will be a slide with an animal. It will be very obvious which slides, as the colour or animal will be the only thing being called out on it.

The colours and animals you see here are for previous years. They are NOT the answers for this year. Please don't put in previous year answers on the form, as we'll just ask you to watch the training again.

Completion Form#

006. The training completion form.

At the end of this training, you're going to get this form in your inbox. You put in the colour, you put in the animal, and that "proves" that you attended the training. Of course, this can be gamed, any attendance tracking can be, we're using the honor system really.

If you don't complete the form, you will get nagging emails from the security team. If you STILL don't fill in the form, we are required to revoke your access to all PagerDuty systems until we have evidence of you completing the training. That's a compliance requirement, and we have had to do that a few times, so please don't be that person.

Green#

007. Green.

So with that, I can tell you that this year's colour is GREEN. Remember green, and when you get the form at the end of this training, fill in green as the colour. This will demonstrate to us that you were here at the start of the training. Hopefully everyone will be able to spell that correctly too.

Source for Colour and Animals#

008. The source of the colour and animal combinations.

People have asked me where I come up with the colours and animals each year, and really it comes from my daughter. She's been watching me put together this talk over the last few months at home, and she keeps asking me if she can help with my "meeting about the aminals".

You can see here that she has a blue llama, and a purple dog, both of which she likes very much. So she chose the colour and animals this year too.

Choose Your Aminal#

009. Choose your aminal.

I'm going to do something a bit different this year and let you see the choices ahead of time. It's going to be one of these five animals that is the answer. Have a quick game, see if you can guess what it is.

The Security Team#

010. The PagerDuty Security Team.

With that out of the way, I'd like to introduce you to the PagerDuty Security Team. Who are the Security Team? All of these lovely faces you see right now are our current security team. I'm going to talk a bit about who we are, what we do, what our philosophy is, and how you can reach out to us if you ever need help with anything.

We've grown quite a bit in the last few years, so we've had to change our ways a bit. We're no longer silly.

The Security Team Is Not Silly#

011. The PagerDuty Security Team are always serious, never silly.

We now have to be 100% serious 100% of the time. There are definitely no moments where we're not a serious security team. So yeah, please don't come to use with silly things. Always be professional.

Our Mission#

012. Mission for the Security Team.

Our mission is twofold. We want to increase customer confidence in our product, and we need to protect our company brand and reputation. It's worth noting that employees are part of that too, you are part of the company, so one of our goals is to protect you.

Make it Easy to do the Right Thing#

013. Make it easy to do the right thing.

And one of our own overwhelming principles is we want to make it easy for you to do the right thing. There's always going to be a security and convenience trade off.

The goal we have is to make those less of an either/or situation. We want to try and make the most secure way also one of the easiest paths forward. That's definitely not always possible, but it's what we're striving for.

What if I do something stupid?#

014. What if I do something stupid?.

A question we get a lot is,

What if I do something stupid?

Adversarial Relationship#

015. We won't speak to you in a condescending tone, amongst other things.

There are a lot of companies out there where the Security Team has an adversarial relationship with the rest of the company. If you do something wrong, you're going to reprimanded, demoted, or possibly even fired. At the very least you'll likely be forced to take additional training of some sort.

That could not possibly be more counter to how the security team operates at PagerDuty. We will not do any of that. Ever.

Assume Good Intentions#

016. Assume good intentions.

We always assume good intentions when security issues are reported to us. If you clicked a link you shouldn't have by mistake, or if you copied customer data somewhere you shouldn't have, we're not going to come down on your like a ton of bricks. We're going to work with you to fix the problem. And we're not going to make you feel bad just because you got scammed.

We're here to help. We're not here to punish people for clicking a wrong link. That's not what we're here for. That's not what we do.

Never Shame#

017. Never shame.

We will also never shame you if you come to us with a problem. Anyone who's come to us with a problem or saying that you've clicked a link you shouldn't have will already know this. Your name isn't going to appear on any report, it won't appear in any post-mortem or after-action-review. We keep it private within the security team.

We also don't go around behind your back. There is no hidden #security-shaming channel on Slack or anything like that.

You'll see a lot of companies out there who will do phishing tests and things like that, where they send you a fake email and if you fall for it then you have to resit the training and do a bunch of other stuff. In other cases you may even get fired, as one particularly egregious example demonstrates. We will never, under any circumstances, as long as I'm still here and breathing, ever allow any of those types of tests against our employees. They go completely against what we stand for, and creates an environment of fear rather than one of trust.

Stop#

018. We may ask you to stop.

Having said all that, we may come to you and abruptly ask you to stop what you're doing. Sometimes in the middle of the security incident we will need to be brief and it may come across like we're being mean or adversarial. That's not the case.

Especially with security incidents, there are strict deadlines of when we have to report issues, and the clock starts ticking the second we discover an issue. So if we're abrupt, that will be why.

We will always come back to you after the fact and help you achieve whatever it is that you were trying to do when we asked you to stop. We're not just going to leave you hanging or anything like that.

Contacting the Security Team#

019. Contacting the Security Team.

If you need to reach out to us, there are a couple of different ways. If you just need to ask us a question or you're curious about something, the #security Slack room is the best method. We also have our email address. But if you need to get us urgently for anything, there's this cool product called PagerDuty that can do it for you. Choose the "Security Team Manual Escalation" service and that will immediately page the security team on-call engineer.

Response will be within a minute or two, so this really is just for urgent requests. Things like, you see someone called hacker logged in on one of our servers, you've found a security vulnerability being exploited, that kind of thing.

Examples of non-urgent issues are a customer wanting to ask a question about security. That's not an urgent request, please send those to email.

Security Team at PagerDuty#

020. Using PagerDuty to raise a security incident.

For those who aren't familiar with PagerDuty, although you really should be, this is what the manual escalation service looks like. We also put the words "Security Incident" and "Emergency" in there for easier searching. The big button over on the right is the one you want to press.

Compliance#

Okay, with all the intro stuff out of the way, let's get stuck into our main first topic of the day, which is Compliance.

Compliance is a really important initiative at PagerDuty. Because by obtaining and maintaining these industry compliances, we not only make our sales cycle so much faster, but we also help to gain the confidence and trust of our customers.

SOC 2 Type II Compliant#

022. We are SOC 2 Type II compliant.

The good news is that as of December 2019, we have achieved SOC 2 Type II compliance. This is a huge achievement and a lot of effort and work when into this from almost every team in the company. So well done!

Type I vs Type II#

023. SOC2 Type I vs Type II.

If you were here least year, you probably remember that at this part of the training we were Type I compliant, and now we're Type II compliant.

The difference is that Type I is a point-in-time snapshot of how we behaved on a specific day. So an auditor would come in for a day, and said "Okay, you do everything you say you do, you're all good".

For Type II, we were audited for a six month period, where auditors assessed whether we do all the things we say we do continually for six months. So as you can imagine, that was a much bigger undertaking.

So We're Done?#

024. So we're done, right?

So cool, we're done right? There's nothing else left for SOC 2 compliance for us to ever worry about?

Not Quite#

025. Not quite.

I'm afraid not quite.

SOC2 is Forever#

026-029. SOC2 is forever...and ever...and ever...

SOC2 is forever...

...and ever...

...and ever...

...and ever.

Stay Compliant#

030. We need to stay compliant.

The key now is that we have to stay compliant. Every six months we'll have another audit period and will be audited at the end of it. We have to maintain all of the same policies, practices, and procedures that we already have. We can't just let up because we're not in an initial crunch period any more.

SOC2 audits are always happening, so if we let up, even for a moment, that's going to end up on the report. The SOC2 reports go to customers, so it can be really embarrassing if things show up on them.

Other Compliances#

031. What about other compliances?.

We also receive a lot of questions about other compliances we're looking to achieve, since it's not all about SOC2.

Other Compliances Redacted#

032-033. Redacted.

GDPR#

034. GDPR. Reference

Not really compliance, but more of a privacy framework and regulation is GDPR, the General Data Protection Regulation from Europe.

Just like SOC2, we cannot give up on maintaining these privacy frameworks, policies, and protections that we have in place. Especially when it comes to GDPR, the fines are ridiculous. Even though we're not a European company, we have a presence there, and have customers there, so we're obligated to maintain the same level of privacy controls.

This is the reason why we have to let users delete all of their information when requested and things like that. I mean, we should do that anyway because it's the right thing to do, but if that wasn't a good enough reason, there's GDPR to help make the decision for us.

CCPA#

035. CCPA. Reference

There's also a new one now, the CCPA, or California Consumer Privacy Act, which is essentially California's version of GDPR.

Which again we are going to fall under as well, and there will be some stringent privacy requirements there too.

California has lots of extra protections when it comes to business that operate online, and not just related to privacy too. So it's important we ensure we follow all those rules.

Compliance Roadmap#

036. An example compliance roadmap.

So when is all this stuff coming? Are we doing all of it? Are we only doing some of it?

Our compliance roadmap is available on the wiki, it is internal only, please never share dates with customers as they can change at any time, so we don't want to start making promises we can't keep.

How Does This Affect Me?#

037. How does this affect me?

So one of the questions you're probably asking yourself is,

How does this affect me?

Compliance Checklists#

038. Compliance checklists.

Well, as we start gaining more and more of these compliances, you're going to start seeing more stringent policies in effect throughout the organization. You'll see things like checklists and changes to our software development lifecycle that add more reviews before systems go into production and things like that.

Just something to be on the lookout for in future as we mature as an organization.

Where Can I Find The Controls?#

039. Where can I find the controls?

Another question we get a lot is,

Where can I find a list of the controls?

Folks are very interested in self-discovering what's involved with SOC2 so they can figure out what they need to do to stay compliant.

SOC2 Report#

040. Redacted.

The best place is probably the SOC2 report itself, which is available internally (and to customers under an NDA). They're a bit difficult to understand, I won't lie, and there's some interpretation and wiggle room involved. But it should give you a general idea of the types of rules that we have to follow.

Compliance Summary#

041. Compliance summary.

So a quick summary of all that:

• SOC2 is never going to end, don't think that it's over.
• We have other compliances on the roadmap.
• GDPR and CCPA privacy regulations are important and we need to keep those in mind when we're building new things.
• Keep in mind that compliance may start to impact your team priorities and roadmaps as we get further along in the processes.

Securely Working From Home#

Our next topic is working from home, which is pretty timely given the current situation in the world. This will be our main deep-dive topic for this year, so we'll be spending most of our time on this.

Office, Hotel, Anywhere...#

043-045. Office, Hotel, Anywhere.

Whether you're working from home, the office, a hotel, or anywhere really, the same things are going to apply for all four situations.

Do I Need to Use a VPN?#

046. Do I need to use a VPN?

One of the questions we've been getting a lot since folks have started working remotely is,

Do I need to use a VPN?

There's a lot of advertisements for Virtual Private Network (VPN) services right now, and we've been getting this question a lot.

Unfortunately it's not a simple yes or no answer. So I'm going to try and explain what a VPN actually is in as much detail as I can, so that you'll understand the tradeoffs involved and whether it makes sense for you.

Explain How The Internet Works#

047. Let me explain how the internet works...

But first I need to step back, and tell you a bit about how the internet works.

You may feel like this is going off the rails, but I assure you we'll come back to the question of VPNs.

Post Office#

048. The internet is like a post office.

So when talking about the internet, I'm going to use the analogy of the Post Office. But instead of having distribution centers and sortation centers, the internet uses "routers". They essentially do the same thing, but with electronic messages instead of physical messages.

HTTP vs HTTPS#

049. HTTP vs HTTPS.

When you're going to a website in your browser, you've probably seen one of these two things. You will likely have seen HTTP (which now gets flagged as "not secure"), or HTTPS (which shows a nice green padlock next to it if it's secure).

But what do they actually mean? What's the difference between these two methods of using the internet?

Not Secure#

050. Not secure.

We're going to look at the "Not Secure" one first, which would be HTTP.

Postcard#

051. Sending a postcard.

Keeping our postal analogy, this is going to be the equivalent of sending a postcard.

Say you want to login to a website. You can write a postcard saying "this is my username, this is my password, please log me in".

Then you're going to send this postcard over the internet (or over the postal system in our analogy).

Movies#

052. Just like the movies.

It's kind of like how you see in the movies, with fancy graphics showing computers talking to each other and things like that, and your message jumps between them.

It's the same thing, just without the fancy graphics.

Sending a Postcard#

053-057. Sending a postcard over the internet.

So, imagine you're here on the left (you're a little Pagey), and you want to login to PagerDuty. You write your postcard and need to find a path to send it over the internet.

It goes between multiple "sortation centers" or routers, and your message gets sent over.

The problem is that because it's just a postcard, anyone along that path can just look at the message and read it as it's going by. So they can steal your information, your username and passwords, without having to do anything. They can just sit there and watch it go past.

That's not great.

Anyone Can Read It#

058. Anyone along the path can read it.

More than that, not only can they read your message, they can also change it without either side of the conversation knowing that it's been changed, or what about it changed.

Received Postcard#

059. "Thanks, you're logged in".

So let's say you login to your bank or a website, and you'll get a postcard back essentially saying "Thanks, you're logged in". But you can't trust any of the content of that message.

Suspicious Postcard Edit#

060. Suspiciously edited postcard.

It could have been modified by anyone along the path, it could have been tampered with, and you would have absolutely no way of knowing.

Cannot Trust Anything#

061. You cannot trust anything.

The key here is what when using the "Not Secure" method, the reason it's not secure is because you cannot trust anything when communicating over that channel. Anyone can read your messages, and anyone can change your messages.

Comcast#

062. Comcast injecting code. Reference

This isn't some theoretical thing either, Comcast were actually doing this back in 2017. They would modify your requests to inject ads onto every site you visited to try and upsell you on new modems. Very not cool.

The Secure Method#

063. The secure method.

Okay, so how do we stop this? We need to use the secure method of accessing the internet, or HTTPS.

Another Postcard#

064. Another postcard being sent.

This is actually not too different from what we were looking at before. It's still a postcard, only now the data is encrypted.

Encrypted Postcard#

065. An encrypted postcard.

The message we're sending is encrypted, along with a few other bits and pieces. But you can still see who the message is going from, and who it's going to. Otherwise how would you deliver the postcard?

Encrypted Response#

066. An encrypted response.

The message back is going to be the same in that the contents will be encrypted so only you know how to read them.

A Secure Path#

067. A secure communication path.

The upshot of this is now no one along the path of this message can read your message, as they have no way of decrypting the contents. More so, if they try to modify you message, you'll know, because you won't be able to decrypt it when it gets to the other end. So not only do we know the contents of our messages are secure, we know they can't be tampered with.

We call this being secure or being encrypted "in-transit", as your message cannot be read or tampered with while it's moving across the internet.

Password Are Safe#

068. Your passwords and personal information are safe.

So this is great, your passwords and personal information are encrypted and safe, even if you're on public WiFi, even if you're connecting from an insecure location! As long as you're doing it over HTTPS, your passwords and personal information are safe.

But...

Information Leakage#

069. An attacker can still see the sender and receiver though.

...anyone can still see the sender and receiver of the message. Otherwise there would be no way to deliver the messages.

So there's a little bit of information leakage there, but it's not the end of the world.

Not Where#

070. But not where on a website.

It's worth noting that when I talk about the sender and receiver, this doesn't mean that people can see the entire path to the website you're vising. They can only see the domain, not the exact page on that site. It's like they can see the building the letter goes to, but not the specific room inside the building.

So all someone would be able to see is that you visit amazon.com, not what product you were looking at, or which user you are, for example.

Why Don't We Always Use HTTPS?#

071. Why don't we always use HTTPS?

This is a huge step up from HTTP, so why don't we always use this?

We Do!#

072. We do!

Well, PagerDuty does! Along with the vast majority of the websites out there. It's getting harder and harder to find websites that don't use HTTPS by default nowadays.

HSTS on PagerDuty#

073. You cannot visit pagerduty.com without HTTPS. Reference

In fact you can try this yourself, you cannot visit pagerduty.com without HTTPS, your browser will refuse to do it^*.

HSTS Preloading#

074. pagerduty.com is preloaded into Chrome. Reference

This is because we are hardcoded into a lot of browsers out there. This is a screenshot of the Google Chrome source code, with pagerduty.com listed in there. Website owners are able to add themselves to this list to ensure that users of the browser can only visit over secure connections.

So even if you try to visit via that original plaintext postcard method I showed before, the browser won't let you send the postcard. It'll just say no.

All Sites for PagerDuty Use HTTPS#

075. All sites for PagerDuty business use HTTPS.

This is true for every service we use for PagerDuty business too. This is one of the reasons that when you want to use a new vendor at PagerDuty, we have to do those security reviews and risk assessments. One of the things we're checking for is whether the vendor encrypts all of their information while it is in transit over the internet.

How Do We Know HTTPS is Safe?#

076. How do we know HTTPS is safe?

But how do we know this is safe? How do we know that our message is reaching the correct place and that it's not some fake site pretending to be PagerDuty just to steal your information?

SSL Certificates#

077. SSL certificates.

For this, we need to talk about something called certificates. You've probably heard about "SSL Certificates" before...

TLS Certificates#

078. TLS certificates.

...or really "TLS Certificates". The term "certificates" is all we really care about for this discussion.

I Am PagerDuty#

079. I am PagerDuty. Signed: Me.

When you visit a website, it shows you a digital certificate saying "I am pagerduty.com. Signed: me".

OK, great. How can we trust that though? They've just printed the certificate and signed it themselves, right?

Certificate Authority#

080. Certificate authority certifies PagerDuty.

In reality, these certificates will be signed by something called a Certificate Authority (CA).

This is an entity that has signed the certificate and is vouching for the authenticity of it. They are saying "Yes, I confirm that it is really pagerduty.com. Signed: Certificate Authority".

It's not like a physical signature, really it's going to be some fancy mathematics, a digital cryptographic signature that can't be faked.

Root CA#

081. Root CA certifies the certificate authority.

But how do we know the certificate authority is trustworthy? Well, they have a certificate themselves, and their certificate will be signed by yet another certificate authority. There can be many certificate authorities in this chain (and the chain can get pretty large).

Ultimately however, you will reach something called a root certificate authority (Root CA).

Root CA's are heavily vetted and audited, and are generally globally recognized as being trustworthy. Your operating system will have a pre-vetted list of these available to use. So Apple and Microsoft, etc. will vet these root certificate authorities for you.

Trust the Authority#

082. "I trust the root authority".

So your operating system will trust one of these root authorities, which in turn signed an intermediary certificate, which in turn signed the pagerduty.com certificate.

Trust PagerDuty#

083. "Which means I also trust PagerDuty".

The ultimate upshot of this is that through this whole chain of trust you know you're speaking to the real pagerduty.com.

Saul Goodman#

084. Saul Goodman knows a guy, who knows a guy, who knows another guy.

I like to equate it to Saul Goodman saying, "I know a guy, who knows a guy, who knows another guy".

Driver License#

085. Driver license.

Looking at this from another point of view, one way I like to think about this idea is to use the example of driving licenses.

Let's say you're at an airport, and you need to prove your identity to an agent there. You might use your drivers license to do that.

So you show them your drivers license. But how can they trust that document?

DMV#

086. DMV certifies driver license.

It will have been issued by some authority, usually the DMV (in the United States at least). That authority has vetted things like your social security number and other forms of identity.

Government#

087. Government certifies DMV.

But then why should they trust the DMV? Well, they've probably been vetted by some higher authority. Usually the Federal or State Government.

Trust Root Authority#

088. "I trust the root authority".

Ultimately the agent trusts that higher authority...

Trust Drivers License#

089. "So I also trust your driver license".

...which in turn means they trust your driving license as a valid form of identification.

MITM#

090. Beware of man-in-the-middle attacks.

One thing to be careful of with all this certificate stuff is something called a "man-in-the-middle attack" (often abbreviated to MITM).

This is when someone (despite the name, it doesn't have to be a man) sits in the middle of a connection between two parties and can eavesdrop or alter the communication.

Honest Rich's CA#

091. Honest Rich's Certificate Authority.

The way this usually presents itself is that you try to visit a site, and it says "I'm pagerduty.com", but their certificate is signed by some certificate authority you've never heard of, "Honest Rich's CA" for example.

Browser Warning#

092. A typical browser warning.

At this point your browser will say "Hrm, this looks suspicious. I had better stop what I'm doing and warn someone". So you'll see something like this error here.

I'm sure you've all seen errors like this in your browser before. This particular one is from Chrome, so it'll look slightly different in other browsers, but the concept is the same.

When this happens, you very much want to click the button that says "Back to safety".

You don't ignore browser warnings and just click whatever gets the page to load, right? RIGHT?

Loki#

093. Loki has never met this certificate authority before.

Your browser is essentially saying "I've never heard of the certificate authority before", or more correctly "I don't trust that this communication is secure". It has found a problem when trying to verify the certificates, and it cannot 100% say for sure that your connection is secure and someone isn't sitting in the middle of your connection and listening in.

There are other reasons to get an error like this, for example if the certificate is out of date (they usually have to be renewed annually), but in every case your browser is warning you that something is wrong and you likely can't fully trust that your connection is as secure as you want it to be.

No Trust#

094. Don't trust Rich's Honest CA.

So your browser says "I'm outta here, I'm not trusting this". And you shouldn't either.

It is possible to bypass these warnings, but you really really really shouldn't do that unless you very much know what you're doing and the full implications of doing it.

Don't Ignore Browser Warnings#

095. Don't ignore browser warnings.

So the key takeaway of all this is to not ignore browser warnings. They're there to protect you. If you ignore them, then the security of your connection cannot be guaranteed.

So What's a VPN?#

096. So what's a VPN then?

Ok, so why am I going on about all this? What does any of this have to do with a VPN? I promised I'd get back to what a VPN is. So let's go back to our postal analogy now and talk about what a VPN does differently than just normal HTTPS.

VPN Postcard#

097. Postcard, but the "from" address is blank.

We're sending the same postcard that we sent before, but now we're going to leave the "from" address blank. And rather than mailing it ourselves...

Put In An Envelope#

098. Put it in an envelope to the VPN provider.

...we're going to pop it into an envelope and send it to someone else, our VPN provider.

OK.. so why are we doing that? That's weird, right?

First Connection#

099. The first connection is only to the VPN provider.

We're essentially making two connections. We're going to make one connection where we send this envelope to the VPN provider. So anyone looking at that connection only sees us sending something to the VPN provider, and they don't know what it is.

VPN Mails the Postcard#

100. VPN mails your postcard the rest of the way.

When the VPN provider gets it, they're going to write their name in the "from" address and they're going to mail it the final leg of the way to the end website.

Second Connection#

101. The second connection is only from VPN provider to destination.

So anyone looking at this connection now only sees that this message came from the VPN provider to the final website.

Mailing Back#

102. On the way back, VPN mails you the postcard.

It's the same when it goes in reverse as well. The website sends the postcard back to the VPN provider.

Return Envelope#

103. You receive envelope from VPN provider.

And the VPN provider pops it into an envelope and sends that envelope back to you.

Two Connections#

104. There are two separate connections.

The upshot is that neither end of the connection is speaking directly to the other end. So as far as the people on the left are concerned, you're only talking to the VPN provider. As far as the website on the right is concerned, the only thing talking to them is the VPN provider.

So now we've gotten rid of that little bit of information leakage that was still there. The "From" and "To" addresses on our postcard.

ISP#

105. You ISP only sees your connection to the VPN provider.

If you're in a cafe and using a VPN, they can now only see your connection to that VPN provider and not which particular sites you are visiting. The same is true if you're at home, your ISP would now only see your connection to the VPN provider and not which sites you're visiting.

Website VPN#

106. Website thinks the request came from the VPN provider.

The website you're visiting doesn't know where you're coming from either. They only see the connection from the VPN provider. So it can provide a level of anonymity.

OK, that sounds great! But it's not without some caveats...

VPN Sees Everything#

107. VPN provider now sees everything.

The VPN provider can now see everything your ISP used to see. They're sitting right in the middle of your connection. It's essentially a man-in-the-middle attack, but one that you've allowed to happen.

You're moving your trust from the ISP to the VPN provider. So if you do use a VPN provider, you have to really trust that provider because they can see the information that's going between the connections that previously only the ISP would have seen.

Are VPNs Useful?#

108. Why are VPNs useful then?

Why are VPNs useful then? Hopefully you can start to see why these things would be useful from a security point of view.

Streaming Service#

109. Streaming service in another country.

One of the main use cases you see is that, hypothetically of course, you want a stream service to think you're in another country. Using a VPN provider can mask your true location or identity.

I'm sure people would never really do that though.

Downloading Things#

110. Downloading things without ISP knowing.

Or maybe, hypothetically, you don't want your ISP to see certain downloads that you're making.

Of course, I'm sure someone would never really have a need for that though.

Whistleblowers#

111. Vital for journalists, whistleblowers, persecuted groups, etc.

It may sound like VPNs are only used to provide anonymity while attempting to do something nefarious, but that's definitely not the case. They are an absolutely vital tool for journalists, whistleblowers, persecuted groups in unfriendly countries, etc.

They are also used all over the corporate world to ensure a baseline level of connection security when access internal networks.

Guarantees Trusted Communication#

112. Guarantees trusted communication.

Because they guarantee (within reason, assuming you trust the provider) that you have trusted communication over an untrusted network. So if you're somewhere where you can't trust the normal connection, you can use a trusted VPN provider to enable secure communication.

Need to Use VPN?#

113. Do I need to use a VPN?

So we finally, eventually, get back to the original question.

Do I need to use a VPN?

It Depends#

114. It depends...

It depends! And I know that's a bit of a wishy-washy answer, but it's true. It depends on what your personal threat model is. What do you consider to be a significant amount of risk to justify the time and energy on using one.

HTTPS Already Safe#

115. HTTPS is already safe.

Remember, if you're using HTTPS, your passwords and personal information are already safe. As long as you don't ignore browser warnings, that information is already safe. The only thing anyone could see is that you're visiting, say amazon.com, not what items you're looking at or anything about who you are.

Care About ISP?#

116. Do you care about your ISP seeing the sites you visit?

What the question really boils down to is, do you care if your ISP, hotel, or airport sees what sites you visit?

For me, I personally tend not to really care about that. But if I were a reporter researching government corruption, I would very much care about it.

It depends on the level of risk you're willing to accept, and every situation is going to be different.

PagerDuty VPN#

117. A PagerDuty VPN?

It may be surprising to learn that we don't currently have a corporate VPN at PagerDuty. The reason we don't provide one is because we don't explicitly need one, and the effort to benefit ratio doesn't currently make sense.

We'd have to built it, maintain it, it would become a single point of failure, we'd have to start troubleshooting issues on it, etc.

One of the reasons other organizations tend to use VPNs is that they trust traffic from the VPN network more than other traffic. We don't, because we use something called a "Zero Trust" approach to our networking at PagerDuty. The network you're coming from gives you no special privileges, everything is treated as untrusted and requires authentication.

In a typical environment, you might authenticate to a VPN, and after that point you'd be able to access all internal systems since you're within the private network. It's like a secure campus with a big fence around it. Once you get through the guards at the fence, you're free to roam around inside the wall as much as you like. At PagerDuty, every internal system is protected directly, so you have to authenticate to it no matter where you're coming from. So rather than a wall around the campus, every single building has it's own fence and security guards. Just because you can access one, doesn't mean you can access another.

So a VPN wouldn't provide us any real benefit considering the model we currently use. It would be useful for travelling employees who wanted a trusted connection, but there's no reason a PagerDuty-specific VPN would be needed for that, as any commercial VPN provider would provide the same benefit.

VPN Advert#

118. A typical VPN advertisement.

Be careful out there as you see ads from VPN providers, as they're really trying to sell you hard on their product, especially nowadays with everyone working from home. They position themselves as the "only secure way" to access the internet.

Already True#

119. Most is already true with HTTPS.

Most of what they claim in ads that a VPN protects you from is already true with HTTPS and you already get the benefits today. They're not lying, it is all technically correct, but it's very misleading.

There's a lot of fear around it, so don't get forced into using one without understanding what they do (and don't) provide when it comes to security.

Misleading Advert#

120. Some ads can be misleading. Reference

In fact, one VPN provider got in trouble in the United Kingdom by the Advertising Standards Authority for creating the misleading impression of risk.

Virtual Private Networks#

121. Virtual Private Networks video from Tom Scott. Reference

So that's all I'm going to say about VPNs. If you want to know more about VPNs, there's a great video from Tom Scott that I recommend on YouTube. It talks about them in non-technical terms, and why the claims of many VPN providers are perhaps exaggerated.

But hopefully now you understand what a VPN is, what protections it provides, what protections you already have without a VPN, and the rationale for why we don't have a corporate VPN at PagerDuty.

Is Zoom Safe?#

122. Is Zoom safe for us to use?

Another question we've been hearing a lot lately is,

Is Zoom safe for us to use?

Yes#

123. Yes.

Yes. It is. This one I actually have a direct answer for.

There have been a lot of stories recently about Zoom and their security posture, things like using ECB-mode encryption (which to my understanding is not terrible for compressed audio/video communication, I'm nothing close to a cryptography expert though so take that view with a giant grain of salt).

But there's a lot of concern along the lines of "They have so many vulnerabilities!!".

Vulnerabilities#

124. Vulnerabilities on chat/video software. Reference

Well, so do everyone. Zoom have just been put under more scrutiny lately, so more things are being found. But every single chat and video solution out there has vulnerabilities. In fact, every piece of software out there has vulnerabilities. I would be immediately suspicious of any software that's been around for a while without a single vulnerability being found.

Zoom actually come out better than the others here, but they also haven't been around as long. So there's lots of factors to consider.

We trust Zoom, we still use Zoom, we're not planning on changing to something else.

Goodnight Moon#

125. Goodnight sense of impending doom. Reference

So say "Goodnight sense of impending doom".

Threat Models#

126. Thread models are important.

But this brings me nicely to another point, in that threat models are important. When we're trying to consider whether to move to a new product, or to change the way we do things, we need to consider what our threat model is.

What is the threat are we trying to defend against? What is the likelihood that threat is going to happen? How devastating would it be if that threat were exploited?

Our customers will be doing this to us too, they're going to be analyzing if PagerDuty presents a significant risk to them or not. For some organizations, using a third-party for their incident response might present too high of a risk for them.

We Are Not Google#

127. We are not Google.

So it's worth remembering. We are not a government agency. We are not a defense contractor. We are not Facebook or Google. We do not have the same problems or risks those entities have. We do not have to defend against the same threats they do.

When you read stories in the news about how a piece of software isn't secure because company X isn't using it any more, think about the threat models involved before you worry about if you should also stop using the same software.

If we were the NSA, we probably would not use Zoom for example. But we're PagerDuty, using Zoom is fine for our threat models.

FUD#

128. Fear, uncertainty, and doubt.

The upshot of all of this is that there's a lot of security news out there. There's a lot of fear, uncertainty, and doubt floating around, especially as everyone has switched to working remotely.

People are rightly concerned when they read about a security issue, and we see this a lot on the security team so we are a bit jaded towards it. But if you're concerned about any of this, perhaps you see some vulnerability announced and you're worried if there's a risk (whether for business or personal), just come and ask us. We'll be happy to point you in the right direction or explain more about it.

Working From Home Summary#

129. Pagey's summary of working remotely.

That was a long section, and our main deep-dive topic of the day, so don't worry, the others will be shorter. Here's a quick summary of what we talked about.

HTTPS is more than enough to protect you in most cases. VPNs are good, if you want to use one and you trust it, go right ahead. We don't have a corporate VPN. Zoom is safe for us to use. And threat models are important. They're how we decided what to work on.

We're never going to get rid of all threats out there, so we have to focus our attention appropriately.

Phishing#

Next topic, phishing!

Be Suspicious#

131. Be suspicious of your email.

If I could sum up phishing in a single sentence, I would probably use "Be suspicious of your email".

Not Sure If#

132. Not sure if...

If I could do it in a single slide it would just be this.

Every email you get, you should be looking at suspiciously. We get quite a few phishing attacks, and they're getting more and more targeted.

Phishing Statistics#

133. Phishing statistics for the last few years.

Last year I showed some statistics on the phishing attacks we received. Here's the comparison for this year.

I did want to discuss some other statistics, but first...

We Never Fall For Phishing#

134. We never fall for phishing, right?

...I wanted to address a piece of feedback from last year's training that troubled me a little. There was a similar sentiment from a few other pieces of feedback essentially saying:

We never fall for phishing so why do we bother spending so much time on it?

I realized that we don't really talk about when people do fall victim to phishing. So let's look at the numbers.

Phishing Victims#

135. We've had phishing victims.

We've had more than 𝑥 employees in the last two years that have fallen victim to phishing scams at PagerDuty.

It's difficult to get an exact number here, because how do you count it? If they clicked a link, do you can that as falling for it?

But however you count it, it is a very non-zero number.

Real Money#

136. Real money was lost.

Real money has been lost. In one instance an employee lost quite a bit of their own personal money due to a phishing attack 😬

Engineering Too#

137. It happens to engineers too, you are not immune.

A lot of the feedback came from the Engineering department as well. I'm not sure if engineers just think that because they see this more that they're more immune to it. But that's not the case. Around 20% (plus or minus a few percent) of the victims were in the engineering department.

There could be many reasons for this of course, maybe engineers just report it more, so take all of these numbers with a grain of salt. The main takeaway should be less about the actual figures, and more about the fact that they are definitely not 0. People have fallen for phishing, and will continue to fall for phishing.

Money, Access, or Sensitive Info#

138. Money, access, or sensitive information.

Engineers can be specifically targeted just like anyone else. When someone attempts a phishing scam, they're usually after one of three things: money, access, or sensitive information. Which means they're going to be targeting our finance team, or engineering teams, or HR, etc.

But anyone can be targeted for this stuff.

Anyone Can Fall For It#

139. Anyone can fall for phishing.

And anyone can fall for it. Don't get complacent thinking you're immune to phishing. You're not. Anyone can fall victim to it.

Only One#

140. It only takes one.

It only takes one. One successful phishing attack, and malware gets installed on someone's laptop, and then an attacker can access a variety of sensitive information about PagerDuty.

So stay vigilant. We do a really good job of reporting phishing here, and that's something we should be proud of. But it doesn't mean none ever get through. That's unfortunately just the nature of the game.

How Did They Get My Email?#

141. How did they get my email?

One question we've gotten a lot is,

How did the attacker even get my email address?

It's Not You#

142. It's not you.

Well, you don't need to have fallen for an attack for that to happen. There are many ways for attackers to get ahold of your email address.

Emails can be guessed from names of employees from LinkedIn, or they can appear on lists of leads when your badge is scanned at conferences, and so on. There are many ways to get (or guess) an email address, so you shouldn't be too surprised when it happens.

One other way attackers can get your email address is when you use that email on another website, and that website gets breached.

Have I Been Pwned#

143. Have I Been Pwned. Reference

There's a great website called Have I Been Pwned (HIBP) from Troy Hunt. It's a legit website, I assure you. But the idea is that you enter your email address and it will show you a list of all the known breaches that email address has appeared in.

It doesn't tell you the password that was potentially leaked, but it gives you a heads up that your personal data may have been leaked.

HIBP Comparison#

144. Unleaked email vs a leaked email. Reference

For example, example@pagerduty.com doesn't show up in any breaches, but the security mailing list does show up in at least one breach.

This is a service we use on the security team here at PagerDuty. We will get a notification any time an email on the pagerduty.com domain appears in a breach, so that we can get a start on taking action such as forcing a password rotation and informing the user.

How Many Emails?#

145. How many PagerDuty addresses are leaked?

But I thought it would be fun for you all to take a guess at how many PagerDuty email addresses you think have appeared in known breaches. Bare in mind the number of employees we currently have.

Lots#

146. Lots.

If you're thinking it's a trick question, and the answer is "it's the number of employees" because all email addresses are guessable, well, you'd be right, that does sound like something I would do. But in this case, you're wrong! It's actually about twice the number of employees we currently have!

All Emails#

147. The list of leaked email addresses.

That's a lot of email addresses.

Why more than the current number of employees? Well, because it will include a lot of past employee email addresses too.

You should always assume that an attacker already knows your email address. But if you were wondering how they got it, this might be how.

Interesting Phishing#

148. Any interesting phishing lately?

OK, let's look at some recent phishing attacks now.

GoDaddy Phishing#

149. GoDaddy phishing example.

I'm going to start with a fun one we got to our email list that handles our domain registrations.

It says that pagerdutysummit.org is about to expire. It does look a little legit, like it comes from GoDaddy.

A couple of things to point out though:

1. There's a literal red flag in the subject 🚩. Which I thought was kind of funny.
2. The key thing here though is that the from address isn't the correct GoDaddy domain.
3. Even more, when you hover over the button, the link goes to a completely different domain.

Those are the big key indicators on this one. But there was another fun indicator we spotted, which is that the support phone number that's listed is actually for the US Naval Observatory talking clock.

Now, we're not expecting you to go and check all the phone numbers in the emails you get of course, but we just thought this was a funny little extra thing.

Very Close#

150. I was very close to falling for it.

The reason I'm pointing this out though is because I came very, very, very close to falling for this one. In fact, I think it's fair to say I did fall for it, I just happened to get lucky because another employee spotted it first.

When this came in, I wasn't really paying attention. I saw it arrive, saw that a domain was expiring, and thought "I'll check that after lunch".

By the time I got back from lunch there was an email reporting it to us as "some pretty good phishing". Of course, then I looked a little closer and thought "Oh yeah, that is pretty good".

I'm relatively convinced that had I not been hungry, I would have almost definitely clicked that link.

So like I said before, anyone can fall for this stuff, it just takes a momentary lapse.

W2 Phishing#

151. A W-2 phishing attempt.

The targeted ones we get nowadays though seem to be more after employee information than information on our customers.

Ones like this tend to be fairly obvious, where it's claiming to be Jennifer (our CEO), but the email is very much not a pagerduty.com one.

They're pretty brazen though. Send all the W-2's for the last two years... that's never going to happen even if it really did come from the CEO, let's be honest.

Payroll Phishing#

152. A payroll phishing attempt.

A really good one we got appeared to be from an employee to our Payroll department, asking about how to make changes to their direct deposit information.

But look at that domain, it's not our real PagerDuty domain, it's pretty close though and easy to miss.

Props to the payroll team for spotting this straight away and sending it to the Security team.

What I liked about this though was that they're not directly asking you to change any information yet, they're just asking what the process is. This is a really good tactic from phishers, as it's laying the groundwork without raising alarm bells immediately.

Operational Security Fail#

153. Oops.

The attacker for this one made some mistakes though, in that they leaked their real email and IP address. We were able to use this information to find reports from other organizations of similar phishing campaigns that they'd been receiving from (presumably) the same attacker.

Hard to Block#

154. Some phishing can be harder to block.

And finally, this one. It's pretty obvious phishing, and I think almost everyone at the company reported this to us at some point.

But the reason I'm calling this out is because while it was low effort, it was surprisingly difficult to block at first because there were so many variations. They changed pretty much everything about this email every time it was sent. The from address, the subject, the content, the name, etc.

They specifically used things like "Microsoft Word", knowing that we wouldn't be able to block things based on that alone without causing legitimate emails that mentions that to also be blocked.

How Do You Block?#

155. How do we actually block them?

Which brings me to a question we get a lot, which is how do we actually block these types of things?

IOCs#

156. Indicators of Compromise.

We document it on the wiki so you can see for yourself!

For each attack we get we look at something called the indicators of compromise, also called "IOCs". These are essentially key indicators or signatures we use to distinguish these messages from all the other ones that would be coming in.

For example, as mentioned before we couldn't just filter every message with the words "Microsoft Word" in, as that would be disastrous and cause mayhem by blocking a wide variety of legitimate emails. So we have to chose the indicators we will use to filter malicious messages carefully.

We use the key indicators to determine the detection pattern we put in place, whether it's a specific bit of text we look for, specific range of IP addresses, or other more advanced (and frankly really annoying) methods.

Email Checklist#

157. An email checklist.

When you get an email, you should go through a quick checklist in your head.

Is it really from pagerduty.com? Is it trying to force me into some action? Is it trying to impose a sense of urgency? And so on.

Are the links going to the right place? And if it's asking for information, confirm with the sender via a separate channel. So if someone emails you asking for all the W-2's, ping them on Slack and ask if it really came from them.

And if you get any emails that you're suspicious of, report them to us and we'll take a look.

How Do I Report Phishing?#

158. How do I report phishing?

Which brings us to, how do you actually report phishing to us?

Forward It#

159. Forward it to us.

No more do you need to worry about headers or viewing the original message or any of that stuff. From now on, just forward the email to the security team and we'll take it from there. We have some lovely new tooling which makes it much easier for us to deal with phishing and we can make everyone's lives a bit easier because of it.

Report Phishing Button#

160. A fancy new "Report Phishing" button.

Coming soon will even be a fancy new "Report Phishing" button in Gmail which will make this even easier. You'd just click that, and our tooling will go and quarantine the email from everyone's inbox once enough people have flagged it.

I'm told it will be "lit" 🔥 as I think the kids say these days.

Phishing Changes#

161. Changes to our phishing processes.

Some new email changes you're also going to see soon are a new category called "Sensitive ⚑", where any email containing sensitive information will be automatically categorized and flagged.

Phishing Changes Redacted#

162-164. Changes to our phishing processes.

After 30 days of it being in your account, the email will get locked away and you'll be prompted to tap your Yubikey (or use whatever secondary authentication device you've specified) to unlock it. The only exception to this are password reset emails, which will be locked immediately.

We've been using it for a while on the security team to test it, and it's awesome. It's a very seamless experience and 99% of the time you won't even notice it's there.

But if an attacker ever gets access to your email, one of the first things they're likely to do is search for sensitive information so they can steal it. So this really limits their ability to do that.

This is something called DLP or "Data Loss Prevention" in case you were wondering.

Phishing Summary#

165. Pagey's summary of phishing.

So to summarize all that:

• Be vigilant, check from addresses.
• Confirm all link destinations in emails.
• Be suspicious of urgent language and confirm in Slack if you're unsure.
• And report all phishing to us, even if it's "obvious", as it helps us to track this stuff.

Authentication#

The next topic is authentication. We covered this topic in immense detail the last few times, so this one is going to be a quick topic this year.

Password Checklist#

167. A password checklist.

Hopefully I convinced you all previously that all your passwords should be long, random, unique, and private.

Shameless Plug (1)#

168. Shameless plug #1. Reference

If you need (re-)convincing of that, our previous trainings are all open-sourced on sudo.pagerduty.com, so you can go and check those out. We have a great detailed analysis of how attackers crack passwords.

Use a Password Manager#

169. Use a password manager.

The upshot of all of it really is to use a password manager.

Normally I'd give you a list of different ones you can use, but we're now changing that advice to just...

Use 1Password#

170. Use 1Password. Reference

...use 1Password. So if you're not sure which one to use, just use 1Password.

We're not sponsored by them or anything, although maybe I should've asked them to sponsor this talk now I think about it. But the reason we recommend them specifically is more in the interest of keeping things simple.

1Password is already provisioned for everyone at PagerDuty, so you might as well use the one we know you have access to, and for which we have all sorts of documentation on how to use.

1Password#

171. 1Password.

What a password manager does, is it takes all of your passwords, stores them for you, and keeps them behind one single "master password". So this is then that one password you have to remember, hence the name.

Nuclear Launch Codes#

172. Nuclear launch codes stored in 1Password.

You can store all of your information there, even things like your driving license info or credit card numbers and so on. I do that and consider it safe enough, but your personal threat model might be different.

Notifies Two-Factor#

173. Notifies about two-factor authentication.

They're really helpful too because they'll warn you about things like enabling two-factor authentication if you don't currently have it enabled and the website you're using supports it.

Notifies Breaches#

174. Notifies about breaches.

They'll even integrate with Have I Been Pwned, that site I talked about earlier (more specifically the Pwned Passwords feature), and will warn you if your account has been breached so you know to rotate the password. They have lots of incredibly useful features.

Creating a Really Good Master Password#

175. How do I create a really good master password?

One question we get a lot is, how can you create a really good master password? Because it is very important that the password protecting your password manager isn't something simple, as otherwise that kind of defeats the whole point.

a7hD%^Ht#0Fd{-1GA8Th#

176. a7hD%^Ht#0Fd{-1GA8Th

Well, we have some tips for that! The way we tend to recommend doing it is, first of all, to generate a random password. There are websites that will do this for you, your password manager will do it too (although obviously you can't store this particular one in there, as you have a bit of a chicken/egg problem).

Split Into Chunks#

177. Split into chunks.

You want to type it out, and split it into chunks of four characters.

Remember Chunks#

178. Remember the chunks.

Then try and remember each of those chunks one at a time. Literally, just sit down and memorize them. You'll be amazed at how quickly you can retain the information. After about 5 minutes, stop and come back to it later. You'll find it's much much easier to remember them in chunks.

PageyBank#

179. Just like a credit card.

This is kind of like the idea of the numbers on a credit card. They're split into four characters so that you don't get lost when you're reading it out, similarly for phone numbers and things like that. It's the same principal, you're just breaking it down into smaller components to make it easier to digest.

Repeat#

180. Repeat it over and over.

And then you just start typing it out in full. Putting those chunks back together and repeat it, repeat it, repeat it. Type it out over and over again in a text editor and build up the muscle memory.

You'll be surprised how quickly this will stick after you've done that.

Chalkboard#

181. Like lines on a chalkboard.

I won't like, it can seem like a bit of a chore at first, it might feel like you're writing lines on a chalkboard. But you'll be amazed at how quickly you're able to remember really complicated passwords this way. Soon enough muscle memory will kick in and you'll have it perfectly memorized. And remember, you only have to do this for one password, your master password. Every other password will be protected by your password manager.

Please don't use this exact password though, because obviously this one is now public.

Use Two-Factor Authentication#

182. Use two-factor authentication.

Another part of authentication is that you need to use two-factor authentication everywhere you possibly can. We enforce it for everything at PagerDuty.

Something You...#

183. Something you know, have, and are.

There are three main factors you can provide when you authenticate with a service. Something you know, something you have, and something you are.

2FA Examples#

184. Examples of each factor.

An example would be that a password is something you know, a phone is something you have, and a fingerprint is something you are.

Two-factor authentication is the concept of providing two different ones of those factors, as it massively improves security.

Don't Use SMS#

185. Don't use SMS (unless you have to).

One important point here is to not use SMS for two-factor authentication. Well, if you literally have no other choice, SMS two-factor is better than no two-factor at all, but you're going to be really hard-pressed to find a situation where only SMS is available.

It's much easier to bypass than other methods, and there are various low-skill attacks that make it particularly vulnerable, such as SIM swapping.

Shamesless Plug (2)#

186. Shameless plug #2. Reference

If you want to know more about why, another shameless plug for our previous training which has a section on this, talking about how to intercept and decode SMS messages.

What Can I Use?#

187. So what can I use?

OK, so you can't use SMS, but what can you use?

TOTP#

188. Time-Based One-Time Password (TOTP).

TOTP, or Time-Based One-Time Password, is one that you can use. These are those six digit numbers that change every 30 seconds that always seem to expire just as you're about to use them.

Using these is perfectly acceptable, no problems if you want to continue to use them.

Push#

189. Push notification.

Push notifications are also perfectly fine. People like these because it works with smart watches and so on.

U2F Key#

190. Universal Two-Factor (U2F) Key.

The one we really recommend though is something called a U2F key, which stands for Universal 2nd Factor. This is, in our opinion, by far the best two-factor method in both convenience and security.

It's literally just a USB stick that you plug in, tap a metal contact on it, and that's your authentication.

U2F Key Nano#

191. Itty bitty tiny U2F key.

There are even small ones that can stay plugged in to your laptop all the time, and you just tap it to let you in. They are really convenient.

U2F Examples#

192. Examples of U2F key sizes.

They come in all shapes and sizes, although a word of warning, I would avoid the particular USB-c one you see here, because they seem to crack and break far more than any of the other types we've used.

It's worth noting while we're only showing Yubikey's here, there are other brands out there. U2F is a standard and not just one company's implementation.

Using U2F#

193-197. How to use U2F at PagerDuty.

But login is very easy! You enter your password as normal, then at the two-factor prompt you just tap your finger on the U2F device and you're in. That's it!

So much more convenient than trying to pull out your phone and enter the six digit number, or getting an SMS message and things like that. It really is that simple.

U2F is Best#

198. U2F is the best.

So while U2F keys are by far the best method (in our opinion), it is still perfectly acceptable to use the other ones we've mentioned, we are not taking those away. But we would highly encourage you to use U2F keys. Seriously, did I mention yet how U2F keys are the most convenient two-factor method? I feel like I might have missed mentioning that.

How Do I Set Up U2F?#

199. How do I set up my U2F key?

This is on the wiki as well, so I'm not going to spend too much time on this. But you're probably wondering how you set these U2F keys up. Well, it's self-service so you can do it yourselves!

Setting Up U2F Key#

200-203. How to configure your U2F key at PagerDuty.

Losing U2F Key#

204. What if I lose my security key?

Something you might be thinking of,

What happens if I lose my security key?

Backup Methods#

205. Have backup methods.

Well, it's kind of the same as if you lose your phone and you're using TOTP codes. Make sure you have some backup authentication mechanism. It could be the same method, maybe you have two U2F keys. It could be one of the other methods. Or it could be some backup codes that you physically print out and keep somewhere safe.

If you only have one method and you lose it, you will be locked out of your account. And that's by design, that's security.

If you're using a PagerDuty system, then you may be in luck as one of our HelpDesk administrators will probably be able to unlock your account. But for your personal accounts, if you don't want to potentially lose access, make sure to set up a backup authentication method.

Work and Personal#

206. You can combine work and personal U2F keys.

One thing worth mentioning is that if you do use U2F keys, you can use the same key for work as well as personal sites. The key identifies you, not your specific employer or anything like that.

And yes, your PagerDuty-issued Yubikeys are yours to keep. You do not return those when you leave, so feel free to use them for personal accounts too. It doesn't grant PagerDuty access to your personal accounts or anything like that.

Authentication Summary#

207. Pagey's summary of authentication.

To summarize: Keep your passwords long, random, unique, and private. Use a password manager because they are great. Always use two-factor authentication (but not SMS unless it's your only option). And we strongly encourage you to use U2F security keys.

Did we mention U2F security keys? You should use those.

Physical Security#

Physical security now! This one might be a bit different than previous years, since everyone is working from home now.

Lock Your Computer#

209. Lock your computer.

But even though you're working from home, you should still lock your computer when you're not at it.

You might be thinking "But Rich, that's crazy, do we really care about an attacker breaking into our homes at the exact time we happen to leave our computer unlocked?"

And you're right, we don't really consider that a high priority threat model.

Little Attackers#

210. There are little attackers everywhere.

However, for any of you who have kids or pets, you have little attackers in your house right now. They will tap buttons on your laptop and delete things without you even knowing it. Ask me how I know.

So it's still a good practice to get into the habit of locking your laptop whenever you're not at it.

Hot Corners#

211. Hot corners.

So one of the things I like to recommend is to use the feature in MacOS called "Hot Corners". I have the top-right of my screen set to put my display to sleep (which locks the device). So now it's essentially muscle memory that whenever I get up from my computer, I throw my cursor into the top-right of the screen and everything locks. It's pretty convenient, and once you have the muscle memory of it you don't really think about it.

There are some apps which can use bluetooth and things like that to detect when your smart watch gets far enough away from your computer and lock it for you too, so there are other options out there. But hot corners have been my preferred method for a while, so I wanted to pass that information on.

Losing Laptop#

212. What do I do if I lose my laptop?

OK, so what happens if you lose your laptop (or even worse, it's stolen!)?

If you lose your laptop, whether at home or the office, you need to let our HelpDesk team know immediately.

Lost Laptop Form#

213. The form for reporting a lost device.

They will direct you to a special link which gives you a quick form to fill in with all the information they need. They have a big checklist of things they'll work through to get things nice and secure, and so they can issue and ship you a new device as quickly as possible.

We have a custom short URL for this form, so it might be worth making a note of this somewhere. But don't worry if you can't remember it, our HelpDesk team can walk you through it if ever happens.

Company Issued Devices#

214. Any company-issued equipment.

While this is definitely the case for any company-issued equipment, you should let us know for any personal devices that had access to things like Slack too, because we can help you protect those and make sure they're locked out from continued access and so on.

Even if you lose a personal device that is completely unrelated to work, feel free to reach out to us and we'll be happy to guide you through what precautions you should take.

Back at Office?#

215. What about when we're back?

What about when we're back at our offices?

Three Llama Distance#

216. Stay at least three llamas away.

First of all, we'll want to maintain a good physical distance of at least three small llamas. If you're not sure how wide a llama is, they're about the same width as a Pagey. Hopefully that clears things up for you.

Tailgating#

217. Watch for tailgaters.

But also all the other usual precautions should be taken. One of which is to stay vigilent to check who's following you through doors. I mean, you should do this at home too to be fair.

If people refuse to show or tap their badge, contact building security. Do not get in their way or attempt to stop them yourselves. It's not your job to put yourself in danger like that.

Carry Your Badge#

218. Always wear and carry your badge.

You should also always wear and carry your badge, and report it to HelpDesk if you lose it. Even if you think you just left it at home that day, they can revoke it just for the day to be sure and re-enable it again when you find it. It's not a big deal, and we'd rather err on the side of caution.

Report Suspicious#

219. Report anything suspicious.

And if you see anything suspicious, report it to building security. Remember, it's not your job to confront people or to put yourself in harms way. Just report it if you're unsure.

Shamesless Plug (3)#

220. Shameless plug #3. Reference

I'm not going to talk any more about physical security, as that was our deep-dive topic last year when I went through how to bypass all the physical controls at our offices and things like that.

Once again, another shameless plug for sudo.pagerduty.com.

Physical Security Summary#

221. Pagey's Summary.

So in summary,

• Always lock your computer, even at home.
• Don't let people tailgate your entry into our offices, and report it to building security if that happens.
• Report anything suspicious.
• Report lost or stolen devices immediately to HelpDesk.

It's also worth noting that if you do lose a personal device and it has nothing to do with PagerDuty, you can still let us know and we'll help you out.

Data Handling#

Our final topic of the day, but no less important, is data handling. No more Data from Star Trek puns this year, I promise.

Keep Data Safe#

223. We need to keep data safe.

The main purpose of our data handling policies are because we need to keep our data safe.

No Secrets Here#

224. This cabinet does not contain classified information.

Not just the illusion of safety, like putting it in a drawer marked "nothing secret in here, we promise" or something like that. We need to really keep our data safe.

Types of Data#

225. What are the types of data?

So the big question is, how do we know what types of information are sensitive? Which information needs protecting, and which doesn't?

This comes down to data classifications, which are very similar to our slide classifications, which you've definitely been checking and looking at on every slide, right?

General Data#

226. General data.

The main one is general data. This is anything that is intentionally available to the public. The key word there being "intentionally". Just because something might leak to the public, doesn't automatically make it general data.

Business Data#

227. Business data.

The next is business data, which is anything used to operate the business. Lists of employees and that kind of thing.

Customer Data#

228. Customer data.

And finally, there's customer data. This is any data that is provided to us by the customer.

Customer Data With PII#

229. Customer data with PII.

There is a subset of that too, which is customer data with PII, which stands for Personally Identifiable Information. This would be things like names, home addresses, phone numbers, and so on.

Other Types#

230. Other sub-categories of data.

There are various other subtypes of data too, things like trade secrets, financial data, personal data, or employee data, and so on.

Confusing#

231. It can get confusing with so many.

There's a lot of these sub-categories and it can get pretty confusing. Most of these would be in the context of business data. So taking business data as a whole group, you would have trade secrets within that, which would contain things like company strategy or source code.

We might have employee data, then more important would be employee personal data, and even more important would be PII for employees.

If this is confusing, you're not alone. We have a full breakdown of the our data classification levels along with the various sub-types/sub-categories of data on our wiki. In general though, you only really need to worry about the main classifications of general, business, or customer data.

Data Handling Policies#

232. Data handling policies.

Why do we classify data at all? Well, it's to inform how to handle that data appropriately. Each level of data needs to be handled differently,

• General Data generally has no limits, you can just share it freely.
• Business Data needs to be encrypted, have authentication, and things like that.
• Customer Data has very stringent checks, we need to have auditing and so on.
• Customer Data with PII has more strict policies around key management, etc.

The fines for mishandling data can be really high, so it's important for us to follow our rules here.

WWTBAGFM#

233. Who wants to be a GDPR-fined millionaire?

It's often difficult to know what types of data things fall under though, so I thought we could play a fun game of "Who Wants to be a GDPR-Fined Millionaire".

WWTBAGFM Employee's Name#

234. What type of data is an Employee's Name?

First question, what type of data would an employee's name be?

WWTBAGFM Employee's Name Answer#

235. Business data.

Hopefully this one is pretty clear based on what we talked about before. This would be considered business data, as it's data used to run our business.

WWTBAGFM Employee's Home Address#

236. What type of data is an Employee's Home Address?.

What about an employee's home address?

WWTBAGFM Employee's Home Address Answer#

237. Business data.

Well, it's PII (Personally Identifiable Information) for an employee, but that is still considered business data. It's a sub-category of business data, but still under the business data umbrella.

WWTBAGFM Customer ARR#

238. What type of data is a Customer's ARR?

What about a customer's ARR? Which stands for Annual Recurring Revenue.

WWTBAGFM Customer ARR Answer#

239. Business data.

Even though the word customer is in there, this would still be considered business data, because it's data used to run our business, it's not information that the customer has provided and entrusted to us.

WWTBAGFM User's Phone Number#

240. What type of data is a User's Phone Number?.

What about a user's phone number?

WWTBAGFM User's Phone Number Answer#

241. Customer data with PII.

Hopefully this one is pretty clear cut. This is data provided to us by the customer, and it's personally identifiable. So this would be customer data with PII.

WWTBAGFM Internal PagerDuty#

242. What type of data is in the internal PagerDuty account?.

What about data in the internal PagerDuty account? Is it customer data? Or is it business data because we use it to run our business?

WWTBAGFM Internal PagerDuty Answer#

243. Customer data or Customer data with PII.

While it might technically be business data, we treat it like customer data. This is because it's too easy to make a mistake if we try to treat internal accounts in our product differently than customer accounts. It's much easier and safer to treat all accounts as if they are customer data and not have to worry about one-off exceptions.

WWTBAGFM Aggregated#

244. What type of data is aggregated data?

What about aggregated data? Metrics and analytics that we merge together from multiple customers?

WWTBAGFM Aggregated Answer#

245. Business or Customer data.

This can be either business or customer data depending on the situation. The general rule is that if it can no longer be tied to an individual customer, it's business data. Otherwise it stays as customer data. So really it depends on how the data was aggregated.

WWTBAGFM User IP#

246. What type of data is a User's IP Address?

What about a user's IP address?

WWTBAGFM User IP Answer#

247. Business, Customer, or Customer with PII.

This one comes up a lot, and it's very difficult to answer because it could be one of three different categories. It depends on a lot of variables, and even the jurisdiction you're in. Search Google for "Is an IP address PII?" and you'll find plenty of nuance without a direct clear answer.

Depends on Context and Jurisdiction#

248. Data classification depends on context and jurisdiction.

What you'll find with a lot of these things is that it depends on context and the jurisdiction you're operating in. Some are very clear cut, and others require reading laws and interpreting them. This is generally when we'll involve our legal team for an opinion on the matter.

So if you're unsure about any of this, come and ask us, and we'll figure it out!

Safely Share Data#

249. How can I safely share data?

And the final question to answer today, how do you safely share data?

Google Drive#

250. Use Google Drive or Box.

For us, you want to use Google Drive or Box. Both of these are tools we have that we have vetted for sharing data both within PagerDuty and externally to customers when needed. Please be sure to limit who has access to the files, don't make them available to the public.

We have checks in place for files which are shared with very wide access, so you can expect a member of the Security Team to reach out if that happens to make sure it wasn't a mistake.

Be Careful#

251. Be careful when sharing data outside PagerDuty.

So be very very careful when sharing data. We have guardrails in place, but they're not foolproof. If you're copying things or sharing with specific people, make sure you share with the correct people and don't rely on email autocompletion to get the correct group. You don't want to share sensitive information with the wrong people by mistake for example.

Data Sharing Checklist#

252. A data sharing checklist.

As with the other things, think about a checklist when you're going to share data. Know the classification, make any redactions, identify who to share it with, and things like that.

Don't Discuss in Public#

253. Don't discuss company information in public.

And remember, don't discuss company information in public! If you start discussing material non-public information in public, you're going to get the SEC after you, and that's something you probably don't want.

See Something, Say Something#

254. If you see something, say something (to the security team).

One final piece of advice for today: If you see something, say something. If you find you have access to information you shouldn't, please let us know. You're not going to get in trouble, and we'll be happy to help close any potential security holes that have been found.

If you have access to a system you shouldn't, that's never going to be your fault, it's our fault for configuring the system in a way that lets you access the information.

Data Handling Summary#

255. Pagey's summary of data handling.

So a summary of all that:

• Know the classification of the data you're handling, if you're unsure, come and ask us!
• Don't move data between system types. No production data on staging or things like that.
• Don't discuss company information in public.
• Share files via vetted tools only.
• Restrict access to shared files, don't make them public.

I Want to Know More#

So that's all the topics I had for us today! You're probably now wondering where you can find out more about all this stuff, right?

Slack Security#

257. The #security channel in Slack.

If you want to know more about any of this, come and join us in Slack in the #security room. Feel free to ask us any questions you have. If you have an important question, the current on-call is always listed in the topic, so you can feel free to @-mention them.

Please follow good Slack etiquette though and don't use @here, because you will notify 100s of people who will be upset about it.

Slack Security Watercooler#

258. The #security-watercooler channel in Slack.

We also have a #security-watercooler room, which is where you can post general security conversation, funny tidbits, links to articles, or any memes you've found.

Internal Wiki#

259. Our internal wiki has lots of useful information.

We also have our internal wiki with lots of information, best practices, design documents, recommended reading, and things like that. It's full of useful information, so definitely check it out if you want to know more about the things we do.

Office Hours#

260. Office hours.

We also have office hours, every Wednesday! Come and say hi and ask any pressing security questions you may have.

The Team, Again#

261. Lovely faces of the security team.

And here's a quick reminder of who everyone on the security team is and all their lovely faces.

Animal Choices#

262. The choices for the animal.

The time has now come. I know you're all waiting for the animal. Remember that it was going to be one of these this year, so think about what your guess was at the start and see if you're right.

Giraffe#

263-267. Giraffe!

For this year, the animal is GIRAFFE!

Any reasonable spelling will be fine. I actually got it wrong myself the first time (fine, the first three times).

That's All Folks#

268. That's all folks!

And that's all we have for you!

I know you didn't have a choice of whether to attend this, but I hope you got something useful out of it. If you have any questions feel free to reach out to us via Slack.

Now get back to work!

Image Credits#

269. Image credits.