Podcast: Break Things on Purpose | Carissa Morrow: Learning to be Resilient

‍

Being new in tech an be intimidating! Thankfully, folks like Carissa Morrow are shining examples of how to come into tech from the ground up. Carissa began with a career shift and just started coding, went through the Boise Codeworks bootcamp, and made the jump to tech. Carissa talks about the resilience it took in her early days, and how those experiences reinforced her attitude on continually learning. Carissa levels a focus on keeping things fresh to avoid the “organizational death spiral” that only a new set of eyes can bring to the table. Check out the conversation for more!

Episode Highlights

In this episode, we cover:

• 00:00:00 - Introduction
• 00:02:00 - Carissa’s first job in tech and first bootcamp
• 00:04:30 - Early Lessons: Carissa breaks production—on a Friday!
• 00:08:40 - Carissa’s work at ClickBank and listening to newer hires
• 00:10:55 - The metrics that Carissa measures and her attitude about constantly learning
• 00:16:45 - Carissa’s Chaos Engineering experiences
• 00:18:25 - Some advice for bringing new folks into the fold
• 00:23:08 - Carissa and ClickBank/Outro

Links:

• ClickBank: https://www.clickbank.com/
• LinkedIn: https://www.linkedin.com/in/carissa-morrow/

Transcript

Carissa: It’s all learning. I mean, technology is never going to stop changing and it’s never going to stop being… a lot to learn, [laugh] so we might as well learn it and try to keep up with the [laugh] times and make our lives easier.

Julie: Welcome to Break Things on Purpose, a podcast about reliability, asking questions, and learning from failure. In this episode, we talked with Carissa Morrow about what it’s like to be new in tech, and how to learn from mistakes and build your skills.

Julie: Carissa, I’m really excited to talk to you. I know we chatted in the past a little bit about some horror stories of breaking production. I think that it’s going to be a lot of fun for our listeners. Why don’t you tell us a little bit about yourself?

Carissa: Yeah, so I actually have only been in this industry about three years. So, I come with kind of a newbie's perspective. I was a certified ophthalmic tech before this. So, completely different field. Hit my ceiling, and my husband said, “You want to try coding?” I said, “Not really.” [laugh]. But I did. And I loved it.

So, long story short, I ended up just signing up for a local boot camp, three-month full stack. And then I got really lucky; when I graduated there and walked into my previous employer’s place. They said, “Do you know what DevOps is?” I said, “I have no idea.” And they still hired me.

And it was really great, really, really great experience. I learned so much in a couple years with them. So, and now I’m here at ClickBank and I’m three years in and trying not to break things every day, especially on a Friday.

Julie: [laugh]. Why? That's the best day to break things, Carissa—

Carissa: [laugh]. No, it’s really not.

Julie: —preferably at 4:45. Well, that’s really amazing. So, that’s quite the jump. And as you mentioned, you started with a boot camp and then ended up at an employer—and so, what was your role? What were you doing in your first role?

Carissa: So, I started on a really small team; there was just three of us including myself. So, I learned pretty much everything from the ground up, knowing nothing coming into DevOps. So, I had, you know, coding background from the boot camp, but I had to learn Python from scratch. And then from there, just kind of learning everything cloud. I had no idea about AWS or Google or anything in the cloud realm.

So, it was very much a rough—very, very rough first year, I had to put my helmet on because it was a very bumpy ride. But I made it and I’ve come out a heck of a lot stronger because of it.

Julie: Well, that’s awesome. How about do you have people that you were working with that are mentoring you?

Carissa: Yep. So, I actually have been very lucky and have a couple of mentors, from not only my previous employer, but also clients that I worked with that have asked to be my mentor and have stuck it out with me, and helped not just in the DevOps realm or the cloud realm, but for me as a person in that growing area. So, it’s been pretty great.

Julie: Well, that’s awesome. And I guess I should give the disclosure that Carissa and I both worked together, for me a couple of jobs ago. And I know that, Carissa, I’ve reached out to you for folks who are interested in the boot camp that you went through. And I know it’s not an advertisement for the boot camp, but I also know that you mentored a friend of mine. Did you want to share where you went?

Carissa: Yeah, definitely. So, I went to Boise CodeWorks, which is a local coding school here in Boise. And they did just move locations, so I’m not quite sure where they’re at now, but they’re definitely in Boise.

Julie: And if I remember correctly, that was a three-month very intensive, full-time boot camp where you really didn’t have time for anything else. Is that right?

Carissa: Yes, it is absolutely 1000% a full-time job for three months. And you will get gray hairs. If you don’t, you’re doing something wrong. [laugh]. Yep.

Julie: So, what would you say is one of the most important things you learned out of that?

Carissa: I would say just learning how to be resilient. It was very easy to want to quit because it was so difficult. And not knowing what it was going to look like when I got out of it, but part of me just wanted to throw my hands up half the time. But pushing through that made it just that much sweeter when I was done.

Julie: Well now, when we were talking before, you mentioned that you broke production once. Do you want to tell me about that—

Carissa: Maybe a few times. [laugh].

Julie: —[crosstalk 00:04:34] a few times? [laugh]. You want to share what happened and maybe what you learned from it.

Carissa: Yeah, yep. So, I was working for a company that we had clients, so it was a lot of client work. And they were an AWS shop, and I was going in to kind of clean up some of their subnets and some of their VPN issues—of course, this is also on a Friday. Yeah. It has to be on a Friday.

Julie: Of course.

Carissa: So, I will never forget, I was sitting outside thinking, “This is going to be a piece of cake.” I went in, I just deleted a subnet, thinking, “That’s fine. Nothing’s going to happen.” Five minutes later Slack’s blowing up, production’s down and, you know, websites not working. Bad. Like, worst-case scenario.

So, back then we had, like, a team of, I think I would say ten, and every single person jumped on because you could tell I was panicking. And they all jumped in and we went step-by-step, tried to figure it out, figured out how we could fix it. But it took a good four hours of traumatizing stress [laugh] before we got it fixed. And then I learned my lesson, you know? Double-triple check before you delete anything and try to just make Fridays read-only if you can. [laugh].

Julie: Well, and I think that’s one of the things right? You always have to have that lesson-learning experience, and it’s going to happen. And showing empathy for friends during that, I think, is the really important piece. And I love the fact that you just talked about how the whole team jumped on because they saw that you were stressed out. Were you in person or remote at the time?

Carissa: I was remote at the time.

Julie: Okay.

Carissa: Yeah. And we were traveling in our RV, so nothing like being out in the woods, panicking by yourself, and [laugh] roaming around.

Julie: So, did you run a postmortem on it?

Carissa: So, back then—actually, we ended up doing that, yes, but that was when I had never really experienced a postmortem before, and that’s one thing that, you know, when we talk about this kind of stuff—and everyone has a horror story or two, but that’s something that I’ve had to learn to get better at is RCAs and postmortems because they’re so important. I think they’re incredibly important. Because these things are going to happen again; they’re going to happen to the best of us. So, definitely, everything is a learning experience. And if it’s not, you’re missing out. So, I try to make everything a learning experience, for sure.

Julie: Absolutely. And that’s one of the things we talk about is now take that, and how do you learn from this? And how do you put the gates in place so that you can’t just delete a subnet? I mean, to be fair, you did it, but were there other things that could have prevented this from happening, some additional checks and balances?

Carissa: Mm-hm.

Julie: And as you mentioned, that’s not the only time that you’ve broken production. But let me ask you was that—did the alerting mechanisms work? Did all of the other—did the monitoring and observability? Like, did everything work correctly, or did you find some holes in that as well?

Carissa: So, that’s a great question. So, this specific client did not use any monitoring tools whatsoever. So—

Julie: Huh.

Carissa: Yeah, so that was one of those unique situations where they just tried to get on their own website and it didn’t work. And then, you know, it was testing and everything was failing. But it was all manual testing. And I actually—believe it or not—I’ve seen that more often than I ever thought I would in the last three years. And so with what you guys do, and kind of what I’m seeing with a bunch of different clients, it’s not just do they have monitoring, it’s how do they use that? And when it’s, kind of, bits and pieces here and there and they’re not using it to their full potential, that’s when a lot of things slip through the cracks. So, I’ve definitely seen a lot of that.

Julie: Absolutely. And it’s interesting because I really think that, especially these advanced organizations, that they’re just going to have all the ducks in the row, all the right monitoring setup, and it turns out that they don’t always have everything set up or set up correctly. And that’s one of the things that we talk about, too, is validating with Chaos Engineering, and looking at how can we make sure it’s not just that our systems are resilient, but that our tools pick things up, that our people and processes work? And I think that’s really important. Now… you’re working at ClickBank today?

Carissa: Mm-hm.

Julie: You want to tell us a little bit about that and about what you do over there?

Carissa: Yep. So, I came on a few months ago as a cloud engineer for their team. And they are—I have actually learned a lot of monitoring tools through what they have already set up. And as they’re growing and continue to grow, I’m learning a lot about what they have in place and maybe how we can improve it. So, not just understanding the metrics has been a learning curve, but understanding what we’re tracking, why, and what’s an emergency—what’s critical, what’s not—all of those things is definitely a huge, huge learning curve.

But regardless of if it’s ClickBank or other companies that I know people that work out or I’ve worked at, everyone knows there’s a humbling aspect when you’re using all these tools. We all want to pretend like we know everything all the time, and so being humble enough to ask the questions of, “Why do we use this? Are we using it to its full potential? And what am I looking at?” That’s how I’ve learned the most, even in the last couple of months here is just asking those very humbling questions.

Julie: Well, I have to say, you know, you mentioned that you are really still new; it’s three years out of school for you doing this, and I think that there actually is quite a lot to be said about listening to newer people because you’re going to ask questions that other folks haven’t thought of, like, the whys. “Why are we doing things this way?” Or, “Why are we tracking that?” And sometimes—I think you’ve probably seen this as organizations—we just get into these habits—

Carissa: Mm-hm.

Julie: —and we do things because somebody who worked here, like, five years ago, set it up that way; we’ve just always done it this way.

Carissa: Mm-hm.

Julie: And it’s a great idea to look into some of our practices and make sure that they’re still serving us. One thing that you mentioned that I love, though, is you said metrics. And metrics are really important when practicing Chaos Engineering because it’s good to know where you are now so that you can see improvement. Can you talk about some of the metrics that you measure or that might be important to ClickBank?

Carissa: Yeah. So, a lot of the things that we measure have to do with orders. So, the big thing with ClickBank with how the model, the infrastructure of this company is set, orders are incredibly important, so between the vendors and the buyers in ClickBank. So, we are always monitoring in great detail how our orders are coming in, going out, all the payment information, you know, make sure everything’s always secure and running smoothly. So, those are where most of our metrics that we watch where those live.

The one thing that I think is—I’ve noticed is really important is whether you’re monitoring one thing or ten, monitor to the best of your ability so that you’re not just buying stuff and using 50% of it. And I think we get really excited when we go and we’re like, “Yes, this is a great third-party tool or third-party—we’re going to use it.” And then 10% of it, you know, you use and the rest of it, it’s like, “That’s really cool. Maybe we’ll do that later, maybe we’ll implement that part of it later.” And that’s something that it’s just, it’s like, I know it’s painful, [laugh] but do it now; get it implemented now and start using it, and then go from there.

But I feel like why do we bother if we’re only going to use 10% to 50% of these amazing things that really make our lives easier, and obviously, more secure and more resilient.

Julie: I think you’re onto something there. That is really good advice. I remember speaking at a conference in New Zealand and one of the speakers there talked about how their organization will buy any new tool that comes out, any and every new tool that comes out. But just buying that—and as you mentioned, just using a tiny, small portion of that tool can really be kind of ridiculous. You’re spending a lot of money on these tools, but then these features were built for a reason, and oftentimes—and I saw this, too, at my past company—folks would purchase our tool, but not realize that our tool did so many other things.

And so then there are multiple tools that are doing the same things within an organization when in reality, if you look at all the features and truly understand a tool—I would say some folks have a hard time with saying well, it just takes too much time to learn all of that. What’s your advice for them?

Carissa: Yeah. I think I’ve caught myself saying that to [laugh] at some point in time. You know, the context-switching, already having our full-time jobs and then bringing on tools, other tools that we need to learn. And it is overwhelming, but my advice is, why make more pain for yourself? [laugh]. Why not make your life easier, just like automation, right?

When you’re automating things, it’s going to be a lot of work up front, but the end goal is make everything more secure, make it easier on yourself, take out the single point of failure or the single-person disaster because they did one wrong thing. Monitoring does the same thing. You know, if you put the investment up ahead of time, if you do it right upfront, it’s going to pay off later.

The other thing I’ve seen, and I’ve been guilty of as well is just looking at it and saying, “Well, it looks like it’s working,” but I don’t really know what I’m looking at. And so going back to that, you know, if you don’t know why things are failing, or what to look out for to catch things from failing, then why even bother having that stuff in front of you? So, it’s a lot of learning. It’s all learning. I mean, technology is never going to stop changing and it’s never going to stop being… a lot to learn, [laugh] so we might as well learn it and try to keep up with the [laugh] times and make our lives easier.

There was actually a—I wrote this quote down because I ran across this last week, and I loved it because we were talking about failures. It said, “Not responding to failures is one characteristic of the organizational death spiral.” And I loved that because I sat there and thought, “Yeah, if you do have a failure, and you think, ‘Well, I have my monitoring tools in place. It looks like it worked itself out. I don’t really know what happened.’ And that continues to happen, and everyone on the team has that same mentality, then eventually, things are going to keep breaking, and it’s going to get worse and worse over time.” And they’re not going to realize that they had a death spiral. [laugh]. So, I just love that quote, I thought that was pretty great.

Julie: I love that as well, who was that from?

Carissa: Oh, I’ll have to pull it up, but it was online somewhere. I was kind of going through—because really bothering me when we were talking about some of our monitoring, and I was asking some kind of deep questions about, why? What’s the critical threshold? What’s the warning? Why are we looking at this? And so I started looking at deeper dives into resiliency, and so that popped up, and I thought that was pretty spot on.

Julie: I love it. We will find the author of that. We’ll post it in the show notes. I think that is an amazing quote. I think I’m going to steal it from you at some point because that’s—it’s very true.

And learning from those failures and understanding that we can prevent failures from occurring, right? So—

Carissa: Absolutely.

Julie: —if you have a failure and you’ve remediated it, and you still want to test to make sure that you’re not going to drift back into that failure, right? Our systems are constantly changing. So, that’s one of the things we talk about with Chaos Engineering, as well, and building that reliability in. Now, have you experienced or practiced Chaos Engineering at all with any of your customers that you’ve worked on, or at ClickBank?

Carissa: There was one, [sigh] one client that we had that I would say yes, but the testing itself needed to be more robust, it needed to be more accurate. It was kind of like an attempt to build testing around—you know, for Chaos Engineering, but looking back now, I wish we would have had more guidance and direction on how to build really strategic testing, not just, “Oh, look, it passed.” It might have been a false pass, [laugh] but it was just kind of absolute basic testing. So, I think there’s a growth with that. Because I’ve talked to a lot of engineers over the years that we say testing is important, right, but then do we actually do it, especially when we’re automating and we’re using all these third-party tools.

A lot of times, I’m going to go with what we don't. We say it’s really important, we see the importance of it, but we don’t actually implement it. And sometimes it’s because we need help to be able to build accurate testing and things that we know really are going to be sustainable testing. So, it’s more of probably an intimidation thing that I’ve seen over the years. And it’s kind of going back to, we don’t like to ask for help a lot of times in this industry, and so that plays a role there. Sometimes we just need help to be able to build these things out so we’re not walking on eggshells waiting for the next thing to break.

Julie: Now, I love it because you’ve drilled down kind of into that a few times about asking for help. And you’ve worked with some folks that I know you’ve done a great job. So far, I’m really impressed just seeing your growth over the last three years because I do remember your first day—

Carissa: Oh—

Julie: [laugh].

Carissa: [laugh]. Oh, God.

Julie: —and seeing you and in these little corner cubes. That was—[laugh]—

Carissa: I was sweating bullets that day.

Julie: —quite a long time ago. What advice would you give to senior folks who are helping newer folks or more junior folks? What would you want them to know about working with newer people?

Carissa: Yeah, that’s a good question. So, in my last job, I actually ended up becoming a lead before I left. And so [sigh] the one thing I learned from my mentor at my previous company that really just brought me up from knowing nothing. One thing I learned from him was, when he looked at me on the first day, he said, “Do not be afraid to ask for help. Period. Just don’t. Because if you don’t, something bad’s going to happen and you’re not going to learn and you’re not going to grow.”

And he also was one that said, “Put your helmet on. It’s going to be a bumpy ride.” [laugh]. And I loved that. He even got me a little, uh—oh, it kind of like—it was a little bobblehead, and it had a helmet. [laugh]. And I thought that was so spot-on.

I think we forget when we get really good at something or we’ve been doing something for a while, as human beings, we forget what it’s like to be new, and to be scared, and to not know what our left and right hand is doing. So, I would say keep that in the forefront of your mind as you’re mentoring people, as you’re helping ramp them up, is they’re going to be afraid to ask questions or remind them it’s okay, and also just taking a step back and remembering when you were really new at something. Because it’s hard to do. We all want to become experts and we don’t want to remember how horrible that felt when we did not know what was in front of us. So, that would be my couple pieces of advice.

Julie: Well, and then kind of circling back to that first time that you broke production, right, and everybody rallied around to help you—which is amazing; I love that—after it was over, what was the culture like? Were they supportive? What happened?

Carissa: Yeah, that’s a really good question because I’ve heard people’s horror stories where it was not a good response afterwards, and they felt even more horrible after it was fixed. And my experience was a complete opposite. The support was just 1000% there. And we even hung out—we started a Zoom call and after we’d fixed it, there were people that hopped back on the call and said, “Let me tell you about my production story.” And we just started swapping horror stories.

And it was 1000% support, but also it was a nice human reminder that we break things and it’s okay. And so that was—it was a pretty great experience, I hope the best—we’re all going to break things, but I hope that everyone gets that experience because the other experience, no fun. You know, we already feel terrible enough after we break it. [laugh].

Julie: I think that’s important. And I love that because that goes back to the embrace failure statement, right? Embrace it, learn from it. If you can take that and learn. And what did you learn? So, you mentioned you learn double, triple, quadruple check.

Carissa: Mm-hm.

Julie: So, have you made that same mistake again?

Carissa: I have not. Knock on wood. I have not. [laugh].

Julie: [crosstalk 00:21:58]

Carissa: [crosstalk 00:21:59]. [laugh].

Julie: It could happen—

Carissa: Yep.

Julie: —as we all are learning so much, sometimes you make the same mistake twice, right?

Carissa: Yeah, absolutely. I would say there’s two things. So, I learned that, and then I also learned that not just double and triple check before you do something, but going back to the don’t be afraid to ask questions, sometimes you have to ask clarifying questions of your client or your customer before you pull the trigger. So, you might say I’ve done this a million times, but sometimes the ask is a little vague. And so, if you don’t ask detailed questions, then yes, you might have done what needed to be done, but not in the way that they hoped for, not in the way that they wanted, your end game results were now not what was hoped for.

So, definitely ask layered questions if you need to. To anyone: To your coworkers, to your manager, to your whoever you’re using your monitoring tools through. Just ask away because it’s better to do it upfront than to just try to get the work done and then, you know, then more fun happens.

Julie: More fun indeed. [laugh].

Carissa: [laugh]. Yes.

Julie: Now, why don’t you tell our listeners who aren’t familiar with ClickBank, do you want to promote them a little bit, talk a little bit about what you’re doing over there?

Carissa: Yeah. So, ClickBank is awesome, which is why I’m there. [laugh]. No, they’re a great company. I’m on a fairly, I wouldn’t say large team, but it’s a good-sized team.

They’re just really good people. I think that’s been one of the things that’s incredibly important to me, and I knew when I was making a switch that everyone talks about, they have a great working environment, they have great work-life balance. And for me, it’s like you can talk the talk, but I want you to walk the walk, as a company. And I want—you know, if you say you’re going to have a family environment, I want to see that. And I have seen that at ClickBank.

It’s been an awesome couple of months. There’s a lot of support on the teams. There’s a lot of great management there, and I’m kind of excited to see where this goes. But coming with a fresh perspective of working at ClickBank, it’s a really great company. I’m happy.

Julie: Well, I love that. And from what I’m aware of, y’all have some positions that are open, so we’ll post a link to ClickBank in the as well. And, Carissa, I just want to thank you for taking the time to be a little vulnerable and talk about your terrifying breaking production experience, but also about why it’s so important to be open to folks asking questions and to show empathy towards those that are learning.

Carissa: Mm-hm. Yeah, absolutely. I think that is the number one thing that’s going to make us all successful. It’s going to make mentors more successful, and they’re going to learn as they’re doing it and it’s going to make—it’s going to build confidence in people that are coming into this industry or that are new in this industry to say, “Not only can I do this, I’m going to be really great. And I’m going to eventually mentor somebody someday.”

Julie: I love that. And thank you. And thank you for spending time with us today. And, folks, you can find Carissa on LinkedIn. Pretty impressed that you’re not on Twitter, so not a huge social media person, so it’s just LinkedIn for Carissa. And with that—

Jason: For links to all the information mentioned, visit our website at gremlin.com/podcast. If you liked this episode, subscribe to the Break Things on Purpose podcast on Spotify, Apple Podcasts, or your favorite podcast platform. Our theme song is called Battle of Pogs by Komiku and is available on loyaltyfreakmusic.com.