Jim Hirschauer: Alright.
Welcome to ShipTalk, the SRE
edition.
I'm Jim Hirschauer, your host
for today.
ShipTalk is a DevOps podcast,
brought to you by Harness, the
software delivery platform, and
the SRE edition focuses on
reliability topics.
My guest today is Hilliary
Lipsig from Red Hat.
Hilliary, welcome to the show.
Hilliary Lipsig: Thanks, Jim.
Happy to be here.
My name's Hillary Lipsig.
I am Chief Mermaid at Red Hat.
And if you look me up on
LinkedIn, that's really there.
If you've seen my talks, that's
my title.
And the reason for that title is
because I am the technical lead
for a small subdivision in Red
Hat called Strategy Enablement
and Architecture out of our
larger service delivery
organization.
And when they said, Hey You
really can't have principal site
reliability engineer on your job
title anymore because you're not
really, you know, an SRE that
way.
What do you think a good
business card title would be for
you?
I said chief Mermaid, officer
because I was finally my
opportunity to grow up and be a
mermaid like I'd always wanted.
So they laughed but we did it.
So that is that is my official
unofficial business card title
which is delightful.
Jim Hirschauer: That's probably
the greatest title I've heard so
far, so congrats on that.
Hilliary Lipsig: Thank you.
I'm very proud of it.
Someday they will probably make
me change it, but today is not
that day.
Jim Hirschauer: Awesome.
Keep it as long as you can, for
sure.
Well Hilliary, you know, we've
talked about the format of the
show We have a couple of
sections that are called just
for fun, and then we get into
the main topic.
So we're gonna start off with a
just for fun section.
And we, when we were talking
earlier, you, you used a word
that I had never heard before,
so I'm not sure if you made up
this word, or at least you're
like a really early adopter of
this new word, but I feel like
we should all know what this
word is.
So why don't you go and explain
that.
Hilliary Lipsig: Sure.
So people tell me, I made up
this word.
I do not believe it, but the
people I thought I learned it
from insisted they learned it
from me.
So who knows where this came
from?
So when you're looking at a pod
in Kubernetes and it has an out
of memory or an OOM somewhere
along the way I said, Ugh,
that's ooming.
And people are like, what?
And I said, yeah, ooming.
It's like, you know, you say oom
for out of memory.
It's, it's ooming, it's a verb.
It's delightful.
And this has caught on to some
degree here in Red Hat.
Others have picked it up.
Again, I'm credited with this.
I'm certain I did not come up
with this.
I must have heard this
somewhere.
So if anybody knows who I
learned it from, they deserve
all the accolades.
But I will tell you that Like I
said, it's delightful.
And I have been on a call with a
customer that was very serious,
very tense.
They were rightfully so, really
upset.
And I used the term and then I
paused to explain the term and
you know, it's not like
magically everything was better,
but like there, suddenly there
were smiles on the call.
People were laughing and it, it
seriously reduced the whole
tension of the situation because
I wasn't taking myself or
anything very seriously.
Or the right level of levity is
really what I wanna say is what
that brings into the
conversation.
Yeah.
And so then we got back to our
very serious,
Jim Hirschauer: discussions.
Well, that's awesome.
I I love the word, and I, I'm
curious though, can you use the
word a little more generically
or does it have to be like,
really related to an out of
memory error?
Can, can I use that word?
If, let's say the system is just
like chugging along and
performing poorly, can I say,
oh, it's, it's ooming.
Is it almost like googling
something?
Hilliary Lipsig: No, I, I do
feel like it is kind of pretty
specific because it maps back to
a very, like, you see OOM
visually on the like OC get pod,
or not oc I'm sorry, that's a
very OpenShift thing.
kubctl, like get pods or
whatever.
Mm-hmm.
I work in OpenShift.
We have our own CLI.
It's opinionated and I have that
completely memorized, much to my
occasional detriment when I
wonder why it doesn't work on
vanilla Kubernetes.
Jim Hirschauer: Right.
All right, good.
So I've learned today that I can
use this word ooming if I see an
out of memory exception, so next
time I see one, that's
definitely, it's ingrained in my
memory now for sure.
I love it.
I feel like I'm just gonna give
you credit cause you're the
first person I've heard it from,
so if anyone asks me, I'm gonna
say I heard it from you.
And origin being unknown, I
think you really deserve the
credit at this point.
If it really starts taking off.
Hilliary Lipsig: I will humbly
accept that credit, but thank
you.
Jim Hirschauer: Okay.
Alright.
So let's get a little more
serious now and, and jump into
our main topic.
So You know, we had a chance
yesterday to chat a little bit
before the show, and you were
talking about observability and,
you know, I've been in and
around the observability space
personally for over a decade.
My background as a practitioner
goes back into the monitoring
realm, which is kind of the, I'd
say the, the granddaddy of, of
modern observability and.
I think most of us that are
either on the show right now or
listening to the show, we all
know how and why we should be
using observability to identify
and resolve production
incidents.
But yesterday you said to me
that we can, and we should be
doing more with all of that
data.
So I'd love for you to explain
your position on that.
Hilliary Lipsig: Sure.
And I'd like to hope this isn't
a controversial position one of
the.
One of the projects that I'm,
I'm working on with Red Hat is
it's a data project and we were
working through the personas of
how, how do we be a more
data-driven organization and,
and what kind of data do we
really need to be that?
And so, and I swear this is not
an original thought.
I'm sure I learned this as from
somebody who was much smarter
than I am.
But.
I said that, you know, one of
the things that we could be
doing is actually demonstrating
our cloud like usage metrics.
So what are we doing with CPU
and what are we doing with
memory?
And then using that to compare
to, you know, when we're pla
when we're doing these plannings
for these systems, we typically
have some sort of profile.
So the first is, of course, that
profiling, like where are we in
our, our usage compared to how
we you know, how we forecasted
to make sure we're, you know,
kind of keeping track of that.
That'll all maps back to spend,
which everybody is really caring
about right now which you should
always care about actually.
And the other thing that I said
was, you know, It's important to
watch trends as well in these
usage and not just like looking
over the course of the month or
course of a day, but looking a
little bit more granularly.
And so there's actually an
example from the SRE team that
I, I joined at Red Hat actually
as to that was a team I came
onto that I'm, I'm no longer a
part of although I still love
them and they love me, I think.
But anyway, the There was some
thresholds that were set up in
Prometheus that would create
warning alerts and it would just
be for things like CPU and
memory and so forth.
And because of these thresholds
we noticed things like memory
leaks in the code.
And there's a couple things that
like happens when a memory leak
is happening, a couple symptoms.
So it's not just that your
memory usage goes up and up and
up.
But actually your CPU will start
churning and churning and
churning even in virtualized
hardware.
It will work harder and harder
to try and reclaim as much
memory as possible to try to
prevent an out of memory.
And so taking a look at that
kind of data, and looking at
that data a little bit more
granularly, we can start
actually looking for issues in
our software.
And of course, so we had set
alerts on that to let us know
that there might be a memory
leak.
But you can actually with more
data, cuz again, that was one
simple Prometheus rule.
You can actually get a little
bit more intelligent about it as
well.
And so, That's one of the things
that I am trying to drive this
kind of a, a cultural push
within Red Hat is and I know I'm
not alone in this, I really
don't wanna say that there's
nobody else doing this.
It's just I've, I've, I've
joined onto this to this type of
an effort.
It's this idea that we should be
using our metrics to look and
see if we can be improving our
software.
Because if we profile our
software and then we run our
software with like, The profiles
we've defined.
So we've got memory limits and
CPU requests and yes, CPU
limits.
I know that CPU limits are like,
people are like, you don't need
them.
I will argue that it's a whole
other 25 minute conversation.
And, but yes, CPU limits as
well.
And so, If we're constantly
running close to your limits or
even occasionally, like hitting
the limits and then things
restart and run fine for a while
those might not necessarily all
result in incidents or downtime
or alerts, but all these things
tell you that your software's
probably not operating as well
as it could be.
And so, We wanna look into why
is our software not operating as
well as it could be Because the
nature of, of working in a cloud
native environment or Kubernetes
is often that I have seen.
Sometimes certain things can
kind of be hidden for a while
because it's highly available
because it restarts and brings
itself back up to a desired
state.
There might not be any seriously
noticeable blip but that doesn't
mean that the blips that exist
are acceptable either.
Jim Hirschauer: Yeah.
So, you know, you make an
interesting point.
Being proactive with this type
of activity.
First of all, most companies
have this type of data.
They have observability
solutions in-house or in place
at least.
And there's a, a wealth of data
at most companies.
What I think is interesting
about this is, is you made the
point really early on that we
should always be concerned with
costs.
And what I've seen happen is to,
over provision your resources
with your cloud computing
provider, or we used to do this,
you know, back in the days of
the plain old data center, we
just used to over provision the
heck out of our hardware so that
we could avoid these issues for
as long as possible, avoid the
out of memory exception from
kicking in and, and causing an
error, causing the impact to our
users.
But that's in my mind,
especially in today's world, is,
completely unacceptable.
That goes out the window and we
have to do a better job, as an
IT community of creating
services that respond and behave
properly, that we don't have to
over provision and, and really
waste a ton of money on.
So I think you're really onto
something here.
You know, just besides the fact
that you can make an overall
better customer experience by
having better performing
software, you can end up saving
your company significant cost
across all of the different
services that most companies
have due to that over
provisioning.
Hilliary Lipsig: Yeah.
And this is actually one of the,
this goes into the, and I, I, I
swear I did not intend to derail
our entire conversation into CPU
limits.
But it actually, it actually
came up very recently and I
explained to somebody all the
reasons why I'm in favor of
them.
And I'll give a, a high level
overview here, which is one, I'm
not only dealing with
virtualized hardware, I'm
dealing with real hardware.
So, I can't just let processes
do whatever they want.
It actually is, is finite.
And if you're overcommitting
your CPU on real hardware,
you're shortening your
hardware's lifespan.
So if you think about how
expensive a server is, first of
all, that's extremely wasteful
to short its lifespan.
Second of all, it's drawing more
energy, which is extremely
wasteful cause it causes more
pull there and more actually.
Impacts the overall ambient
temperature, which means the AC
needs to run wherever that
server is located more often.
So as you said, it's not just
about the immediate company's
fiscal situation, but then we
start getting into the
environmental impacts of not
having performant software.
Which I think about all the
time, because when I'm thinking
about the size of, of the Red
Hat customer base and, and so
forth.
These, this enterprise scales.
These are actually things that
start coming up.
We have projects around
sustainability and, and
environmental impact as I think
a lot of major companies do
these days.
And so it's, these are the types
of things that, you know, we're
caring about.
These are the types of ways
we're wanting to use data.
And since we already have the
data, it.
Honestly would be almost
irresponsible not to use it to
figure out how to make
everything better holistically
and looking at the holistic
picture of our software.
And since I'm somebody with an
operational background, I care
about how it's performing.
Jim Hirschauer: Yeah, makes
sense.
So let me ask you this.
Sometimes it's really easy to
say we should do this thing.
The reality that most companies
are faced with and, and most
SREs or folks who are
responsible for reliability and
performance of overall systems
is that they are typically
pretty far behind in their main
job responsibilities.
So, what advice can you give
folks to start to transition
from using their observability
data in a reactive methodology
where they're just putting out
fires all the time to being able
to truly switch over into this
proactive mode where they're,
looking at things ahead of time
and, and trying to make sure
that the systems are actually
performant in the right way so
that they can right size things.
I know it's, it's just a very
difficult thing to make that
transition in some companies.
So how would you suggest folks
do that?
Hilliary Lipsig: And so I wanna
actually point out here Red Hat
is it's no different.
We're struggling through some of
these same problems right now.
And one of the initiatives I am
responsible for at Red Hat is
something called hybrid, SRE,
which is an interesting name.
And this is the idea that we
stop throwing code over the wall
to SREs.
I spent 11 years in quality
engineering before I moved to
SRE, and I experienced in
quality engineering how bad
throwing code over the wall can
be.
And so what, what I did when I
was head of quality engineering
at my, in my last role was I
actually had the quality
engineers working on the
engineering teams working on
unit testing, making unit tests
a little bit more similar to
kind of more like the, the
standard way of quality
engineering, thinking about
testing with, of like
integration and regression and
putting that, shifting it left
into the unit testing framework.
Especially with microservices,
that's really easy to do.
It's much harder.
It was much harder in the
monolithic architectures.
And we really just need to be
doing the same thing, that same
type of partnership with SRE and
service engineers, like was what
we call them.
And so we've started doing,
we've, we've pushed off this
initiative and it's about
getting engineers closer to
their operations and the
operations folks closer to the
engineering so that you've got
better communication, better
partnership.
And a lot of that is actually
process engineering.
Where we're putting together
processes of like, okay, here's
how we're going to do these
things.
And so what we're doing is by
putting in these processes that
generally make the software more
reliable and give it a, a
baseline higher level of service
maturity out the gate that
actually frees up the SREs to do
a little bit more of that
proactive work.
So it really requires a strong
partnership between SRE and the
software developers who are on
the service kind of dividing and
conquering.
So SRE should be putting out
things and guidance and best
practices to their engineering
teams to say, Hey, in our
environments, this is what we
know.
These are the good patterns that
make for performant software.
And then, It's incumbent on
engineering to deliver software
that matches those patterns.
That helps.
And then it must be an ongoing
partnership of like, okay, how
are we performing?
How many incidents are we
getting?
What kind of trends are we
seeing in the incidents?
There's A great practice that
our platform SRE team does,
which is where they actually
take like some major incidents,
especially anything that's
recurrent.
And they put together what we
call a tiger team and they'll go
and kind of like do a deep dive
and it's cross-functional.
And so that's really, The thing
that I, I have to say is the way
to get there, it's what we've
been doing.
It's what we see working.
We've seen some really great
outcomes of like, you know,
certain failures going away
completely or, you know, being
reduced.
And so, and we also have
requirements of toil.
If you're having SRE, run your
service.
Then our toil levels must be
below 50%.
If you breach that, if your
service breaches it, then we
can't keep our commitments to
our customers, which means that
engineering teams must come on
and start actively doing that
toil until they can basically
help automate it away or resolve
the issues within the service
that is leading to that toil,
that manual labor.
So there's a kind of a lot of
little pieces that go into this
and it's about relationships and
process primarily.
And I would just say you need to
implement these things
iteratively.
If you are looking holistically
at the problems in your
organization or the problems
with your communication, there's
gonna be some really bigger
issues and then there's gonna be
some probably low hanging fruit.
And definitely just do that.
Do that.
Here's a low hanging fruit thing
that we could be doing better.
Let's go solve that.
And I love the cross-functional
the cross-functional tiger teams
for that, which is also
similarly, we have a
cross-functional chaos
engineering game where we play
with services and we break them
and do incident response and
like engineers and CEE and SRE
all take turns playing each
other's role in an incident
response to learn about things.
And we usually get good insights
into our software out of those
games as well.
Which results in better, more re
resilient and reliant software.
Jim Hirschauer: Wow.
I feel like maybe I need to have
you back on the show for a
completely separate show about
that topic chaos engineering
and, and working on resiliency.
That's a huge topic in and of
itself.
Hilliary Lipsig: Yeah, it is.
And these are all just pieces of
how we're putting together this
bigger holistic data story at
Red Hat, right?
That's one of the things that
feeds into our overall service
health index that we're working
towards, and our overall data
story of here's here is how we
are doing, here's how things
are, and then ultimately working
towards even more proactive
work.
Jim Hirschauer: Yeah.
You know so I do work for
Harness and what you said really
resonated with me.
At Harness we are building our
software to align with exactly
with what you were just talking
about.
So, you know, trying to make
sure those processes are in
place and automating the process
across the software delivery
lifecycle to ensure quality
code, reliable code.
Making its way through the, the
life cycle so that when it hits
production, you're in much
better shape overall.
So like from a a philosophical
perspective, I think we're
completely aligned there.
It's really hard to do in
practice, without good tooling
to help you.
Process can be very difficult in
and of itself and it's hard to
control that unless you actually
like automate it in some ways
what we've been finding.
Hilliary Lipsig: Yeah,
absolutely.
A hundred percent.
And there are a lot of pieces of
the things that I talked about
that are automated.
And even as much as the chaos
engineering stuff, right?
Lots of chaos, engineering
things exist.
We run it as a game because it's
a team building as well.
Yeah.
And so that also is fun cuz
people get to be red team and
they have to like reverse
engineer their software.
How am I gonna break it?
We, we run it as a live game,
but there's all kinds of
automation opportunities places
where GitOps can really we use
GitOps to actually solve alerts
in the SRE teams at Red Hat.
So places where things like that
can really come in and, and
bolster the actions.
Automation should really be
about making your humans more
efficient, right?
And so any, any kind of
automated tool that does that is
probably a great idea.
Jim Hirschauer: Absolutely.
Alright, well, we are out of
time on our main topic, and it's
been really interesting and
insightful and I, I really mean
it.
I think I would love to have you
back on the show at a later time
where we could discuss more
about these, you know, chaos
engineering game days that you
all put on.
I'd love to hear some detail
about that.
So if you're willing to love to
have you back on the show.
Hilliary Lipsig: Oh yeah,
absolutely.
That's it was an initiative I'm
extremely proud of.
It was actually one of mine.
It's not like I came up with the
game.
It follows the capture, the flag
style pattern.
Mm-hmm.
But the initiative was mine and
it's caught on and it's gone
very well.
I'm very proud of that one.
Jim Hirschauer: Okay.
Yeah, we will definitely save
that topic and talk about that
on a future show.
Right now we're gonna transition
to just for fun, number two.
So, Hilliary, outside of work,
what's your favorite hobby?
Hilliary Lipsig: My favorite
hobby is called HEMA.
It is Historical European
martial Arts.
And it is a style of sword
fighting.
Specifically I'm learning what
is called the Meyer System, and
it is a late German style of
fencing, like what Knights, you
know, would've done and would've
trained in.
Okay.
And it also, so it includes in
addition to the sword
techniques, also grappling and
wrestling techniques as well.
Oh, wow.
It's very fun.
For people who follow me on
Twitter, you will see there's a
clip from my first tournament,
first match of me just getting
whacked on the head.
Jim Hirschauer: And what's your,
what's your Twitter?
What's your Twitter handle real
quick.
Hilliary Lipsig: Caffeinated
integrations into caffeinate.
It's at int the number two and
then Caffeinate
(@Int2Caffeinate), which I
cannot spell because I have
dyslexia.
Jim Hirschauer: Okay.
No, no problem.
I think people will be able to
find you.
Hilliary Lipsig: Probably.
Probably, yeah.
So It's a very fun somewhat
expensive sport.
We fight with the swords are 48
inches long, so that's four feet
in the Imperial system and about
a meter and a third in the
metric system.
And yeah, so they're kind of
heavy.
They're big.
Yeah.
And it's great stress relief
though because when you have all
the armor on and everybody you
know, is fighting with control,
that's a really big thing with
the, the sport is you must use
control.
Cause the swords are heavy and
dangerous even though they're
training swords.
And then after that it's great
stress relief cuz you're just
hitting your friends with a
really big stick.
Jim Hirschauer: I love it.
It's, I've, I had never heard of
this before you mentioned it.
It's, it's like this whole new
world of activity that I had no
idea existed.
It's amazing.
Hilliary Lipsig: It is an
extremely fun sport.
It is a very friendly sport.
The people in HEMA are, it's
just the type of culture where
after a tournament match, right,
you've just been just wailing on
each other, right.
You hug, you give a just big
hug, just hug it out, right?
Hold the whole thing.
People are smiling, they're
hugging, you're sweaty and
disgusting.
This is a stranger, and you're
still full force as hard as you
can hugging this person because
that's like the type of the type
of attitude that it has.
And if you've done other types
of martial arts, I will tell you
that the tournament vibes are
not the same.
This is a very joyful sport and
some of the absolute coolest
people I have ever met including
my best friend and her husband
and my own husband participate.
Jim Hirschauer: It sounds
amazing from the way you
describe it.
I, I, I definitely wanna check
it out.
I live in Austin, so it's highly
likely there's a, a place near
me that can teach me this.
And it sounds like it's good
exercise.
It's, it's, I'd imagine it's
really strenuous.
Hilliary Lipsig: It is a full
body exercise.
Like you, in order to be doing
this correctly, you must be
engaging like every single
muscle.
I was at a class last night and
everything hurts, but it's like
the good type of hurt.
Jim Hirschauer: Yeah.
Alright.
Okay.
Listen, Hillary, thank you so
much for being on the show,
sharing your new word with us.
I love that"ooming" as a verb.
So remember that our listeners
need to start using that
whenever they see Out Of Memory
exceptions.
I love that you shared a
completely new sport and hobby
with me, so I'm excited to, to
dig into that.
And your main topic was just
incredibly insightful.
So, I think it's something that
I never really considered all
that we can do with
observability data and all
that's possible and all the
really good reasons for us that
we should do it.
So you're very humble, but you
are super insightful.
So I just wanted to thank you
for, for everything that you
shared today.
Hilliary Lipsig: Well, thank you
so much for having me on.
This was a real joy for me and
I'd be happy to come back
anytime.
Jim Hirschauer: Fantastic.
I'm looking forward to it and to
all of our listeners, if you are
an SRE or if you're in a related
role and you want to be a guest
speaker on ShipTalk, please send
an email to podcast@shiptalk.io
and we'll get back to you.
That's all for now.
Until next time.