[00:00:00] Chapter 1: Introduction to Observability
Mirko Novakovic: Hello everybody. My name is Mirko Novakovic. I am co-founder and CEO of Dash0. And welcome to Code RED code because we are talking about code and red stands for request Errors and Duration the Core Metrics of Observability. On this podcast, you will hear from leaders around our industry about what they are building, what's next in observability, and what you can do today to avoid your next outage. Okay, today my guest is Juraci Paixão Kröhling. Juraci is on the governing board of the Open Telemetry Project, and he maintains a few models of the open telemetry collector. He is also a CNCF ambassador and previously a principal engineer at Grafana Labs. And now he has started his own venture, which is called OllyGarden. And we will talk about that a little bit. But welcome to the podcast.
Juraci Paixão Kröhling: Thank you. Mirko. Thank you for having me.
[00:00:56] Chapter 2: Code RED Moment and Early Experience with Observability
Mirko Novakovic: Yeah, great. And we always start with a very simple question. Simple but sometimes complicated. What was your biggest Code RED moment in your career? Career?
Juraci Paixão Kröhling: Yeah, that got me thinking for some time. I had seen so many code reds before either as a user or as a quality engineer. So I was a quality engineer before, but as someone, as a webmaster, as we were called back then I think the biggest one was when I, when I decided to start using some monitoring tooling, and I saw that one batch job that we were we had running, it always got locked for, I don't know, five minutes at the very beginning because of some database transaction. We didn't know that was happening, but it was happening every day at midnight. Nothing would happen for five minutes because something was locking the database and we had no idea, but nobody was complaining. I'm sure that some users were browsing at midnight. But we had no visibility into that until we plugged a specific tooling back then. And then we saw those transactions taking like five minutes during the batch job. So I think that's and that's what got me hooked into observability, actually. I mean, before observability was even a thing. I think but that kind of understanding of your system is what really got me into this area here.
Mirko Novakovic: Yeah, that makes sense. And what was the problem? Do you remember?
Juraci Paixão Kröhling: Yeah. One very eagerly developer locked the transaction before doing a lot of very intensive queries. So they were just doing queries so we could have had something like an optimistic log or just a read log. Right. But it was locking the entire database because it was touching so many tables to perform it, to extract the report. And nothing would happen until the lock was removed. So the fix was easy. The difficult part was taking, like a few months for us to even have visibility into this issue.
[00:02:53] Chapter 3: Journey into OpenTelemetry and Open Source Contributions
Mirko Novakovic: Yeah. No, I get it, I get it. And then you somehow went into the open source OpenTelemetry back then. Opentracing, community. How did he get there?
Juraci Paixão Kröhling: That was by chance. I think as everything. Right? Everything is kind of by chance. At the beginning. I was working at Red hat back then, and I was working on something that I think doesn't exist anymore. JBOS? Well, in Portlets. So Java Portlets. That was there was a specification back then like, I don't know, 20 years ago, 15 years ago to show how applications could be built in Portlets. So you could assemble a set of portlets and you can buy portlets from providers and have it as part of your portal. And we saw that portlets weren't really the future. They were winding down as part of a sister team to where I was working on people were working on, on JBOS operations network. So monitoring for JBOS. JBOS was, is or was an application platform application server for Java applications back then. And I was working at Red hat and then it was very important to provide a management console and a monitoring platform for JBOS. So when we saw Portlets were not the future anymore, and when we saw that we had a, a, an architecture for JBOS operations network that was very old we decided to change things a little bit. And then I got into the monitoring or I think we call it observability now, but it was really monitoring back then in management. And I kind of by chance I stayed in the application performance monitoring side of the things. And it was around the same time that Opentracing was starting to gain some traction. And we decided when we were thinking about a new architecture for this new system, we thought we should probably use is. Opentracing. We should probably use Opentracing primitives in while doing the system.
Juraci Paixão Kröhling: So that's when I started doing Opentracing. So at the very beginning doing Java things. So a instrumentation for Java. So I did an Opentracing EJB, Opentracing, CDI, Opentracing instrumenting things that I knew. Right. So I knew Java was my role back then. So I started doing instrumentation for Java things until we eventually thought that, you know what in this area of tracing, instead of doing our own thing from scratch, how about we just join Uber's Jaeger? Right. So Uber had just a few months earlier announced that they were open sourcing Jaeger. So we went there today and said, you know what? Our architecture for the new version of our tracing system is going to be very like what you have there. Just that ours is in Java, yours is in go. How about we collaborate here? And we will join you in the process of joining Jaeger. We asked them to donate Jaeger to the CNCF so that we would have a common ground for collaboration. So we weren't very comfortable in. Collaborating with Uber directly. So we wanted a neutral ground. And it was, I think, a turbulent moment for Uber back then in 2017 or so. And I think that was one of the places in my career that I'm most proud of, because you were able to get such a great piece of software donated as part of the CNCF, even with the logo, even with the name. And eventually, you know, Jaeger became almost like a synonym for distributed tracing in open source for a long time. I think even today and yeah. So that's how I got into Opentracing and by extension to Jaeger as well.
[00:06:48] Chapter 4: Tracing, Observability, and Industry Evolution
Mirko Novakovic: Yeah. Yuri. Right. Yuri was there.
Juraci Paixão Kröhling: Yeah, exactly.
Mirko Novakovic: Yeah.
Juraci Paixão Kröhling: Yeah, yeah. He's there.
Mirko Novakovic: I mean, you triggered a lot of interesting things of my past when you were you. I haven't heard portal for a long time. I remember Apache JetSpeed. Oh, yeah.
Juraci Paixão Kröhling: Yeah.
Mirko Novakovic: That was, I think at the end it was also kind of the reference implementation of the portal spec, right?
Juraci Paixão Kröhling: Yeah. I think there was one from Payara actually. Or I mean, it's a long time ago, but I used Jetspeed in 2002 or 3 or so, I was working at IBM and they had like turbine Jetspeed and all those cool things back in the portal server. Yeah. Yeah yeah, yeah. Absolutely.
Mirko Novakovic: Absolutely. And then talking about JBOS, I also remember that it was essentially built around a JMXX microkernel at the beginning. Right. So everything was an M-Bean, which essentially is kind of the spec of Java to monitor things. Right. To, to take a Java bean, wrap it somehow and then you can access the metrics. And the whole JBOS server exposed all the metrics through JMXx, right? It was beautiful. Exactly. And I really liked it. And so you could at the time really start monitoring JBOS better than anything else because everything was an ambience, right? And, you could connect to it. And I remember JBOS on that kind of monitoring platform was pretty cool. And you could change things, right. You could also use beans to change values and turn on logging and stuff like that.
Juraci Paixão Kröhling: Yeah, I think that's one thing that we had back then that we eventually ended up by splitting, right. So back then we had this monitoring and management platforms, and it was very common for the monitoring things to change the running system. So JMX was one of those right. So I remember a the very I would say very 2000 UI for the JMX console for JBOS. We would go in there and on the one hand we would see counters and gauges for the internal state of JBOS, and right below that there was a button called deploy new application or remove Application and things like that. Nowadays our monitoring systems, they don't touch like they're there, only observing. They're only looking for things. So they receive telemetry. They we have tools that can collect telemetry out of the application. So in a way they are interfering with application perhaps, but they're not changing the running application. Right. So they're not changing the configuration of anything. They're just collecting data and sending out. I think that was a very good evolution that we did. I, I'm not quite sure where we did this change or when we our mindset changed, but I'm happy that we did it.
Mirko Novakovic: Yeah, absolutely. I think for me, it was also the first time I came across it when I read the dapper paper. Right. And then Ben Sigelman, he started Opentracing and starting pushing that standard. And that's how I came across this whole idea of distributed tracing. At the time, I knew tracing, but it was different before, right? It was not so much focused on spans and the structure of a trace. So it was more individually by the vendors like Appdynamics. Dynatrace had the pure PaaS. Appdynamics has business transaction. So everyone had a proprietary way of doing traces. But a distributed tracing was the new thing. And I remember that I was fascinated and somehow scared as a vendor because it, I didn't really know how this will change the observability world. And today I'm pretty sure that OpenTelemetry will become the default for tracing metrics and logging for sure in the near future. I mean, I can ask you if you think that way, but I guess you do.
Juraci Paixão Kröhling: Yeah, I do, I do. Absolutely. Yeah. So I think the road back in 2017 or so that is when we started doing Opentracing. Right. So my very first KubeCon 2017, here in Berlin was on a talk on how we do Hockler APN with OpenTracing. So that was right before I think we committed to Jaeger already back then. By the time that I gave the talk. But we were in this transition and back then in 2017, it wasn't very clear to people outside of this world what is this span? What is a distributed trace, even? And I remember very clearly talking to someone at that conference here in Berlin, and they were saying, oh, you should do Opentracing for, for the kernel events as well to do so. So that's traces and traces. They all become Opentracing and so on. At that moment it felt like, yeah, that that could be like two steps ahead. But at the same time, I thought as well, that's very different than what is really tracing in that application level. Right. So tracing for the, for the kernel level is something that is very different than tracing for application level and for distributed tracing really. I mean, distributed tracing is not about getting all of the all of the calls that are being made method by method. Right. So this is actually an anti-pattern. So when we instrument all of our systems and instrument method by method, that's really bad. And what we really are interested in when talking about distributed tracing is, let instrument my ins, my outs, and perhaps a couple of spans in between just so that I understand what each service is doing. But it really is just that if I start doing hundreds or instrumentation for every single method call, function, call, that's not going to work for distributed tracing. I agree.
Mirko Novakovic: I have to say, and I didn't want to go down that rabbit hole. But I will ask you because you're so familiar with the topic. I still see that people are struggling, understanding spans and traces as a user because it's actually a very technical way of defining a trace, because a lot of developers, especially if you talk to web developers, they think differently, right? They think in terms of requests or and so there's often a confusion that you say trace and then it's, it's somehow hard to explain what a trace is, because in open OpenTelemetry, the trace is an abstract thing. It's at the end of the day, a collection of spans. And then there's a root span, which essentially is mostly seen as the trace because the root span, if it's an HTTP root span, it looks like an HTTP request. So have you seen the same. And what do you think? I somehow think we need a little bit more semantics around it to describe different types of traces, like an HTTP request, because that's something that I would say users understand much better than. And then you can also add some other semantics like the status code. Right. It's a 404 is 200 okay. And these things to that request, the.
Juraci Paixão Kröhling: Problem begins by the realization that people are not taught monitoring or observability in, in college. Right. So they go to university, they learn how to build operate operating systems. They know how to build databases. They don't know how to observe them. They don't know how to monitor them. This is not taught there. And I think when they realize they need observability in some way, that's already kind of late in the process, like late in their careers or perhaps not late, but not too early in their careers. So it is a topic that it becomes complex. It is complex. We need certainly more abstractions on top of tracing. We have some primitives there. I think we're very good at primitives, but we do need a way to convey that information.
[00:14:50] Chapter 5: Observability Best Practices and Tooling
Juraci Paixão Kröhling: And I think technically it is what we actually need. So we need the abstraction of traces because we are never sure whether a trace is a request that is coming in or it is a batch job that is in here, or if it's a new message that is spawning from a message queue. So we need something that is not a request, but it is a request most of the time.
Juraci Paixão Kröhling: Right. So that's a trace. And the spans are the events that are part of the trace. And perhaps I think Ben, when we, when he coined the term span back then, I think that was the right name for it. But I think nowadays when I'm explaining tracing to other people, I say that spans, they are the events that are relevant for a specific transaction. So traces become transaction in this new lingo, and events become this spans. Now I often even in Portuguese when I'm doing my side project on Fridays on my live streams. I use both terms at the same time. So I use traces and transactions. I use spans and events because I know that if I don't use the terms from dapper people are not going to familiarize themselves with the terms that are being used everywhere in the industry right now. So we need to change the whole industry to use new terms if we need them. But until we do that, it is traces. It is spans.
Mirko Novakovic: Yeah, but I agree, I also use a lot the name transaction for it or sometimes, depending on what I'm looking at, I say request or and I agree events is also a good name. But then you have the confusion with the span events, right? How do you explain any events of an event. So it's it's it's I totally agree I think we have the elements. Technically it's the correct way of doing it, but I think we are still kind of early in defining more semantical way of explaining it in a non-technical way.
Juraci Paixão Kröhling: And yeah.
Mirko Novakovic: That will help users to become familiar with it. But I also agree that especially tracing seems still to be undervalued thing. Right?
Juraci Paixão Kröhling: You know what is interesting? Mirko. I was talking to someone here, here from Berlin the other day, and she was saying that she was working on a tracing first company, so she understood tracing. She knows tracing very well by now. Since a few years now. And then she moved to a company that was not doing tracing. They were doing logs only, and she felt so bad going back, it felt like she was going back, like 20 years because the way that things worked, only log only or only log. First type of application or company is way different than the way things work for a Tracey first. So I think beyond the first the first traction, the first movement like understanding tracing it is hard. But once you do understand tracing, it becomes so much better than anything that we had before. You can extract metrics out of traces. You can extract logs out of traces. You can be your primitive, spans can be your primitive for observability. Of course, there will be a point where you look at something like a red dashboard. You know which metrics you need for your dashboard, right? So you need a request or duration. So when I know what I need, I can go and precompute metrics out of it and store those metrics on a very inexpensive database, like a time series database. But for everything else, I can keep the high cardinality. I can keep the raw events or the raw spans on a specific database or on and a columnar storage, perhaps for my tracing first kind of questions. So for exploration, I use my traces and for my dashboards I can use my metrics.
Mirko Novakovic: I can just tell from what I'm seeing at Dash0, we now have a little bit more than 50 customers. And if I always look at the data they are sending and it's still more logs and metrics.
Juraci Paixão Kröhling: It is.
Mirko Novakovic: Than spans. I have to say, I totally agree with you. But what I'm seeing over years now, when I started Instana in 2015, it was all around tracing, right? I thought tracing is like the thing, right? But I really learned that it's very hard to kind of educate the market around it. And people still do logs a lot, especially developers and SREs, platform engineers. They love dashboards, they love their metrics. And that's kind of the way they think. Yeah.
[00:19:34] Chapter 6: OpenTelemetry Challenges and Opportunities
Juraci Paixão Kröhling: Yeah. I think that there are two main audiences that we are dealing with, right? And at Dash0, you see that every day there are the people who are configuring servers and from servers you get logs, no questions. Right? Perhaps you get some metrics here and there. From legacy applications, you do have logs as well as the first the primitive or the primary type of telemetry. It is logs for more advanced teams from five, perhaps ten years ago, they were doing metrics as well. Nowadays when for for people, for teams that are really avant garde for teams that are really cutting edge of technology, they are doing tracing as a, as a way of understanding their systems logs as well, mostly as a way to do an optimization perhaps, so they, they know how to do logs. They have a very good tooling around logs. So they can do logs very well. So they do as well. But I think what we're going to see in the future is more and more maturity going towards the traces for applications, which is the biggest audience for OpenTelemetry. So sure, of course we can get logs out of servers and convert and send to a back end somewhere, but it's not where OpenTelemetry shines. Opentelemetry really shines for application telemetry and for application telemetry.
Juraci Paixão Kröhling: Things need to be correlated. Things need to be inter connected. And I think that, once people understand and once people make the shift from logs and metrics to traces first and then metrics and logs as an optimization scheme, then I think it will help us as a whole, as a, as a whole. I think one of the difficulties that people had perhaps for, for a couple of years ago was the tooling around tracing. It was either very expensive or it wasn't good enough. And I think that right now things are changing and people have good tooling that is on par with the logging tooling that they had. So on par tooling in terms of collection, on par tooling, in terms of transmission of this data on tooling for storage and search of this data. I think nowadays we have tooling, some tools that are even better than their counterparts for logs. So I think at the moment we start having not one tool that is better than all of the others or two tools. But really an ecosystem of tools that allow people to move from zero to 1 to 100 very quickly and in a way that they understand in a way that they're comfortable with. I think that's what's eventually going to help distributed tracing?
Mirko Novakovic: No, I agree, I hope so. To be honest and looking forward to it. I think about logs, and I was recently thinking a lot about how I mean, at the moment we see opentelemetry in the observability space, right? But I'm talking to a lot of customers, also enterprise customers, and they are complaining that they essentially send the same data to very different tools, right? They send their logs to Splunk or CrowdStrike for, for Siem, and then they send essentially the same data to an observability tool. And they pay twice. Right. And they pay a lot for it twice. So how do you see that? Is that converging at the point that we will also see observability or OpenTelemetry, not only for observability but also for security? Have you seen that already? What? How do you think about it?
Juraci Paixão Kröhling: So the question we got, even when I was doing Jaeger. Right. So I remember a company coming to us on the community and saying, oh, I want to use distributed tracing for an auditing system that I'm building. I'm building a product on top of Jaeger, and that's going to be auditing and Jaeger. Back then, the theory was we transmission of data from applications to an agent. It was done using UDP, so it would just make a package and throw over the wall. And if there's someone to catch this data at the other side of the wall. And if you didn't hit the wall or if everything worked, then telemetry would be there, which is not enough for logit system. So I think the potential for non-observable use cases for that kind of telemetry that we are collecting, it is there since the inception, but I think we within OpenTelemetry, it is very clear to us that what we are doing is application monitoring, right? So we are doing things that if you open OpenTelemetry IO, you can see that we have a very well formulated definition. They're saying this is for for, for the performance of your applications, I think at the very end of the statement, say for performance and troubleshooting of your applications, it is not for security, it is not for auditing, it is not for other non-observable use cases.
Juraci Paixão Kröhling: We know, though, that OpenTelemetry has the potential for other industries. It's just that we have so many things to do. We have so many on our plate that we cannot just focus on everything at once, right? And I think it is the problem that you mentioned, I think it is very valid. I don't have an answer to that. I mean, I think there is a huge potential for disruption for the future when companies can perhaps have a data lake of sorts and have all of those vendors to consume that. So we are seeing some movement there, like with a bring your own bucket kind of scheme, but for that you need for the entire ecosystem to work on a specific format of the data. So perhaps we are going to see something like that for OpenTelemetry like a we have a proposal for OTP Arrows so we could store arrow or parquet files. And if everybody accepts parquet files, then we can perhaps have some something like a data lake. But, you know we as an industry, we still have this mindset that your data is my data. And it's not their data. So we don't share data with other parties, right? So I think that that requires a change in mindset as well.
Mirko Novakovic: And now you started your own company. It's called Ollygarden. You can explain a little bit about the name. I know that you are still in stealth. So you can't talk about all the details, but I will try to figure out what I can get out of you. I mean, we talk very early, so I know what you're doing, but I just don't want to tell too much. So take us a little bit on this journey. You were a principal engineer at Grafana. You were working very deeply in the community, and normally that's my experience. Founders see a problem, and then and then they see the opportunity to fix that problem. And so they found their own business. Right. And it's obviously that also from the name that this is something in the observability OpenTelemetry space. And, so tell us a little bit what you can tell about what you saw and why you started your own journey here.
[00:26:53] Chapter 7: Founding Ollygarden and Problem Solving in Observability
Juraci Paixão Kröhling: You were very right. I mean last year I was at Grafana, and at some point I was talking to my co-founder, now co-founder, but he's an old time friend of mine, and he was always, oh, we should start a company. And, you know, it sees the right way and we should do it. And I was like, no, man. I mean, I'm fine. i'm okay. I'm where I want to be. I'm doing things that I like. I don't want to move at all. But then, you know, it's the the perfect storm is the moment where you see from different perspectives, the same kind of problem, and it all converges into one thing. And that's that was that moment. So around the same time somebody asked me specific question and I was, you know what? You were right. And it was around the collector. Right. So a question around the collector why the collector is so hard to do to manage. It's not hard. I mean, I work on collector everyday. I can do it, like, with a one hand tied to my back. So it's not hard. But then the more I thought about this problem I. The more I was convinced that there was something worth fixing there. And at a specific point in time, I went to one of my one of the people behind OpenTelemetry as well, and I asked them what am I not seeing? Why? Why is this idea so stupid? And they said, no, it's not stupid at all. And that was actually Ben Sigelman. So we talked about him earlier. And he said it's not stupid at all. You should do it. And I'd say, you know what? You're right. And I actually thought about doing that as part of Grafana, but then I realized that a lot of the value proposition for that would be not to be tied to a specific vendor.
Juraci Paixão Kröhling: So I don't want to compete with Grafana. I don't want to compete with Dash0. I don't want to compete with the backends. I think you all are doing a great job there, in creating innovation for data storage, for oxidization and what to do with that data analysis and so on and so forth. I think there is a lot of innovation there already. And what we were seeing is if we saw a space where we could bring a benefit to the whole ecosystem by not competing with the backend. So we are complementing the backends. Right? So that's our main idea. You ask about the name. I it is OpenTelemetry. Of course. I mean, that's all that I know by now. So I, I should not try to do anything else. That's not going to succeed. But I know OpenTelemetry. It can be very opinionated about OpenTelemetry. And I knew that. Ollie. So, Ollie, I don't know if everybody knows that Ollie is not only observability, but he's also OpenTelemetry, right? So o11 y. It's both so it can translate to both. So I knew that Ollie had to be in the name and I was. What is it going to be? All the lads only mate. Ollie. Ollie. What? And I was sitting right here looking at my garden here. And an image of me planting tulips with my kids here in the garden came to my mind, and I thought, you know. Yeah. Ollygarden. So a garden full of observability experiments. So that's it. So that's where the name came from and.
Mirko Novakovic: Sounds good. How do you feel after a few months in? How is the journey going? I when we started this conversation here, before we turned on the microphone, you said you have never felt that productive, right?
Juraci Paixão Kröhling: So yes.
Mirko Novakovic: Tell me about it.
Juraci Paixão Kröhling: Yeah. I mean, I'm learning a lot and things that I never knew that I. That things that I never thought that I would learn like on the legal side of things and accounting, hiring or. I mean, hiring is hard. So quite a lot of things that we need to know now that I need to know now that I didn't have to before. And I think that's one aspect of doing your own thing. Like you have to learn how to do those things. One thing that I realized is it is actually even though it is a lot of work and you learn a lot of things when you have an idea and when you are convinced about this idea you can do it. You can learn it. I mean, those are all things that every one of us can learn. And the risks are not that big. I mean, I thought the risk was going to be so big that, you know, it's going to be impossible to have a company of my own. It's unattainable. And it's not like that. The more that I read right now, it is clearer to me that I should have done it before. It is scary, but it's doable, right? So people who are thinking about starting their own businesses, as long as you are convinced, as long as you have done the research that you should I repeat, what what what? Ben said you should do it.
[00:31:42] Chapter 8: Entrepreneurship, Learning, and Innovation
Juraci Paixão Kröhling: Of course, I'm not successful by any means yet. But I am very confident that we can deliver something of value and we can be successful. And I said that I feel very productive because it's not only learning about legal accounting and things like that. It is really when you have to do everything on your own, you end up learning a lot of new tricks, right? So, and I think we are on a magic moment in our industry right now. I saw a blog post from you the other day on LinkedIn a blog post, a LinkedIn post the other day where you mentioned AI tools and you ask your team to find tooling on AI to be more productive and so on. And I think that that really resonated with me, because I'm using so many AI tools right now and they make me so productive, like code reviews via AI or even coding with AI or creating beautiful websites with AI or I go to Canva or something and suddenly I can have a beautiful design for my meetup here in Berlin, the hotel nights. So, you know, I can create things very quickly. And I think this also means that it's easy to start your business and even if it's very small at the beginning, you can, you can play eye to eye with the big names out there.
Mirko Novakovic: No, absolutely. I think there are two things here where I want to comment is what I've seen in the past years, when I did a lot of angel investments. I think the most challenging part of founding a company is understanding a problem and the solution for the problem, right? And that's what you have because you are very deep into the topic, right? You understood the problem really, really well because you were talking to customers. You were talking to other professionals in that space, and you knew that there is not a right solution there yet. And then you also understood how to fix it, right? If you have that and you can build it yourself, which is also a big advantage, I think if you have that, the rest, as you said, right. You can learn right how to create your incorporation in the US, how to get funding, how and there are people like business angels. We talked a little bit. You talked to other people who can help you, right? I think this community is also very supportive. Right. If you want to, to build a company, I think you always find someone who wants to help you or will help you. So yeah. Congrats that you did that step. Yeah. Thank you. Exciting to watch. And then on the AI topic, I have to say that I was literally very critical looking at this thing. But over the last weeks and months, I'm really getting excited about it, right? Seeing what you can do, we experimented a lot with Claude, the new 3.7 thing on the coding side, but also using it inside of Dash0. And it's amazing, right? This agentic thing, when you give the AI, basically your API, you ask a high level question and then it starts iterating. Calling something gets an error. Calling something different figures out things, learns, and then triggers the next call. It's amazing and scary somehow, right? To see how good and what I kind of like the most is that I understood that it's acting very much like a human right.
Juraci Paixão Kröhling: Yeah, with the reasoning part. Yeah, yeah, the reasoning part.
Mirko Novakovic: Exactly. It really is interacting and you can actually learn something from the AI about the usability of your product, because if the I can't figure out how to do something, you probably also get the same problem with normal users, right? Because they will also try a similar approach. So I'm really enjoying at the moment, seeing those experiments and thinking about how to put that into your product.
Juraci Paixão Kröhling: I mean, it's so exciting, so many in so many angles. One is I used a lot of AI to criticize my plans for the company. Right. So before even talking to you, I had a few I used, I think ChatGPT even back then. So I wrote a notion page with all of my ideas, and I fed that into ChatGPT and said, what are the holes in my plan? What are the things that I'm not seeing? And a lot of it was crap, right? So a lot of it was like pure hallucination. But a few points, they were really good. And it showed me that even though I can discard, like, 50% of what it said. The other 50%. It's really valuable. And you can keep iterating, right? So AI is as good as the data that you put in. And I think it is, you know, the same the same same things that we talked at the very beginning of our career, like bad data in bad data out it. The same happens here, right?
[00:36:37] Chapter 9: Future of AI in Observability and Closing
Juraci Paixão Kröhling: So if you fit into AI that that prompt, you're going to get bad outputs. But the richer that you can, the richer that knowledge that you can provide to, to AI, the richer the outcome that you get. But I think that's one the user side of things now from the service provider side of things, I think I think that it is a very exciting moment for us in the observability area as well. So I'm not a front end developer, but I'm doing the prototypes using those new tools. And when I look at the code, I think somebody, somebody has to maintain this code.
Juraci Paixão Kröhling: Somebody will have to observe this code. So that's when that's why I think observability is going to be so much more important in the future than it is right now, because if I write a code and I see a specific problem on the screen or on my backend, or if I don't see data on database, then I kind of know already in which part of my code the problem is. I can just go there and guess what the problem is and try some fixes and deploy it and done. But if I have not written the code myself, or if I've if I'm not that familiar with the code as I used to be, then I do need way better observability than I have on currently. And I think that's one challenge that we have as service providers for the future is how do I ensure that I'm observing the right things at the right times? Right. So it's not about again, instrumenting every single method call that we have. It's really about understanding on a deeper level what is going on in there. Some tools are good at that. We. So there is a try to fix this problem on some code. No code tooling, and it works. But as someone that is running a critical system in production, we need better tooling. We need better observability than what is currently being provided. So I think it is very exciting to be in this industry at this moment because observability is going to be way more important than it is. No, I totally agree.
Mirko Novakovic: And also you have that's kind of something I need to understand better also is that you have not only the human user. Now you have an AI user.
Juraci Paixão Kröhling: Agents. Yeah.
Mirko Novakovic: Agents to your system. Right. And you also have to think, how can I provide that observability data also to those agents in a way that they can utilize it to help the user understand it? Right. So I think you have different levels of, of, of users now, and it's super exciting. I really love to be in this space right now. And it's fun. So and I think you are probably also thinking the same way in your space where you are in so will be fun seeing when you come out. Is there any plan when you want to have something out there to test, to see?
Juraci Paixão Kröhling: Very soon, very soon. So we were at KubeCon at the beginning of the month. So in, in the first week of April. Yeah. And we've shown our prototypes to a few people. So we have a prototype that we prepared that we deployed at a specific location only for KubeCon people. So my co-founder, Yuri is was showing the UI there to people and picking some, some initial interest to, to get some design partners. So we have some design partners already working with us that we got even before KubeCon. And but I think we are the moment that we get comfortable about the, the, the value that we provide to our users is the moment that we're going.
Mirko Novakovic: To get out of.
Juraci Paixão Kröhling: Stealth, because then we know that we have the right dose of value, of information. I don't want to give much details, but at the moment we have the right set of insights to people. That's the moment we're going to get out of stealth and say, you know what, look at this. This is the result that those people here who helped us build the product, those are the results that they got. So yeah hopefully soon.
Mirko Novakovic: Yeah.
Mirko Novakovic: Juraci, Thank you. It's fun watching you. And I'm cheering from the sideline and seeing what you are building. And hope to see something running soon.
Juraci Paixão Kröhling: Yeah, absolutely. Yeah. We'll keep you updated. Thank you for inviting me.
Mirko Novakovic: Thanks for listening. I'm always sharing new insights and insight and knowledge about observability on LinkedIn. You can follow me there for more. The podcast is produced by Dash0. We make observability easy for every developer.
