[00:00:00] Robby Russell: Welcome to On Rails, the podcast where we dig into the technical decisions behind building and maintaining production. Ruby on Rails Apps. I'm your host Robby Russell. I run Planet Argon and for over 20 years we've helped teams maintain and evolve long live Rails apps. So I tend to approach these conversations through that lens.

[00:00:21] Robby Russell: I was recently joined by Simone Carletti, the CTO of DNSimple. Simone helps lead the engineering team behind a platform that many of us rely on every day, even if we rarely think about it. DNSimple runs a globally distributed DNS system with a surprisingly small team, and Rails sits right at the center as CTO.

[00:00:41] Robby Russell: Simon has worked across the stack from the Rails app to Ling name service to go services, you name it. So he brings a perspective. You don't often hear how a long Live Rails app Codebase can anchor a system that reaches far beyond a typical web app. Simone lives in Italy, but joined us from his holiday trip in [00:01:00] Columbia recently.

[00:01:01] Robby Russell: Alright, check for your belongings. All aboard,

[00:01:08] Robby Russell: Simone Carletti, welcome to On Rails.

[00:01:11] Simone Carletti: Thanks Robby. Thanks for inviting me,

[00:01:13] Robby Russell: Simone. So let's start with this. What keeps you on Rails?

[00:01:18] Simone Carletti: That's, that's a great question. So, um, I think that Rails been really powerful for us all over all these years in terms of providing a stable framework that has pretty much everything that you can think of when it comes to web development and sometimes not just about that.

[00:01:36] Simone Carletti: And sometimes they actually lead by providing you guidance and ideas. The other thing that I feel has been pretty important for us is that it's a very active project. With a large ecosystem. So for a business that is very, very important. I dunno how many times happened to you or to the people listening, but you know, you pick a gem, you pick a library, and uh, you do your best effort to [00:02:00] find something that maybe is developed by two or three people.

[00:02:02] Simone Carletti: So you don't end up in a single developer. The

[00:02:05] one

[00:02:05] Robby Russell: person,

[00:02:05] Simone Carletti: yeah. Yeah. Bus factor one. And then it doesn't matter, you know, the library get replaced or, or there's the new framework. There's something else. And I've been using Rails for even before starting DNSimple, you know, it's more than 15 years and it's still here, it's still very much actual, it's still very much on the edge of the web development and this is really, really important for, for a success of a company.

[00:02:30] Simone Carletti: So I think this is a pretty good foundation for us

[00:02:32] Robby Russell: for some context. You don't have to go through your whole CV necessarily, but what other tech stacks have you worked with prior to Ruby Rails?

[00:02:40] Simone Carletti: So I actually broke production code in more than 16 languages. I, I love programming languages. It's something that, uh, uh, not that much these days, but, uh, when, when I was at university time and, and when, early in my career, I was just trying new languages for fun.

[00:02:56] Simone Carletti: Uh, now I wish I had all that time still to, to try the languages [00:03:00] that, that it's more like the business is forcing me to get there. But, um, I've been very familiar with Ruby, of course. Uh, with go with PHP with Java, I've done a little bit of Python, uh, development. And of course I had some C background as well.

[00:03:16] Simone Carletti: And so when I came to the Ruby language more than 15 years ago, it was really, really refreshing to see all the new tooling and all the new. Passion for, you know, active development. Something that later on I actually discover in other languages, maybe slightly differently. You know, there's a league here that I use Go, but as I said, regardless, Ruby is still very much modern and Rails is still very much modern and active development, and that is very important for a developer and and for a company.

[00:03:47] Simone Carletti: So I think that that's, uh, that's something pretty cool about this framework.

[00:03:51] Robby Russell: Before we start peeling back some layers about DNSimple, can you give our listeners a sense of the engineering team behind [00:04:00] DNSimple today and maybe a quick introduction of what DNSimple or DNSimple does. I do you Absolutely.

[00:04:06] Robby Russell: I say that all the time, by the way. I think recently I was like, oh wait, it's just DNSimple, not DNSimple. But I think I've been saying that that

[00:04:12] Simone Carletti: is correct. Yeah, the

[00:04:13] Robby Russell: whole time. As a customer,

[00:04:15] Simone Carletti: ironically, I think we strive to make DNSimple, but we pick one of the most challenging name because every time people is, is confused about that.

[00:04:24] Simone Carletti: Not at the time the name was evolved because initially it was DNSimple in the sense like we're making the, it's DNSimple, but making DNSimple. But then also we open up to domain name, so DNSimple makes like domain names simple. So there's a bunch of various interpretations and even, even internally.

[00:04:42] Simone Carletti: Uh, but yeah, sure. So, so DNSimple in total is about 25 people. The vast majority is still the engineering team. Uh, we're about 15 people in the engineering team, give or take. Structure of DNSimple is, is composed by, I used to say three main languages. So we have the Ruby [00:05:00] side, which is 50% more or less.

[00:05:02] Simone Carletti: Then we have go for the 30% of our stack, and then we have Ang for the remaining 20% of our stack. Each of these three languages actually represent one of our. Core component of the infrastructure. We, we can elaborate later. Uh, if you're interested in or if it comes through the conversation. Uh, DNSimple provides domain and DNS registration.

[00:05:25] Simone Carletti: So we started, uh, as a authoritative DNS provider. So basically you give us a zone, a DNS zone with the records, and we become the provider that serves the VDNS record for that particular zone. And then we expanded by providing, uh, an additional number of services. All around your online identity and online presence.

[00:05:47] Simone Carletti: So that would be domain registration, DNS hosting. Uh, we also do provide, uh, some email services, some SSL certificate services. And really what we strive to do is to, is to bring you [00:06:00] all the tools that you need to connect together, the services that we, you use and, and right. We, we don't want to become the jacks of all the trades.

[00:06:10] Simone Carletti: The solution for all your online problems. I feel like there are large companies that do email, for example, very well. Mm-hmm. And we don't wanna become your new email provider. But what we do is that we help you to connect all these pieces together with a strong focus, of course, on the domain side. So the domain registration side and the TNS side.

[00:06:29] Robby Russell: Coincidentally, the, when I had my first professional software job, this was back in like 1999, 2000 era. Like the year 2000, I worked at a company and my boss is like, Hey, why don't you, I know that you like Linux. Why don't you spin up a Linux server? You can be responsible for running DNS for all of we had, we had a bunch of domains for the company that I worked for, and you can manage a Linux server run.

[00:06:52] Robby Russell: I think I ran bind on that or something, and so I had a Linux machine in the corner of my cubicle that had a ethernet connection to it that had a [00:07:00] public IP address. I was responsible for hand coding. The DNS bind configurations for a bunch of different domain names, so that was my introduction to working with DNS and stuff like that back in the day.

[00:07:10] Robby Russell: Outta curiosity. What's underneath the hood or was any of DNSimple on top of bind or things like that? How does that look actually from an infrastructure? I'm actually just kind of curious, like what's the underlying 'cause? Basically DNS for anyone listening is like one, are the most used a PII guess in the way, in a way that, you know, protocol.

[00:07:29] Simone Carletti: Yeah, absolutely.

[00:07:30] Robby Russell: Yeah. It's

[00:07:30] Simone Carletti: one of the oldest actually.

[00:07:31] Robby Russell: Yeah. Yeah.

[00:07:32] Simone Carletti: It's, it's one of the oldest and uh, in the essence is also one of the easiest, although. It's actually one of the most complicated because of the number of the RFCs I believe that I was reading last week about, uh, the number of RFCs for, uh, you know, statistics about the number of RFCs per, per topic.

[00:07:51] Simone Carletti: And the DNS space is one of the largest because of it's continuously being refined and, and new records and new additions [00:08:00] because the web op today is not a web op, you know, 20 years ago.

[00:08:03] Robby Russell: No.

[00:08:04] Simone Carletti: So the effect, I keep saying the protocol itself. It's not new. What keeps changing and evolving is the way that you interact with the protocol, right?

[00:08:14] Simone Carletti: So you're right. Uh, in fact, at the, at the very beginning, DNSimple actually started by using power DNS.

[00:08:21] Robby Russell: Okay.

[00:08:21] Simone Carletti: Power DNS is one of the two main DNS servers. You have bind and you have power DNS. Uh, so our initial infrastructure was based on uncast and we had a bunch of Power DS instances distributed in many servers that we were keeping in sync.

[00:08:37] Simone Carletti: But then eventually, it wasn't that much. Later, around 2012, we started hitting some limits of what PowerDNS back in the days was allowing us to do. In particular, uh, exactly speaking about the way that the web evolved, uh, around 2012, we started to see things like services. Coming to fruition that gave you a [00:09:00] host name and you wanted to point your name to the host name.

[00:09:04] Simone Carletti: So rather than pointing to an IP as used to be, you know, to the service, you wanna point to the specific host name that,

[00:09:10] Robby Russell: yeah.

[00:09:10] Simone Carletti: Yeah. That particular service will give you, I mean, now about example, GitHub pages, if we recall, it was giving you github.io,

[00:09:18] Robby Russell: right?

[00:09:18] Simone Carletti: And you had your own username, GitHub io. And so you are pointing your name to that and you're not gonna point to an ip.

[00:09:24] Simone Carletti: And now there's a DNS limitation of the C name record. Yes. Which is what usually people use to link to that, that cannot be used without a records. So in fact, it cannot be used for the apex. So you can have, for example, example, do com pointing to your website, uh, let's say key top. And so we started to think about alternatives way to execute and to provide this kind of integration interaction.

[00:09:49] Simone Carletti: These days there are RFCs for this type of record. So we came with what we call the alias record. Other companies came with a similar name later on, a name alias. Yeah. [00:10:00] Uh, you know, something like that.

[00:10:01] Robby Russell: Very familiar with this. Yeah, yeah,

[00:10:03] Simone Carletti: exactly. And, uh, because it was, it

[00:10:04] Robby Russell: was a challenge at first.

[00:10:06] Simone Carletti: It was absolutely a challenge, but it was a big challenge.

[00:10:09] Simone Carletti: So in essence, that's very simple. But it was a very big challenge to make that work because the way that we had to work with power DNS, uh, is that we had to use pipeline. So we had to inject within the resolution chain, we had to inject some code that says, if it is this type of record, then proceed in this way.

[00:10:27] Simone Carletti: And so the very first pipeline that we built was in Ruby, and it didn't work very well. So we then replaced that with our languages. Uh, but eventually we realized that it was simply not. Really that structure back in the days power with DS to work in this way. So we decided to go with our own. We, we, we tried, we built our own name servers.

[00:10:49] Simone Carletti: Anthony, my, my business partner, he was more knowledgeable than me in lan. And he says like, you know what, as a fan project, I wanna build a name server with, with airline. And that's how it [00:11:00] all started. And so after a couple of years, we actually had our version that was working sufficiently where, well, that we decided, you know what, we're gonna put that in production.

[00:11:10] Simone Carletti: So around 2014 we switched from power DNS to what we call early NS, which, uh, actually internally we run early and simple, which is, uh, a version on top of RDNS, which are our name server, which is also, uh, an open source project. So you can go and download that and you can have your own name servers in your own, uh, airline ecosystem.

[00:11:31] Simone Carletti: And so we've been running it since 2014 in production. We actually recently started. To work on that more actively. So these days in the last year, it's, it's way more active than what I've been, uh, in the past years. More modern, more features because same story, right? Yeah. Uh, pick a language, pick an environment, then time pass, new things comes out, new runtime environment.

[00:11:55] Simone Carletti: Uh, so you, you, you can really stop more than a few years there. Uh, so we are still running it. [00:12:00] We use that as a primary name service software and uh, we still use power DNS though, but just for the secondary part. I see. So for the A XFR integration for what allows customers to synchronize their zone from the end simple, uh, to a third party provider or from a third party provider to push the zone inside the npo, there is a protocol that is called A XFR, which is the standard protocol for synchronization.

[00:12:26] Simone Carletti: And, uh, rather than implementing that internally, we rely on, on power DNS four D uh, client server part.

[00:12:32] Robby Russell: I see.

[00:12:32] Simone Carletti: Essentially

[00:12:33] Robby Russell: outta curiosity, does it help to have two separate DNS. Servers syncing with each other so that you know that you're staying compliant with like all the RFCs and things like that?

[00:12:45] Simone Carletti: Absolutely, absolutely. 100%. Yeah. That's, uh, that's actually one of the main benefits that we got lately because we were able to find, you know what, the DNS space, unfortunately, the number of RFCs. That much, that [00:13:00] actually sometimes creates more chaos than clarity. And so literally in the last six months we have spent a lot of time working on a topic called the DNS space, which is called Empty on Terminals.

[00:13:12] Simone Carletti: And the empty on terminals are a particular, I would say, scenario that happens in the DNS zone in configuration. And we happened to notice that our implementation of the empty non mass was not compliant, uh, just because we were offering secondary DNS integration. And we realized that the same zone, depending on whether you work querying the primary or the secondary was returning in certain circumstances at different responses.

[00:13:39] Simone Carletti: In my opinion, that has been extremely valuable and I can. Relate back just a little, an anecdote. I can relate back to a major project that we have done around 20 14 20 15, which was completely redesigning the APIs, switching from API v1 to API v2, you know, API v1 [00:14:00] were, uh, pretty much built in with the early versions of Rails API.

[00:14:05] Simone Carletti: So it was, you know, the old XML thing, uh, directly inside the controller and multiple responses. And as you start switching from 1.2 to, you know, newer version, things change drastically. So we decided to split off the API in a completely different, uh, sub component of the Rails application. And when we did it, it was actually the first years that we're starting to play around with, go with a different language and.

[00:14:33] Simone Carletti: We decided that we wanted to provide at least three API clients and we started with Ruby. That was the, the main one was, was there since the beginning. Then we decided to go with Go, and it was extremely valuable to build an API and test that. And by test, I mean use it dog food that mm-hmm. Right. Use that using languages and using, uh, programming languages that are significantly and different in the way that they interact.[00:15:00]

[00:15:00] Simone Carletti: So GO is a strictly typed, Ruby is not that complex. Right. And so by measuring, by testing the API with different languages, we were able to find like, Hey, this thing is actually pretty complicated to implement in, in go typing. So we might have to serialize this object in a different way, or we might wanna build this API endpoint in a different way.

[00:15:22] Simone Carletti: And so in many ways Interesting. Yeah, I feel like going with a. Heterogeneous environment, would it be two different name server? Would it be two different, you know, the providers? Would it be two different programming languages actually help you to see things differently because you're not gonna design.

[00:15:44] Simone Carletti: You know, strictly influenced by whatever is the tool, the service or the language that you're using, uh, because you're gonna have more than one. And so your, your brain will, will not focus on one in particular and you are gonna have the chance, in [00:16:00] my opinion, build a better, better outcome with

[00:16:02] Robby Russell: that. No, I think that makes sense for anyone listening out there, if your team has some capacity and the bandwidth to be able to explore those things that definitely can kind of like unlock ideas that might change how you're implementing it in Ruby or Rails and then the other technology stacks that you're also potentially working with as well.

[00:16:18] Robby Russell: So I think now that we have a bit of terrain about DNS and, and, and I would love to go more detail about that. And when it comes to like what parts of Rails you think, I think you mentioned Rails is about, you know, 50% give or take of your technical stack at the moment. So what kind of keeps Rails at the core of a platform that's also using early and go and a very.

[00:16:40] Robby Russell: Distributed infrastructure. I'm assuming you have servers. I dunno if you're running, maybe we can talk, talk a little bit about the, uh, the deployment strategy that you currently have as well there.

[00:16:51] Simone Carletti: Absolutely, absolutely. Yes. So I mean, what keeps Rails here, it works really well. That's, uh, that's really the, the gist of it, right?

[00:16:59] Simone Carletti: [00:17:00] So, um, admittedly we started with the Rails back in 2010 when we launched the And simple. And so the more you move forward, the more it becomes complicated to replace a stack, especially if it works well, you don't have significant reasons to move away over the time we have. Extracted out some of the things that did not work very well in Rails, or things that we had to implement differently because ultimately Ruby and Rails are tools like other languages, right?

[00:17:31] Simone Carletti: So we always want to start from the perspective of what do we need and then how do we implement rather than like, Hey, we have this language and so we have to implement everything with this language or everything with this, uh, framework. So. Right now we, we still use Rails because it, it works really well for the whole front end and main application side.

[00:17:52] Simone Carletti: And, uh, the main application is not just, you know, the front end and the back end and actually is the entry point [00:18:00] for everything into the end. Simple. Uh, so when you manage your DNS records, when you register a domain, uh, and of course, as you may imagine when you go through the billing and all of it, everything passed through the main Rails application, right?

[00:18:14] Simone Carletti: Directly or indirectly would it be through the ui, through the API and so everything vehicles through the Rails application, even the DNS record and then ultimately gets propagated all the way to, to the name service. And so that component. Which is one of the three core component. We have the main Rails application.

[00:18:34] Simone Carletti: Then on the other side we have the name servers, which are the public phase for the DNS queries, and then in the center we have what we call the zone server, and those are the three core component of our infrastructure. Effectively, the, the application is responsible for managing all the interaction with the customer, and then the data is stored in the databases.

[00:18:57] Simone Carletti: Then the zone server is the connection [00:19:00] between the data and the application and the name servers. Imagine that being like sort of like a mam cd, but but designed from scratch to be extremely efficient and focused on serving DNS records. So it is designed to help us replicate the data. Fast across all our edge, across all the, the various data centers that we have all around the world.

[00:19:24] Simone Carletti: And so that sits in the middle. And those three components, as I was mentioning before, actually represent three main languages. The main Rails application will be then the zone server in go and the name server in airline. And each of them, you know, kinda like develop naturally because every language has its own nuances that makes that really great for that particular use case.

[00:19:48] Simone Carletti: Ruby and Rails works really, really well for all the ui, uh, front end and also backend interaction, storing data to the database, sending out to queues, sending out to our synchronous jobs. [00:20:00] Then the name server ER Airline actually turned out from, it was kind of like a, you know, bat and turned out that the pattern matching of Erline is one of the best way that you can take an RFC.

[00:20:13] Simone Carletti: Literally decompose a binary packet of the DNS message by just writing the airline definition copy and pasting from, from the RFC. So airline works really well on that side, and of course it's tolerant. A high concurrency works really, really well for the DNS side, and then go for the zone server. Uh, help us to have a very lightweight process that can load all the DNS zones in, in memory and help us to process that fast and move the data between, you know, one system and the other.

[00:20:46] Robby Russell: How complicated is, like when changes get made for cache expiration? I mean, I can only imagine, like, if you have that, I mean, do you, I don't know if you know off the top of your head, like how many individual records [00:21:00] are we talking about as being distributed across your, all of your, your edges right now?

[00:21:05] Robby Russell: Like, that's like every like www admin dot, you know, it's like.

[00:21:10] Simone Carletti: Yeah,

[00:21:11] Robby Russell: it's a lot.

[00:21:11] Simone Carletti: Absolutely. No, we're definitely talk about several millions of, of records, uh, distributed all around the, the, the, uh, infrastructures. But right now we are doing that really, really fast. Um, we actually spent a lot of time, uh, five, six years ago optimizing the distribution of the changes across the network by making sure that only the delta of the changes gets distributed pretty much as you, as you would normally do.

[00:21:38] Simone Carletti: We it, right? So we don't always distribute the same zone. We used to do that, but we don't always distribute the same zone. Uh, we distribute just a small delta of the changes, actually, not just the changes that you make, because in, in the DNS space, there is an RFC, uh, there is a topic that is called RR set, which is like a well-defined concept.

[00:21:57] Simone Carletti: And so the RR set is the combination [00:22:00] of all the records that have the same type. And, uh, and name. So in other words, you may be changing one record, but in order to make that efficient, we have to distribute all the other records that are somehow part of the same r set, but it's still a very small percentage and have a small amount of data compared to distributing the All own, we actually have some customers, uh, there's one customer in particular that has one zone with over 2 million records in it.

[00:22:26] Simone Carletti: And, um, it would just like kill it. Actually at some point when there were still around 600,000, it started to kill our backup processes, our distribution processes, because serializing all that amount of data, 'cause it's a multi-tenant system, so they were constantly adding and removing records that represent they're all customers.

[00:22:46] Simone Carletti: Yeah, that makes sense. And the changes that they do in their own systems. And so the changes that we're sending were happening faster than the processing time for us to process the whole zone. So we had to re-engineer completely the [00:23:00] distribution of that and that work out really, really well these days.

[00:23:03] Simone Carletti: I think that, um, compared to other providers, we can serve large zones without that much of overhead because we, we really scale pretty much nearly. We don't have, well constantly, we don't have that kind of like, uh, linear growth where the more records you have in the zone, the more, uh, the impact it is. Of course there are still some services that ultimately it's data that you have to transfer.

[00:23:24] Simone Carletti: For example, there are certain scenario where we have to completely refresh the zone whenever we reboot certain machines or whenever we provision certain hardware. So we have to refresh entirely the zones.

[00:23:34] Robby Russell: So we still have to go through all the data

[00:23:36] Simone Carletti: and deal with the law of physics. That was very funny when we realized that transferring all the amount of data from the main data center that is in Chicago to, for example, Tokyo was the main difficulty that let our server not restart properly.

[00:23:51] Simone Carletti: And once we started to actually distribute the data at the various edges, we were able to move from, you know, restarting a machine in [00:24:00] 35 minutes to restarting a machine in 60 seconds. Oh

[00:24:03] Robby Russell: wow.

[00:24:03] Simone Carletti: And there you learn about the law of physics, like data can only travel so fast.

[00:24:08] Robby Russell: That makes sense. This as like a quick little sidebar.

[00:24:11] Robby Russell: What I remember when DNSimple first launched in, you know, I've known Anthony from even before DNSimple in the Rails community. And I remember one of the big benefits of it was that it, it provided an API for developers to manage their DNS. Uh, which is ironic because I still find myself logging into DNSimple and manually adding new records and I'm like, oh yeah, there's a, but.

[00:24:33] Robby Russell: We've done a lot, a bunch of integration where we would use the API so that what we dealt with, like clients that had multi-tenant applications, we had different servers we needed to point them to. We could just manage that through the API and that was like groundbreaking at the time, coincidentally, just to have a simple rest API endpoint.

[00:24:49] Simone Carletti: Yeah.

[00:24:49] Robby Russell: Was so, was so helpful. So thank you to you and Anthony for

[00:24:53] Simone Carletti: Absolutely.

[00:24:53] Robby Russell: For making that much easier.

[00:24:55] Simone Carletti: Appreciate that. And you know, the API are actually the main [00:25:00] entry point for our application, I believe, uh, the last time as which I was checking the stats. I believe we may be serving more API requests than UI request the states.

[00:25:10] Simone Carletti: That's because, you know, when you think about API, you don't have just to think about the normal API integration that you could do with the clients. By the way, we spent a lot of, we invested a lot by building, uh, clients for I believe up to nine different languages, uh, to facilitate. The adoption of of the API and to, to make sure, because that's our niche, right?

[00:25:30] Simone Carletti: DNSimple is a, is a small provider compared to large three players in the industry. And so you need to find your own niche. And, and our niche was really focused on facilitating automation, not just for developers, but for every company that really wants to get on board with, uh, automating stuff. And so the API these days are really, are the major entry point of our application and that's actually where Rails help us really tremendously.

[00:25:59] Simone Carletti: We still [00:26:00] run a little bit of Hanami, which was the framework that, uh, Luca, we did the frame of mind. Luca was actually working the and simple for, for some time while it was developing that framework. So we still have some of that for the API, although there's been some challenge with, uh, version two and the upgrades and all of it.

[00:26:16] Simone Carletti: Also back in the days Rails really didn't have the API sub component that I believe they introduced five to seven years ago with, you know, a lightweight controller that doesn't come with all the extra things that you don't really need for rest API or for any API at all. And so we are still using it, but really Rails has been extremely helpful here to make possible to have a front end an API and all of them sharing the same business logic behind that.

[00:26:46] Simone Carletti: Right, right,

[00:26:46] Robby Russell: right.

[00:26:47] Simone Carletti: So it worked really, really well. And uh, and in these days, I mean, there are declines, but then the whole ecosystem, Kubernetes and then Terraform and then infrastructure code is, is growing really fast [00:27:00] and it's becoming a key instrumental piece of many companies that want to scale, including the simple, it's one of the secret that help us to scale pretty much the remaining as much as possible in using infrastructure code for everything, including managing our own domains through key top.

[00:27:14] Simone Carletti: All of that really funnels through the API.

[00:27:17] Robby Russell: Maybe tell us a little bit about how you approach your, maybe like zooming in a little bit on the Rails app itself. What's that rails layer look like in terms of like how do you organize, sounds like you've got controllers, your, your API endpoints, uh, you mentioned hanai as well.

[00:27:33] Robby Russell: Where does like controllers and business logic kind of emerge in your rails? How do you separate that out and what's a typical like UI through API business object kind of approach look like there?

[00:27:43] Simone Carletti: Right, right. So we started like everyone pretty much having everything in controllers, right? So when we use the first version of the API, the, the controllers were really containing enough business logic and the other half was in models.

[00:27:58] Simone Carletti: Admittedly, this is one of the thing [00:28:00] that eventually as the architecture grow, it can really scale. And, and for us the breaking point was when we started to implement an API once again diversification. We build a PIV two on. Anami and Anami was, was plugged into the main routing system of the Rails application.

[00:28:20] Simone Carletti: So Anami was a sub component of the routing system because it's also a rack application. So you know, you can mount that on the main router of the Rails application. And so when we started doing that, we came to the problem. It's like, okay, now when someone wants to register our domain, they can do that from the API and they can do that from the ui.

[00:28:37] Simone Carletti: If we stuff a bunch of things in business logic inside the controller, it's not gonna work because how do we make that available in the API and vice versa at the same time, we really wanted to keep them simple. There's uh, there's a few things that we don't use about motors. So for example, noop server and all of that.

[00:28:55] Simone Carletti: We use model more in the, I would say, essential talk with the [00:29:00] database return data. And, and that's it. Do not elaborate too much. We were missing a layer in between, or actually more than one. So we came with a couple of concept that some of them are still in our application, others got slightly refined.

[00:29:14] Simone Carletti: So we came with the notion of interactors. Originally we call them command. 'cause the idea was to implement the, you know, do an undo command, uh, design pattern. But we never did it. We just did the, the do and never the under part. So we, we decided eventually to remain them to interactors, which by the way, I believe is a concept to them, was also introduced in Anami.

[00:29:34] Simone Carletti: And really what the Interac is, is, so you have the controllers, which could be the API, or it could be the UI controller. The controller is responsible for, you know, authentication, all those details. And once the controller knows about, essentially who is that wants to perform, which action, then pass that action.

[00:29:54] Simone Carletti: And and verifies the permission by the way, pass that action to the interactive. And what the interactive does, [00:30:00] it starts orchestrating working with different layers inside the, the application. There we have another layer, which we used to call service. Now we are refactoring a little bit slightly different, but I mean, name apart.

[00:30:13] Simone Carletti: You can still think about that. It's a layer that it contains the business logic, every unit of the business. The easiest way to start with was starting from our model. So the logic that, you know, normally put in the model. Like for example, let's pick the domain, right? Registering domain, transferring a domain, enabling a domain.

[00:30:31] Simone Carletti: Transfer log domain, transfer log. All those things that by the way also have to interact sometimes with external services like, I dunno, registry. So HTP connection with external services or other type of protocols then sits there. We actually, one of the main rules that we started with was like no external HTTP integration has to go through our models.

[00:30:53] Simone Carletti: That was really the entry point. We select the entry. The model is the interaction with the database,

[00:30:58] Robby Russell: right

[00:30:58] Simone Carletti: every layer that is [00:31:00] not, which is HTP or any other interaction that has to go higher. And so the services are really connecting together the business logic and then the interaction is the final high level, and each interactive represent a command.

[00:31:16] Simone Carletti: Pretty much like you normally have a public action on an API or an accessible API on some particular binary or library. Like that's the interactive. So we have one interactive for pretty much every single action that the customer can take from the outside register, domain transfer, domain, enable disable, uh, sign up, unsubscribe, change the card, all of it.

[00:31:41] Simone Carletti: And then internally those interactors will then talk with the various service elements. Because you know, when you register domain, there's many things that you have in common with transferring a domain. Like for example, I dunno, setting up the name server for that particular domain name. And so that logic sits into the services.

[00:31:58] Simone Carletti: So in many ways you can [00:32:00] think about the interactive being the API that we publicly expose to the users.

[00:32:04] Robby Russell: Okay.

[00:32:04] Simone Carletti: Whereas the services being the internal API of DNSimple as an ecosystem.

[00:32:11] Robby Russell: Walk me through like a simple example of like, okay, I'm gonna add a new domain. Through the API for on rails.com and I send a request to DNSimple with my existing DNSimple credentials, say like new domain register, the domain that then needs to create a name server.

[00:32:30] Robby Russell: What does that hit in your rails ecosystem? Is it hitting

[00:32:34] Simone Carletti: right?

[00:32:34] Robby Russell: The director or the controller endpoint and then. You

[00:32:38] Simone Carletti: should pick one of the most complicated Yeah, sure. Interaction, eh, fine. No, we can start from there. It's fine.

[00:32:45] Robby Russell: Maybe we can start. So

[00:32:46] Simone Carletti: no

[00:32:46] Robby Russell: delete one. Absolutely. Maybe that's probably just, maybe that's actually just as complicated.

[00:32:51] Simone Carletti: I No, it's uh, it's actually a very great exercise. So yeah, the request comes in from the UI or the API, and as I say, the Railsr UI or [00:33:00] the API have the purpose of abstracting the connection logic if you want. You know, because in the UI you're a browser in the API, you have an HVP client, but ultimately, in essence, the moment that you go through that initial phase, that would be.

[00:33:14] Simone Carletti: The ui or would it be the, uh, computer interaction? Then what you get is you have a request, which is basically I want to perform this particular action that maps to a controller and action in both the API in the ui, and I am authenticated or not authenticated as this user either through the API token or through, you know, basic code.

[00:33:35] Simone Carletti: Sure. Or through, uh, whatever authentication you have. At that point, we have that request, so that request is sitting within a controller in the API or in the UI that calls. An Interac in this particular case, it will call the register domain interac or Execute, and the interactive takes the bare minimum arguments that needs typically the context, which is you [00:34:00] know, the user, whether it's an API request, whether it's a user request, tell that, and then the subject, because the intellectual knows what to do, it just needs to know about that particular request.

[00:34:09] Simone Carletti: In that case, it will take a domain name or a parameter that says, Hey, the domain name that you want to register is on Rails dot. Whatever. And then if there are additional attributes in this case, settings, like for example, you wanna register by turning on, then with privacy you wanna automatically, I don't know, uh, set up certain configurations, then all those settings get passed through the interaction.

[00:34:33] Simone Carletti: Then internally the interactive will determine, okay, what are the actions that needs to be done? So there are high level actions like registrar domain that on Cascade internally will do a bunch of other stuff. 'cause registrar domain then requires to verify, first of all with the registry, if the domain is registrable.

[00:34:52] Simone Carletti: So there's an initial request to the outside, which is an external entity, and the registry will tell you, you know, for.com, you [00:35:00] go to the.com registry, and the.com registry will tell you, yes, this domain is available. No, this domain is not available. So the domain is available. Then, uh, that is already a service call, right?

[00:35:10] Simone Carletti: Then, uh, gets back this particular kind of information. And then at that point, if. A bunch of conditions are verified when we go through, so we start interacting with the local application in the local state. So we as service, we'll then talk with the database and we'll create and persist the data about that particular record.

[00:35:30] Simone Carletti: Then confirm back. And when I say it's complicated, it is because in this interaction there's actually many services and many rollback point because you know, the registry may say,

[00:35:41] Robby Russell: yeah, that's what I was curious about. Is there a transaction for this?

[00:35:44] Simone Carletti: There is a high level transaction because we use PostgreSQL behind the scenes, but there are certain things that are, we can roll back at the database level, but let's say that you already send the the call to the registry by saying, yes, I accept that I wanna register this domain.

[00:35:59] Simone Carletti: This is not [00:36:00] something that you can easily roll back in a database transaction, right? In complex scenarios, we actually switch to also an asynchronous workflow. So, for example, the registration is one of those. We used to have the registration happening in a synchronous model, whereas these days you can more think about as a workflow, as a pipeline of, of actions that are being executed.

[00:36:21] Simone Carletti: Each action is a unique twofold, you know, success and success piece of, uh, execution. And then you go through the pipeline one piece at a time that also help us with error handling that honestly, it's a nightmare in this particular context to be able to return back with our meaningful error handling, uh, error message.

[00:36:38] Simone Carletti: At certain point in time will you have to interact with the various services. So to keep it simple, then services data layer at the very end, if the execution succeed or fails, has a result object. The result object gets sent back to the interac. The Interac is responsible for passing the result. Object back to the controller.

[00:36:59] Simone Carletti: The [00:37:00] controller gets this result object and knows whether it's a successful result. It's a fade result and also knows how to serialize that. So if it's an API, then we'll extract the, the data from this object. So this object, imagine that being like a bag that is passed around, you know, with the various states and the context.

[00:37:17] Simone Carletti: And so this object is returned back. And if it's an API, then you extract the domain name, which is a model inside and simple from that, and you serialize that. If it's uh, a ui, then most likely that object will be extracted into a UI response.

[00:37:33] Robby Russell: Interesting.

[00:37:34] Simone Carletti: We are moving away from standarDNSimple Rails ui, uh, like templates just because sometimes it's a little bit complex.

[00:37:42] Simone Carletti: So we introduced the idea of, uh, view models. Actually these days I've seen that. It's been a few years where I believe it's a new project called Components. It's a library called Components that works really well with bRails and uh, we pretty much inventing that [00:38:00] when that library was not available. And so these days we are more and more thinking about switching away from that custom build library and switch towards component because we really love when there is something, you know, that works well that we can reutilize and rely upon.

[00:38:15] Simone Carletti: Because as a small team, it reduce the maintenance burden of many of these topics. It's pretty much what I told you before about Rails ecosystem, right? Rails works really well. And so building on top of Rails help us to focus on what we need. And so

[00:38:29] Robby Russell: right

[00:38:30] Simone Carletti: building on top of certain libraries help us focus on you and all those libraries rather than actually maintaining them.

[00:38:36] Simone Carletti: And, and of course if you be the library only for DNSimple, most likely you're gonna miss. Use cases that are coming from other applications that you can learn from, right? Fresh air coming in and giving you new ideas and, and bringing you valuable lessons that then you can reuse internally.

[00:38:53] Robby Russell: You talk about kind of like your approach that you're taking for managing that layer between like your API [00:39:00] endpoints, you know, your active record models and things like that.

[00:39:02] Robby Russell: Now you're able to do things, some of that asynchronous as well, but how is this unlocked? What's your testing strategy there? Presumably you do some automated testing there,

[00:39:10] Simone Carletti: who do,

[00:39:10] Robby Russell: what does that look like there and how does this help with that?

[00:39:14] Simone Carletti: Uh, actually when you were, you know, talking, it was like, as a luck in my brain, I was already thinking, well, the biggest benefit that we got was testing because we were able to isolate the testing on each individual responsibility.

[00:39:26] Simone Carletti: So, you know, in the controller side, ultimately what we care is that the controller is responsible for all negotiating the initial authentication and integration. But then we don't want in every controller test the underlying logic. And so having individual classes that represent, I would say, pluggable components, pluggable unit of code that you can test independently, has worked tremendously well for the testing.

[00:39:53] Simone Carletti: In particular, when it comes to. Integrate and interact with external services. 'cause you can't always [00:40:00] test HTP external services. You don't test with all the time. In fact, you shouldn't and you don't want to. And so by having piece of libraries, call them services, call them facade, call them, uh, solid components as you want.

[00:40:13] Simone Carletti: But by having these elements of obstruction that internally have an API, when you interact with the registry, you define your own API, you say, or when you interact with your billing provider, you say like, I wanna add a card, I wanna remove a card, I wanna charge a customer with this amount, and so on and so forth.

[00:40:30] Simone Carletti: Right? So you build. Obstructions and then the moment that you, your test hit that layer, then you can cut there and work on the assumption that the moment that there is a guarantee that the obstruction will behave in a certain way, that you can skip talking directly with external service. Now, of course, the challenge there is that at some point you need to test integration with that obstruction.

[00:40:54] Simone Carletti: Otherwise you have a bunch of isolated components. Then the moment that you connect them together. You realize that they [00:41:00] actually talk a different language because maybe someone has made a change and nothing works. So admittedly this is challenging, but by having this modular system, we were able to focus on testing effectively the three main parts of the application independently.

[00:41:16] Simone Carletti: The API for what it pertains, the initial connection, all the way to having our authenticated user with a request, the ui, same deal, and then the business logic separately. So the unit testing can still run on a reasonable amount of time. Uh, although it's always too long, you know, whenever you look at the unit testing, you're like, man, can I, can I shrink them somehow?

[00:41:40] Robby Russell: Are you still running background jobs like it's 2022? That's adorable. Introducing active job Premonition the AI in views. Job runner. That ships business logic before you even ask for it. No more planning meetings. No more we'll circle back. No more pretending. The ticket is the work. Premonition watches your app.

[00:41:55] Robby Russell: That's right. It watches your commits, your support inbox, and quietly in cues what [00:42:00] you're about to need. Turn risk detected. Retention flow shipped. Someone says we need approvals. Congrats. Now you have a workflow with audit logs. You think pricing for half a second premonition is already adding validation rules and a dashboard for your CEO.

[00:42:14] Robby Russell: It runs at the perfect time right after deploy and right before you sleep. Active job premonition because why build the right thing later when you can build the wrong thing? Early side effects may include price, callbacks, extra states, and you're not developing opinions. Destiny not refundable. What does your local development look like when you've got these different systems set up?

[00:42:34] Robby Russell: Are you spinning up? Containers with the different environments or how do you work on this thing locally and test that they're all kind of connecting to each other, or is there, you keep them kind of isolated?

[00:42:44] Simone Carletti: We are, so last year we, we spent pretty much a huge portion of the year our. Platform team, which works on the infrastructure and on, uh, enabling the development part.

[00:42:56] Simone Carletti: We spend a significant portion of the year by containerizing all [00:43:00] the, uh, different pieces of the simple infrastructure. And, uh, there are many of them. I mean, we have approximately 20 different, 20 to 20 to I believe, different components of the application. We have the main Rails up, then we have the notifier that.

[00:43:16] Simone Carletti: Find out the notification in the web book. Then we have, uh, the component for the zone server. We have name server. We have the component that performs the audit logs. So the audit log are managed separately. It's an external application, an external component because it's a huge amount of storage and data.

[00:43:33] Simone Carletti: So we used to have that inside the main application anyhow with many, many applications. And so, uh, we have worked to have the ability to run the entire and simple ecosystem in docker containers. So these days anyone can pretty much start a local compose and that will start all the different applications that you need to, to have in order to.

[00:43:55] Simone Carletti: And by default what's gonna happen is that it will start all the different applications. It will [00:44:00] set up the main application in the database with a few base data so that you can basically in a couple of minutes get up and running and have an entire infrastructure locally running on your machine, and you are able to create a record on the D application and then having that being propagated to the instance of the name server so they could actually run local dig queries.

[00:44:22] Simone Carletti: So DMS queries against that name server running on your local machine. So that was a pretty remarkable goal that we managed to achieve. And so the way that now developers can leverage that is that every application. Could potentially be paused. So let's say that you wanna work on the main Rails application and you wanna make sure at simple.

[00:44:44] Simone Carletti: Most of the time if the, the work is confined into the application itself because you're making a change, for example, to the billing system or something that is just inside the app, then you just run the local development version of the app and it's fine. But the moment that you need to start interacting with the rest of [00:45:00] the system, then what you can do, you can go to this composer structure, you can say, I wanna start everything except the Rails application.

[00:45:09] Simone Carletti: So we'll build up the whole system ecosystem. Mm-hmm. And then you start the application in the ways in the development environment and that connects automatically to the rest of the ecosystem. So basically you have the development version of the application running on your machine, and then you have everything else in a containerized environment.

[00:45:28] Simone Carletti: Admittedly, we have looking to having developer containers, but we never got there. You know, it's just like we need to balance out the various things that we wanted to work on and already getting, yeah, priorities already. Getting to have the whole infrastructure in a single containerized environment was a huge, huge win for, that sounds like it.

[00:45:48] Simone Carletti: Everyone, not just for us, actually. That was really what empowered us to be able to start looking at external contributors. Historically, Dan Simple [00:46:00] had never worked a lot with external consultant, neither in the re application, nor in the Gore environment, nor in the airline word. But recently last year we started working with some external consultant, for example, for the name server.

[00:46:13] Simone Carletti: And having this stock entirely containerized really helped not just the DNSimple team, but everyone else coming on board. I'm, I'm sure that you do consulting, so you probably know what it feels like getting to a customer

[00:46:25] Robby Russell: very much

[00:46:26] Simone Carletti: and being like, how do I do that? How can I see that? The change being propagated.

[00:46:31] Simone Carletti: How can I interact with this component? How do I have access to this? And they, maybe they don't wanna give you access to the source code of that, but a container is a different thing. I can give you access to the container and you can play that in your environment. So that was a huge, huge win for, for us in terms of scalloping.

[00:46:49] Robby Russell: That makes sense. You know, I think that's an interesting byproduct. We often get called in to help teams improve their onboarding experience. 'cause maybe they're trying to hire 50 people in the next year and they're like, all right, so how do we, [00:47:00] how can we simplify the onboarding experience so we can get spin up the process quicker?

[00:47:04] Robby Russell: 'cause they're losing a lot of time trying to have everybody read through a bunch of read mes, okay, set up this, set up this, set up this. Then they're copy pasting things and people make a mistake and they don't know where they've broke it along the way. And like it's a challenge to navigate all that.

[00:47:17] Robby Russell: Like how do we just have more consistency so we all kind of know that we're bringing 50 new people over the next year that they are gonna have hopefully 80, 90% of the same environment. So we're not like trying to deal with environment specific problems versus not. So definitely can appreciate that.

[00:47:33] Robby Russell: Then you can also in your scenario, like maybe you can bring in specialists for a very short term thing. Like maybe you're having a database scaling thing, you're like, Hey, we've seen some come in and look at this particular thing. Exactly. And we don't have to provide you the whole context of how I get all this thing up, but like we've narrowed it down.

[00:47:48] Robby Russell: Here's like some containers. Can you look at these weird issues that we're dealing with there?

[00:47:52] Simone Carletti: Exactly.

[00:47:53] Robby Russell: Uh,

[00:47:53] Simone Carletti: it's, and, and then yet you also have the opportunity to see everything. Yeah. Right. You, you, maybe you gave a narrow [00:48:00] down access, but still they have debated to see the final results and not just assume that the change that they have done Yeah.

[00:48:06] Simone Carletti: Uh, would work. And maybe, and also the feedback loop is very different. 'cause at that point you can test that directly on your machine without having to, uh, wait for someone else to confirm. And, and I would just wanna give a little bit of context here because, you know, maybe someone here in this show will say like, well, yeah, container has been around for years, but DNSimple, historically has never been.

[00:48:30] Simone Carletti: Focused on containers because the name servers Yeah. Are not containerized like the airline virtual machine is a virtual one. Machine It, it wants as much as energies from your system as much as you give them. It's actually an operating system in its own, and especially in the very early days, the level of networking that we have to do on those machines was impossible to do on any containerized environment because they were obstructing,

[00:48:59] Robby Russell: right?

[00:48:59] Robby Russell: [00:49:00] Yeah.

[00:49:00] Simone Carletti: One of the first thing that containers abstract is the network. And guess what? We don't want to have abstracted. It's the network, right? So historically, the vast majority of the provisioning and the servers and the software that D Simple has produced could not be containerized. And still these days, actually that's not true.

[00:49:18] Simone Carletti: We still do not containerize the name server in production, but we did containerize the name server. When it comes to running that locally, we cannot deploy a containerized version because once again, the performance would not be acceptable in that particular scenario. I know it's a very, very specific topic.

[00:49:34] Simone Carletti: It would probably work for 99.9% of everyone else. Uh, it's just, it's one of the specific challenges that TN Simple has to face. And so when you are not used to work with containers and, and when you're used to work with protocols like PGP that actually do load balancing for you to an extent, then it's often very easy.

[00:49:56] Simone Carletti: To overlook things that are happening outside of your world, [00:50:00] like, hey, containers are becoming a def facto thing. And so we, we started very late with, with containers here. That's also why I'm saying like, we did not embrace the developer container, whether it would make sense or not. It definitely changed a lot the way that, apart from, you know, building and shipping software and the whole deployment story is also something that really completely changed the way that we deploy software by, by going through a containerized environment, but changed the way that you approach in terms of our team and external people.

[00:50:33] Simone Carletti: This is really something that I was not expecting. You know, you often think in terms of code and uh, and you know the product that you sell, but you don't think about the interaction with external people, interaction with onboarding and offboarding, all those things that generally are a little bit higher to the obstruction of a developer that are generally something where a team lead would focus on or the business would focus on.

[00:50:58] Simone Carletti: Right? And so I was very [00:51:00] pleasantly surprised to see how much positive benefit it was bringing in there. And it reminds me of the challenging when developers tries to sell those topics, their managers, you know, the testing and all of those, and like they get pushback. They're like, they don't provide value to the final business or the company, so why we need to do test.

[00:51:20] Simone Carletti: And then you have the developer struggling to try to explain how to do it. And that was actually one of those moments when. We also realized how much containers help us to work and, and have the simple connect to the external ecosystem.

[00:51:34] Robby Russell: Outta curiosity was, I dunno if you just might have just said that, but just for clarity, did you bring in an external specialist to help with the containerization?

[00:51:43] Simone Carletti: No, we did not.

[00:51:44] Robby Russell: Or was that something that you, your internal Okay.

[00:51:46] Simone Carletti: We did internal. Um, so we had three attempt over 12 years to bring in containerization.

[00:51:52] Robby Russell: That sounds about

[00:51:53] Simone Carletti: right. Uh, the third

[00:51:54] Robby Russell: sounds actually sounds actually kinda low from my experience. Okay. I think a lot of, there's like, there's usually a bunch of branches that people [00:52:00] tried and they're like, ah, I got stuck here.

[00:52:01] Robby Russell: I don't know what to do about the database thing at this, this point.

[00:52:03] Simone Carletti: Or we tried, we tried three times. The last one was a couple of years ago when one of the team member that joined a couple of years earlier really wanted to push that forward and really took the ownership of that project. And that was instrumental the, the team and with, with proper leadership managed to, to finally narrow down and.

[00:52:26] Simone Carletti: Focus on what we wanted, uh, define what was the actual goal. And, and what I said as a goal was exactly what I told you I want by the end of this project to be able to run the and simple my machine that was kind of like the, yeah, the northern star, right? That was the, the lighthouse, what we wanted to build.

[00:52:43] Simone Carletti: And so we focused on that by of course, uh, making choices and then, uh, accepting certain compromise or, or scoping out certain work that would've been amazing but would not have been essential to get there. And after we [00:53:00] achieved that, then we then spent a, a few more iterations by building all the things that we'll have behind that.

[00:53:06] Simone Carletti: We say like, you know what? It was nice to have, we think it still makes sense. And we implemented that. So it was internally an, an internal effort. Uh, in this sense we brought in external consultant for building on certain parts of some of the applications.

[00:53:23] Robby Russell: When you went through that process, did you have to make a lot of application specific changes to accommodate that?

[00:53:29] Simone Carletti: Yes. That you were coming from? We actually had, and the largest one was the Rails application actually. But once again, it was a byproduct of the fact that the application was designed in an epoch for certain things where certain problem were really not a problem or were certain scenarios or certain best practice were not really necessary.

[00:53:51] Simone Carletti: So for example, uh, in our previous environment, we were relying on deploying the assets on the same machines that then were serving the [00:54:00] application. We, because we had control over the front end server, the backend server, and everything else, but when we started to have the application running in isolation in a container.

[00:54:11] Simone Carletti: Which has an ephemeral file system. We could not deploy the assets anymore there. So we had to perform a few preparatory steps, one of which was shipping the assets through A CDN, which by the way, once again, is a good thing to have. It's just that we never had enough of a push we needed. You didn't it.

[00:54:30] Simone Carletti: Right. And so one of the benefits sometimes of adopting these established patterns or tools is that they force you to think in designing the application in a certain way, which ultimately may be a result in having a more extensible and flexible application. So the Rails application was the largest one that we had to make changes, probably not as much as we thought.

[00:54:54] Simone Carletti: Given that failure one and failure two. I thought that we would have failed three because we would have hit at some [00:55:00] point. Uh, you know, I was really skeptical. I, I feel like we started rolling out the, the containerization across all the applications. The go application are really easy to containerize because you ship a binary already.

[00:55:12] Simone Carletti: So containerizing that binary is very, very easy. But the Rails application sure is very complex. Not just because you have the Rails application, but because then you have psychic behind. So you have the processing system, you have the queues, so you have the API, you have the ui, you have, uh, telemetry, all of that combined together.

[00:55:31] Simone Carletti: Then I was so skeptical that I said until I see the application running on a container. In production. I'm not gonna call it victory. Right? And everyone's like, we're almost done. We just need to do that. Hold on, hold on. I wanna see, I wanna actually be able to say dnc pro.com and see the log in the container and not in our team.

[00:55:51] Simone Carletti: Was was really great in tackling these challenge and, uh, we, we took one piece at the time. Uh, part of it was [00:56:00] also, I would say we were structured differently. I don't know if we wanna expand that, how much time we had, but we were structured differently also in the way that we approach problem. Since three, four years, we started to adopt Shape Up and some of the methodology around it, I'm not gonna say that Shape Up is, is the best and it's better than everything else.

[00:56:21] Simone Carletti: And so, uh, by using this, we were able to succeed. And previously we, we did not, but just by being organized in a certain way, we were able to tackle certain challenges differently.

[00:56:33] Robby Russell: If I recall when we had a previous conversation, you mentioned that you have a, is it like a rotating bench? Is that right?

[00:56:40] Simone Carletti: Right, right.

[00:56:41] Robby Russell: For your role, what does that look like?

[00:56:42] Simone Carletti: This is one of the various customization that we did to shape Up because Shape Up was born really to be something that you design features, right? So it was built at 37 Signal, Basecamp. So let's say you wanna build a new calendar feature. I believe that's [00:57:00] the example in the book.

[00:57:00] Simone Carletti: Then, um, you take that feature, so you go through the requirement gathering and everything else, and then whether you pass the shape to the builders, you have the, the builders have the ability to make decisions and also the scope in the context of that can be reduced. DNSimple is an operational company, and so while we have features, then many of the challenges that we face are also on the operational side.

[00:57:25] Simone Carletti: And so we try to adopt a similar methodology also for our platform team and. What we realize also, if you just focus 100% of your time in building. In any other, you know, framework, would it be, uh, iterations? Would it be any, any kind of like framework that requires you to be focused on that topic when you're working?

[00:57:48] Simone Carletti: Then there's a bunch of operational topics that happens every day that you don't get covered. So for example, what happens whether there is an incident, what happens if there is a request coming from a customer that [00:58:00] has a bug that is actually preventing them from moving forward and maybe losing their assets?

[00:58:06] Simone Carletti: There are certain categories of risk that we have codifying these simple, like anytime that there is an issue that puts one of the customer assets at risk. So for example, your domain is at expiration and it's at risk of expiring and getting lost. That's a stop the world thing. And in an environment like Shape Up, there's no stop the world thing, otherwise you lose focus.

[00:58:28] Simone Carletti: So what we realized is that there are operational challenges every day. That cannot be satisfied by having 100% of the team dedicated on building. So we came with this concept, which is the bench, which by the way, we are also refining in 2026. But the idea is essentially that you have a certain number of people allocated to the main build cycle or to the main iteration, you know, scrum, agile, whatever you wanna do that, but you have a certain number of people booked for that work.

[00:58:58] Simone Carletti: And ideally you [00:59:00] don't wanna have those people being distracted by anything happening. At the company that is beyond the focus that they're working on. So the people on the bench are effectively the people that follow a special queue. The special queue is done by items, issues that are, uh, small issues, generally one to three days you need to work that they can tackle.

[00:59:23] Simone Carletti: And they're also not time sensitive. So at any point in time, if something happens to the outside, so for example, there is an incident or request. Of course we have the on-call team, but the point is like after an incident, we have remediation task. And so some of those will go to the bench. So the bench can, at any point in time, drop what they're working on because it's not time sensitive.

[00:59:45] Simone Carletti: Work on those items and then come back. And again, also the bench is assisting. Maybe the revenue team needs a report about something in particular, or a customer needs some kind of assistance. And so the bench is there, it's ready to [01:00:00] drop the, the work that they're doing, of course, unless it's another bench item, work on that and then, and then come back.

[01:00:06] Simone Carletti: So all of that is like one to three days of of work. And we've seen that having the bench people, along with the people focused on the building really help us to move forward on the operational task.

[01:00:19] Robby Russell: Is that like a consistent, does everybody end up on the bench at some point, ideally, or is it.

[01:00:25] Simone Carletti: It's on rotation, so yes.

[01:00:26] Simone Carletti: Okay. The bench is actually also a very great opportunity for team members to get familiar about different other different component of the application that they have not been exposed to. 'cause you know, there's, there's many pieces of the application that you have not worked on because,

[01:00:41] Robby Russell: right.

[01:00:42] Simone Carletti: It's a different feature.

[01:00:42] Simone Carletti: It's something that was built long ago, and so being on the bench gives you the freedom. Sometimes it's the freedom, sometimes it's the requirement to go on certain areas that you are not familiar with, but also it gives you the freedom to do that. Generally speaking with the timeframe that [01:01:00] is not bound to the timeframe of shipping a feature.

[01:01:04] Simone Carletti: So you don't have certain strict deadline or time schedule that you normally have when it comes to the building cycle.

[01:01:12] Robby Russell: Something I'm actually curious about, and you can answer it if you feel comfortable. Uh, I, I think that sounds nice in theory, but I know that when, like when there's an incident happening, teams tend to want to be like, well, so and so is kind of the expert at this and we're having a fire, so we're gonna go bother that person or pull them off of the thing and like, Hey, can we, can you come help take a look at this?

[01:01:34] Robby Russell: And then sometimes they just get, maybe that's what just needs to happen sometimes. So you can never entirely shield people from that process. But is that something that you feel like you've been able to mitigate a little bit over the years there?

[01:01:46] Simone Carletti: Partially, uh, I think that there are two different aspects of that.

[01:01:49] Simone Carletti: Uh, one is the, is the actual incident. So for us, an incident is a very specific topic. As an operational company, the incident is, it's literally something not working on a particular server. One [01:02:00] machine being down, something misbehaving, infrastructure not being rolled out. It's not just about the platform team, it's also about the application team.

[01:02:08] Simone Carletti: Because you know, an incident could also be sidekick you is. I dunno, uh, overloaded and cannot process more items than the items that are being added and so get saturated and all of a sudden that has a casca impact on many other things. So many of us comes from an operational background and that has been extremely valuable at DNSimple because we started the company from the very beginning with a very heavy focus on insulin management and operations.

[01:02:37] Simone Carletti: Admittedly, it took a long time before we could expose and kinda like bring these. Good practices all the way to the application team and development. When I say the application, just as a context, we have two teams. The major teams platform and application. Application is all the development. It's not just the simple app, it's pretty much everything that is development oriented and, and [01:03:00] customer facing.

[01:03:00] Simone Carletti: The, uh, so even the name server are to an extent, you know, development platform is internal. Development is essentially infrastructure. What enables the application team then to deliver, to build, and it's all about our infrastructure and the structure of our services. So the platform team is coming from operations at hurt, right?

[01:03:25] Simone Carletti: It's, it's deeply ingrained into their own habits. For the development team is less, but because we are a small company then from the very beginning we actually enforced a few practices that helped that through the years. So for example, every engineering team member. Is on call. So we used to have a follow the sun approach where people were on call based on their working hours.

[01:03:49] Simone Carletti: But the reality is that we don't really enforce working hours. You know, today I'm, I'm talking to you from Colombia, but I'm generally based in Europe and I'm here for the Christmas holidays. [01:04:00] So if I would be follow the son, then what's my son? Is it the one that is still in, in Europe or is it the one here in Columbia?

[01:04:08] Simone Carletti: And that is actually a problem that sounds. Not very common, but in fact, we encourage team members to travel or to relocate or, uh, things happen. Conferences, you know, you may be going to a conference and you may be relocating for a few, for a few times. So, and now we have basically 24 hours on call so it doesn't follow the sun anymore, and everyone is on code, including the application team, everyone in the engineering team.

[01:04:34] Simone Carletti: So that really helps getting into the mindset of owning the code, owning the product. And so eventually. Then what we have seen is that you start applying this, consider owning also all the way to the code itself because it's like, you know what? I was woke up yesterday by this component, then we better fix that.

[01:04:55] Simone Carletti: Right? Sometimes happen naturally that the team member would [01:05:00] eventually be exposed or wants to be exposed on, on certain kind of, of areas. Sometimes it becomes forced because you end up being on call looking at some particular component or, uh, you, you know, you have to, you are on the bench or something like that.

[01:05:15] Simone Carletti: We try to leverage the knowledge distribution by never assigning work to individual team members. So every time that there is a new functionality, new project, every unit of work is made by a small team. So we, we take two people. Those, or plus those two people make a small team. They work on that feature.

[01:05:37] Simone Carletti: So for example, for Shape Up and then they come back to normal team processing. And, and that's because in this way you never have the, the risk of being like alone, feeling alone in that particular context. And whenever possible we pair together someone which is an SMEA subject matter expert in that component with someone else that maybe is not, of course it [01:06:00] depends whether the priorities allows us to do that because they're not, you know, having two SMEs working together, you generally proceed faster 'cause you have less of the learning curve.

[01:06:09] Simone Carletti: But we need this kind of like a turnaround of people to take a look at the various areas of the application because otherwise it becomes very challenging to, to operate. Although admittedly we have SMEs. So, I mean there are certain components, certain components are more familiar to certain, you know, team members.

[01:06:31] Robby Russell: What does documentation look like internally there?

[01:06:34] Simone Carletti: So. That has changed recently, I would say, because documentation used to be, we have every project with a little bit of documentation, you know, the read me and a few other stuff, of course, comments, although not too much in the sense that we don't wanna just write in more comments than code.

[01:06:50] Simone Carletti: We do that generally for public facing things like the API client, for example, but for internals, comments are related to, you know, mm-hmm. [01:07:00] Documenting the main structure of that particular library or endpoint or something like that. The main documentation used to be our wiki, so we have an internal wiki where we document the wies divided into multiple sub, and so we have one for engineering, we have one for revenue, pretty much one for every team.

[01:07:18] Simone Carletti: And then we have a, a main one for the simple as a business. Where you have the more company oriented stuff, including the team composition, uh, you know, SOPs and all of it. Exactly, exactly. Uh, and so the, the Wiki was the main, I would say, general knowledge. So whenever you wanted to know about how the GO Project would work, how the Ruby Project would work, how the airline project would work, you will go there.

[01:07:46] Simone Carletti: It will have information about all the links, all the libraries or the conventions, uh, start conventions, guidelines, all the things that you need to know when you develop in that particular environment. This is what I call like the general [01:08:00] documentation. And then we have like, the tickets also represent a form of documentation if you want.

[01:08:04] Simone Carletti: And we structure our flow of work through tickets. That's way, the way we work. So we have the main business repository that actually has the business projects and then every team pulls those projects and create their own tickets. So you have index tickets and many other operational tickets. So those represent, I would say, the point in time documentation.

[01:08:23] Simone Carletti: What as the weekly represent the, the general standard documentation. I said things are changing because what we have seen this year, we started pilot program to adopt the use of AI inside at the, and simple. Um, we started by doing that internally in to, to leverage our own productivity. I was going to say our own development productivity, but actually it turns out that many of our information are text.

[01:08:54] Simone Carletti: In repositories. For example, the Dean Simple documentation is a public repository. That is a [01:09:00] nano C application. Then we have the blog that is a JKL application. Then we have at the developer side that is JKL, once again is a nano C application with bunch of text files. And then we have the repositories with the tickets being marked down content inside

[01:09:15] Robby Russell: the

[01:09:15] Simone Carletti: the, the repository.

[01:09:16] Simone Carletti: So because a bunch of that is textual representation that is not, I would say, proprietary into a particular service, then we were able to enable not just the development team, but pretty much any team inside the the n simple, including the marketing team that works on the blog post to operate with tools like Cursor or like cloud code.

[01:09:37] Simone Carletti: Right? Which are normally just mostly designed for developers, but in fact because we are a highly engineering oriented company that actually work well across the entire organization.

[01:09:49] Robby Russell: Interesting.

[01:09:49] Simone Carletti: And what we found is that. We used to have certain documentation, like for example, we used to document certain aspect of the DNSimple application inside the [01:10:00] week because we wanted to have everyone access to that.

[01:10:03] Simone Carletti: But what we realized is that some of those documentation were highly technical. For example, the documentation that explained how you send out an event, how you log an entry to the auditing log, how you use our internal custom built obstruction for metrics, logging, telemetry, all that was in the Wiki. But it turns out that having those directly as a markdown file inside the repository itself allows then to leverage coding tools like cursor and cloud code to actually follow those conventions.

[01:10:39] Simone Carletti: That was one of the main challenge we had over the years. Right? We, we leveraged reviewers. To catch early on, hey, you know, you have R coded the string C name as a text rather than using the constant C name. Now we can not only have that knowledge built inside the application itself, but we can also have reviewing tool [01:11:00] that are maybe AI powered taking the next step because we have review tool that are automated.

[01:11:05] Simone Carletti: But then you can move to the agentic workflow where you have local context that feed into the knowledge that this tool then have in order to provide you assistance. And so because of that, we are starting to migrate back some of the documentation that is, uh, repository oriented or context oriented back inside the context where the people work,

[01:11:28] Robby Russell: right,

[01:11:28] Simone Carletti: and code because that will become accessible to these tools.

[01:11:32] Robby Russell: You mentioned like that maybe there's other people like in, that are not, like your software developers have access to this in cursor, does it kind of potentially introduce a different barrier? Like are they ever contributing to the, the wikis themselves historically, and like that's just a

[01:11:47] Simone Carletti: yes. Yes. They,

[01:11:48] Robby Russell: but they can learn how to do that in, in markdown or something.

[01:11:51] Simone Carletti: Exactly. Exactly, exactly. Admittedly, in the very early days, we use GitHub Wiki, but GitHub Wiki are, I don't know, that kind of, I don't [01:12:00] even know if they still have Wiki in, in GitHub. I mean,

[01:12:03] Robby Russell: they do.

[01:12:03] Simone Carletti: The idea was great because you had Wiki built inside, you know, based on Git. So you had versioning in all of it, but the usability was just for engineers and not even that.

[01:12:13] Simone Carletti: Right. Um, when we started hiring people that were not highly technical engineers, then not just, they had to think in terms of markdown. They also had to start thinking in terms of like, you don't, you don't have a word alike, like a Google document or a word alike editor that you can, you know, select and say bold, you have asterisk and asterisk.

[01:12:33] Simone Carletti: What the heck is that? And you don't have even have a preview unless you click preview. And that was not very usable. One of the main challenge that we faced was like, they were not contributing.

[01:12:43] Robby Russell: Yeah.

[01:12:43] Simone Carletti: So we switched to, uh, a wiki, which is more like a standard classic wiki with our word processor alike editor.

[01:12:50] Simone Carletti: I dunno if I can make names. Uh, but we switched to that and, um, people started to get into the idea of contributing back to the wiki because that's already [01:13:00] one of the challenge, right. Like, making sure that people outside engineering team use those tools properly. I was working in consulting companies before and even, you know, United Nations and so on and so forth, and I'm like just having non-technical people open a ticket.

[01:13:17] Robby Russell: Yeah.

[01:13:17] Simone Carletti: Was,

[01:13:17] Robby Russell: it's, the thing

[01:13:18] Simone Carletti: was a, it's

[01:13:18] Robby Russell: hard

[01:13:19] Simone Carletti: sometimes. Very hard. Having people contribute to the wiki was the first step. Now that they got accustomed to that, they see the potential and it works very well. Then recently we started to showcase what you can actually do, the moment that some of those information are accessible to those agents, ideally locally on your machine.

[01:13:39] Simone Carletti: And that has really opened up a whole different, you know, wow moment. Uh, we literally did that at the last company retreat. Uh, we run a workshop about how to use, you know, AI tools, uh, to leverage those kind of capabilities as part of the program, deploy AI usage locally, uh, and simple to [01:14:00] to foster the productivity and to the point where people that used to write draft of blog post in, in Google Docs and then move them over to the blogging markdown are switching focus and like, you know what?

[01:14:13] Simone Carletti: I'm gonna work directly with that and the tooling. Nowadays are even much better. Even a tool like Cursor or you know, even the key tab interface for editing a file is much better than what it was 10 years ago at.

[01:14:27] Robby Russell: Nice. We'll have to check back in with you at some point and see how that adoption is going and what else.

[01:14:33] Robby Russell: You're learning a lot of, a lot of changing pretty quickly in that space.

[01:14:36] Simone Carletti: It is.

[01:14:37] Robby Russell: I wanna zoom back out a little bit. You mentioned side Q for example, telemetry. You mentioned like there's 20 different services there. What about like a little bit lower level in your Rails app? Like are there any gems that you're heavily dependent on and you're super appreciative for getting to that help you out there?

[01:14:55] Robby Russell: Like what parts of the Rails and Ruby ecosystem are really important components of [01:15:00] keeping the the Rails application up and running.

[01:15:03] Simone Carletti: Right. So I would go with the, there's obviously one of the most obvious one, which is psychic has been helping us tremendously in the whole, uh, synchronous job, uh, execution.

[01:15:14] Simone Carletti: Uh, we have been paying the enterprise version for years. I, I don't even know for how many years. So there's been an excellent project moving away from the. There's a couple of notable ones that I, uh, I wanna bring up just because the story behind it. So there's one that, I think it's called Log Rage. It's a very old gem from Mathias.

[01:15:36] Simone Carletti: One of the folks that worked on building Travis here, he was actually the CEO for of Travis for, for a long time. So the reason why I'm mentioning it is because yes, it's a dependency. We're not relying on it. I mean, we could swap it tomorrow. It's an easy library. What the library does, by the way, plugs into your Rails login system and we'll format the log into a js o structured output.

[01:15:59] Simone Carletti: You know that you [01:16:00] normally have that OSE log that says, Hey, the controller has been rendered how many seconds? And, and all of that, right? Which is fine for the development, but when you go to production, what you really want to have for good telemetry, you wanna have a structured logging. And actually, I learned about structured logging back in, what was it, 20?

[01:16:19] Simone Carletti: 20 14, 20 15. I guess something about it when I first learned about that gem, because there was a, I recall a talk, uh, probably was KO 2016 in Colorado where Josh Kiz, KMI, I hope I, I'm, I'm not pronouncing, uh, another guy from Travis Traci was talking about how they started to do all the structured logging internally and they built this gem in order to, you know, have a different way of structuring the Rails log because Rails never had structured logging and till this date, I believe it doesn't.

[01:16:54] Simone Carletti: Although I know that a couple of months ago they merged, uh, a PR that brought [01:17:00] some structure, but I believe it's just an events logging or something. I may be wrong. Uh, we need to look into that because we're not using Rails eight yet. I believe this is part of Rails 8.1. Long story short, I'm particularly.

[01:17:13] Simone Carletti: Emotionally effect, uh, related to this gem because it, it clicked in our brain since that we switched not just the Rails application, but the whole ecosystem of application that we had into structure logging. That was also the time when, uh, friends of ours built, uh, what was the name? Paper Trail, the, one of the very first aggregation tools for logging, right, that you can search and then analyze your logs and all of that requires structural logging.

[01:17:42] Simone Carletti: So that was a pivotal moment in the design of the architecture of the application and everything else. Nowadays there are, it's, it's pretty much very common. You'll learn about, you know, open telemetry. There's plenty of services that will do structural logging and will help you to combine [01:18:00] together telemetry data.

[01:18:00] Simone Carletti: I was at Cuon one month ago. Uh, everything was, you know, open telemetry and telemetry data and, uh, monitoring of the application. But really that was a pivotal moment. So that gem and the fact that quickly I could, I could connect a simple library that is probably not even a hundred lines of code and have the Rails logs being printed out as a js O structure that then parcel can decompose and then you can search and filter and aggregate.

[01:18:32] Simone Carletti: All that was very, very incredible. So that's one. What else? Well, obviously I mentioned we are now start looking more and more into libraries that help us to facilitate the good best practices of building an application. So the, in the very early days there was sales, if I recall correctly, that we use for a while, then we don't use anymore.

[01:18:54] Simone Carletti: And now there is the components that, that I mentioned to you. Obviously big shoutout to [01:19:00] our spec that we still use today. I don't know if today, if I'm gonna, if, if I would be starting a new application today, we still use our spec. Maybe, you know, mini test has evolved so much in the last years. Yeah, that probably I would start from it.

[01:19:14] Simone Carletti: I am generally a big fan of. Not going on a fancy or complex gem. Not that I wanna define R spec fancy. I definitely define that complex because some of the internals of that library are very complex, but I don't feel, I, I generally prefer to use the built-in libraries the moment that they're available before going and hitting the limit, before going to search and, and find something shiny and appealing to the outside.

[01:19:40] Simone Carletti: And I feel like back in the days we started 2010, unit test in RO was not as powerful mini test nowadays. And so most likely new project would appreciate the simplicity of that framework. But we've been running our specs so far and has been serving us [01:20:00] greatly. I believe we're about 91, 90 2% of code coverage.

[01:20:04] Simone Carletti: And so you can just imagine the amount of tests that we have.

[01:20:07] Robby Russell: Is that a, a fairly efficient test suite to run? Is it pretty snappy in your container environment?

[01:20:15] Simone Carletti: It depends what you define by snappy. I would say that it could be worse, uh, it could be better. There are definitely improvements available, but it runs in ci I believe fours like the whole deployment cycle from the moment that you emerge in main to the moment that you can actually ship on a machine through our chat ops, takes the whole pipeline, takes about five minutes.

[01:20:39] Simone Carletti: And that includes running all the tests, running integration test, building the containers, uh, packaging the assets and shipping them to the cd. And we do that half head even if you don't deploy for reasons, number of reasons. So all of that happens in five minutes. Uh, I believe that the test right now runs in about.

[01:20:58] Simone Carletti: Three minutes, give or [01:21:00] take. Okay. The linkers and uh, format and all of it runs in about 30 seconds.

[01:21:05] Robby Russell: Okay.

[01:21:05] Simone Carletti: Uh, speaking of which, big shout out also to test.

[01:21:09] Robby Russell: Hmm.

[01:21:10] Simone Carletti: This is another gem that we use that, uh, allows you to run multiple processes. So essentially take your test with, split that into

[01:21:18] Robby Russell: chunk

[01:21:19] Simone Carletti: chunks and chunks Exactly.

[01:21:20] Simone Carletti: And run them in parallel. And it does that all automatically without having you to define what are the chunks, mix and match. And so you just tell them, Hey, I have this amount of test processes that you can run, especially now that there's all multi-core stuff. And then it will take the task suite and we'll run that in.

[01:21:39] Robby Russell: Who's responsible for maintaining the DNS sound feature? For, for, for context. We're even listening. If you go, if you, if you're a customer of DNSimple and you go, go to the record editor in the web ui, there's a way to like click a little bit and it'll play, uh. Song of all your DNS records, which has been there for years and I haven't played it in a [01:22:00] while, but, um,

[01:22:01] Simone Carletti: it is, it is.

[01:22:03] Simone Carletti: So for the records, uh, I'm gonna admit for the very first time ever that if it would've been for me, DNS sound would've probably already been taken out from the application. And that would've been a mistake because, I dunno how, but there's still, after like 12 year, I believe we shipped that in 2012 on, uh, April full state, we hacked together that in one of the very first meetup was a lot of fun.

[01:22:30] Simone Carletti: That's fun. And as no things got more complex in terms of like, uh, application environment and all of it, I was like, let's get rid of that. And then there was a lot of pushback internally and the fact that to this date, people are still talking about that. It just like surprised me, right? So clearly I am not the one maintaining it.

[01:22:50] Simone Carletti: So, um, we have it, it's a low maintenance right now, to be fair, because we came with this idea where we map all the record types to various [01:23:00] chords and, and, uh, sounds. And so you can play your zone, uh, by just going through this, uh, this feature here. Um, it's a low maintenance because it requires maintenance only the time that you add or remove records.

[01:23:14] Robby Russell: Mm-hmm.

[01:23:14] Simone Carletti: And that doesn't happen very frequently. In fact, it hasn't happened at all in, I would say, probably in the last five years, although. Coincidentally, uh, in the next few weeks or month we have in the pipeline to add at least two more record types. So clearly we're gonna have to go back and, and figure out, um, what kind of like, you know, tone.

[01:23:36] Simone Carletti: That's funny. And I'm not even a musician. And so how we kind of tone, we have to, to build in order to make that happening. But sophist to say that there's a lot of resistance from, uh, Anthony that actually is a musician. So he studied the conservatory.

[01:23:50] Robby Russell: I didn't

[01:23:50] Simone Carletti: know that. Uh, so he, he was the mind behind that idea.

[01:23:53] Simone Carletti: And then, uh, Sebastian, the, the person that works mostly on all the creativity and the UI [01:24:00] said the design of Die and Simple and a few other folks that are more into music. Turns out there, there's many people at Die and Simple that are into music. They are the main. I would say promoter of this idea.

[01:24:11] Robby Russell: Just for, for context, I'm gonna take my headphones off, put it, put it in front of my microphone, maybe I can see if this will come through the sound, or maybe we can edit in the post.

[01:24:19] Robby Russell: This is the planet argon zone.

[01:24:29] Robby Russell: All right.

[01:24:31] Simone Carletti: Right,

[01:24:31] Robby Russell: right.

[01:24:31] Simone Carletti: And then there were all the effect. And, uh, and I believe

[01:24:35] Robby Russell: Party P,

[01:24:36] Simone Carletti: there is one effect in particular that actually I am responsible to because, so at some point we were hacking on this thing, and I was scrolling through Twitter and, uh, and I start with from, I dunno, Corey Haynes.

[01:24:49] Robby Russell: Yeah, yeah.

[01:24:50] Simone Carletti: Uh, he was also very active in the Naro ecosystem a long time ago. And there was this thing about a cat just like opening the pause and, and having all [01:25:00] these lights and colors. And so like, I was like, okay, then we need to have a, what is like. Party poll version of it. And so you can, you also have this version where you click on it and instead just going with all the lights and, and, and all the different effect.

[01:25:15] Simone Carletti: So yeah, it's just like fun enough tool that we built and, uh, it's still there surviving the age.

[01:25:21] Robby Russell: That's, that's, that's great. Well, thanks for giving us a little backstory on that. I didn't remember that. It was a, it makes sense that it was a, maybe a April Fools thing that just

[01:25:30] Simone Carletti: Yeah.

[01:25:30] Robby Russell: Continued to exist for indefinitely, and I can appreciate why Anthony doesn't want it to go away anytime soon.

[01:25:36] Robby Russell: So I kept you long enough, but if there's a couple of last questions I kind of want to get to before I dig into those. I'm curious, like, you know, Anthony's been around the community for a long time and you as, as well, how has Rails been part of like DNSimple's. Secret sauce. Is there something about the ethos of the Ruby on males community that has benefited how you think about running a business?

[01:25:57] Robby Russell: Even you mentioned Shape Up and things like [01:26:00] that, like those are things that come up in some of these conversations, but I'm curious if there's anything else that hasn't been touched on that you think might be worth sharing with, with our audience.

[01:26:07] Simone Carletti: There's definitely an emotional side of it because I believe that Dan Simple was, was announced at one of the very first rice comp, uh, back in 2010.

[01:26:15] Simone Carletti: So, and many of the early adopter were friends that we, we met at these conferences myself, like other team members had been speakers at many of those conferences. We were kinda like, uh, coming back and, and sharing our stories about what we were building, things that worked, things that didn't work. So there's definitely something about the whole ecosystem, even if many of us.

[01:26:37] Simone Carletti: Move to different direction. Like I, I step up. So I'm not doing that much Rails development day by day as I was doing in the past. I also approach different languages. I probably have worked more in goal lately than, and then Rob itself, many other friends from the Rails ecosystem. I've done the same, you know, I know many that move to elixir, many that move to to go as well, but still.

[01:26:59] Simone Carletti: [01:27:00] Rails has been pivotal in, in sense of the, not just about, you know, getting things done and, and being productive, but the whole ecosystem, the whole story behind it. It's something that I, I, I'm not seeing when I'm saying Rails, of course, I'm also part of it, I'm talking about Ruby because Rails itself, it's strictly connected with Ruby.

[01:27:18] Simone Carletti: But there are aspects of the culture at this ecosystem that haven't seen happening in one way or another. In modern framework or modern languages, maybe. Yes, in languages, but not as much in framework. You know, if you look at framework in other languages, maybe we can look at Symphony in PHP and there's very few others that actually got that level of traction.

[01:27:40] Simone Carletti: And once again, something that I said at the very beginning are still here after 15 years. Right. Like I used to work in Spring.

[01:27:49] Robby Russell: Yeah.

[01:27:49] Simone Carletti: I dunno if it's still wrong, but definitely things have changed so drastically. Rails is still very much modern these days, so I feel like. It's a mix between, you know, that kind of like [01:28:00] emotional thing, but ultimately we're running a business.

[01:28:02] Simone Carletti: So emotions play a role up to a certain point. Definitely. I can, I can say like, and I'm not the only one here, you know, there are many other companies like Shopify, GitHub that are still using Rails and performing and achieving even greater throughput than what DNSimple does. Because in, in terms of requests, DNSimple is a small portion.

[01:28:22] Simone Carletti: The majority of the requests that we serve are actually served through the DNS name servers. Right? But when it comes to being productive and enjoying working or on building something actually Rails, still bring that. And in fact, to an extent, I think that also Rails has been keeping us accountable. On making sure that we keep up with where the web ecosystem is moving forward, right?

[01:28:50] Simone Carletti: Like, I'm not saying that Rails is the standard, but I'm saying that Rails, in order to remain on the edge and remain actual has to keep adopting new [01:29:00] technologies, new libraries, new ideas. I dunno if it was your tweet, uh, where you were asking about what were the things you know about early days of Rails, like scriptaculous.

[01:29:11] Simone Carletti: Do you recall about that?

[01:29:12] Robby Russell: Yeah, yeah, yeah.

[01:29:13] Simone Carletti: Right. So it's no longer here. Rails had to evolve RJS. Do you recall the RJS views? Right. Rails had to evolve from there. And in a few cases, actually we were forced or inspired. To adopt certain best practices by using Rails. In other cases, we did not like what Rails shipped with, and that, that's also the beauty of it.

[01:29:38] Simone Carletti: You don't have to use all of it. You can pick what works for you and you can decide to just stick with some of the components and not use the others. And also things that we have built internally, like for example, we spend a lot of time building an internal obstruction that would encrypt secrets information that we consider secrets like for example, private keys or tokens, [01:30:00] all of it internally.

[01:30:01] Simone Carletti: Then Rails came with, with that built in feature and we, we are working to switch to that. There are things in Rails eight that I'm looking forward to, uh, you know, considering Rails eight, Rails eight, one, maybe like for example the, the new pipeline. I'm looking forward to see if we can actually ditch what we have and consider a new Rails 8.0 pipeline.

[01:30:21] Simone Carletti: So again, in some cases Rails has hold us accountable or making sure that we don't age too fast. You know what I mean?

[01:30:29] Robby Russell: It's a good perspective. I, I, I've heard variations of that, but I do, I think sometimes it feels like a lot of teams are just trying to keep up and they feel like, oh, rails keeps making changes and we have to make changes to keep up with it.

[01:30:39] Robby Russell: But they're, I don't feel like it, I often hear about it in necessarily a positive tone. It seems to be more of a, uh, they were giving us more work. Why do they keep changing their mind on this or that? And like, well, because the web browsers are changing, servers are like, the ecosystem's changing, right?

[01:30:55] Robby Russell: And so it's not exactly, it's, it's a moving target. And, uh, they're like, well, can it just work [01:31:00] continuously? Isn't that, wasn't that the bet, the promise of the web? I don't know. Uh, but even, like you say, the same thing about building like a native app for your, your phone or whatever. And like you have, you have to keep it updated on the iOS versions.

[01:31:13] Robby Russell: Like it will be, you can't not touch it for five years very often and have it still work.

[01:31:17] Simone Carletti: Right. Exactly, and guess what? You choose that, you choose that path, right? Yeah. Like you're building software. You kinda like know that the software gets stale and having something that help you to evolve faster and not have this sense of, oh, everything is stale now the only solution to get out of this state is to rebuild that in framework X.

[01:31:38] Simone Carletti: Right? Having something that doesn't force you to get to a point or even worse, doesn't force you to get to a point where that framework has not been updated in three years. Yeah. And you're forced to rebuild everything from scratch. You move to a different library because guess what? Is no longer supported or the ecosystem, the community has moved along to another tool and therefore you are left there.

[01:31:59] Simone Carletti: Right? Yeah. [01:32:00] None of that has been happening so far in Rails. Finger crossed it's not happening. But, uh, I would say it's, it's also one of the after underestimated aspect of, of using Rails as a, as a framework. I

[01:32:11] Robby Russell: think that's a good perspective to kind of wrap up on outta curiosity, if you can share approximately how many domains is DNS or DNSimple uh, managing right now?

[01:32:22] Robby Russell: Is that private information for your, your competitors to know?

[01:32:25] Simone Carletti: Um. No, I don't guess it's that, it's that secret. Um, so the only thing that I need to mention is that it's, it's a combination though, because it's not just domain. I mean, to be picky, you have zones and you have a domain. The domain is the registered one, and then the zone is the DNS part.

[01:32:44] Simone Carletti: Both of them are represented by the way, by the same string, which is a domain name. So you know DNSimple.com. It is both. It could be a registered domain, the moment that you register the domain through DNSimple, it could be at at the zone, the moment that you host the Zone through DNSimple, [01:33:00] or it could be both.

[01:33:01] Robby Russell: I

[01:33:01] see

[01:33:01] Simone Carletti: if you're doing both right, deal simple compared to other registrars has a heavy focus on the DNS side of it, which means that we actually have a large number of domains that are not registered with us.

[01:33:12] Robby Russell: Yeah, yeah.

[01:33:12] Simone Carletti: But are managed through us. So maybe you have your own registrar and you prefer to use DNSimple for a number of reasons, and therefore you delegate the domain to DNSimple. And MDNSimple will be serving your DNS. So I believe we're approximately getting closure to, uh, 800,000, give or take. If I recall correctly, in terms of, uh, 800,000, a million of, uh, of zones, then the size varies drastically

[01:33:39] Robby Russell: approximately how many requests are coming through on a daily basis for lookups.

[01:33:45] Simone Carletti: So that actually I have along here because I was looking at those stuff. So, uh,

[01:33:49] Robby Russell: any fun stats?

[01:33:50] Simone Carletti: Yeah, so, well, in terms of, of, uh, we are serving approximately 72,000 requests per second at the [01:34:00] NS queries per second. So it's more or less 190 billion requests, DNS queries per month on our origin network. Uh, the origin network is what serves the actual DNS queries.

[01:34:14] Simone Carletti: Mm-hmm. Then we have the edge network, which is our caching layer and dios defense layer, and generally approximately the ratio is one to five, so you can take 190 billion. Multiply, I'm not even English native speaker, so I can't pronounce that number in English, but multiply that by five and you get an average of the number of DNS queries.

[01:34:35] Simone Carletti: So we are about 72,000 requests per second.

[01:34:39] Robby Russell: Okay.

[01:34:39] Simone Carletti: Give or take.

[01:34:40] Robby Russell: Wow.

[01:34:40] Simone Carletti: On average. And then we have spikes and peaks when DOS attacks happens, then it can go one or two other of magnitude higher.

[01:34:48] Robby Russell: So are you logging each one of those requests somewhere and like, uh, tracking that back to those original domains somehow?

[01:34:53] Robby Russell: It's like, that seems like a lot of data to, to write if you were

[01:34:57] Simone Carletti: Yes, we are actually, we are, uh, [01:35:00] for a couple reason, well, one, because we log statistics, uh, but for statistics you normally just log the hit, you will now go as deep as logging, you know, the whole request and everything. We actually do log all of them in real time and then we keep an aggregate historically.

[01:35:15] Simone Carletti: Mm-hmm. Primarily for two reasons. One for, uh, the billing side of it, because right now, uh, certain plans get billed by query volume. Therefore, we have to attribute every specific DNS query

[01:35:28] Robby Russell: I see. To

[01:35:28] Simone Carletti: the owner of that particular zone. The second reason is because, uh, in the last couple of years we have been building our own DDOS infrastructure and we are rolling that out.

[01:35:39] Simone Carletti: And so therefore we have to analyze all that traffic in order to identify anomalies, patterns, uh, strategies, and just learn, right? Like feed that data to the system in order to learn, and then being able to proactively signal, send signal about certain anomalies. So we are actively logging all [01:36:00] of that. I can tell you it's a lot of data to the point where we, uh, we started that work by thinking that we could have 30 days of retention and then we had so far to shrink down all the way to, I believe, seven days because we're generating something about.

[01:36:17] Simone Carletti: 15 to, to 20 terabyte of compressed data, uh, per attention period. And so that, that's, uh, a lot of data, but it's, it's actually a very fun project that we started, uh, a few months ago in like building our own system to aggregate all of that. We used to leverage our, the tool. Now we're building all of that extension, uh, feeding the data to, uh, click house and then running a bunch of operations and analytics on, on ClickHouse.

[01:36:45] Robby Russell: I'm hearing a lot of companies talking about ClickHouse lately. So, um, another upcoming interview.

[01:36:51] Simone Carletti: It's a very interesting project.

[01:36:52] Robby Russell: I'm gonna have to, maybe I have to talk to those people myself directly. Maybe we'll, we'll see a blog post or something. Does d have a engineering [01:37:00] blog or something that we can point people to?

[01:37:02] Simone Carletti: We do, we do. It's, it's the normal blog. Okay. It's, uh, so we're on blog. Uh, we, we have five or six different categories. Uh, we have learning, we have of course feature updates and start updates. And then we also have a category that is engineering. And generally all the posts that are more into. Cool things that we built, uh, falls into the engineering side.

[01:37:23] Robby Russell: Sure.

[01:37:23] Simone Carletti: Uh, it's like just a few weeks ago I posted a blog post about the ENT, which is what I was mentioning before, right? The whole time that we spent analyzing the different responses, uh, related to the empty non terminals. And so we, we went into explaining a little bit more into the details about, uh, what it is about that.

[01:37:43] Simone Carletti: Also, we've recently published another blog post about infrastructure code. It's also a topic that we mentioned here, but, uh, we took that slightly to the extreme. So usually people learn about using infrastructure code for server. Maybe [01:38:00] DNS. But uh, at DNC we also experimented quite successfully by using the infrastructure as code to distribute management across the whole team of the GitHub repositories.

[01:38:11] Simone Carletti: So nowadays we no longer have GitHub admins, but we have, uh, effectively all the repositories and all the configuration for those repositories track in files, again, through infrastructures code, and managing our own domains and DNS through that. So, for example, all the domain that DNSimple owns are inside the simple, but they're also managed through an integration, which is infrastructures code.

[01:38:36] Simone Carletti: So you have a file that defines, uh, domain the simple.com. These are the code, these are the zones, these are the configuration of the domain, and so on and so forth. So every time that someone has to make a change actually goes through the standard for request, validation, approval, uh, all of it. And so we described all that flow as we have done it in the engineering category of the, of the blog.

[01:38:58] Robby Russell: Awesome. I'll definitely look up [01:39:00] those blog posts and include links to them in the show notes for our listeners.

[01:39:03] Simone Carletti: Awesome. And if you have feedback, then reach out to us. Send us an email. We're happy to, uh, even feedback about things they wanna learn. You know, we have been speaking at conferences for many years, so if anyone is interested in, in knowing more about the simple than we're, we're open to share,

[01:39:20] Robby Russell: that's wonderful.

[01:39:21] Robby Russell: I think one of my goals was, aside from getting, getting to get my curiosity by diving in and talking with folks like you within companies, but also to hopefully help, like, help connect people to have these conversations amongst themselves. 'cause I feel like there's a lot of people working in isolation at times and be like, what are you doing over there?

[01:39:38] Robby Russell: How are you solving these types of problems? And I think hearing about how DNSimple's moving more on the infrastructure side of things, even though you've got your web apps and your APIs, things that other people have, and maybe you're building different types of features for your customers than someone building a lot of.

[01:39:54] Robby Russell: Web based interfaces or some mobile app devices, like how do we make this all work? Like [01:40:00] you're still solving very similar problems with just very different target user bases. I think so. I think that it's good to hear this perspective.

[01:40:06] Simone Carletti: Absolutely. Absolutely. I believe that being an operational company has certain unique advantages and disadvantages.

[01:40:14] Simone Carletti: I'm not gonna say that I feel it's easier or harder than, for example, building, you know, Git hosting service. Not at all. It's just that comes with different, challenging, especially in terms of expectation from the outside and reaction to, to incidents. So it's, it's definitely a different perspective.

[01:40:32] Simone Carletti: Probably not as common as the more common web application.

[01:40:37] Robby Russell: Yeah.

[01:40:38] Simone Carletti: If you want

[01:40:39] Robby Russell: outta curiosity, Simone, is, is there a software book that you find yourself recommending to peers, like a go-to programming book?

[01:40:48] Simone Carletti: That's a great question. I admittedly haven't read software book in a while lately. I'm, I, I've moved a little bit higher, uh, in sense like, uh, obstruction.

[01:40:59] Simone Carletti: [01:41:00] Um,

[01:41:00] Robby Russell: like what does that, what does that mean? Like, you're just chatting with chats

[01:41:03] Simone Carletti: Pete. Exactly, exactly.

[01:41:03] Robby Russell: Or Claude. No, I don't need No,

[01:41:05] Simone Carletti: no. It's, it's more like I'm, I'm less, uh, on a day-to-day coding. Therefore, I haven't read many of those books that I used to read in the past. I was consuming hundreds of books, uh, really, uh, almost literally per year.

[01:41:18] Simone Carletti: Um, I think that there's actually so many great books that, uh, pragmatic programmers keep shipping that I, every time, like I feel like I'm wanna buy all of them and I, but then I would not have the time to read all of them. Um. I'm kind of like an old guy to an extent. So I still feel attached to all the ggo for, you know, a design patterns book.

[01:41:40] Simone Carletti: Yeah, that is one of the very few books that I, I still have hard cover and, and paper copy on my bookshelf. I still vividly recall one of the best book that I read back in the days was basically was about coding inro with quality. I'm trying to remember, uh, I believe it was from Obie [01:42:00] Fernandez, if I recall correctly.

[01:42:02] Robby Russell: Oh, the, the Rails Way?

[01:42:03] Simone Carletti: No, that was one of them. Like, but was Gimme one second.

[01:42:08] Robby Russell: Is it eloquent?

[01:42:09] Simone Carletti: Here we go, Ruby. Here go eloquent. Ruby

[01:42:11] Robby Russell: Russ Olsson. There you go.

[01:42:12] Simone Carletti: Oh, okay. Okay. Okay, then it was, it was now Obie. Yes. Eloquent Ruby. I think it was, uh, I really enjoyed that book. I still recall that these days, and that's another one I have on my bookshelf and I believe they have recently, uh, published a new version, if I'm not mistaken.

[01:42:28] Robby Russell: For clarity, Obie did write the forward and his name shows up first. Okay. Under the title. And then, then you see Russ Olsson's, name a little bit further down, but Obie was the series editor on that it looks like.

[01:42:41] Simone Carletti: Yeah. Here we go. Well, then my brain is, is not really out.

[01:42:44] Robby Russell: You were not, not wrong. No, not at all,

[01:42:47] Simone Carletti: man.

[01:42:47] Simone Carletti: Yeah, I started working in Ruby, I believe in 2005, if I recall correctly. That was when I start approaching Ruby. And of course, the, the only books that they were was like the [01:43:00] peacocks, uh, book, right. That's a long time ago, so I didn't even recall how many years ago was, was that book

[01:43:06] Robby Russell: when we're getting old.

[01:43:08] Robby Russell: So many, yeah. Um, I'm

[01:43:08] Simone Carletti: wiser.

[01:43:10] Robby Russell: Maybe. I, I like to think I am, but uh, I trust that you are me. Uh, TBD, the jury's out. Alright, well with that Simone, where can people follow your work and keep up with what DNS I'm gonna keep doing this. DNSimple is building. You mentioned the blog, uh, yes. That seems like probably the appropriate place in your API documentation.

[01:43:28] Simone Carletti: Absolutely. Absolutely. So you can keep an eye on the blog. Uh, we post their updates and all of it. Then we on the, on the website by the way, we also have a page that we rolled out, I believe a couple of weeks ago where we are listing all the various podcasts and events where we're been talking through.

[01:43:46] Simone Carletti: So this one will definitely end up there because we, we found that they are a great way, I don't have to tell you that, but they are a great way for, for people to know more about the insights, uh, of it. Pretty much [01:44:00] similar to conferences, right? Just like a different mechanism. But I love to know more about what it's happening inside at companies and just like learn about real world stories rather than just like reading a book that sometimes like tells things that in interior are great, but in practice are impossible or very hard to manage, right?

[01:44:21] Simone Carletti: And so we're being. Having the owner of being president at multiple podcasts. And so we have this page where we list all the conversations and we try to keep up to date with all the things that are happening. And also, um, you know, the, the conferences where we have been speaking also on the blog, we, we generally list them, or in the social media, we link to the various videos and recording.

[01:44:44] Simone Carletti: So that's definitely where you can start from. I'm not gonna give my personal, uh, side because it hasn't been updated in years. I've been dedicating most of my energies in, in the simple actually. So, uh, many of the things that you can read on blog actually comes from things that we normally [01:45:00] post on, maybe on my side, but I'm, I'm happy to share that in the context of and simple.

[01:45:05] Robby Russell: Well, I appreciate that. Well Simone, thank you so much for stopping by to talk shop with us On Rails. I think we've, we've covered a lot of ground today, so I really appreciate that. Hope to talk to you again at some point in the future as well for.

[01:45:17] Simone Carletti: Thank you very much, Robby. Thank you very much for, for having me, and, uh, talk to you soon.

[01:45:21] Robby Russell: Hopefully, we'll, we'll bump into each other at a conference sometime, maybe the next year or so.

[01:45:24] Simone Carletti: Absolutely. I hope so.

[01:45:25] Robby Russell: All right. Cheers.

[01:45:30] Robby Russell: That's it for this episode of On Rails. This podcast is produced by the Rails Foundation with support from its core and contributing members. If you enjoyed the ride, leave a quick review on Apple Podcasts, Spotify, or YouTube. It helps more folks find the show. Again, I'm Robby Russell. Thanks for riding along.

[01:45:47] Robby Russell: See you next time.