[00:00:04.10] - Robby
Welcome to On Rails, the podcast where we dig into the technical decisions behind building and maintaining production Ruby on Rails apps. I'm your host, Robby Russell. And I run Planet Argon, and for over 21 years, we've helped teams maintain and evolve their long-lived Rails apps. So, I tend to approach these conversations through that lens. In this episode, I'm joined by Hussain Akram, an engineering manager at DonorBox, and Antonio Carlos de Souza, DonorBox's chief architect. DonorBox helps nonprofit organizations raise money online and in person, and behind that platform is a Rails application that has been evolving for more than a decade. We'll dig into how their team keeps the application relatively simple while handling serious traffic, how they think about Rails conventions and code design, and some of the ways that they're pulling older parts of their application towards patterns they want the rest of their team to follow. Hussain joins us from Virginia and A.C. joins us from Brazil. All right, check for your belongings. All aboard. Hussain and AC, welcome to On Rails.

[00:01:00.06] - Hussain
Hi, Robby. Thanks for having us.

[00:01:02.20] - AC
Hey, thanks for having me.

[00:01:05.03] - Robby
So let's get things off to start. I'm gonna start with you, Hussain, first. What keeps you on Rails?

[00:01:11.04] - Hussain
I would say typical conventions over configuration stuff, because I started working with Rails when I was in my 3rd or 4th semester in my bachelor's. So it was very easy for me to just follow the conventions rather than going deeper into the stuff, because I was still studying the other courses in parallel, like object-oriented programming, data structures, assembly, C language. So it was so easy for me to build stuff so quicker for my term projects. So that's where I fall in love with this framework and kept using it. It's so easy for me to solve hard problems rather than solving the infrastructure problems, if that makes sense. So if I have some problem, I don't have to think about how I'm gonna go about it infrastructure-wise. So it's just having a mental model and just solving it using Ruby code, which is so much easy to read and write, you know. So yeah, those, those would be the most— Prominent things that I love about Rails and that keeps me using it.

[00:02:17.08] - Robby
Going through the educational process of that and learning to be a software engineer, did you know you wanted to primarily work in the web space or were you kind of just interested in software engineering in general?

[00:02:27.19] - Hussain
Yeah, I was very picky on selecting my first job after my bachelor's. So I searched around what are the companies that are using Rails and the guy who interviewed me was super impressed. by the knowledge that I already have about Rails. And it was smooth experience for onboarding because I already knew the framework. So training was like 2 weeks and then I just jumped right into the project. Yeah.

[00:02:53.08] - Robby
Wow. And just to provide some context for how long ago was that? Like what version of Rails do you recall that was?

[00:03:02.17] - Hussain
It was Rails 3 or 4. It was in 2018 when I graduated. And I got the job even before my last semester, so, mid-2018, yeah.

[00:03:14.13] - Robby
That's great. It's great to hear people be able to graduate and then land a job, getting to use Ruby on Rails right out of school. What about you, AC? What keeps you on Rails?

[00:03:26.11] - AC
By far, the convention. I was a little bit older than Hussain, so, I started working in 2002. And at the time, I was working with Java and for the web. I really enjoyed those information systems in the grad school. I was more, like, trying to get what I can learn. And one thing that always struggles with me was the rollout process of non-web applications. And I was always wondering how people do that. So, at some point in my life, I got to know about the web and how it works. So, it's kind of a— I got in love. And what happened is I was looking around on things to use for the web. I worked with Flash. I don't know if you—

[00:04:22.13] - Robby
Oh, yeah, yeah.

[00:04:23.00] - AC
Ask your parents what it means. I used Java, I used Struts in its first version. So, it's a long time. And one thing that happens in the Java world is that we have infinite discussions about how to build the same thing. The most extreme and anecdotal one I got was when we were discussing which logging framework we should be using. The problem for this conversation is, well, there are 3 main competitors in the Java landscape at that time, Commons Logging, Log4j, and something else. I think it's the default logging from Java itself. The thing is, you need all of them because you have dependencies, and those dependencies will depend on them.

[00:05:21.12] - Robby
They will pick.

[00:05:22.11] - AC
Even the discussion is pointless. And what happened is, in 2009, I guess, I was looking for a platform that I don't have those kinds of pointless discussions so that I can actually do something. And that's when I met Rails. At that point, I was learning about 2 different things that I don't know if you ever heard about it, the TurboGears and Django, both from Python. And I learned about Rails. And guess what? They have a ton of conventions. Like, you don't need to discuss that anymore.

[00:06:08.18] - Hussain
Yeah.

[00:06:09.18] - AC
It's kind of a blessing of having those conversations already settled. For this particular environment where I work. So, it's more like a, yeah, I finally found it, you know? And I keep on that since then.

[00:06:30.11] - Robby
That's great. I think about, you know, that, like, AC, my story is probably more similar to yours. I wasn't doing much Java at the time, but, like, there was a lot of these other, it felt like every project, it was like, all right, well, what are we doing on this project? what do we wanna try this time? Because maybe we weren't happy with the choices the last time, or someone was super curious, or like, or you didn't work on a project that someone else built, you're like, what decisions did they make? And it was always trying to put together this puzzle of like, all right, well, okay, kind of understand how these things work together. And then like, next time I start a new project, I'm gonna do it this way. And like, and it felt like every project was an experiment a little bit. And in tastes and Absolutely.

[00:07:10.22] - AC
And I don't know if you got this as well, but in 2008-ish, the Java world is getting a lot of the design patterns debates. So, instead of reading the design patterns and understanding the concept of, well, if you have this problem, you can implement the solution, and these are the cons, for using that solution. People jump directly to the UML models and said, well, we need to implement that. You don't even have this problem. Why are we doing that to you?

[00:07:54.01] - Robby
That's wild. Do you think you both generally agree on what good Rails looks like? When you're, when it comes to looking at, I don't know how many different applications you've each worked on, and we'll talk in a moment about DonorBox as well, but just like in terms of like knowing that you both kind of like found Rails in very different paths. And, you know, AC, you come from this background of like working with different platforms and for a couple, several years, not that you weren't, Hussain, but it just like in that you were in that, in the, I'm air quoting the real world for audio listeners here of like, you're working on different projects and you're like, okay. Was there also a moment of, like, once you started working with Ruby on Rails, you saw, like, it removed some of the decisions, but then, as you work across different Ruby on Rails codebases, are you getting a sense of, like, oh, this is, like, a good approach to building and scaling a Ruby on Rails application versus what doesn't look like a good approach to that with Ruby on Rails?

[00:08:53.13] - AC
Rails builds you, unless you're not paying attention at all, what you're doing. Rails builds in you a sense of familiarity. So, when you jump from one application to another, even in different teams or even different companies, you kind of can guess where things are for the shape you expect them to be. And this feeling kind of, it's what guides me to the, that's a good approach to using Rails, that's Weird approach. I've worked on companies that they are constantly fighting Rails. As I got to a point, I'd ask them, why exactly did you pick Rails again? Because every single decision Rails has, they try a way to change it. I don't know if people get bored because, well, you know, like, you don't actually have those problems anymore. So, you just need to actually deliver something, or if it's something like they keep saying that they need to build the application and has this feeling of they built it, like they took the decisions.

[00:10:13.02] - Robby
People problem, maybe, or jealous. Who knows? I get that sense.

[00:10:18.18] - AC
But that's the amazing thing. This problem only exists because Rails allows you to actually deliver at the end of the day. So, even after that, you end up still delivering somehow.

[00:10:35.17] - Robby
What about you, Hussain?

[00:10:37.09] - Hussain
I would just say, if you're using Rails, I think, try not to reinvent the wheel every time you solve a problem, and just go around and see the community, talk to the person who are using Rails, or just go search online and see how people solve the issue. Most probably 99%, this problem, specific problem is already solved by a simplest way. So the codebases I have struggled with, this is the issue that I saw in the codebase. They were just reinventing the stuff and it just made codebase so complex, which is hard to, if you just using the conventions that Rails provides. Yeah, it's very hard to, you know, diverge from that line. if you're using Rails, because everything is in front of you. You just need to put the pieces in the right place and you're done. So yeah, that's the main point I wanted to mention here.

[00:11:33.16] - Robby
I'm sure we'll probably get to dig into some more, maybe some tangible examples in a little bit, but I think to provide some context for our listeners, Hussain, maybe you can give us a kind of a quick high-level tour of DonorBox from an engineering perspective. And what does the Rails application actually have to support day in and day out at this point in time?

[00:11:52.23] - Hussain
Sure. So our engineering team is divided into 5 or 6 teams, and those teams surround and have responsibilities on particular aspect of service that we provide to nonprofit organizations. For example, my team is in-person fundraising team, which basically includes everything that has to do with in-person. It can be live kiosk, right on my back. It can be ticketing events, text-to-give, and all that stuff. Other teams are online fundraising team. They handle online fundraising form, our main donation form. And then we have org experience team that handles and provides best tools for us, for other teams to use. And then we have platform team, integrations team for Salesforce and HubSpot and all other stuff. So this is how we have divided our teams, and each team has one project manager, designer, and QA.

[00:12:49.04] - Robby
So in-person literally means being at a real live event for people. So online, we can talk about DonorBox, and in the intro I provided a little bit of an overview of what DonorBox is, but I think about, so you've got online donations, I think people can wrap their head around it, but then you've got, as you pointed it behind you, for our listeners, there's a kiosk in the back corner behind Hussain, and there's in his home office there. So it's facilitating that. Does that have to do anything, maybe get into the weeds of some of the fun details, but I've worked on some event management tools in the past, and with some of these different clients, some of those require having to work around maybe unreliable internet. And at times in those spaces, do you have to do any sort of offline processing or anything like that, or at this point?

[00:13:35.18] - Hussain
Yes. We also have a similar issue where, because kiosks are being used by mosques and churches across the US, And some of the places, they don't have a stable connection. So we ran into this, and Stripe does provide a way to process transactions in an offline mode. So our kiosk also handles those transactions in an offline mode. So basically, we use devices' local storage, store them, and then when the device gets online, we start processing those transactions and just send them to the server. Nice.

[00:14:12.20] - Robby
And you mentioned like 5 or 6 teams approximately. How many people, I don't know if you reminded, did you say that? How many, approximately how many engineers are part of the team right now?

[00:14:21.21] - Hussain
AC, I think 40 to 50.

[00:14:25.07] - AC
50, I guess.

[00:14:26.11] - Hussain
Yeah. I didn't mention the support team. Amazing support team. They serve our customers very closely. So yeah.

[00:14:35.01] - Robby
What was that? That was the support team?

[00:14:36.23] - Hussain
Yeah, so support agents, they are also engineers, uh, and they kind of do, um, high-level debugging on stuff. So if they can understand, fix a bug, and then just push it without disturbing the, you know, uh, relative team. Um, but yeah, if, if the issue is so big or needs, uh, team collaboration, then obviously they escalate it to the respective team.

[00:15:02.22] - Robby
And is this primarily a monolith Rails app, or is this broken up into multiple things, or is that a touchy topic?

[00:15:10.21] - Hussain
Overall, it's a monolith app, but we do have some small repos. For example, our kiosk has code on Android and iOS repos. And then our integrations team, they have their own repo because they have to communicate with Salesforce and all the other stuff. So they have a very simple repo, which communicates through the main monolith using our internal API.

[00:15:34.19] - AC
Yeah. You can think about this as, we have this mothership and some support ships around it. The live kiosk, we have the integrations application that is also separated. So, it's essentially a background job for different integrations. This thing lives aside.

[00:15:57.20] - Robby
And how do you think about, maybe, AC, on your side, when you zoom out across the different product areas, what has this allowed you to remain primarily in the one Rails application? And how do you think about when you're going to carve off, say, something like this integrations part of it?

[00:16:15.06] - AC
It's essentially, can I delete this whole application? If some business changes, can I delete this whole application without anyone noticing? Essentially, this is the question.

[00:16:33.02] - Robby
Have you ever needed to delete anything like that?

[00:16:36.09] - AC
In a way, yes. It's not common behavior. So, the default answer for this is no. If that happens, we probably need to delete the team as well. because it'll be a huge business change. And what happened is, this thing, especially the integration app, for example, it's essentially a background job. So, something happens in the platform, and this something needs to be communicated to a third-party tool that our customers log in there, to do something. There's no UX, essentially, besides mapping what happens on DonorBox to a third-party tool integration. Those are the cases that we think about, oh, okay, maybe this can live in a different application because DonorBox has a ton of PCI, SOC, and whatever certificates we require. It's required from us. And because of those certifications, our process needs some double checks that prevent us from working faster. The easier example we have is for the— it's very easy to explain to an auditor that we create a pull request. We require someone else that's not the author of the pull request to approve that pull request before merging.

[00:18:13.10] - Hussain
Okay.

[00:18:15.10] - AC
That's unbelievably easy to explain to the auditor. If I tell the auditor that, well, we do have this homebrew integration process that runs a ton of checks against your pull request, even before you share it with someone else, it requires the auditor to understand what you're talking about. It makes the audit process way more complicated or longer. balance between wasting time in audit, wasting time on the process daily. We found out during our time that the auditor is way harder than just ask for someone that is already working with you to approve this pull request.

[00:19:03.01] - Robby
That's interesting. So, you're needing— there are some scenarios where you're making architectural choices just to make it Simpler to work through the auditing process?

[00:19:15.12] - AC
Yes. Donorbox has this problem of not following the default and expected path for a startup. Instead of breaking and vanishing in the sand of time, we're thriving. So, we need to accommodate old habits into a new set of requirements that was unexpected.

[00:19:38.00] - Robby
How has Rails been part of that growth story? Do you feel like Rails has helped enable you, or has it been able to stay out of the way, so to speak? So as you're— I know when we were prepping for this conversation, like, it started off with one thing and you've been adding all these other approaches to fundraising and handling donations and things like that through your platform. Where has Rails been really beneficial to you there? Leave it over to you, AC, to start with that.

[00:20:05.06] - AC
We fall back to the story lessons that it was usually Qt on Donorbox. I've been at Donorbox since 2019. So, I end up getting a lot of context of how things happened in the past. What happened is Donorbox was built by its own founder. He learned Rails because it's sexy at the time. And that's what the cool kids are using. The thing is, he's not a programmer by trade. He became a programmer by necessity before Claude, which is a big difference. And the thing is, Rails has a learning path, a clear learning path for someone that never done a web application before, but he's willing to learn. And he walked his path. So, he managed to build a company through essentially Upwork tasks that he managed himself while he wasn't able to build the stuff. When I joined the company, the founder and owner of the company is still doing product deployments. So, Rails literally enabled the company to be built by someone passionate about what they're trying to achieve and who wants to actually do something that is not only useful, but secure and manageable by one person.

[00:21:48.05] - Robby
That's great. So, the The folks building the application weren't historically super experienced software engineers, were able to build the initial versions, get this thing bootstrapped, picked Ruby on Rails because that was what the cool kids were using. And, or at least the other people that they saw were being successful is probably another way to frame that. Like, oh, those people are doing that. I'll just, I'll just follow in their footsteps. I meet a lot of people over the years who were like musicians. Like I was a musician, you know, a musician myself as well. But I think about like, People that didn't have a super deep resume of like architecture, planning, building startups necessarily, like, but they had an idea, looked and saw what people were using, picked Ruby on Rails off the shelf and be like, all right, I'll, I'll pick this up as I go. And then, you know, not every product like that ends up being successful. You know, that, that's okay. But then the journey was that eventually they realized like it was kind of outgrowing them and like, maybe we should hire some people that have even more experience who can continue moving this thing forward, improve it, make it more stable, and help it scale with, as a, as an organization, which is, I think, a great testament to Ruby on Rails that it can do that.

[00:22:56.07] - Robby
Was that an interesting transition? You know, we talked a little bit about what was good Rails or not. And I'm not saying you need to like complain about like what the state of the codebase, because I'm sure every codebase has their problems. And if it's someone's first major mature Rails app that they're building, I'm sure there's going to be some like things you maybe maybe not wanna be super proud of necessarily in retrospect, but you learn from that. And I think that's okay too. But what were some of the types of things that you were, like, coming in and being like, okay, this is how we're gonna strategize on iterating from this point forward, that you can recall?

[00:23:30.22] - AC
Well, a couple of them. Hussain actually has a very good story on that.

[00:23:36.08] - Hussain
So, basically, most of the codebases that I have interacted, they were built a rapid fashion because the owner or co-founder wanted to solve the problem as soon as possible. And Rails gives you that power to, you know, just build and worry about the optimizations and all the other stuff later. So one of the fine examples was our events product, which was a kind of experimental feature at that time. So our founder wanted to give it a try because he heard a lot of nonprofit organizations, they want to organize events for their main fundraisers. And this also relates to our fundraising goal and providing services around that area. So Events product was built in a very rapid fashion, but it was, I think, built within 3 or 4 months. And that came with this cost of doing refactoring and code debt and everything. But still, we we were able to solve that issue gradually because we knew that the events product is serving the nonprofit very well. It's generating a good amount of revenue for the company, and now we can invest on the optimizations. But still, I would say it's a, like, positive side of Rails that it still gave you tools to build something cool in a very small amount of time before, like, a gala season, and then it performed very well, and then we built like a whole team around this.

[00:25:03.17] - Hussain
So, you know, now we have in-person team of like 4 or 5 developers which are managing other stuff along with events. So this happened with the one product, and I have seen it even with the main projects which was built in a very fast fashion, and then the owner or the founder wanted to invest in optimizations later on. I don't think there's any shame That's great.

[00:25:29.16] - Robby
And then, out of curiosity, do you have any approximate stats you can share with us on how many donations are getting processed on an annual or maybe even daily basis, just to provide some context with listeners?

[00:25:43.05] - AC
Sure. I'll start from the edge until the donations. So, 2 very interesting numbers. We processed in the last 30 days, 188 million requests.

[00:26:00.10] - Robby
Okay.

[00:26:01.22] - AC
Filter out any assets.

[00:26:05.00] - Robby
No favicon?

[00:26:07.04] - AC
Yes. No favicon, no JavaScript, no CSS, no images. That's our volume of requests in a month.

[00:26:15.18] - Hussain
Okay.

[00:26:16.19] - Robby
So about 120, almost 120 million Yes. 30 days. Okay. How big of a platform are we hosting this on? Is this, like, spread out across several AWS data centers or...?

[00:26:29.15] - AC
Yeah, it's kind of sad because we only— we use Heroku for the same reason we use Rails.

[00:26:36.17] - Robby
Conventions.

[00:26:38.08] - AC
They're already there and working. You know the story about Heroku being very expensive and stuff, but Well, we don't have a dedicated infrastructure team to keep that thing running. In the last 60 days, we have a spike of 200 donations, 204 donations made in 1 minute on July 15th.

[00:27:04.02] - Hussain
Hmm.

[00:27:05.02] - AC
And no one knows. Literally, no one's noticed. It's not a particular day for anyone in the company. It's just happened.

[00:27:15.00] - Hussain
Yeah, AC keeps sharing this information in our Slack channel. There would be, like, big spikes and nobody noticed, and he will show the Cloudflare stats that while you were sleeping, this is what happened on our server.

[00:27:30.01] - AC
Exactly.

[00:27:31.03] - Robby
So you've got your Heroku Dynos, and then you've mentioned background jobs. What does that actual request lifecycle look like? So maybe from the web request, someone makes a donation, is that then Immediately hitting a Stripe thing or something? Or is that— are you also integrating with several different payment processors as well to make that work? I mean, I'm not sure where your target customer base is. Is that very regional? And like, do you have to have set up separate things in different countries because not everybody has a credit card everywhere? Or is that primarily— are you primarily focused in environments? I only bring that up because I know that some people might integrate with Stripe in some places, but Not everybody has a credit card in all places, and it's not as accessible, and they might need to use PayPal or some other sort of payment methods, or I'm assuming that kind of complexity might show up in this kind of platform, right?

[00:28:19.19] - AC
Absolutely. So, we do have both PayPal and a partner called The Giving Block. The Giving Block helps us with the stock donations and cryptocurrencies. And of course, Stripe has more than credit card, like, way more than that.

[00:28:39.06] - Robby
Yeah, yeah. ACH, all that.

[00:28:41.18] - AC
And what it does is our majority of our donations came through the US, but we have around 40% outside of the US. Europe, United Kingdom, India, We have a lot of donations. Besides the majority of donations came through the United States, we do have a global presence, and Europe being the second most, and their own payment methods, like SEPA, and it's not ACH, they call it XDebit or something close to that.

[00:29:22.22] - Robby
I see. You've worked hard to build your Rails app, but have you thought about what happens after you call .save, an email gets sent, a job gets queued, 3 other records update, and somewhere a webhook fires for reasons nobody remembers. That's why there's Callback Insurance. Callback Insurance protects your team from unexpected side effects caused by before_save, after_commit, and the dreaded after_save inside a concern. So when someone asks, why did changing their address cancel their subscription? You can rest easy knowing you're covered. Callback Insurance. Because you shouldn't have to know what happens after save. Coverage does not include recursive callbacks, callbacks added by gems, or anything described in the code as temporary. Existing condition may apply. Back to that other part of my question, there was, I was wondering around, so if someone makes a web request, how does that, like, that lifecycle kind of look like right now? Is that from there? And then maybe we can talk about from Hussain, your perspective, like what happens on the kiosk? You mentioned like things might be later synced. When does that all kind of connect to from both sides?

[00:30:22.06] - Hussain
Sure.

[00:30:22.22] - AC
So, the request is started on Cloudflare because we have it in front of us. And Cloudflare delivers the form itself. It's either an embedded form or the hosted forms that you have a kind of a webpage for the forms. Most of the time, it's cached. By the way, those are not counted in the previous numbers. The user fulfilled the information, sent the request for making the payment. From the user perspective, for the user browser, we request Stripe for a tokenized— a token of the payment method. And this is the thing that lands on DonorBox infrastructure. So we essentially only have the tokens from Stripe or PayPal. From this moment on, it's essentially Heroku writing down in a Postgres, and that's it. It can't be more simple than that. We do have background jobs for the things that happen after the donation. We try to create the donation as one insert in one table. It makes it faster because it's less locks to create during this process. And what happens is, after the donation is created, We create background jobs to extract from the donation information like the donor. Do we already have this donor in our database, or is it a new one?

[00:31:50.15] - AC
So, if it's a new one, create it. Those are what those background jobs do.

[00:31:57.02] - Robby
I see. That's interesting. That feels like if you were starting to build the app up kind of incrementally, you might not think to do it that way initially, right? If someone were to say, hey, we're gonna be, we're gonna be building a thing and we're gonna like, I don't know, maybe you have one big table with a lot of columns and then you insert that. And that feels a little atypical from like, uh, the Rails way. You might have, you might try to do that stuff in, in, in your initial create statement, and then it might go off and create a couple different objects or something like that, save other data in different tables. So you have this relational thing. Was that something that you were just seeing as a scaling thing? You know, like that's where we're gonna iterate from and we evolved it that way, or was that an upfront plan?

[00:32:36.20] - Hussain
Yeah.

[00:32:36.21] - Robby
Because I think that's where people start to, maybe they prematurely do that and think that, like, well, we're gonna deviate from Rails conventions in that way. But how do you wrap your head around that part of things? If anyone listened, you'd be like, well, that doesn't smell like a typical Rails thing, but maybe that actually is a really good use case if there's a high velocity of throughput.

[00:32:55.21] - AC
The donation is our hottest path by far. And what happened is the application was built having in mind one individual making one donation to an organization. Through time, what happened is this concept of one individual, we started to get stretched for, yeah, if it's a company making a donation, does it count as an individual? It's still a donor. The systems grew around this concept of one person making one donation. And what we measured way later on, like, years before that, is that keeping things this way helped us on the scale. And by the way, we're not Shopify, okay? So, we're not—

[00:33:52.03] - Robby
Yeah, yeah, yeah.

[00:33:52.23] - AC
Scale for human beings, for the rest of us. I was in way too many discussions where people were starting by Rails doesn't scale. And it was like, dude, you have more microservices in your application than customers. What are you talking about? Those conversations are really hard. And for the majority of cases, Rails scales really well and you don't need to do anything. You just need to pay attention. So, we got an issue on scale because we use touch everywhere. Like, you create a donation, we start touching the whole database. Why are you doing that? Because we are trying to update the counter caches.

[00:34:46.12] - Robby
Yeah.

[00:34:46.16] - AC
What is the problem? When you have a really hot path that needs to keep the database inserting one donation after the other, if you stack up a lot of updates, your database is starting to complain. We have a lot of auto-DDoS because we created this problem for ourselves.

[00:35:10.05] - Hussain
Mm-hmm.

[00:35:10.18] - AC
When you have, I don't know, 100 customers, that's fine. You don't even notice. You have 1,000 customers, you see some spikes if you're really paying attention in your logs. Donorbox right now have more than 30,000 in one database for writing.

[00:35:31.10] - Robby
Oh, wow. This is interesting. I think about that path and I think maybe for people listening, I think you mentioned you have this really hot path where you need to keep the inserts As streamlined as possible. Maybe you have this benefit of like, maybe if we bring up something like a Shopify where a lot of things have to happen because you have to double-check, is there inventory, you know, of the product, right? And maybe in a donation thing, it's like, I don't know if there's like campaign type things where there's some specific rules. You're like, well, we can't have this many donations of this tier. Or if it's just like, no, no, the more donations, the better. So let's, let's just make sure money can come in. as smooth as possible, and we'll make sure that we audit this thing and we can track where it came from so we can properly map this later. And, like, the point is, make it easy to get money to the organizations you wanna help out. And so, like, we'll figure the rest of the stuff in the post-processing phase of that.

[00:36:24.21] - AC
Yeah, we are in the, the more the donations, the better, but we do have very specific campaigns, like political ones, that require you Some information that maybe is not there. And those are more tricky in the sense of, well, we need to run a validation before actually accepting the donation and report back to the donor. Those are the most complicated cases for this.

[00:36:52.12] - Hussain
And part of this validation is already being done on the organization level. If the organization is valid and they are compliant with 501 nonprofit organization. So it's easy to assume that their donors will also be legit, but we do have fraud prevention in place that's working fine. It can easily block donors who are just, you know, messing with the donation form. There's not much of validation that we need to do. So it's just very simple. You send the JSON parameters, enqueue the donation job, and yeah, just write in a couple of database tables. Yeah.

[00:37:27.00] - Robby
What are you all using for background job processing at this point?

[00:37:30.13] - AC
We're using Sidekiq and Redis for now. We do have plans to migrate to the full Rails—

[00:37:39.21] - Robby
Solid Queue?

[00:37:40.20] - AC
Yes, Solid Queue. Thanks. We have 2 tests with Solid Queue on the Donorbox right now, one for the communication team, and we do have one test in the integration team. So, essentially, the idea is we have those support ships around the mothership, and we try there first.

[00:38:05.11] - Robby
While you're thinking about and running those tests right now, is there anything that you're encountering when it comes to, like, using Redis or Sidekiq that's, like, kind of, like, well, is it that you're just, like, let's just get on board with what the general Rails conventions, like, this is what's getting shipped by default, or is there anything that you're, like, limitations or some tooling that you kind of aren't seeing there, or benefits that you're looking at?

[00:38:27.23] - AC
One of the main benefits is space for the jobs. The thing that happens, keep nudging your head on that thing, like, hey, think about next time you need to do this. And increasing Redis is, okay, in this site, especially on Heroku, is really expensive. While having a proper database, and I'm not bashing on Redis here, proper database, I mean, a relational database, that's all. I'm not judging anyone. The thing is, having a relational database to do that is way easier, especially when you're starting to query the jobs that run. Mm.

[00:39:13.12] - Robby
Yeah.

[00:39:14.04] - Hussain
One of the other pain points was the job continuations. I think Solid Queue solves that issue. So yeah, that's one motivation, because currently if we restart the server, make a new deployment, all the jobs have to stop and rerun again. There are very few jobs that take a lot of time, but most of the jobs, they get processed within a second or 2. Some jobs get stuck, and then we have an amazing guy as an infrastructure person. So he solves these kind of issues and notifies the team. His name is Akuyas.

[00:39:48.16] - Robby
Thank you for your service.

[00:39:51.02] - Hussain
Yeah, thank you. So yeah, he recently mentioned that we also downgraded our Sidekiq dyno from using 15 gigs of RAM to 2.5 gigs of RAM because we weren't utilizing anything on that.

[00:40:08.05] - Robby
Is that one of those types of things where just like somebody slid the thing up at one point, like, this seems safe, nobody complained about it, then someone was able to start gradually moving that down and be like, let's see what breaks, or, or they felt like they were confident enough to slide it way down to 2.5 and be like, no, no, based off of what we're seeing, you know, you mentioned you don't have like a dedicated DevOps team necessarily there, but it sounds like you do have at least one person there that maybe knows a little bit more and solves the day a little bit here and there. So, what does that workflow look like?

[00:40:37.08] - AC
We'll talk about Zacharias. He's our server shepherd. Essentially, what happened is Zacharias is really, really good on infrastructure. He likes this. So, what happened is that he kept a very close eye on what's happening on the database. Do you have any queries running for more than 30 seconds? Do you have queries, I don't know, not hitting the caches? Do you have queries that load half of the database to the memory before trying to do something? He does that because he actually likes it.

[00:41:20.13] - Hussain
Yeah.

[00:41:20.18] - AC
So, what happened is the whole infrastructure, run by its own. What Zacarias does is making sure we're not overreaching of these capabilities and we can actually be using— I have the number for the last week, for the last month of our median usage of web dynos was 2.5.

[00:41:46.23] - Robby
Oh, okay.

[00:41:47.17] - AC
For the majority of the month, We go from 2 dynos, because I'm too paranoid to have only 1, to 3 dynos, and back to 2. Most of the time, seeing the charts, Heroku increases the dynos when you make a production deployment. So, you made a production deployment, all the cache is invalidated, Heroku thinks you're getting a huge spike of requests, they add you 1 more dyno. 15 minutes later, they're like, oh, okay, sorry, overreacting. They get back to 2. That's how we got to 2.5 dynos in median.

[00:42:24.10] - Robby
So your deploys are triggering that more than anything?

[00:42:27.18] - Hussain
Yeah.

[00:42:29.01] - Robby
When you think about your hosting infrastructure there, how are you— does your team— what's your deployment process look like right now? Are you, before you deploy something into production, are you doing a lot of things in a QA environment? Do you have a dedicated team that are testing? You mentioned earlier, I think it was AC who talked about, Your platform allows your customers to like embed like a form in their website. So maybe like a nonprofit, maybe you can put an embedded form so they can kind of, it feels like it's part of their website, but it's an embed. Maybe it's, I don't know, are those like iframes? Are these JS snippets? Maybe a couple of different versions of those that you might be doing. You can't really predict how much traffic those places are going to get, right? And then necessarily, because, and that's, that's great that you're not having to think about it that much, but I'm curious about. What's that QA workflow look like when you're dealing with like these forms elsewhere? And like, are you using like Heroku preview environments or something like that for that? Or is relying a lot of just internal testing there?

[00:43:24.23] - Robby
How do things get shipped from a new feature or a new button getting added and getting shipped to production?

[00:43:30.10] - AC
Hussain, that's all yours.

[00:43:32.14] - Robby
Yeah.

[00:43:33.06] - Hussain
So our deployment process is very simple and classic in terms of we Raise a PR, and then we have GR columns. Every team has their own Heroku review app where you can deploy and notify your QA to test it out. And then we have really good, close to 100, I think 100% code coverage. And some of the tests that we have written are system tests that actually test your embed versions, different type of configurations and all that stuff. And QA does another excellent job. to retest it, re-embed it in different like WordPress website or some other places to see if it all works well. They are not like frequent moments when we make changes on our donation form. It happens like very rarely, like when we are like pushing major features to the donation form. For example, we relaunched a new version of donation form which was OmniGive, which needed thorough testing from QAs and other teams to make sure that everything works fine on different versions of embedded, you know, versions. After the QA is done, we do code reviews. Every team is, you know, is welcome to review the code of their peers.

[00:44:49.20] - Hussain
And then we have one command which runs all the tests and pushes the results to GitHub, all green, you run another command, deploy, similar to Capistrano that we used to do. So it's like internal shell command that we just run and it deploys to our main staging server, which is used for demo purposes and on our production. Very quick, very simple, and boring.

[00:45:15.15] - Robby
You mentioned keeping it kind of boring. What do you think that allows you to focus? I think some teams are struggling, or maybe not struggling, but they're kind of weighing up like how much they're relying on their local CI versus like doing like all the CI and say on GitHub or stuff like that and GitHub Actions or GitHub Actions has had its reliability issues recently. And hopefully by the time this airs, they'll have solved a lot of that, but you know, it is what it is. And are there any other interesting tooling that your team has built internally? You feel like as a, as a result that because you've kind of kept your infrastructure relatively Boring? Yeah.

[00:45:54.10] - Hussain
So, we were using CircleCI before using LocalCI. With CircleCI, we used to run into this issue where the code coverage would drop if the 2 PRs get merged simultaneously. And when the code is accumulated on the main branch, the coverage would drop on infractions. And people would be fighting with each other, like, on Slack, that I'm unable to deploy the next PR. During the same time, DHH posted this post that he also shifted onto local CI. So that was a big blessing for us to— an excellent idea for us to use that. And then we developed this Rake gem, which basically runs the linting, Brakeman, runs the tests and everything, and pushes the results to GitHub PR. Now only one PR can be merged at one time. And we didn't have any issue related to code coverage drop after that.

[00:46:48.02] - Robby
Interesting.

[00:46:49.00] - AC
I have a couple of numbers about this that maybe sound interesting. We keep track of the DORA metrics in Donorbox. DORA metrics is the effort from Google for measuring the DevOps teams inside Google. Because there are 4 metrics and they are depending on each other, there are less Capable of being cheated or gamified. One of the metrics is the deployment frequency. Last week, we were around 10 deployments a day. Considering 15 developers, it's kind of nice. This integration process that Hussain mentioned, I have a slightly different view on that because I was guilty for introducing this process. What we had before, as Hussain already explained it, was outsourced CI, and it took around 1 hour to run the tests and only run the tests. And I have a different view because I was the person pushing to run the tests locally so that the person doing the thing experiences the problems they are creating. Another way, less sad way of seeing this, is that I'm giving you back the feedback you lost when you put that on CircleCI. And this integration steps is pretty funny because they are a little bit more than just run the tests.

[00:48:22.21] - AC
We run the static analysis, so enforcing the Ruby on Rails directory compliance, So, that's how we can have 15 people without exploding different code designs. We run RuboCop, Brakeman. We make sure that feature branch is updated, bundle style, because, you know, things happen. And then we test the migrations. We drop the developer's database and recreate it, running the migrations from scratch. Why doing that instead of just using setup? So that you check if the migrations are actually working in the process of migrating from a previous version to a new one. Funny thing is, the first version did exactly that. They dropped the database and recreated it. The first thing people complained was about, well, you dropped my database. I lost my test environment, my local test environment.

[00:49:28.21] - Hussain
Yeah, it happened a lot.

[00:49:30.08] - AC
My first question is, hey, hey, sir, do you have time to listen to our lord and savior fixtures? And it's pretty funny because people didn't understand that fixtures is not only about running tests, it's having a set of known situations, scenarios for a database. I could convince people that fixtures and fixtures load is a way better solution than every single person needs to learn about this new scenario and create the scenario locally. The problem is that I have this conversation like 10 years after the application was built. So, we kind of need to catch up fixing the fixtures.

[00:50:19.18] - Hussain
I would also admit that after doing it locally, it also motivated developers to reevaluate the written tests. Because initially, the tests were taking more than 20 minutes. And with that time, every developer, thanks to all the developers at DonorBox, every developer was kind of proactively working through the slow tests, and they were optimizing it, sharing it with peers, and doing it across the project. And now we reduced the time to half. Still, we are continuously working on improving the time to run the whole test suite. If there are any warnings or any deprecation warnings, developers, because you are running the test on your machine, so it's easier for you to just see and ping the relative developer, oh, can you check if there are deprecation warnings? Should we upgrade this gem or something like that, rather than just outsourcing to CI and just continuing on your work.

[00:51:16.09] - Robby
What's always interesting about hearing these, these types of stories is like, that's how we used to do it. And then, you know, we used to run it all locally. And then there was like, oh, great. GitHub provided us a way to run it on their servers, or we had CircleCI or Jenkins, or, and it was also another way to like be another backup thing. Like, well, We assume people are running their tests. We didn't have, I think, enough confidence at the time that our peers were actually running their tests before they would push a change up. And so, it would require the piece in, like, if you're doing, like, a PR, before we had PR requests, even, you know, before we had pull requests, when you get a patch or whatever, or whatever the process, I can't even remember exactly how we used to do it, but you would have to pull down the changes, you run your tests locally, and you were assuming that everybody was running through the steps.

[00:52:05.18] - Hussain
Right.

[00:52:06.03] - Robby
Before they deployed things. And that wasn't always true. That was, that wasn't always true. And so there were definitely times like, well, I just, we shipped this out really quickly because there was a quick bug and then that starts to degrade, that, that consistency kind of goes away. So it was nice to have a nice little green checkmark on GitHub to be like, well, this thing says it's, it's happening. We didn't break anything, hopefully. But then you're saying it was taking an hour to run your test and then maybe locally it was taking 20, 25, whatever minutes. And then maybe you've cut that down even more. 10 minutes or so, is that?

[00:52:36.02] - Hussain
Yeah, 8 minutes, 7 to 8 minutes.

[00:52:38.00] - Robby
8 minutes? All right. And you would have to wait an hour before for CI to do that. So, does that even mean that if you're running it, like, if you sent a PR for somebody, then you're waiting at least an hour before someone's gonna review it?

[00:52:50.11] - AC
That's exactly the problem. Because the tests are taking so long, they became a checkmark you add to the thing, not a developing tool. So, bringing back the CIs to the developers reminds the developers that, hey, guys, that's for you. The customers completely don't care about automated tests. This feature, the company, because I work with companies that I need to fight for writing automated tests, Yeah. I'm not that old, okay? I just happened to be there. And the thing is, people forgot that the automated tests are a tool for development and for developers. And more than that, they didn't see that tool as they see their Code editor, for example. And this is the main analogy I keep doing over and over. Like, dude, you have one tool that the company not only allows you to create, they demand you to create just to make your life easier. Writing tests doesn't make your PM's life easier. It doesn't make your manager's life easier. It makes your life easier because you are the person that's not waking up to fix a bug that ended up in production and no one had no idea how. We mentioned the test coverage earlier that we reached 100%.

[00:54:37.04] - AC
First, guess who pushed this number up? Of course, it was Zacharias because he is the person that, in the whole team, cared the most about those details. And the way you see the test coverage is the information is not the coverage itself, but what's not covered. So, 100% test coverage is more noise than signal. The signal is the list of things that if you change, you don't have any way to automatically check if that thing breaks. That's the way you're using this measurement. Not if you're reaching 100%, but if the list of things that breaks if you change something and you don't have an automated test for checking it out. And the reason for reaching this 100% is, well, I cannot guarantee the test will exercise the thing, but I can guarantee that One of the tests, pass it through that path. So, if even this is breaking, something's really odd in the tests.

[00:55:50.11] - Robby
I'm curious about when your team was putting more of an effort to, and kind of making everybody run their tests themselves locally and having to live with that and learn that, like, it's gonna blow away your database. So, you need your fixtures or your seeds or whatever your team is using locally to make that optimize that workflow. Did your team have to go through any training together or like, like, was there like, here, follow what AC is doing with this type of approach, that seems to be the right path? Or is there like workshops that you were doing to get everybody kind of on the same page? Or did everybody kind of pick it up over time and at their own pace? What would you replicate and what would you maybe do differently next time?

[00:56:29.18] - AC
I think you'd love to see the different perspectives on this. So Hussain, if you'd like to go yours. Please.

[00:56:36.07] - Hussain
At DonorBox, it was like getting everybody on board on this was very easy because we already built this shell tool on GitHub, shared it with everybody to share their opinions. If there were any suggestions, those were addressed. And there was just one single command that we asked. Rather than expecting CI to respond to you, you just run this command and then it will respond to it. Yes, there were some instances where it was frustrating because if there is another server running on another terminal, it would create a conflict and sometimes drop the database. But it doesn't happen every time you run the test. Now, mostly I don't encounter this issue at all. That's a signal that you need to fix something rather than just replacing the whole process. So that happened initially. Tests were slow. And with the time, developers were motivated. I think we wouldn't have this time without moving the test to local if we were just doing on CI. So now our tests are faster. We improved our coverage. And when I joined DonorBox, the coverage stayed, I think, at 86% for a while, but I saw the spike to 100% very quickly within, I think, 6 to 8 months when I Shifted from CI to local.

[00:57:56.09] - Hussain
So that's like a positive thing, but it came with some cost, you know? Yeah.

[00:58:01.04] - Robby
Tell me more about this one shell command. So you mentioned, is it just running a bunch of these different tools at that point, or what else is in there, basically? You mentioned Brakeman and RuboCop and things like that. Is it running all those types of things? Is it similar to that bin/ci script, but just like some other executables?

[00:58:19.06] - AC
Yes. We have that very creative name of integrate.Donorbox. And what happened is we have 12 different steps before you share your code. So, run RuboCop, they fix the things for you. You don't need to worry too much. We don't have crazy requirements on RuboCop. We're running the default Omakase RuboCop version right now.

[00:58:45.10] - Hussain
Okay.

[00:58:46.00] - AC
After a lot of dispute on what you should run. We run the tests, Rake task. We add a bunch of steps there and we keep evolving this task, but only when we actually have a requirement for that. We don't spend too much time on that. To create the first version, it takes some time to list the things you want to do, but it's a Rake task, very straightforward. Not fancy. The fancier thing you do here is requiring the GitHub app installed in your computer. It could be the CLI because we do the authentication to add a mark on the pull request that someone ran this and we use this as a requirement for the merge. For the process of doing that, I presented to the engineering managers, This idea. And my main argument is people are losing track of why you write automated tests. Some developers are getting way too close to see the tests as a checkmark for someone else's checklist. To run this test again in your machine, it reminds you that You are the responsible one for that, not somebody else. And what I had to do was first convince engineering managers of having faith and try to see what happens.

[01:00:28.18] - AC
But again, the test runs take 1 hour. It's really, really hard to convince someone of doing that. And I worked with the company management of requesting time for them to do that. So, if you're not there yet, in the, my test suite takes 1 hour to run, it's easier. If you get to this point, it's doable. You just need to commit yourself.

[01:00:57.05] - Robby
I appreciate that perspective. So, I wanna pivot the conversation a little bit. And one of the things I've been thinking about was in our prep conversation, I think, Hussain, you mentioned that You're in the process of moving parts of DonorBox from Slim templates back to using ERB. So why, out of curiosity, what's kind of motivating that to start off? And then maybe like, what's that process look like over on your end?

[01:01:24.08] - Hussain
Yeah. So this is also our kind of effort to go with pure Rails defaults. Also, we started seeing issues with Slim files related to formatting and everything. So we want our new developers, or when we scale our development team, to make it easier for them to just onboard them with pure Rails defaults, easy to convey that this is, this is how we work. So Slim files was just adding another complexity, and it was also slower than ERB, so there was no point of having this extra pretty layer on top of what we already have with Rails. That was the kind of main thought behind it, in my opinion. AC, do you want to add anything into it?

[01:02:11.04] - AC
It also was an effort from the engineering department perspective of, yeah, we're not saying that you should commit to Rails Default as a rhetorical commitment, But this is our efforts for making this commitment more tangible for everyone. If you need to change a Slim file, convert that to ERB first. And at some point, we could finally remove the Slim gem. Again, it's not about Slim itself, it's that on DonorBox, it wasn't necessary.

[01:02:50.19] - Robby
Were you around when they just started using Slim, or was that just a decision that was made very early on? And like, well, that was, was it just a choice? Well, that's what the cool kids are doing, so I'll just use Slim because that seems to be what my peers are using. Or I think about like that whole HTML templating abstractions that we've, we've gone through back and forth a couple of times. And I was always a little, personally, a little like, well, I don't, what are we doing here? Because I always felt like it was easy for, for me as someone that hires software engineers and front-end developers in particular, what we used to call front-end developers before As much of the JavaScript side of things came into play, but people that could take designs and make HTML and CSS out of that and be like, great. Well, now I, let's also then introduce, now you need to learn this Ruby code to generate the thing that you already knew how to work with. I always felt a little like, I felt like it was a developer solving a problem of them not wanting to write HTML, even though you needed to understand HTML to use it.

[01:03:41.13] - Robby
So it just felt like an extra layer of, not saying complexity, but it maybe it looked cleaner in your, when you're looking at it, I guess. Some people might be listening and be like, well, It's slower, but is it— maybe it actually looks cleaner to them. I don't know.

[01:03:55.19] - Hussain
Some guy can argue that it's easier, it makes it easier for a developer to write it and understand it. But at the same point, I would argue that we need to replicate everything that we show and render on the browser in our code editor. So it's easier to see the structure with all the indentation and everything, rather than configuring different settings for the slim in your code editor to not Remove the spaces at the end and all that stuff. So yeah, that was kind of another pain that you need to go through.

[01:04:27.00] - Robby
Is that not something that you can just run through some sort of like conversion script and just call it a day? If the workflow right now is just, if you're going to have to make changes in your slim file, let's convert it first and then make sure things look okay. Do you feel like the automatic conversion would be kind of a little too risky?

[01:04:46.06] - Hussain
Yeah, I would say that because some parts of the app were very delicate and we had to involve QA or at least a developer to test it before doing anything or trusting too much on online conversion converter tools. Or there are some gems as well to convert the slim files to ERB, but sometimes they make mistakes. It was always a good idea to test it after you convert the slim file.

[01:05:11.08] - Robby
Has your team started playing with any of Marco Roth's, like, Herb at all or anything?

[01:05:20.07] - Hussain
Yes, yes. I've been using it. I really, really appreciate his efforts on Herb. It's amazing. I really enjoyed his talk last year in Amsterdam. He's doing an amazing job at that.

[01:05:32.05] - Robby
Thank you, Marco, on behalf of all of us.

[01:05:35.04] - AC
Yes.

[01:05:35.18] - Robby
I'm curious about, like, this is like a particular project that your team's working on a migration. It's happening over a period of time, and presumably, it'll get done. How does your team stay motivated to keep working on that? Do you, like, is there someone that kind of owns that on your team and, like, reports, like, hey, we're 60% of the way there? Or what's that kind of, like, look like from a team culture perspective of, like, how you champion these types of things?

[01:05:58.00] - AC
This particular migration, I wanted to use as a team-building process. So, I don't want to affect the things you want to, you need to deliver. But I do want that to be a collective ownership of what you're doing. So, what I did is requesting the management to add this as a departmental PR for the rest of the company, the information that the team, the department as a whole is looking for doing this as an internal requirement. For the team, it gives them the peace of mind that they're not doing something else not connected to the company. So, you're doing your job by doing this. We announced it to the whole company, like, yeah, the team will reserve some of their time to do this. If they need to change something on the Slim file, they'll convert to ERB first. The purpose of doing that is, remember that we are one team. Not different people working in their own home. Because Donorbox has this very specific characteristic of being a remote-first company. We have people from California up to Cambodia, and very other layers in between, and very other different time zones in between. Being Pakistan the most famous one of them, or most populated at this point.

[01:07:36.08] - AC
The point of those slim files was, is more about the, guys, we are working together here. And the way of motivating people is a weekly reminder of how many slim files were migrated the last week, how many would change without the migration. I'm not trying to build that as a knife on people's neck, but as a, if you need time, let me know. It's pretty interesting how it worked because in the couple first weeks when I started to post that on the devs channel, we do have people that need to change these Slim files without converting them. And I reach out, try to help. After a couple of weeks, this behavior was gone. And people do bought the idea of let's update them. And that's it. And that's the thing with behavior. Once you change, you don't even remember why you're not doing that in the first place.

[01:08:46.14] - Robby
That's a useful perspective there. I think about how teams try to Like, all right, we're going to make this, we're going to start working through this migration, or you kind of will it into way, or like sometimes like, well, does that team require like an owner or someone championing it? Like, and maybe in this scenario, you're like, if you're personally reaching out to people and be like, hey, I noticed you did something and like, do you have any, I'm happy to help provide a little bit of a, see if there's something we can do to make that easier in the future, you know, what have you. And then like that starts to remind people of that, that process and stuff like that. I think it's helpful to hear like these different perspectives, how teams approach these types of things. Because I would imagine like if an application has been around for a while, like it seems like a good, good tangible example we can wrap our head around migrating back to like ERB or maybe just to ERB. Maybe there wasn't an ERB before. Maybe it's not a migration back, but, but a lot of us deal with a lot of interesting challenges on the front end and our Rails apps because of just the generational things of when, what was the new hot, cool thing of the JavaScript framework at the time, or you built one part of the thing and you're like, we're going to build some new thing.

[01:09:46.21] - Robby
We're going to maybe use a new JavaScript framework for this thing to experiment. Maybe that's going to make things better. And Rails doesn't always get those. It's changed its minds, right? And so we all can't keep up maybe as fast as Rails would like us to at times, or at least it was like, well, this is where we're going. And so not everybody's rebuilding their app every couple of years, like 37signals, and they take a new approach and we get the benefit from seeing that and how they're approaching things. I'm curious, like when you're looking And maybe the question for you, Hussain, initially is, uh, the more DonorBox's application can be more Rails-like or following Rails conventions, does that make it easier for you to bring on new engineers into your teams to be like, and look at other comparison projects, be like, look, let's see how they're doing that as there's open source projects or things like that. And, or other projects you've worked on to, to, to weigh that up.

[01:10:40.13] - Hussain
Yeah. Hiring is also a kind of underrated feature of Rails. It makes a lot easier for you to set the expectations because the guy with like 3, 4, or 2 or 3 years of experience, you already know that he knows the basics, conventions and everything.

[01:10:57.23] - AC
Totally.

[01:10:58.19] - Hussain
So onboard— yeah, yeah. And onboarding goes very smoothly, like in terms of you just need to pass some type of documentation, how we work. We do have like our own document. It's like a, I think, 6 to 8 minutes read that you can go through, that this is how we work. And considering that it only takes like 1 or 2 weeks for a person to get an overall idea how DonorBox, how developers are using Rails at DonorBox, you know? So for me at least, it made so much easier to onboard a new person to the team. They already have the basics because I have heard stories from different people using different frameworks. that it takes a lot of time because, take an example of Node.js, everybody has their own way of building a project. So for, I think, first few weeks, you'll be just juggling around to get an idea, the founder of this project, how he was thinking, why he did this configuration particularly. We don't have this issue on Rails. We should be thankful for that, you know.

[01:12:03.00] - Robby
Very similar to what A.C. was describing in the Java land before he got introduced to Rails. And we used to have to make all these decisions, or your app needs 3 logging libraries because of the dependencies. They depend on different logging tooling, and Rails has removed a lot of those decisions for us. Something else that I wanted to make sure, being mindful of time, is one of the things we had talked about prior to this conversation was, I think there was something related to, was it search and filter pages?

[01:12:35.02] - Hussain
Yeah.

[01:12:35.22] - Robby
In your application? And was that—

[01:12:38.23] - AC
Yes.

[01:12:39.21] - Robby
Tell us more about this. Like, I think, what were you doing? What sort of challenges were you having with that over at DonorBox?

[01:12:46.06] - AC
DonorBox requires a lot of filtering for donor segmentation. The people working on that particular feature, they're trying to create the filters, And creating some ways of preventing a user to create impossible filters. Like, we have a filter for the amount of donations you received, the amount of donations a donor made. So I want to see all the donors that donated more than 5 donations. And less than 2. This kind of logical—

[01:13:30.10] - Robby
Complex query. Yeah.

[01:13:32.14] - AC
But do you see that there is a logical failure here? Like, more than 5 and less than 2.

[01:13:38.21] - Robby
Yep.

[01:13:39.18] - AC
You need some way for preventing these mistakes because someone— you don't necessarily need to try to do something wrong. Maybe you're just writing things down, You roll out your mouse over the increment, decrement stuff. So, to avoid those problems, we're trying to create a self-validation of the filters. The thing is, the way it was created, we're trying to use the flash message to explain what went wrong.

[01:14:15.23] - Robby
Oh, I see.

[01:14:16.16] - AC
To do that, you raise an error. So, essentially, they are controlling the flow of the user by raising exceptions. I tried a couple of times sharing Rails guides, sharing the thing about the ActiveModel, and it didn't work. So, what I did is, I would like to ask you for 3 hours of your time today. I sit down with you and try to show you an alternative way of implementing the same set of filters. But I need you to do a leap of faith because it will sound unbelievably tedious and you have the urge to keep asking me, but it's already working, why Why are you doing that? And I was like, yeah, it's not only working, it's deployed. You know, like, you already delivered the value. I just want to show you there is another way. I never managed to do that in a huge audience. Every time it worked for me, it was when I sit down with the person and I say, I got what you did. Let me show you another way of doing that. I'm not saying it's better. I'm not saying I like mine more. I'm just trying to say that this is different from what you did.

[01:15:41.16] - AC
I want you to judge. The merge button is yours. You do whatever you want with that. But I need to give you these 3 hours. So, I sat down, deleted the whole code from the controllers. It kind of calls the attention when you do that. You're like, people, wait a minute. And what I did was implementing the whole filters as one active method with a very creative name of SearchDonors, because, you know, like, I'm very creative of those names. So, given you're searching donors, that's the name of the filter. Plain Ruby class. And I created a method. I don't remember exactly what was the name of the method. I don't know if it's all_create or something like that. But the thing is, I added the first very, very simple method of filtering by dates. And you add the first date and the last date of the donation, and you filter your donors by that. So it feels kind of awkward in the beginning of the exercise, but I asked them, how do you implement the validation of this? And they go, oh, you can throw an error, yada yada. And I was like, yeah, yeah.

[01:17:01.05] - AC
you also have this Rails feature called ActiveModel. ActiveModel is every nice stuff you have on ActiveRecord without the database. So, let's start with the validations. So, I create a validation using that. And when they saw the errors appear in the form itself without requiring them to do anything, It landed different. It's like, oh, wait a minute. How did that? I didn't do anything. I just followed the regex. It's way less effort. And I keep doing that until I implemented all the filters. The thing is, and it took me 3 hours. And what happened is, during this process, I saw people having the same— because it's 2 guys, they have the same feeling that people keep having when I do that this way. One of them got back to me and said, you're right. I was actually through the whole process thinking that, why are you doing that? It's already working. But you gave me this feeling that if it's taking too long, Or it's too hard, maybe I'm doing something wrong. Because it's not possible that in more than 20 years of Rails, I am the first person with this problem. So, maybe Rails already thought about that.

[01:18:37.18] - AC
You know, this feeling.

[01:18:39.09] - Hussain
The power that ActiveModel and ActiveSupport has, it's super underrated. So, yeah, you can do a lot of stuff using those.

[01:18:47.23] - Robby
I think that's probably a common thing that a lot of teams struggle with is like, you might have, there's already a lot of complexity maybe in a controller action somewhere. Um, I think search is like a really good example of that, where probably the one of the most common things I've, I've seen hundreds and hundreds of Ruby on Rails applications and those really big controller methods. And sometimes people abstract that a little bit and, and come up with some way to approach that. And it's not always super consistent, but, uh, but it seems to be like a little tricky. It ends up, they can be a little fickle and complicated to debug. Like, why am I getting this weird issue? Or, uh, you end up with a lot of weird, like nil messages showing up. One of the probably most common things I see when it comes to like, uh, weird things that people are like, well, it's probably an issue, but maybe not an issue. I'm not going to look into that particular area today, but, uh, we'll look at that at some point. It's probably, it could be like someone just crawling the website, you know, and hitting a search page or something.

[01:19:41.06] - Hussain
And then.

[01:19:42.04] - Robby
It triggers some weird error messages just because it's a bot or something, but knowing when to leverage Rails for these types of things. And if the code is already seemingly working and has already, as you said, provided the value, when do you come back and revisit these things? And so I appreciate DonorBox has like an approach, like in your culture there of, let's go back and revisit these things. Like, we, it's okay. We don't have to be scared to delete this and re-implement this in 3 hours. That could be, you know, and that could unlock things in people's Minds. I'm curious, you know, I think maybe Hussain was mentioning like there you have like a thing that takes like 6 to 8 minutes to read. What's— and how your DonorBox approaches it. Are these like your team's engineering principles and like when to reach for things? And like, does Rails even mention that or? Yes.

[01:20:26.09] - Hussain
Basically it handles everything for a new hire to let them know that, yeah, onboarding guide. It mentions the commands that you need to run. Basically, those are all already mentioned on the GitHub repo as well. But overall idea, how the teams are structured, how the code works, what are the main parts of the app, some tips and tricks for adding new columns in the database, like how to do any kind of optimizations on database levels, a lot of those. And we kind of try to evolve this document with the time. So we revisit it sometimes even, it's been like 2 years, I'm not sure, but I still revisit the same document whenever I'm having any issues. Maybe there's a new developer who raised us some questions. I try to see if it has been answered already in our onboarding guide or not. And then just, I just link that, share the link to that part of the document to this person. So this way we keep evolving this document and it makes easier for us to onboard a new guy, like frictionless way.

[01:21:27.05] - Robby
I'm curious, given that we're having this conversation and it's August 2026, and this episode will probably come out in the next month or so, but I'm curious about, does this tooling also that you've been investing in, are you bringing on agents into your development workflow? Are you using any LLM AI agentic development in your—

[01:21:52.18] - Hussain
Like locally in your—

[01:21:54.01] - Robby
to help you and your engineers work on this? And how is this tooling and the onboarding documentation part of that?

[01:22:00.16] - Hussain
Yeah, so we have put similar efforts into the agent side as well. So we do have Claude and Cursor. Some people prefer Cursor in their development flow, and some people are using Claude. And then we have, thanks to AC, we added AGENTS.mdmd and another couple of files for overall view for agent before writing anything and how to commit, how to write commit messages and all that stuff. And thanks to our org experience team, we do have our internal gem for adding design components which is powered by MCP server. So your agent can ask questions how to add a new button, how to add a modal, you know, this type of questions, and then it will make easier for you you to not spend too much time on the front end and just solve the main problem on the back end, you know. So with the design components, it has made a huge difference in saving so much time and making all the products of the same nature. Previously, they were like a mixture of pages, like using old design and new design, and now it feels very easy to add a new feature. Right now, my team is at the level that when we are going through Figma design, we just thinking and modeling in our mind, okay, I need to add this model, this connection association, and that's it, you know.

[01:23:21.02] - Hussain
And it doesn't take much time to develop as well. So because you have design components in place, you just need to put them in right place and just focus on the backend strategy and structure, and you're done.

[01:23:33.05] - Robby
Do you have much tooling that, or come up with any ways to like say like that you're to feel pretty confident that someone is Uh, that like, uh, maybe not someone, but the agent or the PR is using the, the ideal path going forward. 'Cause maybe they're, you know, a lot of codebases have 3 or 4 equally plausible historical ways of approaching similar things. And you might be like, as a team, be like, we're going to start moving more in this direction, but the agents might be like getting to source a lot of things. How are you wrestling with that right now?

[01:24:05.18] - Hussain
Personally, I would Prefer your first thing as readable code, and it should be domain-driven way because it's a legacy project and most of the code is built in this way. So those are the 2 things I want my agent or anybody's agent to focus on, that code should be readable and it should follow domain-driven design pattern. But there could be different ways. The ways that we can't compromise, they are already mentioned in .md files. So, that will give you, if you are asking your agent to do a quick review, it will ask you, okay, this is what is mentioned on your .md file. Do you still want me to write it this way, or we should rethink about it? You know, that's like a soft warning.

[01:24:51.21] - Robby
Sure, sure.

[01:24:52.22] - Hussain
Yeah.

[01:24:53.17] - AC
What happened is we're documenting the things we don't want to follow anymore for the agents.

[01:24:59.22] - Robby
Is that just something that you're kind of As things pop up, you're like, oh, yeah, we should add it to the list of things to stop following in that direction. Who's deciding that? Maybe that's a complicated question, but—

[01:25:12.11] - AC
I don't know if everyone will have this view that it is collective because there is one particular name that keeps popping up every time you see the commit messages. But what happened is that I reach out for people before changing these documents. Instead of creating a big committee of what are the decisions we should take for this particular thing, I reach out for the people that are actually working on that and request for comments. Like, we're doing this way in this particular side of the code. We have this other way of doing it. What do you think about requesting the LLM agents to, every time we stomp on that, help us to move to this other direction we're looking for? I see the LLM agents as an amplifier of yourself. So it will amplify your knowledge and also amplifies your lack of knowledge. What are we trying to do is having a place for the lack of knowledge or lack of context of one individual has a backup in an agent document. So, that's the overall idea on how this is being built.

[01:26:37.22] - Robby
It's helpful to hear how different teams are navigating that right now because I hear you on the, it's a collective thing and we're all participating, but maybe there is one person that's being a little bit more vocal or being like, hey, what do you think? And you're It sounds like you have a very— you're inviting people. You're not dictating, this is how we're going to do things now. This is AC's way, everybody follow along. Maybe I can have a side conversation with Hussain later and he can tell me otherwise, but I assume that— take that for— we'll just trust that that's actually the case there. But I believe you there. But I'm thinking about, do you have any recommendations for teams right now that are that haven't quite figured out, is it just as simple as like, hey, in your agents.md, there's some— you're annotating certain types of patterns, like, hey, we've done it like this. And so when the agents are looking at things, here's how to avoid— at least ask the soft question, like, are you sure you want us to— Maybe as a better tangible example, do you have something in there for like the slim and ERB thing?

[01:27:41.08] - Robby
And they're like, if a slim file is being modified, raised a question in the chat with, like, maybe we should convert first?

[01:27:51.11] - AC
The sad story here is that we didn't have Claude when we were implementing, we were moving from Slim Files to ERB, which is kind of sad because it helps so much. And what you're doing here on the agents is trying to describe the pieces of the puzzle in their own MD files. I do have a big agents.md. And for those who use Claude, you can always create a—

[01:28:22.19] - Robby
Symbolic link.

[01:28:23.15] - AC
Symbolic link, which is pretty funny when you need to update your CLAUDE.md file, because, air quotes, because Claude tries to update the file, he said, oh, wait a minute, This file doesn't exist. It's a symbolic link. Yeah, sorry about that. But for reading, he didn't notice. So, what I do is having in the agents.md file the core things that you need to use in every question you make about the codebase itself. So, how do I run a test? How do I run the servers? In our case, it's pretty boring because it's the rails test, rails server. So, yeah.

[01:29:06.18] - Hussain
Yeah.

[01:29:07.09] - AC
You got it. And we keep the tech stack, so it helps on building the context for the LLMs. We added the .docker-compose.components gems and how you update them because it helps when they're creating something that requires UX. In the same file, we have the application structure, like how our directories looks like and what is there.

[01:29:36.20] - Robby
Is there a lot of interesting things that are different than a, I'm gonna say, a typical Rails app that you've done differently in your Rails directory structure? Anything for our listeners that stands out kind of interesting?

[01:29:48.17] - Hussain
For example, we don't have services folder, you know, so that's one example. We want every class that you want to write, it can be, if it's associated with the business logic, it can stay in the models directory. So, no need to add another directory in app folder, you know.

[01:30:07.13] - Robby
Do you namespace those at all?

[01:30:09.06] - Hussain
Yeah, we do. But it is still in the models directory. So, it's just conventional stuff like app, models, views, helpers, and those defaults.

[01:30:21.01] - Robby
Anything interesting in your lib directory?

[01:30:23.05] - AC
We have a lot of different integrations. For example, Cloudflare, because Cloudflare maintains the cached forms and hosted pages. So, every time a user, in our case, an organization, our direct customers, they update their form, we need to invalidate the cache. We use the lib as a wrapper around those integrations. So we literally have a code like Cloudflare cache purge this form right here, this campaign form. And the translations happens from DonorBox language to Cloudflare language happens in this call of— I tell you the name of the form. And by this name, the Cloudflare cache purge knows how to get that URL. The cool thing about this is when you're reading the code inside Donorbox where there's this call, it sounds like Cloudflare is just another library that you're using. When you have the update of the form, you have a Cloudflare cache purge with the name of the that form. So we usually call it a slug because you have a dropbox.org/slug of the hosted page or embed slug to get the embedded version. The Cloudflare details didn't leak inside of Dropbox code. And we have a couple of examples like this. The other one is Plivo because we send text messages.

[01:32:10.05] - AC
for donations, and we don't send text messages for requesting donations. We do have this feature of you text DonorBox asking for a form. If you already made a donation in that form, you can make another one through text, but it's always a donor-initiated process, not a—

[01:32:31.22] - Robby
I see. Any other fun, interesting things in your Rails directory structure there, or?

[01:32:37.08] - Hussain
Also, I wanted to mention, we do have one thing that we are kind of demotivating agents to do is adding any new CSS files. So the whole point of having design components is to not write, or at least write minimal CSS, and that CSS should live in the same HTML file. So there shouldn't be like a directory or pathway of all the CSS files.

[01:33:05.10] - Robby
Right.

[01:33:06.09] - Hussain
So if there's something missing in design components, then this is a good point to reach out to the team that I'm using this design component, but it's not supporting and designer built it this way. So it's just like a 15-minute talk with you all 3, and then you make a decision. Okay, we need to evolve our design component or we need to ditch this new requirement.

[01:33:28.00] - Robby
That's interesting. Is that typically coming up when you're working on some new feature requests or something and you're like, Like, how do we—

[01:33:33.02] - Hussain
Yeah, mostly new features.

[01:33:35.05] - Robby
You mentioned you're using Jira. Out of curiosity, when the developer gets a ticket, an engineer on your team has a ticket, is it already kind of determined, like, you should be pointing to this in our design library, or is that something the developer's figuring out as they start working on the ticket?

[01:33:51.12] - Hussain
So basically in the start of the task, we do like an R&D phase where a designer would do like a scheduled couple of meetings to get an overall idea what we want to achieve, what we need to skim before we actually start the design process. And after the design process is done, I involve one developer who is motivated to work on that feature or product, and then we would set up a meeting and then ask any questions. So this developer would take like 2 or 3 hours to go through the design, see if there are any limitations if this is doable, if we need to skim, seeing the deadline and everything and put it in perspective if it will be done before that. If all good, he has answers to all his questions, then I would do another meeting with him just to devise a development plan. It's not a long meeting like 3 hours, but it's like a 30-minute meeting. We would just quickly go through, okay, this is the migration that we need to add, like overall idea, because Engineers are also like mid to senior level engineers at Donorbox. So you just need to give an overall idea that this is my approach.

[01:35:00.08] - Hussain
If you have ideas, then we share it with each other and get to work. I see.

[01:35:05.23] - Robby
Yeah. And how much do you feel like you— how far away do you feel like you are right now? Maybe think about, Hussain, on your side where you have your Linear, your Jira ticket or whatever your team is using and be able to be like, all right, a developer takes that, pops it into their Claude or cursor and be like, all right, implement this, and not needing to do a lot of code review themselves. How far away do you feel like you are right now with that culturally at the moment, or personally maybe as well?

[01:35:36.20] - Hussain
No, I think we are blessed in this regard that developers at Donorbox, they're not trusting too much on their agents. For now, at least. So I'm not very cautious about this particular issue. I do use it for code reviews and all that stuff, but I become hesitant just sharing that review from Claude to, you know, the developer. I need to go through it, ask questions— why did you post this?— and maybe update the .md file so the next time it doesn't make that mistake. And it considers the convention that we are following or the direction that we are following in terms of code quality or code debt or anything like that. So yeah, I'm not much worried about this particular issue.

[01:36:27.23] - Robby
Do you feel like you're seeing a lot of throughput improve? Like you're able to get through more as a team at this point, or is it? I feel like talking to different people on their podcasts and some people are like, oh yeah, we're like, the number of PRs has drastically increased and we have other issues now as a result, but now we have more PRs to review or something like that. Is that showing up yet or?

[01:36:50.12] - Hussain
I think the biggest advantage for us that we have got so far is because the Donorbox project is very big. So it has made and empowered you so much to know what other teams are doing, just by asking AI rather than pinging other devs. You can just have a conversation with it, how the payments are working, how we are charging the invoices, how we are, you know, scheduling everything. Those have empowered you to understand how other teams— if your code or feature is related, it's touching some parts of their code. So rather than having the long meetings, you can just ask AI.

[01:37:28.07] - Robby
Something I'm curious about with that Do you foresee, like, fast forward this progression a couple years, uh, regardless of whether or not we're gonna get to a point where we can just assign an agent a task and be like, great, and then Ship sends 10 PRs where you're not even looking at anything. But if, if your developers are maybe spending a little less time in those maybe longer meetings, do you see there potentially be any interesting things culturally where your team's not checking in with each other more? by themselves and they're just asking the AI for information on what their peers are doing? And do you feel like that could cause any interesting, maybe not a good or bad thing, but just more of like, it's something I'm, I'm always curious about when people are like, oh, I'm not, I'm not needing to ask my coworkers for their input. Like, how, how have we done this in the past? 'Cause I can just ask Claude or Cursor or whatever. And if that, those conversations start to get reduced, then I feel like there's gotta be some sort of like interesting. unexpected consequence of that? Maybe not a bad or good thing, but it just, it'll be different.

[01:38:27.04] - Robby
Like, maybe we don't ask, maybe will it feel like a cohesive team in the same way? Or maybe it'll just, it'll feel different.

[01:38:33.18] - Hussain
Yeah, I think we haven't gotten to that point that AI can answer every question, because there were so many decisions that people made considering the limitations at that time. So AI doesn't have that information. It can go to the commit and PR and all that stuff, but not at the discussion What was made. For example, why Events had a different way of using Rails, why it was so much different. So only a handful of people at Donorbox know this story. So AI can't. So unless you feed it into the .md file, AI can't know why it was built this way. So this is kind of contextual history. I think you need to feed it and you can't trust it too much with everything. But you can ask it, like, what does this class do, or trivial stuff, which takes time for a human to understand.

[01:39:24.23] - Robby
Any thoughts on that, AC? Yeah.

[01:39:27.16] - AC
The agents are a very enhanced Google search. So, the same way you don't approach your colleague to ask for, What is the name exactly of that validation for numeric values that is positive and only integers? You just search for that, looking into the future. Questions like, okay, where should I look for some information? You can start the conversation with the AI and get the information you need to actually have a conversation with a human being. And Use that as a, oh, now I did my homework. This is what I found. Can you help me understand what the hell you did here? Why is it this way? What are you trying to prevent? Because this is the other information that the AI doesn't have.

[01:40:24.10] - Robby
Is there anything that you're kind of like, maybe for each of you, that you've yet to be able to experiment with that you would love to be able to get an opportunity at some point to, like, something kind of coming down the road or something that's been introduced that you're like, Well, we haven't been able to start doing that at DonorBox, but maybe at some point, I'd be really interested to be able to have a chance to play with that. Something in the Ruby or Ruby on Rails ecosystem. And maybe we'll start with you, A.C.

[01:40:51.18] - AC
Right now, we are using this Froala editor, and I'm really interested in the MD editor that was released recently, as recently as Possible, but you got the point. It sounds interesting to check it out. And it's a move back to the, well, if you're gonna write JavaScript, just run JavaScript, don't try to recreate JavaScript in another language behavior that the tricks didn't have. It was that TypeScript stuff. I would love to have the opportunity to test that because it sounds very interesting.

[01:41:30.06] - Robby
Does Lexxy?

[01:41:32.02] - AC
Yeah, yeah.

[01:41:33.15] - Robby
What about you, Hussain?

[01:41:35.09] - Hussain
I think I'm excited to use Solid Queue in our code base very much, especially the job continuations. It's been a while since Fizzyi released their code base online, so it's just so much fun for me to casually read the code base. And another thing I noticed was how they are using the identity So identity is basically is equal to email, and if you do identity.current, your user, you know, and if you do identity.user, it tells you your role. Basically, this, this was very unconventional for me to absorb and digest because coming from the background using Devise and session controllers and all that stuff. And I think this can play very well at DonorBox in terms of logging in can be done from different roles. You can be a fundraiser, you can be an org admin on behalf of organization, you can be a donor. Implementing this would be fun for me, and I would raise my hand to do this.

[01:42:37.14] - Robby
Out of curiosity, I didn't want to go too far down the weeds there into your platform there, but if a single person, like let's say RobbyRussell@gmail.com is a donor to ONE, organization, but I've donated to 10 different organizations, 10 different customers there. Is that linked at all in your system or are those treated as separate donors per customer type of scenario?

[01:43:02.08] - AC
I think you're going to laugh in the end for my answer, but the answer is yes. Great. The thing is, from our DonorBox perspective, a donor is a contact of a person and that organization agenda. The person itself is another thing.

[01:43:26.19] - Robby
I see.

[01:43:27.10] - AC
If you donated to 10 different organizations, you have 10 different donors to represent yourself. But the cool thing is you can manage all those donations through your donor credentials. So you can see all of them, but they can only see their own context for you.

[01:43:51.05] - Robby
Yeah. Okay. And there's no— okay. That's helpful. I was kind of curious about that. But Hussain, do you feel like if you— there's a scenario where you might look to migrate away from something like a Devise?

[01:44:05.15] - Hussain
We're not using Devise at Donorbox. I just gave you an example because Typically on Devise or any, you try to use everything in the user model like authentication fields and everything, but Fizzy gave a very different perspective on how they solved this issue.

[01:44:24.17] - Robby
Yeah.

[01:44:25.02] - Hussain
Using identity, you know, so that was interesting. It's just like an analogy of entering a building. There's a security guy, he just wants to check your ID, doesn't care about about who you are, where you work, and he just lets you in. And then there's different types of areas in the building that you can go in.

[01:44:44.10] - Robby
Yeah. All right. So before I let you both go, here's a question that hopefully is a little less complicated. I'll start with you, Hussain. Is there a programming or software book that you find yourself recommending to other engineers?

[01:44:58.11] - Hussain
I would just recommend the old bible for Rails developers is Agile Web Development. That's a good one. I loved Eloquent Ruby and Practical Object-Oriented Programming by Sandi, I think.

[01:45:14.18] - Robby
Sandi Metz. Great. What about you, AC?

[01:45:17.21] - AC
Well, Hussain stole all my recommendations, so—

[01:45:22.03] - Robby
Plus one.

[01:45:23.10] - AC
I'll go with The Pragmatic Programmer: From Journeyman to Master. It's not a new book, it's kind of old, from Andrew Hunt and David Thomas. You probably heard about David Thomas before. The discussions in this book is about everything you do in a programming environment professionally. You need to weigh pros and cons, and they give you 1,000 examples of that through the book. But the way they describe it is programming is a tool for achieving something you want to do. Sometimes your goal is the journey of programming itself, and that's fair, but they call your attention to things that you just take for granted. So, that will be the reason around my book recommendation.

[01:46:20.08] - Robby
That's a good addition to that. Well, I'll definitely include links to all of those in the show notes for our listeners. And with that, I wanna thank you both, A.C. and Hussain, for stopping by to talk shop with us on Rails. It's been a delight getting to chat with both of you about all the fun stuff you're doing at DonorBox, and hopefully we'll get to chat again at some point, and maybe at a conference or something.

[01:46:38.21] - Hussain
Sure. It's been a pleasure.

[01:46:41.04] - AC
By the way, we'll be on the Rails World in Austin. Yeah.

[01:46:45.15] - Hussain
Good opportunity.

[01:46:46.09] - AC
Excellent.

[01:46:46.23] - Robby
We'll, we'll look forward to, yeah, getting to catch up there. I'm speaking at the end of day 1, I just learned. So, um, well, thank you so much, both of you. Thank you again for stopping by and for joining us On Rails.

[01:46:56.20] - AC
Sure. Thank you so much.

[01:47:00.01] - Robby
That's it for this episode of On Rails. This podcast is produced by the Rails Foundation with support from its core and contributing members. If you enjoyed the ride, leave a quick review on Apple Podcasts, Spotify, or YouTube. It helps more folks find the show. Again, I'm Robby Russell. Thanks for riding along. See you next time.