Robby (00:00:05.18)
Welcome to On Rails, the podcast where we dig into the technical decisions behind building and maintaining production Ruby on Rails apps. I'm your host, Robby Russell. I also run Planet Argon, and for over 20 years we've helped teams maintain and evolve their long-lived Rails apps. So I tend to approach these conversations through that lens. On this episode, I'm joined by Nikki Sutherland, lead engineer at Shop-Ware, which is part of Vehlo. Shop-Ware is a software platform that helps automotive service shops run their businesses. Behind the scenes, Ruby on Rails has played a major role in powering that platform, and it's grown and evolved over the years. In our conversation, Nicki and I dig into how the Rails app has evolved over that time, how their team approaches API design and front-end architecture decisions that have accumulated up over the years, and what it looks like to continue maintaining and improving a mature Rails system with a much smaller team than they once had. Nicki joins us from Philadelphia in the United States. All right, check for your belongings. All aboard. Nicky Sutherland, welcome to On Rails .

Nikky (00:01:03.11)
Thanks, Robby. Happy to be here.

Robby (00:01:05.15)
All right, Nicky. So to kick things off, what keeps you on Rails these days?

Nikky (00:01:10.03)
Oh, it's pure love. I've been on Rails for 12, 13 years now, and every time I drift away, I always come right back because nothing else makes me feel the warm embrace of something I think is really cared for and framework that really makes sense for what I'm trying to do.

Robby (00:01:26.03)
Out of curiosity, what tech stacks did you work with prior to Ruby on Rails ?

Nikky (00:01:30.23)
Well, I got started working on Perl forums back in the 2000s.

Robby (00:01:37.00)
Okay.

Nikky (00:01:37.08)
And that was real fun. And then a lot of WordPress experience after that. And so PHP as well, but that's kind of the general progression.

Robby (00:01:45.18)
Is that back in the Perl CGI era? Script era, or?

Nikky (00:01:50.08)
Yep. CGI bin directory. And it was a forum software, like probably like 3 people used and did not work very well, but it was great at the time.

Robby (00:01:57.22)
Oh, brings back memories myself as well. I did quite a bit of work in Perl actually in the early 2000s, a couple years before I started using Ruby on Rails myself. Did you find any interesting similarities between Perl and Ruby at the time?

Nikky (00:02:09.00)
Not really. I did appreciate, you know, the fact that Rails had an actual ORM, whereas our Perl database was actually a flat text file. They don't believe in actual databases, this particular piece of software.

Robby (00:02:18.18)
So.

Nikky (00:02:21.03)
That was fun. But I do find other languages that I think Rails has a bad rap, but I disagree. I think it's really friendly for what it's trying to do. And I think it's designed with a lot of thought and intent, which is what I really appreciate about it.

Robby (00:02:33.06)
Nice. All right. So why I wanted to have you on the podcast is like, so for listeners who might not be familiar with Shop-Ware, can you give us a little bit of an overview of the platform and how Ruby on Rails kind of fits into the architecture today?

Nikky (00:02:44.15)
Yeah, sure. So we've been around for, a little over a decade now. I think we got started in 2013, and it was a Rails app in the very beginning, Rails , probably version 3 at the time. As the company and the group has grown, Rails has grown along with us. It's really helped us meet the challenges that a growing platform needs and provides the guardrails and recommendations to do things the Rails way, which is the way that we wanted to do them initially.

Robby (00:03:11.10)
What is Shop-Ware actually?

Nikky (00:03:13.07)
You know, there's lots of things in the world that you don't really know about, and I think this is software for auto shops. There's mechanics every day who are working on your car and you bring it in and you have some sort of issues and you have a service advisor who inspects your car and tells you how much money you need to pay to get your, your vehicle operating safely again. It's always more than you'd probably like, but you know, that's a software that helps keep track of that job. You know, you can prove your estimates and then it gets handed off to the technician who then actually does the work and tracks their labor hours. And it's kind of an all-in-one platform to help an auto, an auto shop business kind of run their shop, which is, you know, a lot of like insider industry stuff that we don't think about, but they care a lot about and we try to help them and match their enthusiasm.

Robby (00:03:54.13)
Is this primarily for like internal operations or does it like handle like marketing-related activities as well or kind of across the board?

Nikky (00:04:01.22)
It's a little bit of both. It primarily is for communicating with an actual like vehicle owner. So it's mostly internal, but we can communicate with the vehicle owner if we need to. Like we show them an inspection of the car, which usually includes lots of photos that they take, and we can do some like texting platforms as well. Mm-hmm. But we don't really do much of the marketing side and public websites. That's kind of on the shops to do. And there's other solutions that we, we have for that within our, like our parent company, but we focus on like internal operations mostly.

Robby (00:04:28.23)
So it's like someone brings their car to an automotive shop and they're using this to bring in, do some customer intake, tracking their workflow. Does it handle like billing and things like that as well, or?

Nikky (00:04:41.11)
Yeah, it handles payments, it handles, you know, sometimes there's a down payment for large job, it handles estimates and approving those estimates. You know, if it's a large job, you want to approve that spend before they actually start throwing in like thousands of dollars worth of parts into your vehicle.

Robby (00:04:56.02)
So is this like a SaaS type product then, or?

Nikky (00:04:58.18)
Yeah, exactly.

Robby (00:04:59.12)
Okay, interesting. And so you said that's been around for, you said 12, 13 years, is that right?

Nikky (00:05:03.19)
Or? Mm-hmm, yep.

Robby (00:05:04.20)
Maybe you mentioned this earlier, but when did Rails kind of enter the picture for Shop-Ware?

Nikky (00:05:08.06)
I think it's the very beginning. I've only been around for around 4 years now, but my understanding is the first line of code was Rails code. And we still see some of those 12, 13-year-old pieces of software lines in our codebase still. So it's from the very beginning.

Robby (00:05:23.08)
Are there other tech stacks in the workflow that fit into this as well?

Nikky (00:05:28.17)
Yeah, there's a couple. As with most companies that existed in the 2020s, we have some platform services and some AWS Lambdas and various stacks in there. But it's almost, I would say, 90% plus is a Rails-based codebase.

Robby (00:05:42.21)
Do you know off top of your head, like approximately how large of a Rails codebase these days?

Nikky (00:05:46.22)
I'd say it's a couple hundred thousand lines of code.

Robby (00:05:48.15)
Okay. Overall. And how big of an engineering team is kind of managing that today?

Nikky (00:05:54.00)
We have around 10, includes some contractors as well, but around 10 to 12 usually engineers that we're looking at.

Robby (00:05:59.04)
That definitely helps us, at least helps set the scene for some of the topics I want to dig into. So for developers that might be listening, so Shop-Ware has grown Where do you feel like Rails has been, say, part of the platform secret sauce?

Nikky (00:06:11.01)
The really nice thing about Rails is that it is an opinionated framework, and I think there's a lot of value in having those opinions because you're not having to bike shed everything at that point. You know that as long as you trust the people behind it and the groups behind those frameworks, you don't have to do a ton of second-guessing and research about the decisions that they've made and that they're advocating for. I think it's really helpful in that, as you have this framework, and you are trying to go do something that's extremely against it, you can do that, which you can do in Rails, of course, but it really makes you think and pause about, is this a good idea? Do I need to be really contemplative about this thing before doing it in a way that's not the Rails way? And it's been really helpful for us, because it is that built-in check of, you want to do this thing that will take us 4 times as long, because it's not like the Rails Ruby on Rails way of doing things. Is this worth it or should we maybe rethink how we actually want to implement this?

Robby (00:07:09.22)
When they're thinking about you're adding new functionality or making architectural decisions, do you find yourself mapping out how can we do this with Rails or is it what's the best way to do this and then try to figure out how to do that within the Rails? Is there a direction your team tends to navigate when you think about these things?

Nikky (00:07:25.19)
I don't think it's either or, I think it's a combination of both. I think our engineers are very steeped in the Rails way of doing things and so I think it comes really naturally to them, myself included, to think about these, we are shaped by what we are experienced with and what we've been working in. And so, as we build these architectures out, we're always thinking, I think, of the Rails way, even if it's subconscious, just because that's a way that we're used to doing and we know works well with the system.

Robby (00:07:55.12)
Right, right. You mentioned the ORM and Active Record, and that's a pretty common thing. Are there And maybe, is there something about Active Record that you feel like has really resonated really well with the type of platform that you built there? And follow-up question, are there any other particular Rails features or conventions that you feel like have really paid dividends over time?

Nikky (00:08:17.10)
I think Active Record is huge for us, as is huge for almost anybody who I think uses Rails, in that it does provide this really well-intentioned way of visiting your data layer, and it's a way that you know, can be familiar with developers who are onboarding as well, who maybe not be familiar with Rails. Like we're saying, okay, here's our data layer. Here's how we view it. And it's extremely easy for them to pick up, which is huge for us. And the second part question was what else do we find useful?

Robby (00:08:46.19)
Yeah. What other aspects of Rails that comes kind of bundled with Rails you feel like is really helping pay off for your team?

Nikky (00:08:53.00)
The separation in model-view-controller. Of course, is also huge for us. Like we've seen applications in the past that could be extremely cluttered or there's not like a clear separation between all those things. And there's a really bright line in Rails , as long as you take care to enforce that bright line, that really helps you keep everything like extremely well organized. And especially as your codebase gets larger and larger, it helps you understand like who's the expert in what and who maintains it and where all this application layer should actually be going.

Robby (00:09:22.15)
Do you find that, I don't know if you can compare this to previous Tech stacks or roles that you've been a part of, but do you feel like there is hyper-specialization within your team, within like, given how you're approaching Rails, or do you feel like because of things, how Rails approaches things, that people can move around the codebase a little bit more freely than you might've experienced in other organizations?

Nikky (00:09:45.23)
Yeah, it's great. Our team is, they're all full-stack engineers, myself included, and I truly believe that is correct. Some of us are more experienced in the backend or frontend layers, and as you progress kind of in seniority, that's to be expected, but At the end of the day, the way that the applications are organized and architected means you can work like one day in the front end doing some really application layer stuff, and then next day be working on like HTML or working on like CSS platforms or something, which is really great because it does give that flexibility for engineering staff to be kind of working in all different areas depending on what, what the need is for the business.

Robby (00:10:22.18)
Do you think that's also Given some of the way that Rails approaches the stack, you're able to focus a little bit more on the product than the infrastructure itself. Do you feel like it's kind of helping a lot in that area as well? And do you have people on your team that kind of focus more on the infrastructure side? And are they actively working on the Rails app or are they kind of like a separate team that manages infra?

Nikky (00:10:42.13)
Yeah, it's all the same team. And it is really helpful because the way we monitor applications and view our applications is in a way that's really familiar with us. And especially like our senior Senior+ engineers who've been around a long time really kind of understand how servers behave and application layers behave in a way that makes it so that we can monitor and deploy and deal with our application hosting itself. And it's not perfect. Sometimes dedicated staff are better for that. But for us, I think it works well. And I think it's also really helpful for the engineers to be extremely close to your production instance because if something goes wrong, you know much better of what's happening. If you make a change to the code, you can see how it reacts in production. I think it can help spot issues and performance things really quickly, and you're much more cognizant of what you do have potential downtime down the road if you don't do it in a meaningful way, which I think is great because that's the ultimate accountability tool.

Robby (00:11:37.20)
Sure. I'm curious, how does your team currently thinking about monitoring and observability? Are you leveraging many observability-specific tooling? Things at this point, or where does that kind of fit into there? And then if your team is, one of your engineers is assigned to, let's say, deal with a bug that's only able to be reproduced in production, what's that workflow look like for being able to kind of figure that out? If you don't mind sharing a little bit behind the scenes.

Nikky (00:12:02.11)
Yeah, I was thinking about that. So I think one of the best attributes of Rails in the very beginning is I remember I was doing a small class Probably 2013, the Ruby on Rails, and someone, one of the instructors, pulled out New Relic. He's like, "Hey, check this out," and installed the ActionController keys, installed the gem, and it's probably Rails too at the time. We were all just stunned. We'd never seen anything like that. Most of us were from PHP WordPress land, but to be able to see this app-specific information that actually drills down to your individual controllers and see your actual queries and those explains behind it was like magic. It was absolutely amazing. And I think we still see that today with the observability tools that we have, is they're really, really good. New Relic has obviously gotten into a much larger thing now, but the core product is still there, and we have really close connection with the code and how it behaves in production because of these tools. And I think error reporting also seems a bit magical to me. You know, like Sentry is, I think, kind of the main tool that I've used a lot.

Nikky (00:13:07.12)
It is great, like the ability to see this, error almost real time and in some cases address it before you even hear from support staff, or you can notify them like, hey, in 10 minutes you're going to get probably somebody complaining about this. We've already fixed it, but just to let you know that's already happened. That's great. That really gives so much confidence that we can ship things faster and ship them more reliably and know that we're not causing issues.

Robby (00:13:35.17)
Let's say an issue pops up. It sounds like you're using Sentry there potentially, and something pops up in Sentry. There's an issue. Is your team needing to like figure out things that are going on with production data at all? Are you able to like, what's that workflow for? And also how does your team approach like a healthy Database Seeder strategy for like, let's say local development or testing things out in staging? What sort of tooling has your team put together there? Or is that an area that you're still trying to figure out?

Nikky (00:14:04.05)
I would say it has grown organically over the past decade.

Robby (00:14:08.06)
Like good seeds do?

Nikky (00:14:11.11)
Exactly. Yes. And it's always a challenge, like, because I think a lot of developers don't think about that always. And I think it is important to have a seed that is reliable because one of the best ways I think of approaching some of these things is you do start from scratch. Like you have a new issue, a new bug. Let's wipe the database, let's start our dev environment from scratch so we know everything is clean and ready to go for our next work. And so a clean seed file is a really important part of that. And we have specs that can ensure that the seed file runs cleanly. And if we drop a table and then later don't remove it from our seed file as well, it's going to complain about it. And that spec never even passes whenever you merge that in. That's been really helpful for us, the tooling around that, but Onboarding's always a challenge. We are a smaller team, so we don't onboard that often. And it seems like every time we do, something new and unexpected and fun comes up that we have to contend with. Some new service or some key or some database thing doesn't quite work.

Nikky (00:15:09.23)
But it's a result of we have another organically grown documentation that every developer as they onboard helps add to. And then we are in a state where I myself am actually going through it. I broke my laptop and have to get a new dev environment set up. And it's been really helpful seeing that experience from scratch that they have to actually go through this and make sure everything is working.

Robby (00:15:30.21)
Do you feel like your team is able to, your seeds, you mentioned it's grown organically. Is there typically enough volume to stress test anything, maybe on a performance level locally, or if you've got, say, production issues that seem to be maybe just from the sheer volume, what resources and tooling does your team have access to, to try to mitigate those types of issues?

Nikky (00:15:51.10)
Yeah. We've developed largely through kind of the ease of Ruby on Rails 's scripting capabilities, lots of stress testing scripts that we can run that will quickly seed in a certain database hundreds of thousands of associated records. And that's very helpful because that usually gets us to be at the scale where we can start looking at what Postgres is doing in a near production-like environment. And it's not just relying on sequential scans anymore, but actually having user indexes, which is very helpful. And we have a really good quality assurance team, and they really help out with defects. And as a bug comes in, they usually will give us very good reproduction steps on that. And I don't have too much to say on that. We do things, but there's no great processes in place.

Robby (00:16:39.00)
Okay. That's fair. So for many long-running, say Rails applications, the application's been in development for you know, over 12, 13 years now. So they tend to have a lot of front-end decisions that might date back several eras in the ecosystem. So what does that look like at Shop-Ware today? Is everything up to date on our most recent version of the front-end ecosystem, or maybe not?

Nikky (00:17:02.07)
Yeah, that, that's a story as Ruby on Rails applications have evolved. And I think anyone started in the past decade is experiencing this where The frontend framework decisions that these applications have made were largely best guesses. You kind of pick what framework you think was going to survive in the future and works with what you have right now. And in many cases, in some cases, us included, our initial framework was Backbone, which I don't know if you're familiar with that one, but I think it's quite cool, especially just the Backbone relational component to it, which I think really fits within the Ruby on Rails framework, but it's also unmaintained and doesn't really do the things you want it to do, and no developers know what it does anymore. And so we've been moving over to the React side of things, which works really well for us, but it is different than Backbone. I'd argue quite significantly different. And yeah, as much information that we get from the Rails backend on best practices and ways to do things, I think we didn't get that for a long time from Rails on the frontend. And I know this is not a new story or a new revelation at all.

Nikky (00:18:16.14)
Everyone, I think, has gone their own path, us included. And it's been a journey to get us back to something that makes a little more sense. We're still compiling our assets. It's coupled with our backend releases, but we are working towards decoupling those. But it's a process because there's all the baked-in assumptions there and libraries that are hard to unwind once those decisions have been made.

Robby (00:18:38.23)
So is it safe to assume all that Backbone is in the same repository as— and what about the, what about the React side? Is that something you're currently, is that separate in a separate repository that you're doing that, or are you using like something like React on Rails to kind of have that be part of it?

Nikky (00:18:54.01)
It's the same repository. Okay.

Robby (00:18:56.05)
Are you migrating things from Backbone to React or are you building out new things with React and hoping to eventually get back to, uh, deprecating Backbone at some point?

Nikky (00:19:05.12)
Actively migrating things from Backbone to React. It's a long process. This, as our application is a whole set of extremely competent power users who have a lot of really good workflows built up around a few really core pages, we have to take a lot of, a lot of care to make sure that what we build in React is extremely performant and reliable and works from day one. It doesn't interrupt their workflows. Because we make changes and it could have significant impacts to these businesses. So it's a contemplative, long process to move these from Backbone to React. The things we've done so far, it's been most of the application has moved to React at this point. We get really good feedback from, but it takes time to do.

Robby (00:19:49.18)
I haven't honestly gone back to look at the Backbone website in a while to see if there's been many new versions in like a year or two. And I don't remember what I remember figuring out. But I've always like, well, at the time it was modern. Do you feel like the migration to React versus like, I know that like Rails is maybe encouraging things like Hotwire and things like that. Do you think, do you feel like you're going to end up in a similar situation in 5 years or, or do you feel like React has enough momentum on it with their community there that you're not concerned about that anytime soon?

Nikky (00:20:18.02)
Oh, this, this always keeps me up at night. Uh, we've, we've been burned, burned by this too many times in the past. I think the grizzled veterans have always had that in the back of their head of what happens if this framework goes away? What happens if this goes into maintenance mode? I don't think React's going to get there. I don't think it's going to get there for a very, very long time. I think there's a lot of momentum behind it and corporate backing and a community that keeps things going far into the future. And if something does get deprecated to a point, I think we'll have much more notice and information about it. I just think React has done a very good job of keeping everyone calm and centered and not worried about the rug being pulled out.

Robby (00:20:58.10)
Do you get a sense that if you were to bet that maybe in a year from now you'd feel pretty confident that you could task an LLM code tool to be like, hey, migrate the remainder of our backbone to React given the patterns we've already been using? Or have you been able to experiment with that? And I say that because I know that you have you know, a robust QA department there, that seems like that could potentially be maybe a strategy to experiment with if they could test the before and after, make sure things are still working there.

Nikky (00:21:27.17)
Yeah, it's actually something we were talking about last week internally. You know, there's a lot of views in Backbone that are really simple, simple CRUD, simple things that I think an LLM would do just a fine job converting that over. And some of the really core pages of the app, Maybe not so much. But if we can at least do a lot of these smaller things or more routine conversions out of the way, which I think it should be able to do, I think there's enough backbone training data there and LLMs to be able to handle that, it does free us up to spend more time on the really core parts of the app that probably do require a lot more human intervention to get right because they just are so core to the business and how people use it.

Robby (00:22:09.10)
As your team been adopting and leveraging much on the LLM front at this point, specifically within the context of code generation and working on issues and tickets and things like that?

Nikky (00:22:20.10)
Yeah, we've been dipping our toes into the LLM waters, as I'm sure everyone else is to some extent, and it's been lukewarm water so far for us. As I mentioned, we are a large codebase and there is a lot of work to be done informing the agents and LLM models, like kind of how you interact with our codebase, where our helpers where our service classes live, how to find things. You probably don't need to rewrite that method from scratch because somewhere, someone's already written that. You just need to go know where it is. And we found it helpful for code generation, for sure. I think that the big win in my mind is writing specs. It's been really great to kind of have an idea of something that you want to add to your models, your controllers, or whatever else, and you have it write the specs first, kind of guiding that process, write the code, maybe LLM-assisted, and then go back and revisit the specs and ask, is there anything that we've missed, any edge cases that would've introduced, add those to the codebase as well. That's been great, being able to write these really low-level unit testing.

Nikky (00:23:25.11)
I don't have a problem with having too many unit tests personally. They're extremely fast. I'd rather have an issue of having perhaps a little bit too many than a little bit too few, because I think that's where a lot of your low-level bugs have been found. And the second component to that is I found it useful for a really early pull review look at. There's a feature in Copilot in GitHub land where you can request a review from a PR. And that's great for spotting all syntax issues, unused variables, things that maybe slip through the cracks before a human even gets eyes on it. So that by the time a human sees it, they're not looking for, oh, maybe I need to rename this variable a little bit. They're looking at the much bigger structural things, not getting distracted by it. Misspellings or a method not being maybe used right.

Robby (00:24:10.15)
You mentioned that like lukewarm, is that primarily on the results of just the experiments that you're running or is that kind of lukewarm adoption with just your teammates as humans? Is everybody on your team kind of excited to play with these things or is it kind of a little bit of a mixed bag at the moment? And I know that this, their opinions on this might change in a month or two anyways, but, uh, As mine did over the last 3 or 4 months myself. So share, share what you're willing to share, I suppose.

Nikky (00:24:38.04)
Yeah, it's a journey. And I think the initial kind of use case for a lot of these, and I still think a really powerful use case of it is just, it's a rubber duck. It's something that you can ask questions to, get information from. It's a little bit easier to enter some CLI LLM tool versus searching Stack Overflow or seeing some like search result on Google or DuckDuckGo. That's really been helpful for us. We see a lot of usage where things that might have required a group, stand-up parking lot or something to ask questions about, now they're being resolved on their own. Just when they're stuck, they're like, "Hey, I got a question," and Nila might be able to help out. And if it even helps out some of the time, I think it keeps people from getting blocked and helping them unstick themselves from whatever things they're running into. You know, I think the, the lukewarm part is like, we're still not sure about how much new code will be generated on our mainline application. I think LLMs are great for greenfield apps. Like the other day we had a new API and I was able to create like interface to test it out in like, you know, 10 minutes.

Nikky (00:25:41.01)
I knew what I wanted. I just wanted some simple, like, you know, precompiled code to go in and hit this endpoint and show me the results on the browser. And like, you could just, it could do that like no problem. And it's been useful for us to have from our product team too, to create a lot of proof of concepts and play around with how they want the UI to work. Code generation inside our actual mainline application is a hit or miss for us. It can be useful, especially if we have a new user coming online, working on our platform services or working on Ruby, who's not super familiar with those frameworks. But we're still trying to figure out how to get best generate large swaths of code. And like rearchitect parts of our application that would be in a way that doesn't break what we currently have.

Robby (00:26:23.21)
And do you feel like your test coverage there is substantial enough that you feel like there's a lot of confidence that like if things make changes, you're like, okay, we know things are breaking or when you say you're having like lukewarm on code generation, can you tell me a little bit more? It's just like, it's not quite doing what we would kind of expect it to follow existing patterns or what does that actually look like?

Nikky (00:26:45.15)
It's a lot of trying to figure out, in our existing patterns, how to fit all of the paradigms together in a way that makes sense for us. Part of it is that I think a lot of these applications, Rails applications especially, have grown where your codebase is extremely regimented and separated, like in the separation of concerns, which is Great, but it's to an LLM, that means the files that are approximate to whatever you're asking it are going to be way far away. And it's not always able to fully contemplate what you're trying to do in a large-scale refactor. And I think our test coverage is there, which gives us, we're prepared to do it and we have been trying it out and results have been successful sometimes. But you throw it away and you throw it away until you get something that makes sense and then I think it's useful to do a proof of concept inside our monolith application. You do a POC, you make sure it works, you have some tests for it, and then you say, okay, now we can work on it for reals and give someone the POC code and say, hey, this is probably not that great, but this at least shows that it's working.

Nikky (00:27:56.11)
Let's make it better. Let's improve on it.

Robby (00:27:58.07)
Sure. One of the things you mentioned there was that some of the LLM tooling is helping your team maybe answer some questions themselves before they need to go to I don't know, maybe I'm just assuming like maybe you have some engineering help Slack channel or whatever. Do you feel like there's any weird side effects that like your team is less able to bond and help each other as a result? And like this weird, you feel like you're siloing the work a little bit more as a result of that? I'm not saying that as a good or bad thing. It's just like, have you seen any interesting cultural side effects to that?

Nikky (00:28:28.20)
Yeah, no, I was actually just thinking about that a few minutes ago and It is something that I think we're seeing. We're an all remote team. We really foster like pairing and huddles and collaborative question and answer and let's work together to find, find solutions to these problems and kind of help me figure out what's going on. And I think it does give you those familiarity with your colleagues that makes you more comfortable, like asking those questions. And sometimes I think it can be a siloing effect and you know that the answers you're getting from the other person are probably going to be extremely relevant to what your question is. With an LLM, it's making educated guess, you know, it's doing pattern recognition based off of your code base and you hope it's the right answer. And in some cases, you know, you might be the way that you did it 10 years ago that has 0% of your code is not the way we do it anymore, but the LLM doesn't know that.

Robby (00:29:22.05)
Yeah.

Nikky (00:29:22.14)
But your colleague, a Slack Room boy does.

Robby (00:29:25.12)
I do think that's an interesting thing that I feel like teams are gonna have to wrap their head around and be mindful. Like, okay, how, where, where are we, if we adopt these tools and it starts accelerating some things and we spend more time managing and orchestrating these LLM tools, where are we kind of gelling together as a team and helping each other? You know, sure, the marketing materials on all of these tools are like, you're gonna get to focus on these bigger issue things now. And it's like, I hope maybe we'll, we get there. And I don't know, I'm not an expert on any of this stuff, so take my thoughts with a grain of salt here. But I do wonder, like you mentioned, like, okay, well, how we used to do things 10 years ago and there's plenty of code in your codebase that might be like, this is how we were organizing things then. And now we have like a new architecture approach. Maybe we're using service objects for a long time and we're now trying to do something different over here. So we want anything new like these old things to be kind of in this new pattern.

Robby (00:30:25.07)
Trying to manage the LLMs, make sure you keep doing that. It's not that that problem hadn't already existed with large growing teams because I've had plenty of consulting clients that they're like, we've got 40 engineers and we're struggling to figure out how to keep everybody on the same page about like, we're moving this way. But like, when do they decide to refactor existing approaches? When do they start moving towards the new thing? When do you just make 3 lines of code changes to an old way versus let's revisit this and move it over to the new way. Pattern how we want to start moving this. LLMs, I feel like, might expedite some of that, but also, I'm like, how do we make sure that they're doing the same thing and everybody's on the same page there? But tell me a little bit about like your team's experience with like, are there, are you using service objects or in the past, or like how do you organize a lot of that type of business logic in your codebase?

Nikky (00:31:17.08)
I think there's a few different touch points along the way to make sure that your code and your team stay connected with what you want to be doing. And I think the earliest you catch that, the better. As we get epics and feature work to be done, I like engineers going through and scoping those, writing those tickets out, thinking through the potential from a UI through backend and deployment strategy. I think it's really important to kind of touch base with like, hey, this is how we think this should go. Someone on a team will review that. Usually it's me, but it can be someone else too. To get to, yeah, this is the direction we want to go on this. And there's not going to be any ambiguity about how we want to architect this thing. And then the code is written and through whatever means the code is written. And then the final touchpoint is our code reviews. And I view a PR even more important than they used to be because 10 years ago, you knew that this PR was written by a human or perhaps copy-pasted from Stack Overflow. Or maybe like a really proto, like, Tabnine LLM or something was happening at the time.

Nikky (00:32:24.09)
And now it's code that someone has written, perhaps through any means. And that's kind of our final check to make sure that everything is kind of what we want it to do from an organization architecture perspective. And that's where we have Copilot give it a view for like the small syntax stuff. But like, I really expect like any thing that we push out to production, there's going to be an author, and then there's the PR approver. And I view them as equally responsible for the code that goes out because that is our backstop. And it does give us opportunity to say, hey, I think this should be done a different way, or let's discuss this. You use a certain type of service that we're not using anymore, or we don't really do this anymore. And I think it can help spot things that LLMs are making too. Because sometimes you're just so steeped with whatever you're making, you're not able to fully back out and see what decisions it's making. And that really helps having another person in that loop.

Robby (00:33:26.05)
Ever wish you could ship a feature and have someone talk you through it like you're wandering a conference hallway pretending you're just grabbing water? Introducing the On Rails CLI with onrails space generate:podcast. Just run one command and it generates a brand new podcast episode custom made for the exact feature you're building right now. Are you building approvals into your application? You get a 34-minute deep dive on state machines, audits, and regret. Refactoring a controller? Congrats, you've got a calm voice explaining why it grew and a slightly worried voice asking why it's still alive. And yes, it's like having me and Opie Fernandez in your ears arguing politely about the exact trade-off you're about to make because it's trained on hundreds and hundreds of hallway conversations from conferences, meetups, and those late-night, "Wait, how did you do that?" moments. It even adds tasteful sponsor breaks like this one that you didn't request, but honestly, you kind of deserve. Use the OnRails CLI to generate your next podcast, because pairing on a feature is great, but pairing with me and an AI-generated podcast guest is probably inevitable.

Nikky (00:34:22.14)
Make a thing, confident opinions, gentle disagreement, educational, just use Rails energy. Not responsible for shipping features purely because the episode made it sound easy.

Robby (00:34:28.18)
So I'm going to switch gears a little bit, Nicky. So I know that Shop-Ware, we had a previous conversation and kind of like prepped for this, but I know that Shop-Ware has an API that your customers can interact with, right? And that's part of your architecture. So how does that fit into it? And I know that you also handle a bunch of integrations, but can you tell us a little bit about like how your API is designed and how that's been approached over the years?

Nikky (00:34:53.23)
So we have a public API and it's public in that Both users of our application can use it and also some third-party integrators can as well. And the API has also grown a bit organically as well. And when it was first created, the decision was made to have essentially two separate codebases. We had a codebase for the API and we had a codebase for our, like the rest of the Rails application. We regret this decision. I think we regretted this decision almost the minute it was made. And it causes us a lot of grief in that we essentially have a lot of duplicated logic between the two. You make a model change on your main application, well, now your API application has to know about that. If you have a validation, your API needs to know about that. If you make some sort of service helper that does some sort of dynamic calculation, well, your API needs to know that too. That's been a big morass for us that we're trying to unwind in a way that doesn't take our API down in the meantime.

Robby (00:35:50.12)
I'm assuming that you have a lot of models and things like that in your, your main Rails app that don't, don't exist in the API repository. What's been the pattern? Have you needed to use like Git submodules? I've seen a lot of teams use a lot of different approaches to try to keep their models in sync across two different codebases. Or is it literally just like some copy pasting every once in a while of certain methods or a little bit of everything?

Nikky (00:36:15.08)
There's a fancy way and we have some automatic script syncing that will sync over our model code. And our schema code, our schema file. And that's helpful to keep us somewhat honest. And there's also been some copy pasting too, when we need to bring in some additional, like, service level calculators.

Robby (00:36:33.23)
So how tightly coupled is the Rails application to the API layer then? And how does that workflow that if your end user is hitting your API, is that then connecting to its own connection to the same database, or is it connecting as like a read-only type of thing, or? Or is it talking to your main app as well?

Nikky (00:36:51.10)
Yeah, you nailed it. It's, it's exactly that. There's, there's read-only connections and there's also a main application connection as well, which helps us with ensuring that the write access has data integrity rules that we're trying to enforce.

Robby (00:37:04.09)
So if there's like a POST that comes through your API, what does that lifecycle look like for the request?

Nikky (00:37:11.12)
That makes a separate POST request to our internal monolith. APIs that will then like actually make that request. And then usually on the read back, it will come through our like read-only connection.

Robby (00:37:23.12)
Do you have like a proxy or anything between the, to handle that? Or is it just like a different hostname or how does things get routed based off of the request?

Nikky (00:37:32.01)
They're just secret APIs.

Robby (00:37:33.22)
The secret API is that your API is then passing on the, the, the right action to Rails . Your secret API or is—

Nikky (00:37:42.10)
Yeah, like essentially when the write request comes through, the API app is essentially a proxy for the main application.

Robby (00:37:48.14)
Okay, okay.

Nikky (00:37:49.04)
So they just turn— It just passes the request on without a whole lot of other things.

Robby (00:37:53.16)
Are there internal consumers of that API?

Nikky (00:37:56.01)
Nope.

Robby (00:37:56.13)
Interesting. So you kind of, in retrospect, maybe regret doing that. Is there an effort that you've attempted a couple times at all to try to bring it back all into the same repository or?

Nikky (00:38:08.03)
We're in the nascent stages of rethinking how this should be re-architected, but there's a broad consensus on our team that the next time we need to do a major effort inside API, we'll move it back into our main application code.

Robby (00:38:21.12)
I mean, a lot of the other companies I've talked to so far on the podcast have mentioned GraphQL. Has that been part of the conversation at all?

Nikky (00:38:29.08)
Yeah, so a lot of our API work was done prior to GraphQL really becoming popular, so we're Rails , we're still a REST-based, REST-ish based architecture. And I think it's the same model of what we have now is working for us. There's no need to move us to GraphQL. If we're doing brand new, we would choose GraphQL, I'm sure. But what we have is working and there's not a huge push to move things to a new architecture and there's not really a demand inside our ecosystem for that. We certainly gaze at GraphQL with wistful eyes sometimes at the flexibility and capability it can give us.

Robby (00:39:10.05)
I know that APIs tend to reveal a lot about how systems evolve. And were you around when that decision to have it be a separate repository? That sounds like it's probably pretty recent.

Nikky (00:39:20.19)
I was not. Yeah. A little bit of archeology here.

Robby (00:39:25.14)
Can kind of get why teams might've considered that at one point in time, but It's, it's, I can definitely see that being a challenge and maybe LLMs will help solve this problem for you at some point. And you'll just be like, yeah, just migrate everything over. Right. It's going to be super easy.

Nikky (00:39:38.15)
It's very possible. Yeah.

Robby (00:39:40.03)
So another topic was, you know, while preparing for this conversation, I know, you know, you mentioned, I think the engineering team right now is around 10 people. So, but it's also smaller than it once was. Can you remind me like how large of engineering team did it scale up to at one point? Like how maybe at its peak?

Nikky (00:39:56.07)
Yeah, we were probably double that a few years ago. There's a lot of different concurrent efforts to build new things and get new applications up and running and scaled back a bit. I think it's never great losing colleagues and realizing that that can happen. I think you lose a lot of institutional knowledge in that process despite your best efforts. We try to have really good documentation internally. We try to have Everyone cross-trained. We try really hard not to have silos, but it happens. And when it does, I think Rails can help us gain that knowledge a bit back, in that it at least forces you to be a little bit more organized with your code. And so, you're not completely lost when you're trying to figure out something new, and how it's operating, new to you, I should say, and how it's operating. It's always something that whenever you whenever a colleague moves to some new position, you always lose that, and Rails helps us part of the way, but not all the way.

Robby (00:40:59.04)
Do you feel like with a smaller team, it changed how your team prioritizes work amongst yourself?

Nikky (00:41:04.19)
I think it's just less work scales. I mean, as you envision an application, a mature application that's been deployed, you always have X amount of hours that's set to DevOps and DevOps-like things. On a larger team, you could allocate that out more. I mean, engineers are on call less. Engineers expect X number of hours a week, a month, a quarter, whatever, working on that infrastructure work. But infrastructure work is essentially the same, whether it's a team of 10 or a team of 100. And so as the teams get smaller, you're looking for more ways to make infrastructure be as efficient as possible, because it has more of a cost to the business when you are not taking the care and attention to have that be really efficient. And there's always defects as well that come in, and defects are somewhat correlated to the amount of features you're pushing out. So if you're pushing out less features, your defects should be scaling accordingly. But we also have a lot of integrations with third-party vendors and third-party suppliers, and they are also upgrading their APIs. They're upgrading the way that they handle things. Maybe they're an old SOAP-based interface and they're like, you know, it'd be really great to not use the SOAP interface anymore.

Nikky (00:42:18.04)
Can you upgrade your integration that you built a decade ago? And we're like, sure. But you know, that's something that's spread around is now a smaller team spread that around too. But I think as that happens, there's also like integrating with APIs is really, I consider like an LLM's like bread and butter. Like that, As engineers, we're doing that constantly and being able to just say, like, here's an API. Can you kind of give me a sense of like how it works, what the framework is, what we need to do to do this and integrate that is really helpful, like getting us from like, you know, the first step to maybe the first 10 steps and getting that process solved.

Robby (00:42:54.01)
I think that's a really good example of a good use case for anyone listening out there that's maybe a little skeptical of experimenting with LLMs. Like, I feel like that is A really good use case for like, especially if it's like a public accessible API, you maybe don't need to reinvent the wheel. I'm curious, how does your team think about when you might bring in, say, a Ruby gem that maybe already integrates with that API? If, if that is the case, that might not necessarily be the case with your specialist of an industry that you're working in there, but are you able to leverage Ruby gems very often for things like that? Or does your team have an opinion about Ruby gems, some Ruby gems can become potentially a blocker for us in the future if those, because you're kind of taking a little bit of a bet on, will this gem be maintained in 4 or 5 years when new versions of Ruby come out? Or am I going to be the one having to maintain this myself as a team?

Nikky (00:43:44.22)
It's rare for there to be an off-the-shelf gem for us on a lot of these integrators. Like I can't think of an example when we're able to fully leverage that. Which is nice in some respects, because as you mentioned, we do have control over the integration code. We know exactly what it's doing. We're able to configure things the way we want them to, and that kind of best practices, and be very defensive about it. In a way that you don't always get with that gem, or something to pull for that gem, even though it's not entirely what you're looking for.

Robby (00:44:13.14)
No, I think that's fair. And I think I can imagine, as I said, you're working in a very specialist industry where there might be not a lot of other Ruby teams needing to integrate with these different automotive-related industry platforms, I'm guessing you're integrating with. I don't know, can you give me an example of one of those that, what might be a couple of integrations that would be atypical for a typical Rails developer to ever need to encounter?

Nikky (00:44:37.09)
Yeah, I doubt most Rails developers will be purchasing automotive parts from wholesalers in the way that we would be. And I don't think many developers will be pulling in data from motor vehicle lookups. There's this huge amount of data that the auto industry has around standardization of your vehicle where you look up like your vehicle's VIN and there's like IDs associated with the engine, the transmission, the submodel, the trim, your transmission type, like the amount of data that they pull from these VINs is really incredible. And, you know, there's like, 50 people consuming this probably in the world. And there's just a few providers out there for that. And a lot of the, you know, Shop-Ware is one that's a little bit newer on the block. It is cloud-based. There's a lot of the older automotive platforms. One of the earliest applications of like desktop computers back in the ' 0s was automotive shops because they needed to keep track of these things even back then. And so there's a lot of these shops who are using like, you know, pen and paper, notebook systems or they're using Windows XP or Windows 95-based terminal applications and went away, but they still maintain it because that's the system that works for them.

Robby (00:45:57.14)
Yeah.

Nikky (00:45:58.01)
And migration is huge. And it's all those little nuances about how you deal with that and how you bring a lot of these shops, having them recognize there is a lot of value in moving to a newer system. From what they currently have and what they're running.

Robby (00:46:14.04)
Is it safe to assume that if you're onboarding, is it typically like individual shops are acquiring this, or is it typically like an entity that owns multiple shops or a combination of that?

Nikky (00:46:28.09)
It's a combination of that. And they bring in— they'll typically, unless they are like a pen and paper-based business, they're bringing in data from some other system. And so data migrations are a big part of that where we really believe that you'd be able to both import and export your data. And we really try to make it as easy as possible to bring in your data. But again, there's like a myriad of systems, which some of these require custom care and attention to like bring in their data from whatever database they had in the past. And in some ways, like it's not even really possible. Like it's just so far and so different from what we'd be experiencing. But yeah, I'd say it's a combination of, of like really sophisticated shops coming on board and like shops that are just getting started but know they want to be a sophisticated shop.

Robby (00:47:19.23)
So I imagine that if you've got one of the— your team, your organization has like an onboarding process to bring in and help, help them migrate and import their data into The Thing, and that's part of the, the value out of working with Shop-Ware.

Nikky (00:47:35.00)
Mm-hmm. Okay. Yeah, that's exactly it.

Robby (00:47:39.06)
Out of curiosity, is there a lot of very custom code in your codebase that's specific for like, hey, specific customers or clientele need some specific features that your sales team needed to negotiate? And so are there conditionals in your codebase or feature flags for specific— this huge tire company, we need to do this one special thing. For them? Is there a lot of that in your codebase?

Nikky (00:48:04.01)
We try really hard where if there is a request that someone is asking for, we try really hard to evaluate that. And if it's something that makes sense for a larger group of customers, that's something that's just available to a larger group of customers as well. I want feature A, B, and C. You're like, great, features A and C are super relevant to a lot of people. Feature B, we can get on our roadmap and we can think about it. And I'm not too privy about like how our product team is doing a lot of those prioritizations, but we generally tend not to have like tenant or shop-specific code in our codebase because every time you do that, you're like doubling your complexity of your code.

Robby (00:48:42.09)
But it happens a little bit.

Nikky (00:48:44.13)
It can happen a little bit.

Robby (00:48:45.22)
Yeah, I just, I've just seen, you know, working in the consulting side of things, we'll come in and help companies and I'm always like refreshingly like, I suppose, uh, I guess there's like a little bit of comfort knowing like, oh yeah, like at the end of the day, even like a SaaS platform that's trying to present itself as like, hey, this is out of the box, it's going to do all these things, there still tends to occasionally need to be these few exceptions to the rule that they'll do for like, hey, this is something we're going to spend a bunch of initiative on, this one big customer that's going to bring 50, you know, automotive shops into the platform. It's going to be worth it to have this one thing that we might not be ready to roll out to everybody else, but let's get it in there for them so we can get them in the door and then we'll figure out how we can roll this out for other people or not. But some companies don't always have an opportunity to go back and refactor that or roll it out to everybody else necessarily.

Robby (00:49:38.18)
But anyhow, I was kind of curious about that.

Nikky (00:49:41.23)
Yeah, I mean, we use feature flags a lot and we get a lot of opportunity to roll things out to early adopters. And if someone If they'll be pushing for a feature, then yeah, we can roll it out early because of feature flags and it can go into an early access type thing, which is really helpful for a lot of, for betting how that feature works. And then once it all seems good and everyone's happy, then we can roll that out to the larger groups and all the shops at large.

Robby (00:50:06.23)
Are you using any specific tooling for feature flags or is this something home baked?

Nikky (00:50:13.15)
We use LaunchDarkly.

Robby (00:50:15.14)
Okay.

Nikky (00:50:16.01)
Which has worked out really well for us. It took us a few years to get our workflow down and get everything working the way we wanted it to, but now that we're on it, we find it really, really helpful.

Robby (00:50:25.23)
Who gets to control when things get turned on for everybody else? Is that engineering decision or product team?

Nikky (00:50:32.12)
Yeah, our product team does, which is really great because we're able to do really frequent deploys knowing that new UI changes and logic changes are behind feature flags. and so our product team can roll those out, you know, on a per-tenant level. They can roll them out to everybody. They can turn them on and off, and it's completely decoupled from engineering, which is really, really helpful for us. And we still have some, like, permanent engineering flags as well. So you mentioned we have a lot of third-party integrators, and we have kill switches for those. Like, let's say an integrator goes down, and them being down causes parts of our application to be really slow or non-responsive. We can shut that integration off via feature flag, until that integrator comes back online.

Robby (00:51:10.23)
That's interesting. Do you, what's your deployment strategy look like then? Are you, with your infrastructure, is everything kind of deployed? Are you doing like canary deploys or anything like that?

Nikky (00:51:22.22)
When I first joined, we were doing, in theory, biweekly deploys and releases. And that was a process that everyone hated.

Robby (00:51:35.12)
As far as Rails , every other week.

Nikky (00:51:37.18)
And there was a huge QA regression test happened that lasted like a week. And sometimes they would find bugs and sometimes features would need to go in that were promised. And we'd have to shove them in the last minute and hope they don't break anything and delay the release candidates being cut until they were merged in. And there's just a lot of stress. And if you missed that, then you had to wait 2 weeks for the next release. Rails , and bugs didn't get fixed for 2 weeks unless it was a really important bug and you did a special release for it. And that was not working. It took a ton of engineering time. The QA time was exceptionally large. Product wasn't happy because things were being released really slowly. And a lot of trade-offs are being made when you do a waterfall deploy like that. And so we switched a few years ago. We had a pretty big discussion about the the pros and cons of switching to a daily deployment where the code is deployed daily by an engineer, and almost all of our engineers are deployers, which means they're also on call.

Nikky (00:52:38.14)
They're really close to this production server, which is great. And that's worked out really well for us, which you can push out bugs a lot, bug fixes a lot faster. We can push out product changes incrementally as we finish tickets on a particular feature, we'd start rolling those out and that code is being exercised in production early and it can be turned on for early access for testing tenants way early as well. You know, as these things get rolled out, our product team can verify that on the prod environment in a testing tenant really easily. I think it gives a lot of developers another sense that like this is their backstop is gone. Like there's no more, you know, large manual QA regression testing happening. Rails , for each release, like the code that you write, you really need to think defensively about it and really think about the implications it has for your production queries, for your production third-party integrations. And it's really good. I think we are finding people much more cognizant of those trade-offs and being really, really careful about what they do push out. And overall, I think it's been a big success.

Nikky (00:53:44.03)
We spend a lot of time deploying. We spend much more time doing targeted QA against things that actually matter. We are able to even have, I think, decreased defects because we're not pushing code out anymore to meet this 2-week window. We can just push that when it's ready, and we know that it's really confident in that. And if we need to roll back our code, it's just a day's worth versus 2 weeks.

Robby (00:54:09.00)
I really like to hear that as a reinforcer of better, let's say, habits for software engineers to think like, we're going to be deploying more frequently, So the stuff we're working on really matters. We need to show up, do the work, do our due diligence and push things out and know that we're going to get quicker feedback. Cause I, there's a lot of teams out there that, and maybe a lot of people in the Rails community, maybe listening might be like, ah, that sounds wild. Like why would we wait every couple of weeks to deploy Rails ? Makes it super easy. But there are companies that have to slow things down for regulation reasons, stability things. A lot of, there are reasons for that, like legitimate reasons, but if you don't have to worry about like, we need to queue this all up, because it's really difficult to work on, say like a ticket, or you're going to work on a bug or something, or a new small feature change. And it's going to take a couple of weeks before it goes through the QA and gets deployed. And then maybe a bug comes back.

Robby (00:55:05.12)
What was I working on 3 weeks ago? You know, it's difficult versus like, what was I working on earlier today or yesterday? That context is there. So you can quickly like, oh shit, I, you know, I forgot about that one little thing and you can hopefully get that addressed in the next day or so. That's refreshing to hear that you were, your team was able to do that and to do that with a much smaller team. Do you think if there was any correlation to you having a smaller team makes it easier to do that? Do you feel like when you're a couple times the size that maybe slowing things down is a safer thing to do? Or do you feel like it would be ideal regardless of your team size now?

Nikky (00:55:43.10)
I have a couple thoughts, and I think one of the really interesting ones is that, you know, we are, we are in a business that does really value stability. Like these are, you know, auto shops that really, they're, if our application doesn't work, like their businesses don't run, they're not repairing vehicles, they're not seeing new customers, they can't have customers pay them, they don't know where these cars are, they can't order parts. I guess really huge, like stability is is something that we care a lot about, and we spent a lot of time thinking about, and we take all these incidents really seriously, because there's a real human cost to these things when things aren't working right. Also, I think that team size doesn't matter too much when you're dealing with these more frequent deploys, as long as everyone's on the same page and really buys into the idea that the code that you push is really close to production. There is a QA check, who makes sure that, like, the things you're immediately touching is good, and then you're out. And then, you know, you make really sure that your blast radius— you're very blast radius sensitive because you have to be.

Nikky (00:56:50.15)
Because you know that if things go sideways, like, you're gonna be the one who's getting paged, who's getting asked, like, a day or two later about what happened. As you mentioned, it's also really beneficial because you're super close to the code. You release something one day, You see a new Sentry exception the next day, you're pretty sure you know what you did and how to fix it, versus if it's a couple weeks ago, you might be on a whole different epic at that point, and you don't remember anything what you did. Hopefully you do, but you might not. And so, yeah, it just keeps that loop much, much tighter, which we found really, really helpful for us.

Robby (00:57:23.12)
No, I appreciate that. I'm curious about, you know, you mentioned, you brought up Postgres earlier. Have you found, there to be any, any— have you had any scaling issues at all with your database or needed to do anything? Are you integrating with any other toolings for like reporting type tools that your business product teams can get access to? That maybe, uh, I'm just— I've talked to a couple people on the podcast so far where they've talked about they've needed to, to push some data into different platforms so they can extract and build reports a little quicker so that engineers are having to write a bunch of custom reports themselves and some backend interface themselves? How does your team approach that these days?

Nikky (00:58:03.10)
Yeah, so we love Postgres for a lot of reasons. We don't love scaling Postgres for a lot of reasons, but less reasons why we love it. One of the things that you are taught when you create a Postgres database or any sort of relational database is that connections are really precious. and you want to keep connection counts, like, as low as possible. You want your connections to be utilized near 100% of the time. There's that balance there because those new connections are expensive, and especially in Postgres land, they're really expensive computationally-wise, memory-wise. It's a whole thing. And so the kind of the clinical answer for that is you use a connection proxy. Heroku has one. Most things have it. PgBouncer is, like, kind of the big one for that. But what they don't tell you is those can also have CPU saturation and is extremely not observable when that happens. And we've learned over time that maybe we don't fear connections so much as we used to. I remember, you know, 10, 12 years ago, starting with Heroku , there was the idea that you had like, I think it was like 20 connections or some really small limit.

Nikky (00:59:16.12)
And so we tried really hard to use all of the built-in tools that Rails gives you and connection balancers to get that as low as possible. and then you spool up a bouncer. Now things are different in a weird way. And you say, you know what, I'm going to embrace connections again. We're going to go back and make sure that we're using the Rails way of managing connections. And that's okay. And maybe we'll have a little bit more, but we know that our application is going to be stable and not saturating related middleware that causes our application to basically go down in the meantime.

Robby (00:59:46.09)
Quick question. You're running this on Heroku? Is it safe to say that your team is evaluating your hosting options at this point, or?

Nikky (00:59:56.04)
It is very safe to say yes.

Robby (00:59:58.18)
For our listeners, the content, we're recording this in the middle of March, and I think it was maybe 4 or 6 weeks ago, give or take, Heroku kind of made it very clear that they were not going to be investing a lot more development efforts on evolving the platform period. I think it was kind of like, and you can read between the lines, that like that's potentially an issue. So what are, what is your team, you may, maybe you haven't made a decision on that yet, but how is your team kind of navigating that conversation right now? Have you landed on a destination yet?

Nikky (01:00:30.19)
Yeah, that's, that's something we started pursuing last year. I think we all read the writing on the wall after the last few incidents over the past couple years with Heroku not really expanding their offerings, staying pretty limited in what they were doing. It definitely seems like an application that's in maintenance mode, where they're maintaining what they have, but not really adding new features. And yeah, we're evaluating still, and that's something we're hoping to do within the next year or so. But I think that the very obvious answer is something like AWS or something similar, where we can just run on a container and scale up and down. And do all those things in a way that is more flexible for us too. You know, our workload is extremely time dependent. Like, we mostly run United States-based shops, and those shops are open, you know, 9 to 5 across various time zones. And at night, we have this very large, beefy database that is essentially being unused because nobody's repairing cars at 3 AM Pacific time.

Robby (01:01:31.09)
Sure.

Nikky (01:01:32.12)
And so, I think there's a lot of opportunity for us to kind of have a lot more intelligent resource scaling, the most consistent platform is a little bit more modern than Heroku, which Heroku has essentially remained unevolved for the past 10 or 12 years. I think you open it up today, it'd be very familiar to you if you just learned about it in 2010.

Robby (01:01:53.01)
Agreed. As I was just helping a client, we were literally just working on this couple hours ago before I got on this call, I was helping navigate some, some plans for one of our clients there on that front. But, uh, so curious to see, well, maybe we can check back in with you and see how that migration went. Uh, and out of curiosity, is your team using— if you are using, are you deploying with containers at the moment, or is it just using like the default Heroku stack at this point?

Nikky (01:02:21.15)
We've been spoiled and we've just been using the default Heroku stack, and this containerization effort is is in progress because that's like the first step to getting us into something more, more dependable.

Robby (01:02:30.17)
Okay. Get that. Touching back on the database thing real quick on the Postgres, because I know that you're, you mentioned, uh, multi-tenant. Are you doing that all within Postgres with different, how, how do you, do you split up your database at all between different instances or all primarily running in one just huge database at this point?

Nikky (01:02:48.23)
Yeah, it's one, it's one huge database, which There's a lot of benefits to doing that. It all runs pretty well when it runs well, which is almost all of the time. But there's also some scaling things too. As we've mentioned, we have some, some tenants are quite small and some tenants are really, really large. And so you can imagine your indexes can be potentially a little bit unbalanced and large to maintain, unwieldy to maintain. And we've experimented with some tooling around like Postgres partitioning Rails partition tables based off your tenant ID, which caused issues in Heroku's Postgres because we don't have like full access to all their like really fancy administrative tools. So some things you wanted to do weren't really possible. And so it's like kind of workarounds upon workarounds and they're like, this is not really tenable in Heroku land that we were hoping to be able to do in AWS land because that is something really interested in is having those those much more sharded out and be kind of isolated from each other.

Robby (01:03:49.10)
That's something that definitely a number of the guests so far have talked about needing to kind of split those out a little bit and some of the scaling. But some of those clients or those, those, those guests talked about how they basically were reaching the AWS, like Aurora's limit of how, you know, their size limit. And so they needed to split it up because of just because they were hitting like the size limits within AWS itself. And so maybe a different sort of scaling issue on that front. But knowing that your smaller customers could potentially be impacted because your larger customers at the same time could be, you definitely want to be mindful of those types of things. All right, Nicky, I want to shift gears a little again real quick with you. I'm curious, let's just say hypothetically you had a new engineer starting tomorrow and you needed to be gone for the next, say, 6 months. And it's like very competent Ruby on Rails developer was going to join, they have years of experience, they're going to come in and help work with your team while you're gone on your sabbatical, well, much deserved, by the way.

Robby (01:04:45.10)
And you needed to leave them a letter and be like, hey, new developer, let's call them Shannon. Shannon, you're starting tomorrow and typical Rails app, big app, a lot of typical things you might see. But there's this one very atypical thing that we've done that you might want to know about. What would you leave in that letter for them? About what's going on in the Shop-Ware codebase.

Nikky (01:05:12.03)
We have asynchronous update support, which is a very common thing for applications. And we don't use ActionController to do that. Rails is all built pre-Action Cable. And we have a third-party service provider that provides all of this. And it trips up a lot of people near the codebase because I think the assumption is you're using ActionController or you've converted over to it. And we are not.

Robby (01:05:37.19)
What's this third-party service out of curiosity?

Nikky (01:05:41.02)
I think it's called Stream. Just some people, some org that did these things because Rails didn't have it built in.

Robby (01:05:50.13)
I see.

Nikky (01:05:51.08)
We ship it off to them instead.

Robby (01:05:52.18)
What does that workflow look like then? Is it just passing things? Is it outsourcing some background jobs in a way?

Nikky (01:06:01.03)
Yeah, that, that's essentially it. We have to make an async update. We have this third-party code that we integrate that will open the WebSocket and then it listens updates from this third-party provider. And then we push an update to this provider who presumably runs like Redis or something to, to send those, those updates along. So it's, it's essentially ActionCable, but like we're doing it pre-ActionCable.

Robby (01:06:25.00)
Oh, I see. Okay. Fair enough. So I really do appreciate you coming on the podcast to talk shop with us and kind of share a little bit about the Shop-Ware platform. So I think you're talking shop about the Shop-Ware platform. Um, you've definitely shared a lot of things for, uh, with our, with our audience today on like how a small mighty team can still maintain things even with a smaller team than they maybe used to have. And just getting a sense of how different companies are using Ruby on Rails and deploying that out in the real world. So I have a couple of quick last questions for you, Nikki. Is there a non-technical book that you find yourself recommending to teammates or peers over and over?

Nikky (01:07:01.15)
That's a really good question because I actually have it right here. I wasn't expecting this question to be, to be asked, but I really like a book called Digital Apollo. It's about the design and architecture of the Apollo DSKY, so the guidance computer that was conceived and created in the 1960s, and kind of one of the first pieces of computer hardware that was both mission-critical and really well known to the public. I love it for a lot of reasons. I think it's a great text, kind of reminiscing of where we came from, and how these decisions that we're making today are still really relevant to ones that are being made like 30, 40, 50 years ago. And the things that our predecessors have done are still informing us today.

Robby (01:07:49.05)
Who's the author for that?

Nikky (01:07:50.16)
Let me see that.

Robby (01:07:52.06)
Let me see that book. We got video here.

Nikky (01:07:54.18)
It's just a, I don't have like, we take off our card covers, so. Okay. This is called Digital Apollo by David Mindell.

Robby (01:08:03.02)
Okay. I'll definitely include a link for that for our listeners in the show notes. David Mindell. Excellent. And where can folks follow, does Shop-Ware, does your team have like an engineering blog or anything of that sort? Do you stay active on, online?

Nikky (01:08:18.11)
We, we have no blog. We have no socials.

Robby (01:08:21.15)
Perfect. Well, I'll definitely not include a link to that. I'm sure it's for everybody. And with that, Nicky, it's been such a great conversation. Thank you so much for stopping by to talk shop with us on On Rails.

Nikky (01:08:33.07)
Yeah, great. Thanks, Robby.

Robby (01:08:37.19)
That's it for this episode of On Rails. This podcast is produced by the Rails Foundation with support from its core and contributing members. If you enjoyed the ride, leave a quick review on Apple Podcasts, Spotify, or YouTube. It helps more folks find the show. Again, I'm Robby Russell. Thanks for riding along. See you next time.