[00:00:00] Robby Russell: Welcome to On Rails, the podcast where we dig into the technical decisions behind building and maintaining production. Ruby on Rails Apps. I'm your host, Robby Russell. In this episode, I'm joined by Ryan Stawarz and Austin Story from Doximity, where Rails has been at the heart of their platform for over 15 years.

[00:00:20] Robby Russell: We talk about how they've scaled a single Rails monolith to support well north of a hundred engineers, alongside a few supporting services and why Rails remains one of their secret weapons powering a mobile first platform used by healthcare professionals around the world. We get into how their front end approach has evolved from Backbone to Vue and what led them to adopt GraphQL to better support mobile clients and cross team collaboration.

[00:00:42] Robby Russell: They also share how they approach debugging at scale, evolving patterns across a large organization and what Rails still makes easier even after all these years. Alright, check for your belongings. All aboard. Welcome to On Rails. To start, I'd like to ask each guest, what keeps you on Rails these days? How about we start with you, Ryan?

[00:01:03] Ryan Stawarz: To me it's just, you know, I think Austin probably says something very similar here, but it's, it's just a proven path, right? It allows you to take all the good things and then when something you need isn't there, you can customize, right? So there's a very clean path for integration, for, for going off, per se, the Rails for the things that are custom to your business, logic, your needs, whatever's possible.

[00:01:22] Ryan Stawarz: But it's from what from I've seen, and I keep playing with every platform out there, every kind of package that comes out, and I haven't found anything that has the same comprehensive set of proven paths that allows you to just go from zero to a hundred and then customize where you need it.

[00:01:37] Austin Story: What about you, Austin?

[00:01:38] Austin Story: Yeah, so for me, I think it comes down to the philosophy of the framework, the Oma nature, the Rails doctrine, all of the things that, you know, whenever DHH pushed this thing out, you know, he kept pre tight reigns initially on making sure that it was. Inclusive of the things that matters and adaptive for the things that people need to change.

[00:01:59] Austin Story: And you know, there's been a lot of really excellent core members that have helped push that philosophy along, but being able to take an idea from start to finish to enterprise with the same set of tools that most people know that you bring on is hugely empowering.

[00:02:16] Robby Russell: You know, I'm curious, Ryan, you had mentioned, you know, like a proven path and you sounded like you, you've experimented with a bunch of different frameworks.

[00:02:23] Robby Russell: Are you just the kind of person that likes to go experiment with new things on a regular basis and, and that, is that kind of like your approach or how you think about professional development or anything for yourself? Or are you just like a curious, like what are, what are people working on? A

[00:02:36] Ryan Stawarz: little bit of both.

[00:02:36] Ryan Stawarz: And it's always professional development, right? Like I've been in Ruby for a long time. I used to do Java, low sleep plus plus, right? For, and honestly, like I go out there, I'm like, there's gotta be something else, right? Like somebody's created another framework that's gonna be able to compete with Rails and you just don't see it in the same way of level of productivity.

[00:02:53] Ryan Stawarz: And there are niches of things that you, you'd wanna use for, for different purposes, right? But when you know you're developing for a longer term or, or when you don't know exactly potentially where you're headed, Rails gives you everything you know you're going to need. And as I said before, you can adapt, right?

[00:03:10] Ryan Stawarz: So it's not like I'm looking to replace Rails or look for a different framework altogether. I'm mostly curious and it's, other frameworks are good ways to pick up other languages, right?

[00:03:19] Robby Russell: What about you, Austin? Do you try to keep up, or you're like, no, I'm pretty happy over here until someone says, Hey, I found this amazing new thing that's gonna, like, aid excess or something.

[00:03:27] Robby Russell: I'd say I'm kind

[00:03:28] Austin Story: of in between that. I still experiment, at least like my professional learning. I find that I, I go at bursts like times, whenever I'm really concerned with like Rails, there's times that I'm really concerned with JavaScript or go, and then every time that I come back into Rails, I really appreciate, you know, like, like a train, how on the Rails, like, you know, there's just so much stuff that I don't have to think about whenever I'm in that domain.

[00:03:53] Austin Story: So I do try to take things that other people are doing, integrate them into the various concerns that like my team has in order to make our overall organization go faster. But I, I am very happy with Rails at this point. I've tried so many other things and nothing resonates in the same way.

[00:04:12] Robby Russell: I think this is a good time to pivot over to, I want to talk a little bit just to give our audience some context.

[00:04:19] Robby Russell: You know, I know you both have been at Doximity now since what, I think 2016, and you started the same year within maybe six, nine months of each other, I think, or something like that. And prepping for this conversation. But just to give listeners a bit of perspective, approximately how big was the engineering team back then versus what the size of it is now?

[00:04:39] Ryan Stawarz: I don't, I never kept tabs on numbers. Right. In terms of engineering team, you know, are we measuring web engineers data? That's fair. You know, things like that. So 30 to 50 maybe tops. I'd say 50 seems high, 30 seems low. So somewhere in there

[00:04:54] Robby Russell: I'm thinking maybe number of people that are probably touching the Rails applications, maybe, if that helps.

[00:05:01] Austin Story: Yeah, I'd say less than 20 ish. Yeah. That sounds about right.

[00:05:05] Robby Russell: Is that still true today?

[00:05:06] Austin Story: Yeah.

[00:05:07] Robby Russell: What are we looking at now?

[00:05:09] Austin Story: Probably around 200 ish, somewhere in that area. Wow,

[00:05:11] Robby Russell: okay.

[00:05:12] Ryan Stawarz: Yeah. Touching Rails. A hundred, 150, right. I don't think we have 200 Rails engineers, but I could be wrong on that too.

[00:05:20] Robby Russell: I love that.

[00:05:20] Robby Russell: Like no one really knows how many people are working on their software projects, but I mean, you can go look at your GI commits and probably try to figure out a way to uni identify them and who knows. But you know, so for listeners who might not know that much about Doximity, we'll link over to a case study and you can go look up Doximity to see more about that.

[00:05:36] Robby Russell: But just thinking, I wanted to focus a lot more on like how your team approaches things there, how you've navigated, like the app's been in development for well over a decade at at least, right? So, and, and I don't know if you know what version it might have started on, but for, I know that Doximity has a large Rails monolith and has also been using GraphQL for several years.

[00:05:56] Robby Russell: I know, I don't remember if that was something that was decided before either of you joined or not, but maybe Ryan, you can touch on this first, maybe. What was the story of how GraphQL first got introduced there? What made you both take the lead on that? And I think because you were part of that process, right?

[00:06:10] Ryan Stawarz: Yeah. I think the impetus there was our migration to Vue as well. Hmm. Right. So we were, we were separating out our front end to more, towards more of a single page web app to be, you know, as most teams that started at the time, started with different versions of JavaScript frameworks, if you will call that.

[00:06:28] Ryan Stawarz: Right. We outreach a query. We dabbled with a couple other frameworks, Backbone coffee script or things to help a Rails engineering team adapt to JavaScript, which is a challenge for some developers, right? And so everything across the board, we always ended up adding interactivity to our pages, our webpage, right?

[00:06:47] Ryan Stawarz: And so it always just ended up needing some APIs and we always needed some JavaScript on that front end. And it was never a perfect, not perfect, nothing's ever perfect, but there was no real like great paved path, right? There is, you know, dabbling of JavaScript around. And we wanted to finally pick a richer framework, right?

[00:07:08] Ryan Stawarz: So we did a experiment between, I think it was Angular, Vue and React at the time. And we, we broke across teams and did some prototypes. So that was the impetus, right, of like, okay, well we're gonna create the single page web app. How are we gonna standardize the backend of that? And I think GraphQL was our answer there, right?

[00:07:25] Ryan Stawarz: We were exposed to. We, in our history of our app, we have in the monolith, which we have different apps now, but we have a v1, a v2, a V three, and they're all different flavors. They're very similar in nature. And people, because of the team across their diverse products, people come in, they add, they changed.

[00:07:43] Ryan Stawarz: There wasn't a great way to kind of keep that all in sync, especially if like, okay, if we return a user over here, what is that over there? Right? And we had type issues and we saw, I saw GraphQL as a great way to kind of help us constrain that and keep a single nice, uh, interface on top of that or provide one to our, our native apps to our single page apps in our app, you know, communication if we needed it.

[00:08:07] Ryan Stawarz: Um, and that was kind of the, the, from my perspective, the, the driver Austin, I don't dunno if you recall it differently, right? I tend to

[00:08:15] Austin Story: package this in terms of mobile. I remember Ryan and I read a, an offsite. Waiting on a decision for the backend that we were gonna use for this new Vue application. And Jay, who's just smarter than me, made the decision, Hey, I, I think this is better.

[00:08:30] Austin Story: We should go this route and see how it ends up working. But over the years of talking to, to mobile developers, like one of the hardest things for a Rails application is for somebody to iterate on the backend without accidentally breaking something and crashing the mobile client. And our business is, is primarily a professional medical network, right?

[00:08:54] Austin Story: And that's doctors and PS, physician assistants, they're very busy people. They do use desktops, but the majority of them meet us where they can, which is on their phone, in between visits with clients, you know, or you know, when they're after hours in order to keep up with the news and stuff like that. So keeping the mobile app.

[00:09:15] Austin Story: Running and not crashing was a problem that we had well before I joined Doximity. Just because of how easy it is to iterate on those REST APIs, you end up introducing breaking changes. You don't know what specific flags do. It's really complicated and GraphQL is also complicated, but it does add a layer in between the changes that you're making and the way that mobile's going to be requesting those so that you can make a change and get alerts whenever something's going to break and be like, oh, that's great.

[00:09:45] Austin Story: Thank you for not allowing me to accidentally break our entire iOS client. You know,

[00:09:51] Robby Russell: I'm imagining that the GraphQL interface in that situation, sitting within that monolith Rails app

[00:09:57] Austin Story: initially,

[00:09:58] Robby Russell: yes. What sort of tooling did you use to to accomplish that?

[00:10:02] Austin Story: GraphQL-Ruby

[00:10:03] Robby Russell: And for those that haven't really done a lot of experimenting with that yet, so I'm thinking like a typical Rails app you've got, you know, maybe your approach that you mentioned, like you probably have REST Endpoint and your mobile app was trying to use that.

[00:10:14] Robby Russell: Or maybe at one point and you're like, well this maybe, I would imagine some of the problems there was like trying to sync up people working on the mobile apps. They're like, well we need these things and where do we fetch this? Which REST endpoints And then we gotta call this, and then maybe then you're fetching another endpoint in GraphQL.

[00:10:29] Robby Russell: Let you kind of decide what you wanna pull on that initial grab. Do you feel like Rails is set up well for that? I mean, it's definitely

[00:10:39] Ryan Stawarz: a mind shift and it may or may not fit directly into Rails, depending on your data set is the way I would say it. Right, because. Due to the nature of GraphQL, right? You don't know exactly what the client's gonna ask for, right?

[00:10:52] Ryan Stawarz: So there's a set of data that they could ask for your entire graph, right? And obviously there's constraints in the graph to say, well, don't, you can only return so many nodes, you can only go so deep, right? You, so you can configure all that, but that complexity aside, you really don't know ahead of time what query is going to be made theoretically, right?

[00:11:07] Ryan Stawarz: And you can constrain it. And so I'm not gonna say, you know, there's no way to do it. But generally speaking, you don't set it up that way because that's one of the value adds. So you can easily get into a situation where you're adding n plus ones in places where you didn't expect it to, or like you didn't think about ahead of time.

[00:11:24] Ryan Stawarz: 'cause you didn't expect somebody to make this query that way, theoretically, right? So from a Rails perspective, when you hit a REST API, you know, here's what you're gonna be returning, you know, ahead of time. You should do some analysis of identifying, okay, we need to include this, we need to join this, right?

[00:11:39] Ryan Stawarz: So forth and so on. In GraphQL, you don't necessarily know that. So depending on the data you're exposing and how you're exposing. You may need a different approach. And we ran into this pretty quickly and we, we started using batch loading in certain places. Right. And depending on what was being used or why we would use a batch loader mechanism to, to fetch you data asynchronously in a way.

[00:11:59] Ryan Stawarz: It's actually a really simple pattern. Um, but it's, it's really powerful too.

[00:12:03] Robby Russell: Batch loading. Is that from the, on the client side?

[00:12:06] Ryan Stawarz: Yeah, on the server side. There's a

[00:12:08] Ryan Stawarz:  batch loader.

[00:12:09] Ryan Stawarz: Am I right about the Austin?

[00:12:09] Ryan Stawarz: Is that the name of the Gem?

[00:12:11] Austin Story: Yeah, there's, there's several. The one from Shopify I think is Batch Loader.

[00:12:15] Robby Russell: So how does that work? Is that basically fetching some and then kind of streaming the data back through GraphQL at that point, or

[00:12:21] Ryan Stawarz: No, not really.. So what it actually does is defer the loading right. Um, it's kind of using a block mechanism. It's really clever. If you look at the library, it's not deep science.

[00:12:30] Ryan Stawarz: It just takes a little, your head to wrap around some of the asynchronous of it, but it effectively defers making the query until you actually access the result. So one of the things important to GraphQL, you have to understand the lifecycle of GraphQL request because you go through, you fetch all the data, you go through and actually parcel the nodes and the query, resolve all the nodes.

[00:12:51] Ryan Stawarz: Then you render all those nodes, right? And that's one way to go depending on the library, right? But from a GraphQL-ruby perspective. It may begin that, not exactly right, but the intention is like you go through and you resolve, you identify all the things you wanna get. So during that first step, first of going through that treat, it says, okay, well you want this field, let's go batch load this guy, and it puts it kind of to the side until you actually try to resolve that, until you actually render that to JSON, then it will trigger those calls and asynchronously, or in the background if you got the background, because it's not threaded, so I'm saying it wrong, but effectively it will defer those calls until you actually try to access one of the notes and then it'll fire off all those relevant queries.

[00:13:30] Ryan Stawarz: So it builds up a set of like IDs that you're trying to load effectively and then makes that query all at once.

[00:13:36] Robby Russell: I see. And then, and then, then I'll send that back to the mobile app or whatever your client is in that point. I'm curious about, is almost all of the, your client apps all now using GraphQL or are you still maintaining RESTful any APIs at that point?

[00:13:50] Robby Russell: And did you kind of just migrate away or, or are you kind of doing this in parallel and having to just kind of support both approaches or even more approaches?

[00:13:57] Austin Story: Yeah, we, we still have some mobile applications. I mean, we still have REST, you know, we're not going to get away from REST. Sometimes REST is the, the best choice for what we're doing in general.

[00:14:08] Austin Story: You know, if you talk to a mobile developer, they will tell you, use GraphQL for this. Because I can go in, I can see exactly what's gonna be returned. I could be confident that what's being returned is gonna actually fulfill the contract. And we're able to, to use that in order to bake in a bunch of primitives about request response types, uh, error codes and that sort of stuff that are a little harder to do on the, the REST side of it.

[00:14:34] Robby Russell: I know that many teams will reach to a point where you say GraphQL starts to grow beyond us, say a single schema. So can you walk us through why and how Doximity transitioned over using, is that GraphQL Federation?

[00:14:46] Austin Story: I gave a talk, uh, about our move to GraphQL Federation at RailsConf 2023 And you know, for us, if, if you're looking at GraphQL, it's very simple.

[00:14:55] Austin Story: It is a endpoint that you expose and clients make a request to that endpoint. And if you think of it like a a j response type, you just tell it the keys that you want off of that. And so what the backend is able to do is able to assemble that data however they need it. You know, and that's really great for, you know, places with a whole lot of applications so that they don't have to build, you know, all of these artisanal backends for front ends or, you know, hundreds of different REST endpoints.

[00:15:24] Austin Story: People can grab what they need. For us, that wasn't really the selling point. It was the, the type safety between the mobile clients and, and the Rails application. But what ended up happening is GraphQL was a big head for. Mobile specifically. And because we're mobile first, you know, this was the, the mid 20, 20 10, 20 16, 20 17.

[00:15:44] Austin Story: So many teams started to use this more. You know, the initial push was on Ryan's team to expose like profile pages and, and that sort of stuff ended up working out really well for them. And because we were building this, you know, this Vue side that was forcing the build of GraphQL, we already had mobile support for everything that we built out outta the box, right?

[00:16:04] Austin Story: The APIs were already there. They were gonna use the exact same queries and all that sort of stuff. But as we got more of our teams involved, you know, at this time, like we were really growing the engineering organization, we started to add other business domains there. And as those business domains came in, you know, we had this great tight interface between, Hey, mobile, you know, you're gonna make a request, it's gonna come back from Doximity, it'll work exactly how you expect.

[00:16:28] Austin Story: But then we had to make all these artisanal decisions of like, how do we map in external microservices into Doximity? Can you gimme an example of one? Yeah. So like Doximity, the monolith is what I'm talking about. So how do you in this monolith expose our newsfeed, which is, you know, like a, a stream of, you know, news articles, cards, comments, uh, you know, even ads, that sort of stuff.

[00:16:55] Austin Story: How do you get that into Doximity? And that's where it started to get really complicated. 'cause Doximity does a lot like a, a whole lot. And it was also serving GraphQL. So anytime that a team that didn't have a direct data connection in Doximity, like, you know, wasn't associated with the database or elastic search or something like that, they would have to change their microservice for, you know, their service.

[00:17:18] Austin Story: Uh, most of them exposed a gem, so they have to change their gem and then they'd have to go into Doximity, the monolith and then map those gym things into the types in Doximity. And so that was like the dev. Ergonomics part of it. For teams like Ryan's it, it wasn't, you know, as big of a hit, but for other people that I know, it was a huge hit.

[00:17:39] Austin Story: 'cause they had to make four changes in order to do something that should just be super simple. And then the deaf part of it was not great for those folks. We could have probably limped along with that, but I'd say the bigger thing was resiliency. So because we had all of these transitive dependencies, so Doximity depended on app B, and then app C depended on app B and it was just kind of like a web.

[00:18:06] Austin Story: And what really broke the camel's back for me was when we had an outage for one of our teams that was completely unrelated to anything that they were doing. Like their team was just humming along and just because of some constraints and how many web requests you can have for the way that we were, you know, scaling at the time, it took their service completely down.

[00:18:27] Austin Story: Oh wow. Because of the change. On the database on another team side. We still had a couple apps that were working, but like for all intents and purposes for the physicians using the side, it was now because of the A database migration. So that resiliency issue was like the big thing, you know, trying to pull out the GraphQL portion.

[00:18:46] Austin Story: It's like Doximity could do everything that it does the monolith, but if we could just get it to not do GraphQL, 'cause it was essentially acting as GraphQL Federation right there. You just had to map everything in manually. It's like if we could move that out, that'd be super great.

[00:18:57] Robby Russell: Ryan, I'm curious, did that change how going towards Federation changed how your team thinks about deploying or anything on your end or code ownership?

[00:19:05] Ryan Stawarz: It changed, I think, how we think about services moving forward. It allowed a lot of different opportunities, right? So we had things in our monolith that we always wanted to extract out, but there really just wasn't. A huge value add. They didn't belong in the monolith, but the most of the data was there.

[00:19:22] Ryan Stawarz: And if we extracted it out, we would've had to then create this API layer to talk through Doximity to the GraphQL. Like, so it had this experience. But as soon as we removed that and we had federation, it allowed us to say, well, you know what? We can extract this service out that's totally independent of the monolith.

[00:19:37] Ryan Stawarz: Pull that data outta that database, you know, and now this thing, this other entity, this other service is now has its own resiliency. 'cause it, it can do its own thing a hundred percent. If the model goes down, we have no problems. Right. Which, it's got large uptime, right? It's a really big thing, but when it does, it affects lots of teams.

[00:19:53] Ryan Stawarz: That's not what you want, right? So it did allow us to change how we approach building either new apps and extracting out like our opportunities for saying, well now it makes total sense. Let's live here. Let's take on a project to move that over the next time we revamp that up, or add a new change of UX where it might be we'll take on the, the 30% of time work to, to go move up, you know, make those changes.

[00:20:12] Ryan Stawarz: We need to add resiliency and, and, uh, help out overall. But I would say that the biggest thing it did is as we add new products. We can spin those up as some people think of microservices as well. That's the purchase flow, that's the products flow, right? We don't necessarily approach it from that perspective.

[00:20:28] Ryan Stawarz: We approach it from a product driven perspective of this thing top to bottom can operate independently and so therefore we want it to be its own Rails app. And sure we might get data from somewhere else and we have ways to sync that data over, push that data into those apps, but it, it's always asynchronous so that that secondary service can stay up on its own.

[00:20:48] Ryan Stawarz: And that's really what federation's given us is that ability to launch those products and have that resiliency from day one of like, this is a totally independent service.

[00:20:57] Robby Russell: You know, you mentioned on around the aspect of when you decide to say extract something versus now you might think about like you might decide to not even maybe initially put something into the monolith.

[00:21:08] Robby Russell: Where do you see that line now of like, okay, let's, you're kind of touching on that a bit there around like, well, this seems like it can be a self-contained thing and there might be some shared data, but like, is there a point when you're doing it a little too prematurely or not? Or like, we're, is there like a, is there like a checklist that your team runs through?

[00:21:22] Robby Russell: Like, here's the questions we need to ask ourselves. Like, default we're going to put it in the monolith otherwise unless X, Y, and Z, or is it, we're not gonna put it in the rail step unless x, Y, or Z? How do you kinda approach that?

[00:21:33] Austin Story: It really depends on the context of what we're building out. And I know that's a pretty squishy answer, but if we have something that needs to get out there fast and it needs access to specific types of data, it goes in the monolith and we use, uh, the Packwerk gem, which is, you know, a way to kinda like isolate those quasi domains inside of just a, a single application and.

[00:22:00] Austin Story: We built out a bunch of tooling that allows people to basically do everything that they need to do inside of those packs. And I think that's an excellent pattern. Until we know that something is going to live a long time, it doesn't make sense from a maintenance perspective and a setup and deployment and all that sort of stuff, because those things have real costs.

[00:22:19] Austin Story: You know, every time there's a Rails upgrade or a Ruby upgrade or a package upgrade, you have to touch in apps in order to get that out there. And you have to balance that maintenance with what you're doing. And if what you're doing is tiny and isolated and it can live in the monolith, and you're okay with that, that tie to like, Hey, when if the monolith is ever down, I'm down to, if you're like, okay, that's fine, because we don't wanna spin up a new application.

[00:22:45] Austin Story: But then there's the flip side of that, which is, Hey, we have this app, we have. This app called Residency Navigator, and it allows physicians to come in and see, hey, what residencies are available for me whenever I, I graduate from medical schools, that I can go start my career and shadow the people that I wanna shadow and learn what I wanna learn.

[00:23:04] Austin Story: That app is going to change for its own reasons, and you want the uptime of that to be tied to itself. So like, that's an example of one where we want to put it there in a, a separate service in its own domain. I, I don't like microservices. I, I think it was just a domain, like if it's a, if it's the self-contained, logical unit that, that serves a business purpose.

[00:23:26] Austin Story: Like, like that's generally a good service.

[00:23:28] Robby Russell: Does that sort of process, 'cause it's like, sounds a little squishy in, in some ways, right? Of like, well it's kind of like a, is there like a hunch or does your people on your team teams have to like, propose something and like, this is like, how does, how does that kind of decision end up happening within your organization?

[00:23:45] Austin Story: So I'll, I'll talk about my teams. We by default put it in stuff that we, we already have. And if somebody has a compelling reason for spinning something up, it has to be compelling. Uh, just in aggregate, you know, like we have smart people that I really trust on my team. If we agree that it makes sense to maintain a completely separate application for this thing, then we can do it.

[00:24:07] Robby Russell: What about in your world, Ryan?

[00:24:09] Ryan Stawarz: I would wholeheartedly agree with that, right? Um, when we can incubate it, it's the way I like to say and prove the business case. It's the better way to go, right? With some controlled isolation to how often, right? We try to put it in a little package, if you will, rights knowing that either it's a gonna die and we need to rip it out, or B, we're gonna extract it to its own, its own place eventually.

[00:24:28] Ryan Stawarz: Right. Um, there are some caveats that you asked for a checklist. I mean, one of the things that we'd probably look at is like, if this is a separate service that we're spinning up that contains. Either patient or medical data, right? Needs some special HIPAA compliance, right? We will might say, okay, you know what?

[00:24:42] Ryan Stawarz: We should just start this as a separate service so that we have isolated logging and so forth and so on, right? That would be one special case we'd consider. Um, but for the most part, um, there's a couple things I'd say about Doximity is one, we bridge incredibly pragmatic, right? Like we move fast and you know, and sometimes we get it wrong and we're okay getting it wrong, right?

[00:25:00] Ryan Stawarz: Knowing that, well, this is the best decision right now, so let's just move forward so we can get to the next step and realize, oh, we got it wrong now let's go fix that. Right? And getting it wrong is a good thing. 'cause it means we're in the market, we tested the product, we're doing all the right things.

[00:25:12] Ryan Stawarz: Like we still have that startup mentality of like, what's the quickest thing we can do to get this in somebody's hands so we can prove that we can, should continue doing this rather than an extra six months of infra work. It's, it's not gonna take that long, but I'm being dramatic. But

[00:25:27] Robby Russell: that makes a lot of sense and that really lands, I'm, I keep thinking about how when I talk to different teams.

[00:25:32] Robby Russell: Things that they're, they're often kind of trying to wrap their head around. It's like, well this is the best way of to do something. And it's kind of go back to the, you just being clear about being pragmatic. You're gonna make some mistakes. You can revisit these things. And do you feel like maybe when you put things into, say the monolith you, you have a little bit more flexibility to, you're not having to spend all that other time getting all the other things that would need to get set up to deploy another app.

[00:25:55] Robby Russell: Even if you have that kind of optimize, it's still yet another thing to, to handle and to name, to add new documentation. Still quite a bit of work there, but it's like another thing to add to a long list of additional things to be responsible for an organization. Is that something you feel like as people join your organization, they're able to kind of understand that in a pretty quick amount of time?

[00:26:17] Robby Russell: Or is it just something that kind of ambiently they pick up on over time? Like, well this is seems to be how our department or our teams approaches these types of things versus maybe how they've done it elsewhere.

[00:26:28] Ryan Stawarz: Mean, my answer is I certainly, I believe so, and I certainly hope so. People will coming on of the people I've talked to and they do bring people on, right?

[00:26:36] Ryan Stawarz: There's no hard and fast rule, right? What you would think would be scary for an enterprise of our size. But the reality is right, we have great wiki tons of documentation of, of like where things are at and it's easy to find and people are always amazed when they come on and they search something and they can find, you know, maybe it's not up to date as of yesterday, but it's reasonably up to date and it gives you on the right path and the right person to talk to you.

[00:26:59] Ryan Stawarz: Um, and I think that's what gives us a lot of headway in that direction because again, we're pragmatically move fast. So even if we document everything perfectly, it's wrong three days from now, right? Because we just changed our mind and we're doing something else. And that was the right thing to do then.

[00:27:12] Ryan Stawarz: And now it's not right. I, I can't emphasize that enough. Right? We do things the right thing right now and then we move forward. I think, um, I don't want say specifically, we had a recent product where we launched on a particular backend database, a particular, whatever you wanna call it. We knew click, knew it wasn't gonna scale.

[00:27:28] Ryan Stawarz: Switched again. Two months later, we realized that one wasn't gonna scale where we needed to be, and we did it again, but we had the best choice at the time. 'cause we got us moving forward, getting the direction teams moving, product moving forward. Right. And so is it ideal? No. But is it better than spending three months analyzing the best possible data store that we could have used at the time?

[00:27:45] Ryan Stawarz: Absolutely.

[00:27:47] Robby Russell: You know, Austin, you mentioned that, you know, there's been scenarios where you, it's like maybe rolling out some new features might require several things to get deployed. Maybe your monolith needs to get a deploy added, maybe some other services as well. Can you share an example of like how you ultimately reduced that friction?

[00:28:06] Austin Story: So specifically with, with GraphQL?

[00:28:08] Robby Russell: Mm,

[00:28:08] Austin Story: so this was, was that project? Yeah, I'd be pulling that out to GraphQL Federation. Was the thing that allowed us to do that. So the, the way that GraphQL Yeah, at, at its root works is it produces a schema that's kind of similar, like Schema rb. It's just an artifact of like reflecting on what it can do.

[00:28:26] Austin Story: And GraphQL Federation allows different services, different domains to export a schema and then assemble those. And then it acts kind of like, I don't think it was like a telephone book or, you know, like, like an, an edge router where it's gets a request and it says, okay, the, the Jason document, you know, the keys to that says, Hey, uh, I want a user with a name.

[00:28:49] Austin Story: And then it looks up and its phone book and it says, oh, well the Doximity monolith responds to that. I'll send that request down there, and then it comes back to federation and sends it back to the client. And whenever we did that, these teams, you know, they got the option to make that choice of, you know, do I want to be coupled.

[00:29:08] Austin Story: To other services, or do I want to be able to implement my domain and be independent from everything else? And that's, that's just hugely empowering. It. It's something that I'm, I'm passionate about giving people the tools that, you know, if they want to opt into this path, they can. And, you know, the, that team that was doing the four ships, they still have to ship their service, but after they ship their service, we have automation that automatically gets that schema up to production.

[00:29:35] Austin Story: And then we also handle like the staging and development cases, which are also critical when you're pushing something like that out so that they can make a change and their service and it's available and they make one change and they're done Time.

[00:29:48] Robby Russell: This episode of On Rails is brought to you by accepts nested attributes for, because one form just wasn't enough.

[00:29:55] Robby Russell: It started simple. One model, one form, life was good. Then someone said, actually this needs to update associated records too. And just like that. You work deep in the weeds, nested fields, nested paras, nested regret. That's when I reach for accepts nested attributes for one line in your model, suddenly your form knows how to update child records.

[00:30:17] Robby Russell: No controller gymnastics required. Is it perfect? No. We'll confuse feature you almost definitely, but it works and it keeps your code a little cleaner, at least on the surface. Accepts nested attributes for, because sometimes your form needs a form inside it.

[00:30:35] Robby Russell: I'm gonna pivot and talk a little bit about realtime, say notifications like that. So how does Doximity handle realtime features like notifications or are you using tools like ActionCable or AnyCable or Kafka tools like that?

[00:30:49] Ryan Stawarz: Yes. How do we handle real time? So couple services handle it really well. I would say most of our enterprise doesn't.

[00:30:57] Ryan Stawarz: By design, if that makes sense. In other words, and I have a team going through, working through a feature of this actually from next quarter, I think coming up here. And we're pushing Austin's, I think it's Austin's team to, to set up some more n cable infrastructure for us, right? So we do use n cable, we particularly use it for our dialer products, which is where real time is critical, right?

[00:31:17] Ryan Stawarz: It's, it's our video and voice platform and having, you know, that messaging and connecting different disparate clients together with realtime audio and realtime video is critical. And the other places where we're saying we do fantastic push notifications, right? Which we rely on, I believe Firebase, and we have old source around that.

[00:31:34] Ryan Stawarz: That's one type of real time. But the kind of communication you're talking about, we do have some chat and, and fax services that push notifications out. But if you're on our platform and you have, uh, let's say the messaging client open on the web, we're actually not really doing a great job at the moment with messaging, pushing out real time updates.

[00:31:53] Ryan Stawarz: And that's not. Because we can't is 'cause we just haven't prioritized those features. And mind you, this is 12 years in, right? We did have some polling and it just got expensive. And it's more the nature of how our clients use those apps where we're not doing it. Right. So for example, most clinicians aren't real time messaging.

[00:32:10] Ryan Stawarz: Real time. They're sending messages and expecting a reply back, right? So it's not like we need it and like we're just not doing a good job at it. It's by design that we haven't prioritize those features because our users haven't been like, Hey, why aren't I getting this? Why aren't I doing this? Because we prioritize the features our clinicians are asking for.

[00:32:26] Ryan Stawarz: Right? So, um, that's it from my perspective also, if you have anything to fill out from

[00:32:31] Austin Story: there, but not a ton. I mean we use a combination of both of those. I have call 'em like admin apps. I tend to use the Rails built in, you know, just ActionCable. We also have ActionCable whenever we need to scale something out and.

[00:32:47] Austin Story: There's not really a lot of cases where we have to scale out to that level where it needs to work.

[00:32:52] Robby Russell: For those listening that might not be that familiar with the distinction there, can you tell us a little bit about like what's the difference between ActionCable and ActionCable?

[00:32:59] Austin Story: Yeah. Act ActionCable's built into, to Rails.

[00:33:02] Austin Story: Um, you know, they, they've got a bunch of adapters that you can use. We tend to use like the Redis adapter where you can publish and subscribe to like an abstraction called channels. It's just pub sub on the backend for like, like Redis and then ActionCable is a service that implements that API, but they have things that can handle huge request loads and connection volumes.

[00:33:23] Austin Story: They have a, a go service that you can spin up and then from your Rails app, from your perspective, you're still communicating with like an ActionCable, API. It's sending those messages to the Go Service and then the clients are connecting to go. And that's a lot better whenever you have scale, because anytime that you restart.

[00:33:43] Austin Story: Your application, which happens when you deploy, you're gonna get a bunch of connection severances if you're using your Rails application, which may be fine, but you know, if you're using dialer, that's, uh, less fine, you know, and you're expecting like communication with your clients. So it allows 'em to keep that connection open so that they can get messages about somebody joining or something like that.

[00:34:02] Austin Story: Or a message being sent and that go service stay up, that is ActionCable, uh, and then you can continue shipping your Rails application as normal.

[00:34:11] Robby Russell: Is that something that you're self-hosting or do you rely on a, like a different third party service for that?

[00:34:16] Austin Story: We self-host that.

[00:34:17] Robby Russell: Okay. So you're just running that go service within your infrastructure there.

[00:34:20] Robby Russell: Does that change anything for like in a local developments type of setup?

[00:34:25] Ryan Stawarz: The quick thing for that is we dockerize and containerize our apps. And so from my perspective. As a developer spinning on my environment, I really don't notice because it's just a dependency and it gets spun up and it, it's able to run in a container.

[00:34:39] Ryan Stawarz: Um, and so I don't ever have to worry about that added complexity. Most of our engineers probably don't know it runs if they start on certain apps. Right.

[00:34:46] Robby Russell: So it's kinda like a having a Docker container that just runs memc or something. Exactly. It's there. You don't really think about it. I'm curious at Doximity scale, how do engineers get access to realistic data for debugging?

[00:35:00] Robby Russell: Like, what's that look like? Or is that just, I I, there's a lot of teams, I, I'll talk to you and they're just like, well, eh, it gets a little tricky. It's like you just have the most amazing seed data set that you could you give to everybody, or what does that look like? Do you have internal tools, strategies?

[00:35:15] Ryan Stawarz: Yeah. I, I wish we had the most amazing seed data set as everybody does. Right. I'll let Austin talk to this.

[00:35:21] Austin Story: Yeah. I, I think we do a, a pretty excellent job, really. I tend to jump between all of the apps, you know, so I, I don't have the privilege of creating a bunch of artisanal data. We have two tools I, I think, that work really well together.

[00:35:35] Austin Story: One is a, a series of like seed scripts that generate realistic-ish data for development that covers most of the use cases. And then all of those seed data sets get exported from the database so that you can re-seed pretty fast. And the team constantly, I mean, this is always top of mind because whenever we build products and, and ship them through development and staging, whenever we're QAing those, so somebody's verifying the feature.

[00:36:06] Austin Story: We don't want it to get through those two steps and then break in production. 'cause we didn't cover a use case. And we also have to acknowledge that we can't cover every use case because there's so many permutations of that. But we tried our best to have just an, an excellent base data set and then people can embellish on those as they need with other rake tasks that they have locally.

[00:36:24] Austin Story: One of our teams just built out this engine called dox-tasks. That's, uh, that uses this, this gem that's public from us called rake-ui and that is essentially a collaborator between the applications. And, you know, like a, a simple example would be, you know, let's say you're building out a feature on the newsfeed and locally you wanna test out that you have a, a specific card that another service has with this, you'd be able to pull up the local engine DE is only exposed in development and staging and search through this UI and say, oh, well I would like to see an ad on my newsfeed, click a button and then, you know.

[00:37:04] Austin Story: Scatters out and assembles the right tasks for that. So that's when everything become complicated, you know? Um, just kind of a trade off for having the, the domains with the different concerns and how you finally integrate those together.

[00:37:16] Ryan Stawarz: I'll add to that, Robby. So I mean, from a team's perspective, right?

[00:37:20] Ryan Stawarz: So also to the platform level, we have our SREs that are managing some, a lot of the complexity here as well. So what our teams are responsible for is maintaining their seed scripts, right? So the data that they're re, that they're trying to set up that know that domain really well, right? Whether or not it's profile data or network growth data or, or some of our social updates, information, news feeds, whatever might be, each of those teams are maintaining their own data sets, right?

[00:37:44] Ryan Stawarz: Their own seeds for their own uses, which are also usually included in even some of our specs, right? And so, like the teams are responsible for that. Then, you know, the platform team and our SRA teams provide this infrastructure in this beautiful way for it to automatically be run together and then captured and then reseeded.

[00:38:02] Ryan Stawarz: So our local environment's seeded with data when you spin it up from the beginning, which is incredibly powerful for somebody who just, you know, wants to start up and say, okay, well I need to create users. Right? It's not an empty app. Right? And we're talking any number of containers, like of our services you wanna run.

[00:38:17] Ryan Stawarz: They all have some sort of basic set data to begin with, which is really powerful.

[00:38:23] Robby Russell: Was that something that was around when you started in 2016,

[00:38:28] Ryan Stawarz: always had seed data, right? And it, it's morphed time and time again. And I think we have seed data and new seed data and things like that. Sample data, right? So change names and different usages, and there's a differentiation between like seed data that we classify, like this has to be run in production, this has to be kept up to date.

[00:38:45] Ryan Stawarz: We have little of that and more of, it's more of what we call now sample data, which is like we're generating. Fake data on the fly to, to produce new users. We want a hundred MDs, 10 PAs, nmp, you know, whatever, you know, different set of data you might need to build out your newsfeed and or your social feed, whatever it might be.

[00:39:03] Ryan Stawarz: Right. So we've always had seed data. We've been leaning into it more over the past two or three years as we've, I think more and more containerized and more and more have created more and more services. Right.

[00:39:13] Robby Russell: I'm curious about how do you go about managing at that sort of scale when you make database schema changes that are gonna actually require changing the data and your production data.

[00:39:23] Robby Russell: What's been your tactic? Are you just adding into your database schema, like update column, or is it like, are you using some other approaches to navigate, like, let's say changing some attributes, but you need to move some data around at the same time across your, I'm assuming pretty large dataset. So what does that kind of look like there?

[00:39:40] Ryan Stawarz: I'll chime in real quick and I'll let Austin speak to this as well. But I think there we have some core principles on like. Okay. Migrations over a changing schema. Right? And if you need to change data, there should be a separate way to do that, right? It's generally asynchronous tasks that we run, right?

[00:39:54] Ryan Stawarz: Some break tasks to say, okay, now we have a, a backfill task, or whatever it might be. And so the first part of that, the staging data, we kind of cheat, right? Where it's like we just blow it all away and we run off all the samples nightly and then next day if we wanna regenerate a, a staging environment, in theory it would have all that new seated data.

[00:40:12] Robby Russell: What does that look like there then, Austin? Is it that your, if you make a change, you mentioned like there's asynchronous tasks that need to happen. Are your developers writing things to be like account for if like data is maybe in one act, one column and now is gonna be in a different area of the app or a different column name or something now or something.

[00:40:29] Robby Russell: If it maybe just like using that as a really simple example, we're renaming a column and we need to change, we need to update some data, or we're gonna change the type of column it is. Like what, what's that step process look like in a larger scale?

[00:40:41] Austin Story: So I mean, if somebody was just wanting to rename a column, my first question would be.

[00:40:45] Austin Story: Why are you renaming this column? Uh, and why is that important? But, but now to like get, get to your root there. I think delegation to really smart people is important. We have a, a wiki and a very established culture on data migrations, which have to be very intentional. Like some of our databases have multiple terabytes of data.

[00:41:06] Austin Story: And when you're dealing with scale like that, it's kind of like juggling. You know, Nate, he said, you know, juggling razor blades and you know, you have to be very intentional when you're making those changes. 'cause there's, there's so much data there, you know, that that's not a constraint everywhere. But understand, like the engineers that, that we bring in that are making changes to a production data app have been selected by the tech leads and the umbrella leads as somebody who's intelligent enough to be able to understand that, you know, if something has a foreign key, what are the ramifications of whenever you change this data, you know, from like a production standpoint.

[00:41:42] Austin Story: And we do use, uh, departure gem for production changes. Like if you're making a, a dangerous database that, you know, someone's gonna lock the table, departure wraps, the PT online schema migrator, which is just a, a fancy tool for handling, like migrating tables from A to B when you're gonna lock the table or, you know, need to migrate foreign keys and stuff like that.

[00:42:03] Austin Story: But just training people, pointing 'em to the resources for like, how do you make a, a safe change in this, you know, asynchronous environment where you have an application that loads columns at boot time. And the data store is gonna be changing. We have a very clear, like, I can point somebody to a, we can read this, then make the change, and then we'll step through the process.

[00:42:28] Austin Story: And after somebody goes through that once or twice, then it's, it's just, oh yeah, that's, that's how you do database changes and, and Rails, you know? So, I mean, there, there are some gotchas there for sure, but that knowledge and, and trust and delegation I, I think works really well for us. And then, like Ryan mentioned with the, the seeding, you know, the, the staging data is not important luckily, so we can just rimraf it and start over.

[00:42:52] Ryan Stawarz: So part of our platform, right, when we have to run asynchronous tasks, we have a way to do that in production, right? We have part of our, the way we deploy our entire Maestro platform allows us to say, Hey, here's a task and then allows me to kick off our, uh, that task in production as a separate pod, and it runs through all the changes that needs to do.

[00:43:14] Ryan Stawarz: While it would be like a migration kind of, it's a separate thing so that it's independent of actually making the DDL change, right? Because it could take two days to run, right? We let that pod run for two days and it has happened where we've needed to migrate data for that long. The most important part about that also is that we have a separate part, a platform that allows us to take that same task and say, Hey, next time you deploy to staging, 'cause we have 42 staging environments.

[00:43:38] Ryan Stawarz: I don't know what number it is, but there's like one for every QA engineer, five development shared, some mobile shared, right? There's tons of staging environments. Basically in our platform, YAML file just says, Hey, run this next thing we run so that the migration automatically runs junior deployed to staging.

[00:43:53] Ryan Stawarz: After that, it says, okay, well let me run this rake task effectively, and then it, that takes care of that data in that staging environment. So if you're not in the business of dropping your staging database, which most people aren't, that task actually picks it up and actually migrates that data too. And it's also a great way to test that when we're gonna do this in production's actually gonna work as well.

[00:44:12] Ryan Stawarz: The task you ran just ran in 42 different environments that has 42 different mixes of really bad data to begin with. So it's kind of a limit test, right? Because everyone's deleting things all the time. Right. You can imagine. Um, so it generally works out pretty well.

[00:44:26] Robby Russell: You mentioned, uh, rake tasks. There are, is that a typical go-to approach to like, let's, if you needed to kind of, maybe not necessarily throw away code, but something you need to run once, is that typically how your developers would go about default way to make something change versus some SQL statement somewhere in a database or something?

[00:44:42] Ryan Stawarz: Yeah, that or Thor, we use both.

[00:44:44] Robby Russell: Do you tend to keep those types of one-offs? Like if they run once you get, you test it a couple times or you tend to keep those around in your code base, or do you tend to discard them at some point or just document them? Or do you let them sit around in your lib directory?

[00:44:57] Robby Russell: Like where, where do these things exist

[00:44:59] Austin Story: in a rake one-offs folder or the lib? Uh, one-offs is where a lot of them live. I mean, different app, different teams.

[00:45:07] Ryan Stawarz: I put 'em in different places, but that's, that's where I put them and, and some teams. Do better cleaning 'em up than others. Not because it's right or wrong, but like you clean 'em up monthly, quarterly, yearly.

[00:45:17] Ryan Stawarz: Right? Like, oh yeah, we, that one's been ran, we don't need that anymore. Right. We clean 'em up sometimes. Immediately. Sometimes they hang around. It's not an important artifact, mostly because we have Git. Right. So you can go through that history and find that things out. Yeah. Which is yeah. Key too.

[00:45:31] Robby Russell: And one of the other conversations I was having with uh, Rosa from 37 Signals, I was, I was curious about that.

[00:45:35] Robby Russell: 'cause I'm, well wait, people will leave that stuff around for, and then people are afraid to delete the stuff if they come in later and like, well I don't know what this is, is gonna be used anymore. But you mentioned, do you actually call it like a one-offs folder or something? And then, so it's very clear that this is one time thing.

[00:45:48] Robby Russell: And then do you have other ones, like this is multiple times, like, is that just like the default, like throw it in the one-off folder? And who is the clever person that came up with that idea of just the name, A folder of that?

[00:45:56] Ryan Stawarz: I'll say Austin. I think it was Austin. I dunno who,

[00:46:01] Austin Story: nah, it was, uh, I, I have no idea who came up.

[00:46:03] Austin Story: I mean, but it, it's very descriptive, right? Of one-offs. I like to delete code as soon as I'm not using it because otherwise it. Lives for forever. But that does make it very clear for people in the future that it is a one-off. And we're on our team very strict about having good descriptions and pull requests so that you're not just saying, which did, but like why you're doing something.

[00:46:25] Austin Story: So if somebody's curious, they could just click in there and make sure that this is indeed a one-off. Which, you know, would be bizarre if it wasn't a one-off task that wasn't

[00:46:34] Ryan Stawarz: supposed to be just ran once. Yeah. I mean, in the same regard, similar, like one of the reasons I prefer to lean more towards Thor now is that I think it's easier to test.

[00:46:42] Ryan Stawarz: And so I like to write specs. I'd like to have all my teams write specs against those sort tasks. And so that's another reason why that's really nice to delete them. Mm-hmm. Is because now you're maintaining the specs. The specs start failing. You're like, do we even need this? Why am I updating this spec?

[00:46:54] Ryan Stawarz: All right, let's get rid of it. It doesn't happen often, but that's another artifact that's there to help document what's being run and, and everything. Right. Almost all the one-off tasks have specs associated with them because we wanna make sure they're, they're covering what they need to do as well.

[00:47:07] Ryan Stawarz: Even just 'cause it's one-off doesn't mean we don't wanna test that code. Right.

[00:47:10] Robby Russell: That's true. Do you, in those types of situations, are you typically adding a lot of the, the business logic in, say some models or something and then you're just calling 'em from the right tasks or so you can do the test or are you just writing tests on those?

[00:47:24] Ryan Stawarz: I think that's so bespoke depending on what the task is trying to do. Right? Like if you're migrating data, well, you know, is that the model's responsibility? 'cause that's probably not gonna happen more than once. It's probably gonna end up in that third task. Right. So really just depends on what it is that's being done.

[00:47:38] Ryan Stawarz: Sometimes there is a method right on the, on the model that does some work that you can call and hopefully if it's code you want to live, it would be in the model or in a service class. Right. That would be the ideal case. Certainly there's been larger one-off tasks that it's all throwaway and it's like, yeah, we just need to loop over this crazy set of data, andm a bunch of stupid things together and, and just get it done.

[00:47:59] Robby Russell: For those listening that might not be familiar with Thor. What is Thor?

[00:48:02] Ryan Stawarz: It's similar to rake. It's a command line tool that allows you to run commands effectively, builds up a command list of history, or not history, but a list of commands that you can run and execute just like rake. It has different mechanics.

[00:48:14] Ryan Stawarz: I find that ergonomics to be a little bit simpler to understand, not understand, but the command line part of it for sure. But some reason I've never been able to wrap my head around some of the Rake man line syntax. I don't know why I, it's, it's always kind of drew me nuts. There's like brackets in there sometimes and like just doesn't help me.

[00:48:32] Ryan Stawarz: I don't know.

[00:48:33] Robby Russell: Can appreciate that. You know, I'm curious about maybe for you, Austin, initially is, uh, how do you go about establishing patterns across the teams to kind of enforce some consistency in that large of a code base? Then my follow up is then is like, I would imagine that you've had some patterns that were around at one point that your team decided, Hey, we want to kind of maybe start moving a little bit more in this way.

[00:48:56] Robby Russell: How do we get everybody kind of on board with that or what's that kind of look like there?

[00:49:02] Austin Story: So I think of this in terms of adaptive versus prescriptive and you know, we see all the applications and, and problems that come up in one area or things that's, that's not great in one area, but you don't wanna be prescriptive with everything We hire, you know, we bring people on our team that are smart, they wanna be able to work independently.

[00:49:22] Austin Story: That's kind of like a lot in the Rails ethos of being able to just get some stuff done very quickly. But if we start seeing a lot of problems or people implementing things in different ways. Where, you know, like, let's say, you know, one team is, is building a bridge and then you find out that another team is building a tunnel, uh, under the same body of water.

[00:49:42] Austin Story: It's like, well, you know, let's, uh, let's shore this up so that we're all building towards the same interfaces. And that's where we kind of try to read the tea leaves of like where that is best. Like where, where we can invest our time the best so that we help the organization the most. And we're pretty severe with like, cutting projects that like just come from our head where we're like, oh, you know, we think that this would be better, or we think we should do this thing.

[00:50:09] Austin Story: Like, we try to base it on what we're actually seeing in like either postmortems or proposals or. Code changes or bugs that people are running into, and then aggregate all those together. And at the end we may say, okay, well, you know, kinda like with that GraphQL Federation, it's like, like, look, the ergonomics of this are not great.

[00:50:26] Austin Story: Uh, it's causing outages. People cannot opt out of this. So we need a better solution. And then, you know, cut a project, buy in from stakeholders, explain how the process is gonna go, do the work, cover all the use cases, and, and then ship it out and culturally continue to talk about that thing. And when we put out a new tool out there, you know, from like a, a platform side, something that, that I care deeply about is not just building the thing like field of dream style, where it's like, Hey, if you build it, they'll come build it.

[00:50:57] Austin Story: And then actively recruit people from the organization that, you know, could benefit from the thing that you're, you're using and getting a project with them in order to implement your thing. And whenever you do that, you actually find that. Sometimes you, you didn't get the API, right? You didn't cover all the use cases.

[00:51:16] Austin Story: But whenever you do that about two or three times across the organization, you come up with a tight API that's covering most of the use cases that you're gonna have to deal with. And culturally like the, the people that need it know about it. They know how to use it, they're actually using it, and then other people start seeing the benefits and that's how something gets out there.

[00:51:34] Robby Russell: Ryan, how have you kind of thought about that from your end? Where in the, the teams, did these ideas start to spark up? Is it usually by some lead person or is it just, Austin mentioned reading the tea leaves, like how does that kind of like look to like get these ideas? And then, and then I'm also curious a little bit more about, maybe you could talk about like that many engineers.

[00:51:50] Robby Russell: What sorts of approaches do you have for disseminating ideas like that or sharing like is that internal workshops, events or just Slack channels that people, things like that somehow just happen because. I know with large organizations, it seems like I've work in smaller organizations, so it's a completely different world for me there.

[00:52:08] Robby Russell: So I'm just kinda curious about that.

[00:52:10] Ryan Stawarz: Yeah, so first I'll say in terms of larger organizations, I've worked at lots of larger organizations and Doximity is not the norm from my right. Like, no, we're large, we got plenty of people, don't be wrong. But like we're very flat. Nothing's stopping any engineer to go talk to anybody else in our organization, the business side, our CTO, our CEO, right?

[00:52:33] Ryan Stawarz: Like we're very open. Like you can reach out to anybody and have a conversation. And I encourage people onboard, tell the mentors, like teach 'em to fish, right? Like, don't just go get the answer for them, introduce them to Austin so that they can ask Austin questions later. You know, things like that. Right?

[00:52:49] Ryan Stawarz: And that's really important. So. From that perspective, like, so how do ideas get disseminated? We have several different ways. Often what happens is we will run into a problem on a team and we'll recognize this isn't our domain and it should be solved. Probably other teams are having similar issues. Maybe we'll talk to a couple other people like, Hey, have you seen this?

[00:53:06] Ryan Stawarz: And then we'll, we'll try to figure out, hey, maybe platform can help us build this rather than us. 'cause we don't want to own it when we're done with it. And it probably would benefit everybody in the organization. Right? And that's probably a lot of teams. I encourage teams to reach out the platform mostly because, one, it'll be a better solution because we'll get that tighter API that will fit the real need versus like the bespoke thing that nobody else is gonna use.

[00:53:28] Ryan Stawarz: And then we're gonna have to re-implement it later. And then two, it's also helpful because then we get extra resources from Austin's team to help from platform perspective and pull in some of our business goals and objectives sooner. Which is the most important part. Right. And I, I say it jokingly, but I, I know Austin's team loves working on the products and like.

[00:53:45] Ryan Stawarz: Helping us provide the solutions to our clinicians faster. Right? And that's ultimately what, what we're around for. So, and then finally we have what we call needs assessments and technical proposals. And we just, I mean, it's not super high tech. We don't have this giant SaaS software we pulled in, we have this thing called a Google Sheet, and somebody writes up a needs assessment and they drop it in there and people subscribe to the sheet and it changes.

[00:54:07] Ryan Stawarz: And people read and they're like, oh, that's interesting. I had the same problem. Or, you know what, maybe you should look at this or talk to Soandso. And that's, you know, it works really, really well. And then finally, I would say the, the bread and butter of Doximity is in AC organizations. Our offsites, the best part of our offsites are getting together with different teams and having a conversation, just hearing them talk or like, oh man, I was dealing with this problem.

[00:54:29] Ryan Stawarz: They're like, well, I had the same problem. And then that's where a lot of things ruminate and originate in terms of like come up with solutions and bringing it back to the teams and. And then we'll start writing these assessments or tech proposals or whatever it might be. So

[00:54:43] Robby Russell: That sounds healthy. I'm hoping for those listening, they're like, wow, you know?

[00:54:47] Robby Russell: 'cause I've talked to people that are like, well I don't know who to go to about this. I feel like I've complained about this, or I brought up this issue and it doesn't, I don't know where to go with it. And doesn't seem to like, well it's Stick another department. And there seemed slammed with some other projects that they're focused on and like, so they kind of start to give up a little bit on as advocating for themselves a little bit and like guess like, we'll just live with this for now.

[00:55:06] Robby Russell: And I would imagine that the Doximity's not entirely immune from that ever happening at times. Like when priorities, you know, might pop up and stuff like that. But is there a way about how you're teams openly talk about that? Or do you feel like you're just, or are you immune from that?

[00:55:20] Ryan Stawarz: Certainly not. I immune.

[00:55:21] Ryan Stawarz: I think the way we do quarterly planning helps. Like we commit to three months and sometimes, you know, we can't help you. Like my team, you need something from this particular domain, right? And we're like, we're already over committed. We can't get it done in three months. But guess. Three months is a short period of time.

[00:55:39] Ryan Stawarz: We can do it week one of that next period if we can. Usually most teams have the slack and they can get it done, but if it's really, really that end of year, end of quarter fiscal, whatever is really important at that moment in time, it's not far off that we're gonna be able to get that thing done for you.

[00:55:54] Ryan Stawarz: Right. Whatever might be Correct me, Austin, your take on that as well.

[00:55:58] Austin Story: Yeah, no, I, I agree. I think the big part is the communication and, and trust. We do have a really high trust environment. Not

[00:56:04] Ryan Stawarz: absolutely

[00:56:05] Austin Story: just implicitly, but you know, those offsites really help. And then also, you know, the, the tenure on our teams is, is pretty long.

[00:56:13] Austin Story: Ryan and I have worked on so many projects, like I just, I trust him at this point and like he think, can't put words in his mouth, but, you know, trusts me, you know, and when you have that sort of relationship where. We both know that we're looking out for what's best for everybody and that it's pervasive across our entire organization.

[00:56:31] Austin Story: We are together working to build a medical network and make physicians' lives easier, and those proposals, whenever somebody puts it out there, I, I think that is one of the harder things culturally on my team for people to feel okay with making a proposal for something. Because by default, you know, if somebody just says something's a problem, that's not a big enough bar for them to jump over for us to spend time on it most of the time.

[00:56:58] Austin Story: Mm-hmm. I mean, that, that's not true all the time, but like the majority of the time, if somebody isn't willing to write down what they're thinking and say what the problem is and get feedback from other people in the organization, then you know, it's, it's one of two things. They don't feel comfortable and they need to be empowered.

[00:57:14] Austin Story: Or it's not important enough for them to spend 20 minutes doing that. And so that's a good, I guess like sifter of whether we should even consider spending time on something. And then the quarterly planning is, is huge for us. Our team has been through many different iterations of this, but you know, when I first started it was very simple.

[00:57:31] Austin Story: It was just a, a Google sheet where somebody put like a one sentence, like, we need to do this, and they would pick what they do off of that. And where we've gotten it to, I think make sure that what we pick is actually going to be helpful for the organization. And like, there's four teams on product platform.

[00:57:49] Austin Story: You know, we all have very specific like, engagement areas that we're, we're focused on engagement or, you know, en enhancements, like trying to help the other other teams out. And in that quarterly planning we call out for, for each goal that we're saying we're gonna do, we just, we take it down to math. Hey, you have four people on your team.

[00:58:07] Austin Story: There's 12 weeks in a quarter. Okay, you've got 48 people weeks. Cut that by a third for other time off and, and things that you gotta do, you know, things that come up where you gotta help somebody right now. 'cause we have to prioritize that. Like, that's our responsibility primarily. And these goals that we're coming up with, like they are in pursuit of reducing the amount of time that we have to spend helping people.

[00:58:29] Austin Story: You know, I mean we're, we're always gonna have to help people, but like, if there is a consistent thing that's coming up and we can close that down, then we have less things coming up and more things that we can focus on that are important. So using that, you know, you've got like 30 something weeks and for each project we say what the project is, how it maps to, you know, what our team tries to focus on where it came from, when it's done a description, and that when it's done is like an explicit goalpost.

[00:58:59] Austin Story: Like, so I can give this to any person on the team and they know, okay, these are the things that we're doing, and then how long is it gonna take? And that isn't like a, a feat to the fire. You know, if you say something that's gonna take like four weeks and it takes six weeks, you're not gonna get in trouble.

[00:59:12] Austin Story: But just as you know, mature engineers, we can kind of guess at how long something's going to take. And if they're saying that something's gonna take like 12 weeks and there's like four things that are on the goalpost, you can say, okay, well if we remove this thing here, how would that impact the time commitment on this?

[00:59:30] Austin Story: And then we try to fill up that time accordingly with ordered projects in that list of like, what's the most important thing like that. So

[00:59:38] Robby Russell: is it safe to assume that there's projects that that'll be like mid-production process and then you're like, okay, we gotta put it on hold for a bit to divert for a little bit because something else popped up or whatever, we're gonna need to come back to this next quarter or something.

[00:59:50] Robby Russell: Does your team have a good workflow for like how to put something on hold or is it just, or if you need to pass that work over to another team or another person on your team?

[00:59:59] Austin Story: Uh, for us it depends on how urgent. The need is, if it's an urgent time sensitive need, then we can just push pause on things. But the way that we try to break down projects is shipping the absolute smallest thing that pushes that goal forward and trying to do something every day for that.

[01:00:18] Austin Story: And if you've split that project up, you know, if you've got like a, a massive project like that graph Yellow Federation's a good example. Like we just migrated our analytics pipeline. That was massive, but we just split it into really teeny tiny chunks. Hey, this is like the broader thing that we're doing.

[01:00:31] Austin Story: And then there's like six steps in there and then we just ship those individually and then if we had to pause that we at least have all of the things functioning and working in production the way that we expect 'em to now. And then we can come back to that.

[01:00:44] Robby Russell: Alright, so something I'm kind of curious about is, you know, you touched on earlier on, on like what keeps you on Rails.

[01:00:50] Robby Russell: What is it, do you think about Ruby on Rails that. Do you think it's one of Doximity's secret weapons as in your competitive landscape? Do you kind of think of that Rails has been a big contributor to that?

[01:01:00] Ryan Stawarz: I would a hundred percent say yes to that. Again, it's the pragmatic nature of the framework, right? Of just being able to do what we need to do within the framework and get us there faster.

[01:01:10] Ryan Stawarz: And when we need to deviate, we can deviate. Front end is probably the biggest thing. We probably deviate from the Rails paved path, and we do it very successfully in my opinion. Like we have a set of front end constructs that we built and managed, and it works really well for our teams and I've seen it provide a lot of value.

[01:01:28] Ryan Stawarz: So it's not like it's an impedance, like, oh, we went off, I hate the pun, but we're off the Rails. And now it's like we're having all these challenges, right? I've evaluated a couple of frameworks before where it's like, okay, what if we did this thing differently when we were evaluating Vue for example, it's one of the reasons why I picked Vue is 'cause it allowed us to be a little bit different, but couple of the competitors we were looking at at the time, there was you wouldn't off to side and tried to do something different.

[01:01:50] Ryan Stawarz: Like it got ugly really fast. A lot of it's through the nature of Rails, right? And just the way the framework's composed and built. And obviously the nature of Ruby too. So

[01:02:01] Robby Russell: you mentioned that you had jQuery and um, Backbone and other things like that there, and you ended up on Vue and you've also evaluated Turbo.

[01:02:09] Robby Russell: Like what would it take for you to ever consider migrating away from Vue?

[01:02:14] Ryan Stawarz: I mean, some of our teams don't use Vue, right? There are teams and there are products and there are places where we just don't need it. And it is a heavy framework, right? It is a full on single page web app when you need it, or at least.

[01:02:28] Ryan Stawarz: Sprinklings of it, and you don't always need that. And if you just want a simple RV template or a slim template and a little bit of the, the JavaScript, like we encourage teams to do that. However, there are benefits in our ecosystem. We have a whole analytics framework that allows it to track things inside of view that just isn't the same within our Rails templates, um, using whatever job script might need.

[01:02:50] Ryan Stawarz: But again, you don't always need that, especially for admin tooling or simple pages for users marketing pages. Like there's value in and we definitely don't use it in certain places.

[01:03:03] Robby Russell: Austin, what about you? Do you think Rails is part of Doximity's, like one of your secret weapons as an organization?

[01:03:10] Austin Story: For sure.

[01:03:10] Austin Story: Anytime that we need to get something out there, when we try to use not Rails, it's harder to maintain. Takes a, an enormous amount more work whereas. Because Rails exposes not just like a a, a web framework, but a way to integrate into it very consistently. It's very easy for us to plug things like, you know, Brian mentioned like the front end tooling.

[01:03:32] Austin Story: Like we've got that in an engine now and you add, we call it docs front end and you're able to use Vue where you need to use it, you know, or metrics or tracing or analytics or authentication. All these things are just baked in. And keeping that number of frameworks and supported options down allows us to build really prescriptive tools for the things that are important.

[01:03:55] Robby Russell: Nice. Alright. I have a couple of last questions for you both. Um, we'll start with you Ryan. Is there a software book that you find yourself recommending to peers?

[01:04:04] Ryan Stawarz: That's tough. I mean, no, not generally, right? Like in terms of, is there a book, like, oh, you have to go read that Particular, the book. I haven't read a lot of software engineering books recently, so, yeah, I, I guess I, I should leave it that right.

[01:04:20] Ryan Stawarz: I mean, I think I'm most, like most people, I imagine I get most of my stuff from blogs, YouTube videos, like tutorials, things like that. Like to me, most books are more fluff than it's worth. But these days, at least from a, it's really good for intro. But beyond that, like when you know deep into a topic, you know, you're reading textbooks and that nobody wants to read a textbook, right?

[01:04:40] Ryan Stawarz: I don't, I dunno.

[01:04:41] Robby Russell: What about you Austin?

[01:05:05] Robby Russell: Hmm.

[01:05:52] Robby Russell: Nice. I'll definitely include links to both of those in, or all three of those in the show notes for people. And Austin was referencing my other blog Maintainable, which he was on sometime last year, so I'll include a link to that episode as well. We dig into. Less rail specific stuff, more about software maintainability.

[01:06:08] Robby Russell: I'm curious, Ryan, because I don't wanna leave you hanging without sharing any books, but put you on the spot here. Is there an aspect of Ruby on Rails or part of the framework that you feel like you would be ashamed to almost admit that you don't know that much about, but you're curious to? Maybe one day I'm gonna finally dig up and look that up.

[01:06:26] Ryan Stawarz: So I would say yes to almost all of it. And the reason being is 'cause like I've been, I've been developing with Rails since I think 0.7, right? I started a a side project in 0.7. It was horrible. I didn't know Ruby, so I was learning the wrong thing, right? It was like meta. Awful, right? Like I was doing all sorts of wrong things.

[01:06:44] Ryan Stawarz: But the reason I say yes to everything is because there's so, it's such a rich framework and there's such, there's so much depth to it that like, keeping up with it across every version is really hard. So I absolutely can tell you there are corners of the frameworks that I don't fully understand, I don't know about, right?

[01:07:03] Ryan Stawarz: Or I have to relearn because I haven't done anything with it for three years, five years, whatever it might be, right? And it's shifted a little bit. I have to support notifications, for example, right? I know it's there, I know it's hugely powerful and you can do a lot of things with it. I don't normally have to do anything with it, right?

[01:07:18] Ryan Stawarz: And so it, it generally works under the hood and the libraries I use it, do what they need to do. But if I was going to be critical of the framework in a really good way, it's like it is so rich that it's hard to keep up with and know everything about the framework. And that's not a bad thing. That's what empowers it and makes it so, so powerful.

[01:07:35] Ryan Stawarz: 'cause you only need to use what you need and the other parts fall to the side.

[01:07:40] Robby Russell: It's true. Austin, is there anything specifically within the Rails that you wish you understood a little bit more?

[01:07:45] Austin Story: No,

[01:07:46] Robby Russell: uh, no. You understand it all? I get it. I get it.

[01:07:49] Austin Story: I don't, I don't understand it all. Uh, I'm not like a core ator.

[01:07:52] Austin Story: No, I mean, just our team goes really deep in everything. Uh, I mean there for sure, like action mailer or something that I don't do too often, but we maintain a, of like departure. So having to understand like Rails and databases and controllers and integrations. There's, I mean the, for sure there's, I'm certain there's areas that I don't understand that I've never touched, but we have to go super, super deep all the time in this.

[01:08:16] Robby Russell: I, I'm sure my reason for asking that outta curiosity, it was just more of, uh, for those listening that don't feel like, like, wow, I'm gonna have to learn everything before I can work at some larger organization one day. And I just wanna dispel that a little bit. 'cause there's a lot of things that I don't understand about, you know, admittedly starting this new podcast, I'm like, well, there's a lot of things about Rails.

[01:08:32] Robby Russell: I don't know, like what these people are gonna talk about. So I'm just wanting to be open about that with everybody listening.

[01:08:37] Austin Story: Robby, I was gonna mention to somebody that's just getting started, the best way for me to get super familiar with these things, and I haven't been around it as long as Ryan, I didn't realize you started in 0.7.

[01:08:49] Austin Story: And that's, that's impressive. But you don't have to know everything inside of it. Like, that's the beauty of using Rails. They provide the omakase of this where you know, hey, you need to be able to manage cookies. Okay, we've got an entire framework for that. And if for some weirdo reason you need to opt out of that, you know, go for it.

[01:09:09] Austin Story: But out of the box it just works and dig into the code. You know, I, I, I use pry. I think, uh, debuggers a, a good example of something that I, I haven't really used very much 'cause I've just been so used to using binding.pry. But, you know, whenever you're in a request. The way that, that I learned this stuff really well is I'll just, I'll throw in a debugger statement and I'll just trace every single part and then I'll look at that code in Rails and say, okay, what is that thing doing?

[01:09:36] Austin Story: Why is it doing it that way? And then you can look at the pull-request on the Rails' side to be like, oh, okay. I understand exactly why that's, that's happening there. So you don't have to know everything.

[01:09:44] Robby Russell: That's helpful. And you know, another thing just to kinda add onto that, as for listeners, I think where a lot of times a lot of developers struggle with is that they don't have a opportunity to experiment with something because they don't really have a real problem to specifically solve for it.

[01:09:58] Robby Russell: So it's kind of sometimes a little challenging to wrap your head around like, well it's, I'm gonna go play with this thing, but I don't really have a problem to solve that needs it yet. So it's kinda like, how do you put it in practice? And like, I personally am not the type of developer that can just walk through a tutorial and be like, okay, I guess I picked that up.

[01:10:12] Robby Russell: But I don't know if I'm ever gonna apply that to something. I'm like, I have a problem. Like what type of tools are available to solve that problem? And then I'm gonna go through that process and kind of dig into it at that point. So, and everybody has learns a little bit differently. I know that that's also an aspect of that.

[01:10:25] Austin Story: Our team also helps with Rails upgrades. I, I just wanna make sure I don't sound like I'm like pompous or think that I know everything, but like we are managing the, the Rails upgrades. So like we've been a part of like every major Rails upgrades since I've been here. Like the three to four, four to five, five to 6 7, 7 to eight, seven to eight's going on right now.

[01:10:43] Austin Story: And that resonates with me. It's really just when you have this large application that has thousands of responsibilities and integrations and use cases and you upgrade that thing from, you know, like Rails four to five and you see, oh well the way that the cookie parsing changed, you know, uh, and then digging into that be like, oh, well why is that different and why isn't it working now?

[01:11:04] Austin Story: That's how I learn in practice actually doing stuff and, you know, reading the code as you're like going through requests really helps. But it is that actually using it in a real situation that solidifies everything.

[01:11:15] Robby Russell: Well, Ryan and Austin I have taken up enough of your time today. I want to thank you both for stopping by on Rails, a talk shop with us today, and if people wanna learn more about what you folks are doing at Doximity, direct them to the engineering blog, I believe.

[01:11:28] Ryan Stawarz: Absolutely.

[01:11:28] Robby Russell: Thank you so much.

[01:11:29] Ryan Stawarz: Thank you, Robby.

[01:11:30] Austin Story: Thanks, Robby.

[01:11:34] Robby Russell: That's it for this episode of On Rails. This podcast is produced by The Rails Foundation with support from its core and contributing members. If you enjoyed the ride, leave a quick review on Apple Podcasts, Spotify, or YouTube. It helps more folks find the show. Again, I'm Robby Russell. Thanks for riding along.

[01:11:51] Robby Russell: See you next time.