As a senior engineer, I pride myself on spotting optimization opportunities. Who doesn’t, right? I mean, not these preliminary optimizations, the real ones — impactful ones. Recently, I identified what seemed like a real potential performance improvement in our system. Being the diligent engineer I am, I consulted with a colleague who had previously researched this exact issue.

We already tested this. No need to revisit. Change my mind.

“Don’t bother,” he said, showing me data that suggested the gains would be negligible. I trusted his expertise — he knows his stuff — so I shelved the idea without a second thought.

Fast forward a few weeks. Our Product Leader swoops in with essentially the same optimization idea. I immediately pushed back, citing the “solid data” that proved it wouldn’t work. I should mention — this isn’t your typical product guy who barely knows what a for loop is. He has serious engineering intuition, which, if I’m being honest, only made me more determined to prove that I — the senior engineer — had already considered and dismissed this idea on solid technical grounds.

But this Product Leader didn’t give up easily.

“Are you absolutely sure?” he pressed.

Something about his confidence made me pause. “Fine,” I thought, “I’ll redo the measurements just to prove I’m right.”

Plot twist: I was wrong. Well, sort of.

The optimization did work—significantly. So what happened? As it turns out, the initial proof of concept had some fundamental flaws. The devil is hidden in the details. My engineering colleague who originally researched this was actually so close, but there was a complexity he missed. His PoC was almost flawless; however, he applied the optimization only in dev mode—in production, his code was counteracted by a different part of the system, a system he overlooked. That's why complexity sucks; it fights back.

Anyway, while he was interpreting the data he was surprised as well. It almost looked that it would work – in dev mode it shows promising numbers, but when he deployed the code in production-like environment, nothing. It looks that almost real-life environment makes the difference negligible. So he dismissed it.

This is where Occam’s razor should have come into play. The simplest explanation wasn’t that our optimization idea was fundamentally flawed — it was that something in our testing approach was masking the results. Ironically, it took an ego check for us to start questioning our assumptions. I ran the experiment from scratch and did it more thoroughly this time.

I reserved one day to look into it, and that's when I realized something crucial: our production environment and development environment differed in one critical component. The irony? This component was originally implemented to make the app faster, but it was actually counteracting our optimization by reverting its function—making everything slower. The very thing designed to speed things up was fighting against our new speed improvements.

This time, I approached the problem methodically. I formed a clear hypothesis, ran comprehensive tests in production-like environments, and confirmed what I didn’t want to believe — the optimization would make things much faster (nearly 50% faster in P90).

Armed with data, I swallowed my pride and went back to our Product Leader. “You were right,” I admitted. The best part? I discovered we could implement the optimization as low-hanging fruit without needing massive refactors or architectural changes. From that moment, everything moved at lightning speed. We quickly shipped the code internally, then to selected customers, and finally to production. Each deployment confirmed our findings — the performance gains were real and significant.

In case you’re wondering — when presenting our findings to the broader audience, we conveniently omitted how we had initially dismissed the idea. That became our little professional secret. We documented the learning together and agreed that haste makes waste. If anything, this experience made our collaboration stronger (and gave us something to laugh about in private).

Lessons Learned

This experience was a big slice of humble pie with some useful takeaways:

1. Trust but verify: Even when working with skilled colleagues, critical findings deserve independent validation.

2. Titles don’t confer infallibility: Being a senior engineer doesn’t make me immune to errors or cognitive bias.

3. Resistance to revisiting “solved” problems can be costly: I nearly missed a significant optimization opportunity because I was too comfortable with the initial conclusion.

4. The simplest answer: we tested wrong: Our Product Leader’s instinct to question our conclusion rather than our theory proved invaluable. This is Occam’s razor in action — the simplest explanation is often correct. Our experiment was flawed, not our optimization theory. Our collective intuition that the optimization should work was right all along; we just needed someone to push us to look past our initial test results and verify our approach.

5. System complexity is the eternal enemy: If our development and production environments had been more similar, we would have identified the opportunity immediately. The more moving parts and differences between environments, the harder it becomes to isolate variables and make accurate assessments. Every layer of abstraction or configuration difference adds a potential blind spot.

So, did this humbling experience actually happen, or did I craft it to illustrate important engineering principles? Well, who knows. Either way, the lesson stands: the only thing worse than being wrong is being confidently wrong — especially when you have “senior engineer” in your title.

Recently, I was deep in debugging our fairly large React application that we have here at Outreach when I noticed some unusual behavior from React Developer Tools. Upon investigating further, I discovered that React Developer Tools was attempting to connect to a different React renderer. Which renderer, you might wonder? It turns out, it was attempting to connect to the renderer originating from react-test-renderer.

To put it simply, due to certain race conditions, it was randomly selecting either our main React renderer or the TestRenderer. This behavior was definitely unexpected — after all, the TestRenderer should only be utilized in testing scenarios. So, who was importing it? And why?

To start with, I verified that indeed we had bundled react-test-renderer as part of our production bundle. This raised several concerns, not least of which was the additional 28 Kb of download, parsing, and JavaScript execution burden imposed on our users.

As part of our CI pipeline, we utilize the Webpack Bundle Analyzer to generate outputs. With this tool at hand, I could easily use its built-in search feature to confirm my suspicions. Snap!

Now that we’ve identified the issue, the next step is to uncover how this undesired dependency made its way into the bundle. You might be tempted to simply use the search function in your IDE, and if you’re lucky, you might stumble upon its usage in App.tsx. However, in most cases, it’s not so straightforward. Our monorepo contains over 1 million lines of TypeScript code, which made me quickly realize that a more sophisticated approach was needed. 🤓

Enter Statoscope. All we require are the stats data from Webpack. The process is relatively straightforward (although in our case, we had to chain this command with node and — max-old-space-size to avoid running out of memory — a challenge we managed to overcome 😀).

webpack --json stats.json

Once we possess our stats.json file (ours was a whopping 2.1 GB), it’s a simple matter of dragging and dropping it into Statoscope — rest assured, it’s a local-first application and no data is uploaded, so don’t worry it takes just a few seconds. With Statoscope, we can then examine why the particular module was required.

In our scenario, the root cause was clear: react-test-renderer was being required (as part of React Testing Library) within a util file that was utilized by both tests and app code. Because the code wasn’t effectively tree-shaken (given that it’s a third-party dependency and Webpack, which plays it safe, can’t be sure it doesn’t consist of side effects), it unintentionally became part of the bundle itself.

After fixing the issue by simply splitting the file into two where the one required by the app was not requiring this dependency, React Developer Tools was finally happy again! And as a nice bonus, we shaved off 28 Kb of JavaScript.

P.S: Statoscope also provides CLI tools that can seamlessly integrate into your CI/CD system, ensuring that no such unwanted dependencies make their way into the production bundle. This serves as a valuable safeguard to prevent such occurrences in the future.

As always, I write this article mostly for myself. Not that I would believe that I would suddenly lose my memory and start wondering what I'm doing here. Rather, I continue in my efforts to document what I believe works well, so I have an easier time passing this information around in this format. In the past, I would probably have used Twitter for this, but recently I decided that every time I have a big urge to share something bigger, I would write an article about it. So here we are.

I believe there is no point in doing a fancy introduction of Git to you. You most likely know it very well. Git is one of the most popular distributed version control systems, which aims at tracking changes in your source code and thus making collaboration possible at scale. It would be an understatement to say that Git is popular; it is practically everywhere in modern software development. Let me know if you use something different and the reasons for that in the comments (pro tip: in that case, it might be a good idea to start looking for a new challenge; your career might need it).

Trunk-based development

There are lots of opinions on what is the best way of working with Git at small-to-big software product companies. I have a personal answer for that (and it is not that controversial), it is called trunk-based development. You might be using it, you just don't know it is named like this.

This idea is simple: in order to move fast, you need to ship fast, iterate, and do it all over again. (Shameless plug: you might enjoy my recent article about high-performing product teams and their mindset.) To reiterate what I just said, developers are encouraged to merge small, frequent updates to a "trunk" branch (usually conventionally called the "main" branch). Once the code is merged, it's all in the hands of CI/CD (I have some opinions on this topic as well); tests are run, code is built, and the app is shipped to customers. Everyone loves it. Rinse and repeat.

It might look like this: I create a branch with my feature, iterate on it (more about it later) and once it's ready, we do a review with the team. If everything is all right, we squash-merge it to the trunk branch.

There are many nuances to this one though. Firstly, you can technically do a fast-forward merge, where the commit ID would stay the same.

This is how it would look.

That is good in case you want to optimize for CI/CD runs. It is a good practice to make sure your PR is up to date with the main branch so you can be sure that the integration of your code is flawless. Otherwise, something that looks fine on your branch might be broken once it is put on top of the actual "trunk" branch.

Consider something like this. Here, you can see that my feature is missing code from the commit with ID e which might (or might not) be important.

Anyway, back to my case (see the first image) – I tend to prefer squash merging. Making sure your commits are atomic (a fancy word for a commit that is self-contained) in a way that is worth keeping them in the history can be tedious, and trust me, if you have a big enough team, it might be hard to maintain consistency. To play it safe, I found that squashing the feature into one commit with a descriptive message is much better than adding 10 commits, most of which are "fix" or "wip".

There is one important point I have omitted so far. In order to do this, you have to separate deployment (which, as we said, should happen every time you merge something) from release. What does this mean? Simply put, you should be free to merge imperfect pull requests as soon as they do not break the whole system, putting everything down. How to do that? Feature flags, labs flags, call it what you want. The idea is that you are in control of what is available to the customer. With a system like this, you can eventually achieve rolling out features based on geography, user cohort, and more. The same goes for a fast rollback in case something goes wrong; you can just turn it off again. Boom. This sets a great mindset where the way forward is through iteration. Not even talking about derisking deployments; deployments will be so frequent that it won't be a big deal at all.

Stacked pull requests

There is a second part to what I have shared so far. Hopefully, you got the point with the fast pace of development that trunk-based flow enables. However, there is still at least one important part that we can take a deeper look at: pull requests or merge requests, if you like.

I don't want to explain why pull requests are essential in software development and the culture I'm trying to build as a leader. Not because I'm not passionate about the topic, but because I don't want to bore you to death. However, this is my article. I recently had a discussion where I had to defend pull requests as a source of knowledge sharing and expanding domain knowledge. It's not the only reason we love pull requests. It's also a great tool to make sure we ship the best code we can as a team. What's obvious to one person might be confusing to another. Pull requests are a great place to catch cases like this and make sure the trade-off makes sense, that stuff is documented, and that it is tested. I guess I will write something about code-reviewing culture someday in the future. For now, I guess you get the point.

I wrote a few paragraphs above about atomic commits. To make pull request review easier for your team, I highly recommend switching to atomic pull requests. These pull requests should be small and self-contained, tackling one problem at a time. I also like to introduce either behavioral change or structural change, not both together. There is more to this than meets the eye; in theory, it is simple, but it may sound tedious to juggle all the pull requests and make sure they can be built on top of each other.

Imagine you have two pull requests open. The base branch for the second PR is the first one. This allows you to work with changes from the previous one, but streamline the conversation (and review itself) within the PR only to newly added code because the rest of it is covered in the first pull request (1st-pull-request). With this approach, you can keep building your complex feature and still make sure your pull request reaches the main branch swiftly. Not to mention the fact that smaller PRs are much easier to review. If you're really looking for a thorough review, what's the point in a review anyway? By stacking the pull requests on top of each other, you can resolve them concurrently; once the descendant is mergeable, you can do it, rebase the rest of them, and continue.

But imagine a situation in which someone made a good point about something in your first pull request, so you go and update it.

Now you have to rebase the second one in order to maintain the integrity of the stack. Isn't that too much work?

In a world without stacking, your second pull request would be blocked until you merge the first one, and that can be frustrating.

Yes, there has to be another way. You bet there is one.

Meet Graphite.dev

To my understanding, this flow is quite common in Big Tech; however, I have never worked for one of the mighty list. So, take it with a grain of salt. Though I don't have a reason not to believe it, it is really that productive.

Graphite is a product that enables just that (and more). It is currently in beta (I have some invites left, so if you are curious, feel free to reach out to me!). Their newly redesigned page does a great job of explaining how it works, so start there.

It's mostly two things: a dashboard which offers a slightly different take on Github's Pull Request (with a meme database so you can quickly comment with your favorite GIF). The UI helps you to eventually merge the whole stack with one click, but I rarely want that so I usually stay in Github anyway.

It is the CLI that matters the most. Do you remember the chore I described one must do in order to keep their stacked PR aligned with every commit they do? This heavy lifting is done by the CLI so you can stack and be cool about it.

It is worth saying that Graphite adds a bit of its own terminology to the game. There is "stack," "upstack," and "downstack." In case you use Git directly from the command line, you need to rewire your brain to gt (alias for graphite CLI), which is luckily proxied to Git. Any unrecognized command will be passed to Git. Though, if you are used to Git as I was, it might take some time to adapt.

Here is an example of the workflow to make it more visual.

gt log # this prints current stacks visually
gt add -A # add files, passthrough to git binary
gt branch create -m "my new feature" # create branch from currently staged files
# ... work on stuff
gt add -A
gt commit create -m "additional fix which might go in separate PR"
gt branch split # splits the stack by commits
# ... some time later
gt repo sync # pull latest trunk branch
gt branch checkout # pick stack you want to work on
gt upstack restack # ensure the current branch and each of its descendants is based on its parent, rebasing if necessary.

Don't worry if you are confused; I encourage you to try it. It's easier than it seems. Anyway, one feature I quite like is that whenever you run gt repo sync, it checks for merged/closed PRs and asks you to remove their local branch. With that, it's always easy to check out only relevant branches (stacks).

Every time you restack your stack, Graphite updates/rebases all the PRs affected by the change under the hood. Reviewers also understand that the PR is part of something bigger, as Graphite automatically references relevant pull requests.

Alternatives

Graphite isn't the only option on the market; there's also Sapling from Meta and ghstack. I haven't spent much time with them, but Sapling looks interesting. However, there are some caveats. The great thing about Graphite is that you can easily plug it into your company's workflow without causing any havoc. Pull requests are pretty much normal and easy to review when using Graphite. You can't say the same about Sapling, which has Git interoperability, but it's generally recommended to onboard reviewers as well, which is hard to imagine in my shoes. I'd be curious to learn if someone was bold enough to roll it out in a company with more than 100 engineers (except Meta of course! Haha, you thought you got me, right?).

One important note: all of those are focused on Github. It's not a big deal for me, because for the last 8 years all of the repositories I have contributed to have been there. But it might be a deal breaker for some. Though, the concept should be able to be generalized.

Future

Are you still with me? I hope you are! Graphite is still quite new and I'm curious about how it will shape up. I have to frankly say I'm not getting any extra value from the dashboard; I'd be happy with the CLI only. My secret dream, which I reveal to you as a reward for making it here, is that this CLI could be recreated with Github CLI (gh) using their extensions' capabilities. This would undoubtedly make a great service to the popularity of stacked PRs, as it would decouple the CLI and the Graphite Dashboard. In order to make sure Graphite CLI can update the pull requests based on your stack, you need to authenticate towards their API (thus you need an account there). However, I believe you should be able to do the same thing by calling the Github API directly from the CLI. Well, maybe in the near future, who knows? If you would like to take a stab and kick it off, let me know; I would love to help.

At the very start of my career, it was all about going solo. I started as a solo developer, doing tiny contracts and barely making more money than I would make if I spent the same time at a local McDonald's. No matter what, I loved that - earning money with something that started as a hobby is very empowering and I feel grateful for it.

I quickly realized that it wasn't a way forward forever - at least not for me. If you go solo, it is important to have strong mentors within reach; it is much easier to learn with great teachers around you. The exercise I like to use to check how I am doing is to ask myself if I ever feel like I am the smartest person in the room. If I do, it means I am in the wrong room and I should move on. It doesn't work well for this corner case, right! Being exposed to other, more senior, developers, and other roles in product development was essential for me to broaden the context; it helped me massively to see the value of considering the long-term impact of my work and striving to solve real customer pain points rather than just churning out code. This shift in mindset has been crucial for me in terms of delivering more effective solutions.

Like many developers and engineers from my country, later on I moved to an agency delivering (usually small) software solutions. And honestly, it was great. Here and there I had ad-hoc mentoring sessions with my peers and I feel quite strongly that a good agency is a pretty good starting point for one's career. Looking back at my younger self, I was able to touch a variety of different tech - implement them from scratch into greenfield projects, see them fail and rebuild them again. I was able to experiment with different languages, frameworks, and services. It would take an entirely different article if I dived more into how development, 10 years ago, was for me. However, that's not what I want to write about. At least not today.

The thing different agencies have in common is the fact that it's usually hard to be working on something personally meaningful. Something you can connect to. Yeah, you might be super invested in some new technology and then the client might be so kind as to have you implement it for them. But sooner or later you realize that if you are detached from real customers, you'll start losing interest and, worse, you will solve the wrong problems. You might spend months building the best real-time architecture just to realize there is little to no collaboration happening and simple polling would do the job.

When I was a child, I developed games for fun; Game Maker was my obsession. It was amazing to be able to create games, rules, and universes - whatever I could think of. Maybe also because of this, I realized what I needed was to be directly involved in product development, to be part of something great for the end users, creating emotion and delivering value to them. I know it is tempting to focus only on the technical aspect, but I ultimately think the outcome is all that matters. I'm a missionary, not a mercenary; your users don't care if your Todo app uses Redux or not, they care about getting value from it.

Outcome is the key: it took me a few years in my career to recognize that aiming for outcomes rather than outputs is the key to success and feeling grateful for what you build. Solving some user pain by removing a bunch of code is the nice outcome - I'd even argue that's what I consider the best, and I celebrate every moment when I remove code, especially my own. The difference is that if you are judged by outputs, it is much harder to sell the fact that you removed code; at worst, you might even look incompetent for having written it in the first place. Shame.

If you have never experienced a truly empowered product team, trust me, you are missing out and you should fix it. If you want to read something about this topic, I highly recommend INSPIRED: How to Create Tech Products Customers Love by Marty Cagan.

The book describes, among other great things, four big risks in product development:

• market risk (will the product or service be perceived as valuable by potential buyers and generate sufficient demand?)

• usability risk (will users be able to easily understand and use the product or service as intended?)

• feasibility risk (do we have the necessary resources, skills, and technology to successfully develop and deliver the product or service as planned?)

• business risk (will the product or service support the overall goals and objectives of the business and be financially successful?)

In a nutshell, there are two basic buckets where you might end up as an engineer: a feature team or a product team. Your affiliation determines all the rest of the relationships you might experience while being a member of either one or the other group. Feasibility is a concern of engineering, that's clear. Hopefully, you always have a dedicated designer, so you can cross out the usability risk as well. Although good product designers want to seek solutions, not draw borders and buttons someone sketches out of the blue. Where it starts to be tricky is business and market fit.

In a truly empowered team, this is a concern and explicit responsibility of the product manager. On the other hand, in feature teams, it is usually an executive or another stakeholder pushing the feature onto the roadmap with the expectation of a return on investment, expanding the business, or both. The product manager then serves more like a project manager, trying to handle all the parties and eventually helping the project to get delivered, left at the mercy of stakeholders.

You might be wondering what happens in case the thing you deliver is not performing as expected. You guessed right, it will mostly be your problem. They would blame bad design, that it took too long, or that it was underutilized, or all three together.

On the contrary, in the case of empowered product teams, you do the research as a team (I highly recommend taking a look at the Double Diamond approach), you propose the solution, and it is your responsibility to properly validate the solution and keep iterating until the problem is solved. The goal is to solve the problem, and learning how to do it well enough is part of the journey.

Of course, nothing is that simple in the real world, and the buckets I described are more like a spectrum that your team might be on. However, there is likely space for both types of teams, especially during different times and company phases. It is essential for you to understand where you stand so you can set your expectations correctly.

One last thought I cannot resist pointing out is team play, a crucial part of product teams. By being too restrictive about the roles and responsibilities of each team member, you may not fully utilize their skills and expertise. In my opinion, engineers should be allowed to participate in activities such as product discovery and strategy sessions, as they often have valuable insights as users of the product and may be able to identify opportunities for improvement that others might overlook. Encouraging cross-functional collaboration can lead to more creative and efficient solutions.

With that said, I wouldn't be surprised if a really good craftsman would prefer to work in an empowered product team where they can work on more than just stacked JIRA issues, prioritized by someone who might no longer be with the company.

As you know, things changed at Twitter; I wouldn't say for the better - it's a pretty big shitshow if you ask me. However, that's not really the point. Social media changed, and I'm not sure if it's just me, probably not, but I got to the point where I could barely stand the polarization and hatred in every tiny bit of information I glanced at. The times when I was scrolling through the timeline of cool tech stuff are gone, now everywhere are just fancy clickbait-ish headlines with little or no value.

What is the point? What am I getting back for my time? I asked myself multiple times. The times when I had fun or learned something that would help me progress in my life on Facebook, Instagram, Twitter, LinkedIn, and other sites I used often are gone. I remember pretty clearly that once Instagram added the Stories feature - I got lost - I kinda didn't know how to use it, or maybe I just lost some narcissistic part of my personality; however, I haven't picked it up till this day. Maybe I'm just getting old.

So here's what I plan to do now - whenever I have the itch to open Twitter, I open Reeder instead. I've been collecting blogs and interesting articles for more than a year now and Reeder does an awesome job of that. However, I was always struggling with finding time to go through the things I saved there. Silly, I know. I'm not trying to set any goals or something like this - I guess there's no need to torture my brain with more numbers, badges, and achievements to compete with. It's more about that approach - whenever I decide to consume something, I should try to bet on something worth it. My father was always saying: “One should finish the book, even though they think it's bad. It's good to create a sense of what's bad, to recognize it later”. He's not wrong, at least not entirely - but I'm trying to internalize that once I know it's bad, I'm just going to drop it. Life is too precious to be buried in poor content.

I'd also like to go further and try to establish some habits to actually create things. To get better at writing, for example. After all, writing is quite a big portion of being a senior+ engineer. This post is a kind of contract between me and future me. To have something tangible for my next reflection. We will see. Until then; see you around, my friends.

P.S: The cover image was generated by Stable Diffusion with this prompt: Abstract oil painting of a man who just signed contract with themself as over-the shoulder shot via Diffusion Bee.

One interesting thing I have observed over time is that open-source solutions tend to struggle with bigger code bases. That’s when custom tooling comes into play — or not necessarily. The truth lies somewhere in the middle.

To give you a clearer picture, cloc says we have almost 400k lines of TypeScript in our frontend monorepo.

That mainly comprises the three apps we develop here at Productboard — Portal, Signup, and our main app. To put things into perspective, Github says that for the last week, 35 authors pushed 110 commits to master (yet to be renamed to main— two character less to write, yay!) and 322 commits to all branches, excluding merges. That’s huge! We try to stick with small pull requests, which we then squash and add on top of the master with Kodiak — but more about that later.

To support this heavy machine, we have established an organization I’m leading called Frontend Platform. The team’s mission is to empower engineers and enable productivity. Ultimately, we want to deliver value to the customer, and we want to do it right and fast. That sounds straightforward, but it comes with challenges, especially at this scale. On top of that, we strive to deploy to production multiple times a day. So how exactly do we do it?

With tooling we stand. Without it we fall.

We realized soon enough that without a proper way to map dependencies between parts of our monorepo, we end up with one big ball of mud. Without clear boundaries between modules, it was hard for us to allow teams to experiment with patterns and approaches because it was still part of a monolithic application. So, for the sake of consistency, you would have to create a strategy for everyone.

Another logical consequence of not knowing the dependency graph and not building applications from many independent modules is that you always have to run tests, lint your code , basically everything — just name it — against every line of code. Simply put, you treat the whole monorepo as one big project. And trust me, when you are talking about 400k LoC, it takes some time. In darker days, our frontend feature branch pipeline took us almost an hour to finish.

We had the most obvious problems laid out in front of us. To sum it up, we wanted a unified way to introduce modules that would be easily isolated from the rest of the stack via a public API. Secondly, we wanted a way to track dependencies between those modules, so we could ultimately say what modules you are affecting within your PR and thus exactly what the CI pipeline should test, lint, build, and eventually deploy.

First, we were looking in the direction of scaffolding tools like Yeoman, Hygen, and Plop, but we soon realized they wouldn’t be enough for our use case. Essentially, those tools are focused mainly on scaffolding features, but that’s just one part of the story for us. We wanted to get insights into our monorepo — what’s being built and what dependencies those projects have. We scoped on Rush and later on Nx, before finally deciding to give Nx a shot. It’s feature-rich, open to the community, extensible, and based on battle-tested Angular CLI.

We started in Q1 2020. Fast forward a year, and we have registered more than 230 projects — every one of them benefiting from the ecosystem we put in place. We had to use some shortcuts to be able to work within an already-existing codebase, but in general, we were able to leverage Nx for every part of our codebase.

I guess now’s the right time to dive a bit deeper and explain how it really works — and how it affects our engineers on a daily basis. Let me describe a day in the life of our favorite employee, Joe Doe (you know, my colleagues would be mad at me if I picked one person and not them 🤓)

The story of Joe Doe

It’s a lovely day, and I’m working on an exciting new feature. The first thing I’m going to do is spin up the module for it.

I run a command, write the name of the module, and puff! My files are automatically created. Then I realize that I also want to scaffold a storybook for it, so I write another command, and abracadabra! The storybook is ready to use.

Now it’s time to do some magic. I can write another command that runs tests for my module in watch mode, or I can start the storybook I created earlier and start to prototype the functionality there.

Once my code is ready to be integrated, I simply open the PR. The CI pipeline is configured and powered by Nx, so the tooling automatically runs tests, linter, and similar only on the affected projects in the dependency graph. After a few minutes, I get some info from our bots. It might look like this.

We use https://danger.systems/! 🤘

It’s worth noting that I can access the revision under a special URL, so I can quickly showcase it to my team. Speaking about my team, once they do a review, they might see something like this.

This is custom tooling built on top of @actions/github. Do you want to hear more?

Once the PR is resolved, it’s time to merge it and have a cup of tea. We have robots for that as well — I mean for the merging, not the tea-making. I simply add a label, and Kodiak will do the rest. It integrates master to my PR, waits for the CI to pass (so we can be sure that the integration is OK), places my PR into a queue, and boom! After a couple of minutes, it’s merged.

Don’t worry about that merge of master to my branch. Since the PR is up to date with master when it is being merged, the merge commit will be rebased away. With that, our master branch looks slick and clean — no branches, just pure fantasy!

You might ask why we have this step here. Great question! Before, it often happened that we felt our PR was all right. All the tests were passing. You know the drill. But once we merged it to master and the code was integrated, something went wrong with the newest changes. Our master pipeline started to fail, and everybody got blocked by it. So we decided to protect our master and keep integrating code on feature branches.

One great side effect of this approach is that we don’t need to recheck everything during the master pipeline — it has already passed the checks on the feature branch. That means it’s also much faster and can directly proceed with deployment and e2e tests.

So that’s it for today. My PR is merged. Mission accomplished! Now let’s deploy it.

Back to reality

Joe Doe succeeded and delivered — after all, I like the phrase, “what’s not in the master doesn’t count.” His feature is already on staging (thanks to continuous delivery) and exactly one click away from production — auto-deployment to production and rollback is something we currently have on our roadmap along with incremental builds, for example.

If you want to know more about how Nx can affect your developer experience, give it a shot! You can check out this wonderful presentation by @juristr — Speed up! Incremental Compilation with Nx.

Remember at the beginning of the article when I said that before we introduced Nx, the CI pipeline took almost 50 minutes to complete? Well, where do we stand now — 5 mins? 15 mins? It all depends on the PR. Some teams have AVG feature pipeline speeds of around 7 mins, some 15 mins. It all depends on how big their PRs are and how many projects they’re affecting. Now that our architecture scales, it’s all in their hands!

By the way, our PR lead time oscillates around 24 hours, which is one of our north star we measure, and we spotted significant improvement once we integrated Kodiak into our flow.

I’m aware that I’ve described these topics faster than Flash running through Central City, but I hope you got some idea of how it works here at Productboard. Initially, I was thinking about doing a series on this topic — but frankly, folks, I don’t trust you would finish it. Feel free to prove me wrong in the comments!

Last but not least, I would like to recognize @martin_hotell and the rest of the team, who laid the foundations for all of this to be possible! Thanks, Martin!

I could talk for hours about exactly how we’ve executed on this and what the situation looked like when we decided to establish the team. But for the purposes of this article, I want to focus on one practical way we’ve improved our ability to make impactful changes as we scale.

Addressing those growing pains

A couple of months ago, we realized that our Fronted Architecture meeting (as we called it back in the day) no longer worked. It was great when we were essentially half the size we are right now, but it stopped scaling pretty quickly once we grew. And that’s without going into how switching to a completely asynchronous environment took its toll! Ah, 2020, the year of pushing comfort zones!

I teamed up with my mentor from Plato (Hi, Jennie!) to figure out how we could improve the way we share ideas, give feedback, and make changes as the team grows. We decided that it might be time to completely revamp our RFC process, which dates back to the origin of the Frontend Platform team. That’s right, we actually had an RFC process in place, but it never really took off. This time, we had to do a few things differently.

Designing an RFC process that works

Before, heavily inspired by the OSS community around the Facebook projects (namely React, React Native), we had a similar kind of template versioned in our monorepo. But I could count on the fingers of one hand how many times it was actually used. Luckily for us, as we grew, we had another problem to solve — we needed a record of impactful changes that we could refer back to in the future. Creating a culture around RFCs effectively allowed us to kill two birds with one stone.

One problem I saw with git-versioned RFCs was that the process was very heavy and formal. You had to open a PR, commit it, and so on. Also, it hasn’t scaled for our other projects — we have a monorepo with mostly frontend code, but we have other services in separate repositories.

As I thought about the problem, I wanted to avoid overcomplicating things by introducing new tools (a new process with a new tool? That’s a fast track to hell!). And truth be told, we have enough tools as it is. So, given that we are pretty fond of Notion here at Productboard — actually, we love it! — I thought, why not use Notion as our RFC database?

We revamped the old template and simplified it (heavily inspired by Design Docs at Google). We also emphasized that it’s an informal document — no need to rigidly follow the template, just put your thoughts down so everyone can participate and be on the same page. Simple.

Next, we aligned at a leadership level to support this initiative across our teams. In essence, the process was as simple as this:

Are you gonna deliver something cool? Awesome! Do you need to align with more than your team? Sweet, now document it! If not, is it a huge change that will potentially affect someone outside your domain? Great, go ahead and document it!

Now the last step: How to spread the word? Confluence has its notifications. Notion has a notification system as well, but it might be hard to follow — it can get noisy, and no one really looks at emails. You know what we like besides Notion? Slack!

So we decided to create a simple notification channel in Slack called “n-rfc”. Each time someone adds an “open” document for comment, a new notification appears there. It’s nothing fancy, but it works. And the best part? It’s open-sourced, so you can use it as well.

The system works!

If you are wondering how it performs, I can honestly say that it has exceeded my expectations. We established our new RFC process at the beginning of Q3 2020, and so far, we have created more than 100 documents this way. That’s pretty impressive given that we’re an organization of under 90 engineers.

That’s it for today, folks. Now over to you — how do you approach changes in your organization?

Your yarn.lock file ensures that your package is consistent across installations by storing the versions of which dependencies are installed with your package.

In other words, this means you can be sure that you are working with the same dependencies (and their dependencies) as your colleagues. The same also applies to the CI systems we are using.

Sounds good, right? Sure! But things aren’t always so straightforward. Different versions of Yarn may produce a different yarn.lock. This happened to me when Yarn introduced the integrity field.

When this happens, you might end up requesting changes to your colleagues’ PR to upgrade their Yarn accordingly and regenerate the file to keep the diff clean.

So what now? Should we update the main Readme with the supported version of Yarn? No! We can go one better…

Enforcing the specific version

One way to enforce a version within one project is to use the engines field in package.json (see documentation).

{ "engines": { "yarn": "1.21" } }

This allows you to enforce a specific version of yarn for everyone who will run Yarn commands including add, for example.

In this case, when you try to install a new package, you may get this message.

That’s better. But still, you need to install the requested version manually. Now, let’s imagine that the recommended version will change — you would have to repeat the process again! Frustrating, eh? Don’t worry, there’s a better way…

One Yarn for everyone

This headline may sound silly but bear with me. The authors of Yarn have been facing this specific issue themselves. Thankfully, they’ve come up with a great solution.

Ladies and gentleman, allow me to introduce the best way to keep your Yarn version aligned across the team: yarn policies.

What is Yarn policies all about?

yarn policies set-version offers a simple way to check in your Yarn release within your repository. Once you run it, your configuration will be updated in such a way that anyone running a Yarn command inside the project will always use the version you set.

And how do I use it?

yarn policies set-version

This command will set the latest stable Yarn as the default for everyone working in the same repository. It’s as simple as that.

A Yarn binary snapshot will be stored within .yarn/releases along with the updated yarn-path in the .yarnrc configuration file. Now, you’ll be using the locally “installed” version of Yarn.

And last but not least, make sure to push these changes to your remote, so everyone within the team gets the same benefits — one exact, locally scoped version of Yarn, no matter what version they have installed. No more yarn.lock deviations! Beautiful.

Final touches

If you’re using GitHub, you might be familiar with the language distribution dashboard. This is how our repository looked before we introduced the change I just walked you through. Let’s say you’re using TypeScript, as we do.

And this is how it looks after I merged my lovely changes.

Wait, what? Allow me to explain…

GitHub uses a program called Linguist to calculate code distribution. And by default, it processes all the versioned files.

Thankfully, you can control this process. Simply edit your .gitattributes as follows to ignore this file from the graph.

.yarn/releases/*.js linguist-vendored

And that’s it! If it’s not versioned, it didn’t happen!

Kudos to Martin Hochel for proofread!