At NerdWallet we have a bi-annual review cycle to evaluate how we are doing both as NerdWallet engineers and as professionals. As a part of reviews, we use a standard job architecture when performing evaluations across the organization, with the intent that everyone is judged on a level playing field. After our most recent review cycle, though, we felt that our engineering job architecture was starting to show its age. We last updated it more than two years ago, and it simply wasn’t meeting our needs any more. Originally created for a team of 25 engineers, some of the nuance in the job architecture that was easy to convey with the smaller team was being lost with our larger ~100 person organization. Managers weren’t always evaluating their teams in the same way and some of the engineers were approaching the architecture as a checklist of tasks to execute and less as an embodiment of expectations and values for the engineering team. It was perfectly rational for people to do this. As engineers, a major part of our jobs is to take complex, sometimes nebulous requirements and impose logic and structure on them. So, naturally, people might look at the architecture as a discrete set of steps to take to get to the next level. To address these problems, a cross-section of the engineering team set about rewriting the job architecture to better suit NerdWallet’s needs.
We all agreed that the biggest issue was that people would too often approach the job architecture as a discrete set of steps to accomplish as opposed to a framework outlining a craft to master. This is not surprising, in fact, it’s fairly common to find progression encoded as a series of discrete requirements to achieve a specific goal:
- If you complete 120 units of coursework you qualify for a bachelor’s degree. Completing another 30 units and a thesis will earn you a master’s.
- Pass a test and log 30 hours of flight time to get a recreational pilot’s license.
- Get a bachelor’s degree, pass the CFP™ exam, log three years of relevant experience, and pass a background check to become a Certified Financial Planner.
Was it really so bad to have the job architecture function in a similar fashion? It makes sense to have clear guidelines like this, particularly when you’re dealing with large populations of people. There are approximately 600,000 active pilots in the US. Evaluating them all on personalized, qualitative criteria is unfeasible. Broad-based quantitative standards are necessary for our aviation system to operate with any semblance of efficiency, even if it obscures nuances around individual achievement. It is natural for people to think of their career progression in a similar fashion and for companies to create job architectures using this same model. Designs that work well for larger groups of people, though, have significant issues when applied to much smaller populations:
- A company’s needs, particularly startups, can evolve from year to year, or even quarter to quarter. Even if you can enumerate exactly what you need today, that may change as the company changes.
- Not all skills can be evaluated in a discrete and measurable way. As an example, most companies value clear communication skills, but like most soft skills, evaluating communication ability is largely a subjective exercise.
- All metrics are gameable. For example, as professional engineers, we want low bug counts, but the best way to avoid bugs is to not ship code.
- If you present people progression as a set of hoops to jump through, they will largely focus on jumping through those hoops. This is counter to the learning and growing mindset that we want engineers to adopt when approaching technology and development.
For all these reasons we decided to move away from isolated enumerations of tasks and towards more subjective statements of values and expectations for engineers with examples provided at every level. Additionally, we were much more explicit about how people should use the architecture, since it wasn’t entirely clear before. We want engineers to approach software development as a craft to be mastered as opposed to a certification or license to qualify for. We created the new architecture with this in mind. Code as craft is not a new concept, but it is always one worth reiterating, particularly as technology grows and evolves both inside and outside of NerdWallet.
Of course, this method of evaluation comes with its own challenges. How do you map subjective criteria into discrete job titles? How do you ensure that managers are using identical subjective criteria for evaluating engineers? Most importantly, as with all refactors, how do you know you’re not just introducing a system that is as broken as your previous one, but in different ways?
Issue 1: Leveling
At NerdWallet, we have six levels for individual contributor engineering. Leveling is a blunt instrument, but for any non-trivially sized engineering organization, a necessary evil. Leveling helps you ensure that your team is being properly compensated and helps your engineers demonstrate professional growth when they eventually decide to move on. How do you distinguish work at one level versus another, though, particularly in the context of subjective criteria? We decided to tackle this from two perspectives:
- What is the high level description of a given engineering role, independent of specific values? If done correctly, we should be able to identify a distinct difference between each level to justify maintaining our current set.
- Who are the current engineers at NerdWallet that embodied our engineering values for a given level, and why did we think that was the case? This exercise would help us back into what our expectations were for any given level.
For #1, we came out with the following high-level distinctions.
So, overall, the levels seemed about right. There were some nuances and questions, but it was close enough that it didn’t warrant the disruption necessary to add or remove a level.
For #2, we identified engineers from NerdWallet that were emblematic of their level. We then talked about their specific skills and behaviors, and how they showed up in specific circumstances. One thing that became clear fairly early on, was that even within discrete levels we had well-qualified engineers who showed up quite differently from each other. This led us to the idea of providing wide-ranging examples as a way of demonstrating what was expected without being overly prescriptive. Eventually, this evolved to underpin the format that we settled on (below).
- <broad statement>
- <specific example>
- <specific example>
- <broad statement>
- <broad statement>
- <specific example>
- <specific example>
- Skill/Behavior – A short statement as to what is expected at a given level
- Rationale – More in-depth reasoning and context for the above expectation
- Broad statements – General statements that demonstrate the expectation
- Specific example – Anonymized examples of behaviors we have directly observed from NerdWallet engineers
This format also has the benefit of letting us do minor refactors to individual parts of the architecture. For example, if the rationale is poorly worded, or the examples are misleveled, we can easily move things around without fundamentally changing the architecture. That wasn’t really possible in the previous architecture where changing a single bullet point could significantly change the expectations around a given role.
Issue 2: Fair evaluations
At NerdWallet, we have two broad categories of engineering: Product and Platform. Product engineers typically have more front end skills and are expected to deliver features for our users whereas platform engineers typically have more back end skills and are expected to deliver features for internal stakeholders. These are not formal roles and they are mostly captured as a function of the team you work with at NerdWallet. This particular dichotomy had unintended consequences, though.
At the Staff Engineer level and above, there was originally an expectation that an engineer have impact on the broader engineering team as a whole. This requirement tended to favor platform engineers who have more of an opportunity do wide-ranging work as a function of their role. Product engineers on the other hand are typically focused on their specific domain. The obvious outcome of this disparity was that talented Senior Engineers on the product side would either have to work harder than platform engineers to earn a promotion, or move out of product engineering to focus on platforms. Neither of these outcomes were acceptable to us. Just from a standpoint of fairness, we didn’t want equally talented engineers judged by different standards, and as a user-first company, it was important for us to have a strong product engineering discipline.
At first we considered the possibility of maintaining separate architectures, but we all agreed this would be infeasible. We wanted a single architecture that could flex and grow with the company, not multiple that we would have to manage and coordinate between. We instead decided was that while the intent of the requirement was correct, the way it manifested in the architecture was poorly worded. In particular, we do want Staff Engineers to have reach beyond their immediate team, but there is no reason to constrain that influence to the engineering organization. So, with Staff-level product engineers, they will most likely reach outside of their team when engaging with product managers, designers, and user researchers. More generally, Staff Engineers, are expected to drive improvements in outcomes with their partners and stakeholders, regardless of who they may be.
Issue 3: Validating the refactor
As with all refactors, it was important for us to make sure that we weren’t just trading one set of problems for another. We started by reviewing how the new architecture differed from the old one. To our surprise, there were relatively small differences between the core skills defined in each architecture. The real difference between the two architectures was the amount of context and framing we put around the core skills we were looking for. This gave us confidence that the new architecture was not a radical departure from previous expectations and would provide us with a solid foundation from which we could evolve.
As chance would have it, we finished the first draft of the architecture right before mid-year reviews. It would have been premature to roll it out right away, but it presented an opportunity for us to beta test it with engineering managers. We asked them to use the then-current architecture for official reviews, but to compare it side-by-side with the new architecture so they could provide contextually relevant feedback on what they felt worked and what didn’t. Almost all feedback was accepted with little issue. In the old architecture, changing a single bullet point had a much higher likelihood of changing the responsibilities and expectations for a given role, but the greater verbosity of the new architecture meant that we could much more readily incorporate feedback without altering the underlying intent. After a couple of rounds of feedback from the managers, we decided that the architecture was ready to share with the wider organization.
Rolling it out
Since we changed a great deal about how we present and contextualize the architecture, we had to discuss it in multiple forums to talk not just about what we changed, but why we changed it. Just as the heavier emphasis on subjectivity puts a greater obligation on engineering managers to evaluate each engineer in the same way, there was an obligation on us to help people understand the fundamentals and rationale of the new architecture. This blog post is a part of that effort, revealing our thought processes and reasoning.
Going forward, we intend this architecture to be a living document. Certainly not one that we will make frequent or drastic changes to, as there are benefits to consistency, but one that we will continue to evolve as we better understand the needs of the company and the engineering team.
Ready for that promotion to Principal Platform Engineer? Check out some of the open roles we have on the team here at NerdWallet.