Tech startups face mounting expectations for newer, more intelligent features, faster performance, always-on resiliency, enhanced security and much more – creating an endless need to out-innovate and out-perform the competition.  To keep pace, technical leaders must traverse a vast ecosystem of tools, unpack an evolving set of modern engineering practices, and overcome an unprecedented war for top talent.  We at Sapphire Ventures believe this digital arms race is fundamentally shifting the playbooks for engineering success.

To help support startups as they navigate this complex landscape, Sapphire hosted our first annual Hypergrowth Engineering Summit, a community-focused event that convened over 100 technical leaders (VPEs, CTOs), representing 75+ of some of the most innovative startups on the planet, for a day of TED style talks, panels, in-person networking (and whiskey!).  The purpose of the Summit was to share best practices, discuss future trends and ultimately become better engineers, together.  It featured a luminary speaker lineup, including Dave Rensin (SVP of Eng at Pendo, a Sapphire portfolio company), Charity Majors (co-founder & CTO at Honeycomb), Amit Prakash (co-founder & CTO at ThoughtSpot, a Sapphire portfolio company), Rukmini Reddy (SVP of Eng at Slack), Tramale Turner (CTO at Taxbit, another Sapphire portfolio company), and Angie Ruan (VP of Eng at Chime), just to name a few!  The Summit was sponsored by Sapphire’s Infrastructure & DevOps Center of Excellence, a platform which enables engineering leaders to scale both their organizations and products through strategy guidance, peer learnings, playbooks and partner connectivity.

This blog provides a brief recap of key learnings from the Summit which centered around Optimizing Engineering Organizations, Maturing Builds & Deploys, Software Ownership and Resilience Engineering.  In addition to this summary, we encourage you to check out the video replays to unpack the full breadth and depth of the talks.

Building Mental Models to Predict the Future

Engineering leaders face significant challenges attracting, onboarding and retaining engineers, in the face of a market that is maybe more competitive now than ever.  Dave Rensin (SVP of Eng @ Pendo) gave an inspiring opening keynote about the role that managers must play in reducing anxiety, which he described as “the cost of intelligence.”  When individuals struggle to grasp expectations or the ‘why’ behind decisions at work, they become anxious and, as a result, less productive.  Dave explained that it is the responsibility of managers to help their engineers “build better mental models to predict the future.”

One way to achieve this is to grant engineers autonomy and protect their sense of agency. Avoid top-down mandates (‘thou shalts’), as central authority often creates what Dave described as “malicious compliance.” Managers should endeavor to establish principles, explain the why behind those principles, and incentivize teams to stay within the boundaries of them as they execute on objectives – all other decision making should be distributed, and pushed to the edge wherever possible.

Measuring Engineering Productivity

Though valuable, collecting and reporting on engineering productivity metrics (DORA, Flow) such as cycle time, deploy frequency and success rate, can be met with skepticism by engineers who often perceive them as a ‘big brother’ apparatus, incapable of capturing the essence and art of software development.

While speaking on a panel hosted by Casber Wang (Partner @ Sapphire Ventures), Amit Prakash (CTO @ ThoughtSpot) cautioned that setting productivity targets at an individual level can actually “disincentive collaboration,” creating a myopic focus on specific KPIs (e.g. # of commits) vs functioning efficiently as a collective. Rukmini Reddy (SVP of Eng @ Slack) shared that metrics should be treated as “conversations starters,” useful for identifying anomalies and trends but not to be relied on for unpacking the full story. One way managers can shift the narrative, and make metrics more valuable to individual engineers is by linking them to business outcomes (e.g. tracking customer adoption of a new feature).

Remaining Technical as an Engineering Manager

Elon Musk recently tweeted, “Managers in software must write great software or it’s like being a cavalry captain who can’t ride a horse!”  Whether you agree with this sentiment or not, retaining technical chops is a real challenge for managers at all levels.  To that end, Tramale Turner (CTO @ Taxbit) took the stage to share tips for maintaining technical fitness.  He spoke about the notion of Engineerications – or temporary immersion in a delivery team, with the goal of performing at least one production PR.  He also suggested that experimenting with tech stacks in your free time, along with technical reading and writing, are great ways to remain curious while sharpening overall acumen.

Above all else, Tramale stressed that managers should stop worrying about what others think!  Instead, focus on what matters – providing teams with a best-in-class dev platform, reducing toil, clearly defining objectives and establishing a clear path for career advancement.

Software Ownership

As apps become increasingly complex (e.g. ephemeral cloud assets, loosely coupled architectures), any separation of Dev and Ops can introduce significant inefficiencies. Charity Majors (CTO @ Honeycomb) gave a riveting closing keynote about the importance of ‘software ownership,’ suggesting that engineers (and even managers) who ship code to production should be responsible for that code throughout its lifecycle, including spending some amount of time on-call.  But how do we make on-call more palatable, even for those 10x engineers that feel their time is better served elsewhere?

An important first step is to optimize the way in which alerts are managed. For example, shifting from symptom-based alerting (e.g. CPU is spiking) to SLOs tied to customer experience (and pain).  For every alert type, it is critical to provide an associated link to documentation and runbooks describing debug steps.  Observability and proper instrumentation of code are also paramount and aligning with standards such as OpenTelemetry will unlock community driven innovations and reduce lock-in.

Resilience Engineering & Socio-Technical Complexity

Inevitably, incidents will occur, and each is an opportunity to learn and improve.  Postmortems are a commonly used practice, however they often focus on the technical specifics of a given incident such as root cause.  To further optimize incident management, there are sociotechnical and team collaboration factors to observe and improve.  J. Paul Reed (Senior Applied Resilience Engineer @ Netflix) spoke about the concept of Resilience Engineering, or the capacity of a system and team to adapt to novel issues in real-time.  He emphasized that resilience is a verb, something that teams participate in together, and that every incident is an opportunity to validate and improve a system’s associated capacity for resilience.

Resilience can be observed via the accuracy and efficiency with which information disseminates through an organization and the inter-predictability of how teams will respond in moments of high stress. Inefficiencies can emerge due to a lack of a shared understanding of the sociotechnical system – different personas may have different expectations, and misalignment can lead to wasted time and prolonged outages. Risk Radar meetings are a useful technique for establishing this common ground, and aim to expose thematic, emergent risks which might occur in the future.

Maturing Builds & Deploys:  Monorepo Tooling

Engineers must continuously invent new ways to build and deploy faster, without sacrificing  stability or security.  At the code and build phase, remote caching tools and merge queues are increasingly being used to speed up build times.  David Apirian (CEO @ Trunk.io) shared that these tools are particularly relevant in the context of monorepos, where builds repeatedly pull down the same common packages, and where simultaneous merges can create conflicts that must be thoughtfully orchestrated.

Maturing Builds & Deploys:  Testing in Production

At the deploy phase, shifting right enables teams to invest less resources into recreating ‘production-like’ environments in Dev and Test.  Adam Liu (Associate @ Sapphire Ventures) hosted an intriguing panel about the current state of testing in production, with Rob Zuber (CTO at CircleCI, a Sapphire portfolio company) and Jonathan Nolen (SVP of Eng @ LaunchDarkly).  Rob pointed out that while the cost of fixing errors is generally lower as you shift left, “the cost of finding those errors isn’t always the same.”  Despite best efforts, there is some degree of uncertainty with every deploy, and feature flags and experimentation platforms offer a form of a safety net, reducing time and energy devoted to predicting possible failures prior to deployment.  As Charity suggested, “it’s not a question of whether you should test in production, because you already are!”

Maturing Builds & Deploys:  Supply Chain Security

As more and more code is borrowed from the open source community, it is critical to implement robust security checks throughout the SDLC. Kim Lewandowski (Co-founder and Head of Product @ ChainGuard) shared various techniques for mitigating software supply chain risks, such as treating build systems the same as production systems, using SBOMs for inventory management, and requiring code signing to validate project contributors. She also touched on emerging frameworks like SLSA which offer a valuable guide for implementing and measuring adherence to supply chain security best practices.

Paying Down Technical Debt

Technical debt is an almost unavoidable by-product of growth.  As product direction shifts and active users grow, architectures can struggle to keep pace and become stale over time.  And as developers come and go, they often leave behind technical ‘hot potatoes’ that require significant engineering effort to understand and dissect.

Preeti Kaur (VP of Engineering @ Carta) spoke about the importance of aligning engineering with the business, in order to effectively address technical debt. Simply put, if you cannot get the business excited about the need to solve a given issue, then you should rethink whether it is truly a priority. Engineers should pursue an iterative discovery and planning process, articulate clear metrics highlighting the ‘why’ for addressing a given piece of debt (e.g. reduction in number of tickets), and produce a corresponding set of time estimates that clarify when value will be realized.

In Summary

As startups seek to out-perform the competition, we have and will continue to see developers challenge the status quo, endeavoring to redefine the definition of engineering excellence. We at Sapphire are incredibly excited about the future of engineering practices, app architectures, tools and platforms, and the associated ecosystems that this continuous innovation will spawn.

We had an absolute blast hosting this year’s Summit, doubling down on our commitment to the engineering community.  Sapphire is focused on providing a space for technical leaders to collaborate and share with one another, via our Infrastructure & DevOps Center of Excellence.  To that end, we can’t wait to bring the Hypergrowth Engineering Summit back again next year!!  Until then, we encourage you to check out this year’s session replays, and stay tuned for more content focused on the emerging trends that engineering leaders of tomorrow must embrace.