Most people have seen Steve Ballmer dance about developers developers developers! If you haven’t you absolutely should. Ballmer might have had an interesting way of going about it, but if you are a tech company, you need that enthusiasm for your developers. Sure he may have been talking about external developers, but I firmly believe that feeling should apply to developers at your company too! When you are a small company it might be as simple as part of one of your engineer’s time, and at a larger place it might be a dedicated team, but one thing remains true, you need to give your engineers the right tools to be productive.
I’ve always been interested in developer productivity, right from my time at Twitter. I’m not sure what got me so interested, perhaps that your customers sit right next to you, or that you get to see your product be used, and getting feedback is almost instant. I decided to spend some time writing down some interesting stories through my time focussing on developer productivity and how they had a lot of impact to the companies I was at.
Twitter - My foray into devprod
A couple of months after joining Twitter I noticed you could publish internal chrome extensions and this sounded like a great way for me to try to build something from scratch. I didn’t really have any great ideas at the start, but I really just wanted to build something. In my quest for ideas, I realized that people opened JIRA tickets from Gmail just to click resolve, or look at the status. So I built a chrome extension that scraped the Gmail page when you loaded it and literally pulled JIRA statuses and added a background color to the email row. The extra feature was a button that let you single click close / resolve tickets right from Gmail. I published this extension and sent out an email to a few mailing groups.
Apart from my team, no one seemed to be using the extension. I was pretty bummed, but new projects came along and I got distracted. A few weeks later my manager pulled me aside and said he got an email about the chrome extension. It turns out that enough people were using it that the JIRA team at Twitter saw their servers being bombarded with requests. That’s right, new grad Roopak had never thought about caching or being optimal in any form or manner. Every time a Twitter employee refreshed their Gmail tab, the chrome extension sent an API request for every single email from JIRA 😂. I added some caching and updated the chrome extension, but more importantly, I had built something that people were using.
A few months later I discovered a better problem to solve - testing proof of concept & dev versions of features & services easily. Twitter had a way for their employees to direct their clients (web, android & iOS) to dev servers by setting custom headers. However if you wanted someone to try out what you had built, and there was more than one service & multiple feature switches involved, good luck with making that happen. This time, I recruited a partner-in-crime – @chander. We built “ConCon”, a simple chrome extension that lets you create bundles with the headers and feature flags you wanted to set. We went all out – it worked on iOS, Android, & web. In fact, we made it so simple that if you clicked on a ConCon URL, which looked something like this https://concon.some-twitter-internal-domain.com/bundle_id_here, it would set up your client with all the feature switches & headers.
Chander & I launched this at BlueBird demos, Twitter’s monthly demo hour for the product org. I’m not going to lie, we had literally scripted every single line and joke we were going to make during that presentation. After two weeks of hacking every evening, we were excited to launch this thing! Validation came 2 weeks later when a C-level exec mailed us saying ConCon links don’t work if you weren’t on VPN. I actually don’t remember if we added this functionality, but ConCon was being used by a lot of developers. In fact we saw product managers sending out emails with ConCon links.
The team I was on at Twitter had their direction change every 6 months, and a lot of the stuff I built, I would unship myself. That being said, I am told that Concon is still used at Twitter today about 6 years later. Two weeks of evening hacking, by 2 developers saved hours of time across the company.
AtSpoke - Understanding the importance of local dev
Fast forward a few years and I was at a startup called Spoke. On my first day at Spoke, in a matter of hours, I had my own version of Spoke running and I could test any part of the system easily. Sure I was learning about the company and the product, but what stood out to me was how good the local development setup was. If I didn’t appreciate this then, I definitely appreciate it now. The pace at which we shipped features at Spoke was definitely aided by how thoughtful the team had been about our developer tooling. Needless to say, I did not hesitate to spend more time optimizing our dev tooling. One could say that in some cases I built what we didn’t need for a company of our size, but I learned what was helpful and what was just me exploring.
Bolt - Digging in deeper
When I joined Bolt, I spent the first month of my time optimizing the build pipelines to be faster and then building our local development environment. Engineers at Bolt pushed to staging to test a change. This meant that we literally validated the XKCD comic and Bolt engineers waited in a queue to test changes and for their binaries to be built.
Initially it was me pushing folks to test locally instead of taking over staging. We had a chart in the office where we counted the number of times someone deployed to staging to test. Soon it wasn’t my war anymore, it was the Bolt engineering teams goal to stop testing on staging. We eventually hit our goal of 20 and I immediately made it harder to take over staging.
We didn’t stop there, soon afterward we decided to move our staging environment from manual deploys to CD. One would imagine it was just hooking up a bunch of scripts to call one another. But of course, life is never that easy in practice. I managed to convince Ananya, one of our engineers who had joined recently that this would be a huge win for the team. It’s always fun to see an engineer take on a challenge as their personal goal. She knocked this out in a matter of weeks and proceeded to later make CD for DB migrations too (another story). The number of developer pains we saw reduce over the time we made these changes were huge. Engineers were a lot more productive and the amount of time waiting for nonsense things like CI to build so you could take over staging to test a feature went from > ~40 minutes down to 0 since you could test locally. The amount of time we saved engineers (and how much happiness)
We didn’t have an official dev prod team back then, and we are starting one now, but the fact that we had engineers who wanted to improve our systems made up for the fact that we didn’t have an official team. If one were to go by Peter Seibel’s math we were definitely too early to have a team dedicated to devprod, but it is never too early to focus on your developers. Given our massive growth last year, we are changing that around and staffing up a team to constantly think about developer productivity.
Invest in your engineering team, make your developers' lives better and make your engineering team go further. If you want to talk developer productivity or work on making engineers lives' better hit me up at rv[at]roopakv.com or DM me on twitter @roopakv.