I used to be on a team that configured load balancing for everyone else at the company. Our documents said we needed a week’s notice to add a new backend to our balancers but, truthfully, each request took about half an hour. We needed to provision extra capacity, change configurations, and restart services. It was a frequent task, and a straightforward one for someone who knew what they were doing. Other teams knew that what we were doing wasn’t rocket science, so we regularly got requests like “We’re launching tomorrow, so please set up our load balancing today.” And, usually, we didn’t. Why did we insist on a week’s notice? Because these configurations weren’t our only work. Hundreds of teams used our banks of balancers, and load balancing was just one of the four critical services my team supported. We didn’t want to react constantly: we wanted to plan out our weeks, and to make these configuration changes in batches rather than continually restarting services every time. As a result, we had little sympathy for people who came in hot and angry about why we hadn’t done the thing they’d told us about only a few hours ago. Our team motto became “lack of planning on your part is not an emergency on mine.”
Link:: Staff Engineer’s Path