How do you know your software developer is doing a good job?

You pay a developer every month, or you paid a big project fee last year. Then your partner or your boss asks a simple question: is the developer any good?

Most owners find they can’t really answer. Usually that isn’t because the developer is bad. It’s because nobody agreed how to tell.

So people judge by what the contract counts. On a project: was it delivered on time and on budget? On a monthly fee: does the team look busy? Neither tells you whether the software is doing the job you paid for.

What your contract measures

A fixed price measures whether the promise was kept. The agreed scope, for the agreed price, by the agreed date. That is a fair thing to check. But a system can pass all three and still sit unused, or solve the wrong problem.

The people who study this changed their minds about it. The Standish Group, whose CHAOS reports track how software projects turn out, used to count a project as a success when it was on time, on budget and on target. In 2015 it changed that to on time, on budget, with a satisfactory result. Its reason: many projects had hit all three and still returned no value to the organisation or its users.^⁠[1]

When McKinsey and the University of Oxford studied more than 5,400 IT projects, the large ones, those over US$15 million, ran 45% over budget on average. They also delivered 56% less value than predicted.^⁠[2] The overrun is the number people remember. The missing value is the one nobody was tracking.

A monthly fee, on its own, measures nothing. There is no finish line, so people count what is easy to count: hours, tickets closed, features shipped. A busy team and a useful team are not the same thing.

Both ways of paying can work well. You just have to decide what to look at, because the contract won’t do it for you.

Three questions that work under any contract

Is the business better for it?

You paid for software for a reason. Fewer order mistakes. Staff not typing the same data into two systems. Month-end closed in two days instead of five. Customers booking without calling.

Write down one or two of these before the work starts, using numbers you already track. Check them a month or two after each big release.

Be fair about this one. Your developer can’t move these numbers alone. If your team keeps using the old spreadsheet, the new screen won’t change anything. But a good developer asks what the number is before building, and asks again after release whether it moved.

Does finished work reach your users?

Software teams already have a way to measure this. DORA, a research programme on how software teams deliver, tracks five measures, including how often changes go live, how long a change takes to get there and how often a release breaks something.^⁠[3] Its research finds that teams doing well on them also do better as organisations.^⁠[3]

You don’t need the dashboards. Ask the same things in plain words:

• How often do your users get something new? Live and in use, not “done” on the developer’s laptop.
• How long does a small change take, from the day you ask to the day it is live?
• How often does a release break something that used to work?
• When something breaks, how fast is it put right?

There is no right number for every business. A payroll system used once a month moves differently from an ordering app used all day. Watch the direction. If releases are getting rarer, slower or more painful, something is wrong, whatever the updates say.

Can you see it, and could you leave?

• Can you see what was done without chasing? A short written update, on a fixed day, about things you can open and check.
• Is everything in your company’s name? The code, the database, the hosting, the domain and the app store accounts. More on this in who should own your code, hosting and accounts.
• If you stopped tomorrow, what would you have? A working system and everything another team needs to carry on, or a pile of half-finished work only one person understands?

On a fixed-price project

• Agree how “done” will be checked before work starts. Use real cases from your business that the system must handle. Then acceptance is a test, not an argument.
• See working parts along the way. If nothing can be opened until the last month, you can’t judge the work until it’s too late to change it.
• Check use after go-live, not just delivery. Are people using it? Are the bugs found in the first weeks being fixed as agreed?
• New requests get a price before anyone builds them. See what happens when requirements change.

A fixed project that passes its acceptance tests and gets used every day is a success.

On a monthly engagement

• Every month, something reaches your users. It should be something that mattered more than what is still waiting.
• You set the order. If the developer decides what comes next, you are paying for their priorities.
• Updates come without you asking. On our Managed Delivery, a written update goes out every Monday: what moved, what is ready for you to check, what is waiting on you and what comes next.
• Ask yourself each month: if we stopped now, what would we have?

If your monthly agreement has no lock-in, you can make this call every month. If it locks you in for a year, ask why.

Warning signs under either model

• Updates full of “working on” and “in progress”, with nothing you can open.
• Weeks with nothing released, and no clear reason.
• The same bug coming back.
• Only the developer can release, or the accounts are in their name.
• You hear about problems from your customers first.

None of these proves your developer is bad. Each one is worth a direct question.

Common questions

Should I ask my developer for DORA metrics?

You can. But a list of what went live last month, with dates, tells you most of what you need. If a team can’t produce that list, it has a bigger problem than metrics.

We pay a freelancer by the hour. Does this apply?

Yes. Hours tell you what you paid, not what you got. The three questions are the same.

My developer doesn’t pass these. What now?

Talk first. Ask for the release list, the list of accounts and whose name they are in, and what is waiting on your side. Often the gap is in the reporting, not the work. If it still isn’t clear, get a second opinion before you decide to switch.

How we can help

If you are unsure about your current developer or system, a Review gives you a written second opinion on one decision for RM1,000, or investigates the whole system and what to fix first for RM5,000. If you want a team that answers for the result, Managed Delivery starts at RM13,000 a month, with no minimum term.

The TCE Baby Expo case study below is a monthly engagement where more than 72 releases went out while the expos kept running.