Use AI in software testing without losing your judgement

Choose tasks with reviewable outputs

Useful starting tasks include turning a sanitised requirement into candidate test ideas, grouping duplicate cases, improving the clarity of a bug report and explaining a short code example. Keep the output narrow enough to compare with a known source of truth.

Avoid asking AI to certify that an application is secure or ready for release. Those conclusions require evidence from an agreed assessment. Similarly, a generated test is not an executed test: it may contain an incorrect selector, unsupported API or assertion that verifies the wrong behaviour.

Give requirements and boundaries

Tell the model the feature's rules, user goal and known constraints. Explicitly distinguish facts from unresolved questions. Ask it to mark assumptions and explain the risk behind each proposed case. This makes omissions and unsupported claims easier to spot.

Use invented data or information approved for the tool. Removing a name may not make a log safe if it still contains tokens, identifiers or private business information. Follow your organisation's tool and data policies before submitting code, screenshots or documents.

Review the answer before using it

Check every proposed expectation against the actual requirement. Look for duplicates, missing boundaries and assumptions disguised as facts. Ask whether the test would detect a meaningful failure and whether its data can be set up safely.

For generated code, inspect dependencies, selectors, assertions and any network or file operations. Run it only in an authorised test environment, then deliberately change the expected outcome to confirm that the check can fail. A permanently green test can be as misleading as a noisy failing one.

• Is each expected result supported by a requirement?
• Does each case investigate a distinct risk?
• Can I explain the code and its external effects?
• Does execution produce inspectable evidence?
• What still needs human investigation?

Treat external content as data

A document or message supplied to an AI system can contain instructions attempting to change its behaviour. OWASP identifies prompt injection and sensitive information disclosure among risks associated with large language model applications. This matters when a tool reads external tickets, pages or attachments.

Keep permissions limited and require review for actions with external effects. A test assistant that can draft a report does not automatically need permission to publish it, delete data or run production commands. Separate generation from execution, and record who approved consequential actions.

Further reading: OWASP: Top 10 for Large Language Model Applications

Measure the useful result

Compare a small task with and without AI. Include the time spent reviewing, correcting and executing the result. If the draft is fast but introduces mistakes you struggle to detect, the workflow needs adjustment. Speed alone is not a quality improvement.

Keep a record of which prompts and checks were useful for your work. As your understanding grows, use AI for more demanding tasks while keeping the evidence standard unchanged. The goal is to produce better testing decisions and clearer communication, with you able to explain both.