Musings of a Long-term Evaluator – Rob Walley

Background to involvement

I was introduced to the concept in 1988 while working as Quality Development Manager in Alcoa of Australia in WA.  I received extensive training within Alcoa and outside both on Total Quality Management and Statistics.  Highlights included Dr Edwards Deming’s 4-day seminar in Sydney, Qualcon 1992, Quality Function Deployment and Kaizen Concepts by Masaaki Imai.


I became evaluator in 1994 and went to evaluator training in Melbourne run by the Australian Quality Council and a Baldrige examiner from Tennessee and other experts.  We did role plays and evaluated a fictitious medical organisation.  The training ran over a long weekend and there was special training for the team leaders.

My first evaluation event was visiting 3 organisations in 2 weeks and one of them had locations in Sydney, Perth, Melbourne and Adelaide and another organisation in Melbourne.  I was fortunate to have a well-respected exponent, Brian Thomas in my team.  In the early years that was quite normal as the process was popular and very well supported particularly by the Labour Government led by Senator Button.  At the time applicants had to submit a document outlining their approaches and how successful they had been.  The rules around submissions have changed through the years until recently no submission is required except for a Prize Application.  From their inception until 2000 the Awards were run by the Australian Quality Council (AQC) and after Standards Australia bought AQC in 2001 and then formed SAI Global, they were run by SAI Global.  In 2013 SAI Global relinquished control of the Awards to a not-for-profit organisation called the Australian Organisational Excellence Foundation now called Business excellence Australia (BEA)

How does it work and who does it?

A group of business excellence experts who have all volunteered to be evaluators is appointed to evaluate an applicant.  Ideally the team will be gender-balanced and have a member who works in the relevant industry.  Geographical location is important as that contributes to minimising the cost to the application.  Every evaluator must go through a rigorous interview before they join the panel. They all do the evaluations on a voluntary basis with only their expenses met by the applicant.  Depending on the size of the organisation the groups are usually 2, 4 or 6.  This helps the information gathering which is done in pairs. Before visiting the site, the team reads the organisation’s information and reaches consensus on a general view of the organisation and an identification of the issues that require clarification at a site visit.  Every application has a site visit.

What are we looking for?

The goal of the Awards process is twofold. It is to recognise exceptional performance by organisations and to provide information and suggestions on how an applicant might improve further.  The whole objective of the process is to get Australian economy to improve.

Because the Award evaluation is an encouragement process not an audit, the conversations are somewhat unstructured but are focussed on how the organisation is approaching each issue and how successful it has been.  The way an organisation approaches any aspect of its operation is determined by its leaders and suits their culture.  The framework is based around the concept that in life any approach is a theory until proven.  In other words, the evaluation and feedback process is encouraging the organisation to think through its approach to an aspect of its operation and then devise a process to give effect to that approach and deploy it.  It must however verify that the approach chosen works and then act on that verification.  This is the heart of the evaluation known as ADRI from the first letters of Approach, Deployment, Results and Improvement.  This way is firmly rooted in the scientific method that has improved humanity’s life so impressively particularly in the last 100 years.

The Awards have always been focused on sustainable performance, giving value to customers and stakeholders, the impact of variation and respecting people.  Some of the considerations in the Framework are leading edge and questions about them are often challenging.

One of the interesting aspects of evaluating is that on entering an organisation the level of energy and involvement is evident within a noticeably short time.  The number of organisations I have visited where I have been almost overwhelmed by the extent to which the organisation has been successful is small.  It is most rewarding to participate in those.  By the same token, organisations that are not succeeding are obvious too.  The ambience in the organisation is palpable and that forms part of the evaluation.  This may appear to be a non-scientific process but organisations are social groups of people who come together for a common purpose and the extent to which this is achieved is obvious within the environment of work.  The visit usually commences with a meeting of the senior leadership team making a short presentation and then answering questions of the evaluators.  The body language in this event is extremely important and the observation of this forms part of the data-gathering that is so important to the outcome.

Because the evaluation process revolves around asking questions about people’s lives the questioning style is particularly important.  It must never be judgemental or clouded by the evaluator’s own personal values.  The idea is to uncover things the applicant is doing well and especially when the application of the principles is well executed.  At the same time, the team will suggest areas for improvement that could be considered.  These opportunities are never couched in directive terms.  It is especially important that the organisation consider all the opportunities to determine if they are valid in its own circumstances.  The feedback report the team compiles by consensus is the most valuable part of the process.

It is important that evidence is found to support statements made by the team.  The team will not state a strength or opportunity unless there is evidence to support the statement.  The process has always been focused on collecting evidence of improvement using trend data, preferably over at least three years.

It is incredibly enjoyable visiting an organisation and conversing with employees about the things they are passionate about.  Providing ideas for further improvement is challenging and has to be done in clear and unambiguous terms.  Over the years, this process has stood the test of time and will contribute to the Australian economy well into the future.