Setting Priorities in a National Science Programme

Chrys Horn, Tamsin Braisher and Ilse Breitwieser, May 2008

Printable Version of this Page

This page provides a summary of a more detailed description of our process

Introduction and background

Researchers funded by public money the world over are increasingly asked to work on projects relevant to non-science stakeholders.  Likewise, increased competition for the science dollar requires stakeholders to decide priorities, perhaps as part of a contract, a review, or developing a proposal.  It may also be part of the work completed by funders when they allocate funding.  In situations where a range of interests will be involved in implementing a decision or in signing it off, the process must be transparent and involve the different stakeholders in learning about each other’s perspectives and interests in the decisions to be made.  The question is, “How do you do this in a way that promotes this ‘social learning’ process and that results in outcomes that are at least acceptable to all of those with a stake?”

This article outlines our learning from setting priorities for a biosystematics outcome-based investment (hereafter called the OBI) based at Landcare Research in New Zealand.  The formation of the OBI in 2005 drew together three separate and geographically-dispersed biosystematics teams based around physical collections of biota of national significance in New Zealand, and a number of partner agencies with a need for systematics work.  As part of its contract, the OBI had to prioritise all research work and collections activity based on what would have the greatest effect on environmental outcomes in New Zealand, as well as being of international scientific merit.  There was also a requirement from the funding agency to consider the maintenance of systematics capability.

The priority-setting process had to take account of emotion and history as well as analytical and scientific elements. Tension existed between the different science teams and between the science teams and the end user agencies and between some of the end user agencies. Science staff were anxious about their jobs, and the quality of their science. End user groups were keen to ensure that the science needed for their work was a priority. There was also a need to balance the maintenance of the collections with research priorities, given very tight funding. Thus, our overall process had two major stages: 

Designing the priority setting process

The design of the priority setting process involved all OBI science staff and ??? end user agencies. A series of workshops aimed to: 

Design principles

While a few participants initially favoured a scoring approach for making this decision, most participants thought that discussion and consensus were the ideal way to reach agreement.  They wanted discussion so they could explain their perspectives to others, a fact which points to the importance of learning and deliberation in decision processes and the need to ensure everyone was able to represent their perspectives and learn about the perspectives of others. The social scientist working alongside the process advocated strongly against a scoring approach because this approach does not allow participants access to the views of others.  Thus, while scoring may seem fair and ‘objective’, it is not entirely transparent and makes no allowance for people to change their perspective as they learn more about the intricacies of the system within which the decision is being made.  Previous experience indicates that trying to reach a judgement without dialogue can work accentuate the differences between perspectives without increasing mutual understanding  Thus while we documented criteria for setting the priorities, we specifically refrained from weighting them or using any kind of calculation process. Instead we focused on learning around those criteria as they emerged in discussion.

Figure 1: The Agreed Priority Setting Process

The priority setting process

We aimed to make the decision process as iterative as we could, subject to our contractual and time constraints. All groups (the three science teams within the OBI, and each stakeholder agency) provided the information outlined in Figure 1 and their suggestions for priorities as a first step in the process. This information was collated, and a group of representatives chosen by the various science groups met again with agency stakeholders to discuss those preliminary priorities.  Subsequently, the priorities and the reasons for them (based on the criteria) were documented and sent to participants for further comment.

Using the process

While the overall process was designed by the group, much work was required to operationalise the design.  Significant work was required before the priority-setting workshop.  Agency representatives were asked, via email, for their priorities across a range of work types and taxa.  Science group leaders were also asked about their group’s priorities for research.  Responses, which arrived in a variety of different formats, were collated, summarised and sent out to participants prior to the meeting. 

Only one day was available for discussing the priorities, so it was important to focus group discussions where it mattered.  Thus we needed to find a way to highlight areas of disagreement.  Initially, science team representatives prioritised within science areas a series of post-it notes with the different projects on them.  Once a ‘first cut’ had been achieved this way, the wider group could add in any projects that they thought were missing.  Workshop participants then used sticky dots to indicate where they disagreed with the placement of any particular post-it note.  Thus, a post-it note with many dots placed on it had a high level of disagreement with its placing.  Full group discussion then focused on those post-it notes with dots placed on them.  Post-it notes positioned such that there was general agreement about their placement were not discussed in any detail.  This way, discussion was kept only to those areas that really needed it. 

Reflections on the process

End user representatives noted that all of the potential priorities were important and most were very important to someone, somewhere.  They suggested that the aim of the priority setting process was “minimum dissatisfaction,” given current funding levels. 

No explicit reference was made to the list of criteria in our discussions on the day, but the discussion revolved around the criteria.  Agency representatives discussed those criteria most associated with the needs of their organisation, the capability that they might need to draw on and the use of the information in achieving the organisation’s stated strategic goals. All group members contributed significantly to discussions of priority taxa, priority research areas, and priority products and services. Many questions were asked during this discussion, indicating that participants needed specific information to fill gaps in their understanding of the issues and of each others’ views.  Overall much information was exchanged. 

Agency representatives asked that priorities be discussed throughout the existence of the OBI and not be confined to a ‘one off’ process. In fact, this particular process relied heavily on previous discussions and established relationships between scientists and agency staff.  Without this background, this process would have been much more difficult in the short time frame available.  Ongoing discussion will be necessary to accommodate changes to priorities in response to changing stakeholder environments.

Maintaining balance between the needs of end users and the needs of systematics scientists was important in this process.  Judgement of the balance was also a matter of perspective, with end user groups being concerned that they were heard in the process and science groups also needing to be heard.  Both groups wanted their needs accommodated in the process. In managing this, it was useful to have observers and facilitators who had no stake in the outcomes of the process. 

Representation was an issue for some agency representatives.  Some felt that they could not easily represent the views of their organisation, partly because those views were likely to change as they started to rethink their own priorities.  These same people found the tight time frames problematic because they could not have their written input into the process signed off.  Furthermore, agency representatives found it difficult to represent the views of all individuals or groups within their organisation on priorities.  This was especially problematic when they were uncertain of their expertise relative to those they were representing, or if they disagreed with them.  This points to the difficulty of aligning processes throughout the social system.  It is unlikely that perfect alignment could be achieved given that every organisation has its own trajectory in strategy development.

Comparing large, long-term projects, with smaller, shorter-term projects was challenging.  It was also impossible to come up with a single list of priorities. Agency representatives did not want one priority at the expense of all others. Some agency representatives could not articulate what their research needs would be in the future, but needed to ensure they could call on a range of capabilities, which would therefore require maintenance through a range of potentially lower priority projects.

 This was not an easy set of processes to run. Principles that appear simple, such as transparency, openness in participation, the idea of setting priorities or being strategic, are not always so obvious when they are put into practice.  People’s different understanding of the process, the language used in the process, the different ways they attributed meanings to actions, and the different organisational environments in which they worked all contributed to challenges in this process.  It was seldom clear what the best course of action was at the beginning of each step of the process. The steering team had to make many decisions about how best to structure documents, what questions should be asked, how the workshop should be structured, how the initial list of priorities should be presented to participants, and how much emphasis to give to different parts of the process. Thus new challenges and questions arose frequently as the priority setting process was developed and executed. 

This process was based on the ideal of good participatory practice.  The process was iterative and we changed the plan as we learned about how it was working in practice.  In operationalising the plan, we learned things that then required incorporation into the process.  Much of the success of this work, therefore, arose from a process of planning, monitoring, reflection and adjustment. It required us to work with uncertainty and to manage the balance between keeping the process open and participatory without overloading people unduly.

Throughout this process the steering group had to keep reflecting on the job in hand.  For example, it was easy to get distracted from the priority setting and to start thinking about implementation.  In grappling with this tendency, we realised that we did need to think about implementation issues at times.  For example, one of the criteria was “how feasible is the work and how long will it take to produce meaningful results?”  Also the size of the project affected how sensible it was to prioritise it, particularly for those priorities lower down the list. Therefore, while it was sometimes productive to think at or even beyond the “overlaps,” we also found it constructive to ask ourselves frequently if we really had the right focus for the task in hand. 

Similarly, it was important to separate out the work we were doing setting priorities for the OBI from work that should be completed at national level by high level government agencies. A national biosystematics strategy, for example, would have made this process easier to complete, as it could have taken into account the biosystematics collections and research that sit outside the OBI.  Aspects of our process may be useful for developing such a strategy.

Having a good, independent meeting facilitator familiar with science and the science system, and indeed known to many of the participants, was key to the success of this process.  Facilitators look after the process of a workshop and ensure all participants are able to participate fully.  Without a good facilitator, this process may not have succeeded, particularly in light of the lack of time available for the meetings.  Our facilitator was able to elicit input from people, ensure that all participants could contribute, question and have their perspectives heard and, because he was independent, he was able to make demands of people that would have been questioned were he not independent.

Likewise, another unusual participant in this work was the social scientist who assisted with and documented the process.  She provided independent observation, worked as part of the steering group to bring in knowledge of social processes, challenge assumptions, assist reflection, and assist with problem solving.  The documentation in particular should allow others some benefit from the work that went into developing this process.

Conclusions

Overall this process achieved a set of priorities that stakeholders appeared to find acceptable.  Longer time frames would have allowed participants representing large organisations to consult better with others in the organisation and to allow for another round of discussion or evaluation of the priorities as they were finalised. Likewise, we really could have used more discussion time during the priority setting process itself because some important discussion did not occur.  It is not clear if this would have significantly affected the final priorities of the OBI, or just participant satisfaction with the process.

The overall learning to be taken from this work is the importance of deliberation in making decision processes transparent.  It seems that people can cope with decisions that are less than ideal for them personally if they understand why those decisions have been made and they can see how their views have been included in the process.  This process has highlighted the conversations that occurred through a mixture of face to face dialogue, telephone conversations and written documents and the way in which these conversations have influenced the priorities that have been set. 

The process has involved making the values used in it explicit.  Despite this, we did not attempt to use these values in any “calculative” way and instead relied on the principle that people tend to make complex decisions better if they can move between a holistic, “intuitive” way of working and a more analytical type of thinking in which individual criteria might be assessed.  Our priority setting process aimed to tap into both of these thought types and appeared to work relatively well under difficult conditions.  Further experimentation around this would be beneficial.

Questions remain about the use of criteria in an a priori or a posteriori way. It is possible that we could have simply articulated criteria rather than trying to refine them to a point where we could rank projects has having high medium or low value relative to those criteria.  As one of our end users noted, we simply did not need to go through the criteria systematically; they came up naturally in the discussion that we had.  When the decisions became more difficult, it appeared that more criteria were raised and the discussion was broadened rather than that people wanted to shift to using a more systematic approach of ranking and thinking more in-depth about particular criteria.  This pattern is worth noting as an important observation to take into further studies of decision processes. Despite this articulating the criteria was an important step that had to be done, because when there was debate it was easier to be more systematic in thinking through the issues and it allowed different groups some validity with regards to the criteria important to them. 

A more detailed account of this process is available at: http://www.landcareresearch.co.nz/research/sustainablesoc/social/documents/SystematicsPrioritySettingDetailedProcessPaperOct0808.pdf

This research was funded by the Foundation for Research Science and Techology in the Defining New Zealand’s Land Biota OBI.


Back Contents NextNext
Page last updated: Thursday 29th March 2007

Research areas

Communication and dialogue Evaluation and reflection Indigenous knowledge and values Knowledge management for integration Social aspects of ecosystem management Supporting collaborative approaches

Programmes & OBIs

Building capacity for sustainable development Integrated Catchment Management Low–impact urban design & development Te Tapoitanga Māori: Growing Regional Māori Tourism

Tools & Services

Collaborating with iwi Stakeholder analysis Team–building evaluation guide