Best Practices Self-Assessment

Plan

Implement

Evaluate

Results

The Best Practice Self-Assessment is a tool to assist you in quickly identifying organizational practices that could benefit the most from application of the best practices found in our Best Practice Handbook. The Handbook was developed to address common pitfalls we see when we audit government organizations. By focusing on improving these practices, you can be proactive in mitigating common pitfalls and risks, while advancing greater effectiveness and efficiency.

Once you've completed this brief, 5-minute assessment, you will receive a custom report identifying the sections of the Handbook that will be most useful to you. The practices not listed are those where your organization's practices, most likely, already closely align with the best practices detailed in the Handbook. The practices identified in your report present a good starting point for strengthening your organization.

If you are interested in receiving training or support on any of the best practices found inside the Handbook, you may reach out to our office by selecting the Request Training tool inside our online toolbox. We look forward to collaborating on our collective effort to further drive excellence in Utah government.

1. Does the organization have a Yes No

1a. Has the strategic plan been in the last four years? Yes No

1b. Does the strategic plan use and ? Yes No

2. Does the organization consistently meet for critical processes and activities? Yes No

3. Has the organization clearly defined its and for objectives? Yes No

4. Has the organization clearly articulated that give employees permission and a supportive framework to challenge the status quo, try new ideas, and scale successes? Yes No

5. Does the organization have a function? Yes No

6. Are policies and procedures to the strategic plan? Yes No

7. Have policies and procedures been thoroughly reviewed in the last four years to remove those that are Yes No

8. Do resource allocation decisions rely on and Yes No

9. Is the uniform and are financial data reported on accurately and completely? Yes No

10. Does the organization roles and decision rights? Yes No

11. Have ALL managers received Yes No

12. Are data for accuracy, reliability, and completeness on an annual or semi-annual basis? Yes No

13. Does data have in place? Yes No

14. Does the organization have an Yes No

15. Does the organization have an for different stakeholders? Yes No

16. Does the organization have a Yes No

16a. Is each performance measure in the performance management system and ? Yes No

Results

Based on your responses, we recommend you review the following subsections of the Best Practice Handbook. We encourage you to select 1 to 3 best practices you can focus on applying to your organization. Once successfully implemented, you can select additional best practices to focus on. Check out our suite of online tools for additional assistance, or request training on any of the best practices found in the handbook. For sources and additional details, see the full Best Practice Handbook.

Section 2.1: Develop and Refine a Strategic Plan

Pitfall: Missing or Inadequate Strategic Plan; Outcome: Strategic Plan Directs the Organization

Effective strategic plans shape the vision of the organization and direct actions to provide reasonable assurance that objectives and goals are being met. Developing a strategic plan can clarify organizational priorities and define what the agency seeks to accomplish. Statewide goals require inter-organizational alignment through broad, cross-cutting plans. Utah's Coordinated Action Plan for Water is a good example of several departments coming together to undertake an enterprise-scale endeavor. Though no two plans are alike, each should include a basic set of elements and should use tools to improve the plan's reliability. Executive management should aim to advance efficiency and effectiveness reliably across the organization. Directors should approach their organizations as complex, dynamic systems, with interrelated parts. Omission of key elements, use of vague descriptions, and improper application of tools contribute to ineffective strategic plans.

Ensure Strategic Plans Contain Key Elements That Guide Operations and Provide Transparency to Stakeholders

This handbook can be used as a resource for organizations to define, develop, and implement a strategic plan. Effective strategic plans should involve stakeholders, assess the internal and external environment, and foster alignment. Stakeholders include customers, employees, external agencies, local partners, neighboring businesses, government regulators, and any other parties significantly affected by the strategic plan. The Governor's Office of Planning and Budget (GOPB) issued a Guide to Strategic Planning that specifies eight elements that should be incorporated into strategic plans, as seen in the following figure.

A circle graphic with images that represent each of the 8 elements of a strategic plan: Core Values, Goals, Objectives, Strategies, Performance Measures, Annual Work Plan, Vision Statement, Mission Statement

The US Government Accountability Office (GAO) has also developed a resource to guide agencies in developing essential elements for a strategic plan. GOPB states that strategic plans should be significantly refreshed and reviewed at least every four years.

GAO further recommends,

If there is a significant change in an entity's process, management reviews this process in a timely manner after the change to determine that the control activities are designed and implemented appropriately.

We encourage agencies to revisit their strategic plans often and refer to these resources when making changes. The following information supplements these materials and provides a set of tools that can help leaders achieve clarity, accuracy, and relevance in their strategic plans.

Delivery Models and Performance Metrics Ensure Strategic Plans Are Clear, Concise, and Measurable

Strategic plans are useful only if they are reviewed often and implemented in daily activities. Efforts should be made to simplify and visualize strategic plans-both as a whole and by their individual components. Plans should have sufficient detail to guide operations, but not so much detail that the plan becomes unwieldy. Simplification of plans through visualization tools makes review and refinement easier, and more transparent for stakeholders.

A service delivery model maps out the classic input-process-output-outcome components of a government program. It supports alignment of priorities and sets the stage for evaluative activities by giving management greater insight into operations. It is also used to make management decisions transparent to stakeholders, such as why resources are allocated a certain way. Service delivery models also can empower managers and directors to advocate for more resources or a new allocation strategy. Once the service delivery model is developed, appropriate standards and controls can be established to ensure the necessary components are executed at the right time and at the level of excellence needed to realize goals and objectives. Service delivery models can take many shapes. An example of a service delivery model that simplifies large or complex functions is shown in Figure 2.1.

A diagram modeling the steps to take in a service delivery model — Figure 2.1: Service Delivery Models Are Used to Visually Depict the Relationships between Inputs, Outputs, and Outcomes. Once developed, controls can be put in place to ensure critical steps are taken. These models can be used to add clarity in a concise manner.

Creating service delivery models makes assumptions explicit regarding the relationship between inputs, processes, outputs, and outcomes. This focuses strategic discussions by allowing managers to pinpoint obstacles in the model. While organizations can only influence (but not control) outcomes, outcome metrics should be used to inform refinement of operational activities (see Subsection 4.4 for more on performance measures). Output metrics are important for determining whether strategies produce desired outcomes. If intended outcomes are not being realized despite a well-implemented strategy or program, it should signal to management that the strategy itself needs to be adapted. The GOPB Guide to Strategic Planning differentiates between ends and means. For the purposes of this handbook, ends are synonymous with outcomes, while means are related to the program structure, or the strategies used to improve outcomes. GOPB and the Office of the Legislative Fiscal Analyst (LFA) have provided examples of output and outcome measures that can be used within a service delivery framework, which we have included in Appendix B of this handbook. Establishing these measures is further discussed in Subsections 2.2 and 4.1 of this handbook.

Reliable Cost Estimation Enables Effective Cost Allocation

A significant part of planning is for an organization to establish a long-term, future-oriented vision. When strategic plans lack reliable models of prediction, they risk not having enough resources to carry out the plan. This can undermine objectives across the organization as managers make difficult decisions on what to cut. Because resources will always be scarce to some extent (see Subsection 3.2), the success of planning largely hinges on optimizing strategies for resource allocation. Having a reliable cost estimation tool or process is essential for guiding management decisions during the planning phase.

A reliable cost estimate is the basis for a reasonable budget. Once developed, the cost estimate can be used to make trade-off decisions among cost, schedule, and program requirements. A cost estimate requires well-defined assumptions, sufficient comparable data, and well-trained analysts. Example of Need for a Reliable Cost Estimate: In our 2021 Performance Audit of the Division of Services for People with Disabilities (DSPD), we found that the division had underestimated mandated additional costs of the program. This resulted in budget shortfalls of over $3 million. To control for this kind of error in the future, we recommended adjusting ongoing attrition savings or establishing contingency appropriations internally and with the Legislature. A reliable cost estimate should account for future estimates of total cost of labor, material, and other resources needed over the lifespan of the project. Risk analyses also may be used to anticipate possible contingency funding. The service delivery model described above can be extended by assigning monetary values to resources (inputs) and activities. GAO has identified 12 steps to help organizations develop a reliable cost estimate.

Development of a Reliable Forecast Is Pivotal for Successful Planning, Budgeting, and Management. Forecasting extends beyond cost estimates and can apply to any prediction of future quantity. Forecasting allows decisions to be based on data, void of personal bias and preference. For example, when contemplating new policy, forecasts of the consequences or result of the policy with the current outlook should be considered. Development of a forecasting model should align with the organization's risk tolerance. Organizations should recognize that higher-precision models tend to be less reliable. Choosing smaller confidence intervals can sometimes lead to optimism that later produces unexpected results. This can also occur with misspecification, which refers to omitting a variable that is, in fact, statistically significant. Example of Monitoring Projections: In the 2020 Performance Audit of the Justice Reinvestment Initiative (JRI), auditors found that the initial projection of the future prison population was inaccurate. The original projection used a linear model that followed population growth. Underlying assumptions that ultimately proved untrue had led to incorrect estimates. However, as can be seen in Figure 2.2, the Commission on Criminal and Juvenile Justice continued to monitor the prison population and adapt policy based on the actual population, which was eventually lower than the projection.

Figure 2.2: The Justice Reinvestment Initiative's Original Projection of the Prison Population, Compared with the Actual Prison Population, Illustrates How Monitoring Is Used to Make Adjustments. Forecasts like these are essential for making decisions and should be adjusted regularly when new data become available.

The JRI example highlights how projections need to be monitored and adjusted using new information as it surfaces. Because reducing the prison population was a goal of JRI, this information was also used to guide policy decisions intended to promote the desired outcome. In the JRI example, inaccessibility of treatment among prisoners, along with lack of treatment oversight, were driving factors for higher recidivism than had been expected. Though no model is perfectly accurate, making the assumptions of the forecast model explicit and adjusting the model with new information enhances its reliability. In addition, it is important to know the assumptions of the forecast model in the beginning, as this will increase understanding of what could cause the forecast to change. The Forecast Conversation Guide is an online tool we developed to help leaders and other stakeholders better understand assumptions and how to appropriately interpret a forecast.

Like other parts of a strategic plan, forecasts should be living, and analysts should revisit their forecasts to check their accuracy. Adjusting models can create significant impacts on an organization and on the lives of its employees. Example of the Need to Revisit Prior Forecasts: In 2013, for example, adjusting a forecast model for the Kennecott copper mine after small earthquakes may have saved the lives of local miners. Models predicting rock slope failures typically include the structural geology, groundwater, changing climate, rock mass, rock strength, maximum stress, and applicable seismic events. Small earthquakes changed the integrity of the mine, causing a massive rockslide, as seen in Figure 2.3. Because geotechnical surveyors were aware of the change and had adjusted their models, mine workers were evacuated before the rockslide, and no fatalities occurred.

The Kennecott Copper Mine with an area cirlced in red that includes the source location of a 2013 rockslide, the slide pathway, and the depositional area — Figure 2.3: In 2013, Utah Geotechnical Surveyors Were Able to Predict a Rockslide at the Kennecott Copper Mine. As a result, zero fatalities occurred despite the event.

At a minimum, management should understand the following about forecasting for strategic planning purposes:

What is being forecast
Cost of forecasting and errors
Forecast time horizon
Process of choosing independent variables
Choosing the model
How to decipher forecast results
Importance of recursion
Likelihood that models may evolve
Existing data limitations

To act appropriately on results of the model, management should avoid two common biases: anchoring bias and recency bias.

Anchoring bias relies solely on historical trends, without consideration of recent or forthcoming changes.
Recency bias occurs when decision-makers assume that the most recent trend will continue, ignoring the fact that many economic variables are cyclical.

Five circles with one at the center labeled as organization and the other four cirlces branching out from the organization circle each respectively labeled Labor Force Participation, GDP, Real Income, Fed Funds Rate

Additionally, planning is not a stagnant process; rather, it evolves as circumstances evolve. To ensure plans remain relevant, management should understand core macroeconomic principles and trends, as well as how these factors may affect the operating environments of their organizations. Things like unemployment rates, interest rates, labor force participation, gross domestic product (GDP), and real income can influence staffing decisions, payroll rates, supply costs, and other critical areas of operations. Other organization-specific metrics, like seasonal workers and weather trends for the Department of Agriculture, must be considered.

Along with using proper techniques in modeling, organizations should budget for contingencies or unexpected shocks that forecasting cannot predict. Forecasting is an evolving and flexible process that should respond to changes in data, society, and policy. Analysts, when using scenario-based forecasting techniques, typically present several models. When budgeting, worst-case scenarios should always be given serious consideration. When leadership uses, updates, and understands forecasts properly, data-driven decisions can guide management and promote success in accomplishing objectives.

Baselines and Benchmarks Help
Organizations Make Consistent Progress

Measures gain greater value when contextualized through comparisons. Baselines and benchmarks provide valuable context that managers need to set goals and targets. Without this information, managers must approximate the conditions of the organization and make guesses about what they will be able to achieve. A baseline is used to understand the organization's current status in a particular function or activity. Before implementing a new program or initiative, it is critical to establish a baseline using the same measures that will be used to assess the program's success. Baselines are typically established after planning, but before implementation. Data should be collected utilizing the developed measures for three or more periods prior to implementation. These data are then used to assess progress, including the rate of change in performance over time. Negative Example of Not Calculating the Baseline: In the 2013 Performance Audit of the Department of Workforce Services Work Environment by the Office of the Legislative Auditor General (OLAG), we described complications in the pay-for-performance program because the baseline had not been calculated. More specifically, a lack of baseline data resulted in budgetary shortfalls, forcing management to reduce performance incentives. The audit report further noted,

These incentive reductions are detrimental to the morale of high-performing employees, who receive a reduction in total compensation as management addresses the budget constraints for the program.

Because a baseline was not properly established, management underestimated the cost to incentivize workers, and the costs ultimately exceeded the savings.

A benchmark is a point of reference for establishing targets and assessing performance. Examples include a national average, a standard, or the same measure from similar organizations or programs. Benchmarks provide important context and help organizations know what realistically can be achieved. Benchmarks are sometimes used to plot a course from where the organization is to where it wants to be. In the 2021 Performance Audit of Social Service Agencies' Performance Measures, OLAG discussed the need for "stretch" goals. These targets should cause the organization to stretch to reach them but not be so difficult that they are unattainable.

Baselines, benchmarks, and targets should be numeric where possible-for example a count, a timeframe, a dollar amount, or a percentage. Numeric measures are useful to assess progress and quickly gauge what is working and what is not. Example of the Proper Use of Baselines, Benchmarks, and Targets: For example, Figure 2.4 shows Utah's smoking rates compared with a national benchmark. While Utah's smoking rates are lower than the national average, trends in the smoking rate over time show a steeper decrease at the national level than what is seen locally.

Figure 2.4: Baselines and Benchmarks Are Useful Tools for Gauging Progress toward Goals. The trend lines below show how Utah's smoking rates compare against national benchmarks, as well as Utah's target rates over time.

When paired together, baselines, benchmarks, and targets guide management to set realistic goals, resulting in meaningful progress. Figure 2.4 indicates that once Utah's target was achieved for two consecutive years, it was altered to stretch toward a new target to lower the smoking rate. As targets are achieved, and new stretch goals are set, consistent progress can be achieved. Baselines, benchmarks, and targets are valuable for assessing progress toward long-term goals and determining why objectives are (or are not) achieved.

Qualitative Information Should Accompany and
Complement Quantitative Data Whenever Possible

A focus on quantity without acknowledging or measuring differences in quality ultimately undermines measurement systems. Quality refers to the characteristics or attributes that make a product or service valuable. Without an intentional consideration of quality, quantitative measures are prone to distortion and manipulation (see Subsection 4.4). Although quantitative measures should be used extensively in organizational planning and reporting, qualitative information should complement and enrich those measures. Example Illustrating the Importance of Measuring Quality: To highlight the importance of measuring quality, the following example from our 2016 report, A Performance Audit of the Utah Department of Transportation, describes what was found in the department's design, bid, build (DBB) practices. At the time, the DBB practices at the Utah Department of Transportation (UDOT) incentivized "budget and speed, but not quality." Because quality was omitted from the performance management system, poor-performing contractors continued to bid on and win contracts. UDOT employees reported added maintenance and administrative costs due to these poor-quality contractors.

Reasons for not measuring quality are varied, including:

Difficulty assigning a number to quality
Burden of collection efforts
Concern that the measure does not truly capture quality
Inexperience in creating qualitative measures

Whatever the reason, when quality is disregarded or sidelined, quantity is disproportionately relied on for making decisions. This can lead to distortions in workflows and undermine objectives. Example of the Essential Need for Qualitative Information: For example, in the Performance Audit of the Workforce Services Work Environment, OLAG auditors found that the Department of Workforce Services heavily incentivized case determinations (a quantitative measure) while ignoring important qualitative information. Because cases were limited, employees ended up competing with one another to "cherry-pick" others' cases that were partially finished, while neglecting day-to-day tasks. This practice allowed employees to take full credit for completing cases while doing only part of the work. Other examples included employees encouraging customers to apply for programs for which the employee knew the customers were not qualified, thereby artificially inflating the employee's case determination numbers. Auditors reported that there were no controls in place to prevent employees adding programs to case files without talking to the customer at all. These examples indicate that qualitative measures were being omitted from the performance measurement framework, giving way to distortions and gaming (see Subsection 4.4). Quality and quantity should go hand in hand to ensure that objectives are realized.

Proxy Measures Can Be Considered to Make Measurements Simpler or More Visible. Proxy measures, or measures that are closely related to the subject of interest, may be useful to roughly capture information in a simpler or easier way. Proxy measures may also provide a quantitative measure for something that is difficult to observe. Example of Proper Use of Proxy Measures: For example, the quality of a product or service can be measured, by proxy, from the change in the amount of customer complaints. In another example, UDOT uses "delay time" and "snow removal" measures as proxies for "mobility." Because mobility is difficult to measure directly, proxies are used to gain insight into how well UDOT is improving that outcome. Figure 2.5 shows these proxy measures.

Graphic showing a circle graph with 65% success on mobility performance measures. — Figure 2.5: UDOT's Mobility Performance Measures Are a Good Example of Proxy Measures. Tracking proxy measures like "delay," "reliability," "mode split," and "snow removal" allows UDOT to gauge progress toward its goal to "optimize mobility." Proxy measures provide a quantitative measure of something that is difficult to observe.

While proxy measures can be a creative way to assess certain important goals, they should be used with caution, as they can deviate from what is intended to be measured. Example of Improper Use of a Proxy Measure: For example, as part of the 2022 Performance Audit of Tax Increment Financing, OLAG auditors found a mistake in one of the proxy measures. Orem City University Place Community Development did not require developers to submit an annual jobs report as a condition of receiving the tax increment financing (TIF) incentive. Rather, developers only had to meet square footage requirements for office space. According to the report,

Having the TIF increment tied to square footage rather than to physical job creation generates a disconnect between project area objectives and the reason the increment is paid out.

To address this, we recommended that for developers to receive the incentive, they must tie area plans to the purpose or goal of the city.

Section 2.2: Establish Standards

Pitfall: Missing or Inadequate Standards; Outcome:Critical activities are standardized

A key element in the planning process is the establishment of standards. Standardization is the process of specifying how to complete a particular task or sequence of tasks. Standards ensure that processes and products meet expected values of quality or quantity. With properly established standards, an organization can operate smoothly, establish trust in its field, be held accountable, and achieve its objectives.

Standards serve as the foundation upon which an organization operates. Whereas targets are a goal that the organization or program is aiming to achieve, standards should be the minimum acceptable threshold of quality or quantity. Thus, management should choose standards that can be consistently achieved.

The purpose of standards is to ensure that critical activities occur according to the proper sequence and method. The service delivery model described in Figure 2.1 can help identify standards for the processes being mapped. Having knowledge of what works, and a way to measure or quantify it, is also necessary to develop effective standards. The Results First Clearinghouse Database is a tool developed by Pew Charitable Trusts to guide government entities toward approaches that have demonstrated success and high impact.

Where research is not reliable and the knowledge base is lacking, design and testing can be a useful approach to develop a better understanding of what works. For example, Tennessee's Office of Evidence and Impact, discussed in Subection 4.2, uses an evidence framework to build its own repository of proven programs for its state.

Autonomy and Courage to Make Decisions
Are Essential for Successful Operations

Determining when and where to allow for autonomy is one of the most important policy questions facing directors and managers. Without innovation, the effectiveness of an organization deteriorates over time. To promote innovation, organizations should push decisions down as low as they can while considering the risks associated with greater autonomy. The director of UDOT told us that their management team believes that "The worst decision is the one you do not make." Too many layers of approvals can bog down the decision-making process. It further inhibits innovations as employees' hands become more tied. For this reason, the director at the Department of Government Operations (GovOps) told us they are developing manager training that encourages leaders to empower their staff with greater decision-making authority. They also plan to provide tools to assist managers as they exercise that authority. Ultimately, the goal is to make the approval process less burdensome and to make government more agile and responsive. The director told us that requiring eight approvals to get something done is too much. Another department director said they struggle with managers "punting decisions" to avoid getting blamed for things. This risk-averse approach can undermine objectives and bog down systems. As discussed in Subsection 2.3, calculated risks should be taken at every level of the organization. Leaders should have tolerance of some mistakes when employees are seeking to improve processes in a measured way within their respective duties.

Example of Professional Discretion as Beneficial: In the 2020 Performance Audit of Information Sharing in the Criminal Justice System, 2020, OLAG described the need for professional discretion:

It is important to note that the concept of professional discretion does not run counter to the functions of the criminal justice system.

We further recommended that professionals be provided with accurate and reliable information to guide their judgment and assure accountability for the choices they make.

Standards should be formed with enough specificity to define roles and behaviors, measure performance, and assure accountability. Example of Standards Not Being Rigid Enough: For example, as part of An In-Depth Follow-Up of 911 Audits and Review of 911 Staffing (2020), auditors found that "46 percent of Utah's 911 dispatchers were not in compliance" with medical dispatcher licensing standards. They further found there was inadequate staffing for emergency call centers in some locations. Auditors suggested that management "reconsider and refine minimum standards to ensure the quality of 911 service does not suffer because of inadequate staffing."

On the other hand, when standards are too strict, and discretion too limited, frontline staff may be hindered from doing what is best in a particular situation. Example of Standards Not Being Adequately Utilized: For example, the Board of Pardons and Parole uses the Sentencing Guidelines and Structured Decision-Making (SDM) tools to make release decisions. SDM tools were implemented to unite the board with a common philosophy, goals, and purposes. Despite having SDM tools, which were developed to increase fairness and consistency in release decisions, the board goes with the Sentencing Guidelines 73 percent of the time. By relying heavily on one tool and not the other, the board's ability to clarify and advance its paroling philosophy is diminished.

Section 2.3: Foster Positive Organizational Culture

Pitfall: Toxic work environment; Outcome: Supportive and Innovative Culture

Each organization has a unique culture. The culture of an organization is not only reflected in the strategic plan, but is also interwoven in attitudes, behaviors, decisions of management, and the core values of an organization. As an organization's culture is paramount to its planning and success, fostering a positive organizational culture should be considered in every decision. Innovation is the means to keep strategic plans relevant in a changing world. Without innovation, plans become outdated and less useful. In our survey of Utah's top government leaders, the executive branch ranked lack of innovation as the number one problem related to planning, and the legislative branch ranked it number two. One reason innovation is so vital is because it enables organizations to adapt to shifting challenges. For example, data privacy needs have evolved and become more sophisticated with the shift to digital records. Failure to innovate in this area exposes organizations to external threats. When organizations fail to innovate, they assume risk in the form of declining value, stagnation, and not meeting the needs of the population they serve.

However, innovation brings change, which also carries risk. GAO advises agencies to balance risk appetite against risk tolerance. Whenever possible, it is best to mitigate the risk of change by lowering the stakes. This can be done in several ways. One is small-scale testing of new ideas. Pilot programs, prototypes, and proofs of concept are examples of this. Near real-time monitoring enables organizations to fail quickly, as timely course correction will avoid sinking resources into unsuccessful ideas. Examples Highlighting the Effects of Innovation: Innovation can also make processes more efficient and effective. In the 2017 Performance Audit of Utah's Monetary Bail System, OLAG auditors observed that forfeitures were rarely filed, resulting in a low number of recovered funds. The audit suggested that "the Courts adopt a court date reminder notification system" to address the issue. Similarly, in 2018, the Utah courts were challenged by an increased demand to process divorce cases more quickly. To address the backlog, the courts piloted a triage program to improve processing times. They tested the new process in two judicial districts. After nearly a year of testing, they found that the time needed to process divorce cases had decreased by thirty-eight and forty-six days, on average, in those two districts. Based on the success of the pilot programs, the courts rolled out the practice statewide.

Innovation has been a key focus in Utah policy. To foster more innovation statewide, recent leadership created the new position of chief innovation officer (CINO). Since that time, the CINO has developed best practices for innovating in the public sector and has compiled them into ten guidelines, as seen in Figure 2.6.

Chart describing the ten guidelines: 1. Put customers first with a mindset of serving them. 2.Understand and align efforts with missions, values, and roles. 3.Clearly articulate guardrails that give employees permission and a supportive framework to challenge the status quo, try new ideas, and scalle successes. 4. Develop a system of communication to allow the best innovations to be adopted and employees recognized. 5. Ensure employees have the opportunities and experiences that help them become agile, responsive, resourceful, creative, and innovative. 6. Expect change, and nimbly adapt to it. 7. Start from a mindset of 'yes,' and work towards 'how'. 8. Use data to identify areas needing improvement. 9. Expect mistakes. Accept the risk of prudent failure. Learn and adjust quickly. 10. Work together as One Utah across agenciesand divisions to create a 'no wrong door' mindset and eliminate the 'it's not my job' mentality. — Figure 2.6: Utah's Chief Innovation Officer Developed Ten Guidelines for Innovating in the Public Sector. These can help government entities innovate while mitigating risks.

Small-scale innovation can also benefit a program or organization. One example is modernizing the way information gets communicated. When information can be easily understood, and its value can be leveraged through user interactions, a small innovation can have an outsized impact. Example of Innovating by Changing the Way Information Is Communicated: For example, this graphic illustrates the complexities of the homeless system in Utah. By interacting with different elements of the image, the end user can better understand recidivism, use of shelters, rehousing efforts, and other supportive services. View this interactive data visualization.

By utilizing best practices in innovation, agencies create a better, more efficient workspace for employees and clients. Innovation should be among the first attempts to solve a challenge faced by any organization. Innovation can often address tight budgets, strained capacity, a stressed workforce, and other common issues organizations face. Only after testing and exhausting other options should organizations consider increased resources as a solution. Effective planning should consider how to foster a positive organizational culture, establish appropriate standards, and develop and refine a strategic plan.

Section 3.1: Assure Compliance with Policy and Procedure

Pitfall: persistent or widespread noncompliance; Outcome:reasonable assurance objectives are achieved

Compliance with policy and procedure is the most basic and perhaps most important element of implementation. We often find that organizations have developed plans but have failed to properly implement them. Inadequate policies and a lack of oversight are common issues identified in our audits. Noncompliance can result from these deficiencies. Widespread noncompliance should signal to governing boards and managers that there are issues with either oversight or policies.

When a plan has not been adequately implemented, it is not possible to properly evaluate it. In some cases, leaders may conclude the plan did not work, when in actuality, the plan was never fully implemented. This error can lead management to discard promising practices, tools, and techniques. It is necessary to fully "test" the strategic plan before it can be appropriately assessed for strengths and weaknesses. Example of Repeated Noncompliance: In the 2019 report, A Performance Audit of the University of Utah's Laboratory Safety Practices, auditors found that the University of Utah's lab safety system showed repeat major deficiencies that substantially impacted safety in many of its laboratories from 2016 to 2017. In fact, 49 percent of research groups reviewed in 2016 still had an outstanding major deficiency the following year, as seen in the following figure.

Grapic showing lab beaker icons illustrating 54 repeat deficiencies and 56 deficiencies corrected

Some of the same deficiencies found at the University of Utah labs existed at other universities at the time of serious injury and fatality, illustrating the serious nature of the deficiencies. The audit explained that poor oversight and coordination were the reasons for these major deficiencies not being resolved.

It is necessary to track the level of adherence to policies and procedures for key components of the plan. Doing so will enable management to properly assess strategies, which will in turn improve operations. In 2014, the Utah Legislature passed a major criminal justice reform called the Justice Reinvestment Initiative (JRI). Example of Implementation Status Affecting the Evaluation: In the 2020 Performance Audit of the Justice Reinvestment Initiative, the Office of the Legislative Auditor General (OLAG) found that one in five major policy recommendations had been fully implemented, as shown in the following figure.

JRI Policy Recommendations	Status
Focus Prison Beds on Serious and Violent Offenders	Completed
Ensure Oversight and Accountability	Not Implemented
Support Local Corrections Systems	Not Implemented
Improve and Expand Reentry/Treatment Services	Partly Implemented
Strengthen Probation and Parole Supervision	Partly Implemented

New initiatives and strategies, particularly those that span functional and interagency boundaries, require sustained effort to fully implement. Evaluators should consider the extent to which any project or initiative can be evaluated reliably and credibly. Evaluations should be scoped to ensure that the information obtained, and conclusions drawn, are accurate. Where appropriate, evaluators should include discussion of implementation status of key components. In the case of JRI, we recommended that the initiative be fully implemented by working together at the state and local levels and that information sharing be improved. In short, before a plan can be evaluated, it must be fully implemented.

Section 3.2: Optimize Resources

Pitfall: inefficient use of resources; outcome:resource allocation maximizes impact

After reasonable assurance of compliance is achieved, another function of implementation is ensuring that resources are used to maximize impact. Sometimes an organization struggles to achieve its objectives because it needs to reconsider how it is allocating or leveraging its resources. In some situations, additional resources may be necessary to achieve the objective. Resource allocation decisions should be grounded in relevant data and meaningful performance measures. Resources are likely to be misallocated when measures misrepresent the situation.

For example, in the 2018 Performance Audit of the Division of Juvenile Justice Services, auditors found that Juvenile Justice Services was reporting cost per bed available, rather than the number of juveniles at the facility. Because facilities are rarely at capacity, the true cost of housing and treating one juvenile was understated. Auditors inferred that this measure was a factor driving increased operating expenses of nearly $3 million, despite a decreasing juvenile population. Because total beds did not represent actual needs, some areas received resources to expand capacity, despite low bed utilization in those same areas. OLAG recommended changing the measure to reflect utilization when making allocation decisions. Example of Lack of Oversight: As part of the 2019 Performance Audit of Tooele County's Sale of the Utah Motorsports Campus, auditors found misleading financial reporting, inadequate budgetary oversight, and inadequate monitoring of capital purchases. One example was utility vehicles that were purchased for $103,000 but were rarely used and were eventually resold for a $25,000 loss. We recommended the county adopt adequate internal controls for budgetary oversight to ensure money is appropriately expended.

A primary responsibility of government leaders is the efficient use of taxpayer dollars. Optimization is about maximizing the value of resources. This requires leaders to consider the opportunity costs of their decisions. Optimization is managing assets in a way that supports the strategic plan. It involves balancing competing needs to maximize the return on investments.

Organizational budgets may improperly be made by simply adapting the prior year's budget and carrying it forward to the current year. While this approach may approximate the organizational resources needed, it does not emphasize optimization of funds. It is useful for high-level, what-if analyses, or when a quick estimate is needed. However, organizations should focus on adding resources that add value to the customer and subtracting resources that do not add value. This requires leadership to be forward-thinking and adaptive to change. Resource allocation decisions should adapt to the changing circumstances of the economy and focus on the mission and future of the organization rather than being reactive to short-term shocks. Collaboration between all programs within an organization is key to optimizing the resources available to the organization.

Choosing how to allocate resources involves a cycle of planning, scheduling, tracking, and evaluating. Financial indicators provide vital information for making allocation decisions. The Office of the Legislative Fiscal Analyst (LFA) has developed the Compendium of Budget Information and Budget Dashboards to allow administrators to quickly access and drill-down into the state's financial data. LFA also maintains a Fiscal Health Dashboard that includes key financial indicators such as "obligations," "reserves," "expenditures," and "revenues." These tools for managing and understanding the state's resources are reflected in Figure 3.1.

State of Utah, 2023 Gen. Session Appropriations by Use Tableeau visualization — Figure 3.1: The Office of the Legislative Fiscal Analyst Uses Budget Dashboards to Monitor Utah's Financial Data. Tools such as this one can provide administrators with easily accessible information to guide decisions.

Financial indicators are not always useful on their own, but rather are intended to be compared over time or against alternative options. By positioning financial information in context, irregularities and opportunities for efficiencies can be identified. For example, a program expense ratio is determined by dividing program service expenses by the total expenses. If this ratio is seen increasing over time, it may alert leadership that a program is becoming less self-sufficient and may prompt them to investigate why.

For day-to-day operations, utilization of a resource management tool can assist management with governance and supervision to ensure projects are running efficiently and within budget. If a tool is utilized properly, management can understand what resources are available, whether they are operating in budget, and whether tasks are being completed by projected deadlines. Common tools include a weekly spreadsheet updating management on the progress of a program or assignment. More advanced project management software also exists for complex projects.

To accurately understand the effectiveness and progress of any program, standardizing expenses to a uniform chart of accounts is key. Example of the Need for Uniform Chart of Accounts. As part of the 2022 Performance Audit of Financial Reporting of Public Education, OLAG found that a nonuniform chart of accounts made comparison and best practices among local education agencies (LEAs) difficult due to differences in expense reporting. Auditors therefore recommended that the Utah State Board of Education provide more detailed function codes that include specific definitions to reduce differing LEA interpretation. To guide organizations, accounting codes used for the state's Financial Information Network (FINET) system are available on the Utah Division of Finance website.

Accounting information can increase its value by being comparable. By formatting expenses uniformly with what is done among other agencies, best practices in efficiency can be discovered, both internally and externally. The Utah Division of Finance provides a wealth of information to assist with uniform accounting and allocation. These tools include a help desk, dashboards, accounting practices, the FINET data warehouse, an accounting coordinator team, and an internal control program. Organizations can use these tools for comparing allocation strategies and costs of similar organizations and programs throughout the state. Organizations that do these kinds of financial comparisons are well-positioned to make prudent decisions about resource allocation.

Once financial reporting accounts have been standardized and upheld, leaders should assess the efficiency and effectiveness of their programs over time (see Subsection 4.2). A simple way to do this is to compare costs against outputs and outcomes. For example, the Department of Alcoholic Beverage Services uses an output measure of bottles sold per hour. Data guides the department's decision-making to inform stakeholders and find both external and internal efficiencies in operation.

Section 3.3: Implement and Calibrate Policies and Procedures

Pitfall: missing or inadequate policies and procedures; outcome: policies and procedures fit organizational objectives

After creating a uniform chart of accounts and comparing financial data with those of similar organizations and programs, management will need to calibrate policies and procedures to ensure they are effectively supporting organizational objectives. Policies and procedures are among the most versatile tools available to management. However, they can be a double-edged sword. Policies and procedures can guide, direct, strengthen, and empower the organization to achieve its mission and goals. They also can stifle, constrain, obstruct, and harm an organization in its accomplishment of objectives. As discussed in Subsection 2.1 on developing a strategic plan, service delivery models facilitate tracking organizational outputs, and then connecting those outputs to the overall objectives and outcomes. Once management understands what operational activities are needed to achieve the desired outcomes, the activities should be crafted into policies and procedures. Sometimes, these are referred to as the internal controls of the organization.

Well-defined, clear, and written policies that are regularly reviewed serve as the backbone of a well-run organization. When policies and procedures are too vague, or are missing, management and governing bodies cannot effectively direct the organization. Example of Failed Adherence to Best Practices: For example, in our 2021 Performance Audit of Healthcare in State Prisons, we observed that "inappropriate or inadequate medical care was given in 17 percent of the sampled cases." Examples included administration of the wrong medications, unreasonable delays, and failure to follow national guidelines. There were also instances of "dangerously low and dangerously high" insulin readings for some diabetic patients. The reason for these "symptoms" was that the Clinical Services Bureau (CSB) had not created policies and procedures for managing inmate health and medical care during disruptions or delays to the normal schedule. As a result, we recommended that CSB create policies and procedures for inmate care. With well-developed and implemented policies, many of the pitfalls we find in implementation can be avoided.

In other cases, policies and procedures may be too rigid and restrictive. An indication of this is when policies and procedures undermine desired outcomes. In an analysis of "What Works" in government programs, the Pew Charitable Trusts found eight programs that were shown to negatively impact the intended outcome. When adopting outside programs or policies that have proven effective elsewhere, it is necessary to ensure the program or policy is a good fit for the operating environment it is applied to. One option that organizations may consider when implementing a policy or program is to have a formal process for making exceptions, such as a requirement for written justification. However, situations where exceptions are too common call into question whether the program has been fully implemented or whether the rule should exist at all.

LFA and the Governor's Office of Planning and Budget (GOPB) released the Performance and Efficiency Initiatives Report in December of 2021 as part of House Bill 326 (2021). They describe a vision for Utah government that "will streamline outdated bureaucracy to focus efforts on outcomes." Removing unnecessary policies and procedures can be as effective as adding new ones. In short, outdated policies and procedures run counter to efficient and effective operations.

Organizations should regularly review administrative rules of their organizations to eliminate unnecessary policies and procedures. The Office of Administrative Rules reported to us that the office is working to simplify the process for adjusting administrative rules, which we expect will allow organizations to modify rules more regularly. Because administrative rules have "the effect of law" and are enforceable, it is important that they are logical, understandable, and concise.

When reviewing policies, procedures, and standards, management should determine their continued relevance. They should also be assessed for effectiveness toward risk and achieving objectives. However, it is necessary to consider how these will be enforced and how the level of adherence will be measured. Monitoring systems and practices can enable managers to see what is occurring inside the organization. For example, in the case of the real-time water monitoring system used in Emery County, monitoring agricultural water use reduced the need for salt load to the Colorado River and the need of additional pesticides and herbicides. Policies and procedures that are costly to enforce and have little to do with mission-critical activities should be closely examined. If it is determined that their contribution is minimal, they should be rescinded. Overly bureaucratic environments bog down systems and tie the hands of employees, discouraging creative solutions and meaningful impact.

Obsolete or burdensome policies should be considered for removal. The governor signed an executive order in December 2021 that directed agencies to continually review existing administrative rules and implement a new process for careful consideration and assessment of new rules. Specifically, it ordered agency heads or designees to:

Repeal rules that are no longer necessary.
Amend rules that create unnecessary burdens or regulations for any individual or entity.
Amend rules that are inconsistent with the current edition of the Office of Administrative Rules' Rulewriting Manual for Utah.

Efficiencies are found not only by adapting new innovations and technologies, but also by removing old policies and requirements that are now obsolete. Getting rid of policies, however, often can be met with challenges. For example, legal barriers, individuals averse to change, or benefactors of the status quo may try to prevent outdated or unused policies from changing. The following strategies can help combat resistance to removing obsolete policies.

Sunset requirements - Sunsets require legislators to review newly enacted policies to measure if they are still needed. If the policy or program is not reenacted at the time of sunset, it ends.
Sunset commissions - A working committee of legislators or agency leadership works to find and decommission obsolete policies.
Agency review - Many policies were not enacted by a legislative body, but by former agency leadership. An internal group can determine which policies and procedures are still helpful to meet desired outcomes.

Capacity constraints are an ongoing concern in government, as organizations are expected to do more with less. By simplifying rules, organizations can augment their capacity and prioritize tasks that move objectives forward. Simplifying rules further makes communication of applicable policy easier.

Many policies and procedures add unnecessary burden and do not serve the organization in advancing its mission. We encourage organizations to work internally to identify and eliminate these rules. We also encourage organizations to work with external stakeholders, including the GOPB, LFA, and OLAG when necessary, to facilitate changes to Administrative Rule and Utah Code where warranted.

Section 3.4: Exercise Oversight

Pitfall: Weak Oversight or Audit functions; Outcome: robust audit and monitoring functions

A key component to assure proper implementation is oversight. Lack of oversight is a common failure often identified during our audit process.

Example of a Lack of Oversight: In our 2018 Performance Audit of Utah's Temporary Assistance for Needy Families Program, we cite a lack of essential oversight for contract elements. Contracts totaled nearly $46 million in 2017, but some contracts were not assessed using the expected outcome metrics. As a result, program effectiveness was difficult to monitor and assess. While outputs were often tracked, the audit report notes:

Activities do not prove progress toward achieving desired outcomes, because a program may have a lot of activity without making substantive progress.

Good oversight should ensure that objectives are achieved and outcomes are improved. When assessing contracts, outcomes are a critical measure, because resource allocation and contract award determinations should hinge on contractors' ability to fulfill the purposes of the program.

Oversight is a process for ensuring that organizational objectives are met through the implementation of policies, procedures, and standards. The accountability framework and reporting lines are central to oversight. Management establishes these reporting lines and the organizational structure, while governing boards assess and provide feedback on the design of management.

When oversight occurs at the governance level of an organization, board members should review management's design, implementation, and operation of the internal control system. Internal controls exist to ensure that operations align with the strategic plan. When the governing board is doing the oversight, the process should consist of five elements identified by the US Government Accountability Office, as outlined in the following figure.

Graphic describing the Five elements of the oversight process: 1. Control Environment - Establish integrity and ethical values, establish oversight structure, develop expectations of competence, and maintain accountability to all members of the oversight body and key stakeholders. 2. Risk assessment - Oversee management's assessment of risks to the achievement of objectives, including the potential impact of significant changes, fraud, and management override or internal control. 3. Control Activities - Provide oversight to management in the development and performance of control activities. 4. Information and Communication - Analyze and discuss information relating to the entity's achievement of objectives. 5. Monitoring - Scrutinize the nature and scope of management's monitoring activities as well as management's evaluation and remdiation of indentified deficiencies.

Oversight also can be used in a broader sense to refer to any individual inside an organization tasked with authority and discretion. When those in authority delegate responsibility to other key roles throughout the organizational structure, ownership is always retained by the organization's higher levels.

Improved decision-making in an organization leads to better outcomes. One standard that is often difficult to understand for new employees-and for those who are new to leadership positions-is decision rights. Decision rights are the empowerment to make a decision without authorization from a superior. Management should determine what decisions need to be made by employees and what tools or resources are necessary to assist employees in making those decisions. Ambiguity in decision rights creates an environment where decisions are delayed, which can cause frustration and failure to perform essential functions.

To ensure an organization has clear lines of responsibility, decision rights should be clearly defined in writing and communicated to staff. Written decision rights should include who is responsible for decisions, expected outcomes, specific decisions that require consultation or approval, and an indication of who needs to be informed of outcomes. An organization runs best when strong, transparent decision lines are drawn and when employees know what they are accountable for and whom they are accountable to. If any vagueness in decision rights exists, employees should take the time to understand decision rights and discuss them with direct management prior to making decisions. Empowering employees to know decision rights and holding them accountable can create a workplace of greater agility and responsiveness, while at the same time lightening the burden of management.

Section 3.5: Cultive an Environment of Growth and Retention

Pitfall: stifled employee development; Outcome: engaged and developed workforce

Organizational culture weaves its way through attitudes, behaviors, decisions, and core values of an organization. Because culture is central to every organization, it has a significant influence on implementation. It is therefore critical that management and staff focus on creating a culture of growth and retention. Government agencies rely heavily on human capital to carry out their services. With rare exception, employee compensation makes up the vast majority of agencies' operating budgets. Consequently, every director and manager should see effective human resource management as a crucial factor to the overall success of the organization. Human resource management primarily consists of employing, compensating, and training employees.

Recruitment and Retention Underpin The Success of Every Organization

Cultivating a positive culture begins with establishing a healthy pipeline through recruitment efforts. It then continues by establishing the supports necessary to retain a talented workforce. Example of a Lack of Retention: In OLAG's 2021 report, A Performance Audit of Teacher Retention within Utah's Public Education System, we reference a study that estimated the cost to replace each teacher who leaves exceeds $20,000. We found that in the first five years of the 2016 cohort of teachers, 43 percent exited the field, suggesting a substantial loss of the state's investment in its educators.

Unsuccessful recruitment and retention amplify the challenges to carrying out the goals and objectives set in the strategic plan. In some cases, extensive vacancies have undermined operations. While competitive pay can influence recruitment and retention, other factors also play an important role in attracting and retaining talent. For instance, in the teacher retention example above, we found that for teachers who remained in the field, workload and stress were more important than pay.

Turnover rates vary drastically by department. Some departments, such as the Department of Alcoholic Beverage Services, have high turnover rates, while others, such as the Department of Insurance, have low turnover rates. Recruitment and retention plans should be tailored specifically to the organization and incorporated into the strategic plan during the planning phase. The following provides guidance on ways to recruit and retain talent aside from compensation, which is discussed later in the subsection.

Employee Recognition Can Increase Morale, Motivation, and Performance. Another important factor found in the survey of Utah's teachers was recognition for work well done. Symbolic awards-like certificates, congratulatory cards, notes of appreciation, and public recognition-have been shown to increase morale, motivation, and performance. This is especially true in fields of public service, like social and healthcare work, where employees are motivated by the positive impact they have on others. When recognizing the contributions of employees, factors like senior management's presentation of the recognition, the timing of the award, and making the award public should be considered.

Successful Recruitment Strategies Vary by Industry and Occupation. Hiring the right people with the necessary skillsets is important to the success of any organization. The selection criteria for candidates and resources available will drive most decisions regarding how to engage potential hires. The following figure provides guidance on matching recruitment strategy to the needs of the organization.

The Division of Human Resource Management, in Administrative Rule, offers alternative strategies to filling vacancies. Agencies should utilize as many avenues as necessary to ensure effective recruitment.

Employee Retention Amplifies Organizational Investment. Generally speaking, turnover adversely affects an organization. One study found that 43 percent of employees named organizational culture as the main reason for searching for a new job. While some turnover is normal, high turnover rates can increase costs, damage morale, and harm productivity. Voluntary turnover can occur for a variety of reasons, including a poor match between the job and skills of an employee, lack of growth, internal pay equity, management, and assigned workload. In some cases, turnover may simply be the result of a strong economy or competitive labor market. Identifying the causes of high turnover is a necessary first step to improved retention moving forward. To reduce turnover issues, organizations should utilize and commit to a retention plan.

Financial Compensation Is One of Several Factors
Impacting Employee Recruitment and Retention

Although pay is not the top reason for employee turnover, 19 percent of employees leaving their job reported compensation as the top reason. While funding largely shapes what a government organization can offer its employees, knowing what employees are able to make elsewhere can inform allocation decisions with the funds that have been apportioned. State employers may adjust salary using a market comparability adjustment to a job's salary range. This is based on salary data and other relevant information for similar jobs in the market through an annual compensation benchmark survey or other sources. Although many compensation policies are outside the direct control of management, leadership should have a good pulse on market conditions to advocate for fair employee pay.

To aide in determining compensation, Utah's Department of Workforce Services publishes a wealth of economic information including wages by region, occupation, rankings, and projections of the job market. Wages are shown for inexperienced workers and median earnings. A sample of "five-star" jobs statewide, which are projected to be high in demand and wage, is shown in the following figure.

Occupation Title	Education	Inexperienced Wage	Median Wage
Accountants and Auditors	Bachelor's	$18.37	$30.82
Computer Network Architects	Bachelor's	$34.16	$48.63
Computer Programmers	Bachelor's	$26.93	$45.30
Database Administrators	Bachelor's	$26.02	$45.66
Financial Analysts	Bachelor's	$26.06	$29.28
Human Resources Specialists	Bachelor's	$17.77	$28.05
Lawyers	Doctoral	$32.83	$50.05
Sales Representative	High School	$15.67	$27.50
Web Developer	Associate's	$14.59	$28.62

Compensation is sometimes an issue identified in our audits. Examples of the Need for Competitive Compensation: As part of the 2017 Performance Audit of the Sex Offender Treatment Program, auditors noted difficulty in both the hiring and retention of psychologists in the program. We recommended partnering with the Division of Human Resource Management (DHRM) to determine competitive pay to retain and hire psychologists. There was a similar finding related to Adult Probation and Parole agents, resulting in additional funding allocated for payroll increases in the 2022 General Session. When considering compensation, among other considerations, thought should be given to competitors' wages, internal equity, the level of talent needed for the role, and room for growth.

General funding for compensation is often restricted by the budget constraints of available funding. However, in addition to recruitment and retention plans, DHRM offers guidance for monetary bonuses for both recruitment and retention. The following are appropriate reasons, according to DHRM, to award bonuses and monetary incentives:

Recruitment or Signing Bonus - Management may award a bonus to a qualified job candidate to incentivize the candidate to work for the state.
Retention Bonus - Management may award a bonus to an employee who has unusually high or unique qualifications that are essential for the agency to retain.

These bonuses can help attract and retain talent in some situations. If management has the funds to provide these, it should do so in a targeted way that rewards high-performing employees. However, funding must still come from traditional sources.

Training Should Enhance Employees' Ability to Increase Efficiency And Effectiveness in Carrying Out Their Respective Duties

A lack of training and experience can contribute to poor implementation. An environment of growth cannot be created if employees are not given opportunities to obtain the necessary skills. In 2010, OLAG recommended that all managers participate in training through DHRM. However, in the 2021 In-Depth Follow-Up of the State's Career Service System, auditors found that "Manager training has not substantially increased since 2010." In fact, the audit reported that two out of three managers do not receive management training from DHRM. Management training is a mechanism to improve employee performance and resolve disciplinary issues. Our survey of executive leaders identified that training is also a concern for some members of executive senior management. One leader said their most pressing need was "Opportunities for leadership training." Even so, only one in six employees had an active performance plan in the Utah Performance Management system. In the 2021 audit, we reiterated the 2010 recommendation that all new and current managers attend management training courses, and we encouraged similar training for supervisors and above. GOPB and LFA have made increased management training one of their initiatives.

An organization where employees lack proper training can result in less productivity, employee dissatisfaction, and poor relationships between employees and managers. Effective training can improve an individual's capability, capacity, productivity, and performance. A study performed by the American Society for Training and Development found that 41 percent of employees at companies with poor training planned to leave within the year, indicating that training also affects retention.

Training opportunities should start with orientation and be complemented by in-house training, external training, and a mentoring program. A mentor is a "a trusted, experienced advisor who has direct investment in the development of an employee." Mentors do not necessarily need to be part of a management team; in fact, it is often beneficial for management to choose mentors who are experienced staff.

Mentorship programs have been utilized as part of the onboarding process for new teachers in public education. From the recent Utah Educator Engagement Survey, 80 percent of early career educators in Utah believe that mentoring has a positive impact on their instructional practice and on their acclimation to the profession. The program also allows teachers to stay engaged with students while giving them an opportunity to advance their career.

Training is often recommended as a solution when internal irregularities are found in audits. Example of Training as a Solution: For example, as part of the 2020 Performance Audit of the Office of State Debt Collection, auditors found inequality in settlement practices. As a result, they recommended additional training to establish more consistent debt settlement practices. Although selecting the proper candidate begins with the recruitment and hiring process, management may establish programs for training and staff development that are agency specific or designed for highly specialized or technical jobs and tasks. Training can be used to expand the capacity of the organization's existing workforce. Training may also ensure uniformity in an organization's application of standards and can make an organization more effective in reaching its goals. Effective implementation should consider how to assure compliance, optimize resources, calibrate policies and procedures, exercise oversight, and cultivate an environment of growth and retention.

Section 4.1: Collect and Protect Data

Pitfall: Missing Key Information; Outcome:Relevant and Accurate Information

Evaluation begins with gaining an understanding of what the organization is currently achieving and what benchmarks it hopes to achieve. This cannot be done without first uniformly collecting and protecting data.

Example of Inconsistent Data Reporting: During the 2022 Performance Audit of Financial Reporting in Public Education, auditors found substantial inconsistent reporting among local education agencies (LEAs). As a result, efficiencies among LEAs were difficult to determine. The following figure illustrates how the LEAs' comparison data were essentially unusable.

balance scale showing different teacher roles between two LEAs with LEA 1 on the left side with curriculum, assessment, and instruction in the scale and LEA 2 with instruction

Our audit report recommended that the Utah State Board of Education ensure that financial data integrity issues be resolved through clarification in its Chart of Accounts and through better compliance and monitoring.

Reliable and comparable financial data are critical to decision-making. Processing and using data, despite significant data integrity issues, can aggravate problems by driving incorrect decisions. This creates a scenario where the burden of data collection continues, but the organization is better served, in some circumstances, by not using the data or using it in limited ways. In these situations, either data integrity issues should be resolved, or data collection efforts should stop, as the value of the data does not justify the burden of collection. The following are some approaches organizations can consider for improving data integrity:

Adopt national data standards.
Establish clear procedures for data collection and entry.
Use validation rules to prevent mis-entry.
Use software applications to minimize formatting errors.
Perform ongoing data audits.

Management should review datasets for unexplained outliers and trends that may indicate the presence of integrity issues. Entries of expenses and codes should follow a standard classification guide and checked for misclassification. A significant number of blank entries for required fields is another red flag. To aide in identifying integrity issues, tests for common issues like spurious correlation or misinterpretation of variables can be conducted.

Good data collection affects every other facet of an organization by providing a foundation for data-driven decisions. It allows an organization to understand why past events occurred and gain insights into the future. Data collection should support an ongoing system of inquiry that allows managers to get answers to important organizational questions. As discussed in Subsection 2.1, a service delivery model can set the stage for this system of inquiry. Management should determine what data are necessary to answer their questions and then determine if the data they need is already being collected. Whenever possible, existing data resources should be utilized to reduce adding additional costs and burdens of collection. Common state resources can be found in Appendix C of this handbook.

If the necessary data is not already being collected, management should build in processes of measurement, collection, and record transfer to ensure the data is high-quality. Examples of primary data collection include:

Surveys
Transactional tracking
Observation
Online tracking
Forms
Social medial monitoring

Whether collecting data or using data that have already been collected, data integrity should be a primary consideration so that error-riddled information is not used to make decisions. Data integrity issues that should be addressed include missing data, misconduct, unintended human error, errors when transferring data, and different interpretations of the same concept when entering data.

Data Sharing Creates Organizational Efficiency and
Insight, but Data Security Is Also a Top Priority

Evaluation comes at a cost of employee resources and time. Frequently, internal evaluation follows a formal process (see Subsection 4.2) to yield new insights and meaningful performance measures. However, this cost is reduced, and in some cases eliminated, when organizations share their data with one another. Partnering with other organizations and their staff often can yield new insights for both the receiving and sharing organization. Making data available to others creates an additional incentive for research integrity in analysis and collection. Partnership in research can reduce costs while providing insights that cannot be obtained with only one agency's data. Example of Inconsistent Data Formatting: In 2020, as part of A Performance Audit of the Justice Reinvestment Initiative, auditors encountered difficulty in obtaining inmate data due to formatting issues. Such data would help assess the effectiveness of the Justice Reinvestment Initiative and other treatment programs. Because data uniformity was lacking, the state risked allocating funding to treatment programs that were ineffective.

Sharing data increases its use, which further augments the value of the data. Because data collection comes with a cost, organizations should seek to maximize its value. Additional security precautions should be taken when sensitive data, like health records, are stored or shared. The following figure offers considerations that can inform the development of appropriate security measures.

Risk scale with higher numbers having greater risk: 1. Publicly available and unrestricted data. 2. Unpublished non-sensitive data. 3. Regulated data that could damage the subject's reputation. 4. Sensitive data, including financial data, that could place the subject under civil or criminal liability. 5. Sensitve data that could subject the entity to significant harm.

For example, the Utah Data Research Center, a multiagency research institution housed at the Utah System of Higher Education, shares data with requestors that may include public education, higher education, and workforce information. However, the center uses proxy IDs to replace identifiers like social security numbers, student IDs, full names, and dates of birth. Additionally, if a group has fewer than ten individuals (for example, a race group), data for these individuals are hidden in the data set.

De-identifying sensitive or identifiable information
Establishing guidelines for use
Ensuring that staff members are adequately trained
Separating user accounts
Implementing access controls, which limits data resources to authorized users
Regularly scanning computers for spyware
Using and maintaining antivirus software and firewalls
Password protecting and encrypting data
Proper disposing of sensitive information

Data Collection Should Adhere to Law and
Exclude Personally Identifiable Information When Possible

A government organization that collects personal information on individuals is responsible to protect that information in accordance with privacy laws and regulations. However, fears of misappropriately handling data should not deter organizations from collecting and sharing needed information. Rather, management should work with stakeholders and experts to collect and secure the necessary data. Government is responsible for the safekeeping and protection of information. Afterward, sharing data with relevant stakeholders can and should exist in harmony with data privacy laws. Prior to processing, storing, managing, or analyzing data, it is necessary to ensure the data are ethically and legally sourced. For example, education data are subject to the Family Educational Rights and Privacy Act (FERPA), the Student Data Protection Act, and the Student Privacy Act, which provide guidance for safeguarding sensitive student information. Health records are subject to rules of the Health Insurance Portability and Accountability Act (HIPAA), which protect the privacy of health information of individuals. Personally identifiable information is important to exclude, especially when studying small samples or vulnerable groups. Some common data privacy laws that leadership should be aware of include the Fair Credit Reporting Act, FERPA, Electronic Communication Privacy Act, Children's Online Privacy and Protection Rule, HIPAA, the Code of Federal Regulations 42, and the rules established by the Federal Trade Commission Act.

Unless a grant or program requires the use of personally identifiable information, data should be de-identified, meaning the information cannot be linked back to an individual or individual device. This applies to individual data points and aggregate data. Extra precaution should be taken when multiple databases are joined, which can add information linking to the same personal record. Data that are considered to be sensitive include social security numbers, biometric information, physical health, mental health, genetic information, and information of minors.

Organizations should strive to understand what information is protected, not only to restrict what can be shared, but also to encourage data sharing when appropriate. Example of Siloed Data Storage: For example, as part of the 2020 Performance Audit of Information Sharing in the Criminal Justice System, auditors found that because of siloed data practices, criminal justice personnel, policymakers, and administrators were not always getting timely information to do their work properly. Siloed data limited agencies from getting the information they needed to make critical decisions. To remediate the issue, auditors recommended the establishment of an Information Sharing Environment (ISE) to ease the access of data when legally appropriate, as seen in the following infographic.

Left side of image depicts silos connected together with threads that are also connected to people representing data silos interfering with timely, accurate, and reliable sharing of information with the caption `Disparate decentralized information systems make information exchanges complex, limiting agencies' abilities to get the information they need when they need it`. Right side of informaton depicts a barn with locks and threads connected to the locks and people representing an information sharing environment that would give agencies the timely, accurate, and reliable information they need with the caption `Standards and common procedures, accompanied by a technological infrastructure would enable the seamless and secure sharing of information-information that would still securely reside with the originating agency.`

Data Storage Should Be Guided by the Intended Use

When accessing information is too cumbersome or difficult, evaluations may be stifled. Consequently, management and information professionals should aim to streamline the process for getting the necessary data to do evaluations. They do this by removing barriers that stand in the way of evaluators. For example, data storage should follow the directives of a data use agreement, also known as a memorandum of understanding (MOU). An MOU is a contract that details the requirements for confidentiality, nondisclosure, data sharing, data transfer, and storage. Some MOUs require data to be stored, while others require data to be disposed. Raw data should be accessible only to parties that will be directly working with or reviewing the results of data products. When the rules surrounding access and use of data are unclear, or the process is too slow and cumbersome, evaluation studies are hindered. By establishing clear rules and guidance, management encourages the evidence-building process, which in turn enhances evaluation, decisions, and operations.

Data preservation involves ensuring that data are actively stewarded, managed, and migrated as formats change over time. This typically involves storing data in a repository and converting the data into a sustainable format. A top priority for every organization should be establishing transparent and secure practices for sharing and storing data. Organizations should strive to balance data privacy with data sharing to maximize its usefulness, while ensuring that sensitive data are safeguarded.

Section 4.2: Evaluate the Condition of the Organization

PItfall: uninformed decisions; Outcome: rigorous evidence and analysis

Sometimes the pitfalls in an organization are unknown to management. Evaluations should be used to inform management of the condition of the organization. Not only does management need to know the extent to which they were (or were not) successful in achieving objectives, they need to know why. A mistake managers may make is to conclude that their plan was not effective, when in actuality, the plan was never fully implemented.

Understanding the causes for success or a lack of success is what enables organizations to meaningfully and intentionally improve. When organizations do not understand why they are getting the results that they are, they may still accurately identify issues but will not be able to effectively address them. Example of Untracked Data: For example, as part of the 2019 Performance Audit of the Waste Tire Recycling Fund, auditors found that the Division of Waste Management and Radiation Control was not aware of the issues with existing waste tire piles because the data were not being tracked. The lack of knowledge surrounding the condition of waste tires led to management being reactive, relying on local entities to request assistance when needed. Tire waste adversely affects not only the environment but also the health of individuals. Auditors therefore recommended that "the Division of Waste Management and Radiation Control maintain an up-to-date list to track the presence of waste tire piles statewide." Maintaining a pulse on the condition of an organization can often be the catalyst for successfully achieving objectives.

The first consideration of an evaluation effort should be the level of compliance with policies, procedures, and strategic plans. To determine whether efforts were successful in achieving objectives, management needs to know if noncompliance was a contributing factor. Noncompliance can lead management to discard valuable practices, tools, and techniques. To overcome this pitfall, managers should monitor compliance and request the support of internal audit, where appropriate. Internal audit offices should be able to provide some, if not all, of the information related to compliance. This is one reason we recommend that internal audit directors report directly to executive management.

Organizations Should Assess the Extent to Which Objectives
Are Achieved, Then Optimize Processes to Minimize Waste

Once the level of compliance has been determined, management should seek to understand the extent to which the organization was effective and efficient in achieving objectives. Effectiveness refers to whether objectives were achieved, while efficiency is measured by getting the most from the available resources. The goal should be to assure that objectives are accomplished, and then processes should be optimized to minimize waste. Rigorous analysis and evaluations are an essential part of knowing what works, what does not work, and what to change. There is a wealth of information on enhancing evidence-building practices; this handbook touches on only a few key principles and practices. Organizations may want to expand on what is presented here to maximize the value of their research.

Organizations Should Use Rigorous Evidence and Evaluations to Support Strategic and Design Decisions. When designing a program or strategic plan, managers should look to reports and other similar programs to uncover research and evidence that already exist and could guide the design. Stakeholders should also contribute to the design by articulating their needs and experience. However, rigorous evidence and evaluations to support the design of programs and strategies of the organization do not always exist. In these cases, the organization needs to develop a process for building the needed evidence to inform decisions and increase the likelihood that objectives will be achieved.

When managers and staff take on the role of evaluator, a formal research process can guide inexperienced personnel in producing evidence that is reliable and sound. Experienced staff should oversee this process to ensure that the quality of research is sufficient to be used for making decisions. Not only will this improve decisions made and actions taken, it also will increase buy-in of stakeholders as they better understand the rigor that is informing choices.

For evaluations to be reliable, they need to be objective. Adhering to a standard process prevents evaluations and research from deviating to questionable practices or untrustworthy sources. Generally, research follows a process of defining the issue, doing a literature review, collecting and analyzing data, verifying conclusions, communicating findings, and publishing the results, as shown in the following figure.

Diagram showing the research process with the steps 1. Define the issue. 2. Literature review 3. Data Handling 4. Verification 5. Communication 6. Publication

An ethical evaluation environment promotes truth, fosters collaboration, is accountable to the public, and builds trust with clients and the community. Creating a culture where integrity is the norm should be reflected in everyday work and should be formalized through written organizational policy. Methods should be transparent, rigorously documented, validated, and reviewed. Although research topics may vary widely, a framework for how research is conducted should be established, monitored, and followed within an organization.

Research begins by clearly defining and understanding purpose. Common misunderstandings can be avoided with a clearly specified research objective. Reviewing prior research is often sufficient to answer research questions without additional work. This process can be streamlined through collaboration with subject matter experts, who can quickly point research staff to relevant and credible bodies of work. When a literature review is complete, staff should determine whether the research question has been answered. If additional work is necessary, research staff should take additional steps as needed to properly design programs and inform strategies.

Changes to Strategic Plans and Programs Should Be
Guided by Evaluations and Impact Assessments

Evaluations come in many forms, as do the data that support evaluations. Directors and managers should determine which data points are needed to assess the impact of the initiative or program. They should also determine how the necessary information will be collected. Administrative data may already contain the main data points that are needed. Observations, surveys, or other methods may also need to be considered.

Not all evaluations are equal in their rigor and reliability. Calibrating evaluations to the strength of evidence needed can avoid waste during the evaluation process. Understanding the types of evaluations that exist, and knowing when it is appropriate to use them, can further guide evaluation choices. The amount of resources devoted to the evaluation should be weighed against the value and costs of the program. Other states have strengthened and standardized evaluations. For example, the state of Tennessee has developed a framework to determine the effectiveness of its government programs through formal research practices. Tennessee relies on "evidence-based budgeting," which is designed to standardize the process for generating new knowledge. The research and evidence are then used to inform funding decisions. This model is shown in Figure 4.1.

Pyramid Showing a ranking of evaluations. Starting at the top are three qualifying evaluations with their respective labels: Systematic reviews - draw on multiple experimental studies to form conclusions and consider the quality of included studies. Randmonized Control Trials (RCTs) - randomly assign subjects to treatment and control groups and compare group outcomes of interest. Quasi-Experimental Studies (e.g., non-randomized controlled studies, comparative interrupted time series) - The main difference between RCTs and quasi-experimental studies is that subjects are not randomly assigned to create the comparison groups. QE studies utilize statistical controls to try to create equivalent comparison groups. The bottom of the pyramid has two non-qualifying evaluations: Observation Studies (e.g., cohort studies, case-control studies, case studies) - attempt to understand the outcomes of a group without the use of a comparison group. Expert opinion, logic--based, anecdote - weak forms of evidence without academic research studies to validate them. — Figure 4.1: Rigorous Evaluations Inform Funding Decisions in Tennessee. By gathering evidence, the state can assess programmatic impact.

In Figure 4.1, the higher up on the pyramid, the more robust the evaluation is. By ranking evaluations that have been done, Tennessee decision-makers can weigh the strength of evidence that supports a particular course of action before making a decision. The model developed by Tennessee adheres closely to best practices for research. The Results First Initiative, developed by Pew-MacArthur, also aligns with this model. Organizations may refer to it and use the model where appropriate.

Other states have developed their own evidence-building models, such as Colorado's Evaluation and Action Lab and North Carolina's Project Portal. In 2018, Congress passed the Evidence-Based Policymaking Act , directing federal agencies to use formal data collection and analysis for their programs, policies, and organizations. More can be found on this initiative by reading the U.S. Office of Management and Budget's publication, "Managing Information as a Strategic Resource."

Communication Should Include Actionable Policy Changes. Even the most impactful evaluation results will not lead to positive change unless the results are communicated effectively. As discussed in the next subsection (4.3), knowing the target audience and visualizing the message through interactive dashboards can amplify the impact of the research. Staff should consider the time the audience has to consume the information and adjust the message to accommodate. Additionally, expanding reach via social media and newsletters can further increase the use of research findings

Section 4.3: Communicate Effectively

Pitfall: Missing or ineffective communication; Outcome:Communication tailored to target audience

Effective communication must occur internally and externally to positively impact the accomplishment of organizational objectives. Impactful research and analysis that are communicated ineffectively can get lost in the vast amount of information now easily accessible. By contrast, effective communication can advance the objectives, strategic plan, and goals of an organization. Effective communication should begin by considering the level of knowledge, preferred mediums, and needed frequency of your target audience. These mediums are not limited to research reports and include traditional outlets like newspapers and television, as well as newer outlets like social media. Both the target audience and the research staff benefit when work is presented to the broader public and policymakers. Because of the vast amount of easily accessible information, research is most effective when it is succinct and clear. Data visualizations aid in communicating results effectively and quickly. Communication should be designed for the target audience, data visualizations should convey conclusions, and conclusions should be easily comparable and accessible.

Communication Should Be Designed for the Target Audience

The Office of the Legislative Auditor General (OLAG) often recommends reporting new, more relevant metrics so that the Legislature can effectively craft policy, and so that the public can form accurate opinions. This keeps stakeholders informed and promotes a culture of transparency. Example of Inaccurate Data: For example, as part of the 2017 In-Depth Follow-up of the Division of Drinking Water's Minimum Source Sizing Requirements, auditors found it difficult to recommend policy change, because necessary data were not being reported. Auditors and various water systems expressed concern about the accuracy of the data that were being collected at the time. As a result, auditors recommended reporting peak day demand, average daily demand, and unaccounted water use to the legislature.

Although new reporting is sometimes necessary, the burden of reporting can stifle employee time that might otherwise be used for other essential activities or innovation. Current reports should be periodically reviewed to gauge whether they are being used. Tools like Google Analytics can be employed to understand trends and how often research is read. To reduce burden, obsolete administrative rules should be reviewed and eliminated. More information on this process can be found in Subsection 3.3 of this handbook.

Data Visualization Is Critical to Understanding Conclusions

The background of the target audience is often different from that of an evaluator. Data visualization is critical to understanding the trends and patterns of a data set. Effective visualization lends credibility to conclusions, increases understanding, and drives interest in the research.

Good data visualization should be simple to understand and communicate the conclusion succinctly. Best practices of data visualization include:

Function should always be considered first prior to the form of a visualization.
Visualizations should show an overview of conclusions prior to offering filter options.
Use color or callouts to highlight conclusions.
Color scales should be picked with color-blind individuals in mind (8 percent of men experience color blindness).
Limit color scale to 12, as it is difficult to distinguish differences with greater numbers.
In large sets, sorting values sequentially helps with comparison.
Avoid using pie charts, with some exceptions for percentage breakdowns.
Start the y-axis at 0.
Use 3D visualizations only when illustrating spatial data.
Avoid using tilted text, as it is often not legible.
Use animation to show differences in the same data set.

Conclusions Should Be Clear and Convincing. Results from the evaluation should be clear and convincing. To reduce skepticism, results should be easy to understand. Noncomparable data and vague insights should be avoided. For data used for evaluation to be impactful and meaningful it needs to be uniform and comparable to other data points. Data should be formatted in a way that makes access and usability easy for users unfamiliar with statistics. Interactive data visualizations are especially useful for large or complex data sets. Example of Interactive Data Visualization: For example, as part of the 2022 Performance Audit of Financial Reporting in Public Education, auditors found data tools to be inadequate to answer some of the basic questions about efficiency in public education spending. As part of the audit, auditors showed how tools could be more useful and intuitive through the creation of two dashboards. Auditors recommended that "the Utah State Board of Education modernize the financial data it publishes on its website by providing raw data as well as dashboards for LEAs and stakeholders to assess their performance relative to peers."

Interactive data visualizations should be web-based, as as not to limit users who do not own a specific software or use a specific operating system. Interactive data visualizations encourage end users to explore the data and increase their understanding. Furthermore, end users can filter and manipulate the visualization to get the answers they want. This maximizes the value of the data and extends the reach of the message. View this interactive data visualization.

Research consumption and understanding increase when research prioritizes the communication preferences of the target audience, when visualizations are built for comprehension, and when conclusions are concise and easily comparable.

Section 4.4: Balance and Evaluate Performance Measures

Pitfall: missing or inadequate performance measures; outcome:aligned incentives and feedback loop

Performance measures drive an organization toward its objectives and goals. Evaluating these measures occasionally to ensure they meet the intended objectives is vital to an organization's success. Any single performance measure on its own can be misleading. Good performance management demands careful selection of a balanced set of measures to be meaningful and accurate. Using the service delivery model described in Section 2, management can establish measures that are integral to the overall success of the initiative. However, because industries and tasks vary widely, no single system of performance management can or should be applied to every environment. Consequently, it is the responsibility of organizational leadership to build a performance management system that makes sense for their organization. However, there are certain principals and best practices that should guide leadership in how they structure their performance management system. Leaders are accountable for how well they develop a performance management system that advances the mission of the organization and facilitates the achievement of objectives.

Measures Should Be Organized by Their Intended Purpose

Performance measures influence both organizational activities and organizational decisions. The purpose of a measure should be clearly understood, and measures should be organized around that purpose. For example, two reports released by the Governor's Office of Planning and Budget-the GOPB Guide to Strategic Planning and the Performance Measurement Playbook-describe various types of measures according to their function:

Outcome measures demonstrate how well the agency is achieving the desired outcomes.
Output (operational) measures demonstrate how well the systems are working at producing the desired outcomes.
Individual measures show how each individual is contributing to the systems that produce the outcomes.

We would add "input measures" as a category to understand what resources are going into the system, as this is a critical component for assessing efficiency and effectiveness. To better understand these measures, we will look at each separately.

Outcome Measures Are an Essential Part of the Performance Management Framework. This type of measure assesses the effectiveness of the organization in achieving its goals and advancing its mission. It should be part of the performance management framework. A typical outcome measure is cost-effectiveness, which is a way to relate benefits to costs. It may be viewed as government's "bottom line." Cost-effectiveness is calculated by simply dividing outcome units by input units. For example, a vocational rehabilitation program might measure cost per client placed in suitable employment, or, perhaps better, cost per client successfully employed for at least six months. Outcome measures can help in making resource allocation decisions, as program directors and executive management learn where they can have the greatest impact.

Output Measures Give Insight into the Operations of the Organization. Middle managers and program directors tend to focus on outputs because it is within their control, and outputs are intended to translate into outcomes. Building on the example of vocational rehabilitation mentioned above, an output measure for the program might be number of client appointments in a week. If done correctly, more time with clients should lead to more job placements, which is the desired outcome. Cost-efficiency is determined by dividing the output units by the input units. As part of their oversight role, funding entities are interested in understanding how money was spent; output and cost-efficiency measures may be helpful to report to these stakeholders. However, when combined with outcome and cost-effectiveness measures, a more complete picture is conveyed that can guide decision-makers in the most prudent use of resources. The following figure provides examples of output and outcome measures.

Organization	Goal	Output Measure	Outcome Measure
Department of Natural Resources	Decrease number of out-of-service bathroom facilities	Number of customer notifications received about out-of-service bathroom facilities. Mean and median response time from notification to resolution	Percent Change in out-of-service bathroom facilities
Department of transportation	Reduce road fatalities due to highway infrastructure	Nubmer of hours to clear designated commuter corridors. Nubmer of potholes reduced	Number of fatlities per 100,000 vehicle - miles of travel on state highways

Individual Measures Ensure That Activities Clearly Link to the Operational Measures. This type of measure is most likely to influence employee behavior. Consequently, management should consider whether there are unintentional incentive structures that do not align with organizational goals and objectives. Individual measures are similar to output measures, except they are limited to the output of an individual or team. Examples of individual performance measures include:

The number of lawns mowed or areas cleared of snow for a groundskeeper working for the Division of Facilities Construction and Management.
The number of claims processed and accuracy of figures for a claims analyst at the Utah Office for Victims of Crime.

Some metrics can encourage gaming and should be mitigated, as noted in Section 2 of this handbook. To effectively promote enhanced performance, management should use a balanced set of measures and frequent check-ins to mentor staff. The final part of this subsection further describes undesirable incentive structures and how to avoid unintentionally encouraging them.

Input Measures Are Critical for Understanding the Extent to Which Outputs and Outcomes Are Efficiently Achieved. Resources represent what was available to the organization for carrying out the activities needed to produce government services and products. For example, input measures for building the West Davis Corridor included labor, machinery, contractors, and administration. In some situations, a prorated cost based on use of facilities and other assets may be appropriate. Inputs are a critical component of evaluating both efficiency and effectiveness, as they provide important context for understanding the value of products and services. Input measures can also inform capacity, including capacity constraints that may undermine an organization's ability to achieve its objectives. Because evaluation involves properly identifying the causes of certain desirable or undesirable outcomes, evaluators should use input measures to understand the trade-offs made by management in how resources were allocated, and why those decisions were made. Knowing the purpose of a proposed measure allows evaluators and managers to intentionally select and balance performance measures to create an effective performance management system.

Performance Measures Should Be Assessed to
Ensure Accurate and Useful Information Is Captured

To be useful, performance measures must meet certain criteria. These criteria ensure the information produced is of high quality and is relevant to stakeholder questions and interests. For example, a program director who is seeking to influence smoking cessation needs to have reliable information on the effectiveness of their public health campaign. If the performance measure used does not relay that type of information, or the information is inaccurate, any decision based on the information will be misplaced.

There are five criteria directors and managers should assess their performance measures against:

Meaningful
- Does it capture information about value from taxpayer investment?
- Does it align with the organization's mission and goals?
- Is the measure comparable to baselines or benchmarks?
- Can the measure be used by policy makers to make future decisions?
Valid
- Does it accurately show what it is intended to show?
- Is it the appropriate type of measure (input, output, outcome, etc.)?
- Is it at the appropriate level of detail (for example, not reporting results for a large line item when measuring a small sub-program)?
- Can the results be reported in a timely manner and at regular intervals?
Reliable
- Does it consistently show what it is intended to show?
- Is it responsive to changes in a timely manner?
- Is the methodology well-documented?
Practical
- Can results be collected in a way that is not overly burdensome from an administrative or cost perspective?
- If the ideal data is not available, what information can serve as proxy?
- Does the measure align with the goals of other funding sources (such as the federal government)?
Clear
- Is it well designed?
- Is it written in plain language?
- Can the results be easily interpreted?

By checking measures against these criteria, organizations can improve the accuracy and relevancy of the information that is collected. They can also be sure that the costs of collection do not exceed the benefits of the information obtained.

Directors and Managers Should Design a Set
Of Measures to Produce a Balanced Picture

Not all activities or performance should be measured. Instead, directors and managers should emphasize critical activities, high-risk areas, and information requested by stakeholders when selecting performance measures. The set of measures should be designed to reduce redundancy and increase coverage. Example of Balancing Performance Measures: OLAG's 2021 report, A Performance Audit of Social Service Agencies' Performance Measures, found that assigning a similar number of measures to all line items underrepresented the largest budgets. For example, a Medicaid Services line item of $3.3 billion in fiscal year 2020 had the same number of measures as the Children's Health Insurance program, which was appropriated $139 million. The Governmental Accounting Standards Board (GASB) recommends that "[performance information] should be provided at the most appropriate level of aggregation and disaggregation." As budgets, impacts, complexity, and risks increase, a more balanced approached to measures to increase visibility and make systems more agile and responsive should be applied.

What makes a good set of measures?

Representative
- Is a complete story being told?
- For line items measures, are all budget areas adequately being reported?
- For a program, does it capture the full breadth of what the program does?
- Are risk areas being monitored?
Results-oriented
- Are outcomes emphasized and highlighted?
- Do the measures clearly link inputs, outputs, and outcomes together?
- Can we identify what is an is not working well?
Accessible
- Are measures available to stakeholders when they need them?
- Is it costly in resources, such as staff time, to gather and submit performance information to stakeholders?
- Is the information used in operations consistently to guide decisions?
Useful
- Is it clear what the performance information is communicating?
- Is the information structured to facilitate evaluations and knowledge-building research?
- Are measures revisited to ensure performance information addresses current stakeholder questions or concerns?
Selective
- Are the most critical activies and outcomes being measured?
- Do the benefits of the information collected justify the costs?
- Is this the information needed by stakeholders to make decisions?

According to GASB, a good set of measures should include at least three broad types: efforts (inputs), accomplishments (outputs and outcomes), and the relationship between efforts and accomplishments, such as cost-efficiency and cost-effectiveness measures. Directors and managers are responsible for balancing the set of measures. Knowing who the measures are for and how they will be used can guide management thinking. For example, managers should choose measures that enable them to quickly identify progress toward objectives and targets, and pivot when their efforts are not producing the intended results.

High-Stakes Accountability Can Undermine and
Distort Performance Management Systems

Leaders should recognize that the process of measurement influences behaviors. Eli Goldratt, an organizational improvement consultant, described this, stating,
Tell me how you measure me, and I will tell you how I will behave. Measures and indicators reflect the state of the system and influence it. For the same reason, it is necessary to ensure that performance measures are not easily "gamed." Misaligned incentives should be avoided by making performance rewards and punitive actions reasonable. While accountability and performance management are critical components of government, a singular focus on high-stakes accountability can distort and undermine a performance management system.

There are several ways to mitigate the risk of inadvertently introducing adverse incentives, including:

Choose a variety of well-balanced measures.
Be up-front regarding the intended uses of performance information.
Involve staff in the development process.
Be open to challenges to chosen indicators.
Offer timely and constructive feedback.

Management should take care not to create unintended consequences by producing misaligned incentives for their employees.

Evaluative Activities Should Inform Revisions to Strategic
Plans, Policies and Procedures, Standards, and Goals

Evaluation is a key component of the Management Cycle, but it is not the final step of the process. Evaluation should lead an organization to recalibrate strategic plans, policies and procedures, goals, standards, and objectives to better advance the overarching mission of the organization. All evaluative activities should be designed with this end in mind, including data collection and sharing, the research processes, reporting, and performance management. As rigorous evaluations are used regularly to inform planning and implementation processes, organizations will continuously exhibit gains in the efficiency of their operations and in their effectiveness in advancing their missions.

Agency Information Form

Best Practices Self-Assessment

Results

Section 2.1: Develop and Refine a Strategic Plan

Ensure Strategic Plans Contain Key Elements That Guide Operations and Provide Transparency to Stakeholders

Delivery Models and Performance Metrics Ensure Strategic Plans Are Clear, Concise, and Measurable

Reliable Cost Estimation Enables Effective Cost Allocation

Baselines and Benchmarks Help Organizations Make Consistent Progress

Qualitative Information Should Accompany and Complement Quantitative Data Whenever Possible

Section 2.2: Establish Standards

Autonomy and Courage to Make Decisions Are Essential for Successful Operations

Section 2.3: Foster Positive Organizational Culture

Section 3.1: Assure Compliance with Policy and Procedure

Section 3.2: Optimize Resources

Section 3.3: Implement and Calibrate Policies and Procedures

Section 3.4: Exercise Oversight

Section 3.5: Cultive an Environment of Growth and Retention

Recruitment and Retention Underpin The Success of Every Organization

Financial Compensation Is One of Several Factors Impacting Employee Recruitment and Retention

Training Should Enhance Employees' Ability to Increase Efficiency And Effectiveness in Carrying Out Their Respective Duties

Section 4.1: Collect and Protect Data

Data Sharing Creates Organizational Efficiency and Insight, but Data Security Is Also a Top Priority

Data Collection Should Adhere to Law and Exclude Personally Identifiable Information When Possible

Data Storage Should Be Guided by the Intended Use

Section 4.2: Evaluate the Condition of the Organization

Organizations Should Assess the Extent to Which Objectives Are Achieved, Then Optimize Processes to Minimize Waste

Changes to Strategic Plans and Programs Should Be Guided by Evaluations and Impact Assessments

Section 4.3: Communicate Effectively

Communication Should Be Designed for the Target Audience

Data Visualization Is Critical to Understanding Conclusions

Section 4.4: Balance and Evaluate Performance Measures

Measures Should Be Organized by Their Intended Purpose

Performance Measures Should Be Assessed to Ensure Accurate and Useful Information Is Captured

Directors and Managers Should Design a Set Of Measures to Produce a Balanced Picture

What makes a good set of measures?

High-Stakes Accountability Can Undermine and Distort Performance Management Systems

Evaluative Activities Should Inform Revisions to Strategic Plans, Policies and Procedures, Standards, and Goals

Baselines and Benchmarks Help
Organizations Make Consistent Progress

Qualitative Information Should Accompany and
Complement Quantitative Data Whenever Possible

Autonomy and Courage to Make Decisions
Are Essential for Successful Operations

Financial Compensation Is One of Several Factors
Impacting Employee Recruitment and Retention

Data Sharing Creates Organizational Efficiency and
Insight, but Data Security Is Also a Top Priority

Data Collection Should Adhere to Law and
Exclude Personally Identifiable Information When Possible

Organizations Should Assess the Extent to Which Objectives
Are Achieved, Then Optimize Processes to Minimize Waste

Changes to Strategic Plans and Programs Should Be
Guided by Evaluations and Impact Assessments

Performance Measures Should Be Assessed to
Ensure Accurate and Useful Information Is Captured

Directors and Managers Should Design a Set
Of Measures to Produce a Balanced Picture

High-Stakes Accountability Can Undermine and
Distort Performance Management Systems

Evaluative Activities Should Inform Revisions to Strategic
Plans, Policies and Procedures, Standards, and Goals