Monitoring

4. Monitoring#

How to measure organizational and business success on a shoestring or when you have little to no budget

4.1. Background#

Monitoring, Evaluation, Accountability, and Learning (MEAL) is crucial for ensuring the effectiveness and success of projects, as well as long-term organizational and business viability. MEAL enables organizations to make informed decisions and adapt strategies in real time by

🎯 Systematically tracking progress
🎯 Assessing outcomes and analyzing data
🎯 Maintaining transparency
🎯 Creating a culture of learning, sharing results, and creating opportunities to compare projects across landscapes, hopefully finding solutions to broader conservation challenges

This comprehensive approach helps identify and address issues early and fosters a culture of continuous improvement. Effective MEAL practices enhance accountability to stakeholders, ensure resources are used efficiently, and contribute to the overall sustainability and impact of projects. Despite the importance of MEAL, donors rarely provide adequate funding for it. Given this reality, you must find ways to implement MEAL with whatever resources—personnel, time, and budget—are available. This quick-step guide will get you started.

Definitions

🔭 Monitoring involves tracking project activities and progress.
🔍 Evaluation assesses the effectiveness and impact of the project.
🧮 Accountability ensures that the project is transparent and responsible to stakeholders.
✏️ Learning refers to using insights from monitoring and evaluation to improve future projects and practices.

4.2. Takeaways#

Key points to monitoring, evaluation, accountability, and learning on a shoestring:

Big picture first. There is a tendency to jump straight to metrics and indicators before developing a theory of change and programmatic goals and objectives. Monitoring plans should consider the big picture, so go through the process of developing the program strategy first. However, once created, feel free to change items, such as false assumptions or critical pathways that don’t reflect reality.
Assessment & learning. Link your assessment or theory of change to your monitoring and learning, e.g., Theory -> Monitoring -> Learning.
Time to learn. Build learning into our organizational culture. Plan a retreat to analyze results and write about project learning that you publish in blogs, posts, and journals. Build learning into project closing and developing new projects.
Big investment/big rewards. Building a successful measurement system takes time, staff capacity, and investment. However, the rewards are huge, and the cost dramatically decreases once the system is in place. Don’t fall into the trap of processing, analyzing, and sharing data near the end of a project or phase.
Participatory research and measurement. Partner with science-based organizations to increase your learning and expand capacity, or use citizen science to engage and connect the general public to your mission and collect data/demystify conservation science at the same time. Employing participatory approaches at all levels of evaluation also takes more time but makes for more meaningful, longer-lasting, and effective results in the long term.

4.3. Challenges#

Monitoring could be the subject written about the most but implemented the least in the nonprofit world. Why is this? The challenges involved give an insight:

Monitoring is hard to do and is hard to do right.
Monitoring is boring.
Your organization doesn’t have staff capacity or applied scientists who can conduct monitoring.
Project funding doesn’t include monitoring, and it can be costly.
It could demonstrate that your theory of change, project assumptions/interventions, or organizational mission is incorrect. That’s a big elephant in the room.
Your organization lacks a learning culture.

4.4. Solutions#

How can these challenges be overcome without requiring massive institutional investment and resources? Let’s start with the big picture of starting with the whole and work through other recommendations.

4.4.1. Start with the Whole#

It is critical to work from the big picture to your metrics. Follow a process and be deliberate about creating your success measurement system.

Situation assessment. Utilizing metrics without a measurement system or situation assessment (see the Program Design Chapter) is like building a house without a blueprint. Start with a situation assessment and understand how you, your staff, team, and partners understand the current contextual situation of the system you’re working in and how your project interventions or business application will positively change that system. Do not just assume everyone gets what you’re going to do. For example, if you ask the ten people you work with to define restoration, you might get ten different responses. Memorializing the situation, debating the relationships, and sharing them reach that common understanding.
Results chains. Create results chains from your interventions to project outcomes. For more details on results chains, see the Program Development chapter and the Open Standards (link in resources section).
Metrics. Once results chains are established, add how you will measure intermediate and long-term outcomes.

Workshopping these steps, preferably with staff and partners, can get you to a cost-effective system sooner. Don’t worry if it seems to take a while at the outset. Socializing across staff and partners can seem to delay the process, but it will ultimately create longer-term solutions.

Tip

Invite existing or potential funders to a planning workshop. There’s no guarantee this will lead to funding, but you will certainly engage them, make your project more memorable, and tap into their deep knowledge regarding project success.

4.4.2. Utilize Existing Data Sources#

Review all your existing data sources, as well as free data sources, before you invest in any new data collection efforts. Organizations often collect relevant data that could be used for M&E, but staff responsible for monitoring and evaluation must be made aware of it. For example, if you are seeking to measure return on investment (ROI), your finance team might already have information you can use to crunch ROI figures. Also, investigate open-source data, such as government and citizen scientist databases, before you decide to go out and collect more information. For example, a community forestry initiative seeking to gauge the fire resistance of a stand it manages could search available USGS datasets to find previous fires that may have impacted its site.

4.4.3. Employ Participatory Approaches#

Engaging stakeholders in the M&E process enhances data validity and relevance and reduces data collection and analysis costs. Participatory approaches empower community members, beneficiaries, and partners to contribute their perspectives, insights, and local knowledge, enriching the M&E process. For example, a nonprofit organization conducting a program evaluation might organize focus group discussions, community meetings, or participatory workshops to gather feedback from program participants and community members, thereby reducing the reliance on expensive external consultants.

Participatory approaches may also employ apps to crowd-source data and engage key audiences in your mission. See the next section for suggestions.

4.4.4. Embrace Technology#

Technology offers cost-effective and time-saving data collection, analysis, and reporting solutions. Many open-source and low-cost software, mobile applications, and digital platforms can streamline M&E data collection, facilitate data analysis, and enhance data visualization [Russell, 2024]. For instance, a conservation organization might use a mobile data app such as Survey123, QField, or Kobo Toolbox to collect real-time forest health indicators, utilize GIS software such as ArcGIS Pro and QGIS to analyze, and use cloud-based platforms to store data, eliminating manual data entry and paper-based reporting. App-based data collection also allows multiple users to collect and input data.

ArcGIS Survey 123, from ESRI, is a good example of a mobile app for field data collection and mapping. It can be customized to collect various forest metrics such as tree species, diameter at breast height (DBH), and tree condition, and this information can be fed directly into ArcGIS software. It is easy to use and synchs effortlessly to your desktop or online GIS software. However, it does require a subscription, which can be expensive for some organizations.

Emerging low-cost, high-tech monitoring options

Traditional methods for monitoring forestry, ecology, and fuels can be expensive, time-consuming, and prone to errors when deployed at scale. Additionally, conventional methods can be slow to implement, limiting their usefulness in detecting new ecological changes, and they often do not provide data with enough precision or in a format that can be easily analyzed without extensive human systematization. Despite these limitations, many forest managers continue to use traditional methods because high-tech methods have historically been too costly to deploy. Fortunately, relevant technologies – including terrestrial laser scanning (TLS) and unmanned aerial vehicle (UAV)-borne light detection and ranging (LIDAR) systems – are rapidly becoming affordable options due to advancements in manufacturing processes, increased competition, and economies of scale.

USDA and USFWS researchers have successfully demonstrated using relatively low-cost (~25,000 USD), off-the-shelf TLS units like the Leica BLK360 for forest monitoring [Pokswinski et al., 2021]. Their methodology combines TLS calibrated using traditional, physically sampled forest conditions to simplify and accelerate the extraction of forestry, fuels, and ecological data from a single TLS point cloud. It streamlines data collection, reduces sampling bias, and generates easily processed data for improved modeling and decision-making.

Other researchers have developed a very low-cost system (~1,500 USD) that integrates DJI Livox MID40 laser scanner with an off-the-shelf UAV, and evaluated its capability in estimating both individual tree-level and plot-level forest inventory attributes [Hu et al., 2020]. They comprehensively compared their low-cost and four high-end UAV lidar systems equipped with high-end laser scanners to survey coniferous and broadleaved forest sites. The DJI Livox MID40 system performed comparably to the more expensive systems. Forest researchers and managers will struggle to resist as technology improves and becomes even cheaper.

4.4.5. Build Internal Capacity#

Consider training existing staff to build internal M&E capacity, reducing your reliance on costly external consultants and service providers. By equipping staff with the necessary skills and knowledge, organizations can conduct M&E activities in-house, saving costs and ensuring sustainability. A small non-profit organization might provide M&E training workshops for its staff, empowering them to design surveys, analyze data, and generate reports independently, enhancing organizational efficiency and effectiveness.

4.4.6. Foster Collaborative Partnerships#

Collaborating with other organizations, academic institutions, and government agencies can facilitate resource-sharing, knowledge exchange, and joint M&E initiatives. By pooling resources and expertise, organizations can achieve more comprehensive M&E outcomes at a lower cost. For example, a consortium of NGOs working on a regional conservation project might establish a joint M&E framework, share data, tools, and methodologies, and coordinate evaluation activities to avoid duplication and maximize efficiency.

4.4.7. Seek Creative Ways to Fund Monitoring#

Many funders don’t provide funding for monitoring, but they almost always require project reporting. Incorporating your measurement into reporting may help cover the costs. Educate the funders, too. For example, don’t let them require reporting on ‘acres and bucks’ and explain why those metrics are okay, but don’t tell the full picture about complex states such as forest health or resilience.

Going back to partnering with other organizations is critical here. For example, you could partner with a university on a research project. First, create a research agenda for your work to point researchers to what data and information you need (rather than them coming to you with pre-formed research). Academic researchers will want specific outcomes that might not connect with your data and outcomes needs, but they could help develop technology or collect some data you need.

4.5. Car Analogies#

Originally, we framed organizational monitoring approaches as the extremes of owning a Yugo vs. a Cadillac. This is a poor analogy since Yugos were cheap, broke down frequently, and some consider them the worst cars ever. On the opposite end of the spectrum, Cadillacs are expensive and beyond the purchasing power of many individuals.

When I was at university, starting my senior or fourth year, I owned a simple red Honda Civic hatchback. It was plain but incredibly dependable and gas efficient. I never had a single-engine problem with that car, and it always started, even during the dark, cold months of winter when some autos from the 1970s and 80s rebelled and would not turn over in frigid northern US temperatures.

civic

On the other hand, you may need something equally dependable, affordable, and expansive, such as the workhorse minivan. A Civic will get you reliably where you need to go, whereas a min-van will do that and allow you to bring along your kids, their friends, and all their stuff. I rented a Kia minivan once when the rental agency at the San Diego airport ran out of cars. That thing was a rocket. I had it going 90 mph several times (don’t tell CHP) while driving through the Cleveland National Forest. Vroom vroom!

What is the absolute bare-bones, dependable monitoring system that could be implemented (Civic) vs. a more expansive version (Minivan)? We suggest a couple of examples in the pulldowns below. Both examples are hypothetical stand-thinning projects with prescribed fire applied periodically after thinning for maintenance. The project assumption is that thinning will lead to increased forest health and reduced high-intensity wildfires. Sometimes, you need a dependable Honda Civic:

Honda Civic Monitoring Methods Checklist

Intervention is stand thinning to reduce wildfire risk followed by regular prescribed fire treatments.

Stand inventory by species and dbh every ten years using Kobo Toolbox and QGIS
Photo witness posts at reference and treatment sites
Total acres treated, total acres burned
Total dollars spent

And sometimes you need a Minivan:

minivan

Minivan Monitoring Methods Checklist