Often, model validation is considered just one of many technical elements of the modelling exercise. A model is developed, validated against measurements, and revised as appropriate before being used to fill in conditions where measurements are not available.
In the context of urban air pollution, however, validation has implications beyond the purely technical.
Urban dispersion models often form the basis of policy-related decisions, including on transport planning, building design, exposure assessment, public health and emergency response. If a model has not been validated against relevant evidence, then the decision making which relies on that model is likely to have more uncertainty than the model output implies.
This point is particularly relevant where models are deployed at street level in complex, built environments where small changes in local wind direction, building form or source location can significantly alter the fate of a pollutant.
The Implications for Urban Dispersion Research
Model validation does not simply boil down to a question of whether a model is “good” or “poor”. It is instead more useful in the case of urban dispersion research to ask whether a model is fit for the purpose for which it is intended.
A model might be excellent for screening purposes but poor near a junction, whereas another model might predict concentrations reasonably well overall but perform poorly for short-term peaks. An expensive, detailed computational model might be able to capture flow around a building in detail but could be highly sensitive to the quality of the input data, while a rapid-running model might be very good for use in emergency situations but only if the nature of the assumptions that underlie it are known.
Model validation helps define the applicability of a model, indicating which areas it covers well, where it may be less certain and what type of applications and decisions it is likely to be suitable for.
This was a key question for DAPPLE-HO’s work on short-range urban dispersion, where a series of experiments and evaluation of urban dispersion models was undertaken to assess the applicability of a range of models for emergency response and exposure assessment purposes.
Challenges of Modelling at Street Level
Urban dispersion modelling has a tough job to do: to replicate the behaviour of pollutant plumes in complex built environments where buildings disrupt airflow, streets channel plumes, junctions mix plumes, and emissions change over time. Local meteorology might differ considerably between roof level and street level.
Indeed, the same physical complexity that makes street-level air pollution so difficult to predict also makes validation difficult. The modelling is concerned not only with determining how much pollutant is present but also with understanding how the built environment can influence the way in which the pollutant plume is transported, diluted, trapped and redirected once it has entered the atmosphere.
Given the level of complexity, the difference between the true state of affairs and the results given by the model can be significant at the street level. A plume could be predicted as remaining in one street when it actually exits into another, leading to exposure predictions which do not accurately reflect what has actually happened.
Concentrations could be underpredicted at a critical location, such as a building frontage, bus stop or crossing, leading to erroneous assumptions about exposure levels. Short-term concentration peaks could be smoothed out, leading the model to under-represent the conditions actually experienced by those in the street.
Model validation cannot therefore be considered as just an optional add-on to urban dispersion modelling; it must be regarded as an essential part of assessing whether a model provides useful evidence for decision-making.
False Confidence
Model outputs can sometimes create the appearance of certainty. High-resolution maps, contour lines, and numerical values can create a sense of precision, yet visual intricacy is not indicative of accuracy.
The outputs of models may be spatially or temporally dense, but generated using inaccurate emission rates, simplistic architectural representations, inadequate meteorological datasets, or assumptions of turbulent flow. This lack of verification leaves users unsure whether they are observing a phenomenon being captured by the model, or an approximation that might be plausible but is not correct.
The risk this creates to policy is clear. A planning decision, an emergency preparedness decision, or a public health decision may be influenced by outputs that appear more accurate than they actually are.
The act of comparing how the model behaves and what the model predicts to known evidence reduces this risk. Validation does not eliminate all uncertainty but, by making more explicit the sources of that uncertainty, validation can help reduce it.
Validation can make use of both field data, meaning evidence obtained from measurements at full scale, and laboratory data, meaning evidence obtained in a scaled environment such as a wind tunnel. In practice, these two forms of data offer different strengths.
Field Evidence and Laboratory Evidence
A model will typically need to be compared to evidence from both the real world and a laboratory environment.
The strength of field data is that the environment it captures is a real environment, with real cities, real traffic emissions, real weather conditions, realistic background pollution, and human activity. The problem with field evidence is that, as the environment is so complex, a measurement taken today is unlikely to be replicated again tomorrow as the wind direction changes, traffic patterns alter, or any other variation in conditions occurs.
The strength of evidence from wind tunnel experiments is the repeatability of the results. A scaled physical model of the same geometry can be exposed to the same wind direction and same emission source and allow measurements and testing to take place over and over again.
This provides evidence on whether the urban dispersion model in question captures the physics relevant to a certain wind direction, a certain building form, a certain source location or any combination of those factors, in a more repeatable manner.
For this reason, the relative roles of field data and wind tunnel data are important in urban dispersion modelling: we require both the physical realism of field data and the control over variables that the wind tunnel offers in order to judge a model.
The reality is that no single source of evidence, either wind tunnel or field data, provides the evidence base required to fully validate a model. It is a combination of both that provides that validation.
Wind Tunnel Data as a Validation Tool
A further reason that wind tunnel experiments continue to be valuable within research into urban air pollution is that they provide a means to capture repeatable physical data. A model might be asked to reproduce a defined geometry, a defined wind direction, and a defined source location and, in doing so, provide a comparison with wind tunnel measured flow and concentration data.
That means the wind tunnel is a useful validation tool against which to assess how the model behaves in terms of processes like the recirculation of the flow and concentration patterns within a street canyon, the spreading of the plume, the separation of the flow around a building, or the concentration gradients close to the source region.
In these conditions, we might expect that a model that fails should not be used within a real city setting without caution. However, a model that performs well should also be checked against evidence from the field before it can be considered suitable for use.
The performance of the model within the wind tunnel suggests that, at the very least, a model is capturing some of the physical processes relevant within the urban environment. That is why controlled wind tunnel experiments remain important for evaluating urban air pollution models.
Tracer Experiments and Plume Pathways
Tracer experiments also provide another useful source of information against which to compare a model.
Here, a material can be released in known concentrations and the movement of that material then measured. This enables a clearer validation test than just comparing the model to air pollution measurements on its own, as it means that we are measuring the movement of pollution from a source we have defined more precisely.
Data from urban tracer experiments can reveal flows through street canyons, side streets, junctions and above roofs that otherwise would go undetected.
This is useful for model validation, as a model has to be able to show movement rather than just a concentration at a single point. If tracer evidence shows material moving into another street, then the model needs to be able to replicate that under the appropriate conditions.
This is useful for standard air quality assessments, but even more so when the model is used to understand short-term peaks or emergency releases.
Validation and Emergency Response
Emergency response is perhaps one of the best examples of where validation has policy implications. During an airborne release there may be limited time, incomplete information and a need for rapid decision making. The models may be used to estimate where the material may go, what areas may be impacted and how to prioritise monitoring.
In dense cities, planning for urban airborne releases depends on understanding short-range movement through streets, junctions, building edges and urban canyons, rather than a very broad brush approach to where the wind is blowing.
In these situations, it is not necessary for a model to be perfect, but the limitations do need to be clear. A fast model might be useful if responders know when it will perform well and when it will miss local effects. A detailed model might be valuable for planning and scenario testing but not very useful if it cannot be run quickly enough during an incident.
Validation helps us understand how to use a model before an incident occurs.
Sensor Networks and Testing Models
Sensor networks can also contribute to model validation, provided the data are interpreted with care.
More sensors do not necessarily mean better evidence; sensor placement, calibration, representativeness and uncertainty are all important. A sensor network, however, if designed well, can show spatial patterns and short-term changes that a single fixed monitoring point cannot.
Better urban pollution monitoring can provide data which a model can be tested against, helping to show whether the model is getting the timing and direction right and how it varies. It can also help show where a model might be missing local effects.
The best approach is not sensors as an alternative to a model, nor a model as an alternative to sensors. It is to have a feedback system in which observations can test model predictions, and a model can help explain the observations between the points at which sensors are measuring.
Matching the Model to the Decision
One of the key policy takeaways is that models need to match up with what they are being used for.
A transport planning study may need to know about longer-term changes in exposure along a corridor. A building design assessment may need to understand ventilation and how long pollutants are retained in a street canyon. An emergency response tool may need rapid estimates under uncertainty. A public health analysis may need a carefully considered approach to short and long-term exposure.
Different uses have different requirements and one model may not be appropriate for all of them.
A validation therefore asks:
- What decision are we trying to inform?
- What spatial and temporal scales are significant for that decision?
- Which input uncertainties have the greatest impact on the decision?
- Have you compared the model with any analogous field or laboratory measurements?
- Can the model capture peaks, transport paths and finer-scale variation, or does it just give a regional average?
- How should model uncertainty be reported to policy users?
These are not merely technical questions; they directly determine whether a model can be used to support decision-making.
Physical, Digital and Policy Evidence
Future advances in urban dispersion research will rely more heavily than ever on connections between physical modelling, digital modelling, field evidence and monitoring.
In particular, a combined approach to physical and digital modelling is important for validation. Digital models require physical and field evidence, physical models require comparison with real field data, sensor networks require interpretation, and policy users require clear reporting of uncertainties.
Validation is the process that ties all this together and gives scientific rigour to the modelling process. It turns model output from a “black box” into evidence that can be tested.
Implications for Urban Policy
Viewing model validation as a policy issue has important implications:
- Validation must occur as part of the early stages of any modelled policy. If you want to use the model to support policy, planning, exposure assessment, or emergency response, you need to include evidence that will validate your model as part of the research.
- Validation must be aligned to the purpose for which the model will be used. For example, if you validated the model only against broad-scale averages, you may not be able to use the model with confidence to support street-scale peaks or very short-term releases.
- Uncertainty associated with any given model output must be made clear. Policy users do not need a full technical explanation, but they do need to understand how the model will support their policy decision.
- Monitoring and modelling should also be designed together. Sensor network design and model design must both provide relevant insights into model behaviour rather than simply being used to generate data.
- Model validation needs to be seen as an issue of policy accountability. If your model informs public exposure, urban design or emergency planning, then you must allow the model to be scrutinised and validated.
A Question of Trust and Usefulness
Why does model validation matter? Simply because the outputs of models are used to inform policy. In the context of urban air pollution, those decisions might concern how we manage traffic or design our streets, where people walk, exposure assessment or emergency response planning.
A model of any process need not be perfect to be useful to policy, but it does need to be understood. Its strengths, weaknesses and uncertainties need to be tested against evidence and reported.
This is why DAPPLE’s original work remains relevant, as it highlighted the need to combine information from different sources, such as field campaigns, tracer studies, wind tunnels and numerical modelling, to produce the best evidence.
Model validation is an issue of policy value and purpose because it ensures modelled outputs not only look scientifically sound but are actually useful. In urban environments, this is critical. Better validation leads to better evidence, and evidence leads to better decisions. Better decisions lead to better environments to live, work, and move within.
