---
title: "Rain Data Validation"
author: "Hauke Sonnenberg"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Rain Data Validation}
  %\VignetteEncoding{UTF-8}
  %\VignetteEngine{knitr::rmarkdown}
editor_options: 
  chunk_output_type: console
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

```{r define_paths, include = FALSE}
is_ghactions <- identical(Sys.getenv("CI"), "true")

# Define Paths
PATHS <- kwb.utils::resolve(list(
  servername = Sys.getenv("SERVERNAME", unset = "my-server"),
  flusshygiene = "//<servername>/projekte$/SUW_Department/Projects/FLUSSHYGIENE",
  validation = "<flusshygiene>/Data-Work packages/Daten/Regen/validation",
  desktop = ifelse(kwb.utils::.OStype() %in% c("windows", "linux"), 
                   kwb.utils::desktop(),
                   "macos_desktopdir"),
  r_dev = "<desktop>/R_Development",
  dir_config = "<r_dev>/RScripts/Flusshygiene/config",
  dictonary_file = "<dir_config>/pathDictionary.txt",
  script_1 = "<r_dev>/RScripts/Flusshygiene/readBwbRainFiles.R",
  script_2 = "<r_dev>/RPackages/kwb.rain/inst/extdata/user_validation.R"
))

# Read further paths from a "path dictionary" file
paths_read <- if (!is_ghactions && file.exists(PATHS$dictonary_file)) {
  kwb.utils::resolve(kwb.utils::readDictionary(PATHS$dictonary_file))
} else {
  list(dir.data = "", dir.corr = "")  
}

paths_read$dir.data <- ifelse(
  dir.exists(paths_read$dir.data),
  paths_read$dir.data, 
  "<path_dir.data>"
)

paths_read$dir.corr <- ifelse(
  dir.exists(paths_read$dir.corr),
  paths_read$dir.corr, 
  "<path_dir.corr>"
)
```

This document explains the rain data validation procedure as we started to apply 
within the FLUSSHYGIENE project.

### Rain Data Files

The rain data provided by Berliner Wasserbetriebe (BWB) come in the form of 
Excel files (rain data files) that store for each five minute period the 
cumulative rain height measured by different rain gauges at the end of the 
period. The data are raw, i.e. they represent what was logged by the rain 
gauges. 

### Rain Correction Files

In order to calibrate the rain gauges BWB staff visits the rain gauges on a 
regular basis and applies a certain amount of water to them. This amount of 
water that appears as additional rain height in the data but does not represent 
actual rain needs to be excluded from the rain data. Therefore, BWB provides 
another set of Excel files (rain correction files) that contain for each day and 
each gauge the expected actual total rain height at that gauge and day.

### Rain Data Validation

The task to be performed during "rain data validation" is to 

- find the gauges and days at which the daily rain height calculated from the
rain data files (raw rain height) differ from the corresponding values in the 
rain correction files (actual rain height).

- "distribute" the difference between actual rain height and raw rain height 
that is given on a daily basis to the actual five minute time intervals in 
which the differences most probably occurred.

### Method

We follow a semi-automatic approach in which we use 


- the script ``r kwb.utils::windowsPath(PATHS$script_1)`` to read, reformat and merge rain 
data and correction data and to save the result to RData files and

- the script ``r kwb.utils::windowsPath(PATHS$script_2)`` to read rain data and correction 
data from the RData files and prepare csv files in which the time intervals 
that most probably contain rain heights that need correction are indicated. 

The first script reads 

- rain data files from ``r kwb.utils::windowsPath(paths_read$dir.data)`` and 

- correction files from ``r kwb.utils::windowsPath(paths_read$dir.corr)``.

The user can then check the CSV files created by the second script, edit them 
appropriately and save them under a different name (`userdiff_*.csv` in the
`userdiffs` folder). 

It is **important** to save them out of the folder `autodiffs` because the files 
within this folder are always overwritten by the second script.

After editing the userdiff-files, the script

``r PATHS$script_2``

can be used to apply the corrections defined by the user to the raw data and to 
store the resulting validated data.