Monitoring systems
Contents
Processing and data storage in DB
Information from meters is stored on the server through rsync on a copy of the measure files. This information has the following properties:
- may contains errors
- may be inconsistent: some minutes does not exist or be repeated
- contains raw values, there are only direct measures
- its access and use are tedious
For this reasons, some R scripts have been developed that perform the following features:
- clean errors and process them
- ensure consistency by creating a continuous time reference
- prepare the desired raw values to be stored and process new information from these raw values
- store this information in a MySQL DB
These scripts have been divided in two different families: create and update scripts.
create scripts
The objective of these scripts is to create the DB with all required information, ensuring the previous features. These scripts have the name create.XXXX.R where XXXX is the name of the facility to be stored. The process is divided in the following steps:
| Step 1 | Get the list of files to be processed |
|---|---|
| Step 2 | Get data from all files and store in a huge matrix with consistent information |
| Step 2.1 | Get data from file that can be no consistent and can contain errors |
| Step 2.2 | Clean the raw data solving consistency and errors, this action is performed by the get_data_from_meter function. After that, store this cleaned data into a huge matrix |
| Step 3 (Opt.) | The cleaned and consistent raw data are processed to get processed data (for example: not measured powers, self-consumption, temperature conversion, etc.) |
| Step 4 (Opt.) | The processed data are unify in a single matrix in order to be stored all together. |
| Step 5 | All information is stored on a DB: cleaned and consistent raw data and processed data. If there is previous information on the DB, it is removed. |
Step 3 and 4 are optionals, if there is not processed data they are not required. In general, there DB is create with these tables: one table per meter and a unique table for processed data or for each family of processed data. The meters are independent in order to isolate the measures. The processed table is unique for simplicity and because it usually requires information from several meters. However, the energy monitoring has a processed table for power and another for energy.
The R scripts can be launch with the following command:
Rscript create.XXXX.R
update scripts
The objective of these scripts is to update the DB with all required information, ensuring the previous features. These scripts have the name update.XXXX.R where XXXX is the name of the facility to be stored. The process is divided in the following steps:
| Step 1 | Get the measure file to be processed (today file) |
|---|---|
| Step 2 | Get stored information on the DB of the meter for today |
| Step 3 | Clean the raw data solving consistency and errors of the new information, this action is performed by the update_data_from_meter function. After that, store this cleaned data into a matrix |
| Step 4 | Update DB with the new information |
| Step 5 (Opt.) | Get current stored information from DB for processing |
| Step 6 (Opt.) | Create a consistent time reference for processed data. Check if there is information for all minutes to be processed |
| Step 7 (Opt.) | Process the new data |
| Step 8 (Opt.) | Unify the processed data in a single matrix and update the processed data. This operation depends on the structure of the processed data. |
Step 5,6,7, and 8 are optionals, if there is not processed data they are not required. The processed data has always the last minute where there were available data from all meters, this action ensure the consistency.
The R scripts can be launch with the following command:
Rscript update.XXXX.R
These scripts must be included in the cron of the user. They can be included with the following command:
crontab -l | { cat; echo "* * * * * Rscript /<ADDRESS>/update_data.XXXX.R"; } | crontab -
install scripts
In order to make easier the installation process, there are install bash scripts. They all called as install.XXXX.sh where XXXX is the name of the facility to be stored. These scripts launch the create.XXXX.R script and include the update.XXXX.R script in the user's cron.
