Monitoring systems

Processing and data storage in DB

Information from meters is stored on the server through rsync on a copy of the measure files. This information has the following properties:

• may contains errors
• may be inconsistent: some minutes does not exist or be repeated
• contains raw values, there are only direct measures
• its access and use are tedious

For this reasons, some R scripts have been developed that perform the following features:

• clean errors and process them
• ensure consistency by creating a continuous time reference
• prepare the desired raw values to be stored and process new information from these raw values
• store this information in a MySQL DB

These scripts have been divided in two different families: create and update scripts.

create scripts

The objective of these scripts is create the DB with all required information, ensuring the previous features. The process is divided in the following steps:

Step 1	Get the list of files to be processed
Step 2	Get data from all files and store in a huge matrix with consistent information
Step 2.1	Get data from file that can be no consistent and can contain errors
Step 2.2	Clean the raw data solving consistency and errors, after that, store this cleaned data into a huge matrix
Step 3	The cleaned and consistent raw data are processed to get processed data (for example: not measured powers, self-consumption, temperature conversion, etc.)

They remove the previous DB, get all