An integrated platform for smart energy management: The CC-SEM project

2.1 Project goals and motivation

The main goal of the project is to design a platform that allows the integration of fundamental concepts and tools for energy management in smart cities, using cloud computing, computational intelligence for big data analysis, and software for simulation and optimization of the energy generation and distribution. The aim is providing both users and administrators of the electrical grid a useful set of tools for intelligent planning and organization of the electricity consumption and generation nowadays and in future smart cities. From the point of view of the users, the project proposes the design and management of a smart home controller for electric devices applying IoT related software, and the application of Big Data processing techniques for the analysis of domestic energy consumption and smart planning. From the point of view of the electric grid administrators, novel tools are presented to monitor the state of the network and the overall quality of services, and the use of novel simulation tools is proposed to analyze and foresee the energy demand. This approach is planned to be adopted by electric market regulators in Argentina and Uruguay.

Energy optimization and planning is on the agenda of many countries. Only few solutions implement an easy-to-use platform to be used by both end-users and energy companies integrating hardware, software, and communications. Energy providers have few (or even no) knowledge about electricity utilization in homes. With the current electrical infrastructure, the provider cannot determine if the energy is well-used or wasted. Having a hardware infrastructure that allows obtaining useful information about utilization is the first stage in a global system to optimize energy at homes, with the main goals of reducing the costs of energy consumption and generation, and improve the quality of service offered to the users.

Another important motivation of the project is to conceive a generic set of tools to allow both users and administrators to extract useful information from the raw data measured by the home controller by applying computational intelligence/machine learning techniques. We propose basing the communications on the IoT paradigm to guarantee ubiquitous access to the system, everywhere, anytime, and using a wide range of communication devices (smartphones, tablets, web interface, other management systems, etc.). The applications that integrate the proposed system need to be conceived as a part of a global monitoring/planning system to be used in real-time in modern smart cities.

2.2 Methodology

The proposed methodology, in line with the project goals, is two-fold. On one hand, the methodology is based on missions that help the project members to consolidate a collaboration network. The institutions have been in contact in the past: research groups at UdelaR and Reims have collaborated in research activities related to distributed computing and cloud computing, and research groups at UdelaR and UBA have collaborated in research activities related to high performance/distributed computing and applications. However, CC-SEM is the first initiative to set a common project between the three institutions. On the other hand, the research subject is realistic and represents a real need, as observed in recent contacts with Academia, Industry, and social actors related to energy management and the real implementation of the smart cities paradigm.

Due to the interdisciplinary nature of the project, and the fact that several actors (e.g., users, companies, agencies) are interested in the project outcomes, all results achieved during the project are being rapidly made available to the community via the project website, public repositories, and on seminaries, meetings, and conferences.

4.1 Low cost energy consumption monitor and controller

A prototype for the smart metering system was designed and built according to the general specification from our project. The proposed system integrates three components: i) a specific module and protocol (Energy Efficiency, EFEN), which allows defining/storing user actions and preferences, and compute plannings; ii) the monitor/controller itself, and iii) an interface for communication with the home controllers, based on Khimo framework. These components are implemented in independent modules that allow monitoring, operating, and controlling home devices according to specific rules.

EFEN defines home devices, stores power consumption data, provides an interface for device control, and also computes ad-hoc planning taking into account user preferences. In EFEN, electric devices are grouped in homes, but larger aggregations are also supported: homes can be grouped in buildings and buildings can be grouped in neighborhoods. This categorization allows performing energy planning at different levels, according to the preferences of single users/community users, and/or the needs of electric companies. EFEN also provides an Application Programming Interface (API) meant to implement the integration of computational intelligence algorithms for big data analysis/pattern recognition and energy planning, to be designed in WP 4. EFEN also includes a feasibility check for defined agendas, a tool to simulate historical power consumption time series (useful for verification purposes), energy and cost evaluation, and user satisfaction estimation algorithms.

The controller, presented in Figure 2, is based on a Single Board Computer (Raspberry Pi, providing a flexible and portable solution) that controls other two modules: i) a power meter STPM01/10, integrated using the steval-ipe016v1 board, which performs the measurements using the SPI protocol, and ii) a relay for power supply control. The controller communicates with the central system (Khimo) via the Internet. The system is controlled by the Khimo module, which allows performing communications via the IoT paradigm [13] to guarantee ubiquitous access (everywhere, anytime), and using multiple communication devices (e.g., smartphones, tablets, web interfaces, etc.). Khimo allows remote monitoring and controlling of several devices in real-time.

A specific protocol (EFEN_PROTO) was developed to guarantee efficiency in the communications between Khimo and EFEN, and also to enable controlling devices. Using a bidirectional communication channel, enabling event subscription, and avoiding intermittent queries (i.e., polling) to each controlled device, EFEN_PROTO provides efficiency for gathering power consumption and other information from devices, and also to define actions according to the state of home appliances. Action grouping is applied to deliver messages of the same type. By grouping home appliances according to the actions to perform over them, EFEN_PROTO avoids redundancy, thus improving the communication efficiency.

A greedy algorithm was proposed as a first step to design computational intelligence methods for home energy planning. This greedy algorithm focuses on minimizing power consumption and maximizing user satisfaction, by taking local decisions to build a global agenda and considering the maximum power available is a hard restriction.

The main details about the design of the controller were published in [14] and the application of IoT-based information for designing simple heuristics for smart home energy planning is described in [15]. Further details can be found on the website https://www.fing.edu.uy/inco/grupos/cecal/hpc/EFEHO.

4.2 NBIoT suitability for smart grid metering

To assess the state of smart grids, smart meters are deployed to monitor the grid. Those devices inform magnitudes that can be used to derive information about the whole urban scenario. Thus, the supporting communication network plays a fundamental role to ensure collecting state information. While wired networks entail high economic costs, wireless networks are positioned as a competitive alternative [16]. In particular, the exponential growth of cellular wireless networks establishes a powerful infrastructure for new communication technologies. In recent years, a narrow-band radio technology (less than 200 kHz) has been developed: Narrow Band Internet of the Things (NB-IoT) [17]. It is designed to satisfy requirements of low-bitrate applications, with special emphasis in coverage enhancement, ultra-low power consumption, and massive terminal access.

Another characteristic of this technology is non-latency-sensitivity, despite this, high channel occupation scenarios occur, thus, increasing latency levels over tolerable thresholds. In particular, when a local or regional blackout occurs, a massive number of Outage Restoration and Management (ORM) messages are sent from the smart meters at the households to the energy provider company. Each message has a payload of 200 bit and a maximum latency of 20 s. Latter issue is critical for a Smart Grid platform, ergo any metering communication technology should handle it appropriately. In [18], an assessment of this problem was carried out. Here we extend previous analysis proposing two strategies for handling massive simultaneous ORM messages, guaranteeing that all of these messages will arrive correctly:

Our analysis assumes the following pessimistic scenario: a low signal-to-noise (SNR) ratio of −20dB, just one sub-carrier available for channel usage (single tone), frequency numerology of 3.75 kHz which sets resource unit (RU) time to 32 ms, and re-transmissions are not available (unacknowledged mode).

By taking into account the uplink sub-process in which the smart meter sends a transfer block of 200 bit to the BS we establish a maximum number of RUs that can be transmitted during the first 320 s after the failure occurred in Equation (1).

Assuming each household (HH) has a single smart meter, the maximum number of households that can transmit its ORM message during the latency interval is presented in Equation (2).

RUs _ORM is the number of RUs needed for an ORM message in the established pessimistic scenario. To obtain this value, we developed a lite NB-IoT simulator for the uplink layer based on the Matlab Toolkit module: Uplink Waveform Generation. The link-layer model is implemented according to the NB-IoT physical uplink shared channel (NPUSCH) standard ([19, 20]). Montecarlo simulations are performed over an AWGN channel to obtain block error rate (BLER) information for the different link-layer configurations. The outcome of the simulation is used to trace BLER curves and obtain optimal link settings for target BLERs. This software was previously used [11, 21] to asses strategies for decreasing radio resource consumption. A detailed description of the current version is also presented in [22]. The whole project source code can be found in [23]. Table 1 has an extract of the simulation results.

Table 2 present household densities (ρ _HH ) for different urban scenarios [18]. In NB-IoT, coverage radio is less than 15km, i.e a maximum footprint area of 706km². To guarantee that HH _max smart devices can send its ORM messages concurrently, the footprint can be decreased. Last column of Table 2 shows footprint radio (Fp _r ) for the different environments. The weakness of this strategy is that the number of BSs that should be installed to provide coverage to the original area significantly increases.

Utilizing M2M communication can mitigate latter issue. This strategy is based on (α-1) households sending their information, i.e. ORM messages, to a common neighbor, which in turn process them to generate an aggregated message, that is sent to the common destination.

In this analysis, we suppose that the message does not increase its size with respect to the original ORM message. On the other hand, the consensus time, i.e. the interval in which the smart meters send its messages to its neighbor and it performs the processing for aggregating those messages, is negligible. Both tasks take only a few milliseconds (see [24]), which is orders of magnitudes smaller than the maximum latency of the ORM message.

Equation (3) presents the function that determines BS footprint radio with respect to household density and the aggregation factor α.

Figure 3 and 4 were generated based on the Equation 3, and present different views of the magnitudes involved on it. As can be observed, the augmentation of household density decrease NB-IoT base station footprint radio, but it can be compensated with the M2M aggregation factor α. Particularly, in Figure 3 densest environment, footprint radio reaches 37m when no M2M aggregation is used, and rise up to 299m aggregating 64 devices.

When is not necessary to send HH _max ORM messages concurrently, M2M aggregation can be used to reduce radio resource usage within the maximum latency interval. Therefore, this could increment the number of available RUs which can be used for other user equipment applications. Equation (4) defines the amount of available resource units depending on the transmission conditions:

where TBS is the transfer block size, SNR is the signal-to-noise ratio and BLER is the block error rate. The upper bound of the magnitude presented in Equation (4) can be calculated, it is presented in inequality (5). Figure 5 shows how the upper limit of the available radio resources within the maximum latency interval varies for different number of OMRs and for different M2M aggregation factors.

In conclusion, we address a critical communication metering issue in a Smart Grid platform: to successfully deliver massive concurrent ORM messages according to latency constraints. An assessment of NB-IoT suitability as a communication technology for this matter was performed. We proposed two strategies for adapting NB-IoT capabilities to this latency problem: BS footprint adjustment and M2M aggregation. Our results suggest that NB-IoT is a suitable technology for smart metering over a Smart Grid context, in particular for the platform that is proposed in this work.

4.3 Characterization of domestic energy consumption

In the previous section, we presented our efforts to measure and collect data from a consumer, helping her to identify usage patterns and situations that can potentially reduce the energy consumption. Even though, the domestic energy consumption is not only a consumer problem but also affects the infrastructure on building and neighborhoods, especially in the case of old and overloaded power grid networks. With the massive expansion of smart meters, the opportunity to gather costumers data and use big data algorithms open the door to extract useful information at different levels, from the individual user to the entire grid itself.

In our project, one of the goals is to identify potential peaks that, combined with the consumption from other users, may lead to disturbances in the local distribution grid (in a building or the neighborhood). Hence, we try to predict situations that may stress the grid and trigger measures to avoid the overload, be them passive (warning the users) or active (automatic shutdown of devices). This is indeed a problem that affects some power networks such as those from Buenos Aires, where the fast development of residential areas was not followed by infrastructure investments. Some projects like “Que no se corte” [25] try to handle this problem voluntarily: through a phone application, users sign in and feed basic information such as their location and electrical devices. Later, the “Que no se corte” application would send messages to encourage users to change their usage patterns.

Crowdsourcing efforts such as “Que no se corte” are necessary but they often struggle with the limited information they can explore. Indeed, having access to information on the power infrastructure is not always easy (for example, the location and capacity of the electrical transformers). Also, the usage profile differs from one user to the other. For instance, in the literature we mostly found methodologies to estimate the electric load at the system level. As illustrated by [26, 27], the electrical consumption at the system scale often follows seasonal variations at macro and micro scale (seasons, weekdays, an hour of the day), and the aggregation of several customers profile produces a smooth profile with consistent patterns that favor the forecasting accuracy. Unlike the system-level load, the individual residential consumption depends on the daily routine and users lifestyle but also on other elements that are harder to predict. While it is easy to forecast the consumption of a programmable water heater, it is much harder to estimate the consumption of other devices that may be activated alone or at the same time (Does one always turn on the oven and the dishwasher at the same time?). Some external elements such as the air temperature and the external weather can help to improve the predictions, but they also have a limited impact and correlation, depending on the residential characteristics (solar exposure, gas or electric heating), the thermal inertia of the buildings and the psychological resiliency of the inhabitants.

Several datasets for energy consumption can be found in the literature (e.g., a complete list is presented on http://wiki.nilm.eu/datasets.html). We start analyzing the Individual household electric power consumption dataset [28] as this dataset covers more than three years of consumption of a house located at the south of Paris, France. It presents the overall consumption and the detail of specific sets of devices, with a resolution of one minute between measures.

Our first analysis involved the attempt to extract consumption patterns, like for example the profile for each weekday or for different periods of the day (dawn, morning, afternoon, night), which are reputed to present similar behaviors. Neither of the categories we tested led to conclusive results, as illustrated in Figure 6. Indeed, we see that two different days/periods have different profiles even if they belong to the same categories, making it hard to forecast the residential consumption. We also tried to correlate the consumption with the local weather, but the residence from this dataset seems to rely on other energy sources for heating and cooking (gas or oil), so the electrical consumption profile was driven mostly by less powerful devices that have a strong dependence on the user’s habits. While a few patterns pointed in [29] could be extracted, they have small importance in the overall consumption and do not help to forecast.

As specific seasonal patterns could not help the prediction of the residential consumption, our next approach was to use deep learning techniques such as Long-Short Term Memory (LSTM). LSTM is a type of recurrent neural network designed for sequence problems such as time-series analysis and forecast.

In our specific case, we aimed at training the model to predict the consumption of the residence based on a sequence of previous measurements. This is not the first time LSTM has been used to this purpose [26, 27, 30], comforting our choice. However, the difference is that we are not interested in developing specific models for the forecast. Instead, we focus on the accuracy of simple models that could be later integrated into smart-meters with low CPU requirements. Because of this choice, we implemented a simple LSTM network using the Keras library, training it with the same dataset from Hebrail and Berard. After a few experiments, we chose a pipeline composed of one LSTM layer and two dense layers, as depicted in Figure 7.

Hence, the network has a visible layer with one or several inputs depending on the length of the historical window we want to evaluate (for example, a 24hs windows with). It also includes a hidden layer with 100 LSTM blocks or neurons, a hidden dense layer with an output size of ten and a final dense layer that makes a single value prediction. The default sigmoid activation function is used for the LSTM block, and the last layer has a non-negative activation constraint (https://keras.io/constraints/) to prevent the algorithms to output negative values, which is unrealistic in the case of power consumption. The “inner” dense layer is not required but seems to produce smoother predictions.

The training was made with 30% of the dataset (what roughly corresponds to a year of measures). The network was trained for 150 epochs and a batch size of 1. The resulting trained model was able to give good predictions on the remainder of the dataset (RMSE=22), as illustrated in Figure 8.

One inconvenient of applying deep learning techniques to the consumption of each residential user is that we need sufficient data to train the model. Indeed, good training requires at least a few months of readings, which would delay the start of operations for new costumers. As a consequence, we decide to circumvent this drawback by applying an existing model (from another residence) and verifying its effectiveness. For this reason, we do not try to optimize the parameters or develop more elaborated LSTM networks, but instead, we focus on applying the trained LSTM model over an independent dataset (available at http://cosy.univ-reims.fr/ ~lsteffenel/documents/residential.zip), obtained from a real user thanks to the Linky smart meter [5].

Contrarily to the reference dataset, this residence fully relies on electricity for heating and cooking, which can raise different consumption profiles. Three different intervals were compared as input history for the LSTM model: one day, one week and four weeks (roughly a month), as illustrated in Figure 9. The analysis of these forecasts shows that using only 24hs of history brings insufficient input data, and the forecast tends towards the “persistence” of the previous state. Forecasts with a month of history are better but, on the contrary, tend to smooth the consumption and raise the expectation for the lower values. Finally, the forecasts made with a week of history seem to offer a better trade-off between accuracy and the history length.

While prediction errors still occur (mostly “false positive” forecasts, like for example on the mark 1-20 on the samples), these estimations are good enough to help a recommendation system or to help to detect potential overloads (summing up the expected consumption from several residences). As expected, the RMSE is still important (about 300) but at least we have a baseline model that can be quickly deployed. As soon as more measurements are collected, individual models can be created by incrementally training the model. Also, specific parameter optimizations can be performed to improve the accuracy.

The scalability and portability of the developed solution can be ensured by relying on basic ETL (Extraction, Transformation, Load) using MapReduce, while more advanced operations can be conducted with high-level tools (e.g., Pig or Apache Spark and Tensorflow/Keras). Storage and access to the data can be made using NoSQL databases, which are specially adapted to store data series from sensors and other data sources (e.g., power generation, historical consumption charts, weather forecast) [31].

Further developments can include energy disaggregation, also referred to as non-intrusive load monitoring (NILM). NILM is a technique to make inferences about the different individual loads of the system based on an aggregate energy signal, such as that coming from a whole-home power monitor. NILM is a full research domain with several challenges, such for example the identification of devices signatures even when they do not work always in a constant regime [Figure 10]. Indeed, most devices present acyclic consumption behaviors, like for example the different phases of a washing machine program. An extended library of NILM datasets and papers can be found at http://wiki.nilm.eu/datasets.html.

In the case of our project, we can focus on the identification of power-hungry devices, like heaters and air conditioning systems. Compared to other devices like TVs, lamps or refrigerators, these devices may induce peaks of consumption that may overload the distribution system. By detecting their activity we can send warnings to the users (using the “Que no se corte” application, for example) or even trigger controlled shutdowns thanks to smart power outlets connected to the metering system.

4.4 Renewable energy generation forecast

Although there are some options to produce energy from wind and other sources, photovoltaic (PV) systems are the most likely way to generate centralized or distributed renewable energy in an urban setting in a massive scale. The challenges for adopting higher shares of this type of energy are posed by its intermittence, inherited from solar radiation dependency on local weather, mainly the cloud cover and optical thickness. Aiming to forecast PV generation for a particular system in a specific location, both the resource and the system characteristics should be considered. The capacity of forecasting PV generation for different installations will help manage the electric system reliably and economically.

To this end, in this project, we explore and evaluate different combinations of two free and open-source tools, a Numerical Weather Prediction (NWP) system and a PV modeling library. The Weather Research and Forecasting (WRF) Model is a mesoscale NWP system designed for both atmospheric research and operational forecasting applications [33, 34]. pvlib-python is a community-supported tool that provides a set of functions and classes for simulating the performance of photovoltaic energy systems [35, 36]. pvlib-python was originally ported from the homonymous MATLAB toolbox developed at Sandia National Laboratories and implements many of the models and methods developed at the Labs. The pvlib-python library comes with some functions to retrieve weather forecast data from some particular web services, thus through minimal modifications, it was adapted to read data produced by our installation of the WRF model.

In this work, we present a comparison of the outcomings of two strategies using off-the-shelf configurations of the NWP model to provide radiation input to the PV modeling library. This was intended to set the grounds for future developments, evaluating each approach.

WRF-direct: In this first strategy, we directly use surface radiation direct and diffuse components provided by WRF. These components were affected by the aerosol optical depth (AOD) measured by the SAVER-Net Project [37] at the closest moment [36].

WRF-coverage: In this second case, we post-process WRF output through the NCEP Unified Post Processing System (UPP) [38] obtaining low, mid, and high-level cloud covers. Those cloud coverages are then composed to get a total cloud cover value, and processed through pvlib-python routines to get the surface global radiation [39] which is then decomposed into direct and diffuse components using the DISC model [40].

Figure 11 shows a comparison of the surface global irradiance (GHI) forecasted by each of the two strategies along nine days during December 2017, for a location in Buenos Aires, Argentina. GHI results are compared with measurements provided by the SAVER-Net Project. Minutely measurements were averaged over 12-minute periods to make them coincide with WRF output frequency. This period was selected as it contains days with a clear sky, partial cloudiness and full overcast conditions. NCEP Reanalysis data [41] was used as initial and boundary conditions for WRF. In this setting, WRF is acting as a dynamical interpolator of the reanalysis conditions. Looking at days with clear sky conditions (Dec 18, 20, 24 and 25) reveals that WRF-direct strategy better agrees with measurements than WRF-coverage, exposing the need to take into account the AOD effect. Conversely, days with partial cloudiness (Dec 21 and 23) or overcast (Dec 19) expose a WRF-direct tendency to overestimate radiation, while WRF-coverage gives lower values than, in any case, do not follow measurements much close. There are also days with intermittent cloudiness (Dec 22 and 26) which both strategies fail to detect.

A useful measure to evaluate model behavior is to compare aggregated daily energy. Figure 12 shows the total daily energy predicted by each strategy, and how they compare against measurements. In this case, the tendency is repeated with WRF-direct acting better in clear sky conditions, but always overestimating radiation.

To acquire some insight into models internal behavior, direct normal (DNI) and diffuse (DHI) irradiance components as obtained from both strategies can be analyzed. Results are shown in Figure 13. In both strategies, the diffuse component is increased over the direct component in days with high cloudiness. Strange behavior is observed in clear sky days near noon for the WRF-coverage DNI results, dent-shape limits the maximum value attained. This is compensated by an increase in the DHI component. This appears to be a spurious effect caused by the DISC model implementation. Comparison against DNI or DHI measurements and a deep analysis of the DISC model will be needed to asses which behavior better represents reality.

After global radiation is decomposed in direct and diffuse components, the pipeline to get generation data from weather prediction models for a particular PV system is as follow (see Figure 14]:

To evaluate the performance of the above-mentioned pipeline, power generation measurements from a 19.6kW PV plant property of the Environmental Protection Agency of Buenos Aires city were used as a comparison (https://www.sunnyportal.com/Templates/PublicPageOverview.aspx?page=cbf7bf62-4171-430d-ab0d-46854669a126&plant=e4f89244-d746-42e2-9f72-909d76da9d66). This plant can be regarded as a typical community distributed generation installation. The actual plant is composed of two 5kWp (the ‘p’ letter denotes peak power) inverters connected to 250Wp PV modules and one 8kWp inverter connected to 215Wp PV modules. The simulated PV system was defined inside pvlib-python as composed by one inverter (SMA America STP5000TL) and modules (Amerisolar AS-6P30-250W) arranged in two strings of 11 modules each, equivalent to the circuit of one of the 5kWp inverters of the actual plant. To generalize the results, power generation was normalized concerning peak plant power. The installation 15-minutes normalized power is shown in Figure 15. The outcomes of the forecasting pipeline with inputs provided by the two strategies are also shown therein. In this case, an excellent agreement is given for clear sky days with the WRF-direct pipeline, stressing the importance of the AOD correction. In cloudy days, the results resemble the differences observed in GHI.

As a summarizing measure of this section the daily normalized energy forecasted by each strategy pipeline is compared with the actual generation of the plant in Figure 16. It can be observed that both strategies tend to overestimate generation in cloudy days and underestimate generation in clear-sky days, with WRF-direct better forecasting clear-sky days generation with respect to WRF-coverage. In these cases, the error is less than 10%, but on cloudy days it raises considerably.

The two proposed strategies allow forecasting the generation of individual PV systems. Although figures show 12-min data for convenience, more frequent sampling can be obtained as the actual model time step is smaller, allowing to capture the dynamics of the possible generation. The smallest sampling step not producing spurious dynamics should be evaluated. For modeling multiple systems assigned to the same WRF grid point, for which no difference will be detected in the radiation forecast, equivalent systems could be defined. Results suggest that:

At the moment, development efforts are focused on model calibration and validation. Several configurations and submodel options could be chosen and a systematic error quantification study is underway.