diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 33a67203f..a6cf71304 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -48,6 +48,7 @@ jobs: image: mongo:4.2 ports: - 27017:27017 + strategy: matrix: node-version: @@ -74,6 +75,7 @@ jobs: image: mongo:4.2 ports: - 27017:27017 + steps: - name: Git checkout uses: actions/checkout@v2 diff --git a/doc/caching-strategy.md b/doc/caching-strategy.md new file mode 100644 index 000000000..112248aac --- /dev/null +++ b/doc/caching-strategy.md @@ -0,0 +1,48 @@ +# Configuration Data Caching Strategies + +The IoTAgent Library options to enforce several different data caching strategies to increase throughput. The actual +data caching strategy required in each use case will differ, and will always be a compromise between speed and data +latency. Several options are discussed below. + +## Memory Only + +if `deviceRegistry.type = memory` within the config, a transient memory-based device registry will be used to register +all the devices. This registry will be emptied whenever the process is restarted. Since all data is lost on exit there +is no concept of disaster recovery. **Memory only** should only be used for testing and in scenarios where all +provisioning can be added by a script on start-up. + +## MongoDB + +if `deviceRegistry.type = mongodb` within the config. a MongoDB database will be used to store all the device +information, so it will be persistent from one execution to the other. This offers a disaster recovery mechanism so that +when any IoT Agent goes down, data is not lost. Furthermore since the data is no longer held locally and is always +received from the database this allows multiple IoT Agent instances to access the same data in common. + +### MongoDB without Cache + +This is the default operation mode with Mongo-DB. Whenever a measure is received, provisioning data is requested from +the database. This may become a bottleneck in high availability systems. + +### MongoDB with in-memory Cache + +if `memCache.enabled = true` within the config this provides a transient memory-based cache in front of the mongo-DB +instance. It effectively combines the advantages of fast in-memory access with the reliability of a Mongo-DB database. + +```javascript +memCache = { + enabled: true, + deviceMax: 200, + deviceTTL: 60, + groupMax: 50, + groupTTL: 60, +}; +``` + +The memCache data is not shared across instances and therefore should be reserved to short term data storage. Multiple +IoT Agents would potential hold inconsistent provisioning data until the cache has expired. + +## Bypassing cache + +In some cases consistent provisioning data is more vital than throughput. When creating or updating a provisioned device +or service group adding a `cache` attribute with the value `true` will ensure that the data can be cached, otherwise it +is never placed into a cache and therefore always consistently received from the Mongo-DB instance. diff --git a/doc/development.md b/doc/development.md index 00cd34942..52f4997c4 100644 --- a/doc/development.md +++ b/doc/development.md @@ -48,6 +48,8 @@ Module mocking during testing can be done with [proxyquire](https://github.com/t To run tests, type ```bash +docker run --name mongodb --publish 27017:27017 --detach mongo:4.2 + npm test ``` diff --git a/doc/howto.md b/doc/howto.md index 13df1ccca..2662a0a37 100644 --- a/doc/howto.md +++ b/doc/howto.md @@ -395,11 +395,11 @@ function updateContextHandler(id, type, service, subservice, attributes, callbac } ``` -The updateContext handler deals with the modification requests that arrive at the North Port of the IoT Agent via `/v2/op/update`. It is -invoked once for each entity requested (note that a single request can contain multiple entity updates), with the same -parameters used in the queryContext handler. The only difference is the value of the attributes array, now containing a -list of attribute objects, each containing name, type and value. The handler must also make use of the callback to -return a list of updated attributes. +The updateContext handler deals with the modification requests that arrive at the North Port of the IoT Agent via +`/v2/op/update`. It is invoked once for each entity requested (note that a single request can contain multiple entity +updates), with the same parameters used in the queryContext handler. The only difference is the value of the attributes +array, now containing a list of attribute objects, each containing name, type and value. The handler must also make use +of the callback to return a list of updated attributes. For this handler we have used a helper function called `createQueryFromAttributes()`, that transforms the NGSI representation of the attributes to the UL type expected by the device: diff --git a/doc/installationguide.md b/doc/installationguide.md index 884620250..c1f758b83 100644 --- a/doc/installationguide.md +++ b/doc/installationguide.md @@ -160,7 +160,7 @@ used for the same purpose. For instance: ```javascript { - type: 'mongodb'; + type: "mongodb"; } ``` @@ -204,6 +204,21 @@ used for the same purpose. For instance: } ``` +- **memCache**: Whether to use a memory cache in front of Mongo-DB when using the `mongodb` **deviceRegistry** option + to reduce I/O. This memory cache will hold and serve a set of recently requested groups and devices (up to a given + maximum time-to-live) and return the cached response so long as the value is still within `TTL`. When enabled the + default values are to hold up to 200 devices and 160 groups in memory and retain values for up to 60 seconds. + +```javascript +{ + enabled: true, + deviceMax: 200, + deviceTTL: 60, + groupMax: 50, + groupTTL: 60 +} +``` + - **iotManager**: configures all the information needed to register the IoT Agent in the IoTManager. If this section is present, the IoTA will try to register to a IoTAM in the `host`, `port` and `path` indicated, with the information configured in the object. The IoTAgent URL that will be reported will be the `providedUrl` (described diff --git a/lib/commonConfig.js b/lib/commonConfig.js index 5dc23c76f..81de7a81b 100644 --- a/lib/commonConfig.js +++ b/lib/commonConfig.js @@ -129,6 +129,11 @@ function processEnvironmentVariables() { 'IOTA_AUTH_TOKEN_PATH', 'IOTA_AUTH_PERMANENT_TOKEN', 'IOTA_REGISTRY_TYPE', + 'IOTA_MEMCACHE_ENABLED', + 'IOTA_MEMCACHE_DEVICE_MAX', + 'IOTA_MEMCACHE_DEVICE_TTL', + 'IOTA_MEMCACHE_GROUP_MAX', + 'IOTA_MEMCACHE_GROUP_TTL', 'IOTA_LOG_LEVEL', 'IOTA_TIMESTAMP', 'IOTA_IOTAM_HOST', @@ -243,9 +248,9 @@ function processEnvironmentVariables() { config.contextBroker.jsonLdContext = process.env.IOTA_JSON_LD_CONTEXT.split(',').map((ctx) => ctx.trim()); } - if (Array.isArray(config.contextBroker.jsonLdContext) && config.contextBroker.jsonLdContext.length === 1){ + if (Array.isArray(config.contextBroker.jsonLdContext) && config.contextBroker.jsonLdContext.length === 1) { config.contextBroker.jsonLdContext = config.contextBroker.jsonLdContext[0]; - } + } config.contextBroker.fallbackTenant = process.env.IOTA_FALLBACK_TENANT || config.contextBroker.service || 'iotagent'; @@ -323,6 +328,32 @@ function processEnvironmentVariables() { config.deviceRegistry.type = process.env.IOTA_REGISTRY_TYPE; } + // In Memory Caching policy + config.memCache = config.memCache || {}; + if (process.env.IOTA_MEMCACHE_ENABLED) { + config.memCache.enabled = process.env.IOTA_MEMCACHE_ENABLED === 'true'; + } + const memCache = config.memCache; + if (memCache.enabled) { + memCache.deviceMax = memCache.deviceMax || 200; + memCache.deviceTTL = memCache.deviceTTL || 60; + memCache.groupMax = memCache.groupMax || 50; + memCache.groupTTL = memCache.groupTTL || 60; + + if (process.env.IOTA_MEMCACHE_DEVICE_MAX) { + memCache.deviceMax = process.env.IOTA_MEMCACHE_DEVICE_MAX; + } + if (process.env.IOTA_MEMCACHE_DEVICE_TTL) { + memCache.deviceTTL = process.env.IOTA_MEMCACHE_DEVICE_TTL; + } + if (process.env.IOTA_MEMCACHE_GROUP_MAX) { + memCache.groupMax = process.env.IOTA_MEMCACHE_GROUP_MAX; + } + if (process.env.IOTA_MEMCACHE_GROUP_TTL) { + memCache.groupTTL = process.env.IOTA_MEMCACHE_GROUP_TTL; + } + } + // Log Level configuration if (process.env.IOTA_LOG_LEVEL) { config.logLevel = process.env.IOTA_LOG_LEVEL; @@ -470,7 +501,9 @@ function processEnvironmentVariables() { if (process.env.IOTA_DEFAULT_ENTITY_NAME_CONJUNCTION) { config.defaultEntityNameConjunction = process.env.IOTA_DEFAULT_ENTITY_NAME_CONJUNCTION; } else { - config.defaultEntityNameConjunction = config.defaultEntityNameConjunction ? config.defaultEntityNameConjunction : ':'; + config.defaultEntityNameConjunction = config.defaultEntityNameConjunction + ? config.defaultEntityNameConjunction + : ':'; } } @@ -525,7 +558,6 @@ function ngsiVersion() { return 'unknown'; } - /** * It checks if a combination of typeInformation or common Config is LD * diff --git a/lib/fiware-iotagent-lib.js b/lib/fiware-iotagent-lib.js index cff647468..5a3f86eeb 100644 --- a/lib/fiware-iotagent-lib.js +++ b/lib/fiware-iotagent-lib.js @@ -157,6 +157,9 @@ function doActivate(newConfig, callback) { registry = require('./services/devices/deviceRegistryMongoDB'); groupRegistry = require('./services/groups/groupRegistryMongoDB'); commandRegistry = require('./services/commands/commandRegistryMongoDB'); + + registry.initialiseCaches(newConfig); + groupRegistry.initialiseCaches(newConfig); } else { logger.info(context, 'Falling back to Transient Memory registry for NGSI Library'); diff --git a/lib/model/Device.js b/lib/model/Device.js index 79b7f55fe..d55b120d4 100644 --- a/lib/model/Device.js +++ b/lib/model/Device.js @@ -53,7 +53,8 @@ const Device = new Schema({ autoprovision: Boolean, expressionLanguage: String, explicitAttrs: Group.ExplicitAttrsType, - ngsiVersion: String + ngsiVersion: String, + cache: Boolean }); function load(db) { diff --git a/lib/model/Group.js b/lib/model/Group.js index 8b158c4d7..f440fbc22 100644 --- a/lib/model/Group.js +++ b/lib/model/Group.js @@ -63,7 +63,8 @@ const Group = new Schema({ explicitAttrs: ExplicitAttrsType, defaultEntityNameConjunction: String, ngsiVersion: String, - entityNameExp: String + entityNameExp: String, + cache: Boolean }); function load(db) { diff --git a/lib/services/common/iotManagerService.js b/lib/services/common/iotManagerService.js index 8abd9c4f7..d303472dc 100644 --- a/lib/services/common/iotManagerService.js +++ b/lib/services/common/iotManagerService.js @@ -61,7 +61,8 @@ function register(callback) { expressionLanguage: service.expressionLanguage, defaultEntityNameConjunction: service.defaultEntityNameConjunction, ngsiVersion: service.ngsiVersion, - entityNameExp: service.entityNameExp + entityNameExp: service.entityNameExp, + cache: service.cache }; } diff --git a/lib/services/devices/deviceRegistryMongoDB.js b/lib/services/devices/deviceRegistryMongoDB.js index 4aa2d5d21..a8f9c66b7 100644 --- a/lib/services/devices/deviceRegistryMongoDB.js +++ b/lib/services/devices/deviceRegistryMongoDB.js @@ -30,6 +30,11 @@ const errors = require('../../errors'); const constants = require('../../constants'); const Device = require('../../model/Device'); const async = require('async'); +const cacheManager = require('cache-manager'); + +let memoryCache; +let cache; + let context = { op: 'IoTAgentNGSI.MongoDBDeviceRegister' }; @@ -56,8 +61,46 @@ const attributeList = [ 'timestamp', 'explicitAttrs', 'expressionLanguage', - 'ngsiVersion' + 'ngsiVersion', + 'cache' ]; +/** + * Sets up the in-memory cache for devices, should one be required. + */ +function initialiseCaches(config) { + function isCacheableValue(value) { + if (value !== null && value !== false && value !== undefined) { + return value.cache; + } + return false; + } + + function initialiseMemCache(memCache, isCacheableValue) { + return cacheManager.caching({ + store: 'memory', + max: memCache.deviceMax, + ttl: memCache.deviceTTL, + isCacheableValue + }); + } + + memoryCache = config.memCache.enabled ? initialiseMemCache(config.memCache, isCacheableValue) : undefined; + + if (memoryCache) { + cache = memoryCache; + } else { + cache = undefined; + } +} + +/** + * Empties the memory cache + */ +function clearCache() { + if (cache) { + cache.reset(); + } +} /** * Generates a handler for the save device operations. The handler will take the customary error and the saved device @@ -136,6 +179,7 @@ function removeDevice(id, service, subservice, callback) { callback(new errors.InternalDbError(error)); } else { logger.debug(context, 'Device [%s] successfully removed.', id); + clearCache(); callback(null); } @@ -223,7 +267,20 @@ function getDeviceById(id, service, subservice, callback) { }; context = fillService(context, queryParams); logger.debug(context, 'Looking for device with id [%s].', id); - findOneInMongoDB(queryParams, id, callback); + + if (cache) { + cache.wrap( + JSON.stringify(queryParams), + (cacheCallback) => { + findOneInMongoDB(queryParams, id, cacheCallback); + }, + (error, data) => { + callback(error, data); + } + ); + } else { + findOneInMongoDB(queryParams, id, callback); + } } /** @@ -289,6 +346,7 @@ function update(device, callback) { if (error) { callback(error); } else { + clearCache(); data.lazy = device.lazy; data.active = device.active; data.internalId = device.internalId; @@ -303,6 +361,7 @@ function update(device, callback) { data.registrationId = device.registrationId; data.explicitAttrs = device.explicitAttrs; data.ngsiVersion = device.ngsiVersion; + data.cache = device.cache; /* eslint-disable-next-line new-cap */ const deviceObj = new Device.model(data); @@ -370,3 +429,4 @@ exports.getSilently = getDevice; exports.getByName = alarmsInt(constants.MONGO_ALARM, getByName); exports.getByNameAndType = alarmsInt(constants.MONGO_ALARM, getByNameAndType); exports.clear = alarmsInt(constants.MONGO_ALARM, clear); +exports.initialiseCaches = initialiseCaches; diff --git a/lib/services/groups/groupRegistryMongoDB.js b/lib/services/groups/groupRegistryMongoDB.js index c80d29d0e..4910bc96f 100644 --- a/lib/services/groups/groupRegistryMongoDB.js +++ b/lib/services/groups/groupRegistryMongoDB.js @@ -32,6 +32,10 @@ const constants = require('../../constants'); const errors = require('../../errors'); const Group = require('../../model/Group'); const async = require('async'); +const cacheManager = require('cache-manager'); + +let memoryCache; +let cache; let context = { op: 'IoTAgentNGSI.MongoDBGroupRegister' }; @@ -58,8 +62,45 @@ const attributeList = [ 'expressionLanguage', 'defaultEntityNameConjunction', 'ngsiVersion', - 'entityNameExp' + 'cache' ]; +/** + * Sets up the in-memory cache for service groups, should one be required. + */ +function initialiseCaches(config) { + function isCacheableValue(value) { + if (value !== null && value !== false && value !== undefined) { + return value.cache; + } + return false; + } + + function initialiseMemCache(memCache, isCacheableValue) { + return cacheManager.caching({ + store: 'memory', + max: memCache.groupMax, + ttl: memCache.groupTTL, + isCacheableValue + }); + } + + memoryCache = config.memCache.enabled ? initialiseMemCache(config.memCache, isCacheableValue) : undefined; + + if (memoryCache) { + cache = memoryCache; + } else { + cache = undefined; + } +} + +/** + * Empties the memory cache + */ +function clearCache() { + if (cache) { + cache.reset(); + } +} /** * Generates a handler for the save device group operations. The handler will take the customary error and the saved @@ -248,7 +289,19 @@ function findBy(fields) { context = fillService(context, { service: 'n/a', subservice: 'n/a' }); logger.debug(context, 'Looking for group params %j with queryObj %j', fields, queryObj); - findOneInMongoDB(queryObj, fields, callback); + if (cache) { + cache.wrap( + JSON.stringify(queryObj), + (cacheCallback) => { + findOneInMongoDB(queryObj, fields, cacheCallback); + }, + (error, data) => { + callback(error, data); + } + ); + } else { + findOneInMongoDB(queryObj, fields, callback); + } }; } @@ -263,6 +316,7 @@ function update(id, body, callback) { group[key] = body[key]; } }); + clearCache(); /* eslint-disable-next-line new-cap */ const groupObj = new Group.model(group); groupObj.isNew = false; @@ -285,6 +339,7 @@ function remove(id, callback) { callback(new errors.InternalDbError(error)); } else { logger.debug(context, 'Device [%s] successfully removed.', id); + clearCache(); callback(null, deviceGroup); } }); @@ -318,3 +373,4 @@ exports.getTypeSilently = intoTrans(context, findBy(['type'])); exports.update = alarmsInt(constants.MONGO_ALARM + '_09', intoTrans(context, update)); exports.remove = alarmsInt(constants.MONGO_ALARM + '_10', intoTrans(context, remove)); exports.clear = alarmsInt(constants.MONGO_ALARM + '_11', intoTrans(context, clear)); +exports.initialiseCaches = initialiseCaches; diff --git a/lib/services/northBound/deviceProvisioningServer.js b/lib/services/northBound/deviceProvisioningServer.js index e60dc4984..d2d009bb2 100644 --- a/lib/services/northBound/deviceProvisioningServer.js +++ b/lib/services/northBound/deviceProvisioningServer.js @@ -64,7 +64,8 @@ const provisioningAPITranslation = { explicitAttrs: 'explicitAttrs', expressionLanguage: 'expressionLanguage', ngsiVersion: 'ngsiVersion', - entityNameExp: 'entityNameExp' + entityNameExp: 'entityNameExp', + cache: 'cache' }; /** @@ -143,7 +144,8 @@ function handleProvision(req, res, next) { autoprovision: body.autoprovision, explicitAttrs: body.explicitAttrs, expressionLanguage: body.expressionLanguage, - ngsiVersion: body.ngsiVersion + ngsiVersion: body.ngsiVersion, + cache: body.cache }); } @@ -220,7 +222,8 @@ function toProvisioningAPIFormat(device) { autoprovision: device.autoprovision, expressionLanguage: device.expressionLanguage, explicitAttrs: device.explicitAttrs, - ngsiVersion: device.ngsiVersion + ngsiVersion: device.ngsiVersion, + cache: device.cache }; } diff --git a/lib/templates/createDevice.json b/lib/templates/createDevice.json index d5199c076..9e9f4446e 100644 --- a/lib/templates/createDevice.json +++ b/lib/templates/createDevice.json @@ -47,6 +47,10 @@ "description": "NGSI Interface for this device", "type": "string" }, + "cache": { + "description": "Flag to whether to cache the details of this device in memory", + "type": "boolean" + }, "lazy": { "description": "list of lazy attributes of the devices", "type": "array", diff --git a/lib/templates/createDeviceLax.json b/lib/templates/createDeviceLax.json index 5e64b4bac..b0bf14359 100644 --- a/lib/templates/createDeviceLax.json +++ b/lib/templates/createDeviceLax.json @@ -47,6 +47,10 @@ "description": "NGSI Interface for this device", "type": "string" }, + "cache": { + "description": "Flag to whether to cache the details of this device in memory", + "type": "boolean" + }, "lazy": { "description": "list of lazy attributes of the devices", "type": "array", @@ -120,7 +124,7 @@ "properties": { "object_id": { "description": "ID of the attribute in the device", - "type": "string" + "type": "string" }, "type": { "description": "Type of the attribute in the target entity", diff --git a/lib/templates/deviceGroup.json b/lib/templates/deviceGroup.json index f5788d119..48340d175 100644 --- a/lib/templates/deviceGroup.json +++ b/lib/templates/deviceGroup.json @@ -45,6 +45,10 @@ "description": "NGSI Interface for this group of devices", "type": "string" }, + "cache": { + "description": "Flag to whether to cache the details of this group in memory", + "type": "boolean" + }, "attributes": { "description": "list of active attributes of the devices", "type": "array" @@ -81,4 +85,4 @@ } } } -} \ No newline at end of file +} diff --git a/lib/templates/updateDevice.json b/lib/templates/updateDevice.json index 0b3d901b4..f9a95ad88 100644 --- a/lib/templates/updateDevice.json +++ b/lib/templates/updateDevice.json @@ -203,6 +203,10 @@ "ngsiVersion": { "description": "NGSI Interface for this device", "type": "string" + }, + "cache": { + "description": "Flag to whether to cache the detals of this device in memory", + "type": "boolean" } } } diff --git a/mkdocs.yml b/mkdocs.yml index 94795c5a5..13e033dcf 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -18,10 +18,11 @@ pages: - 'Advanced Topics' : 'advanced-topics.md' - 'Library Functions': 'usermanual.md' - 'Measurement Transformation Expression Language': 'expressionLanguage.md' + - 'Caching Strategies': 'caching-strategy.md' - 'How to develop a new IoT Agent': 'howto.md' - 'North Port - NGSI Interactions': 'northboundinteractions.md' - 'Development Documentation': development.md - - 'Installation & Administration Manual': + - 'Installation & Administration Manual': - 'Installation Guide': 'installationguide.md' - 'Operations (logs & alarms)': 'operations.md' - + diff --git a/package.json b/package.json index a7dbcd4c4..5d310901c 100644 --- a/package.json +++ b/package.json @@ -44,6 +44,7 @@ "dependencies": { "async": "2.6.4", "body-parser": "~1.20.0", + "cache-manager": "^4.1.0", "express": "~4.18.1", "got": "~11.8.5", "jexl": "2.3.0", diff --git a/test/unit/mongodb/mongoDBUtils.js b/test/tools/mongoDBUtils.js similarity index 100% rename from test/unit/mongodb/mongoDBUtils.js rename to test/tools/mongoDBUtils.js diff --git a/test/unit/general/statistics-persistence_test.js b/test/unit/general/statistics-persistence_test.js index f9ca24dfe..89565bf2b 100644 --- a/test/unit/general/statistics-persistence_test.js +++ b/test/unit/general/statistics-persistence_test.js @@ -28,7 +28,7 @@ const commonConfig = require('../../../lib/commonConfig'); const iotAgentLib = require('../../../lib/fiware-iotagent-lib'); const should = require('should'); const async = require('async'); -const mongoUtils = require('../mongodb/mongoDBUtils'); +const mongoUtils = require('../../tools/mongoDBUtils'); const iotAgentConfig = { logLevel: 'FATAL', contextBroker: { diff --git a/test/unit/mongodb/mongodb-cache-test.js b/test/unit/mongodb/mongodb-cache-test.js new file mode 100644 index 000000000..e9b7aa7b7 --- /dev/null +++ b/test/unit/mongodb/mongodb-cache-test.js @@ -0,0 +1,396 @@ +/* + * Copyright 2016 Telefonica Investigación y Desarrollo, S.A.U + * + * This file is part of fiware-iotagent-lib + * + * fiware-iotagent-lib is free software: you can redistribute it and/or + * modify it under the terms of the GNU Affero General Public License as + * published by the Free Software Foundation, either version 3 of the License, + * or (at your option) any later version. + * + * fiware-iotagent-lib is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. + * See the GNU Affero General Public License for more details. + * + * You should have received a copy of the GNU Affero General Public + * License along with fiware-iotagent-lib. + * If not, seehttp://www.gnu.org/licenses/. + * + * For those usages not covered by the GNU Affero General Public License + * please contact with::[contacto@tid.es] + * + * Modified by: Daniel Calvo - ATOS Research & Innovation + */ + +/* eslint-disable no-unused-vars */ + +const iotAgentLib = require('../../../lib/fiware-iotagent-lib'); +const utils = require('../../tools/utils'); +const should = require('should'); +const logger = require('logops'); +const nock = require('nock'); +const mongoUtils = require('../../tools/mongoDBUtils'); +const request = utils.request; + +let contextBrokerMock; +let statusAttributeMock; +const iotAgentConfig = { + contextBroker: { + host: '192.168.1.1', + port: '1026', + ngsiVersion: 'v2' + }, + server: { + port: 4041 + }, + types: { + Light: { + commands: [], + lazy: [ + { + name: 'temperature', + type: 'centigrades' + } + ], + active: [ + { + name: 'pressure', + type: 'Hgmm' + } + ] + }, + Termometer: { + commands: [], + lazy: [ + { + name: 'temp', + type: 'kelvin' + } + ], + active: [] + }, + Motion: { + commands: [], + lazy: [ + { + name: 'moving', + type: 'Boolean' + } + ], + staticAttributes: [ + { + name: 'location', + type: 'Vector', + value: '(123,523)' + } + ], + active: [] + }, + Robot: { + commands: [ + { + name: 'position', + type: 'Array' + } + ], + lazy: [], + staticAttributes: [], + active: [] + } + }, + deviceRegistry: { + type: 'mongodb' + }, + + memCache: { + enabled: true, + deviceSize: 1000, + deviceTTL: 10, + groupSize: 100, + groupTTL: 10 + }, + + mongodb: { + host: 'localhost', + port: '27017', + db: 'iotagent' + }, + service: 'smartgondor', + subservice: 'gardens', + providerUrl: 'http://smartgondor.com', + pollingExpiration: 200, + pollingDaemonFrequency: 20 +}; +const device3 = { + id: 'cachedDevice', + type: 'Robot', + service: 'smartgondor', + subservice: 'gardens', + polling: true, + cache: true +}; + +describe('Mongo-DB in-memory cache ', function () { + beforeEach(function (done) { + logger.setLevel('FATAL'); + + nock.cleanAll(); + + contextBrokerMock = nock('http://192.168.1.1:1026') + .matchHeader('fiware-service', 'smartgondor') + .matchHeader('fiware-servicepath', 'gardens') + .post('/v2/registrations') + .reply(201, null, { Location: '/v2/registrations/6319a7f5254b05844116584d' }); + + contextBrokerMock + .matchHeader('fiware-service', 'smartgondor') + .matchHeader('fiware-servicepath', 'gardens') + .post('/v2/entities?options=upsert') + .reply(204); + + iotAgentLib.activate(iotAgentConfig, done); + }); + + afterEach(function (done) { + delete device3.registrationId; + iotAgentLib.clearAll(function () { + iotAgentLib.deactivate(function () { + mongoUtils.cleanDbs(function () { + nock.cleanAll(); + iotAgentLib.setDataUpdateHandler(); + iotAgentLib.setCommandHandler(); + done(); + }); + }); + }); + }); + + describe('When caching is enabled and a command update arrives to the IoT Agent for a device with polling', function () { + const options = { + url: 'http://localhost:' + iotAgentConfig.server.port + '/v2/op/update', + method: 'POST', + json: { + actionType: 'update', + entities: [ + { + id: 'Robot:cachedDevice', + type: 'Robot', + position: { + type: 'Array', + value: '[28, -104, 23]' + } + } + ] + }, + headers: { + 'fiware-service': 'smartgondor', + 'fiware-servicepath': 'gardens' + } + }; + + beforeEach(function (done) { + statusAttributeMock = nock('http://192.168.1.1:1026') + .matchHeader('fiware-service', 'smartgondor') + .matchHeader('fiware-servicepath', 'gardens') + .patch( + '/v2/entities/Robot:cachedDevice/attrs?type=Robot', + utils.readExampleFile('./test/unit/ngsiv2/examples/contextRequests/updateContextCommandStatus.json') + ) + .reply(204); + + iotAgentLib.register(device3, function (error) { + done(); + }); + }); + + it('should not call the client handler', function (done) { + let handlerCalled = false; + + iotAgentLib.setCommandHandler(function (id, type, service, subservice, attributes, callback) { + handlerCalled = true; + callback(null, { + id, + type, + attributes: [ + { + name: 'position', + type: 'Array', + value: '[28, -104, 23]' + } + ] + }); + }); + + request(options, function (error, response, body) { + should.not.exist(error); + handlerCalled.should.equal(false); + done(); + }); + }); + it('should create the attribute with the "_status" prefix in the Context Broker', function (done) { + iotAgentLib.setCommandHandler(function (id, type, service, subservice, attributes, callback) { + callback(null); + }); + + request(options, function (error, response, body) { + should.not.exist(error); + statusAttributeMock.done(); + done(); + }); + }); + it('should store the commands in the queue', function (done) { + iotAgentLib.setCommandHandler(function (id, type, service, subservice, attributes, callback) { + callback(null); + }); + + request(options, function (error, response, body) { + iotAgentLib.commandQueue('smartgondor', 'gardens', 'cachedDevice', function (error, listCommands) { + should.not.exist(error); + listCommands.count.should.equal(1); + listCommands.commands[0].name.should.equal('position'); + listCommands.commands[0].type.should.equal('Array'); + listCommands.commands[0].value.should.equal('[28, -104, 23]'); + done(); + }); + }); + }); + }); + + describe('When caching is enabled and a command arrives with multiple values in the value field', function () { + const options = { + url: 'http://localhost:' + iotAgentConfig.server.port + '/v2/op/update', + method: 'POST', + json: { + actionType: 'update', + entities: [ + { + id: 'Robot:cachedDevice', + type: 'Robot', + position: { + type: 'Array', + value: { + attr1: 12, + attr2: 24 + } + } + } + ] + }, + headers: { + 'fiware-service': 'smartgondor', + 'fiware-servicepath': 'gardens' + } + }; + + beforeEach(function (done) { + statusAttributeMock = nock('http://192.168.1.1:1026') + .matchHeader('fiware-service', 'smartgondor') + .matchHeader('fiware-servicepath', 'gardens') + .patch( + '/v2/entities/Robot:cachedDevice/attrs?type=Robot', + utils.readExampleFile('./test/unit/ngsiv2/examples/contextRequests/updateContextCommandStatus.json') + ) + .reply(204); + + iotAgentLib.register(device3, function (error) { + done(); + }); + }); + + it('should return a 200 OK both in HTTP and in the status code', function (done) { + iotAgentLib.setCommandHandler(function (id, type, service, subservice, attributes, callback) { + callback(null); + }); + + request(options, function (error, response, body) { + should.not.exist(error); + + response.statusCode.should.equal(204); + + done(); + }); + }); + }); + + describe('When caching is enabled and a polling command expires', function () { + const options = { + url: 'http://localhost:' + iotAgentConfig.server.port + '/v2/op/update', + method: 'POST', + json: { + actionType: 'update', + entities: [ + { + id: 'Robot:cachedDevice', + type: 'Robot', + position: { + type: 'Array', + value: '[28, -104, 23]' + } + } + ] + }, + headers: { + 'fiware-service': 'smartgondor', + 'fiware-servicepath': 'gardens' + } + }; + + beforeEach(function (done) { + statusAttributeMock = nock('http://192.168.1.1:1026') + .matchHeader('fiware-service', 'smartgondor') + .matchHeader('fiware-servicepath', 'gardens') + .post( + '/v2/entities/Robot:cachedDevice/attrs?type=Robot', + utils.readExampleFile('./test/unit/ngsiv2/examples/contextRequests/updateContextCommandStatus.json') + ) + .reply(204); + + statusAttributeMock = nock('http://192.168.1.1:1026') + .matchHeader('fiware-service', 'smartgondor') + .matchHeader('fiware-servicepath', 'gardens') + .patch( + '/v2/entities/Robot:cachedDevice/attrs?type=Robot', + utils.readExampleFile( + './test/unit//ngsiv2/examples/contextRequests/updateContextCommandExpired.json' + ) + ) + .reply(204); + + iotAgentLib.register(device3, function (error) { + done(); + }); + }); + + it('should remove it from the queue', function (done) { + iotAgentLib.setCommandHandler(function (id, type, service, subservice, attributes, callback) { + callback(null); + }); + + request(options, function (error, response, body) { + setTimeout(function () { + iotAgentLib.commandQueue('smartgondor', 'gardens', 'cachedDevice', function (error, listCommands) { + should.not.exist(error); + listCommands.count.should.equal(0); + done(); + }); + }, 300); + }); + }); + + it('should mark it as ERROR in the Context Broker', function (done) { + iotAgentLib.setCommandHandler(function (id, type, service, subservice, attributes, callback) { + callback(null); + }); + + request(options, function (error, response, body) { + setTimeout(function () { + iotAgentLib.commandQueue('smartgondor', 'gardens', 'cachedDevice', function (error, listCommands) { + statusAttributeMock.done(); + done(); + }); + }, 300); + }); + }); + }); +}); diff --git a/test/unit/mongodb/mongodb-group-registry-test.js b/test/unit/mongodb/mongodb-group-registry-test.js index 300ec42db..27c7668c5 100644 --- a/test/unit/mongodb/mongodb-group-registry-test.js +++ b/test/unit/mongodb/mongodb-group-registry-test.js @@ -57,7 +57,7 @@ const iotAgentConfig = { deviceRegistrationDuration: 'P1M' }; const mongo = require('mongodb').MongoClient; -const mongoUtils = require('./mongoDBUtils'); +const mongoUtils = require('../../tools/mongoDBUtils'); const optionsCreation = { url: 'http://localhost:4041/iot/services', method: 'POST', diff --git a/test/unit/mongodb/mongodb-registry-test.js b/test/unit/mongodb/mongodb-registry-test.js index 5d53dfe37..2f381cadb 100644 --- a/test/unit/mongodb/mongodb-registry-test.js +++ b/test/unit/mongodb/mongodb-registry-test.js @@ -30,7 +30,7 @@ const logger = require('logops'); const mongo = require('mongodb').MongoClient; const nock = require('nock'); const async = require('async'); -const mongoUtils = require('./mongoDBUtils'); +const mongoUtils = require('../../tools/mongoDBUtils'); let contextBrokerMock; const iotAgentConfig = { contextBroker: { diff --git a/test/unit/ngsi-ld/lazyAndCommands/active-devices-attribute-update-test.js b/test/unit/ngsi-ld/lazyAndCommands/active-devices-attribute-update-test.js index 0525a5fe6..5462e8e27 100644 --- a/test/unit/ngsi-ld/lazyAndCommands/active-devices-attribute-update-test.js +++ b/test/unit/ngsi-ld/lazyAndCommands/active-devices-attribute-update-test.js @@ -31,7 +31,7 @@ const request = utils.request; const should = require('should'); const logger = require('logops'); const nock = require('nock'); -const mongoUtils = require('../../mongodb/mongoDBUtils'); +const mongoUtils = require('../../../tools/mongoDBUtils'); let contextBrokerMock; const iotAgentConfig = { diff --git a/test/unit/ngsi-ld/lazyAndCommands/command-test.js b/test/unit/ngsi-ld/lazyAndCommands/command-test.js index 7656b8d92..9835b688f 100644 --- a/test/unit/ngsi-ld/lazyAndCommands/command-test.js +++ b/test/unit/ngsi-ld/lazyAndCommands/command-test.js @@ -31,7 +31,7 @@ const request = utils.request; const should = require('should'); const logger = require('logops'); const nock = require('nock'); -const mongoUtils = require('../../mongodb/mongoDBUtils'); +const mongoUtils = require('../../../tools/mongoDBUtils'); const timekeeper = require('timekeeper'); let contextBrokerMock; diff --git a/test/unit/ngsi-ld/lazyAndCommands/lazy-devices-test.js b/test/unit/ngsi-ld/lazyAndCommands/lazy-devices-test.js index 365107d56..396433fd4 100644 --- a/test/unit/ngsi-ld/lazyAndCommands/lazy-devices-test.js +++ b/test/unit/ngsi-ld/lazyAndCommands/lazy-devices-test.js @@ -33,9 +33,9 @@ const apply = async.apply; const should = require('should'); const logger = require('logops'); const nock = require('nock'); -const mongoUtils = require('../../mongodb/mongoDBUtils'); - +const mongoUtils = require('../../../tools/mongoDBUtils'); const timekeeper = require('timekeeper'); + let contextBrokerMock; const iotAgentConfig = { contextBroker: { diff --git a/test/unit/ngsi-ld/lazyAndCommands/polling-commands-test.js b/test/unit/ngsi-ld/lazyAndCommands/polling-commands-test.js index 563e8b415..b96ad7cf9 100644 --- a/test/unit/ngsi-ld/lazyAndCommands/polling-commands-test.js +++ b/test/unit/ngsi-ld/lazyAndCommands/polling-commands-test.js @@ -31,7 +31,7 @@ const request = utils.request; const should = require('should'); const logger = require('logops'); const nock = require('nock'); -const mongoUtils = require('../../mongodb/mongoDBUtils'); +const mongoUtils = require('../../../tools/mongoDBUtils'); let contextBrokerMock; let statusAttributeMock; diff --git a/test/unit/ngsi-mixed/provisioning/ngsi-versioning-test.js b/test/unit/ngsi-mixed/provisioning/ngsi-versioning-test.js index 8e50df2b7..5f73cee71 100644 --- a/test/unit/ngsi-mixed/provisioning/ngsi-versioning-test.js +++ b/test/unit/ngsi-mixed/provisioning/ngsi-versioning-test.js @@ -57,7 +57,7 @@ const iotAgentConfig = { deviceRegistrationDuration: 'P1M' }; const mongo = require('mongodb').MongoClient; -const mongoUtils = require('../../mongodb/mongoDBUtils'); +const mongoUtils = require('../../../tools/mongoDBUtils'); const optionsCreationDefault = { url: 'http://localhost:4041/iot/services', method: 'POST', diff --git a/test/unit/ngsiv2/general/startup-test.js b/test/unit/ngsiv2/general/startup-test.js index 9b90dd372..ebc511e87 100644 --- a/test/unit/ngsiv2/general/startup-test.js +++ b/test/unit/ngsiv2/general/startup-test.js @@ -146,6 +146,39 @@ describe('NGSI-v2 - Startup tests', function () { }); }); + describe('When the IoT Agent is started with memcache environment variables', function () { + beforeEach(function () { + process.env.IOTA_MEMCACHE_ENABLED = 'true'; + process.env.IOTA_MEMCACHE_DEVICE_MAX = 9990; + process.env.IOTA_MEMCACHE_DEVICE_TTL = 99; + process.env.IOTA_MEMCACHE_GROUP_MAX = 90; + process.env.IOTA_MEMCACHE_GROUP_TTL = 9; + }); + + afterEach(function () { + delete process.env.IOTA_MEMCACHE_ENABLED; + delete process.env.IOTA_MEMCACHE_DEVICE_MAX; + delete process.env.IOTA_MEMCACHE_DEVICE_TTL; + delete process.env.IOTA_MEMCACHE_GROUP_MAX; + delete process.env.IOTA_MEMCACHE_GROUP_TTL; + }); + + afterEach(function (done) { + iotAgentLib.deactivate(done); + }); + + it('should load the correct configuration parameters', function (done) { + iotAgentLib.activate(iotAgentConfig, function (error) { + config.getConfig().memCache.enabled.should.equal(true); + config.getConfig().memCache.deviceMax.should.equal('9990'); + config.getConfig().memCache.deviceTTL.should.equal('99'); + config.getConfig().memCache.groupMax.should.equal('90'); + config.getConfig().memCache.groupTTL.should.equal('9'); + done(); + }); + }); + }); + describe('When the IoT Agent is started with mongodb params', function () { beforeEach(function () { process.env.IOTA_MONGO_HOST = 'mongohost'; diff --git a/test/unit/ngsiv2/lazyAndCommands/active-devices-attribute-update-test.js b/test/unit/ngsiv2/lazyAndCommands/active-devices-attribute-update-test.js index 26939172c..a8c6d7cc9 100644 --- a/test/unit/ngsiv2/lazyAndCommands/active-devices-attribute-update-test.js +++ b/test/unit/ngsiv2/lazyAndCommands/active-devices-attribute-update-test.js @@ -31,7 +31,7 @@ const request = utils.request; const should = require('should'); const logger = require('logops'); const nock = require('nock'); -const mongoUtils = require('../../mongodb/mongoDBUtils'); +const mongoUtils = require('../../../tools/mongoDBUtils'); let contextBrokerMock; const iotAgentConfig = { diff --git a/test/unit/ngsiv2/lazyAndCommands/command-test.js b/test/unit/ngsiv2/lazyAndCommands/command-test.js index 12054b639..b7546b5d6 100644 --- a/test/unit/ngsiv2/lazyAndCommands/command-test.js +++ b/test/unit/ngsiv2/lazyAndCommands/command-test.js @@ -31,7 +31,7 @@ const request = utils.request; const should = require('should'); const logger = require('logops'); const nock = require('nock'); -const mongoUtils = require('../../mongodb/mongoDBUtils'); +const mongoUtils = require('../../../tools/mongoDBUtils'); const timekeeper = require('timekeeper'); let contextBrokerMock; diff --git a/test/unit/ngsiv2/lazyAndCommands/lazy-devices-test.js b/test/unit/ngsiv2/lazyAndCommands/lazy-devices-test.js index aca11f0bd..2e756e3bc 100644 --- a/test/unit/ngsiv2/lazyAndCommands/lazy-devices-test.js +++ b/test/unit/ngsiv2/lazyAndCommands/lazy-devices-test.js @@ -33,9 +33,9 @@ const apply = async.apply; const should = require('should'); const logger = require('logops'); const nock = require('nock'); -const mongoUtils = require('../../mongodb/mongoDBUtils'); - +const mongoUtils = require('../../../tools/mongoDBUtils'); const timekeeper = require('timekeeper'); + let contextBrokerMock; const iotAgentConfig = { logLevel: 'FATAL', diff --git a/test/unit/ngsiv2/lazyAndCommands/polling-commands-test.js b/test/unit/ngsiv2/lazyAndCommands/polling-commands-test.js index 83cb52ce8..aa31b478a 100644 --- a/test/unit/ngsiv2/lazyAndCommands/polling-commands-test.js +++ b/test/unit/ngsiv2/lazyAndCommands/polling-commands-test.js @@ -31,7 +31,7 @@ const request = utils.request; const should = require('should'); const logger = require('logops'); const nock = require('nock'); -const mongoUtils = require('../../mongodb/mongoDBUtils'); +const mongoUtils = require('../../../tools/mongoDBUtils'); let contextBrokerMock; let statusAttributeMock;