Spectrino:TinyML Arduino e soluções sem toque baseadas em IoT

• Arduino 33 BLE sense - Used to process the data gathered, classifies the data processes, sends the command according to the logic fed.
• MG959 / MG995 Servo - Heavy duty servo( An external power supply may be applied) - To open and close the gates as per microcontroller command.
• Arducam Mini 2mp plus - Continuous Raw data image accumulation from source.
• Adafruit lithium ion charger - Used to deliver charge through the lithium battery
• Lithium ion Battery - power source
• ESP8266 - 01 - Used for sending data to the ThingSpeak dashboard via WiFi network.
• MAX7219 4 in 1 display - Used for displaying the wait time on the display screen.

#include 

#include "main_functions.h"

#include "detection_responder.h"
#include "image_provider.h"
#include "model_settings.h"
#include "person_detect_model_data.h"
#include "tensorflow/lite/micro/kernels/micro_ops.h"
#include "tensorflow/lite/micro/micro_error_reporter.h"
#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/micro/micro_mutable_op_resolver.h"
#include "tensorflow/lite/schema/schema_generated.h"
#include "tensorflow/lite/version.h" 

void loop() {
 // Get image from provider.
 if (kTfLiteOk !=GetImage(error_reporter, kNumCols, kNumRows, kNumChannels,
 input->data.uint8)) {
 TF_LITE_REPORT_ERROR(error_reporter, "Image capture failed.");
 }

 // Run the model on this input and make sure it succeeds.
 if (kTfLiteOk !=interpreter->Invoke()) {
 TF_LITE_REPORT_ERROR(error_reporter, "Invoke failed.");
 }

 TfLiteTensor* output =interpreter->output(0);

 // Process the inference results.
 uint8_t person_score =output->data.uint8[kPersonIndex];
 uint8_t no_person_score =output->data.uint8[kNotAPersonIndex];
 RespondToDetection(error_reporter, person_score, no_person_score);
} 

#if defined(ARDUINO) &&!defined(ARDUINO_ARDUINO_NANO33BLE)
#define ARDUINO_EXCLUDE_CODE
#endif // defined(ARDUINO) &&!defined(ARDUINO_ARDUINO_NANO33BLE)

#ifndef ARDUINO_EXCLUDE_CODE

// Required by Arducam library
#include 
#include 
#include 
// Arducam library
#include 
// JPEGDecoder library
#include 

// Checks that the Arducam library has been correctly configured
#if !(defined OV2640_MINI_2MP_PLUS)
#error Please select the hardware platform and camera module in the Arduino/libraries/ArduCAM/memorysaver.h
#endif

// The size of our temporary buffer for holding
// JPEG data received from the Arducam module
#define MAX_JPEG_BYTES 4096
// The pin connected to the Arducam Chip Select
#define CS 7

// Camera library instance
ArduCAM myCAM(OV2640, CS);
// Temporary buffer for holding JPEG data from camera
uint8_t jpeg_buffer[MAX_JPEG_BYTES] ={0};
// Length of the JPEG data currently in the buffer
uint32_t jpeg_length =0;

// Get the camera module ready
TfLiteStatus InitCamera(tflite::ErrorReporter* error_reporter) {
 TF_LITE_REPORT_ERROR(error_reporter, "Attempting to start Arducam");
 // Enable the Wire library
 Wire.begin();
 // Configure the CS pin
 pinMode(CS, OUTPUT);
 digitalWrite(CS, HIGH);
 // initialize SPI
 SPI.begin();
 // Reset the CPLD
 myCAM.write_reg(0x07, 0x80);
 delay(100);
 myCAM.write_reg(0x07, 0x00);
 delay(100);
 // Test whether we can communicate with Arducam via SPI
 myCAM.write_reg(ARDUCHIP_TEST1, 0x55);
 uint8_t test;
 test =myCAM.read_reg(ARDUCHIP_TEST1);
 if (test !=0x55) {
 TF_LITE_REPORT_ERROR(error_reporter, "Can't communicate with Arducam");
 delay(1000);
 return kTfLiteError;
 } 

// Switch on the green LED when a person is detected,
 // the red when no person is detected
 if (person_score> no_person_score) {
 digitalWrite(LEDG, LOW);
 digitalWrite(LEDR, HIGH);
 servo_8.write(0); // this servo moves to 0degrees in order to open the mall door when a person is detected to ensure no touch entry system
 count++
 
 } else {
 digitalWrite(LEDG, HIGH);
 digitalWrite(LEDR, LOW);
 servo_8.write(180); // this servo moves to 180degrees when no person is detected
 }

 TF_LITE_REPORT_ERROR(error_reporter, "Person score:%d No person score:%d",
 person_score, no_person_score);
} 


// Now we have let in 50people inside the store, so we set a delay of 15min wait time for others waiting outside to let them in 
// Displaying wait time on the screen every 1min

 if(count =y*50) // when people are detected in multiples of 50, we instruct it to start displaying wait time to other people to wait until 15 min 
 myDisplay.setTextAlignment(PA_CENTER);
 myDisplay.print("Waiting 15min");
 delay(60000); 

Setting up the ThingSpeak Dashboard:

Since the features in Thingspeak Dashboard are limited, I will not be implementing the Time prediction algorithm right now but I am working on the logic to communicate and write data from the Dashboard to microcontroller to perform the time calculation algorithm.

In the ThingSpeak Dashboard, I have added two fields; one for entry and the other one for exit.

The co-ordinates for the store or mall for which the queuing system is displayed, is also added in the form of a map.

The data displayed for the first field is gathered through the entry responding logic and the data displayed for the second field is gathered through the exit responding logic.

This is the snip of the two different logics used in the model.

The ThingSpeak Dashboard can be made available to the the staff of the store to check the number of people entering the store in real time and the number of people exiting the store in real time. This data can also be observed to see the analysis of the data at a given day at a a given time to check and impose further restrictions if required if the limit of people in the store exceeds the expected number of people at a given day

This Dashboard can be viewed here:IoT Dashboard

The following represents the field created for this purpose.

Now, a question might arise that for person detection model that this model can be replaced by ultrasonic or infrared sensors. Flaws in Ultrasonic Sensors or Infrared Based Sensors:These sensors are not exactly accurate and for the real time person count display, these sensors may provide wrong readings. Also, these sensors add as additional hardwares while in Go to Market Solutions, the person detection algorithm can be implemented in existing cameras and can reduce hardware cost. The data from these cameras could be sent to the Arduino BLE sense for central Classification and data processing.

GO TO MARKET &PRACTICALITY:

• Malls and Supermarkets can use this to identify The count of people entering and exiting the mall in real time.
• Implement Strategies using this data to ensure Safety and Compliance with efficient Queue management algorithms.
• Decrease Labour and automate Queue Management Process
• Offer Dashboard to the visitors to monitor the density of people inside the mall and accordingly visit the mall at the safest point of time.
• This product can be used to ensure the visitors that the mall is a safe place and hence, can increase the sales and visits following Government guidelines
• Companies offering Ai and IoT based solutions can invest for mass production and distribution.
• The more the supermarkets using this product, the more the access to data to the government and more the choice to customers to select the preferable safest place in their locality along with the the queue time required for each store can be monitored. This will lead to a wide range of options of supermarkets in the locality comparing the queue time and safety.
• Comparatively affordable solution as compared to manual queuing system and updating information manually to the Dashboard.
• Utilize real-time CCTV footage to impose Queue management in a mall/shop through person detection in terms of timely trends and spatial analysis of person density in the mall.
• Enable Stores to make better, data-driven decisions that ensure your safety and efficient Queues based on autonomous queuing system.

Github Code:Arduino Autonomous TinyML and IoT based queuing system.

Addition to the Existing Person Detection Algorithm- Mask Detection System:

Mask Detection Model based on TinyML :

Dr. Kierstin Kennedy, chief of hospital medicine at the University of Alabama at Birmingham, said, “Masks can protect against any infectious illness that may be spread by droplets. For example, the flu, pertussis (whooping cough), or pneumonia.”

Adding that wearing a cloth mask has benefits beyond slowing the spread of COVID-19, and that source control can reduce the transmission of many other easily spread respiratory infections — the kind that typically render people infectious even before they display symptoms, like influenza.

Until the threat of this pandemic has been neutralized, people should embrace the protection masks allow them to provide to those around them.

After all, it’s not necessarily about you — it’s about everyone you come in contact with.

It’s not at all uncommon to be an asymptomatic carrier of the new coronavirus — which means that even if you have no symptoms at all, you could potentially transmit the virus to someone who could then become gravely ill or even die.

Adhering to this, I decided to increase the necessity of wearing face-masks along with touch-free systems to increase safety in malls and supermarkets. Along with the person detection algorithm, I decided to make a custom face mask detection model which detects face masks and displays this data on the ThingSpeak IoT dashboard to increase awareness among mall staff as well as the visitors coming inside so that they are aware of the time trends when the most number of people are without masks. Through this there is a increases of sense of warning and awareness in people to wear masks. Accordingly, the store staff can keep a monitor on these trends and increase restrictions based on data driven statistics.

Deciding upon the Logic and Dataset of the Model:

This is an overall Logic used in most of face mask detection algorithms. Since, we are deploying this model to an Arduino 33 BLE Sense, the deployment process of this model will vary.

There are two steps involved in constructing the model for Face Mask Detection.

• Training: Here we’ll focus on loading our face mask detection dataset from disk, training a model (using Keras/TensorFlow) on this dataset, and then serializing the face mask detector to disk
• Deployment: Once the face mask detector is trained, we can then move on to loading the mask detector, performing face detection, and then classifying each face as mask or no_mask

Dataset used in training this model:

The dataset used in this process consists of 3500 images but to reduce the size of the model, and feed in accurate model images, I have used 813 images to increase accuracy of the model by decreasing bulk size. This model is an average 676K in size and utilizes nearly 440.3K Ram. Since this model is the optimized version of the original model, the accuracy of the model is 87.47% as compared to 98.15% in the non-optimized one.

The following Softwares have been used in designing this model:

• TensorFlow lite
• ThingSpeak
• Arduino Web Editor

Heading towards designing the model in EdgeImpulse Studio:

Powerful deep learning models (based on artificial neural networks) are now reaching microcontrollers. Over the past year great strides were made in making deep learning models smaller, faster and runnable on embedded hardware through projects like TensorFlow Lite for Microcontrollers, uTensor and Arm’s CMSIS-NN; but building a quality dataset, extracting the right features, training and deploying these models is can still be complicated.

Using Edge Impulse you can now quickly collect real-world sensor data, train ML models on this data in the cloud, and then deploy the model back to your Arduino device. From there you can integrate the model into your Arduino sketches with a single function call.

Step 1 - Acquisition of Data in the Edge Impulse Studio:

Using the dataset of 3500 images, I filtered these images to the best performing images and finally fed in 813 images totally in the Training Data and 487 images in the testing Data. I labelled these classes as mask and no_mask.

Then, I went ahead creating an impulse design which best suited the Model type. For optimal accuracy its recommended to use a standard image size which is 96*96 and also works the best on the Arduino 33 BLE Sense. Since the input type was images, I went ahead and selected "images" in the processing block. For the transfer learning block, the type recommended for image learning is Transfer Learning (Images) which is a Fine tune a pre-trained image model on your data. with Good performance even with relatively small image datasets.

The next step was saving the parameters based on color depth. Here, I have selected RGB because in the dataset I am using for mask detection, color is also an important feature of classification instead of grayscale. In this page, we can also see the raw features with the processed features of the image

After Feature Generation I obtained the classification graph or the feature explorer where I could see the classes based on their classifications. The blue dots represent mask images and the orange dots represent the no_mask images.

In this feature generation, I obtained a fair classification with a distinct classification plot.

Finally, Moving on to the Transfer Learning Plot:

Here, I set the number of training cycles/epochs as 30 to get the highest accuracy with minimum val_loss. It so happens that if we train the model based on many training cycles, the accuracy graph starts to decrement after a certain number of epochs and the val_loss increases. Therefore I decided to limit the epochs to 30 which proved to be perfect. The learning rate is set to 0.0005 which is the default and proves to be the most appropriate. Here, I have used the MobileNetV2 0.35 (final layer:16 neurons, 0.1 dropout) model because this model is comparitively lightweight and accurate.

Finally after completing 30 epochs and 10 epochs of best model performance, I got the accuracy heading to 1.00 and the loss nearly 0 which was 0.0011. The following was the ouput during the training process:

Saving best performing model... Converting TensorFlow Lite float32 model... Converting TensorFlow Lite int8 quantized model with float32 input and output...

• Epoch 9/10 21/21 - 5s - loss:0.0011 - accuracy:1.0000 - val_loss:0.0727 - val_accuracy:0.9755
• Epoch 10/10 21/21 - 5s - loss:0.0012 - accuracy:1.0000 - val_loss:0.0728 - val_accuracy:0.9755 Finished training

This was the final output which I received after the training:

The accuracy to be 92.6% and the loss to be 0.19.

Finally Using the test data, I tested the accuracy and found it to be 98.15%.

Finally since I had the model ready, I deployed it as an Arduino Library with the firmware to be the Arduino 33 BLE Sense:I got the zip folder of the library ready and started to make changes as per our requirements.

For a last confirmation, I live classified the data to ensure that the data is classified properly. I got the perfect results as expected:

Changing the code as per the output required in the model:

Here is a snippet of the main.ino code of the mask_detection model

I have defined the arduino libraries required for the functioning of the model and based on the Tensorflow lite framework, I have designed this model.

This is the loop of some of the main functions in the model which are defined in the libraries. The main.ino code centralises these functions and accordingly loops them with a central code.

void loop() {
 // Get image from provider.
 if (kTfLiteOk !=GetImage(error_reporter, kNumCols, kNumRows, kNumChannels,
 input->data.uint8)) {
 TF_LITE_REPORT_ERROR(error_reporter, "Image capture failed.");
 }

 // Run the model on this input and make sure it succeeds.
 if (kTfLiteOk !=interpreter->Invoke()) {
 TF_LITE_REPORT_ERROR(error_reporter, "Invoke failed.");
 }

 TfLiteTensor* output =interpreter->output(0);

 // Process the inference results.
 uint8_t mask_score =output->data.uint8[kmaskIndex];
 uint8_t no_mask_score =output->data.uint8[kno-maskIndex];
 RespondToDetection(error_reporter, mask_score, no_mask_score);
} 

The processed data of this model can be viewed here :(This file is relatively large and varies as per the dataset size of the model) Arduino_mask_detect_model_data.h

For providing image, I will be using the Arducam mini 2mp plus for visual data input. A snippet from the image_provider.h file is :

#include "image_provider.h"

/*
 * The sample requires the following third-party libraries to be installed and
 * configured:
 *
 * Arducam
 * -------
 * 1. Download https://github.com/ArduCAM/Arduino and copy its `ArduCAM`
 * subdirectory into `Arduino/libraries`. Commit #e216049 has been tested
 * with this code.
 * 2. Edit `Arduino/libraries/ArduCAM/memorysaver.h` and ensure that
 * "#define OV2640_MINI_2MP_PLUS" is not commented out. Ensure all other
 * defines in the same section are commented out.
 *
 * JPEGDecoder
 * -----------
 * 1. Install "JPEGDecoder" 1.8.0 from the Arduino library manager.
 * 2. Edit "Arduino/Libraries/JPEGDecoder/src/User_Config.h" and comment out
 * "#define LOAD_SD_LIBRARY" and "#define LOAD_SDFAT_LIBRARY".
 */

#if defined(ARDUINO) &&!defined(ARDUINO_ARDUINO_NANO33BLE)
#define ARDUINO_EXCLUDE_CODE
#endif // defined(ARDUINO) &&!defined(ARDUINO_ARDUINO_NANO33BLE)

#ifndef ARDUINO_EXCLUDE_CODE

// Required by Arducam library
#include 
#include 
#include 
// Arducam library
#include 
// JPEGDecoder library
#include 

// Checks that the Arducam library has been correctly configured
#if !(defined OV2640_MINI_2MP_PLUS)
#error Please select the hardware platform and camera module in the Arduino/libraries/ArduCAM/memorysaver.h
#endif 

The Arduino_detection_responder.cpp code performs the inference and delivers the main output required for the code. Here, when a person with a mask is detected, we are opening the gate and if a person with no mask is detected, we are closing the gate.

// Switch on the green LED when a mask is detected,
 // the red when no mask is detected
 if (mask_score> no_mask_score) {
 digitalWrite(LEDG, LOW);
 digitalWrite(LEDR, HIGH);
 servo_8.write(0); // this servo moves to 0degrees in order to open the mall door when a person is detected to ensure no touch entry system
 count++
 } else {
 digitalWrite(LEDG, HIGH);
 digitalWrite(LEDR, LOW);
 servo_8.write(180); // this servo moves to 180degrees when no person is detected
 count2++
 } 

The count and count2 are variable integers and increment everytime a mask is detected, or not detected. These counts are then displayed onto the ThingSpeak IoT Dashboard as follows:

The graph displays the mask and no mask count over time. This thingspeak dashboard is setup in the Arduino code as seen here:

#include 
#include 
WiFiMulti WiFiMulti;
const char* ssid ="Yourssid"; // Your SSID (Name of your WiFi)
const char* password ="Wifipass"; //Your Wifi password
const char* host ="api.thingspeak.com/channels/1118220";
String api_key ="9BRPKINQJJT2WMWP"; // Your API Key provied by thingspeak 

The complete code for responder can be viewed here:Arduino_detection_responder.cpp

Github Code can be viewed here:Arduino Mask Detetction

The ThingSpeak dashboard can be viewed here - IoT Dashboard

5th project - Person detection in supermarket Aisles and self-sanitisation system:

Just as malls and supermarkets and retail sectors have opened, a risk of contamination of the virus has increased. The main risk that is faced is in supermarkets and stores. The stores could be related to clothes, food and even electronics, the virus could deposit and stay on these surfaces for a long time.

Early data suggests the new coronavirus can live on surfaces for several days

The new coronavirus is, well, new — and there's still much to learn about how easily the virus can spread via contaminated surfaces. But early evidence indicates that the surface survivability of the new coronavirus is similar to that of SARS, a related coronavirus first identified in 2002. Depending on the surface, the virus can live on surfaces for a few hours or up to several days.

The new coronavirus seems to be able to survive the longest on plastic and stainless steel — potentially as long as three days on these surfaces. It can also live on cardboard for up to 24 hours.

Find out what this means for the things you touch throughout the day, including your:

• Clothes
• Food
• Groceries
• Packages
• Eletrônicos

The coronavirus pandemic has breathed new life into a decades-old technique that can zap viruses and bacteria:ultraviolet light.

Hospitals have been using it for years to cut down on the spread of drug-resistant superbugs and to disinfect surgical suites. But there is now interest in using the technology in spaces like schools, office buildings, and restaurants to help reduce coronavirus transmission once public spaces are open again.

The sanitizing effects of UV lights have been seen with other coronaviruses, including the one that causes severe acute respiratory syndrome (SARS). Studies have shown that it can be used against other coronaviruses. One study found at least 15 minutes of UVC exposure inactivated SARS, making it impossible for the virus to replicate. New York's Metropolitan Transit Authority announced the use of UV light on subway cars, buses, technology centers, and offices. The National Academy of Sciences says although there is no concrete evidence for UV’s effectiveness on the virus that causes COVID-19, it has worked on other similar viruses, so it would likely fight this one too.

Abiding by the given information, Malls and Supermarkets have started imposing solutions based on UV sanitisation on manual basis. This consumes a lot of time and also, the data is unavailable. In manual process, one does not know which area has been exposed to human touch more or which are has the highest risk of contamination.

To automate this process and to provide real time spatial analysis of the data collected and the rate of contamination in an area.

The above images demonstrate the distribution of population across the mall. As it is clearly seen in Figure B, the density near the escalator is comparatively high as compared to the density on the second floor. Manually keeping a track of this continuously changing data in real time and accordingly, sanitising the areas/aisles according to the rate of contamination is not a manual task.

Hence, I decided to make a robust solution for automating this process with the help of TinyML computer Vision &IoT deployed on the Arduino 33 BLE Sense.

The structured framework of this system is as follows:

These models are individually deployed on the Aisles and Areas of the malls and supermarkets close to the Area which has to be sanitised. Since the model utilises person detection Algorithm, the placement of the model is done accordingly.

In this model, for accumulation of Video data, the Arducam mini 2MP Plus is used. The Arducam Mini 2MP Plus continuously gathers visual data and sends this data to the Arduino Nano 33 BLE Sense for processing and classification. The Arduino classifies the data and accordingly processes commands. If a person is detected, the count of people increments by 1 on detection of each person near the Aisle.

A certain threshold for the count of people is set depending upon the rate of contamination based on the object type in that area. The rate of contamination and threshold for food aisle, clothes aisle, sports aisle and Electronic aisle is different.

Accordingly taking into consideration an average threshold, if 25 people are detected, the area is safe. If the count increases to 50, the area is heading towards contamination and the visitors are given an alert. If the count increases to 100, the area is declared contaminated and the people are warned to be careful while touching objects. Finally if the threshold limit is crosses which is 150, the area is autonomously sanitised using UV Light.

The alerts generated are based on LED Colours. Green is an indication for safe, blue is an indication for alert, and Red is an indication for Warning!. These LED based alerts are installed in the respective areas individually.

The above image shows the various aisles in the malls where the system can be installed.

The above image shows the area of installation of Arducam or similar visual data capturing device to cover a wide spectrum of people walking through the aisle.

The above Images show a demo implementation of the UV Sanitisation Lights in suitable areas of the aisles. This UV light covers a large spectrum of area that can be sanitised together.

Implementation of the autonomous sanitisation system.

The following Softwares have been used in designing this model:

• TensorFlow lite
• ThingSpeak
• Arduino Web Editor

In this person detection model, I have used the Pre-trained TensorFlow Person detection model apt for the project. This pre-trained model consists of three classes out of which the third class is with undefined set of data:

"unused",

"person",

"notperson"

In our model we have the Arducam Mini 2mp plus to carry out image intake and this image data with a decent rate of fps is sent to the Arduino Nano 33 BLE Sense for processing and and classification. Since the Microcontroller is capable of providing 256kb RAM, we change the image size of each image to a standard 96*96 for processing and classification. The Arduino Tensorflow Lite network consists of a deep learning framework as:

• Depthwise Conv_2D
• Conv_2D
• AVERAGE Pool_2D
• Flatten layer

This deep learning framework is used to train the Person detection model.

The following is the most important function defined while processing outputs on the Microcontroller via Arduino_detetction_responder.cpp

// Process the inference results.
 uint8_t person_score =output->data.uint8[kPersonIndex];
 uint8_t no_person_score =output->data.uint8[kNotAPersonIndex];
 RespondToDetection(error_reporter, person_score, no_person_score); 

In the following function defining, the person_score , the no_person_score have been defined on the rate of classification of the data.

using these defined functions, I will be using it to give certain outputs on the basis of confidence of the person_score and the no_person_score .

The detection responder logic of the code works in the following way:

├── Person Detection and sanitisation
├── Arducam mini 2MP Plus
│ ├──Image and Video Data to Arduino
├── Arduino BLE 33 Sense
│ ├── processing and classification of the input data
│ │ ├── If person detected, increment the count by 1
│ │ ├── If no person detected, do nothing
│ │ ├── Send the number of people entered count to to ThingSpeak Dashboard via ESP8266 -01
│ │ | ├── If people count is uptil 25
│ │ │ | ├── Indicate the area to be safe by flashing green light

│ │ | ├── If people count is uptil 50
│ │ │ | ├── Indicate the people to be aware by flashing blue light

│ │ | ├── If people count is uptil 100
│ │ │ | ├── Indicate the area to be contaminated by flashing Red light

│ │ | ├── If people count is between 150 to 175
│ │ │ | ├── Sanitise the area by activating the UV light

│ │ │ | ├── Reset the person count to 0 since the area has been sanitised
│ │ └── ...Repeat the loop

According to the logic used in the model, the Arducam Mini 2MP Plus will continuously capture visual data and send this data to the Arduino Nano 33 BLE sense to process and classify. This model size is 125Kb. Once the image is processed, the Arduino starts to classify the data captured. If a person is detected in the model, the person count increases by 1. This person count is continuously sent to the ThingSpeak dashboard via ESP8266- 01 IoT module. The person count enables the supermarket staff with the data of number of visitors in a particular area at any given point of time. The staff can generate data driven decisions to take action in an area if the visitor count is significantly high.

Proceeding to the output, if the person count is between 1 to 25, the Area is declared safe by flashing the green LED. If the person count is between 26 to 50, the visitors are given an alert by flashing BLUE LED. If the count is between 51 to 100, the area is declared contaminated by flashing the RED LED. When the count surpasses a certain threshold, here taken to be 150, the UV light is turned on until the 165th person passes the area. In this way the Area is sanitised. Similarly the count of people is reset to 0 since the area has been sanitised.

Working of the Firmware:

This model comprises of the following firmware used:

• Arduino 33 BLE sense - Used to process the data gathered, classifies the data processes, sends the command according to the logic fed.
• Arducam Mini 2mp plus - Continuous Raw data image accumulation from source.
• Adafruit lithium ion charger - Used to deliver charge through the lithium battery
• Lithium ion Battery - power source
• ESP8266 - 01 - Used for sending data to the ThingSpeak dashboard via WiFi network.
• RGB LED - Used for flashing Signals based on the status of contamination
• UV Light - Used to Sanitise the Area ( Since this is a prototype, an LED is displayed. In the actual solution, the UV light consumes a lot of energy hence an external power source needs to be provided. )

Functioning and Working of Logic in Code:

The following are the Libraries included in themain.ino c ode for functioning of the model.

#include 

#include "main_functions.h"

#include "detection_responder.h"
#include "image_provider.h"
#include "model_settings.h"
#include "person_detect_model_data.h"
#include "tensorflow/lite/micro/kernels/micro_ops.h"
#include "tensorflow/lite/micro/micro_error_reporter.h"
#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/micro/micro_mutable_op_resolver.h"
#include "tensorflow/lite/schema/schema_generated.h"
#include "tensorflow/lite/version.h" 

The main outcome of the model is dependent on the logic fed in detection_responder.cpp. This snippet shows the generalized threshold of the contamination of the object set to a count of 101.

void RespondToDetection(tflite::ErrorReporter* error_reporter,
 uint8_t person_score, uint8_t no_person_score) {
 static bool is_initialized =false;
 if (!is_initialized) {
 // Pins for the built-in RGB LEDs on the Arduino Nano 33 BLE Sense
 pinMode(LEDR, OUTPUT);
 pinMode(LEDG, OUTPUT);
 pinMode(LEDB, OUTPUT);
 is_initialized =true;
 }

 // Note:The RGB LEDs on the Arduino Nano 33 BLE
 // Sense are on when the pin is LOW, off when HIGH.

 // Switch the person/not person LEDs off
 digitalWrite(LEDG, HIGH);
 digitalWrite(LEDR, HIGH);

 // Flash the blue LED after every inference.
 digitalWrite(LEDB, LOW);
 delay(100);
 digitalWrite(LEDB, HIGH);

 // Switch on the green LED when a person is detected,
 // the red when no person is detected
 if (person_score> no_person_score) {
 digitalWrite(LEDG, LOW);
 digitalWrite(LEDR, HIGH);
 count++
 } else {
 digitalWrite(LEDG, HIGH);
 digitalWrite(LEDR, LOW);
 }

 TF_LITE_REPORT_ERROR(error_reporter, "Person score:%d No person score:%d",
 person_score, no_person_score);
}


 if (constrain(count, 1, 50)) {
 digitalWrite(LEDGREEN, LOW); // This is a green LED which means area is not contaminated yet
}
 if (constrain(count, 51, 75)) {
 digitalWrite(LEDBLUE, LOW); // This is a b led so that people are alert that more than 50 people
}
 if (constrain(count, 76,100)) {
 digitalWrite(LEDRED, LOW); // This is a red LED which warns people that this area is contaminated and one should take care while touching objects
}
 if (constrain(count, 101, 110)) {
 digitalWrite(UV, LOW); // This is an Ultraviolet Light which ensures that after 100 people have touched the aisle objects, it sanitises the area with UV light
 }
 if (count> 111) {
 (count =0); // this resets the person count to 0 once the area has been sanitised by UV light
 } 
 } // In this way, a simple math function has been written by me on the arduino nano 33 ble sense which alerts people and sanitizes areas
/* cc - Dhruv Sheth */ 

The below snippet is carried out by the ESP8266 -01 module to send data to the allocated field in the ThingSpeak Dashboard:

void Send_Data()
{

 
 // Use WiFiClient class to create TCP connections
 WiFiClient client;

 const int httpPort =80;

 if (!client.connect(host, httpPort)) {
 Serial.println("connection failed");
 return;
 }
 else
 {
 String data_to_send =api_key;
 data_to_send +="&field1=";
 data_to_send +=String(count);
 data_to_send +="\r\n\";

 client.print("POST /update HTTP/1.1\n");
 client.print("Host:api.thingspeak.com\n");
 client.print("Connection:close\n");
 client.print("X-THINGSPEAKAPIKEY:" + api_key + "\n");
 client.print("Content-Type:application/x-www-form-urlencoded\n");
 client.print("Content-Length:");
 client.print(data_to_send.length());
 client.print("\n\n");
 client.print(data_to_send);

 delay(10); // reduced delay to perform real time data collection
 }

 client.stop();

} 

The threshold for each Aisle inside the mall is set to be different according to the rate of contamination of the model. This data can be altered according to the need of the malls or supermarkets. These thresholds can also be altered according to the time trends and the population density in that area.

Threshold set for the Clothes Aisle:

if (constrain(count, 1, 50)) {
 digitalWrite(LEDGREEN, LOW); // This is a green LED which means area is not contaminated yet
}
 if (constrain(count, 51, 75)) {
 digitalWrite(LEDBLUE, LOW); // This is a b led so that people are alert that more than 50 people
}
 if (constrain(count, 76,100)) {
 digitalWrite(LEDRED, LOW); // This is a red LED which warns people that this area is contaminated and one should take care while touching objects
}
 if (constrain(count, 101, 110)) {
 digitalWrite(UV, LOW); // This is an Ultraviolet Light which ensures that after 100 people have touched the aisle objects, it sanitises the area with UV light
 }
 if (count> 111) {
 (count =0); // this resets the person count to 0 once the area has been sanitised by UV light
 } 

Threshold for the Electronics Aisle, since the virus stays on these surfaces comparatively less, we have set a higher threshold in this case.

if (constrain(count, 1, 75)) {
 digitalWrite(LEDGREEN, LOW); // This is a green LED which means area is not contaminated yet
}
 if (constrain(count, 76, 180)) {
 digitalWrite(LEDBLUE, LOW); // This is a b led so that people are alert that more than 50 people
}
 if (constrain(count, 181,225)) {
 digitalWrite(LEDRED, LOW); // This is a red LED which warns people that this area is contaminated and one should take care while touching objects
}
 if (constrain(count, 225, 240)) {
 digitalWrite(UV, LOW); // This is an Ultraviolet Light which ensures that after 100 people have touched the aisle objects, it sanitises the area with UV light
 }
 if (count> 111) {
 (count =0); // this resets the person count to 0 once the area has been sanitised by UV light
 } 

Threshold set for Food and eatables:, Since these are consumptives, they face a higher risk of contamination. Hence, a lower threshold is set in this case.

if (constrain(count, 1, 25)) {
 digitalWrite(LEDGREEN, LOW); // This is a green LED which means area is not contaminated yet
}
 if (constrain(count, 25, 50)) {
 digitalWrite(LEDBLUE, LOW); // This is a b led so that people are alert that more than 50 people
}
 if (constrain(count, 51,75)) {
 digitalWrite(LEDRED, LOW); // This is a red LED which warns people that this area is contaminated and one should take care while touching objects
}
 if (constrain(count, 76, 90)) {
 digitalWrite(UV, LOW); // This is an Ultraviolet Light which ensures that after 100 people have touched the aisle objects, it sanitises the area with UV light
 }
 if (count> 91) {
 (count =0); // this resets the person count to 0 once the area has been sanitised by UV light
 } 

Setting up the Thingspeak IoT Dashboard:

On the thingspeak dashboard, there are 4 fields which have been added for the 4 aisles that have been set up namely:

• Food-aisle
• Sports-aisle
• Clothes-aisle
• Electronics-aisle

The co-ordinate locations of the mall can be seen in the channel visualization which can be cumulatively displayed on a single dashboard through which the data for each supermarket and mall in a locality can be accessed.

The IoT Dashboard can be viewed here:IoT Dashboard

The fields and channels of aisles can be all displayed together on a single Thingspeak IoT Dashboard.

The above image shows a density plot of the density population in a given region in the store of a mall.

Currently I have been using the ThingSpeak Dashboard to plot the data in a graphical format. But, using the same data and Tableau visualizations, it is possible to plot the given data on the floor plot map of the Supermarket.

This data will be easily accessible to other visitors entering the supermarket and will be able to interpret data driven decisions like visiting the store which has a low population density.

Now a question might arise that what makes the product unique:

Currently, the UV sanitisation system is implemented on manual basis and hence the sanitisation system is carried out manually monitoring these areas. Autonomous UV robots are deployed in malls to gather plot data of mall and map these areas, hence autonomously sanitising these areas. These solutions are not capable of sanitising each aisle and area in the mall along with the flowing population of the people in the mall. The sanitisation is only limited to sanitising the lower part of these aisles and hence the upper parts are left unsanitised. These solutions are also comparatively expensive and they are not capable of continuous data monitoring and sanitisation respective to the rate of contamination. This means that they are unable to monitor the number of people who have passes through a certain area or aisle and hence are unable to sanitise areas according to their contamination.

Go to Market and Viability:

• Malls and Supermarkets can use this to identify The count of people in and density of people in a certain area in a store and impose self sanitisation process within the malls.
• Implement Strategies using this data to ensure Safety and Compliance with efficient Population Density monitoring algorithms.
• Decrease Labour and automate Sanitisation process.
• Offer Dashboard to the visitors to monitor the density of people at a certain area inside the mall and accordingly the visitors can have a view of the store with lowest density population and make decisions based on this data. The visitors can also have a view of the timely trends of sanitisation by viewing the abrupt change of person density graph in an area when sanitisation is conducted. ( Since, each time sanitisation in an area takes place, the person count is reset to 0 because the place is sanitised)

• This product can be used to ensure the visitors that the mall is a safe place and hence, can increase the sales and visits following Government guidelines
• Companies offering Ai and IoT based solutions can invest for mass production and distribution.
• The more the supermarkets using this product, the more the access to data to the government and more the choice to customers to select the preferable safest place in their locality and which store is continuously sanitised to be up to mark in terms of safety. This will lead to a wide range of options of supermarkets in the locality comparing the queue time and safety.
• Comparatively affordable solution as compared to autonomous UV Robot system and highly scalable in terms of data and services with IoT Dashboard provided.
• Utilize real-time CCTV footage to impose autonomous sanitisation system through person detection in terms of timely trends and spatial analysis of person density in the mall.
• Enable Stores to make better, data-driven decisions that ensure your safety and efficient Queues based on autonomous queuing system.

Github Link :https://github.com/dhruvsheth-ai/self-sanitisation-person-detection

PROJECT VIDEO:

Thank you for viewing my project!