Merge branch 'dev' into 'master'

Miroku - The Refactor of all Refactors

See merge request Shinobi-Systems/Shinobi!224

Docker/README.md

@@ -0,0 +1,137 @@
+# Install Shinobi with Docker
+
+### There are three ways!
+
+## Docker Ninja Way
+
+> This method uses `docker-compose` and has the ability to quick install the TensorFlow Object Detection plugin.
+
+```
+bash <(curl -s https://gitlab.com/Shinobi-Systems/Shinobi-Installer/raw/master/shinobi-docker.sh)
+```
+
+## Docker Ninja Way - Version 2
+
+#### Installing Shinobi
+
+> Please remember to check out the Environment Variables table further down this README.
+
+```
+docker run -d --name='Shinobi' -p '8080:8080/tcp' -v "/dev/shm/Shinobi/streams":'/dev/shm/streams':'rw' -v "$HOME/Shinobi/config":'/config':'rw' -v "$HOME/Shinobi/customAutoLoad":'/home/Shinobi/libs/customAutoLoad':'rw' -v "$HOME/Shinobi/database":'/var/lib/mysql':'rw' -v "$HOME/Shinobi/videos":'/home/Shinobi/videos':'rw' -v "$HOME/Shinobi/plugins":'/home/Shinobi/plugins':'rw' -v '/etc/localtime':'/etc/localtime':'ro' shinobisystems/shinobi:dev
+```
+
+#### Installing Object Detection (TensorFlow.js)
+
+> This requires that you add the plugin key to the Shinobi container. This key is generated and displayed in the startup logs of the Object Detection docker container.
+
+- `-p '8082:8082/tcp'` is an optional flag if you decide to run the plugin in host mode.
+- `-e PLUGIN_HOST='10.1.103.113'` Set this as your Shinobi IP Address.
+- `-e PLUGIN_PORT='8080'` Set this as your Shinobi Web Port number.
+
+```
+docker run -d --name='shinobi-tensorflow' -e PLUGIN_HOST='10.1.103.113' -e PLUGIN_PORT='8080' -v "$HOME/Shinobi/docker-plugins/tensorflow":'/config':'rw' shinobisystems/shinobi-tensorflow:latest
+```
+
+- CPU : https://gitlab.com/Shinobi-Systems/docker-plugin-tensorflow.js
+- GPU (NVIDIA CUDA) : https://gitlab.com/Shinobi-Systems/docker-plugin-tensorflow.js/-/tree/gpu
+
+
+## From Source
+> Image is based on Ubuntu Bionic (20.04). Node.js 12 is used. MariaDB and FFmpeg are included.
+
+1. Download Repo
+
+```
+git clone -b dev https://gitlab.com/Shinobi-Systems/Shinobi.git ShinobiSource
+```
+
+2. Enter Repo and Build Image.
+
+```
+cd ShinobiSource
+docker build --tag shinobi-image:1.0 .
+```
+
+3. Create a container with the image.
+
+> This command only works on Linux because of the temporary directory used. This location must exist in RAM. `-v "/dev/shm/shinobiStreams":'/dev/shm/streams':'rw'`. The timezone is also acquired from the host by the volume declaration of `-v '/etc/localtime':'/etc/localtime':'ro'`.
+
+```
+docker run -d --name='Shinobi' -p '8080:8080/tcp' -v "/dev/shm/Shinobi/streams":'/dev/shm/streams':'rw' -v "$HOME/Shinobi/config":'/config':'rw' -v "$HOME/Shinobi/customAutoLoad":'/home/Shinobi/libs/customAutoLoad':'rw' -v "$HOME/Shinobi/database":'/var/lib/mysql':'rw' -v "$HOME/Shinobi/videos":'/home/Shinobi/videos':'rw' -v "$HOME/Shinobi/plugins":'/home/Shinobi/plugins':'rw' -v '/etc/localtime':'/etc/localtime':'ro' shinobi-image:1.0
+```
+
+ > Host mount paths have been updated in this document.
+
+ ### Volumes
+
+ | Volumes                     | Description                                                                                                                                         |
+ |-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------|
+ | /dev/shm/Shinobi/streams     | **IMPORTANT!** This must be mapped to somewhere in the host's RAM. When running this image on Windows you will need to select a different location. |
+ | $HOME/Shinobi/config         | Put `conf.json` or `super.json` files in here to override the default. values.                                                                           |
+ | $HOME/Shinobi/customAutoLoad | Maps to the `/home/Shinobi/libs/customAutoLoad` folder for loading your own modules into Shinobi.                                                                 |
+ | $HOME/Shinobi/database       | A map to `/var/lib/mysql` in the container. This is the database's core files.                                                                      |
+ | $HOME/Shinobi/videos         | A map to `/home/Shinobi/videos`. The storage location of your recorded videos.                                                                      |
+ | $HOME/Shinobi/plugins        | A map to `/home/Shinobi/plugins`. Mapped so that plugins can easily be modified or swapped.                                                         |
+
+### Environment Variables
+
+ | Environment Variable | Description                                                          | Default            |
+ |----------------------|----------------------------------------------------------------------|--------------------|
+ | SUBSCRIPTION_ID      | **THIS IS NOT REQUIRED**. If you are a subscriber to any of the Shinobi services you may use that key as the value for this parameter. If you have donated by PayPal you may use your Transaction ID to activate the license as well. | *None*     |
+ | DB_USER              | Username that the Shinobi process will connect to the database with. | majesticflame      |
+ | DB_PASSWORD          | Password that the Shinobi process will connect to the database with. | *None* |
+ | DB_HOST              | Address that the Shinobi process will connect to the database with.  | localhost          |
+ | DB_DATABASE          | Database that the Shinobi process will interact with.                | ccio               |
+ | DB_DISABLE_INCLUDED     | Disable included database to use your own. Set to `true` to disable.| false |
+ | PLUGIN_KEYS     | The object containing connection keys for plugins running in client mode (non-host, default). | {} |
+ | SSL_ENABLED     | Enable or Disable SSL. | false |
+ | SSL_COUNTRY     | Country Code for SSL. | CA |
+ | SSL_STATE     | Province/State Code for SSL. | BC |
+ | SSL_LOCATION     | Location of where SSL key is being used. | Vancouver |
+ | SSL_ORGANIZATION     | Company Name associated to key. | Shinobi Systems |
+ | SSL_ORGANIZATION_UNIT     | Department associated to key. | IT Department |
+ | SSL_COMMON_NAME     | Common Name associated to key. | nvr.ninja |
+
+ > You must add (to the docker container) `/config/ssl/server.key` and `/config/ssl/server.cert`. The `/config` folder is mapped to `$HOME/Shinobi/config` on the host by default with the quick run methods. Place `key` and `cert` in `$HOME/Shinobi/config/ssl`. If `SSL_ENABLED=true` and these files don't exist they will be generated with `openssl`.
+
+> For those using `DB_DISABLE_INCLUDED=true` please remember to create a user in your databse first. The Docker image will create the `DB_DATABASE` under the specified connection information.
+
+### Tips
+
+Modifying `conf.json` or Superuser credentials.
+> Please read **Volumes** table in this README. conf.json is for general configuration. super.json is for Superuser credential management.
+
+Get Docker Containers
+```
+docker ps -a
+```
+
+Get Images
+```
+docker images
+```
+
+Container Logs
+```
+docker logs /Shinobi
+```
+
+Enter the Command Line of the Container
+```
+docker exec -it /Shinobi /bin/bash
+```
+
+Stop and Remove
+```
+docker stop /Shinobi
+docker rm /Shinobi
+```
+
+**WARNING - DEVELOPMENT ONLY!!!** Kill all Containers and Images
+> These commands will completely erase all of your docker containers and images. **You have been warned!**
+
+```
+docker stop /Shinobi
+docker rm $(docker ps -a -f status=exited -q)
+docker rmi $(docker images -a -q)
+```

Docker/init.sh

@@ -0,0 +1,105 @@
+#!/bin/sh
+set -e
+
+cp sql/framework.sql sql/framework1.sql
+OLD_SQL_USER_TAG="ccio"
+NEW_SQL_USER_TAG="$DB_DATABASE"
+sed -i "s/$OLD_SQL_USER_TAG/$NEW_SQL_USER_TAG/g" sql/framework1.sql
+if [ "$SSL_ENABLED" = "true" ]; then
+    if [ -d /config/ssl ]; then
+        echo "Using provided SSL Key"
+        cp -R /config/ssl ssl
+        SSL_CONFIG='{"key":"./ssl/server.key","cert":"./ssl/server.cert"}'
+    else
+        echo "Making new SSL Key"
+        mkdir -p ssl
+        openssl req -nodes -new -x509 -keyout ssl/server.key -out ssl/server.cert -subj "/C=$SSL_COUNTRY/ST=$SSL_STATE/L=$SSL_LOCATION/O=$SSL_ORGANIZATION/OU=$SSL_ORGANIZATION_UNIT/CN=$SSL_COMMON_NAME"
+        cp -R ssl /config/ssl
+        SSL_CONFIG='{"key":"./ssl/server.key","cert":"./ssl/server.cert"}'
+    fi
+else
+    SSL_CONFIG='{}'
+fi
+if [ "$DB_DISABLE_INCLUDED" = "false" ]; then
+    echo "MariaDB Directory ..."
+    ls /var/lib/mysql
+
+    if [ ! -f /var/lib/mysql/ibdata1 ]; then
+        echo "Installing MariaDB ..."
+        mysql_install_db --user=mysql --datadir=/var/lib/mysql --silent
+    fi
+    echo "Starting MariaDB ..."
+    /usr/bin/mysqld_safe --user=mysql &
+    sleep 5s
+
+    chown -R mysql /var/lib/mysql
+
+    if [ ! -f /var/lib/mysql/ibdata1 ]; then
+        mysql -u root --password="" -e "SET @@SESSION.SQL_LOG_BIN=0;
+        USE mysql;
+        DELETE FROM mysql.user ;
+        DROP USER IF EXISTS 'root'@'%','root'@'localhost','${DB_USER}'@'localhost','${DB_USER}'@'%';
+        CREATE USER 'root'@'%' IDENTIFIED BY '${DB_PASS}' ;
+        CREATE USER 'root'@'localhost' IDENTIFIED BY '${DB_PASS}' ;
+        CREATE USER '${DB_USER}'@'%' IDENTIFIED BY '${DB_PASS}' ;
+        CREATE USER '${DB_USER}'@'localhost' IDENTIFIED BY '${DB_PASS}' ;
+        GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' WITH GRANT OPTION ;
+        GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' WITH GRANT OPTION ;
+        GRANT ALL PRIVILEGES ON *.* TO '${DB_USER}'@'%' WITH GRANT OPTION ;
+        GRANT ALL PRIVILEGES ON *.* TO '${DB_USER}'@'localhost' WITH GRANT OPTION ;
+        DROP DATABASE IF EXISTS test ;
+        FLUSH PRIVILEGES ;"
+    fi
+
+    # Create MySQL database if it does not exists
+    if [ -n "${DB_HOST}" ]; then
+        echo "Wait for MySQL server" ...
+        while ! mysqladmin ping -h"$DB_HOST"; do
+            sleep 1
+        done
+    fi
+
+    echo "Setting up MySQL database if it does not exists ..."
+
+    echo "Create database schema if it does not exists ..."
+    mysql -e "source /home/Shinobi/sql/framework.sql" || true
+
+    echo "Create database user if it does not exists ..."
+    mysql -e "source /home/Shinobi/sql/user.sql" || true
+
+else
+    echo "Create database schema if it does not exists ..."
+    mysql -u "$DB_USER" -h "$DB_HOST" -p"$DB_PASSWORD" --port="$DB_PORT" -e "source /home/Shinobi/sql/framework.sql" || true
+fi
+
+DATABASE_CONFIG='{"host": "'$DB_HOST'","user": "'$DB_USER'","password": "'$DB_PASSWORD'","database": "'$DB_DATABASE'","port":'$DB_PORT'}'
+
+cronKey="$(head -c 1024 < /dev/urandom | sha256sum | awk '{print substr($1,1,29)}')"
+
+cd /home/Shinobi
+mkdir -p libs/customAutoLoad
+if [ -e "/config/conf.json" ]; then
+    cp /config/conf.json conf.json
+fi
+if [ ! -e "./conf.json" ]; then
+    sudo cp conf.sample.json conf.json
+fi
+sudo sed -i -e 's/change_this_to_something_very_random__just_anything_other_than_this/'"$cronKey"'/g' conf.json
+node tools/modifyConfiguration.js cpuUsageMarker=CPU subscriptionId=$SUBSCRIPTION_ID thisIsDocker=true pluginKeys="$PLUGIN_KEYS" db="$DATABASE_CONFIG" ssl="$SSL_CONFIG"
+sudo cp conf.json /config/conf.json
+
+
+echo "============="
+echo "Default Superuser : admin@shinobi.video"
+echo "Default Password : admin"
+echo "Log in at http://HOST_IP:SHINOBI_PORT/super"
+if [ -e "/config/super.json" ]; then
+    cp /config/super.json super.json
+fi
+if [ ! -e "./super.json" ]; then
+    sudo cp super.sample.json super.json
+    sudo cp super.sample.json /config/super.json
+fi
+# Execute Command
+echo "Starting Shinobi ..."
+exec "$@"

Docker/pm2.yml

@@ -0,0 +1,7 @@
+apps:
+  - script   : '/home/Shinobi/camera.js'
+    name     : 'camera'
+    kill_timeout  : 5000
+  - script   : '/home/Shinobi/cron.js'
+    name     : 'cron'
+    kill_timeout  : 5000

Dockerfile

@@ -0,0 +1,108 @@
+FROM node:12.18.3-buster-slim
+
+ENV DB_USER=majesticflame \
+    DB_PASSWORD='' \
+    DB_HOST='localhost' \
+    DB_DATABASE=ccio \
+    DB_PORT=3306 \
+    SUBSCRIPTION_ID=sub_XXXXXXXXXXXX \
+    PLUGIN_KEYS='{}' \
+    SSL_ENABLED='false' \
+    SSL_COUNTRY='CA' \
+    SSL_STATE='BC' \
+    SSL_LOCATION='Vancouver' \
+    SSL_ORGANIZATION='Shinobi Systems' \
+    SSL_ORGANIZATION_UNIT='IT Department' \
+    SSL_COMMON_NAME='nvr.ninja' \
+    DB_DISABLE_INCLUDED=false
+ARG DEBIAN_FRONTEND=noninteractive
+
+RUN mkdir -p /home/Shinobi /config /var/lib/mysql
+
+RUN apt update -y
+RUN apt install wget curl net-tools -y
+
+# Install MariaDB server... the debian way
+RUN if [ "$DB_DISABLE_INCLUDED" = "false" ] ; then set -ex; \
+	{ \
+		echo "mariadb-server" mysql-server/root_password password '${DB_ROOT_PASSWORD}'; \
+		echo "mariadb-server" mysql-server/root_password_again password '${DB_ROOT_PASSWORD}'; \
+	} | debconf-set-selections; \
+	apt-get update; \
+	apt-get install -y \
+		"mariadb-server" \
+        socat \
+	; \
+    find /etc/mysql/ -name '*.cnf' -print0 \
+		| xargs -0 grep -lZE '^(bind-address|log)' \
+		| xargs -rt -0 sed -Ei 's/^(bind-address|log)/#&/'; fi
+
+RUN if [ "$DB_DISABLE_INCLUDED" = "false" ] ; then sed -ie "s/^bind-address\s*=\s*127\.0\.0\.1$/#bind-address = 0.0.0.0/" /etc/mysql/my.cnf; fi
+
+# Install FFmpeg
+
+RUN apt install -y software-properties-common \
+        libfreetype6-dev \
+        libgnutls28-dev \
+        libmp3lame-dev \
+        libass-dev \
+        libogg-dev \
+        libtheora-dev \
+        libvorbis-dev \
+        libvpx-dev \
+        libwebp-dev \
+        libssh2-1-dev \
+        libopus-dev \
+        librtmp-dev \
+        libx264-dev \
+        libx265-dev \
+        yasm && \
+    apt install -y \
+        build-essential \
+        bzip2 \
+        coreutils \
+        gnutls-bin \
+        nasm \
+        tar \
+        x264
+
+RUN apt install -y \
+                ffmpeg \
+                git \
+                make \
+                g++ \
+                gcc \
+                pkg-config \
+                python3 \
+                wget \
+                tar \
+                sudo \
+                xz-utils
+
+
+WORKDIR /home/Shinobi
+COPY . .
+RUN rm -rf /home/Shinobiplugins
+COPY ./plugins  /home/Shinobi/plugins
+RUN chmod -R 777 /home/Shinobi/plugins
+RUN npm i npm@latest -g && \
+    npm install pm2 -g && \
+    npm install --unsafe-perm && \
+    npm audit fix --force
+COPY ./Docker/pm2.yml ./
+
+# Copy default configuration files
+# COPY ./config/conf.json ./config/super.json /home/Shinobi/
+RUN chmod -f +x /home/Shinobi/Docker/init.sh
+
+VOLUME ["/home/Shinobi/videos"]
+VOLUME ["/home/Shinobi/plugins"]
+VOLUME ["/config"]
+VOLUME ["/customAutoLoad"]
+VOLUME ["/var/lib/mysql"]
+
+EXPOSE 8080
+
+ENTRYPOINT ["/home/Shinobi/Docker/init.sh"]
+
+CMD [ "pm2-docker", "pm2.yml" ]

INSTALL/cuda-10-2.sh

@@ -12,7 +12,7 @@ if [ -x "$(command -v apt)" ]; then
 
     sudo apt-get update -y
 
-    sudo apt-get -o Dpkg::Options::="--force-overwrite" install cuda -y --no-install-recommends
+    sudo apt-get -o Dpkg::Options::="--force-overwrite" install cuda-toolkit-10-2 -y --no-install-recommends
     sudo apt-get -o Dpkg::Options::="--force-overwrite" install --fix-broken -y
 
     # Install CUDA DNN

INSTALL/cuda.sh

@@ -12,7 +12,7 @@ if [ -x "$(command -v apt)" ]; then
 
     sudo apt-get update -y
 
-    sudo apt-get -o Dpkg::Options::="--force-overwrite" install cuda -y --no-install-recommends
+    sudo apt-get -o Dpkg::Options::="--force-overwrite" install cuda-toolkit-10-2 -y --no-install-recommends
     sudo apt-get -o Dpkg::Options::="--force-overwrite" install --fix-broken -y
 
     # Install CUDA DNN

README.md

@@ -1,146 +1,77 @@
-# Shinobi Pro 
+# Shinobi Pro
 ### (Shinobi Open Source Software)
 
-Shinobi is the Open Source CCTV Solution written in Node.JS. Designed with multiple account system, Streams by WebSocket, and Save to WebM. Shinobi can record IP Cameras and Local Cameras.
+Shinobi is the Open Source CCTV Solution written in Node.JS. Designed with multiple account system, Streams by WebSocket, and Direct saving to MP4. Shinobi can record IP Cameras and Local Cameras.
 
-<a href="http://shinobi.video/gallery"><img src="https://github.com/ShinobiCCTV/Shinobi/blob/master/web/libs/img/demo.jpg?raw=true"></a>
+## Install and Use
 
-# Key Aspects
+- Installation : http://shinobi.video/docs/start
+- Post-Installation Tutorials : http://shinobi.video/docs/configure
+- Troubleshooting Guide : https://hub.shinobi.video/articles/view/v0AFPFchfVcFGUS
 
-For an updated list of features visit the official website. http://shinobi.video/features
+#### Docker
+- Install with **Docker** : https://gitlab.com/Shinobi-Systems/Shinobi/-/tree/dev/Docker
 
-- Time-lapse Viewer (Watch a hours worth of footage in a few minutes)
-- 2-Factor Authentication
-- Defeats stream limit imposed by browsers
-  - With Base64 (Stream Type) and JPEG Mode (Option)
-- Records IP Cameras and Local Cameras
-- Streams by WebSocket, HLS (includes audio), and MJPEG
-- Save to WebM and MP4
-  - Can save Audio
-- Push Events - When a video is finished it will appear in the dashboard without a refresh
-- Region Motion Detection (Similar to ZoneMinder Zone Detection)
-  - Represented by a Motion Guage on each monitor
-- "No Motion" Notifications
-- 1 Process for Each Camera to do both, Recording and Streaming
-- Timeline for viewing Motion Events and Videos
-- Sub-Accounts with permissions
-  - Monitor Viewing
-  - Monitor Editing
-  - Video Deleting
-  - Separate API keys for sub account
-- Cron Filters can be set based on master account
-- Stream Analyzer built-in (FFprobe GUI)
-- Monitor Groups
-- Can snapshot images from stream directly
-- Lower Bandwith Mode (JPEG Mode)
-  - Snapshot (cgi-bin) must be enabled in Monitor Settings
-- Control Cameras from Interface
-- API
-  - Get videos
-  - Get monitors
-  - Change monitor modes : Disabled, Watch, Record
-  - Embedding streams
-- Dashboard Framework made with Google Material Design Lite, jQuery, and Bootstrap
+## "is my camera supported?"
+
+Ask yourself these questions to get a general sense.
+
+- Does it have ONVIF?
+    - If yes, then it may have H.264 or H.265 streaming capability.
+- Does it have RTSP Protocol for Streaming?
+    - If yes, then it may have H.264 or H.265 streaming capability.
+- Can you stream it in VLC Player?
+    - If yes, use that same URL in Shinobi. You may need to specify the port number when using `rtsp://` protocol.
+- Does it have MJPEG Streaming?
+    - While this would work in Shinobi, it is far from ideal. Please see if any of the prior questions are applicable.
+- Does it have a web interface that you can connect to directly?
+    - If yes, then you may be able to find model information that can be used to search online for a streaming URL.
+
+Configuration Guides : http://shinobi.video/docs/configure
 
 ## Asking for help
 
-Before asking questions it would nice if you read the docs :) http://shinobi.video
+- General Support : https://shinobi.community
+    - Please be sure to read the `#guidelines` channel after joining.
+- Business Inquiries : business@shinobi.video or the Live Chat on https://shinobi.video
 
-After doing so please head on over to the Discord community chat for support. https://discordapp.com/invite/mdhmvuH
+## Support the Development
 
-The Issues section is only for bugs with the software. Comments and feature requests may be closed without comment. http://shinobi.video/docs/contribute
+It's a proven fact that generosity makes you a happier person :) https://www.nature.com/articles/ncomms15964
 
-Please be considerate of developer efforts. If you have simple questions, like "what does this button do?", please be sure to have read the docs entirely before asking. If you would like to skip reading the docs and ask away you can order a support package :) http://shinobi.video/support
+Get a Mobile License to unlock extended features on the Mobile App as well as support the development!
+- Shinobi Mobile App : https://cdn.shinobi.video/installers/ShinobiMobile/
+- Get a Mobile License : https://licenses.shinobi.video/subscribe?planSubscribe=plan_G31AZ9mknNCa6z
 
-## Making Suggestions or Feature Requests
-
-You can post suggestions on the Forum in the Suggestions category. Please do not treat this channel like a "demands" window. Developer efforts are limited. Much more than many alternatives.
-
-when you have a suggestion please try and make the changes yourself then post a pull request to the `dev` branch. Then we can decide if it's a good change for Shinobi. If you don't know how to go about it and want to have me put it higher on my priority list you can order a support package :) Pretty Ferengi of me... but until we live in a world without money please support Shinobi :) Cheers!
-
-http://shinobi.video/support
-
-## Help make Shinobi the best Open Source CCTV Solution.
-Donate - http://shinobi.video/docs/donate
-
-Ordering a License, Paid Support, or anything from <a href="//camera.observer">here</a> will allow a lot more time to be spent on Shinobi.
-
-Order Support - http://shinobi.video/support
-
-# Why make this?
+## Why make this?
 
 http://shinobi.video/why
 
-# What others say
+## Author
 
-> "After trying zoneminder without success (heavy unstable and slow) I passed to Shinobi that despite being young spins a thousand times better (I have a setup with 16 cameras recording in FHD to ~ 10fps on a pentium of ~ 2009 and I turn with load below 1.5)."
+Moe Alam, Shinobi Systems
 
-> *A Reddit user, /r/ItalyInformatica*
+Shinobi is developed by many contributors. Please have a look at the commits to see some of who they are :)
+https://gitlab.com/Shinobi-Systems/Shinobi/-/commits/dev
 
-&nbsp;
-
-> "I would suggest Shinobi as a NVR. It's still in the early days but works a lot better than ZoneMinder for me. I'm able to record 16 cams at 1080p 15fps continously whith no load on server (Pentium E5500 3GB RAM) where zm crashed with 6 cams at 720p. Not to mention the better interface."
-
-> *A Reddit user, /r/HomeNetworking*
-
-# How to Install and Run
-
-> FOR DOCKER USERS : Docker is not officially supported and is not recommended. The kitematic method is provided for those who wish to quickly test Shinobi. The Docker files included in the master and dev branches are maintained by the community. If you would like support with Docker please find a community member who maintains the Docker files or please refer to Docker's forum.
-
-#### Fast Install (The Ninja Way)
-
-1. Become `root` to use the installer and run Shinobi. Use one of the following to do so.
-
-    - Ubuntu 17.04, 17.10
-        - `sudo su`
-    - CentOS 7
-        - `su`
-    - MacOS 10.7(+)
-        - `su`
-2. Download and run the installer.
-
-```
-bash <(curl -s https://gitlab.com/Shinobi-Systems/Shinobi-Installer/raw/master/shinobi-install.sh)
-```
-
-#### Elaborate Installs
-
-Installation Tutorials - http://shinobi.video/docs/start
-
-Troubleshooting Guide - http://shinobi.video/docs/start#trouble-section
-
-# Author
-
-Moe Alam
-
-Follow Shinobi on Twitter https://twitter.com/ShinobiCCTV
-
-Join the Community Chat
-
-<a title="Find me on Discord, Get an Invite" href="https://discordapp.com/invite/mdhmvuH"><img src="https://cdn-images-1.medium.com/max/115/1*OoXboCzk0gYvTNwNnV4S9A@2x.png"></a>
-
-# Support the Development
+## Support the Development
 
 Ordering a certificate or support package greatly boosts development. Please consider contributing :)
 
 http://shinobi.video/support
 
-# Links
+## Links
 
-Documentation - http://shinobi.video/docs
-
-Donate - https://shinobi.video/docs/donate
-
-Tested Cameras and Systems - http://shinobi.video/docs/supported
-
-Features - http://shinobi.video/features
-
-Reddit (Forum) - https://www.reddit.com/r/ShinobiCCTV/
-
-YouTube (Tutorials) - https://www.youtube.com/channel/UCbgbBLTK-koTyjOmOxA9msQ
-
-Discord (Community Chat) - https://discordapp.com/invite/mdhmvuH
-
-Twitter (News) - https://twitter.com/ShinobiCCTV
-
-Facebook (News) - https://www.facebook.com/Shinobi-1223193167773738/?ref=bookmarks
+- Articles : http://hub.shinobi.video/articles
+- Documentation : http://shinobi.video/docs
+- Features List : http://shinobi.video/features
+    - Some features may not be listed.
+- Donation : http://shinobi.video/docs/donate
+- Buy Shinobi Stuff : https://licenses.shinobi.video
+- User Submitted Configurations : http://shinobi.video/docs/cameras
+- Features : http://shinobi.video/features
+- Reddit (Forum) : https://www.reddit.com/r/ShinobiCCTV/
+- YouTube (Tutorials) : https://www.youtube.com/channel/UCbgbBLTK-koTyjOmOxA9msQ
+- Discord (Community Chat) : https://discordapp.com/invite/mdhmvuH
+- Twitter (News) : https://twitter.com/ShinobiCCTV
+- Facebook (News) : https://www.facebook.com/Shinobi-1223193167773738/?ref=bookmarks

camera.js

@@ -1,6 +1,6 @@
 //
 // Shinobi
-// Copyright (C) 2016 Moe Alam, moeiscool
+// Copyright (C) 2020 Moe Alam, moeiscool
 //
 //
 // # Donate
@@ -57,8 +57,6 @@ require('./libs/ffmpeg.js')(s,config,lang,function(ffmpeg){
     require('./libs/events.js')(s,config,lang)
     //recording functions
     require('./libs/videos.js')(s,config,lang)
-    //branding functions and config defaults
-    require('./libs/videoDropInServer.js')(s,config,lang,app,io)
     //plugins : websocket connected services..
     require('./libs/plugins.js')(s,config,lang,io)
     //health : cpu and ram trackers..

cron.js

@@ -135,6 +135,132 @@ s.sqlQuery = function(query,values,onMoveOn){
         }
     })
 }
+const processSimpleWhereCondition = (dbQuery,where,didOne) => {
+    var whereIsArray = where instanceof Array;
+    if(where[0] === 'or' || where.__separator === 'or'){
+        if(whereIsArray){
+            where.shift()
+            dbQuery.orWhere(...where)
+        }else{
+            where = cleanSqlWhereObject(where)
+            dbQuery.orWhere(where)
+        }
+    }else if(!didOne){
+        didOne = true
+        whereIsArray ? dbQuery.where(...where) : dbQuery.where(where)
+    }else{
+        whereIsArray ? dbQuery.andWhere(...where) : dbQuery.andWhere(where)
+    }
+}
+const processWhereCondition = (dbQuery,where,didOne) => {
+    var whereIsArray = where instanceof Array;
+    if(!where[0])return;
+    if(where[0] && where[0] instanceof Array){
+        dbQuery.where(function() {
+            var _this = this
+            var didOneInsideGroup = false
+            where.forEach((whereInsideGroup) => {
+                processWhereCondition(_this,whereInsideGroup,didOneInsideGroup)
+            })
+        })
+    }else if(where[0] && where[0] instanceof Object){
+        dbQuery.where(function() {
+            var _this = this
+            var didOneInsideGroup = false
+            where.forEach((whereInsideGroup) => {
+                processSimpleWhereCondition(_this,whereInsideGroup,didOneInsideGroup)
+            })
+        })
+    }else{
+        processSimpleWhereCondition(dbQuery,where,didOne)
+    }
+}
+const knexError = (dbQuery,options,err) => {
+    console.error('knexError----------------------------------- START')
+    if(config.debugLogVerbose && config.debugLog === true){
+        s.debugLog('s.knexQuery QUERY',JSON.stringify(options,null,3))
+        s.debugLog('STACK TRACE, NOT AN ',new Error())
+    }
+    console.error(err)
+    console.error(dbQuery.toString())
+    console.error('knexError----------------------------------- END')
+}
+const knexQuery = (options,callback) => {
+    try{
+        if(!s.databaseEngine)return// console.log('Database Not Set');
+        // options = {
+        //     action: "",
+        //     columns: "",
+        //     table: ""
+        // }
+        var dbQuery
+        switch(options.action){
+            case'select':
+                options.columns = options.columns.indexOf(',') === -1 ? [options.columns] : options.columns.split(',');
+                dbQuery = s.databaseEngine.select(...options.columns).from(options.table)
+            break;
+            case'count':
+                options.columns = options.columns.indexOf(',') === -1 ? [options.columns] : options.columns.split(',');
+                dbQuery = s.databaseEngine(options.table)
+                dbQuery.count(options.columns)
+            break;
+            case'update':
+                dbQuery = s.databaseEngine(options.table).update(options.update)
+            break;
+            case'delete':
+                dbQuery = s.databaseEngine(options.table)
+            break;
+            case'insert':
+                dbQuery = s.databaseEngine(options.table).insert(options.insert)
+            break;
+        }
+        if(options.where instanceof Array){
+            var didOne = false;
+            options.where.forEach((where) => {
+                processWhereCondition(dbQuery,where,didOne)
+            })
+        }else if(options.where instanceof Object){
+            dbQuery.where(options.where)
+        }
+        if(options.action === 'delete'){
+            dbQuery.del()
+        }
+        if(options.orderBy){
+            dbQuery.orderBy(...options.orderBy)
+        }
+        if(options.groupBy){
+            dbQuery.groupBy(options.groupBy)
+        }
+        if(options.limit){
+            if(`${options.limit}`.indexOf(',') === -1){
+                dbQuery.limit(options.limit)
+            }else{
+                const limitParts = `${options.limit}`.split(',')
+                dbQuery.limit(limitParts[0]).offset(limitParts[1])
+            }
+        }
+        if(config.debugLog === true){
+            console.log(dbQuery.toString())
+        }
+        if(callback || options.update || options.insert || options.action === 'delete'){
+            dbQuery.asCallback(function(err,r) {
+                if(err){
+                    knexError(dbQuery,options,err)
+                }
+                if(callback)callback(err,r)
+                if(config.debugLogVerbose && config.debugLog === true){
+                    s.debugLog('s.knexQuery QUERY',JSON.stringify(options,null,3))
+                    s.debugLog('s.knexQuery RESPONSE',JSON.stringify(r,null,3))
+                    s.debugLog('STACK TRACE, NOT AN ',new Error())
+                }
+            })
+        }
+        return dbQuery
+    }catch(err){
+        if(callback)callback(err,[])
+        knexError(dbQuery,options,err)
+    }
+}
 
 s.debugLog = function(arg1,arg2){
     if(config.debugLog === true){
@@ -236,29 +362,24 @@ const checkFilterRules = function(v,callback){
             var b = v.d.filters[m];
             s.debugLog(b)
             if(b.enabled==="1"){
-                b.ar=[v.ke];
-                b.sql=[];
-                b.where.forEach(function(j,k){
-                    if(j.p1==='ke'){j.p3=v.ke}
-                    switch(j.p3_type){
-                        case'function':
-                            b.sql.push(j.p1+' '+j.p2+' '+j.p3)
-                        break;
-                        default:
-                            b.sql.push(j.p1+' '+j.p2+' ?')
-                            b.ar.push(j.p3)
-                        break;
-                    }
+                const whereQuery = [
+                    ['ke','=',v.ke],
+                    ['status','!=',"0"],
+                    ['details','NOT LIKE','%"archived":"1"%'],
+                ]
+                b.where.forEach(function(condition){
+                    if(condition.p1 === 'ke'){condition.p3 = v.ke}
+                    whereQuery.push([condition.p1,condition.p2 || '=',condition.p3])
                 })
-                b.sql='WHERE ke=? AND status != 0 AND details NOT LIKE \'%"archived":"1"%\' AND ('+b.sql.join(' AND ')+')';
-                if(b.sort_by&&b.sort_by!==''){
-                    b.sql+=' ORDER BY `'+b.sort_by+'` '+b.sort_by_direction
-                }
-                if(b.limit&&b.limit!==''){
-                    b.sql+=' LIMIT '+b.limit
-                }
-                s.sqlQuery('SELECT * FROM Videos '+b.sql,b.ar,function(err,r){
-                     if(r&&r[0]){
+                knexQuery({
+                    action: "select",
+                    columns: "*",
+                    table: "Videos",
+                    where: whereQuery,
+                    orderBy: [b.sort_by,b.sort_by_direction.toLowerCase()],
+                    limit: b.limit
+                },(err,r) => {
+                     if(r && r[0]){
                         if(r.length > 0 || config.debugLog === true){
                             s.cx({f:'filterMatch',msg:r.length+' SQL rows match "'+m+'"',ke:v.ke,time:moment()})
                         }
@@ -309,10 +430,19 @@ const deleteRowsWithNoVideo = function(v,callback){
         )
     ){
         s.alreadyDeletedRowsWithNoVideosOnStart[v.ke]=true;
-        es={};
-        s.sqlQuery('SELECT * FROM Videos WHERE ke=? AND status!=0 AND details NOT LIKE \'%"archived":"1"%\' AND time < ?',[v.ke,s.sqlDate('10 MINUTE')],function(err,evs){
-            if(evs&&evs[0]){
-                es.del=[];es.ar=[v.ke];
+        knexQuery({
+            action: "select",
+            columns: "*",
+            table: "Videos",
+            where: [
+                ['ke','=',v.ke],
+                ['status','!=','0'],
+                ['details','NOT LIKE','%"archived":"1"%'],
+                ['time','<',s.sqlDate('10 MINUTE')],
+            ]
+        },(err,evs) => {
+            if(evs && evs[0]){
+                const videosToDelete = [];
                 evs.forEach(function(ev){
                     var filename
                     var details
@@ -337,8 +467,8 @@ const deleteRowsWithNoVideo = function(v,callback){
                         s.tx({f:'video_delete',filename:filename+'.'+ev.ext,mid:ev.mid,ke:ev.ke,time:ev.time,end:s.moment(new Date,'YYYY-MM-DD HH:mm:ss')},'GRP_'+ev.ke);
                     }
                 });
-                if(es.del.length>0 || config.debugLog === true){
-                    s.cx({f:'deleteNoVideo',msg:es.del.length+' SQL rows with no file deleted',ke:v.ke,time:moment()})
+                if(videosToDelete.length>0 || config.debugLog === true){
+                    s.cx({f:'deleteNoVideo',msg:videosToDelete.length+' SQL rows with no file deleted',ke:v.ke,time:moment()})
                 }
             }
             setTimeout(function(){
@@ -353,7 +483,14 @@ const deleteRowsWithNoVideo = function(v,callback){
 const deleteOldLogs = function(v,callback){
     if(!v.d.log_days||v.d.log_days==''){v.d.log_days=10}else{v.d.log_days=parseFloat(v.d.log_days)};
     if(config.cron.deleteLogs===true&&v.d.log_days!==0){
-        s.sqlQuery("DELETE FROM Logs WHERE ke=? AND `time` < ?",[v.ke,s.sqlDate(v.d.log_days+' DAY')],function(err,rrr){
+        knexQuery({
+            action: "delete",
+            table: "Logs",
+            where: [
+                ['ke','=',v.ke],
+                ['time','<',s.sqlDate(v.d.log_days+' DAY')],
+            ]
+        },(err,rrr) => {
             callback()
             if(err)return console.error(err);
             if(rrr.affectedRows && rrr.affectedRows.length>0 || config.debugLog === true){
@@ -368,7 +505,14 @@ const deleteOldLogs = function(v,callback){
 const deleteOldEvents = function(v,callback){
     if(!v.d.event_days||v.d.event_days==''){v.d.event_days=10}else{v.d.event_days=parseFloat(v.d.event_days)};
     if(config.cron.deleteEvents===true&&v.d.event_days!==0){
-        s.sqlQuery("DELETE FROM Events WHERE ke=? AND `time` < ?",[v.ke,s.sqlDate(v.d.event_days+' DAY')],function(err,rrr){
+        knexQuery({
+            action: "delete",
+            table: "Events",
+            where: [
+                ['ke','=',v.ke],
+                ['time','<',s.sqlDate(v.d.event_days+' DAY')],
+            ]
+        },(err,rrr) => {
             callback()
             if(err)return console.error(err);
             if(rrr.affectedRows && rrr.affectedRows.length > 0 || config.debugLog === true){
@@ -383,7 +527,14 @@ const deleteOldEvents = function(v,callback){
 const deleteOldEventCounts = function(v,callback){
     if(!v.d.event_days||v.d.event_days==''){v.d.event_days=10}else{v.d.event_days=parseFloat(v.d.event_days)};
     if(config.cron.deleteEvents===true&&v.d.event_days!==0){
-        s.sqlQuery("DELETE FROM `Events Counts` WHERE ke=? AND `time` < ?",[v.ke,s.sqlDate(v.d.event_days+' DAY')],function(err,rrr){
+        knexQuery({
+            action: "delete",
+            table: "Events Counts",
+            where: [
+                ['ke','=',v.ke],
+                ['time','<',s.sqlDate(v.d.event_days+' DAY')],
+            ]
+        },(err,rrr) => {
             callback()
             if(err && err.code !== 'ER_NO_SUCH_TABLE')return console.error(err);
             if(rrr.affectedRows && rrr.affectedRows.length > 0 || config.debugLog === true){
@@ -399,7 +550,15 @@ const deleteOldFileBins = function(v,callback){
     if(!v.d.fileBin_days||v.d.fileBin_days==''){v.d.fileBin_days=10}else{v.d.fileBin_days=parseFloat(v.d.fileBin_days)};
     if(config.cron.deleteFileBins===true&&v.d.fileBin_days!==0){
         var fileBinQuery = " FROM Files WHERE ke=? AND `time` < ?";
-        s.sqlQuery("SELECT *"+fileBinQuery,[v.ke,s.sqlDate(v.d.fileBin_days+' DAY')],function(err,files){
+        knexQuery({
+            action: "select",
+            columns: "*",
+            table: "Files",
+            where: [
+                ['ke','=',v.ke],
+                ['time','<',s.sqlDate(v.d.fileBin_days+' DAY')],
+            ]
+        },(err,files) => {
             if(files&&files[0]){
                 //delete the files
                 files.forEach(function(file){
@@ -408,7 +567,14 @@ const deleteOldFileBins = function(v,callback){
                     })
                 })
                 //delete the database rows
-                s.sqlQuery("DELETE"+fileBinQuery,[v.ke,v.d.fileBin_days],function(err,rrr){
+                knexQuery({
+                    action: "delete",
+                    table: "Files",
+                    where: [
+                        ['ke','=',v.ke],
+                        ['time','<',s.sqlDate(v.d.fileBin_days+' DAY')],
+                    ]
+                },(err,rrr) => {
                     callback()
                     if(err)return console.error(err);
                     if(rrr.affectedRows && rrr.affectedRows.length>0 || config.debugLog === true){
@@ -451,7 +617,14 @@ const processUser = function(number,rows){
         if(!v.d.size||v.d.size==''){v.d.size=10000}else{v.d.size=parseFloat(v.d.size)};
         //days to keep videos
         if(!v.d.days||v.d.days==''){v.d.days=5}else{v.d.days=parseFloat(v.d.days)};
-        s.sqlQuery('SELECT * FROM Monitors WHERE ke=?', [v.ke], function(err,rr) {
+        knexQuery({
+            action: "select",
+            columns: "*",
+            table: "Monitors",
+            where: [
+                ['ke','=',v.ke],
+            ]
+        },(err,rr) => {
             if(!v.d.filters||v.d.filters==''){
                 v.d.filters={};
             }
@@ -522,7 +695,14 @@ const clearCronInterval = function(){
 }
 const doCronJobs = function(){
     s.cx({f:'start',time:moment()})
-    s.sqlQuery('SELECT ke,uid,details,mail FROM Users WHERE details NOT LIKE \'%"sub"%\'', function(err,rows) {
+    knexQuery({
+        action: "select",
+        columns: "ke,uid,details,mail",
+        table: "Users",
+        where: [
+            ['details','NOT LIKE','%"sub"%'],
+        ]
+    },(err,rows) => {
         if(err){
             console.error(err)
         }

languages/en_CA.json

@@ -199,6 +199,7 @@
    "File Type": "File Type",
    "Filesize": "Filesize",
    "Video Status": "Video Status",
+   "Custom Auto Load": "Custom Auto Load",
    "Preferences": "Preferences",
    "Equal to": "Equal to",
    "Not Equal to": "Not Equal to",
@@ -481,7 +482,15 @@
    "StreamText": "<p>This section will designate the primary method of streaming out and its settings. This stream will be displayed in the dashboard. If you choose to use HLS, JPEG, or MJPEG then you can consume the stream through other programs.</p><p class=\"h_st_input h_st_jpeg\">Using JPEG stream essentially turns off the primary stream and uses the snapshot bin to get frames.</p>",
    "DetectorText": "<p>When the Width and Height boxes are shown you should set them to 640x480 or below. This will optimize the read speed of frames.</p>",
    "RecordingText": "It is recommended that you set <b>Record File Type</b> to <b class=\"h_t_input h_t_jpeg h_t_socket\">WebM</b><b class=\"h_t_input h_t_mjpeg h_t_h264 h_t_hls h_t_mp4 h_t_local\">MP4</b> and <b>Video Codec</b> to <b class=\"h_t_input h_t_jpeg h_t_socket\">libvpx</b><b class=\"h_t_input h_t_h264 h_t_hls h_t_mp4\">copy or </b><b class=\"h_t_input h_t_mjpeg h_t_h264 h_t_hls h_t_mp4 h_t_local\">libx264</b> because your <b>Input Type</b> is set to <b class=\"h_t_text\"></b>.",
+   "'Already Installing...'": "'Already Installing...'",
+   "Time Created": "Time Created",
+   "Last Modified": "Last Modified",
    "Mode": "Mode",
+   "Run Installer": "Run Installer",
+   "Install": "Install",
+   "Enable": "Enable",
+   "Disable": "Disable",
+   "Delete": "Delete",
    "Name": "Name",
    "Skip Ping": "Skip Ping",
    "Retry Connection": "Retry Connection <small>Number of times allowed to fail</small>",
@@ -578,6 +587,7 @@
    "Attach Video Clip": "Attach Video Clip",
    "Error While Decoding": "Error While Decoding",
    "ErrorWhileDecodingText": "Your hardware may have an unstable connection to the network. Check your network connections.",
+   "ErrorWhileDecodingTextAudio": "Your camera is providing broken data. Try disabling the Audio in the camera's internal settings.",
    "Discord": "Discord",
    "Discord Alert on Trigger": "Discord Alert on Trigger",
    "Allow Next Email": "Allow Next Email <small>in Minutes</small>",
@@ -929,6 +939,8 @@
    "vda": "vda (Apple VDA Hardware Acceleration)",
    "videotoolbox": "videotoolbox",
    "cuvid": "cuvid (NVIDIA NVENC)",
+   "cuda": "cuda (NVIDIA NVENC)",
+   "opencl": "OpenCL",
    "Main": "Main",
    "Storage Location": "Storage Location",
    "Recommended": "Recommended",

libs/auth.js

@@ -8,14 +8,30 @@ module.exports = function(s,config,lang){
     //
     var getUserByUid = function(params,columns,callback){
         if(!columns)columns = '*'
-        s.sqlQuery(`SELECT ${columns} FROM Users WHERE uid=? AND ke=?`,[params.uid,params.ke],function(err,r){
+        s.knexQuery({
+            action: "select",
+            columns: columns,
+            table: "Users",
+            where: [
+                ['uid','=',params.uid],
+                ['ke','=',params.ke],
+            ]
+        },(err,r) => {
             if(!r)r = []
             var user = r[0]
             callback(err,user)
         })
     }
     var getUserBySessionKey = function(params,callback){
-        s.sqlQuery('SELECT * FROM Users WHERE auth=? AND ke=?',[params.auth,params.ke],function(err,r){
+        s.knexQuery({
+            action: "select",
+            columns: '*',
+            table: "Users",
+            where: [
+                ['auth','=',params.auth],
+                ['ke','=',params.ke],
+            ]
+        },(err,r) => {
             if(!r)r = []
             var user = r[0]
             callback(err,user)
@@ -23,7 +39,18 @@ module.exports = function(s,config,lang){
     }
     var loginWithUsernameAndPassword = function(params,columns,callback){
         if(!columns)columns = '*'
-        s.sqlQuery(`SELECT ${columns} FROM Users WHERE mail=? AND (pass=? OR pass=?) LIMIT 1`,[params.username,params.password,s.createHash(params.password)],function(err,r){
+        s.knexQuery({
+            action: "select",
+            columns: columns,
+            table: "Users",
+            where: [
+                ['mail','=',params.username],
+                ['pass','=',params.password],
+                ['or','mail','=',params.username],
+                ['pass','=',s.createHash(params.password)],
+            ],
+            limit: 1
+        },(err,r) => {
             if(!r)r = []
             var user = r[0]
             callback(err,user)
@@ -31,7 +58,15 @@ module.exports = function(s,config,lang){
     }
     var getApiKey = function(params,columns,callback){
         if(!columns)columns = '*'
-        s.sqlQuery(`SELECT ${columns} FROM API WHERE code=? AND ke=?`,[params.auth,params.ke],function(err,r){
+        s.knexQuery({
+            action: "select",
+            columns: columns,
+            table: "API",
+            where: [
+                ['code','=',params.auth],
+                ['ke','=',params.ke],
+            ]
+        },(err,r) => {
             if(!r)r = []
             var apiKey = r[0]
             callback(err,apiKey)
@@ -226,7 +261,14 @@ module.exports = function(s,config,lang){
             }
             var foundUser = function(){
                 if(params.users === true){
-                    s.sqlQuery('SELECT * FROM Users WHERE details NOT LIKE ?',['%"sub"%'],function(err,r) {
+                    s.knexQuery({
+                        action: "select",
+                        columns: "*",
+                        table: "Users",
+                        where: [
+                            ['details','NOT LIKE','%"sub"%'],
+                        ]
+                    },(err,r) => {
                         adminUsersSelected = r
                         success()
                     })

libs/basic.js

@@ -214,20 +214,27 @@ module.exports = function(s,config){
         if(!e){e=''}
         if(config.systemLog===true){
             if(typeof q==='string'&&s.databaseEngine){
-                s.sqlQuery('INSERT INTO Logs (ke,mid,info) VALUES (?,?,?)',['$','$SYSTEM',s.s({type:q,msg:w})]);
+                s.knexQuery({
+                    action: "insert",
+                    table: "Logs",
+                    insert: {
+                        ke: '$',
+                        mid: '$SYSTEM',
+                        info: s.s({type:q,msg:w}),
+                    }
+                })
                 s.tx({f:'log',log:{time:s.timeObject(),ke:'$',mid:'$SYSTEM',time:s.timeObject(),info:s.s({type:q,msg:w})}},'$');
             }
             return console.log(s.timeObject().format(),q,w,e)
         }
     }
     //system log
-    s.debugLog = function(q,w,e){
+    s.debugLog = function(...args){
         if(config.debugLog === true){
-            if(!w){w = ''}
-            if(!e){e = ''}
-            console.log(s.timeObject().format(),q,w,e)
+            var logRow = ([s.timeObject().format()]).concat(...args)
+            console.log(...logRow)
             if(config.debugLogVerbose === true){
-                console.log(new Error())
+                console.log(new Error('VERBOSE STACK TRACE, THIS IS NOT AN '))
             }
         }
     }

libs/cameraThread/singleCamera.js

@@ -18,7 +18,7 @@ var stdioWriters = [];
 
 var writeToStderr = function(text){
   try{
-    stdioWriters[2].write(Buffer.from(`${text}`, 'utf8' ))
+    process.stderr.write(Buffer.from(`${text}`, 'utf8' ))
       // stdioWriters[2].write(Buffer.from(`${new Error('writeToStderr').stack}`, 'utf8' ))
   }catch(err){
     // fs.appendFileSync('/home/Shinobi/test.log',text + '\n','utf8')
@@ -45,10 +45,14 @@ process.on('uncaughtException', function (err) {
     writeToStderr(err.stack);
 });
 const exitAction = function(){
-    if(isWindows){
-        spawn("taskkill", ["/pid", cameraProcess.pid, '/f', '/t'])
-    }else{
-        process.kill(-cameraProcess.pid)
+    try{
+        if(isWindows){
+            spawn("taskkill", ["/pid", cameraProcess.pid, '/f', '/t'])
+        }else{
+            process.kill(-cameraProcess.pid)
+        }
+    }catch(err){
+
     }
 }
 process.on('SIGTERM', exitAction);
@@ -58,7 +62,7 @@ process.on('exit', exitAction);
 for(var i=0; i < stdioPipes; i++){
     switch(i){
       case 0:
-        newPipes[i] = null
+        newPipes[i] = 'pipe'
       break;
       case 1:
         newPipes[i] = 1
@@ -115,7 +119,7 @@ if(rawMonitorConfig.details.detector === '1' && rawMonitorConfig.details.detecto
     writeToStderr(err.stack)
   }
 }
- 
+
 if(rawMonitorConfig.type === 'jpeg'){
     var recordingSnapRequest
     var recordingSnapper
@@ -197,3 +201,13 @@ if(rawMonitorConfig.type === 'jpeg'){
         captureOne()
     },5000)
 }
+
+if(
+    rawMonitorConfig.type === 'dashcam' ||
+    rawMonitorConfig.type === 'socket'
+){
+    process.stdin.on('data',(data) => {
+        //confirmed receiving data this way.
+        cameraProcess.stdin.write(data)
+    })
+}

libs/cameraThread/snapshot.js

@@ -1,7 +1,9 @@
 const fs = require('fs')
 const spawn = require('child_process').spawn
+const isWindows = process.platform === "win32";
 var writeToStderr = function(text){
   // fs.appendFileSync(rawMonitorConfig.sdir + 'errors.log',text + '\n','utf8')
+  process.stderr.write(Buffer.from(`${text}`, 'utf8' ))
 }
 if(!process.argv[2] || !process.argv[3]){
     return writeToStderr('Missing FFMPEG Command String or no command operator')
@@ -16,7 +18,11 @@ const exitAction = function(){
     if(isWindows){
         spawn("taskkill", ["/pid", snapProcess.pid, '/f', '/t'])
     }else{
-        process.kill(-snapProcess.pid)
+        try{
+            process.kill(-snapProcess.pid)
+        }catch(err){
+
+        }
     }
 }
 process.on('SIGTERM', exitAction);
@@ -31,7 +37,6 @@ const temporaryImageFile = jsonData.temporaryImageFile
 const iconImageFile = jsonData.iconImageFile
 const useIcon = jsonData.useIcon
 const rawMonitorConfig = jsonData.rawMonitorConfig
-
 // var writeToStderr = function(text){
 //   process.stderr.write(Buffer.from(text))
 // }
@@ -44,7 +49,7 @@ snapProcess.stdout.on('data',(data)=>{
   writeToStderr(data.toString())
 })
 snapProcess.on('close',function(data){
-  if(useIcon === true){
+  if(useIcon){
     var fileCopy = fs.createReadStream(temporaryImageFile).pipe(fs.createWriteStream(iconImageFile))
     fileCopy.on('close',function(){
         process.exit();

libs/childNode.js

@@ -3,6 +3,7 @@ var http = require('http');
 var https = require('https');
 var express = require('express');
 module.exports = function(s,config,lang,app,io){
+    const { cameraDestroy } = require('./monitor/utils.js')(s,config,lang)
     //setup Master for childNodes
     if(config.childNodes.enabled === true && config.childNodes.mode === 'master'){
         s.childNodes = {};
@@ -66,6 +67,11 @@ module.exports = function(s,config,lang,app,io){
                                     cn.emit('c',{f:'sqlCallback',rows:rows,err:err,callbackId:d.callbackId});
                                 });
                             break;
+                            case'knex':
+                                s.knexQuery(d.options,function(err,rows){
+                                    cn.emit('c',{f:'sqlCallback',rows:rows,err:err,callbackId:d.callbackId});
+                                });
+                            break;
                             case'clearCameraFromActiveList':
                                 if(s.childNodes[ipAddress])delete(s.childNodes[ipAddress].activeCameras[d.ke + d.id])
                             break;
@@ -152,9 +158,9 @@ module.exports = function(s,config,lang,app,io){
                                     extender(d.d,insert)
                                 })
                                 //purge over max
-                                s.purgeDiskForGroup(d)
+                                s.purgeDiskForGroup(d.ke)
                                 //send new diskUsage values
-                                s.setDiskUsedForGroup(d,insert.filesizeMB)
+                                s.setDiskUsedForGroup(d.ke,insert.filesizeMB)
                                 clearTimeout(s.group[d.ke].activeMonitors[d.mid].recordingChecker)
                                 clearTimeout(s.group[d.ke].activeMonitors[d.mid].streamChecker)
                             break;
@@ -213,11 +219,19 @@ module.exports = function(s,config,lang,app,io){
             s.queuedSqlCallbacks[callbackId] = onMoveOn
             s.cx({f:'sql',query:query,values:values,callbackId:callbackId});
         }
-        setInterval(function(){
-            s.cpuUsage(function(cpu){
-                s.cx({f:'cpu',cpu:parseFloat(cpu)})
+        s.knexQuery = function(options,onMoveOn){
+            var callbackId = s.gid()
+            if(typeof onMoveOn !== 'function'){onMoveOn=function(){}}
+            s.queuedSqlCallbacks[callbackId] = onMoveOn
+            s.cx({f:'knex',options:options,callbackId:callbackId});
+        }
+        setInterval(async () => {
+            const cpu = await s.cpuUsage()
+            s.cx({
+                f: 'cpu',
+                cpu: parseFloat(cpu)
             })
-        },2000)
+        },5000)
         childIO.on('connect', function(d){
             console.log('CHILD CONNECTION SUCCESS')
             s.cx({
@@ -241,7 +255,7 @@ module.exports = function(s,config,lang,app,io){
                 break;
                 case'kill':
                     s.initiateMonitorObject(d.d);
-                    s.cameraDestroy(s.group[d.d.ke].activeMonitors[d.d.id].spawn,d.d)
+                    cameraDestroy(d.d)
                     var childNodeIp = s.group[d.d.ke].activeMonitors[d.d.id]
                 break;
                 case'sync':

libs/common.js

@@ -0,0 +1,11 @@
+const async = require("async");
+exports.copyObject = (obj) => {
+  return Object.assign({},obj)
+}
+exports.createQueue = (timeoutInSeconds, queueItemsRunningInParallel) => {
+    return async.queue(function(action, callback) {
+        setTimeout(function(){
+            action(callback)
+        },timeoutInSeconds * 1000 || 1000)
+    },queueItemsRunningInParallel || 3)
+}

libs/control/onvif.js

@@ -9,7 +9,7 @@ module.exports = function(s,config,lang,app,io){
         const controlURLOptions = s.cameraControlOptionsFromUrl(controlBaseUrl,monitorConfig)
         //create onvif connection
         const device = new onvif.OnvifDevice({
-            xaddr : 'http://' + controlURLOptions.host + ':' + controlURLOptions.port + '/onvif/device_service',
+            address : controlURLOptions.host + ':' + controlURLOptions.port,
             user : controlURLOptions.username,
             pass : controlURLOptions.password
         })

libs/control/ptz.js

@@ -1,5 +1,6 @@
 var os = require('os');
 var exec = require('child_process').exec;
+var request = require('request')
 module.exports = function(s,config,lang,app,io){
     const moveLock = {}
     const startMove = async function(options,callback){
@@ -195,24 +196,28 @@ module.exports = function(s,config,lang,app,io){
         }else{
             const controlUrlStopTimeout = parseInt(monitorConfig.details.control_url_stop_timeout) || 1000
             var stopCamera = function(){
-                var stopURL = controlBaseUrl + monitorConfig.details[`control_url_${options.direction}_stop`]
-                var options = s.cameraControlOptionsFromUrl(stopURL,monitorConfig)
-                var requestOptions = {
+                let stopURL = controlBaseUrl + monitorConfig.details[`control_url_${options.direction}_stop`]
+                let controlOptions = s.cameraControlOptionsFromUrl(stopURL,monitorConfig)
+                let requestOptions = {
                     url : stopURL,
-                    method : options.method,
+                    method : controlOptions.method,
                     auth : {
-                        user : options.username,
-                        pass : options.password
+                        user : controlOptions.username,
+                        pass : controlOptions.password
                     }
                 }
                 if(monitorConfig.details.control_digest_auth === '1'){
                     requestOptions.sendImmediately = true
                 }
                 request(requestOptions,function(err,data){
+                    const msg =  {
+                        ok: true,
+                        type:'Control Trigger Ended'
+                    }
                     if(err){
-                        var msg =  {ok:false,type:'Control Error',msg:err}
-                    }else{
-                        var msg =  {ok:true,type:'Control Trigger Ended'}
+                        msg.ok = false
+                        msg.type = 'Control Error'
+                        msg.msg = err
                     }
                     callback(msg)
                     s.userLog(e,msg);
@@ -221,12 +226,14 @@ module.exports = function(s,config,lang,app,io){
             if(options.direction === 'stopMove'){
                 stopCamera()
             }else{
-                var requestOptions = {
+                let controlURL = controlBaseUrl + monitorConfig.details[`control_url_${options.direction}`]
+                let controlOptions = s.cameraControlOptionsFromUrl(controlURL,monitorConfig)
+                let requestOptions = {
                     url: controlURL,
-                    method: controlURLOptions.method,
+                    method: controlOptions.method,
                     auth: {
-                        user: controlURLOptions.username,
-                        pass: controlURLOptions.password
+                        user: controlOptions.username,
+                        pass: controlOptions.password
                     }
                 }
                 if(monitorConfig.details.control_digest_auth === '1'){

libs/customAutoLoad.js

@@ -1,153 +1,434 @@
-var fs = require('fs')
-var express = require('express')
-module.exports = function(s,config,lang,app,io){
-
-    s.customAutoLoadModules = {}
-    s.customAutoLoadTree = {
-        pages: [],
-        PageBlocks: [],
-        LibsJs: [],
-        LibsCss: [],
-        adminPageBlocks: [],
-        adminLibsJs: [],
-        adminLibsCss: [],
-        superPageBlocks: [],
-        superLibsJs: [],
-        superLibsCss: []
+const fs = require('fs-extra');
+const express = require('express')
+const request = require('request')
+const unzipper = require('unzipper')
+const fetch = require("node-fetch")
+const spawn = require('child_process').spawn
+module.exports = async (s,config,lang,app,io) => {
+    const runningInstallProcesses = {}
+    const modulesBasePath = s.mainDirectory + '/libs/customAutoLoad/'
+    const searchText = function(searchFor,searchIn){
+        return searchIn.indexOf(searchFor) > -1
     }
-    var folderPath = s.mainDirectory + '/libs/customAutoLoad'
-    var search = function(searchFor,searchIn){return searchIn.indexOf(searchFor) > -1}
-    fs.readdir(folderPath,function(err,folderContents){
-        if(!err && folderContents){
-            folderContents.forEach(function(filename){
-                s.customAutoLoadModules[filename] = {}
-                var customModulePath = folderPath + '/' + filename
-                if(filename.indexOf('.js') > -1){
-                    s.customAutoLoadModules[filename].type = 'file'
-                    try{
-                        require(customModulePath)(s,config,lang,app,io)
-                    }catch(err){
-                        console.log('Failed to Load Module : ' + filename)
-                        console.log(err)
-                    }
-                }else{
-                    if(fs.lstatSync(customModulePath).isDirectory()){
-                        s.customAutoLoadModules[filename].type = 'folder'
-                        try{
-                            require(customModulePath)(s,config,lang,app,io)
-                            fs.readdir(customModulePath,function(err,folderContents){
-                                folderContents.forEach(function(name){
-                                    switch(name){
-                                        case'web':
-                                            var webFolder = s.checkCorrectPathEnding(customModulePath) + 'web/'
-                                            fs.readdir(webFolder,function(err,webFolderContents){
-                                                webFolderContents.forEach(function(name){
-                                                    switch(name){
-                                                        case'libs':
-                                                        case'pages':
-                                                            if(name === 'libs'){
-                                                                if(config.webPaths.home !== '/'){
-                                                                    app.use('/libs',express.static(webFolder + '/libs'))
-                                                                }
-                                                                app.use(s.checkCorrectPathEnding(config.webPaths.home)+'libs',express.static(webFolder + '/libs'))
-                                                                app.use(s.checkCorrectPathEnding(config.webPaths.admin)+'libs',express.static(webFolder + '/libs'))
-                                                                app.use(s.checkCorrectPathEnding(config.webPaths.super)+'libs',express.static(webFolder + '/libs'))
-                                                            }
-                                                            var libFolder = webFolder + name + '/'
-                                                            fs.readdir(libFolder,function(err,webFolderContents){
-                                                                webFolderContents.forEach(function(libName){
-                                                                    var thirdLevelName = libFolder + libName
-                                                                    switch(libName){
-                                                                        case'js':
-                                                                        case'css':
-                                                                        case'blocks':
-                                                                            fs.readdir(thirdLevelName,function(err,webFolderContents){
-                                                                                webFolderContents.forEach(function(filename){
-                                                                                    var fullPath = thirdLevelName + '/' + filename
-                                                                                    var blockPrefix = ''
-                                                                                    switch(true){
-                                                                                        case search('super.',filename):
-                                                                                            blockPrefix = 'super'
-                                                                                        break;
-                                                                                        case search('admin.',filename):
-                                                                                            blockPrefix = 'admin'
-                                                                                        break;
-                                                                                    }
-                                                                                    switch(libName){
-                                                                                        case'js':
-                                                                                            s.customAutoLoadTree[blockPrefix + 'LibsJs'].push(filename)
-                                                                                        break;
-                                                                                        case'css':
-                                                                                            s.customAutoLoadTree[blockPrefix + 'LibsCss'].push(filename)
-                                                                                        break;
-                                                                                        case'blocks':
-                                                                                            s.customAutoLoadTree[blockPrefix + 'PageBlocks'].push(fullPath)
-                                                                                        break;
-                                                                                    }
-                                                                                })
-                                                                            })
-                                                                        break;
-                                                                        default:
-                                                                            if(libName.indexOf('.ejs') > -1){
-                                                                                s.customAutoLoadTree.pages.push(thirdLevelName)
-                                                                            }
-                                                                        break;
-                                                                    }
-                                                                })
-                                                            })
-                                                        break;
-                                                    }
-                                                })
-                                            })
-                                        break;
-                                        case'languages':
-                                            var languagesFolder = s.checkCorrectPathEnding(customModulePath) + 'languages/'
-                                            fs.readdir(languagesFolder,function(err,files){
-                                                if(err)return console.log(err);
-                                                files.forEach(function(filename){
-                                                    var fileData = require(languagesFolder + filename)
-                                                    var rule = filename.replace('.json','')
-                                                    if(config.language === rule){
-                                                        lang = Object.assign(lang,fileData)
-                                                    }
-                                                    if(s.loadedLanguages[rule]){
-                                                        s.loadedLanguages[rule] = Object.assign(s.loadedLanguages[rule],fileData)
-                                                    }else{
-                                                        s.loadedLanguages[rule] = Object.assign(s.copySystemDefaultLanguage(),fileData)
-                                                    }
-                                                })
-                                            })
-                                        break;
-                                        case'definitions':
-                                            var definitionsFolder = s.checkCorrectPathEnding(customModulePath) + 'definitions/'
-                                            fs.readdir(definitionsFolder,function(err,files){
-                                                if(err)return console.log(err);
-                                                files.forEach(function(filename){
-                                                    var fileData = require(definitionsFolder + filename)
-                                                    var rule = filename.replace('.json','').replace('.js','')
-                                                    if(config.language === rule){
-                                                        s.definitions = s.mergeDeep(s.definitions,fileData)
-                                                    }
-                                                    if(s.loadedDefinitons[rule]){
-                                                        s.loadedDefinitons[rule] = s.mergeDeep(s.loadedDefinitons[rule],fileData)
-                                                    }else{
-                                                        s.loadedDefinitons[rule] = s.mergeDeep(s.copySystemDefaultDefinitions(),fileData)
-                                                    }
-                                                })
-                                            })
-                                        break;
-                                    }
-                                })
+    const extractNameFromPackage = (filePath) => {
+        const filePathParts = filePath.split('/')
+        const packageName = filePathParts[filePathParts.length - 1].split('.')[0]
+        return packageName
+    }
+    const getModulePath = (name) => {
+        return modulesBasePath + name + '/'
+    }
+    const getModule = (moduleName) => {
+        const modulePath = modulesBasePath + moduleName
+        const stats = fs.lstatSync(modulePath)
+        const isDirectory = stats.isDirectory()
+        const newModule = {
+            name: moduleName,
+            path: modulePath + '/',
+            size: stats.size,
+            lastModified: stats.mtime,
+            created: stats.ctime,
+            isDirectory: isDirectory,
+        }
+        if(isDirectory){
+            var hasInstaller = false
+            if(!fs.existsSync(modulePath + '/index.js')){
+                hasInstaller = true
+                newModule.noIndex = true
+            }
+            if(fs.existsSync(modulePath + '/package.json')){
+                hasInstaller = true
+                newModule.properties = getModuleProperties(moduleName)
+            }else{
+                newModule.properties = {
+                    name: moduleName
+                }
+            }
+            newModule.hasInstaller = hasInstaller
+        }else{
+            newModule.isIgnitor = (moduleName.indexOf('.js') > -1)
+            newModule.properties = {
+                name: moduleName
+            }
+        }
+        return newModule
+    }
+    const getModules = (asArray) => {
+        const foundModules = {}
+        fs.readdirSync(modulesBasePath).forEach((moduleName) => {
+            foundModules[moduleName] = getModule(moduleName)
+        })
+        return asArray ? Object.values(foundModules) : foundModules
+    }
+    const downloadModule = (downloadUrl,packageName) => {
+        const downloadPath = modulesBasePath + packageName
+        fs.mkdirSync(downloadPath)
+        return new Promise(async (resolve, reject) => {
+            fs.mkdir(downloadPath, () => {
+                request(downloadUrl).pipe(fs.createWriteStream(downloadPath + '.zip'))
+                .on('finish',() => {
+                    zip = fs.createReadStream(downloadPath + '.zip')
+                    .pipe(unzipper.Parse())
+                    .on('entry', async (file) => {
+                        if(file.type === 'Directory'){
+                            try{
+                                fs.mkdirSync(modulesBasePath + file.path, { recursive: true })
+                            }catch(err){
+
+                            }
+                        }else{
+                            const content = await file.buffer();
+                            fs.writeFile(modulesBasePath + file.path,content,(err) => {
+                                if(err)console.log(err)
                             })
-                        }catch(err){
-                            console.log('Failed to Load Module : ' + filename)
-                            console.log(err)
                         }
+                    })
+                    .promise()
+                    .then(() => {
+                        fs.remove(downloadPath + '.zip', () => {})
+                        resolve()
+                    })
+                })
+            })
+        })
+    }
+    const getModuleProperties = (name) => {
+        const modulePath = getModulePath(name)
+        const propertiesPath = modulePath + 'package.json'
+        const properties = fs.existsSync(propertiesPath) ? s.parseJSON(fs.readFileSync(propertiesPath)) : {
+            name: name
+        }
+        return properties
+    }
+    const installModule = (name) => {
+        return new Promise((resolve, reject) => {
+            if(!runningInstallProcesses[name]){
+                //depending on module this may only work for Ubuntu
+                const modulePath = getModulePath(name)
+                const properties = getModuleProperties(name);
+                const installerPath = modulePath + `INSTALL.sh`
+                const propertiesPath = modulePath + 'package.json'
+                var installProcess
+                // check for INSTALL.sh (ubuntu only)
+                if(fs.existsSync(installerPath)){
+                    installProcess = spawn(`sh`,[installerPath])
+                }else if(fs.existsSync(propertiesPath)){
+                    // no INSTALL.sh found, check for package.json and do `npm install --unsafe-perm`
+                    installProcess = spawn(`npm`,['install','--unsafe-perm','--prefix',modulePath])
+                }else{
+                    resolve()
+                }
+                if(installProcess){
+                    const sendData = (data,channel) => {
+                        const clientData = {
+                            f: 'module-info',
+                            module: name,
+                            process: 'install-' + channel,
+                            data: data.toString(),
+                        }
+                        s.tx(clientData,'$')
+                        s.debugLog(clientData)
                     }
+                    installProcess.stderr.on('data',(data) => {
+                        sendData(data,'stderr')
+                    })
+                    installProcess.stdout.on('data',(data) => {
+                        sendData(data,'stdout')
+                    })
+                    installProcess.on('exit',(data) => {
+                        runningInstallProcesses[name] = null;
+                        resolve()
+                    })
+                    runningInstallProcesses[name] = installProcess
+                }
+            }else{
+                resolve(lang['Already Installing...'])
+            }
+        })
+    }
+    const disableModule = (name,status) => {
+        // set status to `false` to enable
+        const modulePath = getModulePath(name)
+        const properties = getModuleProperties(name);
+        const propertiesPath = modulePath + 'package.json'
+        var packageJson = {
+            name: name
+        }
+        try{
+            packageJson = JSON.parse(fs.readFileSync(propertiesPath))
+        }catch(err){
+
+        }
+        packageJson.disabled = status;
+        fs.writeFileSync(propertiesPath,s.prettyPrint(packageJson))
+    }
+    const deleteModule = (name) => {
+        // requires restart for changes to take effect
+        try{
+            const modulePath = modulesBasePath + name
+            fs.remove(modulePath, (err) => {
+                console.log(err)
+            })
+            return true
+        }catch(err){
+            console.log(err)
+            return false
+        }
+    }
+    const loadModule = (shinobiModule) => {
+        const moduleName = shinobiModule.name
+        s.customAutoLoadModules[moduleName] = {}
+        var customModulePath = modulesBasePath + '/' + moduleName
+        if(shinobiModule.isIgnitor){
+            s.customAutoLoadModules[moduleName].type = 'file'
+            try{
+                require(customModulePath)(s,config,lang,app,io)
+            }catch(err){
+                s.systemLog('Failed to Load Module : ' + moduleName)
+                s.systemLog(err)
+            }
+        }else if(shinobiModule.isDirectory){
+            s.customAutoLoadModules[moduleName].type = 'folder'
+            try{
+                require(customModulePath)(s,config,lang,app,io)
+                fs.readdir(customModulePath,function(err,folderContents){
+                    folderContents.forEach(function(name){
+                        switch(name){
+                            case'web':
+                                var webFolder = s.checkCorrectPathEnding(customModulePath) + 'web/'
+                                fs.readdir(webFolder,function(err,webFolderContents){
+                                    webFolderContents.forEach(function(name){
+                                        switch(name){
+                                            case'libs':
+                                            case'pages':
+                                                if(name === 'libs'){
+                                                    if(config.webPaths.home !== '/'){
+                                                        app.use('/libs',express.static(webFolder + '/libs'))
+                                                    }
+                                                    app.use(s.checkCorrectPathEnding(config.webPaths.home)+'libs',express.static(webFolder + '/libs'))
+                                                    app.use(s.checkCorrectPathEnding(config.webPaths.admin)+'libs',express.static(webFolder + '/libs'))
+                                                    app.use(s.checkCorrectPathEnding(config.webPaths.super)+'libs',express.static(webFolder + '/libs'))
+                                                }
+                                                var libFolder = webFolder + name + '/'
+                                                fs.readdir(libFolder,function(err,webFolderContents){
+                                                    webFolderContents.forEach(function(libName){
+                                                        var thirdLevelName = libFolder + libName
+                                                        switch(libName){
+                                                            case'js':
+                                                            case'css':
+                                                            case'blocks':
+                                                                fs.readdir(thirdLevelName,function(err,webFolderContents){
+                                                                    webFolderContents.forEach(function(filename){
+                                                                        var fullPath = thirdLevelName + '/' + filename
+                                                                        var blockPrefix = ''
+                                                                        switch(true){
+                                                                            case searchText('super.',filename):
+                                                                                blockPrefix = 'super'
+                                                                            break;
+                                                                            case searchText('admin.',filename):
+                                                                                blockPrefix = 'admin'
+                                                                            break;
+                                                                        }
+                                                                        switch(libName){
+                                                                            case'js':
+                                                                                s.customAutoLoadTree[blockPrefix + 'LibsJs'].push(filename)
+                                                                            break;
+                                                                            case'css':
+                                                                                s.customAutoLoadTree[blockPrefix + 'LibsCss'].push(filename)
+                                                                            break;
+                                                                            case'blocks':
+                                                                                s.customAutoLoadTree[blockPrefix + 'PageBlocks'].push(fullPath)
+                                                                            break;
+                                                                        }
+                                                                    })
+                                                                })
+                                                            break;
+                                                            default:
+                                                                if(libName.indexOf('.ejs') > -1){
+                                                                    s.customAutoLoadTree.pages.push(thirdLevelName)
+                                                                }
+                                                            break;
+                                                        }
+                                                    })
+                                                })
+                                            break;
+                                        }
+                                    })
+                                })
+                            break;
+                            case'languages':
+                                var languagesFolder = s.checkCorrectPathEnding(customModulePath) + 'languages/'
+                                fs.readdir(languagesFolder,function(err,files){
+                                    if(err)return console.log(err);
+                                    files.forEach(function(filename){
+                                        var fileData = require(languagesFolder + filename)
+                                        var rule = filename.replace('.json','')
+                                        if(config.language === rule){
+                                            lang = Object.assign(lang,fileData)
+                                        }
+                                        if(s.loadedLanguages[rule]){
+                                            s.loadedLanguages[rule] = Object.assign(s.loadedLanguages[rule],fileData)
+                                        }else{
+                                            s.loadedLanguages[rule] = Object.assign(s.copySystemDefaultLanguage(),fileData)
+                                        }
+                                    })
+                                })
+                            break;
+                            case'definitions':
+                                var definitionsFolder = s.checkCorrectPathEnding(customModulePath) + 'definitions/'
+                                fs.readdir(definitionsFolder,function(err,files){
+                                    if(err)return console.log(err);
+                                    files.forEach(function(filename){
+                                        var fileData = require(definitionsFolder + filename)
+                                        var rule = filename.replace('.json','').replace('.js','')
+                                        if(config.language === rule){
+                                            s.definitions = s.mergeDeep(s.definitions,fileData)
+                                        }
+                                        if(s.loadedDefinitons[rule]){
+                                            s.loadedDefinitons[rule] = s.mergeDeep(s.loadedDefinitons[rule],fileData)
+                                        }else{
+                                            s.loadedDefinitons[rule] = s.mergeDeep(s.copySystemDefaultDefinitions(),fileData)
+                                        }
+                                    })
+                                })
+                            break;
+                        }
+                    })
+                })
+            }catch(err){
+                s.systemLog('Failed to Load Module : ' + moduleName)
+                s.systemLog(err)
+            }
+        }
+    }
+    const moveModuleToNameInProperties = (modulePath,packageRoot,properties) => {
+        return new Promise((resolve,reject) => {
+            const packageRootParts = packageRoot.split('/')
+            const filename = packageRootParts[packageRootParts.length - 1]
+            fs.move(modulePath + packageRoot,modulesBasePath + filename,(err) => {
+                if(packageRoot){
+                    fs.remove(modulePath, (err) => {
+                        if(err)console.log(err)
+                        resolve(filename)
+                    })
+                }else{
+                    resolve(filename)
                 }
             })
-        }else{
-            fs.mkdirSync(folderPath)
+        })
+    }
+    const initializeAllModules = async () => {
+        s.customAutoLoadModules = {}
+        s.customAutoLoadTree = {
+            pages: [],
+            PageBlocks: [],
+            LibsJs: [],
+            LibsCss: [],
+            adminPageBlocks: [],
+            adminLibsJs: [],
+            adminLibsCss: [],
+            superPageBlocks: [],
+            superLibsJs: [],
+            superLibsCss: []
         }
+        fs.readdir(modulesBasePath,function(err,folderContents){
+            if(!err && folderContents.length > 0){
+                getModules(true).forEach((shinobiModule) => {
+                    if(shinobiModule.properties.disabled){
+                        return;
+                    }
+                    loadModule(shinobiModule)
+                })
+            }else{
+                fs.mkdir(modulesBasePath,() => {})
+            }
+        })
+    }
+    /**
+    * API : Superuser : Custom Auto Load Package Download.
+    */
+    app.get(config.webPaths.superApiPrefix+':auth/package/list', async (req,res) => {
+        s.superAuth(req.params, async (resp) => {
+            s.closeJsonResponse(res,{
+                ok: true,
+                modules: getModules()
+            })
+        },res,req)
     })
+    /**
+    * API : Superuser : Custom Auto Load Package Download.
+    */
+    app.post(config.webPaths.superApiPrefix+':auth/package/download', async (req,res) => {
+        s.superAuth(req.params, async (resp) => {
+            try{
+                const url = req.body.downloadUrl
+                const packageRoot = req.body.packageRoot || ''
+                const packageName = req.body.packageName || extractNameFromPackage(url)
+                const modulePath = getModulePath(packageName)
+                await downloadModule(url,packageName)
+                const properties = getModuleProperties(packageName)
+                const newName = await moveModuleToNameInProperties(modulePath,packageRoot,properties)
+                const chosenName = newName ? newName : packageName
+                disableModule(chosenName,true)
+                s.closeJsonResponse(res,{
+                    ok: true,
+                    moduleName: chosenName,
+                    newModule: getModule(chosenName)
+                })
+            }catch(err){
+                s.closeJsonResponse(res,{
+                    ok: false,
+                    error: err
+                })
+            }
+        },res,req)
+    })
+    /**
+    * API : Superuser : Custom Auto Load Package Install.
+    */
+    app.post(config.webPaths.superApiPrefix+':auth/package/install', (req,res) => {
+        s.superAuth(req.params, async (resp) => {
+            const packageName = req.body.packageName
+            const response = {ok: true}
+            const error = await installModule(packageName)
+            if(error){
+                response.ok = false
+                response.msg = error
+            }
+            s.closeJsonResponse(res,response)
+        },res,req)
+    })
+    /**
+    * API : Superuser : Custom Auto Load Package set Status (Enabled or Disabled).
+    */
+    app.post(config.webPaths.superApiPrefix+':auth/package/status', (req,res) => {
+        s.superAuth(req.params, async (resp) => {
+            const status = req.body.status
+            const packageName = req.body.packageName
+            const selection = status == 'true' ? true : false
+            disableModule(packageName,selection)
+            s.closeJsonResponse(res,{ok: true, status: selection})
+        },res,req)
+    })
+    /**
+    * API : Superuser : Custom Auto Load Package Delete
+    */
+    app.post(config.webPaths.superApiPrefix+':auth/package/delete', async (req,res) => {
+        s.superAuth(req.params, async (resp) => {
+            const packageName = req.body.packageName
+            const response = deleteModule(packageName)
+            s.closeJsonResponse(res,{ok: response})
+        },res,req)
+    })
+    /**
+    * API : Superuser : Custom Auto Load Package Reload All
+    */
+    app.post(config.webPaths.superApiPrefix+':auth/package/reloadAll', async (req,res) => {
+        s.superAuth(req.params, async (resp) => {
+            await initializeAllModules();
+            s.closeJsonResponse(res,{ok: true})
+        },res,req)
+    })
+    // Initialize Modules on Start
+    await initializeAllModules();
 }

libs/dropInEvents.js

@@ -183,46 +183,87 @@ module.exports = function(s,config,lang,app,io){
         }
         // FTP Server
         if(config.ftpServer === true){
+            const authenticateConnection = (connection) => {
+                return new Promise((resolve,reject) => {
+                    var username = null;
+                    s.debugLog('client connected: ' + connection.remoteAddress);
+                    connection.on('command:user', function(user, success, failure) {
+                      if (user) {
+                        username = user;
+                        success();
+                      } else {
+                        failure();
+                      }
+                    })
+
+                    connection.on('command:pass', function(password, success, failure) {
+                        s.basicOrApiAuthentication(username,password,function(err,user){
+                            if(user){
+                                connection._user = user
+                                success(username);
+                            } else {
+                                failure();
+                            }
+                            resolve({
+                                ok: !!user,
+                                username: username,
+                                password: password
+                            })
+                        })
+                    })
+                })
+            }
             createDropInEventsDirectory()
             if(!config.ftpServerPort)config.ftpServerPort = 21
             if(!config.ftpServerUrl)config.ftpServerUrl = `ftp://0.0.0.0:${config.ftpServerPort}`
             config.ftpServerUrl = config.ftpServerUrl.replace('{{PORT}}',config.ftpServerPort)
-            const FtpSrv = require('ftp-srv')
-            const ftpServer = new FtpSrv({
-                url: config.ftpServerUrl,
-                log: require('bunyan').createLogger({
-                  name: 'ftp-srv',
-                  level: 100
-                }),
+            const ftpd = require('shinobi-ftpd')
+            const ftpServer = new ftpd.FtpServer(config.ftpServerUrl, Object.assign({
+                getInitialCwd: function(connection, callback) {
+                    callback(null, s.dir.dropInEvents + '/' + connection._user.ke)
+                },
+                getRoot: function() {
+                    return s.dir.dropInEvents
+                },
+                pasvPortRangeStart: 1025,
+                pasvPortRangeEnd: 1050,
+                allowUnauthorizedTls: true,
+                uploadMaxSlurpSize: 7000
+            },config.ftpServerOptions || {}))
+
+            ftpServer.on('error', function(error) {
+                s.debugLog(['FTP Server error:', error]);
             })
 
-            ftpServer.on('login', ({connection, username, password}, resolve, reject) => {
-                s.basicOrApiAuthentication(username,password,function(err,user){
-                    if(user){
-                        connection.on('STOR', (error, fileName) => {
-                            if(!fileName)return;
-                            var pathPieces = fileName.replace(s.dir.dropInEvents,'').split('/')
-                            var ke = pathPieces[0]
-                            var mid = pathPieces[1]
-                            var firstDroppedPart = pathPieces[2]
-                            var monitorEventDropDir = s.dir.dropInEvents + ke + '/' + mid + '/'
-                            var deleteKey = monitorEventDropDir + firstDroppedPart
-                            onFileOrFolderFound(monitorEventDropDir + firstDroppedPart,deleteKey,{ke: ke, mid: mid})
-                        })
-                        resolve({root: s.dir.dropInEvents + user.ke})
-                    }else{
-                        // reject(new Error('Failed Authorization'))
-                    }
-                })
-            })
-            ftpServer.on('client-error', ({connection, context, error}) => {
-                console.log('client-error',error)
-            })
-            ftpServer.listen().then(() => {
-                s.systemLog(`FTP Server running on port ${config.ftpServerPort}...`)
-            }).catch(function(err){
-                s.systemLog(err)
+            ftpServer.on('client:connected', async (connection) => {
+                const response = await authenticateConnection(connection)
+                if(connection._user){
+                    connection.cwd = s.dir.dropInEvents + connection._user.ke
+                    connection.on('file:stor', async (eventType, data) => {
+                        const fileName = data.file
+                        const pathPieces = fileName.substr(1).split('/')
+                        const groupKey = connection._user.ke
+                        const monitorId = pathPieces[0]
+                        const firstDroppedPart = pathPieces[1]
+                        const monitorEventDropDir = s.dir.dropInEvents + groupKey + '/' + monitorId + '/'
+                        const deleteKey = monitorEventDropDir + firstDroppedPart
+                        onFileOrFolderFound(
+                            monitorEventDropDir + firstDroppedPart,
+                            deleteKey,
+                            {
+                                ke: groupKey,
+                                mid: monitorId
+                            }
+                        )
+                    })
+                }else{
+                    s.systemLog(`Failed FTP Login Attempt : ${response.username}/${response.password}`)
+                    throw `Failed to Authenticate FTP : ${response.username}/${response.password}`;
+                }
             })
+
+            ftpServer.listen(config.ftpServerPort)
+            s.systemLog(`FTP Server running on port ${config.ftpServerPort}...`)
         }
         //add extensions
         s.onMonitorInit(onMonitorInit)
@@ -234,7 +275,8 @@ module.exports = function(s,config,lang,app,io){
         if(config.smtpServerHideStartTls === undefined)config.smtpServerHideStartTls = null
         var SMTPServer = require("smtp-server").SMTPServer;
         if(!config.smtpServerPort && (config.smtpServerSsl && config.smtpServerSsl.enabled !== false || config.ssl)){config.smtpServerPort = 465}else if(!config.smtpServerPort){config.smtpServerPort = 25}
-        var smtpOptions = {
+        config.smtpServerOptions = config.smtpServerOptions ? config.smtpServerOptions : {}
+        var smtpOptions = Object.assign({
             logger: config.debugLog || config.smtpServerLog,
             hideSTARTTLS: config.smtpServerHideStartTls,
             onAuth(auth, session, callback) {
@@ -310,7 +352,7 @@ module.exports = function(s,config,lang,app,io){
                     callback()
                 }
             }
-        }
+        },config.smtpServerOptions)
         if(config.smtpServerSsl && config.smtpServerSsl.enabled !== false || config.ssl && config.ssl.cert && config.ssl.key){
             var key = config.ssl.key || fs.readFileSync(config.smtpServerSsl.key)
             var cert = config.ssl.cert || fs.readFileSync(config.smtpServerSsl.cert)

libs/events.js

@@ -170,7 +170,7 @@ module.exports = function(s,config,lang){
             extender(x,d)
         })
     }
-    s.triggerEvent = function(d,forceSave){
+    s.triggerEvent = async (d,forceSave) => {
         var didCountingAlready = false
         var filter = {
             halt : false,
@@ -375,7 +375,9 @@ module.exports = function(s,config,lang){
                 time : s.formattedTime(),
                 frame : s.group[d.ke].activeMonitors[d.id].lastJpegDetectorFrame
             })
-        }else{
+        }
+        //
+        if(currentConfig.detector_use_motion === '0' || d.doObjectDetection !== true ){
             if(currentConfig.det_multi_trig === '1'){
                 s.getCamerasForMultiTrigger(d.mon).forEach(function(monitor){
                     if(monitor.mid !== d.id){
@@ -394,7 +396,16 @@ module.exports = function(s,config,lang){
             }
             //save this detection result in SQL, only coords. not image.
             if(forceSave || (filter.save && currentConfig.detector_save === '1')){
-                s.sqlQuery('INSERT INTO Events (ke,mid,details,time) VALUES (?,?,?,?)',[d.ke,d.id,detailString,eventTime])
+                s.knexQuery({
+                    action: "insert",
+                    table: "Events",
+                    insert: {
+                        ke: d.ke,
+                        mid: d.id,
+                        details: detailString,
+                        time: eventTime,
+                    }
+                })
             }
             if(currentConfig.detector === '1' && currentConfig.detector_notrigger === '1'){
                 s.setNoEventsDetector(s.group[d.ke].rawMonitorConfigurations[d.id])
@@ -438,13 +449,9 @@ module.exports = function(s,config,lang){
             }
             d.currentTime = new Date()
             d.currentTimestamp = s.timeObject(d.currentTime).format()
-            d.screenshotName = 'Motion_'+(d.mon.name.replace(/[^\w\s]/gi,''))+'_'+d.id+'_'+d.ke+'_'+s.formattedTime()
+            d.screenshotName =  d.details.reason + '_'+(d.mon.name.replace(/[^\w\s]/gi,''))+'_'+d.id+'_'+d.ke+'_'+s.formattedTime()
             d.screenshotBuffer = null
 
-            s.onEventTriggerExtensions.forEach(function(extender){
-                extender(d,filter)
-            })
-
             if(filter.webhook && currentConfig.detector_webhook === '1'){
                 var detector_webhook_url = s.addEventDetailsToString(d,currentConfig.detector_webhook_url)
                 var webhookMethod = currentConfig.detector_webhook_method
@@ -464,6 +471,11 @@ module.exports = function(s,config,lang){
                     if(err)s.debugLog(err)
                 })
             }
+
+            for (var i = 0; i < s.onEventTriggerExtensions.length; i++) {
+                const extender = s.onEventTriggerExtensions[i]
+                await extender(d,filter)
+            }
         }
         //show client machines the event
         d.cx={f:'detector_trigger',id:d.id,ke:d.ke,details:d.details,doObjectDetection:d.doObjectDetection};

libs/ffmpeg.js

@@ -103,6 +103,8 @@ module.exports = function(s,config,lang,onFinish){
                 auto: {label:lang['Auto'],value:'auto'},
                 drm: {label:lang['drm'],value:'drm'},
                 cuvid: {label:lang['cuvid'],value:'cuvid'},
+                cuda: {label:lang['cuda'],value:'cuda'},
+                opencl: {label:lang['opencl'],value:'opencl'},
                 vaapi: {label:lang['vaapi'],value:'vaapi'},
                 qsv: {label:lang['qsv'],value:'qsv'},
                 vdpau: {label:lang['vdpau'],value:'vdpau'},
@@ -424,7 +426,7 @@ module.exports = function(s,config,lang,onFinish){
         x.hwaccel = ''
         x.cust_input = ''
         //wallclock fix for strangely long, single frame videos
-        if(e.details.wall_clock_timestamp_ignore !== '1' && e.type === 'h264' && x.cust_input.indexOf('-use_wallclock_as_timestamps 1') === -1){x.cust_input+=' -use_wallclock_as_timestamps 1';}
+        if((config.wallClockTimestampAsDefault || e.details.wall_clock_timestamp_ignore !== '1') && e.type === 'h264' && x.cust_input.indexOf('-use_wallclock_as_timestamps 1') === -1){x.cust_input+=' -use_wallclock_as_timestamps 1';}
         //input - frame rate (capture rate)
         if(e.details.sfps && e.details.sfps!==''){x.input_fps=' -r '+e.details.sfps}else{x.input_fps=''}
         //input - analyze duration

libs/fileBin.js

@@ -1,13 +1,254 @@
 var fs = require('fs')
 var moment = require('moment')
 module.exports = function(s,config,lang,app,io){
-    s.getFileBinDirectory = function(e){
-        if(e.mid&&!e.id){e.id=e.mid}
-        s.checkDetails(e)
-        if(e.details&&e.details.dir&&e.details.dir!==''){
-            return s.checkCorrectPathEnding(e.details.dir)+e.ke+'/'+e.id+'/'
-        }else{
-            return s.dir.fileBin+e.ke+'/'+e.id+'/';
-        }
+    const getFileBinDirectory = function(monitor){
+        return s.dir.fileBin + monitor.ke + '/' + monitor.mid + '/'
     }
+    const getFileBinEntry = (options) => {
+        return new Promise((resolve, reject) => {
+            s.knexQuery({
+                action: "select",
+                columns: "*",
+                table: "Files",
+                where: options
+            },(err,rows) => {
+                if(rows[0]){
+                    resolve(rows[0])
+                }else{
+                    resolve()
+                }
+            })
+        })
+    }
+    const getFileBinEntries = (options) => {
+        return new Promise((resolve, reject) => {
+            s.knexQuery({
+                action: "select",
+                columns: "*",
+                table: "Files",
+                where: options
+            },(err,rows) => {
+                if(rows){
+                    resolve(rows)
+                }else{
+                    resolve([])
+                }
+            })
+        })
+    }
+    const updateFileBinEntry = (options) => {
+        return new Promise((resolve, reject) => {
+            const groupKey = options.ke
+            const monitorId = options.mid
+            const filename = options.name
+            const update = options.update
+            if(!filename){
+                resolve('No Filename')
+                return
+            }
+            if(!update){
+                resolve('No Update Options')
+                return
+            }
+            s.knexQuery({
+                action: "select",
+                columns: "size",
+                table: "Files",
+                where: {
+                    ke: groupKey,
+                    mid: monitorId,
+                    name: filename,
+                }
+            },(err,rows) => {
+                if(rows[0]){
+                    const fileSize = rows[0].size
+                    s.knexQuery({
+                        action: "update",
+                        table: "Files",
+                        where: {
+                            ke: groupKey,
+                            mid: monitorId,
+                            name: filename,
+                        },
+                        update: update
+                    },(err) => {
+                        resolve()
+                        if(update.size){
+                            s.setDiskUsedForGroup(groupKey,-(fileSize/1048576),'fileBin')
+                            s.setDiskUsedForGroup(groupKey,(update.size/1048576),'fileBin')
+                            s.purgeDiskForGroup(groupKey)
+                        }
+                    })
+                }else{
+                    resolve()
+                }
+            })
+        })
+    }
+    const deleteFileBinEntry = (options) => {
+        return new Promise((resolve, reject) => {
+            const groupKey = options.ke
+            const monitorId = options.mid
+            const filename = options.name
+            if(!filename){
+                resolve('No Filename')
+                return
+            }
+            s.knexQuery({
+                action: "select",
+                columns: "size",
+                table: "Files",
+                where: {
+                    ke: groupKey,
+                    mid: monitorId,
+                    name: filename,
+                }
+            },(err,rows) => {
+                if(rows[0]){
+                    const fileSize = rows[0].size
+                    s.knexQuery({
+                        action: "delete",
+                        table: "Files",
+                        where: {
+                            ke: groupKey,
+                            mid: monitorId,
+                            name: filename,
+                        }
+                    },(err) => {
+                        resolve()
+                        s.setDiskUsedForGroup(groupKey,-(fileSize/1048576),'fileBin')
+                        s.purgeDiskForGroup(groupKey)
+                    })
+                }else{
+                    resolve()
+                }
+            })
+        })
+    }
+    const insertFileBinEntry = (options) => {
+        return new Promise((resolve, reject) => {
+            const groupKey = options.ke
+            const monitorId = options.mid
+            const filename = options.name
+            if(!filename){
+                resolve('No Filename')
+                return
+            }
+            const monitorFileBinDirectory = getFileBinDirectory({ke: groupKey,mid: monitorId,})
+            const fileSize = options.size || fs.lstatSync(monitorFileBinDirectory + filename).size
+            const details = options.details instanceof Object ? JSON.stringify(options.details) : options.details
+            const status = options.status || 0
+            const time = options.time || new Date()
+            s.knexQuery({
+                action: "insert",
+                table: "Files",
+                insert: {
+                    ke: groupKey,
+                    mid: monitorId,
+                    name: filename,
+                    size: fileSize,
+                    details: details,
+                    status: status,
+                    time: time,
+                }
+            },(err) => {
+                resolve()
+                s.setDiskUsedForGroup(groupKey,(fileSize/1048576),'fileBin')
+                s.purgeDiskForGroup(groupKey)
+            })
+        })
+    }
+    s.getFileBinDirectory = getFileBinDirectory
+    s.getFileBinEntry = getFileBinEntry
+    s.insertFileBinEntry = insertFileBinEntry
+    s.updateFileBinEntry = updateFileBinEntry
+    s.deleteFileBinEntry = deleteFileBinEntry
+    /**
+    * API : Get fileBin file rows
+     */
+    app.get([config.webPaths.apiPrefix+':auth/fileBin/:ke',config.webPaths.apiPrefix+':auth/fileBin/:ke/:id'],async (req,res) => {
+        s.auth(req.params,(user) => {
+            const userDetails = user.details
+            const monitorId = req.params.id
+            const groupKey = req.params.ke
+            const hasRestrictions = userDetails.sub && userDetails.allmonitors !== '1';
+            s.sqlQueryBetweenTimesWithPermissions({
+                table: 'Files',
+                user: user,
+                groupKey: req.params.ke,
+                monitorId: req.params.id,
+                startTime: req.query.start,
+                endTime: req.query.end,
+                startTimeOperator: req.query.startOperator,
+                endTimeOperator: req.query.endOperator,
+                limit: req.query.limit,
+                endIsStartTo: true,
+                noFormat: true,
+                noCount: true,
+                preliminaryValidationFailed: (
+                    user.permissions.get_monitors === "0"
+                )
+            },(response) => {
+                response.forEach(function(v){
+                    v.details = s.parseJSON(v.details)
+                    v.href = '/'+req.params.auth+'/fileBin/'+req.params.ke+'/'+req.params.id+'/'+v.details.year+'/'+v.details.month+'/'+v.details.day+'/'+v.name;
+                })
+                s.closeJsonResponse(res,{
+                    ok: true,
+                    files: response
+                })
+            })
+        },res,req);
+    });
+    /**
+    * API : Get fileBin file
+     */
+    app.get(config.webPaths.apiPrefix+':auth/fileBin/:ke/:id/:year/:month/:day/:file', async (req,res) => {
+        s.auth(req.params,function(user){
+            var failed = function(){
+                res.end(user.lang['File Not Found'])
+            }
+            if (!s.group[req.params.ke].fileBin[req.params.id+'/'+req.params.file]){
+                const groupKey = req.params.ke
+                const monitorId = req.params.id
+                const monitorRestrictions = s.getMonitorRestrictions(user.details,monitorId)
+                if(user.details.sub && user.details.allmonitors === '0' && (user.permissions.get_monitors === "0" || monitorRestrictions.length === 0)){
+                    s.closeJsonResponse(res,{
+                        ok: false,
+                        msg: lang['Not Permitted']
+                    })
+                    return
+                }
+                s.knexQuery({
+                    action: "select",
+                    columns: "*",
+                    table: "Files",
+                    where: [
+                        ['ke','=',groupKey],
+                        ['mid','=',req.params.id],
+                        ['name','=',req.params.file],
+                        monitorRestrictions
+                    ]
+                },(err,r) => {
+                    if(r && r[0]){
+                        r = r[0]
+                        r.details = JSON.parse(r.details)
+                        req.dir = s.dir.fileBin + req.params.ke + '/' + req.params.id + '/' + r.details.year + '/' + r.details.month + '/' + r.details.day + '/' + req.params.file;
+                        fs.stat(req.dir,function(err,stats){
+                            if(!err){
+                                res.on('finish',function(){res.end()})
+                                fs.createReadStream(req.dir).pipe(res)
+                            }else{
+                                failed()
+                            }
+                        })
+                    }else{
+                        failed()
+                    }
+                })
+            }else{
+                res.end(user.lang['Please Wait for Completion'])
+            }
+        },res,req);
+    });
 }

libs/folders.js

@@ -11,6 +11,7 @@ module.exports = function(s,config,lang){
         }else{
             config.streamDir = config.windowsTempDir
         }
+        config.shmDir = `${s.checkCorrectPathEnding(config.streamDir)}`
         if(!fs.existsSync(config.streamDir)){
             config.streamDir = s.mainDirectory+'/streams/'
         }else{

libs/health.js

@@ -1,85 +1,130 @@
+var fs = require('fs');
 var exec = require('child_process').exec;
 var spawn = require('child_process').spawn;
+const { getCpuUsageOnLinux, getRamUsageOnLinux } = require('./health/utils.js')
 module.exports = function(s,config,lang,io){
     s.heartBeat = function(){
         setTimeout(s.heartBeat, 8000);
         io.sockets.emit('ping',{beat:1});
     }
     s.heartBeat()
-    s.cpuUsage = function(callback){
-        var k = {}
-        switch(s.platform){
-            case'win32':
-                k.cmd = "@for /f \"skip=1\" %p in ('wmic cpu get loadpercentage') do @echo %p%"
-            break;
-            case'darwin':
-                k.cmd = "ps -A -o %cpu | awk '{s+=$1} END {print s}'";
-            break;
-            case'linux':
-                k.cmd = 'top -b -n 2 | awk \'toupper($0) ~ /^.?CPU/ {gsub("id,","100",$8); gsub("%","",$8); print 100-$8}\' | tail -n 1';
-            break;
-            case'freebsd':
-                k.cmd = 'vmstat 1 2 | awk \'END{print 100-$19}\''
-            break;
-	    case'openbsd':
-                k.cmd = 'vmstat 1 2 | awk \'END{print 100-$18}\''
-            break;
+    let hasProcStat = false
+    try{
+        fs.statSync("/proc/stat")
+        hasProcStat = true
+    }catch(err){
+
+    }
+    if(hasProcStat){
+        s.cpuUsage = async () => {
+            const percent = await getCpuUsageOnLinux()
+            return percent
         }
-        if(config.customCpuCommand){
-          exec(config.customCpuCommand,{encoding:'utf8',detached: true},function(err,d){
-              if(s.isWin===true) {
-                  d = d.replace(/(\r\n|\n|\r)/gm, "").replace(/%/g, "")
-              }
-              callback(d)
-              s.onGetCpuUsageExtensions.forEach(function(extender){
-                  extender(d)
-              })
-          })
-        } else if(k.cmd){
-             exec(k.cmd,{encoding:'utf8',detached: true},function(err,d){
-                 if(s.isWin===true){
-                     d=d.replace(/(\r\n|\n|\r)/gm,"").replace(/%/g,"")
-                 }
-                 callback(d)
-                 s.onGetCpuUsageExtensions.forEach(function(extender){
-                     extender(d)
-                 })
-             })
-        } else {
-            callback(0)
+    }else{
+        s.cpuUsage = () => {
+            return new Promise((resolve, reject) => {
+                var k = {}
+                switch(s.platform){
+                    case'win32':
+                        k.cmd = "@for /f \"skip=1\" %p in ('wmic cpu get loadpercentage') do @echo %p%"
+                    break;
+                    case'darwin':
+                        k.cmd = "ps -A -o %cpu | awk '{s+=$1} END {print s}'";
+                    break;
+                    case'linux':
+                        k.cmd = 'top -b -n 2 | awk \'toupper($0) ~ /^.?CPU/ {gsub("id,","100",$8); gsub("%","",$8); print 100-$8}\' | tail -n 1';
+                    break;
+                    case'freebsd':
+                        k.cmd = 'vmstat 1 2 | awk \'END{print 100-$19}\''
+                    break;
+                    case'openbsd':
+                        k.cmd = 'vmstat 1 2 | awk \'END{print 100-$18}\''
+                    break;
+                }
+                if(config.customCpuCommand){
+                  exec(config.customCpuCommand,{encoding:'utf8',detached: true},function(err,d){
+                      if(s.isWin===true) {
+                          d = d.replace(/(\r\n|\n|\r)/gm, "").replace(/%/g, "")
+                      }
+                      resolve(d)
+                      s.onGetCpuUsageExtensions.forEach(function(extender){
+                          extender(d)
+                      })
+                  })
+                } else if(k.cmd){
+                     exec(k.cmd,{encoding:'utf8',detached: true},function(err,d){
+                         if(s.isWin===true){
+                             d=d.replace(/(\r\n|\n|\r)/gm,"").replace(/%/g,"")
+                         }
+                         resolve(d)
+                         s.onGetCpuUsageExtensions.forEach(function(extender){
+                             extender(d)
+                         })
+                     })
+                } else {
+                    resolve(0)
+                }
+            })
         }
     }
-    s.ramUsage = function(callback){
-        k={}
-        switch(s.platform){
-            case'win32':
-                k.cmd = "wmic OS get FreePhysicalMemory /Value"
-            break;
-            case'darwin':
-                k.cmd = "vm_stat | awk '/^Pages free: /{f=substr($3,1,length($3)-1)} /^Pages active: /{a=substr($3,1,length($3-1))} /^Pages inactive: /{i=substr($3,1,length($3-1))} /^Pages speculative: /{s=substr($3,1,length($3-1))} /^Pages wired down: /{w=substr($4,1,length($4-1))} /^Pages occupied by compressor: /{c=substr($5,1,length($5-1)); print ((a+w)/(f+a+i+w+s+c))*100;}'"
-            break;
-            case'freebsd':
-        	    k.cmd = "echo \"scale=4; $(vmstat -H | awk 'END{print $5}')*1024*100/$(sysctl -n hw.physmem)\" | bc"
-            break;
-	    case'openbsd':
-                    k.cmd = "echo \"scale=4; $(vmstat | awk 'END{ gsub(\"M\",\"\",$4); print $4 }')*104857600/$(sysctl -n hw.physmem)\" | bc"
-            break;
-            default:
-                k.cmd = "LANG=C free | grep Mem | awk '{print $7/$2 * 100.0}'";
-            break;
+    let hasProcMeminfo = false
+    try{
+        fs.statSync("/proc/meminfo")
+        hasProcMeminfo = true
+    }catch(err){
+
+    }
+    if(hasProcMeminfo){
+        s.ramUsage = async () => {
+            const used = await getRamUsageOnLinux()
+            return used
         }
-        if(k.cmd){
-             exec(k.cmd,{encoding:'utf8',detached: true},function(err,d){
-                 if(s.isWin===true){
-                     d=(parseInt(d.split('=')[1])/(s.totalmem/1000))*100
-                 }
-                 callback(d)
-                 s.onGetRamUsageExtensions.forEach(function(extender){
-                     extender(d)
-                 })
-             })
-        }else{
-            callback(0)
+    }else{
+        s.ramUsage = () => {
+            return new Promise((resolve, reject) => {
+                k={}
+                switch(s.platform){
+                    case'win32':
+                        k.cmd = "wmic OS get FreePhysicalMemory /Value"
+                    break;
+                    case'darwin':
+                        k.cmd = "vm_stat | awk '/^Pages free: /{f=substr($3,1,length($3)-1)} /^Pages active: /{a=substr($3,1,length($3-1))} /^Pages inactive: /{i=substr($3,1,length($3-1))} /^Pages speculative: /{s=substr($3,1,length($3-1))} /^Pages wired down: /{w=substr($4,1,length($4-1))} /^Pages occupied by compressor: /{c=substr($5,1,length($5-1)); print ((a+w)/(f+a+i+w+s+c))*100;}'"
+                    break;
+                    case'freebsd':
+                	    k.cmd = "echo \"scale=4; $(vmstat -H | awk 'END{print $5}')*1024*100/$(sysctl -n hw.physmem)\" | bc"
+                    break;
+        	        case'openbsd':
+                        k.cmd = "echo \"scale=4; $(vmstat | awk 'END{ gsub(\"M\",\"\",$4); print $4 }')*104857600/$(sysctl -n hw.physmem)\" | bc"
+                    break;
+                    default:
+                        k.cmd = "LANG=C free | grep Mem | awk '{print $7/$2 * 100.0}'";
+                    break;
+                }
+                if(k.cmd){
+                     exec(k.cmd,{encoding:'utf8',detached: true},function(err,d){
+                         if(s.isWin===true){
+                             d=(parseInt(d.split('=')[1])/(s.totalmem/1000))*100
+                         }
+                         resolve(d)
+                         s.onGetRamUsageExtensions.forEach(function(extender){
+                             extender(d)
+                         })
+                     })
+                }else{
+                    resolve(0)
+                }
+            })
         }
     }
+    if(config.childNodes.mode !== 'child'){
+        setInterval(async () => {
+            const cpu = await s.cpuUsage()
+            const ram = await s.ramUsage()
+            s.tx({
+                f: 'os',
+                cpu: cpu,
+                ram: ram
+            },'CPU')
+        },10000)
+    }
 }

libs/health/utils.js

@@ -0,0 +1,66 @@
+// This file's contents were referenced from https://gist.github.com/sidwarkd/9578213
+const fs = require('fs');
+const calculateCPUPercentage = function(oldVals, newVals){
+    var totalDiff = newVals.total - oldVals.total;
+    var activeDiff = newVals.active - oldVals.active;
+    return Math.ceil((activeDiff / totalDiff) * 100);
+};
+function getValFromLine(line){
+  var match = line.match(/[0-9]+/gi);
+  if(match !== null)
+    return parseInt(match[0]);
+  else
+    return null;
+};
+const currentCPUInfo = {
+    total: 0,
+    active: 0
+}
+const lastCPUInfo = {
+    total: 0,
+    active: 0
+}
+exports.getCpuUsageOnLinux = () => {
+    lastCPUInfo.active = currentCPUInfo.active;
+    lastCPUInfo.idle = currentCPUInfo.idle;
+    lastCPUInfo.total = currentCPUInfo.total;
+    return new Promise((resolve,reject) => {
+        const getUsage = function(callback){
+            fs.readFile("/proc/stat" ,'utf8', function(err, data){
+                var lines = data.split('\n');
+                var cpuTimes = lines[0].match(/[0-9]+/gi);
+                currentCPUInfo.total = 0;
+                currentCPUInfo.idle = parseInt(cpuTimes[3]) + parseInt(cpuTimes[4]);
+                for (var i = 0; i < cpuTimes.length; i++){
+                  currentCPUInfo.total += parseInt(cpuTimes[i]);
+                }
+                currentCPUInfo.active = currentCPUInfo.total - currentCPUInfo.idle
+                currentCPUInfo.percentUsed = calculateCPUPercentage(lastCPUInfo, currentCPUInfo);
+                callback(currentCPUInfo.percentUsed)
+            })
+        }
+        getUsage(function(percentage){
+            setTimeout(function(){
+                getUsage(function(percentage){
+                    resolve(percentage);
+                })
+            }, 3000)
+        })
+    })
+}
+exports.getRamUsageOnLinux = () => {
+    return new Promise((resolve,reject) => {
+        fs.readFile("/proc/meminfo", 'utf8', function(err, data){
+            const lines = data.split('\n');
+            const total = Math.floor(getValFromLine(lines[0]) / 1024);
+            const free = Math.floor(getValFromLine(lines[1]) / 1024);
+            const cached = Math.floor(getValFromLine(lines[4]) / 1024);
+            const used = total - free;
+            const percentUsed = Math.ceil(((used - cached) / total) * 100);
+            resolve({
+                used: used,
+                percent: percentUsed,
+            });
+        })
+    })
+}

libs/monitor.js

@@ -1,21 +1,19 @@
-var fs = require('fs');
-var events = require('events');
-var spawn = require('child_process').spawn;
-var exec = require('child_process').exec;
-var Mp4Frag = require('mp4frag');
-var onvif = require('node-onvif');
-var treekill = require('tree-kill');
-var request = require('request');
-var connectionTester = require('connection-tester')
-var SoundDetection = require('shinobi-sound-detection')
-var async = require("async");
-var URL = require('url')
+const fs = require('fs');
+const events = require('events');
+const spawn = require('child_process').spawn;
+const exec = require('child_process').exec;
+const Mp4Frag = require('mp4frag');
+const onvif = require('node-onvif');
+const treekill = require('tree-kill');
+const request = require('request');
+const connectionTester = require('connection-tester')
+const SoundDetection = require('shinobi-sound-detection')
+const async = require("async");
+const URL = require('url')
+const { copyObject, createQueue } = require('./common.js')
 module.exports = function(s,config,lang){
-    const startMonitorInQueue = async.queue(function(action, callback) {
-        setTimeout(function(){
-            action(callback)
-        },2000)
-    }, 3)
+    const { cameraDestroy } = require('./monitor/utils.js')(s,config,lang)
+    const startMonitorInQueue = createQueue(1, 3)
     s.initiateMonitorObject = function(e){
         if(!s.group[e.ke]){s.group[e.ke]={}};
         if(!s.group[e.ke].activeMonitors){s.group[e.ke].activeMonitors={}}
@@ -48,26 +46,26 @@ module.exports = function(s,config,lang){
     }
     s.getMonitorCpuUsage = function(e,callback){
         if(s.group[e.ke].activeMonitors[e.mid] && s.group[e.ke].activeMonitors[e.mid].spawn){
-            var getUsage = function(callback2){
+            const getUsage = function(callback2){
                 s.readFile("/proc/" + s.group[e.ke].activeMonitors[e.mid].spawn.pid + "/stat", function(err, data){
                     if(!err){
-                        var elems = data.toString().split(' ');
-                        var utime = parseInt(elems[13]);
-                        var stime = parseInt(elems[14]);
+                        const elems = data.toString().split(' ');
+                        const utime = parseInt(elems[13]);
+                        const stime = parseInt(elems[14]);
 
                         callback2(utime + stime);
                     }else{
-                        clearInterval(s.group[e.ke].activeMonitors[e.mid].getMonitorCpuUsage)
+                        clearInterval(0)
                     }
                 })
             }
             getUsage(function(startTime){
                 setTimeout(function(){
                     getUsage(function(endTime){
-                        var delta = endTime - startTime;
-                        var percentage = 100 * (delta / 10000);
+                        const delta = endTime - startTime;
+                        const percentage = 100 * (delta / 10000);
                         callback(percentage)
-                    });
+                    })
                 }, 1000)
             })
         }else{
@@ -104,141 +102,152 @@ module.exports = function(s,config,lang){
         });
         return x.ar;
     }
-    s.getRawSnapshotFromMonitor = function(monitor,options,callback){
-        if(!callback){
-            callback = options
-            var options = {flags: ''}
-        }
-        s.checkDetails(monitor)
-        var inputOptions = []
-        var outputOptions = []
-        var streamDir = s.dir.streams + monitor.ke + '/' + monitor.mid + '/'
-        var url = options.url
-        var secondsInward = options.secondsInward || '0'
-        if(secondsInward.length === 1)secondsInward = '0' + secondsInward
-        if(options.flags)outputOptions.push(options.flags)
-        const checkExists = function(streamDir,callback){
-            s.fileStats(streamDir,function(err){
-                var response = false
-                if(err){
-                    // s.debugLog(err)
-                }else{
-                    response = true
-                }
-                callback(response)
-            })
-        }
-        const noIconChecks = function(){
-            const runExtraction = function(){
-                var sendTempImage = function(){
-                  fs.readFile(temporaryImageFile,function(err,buffer){
-                     if(!err){
-                       callback(buffer,false)
-                     }
-                     fs.unlink(temporaryImageFile,function(){})
-                  })
-                }
-                try{
-                    var snapBuffer = []
-                    var temporaryImageFile = streamDir + s.gid(5) + '.jpg'
-                    var iconImageFile = streamDir + 'icon.jpg'
-                    var ffmpegCmd = s.splitForFFPMEG(`-loglevel warning -re -probesize 100000 -analyzeduration 100000 ${inputOptions.join(' ')} -i "${url}" ${outputOptions.join(' ')} -vf "fps=1" -vframes 1 "${temporaryImageFile}"`)
-                    fs.writeFileSync(s.group[monitor.ke].activeMonitors[monitor.id].sdir + 'snapCmd.txt',JSON.stringify({
-                      cmd: ffmpegCmd,
-                      temporaryImageFile: temporaryImageFile,
-                      iconImageFile: iconImageFile,
-                      useIcon: options.useIcon,
-                      rawMonitorConfig: s.group[monitor.ke].rawMonitorConfigurations[monitor.mid],
-                    },null,3),'utf8')
-                    var cameraCommandParams = [
-                      s.mainDirectory + '/libs/cameraThread/snapshot.js',
-                      config.ffmpegDir,
-                      s.group[monitor.ke].activeMonitors[monitor.id].sdir + 'snapCmd.txt'
-                    ]
-                    var snapProcess = spawn('node',cameraCommandParams,{detached: true})
-                    snapProcess.stderr.on('data',function(data){
-                        console.log(data.toString())
-                    })
-                    snapProcess.on('close',function(data){
-                        clearTimeout(snapProcessTimeout)
-                        sendTempImage()
-                    })
-                    var snapProcessTimeout = setTimeout(function(){
-                        var pid = snapProcess.pid
-                        if(s.isWin){
-                            spawn("taskkill", ["/pid", pid, '/t'])
-                        }else{
-                            process.kill(-pid, 'SIGTERM')
-                        }
-                        setTimeout(function(){
-                            if(s.isWin === false){
-                                treekill(pid)
+    s.getStreamsDirectory = (monitor) => {
+        return s.dir.streams + monitor.ke + '/' + monitor.mid + '/'
+    }
+    s.getRawSnapshotFromMonitor = function(monitor,options){
+        return new Promise((resolve,reject) => {
+            options = options instanceof Object ? options : {flags: ''}
+            s.checkDetails(monitor)
+            var inputOptions = []
+            var outputOptions = []
+            var streamDir = s.dir.streams + monitor.ke + '/' + monitor.mid + '/'
+            var url = options.url
+            var secondsInward = options.secondsInward || '0'
+            if(secondsInward.length === 1)secondsInward = '0' + secondsInward
+            if(options.flags)outputOptions.push(options.flags)
+            const checkExists = function(streamDir,callback){
+                s.fileStats(streamDir,function(err){
+                    var response = false
+                    if(err){
+                        // s.debugLog(err)
+                    }else{
+                        response = true
+                    }
+                    callback(response)
+                })
+            }
+            const noIconChecks = function(){
+                const runExtraction = function(){
+                    var sendTempImage = function(){
+                      fs.readFile(temporaryImageFile,function(err,buffer){
+                         if(!err){
+                             resolve({
+                                 screenShot: buffer,
+                                 isStaticFile: false
+                             })
+                         }
+                         fs.unlink(temporaryImageFile,function(){})
+                      })
+                    }
+                    try{
+                        var snapBuffer = []
+                        var temporaryImageFile = streamDir + s.gid(5) + '.jpg'
+                        var iconImageFile = streamDir + 'icon.jpg'
+                        var ffmpegCmd = s.splitForFFPMEG(`-loglevel warning -re -probesize 100000 -analyzeduration 100000 ${inputOptions.join(' ')} -i "${url}" ${outputOptions.join(' ')} -f image2 -an -vf "fps=1" -vframes 1 "${temporaryImageFile}"`)
+                        fs.writeFileSync(s.getStreamsDirectory(monitor) + 'snapCmd.txt',JSON.stringify({
+                          cmd: ffmpegCmd,
+                          temporaryImageFile: temporaryImageFile,
+                          iconImageFile: iconImageFile,
+                          useIcon: options.useIcon,
+                          rawMonitorConfig: s.group[monitor.ke].rawMonitorConfigurations[monitor.mid],
+                        },null,3),'utf8')
+                        var cameraCommandParams = [
+                          s.mainDirectory + '/libs/cameraThread/snapshot.js',
+                          config.ffmpegDir,
+                          s.group[monitor.ke].activeMonitors[monitor.id].sdir + 'snapCmd.txt'
+                        ]
+                        var snapProcess = spawn('node',cameraCommandParams,{detached: true})
+                        snapProcess.stderr.on('data',function(data){
+                            s.debugLog(data.toString())
+                        })
+                        snapProcess.on('close',function(data){
+                            clearTimeout(snapProcessTimeout)
+                            sendTempImage()
+                        })
+                        var snapProcessTimeout = setTimeout(function(){
+                            var pid = snapProcess.pid
+                            if(s.isWin){
+                                spawn("taskkill", ["/pid", pid, '/t'])
                             }else{
-                                snapProcess.kill()
+                                process.kill(-pid, 'SIGTERM')
                             }
-                        },10000)
-                    },30000)
-                }catch(err){
-                    console.log(err)
+                            setTimeout(function(){
+                                if(s.isWin === false){
+                                    treekill(pid)
+                                }else{
+                                    snapProcess.kill()
+                                }
+                            },10000)
+                        },30000)
+                    }catch(err){
+                        console.log(err)
+                    }
+                }
+                if(url){
+                    runExtraction()
+                }else{
+                    checkExists(streamDir + 's.jpg',function(success){
+                        if(success === false){
+                            checkExists(streamDir + 'detectorStream.m3u8',function(success){
+                                if(success === false){
+                                    checkExists(streamDir + 's.m3u8',function(success){
+                                        if(success === false){
+                                            switch(monitor.type){
+                                                case'h264':
+                                                    switch(monitor.protocol){
+                                                        case'rtsp':
+                                                            if(
+                                                                monitor.details.rtsp_transport
+                                                                && monitor.details.rtsp_transport !== ''
+                                                                && monitor.details.rtsp_transport !== 'no'
+                                                            ){
+                                                                inputOptions.push('-rtsp_transport ' + monitor.details.rtsp_transport)
+                                                            }
+                                                        break;
+                                                    }
+                                                break;
+                                            }
+                                            url = s.buildMonitorUrl(monitor)
+                                        }else{
+                                            outputOptions.push(`-ss 00:00:${secondsInward}`)
+                                            url = streamDir + 's.m3u8'
+                                        }
+                                        runExtraction()
+                                    })
+                                }else{
+                                    outputOptions.push(`-ss 00:00:${secondsInward}`)
+                                    url = streamDir + 'detectorStream.m3u8'
+                                    runExtraction()
+                                }
+                            })
+                        }else{
+                            s.readFile(streamDir + 's.jpg',function(err,snapBuffer){
+                                resolve({
+                                    screenShot: snapBuffer,
+                                    isStaticFile: true
+                                })
+                            })
+                        }
+                    })
                 }
             }
-            if(url){
-                runExtraction()
-            }else{
-                checkExists(streamDir + 's.jpg',function(success){
+            if(options.useIcon === true){
+                checkExists(streamDir + 'icon.jpg',function(success){
                     if(success === false){
-                        checkExists(streamDir + 'detectorStream.m3u8',function(success){
-                            if(success === false){
-                                checkExists(streamDir + 's.m3u8',function(success){
-                                    if(success === false){
-                                        switch(monitor.type){
-                                            case'h264':
-                                                switch(monitor.protocol){
-                                                    case'rtsp':
-                                                        if(
-                                                            monitor.details.rtsp_transport
-                                                            && monitor.details.rtsp_transport !== ''
-                                                            && monitor.details.rtsp_transport !== 'no'
-                                                        ){
-                                                            inputOptions.push('-rtsp_transport ' + monitor.details.rtsp_transport)
-                                                        }
-                                                    break;
-                                                }
-                                            break;
-                                        }
-                                        url = s.buildMonitorUrl(monitor)
-                                    }else{
-                                        outputOptions.push(`-ss 00:00:${secondsInward}`)
-                                        url = streamDir + 's.m3u8'
-                                    }
-                                    runExtraction()
-                                })
-                            }else{
-                                outputOptions.push(`-ss 00:00:${secondsInward}`)
-                                url = streamDir + 'detectorStream.m3u8'
-                                runExtraction()
-                            }
-                        })
+                        noIconChecks()
                     }else{
-                        s.readFile(streamDir + 's.jpg',function(err,snapBuffer){
-                            callback(snapBuffer,true)
+                        var snapBuffer = fs.readFileSync(streamDir + 'icon.jpg')
+                        resolve({
+                            screenShot: snapBuffer,
+                            isStaticFile: false
                         })
                     }
                 })
+            }else{
+                noIconChecks()
             }
-        }
-        if(options.useIcon === true){
-            checkExists(streamDir + 'icon.jpg',function(success){
-                if(success === false){
-                    noIconChecks()
-                }else{
-                    var snapBuffer = fs.readFileSync(streamDir + 'icon.jpg')
-                    callback(snapBuffer,false)
-                }
-            })
-        }else{
-            noIconChecks()
-        }
+        })
     }
     s.mergeDetectorBufferChunks = function(monitor,callback){
         var pathDir = s.dir.streams+monitor.ke+'/'+monitor.id+'/'
@@ -363,89 +372,7 @@ module.exports = function(s,config,lang){
         })
         return items
     }
-
-    const cameraDestroy = function(e,p){
-        if(s.group[e.ke]&&s.group[e.ke].activeMonitors[e.id]&&s.group[e.ke].activeMonitors[e.id].spawn !== undefined){
-            const activeMonitor = s.group[e.ke].activeMonitors[e.id];
-            const proc = s.group[e.ke].activeMonitors[e.id].spawn;
-            if(proc){
-                activeMonitor.allowStdinWrite = false
-                s.txToDashcamUsers({
-                    f : 'disable_stream',
-                    ke : e.ke,
-                    mid : e.id
-                },e.ke)
-    //            if(activeMonitor.p2pStream){activeMonitor.p2pStream.unpipe();}
-                try{
-                    proc.removeListener('end',activeMonitor.spawn_exit);
-                    proc.removeListener('exit',activeMonitor.spawn_exit);
-                    delete(activeMonitor.spawn_exit);
-                }catch(er){
-
-                }
-            }
-            if(activeMonitor.audioDetector){
-              activeMonitor.audioDetector.stop()
-              delete(activeMonitor.audioDetector)
-            }
-            activeMonitor.firstStreamChunk = {}
-            clearTimeout(activeMonitor.recordingChecker);
-            delete(activeMonitor.recordingChecker);
-            clearTimeout(activeMonitor.streamChecker);
-            delete(activeMonitor.streamChecker);
-            clearTimeout(activeMonitor.checkSnap);
-            delete(activeMonitor.checkSnap);
-            clearTimeout(activeMonitor.watchdog_stop);
-            delete(activeMonitor.watchdog_stop);
-            delete(activeMonitor.lastJpegDetectorFrame);
-            delete(activeMonitor.detectorFrameSaveBuffer);
-            clearTimeout(activeMonitor.recordingSnapper);
-            clearInterval(activeMonitor.getMonitorCpuUsage);
-            clearInterval(activeMonitor.objectCountIntervals);
-            delete(activeMonitor.onvifConnection)
-            if(activeMonitor.onChildNodeExit){
-                activeMonitor.onChildNodeExit()
-            }
-            activeMonitor.spawn.stdio.forEach(function(stdio){
-              try{
-                stdio.unpipe()
-              }catch(err){
-                console.log(err)
-              }
-            })
-            if(activeMonitor.mp4frag){
-                var mp4FragChannels = Object.keys(activeMonitor.mp4frag)
-                mp4FragChannels.forEach(function(channel){
-                    activeMonitor.mp4frag[channel].removeAllListeners()
-                    delete(activeMonitor.mp4frag[channel])
-                })
-            }
-            if(config.childNodes.enabled === true && config.childNodes.mode === 'child' && config.childNodes.host){
-                s.cx({f:'clearCameraFromActiveList',ke:e.ke,id:e.id})
-            }
-            if(activeMonitor.childNode){
-                s.cx({f:'kill',d:s.cleanMonitorObject(e)},activeMonitor.childNodeId)
-            }else{
-                s.coSpawnClose(e)
-                if(proc && proc.kill){
-                    if(s.isWin){
-                        spawn("taskkill", ["/pid", proc.pid, '/t'])
-                    }else{
-                        proc.kill('SIGTERM')
-                    }
-                    setTimeout(function(){
-                        try{
-                            proc.kill()
-                        }catch(err){
-                            s.debugLog(err)
-                        }
-                    },1000)
-                }
-            }
-        }
-    }
-
-    s.cameraCheckObjectsInDetails = function(e){
+    const checkObjectsInDetails = function(e){
         //parse Objects
         (['detector_cascades','cords','detector_filters','input_map_choices']).forEach(function(v){
             if(e.details && e.details[v]){
@@ -510,28 +437,27 @@ module.exports = function(s,config,lang){
         }
         return options
     }
-    s.cameraSendSnapshot = function(e,options){
-        if(!options)options = {}
+    s.cameraSendSnapshot = async (e,options) => {
+        options = Object.assign({
+            flags: '-s 200x200'
+        },options || {})
         s.checkDetails(e)
-        if(config.doSnapshot === true){
+        if(e.ke && config.doSnapshot === true){
             if(s.group[e.ke] && s.group[e.ke].rawMonitorConfigurations && s.group[e.ke].rawMonitorConfigurations[e.mid] && s.group[e.ke].rawMonitorConfigurations[e.mid].mode !== 'stop'){
                 var pathDir = s.dir.streams+e.ke+'/'+e.mid+'/'
-                s.getRawSnapshotFromMonitor(s.group[e.ke].rawMonitorConfigurations[e.mid],Object.assign({
-                    flags: '-s 200x200'
-                },options),function(data,isStaticFile){
-                    if(data && (data[data.length-2] === 0xFF && data[data.length-1] === 0xD9)){
-                        s.tx({
-                            f: 'monitor_snapshot',
-                            snapshot: data.toString('base64'),
-                            snapshot_format: 'b64',
-                            mid: e.mid,
-                            ke: e.ke
-                        },'GRP_'+e.ke)
-                    }else{
-                        console.log('not image')
-                        s.tx({f:'monitor_snapshot',snapshot:e.mon.name,snapshot_format:'plc',mid:e.mid,ke:e.ke},'GRP_'+e.ke)
-                   }
-                })
+                const {screenShot, isStaticFile} = await s.getRawSnapshotFromMonitor(s.group[e.ke].rawMonitorConfigurations[e.mid],options)
+                if(screenShot && (screenShot[screenShot.length-2] === 0xFF && screenShot[screenShot.length-1] === 0xD9)){
+                    s.tx({
+                        f: 'monitor_snapshot',
+                        snapshot: screenShot.toString('base64'),
+                        snapshot_format: 'b64',
+                        mid: e.mid,
+                        ke: e.ke
+                    },'GRP_'+e.ke)
+                }else{
+                    console.log('not image')
+                    s.tx({f:'monitor_snapshot',snapshot:e.mon.name,snapshot_format:'plc',mid:e.mid,ke:e.ke},'GRP_'+e.ke)
+               }
             }else{
                 s.tx({f:'monitor_snapshot',snapshot:'Disabled',snapshot_format:'plc',mid:e.mid,ke:e.ke},'GRP_'+e.ke)
             }
@@ -539,6 +465,29 @@ module.exports = function(s,config,lang){
             s.tx({f:'monitor_snapshot',snapshot:e.mon.name,snapshot_format:'plc',mid:e.mid,ke:e.ke},'GRP_'+e.ke)
         }
     }
+    s.getCameraSnapshot = async (e,options) => {
+        const getDefaultImage = async () => {
+            return await fs.promises.readFile(config.defaultMjpeg)
+        }
+        options = Object.assign({
+            flags: '-s 200x200'
+        },options || {})
+        if(e.ke && config.doSnapshot === true){
+            if(s.group[e.ke] && s.group[e.ke].rawMonitorConfigurations && s.group[e.ke].rawMonitorConfigurations[e.mid] && s.group[e.ke].rawMonitorConfigurations[e.mid].mode !== 'stop'){
+                var pathDir = s.dir.streams+e.ke+'/'+e.mid+'/'
+                const {screenShot, isStaticFile} = await s.getRawSnapshotFromMonitor(s.group[e.ke].rawMonitorConfigurations[e.mid],options)
+                if(screenShot && (screenShot[screenShot.length-2] === 0xFF && screenShot[screenShot.length-1] === 0xD9)){
+                    return screenShot
+                }else{
+                    return await getDefaultImage()
+               }
+            }else{
+                return await getDefaultImage()
+            }
+        }else{
+            return await getDefaultImage()
+        }
+    }
     const createRecordingDirectory = function(e,callback){
         var directory
         if(e.details && e.details.dir && e.details.dir !== '' && config.childNodes.mode !== 'child'){
@@ -601,11 +550,6 @@ module.exports = function(s,config,lang){
             })
         })
     }
-    // try{
-    //   fs.unlinkSync('/home/Shinobi/test.log')
-    // }catch(err){
-    //
-    // }
     const createCameraFolders = function(e,callback){
         //set the recording directory
         var activeMonitor = s.group[e.ke].activeMonitors[e.id]
@@ -625,6 +569,23 @@ module.exports = function(s,config,lang){
             })
         })
     }
+    const forceMonitorRestart = (monitor,restartMessage) => {
+        const monitorConfig = Object.assign(s.group[monitor.ke].rawMonitorConfigurations[monitor.mid],{})
+        s.sendMonitorStatus({
+            id: monitor.mid,
+            ke: monitor.ke,
+            status: lang.Restarting
+        })
+        launchMonitorProcesses(monitorConfig)
+        s.userLog({
+            ke: monitor.ke,
+            mid: monitor.mid,
+        },restartMessage)
+        s.orphanedVideoCheck({
+            ke: monitor.ke,
+            mid: monitor.mid,
+        },2,null,true)
+    }
     s.stripAuthFromHost = function(e){
         var host = e.host.split('@');
         if(host[1]){
@@ -636,7 +597,7 @@ module.exports = function(s,config,lang){
         }
         return host
     }
-    s.resetRecordingCheck = function(e){
+    const resetRecordingCheck = function(e){
         clearTimeout(s.group[e.ke].activeMonitors[e.id].recordingChecker)
         var cutoff = e.cutoff + 0
         if(e.type === 'dashcam'){
@@ -644,10 +605,15 @@ module.exports = function(s,config,lang){
         }
         s.group[e.ke].activeMonitors[e.id].recordingChecker = setTimeout(function(){
             if(s.group[e.ke].activeMonitors[e.id].isStarted === true && s.group[e.ke].rawMonitorConfigurations[e.id].mode === 'record'){
-                launchMonitorProcesses(s.cleanMonitorObject(e));
-                s.sendMonitorStatus({id:e.id,ke:e.ke,status:lang.Restarting});
-                s.userLog(e,{type:lang['Camera is not recording'],msg:{msg:lang['Restarting Process']}});
-                s.orphanedVideoCheck(e,2,null,true)
+                forceMonitorRestart({
+                    ke: e.ke,
+                    mid: e.id,
+                },{
+                    type: lang['Camera is not recording'],
+                    msg: {
+                        msg: lang['Restarting Process']
+                    }
+                })
             }
         },60000 * cutoff * 1.3);
     }
@@ -655,9 +621,15 @@ module.exports = function(s,config,lang){
         clearTimeout(s.group[e.ke].activeMonitors[e.id].streamChecker)
         s.group[e.ke].activeMonitors[e.id].streamChecker = setTimeout(function(){
             if(s.group[e.ke].activeMonitors[e.id] && s.group[e.ke].activeMonitors[e.id].isStarted === true){
-                launchMonitorProcesses(s.cleanMonitorObject(e));
-                s.userLog(e,{type:lang['Camera is not streaming'],msg:{msg:lang['Restarting Process']}});
-                s.orphanedVideoCheck(e,2,null,true)
+                forceMonitorRestart({
+                    ke: e.ke,
+                    mid: e.id,
+                },{
+                    type: lang['Camera is not streaming'],
+                    msg: {
+                        msg: lang['Restarting Process']
+                    }
+                })
             }
         },60000*1);
     }
@@ -702,7 +674,7 @@ module.exports = function(s,config,lang){
                 if(e.details.loglevel!=='quiet'){
                     s.userLog(e,{type:lang['Process Unexpected Exit'],msg:{msg:lang['Process Crashed for Monitor'],cmd:s.group[e.ke].activeMonitors[e.id].ffmpeg}});
                 }
-                s.fatalCameraError(e,'Process Unexpected Exit');
+                fatalError(e,'Process Unexpected Exit');
                 s.orphanedVideoCheck(e,2,null,true)
                 s.onMonitorUnexpectedExitExtensions.forEach(function(extender){
                     extender(Object.assign(s.group[e.ke].rawMonitorConfigurations[e.id],{}),e)
@@ -712,7 +684,7 @@ module.exports = function(s,config,lang){
         s.group[e.ke].activeMonitors[e.id].spawn.on('end',s.group[e.ke].activeMonitors[e.id].spawn_exit)
         s.group[e.ke].activeMonitors[e.id].spawn.on('exit',s.group[e.ke].activeMonitors[e.id].spawn_exit)
         s.group[e.ke].activeMonitors[e.id].spawn.on('error',function(er){
-            s.userLog(e,{type:'Spawn Error',msg:er});s.fatalCameraError(e,'Spawn Error')
+            s.userLog(e,{type:'Spawn Error',msg:er});fatalError(e,'Spawn Error')
         })
         s.userLog(e,{type:lang['Process Started'],msg:{cmd:s.group[e.ke].activeMonitors[e.id].ffmpeg}})
         if(s.isWin === false){
@@ -764,18 +736,22 @@ module.exports = function(s,config,lang){
                     const count = Object.keys(tagInfo.count)
                     const times = tagInfo.times
                     const realTag = tagInfo.tag
-                    s.sqlQuery('INSERT INTO `Events Counts` (ke,mid,details,time,end,count,tag) VALUES (?,?,?,?,?,?,?)',[
-                        groupKey,
-                        monitorId,
-                        JSON.stringify({
-                            times: times,
-                            count: count,
-                        }),
-                        startTime,
-                        endTime,
-                        count.length,
-                        realTag
-                    ])
+                    s.knexQuery({
+                        action: "insert",
+                        table: "Events Counts",
+                        insert: {
+                            ke: groupKey,
+                            mid: monitorId,
+                            details: JSON.stringify({
+                                times: times,
+                                count: count,
+                            }),
+                            time: startTime,
+                            end: endTime,
+                            count: count.length,
+                            tag: realTag
+                        }
+                    })
                 })
             },60000) //every minute
         }
@@ -1022,7 +998,7 @@ module.exports = function(s,config,lang){
                     }
                     s.group[e.ke].activeMonitors[e.id].detector_motion_count = []
                 })
-                s.resetRecordingCheck(e)
+                resetRecordingCheck(e)
             }
         })
     }
@@ -1033,7 +1009,10 @@ module.exports = function(s,config,lang){
             switch(true){
                 case checkLog(d,'No space left on device'):
                     s.checkUserPurgeLock(e.ke)
-                    s.purgeDiskForGroup(e)
+                    s.purgeDiskForGroup(e.ke)
+                break;
+                case checkLog(d,'error parsing AU headers'):
+                    s.userLog(e,{type:lang['Error While Decoding'],msg:lang.ErrorWhileDecodingTextAudio});
                 break;
                 case checkLog(d,'error while decoding'):
                     s.userLog(e,{type:lang['Error While Decoding'],msg:lang.ErrorWhileDecodingText});
@@ -1058,7 +1037,7 @@ module.exports = function(s,config,lang){
                     //restart
                     setTimeout(function(){
                         s.userLog(e,{type:lang['Connection timed out'],msg:lang['Retrying...']});
-                        s.fatalCameraError(e,'Connection timed out');
+                        fatalError(e,'Connection timed out');
                     },1000)
                 break;
                 // case checkLog(d,'Immediate exit requested'):
@@ -1103,12 +1082,9 @@ module.exports = function(s,config,lang){
             })
         },detector_notrigger_timeout)
     }
-    copyObject = function(obj){
-      return Object.assign({},obj)
-    }
     //set master based process launcher
-    launchMonitorProcesses = function(e){
-      const activeMonitor = s.group[e.ke].activeMonitors[e.id]
+    const launchMonitorProcesses = function(e){
+        const activeMonitor = s.group[e.ke].activeMonitors[e.id]
         // e = monitor object
         clearTimeout(activeMonitor.resetFatalErrorCountTimer)
         activeMonitor.resetFatalErrorCountTimer = setTimeout(()=>{
@@ -1169,7 +1145,7 @@ module.exports = function(s,config,lang){
                     activeMonitor.fswatch = fs.watch(e.dir, {encoding : 'utf8'}, (event, filename) => {
                         switch(event){
                             case'change':
-                                s.resetRecordingCheck(e)
+                                resetRecordingCheck(e)
                             break;
                         }
                     });
@@ -1209,7 +1185,7 @@ module.exports = function(s,config,lang){
                                 createCameraFfmpegProcess(e)
                                 createCameraStreamHandlers(e)
                                 createEventCounter(e)
-                                if(e.type === 'dashcam'){
+                                if(e.type === 'dashcam' || e.type === 'socket'){
                                     setTimeout(function(){
                                         activeMonitor.allowStdinWrite = true
                                         s.txToDashcamUsers({
@@ -1245,7 +1221,7 @@ module.exports = function(s,config,lang){
                                   extender(Object.assign(s.group[e.ke].rawMonitorConfigurations[e.id],{}),e)
                               })
                               s.userLog(e,{type:lang["Ping Failed"],msg:lang.skipPingText1});
-                              s.fatalCameraError(e,"Ping Failed");return;
+                              fatalError(e,"Ping Failed");return;
                           }
                       }
                     if(
@@ -1295,7 +1271,7 @@ module.exports = function(s,config,lang){
                             extender(Object.assign(s.group[e.ke].rawMonitorConfigurations[e.id],{}),e)
                         })
                         s.userLog(e,{type:lang["Ping Failed"],msg:lang.skipPingText1});
-                        s.fatalCameraError(e,"Ping Failed");return;
+                        fatalError(e,"Ping Failed");return;
                     }
                 })
             }else{
@@ -1343,7 +1319,7 @@ module.exports = function(s,config,lang){
             console.log(err)
         }
     }
-    s.fatalCameraError = function(e,errorMessage){
+    const fatalError = function(e,errorMessage){
       const activeMonitor = s.group[e.ke].activeMonitors[e.id]
         clearTimeout(activeMonitor.err_fatal_timeout);
         ++activeMonitor.errorFatalCount;
@@ -1363,27 +1339,27 @@ module.exports = function(s,config,lang){
             extender(Object.assign(s.group[e.ke].rawMonitorConfigurations[e.id],{}),e)
         })
     }
-    s.isWatchCountable = function(d){
-        try{
-            var variableMethodsToAllow = [
-                'mp4ws', //Poseidon over Websocket
-                'flvws',
-                'h265ws',
-            ];
-            var indefiniteIgnore = [
-                'mjpeg',
-                'h264',
-            ];
-            var monConfig = s.group[d.ke].rawMonitorConfigurations[d.id]
-            if(
-                variableMethodsToAllow.indexOf(monConfig.details.stream_type + monConfig.details.stream_flv_type) > -1 &&
-                indefiniteIgnore.indexOf(monConfig.details.stream_type) === -1
-            ){
-                return true
-            }
-        }catch(err){}
-        return false
-    }
+    // s.isWatchCountable = function(d){
+    //     try{
+    //         var variableMethodsToAllow = [
+    //             'mp4ws', //Poseidon over Websocket
+    //             'flvws',
+    //             'h265ws',
+    //         ];
+    //         var indefiniteIgnore = [
+    //             'mjpeg',
+    //             'h264',
+    //         ];
+    //         var monConfig = s.group[d.ke].rawMonitorConfigurations[d.id]
+    //         if(
+    //             variableMethodsToAllow.indexOf(monConfig.details.stream_type + monConfig.details.stream_flv_type) > -1 &&
+    //             indefiniteIgnore.indexOf(monConfig.details.stream_type) === -1
+    //         ){
+    //             return true
+    //         }
+    //     }catch(err){}
+    //     return false
+    // }
     s.addOrEditMonitor = function(form,callback,user){
         var endData = {
             ok: false
@@ -1395,10 +1371,17 @@ module.exports = function(s,config,lang){
         }
         form.mid = form.mid.replace(/[^\w\s]/gi,'').replace(/ /g,'')
         form = s.cleanMonitorObjectForDatabase(form)
-        s.sqlQuery('SELECT * FROM Monitors WHERE ke=? AND mid=?',[form.ke,form.mid],function(er,r){
+        s.knexQuery({
+            action: "select",
+            columns: "*",
+            table: "Monitors",
+            where: [
+                ['ke','=',form.ke],
+                ['mid','=',form.mid],
+            ]
+        },(err,r) => {
             var affectMonitor = false
-            var monitorQuery = []
-            var monitorQueryValues = []
+            var monitorQuery = {}
             var txData = {
                 f: 'monitor_edit',
                 mid: form.mid,
@@ -1416,19 +1399,23 @@ module.exports = function(s,config,lang){
                         form[v] !== false &&
                         form[v] !== `false`
                     ){
-                        monitorQuery.push(v+'=?')
                         if(form[v] instanceof Object){
                             form[v] = s.s(form[v])
                         }
-                        monitorQueryValues.push(form[v])
+                        monitorQuery[v] = form[v]
                     }
                 })
-                monitorQuery = monitorQuery.join(',')
-                monitorQueryValues.push(form.ke)
-                monitorQueryValues.push(form.mid)
                 s.userLog(form,{type:'Monitor Updated',msg:'by user : '+user.uid})
                 endData.msg = user.lang['Monitor Updated by user']+' : '+user.uid
-                s.sqlQuery('UPDATE Monitors SET '+monitorQuery+' WHERE ke=? AND mid=?',monitorQueryValues)
+                s.knexQuery({
+                    action: "update",
+                    table: "Monitors",
+                    update: monitorQuery,
+                    where: [
+                        ['ke','=',form.ke],
+                        ['mid','=',form.mid],
+                    ]
+                })
                 affectMonitor = true
             }else if(
                 !s.group[form.ke].init.max_camera ||
@@ -1436,22 +1423,21 @@ module.exports = function(s,config,lang){
                 Object.keys(s.group[form.ke].activeMonitors).length <= parseInt(s.group[form.ke].init.max_camera)
             ){
                 txData.new = true
-                monitorQueryInsertValues = []
                 Object.keys(form).forEach(function(v){
                     if(form[v] && form[v] !== ''){
-                        monitorQuery.push(v)
-                        monitorQueryInsertValues.push('?')
                         if(form[v] instanceof Object){
                             form[v] = s.s(form[v])
                         }
-                        monitorQueryValues.push(form[v])
+                        monitorQuery[v] = form[v]
                     }
                 })
-                monitorQuery = monitorQuery.join(',')
-                monitorQueryInsertValues = monitorQueryInsertValues.join(',')
                 s.userLog(form,{type:'Monitor Added',msg:'by user : '+user.uid})
                 endData.msg = user.lang['Monitor Added by user']+' : '+user.uid
-                s.sqlQuery('INSERT INTO Monitors ('+monitorQuery+') VALUES ('+monitorQueryInsertValues+')',monitorQueryValues)
+                s.knexQuery({
+                    action: "insert",
+                    table: "Monitors",
+                    insert: monitorQuery
+                })
                 affectMonitor = true
             }else{
                 txData.f = 'monitor_edit_failed'
@@ -1488,7 +1474,7 @@ module.exports = function(s,config,lang){
         e.functionMode = x
         if(!e.mode){e.mode = x}
         s.checkDetails(e)
-        s.cameraCheckObjectsInDetails(e)
+        checkObjectsInDetails(e)
         s.initiateMonitorObject({ke:e.ke,mid:e.id})
         switch(e.functionMode){
             case'watch_on'://live streamers - join
@@ -1640,15 +1626,24 @@ module.exports = function(s,config,lang){
             if(notFound === false){
                 var sqlQuery = 'SELECT * FROM Monitors WHERE ke=? AND '
                 var monitorQuery = []
-                var sqlQueryValues = [groupKey]
                 var monitorPresets = {}
                 preset.details.monitors.forEach(function(monitor){
-                    monitorQuery.push('mid=?')
-                    sqlQueryValues.push(monitor.mid)
+                    const whereConditions = {}
+                    if(monitorQuery.length === 0){
+                        whereConditions.ke = groupKey
+                        monitorQuery.push(['ke','=',groupKey])
+                    }else{
+                        monitorQuery.push(['or','ke','=',groupKey])
+                    }
+                    monitorQuery.push(['mid','=',monitor.mid])
                     monitorPresets[monitor.mid] = monitor
                 })
-                sqlQuery += '('+monitorQuery.join(' OR ')+')'
-                s.sqlQuery(sqlQuery,sqlQueryValues,function(err,monitors){
+                s.knexQuery({
+                    action: "select",
+                    columns: "*",
+                    table: "Monitors",
+                    where: monitorQuery
+                },function(err,monitors){
                     if(monitors && monitors[0]){
                         monitors.forEach(function(monitor){
                             s.checkDetails(monitor)
@@ -1705,6 +1700,40 @@ module.exports = function(s,config,lang){
         })
         return cameras
     }
+    s.getMonitorRestrictions = (permissions,monitorId) => {
+        const monitorRestrictions = []
+        if(
+            !monitorId &&
+            permissions.sub &&
+            permissions.monitors &&
+            permissions.allmonitors !== '1'
+        ){
+            try{
+                permissions.monitors = s.parseJSON(permissions.monitors)
+                permissions.monitors.forEach(function(v,n){
+                    if(n === 0){
+                        monitorRestrictions.push(['mid','=',v])
+                    }else{
+                        monitorRestrictions.push(['or','mid','=',v])
+                    }
+                })
+            }catch(er){
+            }
+        }else if(
+            monitorId && (
+                !permissions.sub ||
+                permissions.allmonitors !== '0' ||
+                permissions.monitors.indexOf(monitorId) >- 1
+            )
+        ){
+            monitorRestrictions.push(['mid','=',monitorId])
+        }else if(
+            !monitorId &&
+            permissions.sub &&
+            permissions.allmonitors !== '0'
+        ){}
+        return monitorRestrictions
+    }
     // s.checkViewerConnectionsForMonitor = function(monitorObject){
     //     var monitorConfig = s.group[monitorObject.ke].rawMonitorConfigurations[monitorObject.mid]
     //     if(monitorConfig.mode === 'start'){

libs/monitor/utils.js

@@ -0,0 +1,89 @@
+module.exports = (s,config,lang) => {
+    const cameraDestroy = function(e,p){
+        if(
+            s.group[e.ke] &&
+            s.group[e.ke].activeMonitors[e.id] &&
+            s.group[e.ke].activeMonitors[e.id].spawn !== undefined
+        ){
+            const activeMonitor = s.group[e.ke].activeMonitors[e.id];
+            const proc = s.group[e.ke].activeMonitors[e.id].spawn;
+            if(proc){
+                activeMonitor.allowStdinWrite = false
+                s.txToDashcamUsers({
+                    f : 'disable_stream',
+                    ke : e.ke,
+                    mid : e.id
+                },e.ke)
+    //            if(activeMonitor.p2pStream){activeMonitor.p2pStream.unpipe();}
+                try{
+                    proc.removeListener('end',activeMonitor.spawn_exit);
+                    proc.removeListener('exit',activeMonitor.spawn_exit);
+                    delete(activeMonitor.spawn_exit);
+                }catch(er){
+
+                }
+            }
+            if(activeMonitor.audioDetector){
+              activeMonitor.audioDetector.stop()
+              delete(activeMonitor.audioDetector)
+            }
+            activeMonitor.firstStreamChunk = {}
+            clearTimeout(activeMonitor.recordingChecker);
+            delete(activeMonitor.recordingChecker);
+            clearTimeout(activeMonitor.streamChecker);
+            delete(activeMonitor.streamChecker);
+            clearTimeout(activeMonitor.checkSnap);
+            delete(activeMonitor.checkSnap);
+            clearTimeout(activeMonitor.watchdog_stop);
+            delete(activeMonitor.watchdog_stop);
+            delete(activeMonitor.lastJpegDetectorFrame);
+            delete(activeMonitor.detectorFrameSaveBuffer);
+            clearTimeout(activeMonitor.recordingSnapper);
+            clearInterval(activeMonitor.getMonitorCpuUsage);
+            clearInterval(activeMonitor.objectCountIntervals);
+            delete(activeMonitor.onvifConnection)
+            if(activeMonitor.onChildNodeExit){
+                activeMonitor.onChildNodeExit()
+            }
+            activeMonitor.spawn.stdio.forEach(function(stdio){
+              try{
+                stdio.unpipe()
+              }catch(err){
+                console.log(err)
+              }
+            })
+            if(activeMonitor.mp4frag){
+                var mp4FragChannels = Object.keys(activeMonitor.mp4frag)
+                mp4FragChannels.forEach(function(channel){
+                    activeMonitor.mp4frag[channel].removeAllListeners()
+                    delete(activeMonitor.mp4frag[channel])
+                })
+            }
+            if(config.childNodes.enabled === true && config.childNodes.mode === 'child' && config.childNodes.host){
+                s.cx({f:'clearCameraFromActiveList',ke:e.ke,id:e.id})
+            }
+            if(activeMonitor.childNode){
+                s.cx({f:'kill',d:s.cleanMonitorObject(e)},activeMonitor.childNodeId)
+            }else{
+                s.coSpawnClose(e)
+                if(proc && proc.kill){
+                    if(s.isWin){
+                        spawn("taskkill", ["/pid", proc.pid, '/t'])
+                    }else{
+                        proc.kill('SIGTERM')
+                    }
+                    setTimeout(function(){
+                        try{
+                            proc.kill()
+                        }catch(err){
+                            s.debugLog(err)
+                        }
+                    },1000)
+                }
+            }
+        }
+    }
+    return {
+        cameraDestroy: cameraDestroy
+    }
+}

libs/notification.js

@@ -12,7 +12,7 @@ module.exports = function(s,config,lang){
                     s.userLog({ke:groupKey,mid:'$USER'},{type:lang.DiscordFailedText,msg:lang.DiscordNotEnabledText})
                     return
                 }
-                var sendBody = Object.assign({
+                const sendBody = Object.assign({
                     color: 3447003,
                     title: 'Alert from Shinobi',
                     description: "",
@@ -23,7 +23,7 @@ module.exports = function(s,config,lang){
                       text: "Shinobi Systems"
                     }
                 },data)
-                var discordChannel = bot.channels.get(s.group[groupKey].init.discordbot_channel)
+                const discordChannel = bot.channels.cache.get(s.group[groupKey].init.discordbot_channel)
                 if(discordChannel && discordChannel.send){
                     discordChannel.send({
                         embed: sendBody,
@@ -45,10 +45,10 @@ module.exports = function(s,config,lang){
                     })
                 }
             }
-            var onEventTriggerBeforeFilterForDiscord = function(d,filter){
+            const onEventTriggerBeforeFilterForDiscord = function(d,filter){
                 filter.discord = true
             }
-            var onEventTriggerForDiscord = function(d,filter){
+            const onEventTriggerForDiscord = async (d,filter) => {
                 // d = event object
                 //discord bot
                 if(filter.discord && s.group[d.ke].discordBot && d.mon.details.detector_discordbot === '1' && !s.group[d.ke].activeMonitors[d.id].detector_discordbot){
@@ -60,28 +60,11 @@ module.exports = function(s,config,lang){
                     }
                     //lock mailer so you don't get emailed on EVERY trigger event.
                     s.group[d.ke].activeMonitors[d.id].detector_discordbot = setTimeout(function(){
-                        //unlock so you can mail again.
                         clearTimeout(s.group[d.ke].activeMonitors[d.id].detector_discordbot);
                         delete(s.group[d.ke].activeMonitors[d.id].detector_discordbot);
                     },detector_discordbot_timeout)
-                    var files = []
-                    var sendAlert = function(){
-                        s.discordMsg({
-                            author: {
-                              name: s.group[d.ke].rawMonitorConfigurations[d.id].name,
-                              icon_url: config.iconURL
-                            },
-                            title: lang.Event+' - '+d.screenshotName,
-                            description: lang.EventText1+' '+d.currentTimestamp,
-                            fields: [],
-                            timestamp: d.currentTime,
-                            footer: {
-                              icon_url: config.iconURL,
-                              text: "Shinobi Systems"
-                            }
-                        },files,d.ke)
-                    }
                     if(d.mon.details.detector_discordbot_send_video === '1'){
+                        // change to function that captures on going video capture, waits, grabs new video file, slices portion (max for transmission) and prepares for delivery
                         s.mergeDetectorBufferChunks(d,function(mergedFilepath,filename){
                             s.discordMsg({
                                 author: {
@@ -103,21 +86,34 @@ module.exports = function(s,config,lang){
                             ],d.ke)
                         })
                     }
-                    s.getRawSnapshotFromMonitor(d.mon,{
+                    const {screenShot, isStaticFile} = await s.getRawSnapshotFromMonitor(d.mon,{
                         secondsInward: d.mon.details.snap_seconds_inward
-                    },function(data){
-                        if(data[data.length - 2] === 0xFF && data[data.length - 1] === 0xD9){
-                            d.screenshotBuffer = data
-                            files.push({
-                                attachment: d.screenshotBuffer,
-                                name: d.screenshotName+'.jpg'
-                            })
-                        }
-                        sendAlert()
                     })
+                    if(screenShot[screenShot.length - 2] === 0xFF && screenShot[screenShot.length - 1] === 0xD9){
+                        d.screenshotBuffer = screenShot
+                        s.discordMsg({
+                            author: {
+                              name: s.group[d.ke].rawMonitorConfigurations[d.id].name,
+                              icon_url: config.iconURL
+                            },
+                            title: lang.Event+' - '+d.screenshotName,
+                            description: lang.EventText1+' '+d.currentTimestamp,
+                            fields: [],
+                            timestamp: d.currentTime,
+                            footer: {
+                              icon_url: config.iconURL,
+                              text: "Shinobi Systems"
+                            }
+                        },[
+                            {
+                                attachment: screenShot,
+                                name: d.screenshotName+'.jpg'
+                            }
+                        ],d.ke)
+                    }
                 }
             }
-            var onTwoFactorAuthCodeNotificationForDiscord = function(r){
+            const onTwoFactorAuthCodeNotificationForDiscord = function(r){
                 // r = user
                 if(r.details.factor_discord === '1'){
                     s.discordMsg({
@@ -136,13 +132,13 @@ module.exports = function(s,config,lang){
                     },[],r.ke)
                 }
             }
-            var loadDiscordBotForUser = function(user){
-                ar=JSON.parse(user.details);
+            const loadDiscordBotForUser = function(user){
+                const userDetails = s.parseJSON(user.details);
                 //discordbot
                 if(!s.group[user.ke].discordBot &&
                    config.discordBot === true &&
-                   ar.discordbot === '1' &&
-                   ar.discordbot_token !== ''
+                   userDetails.discordbot === '1' &&
+                   userDetails.discordbot_token !== ''
                   ){
                     s.group[user.ke].discordBot = new Discord.Client()
                     s.group[user.ke].discordBot.on('ready', () => {
@@ -154,16 +150,16 @@ module.exports = function(s,config,lang){
                             msg: s.group[user.ke].discordBot.user.tag
                         })
                     })
-                    s.group[user.ke].discordBot.login(ar.discordbot_token)
+                    s.group[user.ke].discordBot.login(userDetails.discordbot_token)
                 }
             }
-            var unloadDiscordBotForUser = function(user){
+            const unloadDiscordBotForUser = function(user){
                 if(s.group[user.ke].discordBot && s.group[user.ke].discordBot.destroy){
                     s.group[user.ke].discordBot.destroy()
                     delete(s.group[user.ke].discordBot)
                 }
             }
-            var onDetectorNoTriggerTimeoutForDiscord = function(e){
+            const onDetectorNoTriggerTimeoutForDiscord = function(e){
                 //e = monitor object
                 var currentTime = new Date()
                 if(e.details.detector_notrigger_discord === '1'){
@@ -205,10 +201,18 @@ module.exports = function(s,config,lang){
             if(config.mail.from === undefined){config.mail.from = '"ShinobiCCTV" <no-reply@shinobi.video>'}
             s.nodemailer = require('nodemailer').createTransport(config.mail);
         }
-        var onDetectorNoTriggerTimeoutForEmail = function(e){
+        const onDetectorNoTriggerTimeoutForEmail = function(e){
             //e = monitor object
             if(config.mail && e.details.detector_notrigger_mail === '1'){
-                s.sqlQuery('SELECT mail FROM Users WHERE ke=? AND details NOT LIKE ?',[e.ke,'%"sub"%'],function(err,r){
+                s.knexQuery({
+                    action: "select",
+                    columns: "mail",
+                    table: "Users",
+                    where: [
+                        ['ke','=',e.ke],
+                        ['details','NOT LIKE','%"sub"%'],
+                    ]
+                },(err,r) => {
                     r = r[0]
                         var mailOptions = {
                             from: config.mail.from, // sender address
@@ -229,16 +233,15 @@ module.exports = function(s,config,lang){
                 })
             }
         }
-        var onTwoFactorAuthCodeNotificationForEmail = function(r){
+        const onTwoFactorAuthCodeNotificationForEmail = function(r){
             // r = user object
             if(r.details.factor_mail !== '0'){
-                var mailOptions = {
+                s.nodemailer.sendMail({
                     from: config.mail.from,
                     to: r.mail,
                     subject: r.lang['2-Factor Authentication'],
                     html: r.lang['Enter this code to proceed']+' <b>'+s.factorAuth[r.ke][r.uid].key+'</b>. '+r.lang.FactorAuthText1,
-                };
-                s.nodemailer.sendMail(mailOptions, (error, info) => {
+                }, (error, info) => {
                     if (error) {
                         s.systemLog(r.lang.MailError,error)
                         return
@@ -246,7 +249,7 @@ module.exports = function(s,config,lang){
                 })
             }
         }
-        var onFilterEventForEmail = function(x,d){
+        const onFilterEventForEmail = function(x,d){
             // x = filter function
             // d = filter event object
             if(x === 'email'){
@@ -275,17 +278,25 @@ module.exports = function(s,config,lang){
                 }
             }
         }
-        var onEventTriggerBeforeFilterForEmail = function(d,filter){
+        const onEventTriggerBeforeFilterForEmail = function(d,filter){
             if(d.mon.details.detector_mail === '1'){
                 filter.mail = true
             }else{
                 filter.mail = false
             }
         }
-        var onEventTriggerForEmail = function(d,filter){
+        const onEventTriggerForEmail = async (d,filter) => {
             if(filter.mail && config.mail && !s.group[d.ke].activeMonitors[d.id].detector_mail){
-                s.sqlQuery('SELECT mail FROM Users WHERE ke=? AND details NOT LIKE ?',[d.ke,'%"sub"%'],function(err,r){
-                    r=r[0];
+                s.knexQuery({
+                    action: "select",
+                    columns: "mail",
+                    table: "Users",
+                    where: [
+                        ['ke','=',d.ke],
+                        ['details','NOT LIKE','%"sub"%'],
+                    ]
+                },async (err,r) => {
+                    r = r[0];
                     var detector_mail_timeout
                     if(!d.mon.details.detector_mail_timeout||d.mon.details.detector_mail_timeout===''){
                         detector_mail_timeout = 1000*60*10;
@@ -293,13 +304,12 @@ module.exports = function(s,config,lang){
                         detector_mail_timeout = parseFloat(d.mon.details.detector_mail_timeout)*1000*60;
                     }
                     //lock mailer so you don't get emailed on EVERY trigger event.
-                    s.group[d.ke].activeMonitors[d.id].detector_mail=setTimeout(function(){
+                    s.group[d.ke].activeMonitors[d.id].detector_mail = setTimeout(function(){
                         //unlock so you can mail again.
                         clearTimeout(s.group[d.ke].activeMonitors[d.id].detector_mail);
                         delete(s.group[d.ke].activeMonitors[d.id].detector_mail);
                     },detector_mail_timeout);
-                    var files = []
-                    var sendMail = function(){
+                    const sendMail = function(files){
                         const infoRows = []
                         Object.keys(d.details).forEach(function(key){
                             var value = d.details[key]
@@ -321,7 +331,7 @@ module.exports = function(s,config,lang){
                                 subtitle: 'Shinobi Event',
                                 body: infoRows.join(''),
                             }),
-                            attachments: files
+                            attachments: files || []
                         }, (error, info) => {
                             if (error) {
                                 s.systemLog(lang.MailError,error)
@@ -330,6 +340,7 @@ module.exports = function(s,config,lang){
                         })
                     }
                     if(d.mon.details.detector_mail_send_video === '1'){
+                        // change to function that captures on going video capture, waits, grabs new video file, slices portion (max for transmission) and prepares for delivery
                         s.mergeDetectorBufferChunks(d,function(mergedFilepath,filename){
                             fs.readFile(mergedFilepath,function(err,buffer){
                                 if(buffer){
@@ -354,24 +365,18 @@ module.exports = function(s,config,lang){
                             })
                         })
                     }
-                    if(d.screenshotBuffer){
-                        files.push({
+                    if(!d.screenshotBuffer){
+                        const {screenShot, isStaticFile} = await s.getRawSnapshotFromMonitor(d.mon,{
+                            secondsInward: d.mon.details.snap_seconds_inward
+                        })
+                        d.screenshotBuffer = screenShot
+                    }
+                    sendMail([
+                        {
                             filename: d.screenshotName + '.jpg',
                             content: d.screenshotBuffer
-                        })
-                        sendMail()
-                    }else{
-                        s.getRawSnapshotFromMonitor(d.mon,{
-                            secondsInward: d.mon.details.snap_seconds_inward
-                        },function(data){
-                            d.screenshotBuffer = data
-                            files.push({
-                                filename: d.screenshotName + '.jpg',
-                                content: data
-                            })
-                            sendMail()
-                        })
-                    }
+                        }
+                    ])
                 })
             }
         }

libs/plugins.js

@@ -14,11 +14,14 @@ module.exports = function(s,config,lang,io){
             case's.tx':
                 s.tx(d.data,d.to)
             break;
+            case'log':
+                s.systemLog('PLUGIN : '+d.plug+' : ',d)
+            break;
             case's.sqlQuery':
                 s.sqlQuery(d.query,d.values)
             break;
-            case'log':
-                s.systemLog('PLUGIN : '+d.plug+' : ',d)
+            case's.knexQuery':
+                s.knexQuery(d.options)
             break;
         }
     }

libs/scheduler.js

@@ -3,7 +3,11 @@ module.exports = function(s,config,lang,app,io){
     //Get all Schedules
     s.getAllSchedules = function(callback){
         s.schedules = {}
-        s.sqlQuery('SELECT * FROM Schedules',function(err,rows){
+        s.knexQuery({
+            action: "select",
+            columns: "*",
+            table: "Schedules"
+        },(err,rows) => {
             rows.forEach(function(schedule){
                 s.updateSchedule(schedule)
             })
@@ -141,7 +145,16 @@ module.exports = function(s,config,lang,app,io){
     //
     s.findSchedule = function(groupKey,name,callback){
         //presetQueryVals = [ke, type, name]
-        s.sqlQuery("SELECT * FROM Schedules WHERE ke=? AND name=? LIMIT 1",[groupKey,name],function(err,schedules){
+        s.knexQuery({
+            action: "select",
+            columns: "*",
+            table: "Schedules",
+            where: [
+                ['ke','=',groupKey],
+                ['name','=',name],
+            ],
+            limit: 1
+        },function(err,schedules) {
             var schedule
             var notFound = false
             if(schedules && schedules[0]){
@@ -184,13 +197,18 @@ module.exports = function(s,config,lang,app,io){
                 s.closeJsonResponse(res,endData)
                 return
             }
-            var theQuery = "SELECT * FROM Schedules WHERE ke=?"
-            var theQueryValues = [req.params.ke]
+            var whereQuery = [
+                ['ke','=',req.params.ke]
+            ]
             if(req.params.name){
-                theQuery += ' AND name=?'
-                theQueryValues.push(req.params.name)
+                whereQuery.push(['name','=',req.params.name])
             }
-            s.sqlQuery(theQuery,theQueryValues,function(err,schedules){
+            s.knexQuery({
+                action: "select",
+                columns: "*",
+                table: "Schedules",
+                where: whereQuery,
+            },function(err,schedules) {
                 endData.ok = true
                 schedules = schedules || []
                 schedules.forEach(function(schedule){
@@ -240,7 +258,11 @@ module.exports = function(s,config,lang,app,io){
                                 end: form.end,
                                 enabled: form.enabled
                             }
-                            s.sqlQuery('INSERT INTO Schedules ('+Object.keys(insertData).join(',')+') VALUES (?,?,?,?,?,?)',Object.values(insertData))
+                            s.knexQuery({
+                                action: "insert",
+                                table: "Schedules",
+                                insert: insertData
+                            })
                             s.tx({
                                 f: 'add_schedule',
                                 insertData: insertData,
@@ -253,14 +275,23 @@ module.exports = function(s,config,lang,app,io){
                                 details: s.stringJSON(form.details),
                                 start: form.start,
                                 end: form.end,
-                                enabled: form.enabled,
-                                ke: req.params.ke,
-                                name: req.params.name
+                                enabled: form.enabled
                             }
-                            s.sqlQuery('UPDATE Schedules SET details=?,start=?,end=?,enabled=? WHERE ke=? AND name=?',Object.values(insertData))
+                            s.knexQuery({
+                                action: "update",
+                                table: "Schedules",
+                                update: insertData,
+                                where: [
+                                    ['ke','=',req.params.ke],
+                                    ['name','=',req.params.name],
+                                ]
+                            })
                             s.tx({
                                 f: 'edit_schedule',
-                                insertData: insertData,
+                                insertData: Object.assign(insertData,{
+                                    ke: req.params.ke,
+                                    name: req.params.name,
+                                }),
                                 ke: req.params.ke,
                                 name: req.params.name
                             },'GRP_'+req.params.ke)
@@ -283,7 +314,14 @@ module.exports = function(s,config,lang,app,io){
                             endData.msg = user.lang['Schedule Configuration Not Found']
                             s.closeJsonResponse(res,endData)
                         }else{
-                            s.sqlQuery('DELETE FROM Schedules WHERE ke=? AND name=?',[req.params.ke,req.params.name],function(err){
+                            s.knexQuery({
+                                action: "delete",
+                                table: "Schedules",
+                                where : {
+                                    ke: req.params.ke,
+                                    name: req.params.name,
+                                }
+                            },function(err){
                                 if(!err){
                                     endData.msg = lang["Deleted Schedule Configuration"]
                                     endData.ok = true

libs/sql.js

@@ -61,6 +61,222 @@ module.exports = function(s,config){
         .raw(data.query,data.values)
         .asCallback(callback)
     }, 4);
+    const cleanSqlWhereObject = (where) => {
+        const newWhere = {}
+        Object.keys(where).forEach((key) => {
+            if(key !== '__separator'){
+                const value = where[key]
+                newWhere[key] = value
+            }
+        })
+        return newWhere
+    }
+    const processSimpleWhereCondition = (dbQuery,where,didOne) => {
+        var whereIsArray = where instanceof Array;
+        if(where[0] === 'or' || where.__separator === 'or'){
+            if(whereIsArray){
+                where.shift()
+                dbQuery.orWhere(...where)
+            }else{
+                where = cleanSqlWhereObject(where)
+                dbQuery.orWhere(where)
+            }
+        }else if(!didOne){
+            didOne = true
+            whereIsArray ? dbQuery.where(...where) : dbQuery.where(where)
+        }else{
+            whereIsArray ? dbQuery.andWhere(...where) : dbQuery.andWhere(where)
+        }
+    }
+    const processWhereCondition = (dbQuery,where,didOne) => {
+        var whereIsArray = where instanceof Array;
+        if(!where[0])return;
+        if(where[0] && where[0] instanceof Array){
+            dbQuery.where(function() {
+                var _this = this
+                var didOneInsideGroup = false
+                where.forEach((whereInsideGroup) => {
+                    processWhereCondition(_this,whereInsideGroup,didOneInsideGroup)
+                })
+            })
+        }else if(where[0] && where[0] instanceof Object){
+            dbQuery.where(function() {
+                var _this = this
+                var didOneInsideGroup = false
+                where.forEach((whereInsideGroup) => {
+                    processSimpleWhereCondition(_this,whereInsideGroup,didOneInsideGroup)
+                })
+            })
+        }else{
+            processSimpleWhereCondition(dbQuery,where,didOne)
+        }
+    }
+    const knexError = (dbQuery,options,err) => {
+        console.error('knexError----------------------------------- START')
+        if(config.debugLogVerbose && config.debugLog === true){
+            s.debugLog('s.knexQuery QUERY',JSON.stringify(options,null,3))
+            s.debugLog('STACK TRACE, NOT AN ',new Error())
+        }
+        console.error(err)
+        console.error(dbQuery.toString())
+        console.error('knexError----------------------------------- END')
+    }
+    const knexQuery = (options,callback) => {
+        try{
+            if(!s.databaseEngine)return// console.log('Database Not Set');
+            // options = {
+            //     action: "",
+            //     columns: "",
+            //     table: ""
+            // }
+            var dbQuery
+            switch(options.action){
+                case'select':
+                    options.columns = options.columns.indexOf(',') === -1 ? [options.columns] : options.columns.split(',');
+                    dbQuery = s.databaseEngine.select(...options.columns).from(options.table)
+                break;
+                case'count':
+                    options.columns = options.columns.indexOf(',') === -1 ? [options.columns] : options.columns.split(',');
+                    dbQuery = s.databaseEngine(options.table)
+                    dbQuery.count(options.columns)
+                break;
+                case'update':
+                    dbQuery = s.databaseEngine(options.table).update(options.update)
+                break;
+                case'delete':
+                    dbQuery = s.databaseEngine(options.table)
+                break;
+                case'insert':
+                    dbQuery = s.databaseEngine(options.table).insert(options.insert)
+                break;
+            }
+            if(options.where instanceof Array){
+                var didOne = false;
+                options.where.forEach((where) => {
+                    processWhereCondition(dbQuery,where,didOne)
+                })
+            }else if(options.where instanceof Object){
+                dbQuery.where(options.where)
+            }
+            if(options.action === 'delete'){
+                dbQuery.del()
+            }
+            if(options.orderBy){
+                dbQuery.orderBy(...options.orderBy)
+            }
+            if(options.groupBy){
+                dbQuery.groupBy(options.groupBy)
+            }
+            if(options.limit){
+                if(`${options.limit}`.indexOf(',') === -1){
+                    dbQuery.limit(options.limit)
+                }else{
+                    const limitParts = `${options.limit}`.split(',')
+                    dbQuery.limit(limitParts[0]).offset(limitParts[1])
+                }
+            }
+            if(config.debugLog === true){
+                console.log(dbQuery.toString())
+            }
+            if(callback || options.update || options.insert || options.action === 'delete'){
+                dbQuery.asCallback(function(err,r) {
+                    if(err){
+                        knexError(dbQuery,options,err)
+                    }
+                    if(callback)callback(err,r)
+                    if(config.debugLogVerbose && config.debugLog === true){
+                        s.debugLog('s.knexQuery QUERY',JSON.stringify(options,null,3))
+                        s.debugLog('s.knexQuery RESPONSE',JSON.stringify(r,null,3))
+                        s.debugLog('STACK TRACE, NOT AN ',new Error())
+                    }
+                })
+            }
+            return dbQuery
+        }catch(err){
+            if(callback)callback(err,[])
+            knexError(dbQuery,options,err)
+        }
+    }
+    const getDatabaseRows = function(options,callback){
+        //current cant handle `end` time
+       var whereQuery = [
+           ['ke','=',options.groupKey],
+       ]
+       const monitorRestrictions = options.monitorRestrictions
+       var frameLimit = parseInt(options.limit) || 500
+       const endIsStartTo = options.endIsStartTo
+       const chosenDate = options.date
+       const startDate = options.startDate ? s.stringToSqlTime(options.startDate) : null
+       const endDate = options.endDate ? s.stringToSqlTime(options.endDate) : null
+       const startOperator = options.startOperator || '>='
+       const endOperator = options.endOperator || '<='
+       const rowType = options.rowType || 'rows'
+       if(chosenDate){
+           if(chosenDate.indexOf('-') === -1 && !isNaN(chosenDate)){
+               chosenDate = parseInt(chosenDate)
+           }
+           var selectedDate = chosenDate
+           if(typeof chosenDate === 'string' && chosenDate.indexOf('.') > -1){
+               selectedDate = chosenDate.split('.')[0]
+           }
+           selectedDate = new Date(selectedDate)
+           var utcSelectedDate = new Date(selectedDate.getTime() + selectedDate.getTimezoneOffset() * 60000)
+           startDate = moment(utcSelectedDate).format('YYYY-MM-DD HH:mm:ss')
+           var dayAfter = utcSelectedDate
+           dayAfter.setDate(dayAfter.getDate() + 1)
+           endDate = moment(dayAfter).format('YYYY-MM-DD HH:mm:ss')
+       }
+       if(startDate){
+           if(endDate){
+               whereQuery.push(['time',startOperator,startDate])
+               whereQuery.push([endIsStartTo ? 'time' : 'end',endOperator,endDate])
+           }else{
+               whereQuery.push(['time',startOperator,startDate])
+           }
+       }
+       if(monitorRestrictions && monitorRestrictions.length > 0){
+           whereQuery.push(monitorRestrictions)
+       }
+       if(options.archived){
+           whereQuery.push(['details','LIKE',`%"archived":"1"%`])
+       }
+       if(options.filename){
+           whereQuery.push(['filename','=',options.filename])
+           frameLimit = "1";
+       }
+       options.orderBy = options.orderBy ? options.orderBy : ['time','desc']
+       if(options.count)options.groupBy = options.groupBy ? options.groupBy : options.orderBy[0]
+       knexQuery({
+           action: options.count ? "count" : "select",
+           columns: options.columns || "*",
+           table: options.table,
+           where: whereQuery,
+           orderBy: options.orderBy,
+           groupBy: options.groupBy,
+           limit: frameLimit || '500'
+       },(err,r) => {
+           if(err){
+               callback({
+                   ok: false,
+                   total: 0,
+                   limit: frameLimit,
+                   [rowType]: []
+               })
+           }else{
+               r.forEach(function(file){
+                   file.details = s.parseJSON(file.details)
+               })
+               callback({
+                   ok: true,
+                   total: r.length,
+                   limit: frameLimit,
+                   [rowType]: r
+               })
+           }
+       })
+    }
+    s.knexQuery = knexQuery
+    s.getDatabaseRows = getDatabaseRows
     s.sqlQuery = function(query,values,onMoveOn,hideLog){
         if(!values){values=[]}
         if(typeof values === 'function'){
@@ -109,94 +325,94 @@ module.exports = function(s,config){
         var mySQLtail = ''
         if(config.databaseType === 'mysql'){
             mySQLtail = ' ENGINE=InnoDB DEFAULT CHARSET=utf8'
-        }
-        //add Presets table and modernize
-        var createPresetsTableQuery = 'CREATE TABLE IF NOT EXISTS `Presets` (  `ke` varchar(50) DEFAULT NULL,  `name` text,  `details` text,  `type` varchar(50) DEFAULT NULL)'
-        s.sqlQuery( createPresetsTableQuery + mySQLtail + ';',[],function(err){
-            if(err)console.error(err)
-            if(config.databaseType === 'sqlite3'){
-                var aQuery = "ALTER TABLE Presets RENAME TO _Presets_old;"
-                    aQuery += createPresetsTableQuery
-                    aQuery += "INSERT INTO Presets (`ke`, `name`, `details`, `type`) SELECT `ke`, `name`, `details`, `type` FROM _Presets_old;COMMIT;DROP TABLE _Presets_old;"
-            }else{
-                s.sqlQuery('ALTER TABLE `Presets` CHANGE COLUMN `type` `type` VARCHAR(50) NULL DEFAULT NULL AFTER `details`;',[],function(err){
-                    if(err)console.error(err)
-                },true)
-            }
-        },true)
-        //add Schedules table, will remove in future
-        s.sqlQuery("CREATE TABLE IF NOT EXISTS `Schedules` (`ke` varchar(50) DEFAULT NULL,`name` text,`details` text,`start` varchar(10) DEFAULT NULL,`end` varchar(10) DEFAULT NULL,`enabled` int(1) NOT NULL DEFAULT '1')" + mySQLtail + ';',[],function(err){
-            if(err)console.error(err)
-        },true)
-        //add Timelapses and Timelapse Frames tables, will remove in future
-        s.sqlQuery("CREATE TABLE IF NOT EXISTS `Timelapses` (`ke` varchar(50) NOT NULL,`mid` varchar(50) NOT NULL,`details` longtext,`date` date NOT NULL,`time` timestamp NOT NULL,`end` timestamp NOT NULL,`size` int(11)NOT NULL)" + mySQLtail + ';',[],function(err){
-            if(err)console.error(err)
-        },true)
-        s.sqlQuery("CREATE TABLE IF NOT EXISTS `Timelapse Frames` (`ke` varchar(50) NOT NULL,`mid` varchar(50) NOT NULL,`details` longtext,`filename` varchar(50) NOT NULL,`time` timestamp NULL DEFAULT NULL,`size` int(11) NOT NULL)" + mySQLtail + ';',[],function(err){
-            if(err)console.error(err)
-        },true)
-        //Add index to Videos table
-        s.sqlQuery('CREATE INDEX `videos_index` ON Videos(`time`);',[],function(err){
-            if(err && err.code !== 'ER_DUP_KEYNAME'){
-                console.error(err)
-            }
-        },true)
-        //Add index to Events table
-        s.sqlQuery('CREATE INDEX `events_index` ON Events(`ke`, `mid`, `time`);',[],function(err){
-            if(err && err.code !== 'ER_DUP_KEYNAME'){
-                console.error(err)
-            }
-        },true)
-        //Add index to Logs table
-        s.sqlQuery('CREATE INDEX `logs_index` ON Logs(`ke`, `mid`, `time`);',[],function(err){
-            if(err && err.code !== 'ER_DUP_KEYNAME'){
-                console.error(err)
-            }
-        },true)
-        //Add index to Monitors table
-        s.sqlQuery('CREATE INDEX `monitors_index` ON Monitors(`ke`, `mode`, `type`, `ext`);',[],function(err){
-            if(err && err.code !== 'ER_DUP_KEYNAME'){
-                console.error(err)
-            }
-        },true)
-        //Add index to Timelapse Frames table
-        s.sqlQuery('CREATE INDEX `timelapseframes_index` ON `Timelapse Frames`(`ke`, `mid`, `time`);',[],function(err){
-            if(err && err.code !== 'ER_DUP_KEYNAME'){
-                console.error(err)
-            }
-        },true)
-        //add Cloud Videos table, will remove in future
-        s.sqlQuery('CREATE TABLE IF NOT EXISTS `Cloud Videos` (`mid` varchar(50) NOT NULL,`ke` varchar(50) DEFAULT NULL,`href` text NOT NULL,`size` float DEFAULT NULL,`time` timestamp NULL DEFAULT NULL,`end` timestamp NULL DEFAULT NULL,`status` int(1) DEFAULT \'0\',`details` text)' + mySQLtail + ';',[],function(err){
-            if(err)console.error(err)
-        },true)
-        //add Events Counts table, will remove in future
-        s.sqlQuery('CREATE TABLE IF NOT EXISTS `Events Counts` (`ke` varchar(50) NOT NULL,`mid` varchar(50) NOT NULL,`details` longtext NOT NULL,`time` timestamp NOT NULL DEFAULT current_timestamp(),`end` timestamp NOT NULL DEFAULT current_timestamp(),`count` int(10) NOT NULL DEFAULT 1,`tag` varchar(30) DEFAULT NULL)' + mySQLtail + ';',[],function(err){
-            if(err && err.code !== 'ER_TABLE_EXISTS_ERROR'){
-                console.error(err)
-            }
-            s.sqlQuery('ALTER TABLE `Events Counts`	ADD COLUMN `time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP AFTER `details`;',[],function(err){
-                // console.error(err)
+            //add Presets table and modernize
+            var createPresetsTableQuery = 'CREATE TABLE IF NOT EXISTS `Presets` (  `ke` varchar(50) DEFAULT NULL,  `name` text,  `details` text,  `type` varchar(50) DEFAULT NULL)'
+            s.sqlQuery( createPresetsTableQuery + mySQLtail + ';',[],function(err){
+                if(err)console.error(err)
+                if(config.databaseType === 'sqlite3'){
+                    var aQuery = "ALTER TABLE Presets RENAME TO _Presets_old;"
+                        aQuery += createPresetsTableQuery
+                        aQuery += "INSERT INTO Presets (`ke`, `name`, `details`, `type`) SELECT `ke`, `name`, `details`, `type` FROM _Presets_old;COMMIT;DROP TABLE _Presets_old;"
+                }else{
+                    s.sqlQuery('ALTER TABLE `Presets` CHANGE COLUMN `type` `type` VARCHAR(50) NULL DEFAULT NULL AFTER `details`;',[],function(err){
+                        if(err)console.error(err)
+                    },true)
+                }
             },true)
-        },true)
-        //add Cloud Timelapse Frames table, will remove in future
-        s.sqlQuery('CREATE TABLE IF NOT EXISTS `Cloud Timelapse Frames` (`ke` varchar(50) NOT NULL,`mid` varchar(50) NOT NULL,`href` text NOT NULL,`details` longtext,`filename` varchar(50) NOT NULL,`time` timestamp NULL DEFAULT NULL,`size` int(11) NOT NULL)' + mySQLtail + ';',[],function(err){
-            if(err)console.error(err)
-        },true)
-        //create Files table
-        var createFilesTableQuery = "CREATE TABLE IF NOT EXISTS `Files` (`ke` varchar(50) NOT NULL,`mid` varchar(50) NOT NULL,`name` tinytext NOT NULL,`size` float NOT NULL DEFAULT '0',`details` text NOT NULL,`status` int(1) NOT NULL DEFAULT '0',`time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP)"
-        s.sqlQuery(createFilesTableQuery + mySQLtail + ';',[],function(err){
-            if(err)console.error(err)
-            //add time column to Files table
-            if(config.databaseType === 'sqlite3'){
-                var aQuery = "ALTER TABLE Files RENAME TO _Files_old;"
-                    aQuery += createPresetsTableQuery
-                    aQuery += "INSERT INTO Files (`ke`, `mid`, `name`, `details`, `size`, `status`, `time`) SELECT `ke`, `mid`, `name`, `details`, `size`, `status`, `time` FROM _Files_old;COMMIT;DROP TABLE _Files_old;"
-            }else{
-                s.sqlQuery('ALTER TABLE `Files`	ADD COLUMN `time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP AFTER `status`;',[],function(err){
-                    if(err && err.sqlMessage && err.sqlMessage.indexOf('Duplicate') === -1)console.error(err)
+            //add Schedules table, will remove in future
+            s.sqlQuery("CREATE TABLE IF NOT EXISTS `Schedules` (`ke` varchar(50) DEFAULT NULL,`name` text,`details` text,`start` varchar(10) DEFAULT NULL,`end` varchar(10) DEFAULT NULL,`enabled` int(1) NOT NULL DEFAULT '1')" + mySQLtail + ';',[],function(err){
+                if(err)console.error(err)
+            },true)
+            //add Timelapses and Timelapse Frames tables, will remove in future
+            s.sqlQuery("CREATE TABLE IF NOT EXISTS `Timelapses` (`ke` varchar(50) NOT NULL,`mid` varchar(50) NOT NULL,`details` longtext,`date` date NOT NULL,`time` timestamp NOT NULL,`end` timestamp NOT NULL,`size` int(11)NOT NULL)" + mySQLtail + ';',[],function(err){
+                if(err)console.error(err)
+            },true)
+            s.sqlQuery("CREATE TABLE IF NOT EXISTS `Timelapse Frames` (`ke` varchar(50) NOT NULL,`mid` varchar(50) NOT NULL,`details` longtext,`filename` varchar(50) NOT NULL,`time` timestamp NULL DEFAULT NULL,`size` int(11) NOT NULL)" + mySQLtail + ';',[],function(err){
+                if(err)console.error(err)
+            },true)
+            //Add index to Videos table
+            s.sqlQuery('CREATE INDEX `videos_index` ON Videos(`time`);',[],function(err){
+                if(err && err.code !== 'ER_DUP_KEYNAME'){
+                    console.error(err)
+                }
+            },true)
+            //Add index to Events table
+            s.sqlQuery('CREATE INDEX `events_index` ON Events(`ke`, `mid`, `time`);',[],function(err){
+                if(err && err.code !== 'ER_DUP_KEYNAME'){
+                    console.error(err)
+                }
+            },true)
+            //Add index to Logs table
+            s.sqlQuery('CREATE INDEX `logs_index` ON Logs(`ke`, `mid`, `time`);',[],function(err){
+                if(err && err.code !== 'ER_DUP_KEYNAME'){
+                    console.error(err)
+                }
+            },true)
+            //Add index to Monitors table
+            s.sqlQuery('CREATE INDEX `monitors_index` ON Monitors(`ke`, `mode`, `type`, `ext`);',[],function(err){
+                if(err && err.code !== 'ER_DUP_KEYNAME'){
+                    console.error(err)
+                }
+            },true)
+            //Add index to Timelapse Frames table
+            s.sqlQuery('CREATE INDEX `timelapseframes_index` ON `Timelapse Frames`(`ke`, `mid`, `time`);',[],function(err){
+                if(err && err.code !== 'ER_DUP_KEYNAME'){
+                    console.error(err)
+                }
+            },true)
+            //add Cloud Videos table, will remove in future
+            s.sqlQuery('CREATE TABLE IF NOT EXISTS `Cloud Videos` (`mid` varchar(50) NOT NULL,`ke` varchar(50) DEFAULT NULL,`href` text NOT NULL,`size` float DEFAULT NULL,`time` timestamp NULL DEFAULT NULL,`end` timestamp NULL DEFAULT NULL,`status` int(1) DEFAULT \'0\',`details` text)' + mySQLtail + ';',[],function(err){
+                if(err)console.error(err)
+            },true)
+            //add Events Counts table, will remove in future
+            s.sqlQuery('CREATE TABLE IF NOT EXISTS `Events Counts` (`ke` varchar(50) NOT NULL,`mid` varchar(50) NOT NULL,`details` longtext NOT NULL,`time` timestamp NOT NULL DEFAULT current_timestamp(),`end` timestamp NOT NULL DEFAULT current_timestamp(),`count` int(10) NOT NULL DEFAULT 1,`tag` varchar(30) DEFAULT NULL)' + mySQLtail + ';',[],function(err){
+                if(err && err.code !== 'ER_TABLE_EXISTS_ERROR'){
+                    console.error(err)
+                }
+                s.sqlQuery('ALTER TABLE `Events Counts`	ADD COLUMN `time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP AFTER `details`;',[],function(err){
+                    // console.error(err)
                 },true)
-            }
-        },true)
+            },true)
+            //add Cloud Timelapse Frames table, will remove in future
+            s.sqlQuery('CREATE TABLE IF NOT EXISTS `Cloud Timelapse Frames` (`ke` varchar(50) NOT NULL,`mid` varchar(50) NOT NULL,`href` text NOT NULL,`details` longtext,`filename` varchar(50) NOT NULL,`time` timestamp NULL DEFAULT NULL,`size` int(11) NOT NULL)' + mySQLtail + ';',[],function(err){
+                if(err)console.error(err)
+            },true)
+            //create Files table
+            var createFilesTableQuery = "CREATE TABLE IF NOT EXISTS `Files` (`ke` varchar(50) NOT NULL,`mid` varchar(50) NOT NULL,`name` tinytext NOT NULL,`size` float NOT NULL DEFAULT '0',`details` text NOT NULL,`status` int(1) NOT NULL DEFAULT '0',`time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP)"
+            s.sqlQuery(createFilesTableQuery + mySQLtail + ';',[],function(err){
+                if(err)console.error(err)
+                //add time column to Files table
+                if(config.databaseType === 'sqlite3'){
+                    var aQuery = "ALTER TABLE Files RENAME TO _Files_old;"
+                        aQuery += createPresetsTableQuery
+                        aQuery += "INSERT INTO Files (`ke`, `mid`, `name`, `details`, `size`, `status`, `time`) SELECT `ke`, `mid`, `name`, `details`, `size`, `status`, `time` FROM _Files_old;COMMIT;DROP TABLE _Files_old;"
+                }else{
+                    s.sqlQuery('ALTER TABLE `Files`	ADD COLUMN `time` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP AFTER `status`;',[],function(err){
+                        if(err && err.sqlMessage && err.sqlMessage.indexOf('Duplicate') === -1)console.error(err)
+                    },true)
+                }
+            },true)
+        }
         delete(s.preQueries)
     }
     s.sqlQueryBetweenTimesWithPermissions = (options,callback) => {
@@ -214,114 +430,51 @@ module.exports = function(s,config){
         //     parseRowDetails: true,
         //     rowName: 'counts'
         // }
+        const rowName = options.rowName || 'rows'
+        const preliminaryValidationFailed = options.preliminaryValidationFailed || false
+        if(preliminaryValidationFailed){
+            if(options.noFormat){
+                callback([]);
+            }else{
+                callback({
+                    ok: true,
+                    [rowName]: [],
+                })
+            }
+            return
+        }
         const user = options.user
         const groupKey = options.groupKey
         const monitorId = options.monitorId
-        const limit = options.limit
         const archived = options.archived
         const theTableSelected = options.table
         const endIsStartTo = options.endIsStartTo
         const userDetails = user.details
-        const rowName = options.rowName || 'rows'
-        const preliminaryValidationFailed = options.preliminaryValidationFailed || false
         var endTime = options.endTime
         var startTimeOperator = options.startTimeOperator
         var endTimeOperator = options.endTimeOperator
         var startTime = options.startTime
-        if(preliminaryValidationFailed){
-            callback([]);
-            return
-        }
-        var queryString = 'SELECT * FROM `' + theTableSelected + '` WHERE ke=?'
-        var queryValues = [groupKey]
-        var queryStringCount = 'SELECT COUNT(*) FROM `' + theTableSelected + '` WHERE ke=?'
-        var queryCountValues = [groupKey]
-        if(archived === '1'){
-            queryString += ` AND details LIKE '%"archived":"1"'`
-            queryStringCount += ` AND details LIKE '%"archived":"1"'`
-        }
-        if(!monitorId){
-            if(
-                userDetails.sub &&
-                userDetails.monitors &&
-                userDetails.allmonitors !== '1'
-            ){
-                try{
-                    userDetails.monitors = JSON.parse(userDetails.monitors)
-                }catch(er){}
-                var queryWheres = []
-                userDetails.monitors.forEach(function(v,n){
-                    queryWheres.push('mid=?')
-                    queryValues.push(v)
-                })
-                queryString += ' AND ('+queryWheres.join(' OR ')+')'
-                queryStringCount += ' AND ('+queryWheres.join(' OR ')+')'
-            }
-        }else{
-            if(
-                !userDetails.sub ||
-                userDetails.allmonitors !== '0' ||
-                userDetails.monitors.indexOf(monitorId) >- 1
-            ){
-                queryString += ' and mid=?'
-                queryValues.push(monitorId)
-                queryStringCount += ' and mid=?'
-                queryCountValues.push(monitorId)
-            }else{
-                res.end('[]');
-                return;
-            }
-        }
-        if(startTime || endTime){
-            if(startTime && startTime !== ''){
-                startTime = s.stringToSqlTime(startTime)
-            }
-            if(endTime && endTime !== ''){
-                endTime = s.stringToSqlTime(endTime)
-            }
-            if(!startTimeOperator || startTimeOperator==''){
-                startTimeOperator = startTimeOperator || '>='
-            }
-            if(!endTimeOperator || endTimeOperator==''){
-                endTimeOperator = endTimeOperator || '<='
-            }
-            var theEndParameter = '`end`'
-            if(endIsStartTo){
-                theEndParameter = '`time`'
-            }
-            switch(true){
-                case(startTime && startTime !== '' && endTime && endTime !== ''):
-                    queryString += ' AND `time` '+startTimeOperator+' ? AND '+theEndParameter+' '+endTimeOperator+' ?';
-                    queryStringCount += ' AND `time` '+startTimeOperator+' ? AND '+theEndParameter+' '+endTimeOperator+' ?';
-                    queryValues.push(startTime)
-                    queryValues.push(endTime)
-                    queryCountValues.push(startTime)
-                    queryCountValues.push(endTime)
-                break;
-                case(startTime && startTime !== ''):
-                    queryString += ' AND `time` '+startTimeOperator+' ?';
-                    queryStringCount += ' AND `time` '+startTimeOperator+' ?';
-                    queryValues.push(startTime)
-                    queryCountValues.push(startTime)
-                break;
-                case(endTime && endTime !== ''):
-                    queryString += ' AND '+theEndParameter+' '+endTimeOperator+' ?';
-                    queryStringCount += ' AND '+theEndParameter+' '+endTimeOperator+' ?';
-                    queryValues.push(endTime)
-                    queryCountValues.push(endTime)
-                break;
-            }
-        }
-        queryString += ' ORDER BY `time` DESC';
-        var rowLimit = limit || '100'
-        if(rowLimit !== '0'){
-            queryString += ' LIMIT ' + rowLimit
-        }
-        s.sqlQuery(queryString,queryValues,function(err,r){
+        var limitString = `${options.limit}`
+        const monitorRestrictions = s.getMonitorRestrictions(options.user.details,monitorId)
+        getDatabaseRows({
+            monitorRestrictions: monitorRestrictions,
+            table: theTableSelected,
+            groupKey: groupKey,
+            startDate: startTime,
+            endDate: endTime,
+            startOperator: startTimeOperator,
+            endOperator: endTimeOperator,
+            limit: options.limit,
+            archived: archived,
+            rowType: rowName,
+            endIsStartTo: endIsStartTo
+        },(response) => {
+            const limit = response.limit
+            const r = response[rowName];
             if(!r){
                 callback({
                     total: 0,
-                    limit: rowLimit,
+                    limit: response.limit,
                     skip: 0,
                     [rowName]: []
                 });
@@ -338,22 +491,39 @@ module.exports = function(s,config){
                 }else{
                     callback({
                         ok: true,
+                        limit: response.limit,
                         [rowName]: r,
                         endIsStartTo: endIsStartTo
                     })
                 }
             }else{
-                s.sqlQuery(queryStringCount,queryCountValues,function(err,count){
+                getDatabaseRows({
+                    monitorRestrictions: monitorRestrictions,
+                    columns: 'time',
+                    count: true,
+                    table: theTableSelected,
+                    groupKey: groupKey,
+                    startDate: startTime,
+                    endDate: endTime,
+                    startOperator: startTimeOperator,
+                    endOperator: endTimeOperator,
+                    archived: archived,
+                    type: 'count',
+                    endIsStartTo: endIsStartTo
+                },(response) => {
+                    console.log('count')
+                    console.log(response)
+                    const count = response.count
                     var skipOver = 0
-                    if(rowLimit.indexOf(',') > -1){
-                        skipOver = parseInt(rowLimit.split(',')[0])
-                        rowLimit = parseInt(rowLimit.split(',')[1])
+                    if(limitString.indexOf(',') > -1){
+                        skipOver = parseInt(limitString.split(',')[0])
+                        limitString = parseInt(limitString.split(',')[1])
                     }else{
-                        rowLimit = parseInt(rowLimit)
+                        limitString = parseInt(limitString)
                     }
                     callback({
-                        total: count[0]['COUNT(*)'],
-                        limit: rowLimit,
+                        total: response['count(*)'],
+                        limit: response.limit,
                         skip: skipOver,
                         [rowName]: r,
                         endIsStartTo: endIsStartTo

libs/startup.js

@@ -43,7 +43,11 @@ module.exports = function(s,config,lang,io){
         })
         s.systemLog(lang.startUpText4)
         //preliminary monitor start
-        s.sqlQuery('SELECT * FROM Monitors', function(err,monitors) {
+        s.knexQuery({
+            action: "select",
+            columns: "*",
+            table: "Monitors",
+        },function(err,monitors) {
             foundMonitors = monitors
             if(err){s.systemLog(err)}
             if(monitors && monitors[0]){
@@ -115,9 +119,31 @@ module.exports = function(s,config,lang,io){
         s.group[user.ke].sizeLimit = parseFloat(userDetails.size) || 10000
         s.group[user.ke].sizeLimitVideoPercent = parseFloat(userDetails.size_video_percent) || 90
         s.group[user.ke].sizeLimitTimelapseFramesPercent = parseFloat(userDetails.size_timelapse_percent) || 10
-        s.sqlQuery('SELECT * FROM Videos WHERE ke=? AND status!=?',[user.ke,0],function(err,videos){
-            s.sqlQuery('SELECT * FROM `Timelapse Frames` WHERE ke=?',[user.ke],function(err,timelapseFrames){
-                s.sqlQuery('SELECT * FROM `Files` WHERE ke=?',[user.ke],function(err,files){
+        s.knexQuery({
+            action: "select",
+            columns: "*",
+            table: "Videos",
+            where: [
+                ['ke','=',user.ke],
+                ['status','!=',0],
+            ]
+        },function(err,videos) {
+            s.knexQuery({
+                action: "select",
+                columns: "*",
+                table: "Timelapse Frames",
+                where: [
+                    ['ke','=',user.ke],
+                ]
+            },function(err,timelapseFrames) {
+                s.knexQuery({
+                    action: "select",
+                    columns: "*",
+                    table: "Files",
+                    where: [
+                        ['ke','=',user.ke],
+                    ]
+                },function(err,files) {
                     var usedSpaceVideos = 0
                     var usedSpaceTimelapseFrames = 0
                     var usedSpaceFilebin = 0
@@ -180,7 +206,15 @@ module.exports = function(s,config,lang,io){
             if(s.cloudDiskUseStartupExtensions[storageType])s.cloudDiskUseStartupExtensions[storageType](user,userDetails)
         })
         var loadCloudVideos = function(callback){
-            s.sqlQuery('SELECT * FROM `Cloud Videos` WHERE ke=? AND status!=?',[user.ke,0],function(err,videos){
+            s.knexQuery({
+                action: "select",
+                columns: "*",
+                table: "Cloud Videos",
+                where: [
+                    ['ke','=',user.ke],
+                    ['status','!=',0],
+                ]
+            },function(err,videos) {
                 if(videos && videos[0]){
                     videos.forEach(function(video){
                         var storageType = JSON.parse(video.details).type
@@ -200,7 +234,14 @@ module.exports = function(s,config,lang,io){
             })
         }
         var loadCloudTimelapseFrames = function(callback){
-            s.sqlQuery('SELECT * FROM `Cloud Timelapse Frames` WHERE ke=?',[user.ke],function(err,frames){
+            s.knexQuery({
+                action: "select",
+                columns: "*",
+                table: "Cloud Timelapse Frames",
+                where: [
+                    ['ke','=',user.ke],
+                ]
+            },function(err,frames) {
                 if(frames && frames[0]){
                     frames.forEach(function(frame){
                         var storageType = JSON.parse(frame.details).type
@@ -284,7 +325,14 @@ module.exports = function(s,config,lang,io){
     }
     var loadAdminUsers = function(callback){
         //get current disk used for each isolated account (admin user) on startup
-        s.sqlQuery('SELECT * FROM Users WHERE details NOT LIKE ?',['%"sub"%'],function(err,users){
+        s.knexQuery({
+            action: "select",
+            columns: "*",
+            table: "Users",
+            where: [
+                ['details','NOT LIKE','%"sub"%']
+            ]
+        },function(err,users) {
             if(users && users[0]){
                 users.forEach(function(user){
                     checkedAdminUsers[user.ke] = user
@@ -370,13 +418,6 @@ module.exports = function(s,config,lang,io){
     }
     if(config.childNodes.mode !== 'child'){
         //master node - startup functions
-        setInterval(function(){
-            s.cpuUsage(function(cpu){
-                s.ramUsage(function(ram){
-                    s.tx({f:'os',cpu:cpu,ram:ram},'CPU');
-                })
-            })
-        },10000)
         //hourly check to see if sizePurge has failed to unlock
         //checks to see if request count is the number of monitors + 10
         s.checkForStalePurgeLocks()
@@ -385,7 +426,7 @@ module.exports = function(s,config,lang,io){
         s.databaseEngine = require('knex')(s.databaseOptions)
         //run prerequsite queries
         s.preQueries()
-        setTimeout(function(){
+        setTimeout(() => {
             //check for subscription
             checkSubscription(function(){
                 //check terminal commander
@@ -395,7 +436,7 @@ module.exports = function(s,config,lang,io){
                         //load monitors (for groups)
                         loadMonitors(function(){
                             //check for orphaned videos
-                            checkForOrphanedVideos(function(){
+                            checkForOrphanedVideos(async () => {
                                 s.processReady()
                             })
                         })

libs/timelapse.js

@@ -74,9 +74,13 @@ module.exports = function(s,config,lang,app,io){
         }
     }
     s.insertTimelapseFrameDatabaseRow = function(e,queryInfo,filePath){
-        s.sqlQuery('INSERT INTO `Timelapse Frames` ('+Object.keys(queryInfo).join(',')+') VALUES (?,?,?,?,?,?)',Object.values(queryInfo))
-        s.setDiskUsedForGroup(e,queryInfo.size / 1048576,'timelapeFrames')
-        s.purgeDiskForGroup(e)
+        s.knexQuery({
+            action: "insert",
+            table: "Timelapse Frames",
+            insert: queryInfo
+        })
+        s.setDiskUsedForGroup(e.ke,queryInfo.size / 1048576,'timelapeFrames')
+        s.purgeDiskForGroup(e.ke)
         s.onInsertTimelapseFrameExtensions.forEach(function(extender){
             extender(e,queryInfo,filePath)
         })
@@ -103,11 +107,26 @@ module.exports = function(s,config,lang,app,io){
     s.deleteTimelapseFrameFromCloud = function(e){
         // e = video object
         s.checkDetails(e)
-        var frameSelector = [e.id,e.ke,new Date(e.time)]
-        s.sqlQuery('SELECT * FROM `Cloud Timelapse Frames` WHERE `mid`=? AND `ke`=? AND `time`=?',frameSelector,function(err,r){
-            if(r&&r[0]){
+        var frameSelector = {
+            ke: e.ke,
+            mid: e.id,
+            time: new Date(e.time),
+        }
+        s.knexQuery({
+            action: "select",
+            columns: "*",
+            table: "Cloud Timelapse Frames",
+            where: frameSelector,
+            limit: 1
+        },function(err,r){
+            if(r && r[0]){
                 r = r[0]
-                s.sqlQuery('DELETE FROM `Cloud Timelapse Frames` WHERE `mid`=? AND `ke`=? AND `time`=?',frameSelector,function(){
+                s.knexQuery({
+                    action: "delete",
+                    table: "Cloud Timelapse Frames",
+                    where: frameSelector,
+                    limit: 1
+                },function(){
                     s.onDeleteTimelapseFrameFromCloudExtensionsRunner(e,r)
                 })
             }else{
@@ -131,112 +150,54 @@ module.exports = function(s,config,lang,app,io){
             var hasRestrictions = user.details.sub && user.details.allmonitors !== '1'
             if(
                 user.permissions.watch_videos==="0" ||
-                hasRestrictions && (!user.details.video_view || user.details.video_view.indexOf(req.params.id)===-1)
+                hasRestrictions &&
+                (
+                    !user.details.video_view ||
+                    user.details.video_view.indexOf(req.params.id) === -1
+                )
             ){
-                res.end(s.prettyPrint([]))
+                s.closeJsonResponse(res,[])
                 return
             }
-            req.sql='SELECT * FROM `Timelapse Frames` WHERE ke=?';req.ar=[req.params.ke];
-            if(req.query.archived=='1'){
-                req.sql+=' AND details LIKE \'%"archived":"1"\''
-            }
-            if(!req.params.id){
-                if(user.details.sub&&user.details.monitors&&user.details.allmonitors!=='1'){
-                    try{user.details.monitors=JSON.parse(user.details.monitors);}catch(er){}
-                    req.or=[];
-                    user.details.monitors.forEach(function(v,n){
-                        req.or.push('mid=?');req.ar.push(v)
-                    })
-                    req.sql+=' AND ('+req.or.join(' OR ')+')'
-                }
-            }else{
-                if(!user.details.sub||user.details.allmonitors!=='0'||user.details.monitors.indexOf(req.params.id)>-1){
-                    req.sql+=' and mid=?'
-                    req.ar.push(req.params.id)
-                }else{
-                    res.end('[]');
-                    return;
-                }
-            }
-            var isMp4Call = false
-            if(req.query.mp4){
-                isMp4Call = true
-            }
-            if(req.params.date){
-                if(req.params.date.indexOf('-') === -1 && !isNaN(req.params.date)){
-                    req.params.date = parseInt(req.params.date)
-                }
-                var selectedDate = req.params.date
-                if(typeof req.params.date === 'string' && req.params.date.indexOf('.') > -1){
-                    isMp4Call = true
-                    selectedDate = req.params.date.split('.')[0]
-                }
-                selectedDate = new Date(selectedDate)
-                var utcSelectedDate = new Date(selectedDate.getTime() + selectedDate.getTimezoneOffset() * 60000)
-                req.query.start = moment(utcSelectedDate).format('YYYY-MM-DD HH:mm:ss')
-                var dayAfter = utcSelectedDate
-                dayAfter.setDate(dayAfter.getDate() + 1)
-                req.query.end = moment(dayAfter).format('YYYY-MM-DD HH:mm:ss')
-            }
-            if(req.query.start||req.query.end){
-                if(!req.query.startOperator||req.query.startOperator==''){
-                    req.query.startOperator='>='
-                }
-                if(!req.query.endOperator||req.query.endOperator==''){
-                    req.query.endOperator='<='
-                }
-                if(req.query.start && req.query.start !== '' && req.query.end && req.query.end !== ''){
-                    req.query.start = s.stringToSqlTime(req.query.start)
-                    req.query.end = s.stringToSqlTime(req.query.end)
-                    req.sql+=' AND `time` '+req.query.startOperator+' ? AND `time` '+req.query.endOperator+' ?';
-                    req.ar.push(req.query.start)
-                    req.ar.push(req.query.end)
-                }else if(req.query.start && req.query.start !== ''){
-                    req.query.start = s.stringToSqlTime(req.query.start)
-                    req.sql+=' AND `time` '+req.query.startOperator+' ?';
-                    req.ar.push(req.query.start)
-                }
-            }
-            // if(!req.query.limit||req.query.limit==''){req.query.limit=288}
-            req.sql+=' ORDER BY `time` DESC'
-            s.sqlQuery(req.sql,req.ar,function(err,r){
-                if(isMp4Call){
-                    if(r && r[0]){
-                        s.createVideoFromTimelapse(r,req.query.fps,function(response){
-                            if(response.fileExists){
-                                if(req.query.download){
-                                    res.setHeader('Content-Type', 'video/mp4')
-                                    s.streamMp4FileOverHttp(response.fileLocation,req,res)
-                                }else{
-                                    res.setHeader('Content-Type', 'application/json')
-                                    res.end(s.prettyPrint({
-                                        ok : response.ok,
-                                        fileExists : response.fileExists,
-                                        msg : response.msg,
-                                    }))
-                                }
+            const monitorRestrictions = s.getMonitorRestrictions(user.details,req.params.id)
+            s.getDatabaseRows({
+                monitorRestrictions: monitorRestrictions,
+                table: 'Timelapse Frames',
+                groupKey: req.params.ke,
+                date: req.query.date,
+                startDate: req.query.start,
+                endDate: req.query.end,
+                startOperator: req.query.startOperator,
+                endOperator: req.query.endOperator,
+                limit: req.query.limit,
+                archived: req.query.archived,
+                rowType: 'frames',
+                endIsStartTo: true
+            },(response) => {
+                var isMp4Call = !!(req.query.mp4 || (req.params.date && typeof req.params.date === 'string' && req.params.date.indexOf('.') > -1))
+                if(isMp4Call && response.frames[0]){
+                    s.createVideoFromTimelapse(response.frames,req.query.fps,function(response){
+                        if(response.fileExists){
+                            if(req.query.download){
+                                res.setHeader('Content-Type', 'video/mp4')
+                                s.streamMp4FileOverHttp(response.fileLocation,req,res)
                             }else{
-                                res.setHeader('Content-Type', 'application/json')
-                                res.end(s.prettyPrint({
+                                s.closeJsonResponse(res,{
                                     ok : response.ok,
                                     fileExists : response.fileExists,
                                     msg : response.msg,
-                                }))
+                                })
                             }
-                        })
-                    }else{
-                        res.setHeader('Content-Type', 'application/json');
-                        res.end(s.prettyPrint([]))
-                    }
+                        }else{
+                            s.closeJsonResponse(res,{
+                                ok : response.ok,
+                                fileExists : response.fileExists,
+                                msg : response.msg,
+                            })
+                        }
+                    })
                 }else{
-                    if(r && r[0]){
-                        r.forEach(function(file){
-                            file.details = s.parseJSON(file.details)
-                        })
-                        res.end(s.prettyPrint(r))
-                    }else{
-                        res.end(s.prettyPrint([]))
-                    }
+                    s.closeJsonResponse(res,response.frames)
                 }
             })
         },res,req);
@@ -257,35 +218,18 @@ module.exports = function(s,config,lang,app,io){
                 res.end(s.prettyPrint([]))
                 return
             }
-            req.sql='SELECT * FROM `Timelapse Frames` WHERE ke=?';req.ar=[req.params.ke];
-            if(req.query.archived=='1'){
-                req.sql+=' AND details LIKE \'%"archived":"1"\''
-            }
-            if(!req.params.id){
-                if(user.details.sub&&user.details.monitors&&user.details.allmonitors!=='1'){
-                    try{user.details.monitors=JSON.parse(user.details.monitors);}catch(er){}
-                    req.or=[];
-                    user.details.monitors.forEach(function(v,n){
-                        req.or.push('mid=?');req.ar.push(v)
-                    })
-                    req.sql+=' AND ('+req.or.join(' OR ')+')'
-                }
-            }else{
-                if(!user.details.sub||user.details.allmonitors!=='0'||user.details.monitors.indexOf(req.params.id)>-1){
-                    req.sql+=' and mid=?'
-                    req.ar.push(req.params.id)
-                }else{
-                    res.end('[]');
-                    return;
-                }
-            }
-            req.sql+=' AND filename=?'
-            req.ar.push(req.params.filename)
-            req.sql+=' ORDER BY `time` DESC'
-            s.sqlQuery(req.sql,req.ar,function(err,r){
-                if(r && r[0]){
-                    var frame = r[0]
-                    frame.details = s.parseJSON(frame.details)
+            const monitorRestrictions = s.getMonitorRestrictions(user.details,req.params.id)
+            s.getDatabaseRows({
+                monitorRestrictions: monitorRestrictions,
+                table: 'Timelapse Frames',
+                groupKey: req.params.ke,
+                archived: req.query.archived,
+                filename: req.params.filename,
+                rowType: 'frames',
+                endIsStartTo: true
+            },(response) => {
+                var frame = response.frames[0]
+                if(frame){
                     var fileLocation
                     if(frame.details.dir){
                         fileLocation = `${s.checkCorrectPathEnding(frame.details.dir)}`
@@ -303,11 +247,11 @@ module.exports = function(s,config,lang,app,io){
                             res.on('finish',function(){res.end()})
                             fs.createReadStream(fileLocation).pipe(res)
                         }else{
-                            res.end(s.prettyPrint({ok: false, msg: lang[`Nothing exists`]}))
+                            s.closeJsonResponse(res,{ok: false, msg: lang[`Nothing exists`]})
                         }
                     })
                 }else{
-                    res.end(s.prettyPrint({ok: false, msg: lang[`Nothing exists`]}))
+                    s.closeJsonResponse(res,{ok: false, msg: lang[`Nothing exists`]})
                 }
             })
         },res,req);
@@ -338,7 +282,15 @@ module.exports = function(s,config,lang,app,io){
         if(hoursNow === 1){
             var dateNowMoment = moment(dateNow).utc().format('YYYY-MM-DDTHH:mm:ss')
             var dateMinusOneDay = moment(dateNow).utc().subtract(1, 'days').format('YYYY-MM-DDTHH:mm:ss')
-            s.sqlQuery('SELECT * FROM `Timelapse Frames` WHERE time => ? AND time =< ?',[dateMinusOneDay,dateNowMoment],function(err,frames){
+            s.knexQuery({
+                action: "select",
+                columns: "*",
+                table: "Timelapse Frames",
+                where: [
+                    ['time','=>',dateMinusOneDay],
+                    ['time','=<',dateNowMoment],
+                ]
+            },function(err,frames) {
                 console.log(frames.length)
                 var groups = {}
                 frames.forEach(function(frame){

libs/uploaders/amazonS3.js

@@ -3,8 +3,8 @@ module.exports = function(s,config,lang){
     //Amazon S3
     var beforeAccountSaveForAmazonS3 = function(d){
         //d = save event
-        d.form.details.aws_use_global=d.d.aws_use_global
-        d.form.details.use_aws_s3=d.d.use_aws_s3
+        d.formDetails.aws_use_global=d.d.aws_use_global
+        d.formDetails.use_aws_s3=d.d.use_aws_s3
     }
     var cloudDiskUseStartupForAmazonS3 = function(group,userDetails){
         group.cloudDiskUse['s3'].name = 'Amazon S3'
@@ -100,23 +100,26 @@ module.exports = function(s,config,lang){
                     s.userLog(e,{type:lang['Amazon S3 Upload Error'],msg:err})
                 }
                 if(s.group[e.ke].init.aws_s3_log === '1' && data && data.Location){
-                    var save = [
-                        e.mid,
-                        e.ke,
-                        k.startTime,
-                        1,
-                        s.s({
-                            type : 's3',
-                            location : saveLocation
-                        }),
-                        k.filesize,
-                        k.endTime,
-                        data.Location
-                    ]
-                    s.sqlQuery('INSERT INTO `Cloud Videos` (mid,ke,time,status,details,size,end,href) VALUES (?,?,?,?,?,?,?,?)',save)
-                    s.setCloudDiskUsedForGroup(e,{
-                        amount : k.filesizeMB,
-                        storageType : 's3'
+                    s.knexQuery({
+                        action: "insert",
+                        table: "Cloud Videos",
+                        insert: {
+                            mid: e.mid,
+                            ke: e.ke,
+                            time: k.startTime,
+                            status: 1,
+                            details: s.s({
+                                type : 's3',
+                                location : saveLocation
+                            }),
+                            size: k.filesize,
+                            end: k.endTime,
+                            href: data.Location
+                        }
+                    })
+                    s.setCloudDiskUsedForGroup(e.ke,{
+                        amount: k.filesizeMB,
+                        storageType: 's3'
                     })
                     s.purgeCloudDiskForGroup(e,'s3')
                 }
@@ -142,19 +145,22 @@ module.exports = function(s,config,lang){
                     s.userLog(e,{type:lang['Wasabi Hot Cloud Storage Upload Error'],msg:err})
                 }
                 if(s.group[e.ke].init.aws_s3_log === '1' && data && data.Location){
-                    var save = [
-                        queryInfo.mid,
-                        queryInfo.ke,
-                        queryInfo.time,
-                        s.s({
-                            type : 's3',
-                            location : saveLocation,
-                        }),
-                        queryInfo.size,
-                        data.Location
-                    ]
-                    s.sqlQuery('INSERT INTO `Cloud Timelapse Frames` (mid,ke,time,details,size,href) VALUES (?,?,?,?,?,?)',save)
-                    s.setCloudDiskUsedForGroup(e,{
+                    s.knexQuery({
+                        action: "insert",
+                        table: "Cloud Timelapse Frames",
+                        insert: {
+                            mid: queryInfo.mid,
+                            ke: queryInfo.ke,
+                            time: queryInfo.time,
+                            details: s.s({
+                                type : 's3',
+                                location : saveLocation
+                            }),
+                            size: queryInfo.size,
+                            href: data.Location
+                        }
+                    })
+                    s.setCloudDiskUsedForGroup(e.ke,{
                         amount : s.kilobyteToMegabyte(queryInfo.size),
                         storageType : 's3'
                     },'timelapseFrames')
@@ -405,4 +411,4 @@ module.exports = function(s,config,lang){
          },
        ]
     }
-}
+}

libs/uploaders/backblazeB2.js

@@ -3,8 +3,8 @@ module.exports = function(s,config,lang){
     //Backblaze B2
     var beforeAccountSaveForBackblazeB2 = function(d){
         //d = save event
-        d.form.details.b2_use_global=d.d.b2_use_global
-        d.form.details.use_bb_b2=d.d.use_bb_b2
+        d.formDetails.b2_use_global=d.d.b2_use_global
+        d.formDetails.use_bb_b2=d.d.use_bb_b2
     }
     var cloudDiskUseStartupForBackblazeB2 = function(group,userDetails){
         group.cloudDiskUse['b2'].name = 'Backblaze B2'
@@ -129,23 +129,26 @@ module.exports = function(s,config,lang){
                     }).then(function(resp){
                         if(s.group[e.ke].init.bb_b2_log === '1' && resp.data.fileId){
                             var backblazeDownloadUrl = s.group[e.ke].bb_b2_downloadUrl + '/file/' + s.group[e.ke].init.bb_b2_bucket + '/' + backblazeSavePath
-                            var save = [
-                                e.mid,
-                                e.ke,
-                                k.startTime,
-                                1,
-                                s.s({
-                                    type : 'b2',
-                                    bucketId : resp.data.bucketId,
-                                    fileId : resp.data.fileId,
-                                    fileName : resp.data.fileName
-                                }),
-                                k.filesize,
-                                k.endTime,
-                                backblazeDownloadUrl
-                            ]
-                            s.sqlQuery('INSERT INTO `Cloud Videos` (mid,ke,time,status,details,size,end,href) VALUES (?,?,?,?,?,?,?,?)',save)
-                            s.setCloudDiskUsedForGroup(e,{
+                            s.knexQuery({
+                                action: "insert",
+                                table: "Cloud Videos",
+                                insert: {
+                                    mid: e.mid,
+                                    ke: e.ke,
+                                    time: k.startTime,
+                                    status: 1,
+                                    details: s.s({
+                                        type : 'b2',
+                                        bucketId : resp.data.bucketId,
+                                        fileId : resp.data.fileId,
+                                        fileName : resp.data.fileName
+                                    }),
+                                    size: k.filesize,
+                                    end: k.endTime,
+                                    href: backblazeDownloadUrl
+                                }
+                            })
+                            s.setCloudDiskUsedForGroup(e.ke,{
                                 amount : k.filesizeMB,
                                 storageType : 'b2'
                             })

libs/uploaders/googleDrive.js

@@ -54,8 +54,8 @@ module.exports = (s,config,lang,app,io) => {
     //Google Drive Storage
     var beforeAccountSaveForGoogleDrive = function(d){
         //d = save event
-        d.form.details.googd_use_global = d.d.googd_use_global
-        d.form.details.use_googd = d.d.use_googd
+        d.formDetails.googd_use_global = d.d.googd_use_global
+        d.formDetails.use_googd = d.d.use_googd
     }
     var cloudDiskUseStartupForGoogleDrive = function(group,userDetails){
         group.cloudDiskUse['googd'].name = 'Google Drive Storage'
@@ -157,21 +157,24 @@ module.exports = (s,config,lang,app,io) => {
                   const data = response.data
 
                     if(s.group[e.ke].init.googd_log === '1' && data && data.id){
-                        var save = [
-                            e.mid,
-                            e.ke,
-                            k.startTime,
-                            1,
-                            s.s({
-                                type : 'googd',
-                                id : data.id
-                            }),
-                            k.filesize,
-                            k.endTime,
-                            ''
-                        ]
-                        s.sqlQuery('INSERT INTO `Cloud Videos` (mid,ke,time,status,details,size,end,href) VALUES (?,?,?,?,?,?,?,?)',save)
-                        s.setCloudDiskUsedForGroup(e,{
+                        s.knexQuery({
+                            action: "insert",
+                            table: "Cloud Videos",
+                            insert: {
+                                mid: e.mid,
+                                ke: e.ke,
+                                time: k.startTime,
+                                status: 1,
+                                details: s.s({
+                                    type: 'googd',
+                                    id: data.id
+                                }),
+                                size: k.filesize,
+                                end: k.endTime,
+                                href: ''
+                            }
+                        })
+                        s.setCloudDiskUsedForGroup(e.ke,{
                             amount : k.filesizeMB,
                             storageType : 'googd'
                         })
@@ -208,19 +211,22 @@ module.exports = (s,config,lang,app,io) => {
                     s.userLog(e,{type:lang['Google Drive Storage Upload Error'],msg:err})
                 }
                 if(s.group[e.ke].init.googd_log === '1' && data && data.id){
-                    var save = [
-                        queryInfo.mid,
-                        queryInfo.ke,
-                        queryInfo.time,
-                        s.s({
-                            type : 'googd',
-                            id : data.id,
-                        }),
-                        queryInfo.size,
-                        ''
-                    ]
-                    s.sqlQuery('INSERT INTO `Cloud Timelapse Frames` (mid,ke,time,details,size,href) VALUES (?,?,?,?,?,?)',save)
-                    s.setCloudDiskUsedForGroup(e,{
+                    s.knexQuery({
+                        action: "insert",
+                        table: "Cloud Timelapse Frames",
+                        insert: {
+                            mid: queryInfo.mid,
+                            ke: queryInfo.ke,
+                            time: queryInfo.time,
+                            details: s.s({
+                                type : 'googd',
+                                id : data.id,
+                            }),
+                            size: queryInfo.size,
+                            href: ''
+                        }
+                    })
+                    s.setCloudDiskUsedForGroup(e.ke,{
                         amount : s.kilobyteToMegabyte(queryInfo.size),
                         storageType : 'googd'
                     },'timelapseFrames')

libs/uploaders/s3based.js

@@ -3,8 +3,8 @@ module.exports = function(s,config,lang){
     //Wasabi Hot Cloud Storage
     var beforeAccountSaveForWasabiHotCloudStorage = function(d){
         //d = save event
-        d.form.details.whcs_use_global=d.d.whcs_use_global
-        d.form.details.use_whcs=d.d.use_whcs
+        d.formDetails.whcs_use_global=d.d.whcs_use_global
+        d.formDetails.use_whcs=d.d.use_whcs
     }
     var cloudDiskUseStartupForWasabiHotCloudStorage = function(group,userDetails){
         group.cloudDiskUse['whcs'].name = 'Wasabi Hot Cloud Storage'
@@ -117,21 +117,24 @@ module.exports = function(s,config,lang){
                 if(s.group[e.ke].init.whcs_log === '1' && data && data.Location){
                     var cloudLink = data.Location
                     cloudLink = fixCloudianUrl(e,cloudLink)
-                    var save = [
-                        e.mid,
-                        e.ke,
-                        k.startTime,
-                        1,
-                        s.s({
-                            type : 'whcs',
-                            location : saveLocation
-                        }),
-                        k.filesize,
-                        k.endTime,
-                        cloudLink
-                    ]
-                    s.sqlQuery('INSERT INTO `Cloud Videos` (mid,ke,time,status,details,size,end,href) VALUES (?,?,?,?,?,?,?,?)',save)
-                    s.setCloudDiskUsedForGroup(e,{
+                    s.knexQuery({
+                        action: "insert",
+                        table: "Cloud Videos",
+                        insert: {
+                            mid: e.mid,
+                            ke: e.ke,
+                            time: k.startTime,
+                            status: 1,
+                            details: s.s({
+                                type : 'whcs',
+                                location : saveLocation
+                            }),
+                            size: k.filesize,
+                            end: k.endTime,
+                            href: cloudLink
+                        }
+                    })
+                    s.setCloudDiskUsedForGroup(e.ke,{
                         amount : k.filesizeMB,
                         storageType : 'whcs'
                     })
@@ -159,19 +162,22 @@ module.exports = function(s,config,lang){
                     s.userLog(e,{type:lang['Wasabi Hot Cloud Storage Upload Error'],msg:err})
                 }
                 if(s.group[e.ke].init.whcs_log === '1' && data && data.Location){
-                    var save = [
-                        queryInfo.mid,
-                        queryInfo.ke,
-                        queryInfo.time,
-                        s.s({
-                            type : 'whcs',
-                            location : saveLocation,
-                        }),
-                        queryInfo.size,
-                        data.Location
-                    ]
-                    s.sqlQuery('INSERT INTO `Cloud Timelapse Frames` (mid,ke,time,details,size,href) VALUES (?,?,?,?,?,?)',save)
-                    s.setCloudDiskUsedForGroup(e,{
+                    s.knexQuery({
+                        action: "insert",
+                        table: "Cloud Timelapse Frames",
+                        insert: {
+                            mid: queryInfo.mid,
+                            ke: queryInfo.ke,
+                            time: queryInfo.time,
+                            details: s.s({
+                                type : 'whcs',
+                                location : saveLocation
+                            }),
+                            size: queryInfo.size,
+                            href: data.Location
+                        }
+                    })
+                    s.setCloudDiskUsedForGroup(e.ke,{
                         amount : s.kilobyteToMegabyte(queryInfo.size),
                         storageType : 'whcs'
                     },'timelapseFrames')

libs/uploaders/sftp.js

@@ -7,7 +7,7 @@ module.exports = function(s,config,lang){
     }
     var beforeAccountSaveForSftp = function(d){
         //d = save event
-        d.form.details.use_sftp = d.d.use_sftp
+        d.formDetails.use_sftp = d.d.use_sftp
     }
     var loadSftpForUser = function(e){
         // e = user

libs/uploaders/webdav.js

@@ -4,8 +4,8 @@ module.exports = function(s,config,lang){
     // WebDAV
     var beforeAccountSaveForWebDav = function(d){
         //d = save event
-        d.form.details.webdav_use_global=d.d.webdav_use_global
-        d.form.details.use_webdav=d.d.use_webdav
+        d.formDetails.webdav_use_global=d.d.webdav_use_global
+        d.formDetails.use_webdav=d.d.use_webdav
     }
     var cloudDiskUseStartupForWebDav = function(group,userDetails){
         group.cloudDiskUse['webdav'].name = 'WebDAV'
@@ -81,23 +81,26 @@ module.exports = function(s,config,lang){
                fs.createReadStream(k.dir + k.filename).pipe(wfs.createWriteStream(webdavUploadDir + k.filename))
                if(s.group[e.ke].init.webdav_log === '1'){
                    var webdavRemoteUrl = s.addUserPassToUrl(s.checkCorrectPathEnding(s.group[e.ke].init.webdav_url),s.group[e.ke].init.webdav_user,s.group[e.ke].init.webdav_pass) + s.group[e.ke].init.webdav_dir + e.ke + '/'+e.mid+'/'+k.filename
-                   var save = [
-                       e.mid,
-                       e.ke,
-                       k.startTime,
-                       1,
-                       s.s({
-                           type : 'webdav',
-                           location : webdavUploadDir + k.filename
-                       }),
-                       k.filesize,
-                       k.endTime,
-                       webdavRemoteUrl
-                   ]
-                   s.sqlQuery('INSERT INTO `Cloud Videos` (mid,ke,time,status,details,size,end,href) VALUES (?,?,?,?,?,?,?,?)',save)
-                   s.setCloudDiskUsedForGroup(e,{
-                       amount : k.filesizeMB,
-                       storageType : 'webdav'
+                   s.knexQuery({
+                       action: "insert",
+                       table: "Cloud Videos",
+                       insert: {
+                           mid: e.mid,
+                           ke: e.ke,
+                           time: k.startTime,
+                           status: 1,
+                           details: s.s({
+                               type : 'webdav',
+                               location : webdavUploadDir + k.filename
+                           }),
+                           size: k.filesize,
+                           end: k.endTime,
+                           href: webdavRemoteUrl
+                       }
+                   })
+                   s.setCloudDiskUsedForGroup(e.ke,{
+                       amount: k.filesizeMB,
+                       storageType: 'webdav'
                    })
                    s.purgeCloudDiskForGroup(e,'webdav')
                }

libs/user.js

@@ -2,278 +2,63 @@ var fs = require('fs');
 var events = require('events');
 var spawn = require('child_process').spawn;
 var exec = require('child_process').exec;
+var async = require("async");
 module.exports = function(s,config,lang){
-    s.purgeDiskForGroup = function(e){
-        if(config.cron.deleteOverMax === true && s.group[e.ke] && s.group[e.ke].sizePurgeQueue){
-            s.group[e.ke].sizePurgeQueue.push(1)
-            if(s.group[e.ke].sizePurging !== true){
-                s.group[e.ke].sizePurging = true
-                var finish = function(){
-                    //remove value just used from queue
-                    s.group[e.ke].sizePurgeQueue.shift()
-                    //do next one
-                    if(s.group[e.ke].sizePurgeQueue.length > 0){
-                        checkQueue()
-                    }else{
-                        s.group[e.ke].sizePurging = false
-                        s.sendDiskUsedAmountToClients(e)
-                    }
-                }
-                var checkQueue = function(){
-                    //get first in queue
-                    var currentPurge = s.group[e.ke].sizePurgeQueue[0]
-                    var reRunCheck = function(){}
-                    var deleteSetOfVideos = function(err,videos,storageIndex,callback){
-                        var videosToDelete = []
-                        var queryValues = [e.ke]
-                        var completedCheck = 0
-                        if(videos){
-                            videos.forEach(function(video){
-                                video.dir = s.getVideoDirectory(video) + s.formattedTime(video.time) + '.' + video.ext
-                                videosToDelete.push('(mid=? AND `time`=?)')
-                                queryValues.push(video.mid)
-                                queryValues.push(video.time)
-                                fs.chmod(video.dir,0o777,function(err){
-                                    fs.unlink(video.dir,function(err){
-                                        ++completedCheck
-                                        if(err){
-                                            fs.stat(video.dir,function(err){
-                                                if(!err){
-                                                    s.file('delete',video.dir)
-                                                }
-                                            })
-                                        }
-                                        if(videosToDelete.length === completedCheck){
-                                            videosToDelete = videosToDelete.join(' OR ')
-                                            s.sqlQuery('DELETE FROM Videos WHERE ke =? AND ('+videosToDelete+')',queryValues,function(){
-                                                reRunCheck()
-                                            })
-                                        }
-                                    })
-                                })
-                                if(storageIndex){
-                                    s.setDiskUsedForGroupAddStorage(e,{
-                                        size: -(video.size/1048576),
-                                        storageIndex: storageIndex
-                                    })
-                                }else{
-                                    s.setDiskUsedForGroup(e,-(video.size/1048576))
-                                }
-                                s.tx({
-                                    f: 'video_delete',
-                                    ff: 'over_max',
-                                    filename: s.formattedTime(video.time)+'.'+video.ext,
-                                    mid: video.mid,
-                                    ke: video.ke,
-                                    time: video.time,
-                                    end: s.formattedTime(new Date,'YYYY-MM-DD HH:mm:ss')
-                                },'GRP_'+e.ke)
-                            })
-                        }else{
-                            console.log(err)
-                        }
-                        if(videosToDelete.length === 0){
-                            if(callback)callback()
-                        }
-                    }
-                    var deleteSetOfTimelapseFrames = function(err,frames,storageIndex,callback){
-                        var framesToDelete = []
-                        var queryValues = [e.ke]
-                        var completedCheck = 0
-                        if(frames){
-                            frames.forEach(function(frame){
-                                var selectedDate = frame.filename.split('T')[0]
-                                var dir = s.getTimelapseFrameDirectory(frame)
-                                var fileLocationMid = `${dir}` + frame.filename
-                                framesToDelete.push('(mid=? AND `time`=?)')
-                                queryValues.push(frame.mid)
-                                queryValues.push(frame.time)
-                                fs.unlink(fileLocationMid,function(err){
-                                    ++completedCheck
-                                    if(err){
-                                        fs.stat(fileLocationMid,function(err){
-                                            if(!err){
-                                                s.file('delete',fileLocationMid)
-                                            }
-                                        })
-                                    }
-                                    if(framesToDelete.length === completedCheck){
-                                        framesToDelete = framesToDelete.join(' OR ')
-                                        s.sqlQuery('DELETE FROM `Timelapse Frames` WHERE ke =? AND ('+framesToDelete+')',queryValues,function(){
-                                            reRunCheck()
-                                        })
-                                    }
-                                })
-                                if(storageIndex){
-                                    s.setDiskUsedForGroupAddStorage(e,{
-                                        size: -(frame.size/1048576),
-                                        storageIndex: storageIndex
-                                    },'timelapeFrames')
-                                }else{
-                                    s.setDiskUsedForGroup(e,-(frame.size/1048576),'timelapeFrames')
-                                }
-                                // s.tx({
-                                //     f: 'timelapse_frame_delete',
-                                //     ff: 'over_max',
-                                //     filename: s.formattedTime(video.time)+'.'+video.ext,
-                                //     mid: video.mid,
-                                //     ke: video.ke,
-                                //     time: video.time,
-                                //     end: s.formattedTime(new Date,'YYYY-MM-DD HH:mm:ss')
-                                // },'GRP_'+e.ke)
-                            })
-                        }else{
-                            console.log(err)
-                        }
-                        if(framesToDelete.length === 0){
-                            if(callback)callback()
-                        }
-                    }
-                    var deleteSetOfFileBinFiles = function(err,files,storageIndex,callback){
-                        var filesToDelete = []
-                        var queryValues = [e.ke]
-                        var completedCheck = 0
-                        if(files){
-                            files.forEach(function(file){
-                                var dir = s.getFileBinDirectory(file)
-                                var fileLocationMid = `${dir}` + file.name
-                                filesToDelete.push('(mid=? AND `name`=?)')
-                                queryValues.push(file.mid)
-                                queryValues.push(file.name)
-                                fs.unlink(fileLocationMid,function(err){
-                                    ++completedCheck
-                                    if(err){
-                                        fs.stat(fileLocationMid,function(err){
-                                            if(!err){
-                                                s.file('delete',fileLocationMid)
-                                            }
-                                        })
-                                    }
-                                    if(filesToDelete.length === completedCheck){
-                                        filesToDelete = filesToDelete.join(' OR ')
-                                        s.sqlQuery('DELETE FROM `Files` WHERE ke =? AND ('+filesToDelete+')',queryValues,function(){
-                                            reRunCheck()
-                                        })
-                                    }
-                                })
-                                if(storageIndex){
-                                    s.setDiskUsedForGroupAddStorage(e,{
-                                        size: -(file.size/1048576),
-                                        storageIndex: storageIndex
-                                    },'fileBin')
-                                }else{
-                                    s.setDiskUsedForGroup(e,-(file.size/1048576),'fileBin')
-                                }
-                            })
-                        }else{
-                            console.log(err)
-                        }
-                        if(framesToDelete.length === 0){
-                            if(callback)callback()
-                        }
-                    }
-                    var deleteMainVideos = function(callback){
-                        reRunCheck = function(){
-                            return deleteMainVideos(callback)
-                        }
-                        //run purge command
-                        if(s.group[e.ke].usedSpaceVideos > (s.group[e.ke].sizeLimit * (s.group[e.ke].sizeLimitVideoPercent / 100) * config.cron.deleteOverMaxOffset)){
-                                s.sqlQuery('SELECT * FROM Videos WHERE status != 0 AND details NOT LIKE \'%"archived":"1"%\' AND ke=? AND details NOT LIKE \'%"dir"%\' ORDER BY `time` ASC LIMIT 3',[e.ke],function(err,rows){
-                                    deleteSetOfVideos(err,rows,null,callback)
-                                })
-                        }else{
-                            callback()
-                        }
-                    }
-                    var deleteAddStorageVideos = function(callback){
-                        reRunCheck = function(){
-                            return deleteAddStorageVideos(callback)
-                        }
-                        var currentStorageNumber = 0
-                        var readStorageArray = function(finishedReading){
-                            setTimeout(function(){
-                                reRunCheck = readStorageArray
-                                var storage = s.listOfStorage[currentStorageNumber]
-                                if(!storage){
-                                    //done all checks, move on to next user
-                                    callback()
-                                    return
-                                }
-                                var storageId = storage.value
-                                if(storageId === '' || !s.group[e.ke].addStorageUse[storageId]){
-                                    ++currentStorageNumber
-                                    readStorageArray()
-                                    return
-                                }
-                                var storageIndex = s.group[e.ke].addStorageUse[storageId]
-                                //run purge command
-                                if(storageIndex.usedSpace > (storageIndex.sizeLimit * (storageIndex.deleteOffset || config.cron.deleteOverMaxOffset))){
-                                    s.sqlQuery('SELECT * FROM Videos WHERE status != 0 AND details NOT LIKE \'%"archived":"1"%\' AND ke=? AND details LIKE ? ORDER BY `time` ASC LIMIT 3',[e.ke,`%"dir":"${storage.value}"%`],function(err,rows){
-                                        deleteSetOfVideos(err,rows,storageIndex,callback)
-                                    })
-                                }else{
-                                    ++currentStorageNumber
-                                    readStorageArray()
-                                }
-                            })
-                        }
-                        readStorageArray()
-                    }
-                    var deleteTimelapseFrames = function(callback){
-                        reRunCheck = function(){
-                            return deleteTimelapseFrames(callback)
-                        }
-                        //run purge command
-                        if(s.group[e.ke].usedSpaceTimelapseFrames > (s.group[e.ke].sizeLimit * (s.group[e.ke].sizeLimitTimelapseFramesPercent / 100) * config.cron.deleteOverMaxOffset)){
-                                s.sqlQuery('SELECT * FROM `Timelapse Frames` WHERE ke=? AND details NOT LIKE \'%"archived":"1"%\' ORDER BY `time` ASC LIMIT 3',[e.ke],function(err,frames){
-                                    deleteSetOfTimelapseFrames(err,frames,null,callback)
-                                })
-                        }else{
-                            callback()
-                        }
-                    }
-                    var deleteFileBinFiles = function(callback){
-                        if(config.deleteFileBinsOverMax === true){
-                            reRunCheck = function(){
-                                return deleteSetOfFileBinFiles(callback)
-                            }
-                            //run purge command
-                            if(s.group[e.ke].usedSpaceFileBin > (s.group[e.ke].sizeLimit * (s.group[e.ke].sizeLimitFileBinPercent / 100) * config.cron.deleteOverMaxOffset)){
-                                    s.sqlQuery('SELECT * FROM `Files` WHERE ke=? ORDER BY `time` ASC LIMIT 1',[e.ke],function(err,frames){
-                                        deleteSetOfFileBinFiles(err,frames,null,callback)
-                                    })
-                            }else{
-                                callback()
-                            }
-                        }else{
-                            callback()
-                        }
-                    }
-                    deleteMainVideos(function(){
-                        deleteTimelapseFrames(function(){
-                            deleteFileBinFiles(function(){
-                                deleteAddStorageVideos(function(){
-                                    finish()
+    const {
+        deleteSetOfVideos,
+        deleteSetOfTimelapseFrames,
+        deleteSetOfFileBinFiles,
+        deleteAddStorageVideos,
+        deleteMainVideos,
+        deleteTimelapseFrames,
+        deleteFileBinFiles,
+        deleteCloudVideos,
+        deleteCloudTimelapseFrames,
+    } = require("./user/utils.js")(s,config,lang);
+    let purgeDiskGroup = () => {}
+    const runQuery = async.queue(function(groupKey, callback) {
+        purgeDiskGroup(groupKey,callback)
+    }, 1);
+    if(config.cron.deleteOverMax === true){
+        purgeDiskGroup = (groupKey,callback) => {
+            if(s.group[groupKey]){
+                if(s.group[groupKey].sizePurging !== true){
+                    s.group[groupKey].sizePurging = true
+                    s.debugLog(`${groupKey} deleteMainVideos`)
+                    deleteMainVideos(groupKey,() => {
+                        s.debugLog(`${groupKey} deleteTimelapseFrames`)
+                        deleteTimelapseFrames(groupKey,() => {
+                            s.debugLog(`${groupKey} deleteFileBinFiles`)
+                            deleteFileBinFiles(groupKey,() => {
+                                s.debugLog(`${groupKey} deleteAddStorageVideos`)
+                                deleteAddStorageVideos(groupKey,() => {
+                                    s.group[groupKey].sizePurging = false
+                                    s.sendDiskUsedAmountToClients(groupKey)
+                                    callback();
                                 })
                             })
                         })
                     })
+                }else{
+                    s.sendDiskUsedAmountToClients(groupKey)
                 }
-                checkQueue()
             }
-        }else{
-            s.sendDiskUsedAmountToClients(e)
         }
     }
-    s.setDiskUsedForGroup = function(e,bytes,storagePoint){
+    s.purgeDiskForGroup = (groupKey) => {
+        return runQuery.push(groupKey,function(){
+            //...
+        })
+    }
+    s.setDiskUsedForGroup = function(groupKey,bytes,storagePoint){
         //`bytes` will be used as the value to add or substract
-        if(s.group[e.ke] && s.group[e.ke].diskUsedEmitter){
-            s.group[e.ke].diskUsedEmitter.emit('set',bytes,storagePoint)
+        if(s.group[groupKey] && s.group[groupKey].diskUsedEmitter){
+            s.group[groupKey].diskUsedEmitter.emit('set',bytes,storagePoint)
         }
     }
-    s.setDiskUsedForGroupAddStorage = function(e,data,storagePoint){
-        if(s.group[e.ke] && s.group[e.ke].diskUsedEmitter){
-            s.group[e.ke].diskUsedEmitter.emit('setAddStorage',data,storagePoint)
+    s.setDiskUsedForGroupAddStorage = function(groupKey,data,storagePoint){
+        if(s.group[groupKey] && s.group[groupKey].diskUsedEmitter){
+            s.group[groupKey].diskUsedEmitter.emit('setAddStorage',data,storagePoint)
         }
     }
     s.purgeCloudDiskForGroup = function(e,storageType,storagePoint){
@@ -281,33 +66,44 @@ module.exports = function(s,config,lang){
             s.group[e.ke].diskUsedEmitter.emit('purgeCloud',storageType,storagePoint)
         }
     }
-    s.setCloudDiskUsedForGroup = function(e,usage,storagePoint){
+    s.setCloudDiskUsedForGroup = function(groupKey,usage,storagePoint){
         //`usage` will be used as the value to add or substract
-        if(s.group[e.ke].diskUsedEmitter){
-            s.group[e.ke].diskUsedEmitter.emit('setCloud',usage,storagePoint)
+        if(s.group[groupKey].diskUsedEmitter){
+            s.group[groupKey].diskUsedEmitter.emit('setCloud',usage,storagePoint)
         }
     }
-    s.sendDiskUsedAmountToClients = function(e){
+    s.sendDiskUsedAmountToClients = function(groupKey){
         //send the amount used disk space to connected users
-        if(s.group[e.ke]&&s.group[e.ke].init){
+        if(s.group[groupKey]&&s.group[groupKey].init){
             s.tx({
                 f: 'diskUsed',
-                size: s.group[e.ke].usedSpace,
-                usedSpace: s.group[e.ke].usedSpace,
-                usedSpaceVideos: s.group[e.ke].usedSpaceVideos,
-                usedSpaceFilebin: s.group[e.ke].usedSpaceFilebin,
-                usedSpaceTimelapseFrames: s.group[e.ke].usedSpaceTimelapseFrames,
-                limit: s.group[e.ke].sizeLimit,
-                addStorage: s.group[e.ke].addStorageUse
-            },'GRP_'+e.ke);
+                size: s.group[groupKey].usedSpace,
+                usedSpace: s.group[groupKey].usedSpace,
+                usedSpaceVideos: s.group[groupKey].usedSpaceVideos,
+                usedSpaceFilebin: s.group[groupKey].usedSpaceFilebin,
+                usedSpaceTimelapseFrames: s.group[groupKey].usedSpaceTimelapseFrames,
+                limit: s.group[groupKey].sizeLimit,
+                addStorage: s.group[groupKey].addStorageUse
+            },'GRP_'+groupKey);
         }
     }
     //user log
     s.userLog = function(e,x){
         if(e.id && !e.mid)e.mid = e.id
         if(!x||!e.mid){return}
-        if((e.details&&e.details.sqllog==='1')||e.mid.indexOf('$')>-1){
-            s.sqlQuery('INSERT INTO Logs (ke,mid,info) VALUES (?,?,?)',[e.ke,e.mid,s.s(x)]);
+        if(
+            (e.details && e.details.sqllog === '1') ||
+            e.mid.indexOf('$') > -1
+        ){
+            s.knexQuery({
+                action: "insert",
+                table: "Logs",
+                insert: {
+                    ke: e.ke,
+                    mid: e.mid,
+                    info: s.s(x),
+                }
+            })
         }
         s.tx({f:'log',ke:e.ke,mid:e.mid,log:x,time:s.timeObject()},'GRPLOG_'+e.ke);
     }
@@ -336,20 +132,29 @@ module.exports = function(s,config,lang){
         //save global used space as megabyte value
         s.group[e.ke].usedSpace = s.group[e.ke].usedSpace || ((e.size || 0) / 1048576)
         //emit the changes to connected users
-        s.sendDiskUsedAmountToClients(e)
+        s.sendDiskUsedAmountToClients(e.ke)
     }
     s.loadGroupApps = function(e){
         // e = user
         if(!s.group[e.ke].init){
             s.group[e.ke].init={};
         }
-        s.sqlQuery('SELECT * FROM Users WHERE ke=? AND details NOT LIKE ?',[e.ke,'%"sub"%'],function(ar,r){
+        s.knexQuery({
+            action: "select",
+            columns: "*",
+            table: "Users",
+            where: [
+                ['ke','=',e.ke],
+                ['details','NOT LIKE',`%"sub"%`],
+            ],
+            limit: 1
+        },(err,r) => {
             if(r && r[0]){
                 r = r[0];
-                ar = JSON.parse(r.details);
+                const details = JSON.parse(r.details);
                 //load extenders
                 s.loadGroupAppExtensions.forEach(function(extender){
-                    extender(r,ar)
+                    extender(r,details)
                 })
                 //disk Used Emitter
                 if(!s.group[e.ke].diskUsedEmitter){
@@ -381,82 +186,15 @@ module.exports = function(s,config,lang){
                             break;
                         }
                     })
-                    s.group[e.ke].diskUsedEmitter.on('purgeCloud',function(storageType,storagePoint){
-                        if(config.cron.deleteOverMax === true){
-                            var cloudDisk = s.group[e.ke].cloudDiskUse[storageType]
-                            //set queue processor
-                            var finish=function(){
-                                // s.sendDiskUsedAmountToClients(e)
-                            }
-                            var deleteVideos = function(){
-                                //run purge command
-                                if(cloudDisk.sizeLimitCheck && cloudDisk.usedSpace > (cloudDisk.sizeLimit*config.cron.deleteOverMaxOffset)){
-                                        s.sqlQuery('SELECT * FROM `Cloud Videos` WHERE status != 0 AND ke=? AND details LIKE \'%"type":"'+storageType+'"%\' ORDER BY `time` ASC LIMIT 2',[e.ke],function(err,videos){
-                                            var videosToDelete = []
-                                            var queryValues = [e.ke]
-                                            if(!videos)return console.log(err)
-                                            videos.forEach(function(video){
-                                                video.dir = s.getVideoDirectory(video) + s.formattedTime(video.time) + '.' + video.ext
-                                                videosToDelete.push('(mid=? AND `time`=?)')
-                                                queryValues.push(video.mid)
-                                                queryValues.push(video.time)
-                                                s.setCloudDiskUsedForGroup(e,{
-                                                    amount : -(video.size/1048576),
-                                                    storageType : storageType
-                                                })
-                                                s.deleteVideoFromCloudExtensionsRunner(e,storageType,video)
-                                            })
-                                            if(videosToDelete.length > 0){
-                                                videosToDelete = videosToDelete.join(' OR ')
-                                                s.sqlQuery('DELETE FROM `Cloud Videos` WHERE ke =? AND ('+videosToDelete+')',queryValues,function(){
-                                                    deleteVideos()
-                                                })
-                                            }else{
-                                                finish()
-                                            }
-                                        })
-                                }else{
-                                    finish()
-                                }
-                            }
-                            var deleteTimelapseFrames = function(callback){
-                                reRunCheck = function(){
-                                    return deleteTimelapseFrames(callback)
-                                }
-                                //run purge command
-                                if(cloudDisk.usedSpaceTimelapseFrames > (cloudDisk.sizeLimit * (s.group[e.ke].sizeLimitTimelapseFramesPercent / 100) * config.cron.deleteOverMaxOffset)){
-                                    s.sqlQuery('SELECT * FROM `Cloud Timelapse Frames` WHERE ke=? AND details NOT LIKE \'%"archived":"1"%\' ORDER BY `time` ASC LIMIT 3',[e.ke],function(err,frames){
-                                        var framesToDelete = []
-                                        var queryValues = [e.ke]
-                                        if(!frames)return console.log(err)
-                                        frames.forEach(function(frame){
-                                            frame.dir = s.getVideoDirectory(frame) + s.formattedTime(frame.time) + '.' + frame.ext
-                                            framesToDelete.push('(mid=? AND `time`=?)')
-                                            queryValues.push(frame.mid)
-                                            queryValues.push(frame.time)
-                                            s.setCloudDiskUsedForGroup(e,{
-                                                amount : -(frame.size/1048576),
-                                                storageType : storageType
-                                            })
-                                            s.deleteVideoFromCloudExtensionsRunner(e,storageType,frame)
-                                        })
-                                        s.sqlQuery('DELETE FROM `Cloud Timelapse Frames` WHERE ke =? AND ('+framesToDelete+')',queryValues,function(){
-                                            deleteTimelapseFrames(callback)
-                                        })
-                                    })
-                                }else{
-                                    callback()
-                                }
-                            }
-                            deleteVideos(function(){
-                                deleteTimelapseFrames(function(){
+                    if(config.cron.deleteOverMax === true){
+                        s.group[e.ke].diskUsedEmitter.on('purgeCloud',function(storageType,storagePoint){
+                            deleteCloudVideos(storageType,storagePoint,function(){
+                                deleteCloudTimelapseFrames(storageType,storagePoint,function(){
 
                                 })
                             })
-                        }else{
-                            // s.sendDiskUsedAmountToClients(e)
-                        }
-                    })
+                        })
+                    }
                     //s.setDiskUsedForGroup
                     s.group[e.ke].diskUsedEmitter.on('set',function(currentChange,storageType){
                         //validate current values
@@ -482,7 +220,7 @@ module.exports = function(s,config,lang){
                             break;
                         }
                         //remove value just used from queue
-                        s.sendDiskUsedAmountToClients(e)
+                        s.sendDiskUsedAmountToClients(e.ke)
                     })
                     s.group[e.ke].diskUsedEmitter.on('setAddStorage',function(data,storageType){
                         var currentSize = data.size
@@ -510,63 +248,76 @@ module.exports = function(s,config,lang){
                             break;
                         }
                         //remove value just used from queue
-                        s.sendDiskUsedAmountToClients(e)
+                        s.sendDiskUsedAmountToClients(e.ke)
                     })
                 }
-                Object.keys(ar).forEach(function(v){
-                    s.group[e.ke].init[v] = ar[v]
+                Object.keys(details).forEach(function(v){
+                    s.group[e.ke].init[v] = details[v]
                 })
             }
         })
     }
     s.accountSettingsEdit = function(d,dontRunExtensions){
-        s.sqlQuery('SELECT details FROM Users WHERE ke=? AND uid=?',[d.ke,d.uid],function(err,r){
-            if(r&&r[0]){
-                r=r[0];
-                d.d=JSON.parse(r.details);
-                if(!d.d.sub || d.d.user_change !== "0"){
+        s.knexQuery({
+            action: "select",
+            columns: "details",
+            table: "Users",
+            where: [
+                ['ke','=',d.ke],
+                ['uid','=',d.uid],
+            ]
+        },(err,r) => {
+            if(r && r[0]){
+                r = r[0];
+                const details = JSON.parse(r.details);
+                if(!details.sub || details.user_change !== "0"){
                     if(d.cnid){
-                        if(d.d.get_server_log === '1'){
+                        if(details.get_server_log === '1'){
                             s.clientSocketConnection[d.cnid].join('GRPLOG_'+d.ke)
                         }else{
                             s.clientSocketConnection[d.cnid].leave('GRPLOG_'+d.ke)
                         }
                     }
                     ///unchangeable from client side, so reset them in case they did.
-                    d.form.details=JSON.parse(d.form.details)
+                    var form = d.form
+                    var formDetails = JSON.parse(form.details)
                     if(!dontRunExtensions){
                         s.beforeAccountSaveExtensions.forEach(function(extender){
-                            extender(d)
+                            extender({
+                                form: form,
+                                formDetails: formDetails,
+                                d: details
+                            })
                         })
                     }
                     //admin permissions
-                    d.form.details.permissions=d.d.permissions
-                    d.form.details.edit_size=d.d.edit_size
-                    d.form.details.edit_days=d.d.edit_days
-                    d.form.details.use_admin=d.d.use_admin
-                    d.form.details.use_ldap=d.d.use_ldap
-                    d.form.details.landing_page=d.d.landing_page
+                    formDetails.permissions = details.permissions
+                    formDetails.edit_size = details.edit_size
+                    formDetails.edit_days = details.edit_days
+                    formDetails.use_admin = details.use_admin
+                    formDetails.use_ldap = details.use_ldap
+                    formDetails.landing_page = details.landing_page
                     //check
-                    if(d.d.edit_days == "0"){
-                        d.form.details.days = d.d.days;
+                    if(details.edit_days == "0"){
+                        formDetails.days = details.days;
                     }
-                    if(d.d.edit_size == "0"){
-                        d.form.details.size = d.d.size;
-                        d.form.details.addStorage = d.d.addStorage;
+                    if(details.edit_size == "0"){
+                        formDetails.size = details.size;
+                        formDetails.addStorage = details.addStorage;
                     }
-                    if(d.d.sub){
-                        d.form.details.sub=d.d.sub;
-                        if(d.d.monitors){d.form.details.monitors=d.d.monitors;}
-                        if(d.d.allmonitors){d.form.details.allmonitors=d.d.allmonitors;}
-                        if(d.d.monitor_create){d.form.details.monitor_create=d.d.monitor_create;}
-                        if(d.d.video_delete){d.form.details.video_delete=d.d.video_delete;}
-                        if(d.d.video_view){d.form.details.video_view=d.d.video_view;}
-                        if(d.d.monitor_edit){d.form.details.monitor_edit=d.d.monitor_edit;}
-                        if(d.d.size){d.form.details.size=d.d.size;}
-                        if(d.d.days){d.form.details.days=d.d.days;}
-                        delete(d.form.details.mon_groups)
+                    if(details.sub){
+                        formDetails.sub = details.sub;
+                        if(details.monitors){formDetails.monitors = details.monitors;}
+                        if(details.allmonitors){formDetails.allmonitors = details.allmonitors;}
+                        if(details.monitor_create){formDetails.monitor_create = details.monitor_create;}
+                        if(details.video_delete){formDetails.video_delete = details.video_delete;}
+                        if(details.video_view){formDetails.video_view = details.video_view;}
+                        if(details.monitor_edit){formDetails.monitor_edit = details.monitor_edit;}
+                        if(details.size){formDetails.size = details.size;}
+                        if(details.days){formDetails.days = details.days;}
+                        delete(formDetails.mon_groups)
                     }
-                    var newSize = parseFloat(d.form.details.size) || 10000
+                    var newSize = parseFloat(formDetails.size) || 10000
                     //load addStorageUse
                     var currentStorageNumber = 0
                     var readStorageArray = function(){
@@ -581,7 +332,7 @@ module.exports = function(s,config,lang){
                             readStorageArray()
                             return
                         }
-                        var detailContainer = d.form.details || s.group[r.ke].init
+                        var detailContainer = formDetails || s.group[r.ke].init
                         var storageId = path
                         var detailsContainerAddStorage = s.parseJSON(detailContainer.addStorage)
                         if(!s.group[d.ke].addStorageUse[storageId])s.group[d.ke].addStorageUse[storageId] = {}
@@ -597,20 +348,31 @@ module.exports = function(s,config,lang){
                     }
                     readStorageArray()
                     ///
-                    d.form.details = JSON.stringify(s.mergeDeep(d.d,d.form.details))
+                    formDetails = JSON.stringify(s.mergeDeep(details,formDetails))
                     ///
-                    d.set=[],d.ar=[];
-                    if(d.form.pass&&d.form.pass!==''){d.form.pass=s.createHash(d.form.pass);}else{delete(d.form.pass)};
-                    delete(d.form.password_again);
-                    d.for=Object.keys(d.form);
-                    d.for.forEach(function(v){
-                        d.set.push(v+'=?'),d.ar.push(d.form[v]);
-                    });
-                    d.ar.push(d.ke),d.ar.push(d.uid);
-                    s.sqlQuery('UPDATE Users SET '+d.set.join(',')+' WHERE ke=? AND uid=?',d.ar,function(err,r){
-                        if(!d.d.sub){
-                            var user = Object.assign(d.form,{ke : d.ke})
-                            var userDetails = JSON.parse(d.form.details)
+                    const updateQuery = {}
+                    if(form.pass && form.pass !== ''){
+                        form.pass = s.createHash(form.pass)
+                    }else{
+                        delete(form.pass)
+                    }
+                    delete(form.password_again)
+                    Object.keys(form).forEach(function(key){
+                        const value = form[key]
+                        updateQuery[key] = value
+                    })
+                    s.knexQuery({
+                        action: "update",
+                        table: "Users",
+                        update: updateQuery,
+                        where: [
+                            ['ke','=',d.ke],
+                            ['uid','=',d.uid],
+                        ]
+                    },() => {
+                        if(!details.sub){
+                            var user = Object.assign(form,{ke : d.ke})
+                            var userDetails = JSON.parse(formDetails)
                             s.group[d.ke].sizeLimit = parseFloat(newSize)
                             if(!dontRunExtensions){
                                 s.onAccountSaveExtensions.forEach(function(extender){
@@ -622,7 +384,7 @@ module.exports = function(s,config,lang){
                                 s.loadGroupApps(d)
                             }
                         }
-                        if(d.cnid)s.tx({f:'user_settings_change',uid:d.uid,ke:d.ke,form:d.form},d.cnid)
+                        if(d.cnid)s.tx({f:'user_settings_change',uid:d.uid,ke:d.ke,form:form},d.cnid)
                     })
                 }
             }
@@ -630,7 +392,17 @@ module.exports = function(s,config,lang){
     }
     s.findPreset = function(presetQueryVals,callback){
         //presetQueryVals = [ke, type, name]
-        s.sqlQuery("SELECT * FROM Presets WHERE ke=? AND type=? AND name=? LIMIT 1",presetQueryVals,function(err,presets){
+        s.knexQuery({
+            action: "select",
+            columns: "*",
+            table: "Presets",
+            where: [
+                ['ke','=',presetQueryVals[0]],
+                ['type','=',presetQueryVals[1]],
+                ['name','=',presetQueryVals[2]],
+            ],
+            limit: 1
+        },function(err,presets) {
             var preset
             var notFound = false
             if(presets && presets[0]){

libs/user/utils.js

@@ -0,0 +1,463 @@
+var fs = require('fs');
+module.exports = (s,config,lang) => {
+    const deleteSetOfVideos = function(options,callback){
+        const groupKey = options.groupKey
+        const err = options.err
+        const videos = options.videos
+        const storageIndex = options.storageIndex
+        const reRunCheck = options.reRunCheck
+        var completedCheck = 0
+        var whereGroup = []
+        var whereQuery = [
+            ['ke','=',groupKey],
+        ]
+        if(videos){
+            videos.forEach(function(video){
+                video.dir = s.getVideoDirectory(video) + s.formattedTime(video.time) + '.' + video.ext
+                const queryGroup = {
+                    mid: video.mid,
+                    time: video.time,
+                }
+                if(whereGroup.length > 0)queryGroup.__separator = 'or'
+                whereGroup.push(queryGroup)
+                fs.chmod(video.dir,0o777,function(err){
+                    fs.unlink(video.dir,function(err){
+                        ++completedCheck
+                        if(err){
+                            fs.stat(video.dir,function(err){
+                                if(!err){
+                                    s.file('delete',video.dir)
+                                }
+                            })
+                        }
+                        const whereGroupLength = whereGroup.length
+                        if(whereGroupLength > 0 && whereGroupLength === completedCheck){
+                            whereQuery[1] = whereGroup
+                            s.knexQuery({
+                                action: "delete",
+                                table: "Videos",
+                                where: whereQuery
+                            },(err,info) => {
+                                setTimeout(reRunCheck,1000)
+                            })
+                        }
+                    })
+                })
+                if(storageIndex){
+                    s.setDiskUsedForGroupAddStorage(groupKey,{
+                        size: -(video.size/1048576),
+                        storageIndex: storageIndex
+                    })
+                }else{
+                    s.setDiskUsedForGroup(groupKey,-(video.size/1048576))
+                }
+                s.tx({
+                    f: 'video_delete',
+                    ff: 'over_max',
+                    filename: s.formattedTime(video.time)+'.'+video.ext,
+                    mid: video.mid,
+                    ke: video.ke,
+                    time: video.time,
+                    end: s.formattedTime(new Date,'YYYY-MM-DD HH:mm:ss')
+                },'GRP_'+groupKey)
+            })
+        }else{
+            console.log(err)
+        }
+        if(whereGroup.length === 0){
+            if(callback)callback()
+        }
+    }
+    const deleteSetOfTimelapseFrames = function(options,callback){
+        const groupKey = options.groupKey
+        const err = options.err
+        const frames = options.frames
+        const storageIndex = options.storageIndex
+        var whereGroup = []
+        var whereQuery = [
+            ['ke','=',groupKey],
+            []
+        ]
+        var completedCheck = 0
+        if(frames){
+            frames.forEach(function(frame){
+                var selectedDate = frame.filename.split('T')[0]
+                var dir = s.getTimelapseFrameDirectory(frame)
+                var fileLocationMid = `${dir}` + frame.filename
+                const queryGroup = {
+                    mid: video.mid,
+                    time: video.time,
+                }
+                if(whereGroup.length > 0)queryGroup.__separator = 'or'
+                whereGroup.push(queryGroup)
+                fs.unlink(fileLocationMid,function(err){
+                    ++completedCheck
+                    if(err){
+                        fs.stat(fileLocationMid,function(err){
+                            if(!err){
+                                s.file('delete',fileLocationMid)
+                            }
+                        })
+                    }
+                    const whereGroupLength = whereGroup.length
+                    if(whereGroupLength > 0 && whereGroupLength === completedCheck){
+                        whereQuery[1] = whereGroup
+                        s.knexQuery({
+                            action: "delete",
+                            table: "Timelapse Frames",
+                            where: whereQuery
+                        },() => {
+                            deleteTimelapseFrames(groupKey,callback)
+                        })
+                    }
+                })
+                if(storageIndex){
+                    s.setDiskUsedForGroupAddStorage(groupKey,{
+                        size: -(frame.size/1048576),
+                        storageIndex: storageIndex
+                    },'timelapeFrames')
+                }else{
+                    s.setDiskUsedForGroup(groupKey,-(frame.size/1048576),'timelapeFrames')
+                }
+                // s.tx({
+                //     f: 'timelapse_frame_delete',
+                //     ff: 'over_max',
+                //     filename: s.formattedTime(video.time)+'.'+video.ext,
+                //     mid: video.mid,
+                //     ke: video.ke,
+                //     time: video.time,
+                //     end: s.formattedTime(new Date,'YYYY-MM-DD HH:mm:ss')
+                // },'GRP_'+groupKey)
+            })
+        }else{
+            console.log(err)
+        }
+        if(whereGroup.length === 0){
+            if(callback)callback()
+        }
+    }
+    const deleteSetOfFileBinFiles = function(options,callback){
+        const groupKey = options.groupKey
+        const err = options.err
+        const frames = options.frames
+        const storageIndex = options.storageIndex
+        var whereGroup = []
+        var whereQuery = [
+            ['ke','=',groupKey],
+            []
+        ]
+        var completedCheck = 0
+        if(files){
+            files.forEach(function(file){
+                var dir = s.getFileBinDirectory(file)
+                var fileLocationMid = `${dir}` + file.name
+                const queryGroup = {
+                    mid: file.mid,
+                    name: file.name,
+                }
+                if(whereGroup.length > 0)queryGroup.__separator = 'or'
+                whereGroup.push(queryGroup)
+                fs.unlink(fileLocationMid,function(err){
+                    ++completedCheck
+                    if(err){
+                        fs.stat(fileLocationMid,function(err){
+                            if(!err){
+                                s.file('delete',fileLocationMid)
+                            }
+                        })
+                    }
+                    const whereGroupLength = whereGroup.length
+                    if(whereGroupLength > 0 && whereGroupLength === completedCheck){
+                        whereQuery[1] = whereGroup
+                        s.knexQuery({
+                            action: "delete",
+                            table: "Files",
+                            where: whereQuery
+                        },() => {
+                            deleteFileBinFiles(groupKey,callback)
+                        })
+                    }
+                })
+                if(storageIndex){
+                    s.setDiskUsedForGroupAddStorage(groupKey,{
+                        size: -(file.size/1048576),
+                        storageIndex: storageIndex
+                    },'fileBin')
+                }else{
+                    s.setDiskUsedForGroup(groupKey,-(file.size/1048576),'fileBin')
+                }
+            })
+        }else{
+            console.log(err)
+        }
+        if(whereGroup.length === 0){
+            if(callback)callback()
+        }
+    }
+    const deleteAddStorageVideos = function(groupKey,callback){
+        reRunCheck = function(){
+            s.debugLog('deleteAddStorageVideos')
+            return deleteAddStorageVideos(groupKey,callback)
+        }
+        var currentStorageNumber = 0
+        var readStorageArray = function(){
+            setTimeout(function(){
+                reRunCheck = readStorageArray
+                var storage = s.listOfStorage[currentStorageNumber]
+                if(!storage){
+                    //done all checks, move on to next user
+                    callback()
+                    return
+                }
+                var storageId = storage.value
+                if(storageId === '' || !s.group[groupKey].addStorageUse[storageId]){
+                    ++currentStorageNumber
+                    readStorageArray()
+                    return
+                }
+                var storageIndex = s.group[groupKey].addStorageUse[storageId]
+                //run purge command
+                if(storageIndex.usedSpace > (storageIndex.sizeLimit * (storageIndex.deleteOffset || config.cron.deleteOverMaxOffset))){
+                    s.knexQuery({
+                        action: "select",
+                        columns: "*",
+                        table: "Videos",
+                        where: [
+                            ['ke','=',groupKey],
+                            ['status','!=','0'],
+                            ['details','NOT LIKE',`%"archived":"1"%`],
+                            ['details','LIKE',`%"dir":"${storage.value}"%`],
+                        ],
+                        orderBy: ['time','asc'],
+                        limit: 3
+                    },(err,rows) => {
+                        deleteSetOfVideos({
+                            groupKey: groupKey,
+                            err: err,
+                            videos: rows,
+                            storageIndex: storageIndex,
+                            reRunCheck: () => {
+                                return readStorageArray()
+                            }
+                        },callback)
+                    })
+                }else{
+                    ++currentStorageNumber
+                    readStorageArray()
+                }
+            })
+        }
+        readStorageArray()
+    }
+    const deleteMainVideos = function(groupKey,callback){
+        // //run purge command
+        // s.debugLog('!!!!!!!!!!!deleteMainVideos')
+        // s.debugLog('s.group[groupKey].usedSpaceVideos > (s.group[groupKey].sizeLimit * (s.group[groupKey].sizeLimitVideoPercent / 100) * config.cron.deleteOverMaxOffset)')
+        // s.debugLog(s.group[groupKey].usedSpaceVideos > (s.group[groupKey].sizeLimit * (s.group[groupKey].sizeLimitVideoPercent / 100) * config.cron.deleteOverMaxOffset))
+        // s.debugLog('s.group[groupKey].usedSpaceVideos')
+        // s.debugLog(s.group[groupKey].usedSpaceVideos)
+        // s.debugLog('s.group[groupKey].sizeLimit * (s.group[groupKey].sizeLimitVideoPercent / 100) * config.cron.deleteOverMaxOffset')
+        // s.debugLog(s.group[groupKey].sizeLimit * (s.group[groupKey].sizeLimitVideoPercent / 100) * config.cron.deleteOverMaxOffset)
+        // s.debugLog('s.group[groupKey].sizeLimitVideoPercent / 100')
+        // s.debugLog(s.group[groupKey].sizeLimitVideoPercent / 100)
+        // s.debugLog('s.group[groupKey].sizeLimit')
+        // s.debugLog(s.group[groupKey].sizeLimit)
+        if(s.group[groupKey].usedSpaceVideos > (s.group[groupKey].sizeLimit * (s.group[groupKey].sizeLimitVideoPercent / 100) * config.cron.deleteOverMaxOffset)){
+            s.knexQuery({
+                action: "select",
+                columns: "*",
+                table: "Videos",
+                where: [
+                    ['ke','=',groupKey],
+                    ['status','!=','0'],
+                    ['details','NOT LIKE',`%"archived":"1"%`],
+                    ['details','NOT LIKE',`%"dir"%`],
+                ],
+                orderBy: ['time','asc'],
+                limit: 3
+            },(err,rows) => {
+                deleteSetOfVideos({
+                    groupKey: groupKey,
+                    err: err,
+                    videos: rows,
+                    storageIndex: null,
+                    reRunCheck: () => {
+                        return deleteMainVideos(groupKey,callback)
+                    }
+                },callback)
+            })
+        }else{
+            callback()
+        }
+    }
+    const deleteTimelapseFrames = function(groupKey,callback){
+        //run purge command
+        if(s.group[groupKey].usedSpaceTimelapseFrames > (s.group[groupKey].sizeLimit * (s.group[groupKey].sizeLimitTimelapseFramesPercent / 100) * config.cron.deleteOverMaxOffset)){
+            s.knexQuery({
+                action: "select",
+                columns: "*",
+                table: "Timelapse Frames",
+                where: [
+                    ['ke','=',groupKey],
+                    ['details','NOT LIKE',`%"archived":"1"%`],
+                ],
+                orderBy: ['time','asc'],
+                limit: 3
+            },(err,frames) => {
+                deleteSetOfTimelapseFrames({
+                    groupKey: groupKey,
+                    err: err,
+                    frames: frames,
+                    storageIndex: null
+                },callback)
+            })
+        }else{
+            callback()
+        }
+    }
+    const deleteFileBinFiles = function(groupKey,callback){
+        if(config.deleteFileBinsOverMax === true){
+            //run purge command
+            if(s.group[groupKey].usedSpaceFileBin > (s.group[groupKey].sizeLimit * (s.group[groupKey].sizeLimitFileBinPercent / 100) * config.cron.deleteOverMaxOffset)){
+                s.knexQuery({
+                    action: "select",
+                    columns: "*",
+                    table: "Files",
+                    where: [
+                        ['ke','=',groupKey],
+                    ],
+                    orderBy: ['time','asc'],
+                    limit: 1
+                },(err,frames) => {
+                    deleteSetOfFileBinFiles({
+                        groupKey: groupKey,
+                        err: err,
+                        frames: frames,
+                        storageIndex: null
+                    },callback)
+                })
+            }else{
+                callback()
+            }
+        }else{
+            callback()
+        }
+    }
+    const deleteCloudVideos = function(groupKey,storageType,storagePoint,callback){
+        const whereGroup = []
+        const cloudDisk = s.group[groupKey].cloudDiskUse[storageType]
+        //run purge command
+        if(cloudDisk.sizeLimitCheck && cloudDisk.usedSpace > (cloudDisk.sizeLimit * config.cron.deleteOverMaxOffset)){
+            s.knexQuery({
+                action: "select",
+                columns: "*",
+                table: "Cloud Videos",
+                where: [
+                    ['status','!=','0'],
+                    ['ke','=',groupKey],
+                    ['details','LIKE',`%"type":"${storageType}"%`],
+                ],
+                orderBy: ['time','asc'],
+                limit: 2
+            },function(err,videos) {
+                if(!videos)return console.log(err)
+                var whereQuery = [
+                    ['ke','=',groupKey],
+                ]
+                var didOne = false
+                videos.forEach(function(video){
+                    video.dir = s.getVideoDirectory(video) + s.formattedTime(video.time) + '.' + video.ext
+                    const queryGroup = {
+                        mid: video.mid,
+                        time: video.time,
+                    }
+                    if(whereGroup.length > 0)queryGroup.__separator = 'or'
+                    whereGroup.push(queryGroup)
+                    s.setCloudDiskUsedForGroup(e.ke,{
+                        amount : -(video.size/1048576),
+                        storageType : storageType
+                    })
+                    s.deleteVideoFromCloudExtensionsRunner(e,storageType,video)
+                })
+                const whereGroupLength = whereGroup.length
+                if(whereGroupLength > 0){
+                    whereQuery[1] = whereGroup
+                    s.knexQuery({
+                        action: "delete",
+                        table: "Cloud Videos",
+                        where: whereQuery
+                    },() => {
+                        deleteCloudVideos(groupKey,storageType,storagePoint,callback)
+                    })
+                }else{
+                    callback()
+                }
+            })
+        }else{
+            callback()
+        }
+    }
+    const deleteCloudTimelapseFrames = function(groupKey,storageType,storagePoint,callback){
+        const whereGroup = []
+        var cloudDisk = s.group[e.ke].cloudDiskUse[storageType]
+        //run purge command
+        if(cloudDisk.usedSpaceTimelapseFrames > (cloudDisk.sizeLimit * (s.group[e.ke].sizeLimitTimelapseFramesPercent / 100) * config.cron.deleteOverMaxOffset)){
+            s.knexQuery({
+                action: "select",
+                columns: "*",
+                table: "Cloud Timelapse Frames",
+                where: [
+                    ['ke','=',e.ke],
+                    ['details','NOT LIKE',`%"archived":"1"%`],
+                ],
+                orderBy: ['time','asc'],
+                limit: 3
+            },(err,frames) => {
+                if(!frames)return console.log(err)
+                var whereQuery = [
+                    ['ke','=',e.ke],
+                ]
+                frames.forEach(function(frame){
+                    frame.dir = s.getVideoDirectory(frame) + s.formattedTime(frame.time) + '.' + frame.ext
+                    const queryGroup = {
+                        mid: frame.mid,
+                        time: frame.time,
+                    }
+                    if(whereGroup.length > 0)queryGroup.__separator = 'or'
+                    whereGroup.push(queryGroup)
+                    s.setCloudDiskUsedForGroup(e.ke,{
+                        amount : -(frame.size/1048576),
+                        storageType : storageType
+                    })
+                    s.deleteVideoFromCloudExtensionsRunner(e,storageType,frame)
+                })
+                const whereGroupLength = whereGroup.length
+                if(whereGroupLength > 0){
+                    whereQuery[1] = whereGroup
+                    s.knexQuery({
+                        action: "delete",
+                        table: "Cloud Timelapse Frames",
+                        where: whereQuery
+                    },() => {
+                        deleteCloudTimelapseFrames(groupKey,storageType,storagePoint,callback)
+                    })
+                }else{
+                    callback()
+                }
+            })
+        }else{
+            callback()
+        }
+    }
+    return {
+        deleteSetOfVideos: deleteSetOfVideos,
+        deleteSetOfTimelapseFrames: deleteSetOfTimelapseFrames,
+        deleteSetOfFileBinFiles: deleteSetOfFileBinFiles,
+        deleteAddStorageVideos: deleteAddStorageVideos,
+        deleteMainVideos: deleteMainVideos,
+        deleteTimelapseFrames: deleteTimelapseFrames,
+        deleteFileBinFiles: deleteFileBinFiles,
+        deleteCloudVideos: deleteCloudVideos,
+        deleteCloudTimelapseFrames: deleteCloudTimelapseFrames,
+    }
+}

libs/videoDropInServer.js

@@ -31,8 +31,19 @@ module.exports = function(s,config,lang,app,io){
                                         details.dir = monitor.details.dir
                                     }
                                     var timeNow = new Date(s.nameToTime(filename))
-                                    s.sqlQuery('INSERT INTO `Timelapse Frames` (ke,mid,details,filename,size,time) VALUES (?,?,?,?,?,?)',[ke,mid,s.s(details),filename,fileStats.size,timeNow])
-                                    s.setDiskUsedForGroup(monitor,fileStats.size / 1048576)
+                                    s.knexQuery({
+                                        action: "insert",
+                                        table: "Timelapse Frames",
+                                        insert: {
+                                            ke: ke,
+                                            mid: mid,
+                                            details: s.s(details),
+                                            filename: filename,
+                                            size: fileStats.size,
+                                            time: timeNow,
+                                        }
+                                    })
+                                    s.setDiskUsedForGroup(monitor.ke,fileStats.size / 1048576)
                                 }
                                 // else{
                                 //     s.insertDatabaseRow(

libs/videos.js

@@ -64,17 +64,20 @@ module.exports = function(s,config,lang){
             k.details.dir = e.details.dir
         }
         if(config.useUTC === true)k.details.isUTC = config.useUTC;
-        var save = [
-            e.mid,
-            e.ke,
-            k.startTime,
-            e.ext,
-            1,
-            s.s(k.details),
-            k.filesize,
-            k.endTime,
-        ]
-        s.sqlQuery('INSERT INTO Videos (mid,ke,time,ext,status,details,size,end) VALUES (?,?,?,?,?,?,?,?)',save,function(err){
+        s.knexQuery({
+            action: "insert",
+            table: "Videos",
+            insert: {
+                ke: e.ke,
+                mid: e.mid,
+                time: k.startTime,
+                ext: e.ext,
+                status: 1,
+                details: s.s(k.details),
+                size: k.filesize,
+                end: k.endTime,
+            }
+        },(err) => {
             if(callback)callback(err)
             fs.chmod(k.dir+k.file,0o777,function(err){
 
@@ -90,7 +93,11 @@ module.exports = function(s,config,lang){
         e.dir = s.getVideoDirectory(e)
         k.dir = e.dir.toString()
         if(s.group[e.ke].activeMonitors[e.id].childNode){
-            s.cx({f:'insertCompleted',d:s.group[e.ke].rawMonitorConfigurations[e.id],k:k},s.group[e.ke].activeMonitors[e.id].childNodeId);
+            s.cx({
+                f: 'insertCompleted',
+                d: s.group[e.ke].rawMonitorConfigurations[e.id],
+                k: k
+            },s.group[e.ke].activeMonitors[e.id].childNodeId);
         }else{
             //get file directory
             k.fileExists = fs.existsSync(k.dir+k.file)
@@ -108,7 +115,7 @@ module.exports = function(s,config,lang){
             }
             if(k.fileExists===true){
                 //close video row
-                k.details = {}
+                k.details = k.details && k.details instanceof Object ? k.details : {}
                 k.stat = fs.statSync(k.dir+k.file)
                 k.filesize = k.stat.size
                 k.filesizeMB = parseFloat((k.filesize/1048576).toFixed(2))
@@ -126,33 +133,28 @@ module.exports = function(s,config,lang){
                 if(!e.ext){e.ext = k.filename.split('.')[1]}
                 //send event for completed recording
                 if(config.childNodes.enabled === true && config.childNodes.mode === 'child' && config.childNodes.host){
+                    const response = {
+                        mid: e.mid,
+                        ke: e.ke,
+                        filename: k.filename,
+                        d: s.cleanMonitorObject(e),
+                        filesize: k.filesize,
+                        time: s.timeObject(k.startTime).format('YYYY-MM-DD HH:mm:ss'),
+                        end: s.timeObject(k.endTime).format('YYYY-MM-DD HH:mm:ss')
+                    }
                     fs.createReadStream(k.dir+k.filename,{ highWaterMark: 500 })
                     .on('data',function(data){
-                        s.cx({
+                        s.cx(Object.assign(response,{
                             f:'created_file_chunk',
-                            mid:e.mid,
-                            ke:e.ke,
-                            chunk:data,
-                            filename:k.filename,
-                            d:s.cleanMonitorObject(e),
-                            filesize:e.filesize,
-                            time:s.timeObject(k.startTime).format(),
-                            end:s.timeObject(k.endTime).format()
-                        })
+                            chunk: data,
+                        }))
                     })
                     .on('close',function(){
                         clearTimeout(s.group[e.ke].activeMonitors[e.id].recordingChecker)
                         clearTimeout(s.group[e.ke].activeMonitors[e.id].streamChecker)
-                        s.cx({
+                        s.cx(Object.assign(response,{
                             f:'created_file',
-                            mid:e.id,
-                            ke:e.ke,
-                            filename:k.filename,
-                            d:s.cleanMonitorObject(e),
-                            filesize:k.filesize,
-                            time:s.timeObject(k.startTime).format(),
-                            end:s.timeObject(k.endTime).format()
-                        })
+                        }))
                     })
                 }else{
                     var href = '/videos/'+e.ke+'/'+e.mid+'/'+k.filename
@@ -169,16 +171,16 @@ module.exports = function(s,config,lang){
                         events: k.events && k.events.length > 0 ? k.events : null
                     },'GRP_'+e.ke,'video_view')
                     //purge over max
-                    s.purgeDiskForGroup(e)
+                    s.purgeDiskForGroup(e.ke)
                     //send new diskUsage values
                     var storageIndex = s.getVideoStorageIndex(e)
                     if(storageIndex){
-                        s.setDiskUsedForGroupAddStorage(e,{
+                        s.setDiskUsedForGroupAddStorage(e.ke,{
                             size: k.filesizeMB,
                             storageIndex: storageIndex
                         })
                     }else{
-                        s.setDiskUsedForGroup(e,k.filesizeMB)
+                        s.setDiskUsedForGroup(e.ke,k.filesizeMB)
                     }
                     s.onBeforeInsertCompletedVideoExtensions.forEach(function(extender){
                         extender(e,k)
@@ -211,8 +213,17 @@ module.exports = function(s,config,lang){
             time = e.time
         }
         time = new Date(time)
-        var queryValues = [e.id,e.ke,time];
-        s.sqlQuery('SELECT * FROM Videos WHERE `mid`=? AND `ke`=? AND `time`=?',queryValues,function(err,r){
+        const whereQuery = {
+            ke: e.ke,
+            mid: e.id,
+            time: time,
+        }
+        s.knexQuery({
+            action: "select",
+            columns: "*",
+            table: "Videos",
+            where: whereQuery
+        },(err,r) => {
             if(r && r[0]){
                 r = r[0]
                 fs.chmod(e.dir+filename,0o777,function(err){
@@ -226,14 +237,18 @@ module.exports = function(s,config,lang){
                     },'GRP_'+e.ke);
                     var storageIndex = s.getVideoStorageIndex(e)
                     if(storageIndex){
-                        s.setDiskUsedForGroupAddStorage(e,{
+                        s.setDiskUsedForGroupAddStorage(e.ke,{
                             size: -(r.size / 1048576),
                             storageIndex: storageIndex
                         })
                     }else{
-                        s.setDiskUsedForGroup(e,-(r.size / 1048576))
+                        s.setDiskUsedForGroup(e.ke,-(r.size / 1048576))
                     }
-                    s.sqlQuery('DELETE FROM Videos WHERE `mid`=? AND `ke`=? AND `time`=?',queryValues,function(err){
+                    s.knexQuery({
+                        action: "delete",
+                        table: "Videos",
+                        where: whereQuery
+                    },(err) => {
                         if(err){
                             s.systemLog(lang['File Delete Error'] + ' : '+e.ke+' : '+' : '+e.id,err)
                         }
@@ -254,9 +269,8 @@ module.exports = function(s,config,lang){
     }
     s.deleteListOfVideos = function(videos){
         var deleteSetOfVideos = function(videos){
-            var query = 'DELETE FROM Videos WHERE '
-            var videoQuery = []
-            var queryValues = []
+            const whereQuery = []
+            var didOne = false;
             videos.forEach(function(video){
                 s.checkDetails(video)
                 //e = video object
@@ -277,37 +291,45 @@ module.exports = function(s,config,lang){
                     time = video.time
                 }
                 time = new Date(time)
-                fs.chmod(video.dir+filename,0o777,function(err){
+                fs.chmod(video.dir + filename,0o777,function(err){
                     s.tx({
                         f: 'video_delete',
                         filename: filename,
-                        mid: video.id,
+                        mid: video.mid,
                         ke: video.ke,
                         time: s.nameToTime(filename),
                         end: s.formattedTime(new Date,'YYYY-MM-DD HH:mm:ss')
                     },'GRP_'+video.ke);
                     var storageIndex = s.getVideoStorageIndex(video)
                     if(storageIndex){
-                        s.setDiskUsedForGroupAddStorage(video,{
+                        s.setDiskUsedForGroupAddStorage(video.ke,{
                             size: -(video.size / 1048576),
                             storageIndex: storageIndex
                         })
                     }else{
-                        s.setDiskUsedForGroup(video,-(video.size / 1048576))
+                        s.setDiskUsedForGroup(video.ke,-(video.size / 1048576))
                     }
-                    fs.unlink(video.dir+filename,function(err){
-                        fs.stat(video.dir+filename,function(err){
+                    fs.unlink(video.dir + filename,function(err){
+                        fs.stat(video.dir + filename,function(err){
                             if(!err){
-                                s.file('delete',video.dir+filename)
+                                s.file('delete',video.dir + filename)
                             }
                         })
                     })
                 })
-                videoQuery.push('(`mid`=? AND `ke`=? AND `time`=?)')
-                queryValues = queryValues.concat([video.id,video.ke,time])
+                const queryGroup = {
+                    ke: video.ke,
+                    mid: video.mid,
+                    time: time,
+                }
+                if(whereQuery.length > 0)queryGroup.__separator = 'or'
+                whereQuery.push(queryGroup)
             })
-            query += videoQuery.join(' OR ')
-            s.sqlQuery(query,queryValues,function(err){
+            s.knexQuery({
+                action: "delete",
+                table: "Videos",
+                where: whereQuery
+            },(err) => {
                 if(err){
                     s.systemLog(lang['List of Videos Delete Error'],err)
                 }
@@ -339,11 +361,24 @@ module.exports = function(s,config,lang){
     s.deleteVideoFromCloud = function(e){
         // e = video object
         s.checkDetails(e)
-        var videoSelector = [e.id,e.ke,new Date(e.time)]
-        s.sqlQuery('SELECT * FROM `Cloud Videos` WHERE `mid`=? AND `ke`=? AND `time`=?',videoSelector,function(err,r){
+        const whereQuery = {
+            ke: e.ke,
+            mid: e.mid,
+            time: new Date(e.time),
+        }
+        s.knexQuery({
+            action: "select",
+            columns: "*",
+            table: "Cloud Videos",
+            where: whereQuery
+        },(err,r) => {
             if(r&&r[0]){
                 r = r[0]
-                s.sqlQuery('DELETE FROM `Cloud Videos` WHERE `mid`=? AND `ke`=? AND `time`=?',videoSelector,function(){
+                s.knexQuery({
+                    action: "delete",
+                    table: "Cloud Videos",
+                    where: whereQuery
+                },(err,r) => {
                     s.deleteVideoFromCloudExtensionsRunner(e,r)
                 })
             }else{
@@ -375,18 +410,23 @@ module.exports = function(s,config,lang){
                     }
                     fiveRecentFiles.forEach(function(filename){
                         if(/T[0-9][0-9]-[0-9][0-9]-[0-9][0-9]./.test(filename)){
-                            var queryValues = [
-                                monitor.ke,
-                                monitor.mid,
-                                s.nameToTime(filename)
-                            ]
-                            s.sqlQuery('SELECT * FROM Videos WHERE ke=? AND mid=? AND time=? LIMIT 1',queryValues,function(err,rows){
+                            s.knexQuery({
+                                action: "select",
+                                columns: "*",
+                                table: "Videos",
+                                where: [
+                                    ['ke','=',monitor.ke],
+                                    ['mid','=',monitor.mid],
+                                    ['time','=',s.nameToTime(filename)],
+                                ],
+                                limit: 1
+                            },(err,rows) => {
                                 if(!err && (!rows || !rows[0])){
                                     ++orphanedFilesCount
                                     var video = rows[0]
                                     s.insertCompletedVideo(monitor,{
                                         file : filename
-                                    },function(){
+                                    },() => {
                                         fileComplete()
                                     })
                                 }else{
@@ -407,28 +447,28 @@ module.exports = function(s,config,lang){
         var ext = filePath.split('.')
         ext = ext[ext.length - 1]
         var total = fs.statSync(filePath).size;
-        if (req.headers['range']) {
-            try{
-                var range = req.headers.range;
-                var parts = range.replace(/bytes=/, "").split("-");
-                var partialstart = parts[0];
-                var partialend = parts[1];
-                var start = parseInt(partialstart, 10);
-                var end = partialend ? parseInt(partialend, 10) : total-1;
-                var chunksize = (end-start)+1;
-                var file = fs.createReadStream(filePath, {start: start, end: end});
-                req.headerWrite={ 'Content-Range': 'bytes ' + start + '-' + end + '/' + total, 'Accept-Ranges': 'bytes', 'Content-Length': chunksize, 'Content-Type': 'video/'+ext }
-                req.writeCode=206
-            }catch(err){
-                req.headerWrite={ 'Content-Length': total, 'Content-Type': 'video/'+ext};
-                var file = fs.createReadStream(filePath)
-                req.writeCode=200
-            }
-        } else {
+        // if (req.headers['range']) {
+        //     try{
+        //         var range = req.headers.range;
+        //         var parts = range.replace(/bytes=/, "").split("-");
+        //         var partialstart = parts[0];
+        //         var partialend = parts[1];
+        //         var start = parseInt(partialstart, 10);
+        //         var end = partialend ? parseInt(partialend, 10) : total-1;
+        //         var chunksize = (end-start)+1;
+        //         var file = fs.createReadStream(filePath, {start: start, end: end});
+        //         req.headerWrite={ 'Content-Range': 'bytes ' + start + '-' + end + '/' + total, 'Accept-Ranges': 'bytes', 'Content-Length': chunksize, 'Content-Type': 'video/'+ext }
+        //         req.writeCode=206
+        //     }catch(err){
+        //         req.headerWrite={ 'Content-Length': total, 'Content-Type': 'video/'+ext};
+        //         var file = fs.createReadStream(filePath)
+        //         req.writeCode=200
+        //     }
+        // } else {
             req.headerWrite={ 'Content-Length': total, 'Content-Type': 'video/'+ext};
             var file = fs.createReadStream(filePath)
             req.writeCode=200
-        }
+        // }
         if(req.query.downloadName){
             req.headerWrite['content-disposition']='attachment; filename="'+req.query.downloadName+'"';
         }
@@ -471,7 +511,7 @@ module.exports = function(s,config,lang){
                 if(!fs.existsSync(finalMp4OutputLocation)){
                     var currentFile = 0
                     var completionTimeout
-                    var commandString = `ffmpeg -y -pattern_type glob -f image2pipe -vcodec mjpeg -r ${framesPerSecond} -analyzeduration 10 -i - -q:v 1 -c:v libx264 -r ${framesPerSecond} "${finalMp4OutputLocation}"`
+                    var commandString = `ffmpeg -y -f image2pipe -vcodec mjpeg -r ${framesPerSecond} -analyzeduration 10 -i - -q:v 1 -c:v libx264 -r ${framesPerSecond} "${finalMp4OutputLocation}"`
                     fs.writeFileSync(commandTempLocation,commandString)
                     var videoBuildProcess = spawn('sh',[commandTempLocation])
                     videoBuildProcess.stderr.on('data',function(data){
@@ -487,8 +527,19 @@ module.exports = function(s,config,lang){
                         var timeNow = new Date()
                         var fileStats = fs.statSync(finalMp4OutputLocation)
                         var details = {}
-                        s.sqlQuery('INSERT INTO `Files` (ke,mid,details,name,size,time) VALUES (?,?,?,?,?,?)',[ke,mid,s.s(details),finalFileName + '.mp4',fileStats.size,timeNow])
-                        s.setDiskUsedForGroup({ke: ke},fileStats.size / 1048576,'fileBin')
+                        s.knexQuery({
+                            action: "insert",
+                            table: "Files",
+                            insert: {
+                                ke: ke,
+                                mid: mid,
+                                details: s.s(details),
+                                name: finalFileName + '.mp4',
+                                size: fileStats.size,
+                                time: timeNow,
+                            }
+                        })
+                        s.setDiskUsedForGroup(ke,fileStats.size / 1048576,'fileBin')
                         fs.unlink(commandTempLocation,function(){
 
                         })

libs/webServerAdminPaths.js

@@ -25,15 +25,18 @@ module.exports = function(s,config,lang,app){
             var mail = form.mail || s.getPostData(req,'mail',false)
             if(form){
                 var keys = ['details']
-                var condition = []
-                var value = []
-                keys.forEach(function(v){
-                    condition.push(v+'=?')
-                    if(form[v] instanceof Object)form[v] = JSON.stringify(form[v])
-                    value.push(form[v])
+                const updateQuery = {
+                    details: s.stringJSON(form.details)
+                }
+                s.knexQuery({
+                    action: "update",
+                    table: "Users",
+                    update: updateQuery,
+                    where: [
+                        ['ke','=',req.params.ke],
+                        ['uid','=',uid],
+                    ]
                 })
-                value = value.concat([req.params.ke,uid])
-                s.sqlQuery("UPDATE Users SET "+condition.join(',')+" WHERE ke=? AND uid=?",value)
                 s.tx({
                     f: 'edit_sub_account',
                     ke: req.params.ke,
@@ -42,7 +45,15 @@ module.exports = function(s,config,lang,app){
                     form: form
                 },'ADM_'+req.params.ke)
                 endData.ok = true
-                s.sqlQuery("SELECT * FROM API WHERE ke=? AND uid=?",[req.params.ke,uid],function(err,rows){
+                s.knexQuery({
+                    action: "select",
+                    columns: "*",
+                    table: "API",
+                    where: [
+                        ['ke','=',req.params.ke],
+                        ['uid','=',uid],
+                    ]
+                },function(err,rows){
                     if(rows && rows[0]){
                         rows.forEach(function(row){
                             delete(s.api[row.code])
@@ -71,13 +82,36 @@ module.exports = function(s,config,lang,app){
             var form = s.getPostData(req) || {}
             var uid = form.uid || s.getPostData(req,'uid',false)
             var mail = form.mail || s.getPostData(req,'mail',false)
-            s.sqlQuery('DELETE FROM Users WHERE uid=? AND ke=? AND mail=?',[uid,req.params.ke,mail])
-            s.sqlQuery("SELECT * FROM API WHERE ke=? AND uid=?",[req.params.ke,uid],function(err,rows){
+            s.knexQuery({
+                action: "delete",
+                table: "Users",
+                where: {
+                    ke: req.params.ke,
+                    uid: uid,
+                    mail: mail,
+                }
+            })
+            s.knexQuery({
+                action: "select",
+                columns: "*",
+                table: "API",
+                where: [
+                    ['ke','=',req.params.ke],
+                    ['uid','=',uid],
+                ]
+            },function(err,rows){
                 if(rows && rows[0]){
                     rows.forEach(function(row){
                         delete(s.api[row.code])
                     })
-                    s.sqlQuery('DELETE FROM API WHERE uid=? AND ke=?',[uid,req.params.ke])
+                    s.knexQuery({
+                        action: "delete",
+                        table: "API",
+                        where: {
+                            ke: req.params.ke,
+                            uid: uid,
+                        }
+                    })
                 }
             })
             s.tx({
@@ -112,8 +146,15 @@ module.exports = function(s,config,lang,app){
             var form = s.getPostData(req)
             if(form.mail !== '' && form.pass !== ''){
                 if(form.pass === form.password_again || form.pass === form.pass_again){
-                    s.sqlQuery('SELECT * FROM Users WHERE mail=?',[form.mail],function(err,r) {
-                        if(r&&r[0]){
+                    s.knexQuery({
+                        action: "select",
+                        columns: "*",
+                        table: "Users",
+                        where: [
+                            ['mail','=',form.mail],
+                        ]
+                    },function(err,r){
+                        if(r && r[0]){
                             //found one exist
                             endData.msg = 'Email address is in use.'
                         }else{
@@ -125,7 +166,17 @@ module.exports = function(s,config,lang,app){
                                 sub: "1",
                                 allmonitors: "1"
                             })
-                            s.sqlQuery('INSERT INTO Users (ke,uid,mail,pass,details) VALUES (?,?,?,?,?)',[req.params.ke,newId,form.mail,s.createHash(form.pass),details])
+                            s.knexQuery({
+                                action: "insert",
+                                table: "Users",
+                                insert: {
+                                    ke: req.params.ke,
+                                    uid: newId,
+                                    mail: form.mail,
+                                    pass: s.createHash(form.pass),
+                                    details: details,
+                                }
+                            })
                             s.tx({
                                 f: 'add_sub_account',
                                 details: details,
@@ -199,8 +250,22 @@ module.exports = function(s,config,lang,app){
                     s.userLog(s.group[req.params.ke].rawMonitorConfigurations[req.params.id],{type:'Monitor Deleted',msg:'by user : '+user.uid});
                     req.params.delete=1;s.camera('stop',req.params);
                     s.tx({f:'monitor_delete',uid:user.uid,mid:req.params.id,ke:req.params.ke},'GRP_'+req.params.ke);
-                    s.sqlQuery('DELETE FROM Monitors WHERE ke=? AND mid=?',[req.params.ke,req.params.id])
-    //                s.sqlQuery('DELETE FROM Files WHERE ke=? AND mid=?',[req.params.ke,req.params.id])
+                    s.knexQuery({
+                        action: "delete",
+                        table: "Monitors",
+                        where: {
+                            ke: req.params.ke,
+                            mid: req.params.id,
+                        }
+                    })
+                    // s.knexQuery({
+                    //     action: "delete",
+                    //     table: "Files",
+                    //     where: {
+                    //         ke: req.params.ke,
+                    //         mid: req.params.id,
+                    //     }
+                    // })
                     if(req.query.deleteFiles === 'true'){
                         //videos
                         s.dir.addStorage.forEach(function(v,n){
@@ -250,28 +315,28 @@ module.exports = function(s,config,lang,app){
             }
             var form = s.getPostData(req)
             if(form){
-                var insert = {
+                const insertQuery = {
                     ke : req.params.ke,
                     uid : user.uid,
                     code : s.gid(30),
                     ip : form.ip,
                     details : s.stringJSON(form.details)
                 }
-                var escapes = []
-                Object.keys(insert).forEach(function(column){
-                    escapes.push('?')
-                });
-                s.sqlQuery('INSERT INTO API ('+Object.keys(insert).join(',')+') VALUES ('+escapes.join(',')+')',Object.values(insert),function(err,r){
-                    insert.time = s.formattedTime(new Date,'YYYY-DD-MM HH:mm:ss');
+                s.knexQuery({
+                    action: "insert",
+                    table: "API",
+                    insert: insertQuery
+                },(err,r) => {
+                    insertQuery.time = s.formattedTime(new Date,'YYYY-DD-MM HH:mm:ss');
                     if(!err){
                         s.tx({
                             f: 'api_key_added',
                             uid: user.uid,
-                            form: insert
+                            form: insertQuery
                         },'GRP_' + req.params.ke)
                         endData.ok = true
                     }
-                    endData.api = insert
+                    endData.api = insertQuery
                     s.closeJsonResponse(res,endData)
                 })
             }else{
@@ -305,16 +370,15 @@ module.exports = function(s,config,lang,app){
                     s.closeJsonResponse(res,endData)
                     return
                 }
-                var row = {
-                    ke : req.params.ke,
-                    uid : user.uid,
-                    code : form.code
-                }
-                var where = []
-                Object.keys(row).forEach(function(column){
-                    where.push(column+'=?')
-                })
-                s.sqlQuery('DELETE FROM API WHERE '+where.join(' AND '),Object.values(row),function(err,r){
+                s.knexQuery({
+                    action: "delete",
+                    table: "API",
+                    where: {
+                        ke: req.params.ke,
+                        uid: user.uid,
+                        code: form.code,
+                    }
+                },(err,r) => {
                     if(!err){
                         s.tx({
                             f: 'api_key_deleted',
@@ -345,15 +409,16 @@ module.exports = function(s,config,lang,app){
             var endData = {
                 ok : false
             }
-            var row = {
+            const whereQuery = {
                 ke : req.params.ke,
                 uid : user.uid
             }
-            var where = []
-            Object.keys(row).forEach(function(column){
-                where.push(column+'=?')
-            })
-            s.sqlQuery('SELECT * FROM API WHERE '+where.join(' AND '),Object.values(row),function(err,rows){
+            s.knexQuery({
+                action: "select",
+                columns: "*",
+                table: "API",
+                where: whereQuery
+            },function(err,rows) {
                 if(rows && rows[0]){
                     rows.forEach(function(row){
                         row.details = JSON.parse(row.details)
@@ -383,7 +448,15 @@ module.exports = function(s,config,lang,app){
                 s.closeJsonResponse(res,endData)
                 return
             }
-            s.sqlQuery("SELECT * FROM Presets WHERE ke=? AND type=?",[req.params.ke,'monitorStates'],function(err,presets){
+            s.knexQuery({
+                action: "select",
+                columns: "*",
+                table: "Presets",
+                where: [
+                    ['ke','=',req.params.ke],
+                    ['type','=','monitorStates'],
+                ]
+            },function(err,presets) {
                 if(presets && presets[0]){
                     endData.ok = true
                     presets.forEach(function(preset){
@@ -435,7 +508,11 @@ module.exports = function(s,config,lang,app){
                                 details: s.s(details),
                                 type: 'monitorStates'
                             }
-                            s.sqlQuery('INSERT INTO Presets ('+Object.keys(insertData).join(',')+') VALUES (?,?,?,?)',Object.values(insertData))
+                            s.knexQuery({
+                                action: "insert",
+                                table: "Presets",
+                                insert: insertData
+                            })
                             s.tx({
                                 f: 'add_group_state',
                                 details: details,
@@ -447,7 +524,17 @@ module.exports = function(s,config,lang,app){
                             var details = Object.assign(preset.details,{
                                 monitors : form.monitors
                             })
-                            s.sqlQuery('UPDATE Presets SET details=? WHERE ke=? AND name=?',[s.s(details),req.params.ke,req.params.stateName])
+                            s.knexQuery({
+                                action: "update",
+                                table: "Presets",
+                                update: {
+                                    details: s.s(details)
+                                },
+                                where: [
+                                    ['ke','=',req.params.ke],
+                                    ['name','=',req.params.stateName],
+                                ]
+                            })
                             s.tx({
                                 f: 'edit_group_state',
                                 details: details,
@@ -465,7 +552,15 @@ module.exports = function(s,config,lang,app){
                             endData.msg = user.lang['State Configuration Not Found']
                             s.closeJsonResponse(res,endData)
                         }else{
-                            s.sqlQuery('DELETE FROM Presets WHERE ke=? AND name=?',[req.params.ke,req.params.stateName],function(err){
+                            s.knexQuery({
+                                action: "delete",
+                                table: "Presets",
+                                update: monitorQuery,
+                                where: {
+                                    ke: req.params.ke,
+                                    name: req.params.stateName,
+                                }
+                            },(err) => {
                                 if(!err){
                                     endData.msg = lang["Deleted State Configuration"]
                                     endData.ok = true

libs/webServerStreamPaths.js

@@ -214,6 +214,28 @@ module.exports = function(s,config,lang,app){
         },res,req);
     });
     /**
+    * API : Get JPEG Snapshot
+    */
+    app.get(config.webPaths.apiPrefix+':auth/icon/:ke/:id', function(req,res){
+        s.auth(req.params,async (user) => {
+            if(user.details.sub&&user.details.allmonitors!=='1'&&user.details.monitors&&user.details.monitors.indexOf(req.params.id)===-1){
+                res.end(user.lang['Not Permitted'])
+                return
+            }
+            res.writeHead(200, {
+                'Content-Type': 'image/jpeg',
+                'Cache-Control': 'no-cache',
+                'Pragma': 'no-cache'
+            });
+            res.end(await s.getCameraSnapshot({
+                ke: req.params.ke,
+                mid: req.params.id,
+            },{
+                useIcon: true
+            }))
+        },res,req);
+    });
+    /**
     * API : Get FLV Stream
     */
     app.get([config.webPaths.apiPrefix+':auth/flv/:ke/:id/s.flv',config.webPaths.apiPrefix+':auth/flv/:ke/:id/:channel/s.flv'], function(req,res) {

libs/webServerSuperPaths.js

@@ -7,62 +7,31 @@ var exec = require('child_process').exec;
 var spawn = require('child_process').spawn;
 var execSync = require('child_process').execSync;
 module.exports = function(s,config,lang,app){
+
     /**
     * API : Superuser : Get Logs
     */
-    app.all([config.webPaths.supersuperApiPrefix+':auth/logs'], function (req,res){
+    app.all([config.webPaths.superApiPrefix+':auth/logs'], function (req,res){
         req.ret={ok:false};
         s.superAuth(req.params,function(resp){
-            req.sql='SELECT * FROM Logs WHERE ke=?';req.ar=['$'];
-            if(!req.params.id){
-                if(user.details.sub&&user.details.monitors&&user.details.allmonitors!=='1'){
-                    try{user.details.monitors=JSON.parse(user.details.monitors);}catch(er){}
-                    req.or=[];
-                    user.details.monitors.forEach(function(v,n){
-                        req.or.push('mid=?');req.ar.push(v)
-                    })
-                    req.sql+=' AND ('+req.or.join(' OR ')+')'
-                }
-            }else{
-                if(!user.details.sub||user.details.allmonitors!=='0'||user.details.monitors.indexOf(req.params.id)>-1||req.params.id.indexOf('$')>-1){
-                    req.sql+=' and mid=?';req.ar.push(req.params.id)
-                }else{
-                    res.end('[]');
-                    return;
-                }
-            }
-            if(req.query.start||req.query.end){
-                if(!req.query.startOperator||req.query.startOperator==''){
-                    req.query.startOperator='>='
-                }
-                if(!req.query.endOperator||req.query.endOperator==''){
-                    req.query.endOperator='<='
-                }
-                if(req.query.start && req.query.start !== '' && req.query.end && req.query.end !== ''){
-                    req.query.start = s.stringToSqlTime(req.query.start)
-                    req.query.end = s.stringToSqlTime(req.query.end)
-                    req.sql+=' AND `time` '+req.query.startOperator+' ? AND `time` '+req.query.endOperator+' ?';
-                    req.ar.push(req.query.start)
-                    req.ar.push(req.query.end)
-                }else if(req.query.start && req.query.start !== ''){
-                    req.query.start = s.stringToSqlTime(req.query.start)
-                    req.sql+=' AND `time` '+req.query.startOperator+' ?';
-                    req.ar.push(req.query.start)
-                }
-            }
-            if(!req.query.limit||req.query.limit==''){req.query.limit=50}
-            req.sql+=' ORDER BY `time` DESC LIMIT '+req.query.limit+'';
-            s.sqlQuery(req.sql,req.ar,function(err,r){
-                if(err){
-                    err.sql=req.sql;
-                    res.end(s.prettyPrint(err));
-                    return
-                }
-                if(!r){r=[]}
-                r.forEach(function(v,n){
-                    r[n].info=JSON.parse(v.info)
+            const monitorRestrictions = s.getMonitorRestrictions(user.details,req.params.id)
+            s.getDatabaseRows({
+                monitorRestrictions: monitorRestrictions,
+                table: 'Logs',
+                groupKey: req.params.ke,
+                date: req.query.date,
+                startDate: req.query.start,
+                endDate: req.query.end,
+                startOperator: req.query.startOperator,
+                endOperator: req.query.endOperator,
+                limit: req.query.limit,
+                archived: req.query.archived,
+                endIsStartTo: true
+            },(response) => {
+                response.rows.forEach(function(v,n){
+                    r[n].info = JSON.parse(v.info)
                 })
-                res.end(s.prettyPrint(r))
+                s.closeJsonResponse(res,r)
             })
         },res,req)
     })
@@ -71,11 +40,16 @@ module.exports = function(s,config,lang,app){
     */
     app.all(config.webPaths.superApiPrefix+':auth/logs/delete', function (req,res){
         s.superAuth(req.params,function(resp){
-            s.sqlQuery('DELETE FROM Logs WHERE ke=?',['$'],function(){
-                var endData = {
-                    ok : true
+            s.knexQuery({
+                action: "delete",
+                table: "Logs",
+                where: {
+                    ke: '$'
                 }
-                res.end(s.prettyPrint(endData))
+            },() => {
+                s.closeJsonResponse(res,{
+                    ok : true
+                })
             })
         },res,req)
     })
@@ -99,7 +73,7 @@ module.exports = function(s,config,lang,app){
             var endData = {
                 ok : true
             }
-            res.end(s.prettyPrint(endData))
+            s.closeJsonResponse(res,endData)
         },res,req)
     })
     /**
@@ -124,7 +98,7 @@ module.exports = function(s,config,lang,app){
                 s.systemLog('Flush PM2 Logs',{by:resp.$user.mail,ip:resp.ip})
                 endData.logsOuput = execSync('pm2 flush')
             }
-            res.end(s.prettyPrint(endData))
+            s.closeJsonResponse(res,endData)
         },res,req)
     })
     /**
@@ -145,11 +119,23 @@ module.exports = function(s,config,lang,app){
                     ip: resp.ip,
                     old:jsonfile.readFileSync(s.location.config)
                 })
+                try{
+                    if(config.thisIsDocker){
+                        const dockerConfigFile = '/config/conf.json'
+                        fs.stat(dockerConfigFile,(err) => {
+                            if(!err){
+                                fs.writeFile(dockerConfigFile,JSON.stringify(postBody,null,3),function(){})
+                            }
+                        })
+                    }
+                }catch(err){
+                    console.log(err)
+                }
                 jsonfile.writeFile(s.location.config,postBody,{spaces: 2},function(){
                     s.tx({f:'save_configuration'},'$')
                 })
             }
-            res.end(s.prettyPrint(endData))
+            s.closeJsonResponse(res,endData)
         },res,req)
     })
     /**
@@ -163,22 +149,23 @@ module.exports = function(s,config,lang,app){
             var endData = {
                 ok : true
             }
-            searchQuery = 'SELECT ke,uid,auth,mail,details FROM Users'
-            queryVals = []
+            const whereQuery = []
             switch(req.params.type){
                 case'admin':case'administrator':
-                    searchQuery += ' WHERE details NOT LIKE ?'
-                    queryVals.push('%"sub"%')
+                    whereQuery.push(['details','NOT LIKE','%"sub"%'])
                 break;
                 case'sub':case'subaccount':
-                    searchQuery += ' WHERE details LIKE ?'
-                    queryVals.push('%"sub"%')
+                    whereQuery.push(['details','LIKE','%"sub"%'])
                 break;
             }
-            // ' WHERE details NOT LIKE ?'
-            s.sqlQuery(searchQuery,queryVals,function(err,users) {
+            s.knexQuery({
+                action: "select",
+                columns: "ke,uid,auth,mail,details",
+                table: "Users",
+                where: whereQuery
+            },(err,users) => {
                 endData.users = users
-                res.end(s.prettyPrint(endData))
+                s.closeJsonResponse(res,endData)
             })
         },res,req)
     })
@@ -192,7 +179,7 @@ module.exports = function(s,config,lang,app){
             }
             var form = s.getPostData(req)
             if(form){
-                var currentSuperUserList = jsonfile.readFileSync(s.location.super)
+                var currentSuperUserList = JSON.parse(fs.readFileSync(s.location.super))
                 var currentSuperUser = {}
                 var currentSuperUserPosition = -1
                 //find this user in current list
@@ -230,14 +217,26 @@ module.exports = function(s,config,lang,app){
                     currentSuperUserList.push(currentSuperUser)
                 }
                 //update master list in system
-                jsonfile.writeFile(s.location.super,currentSuperUserList,{spaces: 2},function(){
+                try{
+                    if(config.thisIsDocker){
+                        const dockerSuperFile = '/config/super.json'
+                        fs.stat(dockerSuperFile,(err) => {
+                            if(!err){
+                                fs.writeFile(dockerSuperFile,JSON.stringify(currentSuperUserList,null,3),function(){})
+                            }
+                        })
+                    }
+                }catch(err){
+                    console.log(err)
+                }
+                fs.writeFile(s.location.super,JSON.stringify(currentSuperUserList,null,3),function(){
                     s.tx({f:'save_preferences'},'$')
                 })
             }else{
                 endData.ok = false
                 endData.msg = lang.postDataBroken
             }
-            res.end(s.prettyPrint(endData))
+            s.closeJsonResponse(res,endData)
         },res,req)
     })
     /**
@@ -249,7 +248,7 @@ module.exports = function(s,config,lang,app){
                 ok : false
             }
             var close = function(){
-                res.end(s.prettyPrint(endData))
+                s.closeJsonResponse(res,endData)
             }
             var isCallbacking = false
             var form = s.getPostData(req)
@@ -257,7 +256,14 @@ module.exports = function(s,config,lang,app){
                 if(form.mail !== '' && form.pass !== ''){
                     if(form.pass === form.password_again || form.pass === form.pass_again){
                         isCallbacking = true
-                        s.sqlQuery('SELECT * FROM Users WHERE mail=?',[form.mail],function(err,r) {
+                        s.knexQuery({
+                            action: "select",
+                            columns: "*",
+                            table: "Users",
+                            where: [
+                                ['mail','=',form.mail]
+                            ]
+                        },(err,r) => {
                             if(r&&r[0]){
                                 //found address already exists
                                 endData.msg = lang['Email address is in use.'];
@@ -277,16 +283,17 @@ module.exports = function(s,config,lang,app){
                                     form.details = JSON.stringify(form.details)
                                 }
                                 //write user to db
-                                s.sqlQuery(
-                                    'INSERT INTO Users (ke,uid,mail,pass,details) VALUES (?,?,?,?,?)',
-                                    [
-                                        form.ke,
-                                        form.uid,
-                                        form.mail,
-                                        s.createHash(form.pass),
-                                        form.details
-                                    ]
-                                )
+                                s.knexQuery({
+                                    action: "insert",
+                                    table: "Users",
+                                    insert: {
+                                        ke: form.ke,
+                                        uid: form.uid,
+                                        mail: form.mail,
+                                        pass: s.createHash(form.pass),
+                                        details: form.details
+                                    }
+                                })
                                 s.tx({f:'add_account',details:form.details,ke:form.ke,uid:form.uid,mail:form.mail},'$')
                                 endData.user = Object.assign(form,{})
                                 //init user
@@ -315,12 +322,19 @@ module.exports = function(s,config,lang,app){
                 ok : false
             }
             var close = function(){
-                res.end(s.prettyPrint(endData))
+                s.closeJsonResponse(res,endData)
             }
             var form = s.getPostData(req)
             if(form){
                 var account = s.getPostData(req,'account')
-                s.sqlQuery('SELECT * FROM Users WHERE mail=?',[account.mail],function(err,r) {
+                s.knexQuery({
+                    action: "select",
+                    columns: "*",
+                    table: "Users",
+                    where: [
+                        ['mail','=',account.mail]
+                    ]
+                },(err,r) => {
                     if(r && r[0]){
                         r = r[0]
                         var details = JSON.parse(r.details)
@@ -337,25 +351,16 @@ module.exports = function(s,config,lang,app){
                         }
                         delete(form.password_again);
                         delete(form.pass_again);
-                        var keys = Object.keys(form)
-                        var set = []
-                        var values = []
-                        keys.forEach(function(v,n){
-                            if(
-                                set === 'ke' ||
-                                !form[v]
-                            ){
-                                //skip
-                                return
-                            }
-                            set.push(v+'=?')
-                            if(v === 'details'){
-                                form[v] = s.stringJSON(Object.assign(details,s.parseJSON(form[v])))
-                            }
-                            values.push(form[v])
-                        })
-                        values.push(account.mail)
-                        s.sqlQuery('UPDATE Users SET '+set.join(',')+' WHERE mail=?',values,function(err,r) {
+                        delete(form.ke);
+                        form.details = s.stringJSON(Object.assign(details,s.parseJSON(form.details)))
+                        s.knexQuery({
+                            action: "update",
+                            table: "Users",
+                            update: form,
+                            where: [
+                                ['mail','=',account.mail],
+                            ]
+                        },(err,r) => {
                             if(err){
                                 console.log(err)
                                 endData.error = err
@@ -388,32 +393,78 @@ module.exports = function(s,config,lang,app){
                 ok : true
             }
             var close = function(){
-                res.end(s.prettyPrint(endData))
+                s.closeJsonResponse(res,endData)
             }
             var account = s.getPostData(req,'account')
-            s.sqlQuery('DELETE FROM Users WHERE uid=? AND ke=? AND mail=?',[account.uid,account.ke,account.mail])
-            s.sqlQuery('DELETE FROM API WHERE uid=? AND ke=?',[account.uid,account.ke])
+            s.knexQuery({
+                action: "delete",
+                table: "Users",
+                where: {
+                    ke: account.ke,
+                    uid: account.uid,
+                    mail: account.mail,
+                }
+            })
+            s.knexQuery({
+                action: "delete",
+                table: "API",
+                where:  {
+                    ke: account.ke,
+                    uid: account.uid,
+                }
+            })
             if(s.getPostData(req,'deleteSubAccounts',false) === '1'){
-                s.sqlQuery('DELETE FROM Users WHERE ke=?',[account.ke])
+                s.knexQuery({
+                    action: "delete",
+                    table: "Users",
+                    where:  {
+                        ke: account.ke,
+                    }
+                })
             }
             if(s.getPostData(req,'deleteMonitors',false) == '1'){
-                s.sqlQuery('SELECT * FROM Monitors WHERE ke=?',[account.ke],function(err,monitors){
+                s.knexQuery({
+                    action: "select",
+                    columns: "*",
+                    table: "Monitors",
+                    where:  {
+                        ke: account.ke,
+                    }
+                },(err,monitors) => {
                     if(monitors && monitors[0]){
                         monitors.forEach(function(monitor){
                             s.camera('stop',monitor)
                         })
-                        s.sqlQuery('DELETE FROM Monitors WHERE ke=?',[account.ke])
+                        s.knexQuery({
+                            action: "delete",
+                            table: "Monitors",
+                            where:  {
+                                ke: account.ke,
+                            }
+                        })
                     }
                 })
             }
             if(s.getPostData(req,'deleteVideos',false) == '1'){
-                s.sqlQuery('DELETE FROM Videos WHERE ke=?',[account.ke])
+                s.knexQuery({
+                    action: "delete",
+                    table: "Videos",
+                    where:  {
+                        ke: account.ke,
+                    }
+                })
                 fs.chmod(s.dir.videos+account.ke,0o777,function(err){
                     fs.unlink(s.dir.videos+account.ke,function(err){})
                 })
             }
             if(s.getPostData(req,'deleteEvents',false) == '1'){
-                s.sqlQuery('DELETE FROM Events WHERE ke=?',[account.ke])
+                s.knexQuery({
+                    action: "delete",
+                    table: "Events",
+                    where:  {
+                        ke: account.ke,
+                    }
+                })
             }
             s.tx({f:'delete_account',ke:account.ke,uid:account.uid,mail:account.mail},'$')
             close()
@@ -449,7 +500,11 @@ module.exports = function(s,config,lang,app){
                 if(tableName){
                     var tableIsSelected = s.getPostData(req,tableName) == 1
                     if(tableIsSelected){
-                        s.sqlQuery('SELECT * FROM `' + tableName +'`',[],function(err,dataRows){
+                        s.knexQuery({
+                            action: "select",
+                            columns: "*",
+                            table: tableName
+                        },(err,dataRows) => {
                             endData.database[tableName] = dataRows
                             ++completedTables
                             tableExportLoop(callback)
@@ -551,26 +606,26 @@ module.exports = function(s,config,lang,app){
                             ])
                         break;
                     }
-                    var keysToCheck = []
-                    var valuesToCheck = []
+                    const whereQuery = []
                     fieldsToCheck.forEach(function(key){
-                        keysToCheck.push(key + '= ?')
-                        valuesToCheck.push(row[key])
+                        whereQuery.push([key,'=',row[key]])
                     })
-                    s.sqlQuery('SELECT * FROM ' + tableName + ' WHERE ' + keysToCheck.join(' AND '),valuesToCheck,function(err,selected){
+                    s.knexQuery({
+                        action: "select",
+                        columns: "*",
+                        table: tableName,
+                        where: whereQuery
+                    },(err,selected) => {
                         if(selected && selected[0]){
                             selected = selected[0]
                             rowsExistingAlready[tableName].push(selected)
                             callback()
                         }else{
-                            var rowKeys = Object.keys(row)
-                            var insertEscapes = []
-                            var insertValues = []
-                            rowKeys.forEach(function(key){
-                                insertEscapes.push('?')
-                                insertValues.push(row[key])
-                            })
-                            s.sqlQuery('INSERT INTO ' + tableName + ' (' + rowKeys.join(',') +') VALUES (' + insertEscapes.join(',') + ')',insertValues,function(){
+                            s.knexQuery({
+                                action: "insert",
+                                table: tableName,
+                                insert: row
+                            },(err) => {
                                 if(!err){
                                     ++countOfRowsInserted[tableName]
                                 }
@@ -631,7 +686,7 @@ module.exports = function(s,config,lang,app){
                 ok : true
             }
             s.checkForStalePurgeLocks()
-            res.end(s.prettyPrint(endData))
+            s.closeJsonResponse(res,endData)
         },res,req)
     })
     /**
@@ -669,7 +724,7 @@ module.exports = function(s,config,lang,app){
                 ok : true,
                 childNodes: childNodesJson,
             }
-            res.end(s.prettyPrint(endData))
+            s.closeJsonResponse(res,endData)
         },res,req)
     })
 }

package.json

@@ -15,25 +15,25 @@
   "homepage": "https://gitlab.com/Shinobi-Systems/Shinobi#readme",
   "dependencies": {
     "async": "^3.1.0",
-    "aws-sdk": "^2.279.1",
-    "backblaze-b2": "^1.0.4",
-    "body-parser": "^1.18.3",
-    "connection-tester": "^0.1.1",
-    "cws": "^1.0.0",
-    "discord.js": "^11.3.2",
+    "aws-sdk": "^2.731.0",
+    "backblaze-b2": "^1.5.0",
+    "body-parser": "^1.19.0",
+    "connection-tester": "^0.2.0",
+    "cws": "^1.2.11",
+    "discord.js": "^12.2.0",
     "ejs": "^2.5.5",
     "express": "^4.16.4",
-    "ftp-srv": "^4.3.1",
+    "shinobi-ftpd": "0.2.18",
     "http-proxy": "^1.17.0",
     "jsonfile": "^3.0.1",
-    "knex": "^0.19.5",
-    "ldapauth-fork": "^4.0.2",
-    "moment": "^2.17.0",
+    "knex": "^0.21.4",
+    "ldapauth-fork": "^4.3.3",
+    "moment": "^2.27.0",
     "mp4frag": "^0.2.0",
-    "mysql": "^2.16.0",
+    "mysql": "^2.18.1",
     "node-onvif": "^0.1.7",
     "node-ssh": "^5.1.2",
-    "nodemailer": "^4.0.1",
+    "nodemailer": "^6.4.11",
     "pam-diff": "^1.0.0",
     "path": "^0.12.7",
     "pipe2pam": "^0.6.2",
@@ -41,12 +41,15 @@
     "sat": "^0.7.1",
     "shinobi-sound-detection": "^0.1.8",
     "smtp-server": "^3.5.0",
-    "socket.io": "^2.2.0",
-    "socket.io-client": "^2.2.0",
+    "socket.io": "^2.3.0",
+    "socket.io-client": "^2.3.0",
     "webdav-fs": "^1.11.0",
     "express-fileupload": "^1.1.6-alpha.6",
     "googleapis": "^39.2.0",
-    "tree-kill":"1.2.2"
+    "tree-kill":"1.2.2",
+    "unzipper":"0.10.11",
+    "node-fetch":"2.6.0",
+    "fs-extra": "9.0.1"
   },
   "devDependencies": {},
   "bin": "camera.js",

plugins/pluginBase.js

@@ -237,7 +237,7 @@ module.exports = function(__dirname, config){
                 }
                 processImage(buffer,d,tx,d.frameLocation)
             break;
-            case'frame':''
+            case'frame':
                 try{
                     if(!s.group[d.ke]){
                         s.group[d.ke]={}
@@ -335,7 +335,7 @@ module.exports = function(__dirname, config){
         //start plugin as client
         var retryConnection = 0
         var clearRetryConnectionTimeout
-        maxRetryConnection = config.maxRetryConnection || 5
+        maxRetryConnection = parseInt(config.maxRetryConnection) || 5
         plugLog('Plugin starting as Client, Host Address : '+'ws://'+config.host+':'+config.port)
         if(!config.host){config.host='localhost'}
         const createConnection = () => {

sql/framework.sql

@@ -95,6 +95,7 @@ CREATE TABLE IF NOT EXISTS `Users` (
   `auth` varchar(50) DEFAULT NULL,
   `mail` varchar(100) DEFAULT NULL,
   `pass` varchar(100) DEFAULT NULL,
+  `accountType` int(1) DEFAULT '0',
   `details` longtext,
   UNIQUE KEY `mail` (`mail`)
 ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
@@ -110,7 +111,8 @@ CREATE TABLE IF NOT EXISTS `Videos` (
   `size` float DEFAULT NULL,
   `frames` int(11) DEFAULT NULL,
   `end` timestamp NULL DEFAULT NULL,
-  `status` int(1) DEFAULT '0' COMMENT '0:Building,1:Complete,2:Read,3:Archive',
+  `status` int(1) DEFAULT '0',
+  `archived` int(1) DEFAULT '0',
   `details` text
 ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
 

sql/postgresql/default_data.sql

@@ -1,20 +0,0 @@
--- --------------------------------------------------------
--- Host:                         66.51.132.100
--- Server version:               5.7.16-0ubuntu0.16.04.1 - (Ubuntu)
--- Server OS:                    Linux
--- HeidiSQL Version:             9.3.0.4984
--- --------------------------------------------------------
-
-/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
-/*!40101 SET NAMES utf8mb4 */;
-/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
-/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
--- Dumping data for table ccio.Users: ~0 rows (approximately)
-/*!40000 ALTER TABLE `Users` DISABLE KEYS */;
-INSERT INTO Users (ke, uid, auth, mail, pass, details) VALUES
-	('2Df5hBE', 'XDf5hB3', 'ec49f05c1ddc7d818c61b3343c98cbc6', 'ccio@m03.ca', '5f4dcc3b5aa765d61d8327deb882cf99', '{"days":"10"}');
-INSERT INTO Monitors (mid, ke, name, shto, shfr, details, type, ext, protocol, host, path, port, fps, mode, width, height) VALUES ('bunny', '2Df5hBE', 'Bunny', '[]', '[]', '{"fatal_max":"","notes":"","dir":"","rtsp_transport":"tcp","muser":"","mpass":"","port_force":"0","sfps":"","aduration":"1000000","probesize":"1000000","accelerator":"0","hwaccel":null,"hwaccel_vcodec":"","hwaccel_device":"","stream_type":"hls","stream_mjpeg_clients":"","stream_vcodec":"copy","stream_acodec":"no","hls_time":"","preset_stream":"","hls_list_size":"","signal_check":"","signal_check_log":null,"stream_quality":"","stream_fps":"1","stream_scale_x":"","stream_scale_y":"","rotate_stream":null,"svf":"","stream_timestamp":"0","stream_timestamp_font":"","stream_timestamp_font_size":"","stream_timestamp_color":"","stream_timestamp_box_color":"","stream_timestamp_x":"","stream_timestamp_y":"","stream_watermark":"0","stream_watermark_location":"","stream_watermark_position":null,"snap":"1","snap_fps":"","snap_scale_x":"","snap_scale_y":"","snap_vf":"","vcodec":"copy","crf":"","preset_record":"","acodec":"libvorbis","dqf":null,"cutoff":"10","rotate_record":null,"vf":"","timestamp":"1","timestamp_font":"","timestamp_font_size":"","timestamp_color":"","timestamp_box_color":"","timestamp_x":"","timestamp_y":"","watermark":null,"watermark_location":"","watermark_position":null,"cust_input":"","cust_snap":"","cust_detect":"","cust_stream":"","cust_stream_server":"","cust_record":"","custom_output":"","detector":"0","detector_webhook":null,"detector_webhook_url":"","detector_command_enable":null,"detector_command":"","detector_command_timeout":"","detector_lock_timeout":"","detector_save":null,"detector_frame_save":null,"detector_mail":null,"detector_mail_timeout":"","detector_record_method":null,"detector_trigger":null,"detector_trigger_record_fps":"","detector_timeout":"","watchdog_reset":null,"detector_delete_motionless_videos":null,"detector_send_frames":null,"detector_fps":"","detector_scale_x":"","detector_scale_y":"","detector_use_motion":null,"detector_use_detect_object":null,"detector_frame":null,"detector_sensitivity":"","cords":"","detector_lisence_plate":null,"detector_lisence_plate_country":null,"detector_notrigger":null,"detector_notrigger_mail":null,"detector_notrigger_timeout":"","control":"0","control_base_url":"","control_stop":null,"control_url_stop_timeout":"","control_url_center":"","control_url_left":"","control_url_left_stop":"","control_url_right":"","control_url_right_stop":"","control_url_up":"","control_url_up_stop":"","control_url_down":"","control_url_down_stop":"","control_url_enable_nv":"","control_url_disable_nv":"","control_url_zoom_out":"","control_url_zoom_out_stop":"","control_url_zoom_in":"","control_url_zoom_in_stop":"","groups":"","loglevel":"warning","sqllog":"0","detector_cascades":""}', 'mjpeg', 'mp4', 'http', 'came3.nkansai.ne.jp', '/nphMotionJpeg?Resolution=640x480&Quality=Motion', 81, 15, 'start', 640, 480);
-/*!40000 ALTER TABLE `Users` ENABLE KEYS */;
-/*!40101 SET SQL_MODE=IFNULL(@OLD_SQL_MODE, '') */;
-/*!40014 SET FOREIGN_KEY_CHECKS=IF(@OLD_FOREIGN_KEY_CHECKS IS NULL, 1, @OLD_FOREIGN_KEY_CHECKS) */;
-/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;

sql/postgresql/framework.pgsql

@@ -0,0 +1,218 @@
+/*
+ * PostgresSQL rewrite of framework.sql - dave@dream-tech.com
+ * Placed into open source, no license required here unless you want one, licenses and lawyers 
+ * are the primary bane of good software development. :)
+ *
+ * Trigger code lifted from stack overflow here:
+ * https://stackoverflow.com/questions/9556474/how-do-i-automatically-update-a-timestamp-in-postgresql
+ *
+ * Summary of changes:
+ * a) Removed mysql cruft and comments, no need for 'use'
+ * b) Removed create database statement (I can put one back but usually I create dbs using postgres command line tools:
+ *    e.g. 'createdb foo')
+ * c) Removed all cases of int(\d+) and replaced with just int, postgres does not support those
+ * d) Removed ENGINE=InnoDB
+ * e) Removed default charset statements, Postgresql automatically supports 4-byte UTF8 at database createion
+ * f) Removed backtick quotes and added double quotes
+ * g) All timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP replaced with triggers
+ *    and the ON UPDATE portion removed as postgres doesn't support this sadly
+ * h) tinytext/longtest is changed to just text, generally postgres does a good job of managing arbitrary text columns
+ * i) Enums created the Postgres way by creating a type
+ *
+ * Here's my DB create flow:
+ *    1) become the account that controls pgsql (pgsql superuser)
+ *    2) from that shell prompt, say:
+ *          createuser -p shinobi
+ *       Enter a secure password after this, twice.
+ *    3) from same shell prompt say:
+ *          createdb --owner shinobi --encoding='utf-8' shinobi
+ *    4) now from same shell prompt you can do
+ *          psql shinobi <framework.psql
+ *    Your database is created.
+ *
+ * Extra issues:
+ *    You'll need to do 'npm install pg'
+ */
+
+CREATE OR REPLACE FUNCTION update_time_column()   
+RETURNS TRIGGER AS $$
+BEGIN
+    NEW.time = NOW();
+    RETURN NEW;   
+END;
+$$ language 'plpgsql';
+
+CREATE OR REPLACE FUNCTION upd_end_column()   
+RETURNS TRIGGER AS $$
+BEGIN
+    NEW."end" = NOW();
+    RETURN NEW;   
+END;
+$$ language 'plpgsql';
+
+CREATE TABLE IF NOT EXISTS "API" (
+  "ke" varchar(50) DEFAULT NULL,
+  "uid" varchar(50) DEFAULT NULL,
+  "ip" text,
+  "code" varchar(100) DEFAULT NULL,
+  "details" text,
+  "time" timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
+)  ;
+CREATE TRIGGER update_api_modtime 
+BEFORE UPDATE ON "API"
+FOR EACH ROW EXECUTE PROCEDURE update_time_column();
+
+CREATE TABLE IF NOT EXISTS "Cloud Videos" (
+  "mid" varchar(50) NOT NULL,
+  "ke" varchar(50) DEFAULT NULL,
+  "href" text NOT NULL,
+  "size" float DEFAULT NULL,
+  "time" timestamp NULL DEFAULT NULL,
+  "end" timestamp NULL DEFAULT NULL,
+  "status" int DEFAULT '0',
+  "details" text
+)  ;
+
+/* For status above: COMMENT '0:Complete,1:Read,2:Archive' */
+
+CREATE TABLE IF NOT EXISTS "Events" (
+  "ke" varchar(50) DEFAULT NULL,
+  "mid" varchar(50) DEFAULT NULL,
+  "details" text,
+  "time" timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
+)  ;
+
+CREATE TRIGGER update_events_modtime 
+BEFORE UPDATE ON "Events"
+FOR EACH ROW EXECUTE PROCEDURE update_time_column();
+
+CREATE TABLE IF NOT EXISTS "Logs" (
+  "ke" varchar(50) DEFAULT NULL,
+  "mid" varchar(50) DEFAULT NULL,
+  "info" text,
+  "time" timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP 
+)  ;
+CREATE TRIGGER update_logs_modtime 
+BEFORE UPDATE ON "Logs"
+FOR EACH ROW EXECUTE PROCEDURE update_time_column();
+
+CREATE TABLE IF NOT EXISTS "Monitors" (
+  "mid" varchar(50) DEFAULT NULL,
+  "ke" varchar(50) DEFAULT NULL,
+  "name" varchar(50) DEFAULT NULL,
+  "shto" text,
+  "shfr" text,
+  "details" text,
+  "type" varchar(50) DEFAULT 'jpeg',
+  "ext" varchar(50) DEFAULT 'webm',
+  "protocol" varchar(50) DEFAULT 'http',
+  "host" varchar(100) DEFAULT '0.0.0.0',
+  "path" varchar(100) DEFAULT '/',
+  "port" int DEFAULT '80',
+  "fps" int DEFAULT '1',
+  "mode" varchar(15) DEFAULT NULL,
+  "width" int DEFAULT '640',
+  "height" int DEFAULT '360'
+)  ;
+
+CREATE TYPE presettype AS ENUM('monitor','event','null','user');
+CREATE TABLE IF NOT EXISTS "Presets" (
+  "ke" varchar(50) DEFAULT NULL,
+  "name" text,
+  "details" text,
+  "type" presettype DEFAULT 'null'
+)  ;
+
+CREATE TABLE IF NOT EXISTS "Users" (
+  "ke" varchar(50) DEFAULT NULL,
+  "uid" varchar(50) DEFAULT NULL,
+  "auth" varchar(50) DEFAULT NULL,
+  "mail" varchar(100) DEFAULT NULL,
+  "pass" varchar(100) DEFAULT NULL,
+  "details" text,
+   UNIQUE ("mail")
+)  ;
+
+CREATE TYPE vidtype AS ENUM('webm','mp4','null');
+CREATE TABLE IF NOT EXISTS "Videos" (
+  "mid" varchar(50) DEFAULT NULL,
+  "ke" varchar(50) DEFAULT NULL,
+  "ext" vidtype DEFAULT NULL,
+  "time" timestamp NULL DEFAULT NULL,
+  "duration" float DEFAULT NULL,
+  "size" float DEFAULT NULL,
+  "frames" int DEFAULT NULL,
+  "end" timestamp NULL DEFAULT NULL,
+  "status" int DEFAULT '0',
+  "details" text
+)  ;
+/* For status above, COMMENT '0:Building,1:Complete,2:Read,3:Archive' */
+
+CREATE TABLE IF NOT EXISTS "Files" (
+    "ke" varchar(50) NOT NULL,
+    "mid" varchar(50) NOT NULL,
+    "name" text NOT NULL,
+    "size" float NOT NULL DEFAULT '0',
+    "details" text NOT NULL,
+    "status" int NOT NULL DEFAULT '0',
+    "time" timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
+)  ;
+CREATE TRIGGER update_files_modtime 
+BEFORE UPDATE ON "Files"
+FOR EACH ROW EXECUTE PROCEDURE update_time_column();
+
+CREATE TABLE IF NOT EXISTS "Schedules" (
+  "ke" varchar(50) DEFAULT NULL,
+  "name" text,
+  "details" text,
+  "start" varchar(10) DEFAULT NULL,
+  "end" varchar(10) DEFAULT NULL,
+  "enabled" int NOT NULL DEFAULT '1'
+)  ;
+
+CREATE TABLE IF NOT EXISTS "Timelapses" (
+  "ke" varchar(50) NOT NULL,
+  "mid" varchar(50) NOT NULL,
+  "details" text,
+  "date" date NOT NULL,
+  "time" timestamp NOT NULL,
+  "end" timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
+  "size" int NOT NULL
+)  ;
+CREATE TRIGGER update_timelapses_modtime 
+BEFORE UPDATE ON "Timelapses"
+FOR EACH ROW EXECUTE PROCEDURE upd_end_column();
+
+CREATE TABLE IF NOT EXISTS "Timelapse Frames" (
+  "ke" varchar(50) NOT NULL,
+  "mid" varchar(50) NOT NULL,
+  "details" text,
+  "filename" varchar(50) NOT NULL,
+  "time" timestamp NULL DEFAULT NULL,
+  "size" int NOT NULL
+)  ;
+
+CREATE TABLE IF NOT EXISTS "Cloud Timelapse Frames" (
+  "ke" varchar(50) NOT NULL,
+  "mid" varchar(50) NOT NULL,
+  "href" text NOT NULL,
+  "details" text,
+  "filename" varchar(50) NOT NULL,
+  "time" timestamp DEFAULT NULL,
+  "size" int NOT NULL
+)  ;
+
+CREATE TABLE IF NOT EXISTS "Events Counts" (
+  "ke" varchar(50) NOT NULL,
+  "mid" varchar(50) NOT NULL,
+  "details" text NOT NULL,
+  "end" timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
+  "count" int NOT NULL DEFAULT 1,
+  "tag" varchar(30) DEFAULT NULL
+)  ;
+CREATE TRIGGER update_events_counts_modtime 
+BEFORE UPDATE ON "Events Counts"
+FOR EACH ROW EXECUTE PROCEDURE upd_end_column();
+
+
+

sql/postgresql/framework.sql

@@ -1,104 +1,169 @@
--- --------------------------------------------------------
--- Host:                         192.168.88.37
--- Server version:               10.1.25-MariaDB- - Ubuntu 17.04
--- Server OS:                    debian-linux-gnu
--- HeidiSQL Version:             9.4.0.5125
--- --------------------------------------------------------
-
-/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
-/*!40101 SET NAMES utf8 */;
-/*!50503 SET NAMES utf8mb4 */;
-/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
-/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
-
-
--- Dumping database structure for ccio
-CREATE DATABASE `ccio` /*!40100 DEFAULT CHARACTER SET utf8mb4 */;
-USE ccio;
+-- Aug 5,2020 Rewrite of framework.sql for Postgres
+CREATE DATABASE "ccio";
+\c "ccio";
 
 -- Dumping structure for table ccio.API
-CREATE TABLE IF NOT EXISTS API (
-  ke varchar(50) DEFAULT NULL,
-  uid varchar(50) DEFAULT NULL,
-  ip tinytext,
-  code varchar(100) DEFAULT NULL,
-  details text,
-  time timestamp(0) NULL DEFAULT CURRENT_TIMESTAMP 
+CREATE TABLE IF NOT EXISTS "API" (
+  "ke" varchar(50) DEFAULT NULL,
+  "uid" varchar(50) DEFAULT NULL,
+  "ip" varchar(50),
+  "code" varchar(100) DEFAULT NULL,
+  "details" text,
+  "time" timestamp default current_timestamp
+) ;
+
+-- Data exporting was unselected.
+-- Dumping structure for table ccio.Cloud Videos
+CREATE TABLE IF NOT EXISTS "Cloud Videos" (
+  "mid" varchar(50) NOT NULL,
+  "ke" varchar(50) DEFAULT NULL,
+  "href" text NOT NULL,
+  "size" float DEFAULT NULL,
+  "time" timestamp NULL DEFAULT NULL,
+  "end" timestamp NULL DEFAULT NULL,
+  "status" integer DEFAULT 0,
+  "details" text
 ) ;
 
 -- Data exporting was unselected.
 -- Dumping structure for table ccio.Events
-CREATE TABLE IF NOT EXISTS Events (
-  ke varchar(50) DEFAULT NULL,
-  mid varchar(50) DEFAULT NULL,
-  details text,
-  time timestamp(0) NOT NULL DEFAULT CURRENT_TIMESTAMP 
-)  ;
+CREATE TABLE IF NOT EXISTS "Events" (
+  "ke" varchar(50) DEFAULT NULL,
+  "mid" varchar(50) DEFAULT NULL,
+  "details" text,
+  "time" timestamp default current_timestamp
+) ;
 
 -- Data exporting was unselected.
 -- Dumping structure for table ccio.Logs
-CREATE TABLE IF NOT EXISTS Logs (
-  ke varchar(50) DEFAULT NULL,
-  mid varchar(50) DEFAULT NULL,
-  info text,
-  time timestamp(0) NOT NULL DEFAULT CURRENT_TIMESTAMP 
+CREATE TABLE IF NOT EXISTS "Logs" (
+  "ke" varchar(50) DEFAULT NULL,
+  "mid" varchar(50) DEFAULT NULL,
+  "info" text,
+  "time" timestamp default current_timestamp
 ) ;
 
 -- Data exporting was unselected.
 -- Dumping structure for table ccio.Monitors
-CREATE TABLE IF NOT EXISTS Monitors (
-  mid varchar(50) DEFAULT NULL,
-  ke varchar(50) DEFAULT NULL,
-  name varchar(50) DEFAULT NULL,
-  shto text,
-  shfr text,
-  details longtext,
-  type varchar(50) DEFAULT 'jpeg',
-  ext varchar(50) DEFAULT 'webm',
-  protocol varchar(50) DEFAULT 'http',
-  host varchar(100) DEFAULT '0.0.0.0',
-  path varchar(100) DEFAULT '/',
-  port int DEFAULT '80',
-  fps int DEFAULT '1',
-  mode varchar(15) DEFAULT NULL,
-  width int DEFAULT '640',
-  height int DEFAULT '360'
+CREATE TABLE IF NOT EXISTS "Monitors" (
+  "mid" varchar(50) DEFAULT NULL,
+  "ke" varchar(50) DEFAULT NULL,
+  "name" varchar(50) DEFAULT NULL,
+  "shto" text,
+  "shfr" text,
+  "details" text,
+  "type" varchar(50) DEFAULT 'jpeg',
+  "ext" varchar(50) DEFAULT 'webm',
+  "protocol" varchar(50) DEFAULT 'http',
+  "host" varchar(100) DEFAULT '0.0.0.0',
+  "path" varchar(100) DEFAULT '/',
+  "port" integer DEFAULT '80',
+  "fps" integer DEFAULT '1',
+  "mode" varchar(15) DEFAULT NULL,
+  "width" integer DEFAULT '640',
+  "height" integer DEFAULT '360'
 ) ;
 
 -- Data exporting was unselected.
 -- Dumping structure for table ccio.Presets
-CREATE TABLE IF NOT EXISTS Presets (
-  ke varchar(50) DEFAULT NULL,
-  name text,
-  details text,
-  type enum('monitor','event','user') DEFAULT NULL
-) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+CREATE TABLE IF NOT EXISTS "Presets" (
+  "ke" varchar(50) DEFAULT NULL,
+  "name" text,
+  "details" text,
+  "type" varchar(10) DEFAULT NULL
+) ;
 
 -- Data exporting was unselected.
 -- Dumping structure for table ccio.Users
-CREATE TABLE IF NOT EXISTS Users (
-  ke varchar(50) DEFAULT NULL,
-  uid varchar(50) DEFAULT NULL,
-  auth varchar(50) DEFAULT NULL,
-  mail varchar(100) DEFAULT NULL,
-  pass varchar(100) DEFAULT NULL,
-  details longtext,
-  CONSTRAINT mail UNIQUE  (mail)
+CREATE TABLE IF NOT EXISTS "Users" (
+  "ke" varchar(50) DEFAULT NULL,
+  "uid" varchar(50) DEFAULT NULL,
+  "auth" varchar(50) DEFAULT NULL,
+  "mail" varchar(100) UNIQUE,
+  "pass" varchar(100) DEFAULT NULL,
+  "accountType" integer DEFAULT 0,
+  "details" text
 ) ;
 
 -- Data exporting was unselected.
 -- Dumping structure for table ccio.Videos
-CREATE TABLE IF NOT EXISTS Videos (
-  mid varchar(50) DEFAULT NULL,
-  ke varchar(50) DEFAULT NULL,
-  ext enum('webm','mp4') DEFAULT NULL,
-  time timestamp(0) NULL DEFAULT NULL,
-  duration double precision DEFAULT NULL,
-  size double precision DEFAULT NULL,
-  frames int DEFAULT NULL,
-  end timestamp(0) NULL DEFAULT NULL,
-  status int DEFAULT '0' ,
-  details text
+CREATE TABLE IF NOT EXISTS "Videos" (
+  "mid" varchar(50) DEFAULT NULL,
+  "ke" varchar(50) DEFAULT NULL,
+  "ext" varchar(5) DEFAULT NULL,
+  "time" timestamp NULL DEFAULT NULL,
+  "duration" float DEFAULT NULL,
+  "size" float DEFAULT NULL,
+  "frames" integer DEFAULT NULL,
+  "end" timestamp NULL DEFAULT NULL,
+  "status" integer DEFAULT 0,
+  "archived" integer DEFAULT 0,
+  "details" text
+) ;
+
+-- Data exporting was unselected.
+-- Dumping structure for table ccio.Files
+CREATE TABLE IF NOT EXISTS "Files" (
+    "ke" varchar(50) NOT NULL,
+    "mid" varchar(50) NOT NULL,
+    "name" varchar(50) NOT NULL,
+    "size" float NOT NULL DEFAULT 0,
+    "details" text NOT NULL,
+    "status" integer NOT NULL DEFAULT 0,
+    "time" timestamp default current_timestamp
+) ;
+
+-- Data exporting was unselected.
+-- Dumping structure for table ccio.Schedules
+CREATE TABLE IF NOT EXISTS "Schedules" (
+  "ke" varchar(50) DEFAULT NULL,
+  "name" text,
+  "details" text,
+  "start" varchar(10) DEFAULT NULL,
+  "end" varchar(10) DEFAULT NULL,
+  "enabled" integer NOT NULL DEFAULT 1
+) ;
+
+-- Dumping structure for table ccio.Timelapses
+CREATE TABLE IF NOT EXISTS "Timelapses" (
+  "ke" varchar(50) NOT NULL,
+  "mid" varchar(50) NOT NULL,
+  "details" text,
+  "date" date NOT NULL,
+  "time" timestamp NOT NULL,
+  "end" timestamp default current_timestamp,
+  "size" integer NOT NULL
+) ;
+
+-- Dumping structure for table ccio.Timelapse Frames
+CREATE TABLE IF NOT EXISTS "Timelapse Frames" (
+  "ke" varchar(50) NOT NULL,
+  "mid" varchar(50) NOT NULL,
+  "details" text,
+  "filename" varchar(50) NOT NULL,
+  "time" timestamp NULL DEFAULT NULL,
+  "size" integer NOT NULL
+) ;
+-- Dumping structure for table ccio.Timelapse Frames
+CREATE TABLE IF NOT EXISTS "Cloud Timelapse Frames" (
+    "ke" varchar(50) NOT NULL,
+    "mid" varchar(50) NOT NULL,
+    "href" text NOT NULL,
+    "details" text,
+    "filename" varchar(50) NOT NULL,
+    "time" timestamp NULL DEFAULT NULL,
+    "size" integer NOT NULL
+) ;
+
+-- Dumping structure for table ccio.Events Counts
+CREATE TABLE IF NOT EXISTS "Events Counts" (
+  "ke" varchar(50) NOT NULL,
+  "mid" varchar(50) NOT NULL,
+  "details" text NOT NULL,
+  "time" timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
+  "end" timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
+  "count" integer NOT NULL DEFAULT 1,
+  "tag" varchar(30) DEFAULT NULL
 ) ;
 
 -- Data exporting was unselected.

tools/modifyConfiguration.js

@@ -4,8 +4,7 @@ process.on('uncaughtException', function (err) {
 });
 var configLocation = __dirname+'/../conf.json';
 var fs = require('fs');
-var jsonfile = require("jsonfile");
-var config = jsonfile.readFileSync(configLocation);
+var config = JSON.parse(fs.readFileSync(configLocation));
 var processArgv = process.argv.splice(2,process.argv.length)
 var arguments = {};
 
@@ -57,7 +56,20 @@ processArgv.forEach(function(val) {
     console.log(index + ': ' + value);
 });
 
-jsonfile.writeFile(configLocation,config,{spaces: 2},function(){
+try{
+    if(config.thisIsDocker){
+        const dockerConfigFile = '/config/conf.json'
+        fs.stat(dockerConfigFile,(err) => {
+            if(!err){
+                fs.writeFile(dockerConfigFile,JSON.stringify(config,null,3),function(){})
+            }
+        })
+    }
+}catch(err){
+    console.log(err)
+}
+
+fs.writeFile(configLocation,JSON.stringify(config,null,3),function(){
     console.log('Changes Complete. Here is what it is now.')
     console.log(JSON.stringify(config,null,2))
 })

tools/modifyConfigurationForPlugin.js

@@ -1,81 +1,129 @@
-var fs = require('fs');
-var jsonfile = require("jsonfile");
-var execSync = require('child_process').execSync;
-var anError = function(message,dontShowExample){
-    console.log(message)
-    if(!dontShowExample){
-        console.log('Example of usage :')
-        console.log('node tool/modifyConfigurationForPlugin.js tensorflow key=1234asdfg port=8080')
-    }
-}
-var testValueForObject = function(jsonString){
-    var newValue = jsonString + ''
-    try{
-        newValue = JSON.parse(jsonString)
-    }catch(err){
-
-    }
-    if(typeof newValue === 'object'){
-        return true
-    }
-    return false
-}
-process.on('uncaughtException', function (err) {
-    console.error('Uncaught Exception occured!');
-    console.error(err.stack);
-});
-var targetedPlugin = process.argv[2]
-if(!targetedPlugin || targetedPlugin === '' || targetedPlugin.indexOf('=') > -1){
-    return anError('Specify a plugin folder name as the first argument.')
-}
-var pluginLocation = __dirname + `/../plugins/${targetedPlugin}/`
-fs.stat(pluginLocation,function(err){
-    if(!err){
-        var configLocation = `${pluginLocation}conf.json`
-        var config = jsonfile.readFileSync(configLocation);
-        var processArgv = process.argv.splice(3,process.argv.length)
-        var arguments = {};
-        if(processArgv.length === 0){
-            return anError('No changes made. Add arguments to add or modify.')
-        }
-        processArgv.forEach(function(val) {
-            var theSplit = val.split('=');
-            var index = (theSplit[0] || '').trim();
-            var value = theSplit[1];
-            if(index.indexOf('addToConfig') > -1 || index == 'addToConfig'){
-                try{
-                    value = JSON.parse(value)
-                    config = Object.assign(config,value)
-                }catch(err){
-                    anError('Not a valid Data set. "addToConfig" value must be a JSON string. You may need to wrap it in singles quotes.')
-                }
-            }else{
-                if(value==='DELETE'){
-                    delete(config[index])
-                }else{
-                    if(testValueForObject(value)){
-                        config[index] = JSON.parse(value);
-                    }else{
-                        if(index === 'key'){
-                            console.log(`Modifying main conf.json with updated key.`)
-                            execSync(`node ${__dirname}/modifyConfiguration.js addToConfig='{"pluginKeys":{"${config.plug}":"${value + ''}"}}'`,function(err){
-                                console.log(err)
-                            })
-                            config[index] = value + ''
-                        }else{
-                            config[index] = value
-                        }
-                    }
-                }
-            }
-            console.log(index + ': ' + value);
-        });
-
-        jsonfile.writeFile(configLocation,config,{spaces: 2},function(){
-            console.log('Changes Complete. Here is what it is now.')
-            console.log(JSON.stringify(config,null,2))
-        })
-    }else{
-        anError(`Plugin "${targetedPlugin}" not found.`)
-    }
-})
+var fs = require('fs');
+var execSync = require('child_process').execSync;
+const getConfLocation = () => {
+    let chosenLocation
+    try{
+        chosenLocation = __dirname + `/../plugins/${targetedPlugin}/`
+        fs.statSync(chosenLocation)
+    }catch(err){
+        chosenLocation = __dirname + `/`
+    }
+    return chosenLocation
+}
+const mergeDeep = function(...objects) {
+  const isObject = obj => obj && typeof obj === 'object';
+
+  return objects.reduce((prev, obj) => {
+    Object.keys(obj).forEach(key => {
+      const pVal = prev[key];
+      const oVal = obj[key];
+
+      if (Array.isArray(pVal) && Array.isArray(oVal)) {
+        prev[key] = pVal.concat(...oVal);
+      }
+      else if (isObject(pVal) && isObject(oVal)) {
+        prev[key] = mergeDeep(pVal, oVal);
+      }
+      else {
+        prev[key] = oVal;
+      }
+    });
+
+    return prev;
+  }, {});
+}
+var anError = function(message,dontShowExample){
+    console.log(message)
+    if(!dontShowExample){
+        console.log('Example of usage :')
+        console.log('node tools/modifyConfigurationForPlugin.js tensorflow key=1234asdfg port=8080')
+    }
+}
+var testValueForObject = function(jsonString){
+    var newValue = jsonString + ''
+    try{
+        newValue = JSON.parse(jsonString)
+    }catch(err){
+
+    }
+    if(typeof newValue === 'object'){
+        return true
+    }
+    return false
+}
+process.on('uncaughtException', function (err) {
+    console.error('Uncaught Exception occured!');
+    console.error(err.stack);
+});
+var targetedPlugin = process.argv[2]
+if(!targetedPlugin || targetedPlugin === '' || targetedPlugin.indexOf('=') > -1){
+    return anError('Specify a plugin folder name as the first argument.')
+}
+var pluginLocation = getConfLocation()
+fs.stat(pluginLocation,function(err){
+    if(!err){
+        var configLocation = `${pluginLocation}conf.json`
+        try{
+            var config = JSON.parse(fs.readFileSync(configLocation))
+        }catch(err){
+            try{
+                var config = fs.readFileSync(`${pluginLocation}conf.sample.json`,'utf8')
+                fs.writeFileSync(`${pluginLocation}conf.json`,JSON.stringify(config,null,3),'utf8')
+            }catch(err){
+                var config = {}
+            }
+        }
+        var processArgv = process.argv.splice(3,process.argv.length)
+        var arguments = {};
+        if(processArgv.length === 0){
+            return anError('No changes made. Add arguments to add or modify.')
+        }
+        processArgv.forEach(function(val) {
+            var theSplit = val.split('=');
+            var index = (theSplit[0] || '').trim();
+            var value = theSplit[1];
+            if(index.indexOf('addToConfig') > -1 || index == 'addToConfig'){
+                try{
+                    value = JSON.parse(value)
+                    config = mergeDeep(config,value)
+                }catch(err){
+                    anError('Not a valid Data set. "addToConfig" value must be a JSON string. You may need to wrap it in singles quotes.')
+                }
+            }else{
+                if(value==='DELETE'){
+                    delete(config[index])
+                }else{
+                    if(testValueForObject(value)){
+                        config[index] = JSON.parse(value);
+                    }else{
+                        if(index === 'key'){
+                            const modifyMainFileLocation = `${__dirname}/modifyConfiguration.js`
+                            fs.stat(modifyMainFileLocation,(err) => {
+                                if(!err){
+                                    console.log(`Updating main conf.json with new key.`)
+                                    execSync(`node ${modifyMainFileLocation} addToConfig='{"pluginKeys":{"${config.plug}":"${value + ''}"}}'`,function(err){
+                                        console.log(err)
+                                    })
+                                }else{
+                                    console.log(`Didn't find main conf.json. You may need to update it manually.`)
+                                    console.log(`Docker users using the official Ninja-Docker install method don't need to complete any other configuration.`)
+                                }
+                            })
+                            config[index] = value + ''
+                        }else{
+                            config[index] = value
+                        }
+                    }
+                }
+            }
+            console.log(index + ': ' + value);
+        });
+
+        fs.writeFile(configLocation,JSON.stringify(config,null,3),function(){
+            console.log('Changes Complete. Here is what it is now.')
+            console.log(JSON.stringify(config,null,2))
+        })
+    }else{
+        anError(`Plugin "${targetedPlugin}" not found.`)
+    }
+})

web/libs/css/dash2.basic.css

@@ -182,9 +182,6 @@ img{max-width:100%}
 .dot-orange {background:#c49a68}
 .dot-grey {background:#777}
 
-
-
-
 .os_bars{width:600px;display:inline-block;padding:5px 0 0 10px}
 @media screen and (max-width: 600px){
     .os_bars{width:200px;}
@@ -489,3 +486,66 @@ ul.msg_list li .message {
     margin-left: 10px;
 }
 /* End of custom table sorter */
+
+.row-flex {
+    display: flex;
+}
+
+.row-flex-full-height {
+    display: flex;
+    height: 100%;
+}
+
+.row-flex [class*="col-"]{
+    height: 100%;
+    float: none;
+    overflow: auto;
+}
+
+.row-flex .col-md-1{
+    flex: 1;
+}
+
+.row-flex .col-md-2{
+    flex: 2;
+}
+
+.row-flex .col-md-3{
+    flex: 3;
+}
+
+.row-flex .col-md-4{
+    flex: 4;
+}
+
+.row-flex .col-md-5{
+    flex: 5;
+}
+
+.row-flex .col-md-6{
+    flex: 6;
+}
+
+.row-flex .col-md-7{
+    flex: 7;
+}
+
+.row-flex .col-md-8{
+    flex: 8;
+}
+
+.row-flex .col-md-9{
+    flex: 9;
+}
+
+.row-flex .col-md-10{
+    flex: 10;
+}
+
+.row-flex .col-md-11{
+    flex: 11;
+}
+
+.row-flex .col-md-12{
+    flex: 12;
+}

web/libs/css/dash2.monitors.css

@@ -101,3 +101,23 @@ img.circle-img,div.circle-img{border-radius:50%;height:50px;width:50px}
 .stream-objects .stream-detected-object{position:absolute;top:0;left:0;border:3px solid red;background:transparent;border-radius:5px}
 .stream-objects .stream-detected-point{position:absolute;top:0;left:0;border:3px solid yellow;background:transparent;border-radius:5px}
 .stream-objects .point{position:absolute;top:0;left:0;border:3px solid red;border-radius:50%}
+
+.monitor_item .gps-map {
+    position: absolute;
+    width: 190px;
+    height: 190px;
+    border-radius: 50%;
+    border: 1px solid #333;
+    z-index: 9;
+    bottom: 10px;
+    right: 10px;
+}
+.monitor_item .gps-map-details {
+    position: absolute;
+    padding: 5px 7px;
+    border-radius: 25px;
+    background:rgba(0,0,0,0.5);
+    z-index: 11;
+    top: 10px;
+    right: 10px;
+}

web/libs/css/super.customAutoLoad.css

@@ -0,0 +1,12 @@
+#customAutoLoadList [package-name] .card-body{
+    min-height:auto
+}
+#customAutoLoadList [package-name] .install-output-stdout,
+#customAutoLoadList [package-name] .install-output-stderr
+{
+    max-height: 300px;
+    background: ##f7f7f7;
+    border-radius: 15px;
+    padding: 5px;
+    margin:0;
+}

web/libs/js/dash2.config.js

@@ -15,6 +15,7 @@ $.ccio.HWAccelChoices = [
             auto: {label:lang['Auto'],value:'auto'},
             drm: {label:lang['drm'],value:'drm'},
             cuvid: {label:lang['cuvid'],value:'cuvid'},
+            cuda: {label:lang['cuda'],value:'cuda'},
             vaapi: {label:lang['vaapi'],value:'vaapi'},
             qsv: {label:lang['qsv'],value:'qsv'},
             vdpau: {label:lang['vdpau'],value:'vdpau'},
@@ -68,8 +69,9 @@ window.mergeDeep = function(target, ...sources){
 
   return mergeDeep(target, ...sources);
 }
-window.getApiPrefix = function(){
-    return $.ccio.init('location',$user) + $user.auth_token
+window.getApiPrefix = function(path){
+    var mainPart = $.ccio.init('location',$user) + $user.auth_token
+    return path ? mainPart + '/' + path + '/' + $user.ke : mainPart
 }
 window.chartColors = {
     red: 'rgb(255, 99, 132)',

web/libs/js/dash2.elementbuilder.js

@@ -101,6 +101,12 @@ $.ccio.tm=function(x,d,z,user){
             var dataTarget = '.monitor_item[mid=\''+d.mid+'\'][ke=\''+d.ke+'\'][auth=\''+user.auth_token+'\']';
             tmp+='<div id="monitor_live_'+d.mid+user.auth_token+'" auth="'+user.auth_token+'" mid="'+d.mid+'" ke="'+d.ke+'" mode="'+k.mode+'" class="grid-stack-item monitor_item glM'+d.mid+user.auth_token+'"><div class="grid-stack-item-content">';
             tmp+='<div class="stream-block no-padding mdl-card__media mdl-color-text--grey-50">';
+                tmp+=`<div class="gps-map-info gps-map-details hidden">
+                        <div><i class="fa fa-compass fa-3x gps-info-bearing"></i></div>
+                        <div><i class="fa fa-compass fa-3x gps-info-speed"></i></div>
+                    <div></div>
+                </div>
+                <div class="gps-map gps-map-info hidden" id="gps-map-${d.mid}"></div>`;
             tmp+='<div class="stream-objects"></div>';
             tmp+='<div class="stream-hud">'
             tmp+='<div class="camera_cpu_usage"><div class="progress"><div class="progress-bar progress-bar-danger" role="progressbar"><span></span></div></div></div>';

web/libs/js/dash2.elements.js

@@ -157,12 +157,12 @@ $(document).ready(function(e){
                 e.preventDefault();
                 e.href=$(this).attr('href')
                 var el = $('#video_viewer')
-                var modalBody = el.find('.modal-body')
+                var videoContainer = el.find('.video-container')
                 el.find('.modal-title span').html(e.mon.name+' - '+e.file)
                 var html = '<video class="video_video" video="'+e.href+'" autoplay loop controls><source src="'+e.href+'" type="video/'+e.mon.ext+'"></video><br><small class="msg"></small>'
-                modalBody.html(html)
+                videoContainer.html(html)
                 el.find('video')[0].onerror = function(){
-                    modalBody.find('.msg').text(lang.h265BrowserText1)
+                    videoContainer.find('.msg').text(lang.h265BrowserText1)
                 }
                 el.attr('mid',e.mid);
                 footer = el.find('.modal-footer');
@@ -178,6 +178,28 @@ $(document).ready(function(e){
                         if(d.ok !== true)console.log(d,new Error())
                     })
                 }
+                setTimeout(function(){
+                    destroyGpsHandlerForVideoFile(`video_viewer_gps_map_map`)
+                    var videoElement = videoContainer.find('.video_video')
+                    var gpsContainer = videoContainer.next()
+                    var fullWidth = 'col-md-12'
+                    var partialWidth = 'col-md-8'
+                    createGpsHandlerForVideoFile({
+                        ke: e.ke,
+                        mid: e.mid,
+                        file: e.file,
+                        targetVideoElement: videoElement,
+                        targetMapElement: `video_viewer_gps_map`,
+                    },function(response){
+                        if(response.ok){
+                            videoContainer.addClass(partialWidth).removeClass(fullWidth)
+                            gpsContainer.show()
+                        }else{
+                            videoContainer.addClass(fullWidth).removeClass(partialWidth)
+                            gpsContainer.hide()
+                        }
+                    })
+                },2000)
             break;
             case'delete':
                 e.preventDefault();
@@ -186,7 +208,7 @@ $(document).ready(function(e){
                 console.log('videoLink',videoLink)
                 console.log(href)
                 if(!href){
-                    href = $.ccio.init('location',$.users[e.auth])+e.auth+'/videos/'+e.ke+'/'+e.mid+'/'+e.file+'/delete<% if(config.useUTC === true){%>?isUTC=true<%}%>'
+                    href = $.ccio.init('location',$.users[e.auth])+e.auth+'/videos/'+e.ke+'/'+e.mid+'/'+e.file+'/delete'
                 }
                 console.log(href)
                 $.confirm.e.modal('show');
@@ -412,7 +434,7 @@ $(document).ready(function(e){
                 }
             break;
             case'trigger-event':
-                $.getJSON(getApiPrefix() + '/motion/'+e.ke+'/'+e.mid+'/?data={"plug":"camera1","name":"stairs","reason":"motion","confidence":197.4755859375}',function(d){
+                $.getJSON(getApiPrefix() + '/motion/'+e.ke+'/'+e.mid+'/?data={"plug":"camera1","name":"stairs","reason":"motion","confidence":100}',function(d){
                     $.ccio.log(d)
                 })
             break;
@@ -766,8 +788,10 @@ $(document).ready(function(e){
     })
 
     $('.modal').on('hidden.bs.modal',function(){
-        $(this).find('video').remove();
-        $(this).find('iframe').attr('src','about:blank');
+        var el = $(this)
+        el.find('video').remove();
+        el.find('iframe').attr('src','about:blank');
+        if(el.attr('id') === 'video_viewer')destroyGpsHandlerForVideoFile(`video_viewer_gps_map_map`)
     });
     $('.modal').on('shown.bs.modal',function(){
         e={e:$(this).find('.flex-container-modal-body')}

web/libs/js/dash2.gps.js

@@ -0,0 +1,71 @@
+$(document).ready(function(){
+    window.setRadialBearing = function(iconElement,addedDegrees,titlePrefix){
+        //fa-compass
+        var degrees = -35;
+        degrees += addedDegrees
+        iconElement.css('transform','rotate(' + degrees + 'deg)').attr('title',titlePrefix + addedDegrees)
+    }
+    var createMapElement = function(options){
+        $(`#${options.id}`).html(`<div class="gps-map" id="${options.id}_map" style="min-height: 600px;border-radius:5px;overflow:hidden;"></div>`)
+        var vidViewMap = L.map(options.id + '_map').setView(options.initalView, options.zoom)
+        var mapBoxMarker;
+        L.tileLayer('https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png', {
+            attribution: 'OpenStreet Map'
+        }).addTo(vidViewMap)
+
+        if(options.marker)mapBoxMarker = L.marker(options.marker).addTo(vidViewMap)
+        return {
+            map: vidViewMap,
+            marker: mapBoxMarker,
+        }
+    }
+    window.destroyGpsHandlerForVideoFile = function(mapId){
+        var vidViewMap = $(`#${mapId}`)
+        if (vidViewMap.length > 0) {
+            try{
+                vidViewMap.off();
+                vidViewMap.remove();
+            }catch(err){
+                console.log(err)
+            }
+        }
+    }
+    window.createGpsHandlerForVideoFile = function(options,callback){
+        var groupKey = options.ke
+        var monitorId = options.mid
+        var filename = options.file
+        var videoElement = options.targetVideoElement
+        $.get(getApiPrefix() + '/videos/' + groupKey + '/' + monitorId + '/' + filename + '?json=true',function(video){
+            var response = {ok: false}
+            var gps = video.details.gps
+            if(gps && gps[0]){
+                var gpsPoints = {}
+                var firstMarker = gps[0]
+
+                var videoStartTime = new Date(video.time)
+                $.each(gps,function(n,point){
+                    var pointTime = new Date(point.time)
+                    var seekPosition = (pointTime - videoStartTime) / 1000
+                    gpsPoints[pointTime] = point
+                })
+                response.ok = true
+                response.gpsPoints = gpsPoints
+                callback(response)
+                response.elements = createMapElement({
+                    id: options.targetMapElement,
+                    initalView: [firstMarker.lat,firstMarker.lng],
+                    marker: [firstMarker.lat,firstMarker.lng],
+                    zoom: 2,
+                })
+                videoElement.on('timeupdate',function(){
+                    var point = gpsPoints[parseInt(this.currentTime)]
+                    if(point){
+                        mapBoxMarker.setLatLng([point.lat, point.lng]).update()
+                    }
+                })
+            }else{
+                callback(response)
+            }
+        })
+    }
+})

web/libs/js/dash2.socketio.js

@@ -129,7 +129,7 @@ $.ccio.globalWebsocket=function(d,user){
             $.ccio.pm(3,d.apis,null,user);
             $('.os_platform').html(d.os.platform)
             $('.os_cpuCount').html(d.os.cpuCount)
-            $('.os_totalmem').html((d.os.totalmem/1048576).toFixed(2))
+            $('.os_totalmem').attr('title',`Total : ${(d.os.totalmem/1048576).toFixed(2)}`)
             if(d.os.cpuCount>1){
                 $('.os_cpuCount_trailer').html('s')
             }
@@ -148,13 +148,14 @@ $.ccio.globalWebsocket=function(d,user){
         break;
         case'os'://indicator
             //cpu
-            d.cpu=parseFloat(d.cpu).toFixed(0)+'%';
-            $('.cpu_load .progress-bar').css('width',d.cpu);
-            $('.cpu_load .percent').html(d.cpu);
+            var cpuPercent = parseFloat(d.cpu).toFixed(1) + '%'
+            $('.cpu_load .progress-bar').css('width',cpuPercent)
+            $('.cpu_load .percent').html(cpuPercent)
             //ram
-            d.ram=(100-parseFloat(d.ram)).toFixed(0)+'%';
-            $('.ram_load .progress-bar').css('width',d.ram);
-            $('.ram_load .percent').html(d.ram);
+            var ramPercent = d.ram.percent.toFixed(1) + '%'
+            $('.ram_load .progress-bar').css('width',ramPercent)
+            $('.ram_load .percent').html(ramPercent)
+            $('.ram_load .used').html(d.ram.used.toFixed(2))
         break;
         case'diskUsed':
             if(!d.limit||d.limit===''){d.limit=10000}
@@ -656,6 +657,37 @@ $.ccio.globalWebsocket=function(d,user){
             $.ccio.init('jpegModeAll');
             $('body').addClass('jpegMode')
         break;
+        case'gps':
+            var gps = d.gps
+            var activeMonitor = $.ccio.mon[user.ke + d.mid + user.auth_token]
+            var mapBoxMarker = activeMonitor.mapBoxMarker
+            var monitorElement = $(`.monitor_item[mid="${d.mid}"]`)
+            monitorElement.find(`.gps-map-info`).removeClass('hidden')
+            if(!mapBoxMarker){
+                var mapBox = L.map(`gps-map-${d.mid}`).setView([gps.lat, gps.lng], 5);
+
+                L.tileLayer('https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png', {
+                    attribution: 'OpenStreet Map'
+                }).addTo(mapBox);
+
+                var mapBoxMarker = L.marker([gps.lat, gps.lng]).addTo(mapBox);
+                activeMonitor.mapBoxMarker = mapBoxMarker
+                activeMonitor.mapBoxBearingElement = monitorElement.find(`.gps-info-bearing`)
+                activeMonitor.mapBoxSpeedElement = monitorElement.find(`.gps-info-speed`)
+            }else{
+                mapBoxMarker.setLatLng([gps.lat, gps.lng]).update()
+            }
+            if(gps.bearing){
+                setRadialBearing(activeMonitor.mapBoxBearingElement,gps.bearing,'Bearing : ')
+            }
+            if(gps.speed){
+                setRadialBearing(activeMonitor.mapBoxSpeedElement,gps.speed,'Speed (meters/second) : ')
+            }
+            clearTimeout(activeMonitor.mapBoxTimeout)
+            activeMonitor.mapBoxTimeout = setTimeout(function(){
+                monitorElement.find(`.gps-map-info`).addClass('hidden')
+            },30000)
+        break;
     }
 }
 if(location.search === '?assemble=1'){

web/libs/js/poseidon.js

@@ -461,11 +461,15 @@ var Poseidon = function () {
             //this._video.setAttribute('poster', 'data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iNjQwIiBoZWlnaHQ9IjM0IiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjxnPjxyZWN0IHg9Ii0xIiB5PSItMSIgd2lkdGg9IjY0MiIgaGVpZ2h0PSIzNiIgZmlsbD0ibm9uZSIvPjwvZz48Zz48dGV4dCBmaWxsPSIjMDAwIiBzdHJva2Utd2lkdGg9IjAiIHg9IjE5NiIgeT0iMjYiIGZvbnQtc2l6ZT0iMjYiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2EsIEFyaWFsLCBzYW5zLXNlcmlmIiB0ZXh0LWFuY2hvcj0ic3RhcnQiIHhtbDpzcGFjZT0icHJlc2VydmUiIHN0cm9rZT0iIzAwMCI+cmVxdWVzdGluZyBtaW1lIHR5cGU8L3RleHQ+PC9nPjwvc3ZnPg==');
             this.onMime = this._onMime.bind(this);
             this._socket.addEventListener('mime', this.onMime, { capture: true, passive: true, once: true });
-            this._socket.emit('MP4', this._monitor);
             this.Commander = function (cmd) {
                 this._socket.emit('MP4Command', cmd);
             };
-            this.Commander('mime');
+            let _this = this
+            this._socket.emit('MP4', this._monitor, function(err, res) {
+                if (err) _this._callback('socket error "' + err + '"')
+                else if (res === true) _this.Commander('mime');
+                else _this._callback('socket error "' + res + '"')
+            });
         }
     }, {
         key: '_onSocketDisconnect',

web/libs/js/super.customAutoLoad.js

@@ -0,0 +1,175 @@
+$(document).ready(function(){
+    var loadedModules = {}
+    var listElement = $('#customAutoLoadList')
+    var getModules = function(callback) {
+        $.get(superApiPrefix + $user.sessionKey + '/package/list',callback)
+    }
+    var loadedBlocks = {}
+    var drawModuleBlock = function(module){
+        var humanName = module.properties.name ? module.properties.name : module.name
+        if(listElement.find('[package-name="${module.name}"]').length > 0){
+            var existingElement = listElement.find('[package-name="${module.name}"]')
+            existingElement.find('.title').text(humanName)
+            existingElement.find('[calm-action="status"]').text(module.properties.disabled ? lang.Enable : lang.Disable)
+        }else{
+            listElement.append(`
+                <div class="col-md-12">
+                    <div class="card" package-name="${module.name}">
+                        <div class="card-body">
+                            <div><h4 class="title mt-0">${humanName}</h4></div>
+                            <div><pre><b>${lang['Time Created']} :</b> ${module.created}</pre></div>
+                            <div><pre><b>${lang['Last Modified']} :</b> ${module.lastModified}</pre></div>
+                            <div class="mb-2">
+                                ${!module.isIgnitor ? `
+                                    ${module.hasInstaller ? `
+                                        <a href="#" class="btn btn-sm btn-info" calm-action="install">${lang['Run Installer']}</a>
+                                    ` : ''}
+                                    <a href="#" class="btn btn-sm btn-default" calm-action="status">${module.properties.disabled ? lang.Enable : lang.Disable}</a>
+                                ` : ''}
+                                <a href="#" class="btn btn-sm btn-danger" calm-action="delete">${lang.Delete}</a>
+                            </div>
+                            <div class="pl-2 pr-2">
+                                <div class="install-output row">
+                                    <div class="col-md-6 pr-2"><pre class="install-output-stdout"></pre></div>
+                                    <div class="col-md-6 pl-2"><pre class="install-output-stderr"></pre></div>
+                                </div>
+                            </div>
+                        </div>
+                    </div>
+                </div>`)
+            var newBlock = $(`.card[package-name="${module.name}"]`)
+            loadedBlocks[module.name] = {
+                block: newBlock,
+                stdout: newBlock.find('.install-output-stdout'),
+                stderr: newBlock.find('.install-output-stderr'),
+            }
+        }
+    }
+    var downloadModule = function(url,packageRoot,callback){
+        $.confirm.create({
+            title: 'Module Download',
+            body: `Do you want to download the module from ${url}? `,
+            clickOptions: {
+                class: 'btn-success',
+                title: lang.Download,
+            },
+            clickCallback: function(){
+                $.post(superApiPrefix + $user.sessionKey + '/package/download',{
+                    downloadUrl: url,
+                    packageRoot: packageRoot,
+                },callback)
+            }
+        })
+    }
+    var installModule = function(packageName,callback){
+        $.confirm.create({
+            title: 'Install Module',
+            body: `Do you want to install the module ${packageName}?`,
+            clickOptions: {
+                class: 'btn-success',
+                title: lang.Install,
+            },
+            clickCallback: function(){
+                $.post(superApiPrefix + $user.sessionKey + '/package/install',{
+                    packageName: packageName,
+                },callback)
+            }
+        })
+    }
+    var deleteModule = function(packageName,callback){
+        $.confirm.create({
+            title: 'Delete Module',
+            body: `Do you want to delete the module ${packageName}?`,
+            clickOptions: {
+                class: 'btn-danger',
+                title: lang.Delete,
+            },
+            clickCallback: function(){
+                $.post(superApiPrefix + $user.sessionKey + '/package/delete',{
+                    packageName: packageName,
+                },callback)
+            }
+        })
+    }
+    var setModuleStatus = function(packageName,status,callback){
+        $.post(superApiPrefix + $user.sessionKey + '/package/status',{
+            status: status,
+            packageName: packageName,
+        },callback)
+    }
+    $('body').on(`click`,`[calm-action]`,function(e){
+        e.preventDefault()
+        var el = $(this)
+        var action = el.attr('calm-action')
+        var card = el.parents('[package-name]')
+        console.log(card.length)
+        var packageName = card.attr('package-name')
+        switch(action){
+            case'install':
+                installModule(packageName,function(data){
+                    if(data.ok){
+                        console.log(data)
+                    }
+                })
+            break;
+            case'status':
+                setModuleStatus(packageName,!!!loadedModules[packageName].properties.disabled,function(data){
+                    if(data.ok){
+                        loadedModules[packageName].properties.disabled = !!!loadedModules[packageName].properties.disabled
+                        el.text(loadedModules[packageName].properties.disabled ? lang.Enable : lang.Disable)
+                    }
+                })
+            break;
+            case'delete':
+                deleteModule(packageName,function(data){
+                    if(data.ok){
+                        card.remove()
+                    }
+                })
+            break;
+        }
+    })
+    $('#downloadNewModule').submit(function(e){
+        e.preventDefault();
+        var el = $(this)
+        var form = el.serializeObject()
+        downloadModule(form.downloadUrl,form.packageRoot,function(data){
+            console.log(data)
+            if(data.ok){
+                data.newModule.properties.disabled = true
+                drawModuleBlock(data.newModule)
+            }
+        })
+        return false
+    })
+    setTimeout(function(){
+        getModules(function(data){
+            loadedModules = data.modules
+            console.log(loadedModules)
+            $.each(data.modules,function(n,module){
+                drawModuleBlock(module)
+            })
+        })
+    },2000)
+    $.ccio.ws.on('f',function(data){
+        switch(data.f){
+            case'module-info':
+                var name = data.module
+                switch(data.process){
+                    case'install-stdout':
+                        loadedBlocks[name].stdout.append(`<div class="line">${data.data}</div>`)
+                        // if(loadedBlocks[name].stdout.find('.line').length > 10){
+                        //     loadedBlocks[name].stdout.children().first().remove()
+                        // }
+                    break;
+                    case'install-stderr':
+                        loadedBlocks[name].stderr.append(`<div class="line">${data.data}</div>`)
+                        // if(loadedBlocks[name].stderr.find('.line').length > 10){
+                        //     loadedBlocks[name].stderr.children().first().remove()
+                        // }
+                    break;
+                }
+            break;
+        }
+    })
+})

web/libs/themes/Blazing Blue/style.css

@@ -78,4 +78,7 @@
     background: -o-linear-gradient(90deg, rgba(227, 227, 227, 0.26), rgba(249, 99, 50, 0.95));
     background: -moz-linear-gradient(90deg, rgba(227, 227, 227, 0.26), rgba(249, 99, 50, 0.95));
     background: linear-gradient(0deg, rgba(227, 227, 227, 0.26), rgba(249, 99, 50, 0.95));
-}
+}
+
+[status="1"] .btn[video="launch"],[data-status="1"] .btn[video="launch"]{background:#337ab7!important;border-color:#337ab7!important}
+[status="2"] .btn[launch="video"],[status="2"] .btn[video="launch"],[data-status="2"] .btn[video="launch"]{background:#a59100!important;border-color:#a59100!important}

web/libs/themes/Deep Forest/style.css

@@ -17,6 +17,10 @@
 .btn-success {background: #1d8a70!important;border-color:#1d8a70!important;}
 .btn-danger {background: #bf7573!important;border-color:#bf7573!important;}
 .btn-warning {background: #b3a228!important;border-color:#b3a228!important;}
+
+[status="1"] .btn[video="launch"],[data-status="1"] .btn[video="launch"]{background:#337ab7!important;border-color:#337ab7!important}
+[status="2"] .btn[launch="video"],[status="2"] .btn[video="launch"],[data-status="2"] .btn[video="launch"]{background:#a59100!important;border-color:#a59100!important}
+
 ::-webkit-scrollbar-thumb:hover {
     background-color:#27b392!important;
 }

web/libs/themes/Enterprise Blue/style.css

@@ -38,4 +38,7 @@
 .form-group-group.forestgreen {border-color: #091222;}
 .form-group-group.forestgreen h4 {background: #091222;}
 .dark .form-group-group {background: #18253e;}
-.dark .list-group-item {background: #091222;border-color: #18253e;}
+.dark .list-group-item {background: #091222;border-color: #18253e;}
+
+[status="1"] .btn[video="launch"],[data-status="1"] .btn[video="launch"]{background:#337ab7!important;border-color:#337ab7!important}
+[status="2"] .btn[launch="video"],[status="2"] .btn[video="launch"],[data-status="2"] .btn[video="launch"]{background:#a59100!important;border-color:#a59100!important}

web/libs/themes/Ice/style.css

@@ -20,6 +20,10 @@
 .btn-success {background: #27b392!important;border-color:#27b392!important;}
 .btn-danger {background: #bf7573!important;border-color:#bf7573!important;}
 .btn-warning {background: #b3a228!important;border-color:#b3a228!important;}
+
+[status="1"] .btn[video="launch"],[data-status="1"] .btn[video="launch"]{background:#337ab7!important;border-color:#337ab7!important}
+[status="2"] .btn[launch="video"],[status="2"] .btn[video="launch"],[data-status="2"] .btn[video="launch"]{background:#a59100!important;border-color:#a59100!important}
+
 ::-webkit-scrollbar-thumb:hover {
     background-color:#1f80f9;
 }

web/libs/themes/Night Garden/style.css

@@ -20,6 +20,10 @@
 .btn-success {background: #1d8a70!important;border-color:#1d8a70!important;}
 .btn-danger {background: #bf7573!important;border-color:#bf7573!important;}
 .btn-warning {background: #b3a228!important;border-color:#b3a228!important;}
+
+[status="1"] .btn[video="launch"],[data-status="1"] .btn[video="launch"]{background:#337ab7!important;border-color:#337ab7!important}
+[status="2"] .btn[launch="video"],[status="2"] .btn[video="launch"],[data-status="2"] .btn[video="launch"]{background:#a59100!important;border-color:#a59100!important}
+
 ::-webkit-scrollbar-thumb:hover {
     background-color:#27b392;
 }

web/libs/vendor/leaflet/leaflet.css

@@ -0,0 +1,640 @@
+/* required styles */
+
+.leaflet-pane,
+.leaflet-tile,
+.leaflet-marker-icon,
+.leaflet-marker-shadow,
+.leaflet-tile-container,
+.leaflet-pane > svg,
+.leaflet-pane > canvas,
+.leaflet-zoom-box,
+.leaflet-image-layer,
+.leaflet-layer {
+	position: absolute;
+	left: 0;
+	top: 0;
+	}
+.leaflet-container {
+	overflow: hidden;
+	}
+.leaflet-tile,
+.leaflet-marker-icon,
+.leaflet-marker-shadow {
+	-webkit-user-select: none;
+	   -moz-user-select: none;
+	        user-select: none;
+	  -webkit-user-drag: none;
+	}
+/* Prevents IE11 from highlighting tiles in blue */
+.leaflet-tile::selection {
+	background: transparent;
+}
+/* Safari renders non-retina tile on retina better with this, but Chrome is worse */
+.leaflet-safari .leaflet-tile {
+	image-rendering: -webkit-optimize-contrast;
+	}
+/* hack that prevents hw layers "stretching" when loading new tiles */
+.leaflet-safari .leaflet-tile-container {
+	width: 1600px;
+	height: 1600px;
+	-webkit-transform-origin: 0 0;
+	}
+.leaflet-marker-icon,
+.leaflet-marker-shadow {
+	display: block;
+	}
+/* .leaflet-container svg: reset svg max-width decleration shipped in Joomla! (joomla.org) 3.x */
+/* .leaflet-container img: map is broken in FF if you have max-width: 100% on tiles */
+.leaflet-container .leaflet-overlay-pane svg,
+.leaflet-container .leaflet-marker-pane img,
+.leaflet-container .leaflet-shadow-pane img,
+.leaflet-container .leaflet-tile-pane img,
+.leaflet-container img.leaflet-image-layer,
+.leaflet-container .leaflet-tile {
+	max-width: none !important;
+	max-height: none !important;
+	}
+
+.leaflet-container.leaflet-touch-zoom {
+	-ms-touch-action: pan-x pan-y;
+	touch-action: pan-x pan-y;
+	}
+.leaflet-container.leaflet-touch-drag {
+	-ms-touch-action: pinch-zoom;
+	/* Fallback for FF which doesn't support pinch-zoom */
+	touch-action: none;
+	touch-action: pinch-zoom;
+}
+.leaflet-container.leaflet-touch-drag.leaflet-touch-zoom {
+	-ms-touch-action: none;
+	touch-action: none;
+}
+.leaflet-container {
+	-webkit-tap-highlight-color: transparent;
+}
+.leaflet-container a {
+	-webkit-tap-highlight-color: rgba(51, 181, 229, 0.4);
+}
+.leaflet-tile {
+	filter: inherit;
+	visibility: hidden;
+	}
+.leaflet-tile-loaded {
+	visibility: inherit;
+	}
+.leaflet-zoom-box {
+	width: 0;
+	height: 0;
+	-moz-box-sizing: border-box;
+	     box-sizing: border-box;
+	z-index: 800;
+	}
+/* workaround for https://bugzilla.mozilla.org/show_bug.cgi?id=888319 */
+.leaflet-overlay-pane svg {
+	-moz-user-select: none;
+	}
+
+.leaflet-pane         { z-index: 400; }
+
+.leaflet-tile-pane    { z-index: 200; }
+.leaflet-overlay-pane { z-index: 400; }
+.leaflet-shadow-pane  { z-index: 500; }
+.leaflet-marker-pane  { z-index: 600; }
+.leaflet-tooltip-pane   { z-index: 650; }
+.leaflet-popup-pane   { z-index: 700; }
+
+.leaflet-map-pane canvas { z-index: 100; }
+.leaflet-map-pane svg    { z-index: 200; }
+
+.leaflet-vml-shape {
+	width: 1px;
+	height: 1px;
+	}
+.lvml {
+	behavior: url(#default#VML);
+	display: inline-block;
+	position: absolute;
+	}
+
+
+/* control positioning */
+
+.leaflet-control {
+	position: relative;
+	z-index: 800;
+	pointer-events: visiblePainted; /* IE 9-10 doesn't have auto */
+	pointer-events: auto;
+	}
+.leaflet-top,
+.leaflet-bottom {
+	position: absolute;
+	z-index: 1000;
+	pointer-events: none;
+	}
+.leaflet-top {
+	top: 0;
+	}
+.leaflet-right {
+	right: 0;
+	}
+.leaflet-bottom {
+	bottom: 0;
+	}
+.leaflet-left {
+	left: 0;
+	}
+.leaflet-control {
+	float: left;
+	clear: both;
+	}
+.leaflet-right .leaflet-control {
+	float: right;
+	}
+.leaflet-top .leaflet-control {
+	margin-top: 10px;
+	}
+.leaflet-bottom .leaflet-control {
+	margin-bottom: 10px;
+	}
+.leaflet-left .leaflet-control {
+	margin-left: 10px;
+	}
+.leaflet-right .leaflet-control {
+	margin-right: 10px;
+	}
+
+
+/* zoom and fade animations */
+
+.leaflet-fade-anim .leaflet-tile {
+	will-change: opacity;
+	}
+.leaflet-fade-anim .leaflet-popup {
+	opacity: 0;
+	-webkit-transition: opacity 0.2s linear;
+	   -moz-transition: opacity 0.2s linear;
+	        transition: opacity 0.2s linear;
+	}
+.leaflet-fade-anim .leaflet-map-pane .leaflet-popup {
+	opacity: 1;
+	}
+.leaflet-zoom-animated {
+	-webkit-transform-origin: 0 0;
+	    -ms-transform-origin: 0 0;
+	        transform-origin: 0 0;
+	}
+.leaflet-zoom-anim .leaflet-zoom-animated {
+	will-change: transform;
+	}
+.leaflet-zoom-anim .leaflet-zoom-animated {
+	-webkit-transition: -webkit-transform 0.25s cubic-bezier(0,0,0.25,1);
+	   -moz-transition:    -moz-transform 0.25s cubic-bezier(0,0,0.25,1);
+	        transition:         transform 0.25s cubic-bezier(0,0,0.25,1);
+	}
+.leaflet-zoom-anim .leaflet-tile,
+.leaflet-pan-anim .leaflet-tile {
+	-webkit-transition: none;
+	   -moz-transition: none;
+	        transition: none;
+	}
+
+.leaflet-zoom-anim .leaflet-zoom-hide {
+	visibility: hidden;
+	}
+
+
+/* cursors */
+
+.leaflet-interactive {
+	cursor: pointer;
+	}
+.leaflet-grab {
+	cursor: -webkit-grab;
+	cursor:    -moz-grab;
+	cursor:         grab;
+	}
+.leaflet-crosshair,
+.leaflet-crosshair .leaflet-interactive {
+	cursor: crosshair;
+	}
+.leaflet-popup-pane,
+.leaflet-control {
+	cursor: auto;
+	}
+.leaflet-dragging .leaflet-grab,
+.leaflet-dragging .leaflet-grab .leaflet-interactive,
+.leaflet-dragging .leaflet-marker-draggable {
+	cursor: move;
+	cursor: -webkit-grabbing;
+	cursor:    -moz-grabbing;
+	cursor:         grabbing;
+	}
+
+/* marker & overlays interactivity */
+.leaflet-marker-icon,
+.leaflet-marker-shadow,
+.leaflet-image-layer,
+.leaflet-pane > svg path,
+.leaflet-tile-container {
+	pointer-events: none;
+	}
+
+.leaflet-marker-icon.leaflet-interactive,
+.leaflet-image-layer.leaflet-interactive,
+.leaflet-pane > svg path.leaflet-interactive,
+svg.leaflet-image-layer.leaflet-interactive path {
+	pointer-events: visiblePainted; /* IE 9-10 doesn't have auto */
+	pointer-events: auto;
+	}
+
+/* visual tweaks */
+
+.leaflet-container {
+	background: #ddd;
+	outline: 0;
+	}
+.leaflet-container a {
+	color: #0078A8;
+	}
+.leaflet-container a.leaflet-active {
+	outline: 2px solid orange;
+	}
+.leaflet-zoom-box {
+	border: 2px dotted #38f;
+	background: rgba(255,255,255,0.5);
+	}
+
+
+/* general typography */
+.leaflet-container {
+	font: 12px/1.5 "Helvetica Neue", Arial, Helvetica, sans-serif;
+	}
+
+
+/* general toolbar styles */
+
+.leaflet-bar {
+	box-shadow: 0 1px 5px rgba(0,0,0,0.65);
+	border-radius: 4px;
+	}
+.leaflet-bar a,
+.leaflet-bar a:hover {
+	background-color: #fff;
+	border-bottom: 1px solid #ccc;
+	width: 26px;
+	height: 26px;
+	line-height: 26px;
+	display: block;
+	text-align: center;
+	text-decoration: none;
+	color: black;
+	}
+.leaflet-bar a,
+.leaflet-control-layers-toggle {
+	background-position: 50% 50%;
+	background-repeat: no-repeat;
+	display: block;
+	}
+.leaflet-bar a:hover {
+	background-color: #f4f4f4;
+	}
+.leaflet-bar a:first-child {
+	border-top-left-radius: 4px;
+	border-top-right-radius: 4px;
+	}
+.leaflet-bar a:last-child {
+	border-bottom-left-radius: 4px;
+	border-bottom-right-radius: 4px;
+	border-bottom: none;
+	}
+.leaflet-bar a.leaflet-disabled {
+	cursor: default;
+	background-color: #f4f4f4;
+	color: #bbb;
+	}
+
+.leaflet-touch .leaflet-bar a {
+	width: 30px;
+	height: 30px;
+	line-height: 30px;
+	}
+.leaflet-touch .leaflet-bar a:first-child {
+	border-top-left-radius: 2px;
+	border-top-right-radius: 2px;
+	}
+.leaflet-touch .leaflet-bar a:last-child {
+	border-bottom-left-radius: 2px;
+	border-bottom-right-radius: 2px;
+	}
+
+/* zoom control */
+
+.leaflet-control-zoom-in,
+.leaflet-control-zoom-out {
+	font: bold 18px 'Lucida Console', Monaco, monospace;
+	text-indent: 1px;
+	}
+
+.leaflet-touch .leaflet-control-zoom-in, .leaflet-touch .leaflet-control-zoom-out  {
+	font-size: 22px;
+	}
+
+
+/* layers control */
+
+.leaflet-control-layers {
+	box-shadow: 0 1px 5px rgba(0,0,0,0.4);
+	background: #fff;
+	border-radius: 5px;
+	}
+.leaflet-control-layers-toggle {
+	background-image: url(images/layers.png);
+	width: 36px;
+	height: 36px;
+	}
+.leaflet-retina .leaflet-control-layers-toggle {
+	background-image: url(images/layers-2x.png);
+	background-size: 26px 26px;
+	}
+.leaflet-touch .leaflet-control-layers-toggle {
+	width: 44px;
+	height: 44px;
+	}
+.leaflet-control-layers .leaflet-control-layers-list,
+.leaflet-control-layers-expanded .leaflet-control-layers-toggle {
+	display: none;
+	}
+.leaflet-control-layers-expanded .leaflet-control-layers-list {
+	display: block;
+	position: relative;
+	}
+.leaflet-control-layers-expanded {
+	padding: 6px 10px 6px 6px;
+	color: #333;
+	background: #fff;
+	}
+.leaflet-control-layers-scrollbar {
+	overflow-y: scroll;
+	overflow-x: hidden;
+	padding-right: 5px;
+	}
+.leaflet-control-layers-selector {
+	margin-top: 2px;
+	position: relative;
+	top: 1px;
+	}
+.leaflet-control-layers label {
+	display: block;
+	}
+.leaflet-control-layers-separator {
+	height: 0;
+	border-top: 1px solid #ddd;
+	margin: 5px -10px 5px -6px;
+	}
+
+/* Default icon URLs */
+.leaflet-default-icon-path {
+	background-image: url(images/marker-icon.png);
+	}
+
+
+/* attribution and scale controls */
+
+.leaflet-container .leaflet-control-attribution {
+	background: #fff;
+	background: rgba(255, 255, 255, 0.7);
+	margin: 0;
+	}
+.leaflet-control-attribution,
+.leaflet-control-scale-line {
+	padding: 0 5px;
+	color: #333;
+	}
+.leaflet-control-attribution a {
+	text-decoration: none;
+	}
+.leaflet-control-attribution a:hover {
+	text-decoration: underline;
+	}
+.leaflet-container .leaflet-control-attribution,
+.leaflet-container .leaflet-control-scale {
+	font-size: 11px;
+	}
+.leaflet-left .leaflet-control-scale {
+	margin-left: 5px;
+	}
+.leaflet-bottom .leaflet-control-scale {
+	margin-bottom: 5px;
+	}
+.leaflet-control-scale-line {
+	border: 2px solid #777;
+	border-top: none;
+	line-height: 1.1;
+	padding: 2px 5px 1px;
+	font-size: 11px;
+	white-space: nowrap;
+	overflow: hidden;
+	-moz-box-sizing: border-box;
+	     box-sizing: border-box;
+
+	background: #fff;
+	background: rgba(255, 255, 255, 0.5);
+	}
+.leaflet-control-scale-line:not(:first-child) {
+	border-top: 2px solid #777;
+	border-bottom: none;
+	margin-top: -2px;
+	}
+.leaflet-control-scale-line:not(:first-child):not(:last-child) {
+	border-bottom: 2px solid #777;
+	}
+
+.leaflet-touch .leaflet-control-attribution,
+.leaflet-touch .leaflet-control-layers,
+.leaflet-touch .leaflet-bar {
+	box-shadow: none;
+	}
+.leaflet-touch .leaflet-control-layers,
+.leaflet-touch .leaflet-bar {
+	border: 2px solid rgba(0,0,0,0.2);
+	background-clip: padding-box;
+	}
+
+
+/* popup */
+
+.leaflet-popup {
+	position: absolute;
+	text-align: center;
+	margin-bottom: 20px;
+	}
+.leaflet-popup-content-wrapper {
+	padding: 1px;
+	text-align: left;
+	border-radius: 12px;
+	}
+.leaflet-popup-content {
+	margin: 13px 19px;
+	line-height: 1.4;
+	}
+.leaflet-popup-content p {
+	margin: 18px 0;
+	}
+.leaflet-popup-tip-container {
+	width: 40px;
+	height: 20px;
+	position: absolute;
+	left: 50%;
+	margin-left: -20px;
+	overflow: hidden;
+	pointer-events: none;
+	}
+.leaflet-popup-tip {
+	width: 17px;
+	height: 17px;
+	padding: 1px;
+
+	margin: -10px auto 0;
+
+	-webkit-transform: rotate(45deg);
+	   -moz-transform: rotate(45deg);
+	    -ms-transform: rotate(45deg);
+	        transform: rotate(45deg);
+	}
+.leaflet-popup-content-wrapper,
+.leaflet-popup-tip {
+	background: white;
+	color: #333;
+	box-shadow: 0 3px 14px rgba(0,0,0,0.4);
+	}
+.leaflet-container a.leaflet-popup-close-button {
+	position: absolute;
+	top: 0;
+	right: 0;
+	padding: 4px 4px 0 0;
+	border: none;
+	text-align: center;
+	width: 18px;
+	height: 14px;
+	font: 16px/14px Tahoma, Verdana, sans-serif;
+	color: #c3c3c3;
+	text-decoration: none;
+	font-weight: bold;
+	background: transparent;
+	}
+.leaflet-container a.leaflet-popup-close-button:hover {
+	color: #999;
+	}
+.leaflet-popup-scrolled {
+	overflow: auto;
+	border-bottom: 1px solid #ddd;
+	border-top: 1px solid #ddd;
+	}
+
+.leaflet-oldie .leaflet-popup-content-wrapper {
+	-ms-zoom: 1;
+	}
+.leaflet-oldie .leaflet-popup-tip {
+	width: 24px;
+	margin: 0 auto;
+
+	-ms-filter: "progid:DXImageTransform.Microsoft.Matrix(M11=0.70710678, M12=0.70710678, M21=-0.70710678, M22=0.70710678)";
+	filter: progid:DXImageTransform.Microsoft.Matrix(M11=0.70710678, M12=0.70710678, M21=-0.70710678, M22=0.70710678);
+	}
+.leaflet-oldie .leaflet-popup-tip-container {
+	margin-top: -1px;
+	}
+
+.leaflet-oldie .leaflet-control-zoom,
+.leaflet-oldie .leaflet-control-layers,
+.leaflet-oldie .leaflet-popup-content-wrapper,
+.leaflet-oldie .leaflet-popup-tip {
+	border: 1px solid #999;
+	}
+
+
+/* div icon */
+
+.leaflet-div-icon {
+	background: #fff;
+	border: 1px solid #666;
+	}
+
+
+/* Tooltip */
+/* Base styles for the element that has a tooltip */
+.leaflet-tooltip {
+	position: absolute;
+	padding: 6px;
+	background-color: #fff;
+	border: 1px solid #fff;
+	border-radius: 3px;
+	color: #222;
+	white-space: nowrap;
+	-webkit-user-select: none;
+	-moz-user-select: none;
+	-ms-user-select: none;
+	user-select: none;
+	pointer-events: none;
+	box-shadow: 0 1px 3px rgba(0,0,0,0.4);
+	}
+.leaflet-tooltip.leaflet-clickable {
+	cursor: pointer;
+	pointer-events: auto;
+	}
+.leaflet-tooltip-top:before,
+.leaflet-tooltip-bottom:before,
+.leaflet-tooltip-left:before,
+.leaflet-tooltip-right:before {
+	position: absolute;
+	pointer-events: none;
+	border: 6px solid transparent;
+	background: transparent;
+	content: "";
+	}
+
+/* Directions */
+
+.leaflet-tooltip-bottom {
+	margin-top: 6px;
+}
+.leaflet-tooltip-top {
+	margin-top: -6px;
+}
+.leaflet-tooltip-bottom:before,
+.leaflet-tooltip-top:before {
+	left: 50%;
+	margin-left: -6px;
+	}
+.leaflet-tooltip-top:before {
+	bottom: 0;
+	margin-bottom: -12px;
+	border-top-color: #fff;
+	}
+.leaflet-tooltip-bottom:before {
+	top: 0;
+	margin-top: -12px;
+	margin-left: -6px;
+	border-bottom-color: #fff;
+	}
+.leaflet-tooltip-left {
+	margin-left: -6px;
+}
+.leaflet-tooltip-right {
+	margin-left: 6px;
+}
+.leaflet-tooltip-left:before,
+.leaflet-tooltip-right:before {
+	top: 50%;
+	margin-top: -6px;
+	}
+.leaflet-tooltip-left:before {
+	right: 0;
+	margin-right: -12px;
+	border-left-color: #fff;
+	}
+.leaflet-tooltip-right:before {
+	left: 0;
+	margin-left: -12px;
+	border-right-color: #fff;
+	}

web/pages/blocks/help.ejs

@@ -189,7 +189,7 @@
                                   icon: 'star',
                                   color: 'success',
                                   text: 'Join Public Beta',
-                                  href: 'https://cdn.shinobi.video/installers/Dashcam/',
+                                  href: 'https://cdn.shinobi.video/installers/Byaku/',
                                   class: ''
                               },
                               {

web/pages/blocks/mainpermissions.ejs

@@ -295,7 +295,6 @@ $.aC.e.on('click','.delete',function(e){
     e.html='Do you want to delete <b>'+e.account.mail+'</b>? You cannot recover this account. Files will remain in the filesystem. If you choose to create an account with the same Group Key it will have the previous events activated in that account.'
     $.confirm.body.html(e.html)
     $.confirm.click({title:'Delete',class:'btn-danger'},function(){
-        // $.ccio.cx({f:'accounts',ff:'delete',account:e.account})
         $.post('<%=originalURL%><%=config.webPaths.superApiPrefix%>'+$user.sessionKey+'/accounts/deleteAdmin',{
             account : e.account,
             // "deleteSubAccounts" : "1",

web/pages/blocks/superCustomAutoLoadManager.ejs

@@ -0,0 +1,14 @@
+<link rel="stylesheet" href="<%-window.libURL%>libs/css/super.customAutoLoad.css">
+<form class="form-group-group red pt-4" id="downloadNewModule">
+    <div class="form-group">
+        <input type="text" placeholder="Download URL for Module" class="form-control" name="downloadUrl" />
+    </div>
+    <div class="form-group">
+        <input type="text" placeholder="Subdirectory for Module (Inside the downloaded package)" class="form-control" name="packageRoot" />
+    </div>
+    <div><button type="submit" class="btn btn-block btn-default"><i class="fa fa-download"></i> <%- lang.Download %></button></div>
+</form>
+<div class="form-group-group red pt-4 pb-0 row m-0" id="customAutoLoadList">
+
+</div>
+<script src="<%-window.libURL%>libs/js/super.customAutoLoad.js" type="text/javascript"></script>

web/pages/blocks/videoview.ejs

@@ -66,7 +66,17 @@
         </button>
         <h4 class="modal-title" id="video_viewerLabel"><i class="fa fa-play-circle"></i> &nbsp; <span></span></h4>
       </div>
-      <div class="modal-body text-center"></div>
+      <div class="modal-body text-center">
+          <div class="row">
+              <div class="col-md-8 video-container">
+              </div>
+              <div class="col-md-4">
+                  <div id="video_viewer_gps_map">
+
+                  </div>
+              </div>
+          </div>
+      </div>
       <div class="modal-footer">
         <a class="btn btn-danger pull-left" video="delete" data-dismiss="modal"><%-lang.Delete%></a>
           <div class="btn-group">

web/pages/home.ejs

@@ -16,6 +16,7 @@
 <link rel="stylesheet" href="<%-window.libURL%>libs/css/gridstack-extra.min.css">
 <link rel="stylesheet" href="<%-window.libURL%>libs/css/bootstrap-table.min.css">
 <link rel="stylesheet" href="<%-window.libURL%>libs/css/bootstrap-slider.min.css">
+<link rel="stylesheet" href="<%-window.libURL%>libs/vendor/leaflet/leaflet.css">
 
 <link rel="stylesheet" href="<%-window.libURL%>libs/css/dash2.basic.css">
 <link rel="stylesheet" href="<%-window.libURL%>libs/css/dash2.misc.css">
@@ -72,7 +73,7 @@
                     </div>
                     <div class="ram_load display-table-cell">
                         <span class="pull-right percent"></span>
-                       <label><span class="os_totalmem" style="letter-spacing:2px;font-weight:100"></span> <%-lang.MB%> <%-lang.RAM%></label>
+                       <label><span class="os_totalmem used" style="letter-spacing:2px;font-weight:100"></span> <%-lang.MB%> <%-lang.RAM%></label>
                       <div class="progress">
                         <div class="progress-bar progress-bar-warning" role="progressbar" style="width:0%"></div>
                       </div>
@@ -206,6 +207,7 @@
 <script src="<%-window.libURL%>libs/js/lodash.min.js"></script>
 <script src="<%-window.libURL%>libs/js/gridstack.min.js"></script>
 <script src="<%-window.libURL%>libs/js/gridstack.jQueryUI.min.js"></script>
+<script src="<%-window.libURL%>libs/vendor/leaflet/leaflet.js"></script>
 <script src="<%-window.libURL%>libs/js/basic.js"></script>
 <script><% include ../libs/js/dash2.config.js %></script>
 <script src="<%-window.libURL%>libs/js/dash2.basic.js"></script>
@@ -215,6 +217,7 @@
 <script src="<%-window.libURL%>libs/js/dash2.elements.js"></script>
 <script src="<%-window.libURL%>libs/js/dash2.elementbuilder.js"></script>
 <script src="<%-window.libURL%>libs/js/dash2.init.js"></script>
+<script src="<%-window.libURL%>libs/js/dash2.gps.js"></script>
 <% customAutoLoad.LibsJs.forEach(function(lib){ %>
     <script src="<%-window.libURL%>libs/js/<%-lib%>"></script>
 <% }) %>

web/pages/super.ejs

@@ -40,6 +40,7 @@
         <link rel="stylesheet" href="<%-window.libURL%>libs/css/<%-lib%>">
     <% }) %>
 </head>
+<script>$user=<%-JSON.stringify($user)%></script>
 
 <body class="index-page sidebar-collapse bg-hexagon">
     <!-- Navbar -->
@@ -87,6 +88,9 @@
                                 <li class="nav-item">
                                     <a class="nav-link" data-toggle="tab" href="#changeSuperPreferences" role="tab"><%-lang['Preferences']%></a>
                                 </li>
+                                <li class="nav-item">
+                                    <a class="nav-link" data-toggle="tab" href="#customAutoLoad" role="tab"><%-lang['Custom Auto Load']%></a>
+                                </li>
                             </ul>
                             <div class="card-body">
                                 <!-- Tab panes -->
@@ -101,6 +105,9 @@
                                     <div class="tab-pane text-left" id="system" role="tabpanel">
                                         <% include blocks/superSystemTab.ejs %>
                                     </div>
+                                    <div class="tab-pane text-left" id="customAutoLoad" role="tabpanel">
+                                        <% include blocks/superCustomAutoLoadManager.ejs %>
+                                    </div>
                                     <div class="tab-pane text-left" id="changeSuperPreferences" role="tabpanel">
                                         <% include blocks/changeSuperPreferences.ejs %>
                                     </div>
@@ -151,7 +158,6 @@
     }
 </script>
 
-<script>$user=<%-JSON.stringify($user)%></script>
 <script>
 $.ccio={accounts:{}};$.ls=localStorage;
 if(!$user.lang||$user.lang==''){