ShinobiSystems
/
Shinobi
mirror of https://gitlab.com/Shinobi-Systems/Shinobi.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'dev' into 'dev' 

# Conflicts:
#   languages/ja.json
video-slicer-reencoder
Moe 2022-09-06 20:42:53 +00:00
parent 450ab01b82 4f87800c85
commit b0a7457e5c
633 changed files with 116785 additions and 93288 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@ -11,3 +11,4 @@ npm-debug.log
shinobi.sqlite
dist
._*
generatedLanguageFiles

166
COPYING.md
View File

@ -3,95 +3,119 @@ SHINOBI OPEN SOURCE SOFTWARE LICENSE AGREEMENT
Version 1, 04 June 2018
Copyright (C) 2018 [Shinobi Systems](https://shinobi.systems)
*We'll try to keep it simple. Thanks for using Shinobi Software!*
Defintions.
Definitions.
-----------
In this license, which also serves as a general End User License Agreement [EULA], the following
In this license, which also serves as a general End User License Agreement [EULA], the following
terms shall be interpreted by these definitions:
* "EULA" shall mean this End User Licence Agreement
* "Licensor" shall mean SHINOBI SYSTEMS
* "Licensee" shall mean YOU, or the organisation (if any) on whose behalf YOU are taking the EULA.
"SOFTWARE PRODUCTS" or "SOFTWARE" or "PRODUCTS" shall mean the Software Product this License is
included with and any additional modules or add-ons delivered by Shinobi Systems. The term
"SOFTWARE PRODUCTS" or "SOFTWARE" or "PRODUCTS" shall mean the Software Product this License is
included with and any additional modules or add-ons delivered by Shinobi Systems. The term
"SOFTWARE" includes, to the extent provided by SHINOBI SYSTEMS:
1) any revisions, updates and/or upgrades thereto;
2) any data, image or executable files, databases, data engines, computer software, or similar
2) any data, image or executable files, databases, data engines, computer software, or similar
items customarily used or distributed with computer software products;
3) anything in any form whatsoever intended to be used with or in conjunction with the SOFTWARE;
4) any associated media, documentation (including physical, electronic and on-line) and printed
4) any associated media, documentation (including physical, electronic and on-line) and printed
materials (the "Documentation").
Purpose of the Agreement.
-------------------------
**The Short**: Protect the rights of this Software Product and the LICENSOR.
**The Long**: The LICENSOR grants the LICENSEE a non-exclusive, non-transferable and perpetual
licence to use the SOFTWARE PRODUCTS listed therein and under the terms thereof. By accepting
the terms and conditions established in this agreement, the LICENSEE does not acquire any
ownership of copyright or other intellectual property rights in any part of the SOFTWARE
PRODUCTS. The LICENSEE is only entitled to use the SOFTWARE PRODUCTS in accordance with the
terms and conditions set forth by Shinobi Systems. By using the SOFTWARE PRODUCTS, the
**The Long**: The LICENSOR grants the LICENSEE a non-exclusive, non-transferable and perpetual
licence to use the SOFTWARE PRODUCTS listed therein and under the terms thereof. By accepting
the terms and conditions established in this agreement, the LICENSEE does not acquire any
ownership of copyright or other intellectual property rights in any part of the SOFTWARE
PRODUCTS. The LICENSEE is only entitled to use the SOFTWARE PRODUCTS in accordance with the
terms and conditions set forth by Shinobi Systems. By using the SOFTWARE PRODUCTS, the
LICENSEE agrees to accept the terms and conditions presented.
LICENSEE must purchase the applicable subscription in any other use case unless otherwise
granted. If the use case does not have a subscription applicable please contact a
LICENSEE must purchase the applicable subscription in any other use case unless otherwise
granted. If the use case does not have a subscription applicable please contact a
representative at support@shinobi.systems.
#### Commercial Uses
- Selling usage of the software
- Selling hardware with the software on it
- Using the software in locations that engage in the buying and selling of goods and services
As of 2022-07-12 the noted situations below are seen the same as "Commercial Use".
- 25 Active Monitor Rule : Having at least 25 Active Monitors and not a Primary School or Secondary School. Does not apply to Personal Use.
- 150 Active Monitor Rule : If you have more than 150 Active Monitors please contact support for an Enterprise License. The retail Shinobi Pro license will not be applicable for these installations.
- Used on a Device that was part of a commercial transaction. This can be, but not limited to, being sold or provided additionally to a sale.
#### Conditions for Free (Unpaid) use.
If you fall under any conditions for "Commercial Use" none of the following can apply unless otherwise noted.
- Personal use
- Use in a non commercial area
- Used for non commercial purposes
- When used for research or educational purposes
- Testing Purposes
- Usage by Educational institutions
- Use for Emergency Services and facilties associated like Search and Rescue Services or
Ambulance Services
- Use in Health Care facility like a hospital or walk-in clinic
- Usage by Primary Education or Secondary Education Schools
- Use for Emergency Services and facilities associated like Search and Rescue Services or
Ambulance Services with less than 50 cameras
- Use in Health Care facility like a hospital or walk-in clinic with less than 100 cameras
- Schools and Organizations that have been given exemption
As of 2022-07-12 If you are an organization that falls under one or more of these conditions for free use then you must display that you are using Shinobi Systems software as "shinobi.video". This can be physically on premise or on your organization's public web page.
Falling under the conditions for Free use does not guarantee you a License Key. It simply means you are allowed to use it without paying or trading for that use.
#### Unsure if your use is commercial or non-commercial
Please contact us through the Live Chat of our website and we can negotiate or make exceptions based on the circumstances of your use case.
#### Registration
Schools and Resellers must register with Shinobi Systems at https://licenses.shinobi.video.
In your account please fill out the "Apply for a School License" form or "Apply to be a Reseller" form.
#### Support Services.
The Maintenance and Support Service shall be contracted and provided as per selected plan
The Maintenance and Support Service shall be contracted and provided as per selected plan
agreement, taxes will be included in all prices for Support Services.
Support Services will only provide support services as per the selected agreement.
This is not the entire agreement on support services. You must also review all agreements
This is not the entire agreement on support services. You must also review all agreements
provided with subscription plans provided.
#### Software Product Ownership.
This software is property of Shinobi Systems. LICENSEE must keep all copyright notices
This software is property of Shinobi Systems. LICENSEE must keep all copyright notices
unchanged.
#### Modification of this Software Product.
LICENSEE may modify code for personal use but must provide these changes upon request from
Shinobi Systems or an authorized Shinobi representative. LICENSEE may not alter or change
copyright notices. All code changes by LICENSEE shall fall under the copyright of Shinobi
LICENSEE may modify code for but must provide these changes upon request from
Shinobi Systems or an authorized Shinobi representative. LICENSEE may not alter or change
copyright notices. All code changes by LICENSEE shall fall under the copyright of Shinobi
Systems in the case code modified by LICENSEE is integrated into the official Shinobi code base.
#### Software Product Rebranding or "White-Labelling".
LICENSEE can remove the Shinobi branding from the front end but all copyright notices must
LICENSEE can remove the Shinobi branding from the front end but all copyright notices must
remain unchanged.
#### Software Product Contributions.
All contributed code becomes the property of Shinobi Systems. All contributors give permission
All contributed code becomes the property of Shinobi Systems. All contributors give permission
to Shinobi and Shinobi developers to use the code however it is seen fit.
#### Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM
"AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK
AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE,
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM
"AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK
AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE,
YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
#### Changes to the Agreement.
Shinobi Systems reserves the right to change the license and set of terms at any time.
Continued use is agreement to those possible changes. Changes to this license will be provided
Shinobi Systems reserves the right to change the license and set of terms at any time.
Continued use is agreement to those possible changes. Changes to this license will be provided
in the commit history of the repository it is located in.
#### Legal Proceedings.
@ -100,31 +124,18 @@ All lawsuits must be filed at the Vancouver Court House.
Courthouse Vancouver Robson Square
800 Hornby St, Vancouver, BC V6Z 2C5
#### List of Included Software
#### List of Included Software
This list is completed to best of our knowledge.
This list is completed to the best of our knowledge.
Node.js - https://nodejs.org/en/
MariaDB - https://mariadb.org/
FFmpeg - https://www.ffmpeg.org/
request - https://www.npmjs.com/package/request
Express (npm) - https://expressjs.com/ https://www.npmjs.com/package/express
EJS (npm) - http://ejs.co/ https://www.npmjs.com/package/ejs
pam-diff (npm) (Motion Detector) - https://github.com/kevinGodell/pam-diff
pipe2pam (npm) (for pam-diff) - https://github.com/kevinGodell/pipe2pam
mp4frag (npm) (Poseidon's main engine) - https://github.com/kevinGodell/mp4frag
pam-diff (Motion Detector) - https://github.com/kevinGodell/pam-diff
pipe2pam (for pam-diff) - https://github.com/kevinGodell/pipe2pam
mp4frag (Poseidon's main engine) - https://github.com/kevinGodell/mp4frag
mse-live-player (for mp4frag) - https://github.com/kevinGodell/mse-live-player
pipe2jpeg (npm) - https://github.com/kevinGodell/pipe2jpeg
webdav (npm) - https://www.npmjs.com/package/webdav
jsonfile (npm) - https://www.npmjs.com/package/jsonfile
connectionTester (npm) - https://www.npmjs.com/package/connectionTester
node-onvif (npm) - https://www.npmjs.com/package/node-onvif
knex (npm) - https://www.npmjs.com/package/knex
nodemailer (npm) - https://www.npmjs.com/package/nodemailer
mysql (npm) - https://www.npmjs.com/package/mysql
sqlite3 (npm) - https://www.npmjs.com/package/sqlite3
ldapauth-fork (npm) - https://www.npmjs.com/package/ldapauth-fork
http-proxy (npm) - https://www.npmjs.com/package/http-proxy
pipe2jpeg - https://github.com/kevinGodell/pipe2jpeg
hls.js - https://github.com/video-dev/hls.js/
flv.js - https://github.com/Bilibili/flv.js/
Material Design Lite - https://getmdl.io/
@ -142,3 +153,50 @@ Courthouse Vancouver Robson Square
Moment.js - https://momentjs.com/
Livestamp.js - https://mattbradley.github.io/livestampjs/
Lodash - https://lodash.com/
**npmjs.com packages**
async - https://www.npmjs.com/package/async
aws-sdk - https://www.npmjs.com/package/aws-sdk
backblaze-b2 - https://www.npmjs.com/package/backblaze-b2
body-parser - https://www.npmjs.com/package/body-parser
bson - https://www.npmjs.com/package/bson
connection-tester - https://www.npmjs.com/package/connection-tester
cws - https://www.npmjs.com/package/cws
digest-fetch - https://www.npmjs.com/package/digest-fetch
discord.js - https://www.npmjs.com/package/discord.js
ejs - https://www.npmjs.com/package/ejs
express - https://www.npmjs.com/package/express
express-fileupload - https://www.npmjs.com/package/express-fileupload
fs-extra - https://www.npmjs.com/package/fs-extra
ftp-srv - https://www.npmjs.com/package/ftp-srv
googleapis - https://www.npmjs.com/package/googleapis
http-proxy - https://www.npmjs.com/package/http-proxy
jsonfile - https://www.npmjs.com/package/jsonfile
knex - https://www.npmjs.com/package/knex
ldapauth-fork - https://www.npmjs.com/package/ldapauth-fork
moment - https://www.npmjs.com/package/moment
mp4frag - https://www.npmjs.com/package/mp4frag
mqtt - https://www.npmjs.com/package/mqtt
mysql - https://www.npmjs.com/package/mysql
node-abort-controller - https://www.npmjs.com/package/node-abort-controller
node-fetch - https://www.npmjs.com/package/node-fetch
node-onvif-events - https://www.npmjs.com/package/node-onvif-events
node-ssh - https://www.npmjs.com/package/node-ssh
node-telegram-bot-api - https://www.npmjs.com/package/node-telegram-bot-api
nodemailer - https://www.npmjs.com/package/nodemailer
pam-diff - https://www.npmjs.com/package/pam-diff
path - https://www.npmjs.com/package/path
pipe2pam - https://www.npmjs.com/package/pipe2pam
pixel-change - https://www.npmjs.com/package/pixel-change
pushover-notifications - https://www.npmjs.com/package/pushover-notifications
sat - https://www.npmjs.com/package/sat
shinobi-onvif - https://www.npmjs.com/package/shinobi-onvif
shinobi-sound-detection - https://www.npmjs.com/package/shinobi-sound-detection
shinobi-zwave - https://www.npmjs.com/package/shinobi-zwave
smtp-server - https://www.npmjs.com/package/smtp-server
socket.io - https://www.npmjs.com/package/socket.io
socket.io-client - https://www.npmjs.com/package/socket.io-client
tree-kill - https://www.npmjs.com/package/tree-kill
unzipper - https://www.npmjs.com/package/unzipper
webdav-fs - https://www.npmjs.com/package/webdav-fs

9
Docker/README.md
View File

@ -4,7 +4,11 @@
## Docker Ninja Way
> This method uses `docker-compose` and has the ability to quick install the TensorFlow Object Detection plugin.
> This method uses `docker-compose` and has the ability to quick install the TensorFlow Object Detection plugin. This will build your container from the images hosted on Gitlab.
> **We no longer use Docker Hub** and will not in the foreseeable future.
Docker Image Used : `registry.gitlab.com/shinobi-systems/shinobi:dev`
```
bash <(curl -s https://gitlab.com/Shinobi-Systems/Shinobi-Installer/raw/master/shinobi-docker.sh)
@ -17,10 +21,11 @@ bash <(curl -s https://gitlab.com/Shinobi-Systems/Shinobi-Installer/raw/master/s
> Please remember to check out the Environment Variables table further down this README.
```
docker run -d --name='Shinobi' -p '8080:8080/tcp' -v "/dev/shm/Shinobi/streams":'/dev/shm/streams':'rw' -v "$HOME/Shinobi/config":'/config':'rw' -v "$HOME/Shinobi/customAutoLoad":'/home/Shinobi/libs/customAutoLoad':'rw' -v "$HOME/Shinobi/database":'/var/lib/mysql':'rw' -v "$HOME/Shinobi/videos":'/home/Shinobi/videos':'rw' -v "$HOME/Shinobi/plugins":'/home/Shinobi/plugins':'rw' -v '/etc/localtime':'/etc/localtime':'ro' shinobisystems/shinobi:dev
docker run -d --name='Shinobi' -p '8080:8080/tcp' -v "/dev/shm/Shinobi/streams":'/dev/shm/streams':'rw' -v "$HOME/Shinobi/config":'/config':'rw' -v "$HOME/Shinobi/customAutoLoad":'/home/Shinobi/libs/customAutoLoad':'rw' -v "$HOME/Shinobi/database":'/var/lib/mysql':'rw' -v "$HOME/Shinobi/videos":'/home/Shinobi/videos':'rw' -v "$HOME/Shinobi/plugins":'/home/Shinobi/plugins':'rw' -v '/etc/localtime':'/etc/localtime':'ro' registry.gitlab.com/shinobi-systems/shinobi:dev
```
#### Installing Object Detection (TensorFlow.js)
**DEPRECATED, UPDATED IMAGE COMING SOON**
> This requires that you add the plugin key to the Shinobi container. This key is generated and displayed in the startup logs of the Object Detection docker container.

2
INSTALL/CentOS - Quick Install.sh
View File

@ -85,7 +85,7 @@ echo "========================================================="
#Check if Node.js is installed
if ! [ -x "$(command -v node)" ]; then
echo "Node.js not found, installing..."
sudo curl --silent --location https://rpm.nodesource.com/setup_12.x | sudo bash -
sudo curl --silent --location https://rpm.nodesource.com/setup_16.x | sudo bash -
sudo "$pkgmgr" install nodejs -y -q -e 0
else
echo "Node.js is already installed..."

9
INSTALL/cuda-10.sh
View File

@ -4,10 +4,11 @@ echo "-- Installing CUDA Toolkit and CUDA DNN --"
echo "------------------------------------------"
# Install CUDA Drivers and Toolkit
if [ -x "$(command -v apt)" ]; then
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-repo-ubuntu1804_10.0.130-1_amd64.deb
sudo dpkg -i --force-overwrite cuda-repo-ubuntu1804_10.0.130-1_amd64.deb
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
wget https://cdn.shinobi.video/installers/cuda-repo-ubuntu1804-10-0-local-10.0.130-410.48_1.0-1_amd64.deb -O cuda.deb
dpkg -i cuda.deb
sudo apt-key add /var/cuda-repo-10-0-local-10.0.130-410.48/7fa2af80.pub
sudo apt-get update
sudo apt install cuda-toolkit-10-0
sudo apt-get update -y
sudo apt-get -o Dpkg::Options::="--force-overwrite" install cuda-toolkit-10-0 -y --no-install-recommends

36
INSTALL/cuda-11.sh Normal file
View File

@ -0,0 +1,36 @@
#!/bin/sh
echo "------------------------------------------"
echo "-- Installing CUDA Toolkit and CUDA DNN --"
echo "------------------------------------------"
# Install CUDA Drivers and Toolkit
if [ -x "$(command -v apt)" ]; then
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget http://developer.download.nvidia.com/compute/cuda/11.0.2/local_installers/cuda-repo-ubuntu2004-11-0-local_11.0.2-450.51.05-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2004-11-0-local_11.0.2-450.51.05-1_amd64.deb
sudo apt-key add /var/cuda-repo-ubuntu2004-11-0-local/7fa2af80.pub
sudo apt-get update -y
sudo apt-get -o Dpkg::Options::="--force-overwrite" install cuda-toolkit-11-0 -y --no-install-recommends
sudo apt-get -o Dpkg::Options::="--force-overwrite" install --fix-broken -y
sudo apt install nvidia-utils-495 nvidia-headless-495 -y
# Install CUDA DNN
wget https://cdn.shinobi.video/installers/libcudnn8_8.2.1.32-1+cuda11.3_amd64.deb -O cuda-dnn.deb
sudo dpkg -i cuda-dnn.deb
wget https://cdn.shinobi.video/installers/libcudnn8-dev_8.2.0.53-1+cuda11.3_amd64.deb -O cuda-dnn-dev.deb
sudo dpkg -i cuda-dnn-dev.deb
echo "-- Cleaning Up --"
# Cleanup
sudo rm cuda-dnn.deb
sudo rm cuda-dnn-dev.deb
fi
echo "------------------------------"
echo "Reboot is required. Do it now?"
echo "------------------------------"
echo "(y)es or (N)o. Default is No."
read rebootTheMachineHomie
if [ "$rebootTheMachineHomie" = "y" ] || [ "$rebootTheMachineHomie" = "Y" ]; then
sudo reboot
fi

2
INSTALL/freebsd.sh
View File

@ -46,7 +46,7 @@ npm install --unsafe-perm
# sudo npm audit fix --force
echo "============="
echo "Shinobi - Install PM2"
npm install pm2@3.0.0 -g
npm install pm2@latest -g
if (! -e "./conf.json" ) then
cp conf.sample.json conf.json
endif

2
INSTALL/freenas.csh
View File

@ -18,7 +18,7 @@ npm i npm -g
#There are some errors in here that I don't want you to see. Redirecting to dev null :D
npm install --unsafe-perm > & /dev/null
# npm audit fix --force > & /dev/null
npm install pm2@3.0.0 -g
npm install pm2@latest -g
cp conf.sample.json conf.json
cp super.sample.json super.json
pm2 start camera.js

2
INSTALL/macos-part2.sh
View File

@ -18,7 +18,7 @@ sudo npm install --unsafe-perm
# sudo npm audit fix --unsafe-perm
echo "============="
echo "Shinobi - Install PM2"
sudo npm install pm2@3.0.0 -g
sudo npm install pm2@latest -g
if [ ! -e "./conf.json" ]; then
sudo cp conf.sample.json conf.json
fi

9
INSTALL/macos.sh
View File

@ -10,7 +10,7 @@ echo "Shinobi - Do you want to Install Node.js?"
echo "(y)es or (N)o"
read -r nodejsinstall
if [ "$nodejsinstall" = "y" ]; then
curl -o node-installer.pkg https://nodejs.org/dist/v11.9.0/node-v11.9.0.pkg
curl -o node-installer.pkg https://nodejs.org/dist/v16.15.0/node-v16.15.0.pkg
sudo installer -pkg node-installer.pkg -target /
rm node-installer.pkg
sudo ln -s /usr/local/bin/node /usr/bin/nodejs
@ -34,18 +34,13 @@ if [ "$ffmpeginstall" = "y" ]; then
sudo chmod +x /usr/local/bin/ffserver
fi
echo "============="
if [ ! -e "./shinobi.sqlite" ]; then
sudo npm install jsonfile
sudo cp sql/shinobi.sample.sqlite shinobi.sqlite
sudo node tools/modifyConfiguration.js databaseType=sqlite3
fi
echo "Shinobi - Install NPM Libraries"
sudo npm i npm -g
sudo npm install --unsafe-perm
# sudo npm audit fix --unsafe-perm
echo "============="
echo "Shinobi - Install PM2"
sudo npm install pm2@3.0.0 -g
sudo npm install pm2@latest -g
if [ ! -e "./conf.json" ]; then
sudo cp conf.sample.json conf.json
fi

4
INSTALL/opensuse.sh
View File

@ -34,7 +34,7 @@ echo "(y)es or (N)o"
NODEJSINSTALL=0
read -r nodejsinstall
if [ "$nodejsinstall" = "y" ] || [ "$nodejsinstall" = "Y" ]; then
sudo zypper install -y nodejs11
sudo zypper install -y nodejs16
NODEJSINSTALL=1
fi
echo "============="
@ -89,7 +89,7 @@ npm install --unsafe-perm
# sudo npm audit fix --force
echo "============="
echo "Shinobi - Install PM2"
sudo npm install pm2@3.0.0 -g
sudo npm install pm2@latest -g
echo "Shinobi - Finished"
sudo chmod -R 755 .
touch INSTALL/installed.txt

6
INSTALL/ubuntu-easyinstall.sh
View File

@ -14,11 +14,11 @@ fi
echo "============="
echo " Detecting Ubuntu Version"
echo "============="
declare -i getubuntuversion=$(lsb_release -r | awk '{print $2}' | cut -d . -f1)
getubuntuversion=$(lsb_release -r | awk '{print $2}' | cut -d . -f1)
echo "============="
echo " Ubuntu Version: $getubuntuversion"
echo "============="
if [[ "$getubuntuversion" == "16" || "$getubuntuversion" < "16" ]]; then
if [[ "$getubuntuversion" == "16" || "$getubuntuversion" -le "16" ]]; then
echo "============="
echo "Shinobi - Get FFMPEG 3.x from ppa:jonathonf/ffmpeg-3"
sudo add-apt-repository ppa:jonathonf/ffmpeg-3 -y
@ -154,7 +154,7 @@ echo "============="
#Install PM2
echo "Shinobi - Install PM2"
sudo npm install pm2@3.0.0 -g
sudo npm install pm2@latest -g
if [ ! -e "./conf.json" ]; then
cp conf.sample.json conf.json
fi

6
INSTALL/ubuntu-touchless-iso.sh
View File

@ -15,7 +15,7 @@ echo " Ubuntu Version: $getubuntuversion"
echo "============="
apt update -y
apt update --fix-missing -y
if [ "$getubuntuversion" = "18" ] || [ "$getubuntuversion" > "18" ]; then
if [ "$getubuntuversion" = "18" ] || [ "$getubuntuversion" -le "18" ]; then
apt install sudo wget -y
sudo apt install -y software-properties-common
sudo add-apt-repository universe -y
@ -66,7 +66,7 @@ if ! [ -x "$(command -v npm)" ]; then
fi
sudo apt install make zip -y
if ! [ -x "$(command -v ffmpeg)" ]; then
if [ "$getubuntuversion" = "16" ] || [ "$getubuntuversion" < "16" ]; then
if [ "$getubuntuversion" = "16" ] || [ "$getubuntuversion" -le "16" ]; then
echo "============="
echo "Shinobi - Get FFMPEG 3.x from ppa:jonathonf/ffmpeg-3"
sudo add-apt-repository ppa:jonathonf/ffmpeg-3 -y
@ -100,7 +100,7 @@ sudo npm install --unsafe-perm
# sudo npm audit fix --force
echo "============="
echo "Shinobi - Install PM2"
sudo npm install pm2@3.0.0 -g
sudo npm install pm2@latest -g
echo "Shinobi - Finished"
sudo chmod -R 755 .
touch INSTALL/installed.txt

27
INSTALL/ubuntu-touchless.sh
View File

@ -22,7 +22,7 @@ if [ "$disableIpv6" = "y" ] || [ "$disableIpv6" = "Y" ]; then
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.lo.disable_ipv6=1
fi
if [ "$getubuntuversion" = "18" ] || [ "$getubuntuversion" > "18" ]; then
if [ "$getubuntuversion" = "18" ] || [ "$getubuntuversion" -gt "18" ]; then
apt install sudo wget -y
sudo apt install -y software-properties-common
sudo add-apt-repository universe -y
@ -53,25 +53,20 @@ if ! [ -x "$(command -v ifconfig)" ]; then
echo "Shinobi - Installing Net-Tools"
sudo apt install net-tools -y
fi
if ! [ -x "$(command -v node)" ]; then
echo "============="
echo "Shinobi - Installing Node.js"
wget https://deb.nodesource.com/setup_12.x
chmod +x setup_12.x
./setup_12.x
sudo apt install nodejs -y
sudo apt install node-pre-gyp -y
rm setup_12.x
else
echo "Node.js Found..."
echo "Version : $(node -v)"
fi
echo "============="
echo "Shinobi - Installing Node.js"
wget https://deb.nodesource.com/setup_16.x
chmod +x setup_16.x
./setup_16.x
sudo apt install nodejs -y
sudo apt install node-pre-gyp -y
rm setup_16.x
if ! [ -x "$(command -v npm)" ]; then
sudo apt install npm -y
fi
sudo apt install make zip -y
if ! [ -x "$(command -v ffmpeg)" ]; then
if [ "$getubuntuversion" = "16" ] || [ "$getubuntuversion" < "16" ]; then
if [ "$getubuntuversion" = "16" ] || [ "$getubuntuversion" -le "16" ]; then
echo "============="
echo "Shinobi - Get FFMPEG 3.x from ppa:jonathonf/ffmpeg-3"
sudo add-apt-repository ppa:jonathonf/ffmpeg-3 -y
@ -105,7 +100,7 @@ sudo npm install --unsafe-perm
# sudo npm audit fix --force
echo "============="
echo "Shinobi - Install PM2"
sudo npm install pm2@3.0.0 -g
sudo npm install pm2@latest -g
echo "Shinobi - Finished"
sudo chmod -R 755 .
touch INSTALL/installed.txt

14
INSTALL/ubuntu.sh
View File

@ -13,7 +13,7 @@ getubuntuversion=$(lsb_release -r | awk '{print $2}' | cut -d . -f1)
echo "============="
echo " Ubuntu Version: $getubuntuversion"
echo "============="
if [ "$getubuntuversion" = "18" ] || [ "$getubuntuversion" > "18" ]; then
if [ "$getubuntuversion" = "18" ] || [ "$getubuntuversion" -gt "18" ]; then
apt install sudo wget -y
sudo apt install -y software-properties-common
sudo add-apt-repository universe -y
@ -47,11 +47,11 @@ fi
if ! [ -x "$(command -v node)" ]; then
echo "============="
echo "Shinobi - Installing Node.js"
wget https://deb.nodesource.com/setup_12.x
chmod +x setup_12.x
./setup_12.x
wget https://deb.nodesource.com/setup_16.x
chmod +x setup_16.x
./setup_16.x
sudo apt install nodejs -y
rm setup_12.x
rm setup_16.x
else
echo "Node.js Found..."
echo "Version : $(node -v)"
@ -61,7 +61,7 @@ if ! [ -x "$(command -v npm)" ]; then
fi
sudo apt install make zip -y
if ! [ -x "$(command -v ffmpeg)" ]; then
if [ "$getubuntuversion" = "16" ] || [ "$getubuntuversion" < "16" ]; then
if [ "$getubuntuversion" = "16" ] || [ "$getubuntuversion" -le "16" ]; then
echo "============="
echo "Shinobi - Get FFMPEG 3.x from ppa:jonathonf/ffmpeg-3"
sudo add-apt-repository ppa:jonathonf/ffmpeg-3 -y
@ -112,7 +112,7 @@ sudo npm install --unsafe-perm
# sudo npm audit fix --force
echo "============="
echo "Shinobi - Install PM2"
sudo npm install pm2@3.0.0 -g
sudo npm install pm2@latest -g
echo "Shinobi - Finished"
sudo chmod -R 755 .
touch INSTALL/installed.txt

104
LICENSE.md
View File

@ -42,17 +42,41 @@ representative at support@shinobi.systems.
#### Commercial Uses
- Selling usage of the software
- Selling hardware with the software on it
- Using the software in locations that engage in the buying and selling of goods and services
As of 2022-07-12 the noted situations below are seen the same as "Commercial Use".
- 25 Active Monitor Rule : Having at least 25 Active Monitors and not a Primary School or Secondary School. Does not apply to Personal Use.
- 150 Active Monitor Rule : If you have more than 150 Active Monitors please contact support for an Enterprise License. The retail Shinobi Pro license will not be applicable for these installations.
- Used on a Device that was part of a commercial transaction. This can be, but not limited to, being sold or provided additionally to a sale.
#### Conditions for Free (Unpaid) use.
If you fall under any conditions for "Commercial Use" none of the following can apply unless otherwise noted.
- Personal use
- Use in a non commercial area
- Used for non commercial purposes
- When used for research or educational purposes
- Testing Purposes
- Usage by Educational institutions
- Use for Emergency Services and facilties associated like Search and Rescue Services or
Ambulance Services
- Use in Health Care facility like a hospital or walk-in clinic
- Usage by Primary Education or Secondary Education Schools
- Use for Emergency Services and facilities associated like Search and Rescue Services or
Ambulance Services with less than 50 cameras
- Use in Health Care facility like a hospital or walk-in clinic with less than 100 cameras
- Schools and Organizations that have been given exemption
As of 2022-07-12 If you are an organization that falls under one or more of these conditions for free use then you must display that you are using Shinobi Systems software as "shinobi.video". This can be physically on premise or on your organization's public web page.
Falling under the conditions for Free use does not guarantee you a License Key. It simply means you are allowed to use it without paying or trading for that use.
#### Unsure if your use is commercial or non-commercial
Please contact us through the Live Chat of our website and we can negotiate or make exceptions based on the circumstances of your use case.
#### Registration
Schools and Resellers must register with Shinobi Systems at https://licenses.shinobi.video.
In your account please fill out the "Apply for a School License" form or "Apply to be a Reseller" form.
#### Support Services.
The Maintenance and Support Service shall be contracted and provided as per selected plan
@ -68,7 +92,7 @@ This software is property of Shinobi Systems. LICENSEE must keep all copyright n
unchanged.
#### Modification of this Software Product.
LICENSEE may modify code for but must provide these changes upon request from
LICENSEE may modify code for but must provide these changes upon request from
Shinobi Systems or an authorized Shinobi representative. LICENSEE may not alter or change
copyright notices. All code changes by LICENSEE shall fall under the copyright of Shinobi
Systems in the case code modified by LICENSEE is integrated into the official Shinobi code base.
@ -102,29 +126,16 @@ Courthouse Vancouver Robson Square
#### List of Included Software
This list is completed to best of our knowledge.
This list is completed to the best of our knowledge.
Node.js - https://nodejs.org/en/
MariaDB - https://mariadb.org/
FFmpeg - https://www.ffmpeg.org/
request - https://www.npmjs.com/package/request
Express (npm) - https://expressjs.com/ https://www.npmjs.com/package/express
EJS (npm) - http://ejs.co/ https://www.npmjs.com/package/ejs
pam-diff (npm) (Motion Detector) - https://github.com/kevinGodell/pam-diff
pipe2pam (npm) (for pam-diff) - https://github.com/kevinGodell/pipe2pam
mp4frag (npm) (Poseidon's main engine) - https://github.com/kevinGodell/mp4frag
pam-diff (Motion Detector) - https://github.com/kevinGodell/pam-diff
pipe2pam (for pam-diff) - https://github.com/kevinGodell/pipe2pam
mp4frag (Poseidon's main engine) - https://github.com/kevinGodell/mp4frag
mse-live-player (for mp4frag) - https://github.com/kevinGodell/mse-live-player
pipe2jpeg (npm) - https://github.com/kevinGodell/pipe2jpeg
webdav (npm) - https://www.npmjs.com/package/webdav
jsonfile (npm) - https://www.npmjs.com/package/jsonfile
connectionTester (npm) - https://www.npmjs.com/package/connectionTester
node-onvif (npm) - https://www.npmjs.com/package/node-onvif
knex (npm) - https://www.npmjs.com/package/knex
nodemailer (npm) - https://www.npmjs.com/package/nodemailer
mysql (npm) - https://www.npmjs.com/package/mysql
sqlite3 (npm) - https://www.npmjs.com/package/sqlite3
ldapauth-fork (npm) - https://www.npmjs.com/package/ldapauth-fork
http-proxy (npm) - https://www.npmjs.com/package/http-proxy
pipe2jpeg - https://github.com/kevinGodell/pipe2jpeg
hls.js - https://github.com/video-dev/hls.js/
flv.js - https://github.com/Bilibili/flv.js/
Material Design Lite - https://getmdl.io/
@ -142,3 +153,50 @@ Courthouse Vancouver Robson Square
Moment.js - https://momentjs.com/
Livestamp.js - https://mattbradley.github.io/livestampjs/
Lodash - https://lodash.com/
**npmjs.com packages**
async - https://www.npmjs.com/package/async
aws-sdk - https://www.npmjs.com/package/aws-sdk
backblaze-b2 - https://www.npmjs.com/package/backblaze-b2
body-parser - https://www.npmjs.com/package/body-parser
bson - https://www.npmjs.com/package/bson
connection-tester - https://www.npmjs.com/package/connection-tester
cws - https://www.npmjs.com/package/cws
digest-fetch - https://www.npmjs.com/package/digest-fetch
discord.js - https://www.npmjs.com/package/discord.js
ejs - https://www.npmjs.com/package/ejs
express - https://www.npmjs.com/package/express
express-fileupload - https://www.npmjs.com/package/express-fileupload
fs-extra - https://www.npmjs.com/package/fs-extra
ftp-srv - https://www.npmjs.com/package/ftp-srv
googleapis - https://www.npmjs.com/package/googleapis
http-proxy - https://www.npmjs.com/package/http-proxy
jsonfile - https://www.npmjs.com/package/jsonfile
knex - https://www.npmjs.com/package/knex
ldapauth-fork - https://www.npmjs.com/package/ldapauth-fork
moment - https://www.npmjs.com/package/moment
mp4frag - https://www.npmjs.com/package/mp4frag
mqtt - https://www.npmjs.com/package/mqtt
mysql - https://www.npmjs.com/package/mysql
node-abort-controller - https://www.npmjs.com/package/node-abort-controller
node-fetch - https://www.npmjs.com/package/node-fetch
node-onvif-events - https://www.npmjs.com/package/node-onvif-events
node-ssh - https://www.npmjs.com/package/node-ssh
node-telegram-bot-api - https://www.npmjs.com/package/node-telegram-bot-api
nodemailer - https://www.npmjs.com/package/nodemailer
pam-diff - https://www.npmjs.com/package/pam-diff
path - https://www.npmjs.com/package/path
pipe2pam - https://www.npmjs.com/package/pipe2pam
pixel-change - https://www.npmjs.com/package/pixel-change
pushover-notifications - https://www.npmjs.com/package/pushover-notifications
sat - https://www.npmjs.com/package/sat
shinobi-onvif - https://www.npmjs.com/package/shinobi-onvif
shinobi-sound-detection - https://www.npmjs.com/package/shinobi-sound-detection
shinobi-zwave - https://www.npmjs.com/package/shinobi-zwave
smtp-server - https://www.npmjs.com/package/smtp-server
socket.io - https://www.npmjs.com/package/socket.io
socket.io-client - https://www.npmjs.com/package/socket.io-client
tree-kill - https://www.npmjs.com/package/tree-kill
unzipper - https://www.npmjs.com/package/unzipper
webdav-fs - https://www.npmjs.com/package/webdav-fs

25
UPDATE-v2-to-v3.sh Normal file
View File

@ -0,0 +1,25 @@
echo "============="
echo "Updating Node.js to version 16..."
wget https://deb.nodesource.com/setup_16.x
chmod +x setup_16.x
./setup_16.x
apt install nodejs -y
apt install node-pre-gyp -y
rm setup_16.x
npm i npm -g
echo "============="
echo "Updating PM2..."
npm install pm2@latest -g
echo "============="
echo "Updating Shinobi dependencies..."
git reset --hard
git pull
rm -rf node_modules
npm install
echo "============="
echo "Restarting PM2..."
pm2 update
pm2 restart camera.js

10
camera.js
View File

@ -8,7 +8,7 @@
// Subscribe : https://licenses.shinobi.video/subscribe?planSubscribe=plan_G31AZ9mknNCa6z
// PayPal : paypal@m03.ca
//
const io = new (require('socket.io'))()
const io = new (require('socket.io').Server)()
//process handlers
const s = require('./libs/process.js')(process,__dirname)
//load extender functions
@ -33,6 +33,10 @@ require('./libs/ffmpeg.js')(s,config,lang, async () => {
require('./libs/auth.js')(s,config,lang)
//express web server with ejs
const app = require('./libs/webServer.js')(s,config,lang,io)
//data port
require('./libs/dataPort.js')(s,config,lang,app,io)
//page layout load
require('./libs/definitions.js')(s,config,lang,app,io)
//web server routes : page handling..
require('./libs/webServerPaths.js')(s,config,lang,app,io)
//web server routes for streams : streams..
@ -67,8 +71,6 @@ require('./libs/ffmpeg.js')(s,config,lang, async () => {
require('./libs/rtmpserver.js')(s,config,lang)
//dropInEvents server (file manipulation to create event trigger)
require('./libs/dropInEvents.js')(s,config,lang,app,io)
//form fields to drive the internals
require('./libs/definitions.js')(s,config,lang,app,io)
//notifiers : discord..
require('./libs/notification.js')(s,config,lang)
//branding functions and config defaults
@ -91,4 +93,6 @@ require('./libs/ffmpeg.js')(s,config,lang, async () => {
await require('./libs/startup.js')(s,config,lang)
//p2p, commander
require('./libs/commander.js')(s,config,lang,app)
//cron
require('./libs/cron.js')(s,config,lang)
})

6
conf.sample.json
View File

@ -1,8 +1,14 @@
{
"port": 8080,
"debugLog": false,
"videosDir": "__DIR__/videos",
"passwordType": "sha256",
"detectorMergePamRegionTriggers": true,
"wallClockTimestampAsDefault": true,
"useBetterP2P": true,
"smtpServerOptions": {
"allowInsecureAuth": true
},
"addStorage": [
{"name":"second","path":"__DIR__/videos2"}
],

531
cron.js
View File

@ -1,524 +1,9 @@
process.on('uncaughtException', function (err) {
console.error('uncaughtException',err);
});
const fs = require('fs');
const path = require('path');
const moment = require('moment');
const exec = require('child_process').exec;
const spawn = require('child_process').spawn;
const config = require(process.cwd() + '/conf.json')
//set option defaults
s = {
mainDirectory: process.cwd(),
utcOffset: moment().utcOffset()
};
if(config.cron===undefined)config.cron={};
if(config.cron.deleteOld===undefined)config.cron.deleteOld=true;
if(config.cron.deleteOrphans===undefined)config.cron.deleteOrphans=false;
if(config.cron.deleteNoVideo===undefined)config.cron.deleteNoVideo=true;
if(config.cron.deleteNoVideoRecursion===undefined)config.cron.deleteNoVideoRecursion=false;
if(config.cron.deleteOverMax===undefined)config.cron.deleteOverMax=true;
if(config.cron.deleteLogs===undefined)config.cron.deleteLogs=true;
if(config.cron.deleteEvents===undefined)config.cron.deleteEvents=true;
if(config.cron.deleteFileBins===undefined)config.cron.deleteFileBins=true;
if(config.cron.interval===undefined)config.cron.interval=1;
if(config.databaseType===undefined){config.databaseType='mysql'}
if(config.databaseLogs===undefined){config.databaseLogs=false}
if(config.useUTC===undefined){config.useUTC=false}
if(config.debugLog===undefined){config.debugLog=false}
if(!config.ip||config.ip===''||config.ip.indexOf('0.0.0.0')>-1)config.ip='localhost';
if(!config.videosDir)config.videosDir = s.mainDirectory + '/videos/';
if(!config.binDir){config.binDir = s.mainDirectory + '/fileBin/'}
const {
checkCorrectPathEnding,
generateRandomId,
formattedTime,
localToUtc,
} = require('./libs/basic/utils.js')(s.mainDirectory)
const {
sqlDate,
knexQuery,
knexQueryPromise,
initiateDatabaseEngine
} = require('./libs/sql/utils.js')(s,config)
var theCronInterval = null
const overlapLocks = {}
const alreadyDeletedRowsWithNoVideosOnStart = {}
const videoDirectory = checkCorrectPathEnding(config.videosDir)
const fileBinDirectory = checkCorrectPathEnding(config.binDir)
s.debugLog = function(arg1,arg2){
if(config.debugLog === true){
if(!arg2)arg2 = ''
console.log(arg1,arg2)
}
const { parentPort, isMainThread } = require('worker_threads');
if(isMainThread){
console.log(`Shinobi now runs cron.js as a worker process from camera.js.`)
console.error(`Shinobi now runs cron.js as a worker process from camera.js.`)
setInterval(() => {
// console.log(`Please turn off cron.js process.`)
},1000 * 60 * 60 * 24 * 7)
return;
}
const connectToMainProcess = () => {
const io = require('socket.io-client')('ws://'+config.ip+':'+config.port,{
transports:['websocket']
});
io.on('connect',function(d){
postMessage({
f: 'init',
time: moment()
})
})
io.on('f',function(d){
//command from main process
switch(d.f){
case'start':case'restart':
setIntervalForCron()
break;
case'stop':
clearCronInterval()
break;
}
})
return io
}
const postMessage = (x) => {
x.cronKey = config.cron.key;
return io.emit('cron',x)
}
const sendToWebSocket = (x,y) => {
//emulate master socket emitter
postMessage({f:'s.tx',data:x,to:y})
}
const deleteVideo = (x) => {
postMessage({f:'s.deleteVideo',file:x})
}
const deleteFileBinEntry = (x) => {
postMessage({f:'s.deleteFileBinEntry',file:x})
}
const setDiskUsedForGroup = (groupKey,size,target,videoRow) => {
postMessage({f:'s.setDiskUsedForGroup', ke: groupKey, size: size, target: target, videoRow: videoRow})
}
const getVideoDirectory = function(e){
if(e.mid&&!e.id){e.id=e.mid};
if(e.details&&(e.details instanceof Object)===false){
try{e.details=JSON.parse(e.details)}catch(err){}
}
if(e.details.dir&&e.details.dir!==''){
return checkCorrectPathEnding(e.details.dir)+e.ke+'/'+e.id+'/'
}else{
return videoDirectory + e.ke + '/' + e.id + '/'
}
}
const getFileBinDirectory = function(e){
if(e.mid && !e.id){e.id = e.mid}
return fileBinDirectory + e.ke + '/' + e.id + '/'
}
//filters set by the user in their dashboard
//deleting old videos is part of the filter - config.cron.deleteOld
const checkFilterRules = function(v){
return new Promise((resolve,reject) => {
//filters
v.d.filters = v.d.filters ? v.d.filters : {}
s.debugLog('Checking Basic Filters...')
var keys = Object.keys(v.d.filters)
if(keys.length>0){
keys.forEach(function(m,current){
// b = filter
var b = v.d.filters[m];
s.debugLog(b)
if(b.enabled==="1"){
const whereQuery = [
['ke','=',v.ke],
['status','!=',"0"],
['details','NOT LIKE','%"archived":"1"%'],
]
b.where.forEach(function(condition){
if(condition.p1 === 'ke'){condition.p3 = v.ke}
whereQuery.push([condition.p1,condition.p2 || '=',condition.p3])
})
knexQuery({
action: "select",
columns: "*",
table: "Videos",
where: whereQuery,
orderBy: [b.sort_by,b.sort_by_direction.toLowerCase()],
limit: b.limit
},(err,r) => {
if(r && r[0]){
if(r.length > 0 || config.debugLog === true){
postMessage({f:'filterMatch',msg:r.length+' SQL rows match "'+m+'"',ke:v.ke,time:moment()})
}
b.cx={
f:'filters',
name:b.name,
videos:r,
time:moment(),
ke:v.ke,
id:b.id
};
if(b.archive==="1"){
postMessage({f:'filters',ff:'archive',videos:r,time:moment(),ke:v.ke,id:b.id});
}else if(b.delete==="1"){
postMessage({f:'filters',ff:'delete',videos:r,time:moment(),ke:v.ke,id:b.id});
}
if(b.email==="1"){
b.cx.ff='email';
b.cx.delete=b.delete;
b.cx.mail=v.mail;
b.cx.execute=b.execute;
b.cx.query=b.sql;
postMessage(b.cx);
}
if(b.execute&&b.execute!==""){
postMessage({f:'filters',ff:'execute',execute:b.execute,time:moment()});
}
}
})
}
if(current===keys.length-1){
//last filter
resolve()
}
})
}else{
//no filters
resolve()
}
})
}
const deleteVideosByDays = async (v,days,addedQueries) => {
const groupKey = v.ke;
const whereQuery = [
['ke','=',v.ke],
['time','<', sqlDate(days+' DAY')],
addedQueries
]
const selectResponse = await knexQueryPromise({
action: "select",
columns: "*",
table: "Videos",
where: whereQuery
})
const videoRows = selectResponse.rows
let affectedRows = 0
if(videoRows.length > 0){
let clearSize = 0;
var i;
for (i = 0; i < videoRows.length; i++) {
const row = videoRows[i];
const dir = getVideoDirectory(row)
const filename = formattedTime(row.time) + '.' + row.ext
try{
await fs.promises.unlink(dir + filename)
const fileSizeMB = row.size / 1048576;
setDiskUsedForGroup(groupKey,-fileSizeMB,null,row)
sendToWebSocket({
f: 'video_delete',
filename: filename + '.' + row.ext,
mid: row.mid,
ke: row.ke,
time: row.time,
end: formattedTime(new Date,'YYYY-MM-DD HH:mm:ss')
},'GRP_' + row.ke)
}catch(err){
console.log('Video Delete Error',row)
console.log(err)
}
}
const deleteResponse = await knexQueryPromise({
action: "delete",
table: "Videos",
where: whereQuery
})
affectedRows = deleteResponse.rows || 0
}
return {
ok: true,
affectedRows: affectedRows,
}
}
const deleteOldVideos = async (v) => {
// v = group, admin user
if(config.cron.deleteOld === true){
const daysOldForDeletion = v.d.days && !isNaN(v.d.days) ? parseFloat(v.d.days) : 5
const monitorsIgnored = []
const monitorsResponse = await knexQueryPromise({
action: "select",
columns: "*",
table: "Monitors",
where: [
['ke','=',v.ke],
]
})
const monitorRows = monitorsResponse.rows
var i;
for (i = 0; i < monitorRows.length; i++) {
const monitor = monitorRows[i]
const monitorId = monitor.mid
const details = JSON.parse(monitor.details);
const monitorsMaxDaysToKeep = !isNaN(details.max_keep_days) ? parseFloat(details.max_keep_days) : null
if(monitorsMaxDaysToKeep){
const { affectedRows } = await deleteVideosByDays(v,monitorsMaxDaysToKeep,['mid','=',monitorId])
const hasDeletedRows = affectedRows && affectedRows.length > 0;
if(hasDeletedRows || config.debugLog === true){
postMessage({
f: 'deleteOldVideosByMonitorId',
msg: `${affectedRows} SQL rows older than ${monitorsMaxDaysToKeep} days deleted`,
ke: v.ke,
mid: monitorId,
time: moment(),
})
}
monitorsIgnored.push(['mid','!=',monitorId])
}
}
const { affectedRows } = await deleteVideosByDays(v,daysOldForDeletion,monitorsIgnored)
const hasDeletedRows = affectedRows && affectedRows.length > 0;
if(hasDeletedRows || config.debugLog === true){
postMessage({
f: 'deleteOldVideos',
msg: `${affectedRows} SQL rows older than ${daysOldForDeletion} days deleted`,
ke: v.ke,
time: moment(),
})
}
}
}
//database rows with no videos in the filesystem
const deleteRowsWithNoVideo = function(v){
return new Promise((resolve,reject) => {
if(
config.cron.deleteNoVideo===true&&(
config.cron.deleteNoVideoRecursion===true||
(config.cron.deleteNoVideoRecursion===false&&!alreadyDeletedRowsWithNoVideosOnStart[v.ke])
)
){
alreadyDeletedRowsWithNoVideosOnStart[v.ke]=true;
knexQuery({
action: "select",
columns: "*",
table: "Videos",
where: [
['ke','=',v.ke],
['status','!=','0'],
['details','NOT LIKE','%"archived":"1"%'],
['time','<', sqlDate('10 MINUTE')],
]
},(err,evs) => {
if(evs && evs[0]){
const videosToDelete = [];
evs.forEach(function(ev){
var filename
var details
try{
details = JSON.parse(ev.details)
}catch(err){
if(details instanceof Object){
details = ev.details
}else{
details = {}
}
}
var dir = getVideoDirectory(ev)
filename = formattedTime(ev.time)+'.'+ev.ext
fileExists = fs.existsSync(dir+filename)
if(fileExists !== true){
deleteVideo(ev)
sendToWebSocket({f:'video_delete',filename:filename+'.'+ev.ext,mid:ev.mid,ke:ev.ke,time:ev.time,end: formattedTime(new Date,'YYYY-MM-DD HH:mm:ss')},'GRP_'+ev.ke);
}
});
if(videosToDelete.length > 0 || config.debugLog === true){
postMessage({f:'deleteNoVideo',msg:videosToDelete.length+' SQL rows with no file deleted',ke:v.ke,time:moment()})
}
}
setTimeout(function(){
resolve()
},3000)
})
}else{
resolve()
}
})
}
//info about what the application is doing
const deleteOldLogs = function(v){
return new Promise((resolve,reject) => {
const daysOldForDeletion = v.d.log_days && !isNaN(v.d.log_days) ? parseFloat(v.d.log_days) : 10
if(config.cron.deleteLogs === true && daysOldForDeletion !== 0){
knexQuery({
action: "delete",
table: "Logs",
where: [
['ke','=',v.ke],
['time','<', sqlDate(daysOldForDeletion + ' DAY')],
]
},(err,rrr) => {
resolve()
if(err)return console.error(err);
if(rrr && rrr > 0 || config.debugLog === true){
postMessage({f:'deleteLogs',msg: rrr + ' SQL rows older than ' + daysOldForDeletion + ' days deleted',ke:v.ke,time:moment()})
}
})
}else{
resolve()
}
})
}
//events - motion, object, etc. detections
const deleteOldEvents = function(v){
return new Promise((resolve,reject) => {
const daysOldForDeletion = v.d.event_days && !isNaN(v.d.event_days) ? parseFloat(v.d.event_days) : 10
if(config.cron.deleteEvents === true && daysOldForDeletion !== 0){
knexQuery({
action: "delete",
table: "Events",
where: [
['ke','=',v.ke],
['time','<', sqlDate(daysOldForDeletion + ' DAY')],
]
},(err,rrr) => {
resolve()
if(err)return console.error(err);
if(rrr && rrr > 0 || config.debugLog === true){
postMessage({f:'deleteEvents',msg:rrr + ' SQL rows older than ' + daysOldForDeletion + ' days deleted',ke:v.ke,time:moment()})
}
})
}else{
resolve()
}
})
}
//event counts
const deleteOldEventCounts = function(v){
return new Promise((resolve,reject) => {
const daysOldForDeletion = v.d.event_days && !isNaN(v.d.event_days) ? parseFloat(v.d.event_days) : 10
if(config.cron.deleteEvents === true && daysOldForDeletion !== 0){
knexQuery({
action: "delete",
table: "Events Counts",
where: [
['ke','=',v.ke],
['time','<', sqlDate(daysOldForDeletion + ' DAY')],
]
},(err,rrr) => {
resolve()
if(err && err.code !== 'ER_NO_SUCH_TABLE')return console.error(err);
if(rrr && rrr > 0 || config.debugLog === true){
postMessage({f:'deleteEvents',msg:rrr + ' SQL rows older than ' + daysOldForDeletion + ' days deleted',ke:v.ke,time:moment()})
}
})
}else{
resolve()
}
})
}
//check for temporary files (special archive)
const deleteOldFileBins = function(v){
return new Promise((resolve,reject) => {
const daysOldForDeletion = v.d.fileBin_days && !isNaN(v.d.fileBin_days) ? parseFloat(v.d.fileBin_days) : 10
if(config.cron.deleteFileBins === true && daysOldForDeletion !== 0){
var fileBinQuery = " FROM Files WHERE ke=? AND `time` < ?";
knexQuery({
action: "select",
columns: "*",
table: "Files",
where: [
['ke','=',v.ke],
['time','<', sqlDate(daysOldForDeletion + ' DAY')],
]
},(err,files) => {
if(files && files[0]){
//delete the files
files.forEach(function(file){
deleteFileBinEntry(file)
})
if(config.debugLog === true){
postMessage({
f: 'deleteFileBins',
msg: files.length + ' files older than ' + daysOldForDeletion + ' days deleted',
ke: v.ke,
time: moment()
})
}
}
resolve()
})
}else{
resolve()
}
})
}
//user processing function
const processUser = async (v) => {
if(!v){
//no user object given, end of group list
return
}
s.debugLog(`Group Key : ${v.ke}`)
s.debugLog(`Owner : ${v.mail}`)
if(!overlapLocks[v.ke]){
s.debugLog(`Checking...`)
overlapLocks[v.ke] = true
v.d = JSON.parse(v.details);
try{
await deleteOldVideos(v)
s.debugLog('--- deleteOldVideos Complete')
await deleteOldLogs(v)
s.debugLog('--- deleteOldLogs Complete')
await deleteOldFileBins(v)
s.debugLog('--- deleteOldFileBins Complete')
await deleteOldEvents(v)
s.debugLog('--- deleteOldEvents Complete')
await deleteOldEventCounts(v)
s.debugLog('--- deleteOldEventCounts Complete')
await checkFilterRules(v)
s.debugLog('--- checkFilterRules Complete')
await deleteRowsWithNoVideo(v)
s.debugLog('--- deleteRowsWithNoVideo Complete')
}catch(err){
console.log(`Failed to Complete User : ${v.mail}`)
console.log(err)
}
//done user, unlock current, and do next
overlapLocks[v.ke] = false;
s.debugLog(`Complete Checking... ${v.mail}`)
}else{
s.debugLog(`Locked, Skipped...`)
}
}
//recursive function
const setIntervalForCron = function(){
clearCronInterval()
// theCronInterval = setInterval(doCronJobs,1000 * 10)
theCronInterval = setInterval(doCronJobs,parseFloat(config.cron.interval)*60000*60)
}
const clearCronInterval = function(){
clearInterval(theCronInterval)
}
const doCronJobs = function(){
postMessage({
f: 'start',
time: moment()
})
knexQuery({
action: "select",
columns: "ke,uid,details,mail",
table: "Users",
where: [
['details','NOT LIKE','%"sub"%'],
]
}, async (err,rows) => {
if(err){
console.error(err)
}
if(rows.length > 0){
var i;
for (i = 0; i < rows.length; i++) {
await processUser(rows[i])
}
}
})
}
initiateDatabaseEngine()
const io = connectToMainProcess()
setIntervalForCron()
doCronJobs()
console.log('Shinobi : cron.js started')

5098
definitions/en_CA.js → definitions/base.js
View File

File diff suppressed because it is too large Load Diff

2146
languages/ar.json
View File

File diff suppressed because it is too large Load Diff

1692
languages/cs.json Normal file
View File

File diff suppressed because it is too large Load Diff

779
languages/en_CA.json
View File

File diff suppressed because it is too large Load Diff

1652
languages/it.json Normal file
View File

File diff suppressed because it is too large Load Diff

2145
languages/ru.json
View File

File diff suppressed because it is too large Load Diff

1649
languages/tr.json Normal file
View File

File diff suppressed because it is too large Load Diff

1647
languages/vi.json Normal file
View File

File diff suppressed because it is too large Load Diff

95
libs/auth.js
View File

@ -78,24 +78,21 @@ module.exports = function(s,config,lang){
var isSessionKey = false
if(apiKey){
var sessionKey = params.auth
createSession(apiKey,{
auth: sessionKey,
permissions: s.parseJSON(apiKey.details),
details: {}
})
getUserByUid(apiKey,'mail,details',function(err,user){
if(user){
try{
editSession({
auth: sessionKey
},{
mail: user.mail,
details: s.parseJSON(user.details),
lang: s.getLanguageFile(user.details.lang)
})
}catch(er){
console.log('FAILED TO EDIT',er)
}
createSession(apiKey,{
auth: sessionKey,
permissions: s.parseJSON(apiKey.details),
mail: user.mail,
details: s.parseJSON(user.details),
lang: s.getLanguageFile(user.details.lang)
})
}else{
createSession(apiKey,{
auth: sessionKey,
permissions: s.parseJSON(apiKey.details),
details: {}
})
}
callback(err,s.api[params.auth])
})
@ -132,9 +129,7 @@ module.exports = function(s,config,lang){
var editSession = function(user,additionalData){
if(user){
if(!additionalData)additionalData = {}
Object.keys(additionalData).forEach(function(value,key){
s.api[user.auth][key] = value
})
Object.assign(s.api[user.auth], additionalData)
}
}
var failHttpAuthentication = function(res,req,message){
@ -190,39 +185,37 @@ module.exports = function(s,config,lang){
activeSession.lang = s.copySystemDefaultLanguage()
}
onSuccessComplete(activeSession)
}else{
if(s.api[params.auth] && s.api[params.auth].details){
var activeSession = s.api[params.auth]
onSuccess(activeSession)
if(activeSession.timeout){
resetActiveSessionTimer(activeSession)
}
}else{
if(params.username && params.username !== '' && params.password && params.password !== ''){
loginWithUsernameAndPassword(params,'*',function(err,user){
if(user){
params.auth = user.auth
createSession(user)
resetActiveSessionTimer(s.api[params.auth])
onSuccess(user)
}else{
onFail()
}
})
}else{
loginWithApiKey(params,function(err,user,isSessionKey){
if(isSessionKey)resetActiveSessionTimer(s.api[params.auth])
if(user){
createSession(user,{
auth: params.auth
})
onSuccess(s.api[params.auth])
}else{
onFail()
}
})
}
}else if(s.api[params.auth] && s.api[params.auth].details){
var activeSession = s.api[params.auth]
onSuccess(activeSession)
if(activeSession.timeout){
resetActiveSessionTimer(activeSession)
}
}else if(params.username && params.username !== '' && params.password && params.password !== ''){
loginWithUsernameAndPassword(params,'*',function(err,user){
if(user){
params.auth = user.auth
createSession(user)
resetActiveSessionTimer(s.api[params.auth])
onSuccess(user)
}else{
onFail()
}
})
}else if(params.auth && params.ke){
loginWithApiKey(params,function(err,user,isSessionKey){
if(isSessionKey)resetActiveSessionTimer(s.api[params.auth])
if(user){
createSession(user,{
auth: params.auth
})
onSuccess(s.api[params.auth])
}else{
onFail()
}
})
} else {
onFail()
}
}
//super user authentication handler

13
libs/basic.js
View File

@ -57,12 +57,15 @@ module.exports = function(s,config){
splitted[1] = user + ':' + pass + '@' + splitted[1]
return splitted.join('://')
}
s.checkCorrectPathEnding = function(x){
var length=x.length
if(x.charAt(length-1)!=='/'){
x=x+'/'
s.checkCorrectPathEnding = function(x,reverse){
var newString = `${x}`
var length = x.length
if(reverse && x.charAt(length-1) === '/'){
newString = x.slice(0, -1)
}else if(x.charAt(length-1) !== '/'){
newString = x + '/'
}
return x.replace('__DIR__',s.mainDirectory)
return newString.replace('__DIR__',s.mainDirectory)
}
s.mergeDeep = function(...objects) {
const isObject = obj => obj && typeof obj === 'object';

115
libs/basic/utils.js
View File

@ -1,7 +1,9 @@
const fs = require('fs');
const path = require('path');
const moment = require('moment');
const request = require('request');
const fetch = require('node-fetch');
const { AbortController } = require('node-abort-controller')
const DigestFetch = require('digest-fetch')
module.exports = (processCwd,config) => {
const parseJSON = (string) => {
var parsed
@ -28,6 +30,11 @@ module.exports = (processCwd,config) => {
if(toLowerCase)newString = newString.toLowerCase()
return newString.indexOf(find) > -1
}
function getFileDirectory(filePath){
const fileParts = filePath.split('/')
fileParts.pop();
return fileParts.join('/') + '/';
}
const checkCorrectPathEnding = (x) => {
var length=x.length
if(x.charAt(length-1)!=='/'){
@ -79,6 +86,50 @@ module.exports = (processCwd,config) => {
if(!e){e=new Date};if(!x){x='YYYY-MM-DDTHH-mm-ss'};
return moment(e).format(x);
}
const fetchTimeout = (url, ms, { signal, ...options } = {}) => {
const controller = new AbortController();
const promise = fetch(url, { signal: controller.signal, ...options });
if (signal) signal.addEventListener("abort", () => controller.abort());
const timeout = setTimeout(() => controller.abort(), ms);
return promise.finally(() => clearTimeout(timeout));
}
async function fetchDownloadAndWrite(downloadUrl,outputPath,readFileAfterWrite,options){
const writeStream = fs.createWriteStream(outputPath);
const downloadBuffer = await fetch(downloadUrl,options).then((res) => res.buffer());
writeStream.write(downloadBuffer);
writeStream.end();
if(readFileAfterWrite === 1){
return fs.createReadStream(outputPath)
}else if(readFileAfterWrite === 2){
return downloadBuffer
}
return null
}
function fetchWithAuthentication(requestUrl,options,callback){
let hasDigestAuthEnabled = options.digestAuth;
let theRequester;
const hasUsernameAndPassword = options.username && typeof options.password === 'string'
const requestOptions = {
method : options.method || 'GET'
}
if(typeof options.postData === 'object'){
const formData = new fetch.FormData()
const formKeys = Object.keys(options.postData)
formKeys.forEach(function(key){
const value = formKeys[key]
formData.set(key, value)
})
requestOptions.body = formData
}
if(hasUsernameAndPassword && hasDigestAuthEnabled){
theRequester = (new DigestFetch(options.username, options.password)).fetch
}else if(hasUsernameAndPassword){
theRequester = (new DigestFetch(options.username, options.password, { basic: true })).fetch
}else{
theRequester = fetch
}
return theRequester(requestUrl,requestOptions)
}
const checkSubscription = (subscriptionId,callback) => {
function subscriptionFailed(){
console.error('!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!')
@ -92,12 +143,12 @@ module.exports = (processCwd,config) => {
if(subscriptionId && subscriptionId !== 'sub_XXXXXXXXXXXX' && !config.disableOnlineSubscriptionCheck){
var url = 'https://licenses.shinobi.video/subscribe/check?subscriptionId=' + subscriptionId
var hasSubcribed = false
request(url,{
fetchTimeout(url,30000,{
method: 'GET',
timeout: 30000
}, function(err,resp,body){
})
.then(response => response.text())
.then(function(body){
var json = s.parseJSON(body)
if(err)console.log(err,json)
hasSubcribed = json && !!json.ok
var i;
for (i = 0; i < s.onSubscriptionCheckExtensions.length; i++) {
@ -113,8 +164,12 @@ module.exports = (processCwd,config) => {
}else{
subscriptionFailed()
}
}).catch((err) => {
if(err)console.log(err)
subscriptionFailed()
})
}else{
var i;
for (i = 0; i < s.onSubscriptionCheckExtensions.length; i++) {
const extender = s.onSubscriptionCheckExtensions[i]
hasSubcribed = extender(false,{},subscriptionId)
@ -125,10 +180,53 @@ module.exports = (processCwd,config) => {
callback(hasSubcribed)
}
}
function isEven(value) {
if (value%2 == 0)
return true;
else
return false;
}
function asyncSetTimeout(timeoutAmount) {
return new Promise((resolve,reject) => {
setTimeout(function(){
resolve()
},timeoutAmount)
})
}
function copyFile(inputFilePath,outputFilePath) {
const response = {ok: true}
return new Promise((resolve,reject) => {
function failed(err){
response.ok = false
response.err = err
resolve(response)
}
const readStream = fs.createReadStream(inputFilePath)
const writeStream = fs.createWriteStream(outputFilePath)
writeStream.on('finish', () => {
resolve(response)
})
writeStream.on('error', failed)
readStream.on('error', failed)
readStream.pipe(writeStream)
})
}
function hmsToSeconds(str) {
var p = str.split(':'),
s = 0, m = 1;
while (p.length > 0) {
s += m * parseFloat(p.pop(), 10);
m *= 60;
}
return s;
}
return {
parseJSON: parseJSON,
stringJSON: stringJSON,
stringContains: stringContains,
getFileDirectory: getFileDirectory,
checkCorrectPathEnding: checkCorrectPathEnding,
nameToTime: nameToTime,
mergeDeep: mergeDeep,
@ -137,5 +235,12 @@ module.exports = (processCwd,config) => {
localToUtc: localToUtc,
formattedTime: formattedTime,
checkSubscription: checkSubscription,
isEven: isEven,
fetchTimeout: fetchTimeout,
fetchDownloadAndWrite: fetchDownloadAndWrite,
fetchWithAuthentication: fetchWithAuthentication,
asyncSetTimeout: asyncSetTimeout,
copyFile: copyFile,
hmsToSeconds,
}
}

32
libs/basic/websocketTools.js Normal file
View File

@ -0,0 +1,32 @@
const WebSocket = require('cws');
function createWebSocketServer(options){
const theWebSocket = new WebSocket.Server(options ? options : {
noServer: true
});
theWebSocket.broadcast = function(data) {
theWebSocket.clients.forEach((client) => {
try{
client.sendData(data)
}catch(err){
// console.log(err)
}
})
};
return theWebSocket
}
function createWebSocketClient(connectionHost,options){
const clientConnection = new WebSocket(connectionHost, options.engineOptions);
if(options.onMessage){
const onMessage = options.onMessage;
clientConnection.on('message', message => {
const data = JSON.parse(message);
onMessage(data);
});
}
return clientConnection
}
module.exports = {
createWebSocketServer,
createWebSocketClient,
}

41
libs/branding.js
View File

@ -1,15 +1,46 @@
module.exports = function(s,config,lang,app,io){
if(config.showPoweredByShinobi === undefined){config.showPoweredByShinobi=true}
if(config.poweredByShinobi === undefined){config.poweredByShinobi='Powered by Shinobi.Systems'}
if(config.poweredByShinobiClass === undefined){config.poweredByShinobiClass='margin:15px 0 0 0;text-align:center;color:#777;font-family: sans-serif;text-transform: uppercase;letter-spacing: 3;font-size: 8pt;'}
if(config.webPageTitle === undefined){config.webPageTitle='Shinobi'}
if(config.showLoginCardHeader === undefined){config.showLoginCardHeader=true}
if(config.webFavicon === undefined){config.webFavicon='libs/img/icon/favicon.ico'}
if(config.logoLocation76x76 === undefined){config.logoLocation76x76='libs/img/icon/apple-touch-icon-76x76.png'}
if(config.webFavicon === undefined){config.webFavicon = 'libs/img/icon/favicon.ico'}
if(!config.logoLocationAppleTouchIcon)config.logoLocationAppleTouchIcon = 'libs/img/icon/apple-touch-icon.png';
if(!config.logoLocation57x57)config.logoLocation57x57 = 'libs/img/icon/apple-touch-icon-57x57.png';
if(!config.logoLocation72x72)config.logoLocation72x72 = 'libs/img/icon/apple-touch-icon-72x72.png';
if(!config.logoLocation76x76)config.logoLocation76x76 = 'libs/img/icon/apple-touch-icon-76x76.png';
if(!config.logoLocation114x114)config.logoLocation114x114 = 'libs/img/icon/apple-touch-icon-114x114.png';
if(!config.logoLocation120x120)config.logoLocation120x120 = 'libs/img/icon/apple-touch-icon-120x120.png';
if(!config.logoLocation144x144)config.logoLocation144x144 = 'libs/img/icon/apple-touch-icon-144x144.png';
if(!config.logoLocation152x152)config.logoLocation152x152 = 'libs/img/icon/apple-touch-icon-152x152.png';
if(!config.logoLocation196x196)config.logoLocation196x196 = 'libs/img/icon/favicon-196x196.png';
if(config.logoLocation76x76Link === undefined){config.logoLocation76x76Link='https://shinobi.video'}
if(config.logoLocation76x76Style === undefined){config.logoLocation76x76Style='border-radius:50%'}
if(config.loginScreenBackground === undefined){config.loginScreenBackground='assets/img/splash.avif'}
if(config.showLoginSelector === undefined){config.showLoginSelector=true}
if(config.defaultTheme === undefined)config.defaultTheme = 'Ice-v3';
if(config.socialLinks === undefined){
config.socialLinks = [
{
icon: 'home',
href: 'https://shinobi.video',
title: 'Homepage'
},
{
icon: 'facebook',
href: 'https://www.facebook.com/ShinobiCCTV',
title: 'Facebook'
},
{
icon: 'twitter',
href: 'https://twitter.com/ShinobiCCTV',
title: 'Twitter'
},
{
icon: 'youtube',
href: 'https://www.youtube.com/channel/UCbgbBLTK-koTyjOmOxA9msQ',
title: 'YouTube'
}
]
}
s.getConfigWithBranding = function(domain){
var configCopy = Object.assign({},config)
if(config.brandingConfig && config.brandingConfig[domain]){

42
libs/cameraThread/detector.js
View File

@ -1,30 +1,36 @@
module.exports = function(jsonData,pamDiffResponder){
const {
// see libs/detectorUtils.js for more parameters and functions
//
config,
completeMonitorConfig,
groupKey,
monitorId,
monitorName,
monitorDetails,
completeMonitorConfig,
} = require('./libs/monitorUtils.js')(jsonData)
const {
convertRegionsToTiles,
} = require('./libs/tileCutter.js')
const loadDetectorUtils = require('./libs/detectorUtils.js')
let detectorUtils
let onMotionData = null
if(monitorDetails.detector_motion_tile_mode === '1'){
const {
originalCords,
newRegionsBySquares,
} = convertRegionsToTiles(monitorDetails)
jsonData.rawMonitorConfig.details.cords = newRegionsBySquares;
detectorUtils = loadDetectorUtils(jsonData,pamDiffResponder)
detectorUtils.originalCords = originalCords;
onMotionData = detectorUtils.getTileMotionEvent()
}else{
detectorUtils = loadDetectorUtils(jsonData,pamDiffResponder)
}
const {
pamDetectorIsEnabled,
//
attachPamPipeDrivers,
//
getAcceptedTriggers,
getRegionsWithMinimumChange,
getRegionsBelowMaximumChange,
getRegionsWithThresholdMet,
filterTheNoise,
filterTheNoiseFromMultipleRegions,
//
buildDetectorObject,
buildTriggerEvent,
sendDetectedData,
} = require('./libs/detectorUtils.js')(jsonData,pamDiffResponder)
} = detectorUtils;
return function(cameraProcess){
if(pamDetectorIsEnabled){
attachPamPipeDrivers(cameraProcess)
attachPamPipeDrivers(cameraProcess,onMotionData)
}
}
}

10
libs/cameraThread/libs/dataPortConnection.js Normal file
View File

@ -0,0 +1,10 @@
module.exports = function(jsonData,onConnected,onError,onClose){
const config = jsonData.globalInfo.config;
const dataPortToken = jsonData.dataPortToken;
const CWS = require('cws');
const client = new CWS(`ws://localhost:${config.port}/dataPort`);
if(onError)client.on('error',onError);
if(onClose)client.on('close',onClose);
client.on('open',onConnected);
return client;
}

37
libs/cameraThread/libs/detectorUtils.js
View File

@ -1,5 +1,8 @@
const P2P = require('pipe2pam')
let PamDiff = require('pam-diff')
const {
makeBigMatricesFromSmallOnes,
} = require('./tileCutter.js')
module.exports = function(jsonData,pamDiffResponder,alternatePamDiff){
if(alternatePamDiff)PamDiff = alternatePamDiff;
const noiseFilterArray = {};
@ -7,6 +10,7 @@ module.exports = function(jsonData,pamDiffResponder,alternatePamDiff){
const completeMonitorConfig = jsonData.rawMonitorConfig
const groupKey = completeMonitorConfig.ke
const monitorId = completeMonitorConfig.mid
const monitorName = completeMonitorConfig.name
const monitorDetails = completeMonitorConfig.details
const triggerTimer = {}
let regionJson
@ -71,12 +75,12 @@ module.exports = function(jsonData,pamDiffResponder,alternatePamDiff){
pamDiffResponder(detectorObject)
}
}else{
var sendDetectedData = function(detectorObject){
pamDiffResponder.write(Buffer.from(JSON.stringify(detectorObject)))
}
var sendDetectedData = function(detectorObject){
pamDiffResponder.write(Buffer.from(JSON.stringify(detectorObject)))
}
}
function logData(...args){
process.logData(JSON.stringify(args))
process.logData(args)
}
function getRegionsWithMinimumChange(data){
try{
@ -240,6 +244,28 @@ module.exports = function(jsonData,pamDiffResponder,alternatePamDiff){
pamDiff.on('diff',pamAnalyzer)
cameraProcess.stdio[3].pipe(p2p).pipe(pamDiff)
}
function getTileMotionEvent(){
let pamAnalyzer = function(){}
if(monitorDetails.detector_noise_filter === '1'){
pamAnalyzer = async (data) => {
const acceptedTriggers = getAcceptedTriggers(data.trigger)
const passedFilter = await filterTheNoiseFromMultipleRegions(acceptedTriggers)
if(passedFilter){
const mergedTriggers = mergePamTriggers(acceptedTriggers)
mergedTriggers.matrices = makeBigMatricesFromSmallOnes(mergedTriggers.matrices)
buildTriggerEvent(mergedTriggers)
}
}
}else{
pamAnalyzer = (data) => {
const acceptedTriggers = getAcceptedTriggers(data.trigger)
const mergedTriggers = mergePamTriggers(acceptedTriggers)
mergedTriggers.matrices = makeBigMatricesFromSmallOnes(mergedTriggers.matrices)
buildTriggerEvent(mergedTriggers)
}
}
return pamAnalyzer
}
function createPamDiffRegionArray(regions,globalColorThreshold,globalSensitivity,fullFrame){
var pamDiffCompliantArray = [],
arrayForOtherStuff = [],
@ -256,6 +282,7 @@ module.exports = function(jsonData,pamDiffResponder,alternatePamDiff){
if(!region)return false;
region.polygon = [];
region.points.forEach(function(points){
if(!points || isNaN(points[0]) || isNaN(points[1]))return;
var x = parseFloat(points[0]);
var y = parseFloat(points[1]);
if(x < 0)x = 0;
@ -265,6 +292,7 @@ module.exports = function(jsonData,pamDiffResponder,alternatePamDiff){
y: y
})
})
if(region.polygon.length < 4)return logData(`Failed to Create Region : ${monitorName} : ${region.name}`,region.points);
if(region.sensitivity===''){
region.sensitivity = globalSensitivity
}else{
@ -369,6 +397,7 @@ module.exports = function(jsonData,pamDiffResponder,alternatePamDiff){
getPropertiesFromBlob,
createMatricesFromBlobs,
logData,
getTileMotionEvent,
// parameters
pamDetectorIsEnabled,
noiseFilterArray,

14
libs/cameraThread/libs/monitorUtils.js Normal file
View File

@ -0,0 +1,14 @@
module.exports = (jsonData) => {
const completeMonitorConfig = jsonData.rawMonitorConfig
const groupKey = completeMonitorConfig.ke
const monitorId = completeMonitorConfig.mid
const monitorName = completeMonitorConfig.name
const monitorDetails = completeMonitorConfig.details
return {
completeMonitorConfig,
groupKey,
monitorId,
monitorName,
monitorDetails,
}
}

185
libs/cameraThread/libs/tileCutter.js Normal file
View File

@ -0,0 +1,185 @@
const SAT = require('sat')
const V = SAT.Vector;
const P = SAT.Polygon;
const B = SAT.Box;
function intersectionY(edge, y) {
const [[x1, y1], [x2, y2]] = edge;
const dir = Math.sign(y2 - y1);
if (dir && (y1 - y)*(y2 - y) <= 0) return { x: x1 + (y-y1)/(y2-y1) * (x2-x1), dir };
}
function tilePolygon(points, tileSize){
// https://stackoverflow.com/questions/56827208/spilliting-polygon-into-square
const minY = Math.min(...points.map(p => p[1]));
const maxY = Math.max(...points.map(p => p[1]));
const minX = Math.min(...points.map(p => p[0]));
const gridPoints = [];
for (let y = minY; y <= maxY; y += tileSize) {
// Collect x-coordinates where polygon crosses this horizontal line (y)
const cuts = [];
let prev = null;
for (let i = 0; i < points.length; i++) {
const cut = intersectionY([points[i], points[(i+1)%points.length]], y);
if (!cut) continue;
if (!prev || prev.dir !== cut.dir) cuts.push(cut);
prev = cut;
}
if (prev && prev.dir === cuts[0].dir) cuts.pop();
// Now go through those cuts from left to right toggling whether we are in/out the polygon
let dirSum = 0;
let startX = null;
for (let cut of cuts.sort((a, b) => a.x - b.x)) {
dirSum += cut.dir;
if (dirSum % 2) { // Entering polygon
if (startX === null) startX = cut.x;
} else if (startX !== null) { // Exiting polygon
// Genereate grid points on this horizontal line segement
for (let x = minX + Math.ceil((startX - minX) / tileSize)*tileSize; x <= cut.x; x += tileSize) {
gridPoints.push([x, y]);
}
startX = null;
}
}
}
return gridPoints;
}
function convertStringPoints(oldPoints){
// [["0","0"],["0","150"],["300","150"],["300","0"]]
var newPoints = []
oldPoints.forEach((point) => {
newPoints.push([parseInt(point[0]),parseInt(point[1])])
})
return newPoints
}
function createSquares(gridPoints,imgWidth,imgHeight){
var rows = [];
var n = 0;
var curentLine = gridPoints[0][1]
gridPoints.forEach((point) => {
if(!rows[n])rows[n] = []
rows[n].push(point)
if(curentLine !== point[1]){
curentLine = point[1];
++n;
}
});
var squares = [];
rows.forEach((row,n) => {
for (let i = 0; i < row.length; i += 2) {
if(!rows[n + 1] || !row[i + 1])return;
var square = [row[i],row[i + 1],rows[n + 1][i],rows[n + 1][i + 1]]
squares.push(square)
}
})
return squares
}
const getAllSquaresTouchingRegion = function(region,squares){
var matrixPoints = []
var collisions = []
var polyPoints = []
region.points.forEach(function(point){
polyPoints.push(new V(parseInt(point[0]),parseInt(point[1])))
})
var regionPoly = new P(new V(0,0), polyPoints)
squares.forEach(function(squarePoints){
var firstPoint = squarePoints[0]
var thirdPoint = squarePoints[2]
var squareX = firstPoint[0]
var squareY = firstPoint[1]
var squareWidth = thirdPoint[0] - firstPoint[0]
var squareHeight = thirdPoint[1] - firstPoint[1]
var squarePoly = new B(new V(squareX, squareY), squareWidth, squareHeight).toPolygon()
var response = new SAT.Response()
var collided = SAT.testPolygonPolygon(squarePoly, regionPoly, response)
if(collided === true){
collisions.push(squarePoints)
}
})
return collisions
}
function makeBigMatricesFromSmallOnes(matrices){
var bigMatrices = {}
matrices.forEach(function(matrix,n){
const regionName = matrix.tag
if(!bigMatrices[regionName]){
bigMatrices[regionName] = {
tag: regionName,
x: 9999999999,
y: 9999999999,
width: matrices.length > 1 ? matrices[0].width : 0,
height: matrices.length > 1 ? matrices[0].height : 0,
tilesCounted: 0,
confidence: 0,
}
}
var bigMatrix = bigMatrices[regionName];
bigMatrix.x = bigMatrix.x > matrix.x ? matrix.x : bigMatrix.x;
bigMatrix.y = bigMatrix.y > matrix.y ? matrix.y : bigMatrix.y;
const newWidth = matrix.x - bigMatrix.x
const newHeight = matrix.y - bigMatrix.y
bigMatrix.width = bigMatrix.width < matrix.x ? newWidth === 0 ? matrix.width : newWidth : bigMatrix.width;
bigMatrix.height = bigMatrix.height < matrix.y ? newHeight === 0 ? matrix.height : newHeight : bigMatrix.height;
// bigMatrix.tag = matrix.tag;
bigMatrix.confidence += matrix.confidence;
bigMatrix.tilesCounted += 1;
})
let allBigMatrices = Object.values(bigMatrices)
allBigMatrices.forEach(function(matrix,n){
let bigMatrix = allBigMatrices[n]
bigMatrix.averageConfidence = bigMatrix.confidence / bigMatrix.tilesCounted;
})
return allBigMatrices
}
function convertRegionsToTiles(monitorDetails){
let originalCords;
//force full frame detection to be use for tracking blobs
monitorDetails.detector_frame = '1'
monitorDetails.detector_sensitivity = '1'
monitorDetails.detector_color_threshold = monitorDetails.detector_color_threshold || '7'
try{
monitorDetails.cords = JSON.parse(monitorDetails.cords)
}catch(err){
}
originalCords = Object.values(monitorDetails.cords)
const regionKeys = Object.keys(monitorDetails.cords);
const newRegionsBySquares = {}
try{
regionKeys.forEach(function(regionKey){
const region = monitorDetails.cords[regionKey]
const tileSize = parseInt(region.detector_tile_size) || 20;
const gridPoints = tilePolygon([
[0,0],
[0,height],
[width,height],
[width,0]
],tileSize)
const squares = createSquares(gridPoints,width,height)
const squaresInRegion = getAllSquaresTouchingRegion(region,squares)
squaresInRegion.forEach((square,n) => {
newRegionsBySquares[`${regionKey}_${n}`] = Object.assign({},region,{
"points": square
})
})
})
// jsonData.rawMonitorConfig.details.cords = newRegionsBySquares;
}catch(err){
process.logData(err)
}
// detectorUtils.originalCords = originalCords;
return {
originalCords,
newRegionsBySquares,
}
}
module.exports = {
tilePolygon,
createSquares,
convertRegionsToTiles,
getAllSquaresTouchingRegion,
makeBigMatricesFromSmallOnes,
}

50
libs/cameraThread/singleCamera.js
View File

@ -1,5 +1,4 @@
const fs = require('fs')
const request = require('request')
const exec = require('child_process').exec
const spawn = require('child_process').spawn
const isWindows = (process.platform === 'win32' || process.platform === 'win64')
@ -14,9 +13,34 @@ const stdioPipes = jsonData.pipes || []
var newPipes = []
var stdioWriters = [];
var writeToStderr = function(text){
const { fetchTimeout } = require('../basic/utils.js')(process.cwd(),config)
const dataPort = require('./libs/dataPortConnection.js')(jsonData,
// onConnected
() => {
dataPort.send(jsonData.dataPortToken)
},
// onError
(err) => {
writeToStderr([
'dataPort:Connection:Error',
err
])
},
// onClose
(e) => {
writeToStderr([
'dataPort:Connection:Closed',
e
])
})
var writeToStderr = function(argsAsArray){
try{
process.stderr.write(Buffer.from(`${text}`, 'utf8' ))
process.stderr.write(Buffer.from(`${JSON.stringify(argsAsArray)}`, 'utf8' ))
// dataPort.send({
// f: 'debugLog',
// data: argsAsArray,
// })
// stdioWriters[2].write(Buffer.from(`${new Error('writeToStderr').stack}`, 'utf8' ))
}catch(err){
}
@ -111,7 +135,9 @@ writeToStderr('Thread Opening')
if(rawMonitorConfig.details.detector === '1' && rawMonitorConfig.details.detector_pam === '1'){
try{
const attachPamDetector = require(config.monitorDetectorDaemonPath ? config.monitorDetectorDaemonPath : __dirname + '/detector.js')(jsonData,stdioWriters[3])
const attachPamDetector = require(config.monitorDetectorDaemonPath ? config.monitorDetectorDaemonPath : __dirname + '/detector.js')(jsonData,(detectorObject) => {
dataPort.send(JSON.stringify(detectorObject))
})
attachPamDetector(cameraProcess)
}catch(err){
writeToStderr(err.stack)
@ -119,7 +145,6 @@ if(rawMonitorConfig.details.detector === '1' && rawMonitorConfig.details.detecto
}
if(rawMonitorConfig.type === 'jpeg'){
var recordingSnapRequest
var recordingSnapper
var errorTimeout
var errorCount = 0
@ -137,16 +162,11 @@ if(rawMonitorConfig.type === 'jpeg'){
setTimeout(() => {
if(!cameraProcess.stdio[0])return writeToStderr('No Camera Process Found for Snapper');
const captureOne = function(f){
recordingSnapRequest = request({
url: buildMonitorUrl(rawMonitorConfig),
fetchTimeout(buildMonitorUrl(rawMonitorConfig),15000,{
method: 'GET',
encoding: null,
timeout: 15000
},function(err,data){
if(err){
writeToStderr(JSON.stringify(err))
return;
}
})
.then(response => response.text())
.then(function(body){
// writeToStderr(data.body.length)
cameraProcess.stdio[0].write(data.body)
recordingSnapper = setTimeout(function(){
@ -159,7 +179,7 @@ if(rawMonitorConfig.type === 'jpeg'){
delete(errorTimeout)
},3000)
}
}).on('error', function(err){
}).catch(function(err){
++errorCount
clearTimeout(errorTimeout)
errorTimeout = null

407
libs/childNode.js
View File

@ -1,215 +1,147 @@
var fs = require('fs');
var http = require('http');
var https = require('https');
var express = require('express');
const fs = require('fs');
const url = require('url');
const http = require('http');
const https = require('https');
const express = require('express');
const { createWebSocketServer, createWebSocketClient } = require('./basic/websocketTools.js')
module.exports = function(s,config,lang,app,io){
const { cameraDestroy } = require('./monitor/utils.js')(s,config,lang)
//setup Master for childNodes
if(config.childNodes.enabled === true && config.childNodes.mode === 'master'){
if(
config.childNodes.enabled === true &&
config.childNodes.mode === 'master'
){
const {
getIpAddress,
initiateDataConnection,
initiateVideoTransferConnection,
onWebSocketDataFromChildNode,
onDataConnectionDisconnect,
initiateVideoWriteFromChildNode,
initiateTimelapseFrameWriteFromChildNode,
} = require('./childNode/utils.js')(s,config,lang,app,io)
s.childNodes = {};
var childNodeHTTP = express();
var childNodeServer = http.createServer(app);
var childNodeWebsocket = new (require('socket.io'))()
childNodeServer.listen(config.childNodes.port,config.bindip,function(){
console.log(lang.Shinobi+' - CHILD NODE PORT : '+config.childNodes.port);
});
s.debugLog('childNodeWebsocket.attach(childNodeServer)')
childNodeWebsocket.attach(childNodeServer,{
path:'/socket.io',
transports: ['websocket']
});
//send data to child node function (experimental)
s.cx = function(z,y,x){
if(!z.mid && !z.d){
console.error('Missing ID')
}else if(x){
x.broadcast.to(y).emit('c',z)
}else{
childNodeWebsocket.to(y).emit('c',z)
const childNodesConnectionIndex = {};
const childNodeHTTP = express();
const childNodeServer = http.createServer(app);
const childNodeWebsocket = createWebSocketServer();
const childNodeFileRelay = createWebSocketServer();
childNodeServer.on('upgrade', function upgrade(request, socket, head) {
const pathname = url.parse(request.url).pathname;
if (pathname === '/childNode') {
childNodeWebsocket.handleUpgrade(request, socket, head, function done(ws) {
childNodeWebsocket.emit('connection', ws, request)
})
} else if (pathname === '/childNodeFileRelay') {
childNodeFileRelay.handleUpgrade(request, socket, head, function done(ws) {
childNodeFileRelay.emit('connection', ws, request)
})
} else {
socket.destroy();
}
});
const childNodeBindIP = config.childNodes.ip || config.bindip;
childNodeServer.listen(config.childNodes.port,childNodeBindIP,function(){
console.log(lang.Shinobi+' - CHILD NODE SERVER : ' + config.childNodes.port);
});
//send data to child node function
s.cx = function(data,connectionId){
childNodesConnectionIndex[connectionId].sendJson(data)
}
//child Node Websocket
childNodeWebsocket.on('connection', function (cn) {
childNodeWebsocket.on('connection', function (client, req) {
//functions for dispersing work to child servers;
var ipAddress
cn.on('c',function(d){
if(config.childNodes.key.indexOf(d.socketKey) > -1){
if(!cn.shinobi_child&&d.f=='init'){
ipAddress = cn.request.connection.remoteAddress.replace('::ffff:','')+':'+d.port
cn.ip = ipAddress
cn.shinobi_child = 1
cn.tx = function(z){
cn.emit('c',z)
}
if(!s.childNodes[cn.ip]){
s.childNodes[cn.ip] = {}
};
s.childNodes[cn.ip].dead = false
s.childNodes[cn.ip].cnid = cn.id
s.childNodes[cn.ip].cpu = 0
s.childNodes[cn.ip].ip = ipAddress
s.childNodes[cn.ip].activeCameras = {}
d.availableHWAccels.forEach(function(accel){
if(config.availableHWAccels.indexOf(accel) === -1)config.availableHWAccels.push(accel)
})
cn.tx({
f : 'init_success',
childNodes : s.childNodes
})
s.childNodes[cn.ip].coreCount = d.coreCount
}else{
switch(d.f){
case'cpu':
s.childNodes[ipAddress].cpu = d.cpu;
break;
case'sql':
s.sqlQuery(d.query,d.values,function(err,rows){
cn.emit('c',{f:'sqlCallback',rows:rows,err:err,callbackId:d.callbackId});
});
break;
case'knex':
s.knexQuery(d.options,function(err,rows){
cn.emit('c',{f:'sqlCallback',rows:rows,err:err,callbackId:d.callbackId});
});
break;
case'clearCameraFromActiveList':
if(s.childNodes[ipAddress])delete(s.childNodes[ipAddress].activeCameras[d.ke + d.id])
break;
case'camera':
s.camera(d.mode,d.data)
break;
case's.tx':
s.tx(d.data,d.to)
break;
case's.userLog':
if(!d.mon || !d.data)return console.log('LOG DROPPED',d.mon,d.data);
s.userLog(d.mon,d.data)
break;
case'open_timelapse_file_transfer':
var location = s.getTimelapseFrameDirectory(d.d) + `${d.currentDate}/`
if(!fs.existsSync(location)){
fs.mkdirSync(location)
}
break;
case'created_timelapse_file_chunk':
if(!s.group[d.ke].activeMonitors[d.mid].childNodeStreamWriters[d.filename]){
var dir = s.getTimelapseFrameDirectory(d.d) + `${d.currentDate}/`
s.group[d.ke].activeMonitors[d.mid].childNodeStreamWriters[d.filename] = fs.createWriteStream(dir+d.filename)
}
s.group[d.ke].activeMonitors[d.mid].childNodeStreamWriters[d.filename].write(d.chunk)
break;
case'created_timelapse_file':
if(!s.group[d.ke].activeMonitors[d.mid].childNodeStreamWriters[d.filename]){
return console.log('FILE NOT EXIST')
}
s.group[d.ke].activeMonitors[d.mid].childNodeStreamWriters[d.filename].end()
cn.tx({
f: 'deleteTimelapseFrame',
file: d.filename,
currentDate: d.currentDate,
d: d.d, //monitor config
ke: d.ke,
mid: d.mid
})
s.insertTimelapseFrameDatabaseRow({
ke: d.ke
},d.queryInfo)
break;
case'created_file_chunk':
if(!s.group[d.ke].activeMonitors[d.mid].childNodeStreamWriters[d.filename]){
d.dir = s.getVideoDirectory(s.group[d.ke].rawMonitorConfigurations[d.mid])
if (!fs.existsSync(d.dir)) {
fs.mkdirSync(d.dir, {recursive: true}, (err) => {s.debugLog(err)})
}
s.group[d.ke].activeMonitors[d.mid].childNodeStreamWriters[d.filename] = fs.createWriteStream(d.dir+d.filename)
}
s.group[d.ke].activeMonitors[d.mid].childNodeStreamWriters[d.filename].write(d.chunk)
break;
case'created_file':
if(!s.group[d.ke].activeMonitors[d.mid].childNodeStreamWriters[d.filename]){
return console.log('FILE NOT EXIST')
}
s.group[d.ke].activeMonitors[d.mid].childNodeStreamWriters[d.filename].end();
cn.tx({
f:'delete',
file:d.filename,
ke:d.ke,
mid:d.mid
})
s.txWithSubPermissions({
f:'video_build_success',
hrefNoAuth:'/videos/'+d.ke+'/'+d.mid+'/'+d.filename,
filename:d.filename,
mid:d.mid,
ke:d.ke,
time:d.time,
size:d.filesize,
end:d.end
},'GRP_'+d.ke,'video_view')
//save database row
var insert = {
startTime : d.time,
filesize : d.filesize,
endTime : d.end,
dir : s.getVideoDirectory(d.d),
file : d.filename,
filename : d.filename,
filesizeMB : parseFloat((d.filesize/1048576).toFixed(2))
}
s.insertDatabaseRow(d.d,insert)
s.insertCompletedVideoExtensions.forEach(function(extender){
extender(d.d,insert)
})
//purge over max
s.purgeDiskForGroup(d.ke)
//send new diskUsage values
s.setDiskUsedForGroup(d.ke,insert.filesizeMB)
clearTimeout(s.group[d.ke].activeMonitors[d.mid].recordingChecker)
clearTimeout(s.group[d.ke].activeMonitors[d.mid].streamChecker)
break;
}
}
const ipAddress = getIpAddress(req)
const connectionId = s.gid(10);
s.debugLog('Child Node Connection!',new Date(),ipAddress)
client.id = connectionId;
function onAuthenticate(d){
const data = JSON.parse(d);
const childNodeKeyAccepted = config.childNodes.key.indexOf(data.socketKey) > -1;
if(!client.shinobiChildAlreadyRegistered && data.f === 'init' && childNodeKeyAccepted){
initiateDataConnection(client,req,data,connectionId);
childNodesConnectionIndex[connectionId] = client;
client.removeListener('message',onAuthenticate)
client.on('message',(d) => {
const data = JSON.parse(d);
onWebSocketDataFromChildNode(client,data)
})
}else{
s.debugLog('Child Node Force Disconnected!',new Date(),ipAddress)
client.destroy()
}
}
client.on('message',onAuthenticate)
client.on('close',() => {
onDataConnectionDisconnect(client, req)
})
cn.on('disconnect',function(){
console.log('childNodeWebsocket.disconnect',ipAddress)
if(s.childNodes[ipAddress]){
var monitors = Object.values(s.childNodes[ipAddress].activeCameras)
if(monitors && monitors[0]){
var loadCompleted = 0
var loadMonitor = function(monitor){
setTimeout(function(){
var mode = monitor.mode + ''
var cleanMonitor = s.cleanMonitorObject(monitor)
s.camera('stop',Object.assign(cleanMonitor,{}))
delete(s.group[monitor.ke].activeMonitors[monitor.mid].childNode)
delete(s.group[monitor.ke].activeMonitors[monitor.mid].childNodeId)
setTimeout(function(){
s.camera(mode,cleanMonitor)
++loadCompleted
if(monitors[loadCompleted]){
loadMonitor(monitors[loadCompleted])
}
},1000)
},2000)
}
loadMonitor(monitors[loadCompleted])
})
childNodeFileRelay.on('connection', function (client, req) {
function onAuthenticate(d){
const data = JSON.parse(d);
const childNodeKeyAccepted = config.childNodes.key.indexOf(data.socketKey) > -1;
if(!client.alreadyInitiated && data.fileType && childNodeKeyAccepted){
client.alreadyInitiated = true;
client.removeListener('message',onAuthenticate)
switch(data.fileType){
case'video':
initiateVideoWriteFromChildNode(client,data.options,data.connectionId)
break;
case'timelapseFrame':
initiateTimelapseFrameWriteFromChildNode(client,data.options,data.connectionId)
break;
}
s.childNodes[ipAddress].activeCameras = {}
s.childNodes[ipAddress].dead = true
}else{
client.destroy()
}
})
}
client.on('message',onAuthenticate)
})
}else
//setup Child for childNodes
if(config.childNodes.enabled === true && config.childNodes.mode === 'child' && config.childNodes.host){
s.connected = false;
childIO = require('socket.io-client')('ws://'+config.childNodes.host,{
transports: ['websocket']
});
s.cx = function(x){x.socketKey = config.childNodes.key;childIO.emit('c',x)}
s.tx = function(x,y){s.cx({f:'s.tx',data:x,to:y})}
s.userLog = function(x,y){s.cx({f:'s.userLog',mon:x,data:y})}
if(
config.childNodes.enabled === true &&
config.childNodes.mode === 'child' &&
config.childNodes.host
){
const {
initiateConnectionToMasterNode,
onDisconnectFromMasterNode,
onDataFromMasterNode,
} = require('./childNode/childUtils.js')(s,config,lang,app,io)
s.connectedToMasterNode = false;
let childIO;
function createChildNodeConnection(){
childIO = createWebSocketClient('ws://'+config.childNodes.host + '/childNode',{
onMessage: onDataFromMasterNode
})
childIO.on('open', function(){
console.error(new Date(),'Child Nodes : Connected to Master Node! Authenticating...');
initiateConnectionToMasterNode()
})
childIO.on('close',function(){
onDisconnectFromMasterNode()
setTimeout(() => {
console.error(new Date(),'Child Nodes : Connection to Master Node Closed. Attempting Reconnect...');
createChildNodeConnection()
},3000)
})
childIO.on('error',function(err){
console.error(new Date(),'Child Nodes ERROR : ', err.message);
childIO.close()
})
}
createChildNodeConnection()
function sendDataToMasterNode(data){
childIO.send(JSON.stringify(data))
}
s.cx = sendDataToMasterNode;
// replace internal functions with bridges to master node
s.tx = function(x,y){
sendDataToMasterNode({f:'s.tx',data:x,to:y})
}
s.userLog = function(x,y){
sendDataToMasterNode({f:'s.userLog',mon:x,data:y})
}
s.queuedSqlCallbacks = {}
s.sqlQuery = function(query,values,onMoveOn){
var callbackId = s.gid()
@ -218,84 +150,13 @@ module.exports = function(s,config,lang,app,io){
var onMoveOn = values;
var values = [];
}
if(typeof onMoveOn !== 'function'){onMoveOn=function(){}}
s.queuedSqlCallbacks[callbackId] = onMoveOn
s.cx({f:'sql',query:query,values:values,callbackId:callbackId});
if(typeof onMoveOn === 'function')s.queuedSqlCallbacks[callbackId] = onMoveOn;
sendDataToMasterNode({f:'sql',query:query,values:values,callbackId:callbackId});
}
s.knexQuery = function(options,onMoveOn){
var callbackId = s.gid()
if(typeof onMoveOn !== 'function'){onMoveOn=function(){}}
s.queuedSqlCallbacks[callbackId] = onMoveOn
s.cx({f:'knex',options:options,callbackId:callbackId});
if(typeof onMoveOn === 'function')s.queuedSqlCallbacks[callbackId] = onMoveOn;
sendDataToMasterNode({f:'knex',options:options,callbackId:callbackId});
}
setInterval(async () => {
const cpu = await s.cpuUsage()
s.cx({
f: 'cpu',
cpu: parseFloat(cpu)
})
},5000)
childIO.on('connect', function(d){
console.log('CHILD CONNECTION SUCCESS')
s.cx({
f : 'init',
port : config.port,
coreCount : s.coreCount,
availableHWAccels : config.availableHWAccels
})
})
childIO.on('c', function (d) {
switch(d.f){
case'sqlCallback':
if(s.queuedSqlCallbacks[d.callbackId]){
s.queuedSqlCallbacks[d.callbackId](d.err,d.rows)
delete(s.queuedSqlCallbacks[d.callbackId])
}
break;
case'init_success':
s.connected=true;
s.other_helpers=d.child_helpers;
break;
case'kill':
s.initiateMonitorObject(d.d);
cameraDestroy(d.d)
var childNodeIp = s.group[d.d.ke].activeMonitors[d.d.id]
break;
case'sync':
s.initiateMonitorObject(d.sync);
Object.keys(d.sync).forEach(function(v){
s.group[d.sync.ke].activeMonitors[d.sync.mid][v]=d.sync[v];
});
break;
case'delete'://delete video
s.file('delete',s.dir.videos+d.ke+'/'+d.mid+'/'+d.file)
break;
case'deleteTimelapseFrame'://delete video
var filePath = s.getTimelapseFrameDirectory(d.d) + `${d.currentDate}/` + d.file
s.file('delete',filePath)
break;
case'insertCompleted'://close video
s.insertCompletedVideo(d.d,d.k)
break;
case'cameraStop'://start camera
s.camera('stop',d.d)
break;
case'cameraStart'://start or record camera
s.camera(d.mode,d.d)
break;
}
})
childIO.on('disconnect',function(d){
s.connected = false;
var groupKeys = Object.keys(s.group)
groupKeys.forEach(function(groupKey){
var activeMonitorKeys = Object.keys(s.group[groupKey].activeMonitors)
activeMonitorKeys.forEach(function(monitorKey){
var activeMonitor = s.group[groupKey].activeMonitors[monitorKey]
if(activeMonitor && activeMonitor.spawn && activeMonitor.spawn.close)activeMonitor.spawn.close()
if(activeMonitor && activeMonitor.spawn && activeMonitor.spawn.kill)activeMonitor.spawn.kill()
})
})
})
}
}

149
libs/childNode/childUtils.js Normal file
View File

@ -0,0 +1,149 @@
const fs = require('fs');
const { createWebSocketClient } = require('../basic/websocketTools.js')
module.exports = function(s,config,lang,app,io){
const { cameraDestroy } = require('../monitor/utils.js')(s,config,lang)
var checkHwInterval = null;
function onDataFromMasterNode(d) {
switch(d.f){
case'sqlCallback':
const callbackId = d.callbackId;
if(s.queuedSqlCallbacks[callbackId]){
s.queuedSqlCallbacks[callbackId](d.err,d.rows)
delete(s.queuedSqlCallbacks[callbackId])
}
break;
case'init_success':
console.error(new Date(),'Child Nodes : Authenticated with Master Node!');
s.connectedToMasterNode = true;
s.other_helpers = d.child_helpers;
s.childNodeIdOnMasterNode = d.connectionId
break;
case'kill':
s.initiateMonitorObject(d.d);
cameraDestroy(d.d)
break;
case'sync':
s.initiateMonitorObject(d.sync);
Object.keys(d.sync).forEach(function(v){
s.group[d.sync.ke].activeMonitors[d.sync.mid][v]=d.sync[v];
});
break;
case'delete'://delete video
s.file('delete',s.dir.videos+d.ke+'/'+d.mid+'/'+d.file)
break;
case'deleteTimelapseFrame'://delete timelapse frame
var filePath = s.getTimelapseFrameDirectory(d) + `${d.currentDate}/` + d.file
s.file('delete',filePath)
break;
case'cameraStop'://stop camera
// s.group[d.d.ke].activeMonitors[d.d.mid].masterSaysToStop = true
s.camera('stop',d.d)
break;
case'cameraStart'://start or record camera
s.camera(d.mode,d.d)
let activeMonitor = s.group[d.d.ke].activeMonitors[d.d.mid]
// activeMonitor.masterSaysToStop = false
clearTimeout(activeMonitor.recordingChecker);
clearTimeout(activeMonitor.streamChecker);
break;
}
}
function initiateConnectionToMasterNode(){
s.cx({
f: 'init',
port: config.port,
platform: s.platform,
coreCount: s.coreCount,
totalmem: s.totalmem / 1048576,
availableHWAccels: config.availableHWAccels,
socketKey: config.childNodes.key
})
clearInterval(checkHwInterval)
checkHwInterval = setInterval(() => {
sendCurrentCpuUsage()
sendCurrentRamUsage()
},5000)
}
function onDisconnectFromMasterNode(){
s.connectedToMasterNode = false;
destroyAllMonitorProcesses()
clearInterval(checkHwInterval)
}
function destroyAllMonitorProcesses(){
var groupKeys = Object.keys(s.group)
groupKeys.forEach(function(groupKey){
var activeMonitorKeys = Object.keys(s.group[groupKey].activeMonitors)
activeMonitorKeys.forEach(function(monitorKey){
var activeMonitor = s.group[groupKey].activeMonitors[monitorKey]
if(activeMonitor && activeMonitor.spawn && activeMonitor.spawn.close)activeMonitor.spawn.close()
if(activeMonitor && activeMonitor.spawn && activeMonitor.spawn.kill)activeMonitor.spawn.kill()
})
})
}
async function sendCurrentCpuUsage(){
const percent = await s.cpuUsage();
const use = s.coreCount * (percent / 100)
s.cx({
f: 'cpu',
used: use,
percent: percent
})
}
async function sendCurrentRamUsage(){
const ram = await s.ramUsage()
s.cx({
f: 'ram',
used: ram.used,
percent: ram.percent,
})
}
function createFileTransferToMasterNode(filePath,transferInfo,fileType){
const response = {ok: true}
return new Promise((resolve,reject) => {
const fileTransferConnection = createWebSocketClient('ws://'+config.childNodes.host + '/childNodeFileRelay',{
onMessage: () => {}
})
fileTransferConnection.on('open', function(){
fileTransferConnection.send(JSON.stringify({
fileType: fileType || 'video',
options: transferInfo,
socketKey: config.childNodes.key,
connectionId: s.childNodeIdOnMasterNode,
}))
setTimeout(() => {
fs.createReadStream(filePath)
.on('data',function(data){
fileTransferConnection.send(data)
})
.on('close',function(){
fileTransferConnection.close()
resolve(response)
})
},2000)
})
})
}
async function sendVideoToMasterNode(filePath,options){
const groupKey = options.ke
const monitorId = options.mid
const activeMonitor = s.group[groupKey].activeMonitors[monitorId]
const response = await createFileTransferToMasterNode(filePath,options,'video');
clearTimeout(activeMonitor.recordingChecker);
clearTimeout(activeMonitor.streamChecker);
return response;
}
async function sendTimelapseFrameToMasterNode(filePath,options){
const response = await createFileTransferToMasterNode(filePath,options,'timelapseFrame');
return response;
}
return {
onDataFromMasterNode,
initiateConnectionToMasterNode,
onDisconnectFromMasterNode,
destroyAllMonitorProcesses,
sendCurrentCpuUsage,
sendCurrentRamUsage,
sendVideoToMasterNode,
sendTimelapseFrameToMasterNode,
}
}

358
libs/childNode/utils.js Normal file
View File

@ -0,0 +1,358 @@
const fs = require('fs');
module.exports = function(s,config,lang,app,io){
const masterDoWorkToo = config.childNodes.masterDoWorkToo;
const maxCpuPercent = config.childNodes.maxCpuPercent || 75;
const maxRamPercent = config.childNodes.maxRamPercent || 75;
function getIpAddress(req){
return (req.headers['cf-connecting-ip'] ||
req.headers["CF-Connecting-IP"] ||
req.headers["'x-forwarded-for"] ||
req.connection.remoteAddress).replace('::ffff:','');
}
function initiateDataConnection(client,req,options,connectionId){
const ipAddress = getIpAddress(req)
const webAddress = ipAddress + ':' + options.port
client.ip = webAddress;
client.shinobiChildAlreadyRegistered = true;
client.sendJson = (data) => {
client.send(JSON.stringify(data))
}
if(!s.childNodes[webAddress]){
s.childNodes[webAddress] = {}
};
const activeNode = s.childNodes[webAddress];
activeNode.dead = false
activeNode.cnid = client.id
activeNode.cpu = 0
activeNode.ip = webAddress
activeNode.activeCameras = {}
activeNode.platform = options.platform
activeNode.coreCount = options.coreCount
activeNode.totalmem = options.totalmem
options.availableHWAccels.forEach(function(accel){
if(config.availableHWAccels.indexOf(accel) === -1)config.availableHWAccels.push(accel)
})
client.sendJson({
f : 'init_success',
childNodes : s.childNodes,
connectionId: connectionId,
})
s.debugLog('Authenticated Child Node!',new Date(),webAddress)
return webAddress
}
function onWebSocketDataFromChildNode(client,data){
const activeMonitor = data.ke && data.mid && s.group[data.ke] ? s.group[data.ke].activeMonitors[data.mid] : null;
const webAddress = client.ip;
switch(data.f){
case'cpu':
s.childNodes[webAddress].cpuUsed = data.used;
s.childNodes[webAddress].cpuPercent = data.percent;
break;
case'ram':
s.childNodes[webAddress].ramUsed = data.used;
s.childNodes[webAddress].ramPercent = data.percent;
break;
case'sql':
s.sqlQuery(data.query,data.values,function(err,rows){
client.sendJson({f:'sqlCallback',rows:rows,err:err,callbackId:data.callbackId});
});
break;
case'knex':
s.knexQuery(data.options,function(err,rows){
client.sendJson({f:'sqlCallback',rows:rows,err:err,callbackId:data.callbackId});
});
break;
case'clearCameraFromActiveList':
if(s.childNodes[webAddress])delete(s.childNodes[webAddress].activeCameras[data.ke + data.id])
break;
case'camera':
s.camera(data.mode,data.data)
break;
case's.tx':
s.tx(data.data,data.to)
break;
case's.userLog':
if(!data.mon || !data.data)return console.log('LOG DROPPED',data.mon,data.data);
s.userLog(data.mon,data.data)
break;
}
}
function onDataConnectionDisconnect(client, req){
const webAddress = client.ip;
console.log('childNodeWebsocket.disconnect',webAddress)
if(s.childNodes[webAddress]){
var monitors = Object.values(s.childNodes[webAddress].activeCameras)
if(monitors && monitors[0]){
var loadCompleted = 0
var loadMonitor = function(monitor){
setTimeout(function(){
var mode = monitor.mode + ''
var cleanMonitor = s.cleanMonitorObject(monitor)
s.camera('stop',Object.assign(cleanMonitor,{}))
delete(s.group[monitor.ke].activeMonitors[monitor.mid].childNode)
delete(s.group[monitor.ke].activeMonitors[monitor.mid].childNodeId)
setTimeout(function(){
s.camera(mode,cleanMonitor)
++loadCompleted
if(monitors[loadCompleted]){
loadMonitor(monitors[loadCompleted])
}
},1000)
},2000)
}
loadMonitor(monitors[loadCompleted])
}
s.childNodes[webAddress].activeCameras = {}
s.childNodes[webAddress].dead = true
}
}
function initiateFileWriteFromChildNode(client,data,connectionId,onFinish){
const response = {ok: true}
const groupKey = data.ke
const monitorId = data.mid
const filename = data.filename
const activeMonitor = s.group[groupKey].activeMonitors[monitorId]
const writeDirectory = data.writeDirectory
const fileWritePath = writeDirectory + filename
const writeStream = fs.createWriteStream(fileWritePath)
if (!fs.existsSync(writeDirectory)) {
fs.mkdirSync(writeDirectory, {recursive: true}, (err) => {s.debugLog(err)})
}
activeMonitor.childNodeStreamWriters[filename] = writeStream
client.on('message',(d) => {
writeStream.write(d)
})
client.on('close',(d) => {
setTimeout(() => {
// response.fileWritePath = fileWritePath
// response.writeData = data
// response.childNodeId = connectionId
try{
activeMonitor.childNodeStreamWriters[filename].end();
}catch(err){
}
setTimeout(() => {
delete(activeMonitor.childNodeStreamWriters[filename])
},100)
onFinish(response)
},2000)
})
}
function initiateVideoWriteFromChildNode(client,data,connectionId){
return new Promise((resolve,reject) => {
const groupKey = data.ke
const monitorId = data.mid
const filename = data.filename
const activeMonitor = s.group[groupKey].activeMonitors[monitorId]
const monitorConfig = s.group[groupKey].rawMonitorConfigurations[monitorId]
const videoDirectory = s.getVideoDirectory(monitorConfig)
data.writeDirectory = videoDirectory
initiateFileWriteFromChildNode(client,data,connectionId,(response) => {
//delete video file from child node
s.cx({
f: 'delete',
file: filename,
ke: data.ke,
mid: data.mid
},connectionId)
//
s.txWithSubPermissions({
f:'video_build_success',
hrefNoAuth:'/videos/'+data.ke+'/'+data.mid+'/'+filename,
filename:filename,
mid:data.mid,
ke:data.ke,
time:data.time,
size:data.filesize,
end:data.end
},'GRP_'+data.ke,'video_view')
//save database row
var insert = {
startTime : data.time,
filesize : data.filesize,
endTime : data.end,
dir : videoDirectory,
file : filename,
filename : filename,
filesizeMB : parseFloat((data.filesize/1048576).toFixed(2))
}
s.insertDatabaseRow(monitorConfig,insert)
s.insertCompletedVideoExtensions.forEach(function(extender){
extender(monitorConfig,insert)
})
//purge over max
s.purgeDiskForGroup(data.ke)
//send new diskUsage values
s.setDiskUsedForGroup(data.ke,insert.filesizeMB)
clearTimeout(activeMonitor.recordingChecker)
clearTimeout(activeMonitor.streamChecker)
resolve(response)
})
})
}
function initiateTimelapseFrameWriteFromChildNode(client,data,connectionId){
return new Promise((resolve,reject) => {
const groupKey = data.ke
const monitorId = data.mid
const filename = data.filename
const currentDate = data.currentDate
const activeMonitor = s.group[groupKey].activeMonitors[monitorId]
const monitorConfig = s.group[groupKey].rawMonitorConfigurations[monitorId]
const timelapseFrameDirectory = s.getTimelapseFrameDirectory(monitorConfig) + currentDate + `/`
const fileWritePath = timelapseFrameDirectory + filename
const writeStream = fs.createWriteStream(fileWritePath)
data.writeDirectory = timelapseFrameDirectory
initiateFileWriteFromChildNode(client,data,connectionId,(response) => {
s.cx({
f: 'deleteTimelapseFrame',
file: filename,
currentDate: currentDate,
ke: groupKey,
mid: monitorId
},connectionId)
s.insertTimelapseFrameDatabaseRow({
ke: groupKey
},data.queryInfo)
resolve(response)
})
})
}
function getActiveCameraCount(){
let theCount = 0
Object.keys(s.group).forEach(function(groupKey){
const theGroup = s.group[groupKey]
Object.keys(theGroup.activeMonitors).forEach(function(groupKey){
const activeMonitor = theGroup.activeMonitors[monitorId]
if(
// watching
activeMonitor.statusCode === 2 ||
// recording
activeMonitor.statusCode === 3 ||
// starting
activeMonitor.statusCode === 1 ||
// started
activeMonitor.statusCode === 9
//// Idle, in memory
// activeMonitor.statusCode === 6
){
++theCount
}
})
})
return theCount
}
function bindMonitorToChildNode(options){
const groupKey = options.ke
const monitorId = options.mid
const childNodeSelected = options.childNodeId
const theChildNode = s.childNodes[childNodeSelected]
const activeMonitor = s.group[groupKey].activeMonitors[monitorId];
const monitorConfig = Object.assign({},s.group[groupKey].rawMonitorConfigurations[monitorId])
theChildNode.activeCameras[groupKey + monitorId] = monitorConfig;
activeMonitor.childNode = childNodeSelected
activeMonitor.childNodeId = theChildNode.cnid;
}
function getNodeWithHighestCpuAndRamUse(){
var nodeWithLowestCpuUse = 0
var nodeWithLowestRamUse = 0
const childNodeList = Object.keys(s.childNodes)
childNodeList.forEach(function(webAddress){
const theChildNode = s.childNodes[webAddress]
if(
theChildNode.cpuUsed > nodeWithLowestCpuUse &&
theChildNode.ramUsed > nodeWithLowestRamUse
){
nodeWithLowestCpuUse = theChildNode.cpuUsed + 0.2
nodeWithLowestRamUse = theChildNode.ramUsed + 50
}
})
return {
nodeWithLowestCpuUse,
nodeWithLowestRamUse,
}
}
async function selectNodeForOperation(options){
const groupKey = options.ke
const monitorId = options.mid
const childNodeList = Object.keys(s.childNodes)
if(childNodeList.length > 0){
let childNodeFound = false
let childNodeSelected = null;
var nodeWithLowestActiveCamerasCount = 65535
var nodeWithLowestActiveCameras = null
let nodeWithLowestCpuPercent = 100
let nodeWithLowestRamPercent = 100
let {
nodeWithLowestCpuUse,
nodeWithLowestRamUse,
} = getNodeWithHighestCpuAndRamUse();
childNodeList.forEach(function(webAddress){
const theChildNode = s.childNodes[webAddress]
delete(theChildNode.activeCameras[groupKey + monitorId])
const nodeCameraCount = Object.keys(theChildNode.activeCameras).length
if(
// child node is connected and available
!theChildNode.dead &&
// // look for child node with least number of running cameras
// nodeCameraCount < nodeWithLowestActiveCamerasCount &&
// look for child node with CPU usage below 75% (default)
theChildNode.cpuUsed < nodeWithLowestCpuUse &&
theChildNode.cpuPercent < maxCpuPercent &&
theChildNode.cpuPercent < nodeWithLowestCpuPercent &&
// look for child node with RAM usage below 75% (default)
theChildNode.ramUsed < nodeWithLowestRamUse &&
theChildNode.ramPercent < maxRamPercent &&
theChildNode.ramPercent < nodeWithLowestRamPercent
){
// nodeWithLowestActiveCamerasCount = nodeCameraCount
childNodeSelected = `${webAddress}`
nodeWithLowestCpuUse = theChildNode.cpuUsed
nodeWithLowestCpuPercent = theChildNode.cpuPercent
nodeWithLowestRamUse = theChildNode.ramUsed
nodeWithLowestRamPercent = theChildNode.ramPercent
}
})
if(childNodeSelected && masterDoWorkToo){
// const nodeCameraCount = getActiveCameraCount()
const masterNodeHw = await getHwUsage();
if(
// nodeCameraCount < nodeWithLowestActiveCamerasCount &&
masterNodeHw.cpuUsed < nodeWithLowestCpuUse &&
masterNodeHw.cpuPercent < maxCpuPercent &&
masterNodeHw.cpuPercent < nodeWithLowestCpuPercent &&
// look for child node with RAM usage below 75% (default)
masterNodeHw.ramUsed < nodeWithLowestRamUse &&
masterNodeHw.ramPercent < maxRamPercent &&
masterNodeHw.ramPercent < nodeWithLowestRamPercent
){
// nodeWithLowestActiveCamerasCount = nodeCameraCount
// release child node selection and use master node
childNodeSelected = null
}
}
return childNodeSelected;
}
}
async function getHwUsage(){
const percent = await s.cpuUsage();
const use = s.coreCount * (percent / 100)
const ram = await s.ramUsage()
return {
ramUsed: ram.used,
ramPercent: ram.percent,
cpuUsed: use,
cpuPercent: percent,
}
}
return {
getIpAddress,
initiateDataConnection,
onWebSocketDataFromChildNode,
onDataConnectionDisconnect,
initiateVideoWriteFromChildNode,
initiateTimelapseFrameWriteFromChildNode,
selectNodeForOperation,
bindMonitorToChildNode,
}
}

61
libs/commander.js
View File

@ -6,6 +6,8 @@ module.exports = function(s,config,lang,app){
var runningWorker;
config.machineId = config.p2pApiKey + '' + config.p2pGroupId
config.p2pTargetAuth = config.p2pTargetAuth || s.gid(30)
config.p2pShellAccess = config.p2pShellAccess || false
config.useBetterP2P = config.useBetterP2P === undefined ? true : config.useBetterP2P
if(!config.workerStreamOutHandlers){
config.workerStreamOutHandlers = [
'Base64',
@ -15,11 +17,13 @@ module.exports = function(s,config,lang,app){
}
if(!customerServerList){
config.p2pServerList = {
"vancouver-1": {
"vancouver-1-v2": {
name: 'Vancouver-1',
host: 'p2p-vancouver-1.shinobi.cloud',
p2pPort: '8084',
webPort: '8000',
v2: true,
p2pPort: '80',
webPort: '80',
chartPort: '80',
maxNetworkSpeed: {
up: 5000,
down: 5000,
@ -30,11 +34,13 @@ module.exports = function(s,config,lang,app){
lon: -123.1140607
}
},
"toronto-1": {
"toronto-1-v2": {
name: 'Toronto-1',
host: 'p2p-toronto-1.shinobi.cloud',
p2pPort: '8084',
webPort: '8000',
v2: true,
p2pPort: '80',
webPort: '80',
chartPort: '80',
maxNetworkSpeed: {
up: 5000,
down: 5000,
@ -45,11 +51,13 @@ module.exports = function(s,config,lang,app){
lon: -79.3862837
}
},
"paris-1": {
"paris-1-v2": {
name: 'Paris-1',
host: 'p2p-paris-1.shinobi.cloud',
p2pPort: '8084',
webPort: '8000',
v2: true,
p2pPort: '80',
webPort: '80',
chartPort: '80',
maxNetworkSpeed: {
up: 200,
down: 200,
@ -72,7 +80,16 @@ module.exports = function(s,config,lang,app){
}
});
}
if(!config.p2pHostSelected)config.p2pHostSelected = 'paris-1'
if(!config.p2pHostSelected)config.p2pHostSelected = config.useBetterP2P ? 'paris-1-v2' : 'paris-1'
const p2pServerKeys = Object.keys(config.p2pServerList)
const filteredList = {}
p2pServerKeys.forEach((keyName) => {
const connector = config.p2pServerList[keyName]
if(connector.v2 === !!config.useBetterP2P){
filteredList[keyName] = connector;
}
})
config.p2pServerList = filteredList;
const stopWorker = () => {
if(runningWorker){
runningWorker.postMessage({
@ -82,8 +99,7 @@ module.exports = function(s,config,lang,app){
}
const startWorker = () => {
stopWorker()
// set the first parameter as a string.
const pathToWorkerScript = __dirname + '/commander/worker.js'
const pathToWorkerScript = __dirname + `/commander/${config.useBetterP2P ? 'workerv2' : 'worker'}.js`
const workerProcess = new Worker(pathToWorkerScript)
workerProcess.on('message',function(data){
switch(data.f){
@ -102,33 +118,16 @@ module.exports = function(s,config,lang,app){
lang: lang
})
},2000)
// workerProcess is an Emitter.
// it also contains a direct handle to the `spawn` at `workerProcess.spawnProcess`
return workerProcess
}
const beginConnection = () => {
if(config.p2pTargetGroupId && config.p2pTargetUserId){
runningWorker = startWorker()
}else{
s.knexQuery({
action: "select",
columns: "ke,uid",
table: "Users",
where: [],
limit: 1
},(err,r) => {
const firstUser = r[0]
config.p2pTargetUserId = firstUser.uid
config.p2pTargetGroupId = firstUser.ke
runningWorker = startWorker()
})
}
runningWorker = startWorker()
}
if(config.p2pEnabled){
beginConnection()
}
/**
* API : Superuser : Log delete.
* API : Superuser : Save P2P Server choice
*/
app.post(config.webPaths.superApiPrefix+':auth/p2p/save', function (req,res){
s.superAuth(req.params,async (resp) => {

53
libs/commander/worker.js
View File

@ -1,5 +1,5 @@
const { parentPort } = require('worker_threads');
const request = require('request');
const fetch = require('node-fetch');
const socketIOClient = require('socket.io-client');
const p2pClientConnectionStaticName = 'Commander'
const p2pClientConnections = {}
@ -58,15 +58,28 @@ const initialize = (config,lang) => {
if(method === 'GET' && data){
requestEndpoint += '?' + createQueryStringFromObject(data)
}
const theRequest = request(requestEndpoint,{
const theRequest = fetch(requestEndpoint,{
method: method,
json: method !== 'GET' ? (data ? data : null) : null
}, typeof callback === 'function' ? (err,resp,body) => {
// var json = parseJSON(body)
if(err)console.error(err,data)
callback(err,body,resp)
} : null)
.on('data', onDataReceived);
headers: {
'Accept': 'application/json',
'Content-Type': 'application/json'
},
body: method !== 'GET' ? JSON.stringify(data ? data : null) : null
})
.then(res => {
res.body.on('data', onDataReceived);
return res
});
if(typeof callback === 'function'){
theRequest.then(res => res.json())
.then(json => {
callback(null,json);
})
}
theRequest.catch((err) => {
console.error(err);
if(typeof callback === 'function')callback(err,null);
});
return theRequest
}
const createShinobiSocketConnection = (connectionId) => {
@ -88,12 +101,15 @@ const initialize = (config,lang) => {
}
//
function startBridge(noLog){
s.debugLog('p2p',`Connecting to ${selectedHost}...`)
console.log('p2p',`Connecting to ${selectedHost}...`)
if(connectionToP2PServer && connectionToP2PServer.connected){
connectionToP2PServer.allowDisconnect = true;
connectionToP2PServer.disconnect()
}
connectionToP2PServer = socketIOClient('ws://' + selectedHost, {transports:['websocket']});
connectionToP2PServer = socketIOClient('ws://' + selectedHost, {
transports:['websocket'],
reconnection: false
});
if(!config.p2pApiKey){
if(!noLog)s.systemLog('p2p',`Please fill 'p2pApiKey' in your conf.json.`)
}
@ -201,12 +217,7 @@ const initialize = (config,lang) => {
})
});
([
'h265',
'Base64',
'FLV',
'MP4',
]).forEach((target) => {
config.workerStreamOutHandlers.forEach((target) => {
connectionToP2PServer.on(target,(initData) => {
if(connectedUserWebSockets[initData.auth]){
const clientConnectionToMachine = createShinobiSocketConnection(initData.auth + initData.ke + initData.id)
@ -256,7 +267,9 @@ const initialize = (config,lang) => {
connectionToP2PServer.on('disconnect',onDisconnect)
}
startBridge()
setInterval(() => {
startBridge(true)
},1000 * 60 * 60 * 15)
setInterval(function(){
if(!connectionToP2PServer || !connectionToP2PServer.connected){
connectionToP2PServer.connect()
}
},1000 * 60 * 15)
}

338
libs/commander/workerv2.js Normal file
View File

@ -0,0 +1,338 @@
const { parentPort } = require('worker_threads');
process.on("uncaughtException", function(error) {
console.error(error);
});
let remoteConnectionPort = 8080
let config = {}
let lang = {}
const net = require("net")
const bson = require('bson')
const WebSocket = require('cws')
const s = {
debugLog: (...args) => {
parentPort.postMessage({
f: 'debugLog',
data: args
})
},
systemLog: (...args) => {
parentPort.postMessage({
f: 'systemLog',
data: args
})
},
}
parentPort.on('message',(data) => {
switch(data.f){
case'init':
config = Object.assign({},data.config)
lang = Object.assign({},data.lang)
remoteConnectionPort = config.ssl ? config.ssl.port || 443 : config.port || 8080
initialize()
break;
case'exit':
s.debugLog('Closing P2P Connection...')
process.exit(0)
break;
}
})
var socketCheckTimer = null
var heartbeatTimer = null
var heartBeatCheckTimout = null
var onClosedTimeout = null
let stayDisconnected = false
const requestConnections = {}
const requestConnectionsData = {}
function getRequestConnection(requestId){
return requestConnections[requestId] || {
write: () => {}
}
}
function clearAllTimeouts(){
clearInterval(heartbeatTimer)
clearTimeout(heartBeatCheckTimout)
clearTimeout(onClosedTimeout)
}
function startConnection(p2pServerAddress,subscriptionId){
let tunnelToP2P
stayDisconnected = false
const allMessageHandlers = []
async function startWebsocketConnection(key,callback){
s.debugLog(`startWebsocketConnection EXECUTE`,new Error())
console.log('P2P : Connecting to Konekta P2P Server...')
function createWebsocketConnection(){
clearAllTimeouts()
return new Promise((resolve,reject) => {
try{
stayDisconnected = true
if(tunnelToP2P)tunnelToP2P.close()
}catch(err){
console.log(err)
}
tunnelToP2P = new WebSocket(p2pServerAddress);
stayDisconnected = false;
tunnelToP2P.on('open', function(){
resolve(tunnelToP2P)
})
tunnelToP2P.on('error', (err) => {
console.log(`P2P tunnelToP2P Error : `,err)
console.log(`P2P Restarting...`)
// disconnectedConnection()
})
tunnelToP2P.on('close', () => {
console.log(`P2P Connection Closed!`)
clearAllTimeouts()
// onClosedTimeout = setTimeout(() => {
// disconnectedConnection();
// },5000)
});
tunnelToP2P.onmessage = function(event){
const data = bson.deserialize(Buffer.from(event.data))
allMessageHandlers.forEach((handler) => {
if(data.f === handler.key){
handler.callback(data.data,data.rid)
}
})
}
clearInterval(socketCheckTimer)
socketCheckTimer = setInterval(() => {
s.debugLog('Tunnel Ready State :',tunnelToP2P.readyState)
if(tunnelToP2P.readyState !== 1){
s.debugLog('Tunnel NOT Ready! Reconnecting...')
disconnectedConnection()
}
},1000 * 60)
})
}
function disconnectedConnection(code,reason){
s.debugLog('stayDisconnected',stayDisconnected)
clearAllTimeouts()
s.debugLog('DISCONNECTED!')
if(stayDisconnected)return;
s.debugLog('RESTARTING!')
setTimeout(() => {
if(tunnelToP2P && tunnelToP2P.readyState !== 1)startWebsocketConnection()
},2000)
}
s.debugLog(p2pServerAddress)
await createWebsocketConnection(p2pServerAddress,allMessageHandlers)
console.log('P2P : Connected! Authenticating...')
sendDataToTunnel({
subscriptionId: subscriptionId
})
clearInterval(heartbeatTimer)
heartbeatTimer = setInterval(() => {
sendDataToTunnel({
f: 'ping',
})
}, 1000 * 10)
setTimeout(() => {
if(tunnelToP2P.readyState !== 1)refreshHeartBeatCheck()
},5000)
}
function sendDataToTunnel(data){
tunnelToP2P.send(
bson.serialize(data)
)
}
startWebsocketConnection()
function onIncomingMessage(key,callback){
allMessageHandlers.push({
key: key,
callback: callback,
})
}
function outboundMessage(key,data,requestId){
sendDataToTunnel({
f: key,
data: data,
rid: requestId
})
}
async function createRemoteSocket(host,port,requestId,initData){
// if(requestConnections[requestId]){
// remotesocket.off('data')
// remotesocket.off('drain')
// remotesocket.off('close')
// requestConnections[requestId].end()
// }
const responseTunnel = await getResponseTunnel(requestId)
let remotesocket = new net.Socket();
remotesocket.on('ready',() => {
remotesocket.write(initData.buffer)
})
remotesocket.on('error',(err) => {
s.debugLog('createRemoteSocket ERROR',err)
})
remotesocket.on('data', function(data) {
requestConnectionsData[requestId] = data.toString()
responseTunnel.send('data',data)
})
remotesocket.on('drain', function() {
responseTunnel.send('resume',{})
});
remotesocket.on('close', function() {
delete(requestConnectionsData[requestId])
responseTunnel.send('end',{})
setTimeout(() => {
if(
responseTunnel &&
(responseTunnel.readyState === 0 || responseTunnel.readyState === 1)
){
responseTunnel.close()
}
},5000)
});
remotesocket.connect(port || remoteConnectionPort, host || 'localhost');
requestConnections[requestId] = remotesocket
return remotesocket
}
function writeToServer(data,requestId){
var flushed = getRequestConnection(requestId).write(data.buffer)
if (!flushed) {
outboundMessage('pause',{},requestId)
}
}
function refreshHeartBeatCheck(){
clearTimeout(heartBeatCheckTimout)
heartBeatCheckTimout = setTimeout(() => {
startWebsocketConnection()
},1000 * 10 * 1.5)
}
// onIncomingMessage('connect',(data,requestId) => {
// console.log('New Request Incoming',requestId)
// await createRemoteSocket('172.16.101.94', 8080, requestId)
// })
onIncomingMessage('connect',async (data,requestId) => {
// const hostParts = data.host.split(':')
// const host = hostParts[0]
// const port = parseInt(hostParts[1]) || 80
s.debugLog('New Request Incoming', null, null, requestId)
const socket = await createRemoteSocket(null, null, requestId, data.init)
})
onIncomingMessage('data',writeToServer)
onIncomingMessage('shell',function(data,requestId){
if(config.p2pShellAccess === true){
const execCommand = data.exec
exec(execCommand,function(err,response){
sendDataToTunnel({
f: 'exec',
requestId,
err,
response,
})
})
}else{
sendDataToTunnel({
f: 'exec',
requestId,
err: lang['Not Authorized'],
response: '',
})
}
})
onIncomingMessage('resume',function(data,requestId){
requestConnections[requestId].resume()
})
onIncomingMessage('pause',function(data,requestId){
requestConnections[requestId].pause()
})
onIncomingMessage('pong',function(data,requestId){
refreshHeartBeatCheck()
s.debugLog('Heartbeat')
})
onIncomingMessage('init',function(data,requestId){
console.log(`P2P : Authenticated!`)
})
onIncomingMessage('end',function(data,requestId){
try{
requestConnections[requestId].end()
}catch(err){
s.debugLog(`Reqest Failed to END ${requestId}`)
s.debugLog(`Failed Request ${requestConnectionsData[requestId]}`)
delete(requestConnectionsData[requestId])
s.debugLog(err)
// console.log('requestConnections',requestConnections)
}
})
onIncomingMessage('disconnect',function(data,requestId){
console.log(`FAILED LICENSE CHECK ON P2P`)
const retryLater = data && data.retryLater;
stayDisconnected = !retryLater
if(retryLater)console.log(`Retrying P2P Later...`)
})
}
const responseTunnels = {}
async function getResponseTunnel(originalRequestId){
return responseTunnels[originalRequestId] || await createResponseTunnel(originalRequestId)
}
function createResponseTunnel(originalRequestId){
const responseTunnelMessageHandlers = []
function onMessage(key,callback){
responseTunnelMessageHandlers.push({
key: key,
callback: callback,
})
}
return new Promise((resolve,reject) => {
const responseTunnel = new WebSocket(config.selectedHost);
function sendToResponseTunnel(data){
responseTunnel.send(
bson.serialize(data)
)
}
function sendData(key,data){
sendToResponseTunnel({
f: key,
data: data,
rid: originalRequestId
})
}
responseTunnel.on('error', (err) => {
s.debugLog('responseTunnel ERROR',err)
})
responseTunnel.on('open', function(){
sendToResponseTunnel({
responseTunnel: originalRequestId,
subscriptionId: config.p2pApiKey,
})
})
responseTunnel.on('close', function(){
delete(responseTunnels[originalRequestId])
})
onMessage('ready', function(){
const finalData = {
onMessage,
send: sendData,
sendRaw: sendToResponseTunnel,
close: responseTunnel.close
}
responseTunnels[originalRequestId] = finalData;
resolve(finalData)
})
responseTunnel.onmessage = function(event){
const data = bson.deserialize(Buffer.from(event.data))
responseTunnelMessageHandlers.forEach((handler) => {
if(data.f === handler.key){
handler.callback(data.data,data.rid)
}
})
}
})
}
function closeResponseTunnel(originalRequestId){
// also should be handled server side
try{
responseTunnels[originalRequestId].close()
}catch(err){
s.debugLog('closeResponseTunnel',err)
}
}
function initialize(){
const selectedP2PServerId = config.p2pServerList[config.p2pHostSelected] ? config.p2pHostSelected : Object.keys(config.p2pServerList)[0]
const p2pServerDetails = config.p2pServerList[selectedP2PServerId]
const selectedHost = 'ws://' + p2pServerDetails.host + ':' + p2pServerDetails.p2pPort
config.selectedHost = selectedHost
startConnection(selectedHost,config.p2pApiKey)
}

19
libs/common.js
View File

@ -1,4 +1,5 @@
const async = require("async");
const fetch = require("node-fetch");
const mergeDeep = function(...objects) {
const isObject = obj => obj && typeof obj === 'object';
@ -21,7 +22,25 @@ const mergeDeep = function(...objects) {
return prev;
}, {});
}
const getBuffer = async (url) => {
try {
const response = await fetch(url);
const arrayBuffer = await response.arrayBuffer();
return Buffer.from(arrayBuffer);
} catch (error) {
return { error };
}
};
function addCredentialsToUrl(options){
const streamUrl = options.url
const username = options.username
const password = options.password
const urlParts = streamUrl.split('://')
return [urlParts[0],'://',`${username}:${password}@`,urlParts[1]].join('')
}
module.exports = {
addCredentialsToUrl,
getBuffer: getBuffer,
mergeDeep: mergeDeep,
validateIntValue: (value) => {
const newValue = !isNaN(parseInt(value)) ? parseInt(value) : null

12
libs/config.js
View File

@ -31,8 +31,8 @@ module.exports = function(s){
if(config.cron.deleteOverMaxOffset === undefined)config.cron.deleteOverMaxOffset=0.9;
if(config.cron.deleteLogs === undefined)config.cron.deleteLogs=true;
if(config.cron.deleteEvents === undefined)config.cron.deleteEvents=true;
if(config.cron.deleteFileBinsOverMax === undefined)config.cron.deleteFileBins=true;
if(config.deleteFileBins === undefined)config.deleteFileBinsOverMax=true;
if(config.cron.deleteFileBins === undefined)config.cron.deleteFileBins=true;
if(config.deleteFileBinsOverMax === undefined)config.deleteFileBinsOverMax=true;
if(config.cron.interval === undefined)config.cron.interval=1;
if(config.databaseType === undefined){config.databaseType='mysql'}
if(config.pluginKeys === undefined)config.pluginKeys={};
@ -45,6 +45,7 @@ module.exports = function(s){
if(config.orphanedVideoCheckMax === undefined){config.orphanedVideoCheckMax = 2}
if(config.detectorMergePamRegionTriggers === undefined){config.detectorMergePamRegionTriggers = false}
if(config.probeMonitorOnStart === undefined){config.probeMonitorOnStart = true}
if(config.showLoginTypeSelector === undefined){config.showLoginTypeSelector = true}
//Child Nodes
if(config.childNodes === undefined)config.childNodes = {};
//enabled
@ -215,12 +216,5 @@ module.exports = function(s){
}
]
}
if(config.cron.key === 'change_this_to_something_very_random__just_anything_other_than_this'){
console.error('!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!')
console.error('!! Change your cron key in your conf.json. !!')
console.error(`!! If you're running Shinobi remotely you should do this now. !!`)
console.error('!! You can do this in the Super User panel or from terminal. !!')
console.error('!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!')
}
return config
}

29
libs/control.js
View File

@ -1,6 +1,35 @@
var os = require('os');
var exec = require('child_process').exec;
module.exports = function(s,config,lang,app,io){
const {
startMove,
stopMove,
ptzControl,
} = require('./control/ptz.js')(s,config,lang)
require('./control/onvif.js')(s,config,lang,app,io)
require('./control/zwave.js')(s,config,lang,app,io)
// const ptz = require('./control/ptz.js')(s,config,lang,app,io)
s.onOtherWebSocketMessages((data,connection) => {
switch(data.f){
case'startMove':
startMove(data).then((response) => {
s.debugLog(response)
})
break;
case'stopMove':
stopMove(data).then((response) => {
s.debugLog(response)
})
break;
case'control':
ptzControl(data,function(msg){
s.userLog(data,msg);
connection.emit('f',{
f: 'control',
response: msg
})
})
break;
}
})
}

26
libs/control/onvif.js
View File

@ -1,6 +1,9 @@
var os = require('os');
var exec = require('child_process').exec;
const onvif = require("shinobi-onvif");
const {
addCredentialsToUrl,
} = require('../common.js')
module.exports = function(s,config,lang,app,io){
const {
createSnapshot,
@ -123,13 +126,28 @@ module.exports = function(s,config,lang,app,io){
}
}
async function getSnapshotFromOnvif(onvifOptions){
return await createSnapshot({
output: ['-s 400x400'],
url: addCredentialsToStreamLink({
let theUrl;
if(onvifOptions.mid && onvifOptions.ke){
const groupKey = onvifOptions.ke
const monitorId = onvifOptions.mid
const theDevice = s.group[groupKey].activeMonitors[monitorId].onvifConnection
theUrl = addCredentialsToUrl({
username: onvifOptions.username,
password: onvifOptions.password,
url: (await theDevice.services.media.getSnapshotUri({
ProfileToken : theDevice.current_profile.token,
})).GetSnapshotUriResponse.MediaUri.Uri
});
}else{
theUrl = addCredentialsToStreamLink({
username: onvifOptions.username,
password: onvifOptions.password,
url: onvifOptions.uri
}),
})
}
return await createSnapshot({
output: ['-s 400x400'],
url: theUrl,
})
}
/**

525
libs/control/ptz.js
View File

@ -1,300 +1,299 @@
var os = require('os');
var exec = require('child_process').exec;
var request = require('request')
module.exports = function(s,config,lang){
const { fetchWithAuthentication, asyncSetTimeout } = require('../basic/utils.js')(process.cwd(),config)
const moveLock = {}
const ptzTimeoutsUntilResetToHome = {}
const sliceUrlAuth = (url) => {
return /^(.+?\/\/)(?:.+?:.+?@)?(.+)$/.exec(url).slice(1).join('')
}
const startMove = async function(options,callback){
const device = s.group[options.ke].activeMonitors[options.id].onvifConnection
if(!device){
function getGenericControlParameters(optionsm,urlType){
const monitorConfig = s.group[options.ke].rawMonitorConfigurations[options.id]
const controlUrlMethod = monitorConfig.details.control_url_method || 'GET'
const controlBaseUrl = monitorConfig.details.control_base_url || s.buildMonitorUrl(monitorConfig, true)
let theURL;
if(urlType === 'start'){
theURL = controlBaseUrl + monitorConfig.details[`control_url_${options.direction}`]
}else{
theURL = controlBaseUrl + monitorConfig.details[`control_url_${options.direction}_stop`]
}
let controlOptions = s.cameraControlOptionsFromUrl(theURL,monitorConfig);
const hasDigestAuthEnabled = monitorConfig.details.control_digest_auth === '1'
const requestUrl = controlBaseUrl + controlOptions.path
return {
monitorConfig,
controlUrlMethod,
controlBaseUrl,
theURL,
controlOptions,
hasDigestAuthEnabled,
requestUrl,
}
}
function moveGeneric(options,doStart){
if(!s.group[options.ke] || !s.group[options.ke].activeMonitors[options.id]){return}
return new Promise((resolve,reject) => {
if(doStart)moveLock[options.ke + options.id] = true;
const {
monitorConfig,
controlUrlMethod,
controlBaseUrl,
theURL,
controlOptions,
hasDigestAuthEnabled,
requestUrl,
} = getGenericControlParameters(options,'start')
const response = {
ok: true,
type: lang[doStart ? 'Control Triggered' : 'Control Trigger Ended']
}
const theRequest = fetchWithAuthentication(requestUrl,{
method: controlUrlMethod || controlOptions.method,
digestAuth: hasDigestAuthEnabled,
body: controlOptions.postData || null
});
theRequest.then(res => res.text())
.then((data) => {
if(doStart){
const stopCommandEnabled = monitorConfig.details.control_stop === '1' || monitorConfig.details.control_stop === '2';
if(stopCommandEnabled && options.direction !== 'center'){
s.userLog(monitorConfig,{type: lang['Control Trigger Started']});
}else{
moveLock[options.ke + options.id] = false
s.userLog(monitorConfig,response);
}
}else{
moveLock[options.ke + options.id] = false
s.userLog(monitorConfig,response);
}
resolve(response)
});
theRequest.catch((err) => {
response.ok = false
response.type = lang['Control Error']
response.msg = err
resolve(response)
})
})
}
function getOnvifControlOptions(options){
const monitorConfig = s.group[options.ke].rawMonitorConfigurations[options.id]
const invertedVerticalAxis = monitorConfig.details.control_invert_y === '1'
const turnSpeed = parseFloat(monitorConfig.details.control_turn_speed) || 0.1
const controlOptions = {
Velocity : {}
}
if(options.axis){
options.axis.forEach((axis) => {
controlOptions.Velocity[axis.direction] = axis.amount < 0 ? -turnSpeed : axis.amount > 0 ? turnSpeed : 0
})
}else{
const onvifDirections = {
"left": [-turnSpeed,'x'],
"right": [turnSpeed,'x'],
"down": [invertedVerticalAxis ? turnSpeed : -turnSpeed,'y'],
"up": [invertedVerticalAxis ? -turnSpeed : turnSpeed,'y'],
"zoom_in": [turnSpeed,'z'],
"zoom_out": [-turnSpeed,'z']
}
const direction = onvifDirections[options.direction]
controlOptions.Velocity[direction[1]] = direction[0]
}
(['x','y','z']).forEach(function(axis){
if(!controlOptions.Velocity[axis])
controlOptions.Velocity[axis] = 0
})
return controlOptions
}
async function startMoveOnvif(options,callback){
const controlOptions = getOnvifControlOptions(options);
let activeMonitor = s.group[options.ke].activeMonitors[options.id]
let device = activeMonitor.onvifConnection
if(
!device ||
!device.current_profile ||
!device.current_profile.token
){
const response = await s.createOnvifDevice({
ke: options.ke,
id: options.id,
})
const device = s.group[options.ke].activeMonitors[options.id].onvifConnection
device = activeMonitor.onvifConnection
}
options.controlOptions.ProfileToken = device.current_profile.token
s.runOnvifMethod({
auth: {
ke: options.ke,
id: options.id,
action: 'continuousMove',
service: 'ptz',
},
options: options.controlOptions,
},callback)
function returnResponse(){
return new Promise((resolve,reject) => {
controlOptions.ProfileToken = device.current_profile.token
s.runOnvifMethod({
auth: {
ke: options.ke,
id: options.id,
action: 'continuousMove',
service: 'ptz',
},
options: controlOptions,
},resolve)
})
}
return await returnResponse();
}
const stopMove = function(options,callback){
const device = s.group[options.ke].activeMonitors[options.id].onvifConnection
try{
function stopMoveOnvif(options){
return new Promise((resolve,reject) => {
const device = s.group[options.ke].activeMonitors[options.id].onvifConnection
try{
s.runOnvifMethod({
auth: {
ke: options.ke,
id: options.id,
action: 'stop',
service: 'ptz',
},
options: {
'PanTilt': true,
'Zoom': true,
ProfileToken: device.current_profile.token
},
},resolve)
}catch(err){
resolve({ok: false})
}
})
}
function relativeMoveOnvif(options){
return new Promise((resolve,reject) => {
const controlOptions = getOnvifControlOptions(options);
controlOptions.Speed = {'x': 1, 'y': 1, 'z': 1}
controlOptions.Translation = Object.assign(controlOptions.Velocity,{})
delete(controlOptions.Velocity)
moveLock[options.ke + options.id] = true
s.runOnvifMethod({
auth: {
ke: options.ke,
id: options.id,
action: 'stop',
action: 'relativeMove',
service: 'ptz',
},
options: {
'PanTilt': true,
'Zoom': true,
ProfileToken: device.current_profile.token
},
},callback)
}catch(err){
callback({ok: false})
}
}
const moveOnvifCamera = function(options,callback){
const monitorConfig = s.group[options.ke].rawMonitorConfigurations[options.id]
const invertedVerticalAxis = monitorConfig.details.control_invert_y === '1'
const turnSpeed = parseFloat(monitorConfig.details.control_turn_speed) || 0.1
const controlUrlStopTimeout = parseInt(monitorConfig.details.control_url_stop_timeout) || 1000
switch(options.direction){
case'center':
moveLock[options.ke + options.id] = true
moveToPresetPosition({
ke: options.ke,
id: options.id,
},(endData) => {
moveLock[options.ke + options.id] = false
callback({type:'Moving to Home Preset', response: endData})
})
break;
case'stopMove':
callback({type:'Control Trigger Ended'})
stopMove({
ke: options.ke,
id: options.id,
},(response) => {
moveLock[options.ke + options.id] = false
})
break;
default:
try{
var controlOptions = {
Velocity : {}
}
if(options.axis){
options.axis.forEach((axis) => {
controlOptions.Velocity[axis.direction] = axis.amount < 0 ? -turnSpeed : axis.amount > 0 ? turnSpeed : 0
})
options: controlOptions,
},(response) => {
if(response.ok){
resolve({type: 'Control Triggered'})
}else{
var onvifDirections = {
"left": [-turnSpeed,'x'],
"right": [turnSpeed,'x'],
"down": [invertedVerticalAxis ? turnSpeed : -turnSpeed,'y'],
"up": [invertedVerticalAxis ? -turnSpeed : turnSpeed,'y'],
"zoom_in": [turnSpeed,'z'],
"zoom_out": [-turnSpeed,'z']
}
var direction = onvifDirections[options.direction]
controlOptions.Velocity[direction[1]] = direction[0]
resolve({type: 'Control Triggered', error: response.error})
}
(['x','y','z']).forEach(function(axis){
if(!controlOptions.Velocity[axis])
controlOptions.Velocity[axis] = 0
})
if(monitorConfig.details.control_stop === '1'){
moveLock[options.ke + options.id] = false
})
})
}
function moveOnvifCamera(options,doMove){
return new Promise((resolve,reject) => {
const monitorConfig = s.group[options.ke].rawMonitorConfigurations[options.id]
const controlUrlStopTimeout = parseInt(monitorConfig.details.control_url_stop_timeout) || 1000
options.direction = doMove ? options.direction : 'stopMove';
switch(options.direction){
case'center':
moveLock[options.ke + options.id] = true
startMove({
moveToPresetPosition({
ke: options.ke,
id: options.id,
controlOptions: controlOptions
},(response) => {
if(response.ok){
if(controlUrlStopTimeout != '0'){
setTimeout(function(){
stopMove({
ke: options.ke,
id: options.id,
},(response) => {
if(!response.ok){
s.systemLog(response)
}
moveLock[options.ke + options.id] = false
})
callback({type: 'Control Triggered'})
},controlUrlStopTimeout)
}
}else{
s.debugLog(response)
}
},(endData) => {
moveLock[options.ke + options.id] = false
resolve({ type: lang['Moving to Home Preset'], response: endData })
})
}else{
controlOptions.Speed = {'x': 1, 'y': 1, 'z': 1}
controlOptions.Translation = Object.assign(controlOptions.Velocity,{})
delete(controlOptions.Velocity)
moveLock[options.ke + options.id] = true
s.runOnvifMethod({
auth: {
ke: options.ke,
id: options.id,
action: 'relativeMove',
service: 'ptz',
},
options: controlOptions,
},(response) => {
if(response.ok){
callback({type: 'Control Triggered'})
}else{
callback({type: 'Control Triggered', error: response.error})
}
break;
case'stopMove':
resolve({ type: lang['Control Trigger Ended'] })
stopMoveOnvif({
ke: options.ke,
id: options.id,
}).then((response) => {
moveLock[options.ke + options.id] = false
})
}
}catch(err){
console.log(err)
console.log(new Error())
break;
default:
try{
moveLock[options.ke + options.id] = true
startMoveOnvif(options).then((moveResponse) => {
if(!moveResponse.ok){
s.debugLog('ONVIF Move Error',moveResponse)
}
resolve(moveResponse)
})
}catch(err){
console.log(err)
console.log(new Error())
}
break;
}
break;
})
}
async function startMove(options){
const monitorConfig = s.group[options.ke].rawMonitorConfigurations[options.id]
const controlUrlMethod = monitorConfig.details.control_url_method || 'GET'
if(controlUrlMethod === 'ONVIF'){
return await moveOnvifCamera(options,true);
}else{
return await moveGeneric(options,true);
}
}
const ptzControl = async function(options,callback){
async function stopMove(options){
const monitorConfig = s.group[options.ke].rawMonitorConfigurations[options.id]
const controlUrlMethod = monitorConfig.details.control_url_method || 'GET'
if(controlUrlMethod === 'ONVIF'){
return await moveOnvifCamera(options,false);
}else{
return await moveGeneric(options,false);
}
}
async function ptzControl(options,callback){
if(!s.group[options.ke] || !s.group[options.ke].activeMonitors[options.id]){return}
const monitorConfig = s.group[options.ke].rawMonitorConfigurations[options.id]
const controlUrlMethod = monitorConfig.details.control_url_method || 'GET'
const controlBaseUrl = monitorConfig.details.control_base_url || s.buildMonitorUrl(monitorConfig, true)
const controlUrlStopTimeout = options.moveTimeout || parseInt(monitorConfig.details.control_url_stop_timeout) || 1000
const stopCommandEnabled = monitorConfig.details.control_stop === '1' || monitorConfig.details.control_stop === '2';
if(monitorConfig.details.control !== "1"){
s.userLog(monitorConfig,{type:lang['Control Error'],msg:lang.ControlErrorText1});
return
s.userLog(monitorConfig,{
type: lang['Control Error'],
msg: lang.ControlErrorText1
});
return {
ok: false,
msg: lang.ControlErrorText1
}
}
if(monitorConfig.details.control_url_stop_timeout === '0' && monitorConfig.details.control_stop === '1' && s.group[options.ke].activeMonitors[options.id].ptzMoving === true){
options.direction = 'stopMove'
s.group[options.ke].activeMonitors[options.id].ptzMoving = false
}else{
s.group[options.ke].activeMonitors[options.id].ptzMoving = true
let response = {
direction: options.direction,
}
if(controlUrlMethod === 'ONVIF'){
try{
//create onvif connection
if(
!s.group[options.ke].activeMonitors[options.id].onvifConnection ||
!s.group[options.ke].activeMonitors[options.id].onvifConnection.current_profile ||
!s.group[options.ke].activeMonitors[options.id].onvifConnection.current_profile.token
){
const response = await s.createOnvifDevice({
ke: options.ke,
id: options.id,
})
if(response.ok){
moveOnvifCamera({
ke: options.ke,
id: options.id,
direction: options.direction,
axis: options.axis,
},(msg) => {
msg.msg = options.direction
callback(msg)
})
}else{
s.userLog(options,{type:lang['Control Error'],msg:response.error})
}
if(options.direction === 'center'){
response.moveResponse = await moveOnvifCamera(options,true)
}else if(stopCommandEnabled){
response.moveResponse = await moveOnvifCamera(options,true)
if(options.direction !== 'stopMove' && options.direction !== 'center'){
await asyncSetTimeout(controlUrlStopTimeout)
response.stopMoveResponse = await moveOnvifCamera(options,false)
response.ok = response.moveResponse.ok && response.stopMoveResponse.ok;
}else{
moveOnvifCamera({
ke: options.ke,
id: options.id,
direction: options.direction,
axis: options.axis,
},(msg) => {
if(!msg.msg)msg.msg = {direction: options.direction}
callback(msg)
})
response.ok = response.moveResponse.ok;
}
}catch(err){
s.debugLog(err)
callback({
type: lang['Control Error'],
msg: {
msg: lang.ControlErrorText2,
error: err,
direction: options.direction
}
})
}else{
response = await relativeMoveOnvif(options);
}
}else{
const controlUrlStopTimeout = parseInt(monitorConfig.details.control_url_stop_timeout) || 1000
var stopCamera = function(){
let stopURL = controlBaseUrl + monitorConfig.details[`control_url_${options.direction}_stop`]
let controlOptions = s.cameraControlOptionsFromUrl(stopURL,monitorConfig)
let requestOptions = {
url : controlBaseUrl + controlOptions.path,
method : controlOptions.method
}
if(controlOptions.username && controlOptions.password){
requestOptions.auth = {
user: controlOptions.username,
pass: controlOptions.password
}
}
if(controlOptions.postData){
requestOptions.form = controlOptions.postData
}
if(monitorConfig.details.control_digest_auth === '1'){
requestOptions.uri = sliceUrlAuth(requestOptions.url);
delete requestOptions.url;
requestOptions.auth.sendImmediately = false;
}
request(requestOptions,function(err,data){
const msg = {
ok: true,
type:'Control Trigger Ended'
}
if(err){
msg.ok = false
msg.type = 'Control Error'
msg.msg = err
}
moveLock[options.ke + options.id] = false
callback(msg)
s.userLog(monitorConfig,msg);
})
}
if(options.direction === 'stopMove'){
stopCamera()
response = await moveGeneric(options,false)
}else{
let controlURL = controlBaseUrl + monitorConfig.details[`control_url_${options.direction}`]
let controlOptions = s.cameraControlOptionsFromUrl(controlURL,monitorConfig)
let requestOptions = {
url: controlBaseUrl + controlOptions.path,
method: controlOptions.method
// left, right, up, down, center
response.moveResponse = await moveGeneric(options,true)
if(stopCommandEnabled){
await asyncSetTimeout(controlUrlStopTimeout)
response.stopMoveResponse = await moveGeneric(options,false)
response.ok = response.moveResponse.ok && response.stopMoveResponse.ok;
}else{
response.ok = response.moveResponse.ok;
}
if(controlOptions.username && controlOptions.password){
requestOptions.auth = {
user: controlOptions.username,
pass: controlOptions.password
}
}
if(controlOptions.postData){
requestOptions.form = controlOptions.postData
}
if(monitorConfig.details.control_digest_auth === '1'){
requestOptions.uri = sliceUrlAuth(requestOptions.url);
delete requestOptions.url;
requestOptions.auth.sendImmediately = false;
}
moveLock[options.ke + options.id] = true
request(requestOptions,function(err,data){
if(err){
callback({ok:false,type:'Control Error',msg:err})
return
}
if(monitorConfig.details.control_stop == '1' && options.direction !== 'center' ){
s.userLog(monitorConfig,{type:'Control Triggered Started'});
if(controlUrlStopTimeout > 0){
setTimeout(function(){
stopCamera()
},controlUrlStopTimeout)
}
}else{
moveLock[options.ke + options.id] = false
callback({ok:true,type:'Control Triggered'})
}
})
}
}
if(callback)callback(response);
return response;
}
const getPresetPositions = (options,callback) => {
const profileToken = options.ProfileToken || "__CURRENT_TOKEN"
@ -313,7 +312,6 @@ module.exports = function(s,config,lang){
const setPresetForCurrentPosition = (options,callback) => {
const nonStandardOnvif = s.group[options.ke].rawMonitorConfigurations[options.id].details.onvif_non_standard === '1'
const profileToken = options.ProfileToken || "__CURRENT_TOKEN"
console.log(options.PresetToken)
s.runOnvifMethod({
auth: {
ke: options.ke,
@ -380,7 +378,6 @@ module.exports = function(s,config,lang){
const imageCenterY = imgHeight / 2
const matrices = event.details.matrices || []
const largestMatrix = getLargestMatrix(matrices.filter(matrix => trackingTarget.indexOf(matrix.tag) > -1))
// console.log(matrices.find(matrix => matrix.tag === 'person'))
if(!largestMatrix)return;
const monitorConfig = s.group[event.ke].rawMonitorConfigurations[event.id]
const invertedVerticalAxis = monitorConfig.details.control_invert_y === '1'
@ -389,6 +386,10 @@ module.exports = function(s,config,lang){
const matrixCenterY = largestMatrix.y + (largestMatrix.height / 2)
const rawDistanceX = (matrixCenterX - imageCenterX)
const rawDistanceY = (matrixCenterY - imageCenterY)
const percentX = parseFloat((rawDistanceX / imgWidth).toFixed(2));
const percentY = parseFloat((rawDistanceY / imgHeight).toFixed(2));
const turnSpeedX = parseFloat(monitorConfig.details.control_turn_speed) || 0.1
const turnSpeedY = parseFloat(monitorConfig.details.control_turn_speed) || 0.1
const distanceX = imgWidth / rawDistanceX
const distanceY = imgHeight / rawDistanceY
const axisX = rawDistanceX > thresholdX || rawDistanceX < -thresholdX ? distanceX : 0
@ -396,11 +397,11 @@ module.exports = function(s,config,lang){
if(axisX !== 0 || axisY !== 0){
ptzControl({
axis: [
{direction: 'x', amount: axisX === 0 ? 0 : axisX > 0 ? turnSpeed : -turnSpeed},
{direction: 'y', amount: axisY === 0 ? 0 : axisY > 0 ? invertedVerticalAxis ? -turnSpeed : turnSpeed : invertedVerticalAxis ? turnSpeed : -turnSpeed},
{direction: 'x', amount: axisX === 0 ? 0 : axisX > 0 ? turnSpeedX : -turnSpeedX},
{direction: 'y', amount: axisY === 0 ? 0 : axisY > 0 ? invertedVerticalAxis ? -turnSpeedY : turnSpeedY : invertedVerticalAxis ? turnSpeedY : -turnSpeedY},
{direction: 'z', amount: 0},
],
// axis: [{direction: 'x', amount: 1.0}],
moveTimeout: 500,
id: event.id,
ke: event.ke
},(msg) => {
@ -413,12 +414,12 @@ module.exports = function(s,config,lang){
}
}
return {
ptzControl: ptzControl,
startMove: startMove,
stopMove: stopMove,
getPresetPositions: getPresetPositions,
setPresetForCurrentPosition: setPresetForCurrentPosition,
moveToPresetPosition: moveToPresetPosition,
moveCameraPtzToMatrix: moveCameraPtzToMatrix
startMove,
stopMove,
ptzControl,
getPresetPositions,
setPresetForCurrentPosition,
moveToPresetPosition,
moveCameraPtzToMatrix,
}
}

134
libs/control/zwave.js Normal file
View File

@ -0,0 +1,134 @@
const zWaveAPI = require('shinobi-zwave')
module.exports = async (s,config,lang,app,io) => {
const addCredentialsToHostLink = (url,username,password) => {
if(url.indexOf('@') > -1){
return url
}else if(username){
const urlParts = url.split('://')
return [urlParts[0],'://',`${username}:${password || ''}@`,urlParts[1]].join('')
}else{
return url
}
}
function loadApplicationForGroup(user){
const userDetails = s.parseJSON(user.details);
s.debugLog('Z-Wave','Loading API',user.ke)
if(
!s.group[user.ke].zwave &&
userDetails.zwave === '1' &&
userDetails.zwave_host
){
const zWaveHost = addCredentialsToHostLink(
userDetails.zwave_host || config.zWaveHost,
userDetails.zwave_user,
userDetails.zwave_pass
).replace(/\/$/, '');
s.group[user.ke].zwave = zWaveAPI(zWaveHost,config.debugLogZwave || false)
s.debugLog('Z-Wave','Loaded API',zWaveHost)
}
}
function unloadApplicationForGroup(user){
s.group[user.ke].zwave = null
}
/**
* API : Z-Wave HTTP Handler
*/
function httpApiHandler(req,res){
s.auth(req.params,async (user) => {
const endData = {ok: true}
const actionFunction = req.params.action
const arguments = s.getPostData(req) || []
try{
const groupKey = req.params.ke
endData.response = await s.group[groupKey].zwave[actionFunction](...arguments)
}catch(err){
endData.ok = false
endData.err = err
s.debugLog(err)
}
s.closeJsonResponse(res,endData)
},res,req);
}
app.get(config.webPaths.apiPrefix+':auth/zwave/:ke/:action',httpApiHandler)
app.post(config.webPaths.apiPrefix+':auth/zwave/:ke/:action',httpApiHandler)
app.get(config.webPaths.apiPrefix+':auth/zwaveRaw/:ke',(req,res) => {
s.auth(req.params,async (user) => {
const groupKey = req.params.ke
const pathString = s.getPostData(req,'path')
if(!pathString){
res.end(lang['Not Found'])
return
}
s.group[groupKey].zwave.httpRequest(pathString).pipe(res)
},res,req);
})
s.definitions["Account Settings"].blocks["Z-Wave"] = {
"evaluation": "$user.details.use_zwave !== '0'",
"name": lang['Z-Wave'],
"id":"accSectionZwave",
"isSection": true,
"color": "blue",
"info": [
{
"name": "detail=zwave",
"selector":"u_zwave_bot",
"field": lang.Enabled,
"default": "0",
"example": "",
"fieldType": "select",
"possible": [
{
"name": lang.No,
"value": "0"
},
{
"name": lang.Yes,
"value": "1"
}
]
},
{
hidden: true,
"name": "detail=zwave_host",
"placeholder": "https://localhost:8083",
"field": lang.Host,
"form-group-class":"u_zwave_bot_input u_zwave_bot_1",
},
{
hidden: true,
"name": "detail=zwave_user",
"placeholder": lang["Username"],
"field": lang["Username"],
"form-group-class":"u_zwave_bot_input u_zwave_bot_1",
},
{
hidden: true,
"name": "detail=zwave_pass",
"placeholder": lang["Password"],
"fieldType": "password",
"field": lang["Password"],
"form-group-class":"u_zwave_bot_input u_zwave_bot_1",
}
]
}
s.definitions["Z-Wave Manager"] = {
"name": lang["Z-Wave Manager"],
blocks: {
"Container1": {
"evaluation": "$user.details.use_zwave !== '0'",
noHeader: true,
noDefaultSectionClasses: true,
"color": "green",
"section-pre-class": "col-md-8 search-parent",
"info": [
{
"id": "zwaveDevices",
"fieldType": "div",
},
]
}
}
}
s.loadGroupAppExtender(loadApplicationForGroup)
s.unloadGroupAppExtender(unloadApplicationForGroup)
}

63
libs/cron.js Normal file
View File

@ -0,0 +1,63 @@
const { Worker } = require('worker_threads');
const moment = require('moment');
module.exports = (s,config,lang) => {
const {
legacyFilterEvents
} = require('./events/utils.js')(s,config,lang)
if(config.doCronAsWorker===undefined)config.doCronAsWorker = true;
const startWorker = () => {
const pathToWorkerScript = __dirname + `/cron/worker.js`
const workerProcess = new Worker(pathToWorkerScript)
workerProcess.on('message',function(data){
if(data.time === 'moment()')data.time = moment();
switch(data.f){
case'debugLog':
s.debugLog(...data.data)
break;
case'systemLog':
s.systemLog(...data.data)
break;
case'filters':
legacyFilterEvents(data.ff,data)
break;
case's.tx':
s.tx(data.data,data.to)
break;
case's.deleteVideo':
s.deleteVideo(data.file)
break;
case's.deleteFileBinEntry':
s.deleteFileBinEntry(data.file)
break;
case's.setDiskUsedForGroup':
function doOnMain(){
s.setDiskUsedForGroup(data.ke,data.size,data.target || undefined)
}
if(data.videoRow){
let storageIndex = s.getVideoStorageIndex(data.videoRow);
if(storageIndex){
s.setDiskUsedForGroupAddStorage(data.ke,{
size: data.size,
storageIndex: storageIndex
})
}else{
doOnMain()
}
}else{
doOnMain()
}
break;
default:
s.systemLog('CRON.js MESSAGE : ',data)
break;
}
})
setTimeout(() => {
workerProcess.postMessage({
f: 'init',
})
},2000)
return workerProcess
}
if(config.doCronAsWorker === true)startWorker()
}

612
libs/cron/worker.js Normal file
View File

@ -0,0 +1,612 @@
const fs = require('fs');
const path = require('path');
const moment = require('moment');
const exec = require('child_process').exec;
const spawn = require('child_process').spawn;
const { parentPort, isMainThread } = require('worker_threads');
const config = require(process.cwd() + '/conf.json')
process.on('uncaughtException', function (err) {
errorLog('uncaughtException',err);
});
if(isMainThread){
console.log(`Shinobi now runs cron.js as child process.`)
console.error(`Shinobi now runs cron.js as child process.`)
setInterval(() => {
// console.log(`Please turn off cron.js process.`)
},1000 * 60 * 60 * 24 * 7)
return;
}
function setDefaultConfigOptions(){
if(config.cron===undefined)config.cron={};
if(config.cron.deleteOld===undefined)config.cron.deleteOld=true;
if(config.cron.deleteOrphans===undefined)config.cron.deleteOrphans=false;
if(config.cron.deleteNoVideo===undefined)config.cron.deleteNoVideo=true;
if(config.cron.deleteNoVideoRecursion===undefined)config.cron.deleteNoVideoRecursion=false;
if(config.cron.deleteOverMax===undefined)config.cron.deleteOverMax=true;
if(config.cron.deleteLogs===undefined)config.cron.deleteLogs=true;
if(config.cron.deleteTimelpaseFrames===undefined)config.cron.deleteTimelpaseFrames=true;
if(config.cron.deleteEvents===undefined)config.cron.deleteEvents=true;
if(config.cron.deleteFileBins===undefined)config.cron.deleteFileBins=true;
if(config.cron.interval===undefined)config.cron.interval=1;
if(config.databaseType===undefined){config.databaseType='mysql'}
if(config.databaseLogs===undefined){config.databaseLogs=false}
if(config.debugLog===undefined){config.debugLog=false}
if(!config.ip||config.ip===''||config.ip.indexOf('0.0.0.0')>-1)config.ip='localhost';
if(!config.videosDir)config.videosDir = process.cwd() + '/videos/';
if(!config.binDir){config.binDir = process.cwd() + '/fileBin/'}
}
parentPort.on('message',(data) => {
switch(data.f){
case'init':
setDefaultConfigOptions()
beginProcessing()
break;
case'start':case'restart':
setIntervalForCron()
break;
case'stop':
clearCronInterval()
break;
}
})
function debugLog(...args){
if(config.debugLog === true){
console.log(...([`CRON.js DEBUG LOG ${new Date()}`].concat(args)))
}
}
function normalLog(...args){
console.log(...([`CRON.js LOG ${new Date()}`].concat(args)))
}
function errorLog(...args){
console.error(...([`CRON.js ERROR LOG ${new Date()}`].concat(args)))
}
const s = {
debugLog,
}
function beginProcessing(){
normalLog(`Worker Processing!`)
const {
checkCorrectPathEnding,
generateRandomId,
formattedTime,
localToUtc,
} = require('../basic/utils.js')(process.cwd())
const {
sqlDate,
knexQuery,
knexQueryPromise,
initiateDatabaseEngine
} = require('../sql/utils.js')(s,config)
var theCronInterval = null
const overlapLocks = {}
const alreadyDeletedRowsWithNoVideosOnStart = {}
const videoDirectory = checkCorrectPathEnding(config.videosDir)
const fileBinDirectory = checkCorrectPathEnding(config.binDir)
const postMessage = (data) => {
parentPort.postMessage(data)
}
const sendToWebSocket = (x,y) => {
//emulate master socket emitter
postMessage({f:'s.tx',data:x,to:y})
}
const deleteVideo = (x) => {
postMessage({f:'s.deleteVideo',file:x})
}
const deleteFileBinEntry = (x) => {
postMessage({f:'s.deleteFileBinEntry',file:x})
}
const setDiskUsedForGroup = (groupKey,size,target,videoRow) => {
postMessage({f:'s.setDiskUsedForGroup', ke: groupKey, size: size, target: target, videoRow: videoRow})
}
const getVideoDirectory = function(e){
if(e.mid&&!e.id){e.id=e.mid};
if(e.details&&(e.details instanceof Object)===false){
try{e.details=JSON.parse(e.details)}catch(err){}
}
if(e.details.dir&&e.details.dir!==''){
return checkCorrectPathEnding(e.details.dir)+e.ke+'/'+e.id+'/'
}else{
return videoDirectory + e.ke + '/' + e.id + '/'
}
}
const getTimelapseFrameDirectory = function(e){
if(e.mid&&!e.id){e.id=e.mid}
if(e.details&&(e.details instanceof Object)===false){
try{e.details=JSON.parse(e.details)}catch(err){}
}
if(e.details&&e.details.dir&&e.details.dir!==''){
return checkCorrectPathEnding(e.details.dir)+e.ke+'/'+e.id+'_timelapse/'
}else{
return videoDirectory + e.ke + '/' + e.id + '_timelapse/'
}
}
const getFileBinDirectory = function(e){
if(e.mid && !e.id){e.id = e.mid}
return fileBinDirectory + e.ke + '/' + e.id + '/'
}
//filters set by the user in their dashboard
//deleting old videos is part of the filter - config.cron.deleteOld
const checkFilterRules = function(v){
return new Promise((resolve,reject) => {
//filters
v.d.filters = v.d.filters ? v.d.filters : {}
debugLog('Checking Basic Filters...')
var keys = Object.keys(v.d.filters)
if(keys.length>0){
keys.forEach(function(m,current){
// b = filter
var b = v.d.filters[m];
debugLog(b)
if(b.enabled==="1"){
const whereQuery = [
['ke','=',v.ke],
['status','!=',"0"],
['archive','!=',`1`],
]
b.where.forEach(function(condition){
if(condition.p1 === 'ke'){condition.p3 = v.ke}
whereQuery.push([condition.p1,condition.p2 || '=',condition.p3])
})
knexQuery({
action: "select",
columns: "*",
table: "Videos",
where: whereQuery,
orderBy: [b.sort_by,b.sort_by_direction.toLowerCase()],
limit: b.limit
},(err,r) => {
if(r && r[0]){
if(r.length > 0 || config.debugLog === true){
postMessage({f:'filterMatch',msg:r.length+' SQL rows match "'+m+'"',ke:v.ke,time:'moment()'})
}
b.cx={
f:'filters',
name:b.name,
videos:r,
time:'moment()',
ke:v.ke,
id:b.id
};
if(b.archive==="1"){
postMessage({f:'filters',ff:'archive',videos:r,time:'moment()',ke:v.ke,id:b.id});
}else if(b.delete==="1"){
postMessage({f:'filters',ff:'delete',videos:r,time:'moment()',ke:v.ke,id:b.id});
}
if(b.email==="1"){
b.cx.ff='email';
b.cx.delete=b.delete;
b.cx.mail=v.mail;
b.cx.execute=b.execute;
b.cx.query=b.sql;
postMessage(b.cx);
}
if(b.execute&&b.execute!==""){
postMessage({f:'filters',ff:'execute',execute:b.execute,time:'moment()'});
}
}
})
}
if(current===keys.length-1){
//last filter
resolve()
}
})
}else{
//no filters
resolve()
}
})
}
const deleteVideosByDays = async (v,days,addedQueries) => {
const groupKey = v.ke;
const whereQuery = [
['ke','=',v.ke],
['archive','!=',`1`],
['time','<', sqlDate(days+' DAY')],
addedQueries
]
const selectResponse = await knexQueryPromise({
action: "select",
columns: "*",
table: "Videos",
where: whereQuery
})
const videoRows = selectResponse.rows
let affectedRows = 0
if(videoRows.length > 0){
let clearSize = 0;
var i;
for (i = 0; i < videoRows.length; i++) {
const row = videoRows[i];
const dir = getVideoDirectory(row)
const filename = formattedTime(row.time) + '.' + row.ext
try{
await fs.promises.unlink(dir + filename)
const fileSizeMB = row.size / 1048576;
setDiskUsedForGroup(groupKey,-fileSizeMB,null,row)
sendToWebSocket({
f: 'video_delete',
filename: filename + '.' + row.ext,
mid: row.mid,
ke: row.ke,
time: row.time,
end: formattedTime(new Date,'YYYY-MM-DD HH:mm:ss')
},'GRP_' + row.ke)
}catch(err){
normalLog('Video Delete Error',row)
normalLog(err)
}
}
const deleteResponse = await knexQueryPromise({
action: "delete",
table: "Videos",
where: whereQuery
})
affectedRows = deleteResponse.rows || 0
}
return {
ok: true,
affectedRows: affectedRows,
}
}
const deleteOldVideos = async (v) => {
// v = group, admin user
if(config.cron.deleteOld === true){
const daysOldForDeletion = v.d.days && !isNaN(v.d.days) ? parseFloat(v.d.days) : 5
const monitorsIgnored = []
const monitorsResponse = await knexQueryPromise({
action: "select",
columns: "*",
table: "Monitors",
where: [
['ke','=',v.ke],
]
})
const monitorRows = monitorsResponse.rows
var i;
for (i = 0; i < monitorRows.length; i++) {
const monitor = monitorRows[i]
const monitorId = monitor.mid
const details = JSON.parse(monitor.details);
const monitorsMaxDaysToKeep = !isNaN(details.max_keep_days) ? parseFloat(details.max_keep_days) : null
if(monitorsMaxDaysToKeep){
const { affectedRows } = await deleteVideosByDays(v,monitorsMaxDaysToKeep,['mid','=',monitorId])
const hasDeletedRows = affectedRows && affectedRows.length > 0;
if(hasDeletedRows || config.debugLog === true){
postMessage({
f: 'deleteOldVideosByMonitorId',
msg: `${affectedRows} SQL rows older than ${monitorsMaxDaysToKeep} days deleted`,
ke: v.ke,
mid: monitorId,
time: 'moment()',
})
}
monitorsIgnored.push(['mid','!=',monitorId])
}
}
const { affectedRows } = await deleteVideosByDays(v,daysOldForDeletion,monitorsIgnored)
const hasDeletedRows = affectedRows && affectedRows.length > 0;
if(hasDeletedRows || config.debugLog === true){
postMessage({
f: 'deleteOldVideos',
msg: `${affectedRows} SQL rows older than ${daysOldForDeletion} days deleted`,
ke: v.ke,
time: 'moment()',
})
}
}
}
//database rows with no videos in the filesystem
const deleteRowsWithNoVideo = function(v){
return new Promise((resolve,reject) => {
if(
config.cron.deleteNoVideo===true&&(
config.cron.deleteNoVideoRecursion===true||
(config.cron.deleteNoVideoRecursion===false&&!alreadyDeletedRowsWithNoVideosOnStart[v.ke])
)
){
alreadyDeletedRowsWithNoVideosOnStart[v.ke]=true;
knexQuery({
action: "select",
columns: "*",
table: "Videos",
where: [
['ke','=',v.ke],
['status','!=','0'],
['archive','!=',`1`],
['time','<', sqlDate('10 MINUTE')],
]
},(err,evs) => {
if(evs && evs[0]){
const videosToDelete = [];
evs.forEach(function(ev){
var filename
var details
try{
details = JSON.parse(ev.details)
}catch(err){
if(details instanceof Object){
details = ev.details
}else{
details = {}
}
}
var dir = getVideoDirectory(ev)
filename = formattedTime(ev.time)+'.'+ev.ext
fileExists = fs.existsSync(dir+filename)
if(fileExists !== true){
deleteVideo(ev)
sendToWebSocket({f:'video_delete',filename:filename+'.'+ev.ext,mid:ev.mid,ke:ev.ke,time:ev.time,end: formattedTime(new Date,'YYYY-MM-DD HH:mm:ss')},'GRP_'+ev.ke);
}
});
if(videosToDelete.length > 0 || config.debugLog === true){
postMessage({f:'deleteNoVideo',msg:videosToDelete.length+' SQL rows with no file deleted',ke:v.ke,time:'moment()'})
}
}
setTimeout(function(){
resolve()
},3000)
})
}else{
resolve()
}
})
}
//info about what the application is doing
const deleteOldLogs = function(v){
return new Promise((resolve,reject) => {
const daysOldForDeletion = v.d.log_days && !isNaN(v.d.log_days) ? parseFloat(v.d.log_days) : 10
if(config.cron.deleteLogs === true && daysOldForDeletion !== 0){
knexQuery({
action: "delete",
table: "Logs",
where: [
['ke','=',v.ke],
['time','<', sqlDate(daysOldForDeletion + ' DAY')],
]
},(err,rrr) => {
resolve()
if(err)return errorLog(err);
if(rrr && rrr > 0 || config.debugLog === true){
postMessage({f:'deleteLogs',msg: rrr + ' SQL rows older than ' + daysOldForDeletion + ' days deleted',ke:v.ke,time:'moment()'})
}
})
}else{
resolve()
}
})
}
//still images saved
const deleteOldTimelapseFrames = async function(v){
const daysOldForDeletion = v.d.timelapseFrames_days && !isNaN(v.d.timelapseFrames_days) ? parseFloat(v.d.timelapseFrames_days) : 60
if(config.cron.deleteTimelpaseFrames === true && daysOldForDeletion !== 0){
const groupKey = v.ke;
const whereQuery = [
['ke','=',v.ke],
['archive','!=',`1`],
['time','<', sqlDate(daysOldForDeletion+' DAY')],
]
const selectResponse = await knexQueryPromise({
action: "select",
columns: "*",
table: "Timelapse Frames",
where: whereQuery
})
const videoRows = selectResponse.rows
let affectedRows = 0
if(videoRows.length > 0){
const foldersDeletedFrom = [];
let clearSize = 0;
var i;
for (i = 0; i < videoRows.length; i++) {
const row = videoRows[i];
const dir = getTimelapseFrameDirectory(row)
const filename = row.filename
const theDate = filename.split('T')[0]
const enclosingFolder = `${dir}/${theDate}/`
try{
const fileSizeMB = row.size / 1048576;
setDiskUsedForGroup(groupKey,-fileSizeMB,null,row)
sendToWebSocket({
f: 'timelapse_frame_delete',
filename: filename,
mid: row.mid,
ke: groupKey,
time: row.time,
details: row.details,
},'GRP_' + groupKey)
await fs.promises.unlink(`${enclosingFolder}${filename}`)
if(foldersDeletedFrom.indexOf(enclosingFolder) === -1)foldersDeletedFrom.push(enclosingFolder);
}catch(err){
normalLog('Timelapse Frame Delete Error',row)
normalLog(err)
}
}
for (i = 0; i < foldersDeletedFrom.length; i++) {
const folderPath = foldersDeletedFrom[i];
const folderIsEmpty = (await fs.promises.readdir(folderPath)).filter(file => file.indexOf('.jpg') > -1).length === 0;
if(folderIsEmpty){
await fs.promises.rm(folderPath, { recursive: true, force: true })
}
}
const deleteResponse = await knexQueryPromise({
action: "delete",
table: "Timelapse Frames",
where: whereQuery
})
affectedRows = deleteResponse.rows || 0
}
return {
ok: true,
affectedRows: affectedRows,
}
}
return {
ok: false
}
}
//events - motion, object, etc. detections
const deleteOldEvents = function(v){
return new Promise((resolve,reject) => {
const daysOldForDeletion = v.d.event_days && !isNaN(v.d.event_days) ? parseFloat(v.d.event_days) : 10
if(config.cron.deleteEvents === true && daysOldForDeletion !== 0){
knexQuery({
action: "delete",
table: "Events",
where: [
['ke','=',v.ke],
['archive','!=',`1`],
['time','<', sqlDate(daysOldForDeletion + ' DAY')],
]
},(err,rrr) => {
resolve()
if(err)return errorLog(err);
if(rrr && rrr > 0 || config.debugLog === true){
postMessage({f:'deleteEvents',msg:rrr + ' SQL rows older than ' + daysOldForDeletion + ' days deleted',ke:v.ke,time:'moment()'})
}
})
}else{
resolve()
}
})
}
//event counts
const deleteOldEventCounts = function(v){
return new Promise((resolve,reject) => {
const daysOldForDeletion = v.d.event_days && !isNaN(v.d.event_days) ? parseFloat(v.d.event_days) : 10
if(config.cron.deleteEvents === true && daysOldForDeletion !== 0){
knexQuery({
action: "delete",
table: "Events Counts",
where: [
['ke','=',v.ke],
['time','<', sqlDate(daysOldForDeletion + ' DAY')],
]
},(err,rrr) => {
resolve()
if(err && err.code !== 'ER_NO_SUCH_TABLE')return errorLog(err);
if(rrr && rrr > 0 || config.debugLog === true){
postMessage({f:'deleteEvents',msg:rrr + ' SQL rows older than ' + daysOldForDeletion + ' days deleted',ke:v.ke,time:'moment()'})
}
})
}else{
resolve()
}
})
}
//check for temporary files (special archive)
const deleteOldFileBins = function(v){
return new Promise((resolve,reject) => {
const daysOldForDeletion = v.d.fileBin_days && !isNaN(v.d.fileBin_days) ? parseFloat(v.d.fileBin_days) : 10
if(config.cron.deleteFileBins === true && daysOldForDeletion !== 0){
var fileBinQuery = " FROM Files WHERE ke=? AND `time` < ?";
knexQuery({
action: "select",
columns: "*",
table: "Files",
where: [
['ke','=',v.ke],
['archive','!=',`1`],
['time','<', sqlDate(daysOldForDeletion + ' DAY')],
]
},(err,files) => {
if(files && files[0]){
//delete the files
files.forEach(function(file){
deleteFileBinEntry(file)
})
if(config.debugLog === true){
postMessage({
f: 'deleteFileBins',
msg: files.length + ' files older than ' + daysOldForDeletion + ' days deleted',
ke: v.ke,
time: 'moment()'
})
}
}
resolve()
})
}else{
resolve()
}
})
}
//user processing function
const processUser = async (v) => {
if(!v){
//no user object given, end of group list
return
}
debugLog(`Group Key : ${v.ke}`)
debugLog(`Owner : ${v.mail}`)
if(!overlapLocks[v.ke]){
debugLog(`Checking...`)
overlapLocks[v.ke] = true
v.d = JSON.parse(v.details);
try{
await deleteOldVideos(v)
debugLog('--- deleteOldVideos Complete')
await deleteOldTimelapseFrames(v)
debugLog('--- deleteOldTimelapseFrames Complete')
await deleteOldLogs(v)
debugLog('--- deleteOldLogs Complete')
await deleteOldFileBins(v)
debugLog('--- deleteOldFileBins Complete')
await deleteOldEvents(v)
debugLog('--- deleteOldEvents Complete')
await deleteOldEventCounts(v)
debugLog('--- deleteOldEventCounts Complete')
await checkFilterRules(v)
debugLog('--- checkFilterRules Complete')
await deleteRowsWithNoVideo(v)
debugLog('--- deleteRowsWithNoVideo Complete')
}catch(err){
normalLog(`Failed to Complete User : ${v.mail}`)
normalLog(err)
}
//done user, unlock current, and do next
overlapLocks[v.ke] = false;
debugLog(`Complete Checking... ${v.mail}`)
}else{
debugLog(`Locked, Skipped...`)
}
}
//recursive function
const setIntervalForCron = function(){
clearCronInterval()
// theCronInterval = setInterval(doCronJobs,1000 * 10)
theCronInterval = setInterval(doCronJobs,parseFloat(config.cron.interval)*60000*60)
}
const clearCronInterval = function(){
clearInterval(theCronInterval)
}
const doCronJobs = function(){
postMessage({
f: 'start',
time: 'moment()'
})
knexQuery({
action: "select",
columns: "ke,uid,details,mail",
table: "Users",
where: [
['details','NOT LIKE','%"sub"%'],
]
}, async (err,rows) => {
if(err){
errorLog(err)
}
if(rows.length > 0){
var i;
for (i = 0; i < rows.length; i++) {
await processUser(rows[i])
}
}
})
}
initiateDatabaseEngine()
setIntervalForCron()
doCronJobs()
}

104
libs/customAutoLoad.js
View File

@ -1,10 +1,10 @@
const fs = require('fs-extra');
const express = require('express')
const request = require('request')
const unzipper = require('unzipper')
const fetch = require("node-fetch")
const spawn = require('child_process').spawn
module.exports = async (s,config,lang,app,io) => {
const { fetchDownloadAndWrite } = require('./basic/utils.js')(process.cwd(),config)
s.debugLog(`+++++++++++CustomAutoLoad Modules++++++++++++`)
const runningInstallProcesses = {}
const modulesBasePath = __dirname + '/customAutoLoad/'
@ -18,7 +18,7 @@ module.exports = async (s,config,lang,app,io) => {
}
const getModule = (moduleName) => {
s.debugLog(`+++++++++++++++++++++++`)
s.debugLog(`Loading : ${moduleName}`)
s.debugLog(`Getting Module : ${moduleName}`)
const modulePath = modulesBasePath + moduleName
const stats = fs.lstatSync(modulePath)
const isDirectory = stats.isDirectory()
@ -65,10 +65,9 @@ module.exports = async (s,config,lang,app,io) => {
fs.mkdirSync(downloadPath)
return new Promise(async (resolve, reject) => {
fs.mkdir(downloadPath, () => {
request(downloadUrl).pipe(fs.createWriteStream(downloadPath + '.zip'))
.on('finish',() => {
zip = fs.createReadStream(downloadPath + '.zip')
.pipe(unzipper.Parse())
fetchDownloadAndWrite(downloadUrl,downloadPath + '.zip', 1)
.then((readStream) => {
readStream.pipe(unzipper.Parse())
.on('entry', async (file) => {
if(file.type === 'Directory'){
try{
@ -176,7 +175,9 @@ module.exports = async (s,config,lang,app,io) => {
}
}
const loadModule = (shinobiModule) => {
s.debugLog(`+++++++++++++++++++++++`)
const moduleName = shinobiModule.name
s.debugLog(`Loading Module : ${moduleName}`)
s.customAutoLoadModules[moduleName] = {}
var customModulePath = modulesBasePath + '/' + moduleName
s.debugLog(customModulePath)
@ -208,8 +209,13 @@ module.exports = async (s,config,lang,app,io) => {
app.use('/libs',express.static(webFolder + '/libs'))
}
app.use(s.checkCorrectPathEnding(config.webPaths.home)+'libs',express.static(webFolder + '/libs'))
app.use(s.checkCorrectPathEnding(config.webPaths.admin)+'libs',express.static(webFolder + '/libs'))
app.use(s.checkCorrectPathEnding(config.webPaths.super)+'libs',express.static(webFolder + '/libs'))
}else if(name === 'assets'){
if(config.webPaths.home !== '/'){
app.use('/assets',express.static(webFolder + '/assets'))
}
app.use(s.checkCorrectPathEnding(config.webPaths.home)+'assets',express.static(webFolder + '/assets'))
app.use(s.checkCorrectPathEnding(config.webPaths.super)+'assets',express.static(webFolder + '/assets'))
}
var libFolder = webFolder + name + '/'
fs.readdir(libFolder,function(err,webFolderContents){
@ -225,23 +231,34 @@ module.exports = async (s,config,lang,app,io) => {
var fullPath = thirdLevelName + '/' + filename
var blockPrefix = ''
switch(true){
case filename.contains('super.'):
case filename.indexOf('super.') > -1:
blockPrefix = 'super'
break;
case filename.contains('admin.'):
blockPrefix = 'admin'
break;
}
switch(libName){
case'js':
s.customAutoLoadTree[blockPrefix + 'LibsJs'].push(filename)
break;
case'css':
s.customAutoLoadTree[blockPrefix + 'LibsCss'].push(filename)
break;
case'blocks':
s.customAutoLoadTree[blockPrefix + 'PageBlocks'].push(fullPath)
break;
if(name === 'libs'){
switch(libName){
case'js':
s.customAutoLoadTree[blockPrefix + 'LibsJs'].push(filename)
break;
case'css':
s.customAutoLoadTree[blockPrefix + 'LibsCss'].push(filename)
break;
case'blocks':
s.customAutoLoadTree[blockPrefix + 'PageBlocks'].push(fullPath)
break;
}
}else if(name === 'assets'){
switch(libName){
case'js':
s.customAutoLoadTree[blockPrefix + 'AssetsJs'].push(filename)
break;
case'css':
s.customAutoLoadTree[blockPrefix + 'AssetsCss'].push(filename)
break;
case'blocks':
s.customAutoLoadTree[blockPrefix + 'PageBlocks'].push(fullPath)
break;
}
}
})
})
@ -278,22 +295,24 @@ module.exports = async (s,config,lang,app,io) => {
})
break;
case'definitions':
var definitionsFolder = s.checkCorrectPathEnding(customModulePath) + 'definitions/'
fs.readdir(definitionsFolder,function(err,files){
if(err)return console.log(err);
files.forEach(function(filename){
var fileData = require(definitionsFolder + filename)
var rule = filename.replace('.json','').replace('.js','')
if(config.language === rule){
s.definitions = s.mergeDeep(s.definitions,fileData)
}
if(s.loadedDefinitons[rule]){
s.loadedDefinitons[rule] = s.mergeDeep(s.loadedDefinitons[rule],fileData)
}else{
s.loadedDefinitons[rule] = s.mergeDeep(s.copySystemDefaultDefinitions(),fileData)
}
})
})
console.error('This Method has been deprecated. Could not load : ', customModulePath + 'defintions/')
console.error('Make your module\'s index.js file make the changes directly.')
// var definitionsFolder = s.checkCorrectPathEnding(customModulePath) + 'definitions/'
// fs.readdir(definitionsFolder,function(err,files){
// if(err)return console.log(err);
// files.forEach(function(filename){
// var fileData = require(definitionsFolder + filename)
// var rule = filename.replace('.json','').replace('.js','')
// if(config.language === rule){
// s.definitions = s.mergeDeep(s.definitions,fileData)
// }
// if(s.loadedDefinitons[rule]){
// s.loadedDefinitons[rule] = s.mergeDeep(s.loadedDefinitons[rule],fileData)
// }else{
// s.loadedDefinitons[rule] = s.mergeDeep(s.copySystemDefaultDefinitions(),fileData)
// }
// })
// })
break;
}
})
@ -327,13 +346,14 @@ module.exports = async (s,config,lang,app,io) => {
PageBlocks: [],
LibsJs: [],
LibsCss: [],
adminPageBlocks: [],
adminLibsJs: [],
adminLibsCss: [],
AssetsJs: [],
AssetsCss: [],
superPageBlocks: [],
superLibsJs: [],
superRawJs: [],
superLibsCss: []
superLibsCss: [],
superAssetsJs: [],
superAssetsCss: []
}
fs.readdir(modulesBasePath,function(err,folderContents){
if(!err && folderContents.length > 0){
@ -342,6 +362,8 @@ module.exports = async (s,config,lang,app,io) => {
return;
}
loadModule(shinobiModule)
s.reloadLanguages()
s.reloadDefinitions()
})
}else{
fs.mkdir(modulesBasePath,() => {})

62
libs/dataPort.js Normal file
View File

@ -0,0 +1,62 @@
const { createWebSocketServer } = require('./basic/websocketTools.js')
module.exports = function(s,config,lang,app,io){
const {
triggerEvent,
} = require('./events/utils.js')(s,config,lang)
s.dataPortTokens = {}
const theWebSocket = createWebSocketServer()
function setClientKillTimerIfNotAuthenticatedInTime(client){
client.killTimer = setTimeout(function(){
client.terminate()
},10000)
}
function clearKillTimer(client){
clearTimeout(client.killTimer)
}
theWebSocket.on('connection',(client) => {
// client.send(someDataToSendAsStringOrBinary)
setClientKillTimerIfNotAuthenticatedInTime(client)
function onAuthenticate(data){
clearKillTimer(client)
if(data in s.dataPortTokens){
client.removeListener('message', onAuthenticate);
client.on('message', onAuthenticatedData)
delete(s.dataPortTokens[data]);
}else{
client.terminate()
}
}
function onAuthenticatedData(jsonData){
const data = JSON.parse(jsonData);
switch(data.f){
case'trigger':
triggerEvent(data)
break;
case's.tx':
s.tx(data.data,data.to)
break;
case'debugLog':
s.debugLog(data.data)
break;
default:
console.log(data)
break;
}
s.onDataPortMessageExtensions.forEach(function(extender){
extender(data)
})
}
client.on('message', onAuthenticate)
client.on('close', () => {
clearTimeout(client.killTimer)
})
client.on('disconnect', () => {
clearTimeout(client.killTimer)
})
})
s.onHttpRequestUpgrade('/dataPort',(request, socket, head) => {
theWebSocket.handleUpgrade(request, socket, head, function done(ws) {
theWebSocket.emit('connection', ws, request)
})
})
}

20
libs/database/utils.js
View File

@ -87,11 +87,11 @@ module.exports = function(s,config){
const knexError = (dbQuery,options,err) => {
console.error('knexError----------------------------------- START')
if(config.debugLogVerbose && config.debugLog === true){
s.debugLog('s.knexQuery QUERY',JSON.stringify(options,null,3))
s.debugLog('STACK TRACE, NOT AN ',new Error())
}
console.error(err)
console.error(dbQuery.toString())
// CANNOT USE `dbQuery.toString()` because it breaks the query
console.error(options)
console.error('knexError----------------------------------- END')
}
const knexQuery = (options,callback) => {
@ -140,7 +140,7 @@ module.exports = function(s,config){
if(options.groupBy){
dbQuery.groupBy(options.groupBy)
}
if(options.limit){
if(options.limit && options.limit !== '0'){
if(`${options.limit}`.indexOf(',') === -1){
dbQuery.limit(options.limit)
}else{
@ -149,7 +149,8 @@ module.exports = function(s,config){
}
}
if(config.debugLog === true){
console.log(dbQuery.toString())
// CANNOT USE `dbQuery.toString()` because it breaks the query
console.log(options)
}
if(callback || options.update || options.insert || options.action === 'delete'){
dbQuery.asCallback(function(err,r) {
@ -177,6 +178,7 @@ module.exports = function(s,config){
]
const monitorRestrictions = options.monitorRestrictions
var frameLimit = options.limit
const noLimit = options.noLimit === '1'
const endIsStartTo = options.endIsStartTo
const chosenDate = options.date
const startDate = options.startDate ? stringToSqlTime(options.startDate) : null
@ -211,12 +213,13 @@ module.exports = function(s,config){
whereQuery.push(monitorRestrictions)
}
if(options.archived){
whereQuery.push(['details','LIKE',`%"archived":"1"%`])
whereQuery.push(['archive','=',`1`])
}
if(options.filename){
whereQuery.push(['filename','=',options.filename])
frameLimit = "1";
}
if(noLimit)frameLimit = '0';
options.orderBy = options.orderBy ? options.orderBy : ['time','desc']
if(options.count)options.groupBy = options.groupBy ? options.groupBy : options.orderBy[0]
knexQuery({
@ -328,7 +331,10 @@ module.exports = function(s,config){
var endTimeOperator = options.endTimeOperator
var startTime = options.startTime
var limitString = `${options.limit}`
const monitorRestrictions = s.getMonitorRestrictions(options.user.details,monitorId)
const {
monitorPermissions,
monitorRestrictions,
} = s.getMonitorsPermitted(user.details,monitorId)
getDatabaseRows({
monitorRestrictions: monitorRestrictions,
table: theTableSelected,
@ -337,7 +343,7 @@ module.exports = function(s,config){
endDate: endTime,
startOperator: startTimeOperator,
endOperator: endTimeOperator,
limit: options.limit,
limit: options.noLimit === '1' ? '0' : options.limit,
archived: archived,
rowType: rowName,
endIsStartTo: endIsStartTo

39
libs/definitions.js
View File

@ -1,25 +1,19 @@
var fs = require('fs')
var express = require('express')
const {
mergeDeep
} = require('./common.js')
const frameworkBase = require(`../definitions/base.js`)
module.exports = function(s,config,lang,app,io){
s.location.definitions = s.mainDirectory+'/definitions'
try{
var definitions = require(s.location.definitions+'/'+config.language+'.js')(s,config,lang)
}catch(er){
console.error(er)
console.log('There was an error loading your definition file.')
try{
var definitions = require(s.location.definitions+'/en_CA.js')(s,config,lang)
}catch(er){
console.error(er)
console.log('There was an error loading your definition file.')
var definitions = require(s.location.definitions+'/en_CA.json');
}
function getFramework(languageFile){
return frameworkBase(s,config,languageFile)
}
const defaultFramework = getFramework(lang)
//load defintions dynamically
s.definitions = definitions
s.definitions = defaultFramework
s.copySystemDefaultDefinitions = function(){
//en_CA
return Object.assign(s.definitions,{})
return Object.assign({},defaultFramework)
}
s.loadedDefinitons={}
s.loadedDefinitons[config.language] = s.copySystemDefaultDefinitions()
@ -28,10 +22,15 @@ module.exports = function(s,config,lang,app,io){
var file = s.loadedDefinitons[rule]
if(!file){
try{
s.loadedDefinitons[rule] = require(s.location.definitions+'/'+rule+'.js')(s,config,lang)
s.loadedDefinitons[rule] = Object.assign(s.copySystemDefaultDefinitions(),s.loadedDefinitons[rule])
// console.log(getFramework(lang))
s.loadedDefinitons[rule] = Object.assign(
{},
s.copySystemDefaultDefinitions(),
getFramework(s.getLanguageFile(rule))
);
file = s.loadedDefinitons[rule]
}catch(err){
console.error(err)
file = s.copySystemDefaultDefinitions()
}
}
@ -40,5 +39,9 @@ module.exports = function(s,config,lang,app,io){
}
return file
}
return definitions
s.reloadDefinitions = function(){
s.loadedDefinitons = {};
s.loadedDefinitons[config.language] = s.copySystemDefaultDefinitions()
}
return defaultFramework
}

10
libs/dropInEvents.js
View File

@ -188,14 +188,21 @@ module.exports = function(s,config,lang,app,io){
createDropInEventsDirectory()
if(!config.ftpServerPort)config.ftpServerPort = 21
if(!config.ftpServerUrl)config.ftpServerUrl = `ftp://0.0.0.0:${config.ftpServerPort}`
if(!config.ftpServerPasvUrl)config.ftpServerPasvUrl = config.ftpServerUrl.replace(/.*:\/\//, '').replace(/:.*/, '');
if(!config.ftpServerPasvMinPort)config.ftpServerPasvMinPort = 10050;
if(!config.ftpServerPasvMaxPort)config.ftpServerPasvMaxPort = 10100;
config.ftpServerUrl = config.ftpServerUrl.replace('{{PORT}}',config.ftpServerPort)
const FtpSrv = require('ftp-srv')
const ftpServer = new FtpSrv({
url: config.ftpServerUrl,
// pasv_url must be set to enable PASV; ftp-srv uses its known IP if given 127.0.0.1,
// and smart clients will ignore the IP anyway. Some Dahua IP cams require PASV mode.
// ftp-srv just wants an IP only (no protocol or port)
pasv_url: config.ftpServerUrl.replace(/.*:\/\//, '').replace(/:.*/, ''),
pasv_url: config.ftpServerPasvUrl,
pasv_min: config.ftpServerPasvMinPort,
pasv_max: config.ftpServerPasvMaxPort,
greeting: "Shinobi FTP dropInEvent Server says hello!",
log: require('bunyan').createLogger({
name: 'ftp-srv',
level: 100
@ -331,4 +338,5 @@ module.exports = function(s,config,lang,app,io){
s.systemLog(`SMTP Server running on port ${config.smtpServerPort}...`)
})
}
require('./dropInEvents/mqtt.js')(s,config,lang,app,io)
}

230
libs/dropInEvents/mqtt.js Normal file
View File

@ -0,0 +1,230 @@
module.exports = (s,config,lang,app,io) => {
if(config.mqttClient === true){
console.log('Loading MQTT Inbound Connectivity...')
const mqtt = require('mqtt')
const {
triggerEvent,
} = require('../events/utils.js')(s,config,lang)
function sendPlainEvent(options){
const groupKey = options.ke
const monitorId = options.mid || options.id
const subKey = options.subKey
const endpoint = options.host
triggerEvent({
id: monitorId,
ke: groupKey,
details: {
confidence: 100,
name: 'mqtt',
plug: endpoint,
reason: subKey
},
},config.mqttEventForceSaveEvent)
}
function sendFrigateEvent(data,options){
const groupKey = options.ke
const monitorId = options.mid || options.id
const subKey = options.subKey
const endpoint = options.host
const frigateMatrix = data.after || data.before
const confidenceScore = frigateMatrix.top_score * 100
const activeZones = frigateMatrix.entered_zones.join(', ')
const shinobiMatrix = {
x: frigateMatrix.box[0],
y: frigateMatrix.box[1],
width: frigateMatrix.box[2],
height: frigateMatrix.box[3],
tag: frigateMatrix.label,
confidence: confidenceScore,
}
triggerEvent({
id: monitorId,
ke: groupKey,
details: {
confidence: confidenceScore,
name: 'mqtt-'+endpoint,
plug: subKey,
reason: activeZones,
matrices: [shinobiMatrix]
},
},config.mqttEventForceSaveEvent)
}
function createMqttSubscription(options){
const mqttEndpoint = options.host
const subKey = options.subKey
const groupKey = options.ke
const onData = options.onData || function(){}
s.debugLog('Connecting.... mqtt://' + mqttEndpoint)
const client = mqtt.connect('mqtt://' + mqttEndpoint)
client.on('connect', function () {
s.debugLog('Connected! mqtt://' + mqttEndpoint)
client.subscribe(subKey, function (err) {
if (err) {
s.debugLog(err)
s.userLog({
ke: groupKey,
mid: '$USER'
},{
type: lang['MQTT Error'],
msg: err
})
}else{
client.on('message', function (topic, message) {
const data = s.parseJSON(message.toString())
onData(data)
})
}
})
})
return client
}
// const onEventTrigger = async () => {}
// const onMonitorUnexpectedExit = (monitorConfig) => {}
const loadMqttListBotForUser = function(user){
const groupKey = user.ke
const userDetails = s.parseJSON(user.details);
if(userDetails.mqttclient === '1'){
const mqttClientList = userDetails.mqttclient_list || []
if(!s.group[groupKey].mqttSubscriptions)s.group[groupKey].mqttSubscriptions = {};
const mqttSubs = s.group[groupKey].mqttSubscriptions
mqttClientList.forEach(function(row,n){
try{
const mqttSubId = `${row.host} ${row.subKey}`
const messageConversionTypes = row.type || []
const monitorsToTrigger = row.monitors || []
const triggerAllMonitors = monitorsToTrigger.indexOf('$all') > -1
const doActions = []
const onData = (data) => {
doActions.forEach(function(theAction){
theAction(data)
})
s.debugLog('MQTT Data',row,data)
}
if(mqttSubs[mqttSubId]){
mqttSubs[mqttSubId].end()
delete(mqttSubs[mqttSubId])
}
messageConversionTypes.forEach(function(type){
switch(type){
case'plain':
doActions.push(function(data){
// data is unused for plain event.
let listOfMonitors = monitorsToTrigger
if(triggerAllMonitors){
const activeMonitors = Object.keys(s.group[groupKey].activeMonitors)
listOfMonitors = activeMonitors
}
listOfMonitors.forEach(function(monitorId){
sendPlainEvent({
host: row.host,
subKey: row.subKey,
ke: groupKey,
mid: monitorId
})
})
})
break;
case'frigate':
// https://docs.frigate.video/integrations/mqtt/#frigateevents
doActions.push(function(data){
// this handler requires using frigate/events
// only "new" events will be captured.
if(data.type === 'new'){
let listOfMonitors = monitorsToTrigger
if(triggerAllMonitors){
const activeMonitors = Object.keys(s.group[groupKey].activeMonitors)
listOfMonitors = activeMonitors
}
listOfMonitors.forEach(function(monitorId){
sendFrigateEvent(data,{
host: row.host,
subKey: row.subKey,
ke: groupKey,
mid: monitorId
})
})
}
})
break;
}
})
mqttSubs[mqttSubId] = createMqttSubscription({
host: row.host,
subKey: row.subKey,
ke: groupKey,
onData: onData,
})
}catch(err){
s.debugLog(err)
// s.systemLog(err)
}
})
}
}
const unloadMqttListBotForUser = function(user){
const groupKey = user.ke
const mqttSubs = s.group[groupKey].mqttSubscriptions || {}
Object.keys(mqttSubs).forEach(function(mqttSubId){
try{
mqttSubs[mqttSubId].end()
}catch(err){
s.debugLog(err)
// s.userLog({
// ke: groupKey,
// mid: '$USER'
// },{
// type: lang['MQTT Error'],
// msg: err
// })
}
delete(mqttSubs[mqttSubId])
})
}
const onBeforeAccountSave = function(data){
data.d.mqttclient_list = []
}
s.loadGroupAppExtender(loadMqttListBotForUser)
s.unloadGroupAppExtender(unloadMqttListBotForUser)
s.beforeAccountSave(onBeforeAccountSave)
// s.onEventTrigger(onEventTrigger)
// s.onMonitorUnexpectedExit(onMonitorUnexpectedExit)
s.definitions["Account Settings"].blocks["MQTT Inbound"] = {
"evaluation": "$user.details.use_mqttclient !== '0'",
"name": lang['MQTT Inbound'],
"color": "green",
"info": [
{
"name": "detail=mqttclient",
"selector":"u_mqttclient",
"field": lang.Enabled,
"default": "0",
"example": "",
"fieldType": "select",
"possible": [
{
"name": lang.No,
"value": "0"
},
{
"name": lang.Yes,
"value": "1"
}
]
},
{
"fieldType": "btn",
"class": `btn-success mqtt-add-row`,
"btnContent": `<i class="fa fa-plus"></i> &nbsp; ${lang['Add']}`,
},
{
"id": "mqttclient_list",
"fieldType": "div",
},
{
"fieldType": "script",
"src": "assets/js/bs5.mqtt.js",
}
]
}
}
}

2
libs/events.js
View File

@ -1,3 +1,3 @@
module.exports = function(s,config,lang){
// all contents moved to libs/events/utils.js
require('./events/onvif.js')(s,config,lang)
}

109
libs/events/onvif.js Normal file
View File

@ -0,0 +1,109 @@
module.exports = function(s,config,lang){
const {
triggerEvent,
} = require('./utils.js')(s,config,lang)
const onvifEvents = require("node-onvif-events");
const onvifEventIds = []
const onvifEventControllers = {}
const startMotion = async (onvifId,monitorConfig) => {
const groupKey = monitorConfig.ke
const monitorId = monitorConfig.mid
const onvifIdKey = `${monitorConfig.mid}${monitorConfig.ke}`
const controlBaseUrl = monitorConfig.details.control_base_url || s.buildMonitorUrl(monitorConfig, true)
const controlURLOptions = s.cameraControlOptionsFromUrl(controlBaseUrl,monitorConfig)
const onvifPort = parseInt(monitorConfig.details.onvif_port) || 8000
let options = {
id: onvifId,
hostname: controlURLOptions.host,
username: controlURLOptions.username,
password: controlURLOptions.password,
port: onvifPort,
};
const detector = onvifEventControllers[onvifIdKey] || await onvifEvents.MotionDetector.create(options.id, options);
function onvifEventLog(type,data){
s.userLog({
ke: groupKey,
mid: monitorId
},{
type: type,
msg: data
})
}
onvifEventLog(`ONVIF Event Detection Listening!`)
try {
detector.listen((motion) => {
if (motion) {
// onvifEventLog(`ONVIF Event Detected!`)
triggerEvent({
f: 'trigger',
id: monitorId,
ke: groupKey,
details:{
plug: 'onvifEvent',
name: 'onvifEvent',
reason: 'motion',
confidence: 100,
// reason: 'object',
// matrices: [matrix],
// imgHeight: img.height,
// imgWidth: img.width,
}
})
// } else {
// onvifEventLog(`ONVIF Event Stopped`)
}
});
} catch(e) {
onvifEventLog(`ONVIF Event Error`,e)
}
return detector
}
function initializeOnvifEvents(monitorConfig){
const monitorMode = monitorConfig.mode
const groupKey = monitorConfig.ke
const monitorId = monitorConfig.mid
const hasOnvifEventsEnabled = monitorConfig.details.is_onvif === '1' && monitorConfig.details.onvif_events === '1';
if(hasOnvifEventsEnabled){
const onvifIdKey = `${monitorConfig.mid}${monitorConfig.ke}`
let onvifId = onvifEventIds.indexOf(onvifIdKey)
if(onvifEventIds.indexOf(onvifIdKey) === -1){
onvifId = onvifEventIds.length;
onvifEventIds.push(onvifIdKey);
}
try{
onvifEventControllers[onvifIdKey].close()
s.debugLog('ONVIF Event Module Warning : This could cause a memory leak?')
}catch(err){
s.debugLog('ONVIF Event Module Error', err);
}
delete(onvifEventControllers[onvifIdKey])
if(monitorMode !== 'stop'){
startMotion(onvifId,monitorConfig).then((detector) => {
onvifEventControllers[onvifIdKey] = detector;
})
}
}
}
s.onMonitorStart((monitorConfig) => {
initializeOnvifEvents(monitorConfig)
})
const connectionInfoArray = s.definitions["Monitor Settings"].blocks["Detector"].info
connectionInfoArray.splice(2, 0, {
"name": "detail=onvif_events",
"field": lang['ONVIF Events'],
"default": "0",
"form-group-class": "h_onvif_input h_onvif_1",
"form-group-class-pre-layer": "h_det_input h_det_1",
"fieldType": "select",
"possible": [
{
"name": lang.No,
"value": "0"
},
{
"name": lang.Yes,
"value": "1"
}
]
});
}

148
libs/events/utils.js
View File

@ -3,7 +3,6 @@ const moment = require('moment');
const execSync = require('child_process').execSync;
const exec = require('child_process').exec;
const spawn = require('child_process').spawn;
const request = require('request');
const imageSaveEventLock = {};
// Matrix In Region Libs >
const SAT = require('sat')
@ -12,6 +11,9 @@ const P = SAT.Polygon;
const B = SAT.Box;
// Matrix In Region Libs />
module.exports = (s,config,lang,app,io) => {
// Event Filters >
const acceptableOperators = ['indexOf','!indexOf','===','!==','>=','>','<','<=']
// Event Filters />
const {
splitForFFPMEG
} = require('../ffmpeg/utils.js')(s,config,lang)
@ -19,8 +21,13 @@ module.exports = (s,config,lang,app,io) => {
moveCameraPtzToMatrix
} = require('../control/ptz.js')(s,config,lang)
const {
cutVideoLength
cutVideoLength,
reEncodeVideoAndBinOriginalAddToQueue
} = require('../video/utils.js')(s,config,lang)
const {
isEven,
fetchTimeout,
} = require('../basic/utils.js')(process.cwd(),config)
async function saveImageFromEvent(options,frameBuffer){
const monitorId = options.mid || options.id
const groupKey = options.ke
@ -64,6 +71,9 @@ module.exports = (s,config,lang,app,io) => {
var newString = string + ''
var d = Object.assign(eventData,addOps)
var detailString = s.stringJSON(d.details)
var tag = detailString.matrices
&& detailString.matrices[0]
&& detailString.matrices[0].tag;
newString = newString
.replace(/{{TIME}}/g,d.currentTimestamp)
.replace(/{{REGION_NAME}}/g,d.details.name)
@ -71,7 +81,10 @@ module.exports = (s,config,lang,app,io) => {
.replace(/{{MONITOR_ID}}/g,d.id)
.replace(/{{MONITOR_NAME}}/g,s.group[d.ke].rawMonitorConfigurations[d.id].name)
.replace(/{{GROUP_KEY}}/g,d.ke)
.replace(/{{DETAILS}}/g,detailString)
.replace(/{{DETAILS}}/g,detailString);
if(tag){
newString = newString.replace(/{{TAG}}/g,tag)
}
if(d.details.confidence){
newString = newString
.replace(/{{CONFIDENCE}}/g,d.details.confidence)
@ -146,7 +159,6 @@ module.exports = (s,config,lang,app,io) => {
}
const addToEventCounter = (eventData) => {
const eventsCounted = s.group[eventData.ke].activeMonitors[eventData.id].detector_motion_count
s.debugLog(`addToEventCounter`,eventData,eventsCounted.length)
eventsCounted.push(eventData)
}
const clearEventCounter = (groupKey,monitorId) => {
@ -187,8 +199,20 @@ module.exports = (s,config,lang,app,io) => {
Object.keys(filters).forEach(function(key){
var conditionChain = {}
var dFilter = filters[key]
if(dFilter.enabled === '0')return;
var numberOfOpenAndCloseBrackets = 0
dFilter.where.forEach(function(condition,place){
conditionChain[place] = {ok:false,next:condition.p4,matrixCount:0}
const hasOpenBracket = condition.openBracket === '1';
const hasCloseBracket = condition.closeBracket === '1';
conditionChain[place] = {
ok: false,
next: condition.p4,
matrixCount: 0,
openBracket: hasOpenBracket,
closeBracket: hasCloseBracket,
}
if(hasOpenBracket)++numberOfOpenAndCloseBrackets;
if(hasCloseBracket)++numberOfOpenAndCloseBrackets;
if(d.details.matrices)conditionChain[place].matrixCount = d.details.matrices.length
var modifyFilters = function(toCheck,matrixPosition){
var param = toCheck[condition.p1]
@ -210,7 +234,12 @@ module.exports = (s,config,lang,app,io) => {
pass()
}
break;
default:
case'===':
case'!==':
case'>=':
case'>':
case'<':
case'<=':
if(eval('param '+condition.p2+' "'+condition.p3.replace(/"/g,'\\"')+'"')){
pass()
}
@ -243,7 +272,7 @@ module.exports = (s,config,lang,app,io) => {
var atSecond = parseInt(doAtTime[2]) - 1 || timeNow.getSeconds()
var nowAddedInSeconds = atHourNow * 60 * 60 + atMinuteNow * 60 + atSecondNow
var conditionAddedInSeconds = atHour * 60 * 60 + atMinute * 60 + atSecond
if(eval('nowAddedInSeconds '+condition.p2+' conditionAddedInSeconds')){
if(acceptableOperators.indexOf(condition.p2) > -1 && eval('nowAddedInSeconds '+condition.p2+' conditionAddedInSeconds')){
conditionChain[place].ok = true
}
}
@ -254,20 +283,29 @@ module.exports = (s,config,lang,app,io) => {
}
})
var conditionArray = Object.values(conditionChain)
var validationString = ''
var validationString = []
var allowBrackets = false;
if (numberOfOpenAndCloseBrackets === 0 || isEven(numberOfOpenAndCloseBrackets)){
allowBrackets = true;
}else{
s.userLog(d,{type:lang["Event Filter Error"],msg:lang.eventFilterErrorBrackets})
}
conditionArray.forEach(function(condition,number){
validationString += condition.ok+' '
validationString.push(`${allowBrackets && condition.openBracket ? '(' : ''}${condition.ok}${allowBrackets && condition.closeBracket ? ')' : ''}`);
if(conditionArray.length-1 !== number){
validationString += condition.next+' '
validationString.push(condition.next)
}
})
if(eval(validationString)){
if(eval(validationString.join(' '))){
if(dFilter.actions.halt !== '1'){
delete(dFilter.actions.halt)
Object.keys(dFilter.actions).forEach(function(key){
var value = dFilter.actions[key]
filter[key] = parseValue(key,value)
})
if(dFilter.actions.record === '1'){
filter.forceRecord = true
}
}else{
filter.halt = true
}
@ -362,7 +400,7 @@ module.exports = (s,config,lang,app,io) => {
matrices: eventDetails.matrices || [],
},d.frame)
}
if(forceSave || (filter.save && monitorDetails.detector_save === '1')){
if(forceSave || (filter.save || monitorDetails.detector_save === '1')){
s.knexQuery({
action: "insert",
table: "Events",
@ -370,7 +408,7 @@ module.exports = (s,config,lang,app,io) => {
ke: d.ke,
mid: d.id,
details: detailString,
time: eventTime,
time: s.formattedTime(eventTime),
}
})
}
@ -384,12 +422,12 @@ module.exports = (s,config,lang,app,io) => {
detector_timeout = parseFloat(monitorDetails.detector_timeout)
}
if(
filter.record &&
(filter.forceRecord || (filter.record && monitorDetails.detector_trigger === '1')) &&
monitorConfig.mode === 'start' &&
monitorDetails.detector_trigger === '1' &&
(monitorDetails.detector_record_method === 'sip' || monitorDetails.detector_record_method === 'hot')
){
createEventBasedRecording(d,moment(eventTime).subtract(5,'seconds').format('YYYY-MM-DDTHH-mm-ss'))
const secondBefore = (parseInt(monitorDetails.detector_buffer_seconds_before) || 5) + 1
createEventBasedRecording(d,moment(eventTime).subtract(secondBefore,'seconds').format('YYYY-MM-DDTHH-mm-ss'))
}
d.currentTime = eventTime
d.currentTimestamp = s.timeObject(d.currentTime).format()
@ -401,14 +439,17 @@ module.exports = (s,config,lang,app,io) => {
var detector_webhook_url = addEventDetailsToString(d,monitorDetails.detector_webhook_url)
var webhookMethod = monitorDetails.detector_webhook_method
if(!webhookMethod || webhookMethod === '')webhookMethod = 'GET'
request(detector_webhook_url,{method: webhookMethod,encoding:null},function(err,data){
if(err){
s.userLog(d,{type:lang["Event Webhook Error"],msg:{error:err,data:data}})
}
fetchTimeout(detector_webhook_url,10000,{
method: webhookMethod
}).catch((err) => {
s.userLog(d,{type:lang["Event Webhook Error"],msg:{error:err,data:data}})
})
}
if(filter.command && monitorDetails.detector_command_enable === '1' && !s.group[d.ke].activeMonitors[d.id].detector_command){
if(
filter.command ||
(monitorDetails.detector_command_enable === '1' && !s.group[d.ke].activeMonitors[d.id].detector_command)
){
s.group[d.ke].activeMonitors[d.id].detector_command = s.createTimeout('detector_command',s.group[d.ke].activeMonitors[d.id],monitorDetails.detector_command_timeout,10)
var detector_command = addEventDetailsToString(d,monitorDetails.detector_command)
if(detector_command === '')return
@ -431,23 +472,27 @@ module.exports = (s,config,lang,app,io) => {
if(activeMonitor && activeMonitor.eventBasedRecording && activeMonitor.eventBasedRecording.process){
const eventBasedRecording = activeMonitor.eventBasedRecording
const monitorConfig = s.group[groupKey].rawMonitorConfigurations[monitorId]
const videoLength = monitorConfig.details.detector_send_video_length
const videoLength = parseInt(monitorConfig.details.detector_send_video_length) || 10
const recordingDirectory = s.getVideoDirectory(monitorConfig)
const fileTime = eventBasedRecording.lastFileTime
const filename = `${fileTime}.mp4`
response.filename = `${filename}`
response.filePath = `${recordingDirectory}${filename}`
eventBasedRecording.process.on('close',function(){
eventBasedRecording.process.on('exit',function(){
setTimeout(async () => {
if(!isNaN(videoLength)){
const cutResponse = await cutVideoLength({
ke: groupKey,
mid: monitorId,
filePath: response.filePath,
cutLength: parseInt(videoLength),
cutLength: videoLength,
})
response.filename = cutResponse.filename
response.filePath = cutResponse.filePath
if(cutResponse.ok){
response.filename = cutResponse.filename
response.filePath = cutResponse.filePath
}else{
s.debugLog('cutResponse',cutResponse)
}
}
resolve(response)
},1000)
@ -489,14 +534,26 @@ module.exports = (s,config,lang,app,io) => {
var ffmpegError = ''
var error
var filename = fileTime + '.mp4'
let outputMap = `-map 0:0 `
const analyzeDuration = parseInt(monitorDetails.event_record_aduration) || 1000
const probeSize = parseInt(monitorDetails.event_record_probesize) || 32
s.userLog(d,{
type: logTitleText,
msg: lang["Started"]
})
//-t 00:'+s.timeObject(new Date(detector_timeout * 1000 * 60)).format('mm:ss')+'
if(
monitorDetails.detector_buffer_acodec &&
monitorDetails.detector_buffer_acodec !== 'no' &&
monitorDetails.detector_buffer_acodec !== 'auto'
){
outputMap += `-map 0:1 `
}
const ffmpegCommand = `-loglevel warning -live_start_index -99999 -analyzeduration ${analyzeDuration} -probesize ${probeSize} -re -i "${s.dir.streams+d.ke+'/'+d.id}/detectorStream.m3u8" ${outputMap}-movflags faststart+frag_keyframe+empty_moov -fflags +igndts -c:v copy -c:a aac -strict -2 -strftime 1 -y "${s.getVideoDirectory(monitorConfig) + filename}"`
s.debugLog(ffmpegCommand)
activeMonitor.eventBasedRecording.process = spawn(
config.ffmpegDir,
splitForFFPMEG(('-loglevel warning -analyzeduration 1000000 -probesize 1000000 -re -i "'+s.dir.streams+d.ke+'/'+d.id+'/detectorStream.m3u8" -movflags faststart+frag_keyframe+empty_moov -fflags +igndts -c:v copy -strftime 1 "'+s.getVideoDirectory(monitorConfig) + filename + '"'))
splitForFFPMEG(ffmpegCommand)
)
activeMonitor.eventBasedRecording.process.stderr.on('data',function(data){
s.userLog(d,{
@ -515,7 +572,24 @@ module.exports = (s,config,lang,app,io) => {
}
s.insertCompletedVideo(monitorConfig,{
file : filename,
})
},function(err,response){
const autoCompressionEnabled = monitorDetails.auto_compress_videos === '1';
if(autoCompressionEnabled){
reEncodeVideoAndBinOriginalAddToQueue({
video: response.insertQuery,
targetVideoCodec: 'vp9',
targetAudioCodec: 'libopus',
targetQuality: '-q:v 1 -q:a 1',
targetExtension: 'webm',
doSlowly: false,
automated: true,
}).then((encodeResponse) => {
s.debugLog('Complete Automatic Compression',encodeResponse)
}).catch((err) => {
console.log(err)
})
}
});
s.userLog(d,{
type: logTitleText,
msg: lang["Detector Recording Complete"]
@ -594,12 +668,13 @@ module.exports = (s,config,lang,app,io) => {
halt : false,
addToMotionCounter : true,
useLock : true,
save : true,
webhook : true,
command : true,
save : false,
webhook : false,
command : false,
record : true,
forceRecord : false,
indifference : false,
countObjects : true
countObjects : false
}
if(!s.group[d.ke] || !s.group[d.ke].activeMonitors[d.id]){
return s.systemLog(lang['No Monitor Found, Ignoring Request'])
@ -608,14 +683,14 @@ module.exports = (s,config,lang,app,io) => {
if(!monitorConfig){
return s.systemLog(lang['No Monitor Found, Ignoring Request'])
}
const activeMonitor = s.group[d.ke].activeMonitors[d.id]
const monitorDetails = monitorConfig.details
s.onEventTriggerBeforeFilterExtensions.forEach(function(extender){
extender(d,filter)
})
const eventDetails = d.details
const passedEventFilters = checkEventFilters(d,monitorDetails,filter)
if(!passedEventFilters)return
const detailString = JSON.stringify(eventDetails)
const passedEventFilters = checkEventFilters(d,activeMonitor.details,filter)
if(!passedEventFilters)return;
const eventTime = new Date()
if(
filter.addToMotionCounter &&
@ -638,8 +713,7 @@ module.exports = (s,config,lang,app,io) => {
addToEventCounter(d)
}
if(
filter.countObjects &&
monitorDetails.detector_obj_count === '1' &&
(filter.countObjects || monitorDetails.detector_obj_count === '1') &&
monitorDetails.detector_obj_count_in_region !== '1'
){
didCountingAlready = true

234
libs/extenders.js
View File

@ -1,186 +1,66 @@
module.exports = function(s,config){
////// USER //////
s.onSocketAuthenticationExtensions = []
s.onSocketAuthentication = function(callback){
s.onSocketAuthenticationExtensions.push(callback)
}
//
s.onUserLogExtensions = []
s.onUserLog = function(callback){
s.onUserLogExtensions.push(callback)
}
//
s.loadGroupExtensions = []
s.loadGroupExtender = function(callback){
s.loadGroupExtensions.push(callback)
}
//
s.loadGroupAppExtensions = []
s.loadGroupAppExtender = function(callback){
s.loadGroupAppExtensions.push(callback)
}
//
s.unloadGroupAppExtensions = []
s.unloadGroupAppExtender = function(callback){
s.unloadGroupAppExtensions.push(callback)
}
//
s.cloudDisksLoaded = []
s.cloudDisksLoader = function(storageType){
s.cloudDisksLoaded.push(storageType)
}
//
s.onAccountSaveExtensions = []
s.onAccountSave = function(callback){
s.onAccountSaveExtensions.push(callback)
}
//
s.beforeAccountSaveExtensions = []
s.beforeAccountSave = function(callback){
s.beforeAccountSaveExtensions.push(callback)
}
//
s.onTwoFactorAuthCodeNotificationExtensions = []
s.onTwoFactorAuthCodeNotification = function(callback){
s.onTwoFactorAuthCodeNotificationExtensions.push(callback)
}
//
s.onStalePurgeLockExtensions = []
s.onStalePurgeLock = function(callback){
s.onStalePurgeLockExtensions.push(callback)
}
//
s.cloudDiskUseStartupExtensions = {}
s.cloudDiskUseOnGetVideoDataExtensions = {}
function createExtension(nameOfExtension,nameOfExtensionContainer,objective){
nameOfExtensionContainer = nameOfExtensionContainer || `${nameOfExtension}Extensions`
if(objective){
s[nameOfExtensionContainer] = []
s[nameOfExtension] = function(nameOfCallback,callback){
s[nameOfExtensionContainer][nameOfCallback] = callback
}
}else{
s[nameOfExtensionContainer] = []
s[nameOfExtension] = function(callback){
s[nameOfExtensionContainer].push(callback)
}
}
}
////// USER //////
createExtension(`onSocketAuthentication`)
createExtension(`onUserLog`)
createExtension(`loadGroupExtender`,`loadGroupExtensions`)
createExtension(`loadGroupAppExtender`,`loadGroupAppExtensions`)
createExtension(`unloadGroupAppExtender`,`unloadGroupAppExtensions`)
createExtension(`cloudDisksLoader`,`cloudDisksLoaded`)
createExtension(`onAccountSave`)
createExtension(`beforeAccountSave`)
createExtension(`onTwoFactorAuthCodeNotification`)
createExtension(`onStalePurgeLock`)
////// EVENTS //////
s.onEventTriggerExtensions = []
s.onEventTrigger = function(callback){
s.onEventTriggerExtensions.push(callback)
}
s.onEventTriggerBeforeFilterExtensions = []
s.onEventTriggerBeforeFilter = function(callback){
s.onEventTriggerBeforeFilterExtensions.push(callback)
}
s.onFilterEventExtensions = []
s.onFilterEvent = function(callback){
s.onFilterEventExtensions.push(callback)
}
createExtension(`onEventTrigger`)
createExtension(`onEventTriggerBeforeFilter`)
createExtension(`onFilterEvent`)
////// MONITOR //////
s.onMonitorInitExtensions = []
s.onMonitorInit = function(callback){
s.onMonitorInitExtensions.push(callback)
}
//
s.onMonitorStartExtensions = []
s.onMonitorStart = function(callback){
s.onMonitorStartExtensions.push(callback)
}
//
s.onMonitorStopExtensions = []
s.onMonitorStop = function(callback){
s.onMonitorStopExtensions.push(callback)
}
//
s.onMonitorSaveExtensions = []
s.onMonitorSave = function(callback){
s.onMonitorSaveExtensions.push(callback)
}
//
s.onMonitorUnexpectedExitExtensions = []
s.onMonitorUnexpectedExit = function(callback){
s.onMonitorUnexpectedExitExtensions.push(callback)
}
//
s.onDetectorNoTriggerTimeoutExtensions = []
s.onDetectorNoTriggerTimeout = function(callback){
s.onDetectorNoTriggerTimeoutExtensions.push(callback)
}
//
s.onFfmpegCameraStringCreationExtensions = []
s.onFfmpegCameraStringCreation = function(callback){
s.onFfmpegCameraStringCreationExtensions.push(callback)
}
//
s.onMonitorPingFailedExtensions = []
s.onMonitorPingFailed = function(callback){
s.onMonitorPingFailedExtensions.push(callback)
}
//
s.onMonitorDiedExtensions = []
s.onMonitorDied = function(callback){
s.onMonitorDiedExtensions.push(callback)
}
createExtension(`onMonitorInit`)
createExtension(`onMonitorStart`)
createExtension(`onMonitorStop`)
createExtension(`onMonitorSave`)
createExtension(`onMonitorUnexpectedExit`)
createExtension(`onDetectorNoTriggerTimeout`)
createExtension(`onFfmpegCameraStringCreation`)
createExtension(`onFfmpegBuildMainStream`)
createExtension(`onFfmpegBuildStreamChannel`)
createExtension(`onMonitorPingFailed`)
createExtension(`onMonitorDied`)
createExtension(`onMonitorCreateStreamPipe`)
///////// SYSTEM ////////
s.onProcessReadyExtensions = []
s.onProcessReady = function(callback){
s.onProcessReadyExtensions.push(callback)
}
//
s.onProcessExitExtensions = []
s.onProcessExit = function(callback){
s.onProcessExitExtensions.push(callback)
}
//
s.onBeforeDatabaseLoadExtensions = []
s.onBeforeDatabaseLoad = function(callback){
s.onBeforeDatabaseLoadExtensions.push(callback)
}
//
s.onFFmpegLoadedExtensions = []
s.onFFmpegLoaded = function(callback){
s.onFFmpegLoadedExtensions.push(callback)
}
//
s.beforeMonitorsLoadedOnStartupExtensions = []
s.beforeMonitorsLoadedOnStartup = function(callback){
s.beforeMonitorsLoadedOnStartupExtensions.push(callback)
}
//
s.onWebSocketConnectionExtensions = []
s.onWebSocketConnection = function(callback){
s.onWebSocketConnectionExtensions.push(callback)
}
//
s.onWebSocketDisconnectionExtensions = []
s.onWebSocketDisconnection = function(callback){
s.onWebSocketDisconnectionExtensions.push(callback)
}
//
s.onWebsocketMessageSendExtensions = []
s.onWebsocketMessageSend = function(callback){
s.onWebsocketMessageSendExtensions.push(callback)
}
//
s.onGetCpuUsageExtensions = []
s.onGetCpuUsage = function(callback){
s.onGetCpuUsageExtensions.push(callback)
}
//
s.onGetRamUsageExtensions = []
s.onGetRamUsage = function(callback){
s.onGetRamUsageExtensions.push(callback)
}
//
s.onSubscriptionCheckExtensions = []
s.onSubscriptionCheck = function(callback){
s.onSubscriptionCheckExtensions.push(callback)
}
//
createExtension(`onProcessReady`)
createExtension(`onProcessExit`)
createExtension(`onBeforeDatabaseLoad`)
createExtension(`onFFmpegLoaded`)
createExtension(`beforeMonitorsLoadedOnStartup`)
createExtension(`onWebSocketConnection`)
createExtension(`onWebSocketDisconnection`)
createExtension(`onWebsocketMessageSend`)
createExtension(`onOtherWebSocketMessages`)
createExtension(`onGetCpuUsage`)
createExtension(`onGetRamUsage`)
createExtension(`onSubscriptionCheck`)
createExtension(`onDataPortMessage`)
createExtension(`onHttpRequestUpgrade`,null,true)
/////// VIDEOS ////////
s.insertCompletedVideoExtensions = []
s.insertCompletedVideoExtender = function(callback){
s.insertCompletedVideoExtensions.push(callback)
}
s.onBeforeInsertCompletedVideoExtensions = []
s.onBeforeInsertCompletedVideo = function(callback){
s.onBeforeInsertCompletedVideoExtensions.push(callback)
}
createExtension(`insertCompletedVideoExtender`,`insertCompletedVideoExtensions`)
createExtension(`onBeforeInsertCompletedVideo`)
/////// TIMELAPSE ////////
s.onInsertTimelapseFrameExtensions = []
s.onInsertTimelapseFrame = function(callback){
s.onInsertTimelapseFrameExtensions.push(callback)
}
createExtension(`onInsertTimelapseFrame`)
}

92
libs/ffmpeg.js
View File

@ -26,9 +26,15 @@ module.exports = async (s,config,lang,onFinish) => {
s.ffmpeg = function(e){
try{
const activeMonitor = s.group[e.ke].activeMonitors[e.mid];
const dataPortToken = s.gid(10);
s.dataPortTokens[dataPortToken] = {
type: 'cameraThread',
ke: e.ke,
mid: e.mid,
}
const ffmpegCommand = [`-progress pipe:5`];
([
buildMainInput(e),
const allOutputs = [
buildMainStream(e),
buildJpegApiOutput(e),
buildMainRecording(e),
@ -36,45 +42,53 @@ module.exports = async (s,config,lang,onFinish) => {
buildMainDetector(e),
buildEventRecordingOutput(e),
buildTimelapseOutput(e),
]).forEach(function(commandStringPart){
ffmpegCommand.push(commandStringPart)
})
s.onFfmpegCameraStringCreationExtensions.forEach(function(extender){
extender(e,ffmpegCommand)
})
const stdioPipes = createPipeArray(e)
const ffmpegCommandString = ffmpegCommand.join(' ')
//hold ffmpeg command for log stream
s.group[e.ke].activeMonitors[e.mid].ffmpeg = sanitizedFfmpegCommand(e,ffmpegCommandString)
//clean the string of spatial impurities and split for spawn()
const ffmpegCommandParsed = splitForFFPMEG(ffmpegCommandString)
try{
fs.unlinkSync(e.sdir + 'cmd.txt')
}catch(err){
}
fs.writeFileSync(e.sdir + 'cmd.txt',JSON.stringify({
cmd: ffmpegCommandParsed,
pipes: stdioPipes.length,
rawMonitorConfig: s.group[e.ke].rawMonitorConfigurations[e.id],
globalInfo: {
config: config,
isAtleatOneDetectorPluginConnected: s.isAtleatOneDetectorPluginConnected
}
},null,3),'utf8')
var cameraCommandParams = [
config.monitorDaemonPath ? config.monitorDaemonPath : __dirname + '/cameraThread/singleCamera.js',
config.ffmpegDir,
e.sdir + 'cmd.txt'
]
const cameraProcess = spawn('node',cameraCommandParams,{detached: true,stdio: stdioPipes})
if(config.debugLog === true){
cameraProcess.stderr.on('data',(data) => {
console.log(`${e.ke} ${e.mid}`)
console.log(data.toString())
];
if(allOutputs.filter(output => !!output).length > 0){
([
buildMainInput(e),
]).concat(allOutputs).forEach(function(commandStringPart){
ffmpegCommand.push(commandStringPart)
})
s.onFfmpegCameraStringCreationExtensions.forEach(function(extender){
extender(e,ffmpegCommand)
})
const stdioPipes = createPipeArray(e)
const ffmpegCommandString = ffmpegCommand.join(' ')
//hold ffmpeg command for log stream
activeMonitor.ffmpeg = sanitizedFfmpegCommand(e,ffmpegCommandString)
//clean the string of spatial impurities and split for spawn()
const ffmpegCommandParsed = splitForFFPMEG(ffmpegCommandString)
try{
fs.unlinkSync(e.sdir + 'cmd.txt')
}catch(err){
}
fs.writeFileSync(e.sdir + 'cmd.txt',JSON.stringify({
dataPortToken: dataPortToken,
cmd: ffmpegCommandParsed,
pipes: stdioPipes.length,
rawMonitorConfig: s.group[e.ke].rawMonitorConfigurations[e.id],
globalInfo: {
config: config,
isAtleatOneDetectorPluginConnected: s.isAtleatOneDetectorPluginConnected
}
},null,3),'utf8')
var cameraCommandParams = [
config.monitorDaemonPath ? config.monitorDaemonPath : __dirname + '/cameraThread/singleCamera.js',
config.ffmpegDir,
e.sdir + 'cmd.txt'
]
const cameraProcess = spawn('node',cameraCommandParams,{detached: true,stdio: stdioPipes})
if(config.debugLog === true && config.debugLogMonitors === true){
cameraProcess.stderr.on('data',(data) => {
console.log(`${e.ke} ${e.mid}`)
console.log(data.toString())
})
}
return cameraProcess
}else{
return null
}
return cameraProcess
}catch(err){
s.systemLog(err)
return null

102
libs/ffmpeg/builders.js
View File

@ -9,6 +9,8 @@ module.exports = (s,config,lang) => {
const {
validateDimensions,
} = require('./utils.js')(s,config,lang)
if(!config.outputsWithAudio)config.outputsWithAudio = ['hls','flv','mp4','rtmp'];
if(!config.outputsNotCapableOfPresets)config.outputsNotCapableOfPresets = [];
const hasCudaEnabled = (monitor) => {
return monitor.details.accelerator === '1' && monitor.details.hwaccel === 'cuvid' && monitor.details.hwaccel_vcodec === ('h264_cuvid' || 'hevc_cuvid' || 'mjpeg_cuvid' || 'mpeg4_cuvid')
}
@ -183,7 +185,7 @@ module.exports = (s,config,lang) => {
const createStreamChannel = function(e,number,channel){
//`e` is the monitor object
//`x` is an object used to contain temporary values.
const channelStreamDirectory = !isNaN(parseInt(number)) ? `${e.sdir}channel${number}/` : e.sdir
const channelStreamDirectory = !isNaN(parseInt(number)) ? `${e.sdir || s.getStreamsDirectory(e)}channel${number}/` : e.sdir
if(channelStreamDirectory !== e.sdir && !fs.existsSync(channelStreamDirectory)){
try{
fs.mkdirSync(channelStreamDirectory)
@ -203,7 +205,7 @@ module.exports = (s,config,lang) => {
const streamType = channel.stream_type ? channel.stream_type : 'hls'
const videoFps = !isNaN(parseFloat(channel.stream_fps)) && channel.stream_fps !== '0' ? parseFloat(channel.stream_fps) : streamType === 'rtmp' ? '30' : null
const inputMap = buildInputMap(e,e.details.input_map_choices[`stream_channel-${channelNumber}`])
const outputCanHaveAudio = (streamType === 'hls' || streamType === 'mp4' || streamType === 'flv' || streamType === 'h265' || streamType === 'rtmp')
const outputCanHaveAudio = config.outputsWithAudio.indexOf(streamType) > -1;
const outputRequiresEncoding = streamType === 'mjpeg' || streamType === 'b64'
const outputIsPresetCapable = outputCanHaveAudio
const { videoWidth, videoHeight } = validateDimensions(channel.stream_scale_x,channel.stream_scale_y)
@ -246,7 +248,7 @@ module.exports = (s,config,lang) => {
streamFilters.push(channel.stream_vf)
}
if(outputIsPresetCapable){
const streamPreset = streamType !== 'h265' && channel.preset_stream ? channel.preset_stream : null
const streamPreset = config.outputsNotCapableOfPresets.indexOf(streamType) === -1 && channel.preset_stream ? channel.preset_stream : null
if(streamPreset){
streamFlags.push(`-preset ${streamPreset}`)
}
@ -287,18 +289,18 @@ module.exports = (s,config,lang) => {
streamFlags.push(`-g 1`)
}
}
streamFlags.push(`-f hls -hls_time ${hlsTime} -hls_list_size ${hlsListSize} -start_number 0 -hls_allow_cache 0 -hls_flags +delete_segments+omit_endlist "${channelStreamDirectory}s.m3u8"`)
streamFlags.push(`-f hls -hls_time ${hlsTime} -hls_list_size ${hlsListSize} -start_number 0 -hls_allow_cache 0 -hls_flags +delete_segments+omit_endlist+discont_start "${channelStreamDirectory}s.m3u8"`)
break;
case'mjpeg':
streamFlags.push(`-an -c:v mjpeg -f mpjpeg -boundary_tag shinobi pipe:${number}`)
break;
case'h265':
streamFlags.push(`-movflags +frag_keyframe+empty_moov+default_base_moof -metadata title="Shinobi H.265 Stream" -reset_timestamps 1 -f hevc pipe:${number}`)
break;
case'b64':case'':case undefined:case null://base64
streamFlags.push(`-an -c:v mjpeg -f image2pipe pipe:${number}`)
break;
}
s.onFfmpegBuildStreamChannelExtensions.forEach(function(extender){
extender(streamType,streamFlags,number,e)
});
return ' ' + streamFlags.join(' ')
}
const buildMainInput = function(e){
@ -375,7 +377,7 @@ module.exports = (s,config,lang) => {
//x = temporary values
const streamFlags = []
const streamType = e.details.stream_type ? e.details.stream_type : 'hls'
if(streamType !== 'jpeg'){
if(streamType !== 'jpeg' && streamType !== 'useSubstream'){
const isCudaEnabled = hasCudaEnabled(e)
const streamFilters = []
const videoCodecisCopy = e.details.stream_vcodec === 'copy'
@ -384,7 +386,7 @@ module.exports = (s,config,lang) => {
const videoQuality = e.details.stream_quality ? e.details.stream_quality : '1'
const videoFps = !isNaN(parseFloat(e.details.stream_fps)) && e.details.stream_fps !== '0' ? parseFloat(e.details.stream_fps) : null
const inputMap = buildInputMap(e,e.details.input_map_choices.stream)
const outputCanHaveAudio = (streamType === 'hls' || streamType === 'mp4' || streamType === 'flv' || streamType === 'h265')
const outputCanHaveAudio = config.outputsWithAudio.indexOf(streamType) > -1;
const outputRequiresEncoding = streamType === 'mjpeg' || streamType === 'b64'
const outputIsPresetCapable = outputCanHaveAudio
const { videoWidth, videoHeight } = validateDimensions(e.details.stream_scale_x,e.details.stream_scale_y)
@ -429,7 +431,7 @@ module.exports = (s,config,lang) => {
streamFilters.push(e.details.stream_vf)
}
if(outputIsPresetCapable){
const streamPreset = streamType !== 'h265' && e.details.preset_stream ? e.details.preset_stream : null
const streamPreset = config.outputsNotCapableOfPresets.indexOf(streamType) === -1 && e.details.preset_stream ? e.details.preset_stream : null
if(streamPreset){
streamFlags.push(`-preset ${streamPreset}`)
}
@ -460,18 +462,18 @@ module.exports = (s,config,lang) => {
streamFlags.push(`-g 1`)
}
}
streamFlags.push(`-f hls -hls_time ${hlsTime} -hls_list_size ${hlsListSize} -start_number 0 -hls_allow_cache 0 -hls_flags +delete_segments+omit_endlist "${e.sdir}s.m3u8"`)
streamFlags.push(`-f hls -hls_time ${hlsTime} -hls_list_size ${hlsListSize} -start_number 0 -hls_allow_cache 0 -hls_flags +delete_segments+omit_endlist+discont_start "${e.sdir}s.m3u8"`)
break;
case'mjpeg':
streamFlags.push(`-an -c:v mjpeg -f mpjpeg -boundary_tag shinobi pipe:1`)
break;
case'h265':
streamFlags.push(`-movflags +frag_keyframe+empty_moov+default_base_moof -metadata title="Shinobi H.265 Stream" -reset_timestamps 1 -f hevc pipe:1`)
break;
case'b64':case'':case undefined:case null://base64
streamFlags.push(`-an -c:v mjpeg -f image2pipe pipe:1`)
break;
}
s.onFfmpegBuildMainStreamExtensions.forEach(function(extender){
extender(streamType,streamFlags,e)
});
if(e.details.custom_output){
streamFlags.push(e.details.custom_output)
}
@ -652,20 +654,20 @@ module.exports = (s,config,lang) => {
if(objectDetectorOutputIsEnabled){
addObjectDetectorInputMap()
addObjectDetectValues()
detectorFlags.push('-an -f singlejpeg pipe:4')
detectorFlags.push('-an -f mjpeg pipe:4')
}
}else if(sendFramesToObjectDetector){
addObjectDetectorInputMap()
addObjectDetectValues()
detectorFlags.push('-an -f singlejpeg pipe:4')
detectorFlags.push('-an -f mjpeg pipe:4')
}else{
addInputMap()
detectorFlags.push('-an -f singlejpeg pipe:4')
detectorFlags.push('-an -f mjpeg pipe:4')
}
}else if(sendFramesToObjectDetector){
addObjectDetectorInputMap()
addObjectDetectValues()
detectorFlags.push('-an -f singlejpeg pipe:4')
detectorFlags.push('-an -f mjpeg pipe:4')
}
return detectorFlags.join(' ')
}
@ -677,13 +679,17 @@ module.exports = (s,config,lang) => {
const isCudaEnabled = hasCudaEnabled(e)
const outputFilters = []
var videoCodec = e.details.detector_buffer_vcodec
var liveStartIndex = e.details.detector_buffer_live_start_index || '-3'
var audioCodec = e.details.detector_buffer_acodec ? e.details.detector_buffer_acodec : 'no'
const videoCodecisCopy = videoCodec === 'copy'
const videoFps = !isNaN(parseFloat(e.details.stream_fps)) && e.details.stream_fps !== '0' ? parseFloat(e.details.stream_fps) : null
const inputMap = buildInputMap(e,e.details.input_map_choices.detector_sip_buffer)
const { videoWidth, videoHeight } = validateDimensions(e.details.event_record_scale_x,e.details.event_record_scale_y)
const hlsTime = !isNaN(parseInt(e.details.detector_buffer_hls_time)) ? `${parseInt(e.details.detector_buffer_hls_time)}` : '2'
const hlsListSize = !isNaN(parseInt(e.details.detector_buffer_hls_list_size)) ? `${parseInt(e.details.detector_buffer_hls_list_size)}` : '4'
// const hlsListSize = !isNaN(parseInt(e.details.detector_buffer_hls_list_size)) ? `${parseInt(e.details.detector_buffer_hls_list_size)}` : '4'
const secondsBefore = parseInt(e.details.detector_buffer_seconds_before) || 5
let hlsListSize = parseInt(secondsBefore * 0.6)
hlsListSize = hlsListSize < 3 ? 3 : hlsListSize;
if(inputMap)outputFlags.push(inputMap)
if(e.details.cust_sip_record)outputFlags.push(e.details.cust_sip_record)
if(videoCodec === 'auto'){
@ -737,7 +743,7 @@ module.exports = (s,config,lang) => {
outputFlags.push(`-g 1`)
}
}
outputFlags.push(`-f hls -live_start_index -3 -hls_time ${hlsTime} -hls_list_size ${hlsListSize} -start_number 0 -hls_allow_cache 0 -hls_flags +delete_segments+omit_endlist "${e.sdir}detectorStream.m3u8"`)
outputFlags.push(`-f hls -live_start_index -3 -hls_time ${hlsTime} -hls_list_size ${hlsListSize} -start_number 0 -hls_allow_cache 0 -hls_flags +delete_segments+omit_endlist+discont_start "${e.sdir}detectorStream.m3u8"`)
}
return outputFlags.join(' ')
}
@ -757,11 +763,63 @@ module.exports = (s,config,lang) => {
if(videoFilters.length > 0){
videoFlags.push(`-vf "${videoFilters.join(',')}"`)
}
videoFlags.push(`-f singlejpeg -an -q:v 1 pipe:7`)
videoFlags.push(`-f mjpeg -an -q:v 1 pipe:7`)
return videoFlags.join(' ')
}
return ``
}
const getDefaultSubstreamFields = function(monitor){
const subStreamFields = s.parseJSON(monitor.details.substream || {input:{},output:{}})
const inputAndConnectionFields = Object.assign({
"type":"h264",
"fulladdress":"",
"sfps":"",
"aduration":"",
"probesize":"",
"stream_loop":"0",
"rtsp_transport":"",
"accelerator":"0",
"hwaccel":"",
"hwaccel_vcodec":"auto",
"hwaccel_device":"",
"cust_input":""
},subStreamFields.input);
const outputFields = Object.assign({
"stream_type":"hls",
"rtmp_server_url":"",
"rtmp_stream_key":"",
"stream_mjpeg_clients":"",
"stream_vcodec":"copy",
"stream_acodec":"no",
"stream_fps":"",
"hls_time":"",
"preset_stream":"",
"hls_list_size":"",
"stream_quality":"",
"stream_v_br":"",
"stream_a_br":"",
"stream_scale_x":"",
"stream_scale_y":"",
"rotate_stream":"no",
"svf":"",
"cust_stream":""
},subStreamFields.output);
return {
inputAndConnectionFields,
outputFields,
}
}
const buildSubstreamString = function(channelNumber,monitor){
let ffmpegParts = []
const {
inputAndConnectionFields,
outputFields,
} = getDefaultSubstreamFields(monitor)
ffmpegParts.push(`-loglevel ${monitor.details.loglevel || 'warning'}`)
ffmpegParts.push(createInputMap(monitor,channelNumber,inputAndConnectionFields))
ffmpegParts.push(createStreamChannel(monitor,channelNumber,outputFields))
return ffmpegParts.join(' ')
}
return {
createStreamChannel: createStreamChannel,
buildMainInput: buildMainInput,
@ -772,5 +830,7 @@ module.exports = (s,config,lang) => {
buildMainDetector: buildMainDetector,
buildEventRecordingOutput: buildEventRecordingOutput,
buildTimelapseOutput: buildTimelapseOutput,
getDefaultSubstreamFields: getDefaultSubstreamFields,
buildSubstreamString: buildSubstreamString,
}
}

16
libs/ffmpeg/subStream.js Normal file
View File

@ -0,0 +1,16 @@
function createStringForSubstreamProcess(options){
let options = {
"type":"h264",
"fulladdress":"",
"sfps":"",
"aduration":"",
"probesize":"",
"stream_loop":"0",
"rtsp_transport":"",
"accelerator":"0",
"hwaccel":"auto",
"hwaccel_vcodec":"auto",
"hwaccel_device":"",
"cust_input":""
}
}

101
libs/ffmpeg/utils.js
View File

@ -14,48 +14,63 @@ module.exports = (s,config,lang) => {
var endData = {ok: false, result: {}}
if(!url){
endData.error = 'Missing URL'
callback(endData)
return
if(callback)callback(endData)
return {
result: {
format: {
duration: 1
}
}
}
}
if(!forceOverlap && activeProbes[auth]){
endData.error = 'Account is already probing'
callback(endData)
return
}
activeProbes[auth] = 1
var stderr = ''
var stdout = ''
const probeCommand = splitForFFPMEG(`${customInput ? customInput + ' ' : ''}-analyzeduration 10000 -probesize 10000 -v quiet -print_format json -show_format -show_streams -i "${url}"`)
var processTimeout = null
var ffprobeLocation = config.ffmpegDir.split('/')
ffprobeLocation[ffprobeLocation.length - 1] = 'ffprobe'
ffprobeLocation = ffprobeLocation.join('/')
const probeProcess = spawn(ffprobeLocation, probeCommand)
const finishReponse = () => {
delete(activeProbes[auth])
if(!stdout){
endData.error = stderr
}else{
endData.ok = true
endData.result = s.parseJSON(stdout)
if(callback)callback(endData)
return {
result: {
format: {
duration: 1
}
}
}
endData.probe = probeCommand
callback(endData)
}
probeProcess.stderr.on('data',function(data){
stderr += data.toString()
return new Promise((resolve) => {
activeProbes[auth] = 1
var stderr = ''
var stdout = ''
const probeCommand = splitForFFPMEG(`${customInput ? customInput + ' ' : ''}-analyzeduration 10000 -probesize 10000 -v quiet -print_format json -show_format -show_streams -i "${url}"`)
var processTimeout = null
var ffprobeLocation = config.ffmpegDir.split('/')
ffprobeLocation[ffprobeLocation.length - 1] = 'ffprobe'
ffprobeLocation = ffprobeLocation.join('/')
const probeProcess = spawn(ffprobeLocation, probeCommand)
const finishReponse = () => {
delete(activeProbes[auth])
if(!stdout){
endData.error = stderr
}else{
endData.ok = true
endData.result = s.parseJSON(stdout)
}
endData.probe = probeCommand
if(callback)callback(endData)
resolve(endData)
}
probeProcess.stderr.on('data',function(data){
stderr += data.toString()
})
probeProcess.stdout.on('data',function(data){
stdout += data.toString()
})
probeProcess.on('close',function(){
clearTimeout(processTimeout)
finishReponse()
})
processTimeout = setTimeout(() => {
treekill(probeProcess.pid)
finishReponse()
},4000)
})
probeProcess.stdout.on('data',function(data){
stdout += data.toString()
})
probeProcess.on('close',function(){
clearTimeout(processTimeout)
finishReponse()
})
processTimeout = setTimeout(() => {
treekill(probeProcess.pid)
finishReponse()
},4000)
}
const probeMonitor = (monitor,timeoutAmount,forceOverlap) => {
return new Promise((resolve,reject) => {
@ -130,7 +145,11 @@ module.exports = (s,config,lang) => {
}else{
const details = s.parseJSON(configPartial.details)
Object.keys(details).forEach((key) => {
activeMonitor.details[key] = details[key]
try{
activeMonitor.details[key] = details[key]
}catch(err){
console.log(err)
}
})
}
})
@ -155,11 +174,11 @@ module.exports = (s,config,lang) => {
}
return sanitizedCmd
}
const createPipeArray = function(e){
const createPipeArray = function(e, amountToAdd){
const stdioPipes = [];
var times = config.pipeAddition;
if(e.details.stream_channels){
times+=e.details.stream_channels.length
var times = amountToAdd ? amountToAdd + config.pipeAddition : config.pipeAddition;
if(e.details && e.details.stream_channels){
times += e.details.stream_channels.length
}
for(var i=0; i < times; i++){
stdioPipes.push('pipe')

179
libs/fileBin.js
View File

@ -163,11 +163,34 @@ module.exports = function(s,config,lang,app,io){
})
})
}
function archiveFileBinEntry(file,unarchive){
return new Promise((resolve) => {
s.knexQuery({
action: "update",
table: 'Files',
update: {
archive: unarchive ? '0' : 1
},
where: {
ke: file.ke,
mid: file.mid,
name: file.name,
}
},function(err){
resolve({
ok: !err,
err: err,
archived: !unarchive
})
})
})
}
s.getFileBinDirectory = getFileBinDirectory
s.getFileBinEntry = getFileBinEntry
s.insertFileBinEntry = insertFileBinEntry
s.updateFileBinEntry = updateFileBinEntry
s.deleteFileBinEntry = deleteFileBinEntry
s.archiveFileBinEntry = archiveFileBinEntry
/**
* API : Get fileBin file rows
*/
@ -187,6 +210,7 @@ module.exports = function(s,config,lang,app,io){
startTimeOperator: req.query.startOperator,
endTimeOperator: req.query.endOperator,
limit: req.query.limit,
archived: req.query.archived,
endIsStartTo: true,
noFormat: true,
noCount: true,
@ -196,7 +220,7 @@ module.exports = function(s,config,lang,app,io){
},(response) => {
response.forEach(function(v){
v.details = s.parseJSON(v.details)
v.href = '/'+req.params.auth+'/fileBin/'+req.params.ke+'/'+req.params.id+'/'+v.details.year+'/'+v.details.month+'/'+v.details.day+'/'+v.name;
v.href = '/'+req.params.auth+'/fileBin/'+req.params.ke+'/'+req.params.id+'/'+v.name;
})
s.closeJsonResponse(res,{
ok: true,
@ -208,55 +232,124 @@ module.exports = function(s,config,lang,app,io){
/**
* API : Get fileBin file
*/
app.get([
config.webPaths.apiPrefix+':auth/fileBin/:ke/:id/:file',
config.webPaths.apiPrefix+':auth/fileBin/:ke/:id/:year/:month/:day/:file',
], async (req,res) => {
app.get(config.webPaths.apiPrefix+':auth/fileBin/:ke/:id/:file', async (req,res) => {
s.auth(req.params,function(user){
var failed = function(){
res.end(user.lang['File Not Found'])
}
if (!s.group[req.params.ke].fileBin[req.params.id+'/'+req.params.file]){
const groupKey = req.params.ke
const monitorId = req.params.id
const monitorRestrictions = s.getMonitorRestrictions(user.details,monitorId)
if(user.details.sub && user.details.allmonitors === '0' && (user.permissions.get_monitors === "0" || monitorRestrictions.length === 0)){
s.closeJsonResponse(res,{
ok: false,
msg: lang['Not Permitted']
})
return
}
s.knexQuery({
action: "select",
columns: "*",
table: "Files",
where: [
['ke','=',groupKey],
['mid','=',req.params.id],
['name','=',req.params.file],
monitorRestrictions
]
},(err,r) => {
if(r && r[0]){
r = r[0]
r.details = JSON.parse(r.details)
const filePath = s.dir.fileBin + req.params.ke + '/' + req.params.id + (r.details.year && r.details.month && r.details.day ? '/' + r.details.year + '/' + r.details.month + '/' + r.details.day : '') + '/' + req.params.file;
fs.stat(filePath,function(err,stats){
if(!err){
const groupKey = req.params.ke
const monitorId = req.params.id
const {
monitorPermissions,
monitorRestrictions,
} = s.getMonitorsPermitted(user.details,monitorId)
const {
isRestricted,
isRestrictedApiKey,
apiKeyPermissions,
} = s.checkPermission(user)
if(
isRestrictedApiKey && apiKeyPermissions.watch_videos_disallowed ||
isRestricted && !monitorPermissions[`${monitorId}_video_view`]
){
s.closeJsonResponse(res,{ok: false, msg: lang['Not Authorized']});
return
}
s.knexQuery({
action: "select",
columns: "*",
table: "Files",
where: [
['ke','=',groupKey],
['mid','=',req.params.id],
['name','=',req.params.file],
monitorRestrictions
]
},(err,r) => {
if(r && r[0]){
r = r[0]
r.details = JSON.parse(r.details)
const filename = req.params.file
const filePath = s.dir.fileBin + req.params.ke + '/' + req.params.id + (r.details.year && r.details.month && r.details.day ? '/' + r.details.year + '/' + r.details.month + '/' + r.details.day : '') + '/' + filename;
fs.stat(filePath,function(err,stats){
if(!err){
if(filename.endsWith('.mp4')){
s.streamMp4FileOverHttp(filePath,req,res,!!req.query.pureStream)
}else{
res.on('finish',function(){res.end()})
fs.createReadStream(filePath).pipe(res)
}else{
failed()
}
})
}else{
failed()
}
})
}else{
res.end(user.lang['Please Wait for Completion'])
}
}else{
failed()
}
})
}else{
failed()
}
})
},res,req);
});
/**
* API : Modify fileBin File
*/
app.get(config.webPaths.apiPrefix+':auth/fileBin/:ke/:id/:file/:mode', function (req,res){
let response = { ok: false };
res.setHeader('Content-Type', 'application/json');
s.auth(req.params,function(user){
const monitorId = req.params.id
const groupKey = req.params.ke
const filename = req.params.file
const {
monitorPermissions,
monitorRestrictions,
} = s.getMonitorsPermitted(user.details,monitorId)
const {
isRestricted,
isRestrictedApiKey,
apiKeyPermissions,
} = s.checkPermission(user);
if(
isRestrictedApiKey && apiKeyPermissions.delete_videos_disallowed ||
isRestricted && !monitorPermissions[`${monitorId}_video_delete`]
){
s.closeJsonResponse(res,{ok: false, msg: lang['Not Authorized']});
return
}
s.knexQuery({
action: "select",
columns: "*",
table: 'Files',
where: [
['ke','=',groupKey],
['mid','=',monitorId],
['name','=',filename]
],
limit: 1
},async (err,r) => {
if(r && r[0]){
const file = r[0];
var details = s.parseJSON(r.details) || {}
switch(req.params.mode){
case'archive':
response.ok = true
const unarchive = s.getPostData(req,'unarchive') == '1';
const archiveResponse = await archiveFileBinEntry(file,unarchive)
response.ok = archiveResponse.ok
response.archived = archiveResponse.archived
break;
case'delete':
response.ok = true;
await s.deleteFileBinEntry(file)
break;
default:
response.msg = user.lang.modifyVideoText1;
break;
}
}else{
response.msg = user.lang['No such file'];
}
s.closeJsonResponse(res,response);
})
},res,req);
})
}

36
libs/health.js
View File

@ -55,6 +55,9 @@ module.exports = function(s,config,lang,io){
exec(k.cmd,{encoding:'utf8',detached: true},function(err,d){
if(s.isWin===true){
d=d.replace(/(\r\n|\n|\r)/gm,"").replace(/%/g,"")
}else if(s.platform == 'darwin') {
// on macos the cpu% is per core, so a 10 core machine can go up to 1000%
d=parseFloat(d.trim()) / s.coreCount
}
resolve(d)
s.onGetCpuUsageExtensions.forEach(function(extender){
@ -100,18 +103,31 @@ module.exports = function(s,config,lang,io){
k.cmd = "LANG=C free | grep Mem | awk '{print $7/$2 * 100.0}'";
break;
}
let percent = 0
let used = 0
if(k.cmd){
exec(k.cmd,{encoding:'utf8',detached: true},function(err,d){
if(s.isWin===true){
d=(parseInt(d.split('=')[1])/(s.totalmem/1000))*100
}
resolve(d)
s.onGetRamUsageExtensions.forEach(function(extender){
extender(d)
})
})
exec(k.cmd,{encoding:'utf8',detached: true},function(err,d){
if(s.isWin===true){
const freeMb = parseInt(d.split('=')[1].trim()) / 1024
const totalMemInMb = s.totalmem/1024/1024
used = totalMemInMb - freeMb
percent=(used/totalMemInMb)*100
} else {
percent = d.trim()
}
resolve({
used,
percent
})
s.onGetRamUsageExtensions.forEach(function(extender){
extender(percent)
})
})
}else{
resolve(0)
resolve({
used,
percent
})
}
})
}

6
libs/health/utils.js
View File

@ -16,14 +16,11 @@ const currentCPUInfo = {
total: 0,
active: 0
}
const lastCPUInfo = {
let lastCPUInfo = {
total: 0,
active: 0
}
exports.getCpuUsageOnLinux = () => {
lastCPUInfo.active = currentCPUInfo.active;
lastCPUInfo.idle = currentCPUInfo.idle;
lastCPUInfo.total = currentCPUInfo.total;
return new Promise((resolve,reject) => {
const getUsage = function(callback){
fs.readFile("/proc/stat" ,'utf8', function(err, data){
@ -36,6 +33,7 @@ exports.getCpuUsageOnLinux = () => {
}
currentCPUInfo.active = currentCPUInfo.total - currentCPUInfo.idle
currentCPUInfo.percentUsed = calculateCPUPercentage(lastCPUInfo, currentCPUInfo);
lastCPUInfo = Object.assign({},currentCPUInfo)
callback(currentCPUInfo.percentUsed)
})
}

31
libs/language.js
View File

@ -3,17 +3,23 @@ module.exports = function(s,config){
if(!config.language){
config.language='en_CA'
}
try{
var lang = require(s.location.languages+'/'+config.language+'.json');
}catch(er){
console.error(er)
console.log('There was an error loading your language file.')
var lang = require(s.location.languages+'/en_CA.json');
var lang = {};
function getLanguageData(choice){
let gotLang = {}
try{
eval(`gotLang = ${fs.readFileSync(s.location.languages+'/'+choice+'.json','utf8')}`)
}catch(er){
console.error(er)
console.log('There was an error loading your language file.')
eval(`gotLang = ${fs.readFileSync(s.location.languages+'/en_CA.json','utf8')}`)
}
return gotLang
}
lang = getLanguageData(config.language)
//load languages dynamically
s.copySystemDefaultLanguage = function(){
//en_CA
return Object.assign(lang,{})
return Object.assign({},getLanguageData(config.language))
}
s.listOfPossibleLanguages = []
fs.readdirSync(s.mainDirectory + '/languages').forEach(function(filename){
@ -28,12 +34,15 @@ module.exports = function(s,config){
s.getLanguageFile = function(rule){
if(rule && rule !== ''){
var file = s.loadedLanguages[file]
s.debugLog(file)
if(!file){
try{
s.loadedLanguages[rule] = require(s.location.languages+'/'+rule+'.json')
s.loadedLanguages[rule] = Object.assign(s.copySystemDefaultLanguage(),s.loadedLanguages[rule])
let newLang = {}
eval(`newLang = ${fs.readFileSync(s.location.languages+'/'+rule+'.json','utf8')}`)
s.loadedLanguages[rule] = Object.assign(s.copySystemDefaultLanguage(),newLang)
file = s.loadedLanguages[rule]
}catch(err){
console.error(err)
file = s.copySystemDefaultLanguage()
}
}
@ -42,5 +51,9 @@ module.exports = function(s,config){
}
return file
}
s.reloadLanguages = function(){
s.loadedLanguages = {};
s.loadedLanguages[config.language] = s.copySystemDefaultLanguage()
}
return lang
}

504
libs/monitor.js
View File

@ -5,16 +5,22 @@ const exec = require('child_process').exec;
const Mp4Frag = require('mp4frag');
const onvif = require("shinobi-onvif");
const treekill = require('tree-kill');
const request = require('request');
const connectionTester = require('connection-tester')
const SoundDetection = require('shinobi-sound-detection')
const async = require("async");
const URL = require('url')
const {
Worker
} = require('worker_threads');
const { copyObject, createQueue, queryStringToObject, createQueryStringFromObject } = require('./common.js')
module.exports = function(s,config,lang){
const { fetchTimeout } = require('./basic/utils.js')(process.cwd(),config)
const isMasterNode = (
(
config.childNodes.enabled === true &&
config.childNodes.mode === 'master'
) ||
config.childNodes.enabled === false
);
const {
probeMonitor,
getStreamInfoFromProbe,
@ -27,6 +33,11 @@ module.exports = function(s,config,lang){
processKill,
cameraDestroy,
monitorConfigurationMigrator,
attachStreamChannelHandlers,
setActiveViewer,
getActiveViewerCount,
destroySubstreamProcess,
attachMainProcessHandlers,
} = require('./monitor/utils.js')(s,config,lang)
const {
addEventDetailsToString,
@ -37,15 +48,21 @@ module.exports = function(s,config,lang){
setPresetForCurrentPosition
} = require('./control/ptz.js')(s,config,lang)
const {
scanForOrphanedVideos
scanForOrphanedVideos,
reEncodeVideoAndBinOriginalAddToQueue,
} = require('./video/utils.js')(s,config,lang)
const {
selectNodeForOperation,
bindMonitorToChildNode
} = require('./childNode/utils.js')(s,config,lang)
const startMonitorInQueue = createQueue(1, 3)
s.initiateMonitorObject = function(e){
if(!s.group[e.ke]){s.group[e.ke]={}};
if(!s.group[e.ke].activeMonitors){s.group[e.ke].activeMonitors={}}
if(!s.group[e.ke].activeMonitors[e.mid]){s.group[e.ke].activeMonitors[e.mid]={}}
const activeMonitor = s.group[e.ke].activeMonitors[e.mid]
activeMonitor.ke = e.ke
activeMonitor.mid = e.mid
if(!activeMonitor.streamIn){activeMonitor.streamIn={}};
if(!activeMonitor.emitterChannel){activeMonitor.emitterChannel={}};
if(!activeMonitor.mp4frag){activeMonitor.mp4frag={}};
@ -53,7 +70,7 @@ module.exports = function(s,config,lang){
if(!activeMonitor.contentWriter){activeMonitor.contentWriter={}};
if(!activeMonitor.childNodeStreamWriters){activeMonitor.childNodeStreamWriters={}};
if(!activeMonitor.eventBasedRecording){activeMonitor.eventBasedRecording={}};
if(!activeMonitor.watch){activeMonitor.watch={}};
if(!activeMonitor.watch){activeMonitor.watch = []};
if(!activeMonitor.fixingVideos){activeMonitor.fixingVideos={}};
// if(!activeMonitor.viewerConnection){activeMonitor.viewerConnection={}};
// if(!activeMonitor.viewerConnectionCount){activeMonitor.viewerConnectionCount=0};
@ -138,7 +155,7 @@ module.exports = function(s,config,lang){
return x.ar;
}
s.getStreamsDirectory = (monitor) => {
return s.dir.streams + monitor.ke + '/' + monitor.mid + '/'
return s.dir.streams + monitor.ke + '/' + (monitor.mid || monitor.id) + '/'
}
s.getRawSnapshotFromMonitor = function(monitor,options){
return new Promise((resolve,reject) => {
@ -185,7 +202,7 @@ module.exports = function(s,config,lang){
var snapBuffer = []
var temporaryImageFile = streamDir + s.gid(5) + '.jpg'
var iconImageFile = streamDir + 'icon.jpg'
var ffmpegCmd = splitForFFPMEG(`-loglevel warning -re -probesize 100000 -analyzeduration 100000 ${inputOptions.join(' ')} -i "${url}" ${outputOptions.join(' ')} -f image2 -an -vf "fps=1" -vframes 1 "${temporaryImageFile}"`)
var ffmpegCmd = splitForFFPMEG(`-y -loglevel warning -re ${inputOptions.join(' ')} -i "${url}" ${outputOptions.join(' ')} -f image2 -an -frames:v 1 "${temporaryImageFile}"`)
checkExists(streamDir, function(success) {
if (success === false) {
fs.mkdirSync(streamDir, {recursive: true}, (err) => {s.debugLog(err)})
@ -500,20 +517,7 @@ module.exports = function(s,config,lang){
s.checkDetails(e)
if(e.ke && config.doSnapshot === true){
if(s.group[e.ke] && s.group[e.ke].rawMonitorConfigurations && s.group[e.ke].rawMonitorConfigurations[e.mid] && s.group[e.ke].rawMonitorConfigurations[e.mid].mode !== 'stop'){
if(s.group[e.ke].activeMonitors[e.mid].onvifConnection){
const screenShot = await s.getSnapshotFromOnvif({
username: onvifUsername,
password: onvifPassword,
uri: cameraResponse.uri,
});
s.tx({
f: 'monitor_snapshot',
snapshot: screenShot.toString('base64'),
snapshot_format: 'b64',
mid: e.mid,
ke: e.ke
},'GRP_'+e.ke)
}else{
async function getRaw(){
var pathDir = s.dir.streams+e.ke+'/'+e.mid+'/'
const {screenShot, isStaticFile} = await s.getRawSnapshotFromMonitor(s.group[e.ke].rawMonitorConfigurations[e.mid],options)
if(screenShot){
@ -527,7 +531,27 @@ module.exports = function(s,config,lang){
}else{
s.debugLog('Damaged Snapshot Data')
s.tx({f:'monitor_snapshot',snapshot:e.mon.name,snapshot_format:'plc',mid:e.mid,ke:e.ke},'GRP_'+e.ke)
}
}
}
if(s.group[e.ke].activeMonitors[e.mid].onvifConnection){
try{
const screenShot = await s.getSnapshotFromOnvif({
ke: e.ke,
mid: e.mid,
});
s.tx({
f: 'monitor_snapshot',
snapshot: screenShot.toString('base64'),
snapshot_format: 'b64',
mid: e.mid,
ke: e.ke
},'GRP_'+e.ke)
}catch(err){
s.debugLog(err)
await getRaw()
}
}else{
await getRaw()
}
}else{
s.tx({f:'monitor_snapshot',snapshot:'Disabled',snapshot_format:'plc',mid:e.mid,ke:e.ke},'GRP_'+e.ke)
@ -587,7 +611,7 @@ module.exports = function(s,config,lang){
}
const createTimelapseDirectory = function(e,callback){
var directory = s.getTimelapseFrameDirectory(e)
fs.mkdir(directory,function(err){
fs.mkdir(directory,{ recursive: true },function(err){
s.handleFolderError(err)
callback(err,directory)
})
@ -752,55 +776,8 @@ module.exports = function(s,config,lang){
code: e.wantedStatusCode
});
//on unexpected exit restart
s.group[e.ke].activeMonitors[e.id].spawn_exit = function(){
if(s.group[e.ke].activeMonitors[e.id].isStarted === true){
if(e.details.loglevel!=='quiet'){
s.userLog(e,{type:lang['Process Unexpected Exit'],msg:{msg:lang.unexpectedExitText,cmd:s.group[e.ke].activeMonitors[e.id].ffmpeg}});
}
fatalError(e,'Process Unexpected Exit');
scanForOrphanedVideos(e,{
forceCheck: true,
checkMax: 2
})
s.onMonitorUnexpectedExitExtensions.forEach(function(extender){
extender(Object.assign(s.group[e.ke].rawMonitorConfigurations[e.id],{}),e)
})
}
}
s.group[e.ke].activeMonitors[e.id].spawn.on('end',s.group[e.ke].activeMonitors[e.id].spawn_exit)
s.group[e.ke].activeMonitors[e.id].spawn.on('exit',s.group[e.ke].activeMonitors[e.id].spawn_exit)
s.group[e.ke].activeMonitors[e.id].spawn.on('error',function(er){
s.userLog(e,{type:'Spawn Error',msg:er});fatalError(e,'Spawn Error')
})
s.userLog(e,{type:lang['Process Started'],msg:{cmd:s.group[e.ke].activeMonitors[e.id].ffmpeg}})
if(s.isWin === false){
var strippedHost = s.stripAuthFromHost(e)
var sendProcessCpuUsage = function(){
s.getMonitorCpuUsage(e,function(percent){
s.group[e.ke].activeMonitors[e.id].currentCpuUsage = percent
s.tx({
f: 'camera_cpu_usage',
ke: e.ke,
id: e.id,
percent: percent
},'MON_STREAM_'+e.ke+e.id)
})
}
clearInterval(s.group[e.ke].activeMonitors[e.id].getMonitorCpuUsage)
s.group[e.ke].activeMonitors[e.id].getMonitorCpuUsage = setInterval(function(){
if(e.details.skip_ping !== '1'){
connectionTester.test(strippedHost,e.port,2000,function(err,response){
if(response.success){
sendProcessCpuUsage()
}else{
launchMonitorProcesses(e)
}
})
}else{
sendProcessCpuUsage()
}
},1000 * 60)
}
if(s.group[e.ke].activeMonitors[e.id].spawn)attachMainProcessHandlers(e,fatalError)
return s.group[e.ke].activeMonitors[e.id].spawn
}
const createEventCounter = function(monitor){
if(monitor.details.detector_obj_count === '1'){
@ -923,42 +900,16 @@ module.exports = function(s,config,lang){
//frames from motion detect
if(e.details.detector_pam === '1'){
// s.group[e.ke].activeMonitors[e.id].spawn.stdio[3].pipe(s.group[e.ke].activeMonitors[e.id].p2p).pipe(s.group[e.ke].activeMonitors[e.id].pamDiff)
s.group[e.ke].activeMonitors[e.id].spawn.stdio[3].on('data',function(buf){
let theJson
try{
buf.toString().split('}{').forEach((object,n)=>{
theJson = object
if(object.substr(object.length - 1) !== '}')theJson += '}'
if(object.substr(0,1) !== '{')theJson = '{' + theJson
try{
var data = JSON.parse(theJson)
}catch(err){
var data = JSON.parse(theJson + '}')
}
switch(data.f){
case'trigger':
triggerEvent(data)
break;
case's.tx':
s.tx(data.data,data.to)
break;
}
})
}catch(err){
console.log(theJson)
console.log('There was an error parsing a detector event')
console.log(err)
}
})
// spawn.stdio[3] is deprecated and now motion events are handled by dataPort
if(e.details.detector_use_detect_object === '1' && e.details.detector_use_motion === '1' ){
s.group[e.ke].activeMonitors[e.id].spawn.stdio[4].on('data',function(data){
onDetectorJpegOutputSecondary(e,data)
})
}else{
s.group[e.ke].activeMonitors[e.id].spawn.stdio[4].on('data',function(data){
s.group[e.ke].activeMonitors[e.id].spawn.stdio[4].on('data',function(data){
onDetectorJpegOutputAlone(e,data)
})
}
}
}else if(e.details.detector_use_detect_object === '1' && e.details.detector_send_frames !== '1'){
s.group[e.ke].activeMonitors[e.id].spawn.stdio[4].on('data',function(data){
onDetectorJpegOutputSecondary(e,data)
@ -970,8 +921,9 @@ module.exports = function(s,config,lang){
}
}
//frames to stream
var frameToStreamPrimary
switch(e.details.stream_type){
var frameToStreamPrimary;
const streamType = e.details.stream_type;
switch(streamType){
case'mp4':
delete(s.group[e.ke].activeMonitors[e.id].mp4frag['MAIN'])
if(!s.group[e.ke].activeMonitors[e.id].mp4frag['MAIN'])s.group[e.ke].activeMonitors[e.id].mp4frag['MAIN'] = new Mp4Frag()
@ -996,12 +948,6 @@ module.exports = function(s,config,lang){
s.group[e.ke].activeMonitors[e.id].emitter.emit('data',d)
}
break;
case'h265':
frameToStreamPrimary = function(d){
resetStreamCheck(e)
s.group[e.ke].activeMonitors[e.id].emitter.emit('data',d)
}
break;
case'b64':case undefined:case null:case'':
var buffer
frameToStreamPrimary = function(d){
@ -1018,50 +964,28 @@ module.exports = function(s,config,lang){
}
break;
}
s.onMonitorCreateStreamPipeExtensions.forEach(function(extender){
if(!frameToStreamPrimary)frameToStreamPrimary = extender(streamType,e,resetStreamCheck)
});
if(frameToStreamPrimary){
s.group[e.ke].activeMonitors[e.id].spawn.stdout.on('data',frameToStreamPrimary)
}
if(e.details.stream_channels && e.details.stream_channels !== ''){
var createStreamEmitter = function(channel,number){
var pipeNumber = number+config.pipeAddition;
if(!s.group[e.ke].activeMonitors[e.id].emitterChannel[pipeNumber]){
s.group[e.ke].activeMonitors[e.id].emitterChannel[pipeNumber] = new events.EventEmitter().setMaxListeners(0);
}
var frameToStreamAdded
switch(channel.stream_type){
case'mp4':
delete(s.group[e.ke].activeMonitors[e.id].mp4frag[pipeNumber])
if(!s.group[e.ke].activeMonitors[e.id].mp4frag[pipeNumber])s.group[e.ke].activeMonitors[e.id].mp4frag[pipeNumber] = new Mp4Frag();
s.group[e.ke].activeMonitors[e.id].spawn.stdio[pipeNumber].pipe(s.group[e.ke].activeMonitors[e.id].mp4frag[pipeNumber],{ end: false })
break;
case'mjpeg':
frameToStreamAdded = function(d){
s.group[e.ke].activeMonitors[e.id].emitterChannel[pipeNumber].emit('data',d)
}
break;
case'flv':
frameToStreamAdded = function(d){
if(!s.group[e.ke].activeMonitors[e.id].firstStreamChunk[pipeNumber])s.group[e.ke].activeMonitors[e.id].firstStreamChunk[pipeNumber] = d;
frameToStreamAdded = function(d){
s.group[e.ke].activeMonitors[e.id].emitterChannel[pipeNumber].emit('data',d)
}
frameToStreamAdded(d)
}
break;
case'h264':
frameToStreamAdded = function(d){
s.group[e.ke].activeMonitors[e.id].emitterChannel[pipeNumber].emit('data',d)
}
break;
}
if(frameToStreamAdded){
s.group[e.ke].activeMonitors[e.id].spawn.stdio[pipeNumber].on('data',frameToStreamAdded)
}
}
e.details.stream_channels.forEach(createStreamEmitter)
e.details.stream_channels.forEach((fields,number) => {
attachStreamChannelHandlers({
ke: e.ke,
mid: e.id,
fields: fields,
number: number,
ffmpegProcess: s.group[e.ke].activeMonitors[e.id].spawn,
})
})
}
}
const catchNewSegmentNames = function(e){
const monitorConfig = s.group[e.ke].rawMonitorConfigurations[e.id]
const monitorDetails = monitorConfig.details
const autoCompressionEnabled = monitorDetails.auto_compress_videos === '1'
var checkLog = function(d,x){return d.indexOf(x)>-1}
s.group[e.ke].activeMonitors[e.id].spawn.stdio[8].on('data',function(d){
d=d.toString();
@ -1070,7 +994,7 @@ module.exports = function(s,config,lang){
s.insertCompletedVideo(e,{
file: filename,
events: s.group[e.ke].activeMonitors[e.id].detector_motion_count
},function(err){
},function(err,response){
s.userLog(e,{type:lang['Video Finished'],msg:{filename:d}})
if(
e.details.detector === '1' &&
@ -1088,6 +1012,21 @@ module.exports = function(s,config,lang){
ke : e.ke,
id : e.id
})
}else if(autoCompressionEnabled){
s.debugLog('Queue Automatic Compression',response.insertQuery)
reEncodeVideoAndBinOriginalAddToQueue({
video: response.insertQuery,
targetVideoCodec: 'vp9',
targetAudioCodec: 'libopus',
targetQuality: '-q:v 1 -q:a 1',
targetExtension: 'webm',
doSlowly: false,
automated: true,
}).then((encodeResponse) => {
s.debugLog('Complete Automatic Compression',encodeResponse)
}).catch((err) => {
console.log(err)
})
}
s.group[e.ke].activeMonitors[e.id].detector_motion_count = []
})
@ -1158,11 +1097,11 @@ module.exports = function(s,config,lang){
s.group[e.ke].activeMonitors[monitorId].detector_notrigger_webhook = s.createTimeout('detector_notrigger_webhook',s.group[e.ke].activeMonitors[monitorId],currentConfig.detector_notrigger_webhook_timeout,10)
var detector_notrigger_webhook_url = addEventDetailsToString(e,currentConfig.detector_notrigger_webhook_url)
var webhookMethod = currentConfig.detector_notrigger_webhook_method
if(!webhookMethod || webhookMethod === '')webhookMethod = 'GET'
request(detector_notrigger_webhook_url,{method: webhookMethod,encoding:null},function(err,data){
if(err){
s.userLog(d,{type:lang["Event Webhook Error"],msg:{error:err,data:data}})
}
if(!webhookMethod || webhookMethod === '')webhookMethod = 'GET';
fetchTimeout(detector_notrigger_webhook_url,10000,{
method: webhookMethod
}).catch((err) => {
s.userLog(d,{type:lang["Event Webhook Error"],msg:{error:err,data:data}})
})
}
if(currentConfig.detector_notrigger_command_enable === '1' && !s.group[e.ke].activeMonitors[monitorId].detector_notrigger_command){
@ -1278,27 +1217,29 @@ module.exports = function(s,config,lang){
if(pingResponse.success === true){
activeMonitor.isRecording = true
try{
createCameraFfmpegProcess(e)
createCameraStreamHandlers(e)
var mainProcess = createCameraFfmpegProcess(e)
createEventCounter(e)
if(e.type === 'dashcam' || e.type === 'socket'){
setTimeout(function(){
activeMonitor.allowStdinWrite = true
s.txToDashcamUsers({
f : 'enable_stream',
ke : e.ke,
mid : e.id
},e.ke)
},30000)
}
if(
e.functionMode === 'record' ||
e.type === 'mjpeg' ||
e.type === 'h264' ||
e.type === 'local'
){
catchNewSegmentNames(e)
cameraFilterFfmpegLog(e)
if(mainProcess){
createCameraStreamHandlers(e)
if(e.type === 'dashcam' || e.type === 'socket'){
setTimeout(function(){
activeMonitor.allowStdinWrite = true
s.txToDashcamUsers({
f : 'enable_stream',
ke : e.ke,
mid : e.id
},e.ke)
},30000)
}
if(
e.functionMode === 'record' ||
e.type === 'mjpeg' ||
e.type === 'h264' ||
e.type === 'local'
){
catchNewSegmentNames(e)
cameraFilterFfmpegLog(e)
}
}
clearTimeout(activeMonitor.onMonitorStartTimer)
activeMonitor.onMonitorStartTimer = setTimeout(() => {
@ -1349,6 +1290,8 @@ module.exports = function(s,config,lang){
//data, options
d : s.group[e.ke].rawMonitorConfigurations[e.id]
},activeMonitor.childNodeId)
clearTimeout(activeMonitor.recordingChecker);
clearTimeout(activeMonitor.streamChecker);
}
if(
e.type !== 'socket' &&
@ -1364,37 +1307,21 @@ module.exports = function(s,config,lang){
}
try{
if(config.childNodes.enabled === true && config.childNodes.mode === 'master'){
var copiedMonitorObject = s.cleanMonitorObject(s.group[e.ke].rawMonitorConfigurations[e.id])
var childNodeList = Object.keys(s.childNodes)
if(childNodeList.length > 0){
e.childNodeFound = false
var selectNode = function(ip){
e.childNodeFound = true
e.childNodeSelected = ip
}
var nodeWithLowestActiveCamerasCount = 65535
var nodeWithLowestActiveCameras = null
childNodeList.forEach(function(ip){
delete(s.childNodes[ip].activeCameras[e.ke+e.id])
var nodeCameraCount = Object.keys(s.childNodes[ip].activeCameras).length
if(!s.childNodes[ip].dead && nodeCameraCount < nodeWithLowestActiveCamerasCount && s.childNodes[ip].cpu < 75){
nodeWithLowestActiveCamerasCount = nodeCameraCount
nodeWithLowestActiveCameras = ip
}
})
if(nodeWithLowestActiveCameras)selectNode(nodeWithLowestActiveCameras)
if(e.childNodeFound === true){
s.childNodes[e.childNodeSelected].activeCameras[e.ke+e.id] = copiedMonitorObject
activeMonitor.childNode = e.childNodeSelected
activeMonitor.childNodeId = s.childNodes[e.childNodeSelected].cnid;
s.cx({f:'sync',sync:s.group[e.ke].rawMonitorConfigurations[e.id],ke:e.ke,mid:e.id},activeMonitor.childNodeId);
selectNodeForOperation({
ke: e.ke,
mid: e.id,
}).then((selectedNode) => {
if(selectedNode){
bindMonitorToChildNode({
ke: e.ke,
mid: e.id,
childNodeId: selectedNode,
})
doOnChildMachine()
}else{
startMonitorInQueue.push(doOnThisMachine,function(){})
}
}else{
startMonitorInQueue.push(doOnThisMachine,function(){})
}
});
}else{
startMonitorInQueue.push(doOnThisMachine,function(){})
}
@ -1568,26 +1495,23 @@ module.exports = function(s,config,lang){
s.initiateMonitorObject({ke:e.ke,mid:e.id})
switch(e.functionMode){
case'watch_on'://live streamers - join
if(!cn.monitorsCurrentlyWatching){cn.monitorsCurrentlyWatching = {}}
if(!cn.monitorsCurrentlyWatching[e.id]){cn.monitorsCurrentlyWatching[e.id]={ke:e.ke}}
s.group[e.ke].activeMonitors[e.id].watch[cn.id]={};
var numberOfViewers = Object.keys(s.group[e.ke].activeMonitors[e.id].watch).length
s.tx({
viewers: numberOfViewers,
ke: e.ke,
id: e.id
},'MON_'+e.ke+e.id)
if(!cn.monitorsCurrentlyWatching){cn.monitorsCurrentlyWatching = {}}
if(!cn.monitorsCurrentlyWatching[e.id]){cn.monitorsCurrentlyWatching[e.id]={ke:e.ke}}
setActiveViewer(e.ke,e.id,cn.id,true)
s.group[e.ke].activeMonitors[e.id].allowDestroySubstream = false
clearTimeout(s.group[e.ke].activeMonitors[e.id].noViewerCountDisableSubstream)
break;
case'watch_off'://live streamers - leave
if(cn.monitorsCurrentlyWatching){delete(cn.monitorsCurrentlyWatching[e.id])}
var numberOfViewers = 0
delete(s.group[e.ke].activeMonitors[e.id].watch[cn.id]);
numberOfViewers = Object.keys(s.group[e.ke].activeMonitors[e.id].watch).length
s.tx({
viewers: numberOfViewers,
ke: e.ke,
id: e.id
},'MON_'+e.ke+e.id)
setActiveViewer(e.ke,e.id,cn.id,false)
clearTimeout(s.group[e.ke].activeMonitors[e.id].noViewerCountDisableSubstream)
s.group[e.ke].activeMonitors[e.id].noViewerCountDisableSubstream = setTimeout(async () => {
let currentCount = getActiveViewerCount(e.ke,e.id)
if(currentCount === 0 && s.group[e.ke].activeMonitors[e.id].subStreamProcess){
s.group[e.ke].activeMonitors[e.id].allowDestroySubstream = true
await destroySubstreamProcess(s.group[e.ke].activeMonitors[e.id])
}
},10000)
break;
case'restart'://restart monitor
s.sendMonitorStatus({
@ -1605,13 +1529,13 @@ module.exports = function(s,config,lang){
if(!s.group[e.ke]||!s.group[e.ke].activeMonitors[e.id]){return}
if(config.childNodes.enabled === true && config.childNodes.mode === 'master' && s.group[e.ke].activeMonitors[e.id].childNode && s.childNodes[s.group[e.ke].activeMonitors[e.id].childNode].activeCameras[e.ke+e.id]){
s.group[e.ke].activeMonitors[e.id].isStarted = false
s.cx({f:'sync',sync:s.group[e.ke].rawMonitorConfigurations[e.id],ke:e.ke,mid:e.id},s.group[e.ke].activeMonitors[e.id].childNodeId);
s.cx({
//function
f : 'cameraStop',
//data, options
d : s.group[e.ke].rawMonitorConfigurations[e.id]
},s.group[e.ke].activeMonitors[e.id].childNodeId)
s.cx({f:'sync',sync:s.group[e.ke].rawMonitorConfigurations[e.id],ke:e.ke,mid:e.id},s.group[e.ke].activeMonitors[e.id].childNodeId);
}else{
closeEventBasedRecording(e)
if(s.group[e.ke].activeMonitors[e.id].fswatch){s.group[e.ke].activeMonitors[e.id].fswatch.close();delete(s.group[e.ke].activeMonitors[e.id].fswatch)}
@ -1653,15 +1577,17 @@ module.exports = function(s,config,lang){
status: wantedStatus,
code: wantedStatusCode,
})
setTimeout(() => {
scanForOrphanedVideos({
ke: e.ke,
mid: e.id,
},{
forceCheck: true,
checkMax: 2
})
},2000)
if(isMasterNode){
setTimeout(() => {
scanForOrphanedVideos({
ke: e.ke,
mid: e.id,
},{
forceCheck: true,
checkMax: 2
})
},2000)
}
clearTimeout(s.group[e.ke].activeMonitors[e.id].onMonitorStartTimer)
s.onMonitorStopExtensions.forEach(function(extender){
extender(Object.assign(s.group[e.ke].rawMonitorConfigurations[e.id],{}),e)
@ -1676,6 +1602,15 @@ module.exports = function(s,config,lang){
//stop action, monitor already started or recording
return
}
if(activeMonitor.masterSaysToStop === true){
s.sendMonitorStatus({
id: e.id,
ke: e.ke,
status: lang.Stopped,
code: 5,
})
return;
}
if(config.probeMonitorOnStart === true){
const probeResponse = await probeMonitor(s.group[e.ke].rawMonitorConfigurations[e.id],2000,true)
const probeStreams = getStreamInfoFromProbe(probeResponse.result)
@ -1856,6 +1791,7 @@ module.exports = function(s,config,lang){
monitorRestrictions.push(['or','mid','=',v])
}
})
console.log(monitorRestrictions)
}catch(er){
}
}else if(
@ -1873,20 +1809,104 @@ module.exports = function(s,config,lang){
){}
return monitorRestrictions
}
// s.checkViewerConnectionsForMonitor = function(monitorObject){
// var monitorConfig = s.group[monitorObject.ke].rawMonitorConfigurations[monitorObject.mid]
// if(monitorConfig.mode === 'start'){
//
// }
// }
// s.addViewerConnectionForMonitor = function(monitorObject,viewerDetails){
// s.group[monitorObject.ke].activeMonitors[monitorObject.mid].viewerConnection[viewerDetails.viewerId] = viewerDetails
// s.group[monitorObject.ke].activeMonitors[monitorObject.mid].viewerConnectionCount += 1
// return s.group[monitorObject.ke].activeMonitors[monitorObject.mid].viewerConnectionCount
// }
// s.removeViewerConnectionForMonitor = function(monitorObject,viewerDetails){
// delete(s.group[monitorObject.ke].activeMonitors[monitorObject.mid].viewerConnection[viewerDetails.viewerId])
// s.group[monitorObject.ke].activeMonitors[monitorObject.mid].viewerConnectionCount -= 1
// return s.group[monitorObject.ke].activeMonitors[monitorObject.mid].viewerConnectionCount
// }
s.checkPermission = (user) => {
// provide "user" object given from "s.auth"
const isSubAccount = !!user.details.sub
const response = {
isSubAccount,
hasAllPermissions: isSubAccount && user.details.allmonitors === '1',
isRestricted: isSubAccount && user.details.allmonitors !== '1',
isRestrictedApiKey: false,
apiKeyPermissions: {},
userPermissions: {},
}
const permissions = user.permissions
const details = user.details;
[
'auth_socket',
'get_monitors',
'control_monitors',
'get_logs',
'watch_stream',
'watch_snapshot',
'watch_videos',
'delete_videos',
].forEach((key) => {
const permissionOff = permissions[key] === '0';
response.apiKeyPermissions[key] = permissions[key] === '1';
response.apiKeyPermissions[`${key}_disallowed`] = permissionOff;
response.isRestrictedApiKey = response.isRestrictedApiKey || permissionOff;
});
// Base Level Permissions
// allmonitors : All Monitors and Privileges
// monitor_create : Can Create and Delete Monitors
// user_change : Can Change User Settings
// view_logs : Can View Logs
[
'allmonitors',
'monitor_create',
'user_change',
'view_logs',
].forEach((key) => {
response.userPermissions[key] = details[key] === '1' || !details[key];
response.userPermissions[`${key}_disallowed`] = details[key] === '0';
});
return response
}
s.getMonitorsPermitted = (userDetails,monitorId) => {
const monitorRestrictions = []
const monitors = {}
function setMonitorPermissions(mid){
// monitors : Can View Monitor
// monitor_edit : Can Edit Monitor (Delete as well)
// video_view : Can View Videos and Events
// video_delete : Can Delete Videos and Events
[
'monitors',
'monitor_edit',
'video_view',
'video_delete',
].forEach((key) => {
monitors[`${mid}_${key}`] = userDetails[key] && userDetails[key].indexOf(mid) > -1 || false;
});
return true
}
function addToQuery(mid,n){
if(n === 0){
monitorRestrictions.push(['mid','=',mid])
}else{
monitorRestrictions.push(['or','mid','=',mid])
}
};
if(
!monitorId &&
userDetails.sub &&
userDetails.monitors &&
userDetails.allmonitors !== '1'
){
try{
userDetails.monitors = s.parseJSON(userDetails.monitors)
userDetails.monitors.forEach(function(v,n){
setMonitorPermissions(v)
addToQuery(v,n)
})
}catch(err){
s.debugLog(err)
}
}else if(
monitorId && (
!userDetails.sub ||
userDetails.allmonitors !== '0' ||
userDetails.monitors.indexOf(monitorId) >- 1
)
){
setMonitorPermissions(monitorId)
addToQuery(monitorId,0)
}
return {
monitorPermissions: monitors,
// queryConditions
monitorRestrictions: monitorRestrictions,
}
}
}

400
libs/monitor/utils.js
View File

@ -1,21 +1,39 @@
const fs = require('fs');
const treekill = require('tree-kill');
const spawn = require('child_process').spawn;
const events = require('events');
const Mp4Frag = require('mp4frag');
const streamViewerCountTimeouts = {}
module.exports = (s,config,lang) => {
const {
scanForOrphanedVideos
} = require('../video/utils.js')(s,config,lang)
const {
createPipeArray,
splitForFFPMEG,
sanitizedFfmpegCommand,
} = require('../ffmpeg/utils.js')(s,config,lang)
const {
buildSubstreamString,
getDefaultSubstreamFields,
} = require('../ffmpeg/builders.js')(s,config,lang)
const getUpdateableFields = require('./updatedFields.js')
const processKill = (proc) => {
const response = {ok: true}
return new Promise((resolve,reject) => {
if(!proc){
resolve(response)
return
}
function sendError(err){
response.ok = false
response.err = err
resolve(response)
}
try{
proc.stdin.write("q\r\n")
if(proc && proc.stdin) {
proc.stdin.write("q\r\n");
}
setTimeout(() => {
if(proc && proc.kill){
if(s.isWin){
@ -86,13 +104,17 @@ module.exports = (s,config,lang) => {
if(activeMonitor.onChildNodeExit){
activeMonitor.onChildNodeExit()
}
activeMonitor.spawn.stdio.forEach(function(stdio){
try{
stdio.unpipe()
}catch(err){
console.log(err)
}
})
try{
activeMonitor.spawn.stdio.forEach(function(stdio){
try{
stdio.unpipe()
}catch(err){
console.log(err)
}
})
}catch(err){
// s.debugLog(err)
}
if(activeMonitor.mp4frag){
var mp4FragChannels = Object.keys(activeMonitor.mp4frag)
mp4FragChannels.forEach(function(channel){
@ -108,6 +130,10 @@ module.exports = (s,config,lang) => {
}else{
processKill(proc).then((response) => {
s.debugLog(`cameraDestroy`,response)
activeMonitor.allowDestroySubstream = true
destroySubstreamProcess(activeMonitor).then((response) => {
if(response.hadSubStream)s.debugLog(`cameraDestroy`,response.closeResponse)
})
})
}
}
@ -136,7 +162,7 @@ module.exports = (s,config,lang) => {
})
}
const temporaryImageFile = streamDir + s.gid(5) + '.jpg'
const ffmpegCmd = splitForFFPMEG(`-loglevel warning -re -stimeout 30000000 -probesize 100000 -analyzeduration 100000 ${inputOptions.join(' ')} -i "${url}" ${outputOptions.join(' ')} -f image2 -an -vf "fps=1" -vframes 1 "${temporaryImageFile}"`)
const ffmpegCmd = splitForFFPMEG(`-y -loglevel warning -re ${inputOptions.join(' ')} -i "${url}" ${outputOptions.join(' ')} -f image2 -an -frames:v 1 "${temporaryImageFile}"`)
const snapProcess = spawn('ffmpeg',ffmpegCmd,{detached: true})
snapProcess.stderr.on('data',function(data){
// s.debugLog(data.toString())
@ -195,11 +221,367 @@ module.exports = (s,config,lang) => {
}
})
}
const spawnSubstreamProcess = function(e){
// e = monitorConfig
try{
const groupKey = e.ke
const monitorId = e.mid
const monitorConfig = Object.assign({},s.group[groupKey].rawMonitorConfigurations[monitorId])
const monitorDetails = monitorConfig.details
const activeMonitor = s.group[e.ke].activeMonitors[e.mid]
const channelNumber = 1 + (monitorDetails.stream_channels || []).length
const ffmpegCommand = [`-progress pipe:5`];
const logLevel = monitorDetails.loglevel ? e.details.loglevel : 'warning'
const stdioPipes = createPipeArray({}, 2)
const substreamConfig = monitorConfig.details.substream
substreamConfig.input.type = !substreamConfig.input.fulladdress ? monitorConfig.type : substreamConfig.input.type || monitorConfig.details.rtsp_transport
substreamConfig.input.fulladdress = substreamConfig.input.fulladdress || s.buildMonitorUrl(monitorConfig)
substreamConfig.input.rtsp_transport = substreamConfig.input.rtsp_transport || monitorConfig.details.rtsp_transport
const {
inputAndConnectionFields,
outputFields,
} = getDefaultSubstreamFields(monitorConfig);
([
buildSubstreamString(channelNumber + config.pipeAddition,e),
]).forEach(function(commandStringPart){
ffmpegCommand.push(commandStringPart)
});
const ffmpegCommandString = ffmpegCommand.join(' ')
activeMonitor.ffmpegSubstream = sanitizedFfmpegCommand(e,ffmpegCommandString)
const ffmpegCommandParsed = splitForFFPMEG(ffmpegCommandString)
activeMonitor.subStreamChannel = channelNumber;
s.userLog({
ke: e.ke,
mid: e.mid,
},
{
type: lang["Substream Process"],
msg: {
msg: lang["Process Started"],
cmd: ffmpegCommandString,
},
});
const subStreamProcess = spawn(config.ffmpegDir,ffmpegCommandParsed,{detached: true,stdio: stdioPipes})
attachStreamChannelHandlers({
ke: e.ke,
mid: e.mid,
fields: Object.assign({},inputAndConnectionFields,outputFields),
number: activeMonitor.subStreamChannel,
ffmpegProcess: subStreamProcess,
})
if(config.debugLog === true){
subStreamProcess.stderr.on('data',(data) => {
console.log(`${e.ke} ${e.mid}`)
console.log(data.toString())
})
}
if(logLevel !== 'quiet'){
subStreamProcess.stderr.on('data',(data) => {
s.userLog({
ke: e.ke,
mid: e.mid,
},{
type: lang["Substream Process"],
msg: data.toString()
})
})
}
subStreamProcess.on('close',(data) => {
if(!activeMonitor.allowDestroySubstream){
subStreamProcess.stderr.on('data',(data) => {
s.userLog({
ke: e.ke,
mid: e.mid,
},
{
type: lang["Substream Process"],
msg: lang["Process Crashed for Monitor"],
})
})
setTimeout(() => {
spawnSubstreamProcess(e)
},2000)
}
})
activeMonitor.subStreamProcess = subStreamProcess
s.tx({
f: 'substream_start',
mid: e.mid,
ke: e.ke,
channel: activeMonitor.subStreamChannel
},'GRP_'+e.ke);
return subStreamProcess
}catch(err){
s.systemLog(err)
return null
}
}
const destroySubstreamProcess = async function(activeMonitor){
// e = monitorConfig.details.substream
const response = {
hadSubStream: false,
alreadyClosing: false
}
try{
if(activeMonitor.subStreamProcessClosing){
response.alreadyClosing = true
}else if(activeMonitor.subStreamProcess){
activeMonitor.subStreamProcessClosing = true
activeMonitor.subStreamChannel = null;
const closeResponse = await processKill(activeMonitor.subStreamProcess)
response.hadSubStream = true
response.closeResponse = closeResponse
delete(activeMonitor.subStreamProcess)
s.tx({
f: 'substream_end',
mid: activeMonitor.mid,
ke: activeMonitor.ke
},'GRP_'+activeMonitor.ke);
activeMonitor.subStreamProcessClosing = false
}
}catch(err){
s.debugLog('destroySubstreamProcess',err)
}
return response
}
function attachStreamChannelHandlers(options){
const fields = options.fields
const number = options.number
const ffmpegProcess = options.ffmpegProcess
const activeMonitor = s.group[options.ke].activeMonitors[options.mid]
const pipeNumber = number + config.pipeAddition;
if(!activeMonitor.emitterChannel[pipeNumber]){
activeMonitor.emitterChannel[pipeNumber] = new events.EventEmitter().setMaxListeners(0);
}
let frameToStreamAdded
switch(fields.stream_type){
case'mp4':
delete(activeMonitor.mp4frag[pipeNumber])
if(!activeMonitor.mp4frag[pipeNumber])activeMonitor.mp4frag[pipeNumber] = new Mp4Frag();
ffmpegProcess.stdio[pipeNumber].pipe(activeMonitor.mp4frag[pipeNumber],{ end: false })
break;
case'mjpeg':
frameToStreamAdded = function(d){
activeMonitor.emitterChannel[pipeNumber].emit('data',d)
}
break;
case'flv':
frameToStreamAdded = function(d){
if(!activeMonitor.firstStreamChunk[pipeNumber])activeMonitor.firstStreamChunk[pipeNumber] = d;
frameToStreamAdded = function(d){
activeMonitor.emitterChannel[pipeNumber].emit('data',d)
}
frameToStreamAdded(d)
}
break;
case'h264':
frameToStreamAdded = function(d){
activeMonitor.emitterChannel[pipeNumber].emit('data',d)
}
break;
}
if(frameToStreamAdded){
ffmpegProcess.stdio[pipeNumber].on('data',frameToStreamAdded)
}
}
function setActiveViewer(groupKey,monitorId,connectionId,isBeingAdded){
const viewerList = s.group[groupKey].activeMonitors[monitorId].watch;
if(isBeingAdded){
if(viewerList.indexOf(connectionId) > -1)viewerList.push(connectionId);
}else{
viewerList.splice(viewerList.indexOf(connectionId), 1)
}
const numberOfViewers = viewerList.length
s.tx({
f: 'viewer_count',
viewers: numberOfViewers,
ke: groupKey,
id: monitorId
},'MON_' + groupKey + monitorId)
return numberOfViewers;
}
function getActiveViewerCount(groupKey,monitorId){
const viewerList = s.group[groupKey].activeMonitors[monitorId].watch;
const numberOfViewers = viewerList.length
return numberOfViewers;
}
function setTimedActiveViewerForHttp(req){
const groupKey = req.params.ke
const connectionId = req.params.auth
const loggedInUser = s.group[groupKey].users[connectionId]
if(!loggedInUser){
const monitorId = req.params.id
const viewerList = s.group[groupKey].activeMonitors[monitorId].watch
const theViewer = viewerList[connectionId]
if(!theViewer){
setActiveViewer(groupKey,monitorId,connectionId,true)
}
clearTimeout(streamViewerCountTimeouts[req.originalUrl])
streamViewerCountTimeouts[req.originalUrl] = setTimeout(() => {
setActiveViewer(groupKey,monitorId,connectionId,false)
},5000)
}else{
s.debugLog(`User is Logged in, Don't add to viewer count`);
}
}
function attachMainProcessHandlers(e,fatalError){
s.group[e.ke].activeMonitors[e.id].spawn_exit = function(){
if(s.group[e.ke].activeMonitors[e.id].isStarted === true){
if(e.details.loglevel!=='quiet'){
s.userLog(e,{type:lang['Process Unexpected Exit'],msg:{msg:lang.unexpectedExitText,cmd:s.group[e.ke].activeMonitors[e.id].ffmpeg}});
}
fatalError(e,'Process Unexpected Exit');
scanForOrphanedVideos(e,{
forceCheck: true,
checkMax: 2
})
s.onMonitorUnexpectedExitExtensions.forEach(function(extender){
extender(Object.assign(s.group[e.ke].rawMonitorConfigurations[e.id],{}),e)
})
}
}
s.group[e.ke].activeMonitors[e.id].spawn.on('end',s.group[e.ke].activeMonitors[e.id].spawn_exit)
s.group[e.ke].activeMonitors[e.id].spawn.on('exit',s.group[e.ke].activeMonitors[e.id].spawn_exit)
s.group[e.ke].activeMonitors[e.id].spawn.on('error',function(er){
s.userLog(e,{type:'Spawn Error',msg:er});fatalError(e,'Spawn Error')
})
s.userLog(e,{type:lang['Process Started'],msg:{cmd:s.group[e.ke].activeMonitors[e.id].ffmpeg}})
// if(s.isWin === false){
// var strippedHost = s.stripAuthFromHost(e)
// var sendProcessCpuUsage = function(){
// s.getMonitorCpuUsage(e,function(percent){
// s.group[e.ke].activeMonitors[e.id].currentCpuUsage = percent
// s.tx({
// f: 'camera_cpu_usage',
// ke: e.ke,
// id: e.id,
// percent: percent
// },'MON_STREAM_'+e.ke+e.id)
// })
// }
// clearInterval(s.group[e.ke].activeMonitors[e.id].getMonitorCpuUsage)
// s.group[e.ke].activeMonitors[e.id].getMonitorCpuUsage = setInterval(function(){
// if(e.details.skip_ping !== '1'){
// connectionTester.test(strippedHost,e.port,2000,function(err,response){
// if(response.success){
// sendProcessCpuUsage()
// }else{
// launchMonitorProcesses(e)
// }
// })
// }else{
// sendProcessCpuUsage()
// }
// },1000 * 60)
// }
}
async function deleteMonitorData(groupKey,monitorId){
// deleteVideos
// deleteFileBinFiles
// deleteTimelapseFrames
async function deletePath(thePath){
try{
await fs.promises.stat(thePath)
await fs.promises.rm(thePath, {recursive: true})
}catch(err){
}
}
async function deleteFromTable(tableName){
await s.knexQueryPromise({
action: "delete",
table: tableName,
where: {
ke: groupKey,
mid: monitorId,
}
})
}
async function getSizeFromTable(tableName){
const response = await s.knexQueryPromise({
action: "select",
columns: "size",
table: tableName,
where: {
ke: groupKey,
mid: monitorId,
}
})
const rows = response.rows
let size = 0
for (let i = 0; i < rows.length; i++) {
const row = rows[i]
size += row.size
}
return size
}
async function adjustSpaceCounterForTableWithAddStorage(tableName,storageType){
// does normal videos and addStorage
const response = await s.knexQueryPromise({
action: "select",
columns: "ke,mid,details,size",
table: tableName || 'Videos',
where: {
ke: groupKey,
mid: monitorId,
}
})
const rows = response.rows
for (let i = 0; i < rows.length; i++) {
const video = rows[i]
const storageIndex = s.getVideoStorageIndex(video)
if(storageIndex){
s.setDiskUsedForGroupAddStorage(video.ke,{
size: -(video.size / 1048576),
storageIndex: storageIndex
},storageType)
}else{
s.setDiskUsedForGroup(video.ke,-(video.size / 1048576),storageType)
}
}
}
async function adjustSpaceCounter(tableName,storageType){
const amount = await getSizeFromTable(tableName)
s.setDiskUsedForGroup(groupKey,-amount,storageType)
}
const videosDir = s.dir.videos + `${groupKey}/${monitorId}`
const binDir = s.dir.fileBin + `${groupKey}/${monitorId}`
// videos and addStorage
await adjustSpaceCounterForTableWithAddStorage('Timelapse Frames','timelapeFrames')
await adjustSpaceCounterForTableWithAddStorage('Videos')
await deleteFromTable('Videos')
await deletePath(videosDir)
for (let i = 0; i < s.dir.addStorage.length; i++) {
const storage = s.dir.addStorage[i]
const addStorageDir = storage.path + groupKey + '/' + monitorId
await deletePath(addStorageDir)
await deletePath(addStorageDir + '_timelapse')
}
// timelapse frames
await adjustSpaceCounter('Timelapse Frames','timelapeFrames')
await deleteFromTable('Timelapse Frames')
await deletePath(videosDir + '_timelapse')
// fileBin
await adjustSpaceCounter('Files','fileBin')
await deleteFromTable('Files')
await deletePath(binDir)
}
return {
deleteMonitorData,
cameraDestroy: cameraDestroy,
createSnapshot: createSnapshot,
processKill: processKill,
addCredentialsToStreamLink: addCredentialsToStreamLink,
monitorConfigurationMigrator: monitorConfigurationMigrator,
spawnSubstreamProcess: spawnSubstreamProcess,
destroySubstreamProcess: destroySubstreamProcess,
attachStreamChannelHandlers: attachStreamChannelHandlers,
setActiveViewer: setActiveViewer,
getActiveViewerCount: getActiveViewerCount,
setTimedActiveViewerForHttp: setTimedActiveViewerForHttp,
attachMainProcessHandlers: attachMainProcessHandlers,
}
}

12
libs/notification.js
View File

@ -10,8 +10,18 @@ module.exports = function(s,config,lang){
d.screenshotBuffer = screenShot
}
}
require('./notifications/email.js')(s,config,lang,getSnapshot)
if(
config.mail &&
config.mail.auth &&
config.mail.auth.user !== 'your_email@gmail.com' &&
config.mail.auth.pass !== 'your_password_or_app_specific_password'
){
require('./notifications/email.js')(s,config,lang,getSnapshot)
}
require('./notifications/emailByUser.js')(s,config,lang,getSnapshot)
require('./notifications/discordBot.js')(s,config,lang,getSnapshot)
require('./notifications/telegram.js')(s,config,lang,getSnapshot)
require('./notifications/pushover.js')(s,config,lang,getSnapshot)
require('./notifications/webhook.js')(s,config,lang,getSnapshot)
require('./notifications/mqtt.js')(s,config,lang,getSnapshot)
}

69
libs/notifications/discordBot.js
View File

@ -48,14 +48,14 @@ module.exports = function(s,config,lang,getSnapshot){
}
}
const onEventTriggerBeforeFilterForDiscord = function(d,filter){
filter.discord = true
filter.discord = false
}
const onEventTriggerForDiscord = async (d,filter) => {
const monitorConfig = s.group[d.ke].rawMonitorConfigurations[d.id]
// d = event object
//discord bot
const isEnabled = monitorConfig.details.detector_discordbot === '1' || monitorConfig.details.notify_discord === '1'
if(filter.discord && s.group[d.ke].discordBot && isEnabled && !s.group[d.ke].activeMonitors[d.id].detector_discordbot){
const isEnabled = filter.discord || monitorConfig.details.detector_discordbot === '1' || monitorConfig.details.notify_discord === '1'
if(s.group[d.ke].discordBot && isEnabled && !s.group[d.ke].activeMonitors[d.id].detector_discordbot){
var detector_discordbot_timeout
if(!monitorConfig.details.detector_discordbot_timeout||monitorConfig.details.detector_discordbot_timeout===''){
detector_discordbot_timeout = 1000 * 60 * 10;
@ -66,6 +66,28 @@ module.exports = function(s,config,lang,getSnapshot){
clearTimeout(s.group[d.ke].activeMonitors[d.id].detector_discordbot);
s.group[d.ke].activeMonitors[d.id].detector_discordbot = null
},detector_discordbot_timeout)
await getSnapshot(d,monitorConfig)
if(d.screenshotBuffer){
sendMessage({
author: {
name: s.group[d.ke].rawMonitorConfigurations[d.id].name,
icon_url: config.iconURL
},
title: lang.Event+' - '+d.screenshotName,
description: lang.EventText1+' '+d.currentTimestamp,
fields: [],
timestamp: d.currentTime,
footer: {
icon_url: config.iconURL,
text: "Shinobi Systems"
}
},[
{
attachment: d.screenshotBuffer,
name: d.screenshotName+'.jpg'
}
],d.ke)
}
if(monitorConfig.details.detector_discordbot_send_video === '1'){
let videoPath = null
let videoName = null
@ -102,28 +124,6 @@ module.exports = function(s,config,lang,getSnapshot){
],d.ke)
}
}
await getSnapshot(d,monitorConfig)
if(d.screenshotBuffer){
sendMessage({
author: {
name: s.group[d.ke].rawMonitorConfigurations[d.id].name,
icon_url: config.iconURL
},
title: lang.Event+' - '+d.screenshotName,
description: lang.EventText1+' '+d.currentTimestamp,
fields: [],
timestamp: d.currentTime,
footer: {
icon_url: config.iconURL,
text: "Shinobi Systems"
}
},[
{
attachment: d.screenshotBuffer,
name: d.screenshotName+'.jpg'
}
],d.ke)
}
}
}
const onTwoFactorAuthCodeNotificationForDiscord = function(r){
@ -366,6 +366,25 @@ module.exports = function(s,config,lang,getSnapshot){
}
]
}
s.definitions["Event Filters"].blocks["Action for Selected"].info.push({
"name": "actions=discord",
"field": lang['Discord'],
"fieldType": "select",
"form-group-class": "actions-row",
"default": "",
"example": "1",
"possible": [
{
"name": lang['Original Choice'],
"value": "",
"selected": true
},
{
"name": lang.Yes,
"value": "1",
}
]
})
}catch(err){
console.log(err)
console.log('Could not start Discord bot, please run "npm install discord.js" inside the Shinobi folder.')

159
libs/notifications/email.js
View File

@ -29,22 +29,22 @@ module.exports = function(s,config,lang,getSnapshot){
]
},(err,r) => {
r = r[0]
var mailOptions = {
from: config.mail.from, // sender address
to: checkEmail(r.mail), // list of receivers
subject: lang.NoMotionEmailText1+' '+e.name+' ('+e.id+')', // Subject line
html: '<i>'+lang.NoMotionEmailText2+' ' + (e.details.detector_notrigger_timeout || 10) + ' '+lang.minutes+'.</i>',
var mailOptions = {
from: config.mail.from, // sender address
to: checkEmail(r.mail), // list of receivers
subject: lang.NoMotionEmailText1+' '+e.name+' ('+e.id+')', // Subject line
html: '<i>'+lang.NoMotionEmailText2+' ' + (e.details.detector_notrigger_timeout || 10) + ' '+lang.minutes+'.</i>',
}
mailOptions.html+='<div><b>'+lang['Monitor Name']+' </b> : '+e.name+'</div>'
mailOptions.html+='<div><b>'+lang['Monitor ID']+' </b> : '+e.id+'</div>'
sendMessage(mailOptions, (error, info) => {
if (error) {
s.systemLog('detector:notrigger:sendMail',error)
s.tx({f:'error',ff:'detector_notrigger_mail',id:e.id,ke:e.ke,error:error},'GRP_'+e.ke);
return ;
}
mailOptions.html+='<div><b>'+lang['Monitor Name']+' </b> : '+e.name+'</div>'
mailOptions.html+='<div><b>'+lang['Monitor ID']+' </b> : '+e.id+'</div>'
sendMessage(mailOptions, (error, info) => {
if (error) {
s.systemLog('detector:notrigger:sendMail',error)
s.tx({f:'error',ff:'detector_notrigger_mail',id:e.id,ke:e.ke,error:error},'GRP_'+e.ke);
return ;
}
s.tx({f:'detector_notrigger_mail',id:e.id,ke:e.ke,info:info},'GRP_'+e.ke);
})
s.tx({f:'detector_notrigger_mail',id:e.id,ke:e.ke,info:info},'GRP_'+e.ke);
})
})
}
}
@ -94,16 +94,11 @@ module.exports = function(s,config,lang,getSnapshot){
}
}
const onEventTriggerBeforeFilterForEmail = function(d,filter){
const monitorConfig = s.group[d.ke].rawMonitorConfigurations[d.id]
if(monitorConfig.details.detector_mail === '1'){
filter.mail = true
}else{
filter.mail = false
}
filter.mail = false
}
const onEventTriggerForEmail = async (d,filter) => {
const monitorConfig = s.group[d.ke].rawMonitorConfigurations[d.id]
if(filter.mail && config.mail && !s.group[d.ke].activeMonitors[d.id].detector_mail){
if((filter.mail || monitorConfig.details.detector_mail === '1') && config.mail && !s.group[d.ke].activeMonitors[d.id].detector_mail){
s.knexQuery({
action: "select",
columns: "mail",
@ -154,6 +149,13 @@ module.exports = function(s,config,lang,getSnapshot){
}
})
}
await getSnapshot(d,monitorConfig)
sendMail([
{
filename: d.screenshotName + '.jpg',
content: d.screenshotBuffer
}
])
if(monitorConfig.details.detector_mail_send_video === '1'){
let videoPath = null
let videoName = null
@ -193,13 +195,6 @@ module.exports = function(s,config,lang,getSnapshot){
})
}
}
await getSnapshot(d,monitorConfig)
sendMail([
{
filename: d.screenshotName + '.jpg',
content: d.screenshotBuffer
}
])
})
}
}
@ -245,6 +240,112 @@ module.exports = function(s,config,lang,getSnapshot){
s.onFilterEvent(onFilterEventForEmail)
s.onDetectorNoTriggerTimeout(onDetectorNoTriggerTimeoutForEmail)
s.onMonitorUnexpectedExit(onMonitorUnexpectedExitForEmail)
s.definitions['Account Settings'].blocks['2-Factor Authentication'].info.push( {
"name": "detail=factor_mail",
"field": `${lang.Email} (${lang['System Level']})`,
"description": "Send 2-Factor Authentication codes to the email address of the account.",
"default": "1",
"example": "",
"fieldType": "select",
"possible": [
{
"name": lang.No,
"value": "0"
},
{
"name": lang.Yes,
"value": "1"
}
]
});
s.definitions["Event Filters"].blocks["Action for Selected"].info.push( {
"name": "actions=mail",
"field": `${lang['Email on Trigger']} (${lang['System Level']})`,
"fieldType": "select",
"form-group-class": "actions-row",
"default": "",
"example": "1",
"possible": [
{
"name": lang['Original Choice'],
"value": "",
"selected": true
},
{
"name": lang.Yes,
"value": "1",
}
]
})
s.definitions['Monitor Settings'].blocks['Notifications'].info[0].info.push(
{
"name": "detail=notify_email",
"field": `${lang.Email} (${lang['System Level']})`,
"default": "0",
"fieldType": "select",
"possible": [
{
"name": lang.No,
"value": "0"
},
{
"name": lang.Yes,
"value": "1"
}
]
},
)
s.definitions['Monitor Settings'].blocks['Notifications'].info.push(
{
isFormGroupGroup: true,
name: `${lang.Email} (${lang['System Level']})`,
color: 'blue',
'section-class': 'h_det_input h_det_1',
info: [
{
"name": "detail=detector_mail",
"field": lang['Email on Trigger'],
"description": "Recieve an email of an image during a motion event to the master account for the camera group. You must setup SMTP details in conf.json.",
"default": "0",
"selector": "h_det_email",
"fieldType": "select",
"possible": [
{
"name": lang.No,
"value": "0"
},
{
"name": lang.Yes,
"value": "1"
}
]
},
{
"name": "detail=detector_mail_timeout",
"field": lang['Allow Next Email'],
"description": "The amount of time until a trigger is allowed to send another email with motion details and another image.",
"default": "10",
},
{
"name": "detail=detector_notrigger_mail",
"field": lang['No Trigger'],
"description": "If motion has not been detected after the timeout period you will recieve an email.",
"default": "0",
"fieldType": "select",
"possible": [
{
"name": lang.No,
"value": "0"
},
{
"name": lang.Yes,
"value": "1"
}
]
},
],
}
);
}catch(err){
console.log(err)
}

404
libs/notifications/emailByUser.js Normal file
View File

@ -0,0 +1,404 @@
var fs = require('fs');
const {
template,
checkEmail,
} = require("./emailUtils.js")
module.exports = function (s, config, lang, getSnapshot) {
const { getEventBasedRecordingUponCompletion } = require('../events/utils.js')(s, config, lang);
const nodeMailer = require('nodemailer');
try {
const sendMessage = async function (sendBody, files, groupKey) {
const transporter = s.group[groupKey].emailClient;
if (!transporter) {
s.userLog(
{ ke: groupKey, mid: '$USER' },
{
type: lang.NotifyErrorText,
msg: {
msg: lang.AppNotEnabledText,
app: lang.Email
},
}
);
return;
}
try {
const emailClientOptions = s.group[groupKey].emailClientOptions;
const appOptions = emailClientOptions.transport;
const sendTo = emailClientOptions.sendTo;
sendTo.forEach((reciepientAddress) => {
const sendData = {
from: `"${config.mailFromName || 'shinobi.video'}" <${appOptions.auth.user}>`,
to: reciepientAddress,
subject: sendBody.subject,
html: sendBody.html,
attachments: files || []
};
transporter.sendMail(sendData, function (err, result) {
if (err) {
throw err;
}
s.userLog(result);
s.debugLog(result);
});
})
console.log(sendBody)
} catch (err) {
s.debugLog(err)
s.userLog(
{ ke: groupKey, mid: '$USER' },
{ type: lang.NotifyErrorText, msg: err }
);
}
};
const loadAppForUser = function (user) {
const userDetails = s.parseJSON(user.details);
const optionsHost = userDetails.emailClient_host
const optionsUser = userDetails.emailClient_user
const optionsSendTo = userDetails.emailClient_sendTo || ''
if (
!s.group[user.ke].emailClient &&
userDetails.emailClient === '1' &&
optionsHost &&
optionsUser &&
optionsSendTo
){
const optionsPass = userDetails.emailClient_pass || ''
const optionsSecure = userDetails.emailClient_secure === '1' ? true : false
const optionsPort = isNaN(userDetails.emailClient_port) ? (optionsSecure ? 465 : 587) : parseInt(userDetails.emailClient_port)
const clientOptions = {
host: optionsHost,
port: optionsPort,
secure: optionsSecure,
auth: {
user: optionsUser,
pass: optionsPass
}
}
s.group[user.ke].emailClientOptions = {
transport: clientOptions,
sendTo: optionsSendTo.split(',').map((text) => {return text.trim()}),
}
s.group[user.ke].emailClient = nodeMailer.createTransport(clientOptions)
}
};
const unloadAppForUser = function (user) {
if (
s.group[user.ke].emailClient &&
s.group[user.ke].emailClient.close
) {
s.group[user.ke].emailClient.close();
}
delete s.group[user.ke].emailClient;
delete s.group[user.ke].emailClientOptions;
};
const onTwoFactorAuthCodeNotificationForApp = function (r) {
// r = user
if (r.details.factor_emailClient === '1') {
sendMessage({
subject: r.lang['2-Factor Authentication'],
html: template.createFramework({
title: r.lang['2-Factor Authentication'],
subtitle: r.lang['Enter this code to proceed'],
body: '<b style="font-size: 20pt;">'+s.factorAuth[r.ke][r.uid].key+'</b><br><br>'+r.lang.FactorAuthText1,
}),
},[],r.ke);
}
};
const onEventTriggerForApp = async (d, filter) => {
const monitorConfig = s.group[d.ke].rawMonitorConfigurations[d.id];
// d = event object
if (
s.group[d.ke].emailClient &&
(filter.emailClient || monitorConfig.details.notify_emailClient === '1') &&
!s.group[d.ke].activeMonitors[d.id].detector_emailClient
) {
var detector_emailClient_timeout;
if (!monitorConfig.details.detector_emailClient_timeout){
detector_emailClient_timeout = 1000 * 60 * 10
}else{
detector_emailClient_timeout = parseFloat(monitorConfig.details.detector_emailClient_timeout) * 1000 * 60
}
s.group[d.ke].activeMonitors[d.id].detector_emailClient = setTimeout(function () {
s.group[d.ke].activeMonitors[d.id].detector_emailClient = null;
}, detector_emailClient_timeout);
// lock passed
const sendMail = function(files){
const infoRows = []
Object.keys(d.details).forEach(function(key){
var value = d.details[key]
var text = value
if(value instanceof Object){
text = JSON.stringify(value,null,3)
}
infoRows.push(template.createRow({
title: key,
text: text
}))
})
sendMessage({
subject: lang.Event+' - '+d.screenshotName,
html: template.createFramework({
title: lang.EventText1 + ' ' + d.currentTimestamp,
subtitle: lang.Event,
body: infoRows.join(''),
}),
},files || [],d.ke)
}
await getSnapshot(d,monitorConfig)
sendMail([
{
filename: d.screenshotName + '.jpg',
content: d.screenshotBuffer
}
])
if(monitorConfig.details.detector_mail_send_video === '1'){
let videoPath = null
let videoName = null
const eventBasedRecording = await getEventBasedRecordingUponCompletion({
ke: d.ke,
mid: d.mid
})
if(eventBasedRecording.filePath){
videoPath = eventBasedRecording.filePath
videoName = eventBasedRecording.filename
}else{
const siftedVideoFileFromRam = await s.mergeDetectorBufferChunks(d)
videoPath = siftedVideoFileFromRam.filePath
videoName = siftedVideoFileFromRam.filename
}
if(videoPath){
fs.readFile(mergedFilepath,function(err,buffer){
if(buffer){
sendMail([
{
filename: videoName,
content: buffer
}
])
}
})
}
}
}
};
const onEventTriggerBeforeFilterForApp = function (d, filter) {
filter.emailClient = false;
};
const onDetectorNoTriggerTimeoutForApp = function (e) {
//e = monitor object
var currentTime = new Date();
if (e.details.detector_notrigger_emailClient === '1') {
var html =
'*' +
lang.NoMotionEmailText2 +
' ' +
(e.details.detector_notrigger_timeout || 10) +
' ' +
lang.minutes +
'.*\n';
html +=
'**' + lang['Monitor Name'] + '** : ' + e.name + '\n';
html += '**' + lang['Monitor ID'] + '** : ' + e.id + '\n';
html += currentTime;
sendMessage({
subject: lang['"No Motion" Detector'],
html: template.createFramework({
title: lang['"No Motion" Detector'],
subtitle: 'Shinobi Event',
body: html,
}),
},[],e.ke);
}
};
const onMonitorUnexpectedExitForApp = (monitorConfig) => {
if (
monitorConfig.details.notify_emailClient === '1' &&
monitorConfig.details.notify_onUnexpectedExit === '1'
){
const ffmpegCommand = s.group[monitorConfig.ke].activeMonitors[monitorConfig.mid].ffmpeg
const subject = lang['Process Unexpected Exit'] + ' : ' + monitorConfig.name
const currentTime = new Date();
sendMessage({
subject: subject,
html: template.createFramework({
title: subject,
subtitle: lang['Process Crashed for Monitor'],
body: ffmpegCommand,
footerText: currentTime
}),
},[],monitorConfig.ke);
}
};
s.loadGroupAppExtender(loadAppForUser);
s.unloadGroupAppExtender(unloadAppForUser);
s.onTwoFactorAuthCodeNotification(onTwoFactorAuthCodeNotificationForApp);
s.onEventTrigger(onEventTriggerForApp);
s.onEventTriggerBeforeFilter(onEventTriggerBeforeFilterForApp);
s.onDetectorNoTriggerTimeout(onDetectorNoTriggerTimeoutForApp);
s.onMonitorUnexpectedExit(onMonitorUnexpectedExitForApp);
s.definitions['Monitor Settings'].blocks['Notifications'].info[0].info.push({
name: 'detail=notify_emailClient',
field: lang.Email,
description: '',
default: '0',
example: '',
selector: 'h_det_emailClient',
fieldType: 'select',
possible: [
{
name: lang.No,
value: '0',
},
{
name: lang.Yes,
value: '1',
},
],
});
s.definitions['Monitor Settings'].blocks['Notifications'].info.push(
{
evaluation: "$user.details.use_emailClient !== '0'",
isFormGroupGroup: true,
name: lang.Email,
color: 'blue',
'section-class': 'h_det_emailClient_input h_det_emailClient_1',
info: [
{
name: 'detail=detector_emailClient_timeout',
field: `${lang['Allow Next Alert']} (${lang['on Event']})`,
default: '10',
},
],
}
);
s.definitions['Account Settings'].blocks['2-Factor Authentication'].info.push({
name: 'detail=factor_emailClient',
field: lang.Email,
default: '1',
example: '',
fieldType: 'select',
possible: [
{
name: lang.No,
value: '0',
},
{
name: lang.Yes,
value: '1',
},
],
});
s.definitions['Account Settings'].blocks['Email'] = {
evaluation: "$user.details.use_emailClient !== '0'",
name: lang.Email,
id: lang.Email,
color: 'blue',
info: [
{
name: 'detail=emailClient',
selector: 'u_emailClient',
field: lang.Enabled,
default: '0',
example: '',
fieldType: 'select',
possible: [
{
name: lang.No,
value: '0',
},
{
name: lang.Yes,
value: '1',
},
],
},
{
hidden: true,
field: lang.Host,
name: 'detail=emailClient_host',
example: 'smtp.gmail.com',
'form-group-class': 'u_emailClient_input u_emailClient_1',
},
{
hidden: true,
field: lang.Port,
name: 'detail=emailClient_port',
example: '587',
'form-group-class': 'u_emailClient_input u_emailClient_1',
},
{
hidden: true,
name: 'detail=emailClient_secure',
'form-group-class': 'u_emailClient_input u_emailClient_1',
field: lang.Secure,
default: '0',
example: '',
fieldType: 'select',
possible: [
{
name: lang.No,
value: '0',
},
{
name: lang.Yes,
value: '1',
},
],
},
{
hidden: true,
field: lang.Email,
name: 'detail=emailClient_user',
example: 'test@gmail.com',
'form-group-class': 'u_emailClient_input u_emailClient_1',
},
{
hidden: true,
field: lang.Password,
fieldType: 'password',
name: 'detail=emailClient_pass',
'form-group-class': 'u_emailClient_input u_emailClient_1',
},
{
hidden: true,
field: lang['Send to'],
name: 'detail=emailClient_sendTo',
example: 'testrecipient@gmail.com',
'form-group-class': 'u_emailClient_input u_emailClient_1',
},
],
};
s.definitions["Event Filters"].blocks["Action for Selected"].info.push({
"name": "actions=emailClient",
"field": lang['Email'],
"fieldType": "select",
"form-group-class": "actions-row",
"default": "",
"example": "1",
"possible": [
{
"name": lang['Original Choice'],
"value": "",
"selected": true
},
{
"name": lang.Yes,
"value": "1",
}
]
})
} catch (err) {
console.log(err);
console.log('Could not engage Email notifications.');
}
};

428
libs/notifications/mqtt.js Normal file
View File

@ -0,0 +1,428 @@
var fs = require("fs")
module.exports = function(s,config,lang,getSnapshot){
if(config.mqttClient === true){
console.log('Loading MQTT Outbound Connectivity...')
const mqtt = require('mqtt')
const {
getEventBasedRecordingUponCompletion,
} = require('../events/utils.js')(s,config,lang)
try{
function createMqttSubscription(options){
let mqttEndpoint = options.host
const username = options.username || ''
const password = options.password || ''
const subKey = options.subKey
const pubKey = options.pubKey
const groupKey = options.ke
const onData = options.onData || function(){}
function mqttUserLog(type,data){
s.userLog({
ke: groupKey,
mid: '$USER'
},{
type: type,
msg: data
})
}
if(mqttEndpoint.indexOf('://') === -1){
mqttEndpoint = `mqtt://${mqttEndpoint}`
}
mqttUserLog('Connecting... ' + mqttEndpoint)
const client = mqtt.connect(mqttEndpoint,{
clean: true,
username: username,
password: password,
clientId: `shinobi_${Math.random().toString(16).substr(2, 8)}`,
reconnectPeriod: 10000, // 10 seconds
});
client.on('reconnect', (e) => mqttUserLog(`MQTT Reconnected`))
client.on('disconnect', (e) => mqttUserLog(`MQTT Disconnected`))
client.on('offline', (e) => mqttUserLog(`MQTT Offline`))
client.on('error', (e) => mqttUserLog(`MQTT Error`,e))
client.on('connect', function () {
mqttUserLog('Connected! ' + mqttEndpoint)
client.subscribe(pubKey, function (err) {
if (err) {
s.debugLog(err)
s.userLog({
ke: groupKey,
mid: '$USER'
},{
type: lang['MQTT Error'],
msg: err
})
}else{
client.on('message', function (topic, message) {
const data = s.parseJSON(message.toString())
onData(data)
})
}
})
})
return client
}
function sendToMqttConnections(groupKey,eventName,addedArgs,checkMonitors){
try{
(s.group[groupKey].mqttOutbounderKeys || []).forEach(function(key){
const outBounder = s.group[groupKey].mqttOutbounders[key]
const theAction = outBounder.eventHandlers[eventName]
if(!theAction)return;
if(checkMonitors){
const monitorsToRead = outBounder.monitorsToRead
const firstArg = addedArgs[0]
const monitorId = firstArg.mid || firstArg.id
if(monitorsToRead.indexOf(monitorId) > -1 || monitorsToRead.indexOf('$all') > -1)theAction(...addedArgs);
}else{
theAction(...addedArgs)
}
})
}catch(err){
s.debugLog(err)
}
}
const sendMessage = async function(options,data){
const sendBody = s.stringJSON(data)
const groupKey = options.ke
const subId = options.subId
const publishTo = options.to
try{
s.group[groupKey].mqttOutbounders[subId].client.publish(publishTo,sendBody)
}catch(err){
s.debugLog('MQTT Error',err)
s.userLog({ke:groupKey,mid:'$USER'},{type:lang['MQTT Error'],msg:err})
}
}
const onEventTriggerBeforeFilter = function(d,filter){
filter.mqttout = false
}
const onDetectorNoTriggerTimeout = function(e){
if(e.details.detector_notrigger_mqttout === '1'){
const groupKey = e.ke
sendToMqttConnections(groupKey,'onDetectorNoTriggerTimeout',[e],true)
}
}
const onEventTrigger = (d,filter) => {
const monitorConfig = s.group[d.ke].rawMonitorConfigurations[d.id]
if((filter.mqttout || monitorConfig.details.notify_mqttout === '1') && !s.group[d.ke].activeMonitors[d.id].detector_mqttout){
var detector_mqttout_timeout
if(!monitorConfig.details.detector_mqttout_timeout||monitorConfig.details.detector_mqttout_timeout===''){
detector_mqttout_timeout = 1000 * 60 * 10;
}else{
detector_mqttout_timeout = parseFloat(monitorConfig.details.detector_mqttout_timeout) * 1000 * 60;
}
s.group[d.ke].activeMonitors[d.id].detector_mqttout = setTimeout(function(){
clearTimeout(s.group[d.ke].activeMonitors[d.id].detector_mqttout);
s.group[d.ke].activeMonitors[d.id].detector_mqttout = null
},detector_mqttout_timeout)
//
const groupKey = d.ke
sendToMqttConnections(groupKey,'onEventTrigger',[d,filter],true)
}
}
const onMonitorSave = (monitorConfig) => {
const groupKey = monitorConfig.ke
sendToMqttConnections(groupKey,'onMonitorSave',[monitorConfig],true)
}
const onMonitorStart = (monitorConfig) => {
const groupKey = monitorConfig.ke
sendToMqttConnections(groupKey,'onMonitorStart',[monitorConfig],true)
}
const onMonitorStop = (monitorConfig) => {
const groupKey = monitorConfig.ke
sendToMqttConnections(groupKey,'onMonitorStop',[monitorConfig],true)
}
const onMonitorDied = (monitorConfig) => {
const groupKey = monitorConfig.ke
sendToMqttConnections(groupKey,'onMonitorDied',[monitorConfig],true)
}
const onAccountSave = (activeGroup,userDetails,user) => {
const groupKey = user.ke
sendToMqttConnections(groupKey,'onAccountSave',[activeGroup,userDetails,user])
}
const onUserLog = (logEvent) => {
const groupKey = logEvent.ke
if(groupKey.indexOf('$') === -1){
sendToMqttConnections(groupKey,'onUserLog',[logEvent])
}else{
s.debugLog(`Failed sendToMqttConnections onUserLog : ${groupKey}`)
}
}
const onTwoFactorAuthCodeNotification = function(user){
const groupKey = user.ke
if(user.details.factor_mqttout === '1'){
sendToMqttConnections(groupKey,'onTwoFactorAuthCodeNotification',[user],true)
}
}
const loadMqttListBotForUser = function(user){
const groupKey = user.ke
const userDetails = s.parseJSON(user.details);
if(!s.group[groupKey].mqttOutbounders)s.group[groupKey].mqttOutbounders = {};
const mqttSubs = s.group[groupKey].mqttOutbounders
if(userDetails.mqttout === '1' && Object.keys(mqttSubs).length === 0){
const mqttClientList = userDetails.mqttout_list || []
mqttClientList.forEach(function(row,n){
try{
const mqttSubId = `${row.host} ${row.pubKey}`
const message = row.type || []
const eventsToAttachTo = row.msgFor || []
const monitorsToRead = row.monitors || []
mqttSubs[mqttSubId] = {
eventHandlers: {}
};
mqttSubs[mqttSubId].client = createMqttSubscription({
username: row.username,
password: row.password,
host: row.host,
pubKey: row.pubKey,
ke: groupKey,
});
const msgOptions = {
ke: groupKey,
subId: mqttSubId,
to: row.pubKey,
}
const titleLegend = {
onMonitorSave: lang['Monitor Edit'],
onMonitorStart: lang['Monitor Start'],
onMonitorStop: lang['Monitor Stop'],
onMonitorDied: lang['Monitor Died'],
onEventTrigger: lang['Event'],
onDetectorNoTriggerTimeout: lang['"No Motion" Detector'],
onAccountSave: lang['Account Save'],
onUserLog: lang['User Log'],
onTwoFactorAuthCodeNotification: lang['2-Factor Authentication'],
}
eventsToAttachTo.forEach(function(eventName){
let theAction = function(){}
switch(eventName){
case'onEventTrigger':
theAction = function(d,filter){
const eventObject = Object.assign({},d)
delete(eventObject.frame);
sendMessage(msgOptions,{
title: titleLegend[eventName],
name: eventName,
data: eventObject,
time: new Date(),
})
}
break;
case'onAccountSave':
theAction = function(activeGroup,userDetails,user){
sendMessage(msgOptions,{
title: titleLegend[eventName],
name: eventName,
data: {
mail: user.mail,
ke: user.ke,
},
time: new Date(),
})
}
break;
case'userLog':
theAction = function(logEvent){
sendMessage(msgOptions,{
title: titleLegend[eventName],
name: eventName,
data: logEvent,
time: new Date(),
})
}
break;
case'onTwoFactorAuthCodeNotification':
theAction = function(user){
sendMessage(msgOptions,{
title: titleLegend[eventName],
name: eventName,
data: {
code: s.factorAuth[user.ke][user.uid].key
},
time: new Date(),
})
}
break;
case'onMonitorSave':
case'onMonitorStart':
case'onMonitorStop':
case'onMonitorDied':
case'onDetectorNoTriggerTimeout':
theAction = function(monitorConfig){
//e = monitor object
sendMessage(msgOptions,{
title: titleLegend[eventName],
name: eventName,
data: {
name: monitorConfig.name,
monitorId: monitorConfig.mid || monitorConfig.id,
},
time: new Date(),
})
}
break;
}
mqttSubs[mqttSubId].eventHandlers[eventName] = theAction
})
mqttSubs[mqttSubId].monitorsToRead = monitorsToRead;
}catch(err){
s.debugLog(err)
// s.systemLog(err)
}
})
s.group[groupKey].mqttOutbounderKeys = Object.keys(s.group[groupKey].mqttOutbounders)
}else{
s.group[groupKey].mqttOutbounderKeys = []
}
}
const unloadMqttListBotForUser = function(user){
const groupKey = user.ke
const mqttSubs = s.group[groupKey].mqttOutbounders || {}
Object.keys(mqttSubs).forEach(function(mqttSubId){
try{
mqttSubs[mqttSubId].client.end()
}catch(err){
s.debugLog(err)
// s.userLog({
// ke: groupKey,
// mid: '$USER'
// },{
// type: lang['MQTT Error'],
// msg: err
// })
}
delete(mqttSubs[mqttSubId])
})
}
const onBeforeAccountSave = function(data){
data.d.mqttout_list = []
}
s.loadGroupAppExtender(loadMqttListBotForUser)
s.unloadGroupAppExtender(unloadMqttListBotForUser)
s.beforeAccountSave(onBeforeAccountSave)
s.onTwoFactorAuthCodeNotification(onTwoFactorAuthCodeNotification)
s.onEventTrigger(onEventTrigger)
s.onEventTriggerBeforeFilter(onEventTriggerBeforeFilter)
s.onDetectorNoTriggerTimeout(onDetectorNoTriggerTimeout)
s.onMonitorSave(onMonitorSave)
s.onMonitorStart(onMonitorStart)
s.onMonitorStop(onMonitorStop)
s.onMonitorDied(onMonitorDied)
s.onUserLog(onUserLog)
s.definitions["Monitor Settings"].blocks["Notifications"].info[0].info.push(
{
"name": "detail=notify_mqttout",
"field": lang['MQTT Outbound'],
"description": "",
"default": "0",
"example": "",
"selector": "h_det_mqttout",
"fieldType": "select",
"possible": [
{
"name": lang.No,
"value": "0"
},
{
"name": lang.Yes,
"value": "1"
}
]
},
)
s.definitions["Monitor Settings"].blocks["Notifications"].info.push({
"evaluation": "$user.details.use_mqttout !== '0'",
isFormGroupGroup: true,
"name": lang['MQTT Outbound'],
"color": "blue",
"section-class": "h_det_mqttout_input h_det_mqttout_1",
"info": [
{
"name": "detail=detector_mqttout_timeout",
"field": lang['Allow Next Alert'] + ` (${lang['on Event']})`,
"description": "",
"default": "10",
"example": "",
"possible": ""
},
]
})
s.definitions["Account Settings"].blocks["2-Factor Authentication"].info.push({
"name": "detail=factor_mqttout",
"field": lang['MQTT Outbound'],
"default": "1",
"example": "",
"fieldType": "select",
"possible": [
{
"name": lang.No,
"value": "0"
},
{
"name": lang.Yes,
"value": "1"
}
]
})
s.definitions["Account Settings"].blocks["MQTT Outbound"] = {
"evaluation": "$user.details.use_mqttout !== '0'",
"name": lang['MQTT Outbound'],
"color": "blue",
"info": [
{
"name": "detail=mqttout",
"selector":"u_mqttout",
"field": lang.Enabled,
"default": "0",
"example": "",
"fieldType": "select",
"possible": [
{
"name": lang.No,
"value": "0"
},
{
"name": lang.Yes,
"value": "1"
}
]
},
{
"fieldType": "btn",
"class": `btn-success mqtt-out-add-row`,
"btnContent": `<i class="fa fa-plus"></i> &nbsp; ${lang['Add']}`,
},
{
"id": "mqttout_list",
"fieldType": "div",
},
{
"fieldType": "script",
"src": "assets/js/bs5.mqttOut.js",
}
]
}
s.definitions["Event Filters"].blocks["Action for Selected"].info.push({
"name": "actions=mqttout",
"field": lang['MQTT Outbound'],
"fieldType": "select",
"form-group-class": "actions-row",
"default": "",
"example": "1",
"possible": [
{
"name": lang['Original Choice'],
"value": "",
"selected": true
},
{
"name": lang.Yes,
"value": "1",
}
]
})
}catch(err){
console.error(err)
console.log('Could not start MQTT Outbound Handling.')
}
}
}

30
libs/notifications/pushover.js
View File

@ -12,8 +12,8 @@ module.exports = function (s, config, lang, getSnapshot) {
s.userLog(
{ ke: groupKey, mid: '$USER' },
{
type: lang.NotifyErrorText,
msg: lang.DiscordNotEnabledText,
type: lang.PushoverNotifyErrorText,
msg: lang.PushoverNotEnabledText,
}
);
return;
@ -112,9 +112,8 @@ module.exports = function (s, config, lang, getSnapshot) {
s.group[d.ke].rawMonitorConfigurations[d.id];
// d = event object
if (
filter.pushover &&
s.group[d.ke].pushover &&
monitorConfig.details.notify_pushover === '1' &&
(filter.pushover || monitorConfig.details.notify_pushover === '1') &&
!s.group[d.ke].activeMonitors[d.id].detector_pushover
) {
var detector_pushover_timeout;
@ -163,7 +162,7 @@ module.exports = function (s, config, lang, getSnapshot) {
};
const onEventTriggerBeforeFilterForPushover = function (d, filter) {
filter.pushover = true;
filter.pushover = false;
};
const onDetectorNoTriggerTimeoutForPushover = function (e) {
@ -339,6 +338,25 @@ module.exports = function (s, config, lang, getSnapshot) {
},
],
};
s.definitions["Event Filters"].blocks["Action for Selected"].info.push({
"name": "actions=pushover",
"field": lang['Pushover'],
"fieldType": "select",
"form-group-class": "actions-row",
"default": "",
"example": "1",
"possible": [
{
"name": lang['Original Choice'],
"value": "",
"selected": true
},
{
"name": lang.Yes,
"value": "1",
}
]
})
} catch (err) {
console.log(err);
console.log(
@ -346,4 +364,4 @@ module.exports = function (s, config, lang, getSnapshot) {
);
}
}
};
};

66
libs/notifications/telegram.js
View File

@ -1,4 +1,11 @@
var fs = require("fs")
// function asyncSetTimeout(timeout){
// return new Promise((resolve,reject) => {
// setTimeout(() => {
// resolve()
// },timeout || 1000)
// })
// }
module.exports = function(s,config,lang,getSnapshot){
const {
getEventBasedRecordingUponCompletion,
@ -30,6 +37,7 @@ module.exports = function(s,config,lang,getSnapshot){
})
}
}catch(err){
s.debugLog('Telegram Error',err)
s.userLog({ke:groupKey,mid:'$USER'},{type:lang.NotifyErrorText,msg:err})
}
}else{
@ -43,13 +51,13 @@ module.exports = function(s,config,lang,getSnapshot){
}
}
const onEventTriggerBeforeFilterForTelegram = function(d,filter){
filter.telegram = true
filter.telegram = false
}
const onEventTriggerForTelegram = async (d,filter) => {
const monitorConfig = s.group[d.ke].rawMonitorConfigurations[d.id]
// d = event object
//telegram bot
if(filter.telegram && s.group[d.ke].telegramBot && monitorConfig.details.notify_telegram === '1' && !s.group[d.ke].activeMonitors[d.id].detector_telegrambot){
if(s.group[d.ke].telegramBot && (filter.telegram || monitorConfig.details.notify_telegram === '1') && !s.group[d.ke].activeMonitors[d.id].detector_telegrambot){
var detector_telegrambot_timeout
if(!monitorConfig.details.detector_telegrambot_timeout||monitorConfig.details.detector_telegrambot_timeout===''){
detector_telegrambot_timeout = 1000 * 60 * 10;
@ -60,7 +68,21 @@ module.exports = function(s,config,lang,getSnapshot){
clearTimeout(s.group[d.ke].activeMonitors[d.id].detector_telegrambot);
s.group[d.ke].activeMonitors[d.id].detector_telegrambot = null
},detector_telegrambot_timeout)
await getSnapshot(d,monitorConfig)
if(d.screenshotBuffer){
sendMessage({
title: lang.Event+' - '+d.screenshotName,
description: lang.EventText1+' '+d.currentTimestamp,
},[
{
type: 'photo',
attachment: d.screenshotBuffer,
name: d.screenshotName+'.jpg'
}
],d.ke)
}
if(monitorConfig.details.detector_telegrambot_send_video === '1'){
// await asyncSetTimeout(3000)
let videoPath = null
let videoName = null
const eventBasedRecording = await getEventBasedRecordingUponCompletion({
@ -87,19 +109,6 @@ module.exports = function(s,config,lang,getSnapshot){
],d.ke)
}
}
await getSnapshot(d,monitorConfig)
if(d.screenshotBuffer){
sendMessage({
title: lang.Event+' - '+d.screenshotName,
description: lang.EventText1+' '+d.currentTimestamp,
},[
{
type: 'photo',
attachment: d.screenshotBuffer,
name: d.screenshotName+'.jpg'
}
],d.ke)
}
}
}
const onTwoFactorAuthCodeNotificationForTelegram = function(r){
@ -267,7 +276,7 @@ module.exports = function(s,config,lang,getSnapshot){
"default": "",
"example": "",
"possible": ""
},
},
{
hidden: true,
"name": "detail=telegrambot_channel",
@ -281,8 +290,31 @@ module.exports = function(s,config,lang,getSnapshot){
}
]
}
s.definitions["Event Filters"].blocks["Action for Selected"].info.push({
"name": "actions=telegram",
"field": lang['Telegram'],
"fieldType": "select",
"form-group-class": "actions-row",
"default": "",
"example": "1",
"possible": [
{
"name": lang.Default,
"value": "",
"selected": true
},
{
"name": lang.No,
"value": "0",
},
{
"name": lang.Yes,
"value": "1",
}
]
})
}catch(err){
console.log(err)
console.error(err)
console.log('Could not start Telegram bot, please run "npm install node-telegram-bot-api" inside the Shinobi folder.')
}
}

289
libs/notifications/webhook.js Normal file
View File

@ -0,0 +1,289 @@
const fetch = require('node-fetch');
const FormData = require('form-data');
module.exports = function(s,config,lang,getSnapshot){
function replaceQueryStringValues(webhookEndpoint,data){
let newString = webhookEndpoint
.replace(/{{INNER_EVENT_TITLE}}/g,data.title)
.replace(/{{INNER_EVENT_INFO}}/g,s.stringJSON(data.info));
return newString;
}
const sendMessage = function(sendBody,files,groupKey){
let webhookEndpoint = s.group[groupKey].init.global_webhook_url;
if(!webhookEndpoint){
s.userLog({
ke: groupKey,
mid: '$USER'
},{
type: lang.NotifyErrorText,
infoType: 'global_webhook',
msg: lang['Invalid Settings']
})
return new Promise((resolve,reject) => {
resolve({
error: lang['Invalid Settings'],
ok: false,
})
})
}
const doPostMethod = s.group[groupKey].init.global_webhook_method === 'post';
// const includeSnapshot = s.group[groupKey].init.global_webhook_include_image === '1';
const webhookInfoData = {
info: sendBody,
files: [],
}
if(files){
const formData = new FormData();
files.forEach(async (file,n) => {
switch(file.type){
case'video':
// video cannot be sent this way unless POST
if(doPostMethod){
const fileName = file.name
formData.append(`file${n + 1}`, file.attachment, {
contentType: 'video/mp4',
name: fileName,
filename: fileName,
});
webhookInfoData.files.push(fileName)
}
break;
case'photo':
if(doPostMethod){
const fileName = file.name
formData.append(`file${n + 1}`, file.attachment, {
contentType: 'image/jpeg',
name: fileName,
filename: fileName,
});
webhookInfoData.files.push(fileName)
}else{
const base64StringofImage = file.attachment.toString('base64')
webhookInfoData.files.push(base64StringofImage)
}
break;
}
})
}else{
delete(webhookInfoData.files)
}
webhookEndpoint = replaceQueryStringValues(webhookEndpoint,{
title: sendBody.title,
info: webhookInfoData,
});
return new Promise((resolve,reject) => {
const response = {
ok: true,
}
fetch(webhookEndpoint,doPostMethod ? {
method: 'POST',
body: formData
} : undefined)
.then(res => res.text())
.then((text) => {
response.response = text;
resolve(response)
})
.catch((err) => {
response.ok = false;
response.error = err;
s.userLog({
ke: groupKey,
mid: '$USER'
},{
type: lang.NotifyErrorText,
infoType: 'global_webhook',
msg: err
})
resolve(response)
})
})
}
const onEventTriggerBeforeFilterForGlobalWebhook = function(d,filter){
filter.global_webhook = false
}
const onEventTriggerForGlobalWebhook = async (d,filter) => {
let filesSent = 0;
const monitorConfig = s.group[d.ke].rawMonitorConfigurations[d.id]
// d = event object
if((filter.global_webhook || monitorConfig.details.notify_global_webhook === '1') && !s.group[d.ke].activeMonitors[d.id].detector_global_webhook){
var detector_global_webhook_timeout
if(!monitorConfig.details.detector_global_webhook_timeout||monitorConfig.details.detector_global_webhook_timeout===''){
detector_global_webhook_timeout = 1000 * 60 * 10;
}else{
detector_global_webhook_timeout = parseFloat(monitorConfig.details.detector_global_webhook_timeout) * 1000 * 60;
}
s.group[d.ke].activeMonitors[d.id].detector_global_webhook = setTimeout(function(){
clearTimeout(s.group[d.ke].activeMonitors[d.id].detector_global_webhook);
s.group[d.ke].activeMonitors[d.id].detector_global_webhook = null
},detector_global_webhook_timeout)
await getSnapshot(d,monitorConfig)
if(d.screenshotBuffer){
sendMessage({
title: lang.Event+' - '+d.screenshotName,
description: lang.EventText1+' '+d.currentTimestamp,
},[
{
type: 'photo',
attachment: d.screenshotBuffer,
name: d.screenshotName+'.jpg'
}
],d.ke)
++filesSent;
}
if(filesSent === 0){
sendMessage({
title: lang.Event,
description: lang.EventText1+' '+d.currentTimestamp,
eventDetails: d.details
},[],d.ke)
}
}
}
const onTwoFactorAuthCodeNotificationForGlobalWebhook = function(r){
// r = user
if(r.details.factor_global_webhook === '1'){
sendMessage({
title: r.lang['Enter this code to proceed'],
description: '**'+s.factorAuth[r.ke][r.uid].key+'** '+r.lang.FactorAuthText1,
},[],r.ke)
}
}
// const onDetectorNoTriggerTimeoutForGlobalWebhook = function(e){
// //e = monitor object
// var currentTime = new Date()
// if(e.details.detector_notrigger_global_webhook === '1'){
// var html = '*'+lang.NoMotionEmailText2+' ' + (e.details.detector_notrigger_timeout || 10) + ' '+lang.minutes+'.*\n'
// html += '**' + lang['Monitor Name'] + '** : '+e.name + '\n'
// html += '**' + lang['Monitor ID'] + '** : '+e.id + '\n'
// html += currentTime
// sendMessage({
// title: lang['\"No Motion"\ Detector'],
// description: html,
// },[],e.ke)
// }
// }
const onMonitorUnexpectedExitForGlobalWebhook = (monitorConfig) => {
if(monitorConfig.details.notify_global_webhook === '1' && monitorConfig.details.notify_onUnexpectedExit === '1'){
const ffmpegCommand = s.group[monitorConfig.ke].activeMonitors[monitorConfig.mid].ffmpeg
const description = lang['Process Crashed for Monitor'] + '\n' + ffmpegCommand
const currentTime = new Date()
sendMessage({
title: lang['Process Unexpected Exit'] + ' : ' + monitorConfig.name,
description: description,
},[],monitorConfig.ke)
}
}
s.onTwoFactorAuthCodeNotification(onTwoFactorAuthCodeNotificationForGlobalWebhook)
s.onEventTrigger(onEventTriggerForGlobalWebhook)
s.onEventTriggerBeforeFilter(onEventTriggerBeforeFilterForGlobalWebhook)
// s.onDetectorNoTriggerTimeout(onDetectorNoTriggerTimeoutForGlobalWebhook)
s.onMonitorUnexpectedExit(onMonitorUnexpectedExitForGlobalWebhook)
s.definitions["Monitor Settings"].blocks["Notifications"].info[0].info.push(
{
"name": "detail=notify_global_webhook",
"field": lang.Webhook,
"description": "",
"default": "0",
"example": "",
"selector": "h_det_global_webhook",
"fieldType": "select",
"possible": [
{
"name": lang.No,
"value": "0"
},
{
"name": lang.Yes,
"value": "1"
}
]
},
)
s.definitions["Account Settings"].blocks["2-Factor Authentication"].info.push({
"name": "detail=factor_global_webhook",
"field": lang.Webhook,
"default": "1",
"example": "",
"fieldType": "select",
"possible": [
{
"name": lang.No,
"value": "0"
},
{
"name": lang.Yes,
"value": "1"
}
]
})
s.definitions["Account Settings"].blocks["Webhook"] = {
"evaluation": "$user.details.use_global_webhook !== '0'",
"name": lang.Webhook,
"color": "blue",
info: [
{
"name": "detail=global_webhook",
"selector":"u_global_webhook",
"field": lang.Enabled,
"default": "0",
"example": "",
"fieldType": "select",
"possible": [
{
"name": lang.No,
"value": "0"
},
{
"name": lang.Yes,
"value": "1"
}
]
},
{
hidden: true,
"name": "detail=global_webhook_url",
"placeholder": "http://your-webhook-point/onEvent/{{INNER_EVENT_TITLE}}?info={{INNER_EVENT_INFO}}",
"field": lang["Webhook URL"],
"form-group-class":"u_global_webhook_input u_global_webhook_1",
},
{
hidden: true,
"name": "detail=factor_global_webhook",
"field": lang["2-Factor Authentication"],
"form-group-class":"u_global_webhook_input u_global_webhook_1",
"default": "1",
"example": "",
"fieldType": "select",
"possible": [
{
"name": lang.No,
"value": "0"
},
{
"name": lang.Yes,
"value": "1"
}
]
}
]
}
s.definitions["Event Filters"].blocks["Action for Selected"].info.push({
"name": "actions=global_webhook",
"field": lang['Webhook'],
"fieldType": "select",
"form-group-class": "actions-row",
"default": "",
"example": "1",
"possible": [
{
"name": lang['Original Choice'],
"value": "",
"selected": true
},
{
"name": lang.Yes,
"value": "1",
}
]
})
}

74
libs/onvifDeviceManager.js
View File

@ -17,16 +17,36 @@ const {
} = require('./onvifDeviceManager/utils.js')
module.exports = function(s,config,lang,app,io){
async function getOnvifDevice(groupKey,monitorId){
const onvifDevice = s.group[groupKey].activeMonitors[monitorId].onvifConnection || (await s.createOnvifDevice({id: monitorId, ke: groupKey})).device
return onvifDevice
}
/**
* API : Get ONVIF Data from Camera
*/
app.get(config.webPaths.apiPrefix+':auth/onvifDeviceManager/:ke/:id',function (req,res){
s.auth(req.params,async (user) => {
const groupKey = req.params.ke
const monitorId = req.params.id
const {
monitorPermissions,
monitorRestrictions,
} = s.getMonitorsPermitted(user.details,monitorId)
const {
isRestricted,
isRestrictedApiKey,
apiKeyPermissions,
} = s.checkPermission(user)
if(
isRestrictedApiKey && apiKeyPermissions.get_monitors_disallowed ||
isRestricted && !monitorPermissions[`${monitorId}_monitors`]
){
s.closeJsonResponse(res,{ok: false, msg: lang['Not Authorized']});
return
}
const endData = {ok: true}
try{
const groupKey = req.params.ke
const monitorId = req.params.id
const onvifDevice = s.group[groupKey].activeMonitors[monitorId].onvifConnection
const onvifDevice = await getOnvifDevice(groupKey,monitorId)
const cameraInfo = await getUIFieldValues(onvifDevice)
endData.onvifData = cameraInfo
}catch(err){
@ -42,12 +62,30 @@ module.exports = function(s,config,lang,app,io){
*/
app.post(config.webPaths.apiPrefix+':auth/onvifDeviceManager/:ke/:id/save',function (req,res){
s.auth(req.params,async (user) => {
const groupKey = req.params.ke
const monitorId = req.params.id
const {
monitorPermissions,
monitorRestrictions,
} = s.getMonitorsPermitted(user.details,monitorId);
const {
isRestricted,
isRestrictedApiKey,
apiKeyPermissions,
} = s.checkPermission(user);
if(
isRestrictedApiKey && apiKeyPermissions.control_monitors_disallowed
){
s.closeJsonResponse(res,{
ok: false,
msg: lang['Not Authorized']
});
return
}
const endData = {ok: true}
const responses = {}
try{
const groupKey = req.params.ke
const monitorId = req.params.id
const onvifDevice = s.group[groupKey].activeMonitors[monitorId].onvifConnection
const onvifDevice = await getOnvifDevice(groupKey,monitorId)
const form = s.getPostData(req)
const videoToken = form.VideoConfiguration && form.VideoConfiguration.videoToken ? form.VideoConfiguration.videoToken : null
if(form.DateandTime){
@ -96,11 +134,29 @@ module.exports = function(s,config,lang,app,io){
*/
app.get(config.webPaths.apiPrefix+':auth/onvifDeviceManager/:ke/:id/reboot',function (req,res){
s.auth(req.params,async (user) => {
const groupKey = req.params.ke
const monitorId = req.params.id
const {
monitorPermissions,
monitorRestrictions,
} = s.getMonitorsPermitted(user.details,monitorId);
const {
isRestricted,
isRestrictedApiKey,
apiKeyPermissions,
} = s.checkPermission(user);
if(
isRestrictedApiKey && apiKeyPermissions.control_monitors_disallowed
){
s.closeJsonResponse(res,{
ok: false,
msg: lang['Not Authorized']
});
return
}
const endData = {ok: true}
try{
const groupKey = req.params.ke
const monitorId = req.params.id
const onvifDevice = s.group[groupKey].activeMonitors[monitorId].onvifConnection
const onvifDevice = await getOnvifDevice(groupKey,monitorId)
const cameraInfo = await rebootCamera(onvifDevice)
endData.onvifData = cameraInfo
}catch(err){

1
libs/onvifDeviceManager/utils.js
View File

@ -149,6 +149,7 @@ const getDeviceInformation = async (onvifDevice,options) => {
response.ok = false
response.error = err.stack.toString().toString()
s.debugLog(err)
s.debugLog(onvifDevice)
}
return response
}

10
libs/plugins/superUser.js
View File

@ -1,13 +1,12 @@
const fs = require('fs-extra');
const express = require('express')
const request = require('request')
const unzipper = require('unzipper')
const fetch = require("node-fetch")
const spawn = require('child_process').spawn
const {
Worker
} = require('worker_threads');
module.exports = async (s,config,lang,app,io,currentUse) => {
const { fetchDownloadAndWrite } = require('../basic/utils.js')(process.cwd(),config)
const {
currentPluginCpuUsage,
currentPluginGpuUsage,
@ -96,10 +95,9 @@ module.exports = async (s,config,lang,app,io,currentUse) => {
fs.mkdirSync(downloadPath)
return new Promise(async (resolve, reject) => {
fs.mkdir(downloadPath, () => {
request(downloadUrl).pipe(fs.createWriteStream(downloadPath + '.zip'))
.on('finish',() => {
zip = fs.createReadStream(downloadPath + '.zip')
.pipe(unzipper.Parse())
fetchDownloadAndWrite(downloadUrl,downloadPath + '.zip', 1)
.then((readStream) => {
readStream.pipe(unzipper.Parse())
.on('entry', async (file) => {
if(file.type === 'Directory'){
try{

14
libs/scanners.js
View File

@ -21,6 +21,20 @@ module.exports = function(s,config,lang,app,io){
*/
app.get(config.webPaths.apiPrefix+':auth/probe/:ke',function (req,res){
s.auth(req.params,function(user){
const {
isRestricted,
isRestrictedApiKey,
apiKeyPermissions,
} = s.checkPermission(user);
if(
isRestrictedApiKey && apiKeyPermissions.control_monitors_disallowed
){
s.closeJsonResponse(res,{
ok: false,
msg: lang['Not Authorized']
});
return
}
ffprobe(req.query.url,req.params.auth,(endData) => {
s.closeJsonResponse(res,endData)
})

26
libs/scanners/utils.js
View File

@ -1,7 +1,9 @@
var os = require('os');
const onvif = require("shinobi-onvif");
const {
addCredentialsToUrl,
stringContains,
getBuffer,
} = require('../common.js')
module.exports = (s,config,lang) => {
const ipRange = (start_ip, end_ip) => {
@ -119,6 +121,7 @@ module.exports = (s,config,lang) => {
ProfileToken : device.current_profile.token,
Protocol : 'RTSP'
})
var cameraResponse = {
ip: camera.ip,
port: camera.port,
@ -142,16 +145,17 @@ module.exports = (s,config,lang) => {
}
responseList.push(cameraResponse)
var imageSnap
if(cameraResponse.uri){
try{
imageSnap = (await s.getSnapshotFromOnvif({
username: onvifUsername,
password: onvifPassword,
uri: cameraResponse.uri,
})).toString('base64');
}catch(err){
s.debugLog(err)
}
try{
const snapUri = addCredentialsToUrl({
username: onvifUsername,
password: onvifPassword,
url: (await device.services.media.getSnapshotUri({
ProfileToken : device.current_profile.token,
})).data.GetSnapshotUriResponse.MediaUri.Uri,
});
imageSnap = (await getBuffer(snapUri)).toString('base64');
}catch(err){
s.debugLog(err)
}
if(foundCameraCallback)foundCameraCallback(Object.assign(cameraResponse,{f: 'onvif', snapShot: imageSnap}))
}catch(err){
@ -185,7 +189,7 @@ module.exports = (s,config,lang) => {
error: errorMessage
})
}
s.debugLog(err)
if(config.debugLogVerbose)s.debugLog(err);
}
})
return responseList

21
libs/scheduler.js
View File

@ -8,10 +8,11 @@ module.exports = function(s,config,lang,app,io){
columns: "*",
table: "Schedules"
},(err,rows) => {
rows.forEach(function(schedule){
rows && rows.forEach(function(schedule){
s.updateSchedule(schedule)
})
if(callback)callback()
if(callback && typeof callback === 'function')callback()
})
}
//update schedule
@ -193,9 +194,11 @@ module.exports = function(s,config,lang,app,io){
var endData = {
ok : false
}
if(user.details.sub){
endData.msg = user.lang['Not Permitted']
s.closeJsonResponse(res,endData)
const {
isSubAccount,
} = s.checkPermission(user)
if(isSubAccount){
s.closeJsonResponse(res,{ok: false, msg: lang['Not an Administrator Account']});
return
}
var whereQuery = [
@ -233,9 +236,11 @@ module.exports = function(s,config,lang,app,io){
var endData = {
ok : false
}
if(user.details.sub){
endData.msg = user.lang['Not Permitted']
s.closeJsonResponse(res,endData)
const {
isSubAccount,
} = s.checkPermission(user)
if(isSubAccount){
s.closeJsonResponse(res,{ok: false, msg: lang['Not an Administrator Account']});
return
}
switch(req.params.action){

48
libs/shinobiHub.js
View File

@ -1,7 +1,10 @@
var fs = require('fs')
var request = require('request')
const fs = require('fs')
const fetch = require('node-fetch')
module.exports = function(s,config,lang,app,io){
if(config.shinobiHubEndpoint === undefined){config.shinobiHubEndpoint = `https://hub.shinobi.video/`}else{config.shinobiHubEndpoint = s.checkCorrectPathEnding(config.shinobiHubEndpoint)}
const {
fetchWithAuthentication,
} = require('./basic/utils.js')(process.cwd(),config)
var stripUsernameAndPassword = function(string,username,password){
if(username)string = string.split(username).join('_USERNAME_')
if(password)string = string.split(password).join('_PASSWORD_')
@ -53,20 +56,27 @@ module.exports = function(s,config,lang,app,io){
json: JSON.stringify(monitorConfig)
})
if(validated.ok === true){
request.post({
url: `${config.shinobiHubEndpoint}api/${shinobiHubApiKey}/postConfiguration`,
form: {
"type": type,
"brand": monitorConfig.ke,
"name": monitorConfig.name,
"description": "Backup at " + (new Date()),
"json": validated.json,
"details": JSON.stringify({
// maybe ip address?
})
fetchWithAuthentication(
`${config.shinobiHubEndpoint}api/${shinobiHubApiKey}/postConfiguration`,
{
method: 'POST',
postData: {
"type": type,
"brand": monitorConfig.ke,
"name": monitorConfig.name,
"description": "Backup at " + (new Date()),
"json": validated.json,
"details": JSON.stringify({
// maybe ip address?
})
}
}
}, function(err,httpResponse,body){
callback(err,s.parseJSON(body) || {ok: false})
).then(res => res.text())
.then((data) => {
callback(null,s.parseJSON(data) || {ok: false})
})
.catch((err) => {
callback(err,{ok: false})
})
}else{
callback(new Error(validated.msg),{ok: false})
@ -78,7 +88,7 @@ module.exports = function(s,config,lang,app,io){
// s.userLog({ke:monitorConfig.ke,mid:'$USER'},{type:lang['Websocket Connected'],msg:{for:lang['Superuser'],id:cn.mail,ip:cn.ip}})
})
}
if(s.group[monitorConfig.ke] && s.group[monitorConfig.ke].init.shinobihub === '1'){
if(s.group[monitorConfig.ke] && s.group[monitorConfig.ke].init && s.group[monitorConfig.ke].init.shinobihub === '1'){
uploadConfiguration(s.group[monitorConfig.ke].init.shinobihub_key,'cam',monitorConfig,() => {
// s.userLog({ke:monitorConfig.ke,mid:'$USER'},{type:lang['Websocket Connected'],msg:{for:lang['Superuser'],id:cn.mail,ip:cn.ip}})
})
@ -100,7 +110,11 @@ module.exports = function(s,config,lang,app,io){
queryString.push(key + '=' + value)
})
}
request(`${config.shinobiHubEndpoint}api/${shinobiHubApiKey}/getConfiguration/${req.params.type}${req.params.id ? '/' + req.params.id : ''}${queryString.length > 0 ? '?' + queryString.join('&') : ''}`).pipe(res)
const configUrl = `${config.shinobiHubEndpoint}api/${shinobiHubApiKey}/getConfiguration/${req.params.type}${req.params.id ? '/' + req.params.id : ''}${queryString.length > 0 ? '?' + queryString.join('&') : ''}`
fetch(configUrl).then(actual => {
actual.headers.forEach((v, n) => res.setHeader(n, v));
actual.body.pipe(res);
})
}else{
s.closeJsonResponse(res,{
ok: false,

136
libs/socketio.js
View File

@ -7,12 +7,12 @@ const {
stringToSqlTime,
} = require('./common.js')
module.exports = function(s,config,lang,io){
const {
legacyFilterEvents
} = require('./events/utils.js')(s,config,lang)
const {
ptzControl
} = require('./control/ptz.js')(s,config,lang)
const {
legacyFilterEvents
} = require('./events/utils.js')(s,config,lang)
s.clientSocketConnection = {}
//send data to socket client function
s.tx = function(z,y,x){
@ -144,39 +144,6 @@ module.exports = function(s,config,lang,io){
////socket controller
io.on('connection', function (cn) {
var tx;
//unique h265 socket stream
cn.on('h265',function(d){
if(!s.group[d.ke]||!s.group[d.ke].activeMonitors||!s.group[d.ke].activeMonitors[d.id]){
cn.disconnect();return;
}
cn.ip=cn.request.connection.remoteAddress;
var toUTC = function(){
return new Date().toISOString();
}
var tx=function(z){cn.emit('data',z);}
const onFail = (msg) => {
tx({f:'stop_reconnect',msg:msg,token_used:d.auth,ke:d.ke});
cn.disconnect();
}
const onSuccess = (r) => {
r = r[0];
const Emitter = createStreamEmitter(d,cn)
validatedAndBindAuthenticationToSocketConnection(cn,d,true)
var contentWriter
cn.closeSocketVideoStream = function(){
Emitter.removeListener('data', contentWriter);
}
Emitter.on('data',contentWriter = function(base64){
tx(base64)
})
}
//check if auth key is user's temporary session key
if(s.group[d.ke]&&s.group[d.ke].users&&s.group[d.ke].users[d.auth]){
onSuccess(s.group[d.ke].users[d.auth]);
}else{
streamConnectionAuthentication(d,cn.ip).then(onSuccess).catch(onFail)
}
})
//unique Base64 socket stream
cn.on('Base64',function(d){
if(!s.group[d.ke]||!s.group[d.ke].activeMonitors||!s.group[d.ke].activeMonitors[d.id]){
@ -360,7 +327,7 @@ module.exports = function(s,config,lang,io){
if(s.group[d.ke].users[d.auth].details.get_server_log!=='0'){
cn.join('GRPLOG_'+d.ke)
}
s.group[d.ke].users[d.auth].lang=s.getLanguageFile(s.group[d.ke].users[d.auth].details.lang)
s.group[d.ke].users[d.auth].lang = s.getLanguageFile(s.group[d.ke].users[d.auth].details.lang)
s.userLog({ke:d.ke,mid:'$USER'},{type:s.group[d.ke].users[d.auth].lang['Websocket Connected'],msg:{mail:r.mail,id:d.uid,ip:cn.ip}})
if(!s.group[d.ke].activeMonitors){
s.group[d.ke].activeMonitors={}
@ -380,7 +347,7 @@ module.exports = function(s,config,lang,io){
}
})
try{
Object.values(s.group[d.ke].rawMonitorConfigurations).forEach((monitor) => {
(s.group[d.ke] ? Object.values(s.group[d.ke].rawMonitorConfigurations) : []).forEach((monitor) => {
s.cameraSendSnapshot({
mid: monitor.mid,
ke: monitor.ke,
@ -400,7 +367,8 @@ module.exports = function(s,config,lang,io){
return;
}
if((d.id||d.uid||d.mid)&&cn.ke){
try{
try{
d.callbackResponse = {ok: true}
switch(d.f){
case'monitorOrder':
if(d.monitorOrder && d.monitorOrder instanceof Object){
@ -443,8 +411,18 @@ module.exports = function(s,config,lang,io){
]
},(err,r) => {
if(r && r[0]){
const monitorListOrder = {}
const orderKeys = Object.keys(d.monitorListOrder)
details = JSON.parse(r[0].details)
details.monitorListOrder = d.monitorListOrder
orderKeys.forEach((orderKey) => {
const monitorIds = d.monitorListOrder[orderKey]
const uniqueList = {}
monitorIds.forEach((monitorId) => {
uniqueList[monitorId] = 1
})
monitorListOrder[orderKey] = Object.keys(uniqueList)
})
details.monitorListOrder = monitorListOrder
s.knexQuery({
action: "update",
table: "Users",
@ -653,12 +631,6 @@ module.exports = function(s,config,lang,io){
break;
}
break;
case'control':
ptzControl(d,function(msg){
s.userLog(d,msg)
tx({f:'control',response:msg})
})
break;
case'jpeg_off':
delete(cn.jpeg_on);
if(cn.monitorsCurrentlyWatching){
@ -692,6 +664,7 @@ module.exports = function(s,config,lang,io){
f: 'monitor_watch_on',
id: d.id,
ke: d.ke,
subStreamChannel: s.group[d.ke].activeMonitors[d.id].subStreamChannel,
warnings: s.group[d.ke].activeMonitors[d.id].warnings || []
})
s.camera('watch_on',d,cn)
@ -721,6 +694,18 @@ module.exports = function(s,config,lang,io){
break;
}
break;
default:
s.onOtherWebSocketMessagesExtensions.forEach(function(extender){
extender(d,cn,tx)
})
break;
}
if(d.callbackId && !d.hasResponded){
tx({
f:'callback',
callbackId: d.callbackId,
args: [d.callbackResponse]
})
}
}catch(er){
s.systemLog('ERROR CATCH 1',er)
@ -959,63 +944,6 @@ module.exports = function(s,config,lang,io){
}
})
//functions for retrieving cron announcements
cn.on('cron',function(d){
if(d.f==='init'){
if(config.cron.key){
if(config.cron.key===d.cronKey){
s.cron={started:moment(),last_run:moment(),id:cn.id};
}else{
cn.disconnect()
}
}else{
s.cron={started:moment(),last_run:moment(),id:cn.id};
}
}else{
if(s.cron&&cn.id===s.cron.id){
delete(d.cronKey)
switch(d.f){
case'filters':
legacyFilterEvents(d.ff,d)
break;
case's.tx':
s.tx(d.data,d.to)
break;
case's.deleteVideo':
s.deleteVideo(d.file)
break;
case's.deleteFileBinEntry':
s.deleteFileBinEntry(d.file)
break;
case's.setDiskUsedForGroup':
function doOnMain(){
s.setDiskUsedForGroup(d.ke,d.size,d.target || undefined)
}
if(d.videoRow){
let storageIndex = s.getVideoStorageIndex(d.videoRow);
if(storageIndex){
s.setDiskUsedForGroupAddStorage(d.ke,{
size: d.size,
storageIndex: storageIndex
})
}else{
doOnMain()
}
}else{
doOnMain()
}
break;
case'start':case'end':
d.mid='_cron';s.userLog(d,{type:'cron',msg:d.msg})
break;
default:
s.systemLog('CRON : ',d)
break;
}
}else{
cn.disconnect()
}
}
})
cn.on('disconnect', function () {
if(cn.socketVideoStream){
cn.closeSocketVideoStream()
@ -1050,7 +978,7 @@ module.exports = function(s,config,lang,io){
delete(s.clientSocketConnection[cn.id])
})
s.onWebSocketConnectionExtensions.forEach(function(extender){
extender(cn)
extender(cn,validatedAndBindAuthenticationToSocketConnection,createStreamEmitter)
})
});
}

60
libs/sql.js
View File

@ -162,6 +162,66 @@ module.exports = function(s,config){
}catch(err){
console.log(err)
}
try{
await s.databaseEngine.schema.table('Videos', table => {
table.string('objects')
})
}catch(err){
if(err && err.code !== 'ER_DUP_FIELDNAME'){
s.debugLog(err)
}
}
try{
await s.databaseEngine.schema.table('Videos', table => {
table.tinyint('archive',1).defaultTo(0)
})
}catch(err){
if(err && err.code !== 'ER_DUP_FIELDNAME'){
s.debugLog(err)
}
}
try{
await s.databaseEngine.schema.table('Monitors', table => {
table.string('saveDir',255).defaultTo('')
})
}catch(err){
if(err && err.code !== 'ER_DUP_FIELDNAME'){
s.debugLog(err)
}
}
try{
await s.databaseEngine.schema.table('Timelapse Frames', table => {
table.tinyint('archive',1).defaultTo(0)
table.string('saveDir',255).defaultTo('')
})
}catch(err){
if(err && err.code !== 'ER_DUP_FIELDNAME'){
s.debugLog(err)
}
}
try{
await s.databaseEngine.schema.table('Events', table => {
table.tinyint('archive',1).defaultTo(0)
})
}catch(err){
if(err && err.code !== 'ER_DUP_FIELDNAME'){
s.debugLog(err)
}
}
try{
s.databaseEngine.schema.table('Files', table => {
table.tinyint('archive',1).defaultTo(0)
}).then(() => {
console.log(`archive added to Files table`)
}).catch((err) => {
if(err && err.code !== 'ER_DUP_FIELDNAME'){
console.log('error')
console.log(err)
}
})
}catch(err){
console.log(err)
}
delete(s.preQueries)
}
}

22
libs/startup.js
View File

@ -11,6 +11,9 @@ module.exports = function(s,config,lang,io){
const {
checkSubscription
} = require('./basic/utils.js')(process.cwd(),config)
const {
checkForStaticUsers
} = require('./user/startup.js')(s,config,lang,io)
return new Promise((resolve, reject) => {
var checkedAdminUsers = {}
console.log('FFmpeg version : '+s.ffmpegVersion)
@ -84,7 +87,7 @@ module.exports = function(s,config,lang,io){
status: 'Stopped',
code: 5
});
var monObj = Object.assign(monitor,{id : monitor.mid})
const monObj = Object.assign({},monitor,{id : monitor.mid})
s.camera(monitor.mode,monObj)
checkAnother()
},1000)
@ -262,11 +265,15 @@ module.exports = function(s,config,lang,io){
},function(err,frames) {
if(frames && frames[0]){
frames.forEach(function(frame){
var storageType = JSON.parse(frame.details).type
if(!storageType)storageType = 's3'
var frameSize = frame.size / 1048576
user.cloudDiskUse[storageType].usedSpace += frameSize
user.cloudDiskUse[storageType].usedSpaceTimelapseFrames += frameSize
try{
var storageType = JSON.parse(frame.details).type
if(!storageType)storageType = 's3'
var frameSize = frame.size / 1048576
user.cloudDiskUse[storageType].usedSpace += frameSize
user.cloudDiskUse[storageType].usedSpaceTimelapseFrames += frameSize
}catch(err){
s.debugLog(err)
}
})
}
callback()
@ -409,7 +416,8 @@ module.exports = function(s,config,lang,io){
s.databaseEngine = require('knex')(s.databaseOptions)
//run prerequsite queries
s.preQueries()
setTimeout(() => {
setTimeout(async () => {
await checkForStaticUsers()
//check for subscription
checkSubscription(config.subscriptionId,function(hasSubcribed){
config.userHasSubscribed = hasSubcribed

575
libs/timelapse.js
View File

@ -1,7 +1,25 @@
var fs = require('fs')
var moment = require('moment')
var express = require('express')
const fs = require('fs')
const moment = require('moment')
const express = require('express')
const exec = require('child_process').exec;
const spawn = require('child_process').spawn;
const events = require('events');
module.exports = function(s,config,lang,app,io){
const {
sendTimelapseFrameToMasterNode,
} = require('./childNode/childUtils.js')(s,config,lang)
const {
splitForFFPMEG,
} = require('./ffmpeg/utils.js')(s,config,lang)
const {
getFileDirectory,
} = require('./basic/utils.js')(process.cwd(),config)
const {
processKill,
} = require('./monitor/utils.js')(s,config,lang)
const {
stitchMp4Files,
} = require('./video/utils.js')(s,config,lang)
const timelapseFramesCache = {}
const timelapseFramesCacheTimeouts = {}
s.getTimelapseFrameDirectory = function(e){
@ -22,8 +40,8 @@ module.exports = function(s,config,lang,app,io){
if(e.details && e.details.dir && e.details.dir !== ''){
details.dir = e.details.dir
}
var timeNow = eventTime || new Date()
var queryInfo = {
const timeNow = eventTime || new Date()
const queryInfo = {
ke: e.ke,
mid: e.id,
details: s.s(details),
@ -32,45 +50,16 @@ module.exports = function(s,config,lang,app,io){
time: timeNow
}
if(config.childNodes.enabled === true && config.childNodes.mode === 'child' && config.childNodes.host){
var currentDate = s.formattedTime(queryInfo.time,'YYYY-MM-DD')
s.cx({
f: 'open_timelapse_file_transfer',
var currentDate = s.formattedTime(timeNow,'YYYY-MM-DD')
const childNodeData = {
ke: e.ke,
mid: e.id,
d: s.group[e.ke].rawMonitorConfigurations[e.id],
time: currentDate,
filename: filename,
currentDate: currentDate,
queryInfo: queryInfo
})
var formattedTime = s.timeObject(timeNow).format()
fs.createReadStream(filePath,{ highWaterMark: 500 })
.on('data',function(data){
s.cx({
f: 'created_timelapse_file_chunk',
ke: e.ke,
mid: e.id,
time: formattedTime,
filesize: e.filesize,
chunk: data,
d: s.group[e.ke].rawMonitorConfigurations[e.id],
filename: filename,
currentDate: currentDate,
queryInfo: queryInfo
})
})
.on('close',function(){
s.cx({
f: 'created_timelapse_file',
ke: e.ke,
mid: e.id,
time: formattedTime,
filesize: e.filesize,
d: s.group[e.ke].rawMonitorConfigurations[e.id],
filename: filename,
currentDate: currentDate,
queryInfo: queryInfo
})
})
}
sendTimelapseFrameToMasterNode(filePath,childNodeData)
}else{
s.insertTimelapseFrameDatabaseRow(e,queryInfo,filePath)
}
@ -159,8 +148,14 @@ module.exports = function(s,config,lang,app,io){
table: "Timelapse Frames",
where: frameSelector,
limit: 1
},function(){
},async function(){
s.setDiskUsedForGroup(e.ke,-(r.size / 1048576),'timelapeFrames')
s.file('delete',e.fileLocation)
const fileDirectory = getFileDirectory(folderPath);
const folderIsEmpty = (await fs.promises.readdir(folderPath)).filter(file => file.indexOf('.jpg') > -1).length === 0;
if(folderIsEmpty){
await fs.rmdir(folderPath, { recursive: true })
}
})
}else{
// console.log('Delete Failed',e)
@ -168,6 +163,261 @@ module.exports = function(s,config,lang,app,io){
}
})
}
function splitArrayIntoMultiple(bigarray,size){
size = size || 80;
var arrayOfArrays = [];
for (var i=0; i<bigarray.length; i+=size) {
arrayOfArrays.push(bigarray.slice(i,i+size));
}
return arrayOfArrays
}
async function createTemporaryInputFile(frames,concatListFile){
const concatFiles = []
const fileList = []
frames.forEach(function(frame,frameNumber){
var selectedDate = frame.filename.split('T')[0]
var fileLocationMid = `${frame.ke}/${frame.mid}_timelapse/${selectedDate}/`
frame.details = s.parseJSON(frame.details)
var fileLocation
if(frame.details.dir){
fileLocation = `${s.checkCorrectPathEnding(frame.details.dir)}`
}else{
fileLocation = `${s.dir.videos}`
}
fileLocation = `${fileLocation}${fileLocationMid}${frame.filename}`
try{
fs.statSync(fileLocation)
concatFiles.push(`file '${fileLocation}'`)
fileList.push(`${fileLocation}`)
}catch(err){
s.debugLog(`Failed to read frame for Timelapse build`)
}
})
await fs.promises.writeFile(concatListFile,concatFiles.join('\n'))
return fileList
}
async function createTemporaryInputFileForStitched(videosPathsList,concatListFile){
const concatFiles = []
const fileList = []
videosPathsList.forEach(function(videoPath){
try{
fs.statSync(videoPath)
concatFiles.push(`file '${videoPath}'`)
fileList.push(`${videoPath}`)
}catch(err){
s.debugLog(`Failed to read segment for Timelapse build`)
}
})
s.debugLog(concatFiles)
await fs.promises.writeFile(concatListFile,concatFiles.join('\n'))
return fileList
}
function buildVideoSegmentFromFrames(options){
return new Promise((resolve,reject) => {
const frames = options.frames
const ke = frames[0].ke
const mid = frames[0].mid
const concatListFile = options.listFile
createTemporaryInputFile(frames,concatListFile).then((framesAccepted) => {
var completionTimeout
const framesPerSecond = options.fps
const finalMp4OutputLocation = options.output
const onPercentChange = options.onPercentChange
const numberOfFrames = framesAccepted.length
const commandString = `-y -threads 1 -re -f concat -safe 0 -r ${framesPerSecond} -i "${concatListFile}" -q:v 1 -c:v libx264 -preset ultrafast -r ${framesPerSecond} "${finalMp4OutputLocation}"`
s.debugLog("ffmpeg",commandString)
const videoBuildProcess = spawn(config.ffmpegDir,splitForFFPMEG(commandString))
videoBuildProcess.stdout.on('data',function(data){
s.debugLog('stdout',finalMp4OutputLocation,data.toString())
})
videoBuildProcess.stderr.on('data',function(data){
const text = data.toString()
if(text.startsWith('frame=')){
const currentFrame = parseInt(text.split(/(\s+)/)[2])
const percent = (currentFrame / numberOfFrames * 100).toFixed(1)
onPercentChange(percent,currentFrame)
}
clearTimeout(completionTimeout)
completionTimeout = setTimeout(function(){
s.debugLog('videoBuildProcess completionTimeout',finalMp4OutputLocation)
processKill(videoBuildProcess)
},20000)
})
videoBuildProcess.on('exit',async function(data){
clearTimeout(completionTimeout)
resolve()
await fs.promises.unlink(concatListFile)
})
})
})
}
async function chunkFramesAndBuildMultipleVideosThenSticth(options){
// a single video with too many frames makes the video unplayable, this is the fix.
const frames = options.frames
const ke = frames[0].ke
const mid = frames[0].mid
const finalFileName = options.finalFileName
const concatListFile = options.listFile
const framesPerSecond = options.fps
const finalMp4OutputLocation = options.output
const onPercentChange = options.onPercentChange
const frameChunks = splitArrayIntoMultiple(frames,80)
const numberOfSets = frameChunks.length
const filePathsList = []
for (let i = 0; i < numberOfSets; i++) {
var frameSet = frameChunks[i]
var numberOfFrames = frameSet.length
var segmentFileOutput = `${s.dir.streams}${ke}/${mid}/${s.gid(10)}.mp4`
filePathsList.push(segmentFileOutput)
await buildVideoSegmentFromFrames({
frames: frameSet,
listFile: `${concatListFile}${i}`,
fps: framesPerSecond,
output: segmentFileOutput,
onPercentChange: (percent,currentFrame) => {
const overallPercent = ((percent / numberOfSets) + (i * (100 / numberOfSets))).toFixed(1);
s.tx({
f: 'timelapse_build_percent',
ke: ke,
mid: mid,
name: finalFileName,
percent: overallPercent,
},'GRP_'+ke);
if(percent == 100){
s.debugLog('videoBuildProcess 100%',finalMp4OutputLocation)
}
s.debugLog(`Piece ${i}`,`${currentFrame} / ${numberOfFrames}`,`${percent}%`)
},
})
}
s.debugLog('videoBuildProcess Stitching...',finalMp4OutputLocation)
await createTemporaryInputFileForStitched(filePathsList,concatListFile)
await stitchMp4Files({
listFile: concatListFile,
output: finalMp4OutputLocation,
})
await fs.promises.rm(concatListFile)
for (let i = 0; i < filePathsList.length; i++) {
var segmentFileOutput = filePathsList[i]
await fs.promises.rm(segmentFileOutput)
}
s.debugLog('videoBuildProcess Stitching Complete!',finalMp4OutputLocation)
}
async function createVideoFromTimelapse(timelapseFrames,framesPerSecond){
s.debugLog("Building Timelapse Frames Video",timelapseFrames.length)
framesPerSecond = !isNaN(framesPerSecond) ? framesPerSecond : parseInt(framesPerSecond) || 2
const frames = timelapseFrames.reverse()
const numberOfFrames = timelapseFrames.length
const ke = frames[0].ke
const mid = frames[0].mid
const activeMonitor = s.group[ke].activeMonitors[mid]
const finalFileName = `${s.md5(JSON.stringify(frames))}-${framesPerSecond}fps.mp4`
const finalMp4OutputLocation = `${s.dir.fileBin}${ke}/${mid}/${finalFileName}`
const finalFileAlreadyExist = fs.existsSync(finalMp4OutputLocation)
const concatListFile = `${s.dir.streams}${ke}/${mid}/mergeJpegs_${finalFileName}.txt`
const response = {
ok: false,
ke: ke,
mid: mid,
name: finalFileName,
}
s.debugLog("activeMonitor.buildingTimelapseVideo",!!activeMonitor.buildingTimelapseVideo)
if(activeMonitor.buildingTimelapseVideo){
s.debugLog("Timelapse Frames Video Building Already",finalMp4OutputLocation)
return activeMonitor.buildingTimelapseVideo
}
s.debugLog("finalFileAlreadyExist",finalFileAlreadyExist)
if(finalFileAlreadyExist){
s.debugLog("Timelapse Frames Video finalFileAlreadyExist",finalMp4OutputLocation)
response.fileExists = true
response.msg = lang['Already exists']
return response
}
if(frames.length < framesPerSecond){
response.msg = lang.notEnoughFramesText1
return response
}
activeMonitor.buildingTimelapseVideo = response
chunkFramesAndBuildMultipleVideosThenSticth({
frames: frames,
listFile: concatListFile,
fps: framesPerSecond,
output: finalMp4OutputLocation,
finalFileName: finalFileName
}).then(async () => {
// videoBuildProcess exit
s.debugLog('videoBuildProcess exit',finalMp4OutputLocation)
const timeNow = new Date()
const fileStats = await fs.promises.stat(finalMp4OutputLocation)
const details = {
start: frames[0].time,
end: frames[frames.length - 1].time,
}
s.knexQuery({
action: "insert",
table: "Files",
insert: {
ke: ke,
mid: mid,
details: s.s(details),
name: finalFileName,
size: fileStats.size,
time: timeNow,
}
})
s.setDiskUsedForGroup(ke,fileStats.size / 1048576,'fileBin')
s.purgeDiskForGroup(ke)
s.tx({
f: 'fileBin_item_added',
ke: ke,
mid: mid,
details: details,
name: finalFileName,
size: fileStats.size,
time: timeNow,
timelapseVideo: true,
},'GRP_'+ke);
delete(activeMonitor.buildingTimelapseVideo)
s.debugLog("Timelapse Frames Video Done!",finalMp4OutputLocation)
})
response.ok = true
response.msg = `${lang.Building}... ${lang['Please Wait...']}`
return response
}
function initiateTimelapseVideoBuild({
groupKey,
monitorId,
framesPerSecond,
framesPosted,
}){
return new Promise((resolve,reject) => {
let response = {ok: false}
if(!monitorId){
response.msg = lang['No Monitor Found, Ignoring Request']
resolve(response)
}else{
const frames = []
var n = 0
framesPosted.forEach((frame) => {
var firstParam = [['ke','=',groupKey],['mid','=',monitorId],['filename','=',frame.filename]]
if(n !== 0)firstParam[0] = (['or']).concat(firstParam[0])
frames.push(...firstParam)
++n
})
s.knexQuery({
action: "select",
columns: "*",
table: "Timelapse Frames",
where: frames
},async (err,r) => {
if(r.length > 0){
response = await createVideoFromTimelapse(r.reverse(),framesPerSecond)
}
resolve(response)
})
}
})
}
// Web Paths
// // // // //
/**
@ -177,61 +427,57 @@ module.exports = function(s,config,lang,app,io){
config.webPaths.apiPrefix+':auth/timelapse/:ke',
config.webPaths.apiPrefix+':auth/timelapse/:ke/:id',
config.webPaths.apiPrefix+':auth/timelapse/:ke/:id/:date',
config.webPaths.apiPrefix+':auth/cloudTimelapse/:ke',
config.webPaths.apiPrefix+':auth/cloudTimelapse/:ke/:id',
config.webPaths.apiPrefix+':auth/cloudTimelapse/:ke/:id/:date',
], function (req,res){
res.setHeader('Content-Type', 'application/json');
s.auth(req.params,function(user){
var hasRestrictions = user.details.sub && user.details.allmonitors !== '1'
const monitorId = req.params.id
const groupKey = req.params.ke
const {
monitorPermissions,
monitorRestrictions,
} = s.getMonitorsPermitted(user.details,monitorId)
const {
isRestricted,
isRestrictedApiKey,
apiKeyPermissions,
} = s.checkPermission(user);
if(
user.permissions.watch_videos==="0" ||
hasRestrictions &&
(
!user.details.video_view ||
user.details.video_view.indexOf(req.params.id) === -1
isRestrictedApiKey && apiKeyPermissions.watch_videos_disallowed ||
isRestricted && (
monitorId && !monitorPermissions[`${monitorId}_video_view`] ||
monitorRestrictions.length === 0
)
){
s.closeJsonResponse(res,[])
s.closeJsonResponse(res,{ok: false, msg: lang['Not Authorized'], frames: []});
return
}
const monitorRestrictions = s.getMonitorRestrictions(user.details,req.params.id)
var origURL = req.originalUrl.split('/')
var videoParam = origURL[origURL.indexOf(req.params.auth) + 1]
var dataSet = 'Timelapse Frames'
switch(videoParam){
case'cloudTimelapse':
dataSet = 'Cloud Timelapse Frames'
break;
}
s.getDatabaseRows({
monitorRestrictions: monitorRestrictions,
table: 'Timelapse Frames',
table: dataSet,
groupKey: req.params.ke,
date: req.query.date,
startDate: req.query.start,
endDate: req.query.end,
startOperator: req.query.startOperator,
endOperator: req.query.endOperator,
noLimit: req.query.noLimit,
limit: req.query.limit,
archived: req.query.archived,
rowType: 'frames',
endIsStartTo: true
},(response) => {
var isMp4Call = !!(req.query.mp4 || (req.params.date && typeof req.params.date === 'string' && req.params.date.indexOf('.') > -1))
if(isMp4Call && response.frames[0]){
s.createVideoFromTimelapse(response.frames,req.query.fps,function(response){
if(response.fileExists){
if(req.query.download){
res.setHeader('Content-Type', 'video/mp4')
s.streamMp4FileOverHttp(response.fileLocation,req,res)
}else{
s.closeJsonResponse(res,{
ok : response.ok,
fileExists : response.fileExists,
msg : response.msg,
})
}
}else{
s.closeJsonResponse(res,{
ok : response.ok,
fileExists : response.fileExists,
msg : response.msg,
})
}
})
}else{
s.closeJsonResponse(res,response.frames)
}
s.closeJsonResponse(res,response.frames)
})
},res,req);
});
@ -244,54 +490,34 @@ module.exports = function(s,config,lang,app,io){
], function (req,res){
res.setHeader('Content-Type', 'application/json');
s.auth(req.params,function(user){
var hasRestrictions = user.details.sub && user.details.allmonitors !== '1'
const groupKey = req.params.ke
const monitorId = req.params.id
const actionParameter = !!req.params.action
const {
monitorPermissions,
monitorRestrictions,
} = s.getMonitorsPermitted(user.details,monitorId)
const {
isRestricted,
isRestrictedApiKey,
apiKeyPermissions,
} = s.checkPermission(user)
if(
user.permissions.watch_videos==="0" ||
hasRestrictions &&
(
!user.details.video_view ||
user.details.video_view.indexOf(req.params.id) === -1
)
isRestrictedApiKey && apiKeyPermissions.delete_videos_disallowed ||
isRestricted && !monitorPermissions[`${monitorId}_video_delete`]
){
s.closeJsonResponse(res,[])
return
}
const monitorRestrictions = s.getMonitorRestrictions(user.details,req.params.id)
if(monitorRestrictions.length === 0){
s.closeJsonResponse(res,{
ok: false
})
s.closeJsonResponse(res,{ok: false, msg: lang['Not Authorized']});
return
}
const framesPerSecond = s.getPostData(req, 'fps')
const framesPosted = s.getPostData(req, 'frames', true) || []
const frames = []
var n = 0
framesPosted.forEach((frame) => {
var firstParam = ['ke','=',req.params.ke]
if(n !== 0)firstParam = (['or']).concat(firstParam)
frames.push(firstParam,['mid','=',req.params.id],['filename','=',frame.filename])
++n
})
s.knexQuery({
action: "select",
columns: "*",
table: "Timelapse Frames",
where: frames
},(err,r) => {
if(r.length === 0){
s.closeJsonResponse(res,{
ok: false
})
return
}
s.createVideoFromTimelapse(r.reverse(),s.getPostData(req, 'fps'),function(response){
s.closeJsonResponse(res,{
ok : response.ok,
filename : response.filename,
fileExists : response.fileExists,
msg : response.msg,
})
})
initiateTimelapseVideoBuild({
groupKey,
monitorId,
framesPosted,
framesPerSecond,
}).then((buildResponse) => {
s.closeJsonResponse(res,buildResponse)
})
},res,req);
});
@ -304,15 +530,31 @@ module.exports = function(s,config,lang,app,io){
], function (req,res){
res.setHeader('Content-Type', 'application/json');
s.auth(req.params,function(user){
var hasRestrictions = user.details.sub && user.details.allmonitors !== '1'
const groupKey = req.params.ke
const monitorId = req.params.id
const actionParameter = !!req.params.action
const {
monitorPermissions,
monitorRestrictions,
} = s.getMonitorsPermitted(user.details,monitorId)
const {
isRestricted,
isRestrictedApiKey,
apiKeyPermissions,
} = s.checkPermission(user)
if(
user.permissions.watch_videos==="0" ||
hasRestrictions && (!user.details.video_view || user.details.video_view.indexOf(req.params.id)===-1)
actionParameter && (
isRestrictedApiKey && apiKeyPermissions.delete_videos_disallowed ||
isRestricted && !monitorPermissions[`${monitorId}_video_delete`]
) ||
!actionParameter && (
isRestrictedApiKey && apiKeyPermissions.watch_videos_disallowed ||
isRestricted && monitorId && !monitorPermissions[`${monitorId}_video_view`]
)
){
res.end(s.prettyPrint([]))
s.closeJsonResponse(res,{ok: false, msg: lang['Not Authorized']});
return
}
const monitorRestrictions = s.getMonitorRestrictions(user.details,req.params.id)
const cacheKey = req.params.ke + req.params.id + req.params.filename
const processFrame = (frame) => {
var fileLocation
@ -326,7 +568,7 @@ module.exports = function(s,config,lang,app,io){
selectedDate = req.params.filename.split('T')[0]
}
fileLocation = `${fileLocation}${frame.ke}/${frame.mid}_timelapse/${selectedDate}/${req.params.filename}`
if(req.params.action === 'delete'){
if(actionParameter === 'delete'){
deleteTimelapseFrame({
ke: frame.ke,
mid: frame.mid,
@ -356,6 +598,7 @@ module.exports = function(s,config,lang,app,io){
groupKey: req.params.ke,
archived: req.query.archived,
filename: req.params.filename,
limit: 1,
rowType: 'frames',
endIsStartTo: true
},(response) => {
@ -393,39 +636,63 @@ module.exports = function(s,config,lang,app,io){
})
},res,req);
});
var buildTimelapseVideos = function(){
var dateNow = new Date()
var hoursNow = dateNow.getHours()
if(hoursNow === 1){
var dateNowMoment = moment(dateNow).utc().format('YYYY-MM-DDTHH:mm:ss')
var dateMinusOneDay = moment(dateNow).utc().subtract(1, 'days').format('YYYY-MM-DDTHH:mm:ss')
s.knexQuery({
action: "select",
columns: "*",
table: "Timelapse Frames",
where: [
['time','=>',dateMinusOneDay],
['time','=<',dateNowMoment],
]
},function(err,frames) {
var groups = {}
frames.forEach(function(frame){
if(groups[frame.ke])groups[frame.ke] = {}
if(groups[frame.ke][frame.mid])groups[frame.ke][frame.mid] = []
groups[frame.ke][frame.mid].push(frame)
s.onOtherWebSocketMessages((d,connection) => {
switch(d.f){
case'timelapseVideoBuild':
initiateTimelapseVideoBuild({
groupKey: d.ke,
monitorId: d.mid,
framesPosted: d.frames,
framesPerSecond: d.fps,
}).then((buildResponse) => {
s.tx({
f: 'timelapse_build_requested',
ke: d.ke,
mid: d.mid,
buildResponse: buildResponse,
},'GRP_'+d.ke);
})
Object.keys(groups).forEach(function(groupKey){
Object.keys(groups[groupKey]).forEach(function(monitorId){
var frameSet = groups[groupKey][monitorId]
s.createVideoFromTimelapse(frameSet,30,function(response){
if(response.ok){
}
})
})
})
})
break;
}
})
function buildTimelapseVideos(){
return new Promise((resolve,reject) => {
var dateNow = new Date()
var hoursNow = dateNow.getHours()
if(hoursNow === 1){
var dateNowMoment = moment(dateNow).utc().format('YYYY-MM-DDTHH:mm:ss')
var dateMinusOneDay = moment(dateNow).utc().subtract(1, 'days').format('YYYY-MM-DDTHH:mm:ss')
s.knexQuery({
action: "select",
columns: "*",
table: "Timelapse Frames",
where: [
['time','=>',dateMinusOneDay],
['time','=<',dateNowMoment],
]
},async function(err,frames) {
var groups = {}
frames.forEach(function(frame){
if(groups[frame.ke])groups[frame.ke] = {}
if(groups[frame.ke][frame.mid])groups[frame.ke][frame.mid] = []
groups[frame.ke][frame.mid].push(frame)
})
const groupKeys = Object.keys(groups);
for (let i = 0; i < groupKeys.length; i++) {
const groupKey = groupKeys[i]
const monitorIds = Object.keys(groups[groupKey]);
for (let ii = 0; ii < monitorIds.length; ii++) {
const monitorId = monitorIds[ii]
const frameSet = groups[groupKey][monitorId]
await createVideoFromTimelapse(frameSet,30)
}
}
resolve()
})
}else{
resolve()
}
})
}
// Auto Build Timelapse Videos
if(config.autoBuildTimelapseVideosDaily === true){

2
libs/uploaders.js
View File

@ -15,6 +15,6 @@ module.exports = function(s,config,lang,app,io){
Object.keys(loadedLibraries).forEach((key) => {
var loadedLib = loadedLibraries[key](s,config,lang,app,io)
loadedLib.isFormGroupGroup = true
s.uploaderFields.push(loadedLib)
s.definitions["Account Settings"].blocks["Uploaders"].info.push(loadedLib)
})
}

55
libs/uploaders/amazonS3.js
View File

@ -1,12 +1,12 @@
var fs = require('fs');
module.exports = function(s,config,lang){
//Amazon S3
var beforeAccountSaveForAmazonS3 = function(d){
function beforeAccountSave(d){
//d = save event
d.formDetails.aws_use_global=d.d.aws_use_global
d.formDetails.use_aws_s3=d.d.use_aws_s3
}
var cloudDiskUseStartupForAmazonS3 = function(group,userDetails){
function cloudDiskUseStartup(group,userDetails){
group.cloudDiskUse['s3'].name = 'Amazon S3'
group.cloudDiskUse['s3'].sizeLimitCheck = (userDetails.use_aws_s3_size_limit === '1')
if(!userDetails.aws_s3_size_limit || userDetails.aws_s3_size_limit === ''){
@ -15,7 +15,7 @@ module.exports = function(s,config,lang){
group.cloudDiskUse['s3'].sizeLimit = parseFloat(userDetails.aws_s3_size_limit)
}
}
var loadAmazonS3ForUser = function(e){
function loadGroupApp(e){
// e = user
var userDetails = JSON.parse(e.details)
if(userDetails.aws_use_global === '1' && config.cloudUploaders && config.cloudUploaders.AmazonS3){
@ -54,11 +54,11 @@ module.exports = function(s,config,lang){
s.group[e.ke].aws_s3 = new s.group[e.ke].aws.S3();
}
}
var unloadAmazonS3ForUser = function(user){
function unloadGroupApp(user){
s.group[user.ke].aws = null
s.group[user.ke].aws_s3 = null
}
var deleteVideoFromAmazonS3 = function(e,video,callback){
function deleteVideo(e,video,callback){
// e = user
try{
var videoDetails = JSON.parse(video.details)
@ -68,6 +68,9 @@ module.exports = function(s,config,lang){
if(!videoDetails.location){
videoDetails.location = video.href.split('.amazonaws.com')[1]
}
if(videoDetails.type !== 's3'){
return
}
s.group[e.ke].aws_s3.deleteObject({
Bucket: s.group[e.ke].init.aws_s3_bucket,
Key: videoDetails.location,
@ -76,7 +79,7 @@ module.exports = function(s,config,lang){
callback()
});
}
var uploadVideoToAmazonS3 = function(e,k){
function uploadVideo(e,k){
//e = video object
//k = temporary values
if(!k)k={};
@ -92,9 +95,8 @@ module.exports = function(s,config,lang){
s.group[e.ke].aws_s3.upload({
Bucket: s.group[e.ke].init.aws_s3_bucket,
Key: saveLocation,
Body:fileStream,
ACL:'public-read',
ContentType:'video/'+ext
Body: fileStream,
ContentType: 'video/'+ext
},function(err,data){
if(err){
s.userLog(e,{type:lang['Amazon S3 Upload Error'],msg:err})
@ -114,7 +116,7 @@ module.exports = function(s,config,lang){
}),
size: k.filesize,
end: k.endTime,
href: data.Location
href: ''
}
})
s.setCloudDiskUsedForGroup(e.ke,{
@ -126,7 +128,7 @@ module.exports = function(s,config,lang){
})
}
}
var onInsertTimelapseFrame = function(monitorObject,queryInfo,filePath){
function onInsertTimelapseFrame(monitorObject,queryInfo,filePath){
var e = monitorObject
if(s.group[e.ke].aws_s3 && s.group[e.ke].init.use_aws_s3 !== '0' && s.group[e.ke].init.aws_s3_save === '1'){
var fileStream = fs.createReadStream(filePath)
@ -152,6 +154,7 @@ module.exports = function(s,config,lang){
mid: queryInfo.mid,
ke: queryInfo.ke,
time: queryInfo.time,
filename: queryInfo.filename,
details: s.s({
type : 's3',
location : saveLocation
@ -169,7 +172,7 @@ module.exports = function(s,config,lang){
})
}
}
var onDeleteTimelapseFrameFromCloud = function(e,frame,callback){
function onDeleteTimelapseFrameFromCloud(e,frame,callback){
// e = user
try{
var frameDetails = JSON.parse(frame.details)
@ -190,18 +193,30 @@ module.exports = function(s,config,lang){
callback()
});
}
function onGetVideoData(video){
const videoDetails = s.parseJSON(video.details)
return new Promise((resolve, reject) => {
const saveLocation = videoDetails.location
var fileStream = s.group[video.ke].aws_s3.getObject({
Bucket: s.group[video.ke].init.aws_s3_bucket,
Key: saveLocation,
}).createReadStream();
resolve(fileStream)
})
}
//amazon s3
s.addCloudUploader({
name: 's3',
loadGroupAppExtender: loadAmazonS3ForUser,
unloadGroupAppExtender: unloadAmazonS3ForUser,
insertCompletedVideoExtender: uploadVideoToAmazonS3,
deleteVideoFromCloudExtensions: deleteVideoFromAmazonS3,
cloudDiskUseStartupExtensions: cloudDiskUseStartupForAmazonS3,
beforeAccountSave: beforeAccountSaveForAmazonS3,
onAccountSave: cloudDiskUseStartupForAmazonS3,
loadGroupAppExtender: loadGroupApp,
unloadGroupAppExtender: unloadGroupApp,
insertCompletedVideoExtender: uploadVideo,
deleteVideoFromCloudExtensions: deleteVideo,
cloudDiskUseStartupExtensions: cloudDiskUseStartup,
beforeAccountSave: beforeAccountSave,
onAccountSave: cloudDiskUseStartup,
onInsertTimelapseFrame: onInsertTimelapseFrame,
onDeleteTimelapseFrameFromCloud: onDeleteTimelapseFrameFromCloud
onDeleteTimelapseFrameFromCloud: onDeleteTimelapseFrameFromCloud,
onGetVideoData
})
//return fields that will appear in settings
return {

4
libs/uploaders/backblazeB2.js
View File

@ -57,6 +57,10 @@ module.exports = function(s,config,lang){
b2.listBuckets().then(function(resp){
var buckets = resp.buckets
var bucketN = -2
if(!buckets){
s.userLog({mid:'$USER',ke:e.ke},{type: lang['Backblaze Error'],msg: lang['Not Authorized']})
return
}
buckets.forEach(function(item,n){
if(item.bucketName === userDetails.bb_b2_bucket){
bucketN = n

3
libs/uploaders/googleDrive.js
View File

@ -118,6 +118,9 @@ module.exports = (s,config,lang,app,io) => {
var deleteVideoFromGoogleDrive = function(groupKey,video,callback){
// e = user
var videoDetails = s.parseJSON(video.details)
if(videoDetails.type !== 'googd'){
return
}
s.group[groupKey].googleDrive.files.delete({
fileId: videoDetails.id
}, function(err, resp){

23
libs/uploaders/s3based.js
View File

@ -84,6 +84,9 @@ module.exports = function(s,config,lang){
if(!videoDetails.location){
videoDetails.location = video.href.split(locationUrl)[1]
}
if(videoDetails.type !== 'whcs'){
return
}
s.group[e.ke].whcs.deleteObject({
Bucket: s.group[e.ke].init.whcs_bucket,
Key: videoDetails.location,
@ -114,9 +117,8 @@ module.exports = function(s,config,lang){
s.group[e.ke].whcs.upload({
Bucket: bucketName,
Key: saveLocation,
Body:fileStream,
ACL:'public-read',
ContentType:'video/'+ext
Body: fileStream,
ContentType: 'video/'+ext
},options,function(err,data){
if(err){
console.error(err)
@ -177,6 +179,7 @@ module.exports = function(s,config,lang){
mid: queryInfo.mid,
ke: queryInfo.ke,
time: queryInfo.time,
filename: queryInfo.filename,
details: s.s({
type : 'whcs',
location : saveLocation
@ -231,6 +234,17 @@ module.exports = function(s,config,lang){
}
return cloudLink
}
function onGetVideoData(video){
const videoDetails = s.parseJSON(video.details)
return new Promise((resolve, reject) => {
const saveLocation = videoDetails.location
var fileStream = s.group[video.ke].whcs.getObject({
Bucket: s.group[video.ke].init.whcs_bucket,
Key: saveLocation,
}).createReadStream();
resolve(fileStream)
})
}
//wasabi
s.addCloudUploader({
name: 'whcs',
@ -242,7 +256,8 @@ module.exports = function(s,config,lang){
beforeAccountSave: beforeAccountSaveForWasabiHotCloudStorage,
onAccountSave: cloudDiskUseStartupForWasabiHotCloudStorage,
onInsertTimelapseFrame: onInsertTimelapseFrame,
onDeleteTimelapseFrameFromCloud: onDeleteTimelapseFrameFromCloud
onDeleteTimelapseFrameFromCloud: onDeleteTimelapseFrameFromCloud,
onGetVideoData
})
return {
"evaluation": "details.use_whcs !== '0'",

70
libs/uploaders/webdav.js
View File

@ -41,11 +41,10 @@ module.exports = function(s,config,lang){
userDetails.webdav_dir='/'
}
userDetails.webdav_dir = s.checkCorrectPathEnding(userDetails.webdav_dir)
s.group[e.ke].webdav = webdav(
userDetails.webdav_url,
userDetails.webdav_user,
userDetails.webdav_pass
)
s.group[e.ke].webdav = webdav.createAdapter(userDetails.webdav_url, {
username: userDetails.webdav_user,
password: userDetails.webdav_pass
})
}
}
var unloadWebDavForUser = function(user){
@ -95,7 +94,7 @@ module.exports = function(s,config,lang){
}),
size: k.filesize,
end: k.endTime,
href: webdavRemoteUrl
href: ''
}
})
s.setCloudDiskUsedForGroup(e.ke,{
@ -158,6 +157,62 @@ module.exports = function(s,config,lang){
}
}
}
function onInsertTimelapseFrame(monitorObject,queryInfo,filePath){
var e = monitorObject
if(s.group[e.ke].webdav && s.group[e.ke].init.use_webdav !== '0' && s.group[e.ke].init.webdav_save === '1'){
const wfs = s.group[e.ke].webdav
const saveLocation = s.group[e.ke].init.webdav_dir+e.ke+'/'+e.mid+'_timelapse/' + queryInfo.filename
fs.createReadStream(filePath).pipe(wfs.createWriteStream(saveLocation))
if(s.group[e.ke].init.webdav_log === '1'){
s.knexQuery({
action: "insert",
table: "Cloud Timelapse Frames",
insert: {
mid: queryInfo.mid,
ke: queryInfo.ke,
time: queryInfo.time,
filename: queryInfo.filename,
details: s.s({
type : 'webdav',
location : saveLocation
}),
size: queryInfo.size,
href: ''
}
})
s.setCloudDiskUsedForGroup(e.ke,{
amount : s.kilobyteToMegabyte(queryInfo.size),
storageType : 'webdav'
},'timelapseFrames')
s.purgeCloudDiskForGroup(e,'webdav','timelapseFrames')
}
}
}
function onDeleteTimelapseFrameFromCloud(e,frame,callback){
// e = user
try{
var frameDetails = JSON.parse(frame.details)
}catch(err){
var frameDetails = frame.details
}
if(frameDetails.type !== 'webdav'){
return
}
if(!frameDetails.location){
frameDetails.location = frame.href.split(locationUrl)[1]
}
s.group[e.ke].webdav.unlink(frameDetails.location, function(err) {
if (err) console.log(frameDetails.location,err)
callback()
})
}
async function onGetVideoData(video){
const wfs = s.group[video.ke].webdav
const videoDetails = s.parseJSON(video.details)
const saveLocation = videoDetails.location
const fileStream = wfs.createReadStream(saveLocation);
return fileStream
}
//webdav
s.addCloudUploader({
name: 'webdav',
@ -168,6 +223,9 @@ module.exports = function(s,config,lang){
cloudDiskUseStartupExtensions: cloudDiskUseStartupForWebDav,
beforeAccountSave: beforeAccountSaveForWebDav,
onAccountSave: cloudDiskUseStartupForWebDav,
onInsertTimelapseFrame,
onDeleteTimelapseFrameFromCloud,
onGetVideoData
})
return {
"evaluation": "details.use_webdav !== '0'",

28
libs/user.js
View File

@ -14,6 +14,7 @@ module.exports = function(s,config,lang){
deleteFileBinFiles,
deleteCloudVideos,
deleteCloudTimelapseFrames,
resetAllStorageCounters,
} = require("./user/utils.js")(s,config,lang);
let purgeDiskGroup = () => {}
const runQuery = async.queue(function(groupKey, callback) {
@ -218,15 +219,19 @@ module.exports = function(s,config,lang){
}
//change global size value
s.group[e.ke].usedSpace += currentChange
s.group[e.ke].usedSpace = s.group[e.ke].usedSpace < 0 ? 0 : s.group[e.ke].usedSpace
switch(storageType){
case'timelapeFrames':
s.group[e.ke].usedSpaceTimelapseFrames += currentChange
s.group[e.ke].usedSpaceTimelapseFrames = s.group[e.ke].usedSpaceTimelapseFrames < 0 ? 0 : s.group[e.ke].usedSpaceTimelapseFrames
break;
case'fileBin':
s.group[e.ke].usedSpaceFilebin += currentChange
s.group[e.ke].usedSpaceFilebin = s.group[e.ke].usedSpaceFilebin < 0 ? 0 : s.group[e.ke].usedSpaceFilebin
break;
default:
s.group[e.ke].usedSpaceVideos += currentChange
s.group[e.ke].usedSpaceVideos = s.group[e.ke].usedSpaceVideos < 0 ? 0 : s.group[e.ke].usedSpaceVideos
break;
}
//remove value just used from queue
@ -267,6 +272,12 @@ module.exports = function(s,config,lang){
}
})
}
function filterMonitorListOrder(groupKey,details){
const loadedMonitors = s.group[groupKey].rawMonitorConfigurations
var monitorListOrder = (details.monitorListOrder && details.monitorListOrder[0] ? details.monitorListOrder[0] : []).filter(monitorId => !!loadedMonitors[monitorId]);
monitorListOrder = [...new Set(monitorListOrder)];
return monitorListOrder
}
s.accountSettingsEdit = function(d,dontRunExtensions){
s.knexQuery({
action: "select",
@ -302,6 +313,7 @@ module.exports = function(s,config,lang){
}
//admin permissions
formDetails.permissions = details.permissions
formDetails.max_camera = details.max_camera
formDetails.edit_size = details.edit_size
formDetails.edit_days = details.edit_days
formDetails.use_admin = details.use_admin
@ -358,7 +370,9 @@ module.exports = function(s,config,lang){
}
readStorageArray()
///
formDetails = JSON.stringify(s.mergeDeep(details,formDetails))
formDetails = s.mergeDeep(details,formDetails)
if(formDetails.monitorListOrder)formDetails.monitorListOrder[0] = filterMonitorListOrder(d.ke,formDetails);
formDetailsString = JSON.stringify(s.mergeDeep(details,formDetails))
///
const updateQuery = {}
if(form.pass && form.pass !== ''){
@ -371,7 +385,7 @@ module.exports = function(s,config,lang){
const value = form[key]
updateQuery[key] = value
})
updateQuery.details = formDetails
updateQuery.details = formDetailsString
s.knexQuery({
action: "update",
table: "Users",
@ -381,20 +395,20 @@ module.exports = function(s,config,lang){
['uid','=',d.uid],
]
},() => {
const user = Object.assign({ke : d.ke},form)
if(!details.sub){
var user = Object.assign(form,{ke : d.ke})
var userDetails = JSON.parse(formDetails)
s.group[d.ke].sizeLimit = parseFloat(newSize)
resetAllStorageCounters(d.ke)
if(!dontRunExtensions){
s.onAccountSaveExtensions.forEach(function(extender){
extender(s.group[d.ke],userDetails,user)
})
s.unloadGroupAppExtensions.forEach(function(extender){
extender(user)
})
s.loadGroupApps(d)
}
}
s.onAccountSaveExtensions.forEach(function(extender){
extender(s.group[d.ke],formDetails,user)
})
if(d.cnid)s.tx({f:'user_settings_change',uid:d.uid,ke:d.ke,form:form},d.cnid)
})
}

45
libs/user/startup.js Normal file
View File

@ -0,0 +1,45 @@
module.exports = function(s,config,lang,io){
const {
createAdminUser
} = require("./utils.js")(s,config,lang);
function checkStaticUser(staticUser){
return new Promise((resolve,reject) => {
const whereQuery = {
mail: staticUser.mail
}
if(staticUser.ke)whereQuery.ke = staticUser.ke
s.knexQuery({
action: "select",
columns: "mail,ke,uid",
table: "Users",
where: whereQuery,
limit: 1,
},function(err,users) {
resolve(users[0])
})
})
}
async function checkForStaticUsers(){
if(config.staticUsers){
try{
for (let i = 0; i < config.staticUsers.length; i++) {
const staticUser = config.staticUsers[i]
s.debugLog(`Checking Static User...`,staticUser.mail)
const userExists = await checkStaticUser(staticUser)
if(!userExists){
s.debugLog(`Static User does not exist, creating...`)
const creationResponse = await createAdminUser(staticUser)
s.debugLog(`Static User created!`,creationResponse)
}else{
s.debugLog(`Static User exists!`)
}
}
}catch(err){
s.debugLog(`Static User check error!`,err)
}
}
}
return {
checkForStaticUsers,
}
}

Some files were not shown because too many files have changed in this diff Show More