As sessions in pgAdmin4 are filesystem based session, they need locking
for avoiding the access from multiple threads, specially running as an
WSGI application.
Fixes#3547
- Extract SQLEditor.execute and SQLEditor._poll into their own files and add test around them
- Extract SQLEditor backend functions that start executing query to their own files and add tests around it
- Move the Explain SQL from the front-end and now pass the Explain plan parameters as a JSON object in the start query call.
- Extract the compile_template_name into a function that can be used by the different places that try to select the version of the template and the server type
Because - the get_locale(...) function was being executed before any
request, and user login after first request. Hence - the values for
the labels in the Preferences are not being translated properly.
Fetch the raw value of user_language from configuration in case of
runtime/non-server mode to fix the issue.
This issue was caused because we recently added session_write_delay in session.
So session won't be written/updated to disk from memory until specified seconds are elapsed.
However we must forcefully write/update session to disk if user loges in or out irrespective of session_write_delay to keep sessions from memory and disk in sync as user logged in status is kept in session.
Session object is updated, everytime a request is being served, and
that was forcing the session object dumped on the dist on every request.
On windows, it was causing issues on slower system on startup. Because -
windows file system locks the file, when it is opened by any
application. And, frequent requests on the pgAdmin main UI rendering
was causing issues, because of that.
In order to resolve the issue, we will not write the session data on
disk for every request, but - only after certain delay (in seconds),
from it was last written. It can be configured using the attribute
'PGADMIN_SESSION_DISK_WRITE_DELAY' in the configuration file,
default vaule for the delay is 10.
(i.e. 10 seconds)
This allows the user to create partitioned table and it's partitions.
- Edit partitions, Create/Listing of constraints, rules, triggers under partitions. (Implemented by Harshal)
- Updating browser tree node when create/attach/detach partitions from table dialog.(Implemented by Ashesh)
Created a url_for function module to be used by different javascript
modules to use the same functionality of flask at client side.
A python module can expose the list of endpoints, used by its
javascripts, by overriding the 'get_exposed_url_endpoints(self)'
function.
In this patch, we have modified all the browser modules to use this
function. It will allow us to move the majority of the javascript
modules of browser nodes from templates directory to the static
directory.
TODO::
- Move these javascripts modules to the static directory.
- Use this function in all the applicable javascript modules.
e.g. tools, miscellaneous modules
1) To handle non-ascii filenames which we set from table name. Fixes#2314
2) To handle non-ascii query data. Fixes#2253
3) To dump JSON type columns properly in csv. Fixes#2360
In order to resolve the non-ascii characters in path (in user directory,
storage path, etc) on windows, we have converted the path into the
short-path, so that - we don't need to deal with the encoding issues
(specially with Python 2).
We've resolved majority of the issues with this patch.
We still need couple issues to resolve after this in the same area.
TODO
* Add better support for non-ascii characters in the database name on
windows with Python 3
* Improve the messages created after the background processes by
different modules (such as Backup, Restore, Import/Export, etc.),
which does not show short-paths, and xml representable characters for
non-ascii characters, when found in the database objects, and the file
PATH.
Fixes#2174, #1797, #2166, #1940
Initial patch by: Surinder Kumar
Reviewed by: Murtuza Zabuawala
Using 'psycopg2.extensions.UNICODE' (for Python < 3) in the psycopg2
driver for proper conversation of unicode characters. Also - adjusted
the string typecaster to take care of different character types (char,
character, text, name, character varying, and their array types).
Reviewed by: Dave Page, Murtuza Zabuawala & Akshay Joshi
- don't spin up app and chromedriver between each test
- catching signals also tears down the app
- do layout reset between tests, but assume that tests will not leave a modal opened.
Use selenium built-in waiting function and fix flakiness around clicking the alertify OK button
- we think the OK button does not have its event bound when it is created.
If you see more flakiness around clicking the alertify OK button, let us know. The element is clickable but we have to arbitrarily wait for the event to be bound and that timing may vary system to system.
The feature tests are about 7 seconds faster now.
Tira & Joao
To chase down a set of tests that were failing together, we wanted to run
deeper trees of tests. We changed the behavior of the --pkg argument to
runtests.py so it runs tests under a package and its subpackages, instead
of for a single node.
It gave us more flexibility in what tests to run. This also includes the
--exclude option.
To run tests for a single node, ".tests" can be appended to the package path.
Tira & George.
1. Code changes (due to drop objects functionality).
2. Quoting for database names in drop databases.
3. Code changes for import errors for pickle_path and advanced_config variables.
Made backend changes for:
* Taking care of the connection status in the psycopg2 driver. And, when
the connection is lost, it throws a exception with 503 http status
message, and connection lost information in it.
* Allowing the flask application to propagate the exceptions even in the
release mode.
* Utilising the existing password (while reconnection, if not
disconnected explicitly).
* Introduced a new ajax response message 'service_unavailable' (http
status code: 503), which suggests temporary service unavailable.
Client (front-end) changes:
* To handle the connection lost of a database server for different
operations by generating proper events, and handle them properly.
Removed the connection status check code from different nodes, so that
- it generates the proper exception, when accessing the non-alive
connection.
Fixes#1387
1. The user will specify the tablespace path in test_config.json.in
2. If tablespace path not found, skip the test cases for that server(Only tablespace test cases)
3. Add the skipped test summary in the test result. (Now it's showing on console + in log file, but need to update in a final enhanced test summary report. Which is research point we will work on that after finishing all nodes API test cases)
4. Removed the test_ prefix from the values in the config files.
5. Add tablespace and roles tests
- Test framework support API testing with multiple server for this we need to modify test_config.json(for user it’s test_config.json.in) and test_advanced_config.json(for user it’s test_advanced_config.json.in). Server details of PG and PPAS are included in both .in files.
- Removed the logic of logging in the test client on each test scenario(As per Khushboo's comment in previous email). We need this logic in test cases under ‘browser/tests/’ as for test scenarios like change password and invalid login test cases as test client should be logged out first. So, as per this the code is slightly modified in ‘browser/tests/’.
optimized to work from multiple threads. It has too many frequent
transaction from multiple threads, and that tends to result into the
'database is locked' error of sqlite.
With the new implemenation, we're using the caching mechanism, which
keep the data in the memory all the time, and saves it on request
completion, and loads it only for the first time. Also, it will storage
the data using pickle, which will be faster than accessing sqlite.
Fixes#1329
1) No handling for INTERVAL type datetime.
For example: executing query
SELECT INTERVAL '15 minutes';
throws json serialization error, because it returns time in timedelta format which is not handled.
Added support to handle timedelta datetime format in DataTypeJSONEncoder class
2) When we try to get BC dates from database raises ValueError: year is out of range
For eg:
SELECT TIMESTAMP '0044-03-15 10:00:00 BC',
It is because pyscopg2 doesn't handle BC datetime format.
So we have defined our method which type cast the datetime value to string in pyscopg2 overriding default behaviour.
Reference:
http://initd.org/psycopg/docs/advanced.html#type-casting-from-sql-to-pythonccf3693be6/pgcli/pgexecute.py
1) Unable to select sql file through query tool on windows OS. In file_manager.js, we are stripping initial slash '/' from the path obtained, but we should not strip if it is full path like 'c:/path/to/dir/filename.ext'
2) Handle directory path if STORAGE_DIR is None. Proper checks are added.
As we convert the binary password to string during storing the
connection information, we also need to convert it back to byte-arrays
during restoring the connections.
Hence - decode the passowrd using 'utf-8' encoding during storing the
connection information, and encode it during restoring the connections.
Tweaked by Ashesh Vashi to integrate the backgroud process, and also
with some improvements as stated below:
* Resolved an issue loading existing preference.
* Improved the background process observer/executor for supporting
detalied view.
* Added the utility path preferences in the ServerType class.
operation like backup, restore, etc within it.
Also:
* improvised the color combination of the background process logger.
* Removed an unnecessary print statement from the
get_storage_directory(..) function, also return None if STORAGE_DIR
is set to None.
selection, creation, upload/download files/directories resides on the
server side.
This will be useful for file selection/creation for different server
side utilites like pg_dump, pg_dumpall, pg_restore.
server.
Current implementation keeps the cursor object in the 'g' (current
request context) object identified by the connection-id.
But - it fails to identify a different connection object request, when
we have same database name in different database server, it does not
able to identify it as separate request, Hence - now we will use
server-id qualified identifier for the same object.
Thanks Neel Patel for pointing out the issue.
required by its javascript module.
This will allow us to load the javascript modules as a static file, and
not as a Jinja2 template. This will increase the load time, as it will
decrease number of templates to be processed during loading those
javascripts.
DriverRegistry is executed second time.
Also, initialize the driver before registering different blueprints,
which uses those driver inside them.
Thanks Khushboo for reporting the issue.
At the moment, we will only allow to fetch status messages from the
asynchronous connection only, later - we may implement to fetch the
status message from the normal connection too.
configuration database due to unable to identify the change properly.
Thanks Dave for reporting.
This also includes a support to use a 'text' type of preference, and
show help string next to the control.
This commit takes care of the following issues/improvements.
* Adding missing imports for unauthorised in database module
* Node under Servers Nodes (i.e. Databases, tablespaces, roles nodes)
need to be inherited from PGChildNodeView (and, not from NodeView) for
adding server version check for their children.
* Adding statistics for database, and tablespaces in its node (not, yet
in UI)
* Renaming the camel case methods with proper name.
(i.e. getSQL -> get_sql, getNewSQL -> get_new_sql, etc.)
* Fixed the functions going beyond the text limit (column: 80) in
Databases, Roles & Tablespaces modules.
* Fixed the node method of Database module, which was not tested ever.
* We do not need separate SQL template for fetching the name (i.e.
get_name.sql), using the 'nodes.sql' for the same.
* Optimise the query for fetching ACLs for the database node, we didn't
require to join certain tables, while fetching only the ACLs.
* Introduced the list of the ACLs (regular and default ACLs for
different type) supported by each version [Databases Module].
* Renamed the templates 'get_nodes.sql' to' nodes.sql' to make it
consistent with other modules.
* Removed the checks for the authentication table use, as we don't need
to expose the password to the users (even the encrypted MD5). Using
the pg_roles view always for fetching roles/users information now.
* Resolved some typos in unreachable (specially the exceptions
catchment area.)
* Logging the exception in the application.
* Using qtLiteral, qtIdent properly in the templates (do not assume
about the types of data.)
* Using tsid as identifier instead of did for the tablespaces.
* Using nodes method of tablespace view for fetching individual node
information.
* Removing the hardcoded node information from the 'parse_priv_to_db'
function, and pass on allowed ACLs by the caller nodes.
* Using 'nodes.sql' to fetch name of the template instead of writing
that in the delete.sql template, which is definitely wrong place to
fetch the name of the object. [Tablespace Module]
of double quote within it.
The 'replace' function of immutable type string do create a new
instance of type string, which needed to be reassgined to the variable
'value'.
Reported By: Murtuza Zabuawala
It was due to wrong index used for the COL_NAME_KEYWORD, used by the
'needsQuoting' function.
Also, replaced a tab with a space in the generate_keyword.py file.
Reported by: Sanket Mehta, Khushboo Vashi
with schema specified object name. This will also works with
EntepriseDB's package too.
Also, added dependency on the current connection object, it will allow
us to use the keywords list specific to the database server in futuer.
You can use the qtIdent template filter as:
conn|qtIdent(name)
or,
conn|qtIdent(schema_name, name)
inputs.
In order to do the proper quoting around the identifier, different type,
and literal, we introduced respective functions qtIdent, qtTypeIdent,
qtLiteral in psycopg2 driver. Also, introduced them as the Jinja's
custom filter for using it directly inside the templates.
Also, created an utility - generate_keywords.py in order to generate
keyword lists from the latest PostgreSQL installation.
Introduced a new ajax function to return the HTTP Status code 410 (GONE).
Changed the location of the javascript from each modules, it loads.
Added .swp, .swo files in the .gitignore file
- Allow to release connection using the database OID (did).
- Generate correct url for the collection nodes.
- Removed the server-type from the collection node module, from the
generate_browser_collection_node(...) function.
- Show version string, and not version integer in the server properties
(when connected).
connect to the database server. Also, added modified the way, we do
check the node is supported by the server.
Instead of creating separate blueprint for the server types, they will
be independently works. In order to add different variant of the
PostgreSQL, we need to extend the ServerType class, and override the
'instanceOf' function for identification using version string. Please
take a look at the ppas.py for the example.
During checking the back-end support for the node, we will also check
the server type (variant) along with the version within the range of
maximum and minimum version for the node. And, the same support added
for the schema attributes in front-end (JavaScript).
Really thankful to Khushboo Vashi for her initial work in front-end. I
took it further from there.
driver)
1. Update correct variable for database information in the connection
manager.
2. Raise exception, when database not found, instead of return the
(boolean, error message) tupple.
manager, once the connection to the maintenance database has been made,
because - we will have did (i.e. Database OID) most of the time for the
any node, and not its name as identifier.
This will allow us to work directly with OID, and we will not need to
bother about renaming of the database name.
server connection.
The BaseDriver and BaseConnection are two abstract classes, which allows
us to replace the existing driver with the currently used. The current
implementation supports to connect the PostgreSQL and Postgres Plus
Advanced Server using the psycopg2 driver.
(lazy loading) using the require.js. This allows us to load the
javascript required for any node, only when it was loaded in the browser
tree. Also, introduced the mechanism to show/edit/create of any node in
a tab panel (wcDocker.Panel).
Akshay Joshi
Aditya Toshniwal
Harshal Dhumal
Usman Muzaffar
Dave Page
Khushboo Vashi
Anthony Emengo
Joao Pedro De Almeida Pereira
Murtuza Zabuawala
Murtuza Zabuawala
Alexander Lakhin
Teng Zhang
Atul Sharma
Shruti B Iyer
Ashesh Vashi
Sarah McAlear
Navnath Gadakh
RAJASEKAR G
Jonas Thelemann
Surinder Kumar
Atira Odhner
George Gelashvili
George Gelashvili
Murtuza Zabuawala
Karen Blatchley
Priyanka Shendge
Neel Patel
Sanket Mehta