FromDual: PostgreSQL Tech-Feed (en)

Database Index Optimizer

Wed, 15 Jul 2026 16:55:00 +0200

Recently, a client asked me if the “time-consuming” task of checking indexes could be left to an index optimizer. Of course it can…

What exactly do we want to check?

Tables without a Primary Key
Duplicate indexes
Partially redundant indexes
Unused indexes

MariaDB, MySQL, and Percona Server

Tables without a Primary Key

SQL> SELECT DISTINCT t.table_schema, t.table_name
 FROM information_schema.tables AS t
 LEFT JOIN information_schema.columns AS c ON t.table_schema = c.table_schema AND t.table_name = c.table_name
 AND c.column_key = "PRI"
 WHERE t.table_schema NOT IN ('information_schema', 'mysql', 'performance_schema')
 AND c.table_name IS NULL AND t.table_type NOT IN('VIEW', 'SEQUENCE')
 AND t.table_schema = 'testtest'
;
+--------------+------------+
| table_schema | table_name |
+--------------+------------+
| testtest | archived |
+--------------+------------+
1 row in set

Source: Tables without a Primary Key

Duplicate indexes

SQL> SELECT table_name, redundant_index_name, redundant_index_columns, dominant_index_name, dominant_index_columns, sql_drop_index
 FROM sys.schema_redundant_indexes
 WHERE redundant_index_columns = dominant_index_columns
 AND table_schema = 'testtest'
;
+------------+----------------------+-------------------------+---------------------+------------------------+------------------------------------------------------+
| table_name | redundant_index_name | redundant_index_columns | dominant_index_name | dominant_index_columns | sql_drop_index |
+------------+----------------------+-------------------------+---------------------+------------------------+------------------------------------------------------+
| archived | dupl2 | category_id | dupl1 | category_id | ALTER TABLE `testtest`.`archived` DROP INDEX `dupl2` |
+------------+----------------------+-------------------------+---------------------+------------------------+------------------------------------------------------+
1 row in set

Source: Duplicate and redundant indices

Partially redundant indexes

SQL> SELECT table_name, redundant_index_name, redundant_index_columns, dominant_index_name, dominant_index_columns, sql_drop_index
 FROM sys.schema_redundant_indexes
 WHERE table_schema = 'testtest'
;
+-------------------+----------------------+-------------------------+---------------------+---------------------------------+-------------------------------------------------------------------+
| table_name | redundant_index_name | redundant_index_columns | dominant_index_name | dominant_index_columns | sql_drop_index |
+-------------------+----------------------+-------------------------+---------------------+---------------------------------+-------------------------------------------------------------------+
| access | customer | customer | customer_2 | customer,callerid_internal | ALTER TABLE `testtest`.`access` DROP INDEX `customer` |
| access | customer | customer | customer_3 | customer,callerid_external | ALTER TABLE `testtest`.`access` DROP INDEX `customer` |
| active_customers | uniqueid | uniqueid | PRIMARY | uniqueid,scustomer | ALTER TABLE `testtest`.`active_customers` DROP INDEX `uniqueid` |
| analytics_include | analytics | analytics | PRIMARY | analytics,feature,dtype,dnumber | ALTER TABLE `testtest`.`analytics_include` DROP INDEX `analytics` |
| archived | dupl2 | category_id | dupl1 | category_id | ALTER TABLE `testtest`.`archived` DROP INDEX `dupl2` |
...
| texts_media | uniqueid | uniqueid | PRIMARY | uniqueid,filename | ALTER TABLE `testtest`.`texts_media` DROP INDEX `uniqueid` |
| unlimited_access | customer | customer | customer_2 | customer,callerid_internal | ALTER TABLE `testtest`.`unlimited_access` DROP INDEX `customer` |
| unlimited_access | customer | customer | customer_3 | customer,callerid_external | ALTER TABLE `testtest`.`unlimited_access` DROP INDEX `customer` |
+-------------------+----------------------+-------------------------+---------------------+---------------------------------+-------------------------------------------------------------------+
26 rows in set

Source: Duplicate and redundant indices

Unused indexes

SQL> SELECT object_name, index_name
 FROM sys.schema_unused_indexes
 WHERE object_schema = 'testtest'
;
+------------------------+------------------------+
| object_name | index_name |
+------------------------+------------------------+
| access | customer_3 |
| access | customer_2 |
| actions | class |
| actions | action |
| active | channel |
...
| urls | customer |
| voucher_batches | customer |
| vouchers | batch |
+------------------------+------------------------+
413 rows in set

Note:

For MariaDB, the PERFORMANCE_SCHEMA must be enabled first.
The information is accurate as of the last database restart. If an index was last used BEFORE the most recent restart, it will be shown here as unused.

Source: Unused indexes

And now with PostgreSQL

Tables without a Primary Key

SQL> SELECT tab.table_schema, tab.table_name
 FROM information_schema.tables tab
 LEFT JOIN information_schema.table_constraints tco
 ON tab.table_schema = tco.table_schema
 AND tab.table_name = tco.table_name 
 AND tco.constraint_type = 'PRIMARY KEY'
 WHERE tab.table_type = 'BASE TABLE'
 AND tab.table_schema NOT IN ('pg_catalog', 'information_schema')
 AND tco.constraint_name IS NULL
 ORDER BY table_schema, table_name
;
 table_schema | table_name 
--------------+------------
 public | archived
(1 row)

Source: Find tables without primary keys (PKs) in PostgreSQL database

Duplicate indexes

Based on the MySQL sys schema:

SQL> WITH schema_flattened_keys AS (
 SELECT sai.relid, sai.indexrelid
 , sai.schemaname AS table_schema, sai.relname AS table_name, sai.indexrelname AS index_name
 , CASE pi.indisunique WHEN 'f' THEN 1 ELSE 0 END AS non_unique
 , index_columns.columns AS index_columns
 FROM pg_stat_all_indexes AS sai
 JOIN pg_index AS pi ON pi.indexrelid = sai.indexrelid
 JOIN (
 SELECT attrelid, string_agg(attname, ',' ORDER BY attnum ASC) AS columns
 FROM pg_attribute GROUP BY attrelid
 ) AS index_columns ON index_columns.attrelid = sai.indexrelid
 WHERE sai.schemaname NOT IN ('pg_toast', 'pg_catalog')
)
SELECT redundant_keys.table_schema AS table_schema, redundant_keys.table_name AS table_name, redundant_keys.index_name AS redundant_index_name
 , redundant_keys.index_columns AS redundant_index_columns, redundant_keys.non_unique AS redundant_index_non_unique
 , dominant_keys.index_name AS dominant_index_name, dominant_keys.index_columns AS dominant_index_columns, dominant_keys.non_unique AS dominant_index_non_unique
 , CONCAT('ALTER TABLE ', redundant_keys.table_schema, '.', redundant_keys.table_name, ' DROP INDEX ', redundant_keys.index_name, '') AS sql_drop_index
 FROM schema_flattened_keys redundant_keys
 JOIN schema_flattened_keys dominant_keys ON redundant_keys.table_schema = dominant_keys.table_schema AND redundant_keys.table_name = dominant_keys.table_name
 WHERE (redundant_keys.index_name <> dominant_keys.index_name
 AND ((redundant_keys.index_columns = dominant_keys.index_columns)
 AND ((redundant_keys.non_unique > dominant_keys.non_unique) OR (redundant_keys.non_unique = dominant_keys.non_unique))
 )
 OR ((POSITION(CONCAT(redundant_keys.index_columns,',') IN dominant_keys.index_columns) = 1) AND (redundant_keys.non_unique = 1))
 OR ((POSITION(CONCAT(dominant_keys.index_columns,',') IN redundant_keys.index_columns) = 1) AND (dominant_keys.non_unique = 0))
 )
 AND redundant_keys.index_columns = dominant_keys.index_columns
;
 table_schema | table_name | redundant_index_name | redundant_index_columns | redundant_index_non_unique | dominant_index_name | dominant_index_columns | dominant_index_non_unique | sql_drop_index 
--------------+------------+----------------------+-------------------------+----------------------------+---------------------+------------------------+---------------------------+----------------------------------------------
 public | archived | dupl1 | category_id | 1 | dupl2 | category_id | 1 | ALTER TABLE public.archived DROP INDEX dupl1
 public | archived | dupl2 | category_id | 1 | dupl1 | category_id | 1 | ALTER TABLE public.archived DROP INDEX dupl2
(2 rows)

Source: Duplicate and redundant indices

Partially redundant indexes

Based on the MySQL sys schema:

SQL> WITH schema_flattened_keys AS (
 SELECT sai.relid, sai.indexrelid
 , sai.schemaname AS table_schema, sai.relname AS table_name, sai.indexrelname AS index_name
 , CASE pi.indisunique WHEN 'f' THEN 1 ELSE 0 END AS non_unique
 , index_columns.columns AS index_columns
 FROM pg_stat_all_indexes AS sai
 JOIN pg_index AS pi ON pi.indexrelid = sai.indexrelid
 JOIN (
 SELECT attrelid, string_agg(attname, ',' ORDER BY attnum ASC) AS columns
 FROM pg_attribute GROUP BY attrelid
 ) AS index_columns ON index_columns.attrelid = sai.indexrelid
 WHERE sai.schemaname NOT IN ('pg_toast', 'pg_catalog')
)
SELECT redundant_keys.table_schema AS table_schema, redundant_keys.table_name AS table_name, redundant_keys.index_name AS redundant_index_name
 , redundant_keys.index_columns AS redundant_index_columns, redundant_keys.non_unique AS redundant_index_non_unique
 , dominant_keys.index_name AS dominant_index_name, dominant_keys.index_columns AS dominant_index_columns, dominant_keys.non_unique AS dominant_index_non_unique
 , CONCAT('ALTER TABLE ', redundant_keys.table_schema, '.', redundant_keys.table_name, ' DROP INDEX ', redundant_keys.index_name, '') AS sql_drop_index
 FROM schema_flattened_keys redundant_keys
 JOIN schema_flattened_keys dominant_keys ON redundant_keys.table_schema = dominant_keys.table_schema AND redundant_keys.table_name = dominant_keys.table_name
 WHERE (redundant_keys.index_name <> dominant_keys.index_name
 AND ((redundant_keys.index_columns = dominant_keys.index_columns)
 AND ((redundant_keys.non_unique > dominant_keys.non_unique) OR (redundant_keys.non_unique = dominant_keys.non_unique))
 )
 OR ((POSITION(CONCAT(redundant_keys.index_columns,',') IN dominant_keys.index_columns) = 1) AND (redundant_keys.non_unique = 1))
 OR ((POSITION(CONCAT(dominant_keys.index_columns,',') IN redundant_keys.index_columns) = 1) AND (dominant_keys.non_unique = 0))
 )
;
 table_schema | table_name | redundant_index_name | redundant_index_columns | redundant_index_non_unique | dominant_index_name | dominant_index_columns | dominant_index_non_unique | sql_drop_index 
--------------+-----------------------+------------------------------------------+-------------------------+----------------------------+-------------------------------------------------+-----------------------------------------+---------------------------+---------------------------------------------------------------------------------------------
 public | numbers | numbers_customer_idx | customer | 1 | numbers_customer_text_dtype_text_dnumber_idx | customer,text_dtype,text_dnumber | 1 | ALTER TABLE public.numbers DROP INDEX numbers_customer_idx
 public | numbers | numbers_customer_idx | customer | 1 | numbers_customer_fax_dtype_fax_dnumber_idx | customer,fax_dtype,fax_dnumber | 1 | ALTER TABLE public.numbers DROP INDEX numbers_customer_idx
 public | numbers | numbers_customer_idx | customer | 1 | numbers_pkey | customer,stype,snumber | 0 | ALTER TABLE public.numbers DROP INDEX numbers_customer_idx
 public | numbers | numbers_dtype_idx | dtype | 1 | numbers_dtype_dnumber_idx | dtype,dnumber | 1 | ALTER TABLE public.numbers DROP INDEX numbers_dtype_idx
 public | number_callers | number_callers_dtype_idx | dtype | 1 | number_callers_dtype_dnumber_idx | dtype,dnumber | 1 | ALTER TABLE public.number_callers DROP INDEX number_callers_dtype_idx
 public | prefixes | prefixes_customer_idx | customer | 1 | prefixes_customer_dtype_dnumber_idx | customer,dtype,dnumber | 1 | ALTER TABLE public.prefixes DROP INDEX prefixes_customer_idx
 public | number_times | number_times_dtype_idx | dtype | 1 | number_times_dtype_dnumber_idx | dtype,dnumber | 1 | ALTER TABLE public.number_times DROP INDEX number_times_dtype_idx
 public | phones | phones_customer_idx | customer | 1 | phones_customer_callerid_location_idx | customer,callerid_location | 1 | ALTER TABLE public.phones DROP INDEX phones_customer_idx
 public | phones | phones_customer_idx | customer | 1 | phones_customer_callerid_external_idx | customer,callerid_external | 1 | ALTER TABLE public.phones DROP INDEX phones_customer_idx
 public | phones | phones_customer_idx | customer | 1 | phones_customer_callerid_internal_idx | customer,callerid_internal | 1 | ALTER TABLE public.phones DROP INDEX phones_customer_idx
 public | phones_hardware | phones_hardware_phone_idx | phone | 1 | phones_hardware_phone_hardware_address_idx | phone,hardware_address | 0 | ALTER TABLE public.phones_hardware DROP INDEX phones_hardware_phone_idx
 public | speeddials | speeddials_stype_idx | stype | 1 | speeddials_stype_snumber_idx | stype,snumber | 1 | ALTER TABLE public.speeddials DROP INDEX speeddials_stype_idx
 public | speeddials | speeddials_dtype_idx | dtype | 1 | speeddials_dtype_dnumber_idx | dtype,dnumber | 1 | ALTER TABLE public.speeddials DROP INDEX speeddials_dtype_idx
 public | mailbox_destinations | mailbox_destinations_context_mailbox_idx | context,mailbox | 1 | mailbox_destinations_pkey | context,mailbox,dcustomer,dtype,dnumber | 0 | ALTER TABLE public.mailbox_destinations DROP INDEX mailbox_destinations_context_mailbox_idx
 public | outgroup_times | outgroup_times_outgroup_idx | outgroup | 1 | outgroup_times_outgroup_name_idx | outgroup,name | 0 | ALTER TABLE public.outgroup_times DROP INDEX outgroup_times_outgroup_idx
 public | ingroup_times | ingroup_times_ingroup_idx | ingroup | 1 | ingroup_times_ingroup_name_idx | ingroup,name | 0 | ALTER TABLE public.ingroup_times DROP INDEX ingroup_times_ingroup_idx
 public | active_customers | active_customers_uniqueid_idx | uniqueid | 1 | active_customers_pkey | uniqueid,scustomer | 0 | ALTER TABLE public.active_customers DROP INDEX active_customers_uniqueid_idx
 public | access | access_customer_idx | customer | 1 | access_customer_callerid_external_idx | customer,callerid_external | 1 | ALTER TABLE public.access DROP INDEX access_customer_idx
 public | access | access_customer_idx | customer | 1 | access_customer_callerid_internal_idx | customer,callerid_internal | 1 | ALTER TABLE public.access DROP INDEX access_customer_idx
 public | unlimited_access | unlimited_access_customer_idx | customer | 1 | unlimited_access_customer_callerid_external_idx | customer,callerid_external | 1 | ALTER TABLE public.unlimited_access DROP INDEX unlimited_access_customer_idx
 public | unlimited_access | unlimited_access_customer_idx | customer | 1 | unlimited_access_customer_callerid_internal_idx | customer,callerid_internal | 1 | ALTER TABLE public.unlimited_access DROP INDEX unlimited_access_customer_idx
 public | texts | texts_dcustomer_idx | dcustomer | 1 | texts_dcustomer_dtype_dnumber_idx | dcustomer,dtype,dnumber | 1 | ALTER TABLE public.texts DROP INDEX texts_dcustomer_idx
 public | texts_media | texts_media_uniqueid_idx | uniqueid | 1 | texts_media_pkey | uniqueid,filename | 0 | ALTER TABLE public.texts_media DROP INDEX texts_media_uniqueid_idx
 public | number_calleridgroups | number_calleridgroups_dtype_idx | dtype | 1 | number_calleridgroups_dtype_dnumber_idx | dtype,dnumber | 1 | ALTER TABLE public.number_calleridgroups DROP INDEX number_calleridgroups_dtype_idx
 public | analytics_include | analytics_i | analytics | 1 | analytics_include_pkey | analytics,feature,dtype,dnumber | 0 | ALTER TABLE public.analytics_include DROP INDEX analytics_i
 public | archived | dupl1 | category_id | 1 | dupl2 | category_id | 1 | ALTER TABLE public.archived DROP INDEX dupl1
 public | archived | dupl2 | category_id | 1 | dupl1 | category_id | 1 | ALTER TABLE public.archived DROP INDEX dupl2
(27 rows)

Source: Duplicate and redundant indices

Unused indexes

SQL> SELECT relid::regclass AS table, indexrelid::regclass AS index
 , pg_size_pretty(pg_relation_size(indexrelid::regclass)) AS index_size
 , idx_tup_read, idx_tup_fetch, idx_scan
 FROM pg_stat_user_indexes 
 JOIN pg_index USING (indexrelid) 
 WHERE idx_scan = 0 
 AND indisunique IS FALSE
;
 table | index | index_size | idx_tup_read | idx_tup_fetch | idx_scan 
------------------------+-----------------------------------------------------------------+------------+--------------+---------------+----------
 customers | customers_prefix_idx | 16 kB | 0 | 0 | 0
 customers | customers_parent_idx | 16 kB | 0 | 0 | 0
 customers | customers_email_idx | 16 kB | 0 | 0 | 0
 customers | customers_affiliate_customer_idx | 16 kB | 0 | 0 | 0
 customers | customers_bill_ref_idx | 16 kB | 0 | 0 | 0
...
 analytics_include | analytics_i | 8192 bytes | 0 | 0 | 0
 archived | dupl1 | 8192 bytes | 0 | 0 | 0
 archived | dupl2 | 8192 bytes | 0 | 0 | 0
(413 rows)

Sources:

PG Assistant

At the Swiss PGDay2026(s), Bertrand Hartwig presented his tool PG Assistant. In that context, I wanted to try it out right away…

PG Assistant was able to find missing Primary Keys and duplicate indexes. It didn’t show me any partially redundant or unused indexes, but that might just be on my end…

PG Assistant: Dashboard / Dev advisor

PG Assistant: Global Advisor / Dev advisor

PG Assistant: Strictly duplicate unused index

Intallation of PG Assistant

$ apt update
$ apt install python3 python3.13-venv unzip pip
$ wget https://github.com/beh74/pgassistant-community/archive/refs/heads/main.zip
$ unzip main.zip 
$ cd pgassistant-community-main/
$ python3 -m venv env
$ source env/bin/activate
$ pip3 install -r requirements.txt
$ export FLASK_APP=run.py
$ flask run --host=0.0.0.0 --port=80

Then connect to the displayed URL using a web browser.

A user must be created in the database first:

SQL> CREATE ROLE pgassistant WITH LOGIN SUPERUSER PASSWORD 'secret';

and the pg_hba.conf file must be adapted.

Addendum

You can find unused indexes using the PG Assistant as follows: Database Objects ➜ Indexes ➜ Status: Unused ➜ “NO INDEX ACTIVITY”

PG Assistant: Unused Indexes

MariaDB/MySQL Environment MyEnv 3.0.0 has been released

Mon, 23 Mar 2026 17:42:00 +0100

FromDual has the pleasure to announce the release of the new version 3.0.0 of its popular MariaDB, MySQL and PostgreSQL multi-instance environment MyEnv.

The new MyEnv can be downloaded here. How to install MyEnv is described in the MyEnv Installation Guide.

In the inconceivable case that you find a bug in the MyEnv please report it to us by sending an email.

Any feedback, statements and testimonials are welcome as well! Please send them to us.

Upgrade from 2.x to 3.0

Please check the MyEnv Installation Guide.

Changes in MyEnv 3.0.0

MyEnv

Template warning improved.
Distro version in --version added.
Check MyEnv configuration permissions.
#fd increased for MyEnv.
myenv.conf should have more secure permissions now.
Situation caught when my.cnf is missing in myenv.conf.
Directories home and run moved to dba and myenv.
Unit file mariadb.service and mysql.service replaced by dba.service.
User mysql replaced by dba in template.
start_stop fixed warning in case argv[1] is missing.
sys_uid filter fixed for Rocky Linux.
dba unit file added to package.
Nagios plugins detection removed from showMyEnvVersion.
Old SysV init files removed and replaced by Systemd unit files.
dba user was introduced and check for system user added.
User dba changed an cosmetic fixes.

MyEnv Installer

2 concurrent installMyEnv versions cannot run any more.
Directroy binlog, cgroups and angel removed from postgresql type installation.
Wrapper script installMyEnv.sh removed.
Cosmetics fixed in installer.
Installation made more mysql friendly.
libaio1t64 considered during installation recommendations on DEB systems.
Next free port suggestion during installMyEnv improved. It will suggest the first free port now.
apt-get and yum replaced by apt and dnf.

MyEnv Utilities

insert_test.sh made PostgreSQL ready.

PostgreSQL

PostgreSQL instance is stopped with fast instead of immediate now.
show_create_table.sh for PostgreSQL made nicer.
PostgreSQL status.sql added.
Minor fixes for PostgreSQL.

General

CHANGELOG updated.
rc made unique.
Minor bugs fixed.
Copyright year updated from 2024 to 2026.
mkdir changed from /bin to /usr/bin which is the new/right standard on all our 3 supported distributions.

Documentation

README updated.
Installation documentation improved, library related stuff documented.
PostgreSQL added to documentation.
lsb_release removed from documentation.
Documentation restructured.
Documentation made more resilient agaist errors.
Documentation process improved.
Documentation moved completely to asciidoc.

Packaging

Package list completed.
Makefile fixed.
Old distro stuff and initV stuff removed.
Build scripts fixed.

For subscriptions of commercial use of MyEnv please get in contact with us.

FromDual Performance Monitor 2.2.1 has been released

Thu, 19 Feb 2026 18:18:00 +0100

FromDual has the pleasure to announce the release of the new version 2.2.1 of its popular Database Performance Monitor for MariaDB, Galera Cluster, MySQL and PostgreSQL fpmmm.

The FromDual Performance Monitor enables Database and System Administrators to monitor and understand what is going on inside their databases and on the machines where the databases reside.

More information you can find here: FromDual Performance Monitor.

Download

The new FromDual Performance Monitor can be downloaded from our Sofware Download page or you can use our repositories. How to install and use the FromDual Performance Monitor is documented in the Documentation.

In the inconceivable case that you find a bug in the FromDual Performance Monitor please report it to us by sending an email.

Any feedback, statements and testimonials are welcome as well! Please send them to us.

Monitoring as a Service (MaaS)

You do not want to set-up your database monitoring yourself? No problem: Choose our Monitoring as a Service (MaaS) to safe time and costs!

Installation of Performance Monitor 2.2.1

How to install the FromDual Performance Monitor you can find in the Installation Guide.

Upgrade of fpmmm tar ball from 1.x to 2.2.1

There are some changes in the configuration file (fpmmm.conf):

The access rights should be change as follows: chmod 600 /etc/fpmmm.conf
The key Methode was spelled wrong in the configuration file. It was renamed to Method.
The key PidFile is ambiguous which could lead to problems and bugs. Thus it was changed to either MyPidFile for fpmmm and DbPidFile for the database.

Upgrade with DEB/RPM packages should happen automatically. For tar balls follow this instruction:

$ cd /opt
$ tar xf /download/fpmmm-2.2.1.tar.gz
$ rm -f fpmmm
$ ln -s fpmmm-2.2.1 fpmmm

Changes in FromDual Performance Monitor 2.2.1

These release notes include both the changes that came with version 2.2.0 and version 2.2.1.

This release contains new features and various bug fixes.

You can verify your current FromDual Performance Monitor version with the following command:

$ /opt/fpmmm/bin/fpmmm --version

General

Updated to latest myEnv library.
PHP 8.5 incompatibilities fixed.
Typos fixed.
Error messages improved.
Function real_connect warnings send to console are suppressed now.
Connection problems timeout reduced so in case of troubles we should see more and earlier…
Other cosmetic errors and debugging information fixed.
Data are gathered and set to zero even thought database is not reachable.
Indention of logged messages fixed.
Function exit is logged now as well.
SSL connection handling added.
Fix of error: array_sum(): Addition is not supported on type string in warning after upgrade to Ubuntu 24.04/PHP 8.3.
Error log parsing had problems with huge error logs. Now we have added a size barrier.
Function getDistributions updated/cleaned-up.
Command lsb_release removed.
Documentation added.
Nagios: Tests fixed for MariaDB 11.8.

Templates

Server: Available I/O system information added to each I/O system on top, pages named.
InnoDB: Pages named, row write operations graph added.
MySQL: Some graphs and query dashboard made nicer.

Agent

none

Server

Items FromDual.MySQL.server.disk.avg_io_read_wait and FromDual.MySQL.server.disk.avg_io_write_wait removed because they are showing completely wrong values. Use FromDual.MySQL.server.disk.r_await and FromDual.MySQL.server.disk.w_await instead.
Workaround for missing cpuinfo old cachefile implemented.

Galera

Old style variable fixed which causes problems with newer version.
Default values on database stop added.
Workaround for cut wsrep_provider_options bug in MySQL Galera Cluster added.

InnoDB

Variable innodb_log_file_size made consistent for MariaDB and MySQL.
Deprecated and removed variable innodb_log_files_in_group removed.
Fix for innodb_log_file_size in MySQL 9.4.
Log occupancy graph added and graph added to dashboard.
Variable tx_isolation replaced by transaction isolation which is deprecated in MariaDB 11.2 and MySQL 5.7.

MySQL

Variable vendor_versions_behind special case caught.
Connection charset changed from utf8 to utf8mb4 due to errors in MariaDB 11.8.
Template pages named.

Process

none

Security

Module improved for new behaviour in MariaDB 11.8.

Master

Wrong version check for master fixed.

Slave

Slave lagging problem fixed.
Wrong version check for slave fixed.
MySQL 8.4 commands added for replication monitoring.

Backup

none

PostgreSQL

Rudimentary PostgreSQL monitoring added.

Packaging

RHEL 8 added again.
RPM spec adapted for RHEL 10.
SNMP library updated.
Debian 10 and RHEL 7 removed.
DEB sign stuff added.

For subscriptions of commercial use of fpmmm please get in contact with us.

What is the quickest way to load data into the database?

Wed, 11 Feb 2026 10:04:00 +0100

We had some really exciting problems to solve for the last customer! Especially because the database wasn’t exactly small.

Here are some key data: CPU: 2 sockets x 24 cores x 2 threads = 96 vCores, 756 G RAM, 2 x 10 Tbyte PCIe SSD in RAID-10 and 7 Tbyte data, several thousand clients, rapidly growing.

The current throughput: 1 M SELECT/min, 56 k INSERT/min, 44 k UPDATE/min, 7 k DELETE/min averaged over 30 days. With a strong upward trend. Application and queries not consistently optimised. Database configuration: ‘state of the art’ not verified with benchmarks. CPU utilisation approx. 50% on average, more at peak times. I/O system still has available resources.

The customer collects position and other device data and stores it in the database. In other words, a classic IoT problem (with time series, index clustered table, etc.).

The question he has asked is: What is the fastest way to copy data from one table (pending data, a kind of queue) to another table (final data, per client)?

The data flow looks something like this:

+------------+
| IoT Device |--+
+------------+ \
 \ +-----+
+------------+ \ | AS | +--------------+ Processing +------------+
| IoT Device |------+-->| |-->| Pending data |------------->| Final data |
+------------+ / | 400 | +--------------+ of data +------------+
 / +-----+
+------------+ /
| IoT Device |--+
+------------+

3 different variants to copy the data were available for selection.

Variant 1: `INSERT` and `DELETE` (simplest form)

The simplest variant is a simple INSERT and DELETE. This variant is particularly problematic because MariaDB/MySQL and PostgreSQL have AUTOCOMMIT enabled by default (here, here, here and here).

To help you visualise this a little better, here is some pseudocode:

// 20k rows
for (i = 1; i <= 2000; i++) {

 SELECT * FROM pending LIMIT 10;
 foreach ( row ) {
 INSERT INTO final;
 -- implicit COMMIT
 DELETE FROM pending WHERE id = row[id];
 -- implicit COMMIT
 }
}

So if we want to copy 20 k rows, this variant causes: 40 k COMMITs (fsync) and 42 k network round trips!

Variant 2: `START TRANSACTION` and `INSERT` ad `DELETE`

This variant is used by more experienced database developers. Here is the corresponding pseudocode:

// 20k rows
for (i = 1; i <= 2000; i++) {

 SELECT * FROM pending LIMIT 10;
 START TRANSACTION;
 foreach ( row ) {
 INSERT INTO final;
 DELETE FROM pending WHERE id = row[id];
 }
 COMMIT;
}

If we want to copy 20 k rows in this example, this variant only causes 2 k COMMITs (fsync)! So 20 times less! But 46 k network round trips (10% more).

Variant 3: `START TRANSACTION` and optimised `INSERT` and `DELETE`

This variant is a little more demanding in terms of programming. It is used if you want to get a little closer to the limits of what is possible. Here is the pseudo code:

// 20k rows
for (i = 1; i <= 2000; i++) {

 SELECT * FROM pending LIMIT 10;
 START TRANSACTION;
 INSERT INTO final (), (), (), (), (), (), (), (), (), ();
 DELETE FROM pending WHERE id = IN (...);
 COMMIT;
}

And this 3rd variant also only causes 2 k COMMIT’s (fsync) with 20 k rows, but saves the loop via the INSERT and DELETE statements in the database. So on the one hand we save network round trips (only 10 k) and CPU cycles on the database (which are difficult to scale) for parsing the queries.

Test set-up

To test the whole thing, we have prepared a small script: load_data.php

Preparation and execution with MariaDB/MySQL

You can execute these tests yourself with the following commands:

SQL> CREATE DATABASE test;
SQL> CREATE USER 'app'@'127.0.0.1' IDENTIFIED BY 'secret';
SQL> GRANT ALL ON *.* TO 'app'@'127.0.0.1';

$ ./load_data.php --database-type=mysql --database=test --host=127.0.0.1 --port=3306 --user=app --password=secret --prepare

$ for i in $(seq 5) ; do
 ./load_data.php --database-type=mysql --database=test --host=127.0.0.1 --port=3306 --user=app --password=secret --run --variant=1
 ./load_data.php --database-type=mysql --database=test --host=127.0.0.1 --port=3306 --user=app --password=secret --run --variant=2
 ./load_data.php --database-type=mysql --database=test --host=127.0.0.1 --port=3306 --user=app --password=secret --run --variant=3
done

$ ./load_data.php --database-type=mysql --database=test --host=127.0.0.1 --port=3306 --user=app --password=secret --clean-up

Everyone can work out the measured values themselves with the corresponding test script.

Preparation and execution with PostgreSQL

You can execute these tests yourself with the following commands:

postgres# CREATE DATABASE test;
postgres# CREATE USER app PASSWORD 'secret';
postgres# GRANT ALL ON DATABASE test TO app;
postgres# GRANT ALL ON SCHEMA public TO app;

$ ./load_data.php --database-type=postgresql --database=test --host=127.0.0.1 --port=5432 --user=app --password=secret --prepare

$ for i in $(seq 5) ; do
 ./load_data.php --database-type=postgresql --database=test --host=127.0.0.1 --port=5432 --user=app --password=secret --run --variant=1
 ./load_data.php --database-type=postgresql --database=test --host=127.0.0.1 --port=5432 --user=app --password=secret --run --variant=2
 ./load_data.php --database-type=postgresql --database=test --host=127.0.0.1 --port=5432 --user=app --password=secret --run --variant=3
done

$ ./load_data.php --database-type=postgresql --database=test --host=127.0.0.1 --port=5432 --user=app --password=secret --clean-up

Anyone can work out the measured values themselves with the corresponding test script.

Results

To avoid unnecessary discussions, we have ‘only’ listed the relative performance (runtime) here, as MarkC has been doing recently. We are happy to provide our measured values bilaterally. However, they can be easily reproduced with the test script itself.

Less is better:

	Variant 1	Variant 2	Variant 3
MariaDB 11.8, avg(5)	100.0%	7.5%	6.2%
PostgreSQL 19dev, avg(5)	100.0%	11.2%	7.0%

Attention: The values of MariaDB/MySQL and PostgreSQL can NOT be compared directly!

And here is the graphical evaluation:

Remarks

With faster discs, the difference between 1 and 2/3 would probably not have been quite so significant. Customer tests have shown a difference of ‘only’ about a factor of 5 (instead of a factor of 9 to 16).

There are certainly other ways in which this loading process can be optimised. Here are a few that come to mind:

INSERT INTO ... SELECT * FROM
LOAD DATA INFILE/COPY, if possible
Prepared statements
Server side Stored Language (SQL/PSM, PL/pgSQL, …) :-(
PDO-Fetch of the results?
etc.

Maybe I should look for a profiler (Xdebug or xhprof)?

Further contributions

This page was translated using deepl.com.

How much space does NULL need?

Sun, 08 Feb 2026 16:15:00 +0100

The last time I consulted a customer, he came up to me beaming with joy and said that he had taken my advice and changed all the primary key columns from BIGINT (8 bytes) to INT (4 bytes) and that had made a big difference! His MySQL 8.4 database is now 750 Gbyte smaller (from 5.5 Tbyte). Nice!

And yes, I know that contradicts the recommendations of some of my PostgreSQL colleagues (here and here). In the MySQL world, more emphasis is placed on such things (source):

Use the most efficient (smallest) data types possible. MySQL has many specialized types that save disk space and memory. For example, use the smaller integer types if possible to get smaller tables

Also, InnoDB works a wee bit differently (index clustered table and primary key in all secondary keys) than PostgreSQL (heap table, indices with row pointer (ctid)).

But that’s not really the issue. Immediately afterwards, he asked me whether the deletion of columns of type DOUBLE (8 bytes, in PostgreSQL-speak DOUBLE PRECISION) would also save space or whether he should rather drop the columns straight away. My first reflex response to DOUBLE was: NULL is good, followed by OPTIMIZE TABLE (VACUUM FULL in PostgreSQL parlance). But the second thought was, DOUBLE is a data type of fixed length, does NULL also apply there or only for data types with variable length? Caution is the mother of the porcelain box! Love to consult the manual first…

And there it says (source):

Declare columns to be NOT NULL if possible. It makes SQL operations faster, by enabling better use of indexes and eliminating overhead for testing whether each value is NULL. You also save some storage space, one bit per column. If you really need NULL values in your tables, use them. Just avoid the default setting that allows NULL values in every column.

and (source):

The variable-length part of the record header contains a bit vector for indicating NULL columns. … Columns that are NULL do not occupy space other than the bit in this vector. The variable-length part of the header also contains the lengths of variable-length columns. Each length takes one or two bytes, depending on the maximum length of the column. If all columns in the index are NOT NULL and have a fixed length, the record header has no variable-length part.

Experiment with MariaDB/MySQL

Test setup

Somehow the description is a bit too complicated for me. Perhaps a small sketch would help? So let’s give it a try:

SQL> -- DROP TABLE IF EXISTS tracking;

SQL> CREATE TABLE tracking (
 id INT UNSIGNED NOT NULL PRIMARY KEY AUTO_INCREMENT
, d0 DOUBLE, d1 DOUBLE, d2 DOUBLE, d3 DOUBLE, d4 DOUBLE
, d5 DOUBLE, d6 DOUBLE, d7 DOUBLE, d8 DOUBLE, d9 DOUBLE
);

SQL> INSERT INTO tracking SELECT NULL, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0;
SQL> INSERT INTO tracking SELECT NULL, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 FROM tracking;
... bis 16 M rows

The table is approx. 1.8 Gbyte in size for both MariaDB and MySQL with 16 M rows. Since this information is only given very imprecisely in INFORMATION_SCHEMA, let’s take a look at the file system:

MariaDB 11.8:

SQL> system ls -l tracking.ibd
-rw-rw---- 1 mysql mysql 1206 Feb 7 10:28 tracking.frm
-rw-rw---- 1 mysql mysql 1933574144 Feb 7 10:32 tracking.ibd

MySQL 8.4:

SQL> system ls -l tracking.ibd
-rw-r----- 1 mysql mysql 1929379840 Feb 7 10:33 tracking.ibd

Defragment the table

Then we ‘defragment’ the table with the OPTIMIZE TABLE command:

SQL> OPTIMIZE TABLE tracking;
+---------------+----------+----------+-------------------------------------------------------------------+
| Table | Op | Msg_type | Msg_text |
+---------------+----------+----------+-------------------------------------------------------------------+
| test.tracking | optimize | note | Table does not support optimize, doing recreate + analyze instead |
| test.tracking | optimize | status | OK |
+---------------+----------+----------+-------------------------------------------------------------------+

Attention: The table is copied once! It therefore needs twice the amount of disc space for a short time! This can be observed while the OPTIMIZE TABLE command is running:

MariaDB:

$ watch -d -n 1 'ls -l trac* \#*'
-rw-rw---- 1 mysql mysql 1206 Feb 7 10:39 '#sql-alter-d57-8c.frm'
-rw-rw---- 1 mysql mysql 968884224 Feb 7 10:39 '#sql-alter-d57-8c.ibd'
-rw-rw---- 1 mysql mysql 1206 Feb 7 10:28 tracking.frm
-rw-rw---- 1 mysql mysql 1933574144 Feb 7 10:32 tracking.ibd

MySQL:

$ watch -d -n 1 'ls -l trac* \#*'
-rw-r----- 1 mysql mysql 369098752 Feb 7 10:40 #sql-ib1594-4164062678.ibd
-rw-r----- 1 mysql mysql 1929379840 Feb 7 10:33 tracking.ibd

The result is amazing! With MariaDB, the table has remained somewhat the same size:

-rw-rw---- 1 mysql mysql 1206 Feb 7 10:39 tracking.frm
-rw-rw---- 1 mysql mysql 1912602624 Feb 7 10:39 tracking.ibd

With MySQL, on the other hand, the table has actually grown after the ‘defragmentation’, namely by approx. 14%:

-rw-r----- 1 mysql mysql 2197815296 Feb 7 10:41 tracking.ibd

If we execute the OPTIMIZE TABLE command again, the size remains constant for both MariaDB and MySQL:

MariaDB:

-rw-rw---- 1 mysql mysql 1206 Feb 7 10:46 tracking.frm
-rw-rw---- 1 mysql mysql 1912602624 Feb 7 10:48 tracking.ibd

MySQL:

-rw-r----- 1 mysql mysql 2197815296 Feb 7 10:48 tracking.ibd

Attempt 1: `NULL` out

Now we NULL out the values:

SQL> UPDATE tracking
SET d0 = NULL, d1 = NULL, d2 = NULL, d3 = NULL, d4 = NULL
 , d5 = NULL, d6 = NULL, d7 = NULL, d8 = NULL, d9 = NULL
;

After this step, the sizes of the files have even grown slightly:

MariaDB (+1.3%):

-rw-rw---- 1 mysql mysql 1206 Feb 7 10:49 tracking.frm
-rw-rw---- 1 mysql mysql 1937768448 Feb 7 11:04 tracking.ibd

MySQL (+0.2%):

-rw-r----- 1 mysql mysql 2202009600 Feb 7 11:04 tracking.ibd

We then defragment the table again with the OPTIMIZE TABLE command. The tables shrink as expected.

MariaDB (to 23%):

-rw-rw---- 1 mysql mysql 1206 Feb 7 11:09 tracking.frm
-rw-rw---- 1 mysql mysql 448790528 Feb 7 11:10 tracking.ibd

MySQL (to 24%):

-rw-r----- 1 mysql mysql 520093696 Feb 7 11:10 tracking.ibd

OPTIMIZE TABLE again does NOT change the file size any more…

Attempt 2: Deleting the columns

Now we try the whole thing again with the DROP COLUMN command. The starting position is again the same as described above:

MariaDB:

-rw-rw---- 1 mysql mysql 1206 Feb 7 11:15 tracking.frm
-rw-rw---- 1 mysql mysql 1933574144 Feb 7 11:18 tracking.ibd

MySQL:

-rw-r----- 1 mysql mysql 1929379840 Feb 7 11:19 tracking.ibd

After the OPTIMIZE TABLE command, the values look similar to the first attempt:

MariaDB:

-rw-rw---- 1 mysql mysql 1206 Feb 7 11:20 tracking.frm
-rw-rw---- 1 mysql mysql 1912602624 Feb 7 11:21 tracking.ibd

MySQL:

-rw-r----- 1 mysql mysql 2197815296 Feb 7 11:21 tracking.ibd

OPTIMIZE TABLE again also brings no further changes, as above:

MariaDB:

-rw-rw---- 1 mysql mysql 1206 Feb 7 11:22 tracking.frm
-rw-rw---- 1 mysql mysql 1912602624 Feb 7 11:23 tracking.ibd

MySQL:

-rw-r----- 1 mysql mysql 2197815296 Feb 7 11:24 tracking.ibd

And now the actual second attempt with dropping the columns:

SQL> ALTER TABLE tracking
 DROP COLUMN d0, DROP COLUMN d1, DROP COLUMN d2, DROP COLUMN d3, DROP COLUMN d4
, DROP COLUMN d5, DROP COLUMN d6, DROP COLUMN d7, DROP COLUMN d8, DROP COLUMN d9
;

The first thing we notice is that the command is INSTANTANEOUS, i.e. it does not make any changes to the data but only changes the metadata. On the one hand, this is good, as it minimises the impact on the application. On the other hand, it also means that no space is saved.

So let’s get to grips with the whole thing again with the OPTIMIZE TABLE command:

MariaDB (to 93%):

-rw-rw---- 1 mysql mysql 925 Feb 7 11:28 tracking.frm
-rw-rw---- 1 mysql mysql 415236096 Feb 7 11:29 tracking.ibd

MySQL (to 92%):

-rw-r----- 1 mysql mysql 478150656 Feb 7 11:28 tracking.ibd

Conclusion

Both, dropping the columns and the NULL out of columns save a significant amount of space. Dropping the columns saves about 7% more space than NULL them out. If it is possible from an application point of view, you should therefore drop columns that are no longer required, or if not possible, at least NULL them out.

Experiment with PostgreSQL

And now let’s take a look at the whole thing with PostgreSQL 19devel.

Test setup

The test setup is analogous to MariaDB/MySQL:

postgres=# -- DROP TABLE IF EXISTS tracking;

postgres=# CREATE TABLE tracking (
 id SERIAL PRIMARY KEY
, d0 DOUBLE PRECISION, d1 DOUBLE PRECISION, d2 DOUBLE PRECISION, d3 DOUBLE PRECISION, d4 DOUBLE PRECISION
, d5 DOUBLE PRECISION, d6 DOUBLE PRECISION, d7 DOUBLE PRECISION, d8 DOUBLE PRECISION, d9 DOUBLE PRECISION
);

postgres=# \timing

postgres=# INSERT INTO tracking (d0, d1, d2, d3, d4, d5, d6, d7, d8, d9)
 SELECT 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0;
postgres=# INSERT INTO tracking (d0, d1, d2, d3, d4, d5, d6, d7, d8, d9)
 SELECT 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 FROM tracking;
... bis 16 M rows

Firstly, we want to know how big the table has actually become. PostgreSQL seems to know this information very precisely:

postgres=# SELECT pg_relation_size('tracking') AS tab_siz
 , pg_size_pretty(pg_relation_size('tracking')) AS tab_siz_prtty
 , pg_indexes_size('tracking') AS idx_siz
 , pg_size_pretty(pg_indexes_size('tracking')) AS idx_siz_prtty
 , pg_relation_size('tracking') + pg_indexes_size('tracking') AS tab_and_idx_siz
 , pg_size_pretty(pg_relation_size('tracking') + pg_indexes_size('tracking')) AS tab_and_idx_siz_prtty
 , pg_total_relation_size('tracking') AS tot_rel_siz
 , pg_size_pretty(pg_total_relation_size('tracking')) AS tot_rel_siz_prtty
;
 tab_siz | tab_siz_prtty | idx_siz | idx_siz_prtty | tab_and_idx_siz | tab_and_idx_siz_prtty | tot_rel_siz | tot_rel_siz_prtty
------------+---------------+-----------+---------------+-----------------+-----------------------+-------------+-------------------
 1963417600 | 1872 MB | 376856576 | 359 MB | 2340274176 | 2232 MB | 2340798464 | 2232 MB

Then we want to know where these files can be found in the file system:

postgres=# SELECT oid AS db_oid FROM pg_database WHERE datname = current_database();
 db_oid
--------
 5

postgres=# SELECT oid AS table_oid, relname, relnamespace, relfilenode
 FROM pg_class WHERE relname = 'tracking';
 table_oid | relname | relnamespace | relfilenode
-----------+----------+--------------+-------------
 40965 | tracking | 2200 | 40965

postgres=# SELECT i.indexrelid::regclass as index_name, i.indexrelid as index_oid
 FROM pg_index i
 JOIN pg_class c ON i.indrelid = c.oid
 WHERE c.relname = 'tracking';
 index_name | index_oid
---------------+-----------
 tracking_pkey | 40970

postgres=# SELECT pg_relation_filepath('tracking');
 pg_relation_filepath
----------------------
 base/5/40965

Table and index size in the file system:

$ ls -ltr 40965* 40970*
-rw------- 1 mysql mysql 40960 Feb 7 18:33 40965_vm
-rw------- 1 mysql mysql 499712 Feb 7 18:33 40965_fsm
-rw------- 1 mysql mysql 889675776 Feb 7 18:34 40965.1
-rw------- 1 mysql mysql 1073741824 Feb 7 18:34 40965
-rw------- 1 mysql mysql 376856576 Feb 7 18:35 40970

*_fsm means “free space map”
*_vm means “visibility map”
*.1 means 2nd segment of the object (table or index)

PostgreSQL seems to work with segments of 1 Gbyte by default and, unlike MariaDB/MySQL (INFORMATION_SCHEMA), knows exactly how large its files are. And the discrepancy from above (between tot_rel_siz and tab_and_idx_siz) can be explained by the fsm and vm files.

The PostgreSQL equivalent of the MariaDB/MySQL OPTIMIZE TABLE is the VACUUM FULL command:

postgres=# VACUUM FULL tracking;

$ ls -ltr
-rw------- 1 mysql mysql 40960 Feb 7 18:39 40965_vm
-rw------- 1 mysql mysql 499712 Feb 7 18:39 40965_fsm
-rw------- 1 mysql mysql 1073741824 Feb 7 18:39 40965
-rw------- 1 mysql mysql 889675776 Feb 7 18:39 40965.1
-rw------- 1 mysql mysql 1073741824 Feb 7 18:39 40972
-rw------- 1 mysql mysql 889675776 Feb 7 18:39 40972.1
-rw------- 1 mysql mysql 0 Feb 7 18:39 40975

...

-rw------- 1 mysql mysql 49152 Feb 7 18:39 2704
-rw------- 1 mysql mysql 32768 Feb 7 18:39 2703
-rw------- 1 mysql mysql 32768 Feb 7 18:39 2696
-rw------- 1 mysql mysql 65536 Feb 7 18:39 2674
-rw------- 1 mysql mysql 81920 Feb 7 18:39 2673
-rw------- 1 mysql mysql 98304 Feb 7 18:39 2659
-rw------- 1 mysql mysql 139264 Feb 7 18:39 2658
-rw------- 1 mysql mysql 24576 Feb 7 18:39 2619_fsm
-rw------- 1 mysql mysql 163840 Feb 7 18:39 2619
-rw------- 1 mysql mysql 106496 Feb 7 18:39 2608
-rw------- 1 mysql mysql 491520 Feb 7 18:39 1249
-rw------- 1 mysql mysql 122880 Feb 7 18:39 1247
-rw------- 1 mysql mysql 32768 Feb 7 18:39 2662
-rw------- 1 mysql mysql 114688 Feb 7 18:39 1259
-rw------- 1 mysql mysql 16384 Feb 7 18:39 3455
-rw------- 1 mysql mysql 49152 Feb 7 18:39 2663
-rw------- 1 mysql mysql 1073741824 Feb 7 18:39 40972
-rw------- 1 mysql mysql 889675776 Feb 7 18:39 40972.1
-rw------- 1 mysql mysql 376864768 Feb 7 18:39 40975
-rw------- 1 mysql mysql 0 Feb 7 18:39 40965
-rw------- 1 mysql mysql 0 Feb 7 18:39 40970

The first thing you notice is that PostgreSQL touches quite a few files and the ‘free space map’ file has disappeared. In contrast to MariaDB/MySQL, the table segments have remained the same size. You can also see that the old table has ‘disappeared’ (40965, 40970) and a new one has been created (40972 and 40975). The VACUUM FULL command in PostgreSQL also creates a copy of the data, as in MariaDB/MySQL.

postgres=# SELECT pg_relation_size('tracking') AS tab_siz
 , pg_size_pretty(pg_relation_size('tracking')) AS tab_siz_prtty
 , pg_indexes_size('tracking') AS idx_siz
 , pg_size_pretty(pg_indexes_size('tracking')) AS idx_siz_prtty
 , pg_relation_size('tracking') + pg_indexes_size('tracking') AS tab_and_idx_siz
 , pg_size_pretty(pg_relation_size('tracking') + pg_indexes_size('tracking')) AS tab_and_idx_siz_prtty
 , pg_total_relation_size('tracking') AS tot_rel_siz
 , pg_size_pretty(pg_total_relation_size('tracking')) AS tot_rel_siz_prtty
;
 tab_siz | tab_siz_prtty | idx_siz | idx_siz_prtty | tab_and_idx_siz | tab_and_idx_siz_prtty | tot_rel_siz | tot_rel_siz_prtty
------------+---------------+-----------+---------------+-----------------+-----------------------+-------------+-------------------
 1963417600 | 1872 MB | 376864768 | 359 MB | 2340282368 | 2232 MB | 2340282368 | 2232 MB

The following query helps to understand which other files/objects have been created:

postgres=# SELECT c.oid, c.relname, ns.nspname
FROM pg_class AS c
JOIN pg_namespace AS ns ON ns.oid = c.relnamespace
WHERE c.oid IN (2704, 2703, 2696, 2674, 2673, 2659, 2658, 2619, 2608, 1249, 1247, 40972, 2662, 1259, 3455, 2663, 40975, 40965, 40970)
;
 oid | relname | nspname
-------+-----------------------------------+------------
 40965 | tracking | public
 40970 | tracking_pkey | public
 2619 | pg_statistic | pg_catalog
 1247 | pg_type | pg_catalog
 2703 | pg_type_oid_index | pg_catalog
 2704 | pg_type_typname_nsp_index | pg_catalog
 2658 | pg_attribute_relid_attnam_index | pg_catalog
 2659 | pg_attribute_relid_attnum_index | pg_catalog
 2662 | pg_class_oid_index | pg_catalog
 2663 | pg_class_relname_nsp_index | pg_catalog
 3455 | pg_class_tblspc_relfilenode_index | pg_catalog
 2696 | pg_statistic_relid_att_inh_index | pg_catalog
 2673 | pg_depend_depender_index | pg_catalog
 2674 | pg_depend_reference_index | pg_catalog
 1249 | pg_attribute | pg_catalog
 1259 | pg_class | pg_catalog
 2608 | pg_depend | pg_catalog

postgres=# SELECT i.indexrelid::regclass as index_name, i.indexrelid as index_oid, ns.nspname
 FROM pg_index i
 JOIN pg_class c ON i.indrelid = c.oid
 JOIN pg_namespace AS ns ON ns.oid = c.relnamespace
 WHERE c.oid IN (2704, 2703, 2696, 2674, 2673, 2659, 2658, 2619, 2608, 1249, 1247, 40972, 2662, 1259, 3455, 2663, 40975, 40965, 40970)
;
 index_name | index_oid | nspname
-----------------------------------+-----------+------------
 pg_type_typname_nsp_index | 2704 | pg_catalog
 pg_attribute_relid_attnam_index | 2658 | pg_catalog
 tracking_pkey | 40970 | public
 pg_class_relname_nsp_index | 2663 | pg_catalog
 pg_class_tblspc_relfilenode_index | 3455 | pg_catalog
 pg_type_oid_index | 2703 | pg_catalog
 pg_attribute_relid_attnum_index | 2659 | pg_catalog
 pg_statistic_relid_att_inh_index | 2696 | pg_catalog
 pg_depend_depender_index | 2673 | pg_catalog
 pg_depend_reference_index | 2674 | pg_catalog
 pg_class_oid_index | 2662 | pg_catalog

Attempt 1: `NULL` out

Then we also NULL the columns in PostgreSQL. From here on, we save the view of the file system, as PostgreSQL seems to know the file sizes exactly, as we have seen above:

postgres=# UPDATE tracking
SET d0 = NULL, d1 = NULL, d2 = NULL, d3 = NULL, d4 = NULL
 , d5 = NULL, d6 = NULL, d7 = NULL, d8 = NULL, d9 = NULL
;

postgres=# SELECT pg_relation_size('tracking') AS tab_siz
 , pg_size_pretty(pg_relation_size('tracking')) AS tab_siz_prtty
 , pg_indexes_size('tracking') AS idx_siz
 , pg_size_pretty(pg_indexes_size('tracking')) AS idx_siz_prtty
 , pg_relation_size('tracking') + pg_indexes_size('tracking') AS tab_and_idx_siz
 , pg_size_pretty(pg_relation_size('tracking') + pg_indexes_size('tracking')) AS tab_and_idx_siz_prtty
 , pg_total_relation_size('tracking') AS tot_rel_siz
 , pg_size_pretty(pg_total_relation_size('tracking')) AS tot_rel_siz_prtty
;
 tab_siz | tab_siz_prtty | idx_siz | idx_siz_prtty | tab_and_idx_siz | tab_and_idx_siz_prtty | tot_rel_siz | tot_rel_siz_prtty
------------+---------------+-----------+---------------+-----------------+-----------------------+-------------+-------------------
 2695716864 | 2571 MB | 753696768 | 719 MB | 3449413632 | 3290 MB | 3450101760 | 3290 MB

Here we see that the table segments grow massively (+37%), which is called ‘bloat’ in PostgreSQL terminology. The MVCC implementation of PostgreSQL stores both the old and never new version of the row ‘in-place’ directly in the table, in contrast to MariaDB/MySQL which stores the old version in UNDO space and the new row ‘in-place’. The index file also increases significantly (+100%). We need to do more research to find out why this is the case. In addition, a ‘free space map’ is created again (difference between tot_rel_siz and tab_and_idx_siz).

A subsequent VACUUM FULL reduces the table (to 28%) and the index (to 50%) again in relation to the previous size:

postgres=# VACUUM FULL tracking;

postgres=# SELECT pg_relation_size('tracking') AS tab_siz
 , pg_size_pretty(pg_relation_size('tracking')) AS tab_siz_prtty
 , pg_indexes_size('tracking') AS idx_siz
 , pg_size_pretty(pg_indexes_size('tracking')) AS idx_siz_prtty
 , pg_relation_size('tracking') + pg_indexes_size('tracking') AS tab_and_idx_siz
 , pg_size_pretty(pg_relation_size('tracking') + pg_indexes_size('tracking')) AS tab_and_idx_siz_prtty
 , pg_total_relation_size('tracking') AS tot_rel_siz
 , pg_size_pretty(pg_total_relation_size('tracking')) AS tot_rel_siz_prtty
;
 tab_siz | tab_siz_prtty | idx_siz | idx_siz_prtty | tab_and_idx_siz | tab_and_idx_siz_prtty | tot_rel_siz | tot_rel_siz_prtty
-----------+---------------+-----------+---------------+-----------------+-----------------------+-------------+-------------------
 742916096 | 709 MB | 376864768 | 359 MB | 1119780864 | 1068 MB | 1119780864 | 1068 MB

and also in relation to the original size, the table (to 38%) and the index (to 100%) become smaller again. Why the index has remained the same size and only the table has shrunk remains to be investigated…

Experiment 2: Deleting the columns

The columns are then dropped with DROP COLUMN.

postgres=# ALTER TABLE tracking
 DROP COLUMN d0, DROP COLUMN d1, DROP COLUMN d2, DROP COLUMN d3, DROP COLUMN d4
, DROP COLUMN d5, DROP COLUMN d6, DROP COLUMN d7, DROP COLUMN d8, DROP COLUMN d9
;

As the response was immediate, it can be assumed that this operation is also instantaneous. Unfortunately, I couldn’t find anything about this in the PostgreSQL documentation.

Nothing has changed significantly in terms of size, which is actually to be expected with an instant operation. However, the fact that the size did not change after the VACUUM FULL command was a little surprising:

 tab_siz | tab_siz_prtty | idx_siz | idx_siz_prtty | tab_and_idx_siz | tab_and_idx_siz_prtty | tot_rel_siz | tot_rel_siz_prtty
-----------+---------------+-----------+---------------+-----------------+-----------------------+-------------+-------------------
 742916096 | 709 MB | 376864768 | 359 MB | 1119780864 | 1068 MB | 1120010240 | 1068 MB

postgres=# VACUUM FULL tracking;

 tab_siz | tab_siz_prtty | idx_siz | idx_siz_prtty | tab_and_idx_siz | tab_and_idx_siz_prtty | tot_rel_siz | tot_rel_siz_prtty
-----------+---------------+-----------+---------------+-----------------+-----------------------+-------------+-------------------
 742916096 | 709 MB | 376864768 | 359 MB | 1119780864 | 1068 MB | 1119780864 | 1068 MB

Remarks

Locking in PostgreSQL works as follows:

VACUUM Concurrent DML commands are possible similar to the MariaDB/MySQL OPTIMIZE TABLE command. However, the result is not quite the same.
VACUUM FULL causes an ACCESS EXCLUSIVE lock. Similar to the MariaDB/MySQL 5.5 and older OPTIMIZE TABLE command. DML and SELECT commands are NOT permitted.

Sources

Additional attempts

Instead of 0.0, NULL was filled into the columns d0 - d9. The table remained small (tot_rel_siz_prtty = 1068 MB). It is therefore also worth saving NULL instead of dummy values with PostgreSQL.
The columns d0 - d9 were created with DOUBLE PRECISION NOT NULL and the values 0.0 were filled. No effect: The table remained large (tot_rel_siz_prtty = 2232 MB).

This page was translated using deepl.com.

Someone is deleting my shared memory segments!

Sun, 08 Feb 2026 06:27:00 +0100

When we work with PostgreSQL under our myEnv, we regularly get shared memory segment errors. Example:

psql: error: connection to server on socket "/tmp/.s.PGSQL.5433" failed:
FATAL: could not open shared memory segment "/PostgreSQL.4220847662":
No such file or directory

or we see similar messages in the PostgreSQL error log:

ERROR: could not open shared memory segment "/PostgreSQL.4220847662":
No such file or directory

Because I am a MariaDB/MySQL admin, I am not very familiar with shared memory problems (MariaDB/MySQL does not work with shared memory). Fortunately, a search on the Internet led us on the right track (source). It is noted there:

The documentation of systemd states that this only happens for non-system users. Can you check whether your “postgres” user (or whatever you are using) is a system user?

Linux System User

First I had to find out what a system user under Linux actually is. I found an answer here: What’s the difference between a normal user and a system user?.

That is not a technical difference but an organizational decision. E.g. it makes sense to show normal users in a login dialog (so that you can click them instead of having to type the user name) but it wouldn’t to show system accounts (the UIDs under which daemons and other automatic processes run) there.

The LSB standard says: User ID Ranges:

The system User IDs from 0 to 99 should be statically allocated by the system, and shall not be created by applications.
The system User IDs from 100 to 499 should be reserved for dynamic allocation by system administrators and post install scripts using useradd.

On my Ubuntu system it looks like this:

$ grep SYS_ /etc/login.defs
#SYS_UID_MIN 100
#SYS_UID_MAX 999
#SYS_GID_MIN 100
#SYS_GID_MAX 999

This would be correct for the PostgreSQL user:

$ id postgres
uid=130(postgres) gid=142(postgres) groups=142(postgres),116(ssl-cert)

But since the PostgreSQL instance in question runs under our myEnv, that’s different:

$ id dba
uid=1001(dba) gid=1001(dba) groups=1001(dba)

So now we have two options:

We change SYS_UID_MAX and SYS_GID_MAX to 1001 (simple variant).
Or we change the UID and the GID of our user dba to less than 1000.

Simple variant: Change `SYS_UID_MAX` and `SYS_GID_MAX` to 1001

# /etc/login.defs
SYS_UID_MAX 1001
SYS_GID_MAX 1001

To be on the safe side, the machine was rebooted. But that did not help: After a short time, the same errors occur again.

More complicated variant: Changing the `UID` from 1001 to 990

$ cat /etc/passwd
...
polkitd:x:997:997:User for polkitd:/:/usr/sbin/nologin
systemd-coredump:x:998:998:systemd Core Dumper:/:/usr/sbin/nologin
tomcat:x:999:999:Apache Tomcat:/:/sbin/nologin
oli:x:1000:1000:Oli Sennhauser,,,:/home/oli:/bin/bash
dba:x:1001:1001:DBA user:/home/dba:/bin/bash
...

$ id dba
uid=1001(dba) gid=1001(dba) groups=1001(dba)

To do this, all processes of this user must be stopped!

$ usermod --uid 990 dba
$ groupmod --gid 990 dba

$ find / -user 1001 -exec chown --no-dereference dba {} \;
$ find / -group 1001 -exec chgrp --no-dereference dba {} \;

This seems to have solved the problem!

Additional information

A source mentioned above also recommended setting the following parameters in systemd-logind:

# /etc/systemd/system/systemd-logind.service.d/override.conf
RemoveIPC=no
RuntimeDirectorySize=1%

However, this measure is no longer necessary as it works without this change.

This page was translated using deepl.com.

Load CSV files into the database

Fri, 06 Feb 2026 18:04:00 +0100

Recently, I wanted to display the places of residence of the members of my club on a map for a personal gimmick (IGOC members). I knew the addresses of the club members. But not the coordinates of their places of residence.

So I went in search of the coordinates and found what I was looking for at the Federal Office of Topography (swisstopo).

The data is available there as a CSV file. Details here: Swiss town coordinates.

How do I load this data into a database?

Loading the data with MariaDB/MySQL

MariaDB and MySQL have the LOAD DATA INFILE command:

SQL> DROP TABLE IF EXISTS wgs84;

SQL> -- SET GLOBAL local_infile = ON; -- Only needed with MySQL

SQL> CREATE TABLE wgs84 (
 ortschaftsname VARCHAR(32)
, plz4 SMALLINT
, zusatzziffer SMALLINT
, zip_id SMALLINT UNSIGNED
, gemeindename VARCHAR(32)
, bfs_nr SMALLINT
, kantonskuerzel CHAR(2)
, adressenanteil varchar(8)
, e DOUBLE
, n DOUBLE
, sprache VARCHAR(8)
, validity VARCHAR(12)
);

SQL> -- TRUNCATE TABLE wgs84;

SQL> LOAD DATA LOCAL INFILE '/tmp/AMTOVZ_CSV_WGS84/AMTOVZ_CSV_WGS84.csv'
INTO TABLE wgs84
FIELDS TERMINATED BY ';'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
;
Query OK, 5713 rows affected
Records: 5713 Deleted: 0 Skipped: 0 Warnings: 0

You can then query the data in the database:

SQL> SELECT * FROM wgs84 ORDER BY ortschaftsname LIMIT 5;
+----------------+------+--------------+--------+--------------+--------+----------------+----------------+-------------------+--------------------+---------+------------+
| ortschaftsname | plz4 | zusatzziffer | zip_id | gemeindename | bfs_nr | kantonskuerzel | adressenanteil | e | n | sprache | validity |
+----------------+------+--------------+--------+--------------+--------+----------------+----------------+-------------------+--------------------+---------+------------+
| Aadorf | 8355 | 0 | 4672 | Aadorf | 4551 | TG | 96.802 % | 8.903193007810433 | 47.491079014637265 | de | 2008-07-01 |
| Aadorf | 8355 | 0 | 4672 | Elgg | 294 | ZH | 3.198 % | 8.89206766645808 | 47.4933781685032 | de | 2008-07-01 |
| Aarau | 5000 | 0 | 2913 | Aarau | 4001 | AG | 99.713 % | 8.048148371736266 | 47.38973523857376 | de | 2008-07-01 |
| Aarau | 5000 | 0 | 2913 | Suhr | 4012 | AG | 0.287 % | 8.059410934099922 | 47.383298214804334 | de | 2008-07-01 |
| Aarau | 5004 | 0 | 2932 | Aarau | 4001 | AG | 100 % | 8.060698546432551 | 47.400587704180744 | de | 2008-07-01 |
+----------------+------+--------------+--------+--------------+--------+----------------+----------------+-------------------+--------------------+---------+------------+
5 rows in set

Or something more precise:

SQL> SELECT ortschaftsname AS city, plz4 AS city_code, e AS lon, n AS lat
 FROM wgs84 WHERE plz4 IN (8280, 4663, 6043);
+-------------+-----------+-------------------+--------------------+
| city | city_code | lon | lat |
+-------------+-----------+-------------------+--------------------+
| Aarburg | 4663 | 7.904271716719409 | 47.321443418782955 |
| Aarburg | 4663 | 7.889249714098425 | 47.313536073562474 |
| Aarburg | 4663 | 7.880309179095798 | 47.31255194439023 |
| Adligenswil | 6043 | 8.364849060491428 | 47.07037816052481 |
| Kreuzlingen | 8280 | 9.173740257895282 | 47.64491046067056 |
| Kreuzlingen | 8280 | 9.159171428030783 | 47.654149879509134 |
| Kreuzlingen | 8280 | 9.204470741840725 | 47.639949130372145 |
+-------------+-----------+-------------------+--------------------+
7 rows in set (0.003 sec)

I will leave it to the reader to clean out the duplicates… :-)

So far so good, now to the finer points::

Differences between MariaDB and MySQL

The procedure described above works perfectly with MariaDB 11.4 and 11.8. There are small differences with MySQL 8.4:

The first error message that prevents loading is this one:

ERROR 3948 (42000): Loading local data is disabled; this must be enabled on both the client and server sides

It can be bypassed relatively easily with the command:

SQL> SET GLOBAL local_infile = ON;

The next attempt will fail as follows:

ERROR 2068 (HY000): LOAD DATA LOCAL INFILE file request rejected due to restrictions on access.

This problem can be solved by starting the MySQL client as follows:

$ mysql --local-infile=1 --user=root test

Sources

MariaDB: LOAD DATA INFILE
MySQL: LOAD DATA Statement

Loading the data with PostgreSQL

PostgreSQL has the command COPY ... FROM:

postgres=# DROP TABLE IF EXISTS wgs84;

postgres=# CREATE TABLE wgs84 (
 ortschaftsname VARCHAR(32)
, plz4 SMALLINT
, zusatzziffer SMALLINT
, zip_id INT
, gemeindename VARCHAR(32)
, bfs_nr SMALLINT
, kantonskuerzel CHAR(2)
, adressenanteil varchar(8)
, e DOUBLE PRECISION
, n DOUBLE PRECISION
, sprache VARCHAR(8)
, validity VARCHAR(12)
);

postgres=# -- TRUNCATE TABLE wgs84;

postgres=# COPY wgs84
FROM '/tmp/AMTOVZ_CSV_WGS84/AMTOVZ_CSV_WGS84.csv'
DELIMITER ';'
CSV HEADER
;
COPY 5713

Here, too, we receive the result as expected in the usual PostgreSQL form:

postgres=# SELECT ortschaftsname AS city, plz4 AS city_code, e AS lon, n AS lat
 FROM wgs84 WHERE plz4 IN (8280, 4663, 6043);
 city | city_code | lon | lat
-------------+-----------+-------------------+--------------------
 Aarburg | 4663 | 7.904271716719409 | 47.321443418782955
 Aarburg | 4663 | 7.889249714098425 | 47.313536073562474
 Aarburg | 4663 | 7.880309179095798 | 47.31255194439023
 Adligenswil | 6043 | 8.36487538940682 | 47.07037794822416
 Kreuzlingen | 8280 | 9.173740257895282 | 47.64491046067056
 Kreuzlingen | 8280 | 9.159171428030783 | 47.654149879509134
 Kreuzlingen | 8280 | 9.204470741840725 | 47.639949130372145
(7 rows)

Sources

PostgreSQL: COPY

Small differences between MariaDB/MySQL and PostgreSQL

Basically, the load command is completely different in the two database worlds.

With MariaDB and PostgreSQL, the commands run “out-of-the-box”. MySQL has two additional security hurdles built in here.

PostgreSQL does not recognise UNSIGNED integer data types, so the next largest data type (INT) must be used, which is a little less space-saving than with MariaDB/MySQL.

Remarks

When we did the same test a few days ago, there was still a loading error. So it seems that the data source has also changed slightly…

I have not found out quickly whether there is an SQL standard for these load commands and if so, whether MariaDB/MySQL or PostgreSQL are standard-compliant here.

And of course there are other ways to get your CSV data into the database…

The tools mariadb-import/mysqlimport are used if you want to do this from the command line. The CSV Storage Engine can also be misused for this purpose (see here for details). An officially supported variant is the MariaDB CONNECT Storage Engine with the CSV type (see here):

SQL> INSTALL SONAME 'ha_connect';

SQL> CREATE TABLE wgs84_fdw
ENGINE = CONNECT
table_type = CSV
file_name='/tmp/AMTOVZ_CSV_WGS84/AMTOVZ_CSV_WGS84.csv'
header = 1
sep_char = ';'
quoted = 0;

SQL> INSERT INTO wgs84 SELECT * FROM wgs84_fdw;

Unfortunately, it looks like the CONNECT Storage Engine will no longer be supported by MariaDB! And the mydumper/myloader tool also seems to be able to handle CSV files.

And of course the whole thing can also be solved using applications…

With PostgreSQL there are the following options:

postgres=# \copy wgs84 FROM '/tmp/AMTOVZ_CSV_WGS84/AMTOVZ_CSV_WGS84.csv' DELIMITER ';' CSV HEADER

then from the shell:

$ psql --user=dba -c "\copy wgs84 FROM '/tmp/AMTOVZ_CSV_WGS84/AMTOVZ_CSV_WGS84.csv' DELIMITER ';' CSV HEADER"

And the variant via the Foreign Data Wrapper (FWD). But I have not tried this:

postgres=# CREATE EXTENSION postgres_fdw;

postgres=# CREATE SERVER foreign_server
 FOREIGN DATA WRAPPER postgres_fdw
 OPTIONS (
 datasource 'CSV:/tmp/AMTOVZ_CSV_WGS84/AMTOVZ_CSV_WGS84.csv',
 format 'CSV'
 )
;

postgres=# CREATE USER MAPPING FOR local_user
 SERVER foreign_server
 OPTIONS (user 'foreign_user', password 'password')
;

postgres=# CREATE FOREIGN TABLE foreign_table (
 id integer NOT NULL,
 data text
)
 SERVER foreign_server
 OPTIONS (schema_name 'some_schema', table_name 'some_table')
;

Addendum

The MariaDB/MySQL data type DOUBLE is called DOUBLE PRECISION in ProsgreSQL.

This page was translated using deepl.com.

MariaDB/MySQL Environment MyEnv 2.1.1 has been released

Fri, 12 Dec 2025 17:43:00 +0100

FromDual has the pleasure to announce the release of the new version 2.1.1 of its popular MariaDB, Galera Cluster and MySQL multi-instance environment MyEnv.

The new MyEnv can be downloaded here. How to install MyEnv is described in the MyEnv Installation Guide.

In the inconceivable case that you find a bug in MyEnv please report it to us by sending an email.

Any feedback, statements and testimonials are welcome as well! Please send them to us.

Upgrade from 1.1.x to 2.0

Please look at the MyEnv 2.0.0 Release Notes.

Upgrade from 2.0.x to 2.1.1

$ sudo -i -u mysql
$ cd ${HOME}/product
$ tar xf /download/myenv-2.1.1.tar.gz
$ rm -f myenv
$ ln -s myenv-2.1.1 myenv

Plug-ins

If you are using plug-ins for showMyEnvStatus create all the links in the new directory structure:

$ cd ${HOME}/product/myenv
$ ln -s ../../utl/oem_agent.php plg/showMyEnvStatus/

Upgrade of the instance directory structure

From MyEnv 1.0 to 2.0 the directory structure of instances has fundamentally changed. Nevertheless MyEnv 2.0 works fine with MyEnv 1.0 directory structures.

Changes in MyEnv 2.1.1

MyEnv

addInstance for PostgreSQL added.
Basic PostgreSQL functionality implemented.
up working now.
stop, start, restart and status implemented for PostgreSQL.
Version extraction for PostgreSQL added.
Function extractVersion rewritten so section config is passed and not basedir any more.
Configuration type mysqld changed to mysql and now also usable for mariadb and postgresql.
New variant of version comment added for MySQL 8.0.
Comment added to my_exec().

MyEnv Installer

mysql/mariadb fork installation implemented and tested.
Example in installMyEnv corrected/improved.
Made nasty warning during installation of new instance go away.

MyEnv Utilities

New script added.
SSL added to monitor.
Slave monitor made MySQL 8+ ready.
checksum_table works now also with tables consisting of protected keywords.
More debugging info added.
All scripts checked by shellcheck.
Missing runtime directory is caught and fixed now.
shellcheck suggestions applied.
checksum_table.sh added.
Utility scripts brought to new state.
Build slave script added.
table diff refactored.
Chunking added.
table_diff is now ready for chunking.
Faster checksum implemented and output shortened.
Row by row crc32 checksum.
Moved the checksum table in its own function.
table_diff.php added.

PostgreSQL

show_create_table.sh for PostgreSQL added.
PostgreSQL files added here until we have a better location.

General

Code clean-up and configuration check added.
Recursive directory removal improved.
rc made unique.
Minor typos fixed.
my.cnf.template cleaned-up and synced with website.
2 bugs with PHP 8.5 fixed.
Libraries updated.
Some return codes can be ignored because Redhat tools return error codes > 100 as OK.

Documentation

Documentation added and ready to start.

Packaging

Distro versions centralized in one place.
Zabbix repo added for Ubuntu 24.04.
Rocky 10 and Debian 11 added.
lxc replaced by incus, distros cleaned-up.
Ubuntu 24.04 and MariaDB 10.11 and 11.4 enabled.
brman as build project added.
glb included in build scripts.
restartContainer implemented.
gid and uid added to pushFileToContainer.
Build cleaned-up.
Build scrips improved.
Container file push and pull added.
Build infrastructure reorganized.
stopContainer and startContainer added.
Container library started.
Debian 10 removed.
Update container script is waiting to avoid infrastructure build failure.
Old package for RPM changed.
install_base separted from mysql_home.
Installation moved from /home/mysql to /opt.
Fix DEB package.
Ubuntu 22.04 added for package build.
Bug in Debian build script fixed.
Ubuntu 24.04 added to build infrastructure.
recreate_build_infrastructure.sh rewritten in PHP.
update_container_templates.sh rewritten in PHP.
Made package build infrastructure more generic.

For subscriptions of commercial use of MyEnv please get in contact with us.

FromDual: PostgreSQL Tech-Feed (en)

Database Index Optimizer

MariaDB, MySQL, and Percona Server

Tables without a Primary Key

Duplicate indexes

Partially redundant indexes

Unused indexes

And now with PostgreSQL

Tables without a Primary Key

Duplicate indexes

Partially redundant indexes

Unused indexes

PG Assistant

Intallation of PG Assistant

Addendum

MariaDB/MySQL Environment MyEnv 3.0.0 has been released

Upgrade from 2.x to 3.0

Changes in MyEnv 3.0.0

MyEnv

MyEnv Installer

MyEnv Utilities

PostgreSQL

General

Documentation

Packaging

FromDual Performance Monitor 2.2.1 has been released

Download

Monitoring as a Service (MaaS)

Installation of Performance Monitor 2.2.1

Upgrade of fpmmm tar ball from 1.x to 2.2.1

Changes in FromDual Performance Monitor 2.2.1

General

Templates

Agent

Server

Galera

InnoDB

MySQL

Process

Security

Master

Slave

Backup

PostgreSQL

Packaging

What is the quickest way to load data into the database?

Variant 1: INSERT and DELETE (simplest form)

Variant 2: START TRANSACTION and INSERT ad DELETE

Variant 3: START TRANSACTION and optimised INSERT and DELETE

Test set-up

Preparation and execution with MariaDB/MySQL

Preparation and execution with PostgreSQL

Results

Remarks

Further contributions

How much space does NULL need?

Experiment with MariaDB/MySQL

Test setup

Defragment the table

Attempt 1: NULL out

Attempt 2: Deleting the columns

Conclusion

Experiment with PostgreSQL

Test setup

Attempt 1: NULL out

Experiment 2: Deleting the columns

Remarks

Sources

Additional attempts

Someone is deleting my shared memory segments!

Linux System User

Simple variant: Change SYS_UID_MAX and SYS_GID_MAX to 1001

More complicated variant: Changing the UID from 1001 to 990

Additional information

Load CSV files into the database

Loading the data with MariaDB/MySQL

Differences between MariaDB and MySQL

Sources

Loading the data with PostgreSQL

Sources

Variant 1: `INSERT` and `DELETE` (simplest form)

Variant 2: `START TRANSACTION` and `INSERT` ad `DELETE`

Variant 3: `START TRANSACTION` and optimised `INSERT` and `DELETE`

Attempt 1: `NULL` out

Attempt 1: `NULL` out

Simple variant: Change `SYS_UID_MAX` and `SYS_GID_MAX` to 1001

More complicated variant: Changing the `UID` from 1001 to 990