All versions
Select version:
Alan Mologorsky committed 2026-05-152026-05-15 12:13
feat(cmapi, install_es, upgrades): MCOL-6178 MCOL-6192: Overhaul install_es upgrade/downgrade pipeline

Add Upgrade Agent — a standalone FastAPI+uvicorn HTTPS microservice that
runs on each node during upgrades, independent of the CMAPI version being
installed. The agent provides endpoints to fix configs, restart CMAPI, and
auto-shuts down after a configurable timeout.

Upgrade Agent endpoints:
- POST /fix-mariadb-cli-config — remove unsupported CLI options after downgrade.
- POST /fix-columnstore-cnf — add loose- prefix and merge user values from .bak.
- POST /restart-cmapi — restart mariadb-columnstore-cmapi systemd service.
- GET /health — liveness probe.
- POST /shutdown — graceful stop.

Upgrade/downgrade orchestration (install_es):
- Integrate upgrade agent into the flow: start agents on all nodes before
  package changes, use them to fix configs and restart CMAPI afterwards,
  wait for readiness, then stop agents.
- Add --skip-cmapi flag to skip CMAPI upgrade entirely.
- Don't downgrade CMAPI by default; add --allow-cmapi-downgrade to opt in.
- Fix dry-run vs real execution package ordering.
- Clean up stale MariaDB repo files (deb and rpm) before installing new repo.
- Accept version prefixes (e.g. 10.6) and resolve to latest matching version.
- Fix latest version resolution to return real latest for the target OS.
- Fix unremoved mariadb-server-xx.xx meta-package on Debian during downgrade.
- Fix unremoved dependencies during downgrade.
- Fix missing columnstore package for specific OS in a repo version that exists.
- Fix upgrading for 11.4 and 11.8 where mariadb-server-xx.xx doesn't exist.
- Fix preventing downgrade between major versions.
- Fix version mismatch detection.
- Fix case when columnstore package for specific OS doesn't exist in existing repo
  with that version (mdb version).
- MCOL-6192 now MDB restarts even if downgrade happened.
- Add fallback .cnf backup logic in `mcs install_es` for older columnstore versions that
  lack the new postrm logic.
- Automatically add loose- prefix to any `columnstore_*` variable in columnstore.cnf.
- Add auto merge columnstore.cnf file from newly installable package and from removed package (.cnf.bak)

Packaging (postrm):
- Convert debian postrm to .in template with @MARIADB_MYCNFDIR@ substitution.
- Rename columnstore.cnf to .bak on Columnstore removal to prevent MariaDB startup
  failures from unknown columnstore_* variables.
- Add echo for cases when config renamed to .bak to inform user about .cnf backup.

Process manager refactoring:
- Extract BaseNonMCSProcessManager base class for non-MCS systemd services.
- MDBProcessManager now inherits from base with process_name='mariadb'.
- Add CmapiProcessManager (process_name=mariadb-columnstore-cmapi).
- Add restart() method.

CMAPI API extensions:
- Add start-upgrade-agent endpoint on node and cluster controllers.
- Add UpgradeAgentClient in api_clients with full HTTPS support.

Invariant checks:
- Rename config key to invariant_checks_enforce, default to false.
- Checks always run; failures log warnings unless enforce mode is on.
- Cache enforce flag at startup via init_invariant_checks_mode().

Logging:
- Add SharedStorageAccessFilter to redirect periodic shared-storage check
  messages from main cmapi_server.log to dedicated shared_storage_monitor log.
- Consistent upgrade log file paths.

Dependencies:
- Add fastapi==0.115.6, uvicorn==0.34.0 (+ anyio, starlette, h11).

Minor:
- Rename CERT_FILENAME to CMAPI_CERT_PATH, fix certificate typos.
- Update README.md, README_DEV.md, mcs.1 man page.
  mariadb-corporation/mariadb-columnstore-engine
6e814a195d762543dc6eb366dd26ff902f9488d8 GitHub
+2350
±297
-130
Serguey Zefirov committed 2026-05-152026-05-15 11:49
fix(cpimport): MCOL-6326 - cpimport should invalidate partially rolled back extents

This patch adds invalidation logic for partially rolled back extents
during rollback after a failure. Previously only HWMs were set, now
ranges are specificially invalidated too.
  mariadb-corporation/mariadb-columnstore-engine
b9216894d8859c2987a2d10f8209dc21742653bf GitHub
+61
±3
-
Vasily Kozhukhovskiy committed 2026-05-052026-05-05 11:21
MCOL-6332 Fix ORDER BY using numeric positional column reference and DESC order on non-grouped columns in aggregation queries

When a query uses aggregation without GROUP BY (e.g.
SELECT COUNT(a), b FROM t ORDER BY 1, 2 DESC), the RBO wraps
non-grouped columns in AggregateColumn(SELECT_SOME). The SELECT-list
and ORDER BY wrappers for the same column could receive different
expressionIds because AggregateColumn::operator== compares the asc
field, which differs for DESC ORDER BY columns (at the same time a
separate column expression is not added to the jobInfo.deliveredCols
col list when numeric column references are used in ORDER BY clause).
This causes addOrderByAndLimit to fail with "column is not found in info map"
when looking up the unregistered ORDER BY expressionId.

Normalize the asc field to a canonical value before dedup comparison
in wrapColumn(). Restore the real sort direction on both the wrapper and
its inner column afterwards so ORDER BY processing sees the correct direction.
aleksei.bukhalov committed 2026-05-042026-05-04 12:32
ci(drone): run full regression on custom builds, skip mtr

For custom-triggered pipelines:
- include the full nightly regression test list (test000..test500)
  instead of the minimal pull_request set (test000+test001), so that
  ad-hoc runs can verify regression-test fixes end-to-end.
- skip mtr / mtrlog / publish-mtrlog steps entirely (and the rocky:8
  gcc-toolset multi-node MTR + dockerhub publish path) to save build
  time when the goal is regression validation only.
- drop 'mtr' from the first regression step's depends_on on custom,
  so regression starts as soon as 'publish pkg' / 'publish cmapi build'
  are ready.

cron and pull_request pipelines are unchanged.

Used to validate the bug3350.sql.ref.log fix on the matching
MCOL-6340-fix-tests branch in the regression-test repo.
  mariadb-corporation/mariadb-columnstore-engine
ec308c30585adbaf99d1ae377be75749cd9fd02f GitHub
Timofey Turenko committed 2026-05-022026-05-02 22:08
Merge pull request #3998 from mariadb-corporation/MCOL-6346-use_server_10.6.15

Mcol 6346 use server 10.6.15
  mariadb-corporation/mariadb-columnstore-engine
ff8960980cf40ab82e83b65e11aca99ff76c8442 GitHub
Alan Mologorsky committed 2026-05-012026-05-01 03:28
MCOL-6178: Another bunch of changes related to cmapi codebase sync from 23.10 to 23.02

PR#3903 contains all the change set for STABLE-23.10
  mariadb-corporation/mariadb-columnstore-engine
55463660a81fd86ee2d09e450269d16ba784dfdd GitHub
+2431
±305
-127
aleksei.bukhalov committed 2026-04-302026-04-30 18:37
fix(mtr): normalize MTR identifiers to lowercase

Lowercase ~77 mixed-case TABLE/DATABASE/VIEW idents across 119
.test/.result files in basic/bugfixes/devregression/future
(MCOL5142_LEFT_JOIN, FOO, bug5628A, vwBug*, TPC-H LINEITEM/PART/
PARTSUPP, MCOL_669_B vs declared _b, MCOL3434CS, MCOL_736, ...).

Also drop the cnf-mutating sed + restart block from
setup/regression_env_setup.test (the suites no longer depend on
runtime case-folding).

Per-suite suite.opt --lower_case_table_names=2 entries kept as
defense-in-depth.

Ref: MCOL-6340.
aleksei.bukhalov committed 2026-04-302026-04-30 18:37
fix(mtr): stabilize flaky tests MCOL-3785, MCOL-5669, MCOL-5963

Test-level fixes; none mask data bugs.

- MCOL-3785: SELECT lacked ORDER BY — add ORDER BY c0.
- MCOL-5669: UNION lacked ORDER BY → ordering varied across CES
  paths (add ORDER BY 1,2,3,4); sum(volume/9000) drifts 1 ULP in
  last DECIMAL digit through InnoDB→CES (mask 4th decimal via
  --replace_regex).
- MCOL-5963: IF(0=1, int_const, time_col) emits HH:MM:SS.0 via
  CES vs HH:MM:SS via direct InnoDB — keep HH:MM:SS in .result,
  --replace_regex strips optional .0.

Ref: MCOL-6340.
  mariadb-corporation/mariadb-columnstore-engine
46cdd100574b62cbe075f277eb41782ce9c1f523 GitHub
+16
±48
-
aleksei.bukhalov committed 2026-04-302026-04-30 18:37
fix(mtr): disable MCOL-6321-test-em-rebuild

Crashes PrimitiveServer on 4MB+ longtext (MCOL-6302). Disable
until that is fixed.

Ref: MCOL-6302, MCOL-6321.
  mariadb-corporation/mariadb-columnstore-engine
8027d1a07a5149006cd3c27552f15ece73dfddb9 GitHub
aleksei.bukhalov committed 2026-04-302026-04-30 18:37
fix(mtr): refactor cross-engine setup, drop runner-level overrides

- include/cross_engine.inc — idempotent CREATE USER cejuser for
  'localhost' + '127.0.0.1', GRANT, FLUSH.
- include/drop_cross_engine.inc — idempotent DROP USER IF EXISTS
  for both hosts.
- 24 tests now --source these includes instead of inline copies
  (-596/+82).
- run_mtr.sh — stop writing lower_case_table_names=2 and
  loose-columnstore_innodb_queries_use_mcs=ON to a cnf drop-in
  before the first restart. Both are applied per-suite via
  suite.opt (mariadbd restarts between suites). Drop the
  diagnostic recheck block.

Ref: MCOL-6340.
  mariadb-corporation/mariadb-columnstore-engine
86344782246bcc1416d7369e4f19cc1511f86b83 GitHub
+27
±55
-541
Alan Mologorsky committed 2026-04-292026-04-29 16:38
fix(cmapi, cs_package_manager): MCOL-6178 Add --product argument to cs_package_manager for dev packages.
  mariadb-corporation/mariadb-columnstore-engine
35d70d5836e15f436861e8b01029946b79d05e4c GitHub
Alan Mologorsky committed 2026-04-292026-04-29 14:14
fix(engine, load_from_s3): MCOL-6146 Add mode 2 and 3 to load_from_s3.

Fix cmapi s3 data load.
Fix engine load_from_s3 implementation.
  mariadb-corporation/mariadb-columnstore-engine
f82109dcd317aa0a4d3a2effe425ddf1c08554fe GitHub
+6
±8
-
Akhmad O. committed 2026-04-282026-04-28 09:05
fix(funcexp): MCOL-6336 fix DATE_SUB/DATE_ADD with INTERVAL NULL crashes with BatchPrimitiveProcessor error instead of returning NULL (#3991)
  mariadb-corporation/mariadb-columnstore-engine
e565ad012ad6bc73637a1f950ac8a897f3004997 GitHub
+36
±2
-
Leonid Fedorov committed 2026-04-222026-04-22 12:39
fix(rebuildEM): apply HWM once per segment file after all extents exist

ExtentMap::setLocalHWM() stores newHWM on the extent with the highest
blockOffset in (oid, partition, segment) and resets the previous
HWM-bearing extent to 0. Calling it per-extent inside the creation loop
therefore races with itself: addInvisibleLBIDs() appends entries with
artificial LBIDs greater than any real one and hwm=0, they sort last,
and their setLocalHWM(newHWM=0) call wipes the real HWM that had
already been stored on an earlier iteration. PrimProc then reads
getLocalHWM()==0, computes a negative range.size and aborts with
MCS-2031 "Blocks are missing".

Rebuild now works in two passes:
  1. create every extent and call setLocalHWM(newHWM=0) only to flip
     the extent status to AVAILABLE;
  2. walk the extent map a second time and call setLocalHWM once per
     unique (oid, partition, segment), taking the HWM from
     oidHWMs[oid] (regular OIDs) or fileId.hwm (system OIDs).

Regression test: bugfixes.MCOL-6321-test-em-rebuild.
  mariadb-corporation/mariadb-columnstore-engine
16dd19bcfe3c8a1bcb8b6c6ba4f4e54551909c2e GitHub
Leonid Fedorov committed 2026-04-222026-04-22 12:39
fix(plugin): skip hash_sort() when MariaDBHasher::add is given an empty string

Columnstore rowgroup::Row represents NULL / empty string columns as
{str=nullptr, length=0}. utils::ConstString even documents that its
internal pointer 'can be NULL now'. When such a row reaches
datatypes::MariaDBHasher::add() we used to forward that {nullptr, 0}
straight into cs->hash_sort() → my_hash_sort_utf8mb3() →
skip_trailing_space(ptr=0x0, len=0), which

  * is undefined behaviour (applying zero offset to a null pointer --
    the same UB MDEV-35620 was hardening against on the server side);
  * fires the DBUG_ASSERT(ptr) that MDEV-35620 added to
    sql/strings_def.h:85 in debug builds, aborting ExeMgr / PrimProc.

Reproducer is basic.MCOL-5250_disk_based_distinct: a SELECT DISTINCT on
a CHAR/VARCHAR/TEXT column that can be NULL routes through
joblist::LimitedOrderBy::processRow ->
ordering::IdbOrderBy::Hasher::operator() -> rowgroup::Row::hash ->
Row::colUpdateHasher -> MariaDBHasher::add(cs, ConstString{nullptr, 0}).
In a _GLIBCXX_DEBUG / -DDBUG build this trips the assert and the server
returns 'InetStreamSocket::readToMagic: Remote is closed' to the client.

Fix: hashing an empty string is a no-op in every MariaDB collation
(my_hash_sort_* / skip_trailing_space just return when len==0), so
short-circuit in MariaDBHasher::add when length==0 before we ever hand
the null pointer to MariaDB. Keeps hash stability for non-NULL inputs
(the fast path is untouched) and eliminates the nullptr-with-offset UB
for the NULL-string case.

Verified with mtr: basic.MCOL-5250_disk_based_distinct now passes
(previously crashed ExeMgr with signal 6).
  mariadb-corporation/mariadb-columnstore-engine
f2eb84cdfe048ba323b3e2262776c7490704ae1e GitHub
Leonid Fedorov committed 2026-04-222026-04-22 12:39
fix(plugin): do not mutate table->read_set bits in bulk insert

During cpimport bulk insertion ha_mcs_impl_start_bulk_insert()/
ha_mcs_impl_end_bulk_insert() used to call bitmap_set_all() and
bitmap_clear_all() directly on table->read_set so that Field::val_str()
would work for every column from ColWriteBatchString().

That breaks whenever the server had aliased read_set onto a shared,
server-owned bitmap. TABLE::use_all_columns() does exactly that:

    inline void use_all_columns()
    {
      column_bitmaps_set(&s->all_set, &s->all_set);
    }

and copy_data_between_tables() in sql/sql_table.cc calls it on the new
table just before the bulk-insert loop for every ALTER TABLE that copies
rows (ENGINE=columnstore, ADD/DROP/CHANGE COLUMN on a Columnstore
table, ...). bitmap_clear_all(table->read_set) then wipes
TABLE_SHARE::all_set, and the next handler::ha_reset() triggered from
THD::drop_temporary_table() fires

    sql/handler.cc:7829: int handler::ha_reset():
      Assertion `bitmap_is_set_all(&table->s->all_set)' failed.

crashing the server in debug builds. The assertion is a long-standing
invariant of TABLE_SHARE::all_set ('all bits always set'); it only
started firing reliably once the plugin started being compiled with the
same _GLIBCXX_DEBUG / _GLIBCXX_ASSERTIONS flags as the server, because
the ABI mismatch had previously been crashing the server earlier on an
unrelated path.

Fix: instead of mutating the bits, swap the pointer.
ha_mcs_impl_start_bulk_insert() stashes the original table->read_set in
cal_connection_info::bulkInsertSavedReadSet and points table->read_set
at &table->s->all_set for the duration of the bulk insert (its bits are
already guaranteed to be all-set). ha_mcs_impl_end_bulk_insert() (and
every early-return/catch in start that used to clear_all) restores the
saved pointer. This preserves the semantics ColWriteBatchString() needs
while never touching server-owned bitmap bits.

Reproduced and fixed by mtr:
  basic.mcol_2000, bugfixes.MCOL_5175, bugfixes.MCOL-4758,
  bugfixes.MCOL-5779 now pass (all previously crashed in ha_reset).
  basic.mcol641-create no longer hits ha_reset; it now trips a
  separate, unrelated assertion in ha_mcs_ddl.cpp that was masked
  by this crash and is tracked separately.
  mariadb-corporation/mariadb-columnstore-engine
c8ea47b22717b3750f70a00c8f2ed2de43f96388 GitHub
+56
±10
-
Leonid Fedorov committed 2026-04-222026-04-22 12:23
fix(funcexp): MCOL-6173 add Func_from_unixtime::getTimestampIntVal

In MariaDB 11.8 Item_func_from_unixtime inherits from
Item_timestampfunc and now reports MYSQL_TYPE_TIMESTAMP; the
pushed-down FunctionColumn's resultType().colDataType became
TIMESTAMP. Without an explicit getTimestampIntVal override
Func_from_unixtime fell back to intToTimestamp(getIntVal()),
stuffing a 14-digit YYYYMMDDHHmmss integer into the TimeStamp
bit layout (msec:20 | second:44) and producing the same bogus
"1970-08-06 10:44:40" for every row.

Extract the (val, msec) extraction into getUnixTimestampParts
and add a real getTimestampIntVal that returns a properly
packed Calpont TimeStamp — second is the raw Unix seconds
(server applies time zone), msecond holds the fractional us.

Re-enable the portable mcs247_from_unixtime_funtion2 test on
11.8, and make it tolerate FROM_UNIXTIME(0)=NULL / the new
"Truncated incorrect unixtime value: '0.0'" warning.
  mariadb-corporation/mariadb-columnstore-engine
e5eb8d11ebcc004b3ac161b3624744a016003833 GitHub
+52
±7
-
Leonid Fedorov committed 2026-04-212026-04-21 13:17
fix(plugin): MCOL-6311: Fix UPDATE/DELETE with IN-subqueries on cross-engine and self-join tables

Three separate issues prevented UPDATE/DELETE statements with IN-subqueries
from working correctly:

1. ExeMgr Error 999 on UPDATE/DELETE with IN-subquery (MCOL-4868/MCOL-6311).
   The server's check_and_do_in_subquery_rewrites() was converting IN-subqueries
   into semijoin candidates, bypassing select_transformer() and leaving
   Item_in_subselect unwrapped by Item_in_optimizer. Columnstore's gp_walk
   could not serialize the bare Item_in_subselect, causing ExeMgr Error 999.
   Fix: call mutate_optimizer_flags() for UPDATE/DELETE in ha_mcs::open() and
   discover_check_version(), same as for prepared statements. This disables
   semijoin and uses the MATERIALIZATION strategy, which correctly wraps the
   subquery via select_transformer().

2. "Character set '#0' is not a compiled character set" on cross-engine DELETE.
   When PredicateOperator is deserialized in ExeMgr/DMLProc, integer column
   predicates have charsetNumber=0 (unset). get_charset(0) triggers my_error()
   which propagates as a statement failure.
   Fix: return my_charset_bin directly when charsetNumber is 0 in
   get_charset_or_bin().

3. Server crash on self-join DELETE (DELETE FROM t WHERE a IN (SELECT a FROM t)).
   The server's Sql_cmd_dml::execute_inner() calls find_single_select_handler()
   for multi-table DELETEs, which invokes Columnstore's select_handler. The
   select_handler's prepare() tries to create a temp table, but the DELETE's
   SELECT_LEX has no fields set up for this, causing an assertion failure in
   Create_tmp_table::start (m_alloced_field_count == 0).
   Fix: skip select_handler creation for DELETE statements in
   create_columnstore_select_handler_(). DELETEs use the doUpdateDelete path
   via ha_mcs::rnd_init instead.

Additionally:
- Fix processWhere() to prioritize the isUpdateDelete path over join->conds
  at the outer query level. After MDEV-25008 the server creates a JOIN for
  UPDATE/DELETE during prepare, making join non-NULL, but join->conds may
  contain transformations incompatible with Columnstore DML processing.
- Fix run_mtr.sh: remove stale mysql-test/suite/columnstore symlink instead
  of creating it; having symlinks in both mariadb-test/suite/ and
  mysql-test/suite/ causes mtr_cases.pm to crash on duplicate suite paths.
- Remove disabled.def entry for MCOL-4868.
  mariadb-corporation/mariadb-columnstore-engine
a11e5f5673b7d71ec14e6300a81e3bb00a6b7dfc GitHub
+35
±9
-2
Serguey Zefirov committed 2026-04-202026-04-20 15:22
fix(cpimport): MCOL-6326 - cpimport should invalidate partially rolled back extents

This patch adds invalidation logic for partially rolled back extents
during rollback after a failure. Previously only HWMs were set, now
ranges are specificially invalidated too.

Test is in tests/custom/cpimport-rollback.sh, not in MTR.
  mariadb-corporation/mariadb-columnstore-engine
ff7cfc9f0efacc9588bef5fdaca2cc42b9b23916 GitHub
+63
±3
-