pt-table-checksum fails to find pxc nodes when wsrep_node_incoming_address is set to AUTO

Bug #1399789 reported by Marc Castrovinci
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona Toolkit moved to https://jira.percona.com/projects/PT
Fix Released
High
Frank Cizmich

Bug Description

Per this fix https://bugs.launchpad.net/percona-toolkit/+bug/1217466 , wsrep_node_incoming_address is now being used as a way to uniquely identify a cluster node.

However, our cluster sets this value to AUTO, so all the nodes are being seen as non-unique causing no nodes to be found. If I downgrade to version 2.2.11, the error goes away because server-id is being used again.

A possible fix could be to check the server-id first, and if thats not unique then check wsrep_node_incoming_address ( or visa versa )

percona-toolkit version: 2.2.12
 Server version: 5.6.20-68.0-56-log Percona XtraDB Cluster (GPL), Release 25.7, wsrep_25.7.r4126

Reproduce:

root@CCCCCCCCC:~# pt-table-checksum --version
pt-table-checksum 2.2.12
root@CCCCCCCCCC:~# pt-table-checksum --recursion-method=cluster u=XXXXXX,p=XXXXXXX= --empty-replicate-table
Diffs cannot be detected because no cluster nodes were found. Please read the --recursion-method documentation for information.
CCCCCCCCCCCCCCC is a cluster node but no other nodes or regular replicas were found. Use --recursion-method=dsn to specify the other nodes in the cluster.
root@CCCCCCCCCCC:~# mysql -e "show variables like 'wsrep_node_incoming_address';"
+-----------------------------+-------+
| Variable_name | Value |
+-----------------------------+-------+
| wsrep_node_incoming_address | AUTO |
+-----------------------------+-------+

Downgraded:

root@AAAAAAAA:/tmp# pt-table-checksum --recursion-method=cluster u=XXXXXXX,p=XXXXXXX --empty-replicate-table
Not checking replica lag on BBBBBBBBBBBBBBBBBBBBBBBBB because it is a cluster node.
Not checking replica lag on AAAAAAAAAAAAAAAAAAAAAAAA because it is a cluster node.
            TS ERRORS DIFFS ROWS CHUNKS SKIPPED TIME TABLE
12-05T20:37:54 0 0 0 1 0 0.047 mysql.columns_priv
12-05T20:37:54 0 0 2 1 0 0.067 mysql.db
12-05T20:37:54 0 0 0 1 0 0.086 mysql.event
12-05T20:37:54 0 0 0 1 0 0.043 mysql.func
12-05T20:37:55 0 0 40 1 0 0.076 mysql.help_category
12-05T20:37:55 0 0 485 1 0 0.099 mysql.help_keyword
12-05T20:37:55 0 0 1090 1 0 0.045 mysql.help_relation
12-05T20:37:55 0 0 533 1 0 0.044 mysql.help_topic
12-05T20:37:55 0 0 0 1 0 0.036 mysql.ndb_binlog_index
12-05T20:37:55 0 0 0 1 0 0.047 mysql.plugin
12-05T20:37:55 0 0 0 1 0 0.042 mysql.proc
12-05T20:37:55 0 0 0 1 0 0.046 mysql.procs_priv
12-05T20:37:55 0 0 2 1 0 0.041 mysql.proxies_priv
12-05T20:37:55 0 0 0 1 0 0.051 mysql.servers
12-05T20:37:55 0 0 0 1 0 0.039 mysql.tables_priv
12-05T20:37:55 0 0 0 1 0 0.047 mysql.time_zone
12-05T20:37:55 0 0 0 1 0 0.051 mysql.time_zone_leap_second
12-05T20:37:56 0 0 0 1 0 0.893 mysql.time_zone_name
12-05T20:37:56 0 0 0 1 0 0.050 mysql.time_zone_transition
12-05T20:37:56 0 0 0 1 0 0.037 mysql.time_zone_transition_type
12-05T20:37:56 0 0 7 1 0 0.044 mysql.user
root@AAAAAAAAAAA:/tmp# mysql -e "show variables like 'wsrep_node_incoming_address';"
+-----------------------------+-------+
| Variable_name | Value |
+-----------------------------+-------+
| wsrep_node_incoming_address | AUTO |
+-----------------------------+-------+

Related branches

Changed in percona-toolkit:
status: New → In Progress
milestone: none → 2.2.13
importance: Undecided → High
assignee: nobody → Frank Cizmich (frank-cizmich)
Revision history for this message
Frank Cizmich (frank-cizmich) wrote :

Attached is a version that uses a combination of variables to obtain a unique id.

tags: added: pt-table-checksum pxc
Revision history for this message
Kenny Gryp (gryp) wrote :

Arunjith had it too with a customer. I made a very quick patch that solved it for that case, changing to wsrep_node_address instead of wsrep_node_incoming_address

Changed in percona-toolkit:
status: In Progress → Fix Committed
Changed in percona-toolkit:
status: Fix Committed → Fix Released
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PT-384

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.