Add support for MySQL 8.0

[jeremycole]

Add support for MySQL 8.0. There are a number of changes needed to support MySQL 8.0, and some new data structures to implement:

• Support new headers in FSP pages
• Support for reconstructing full BLOB content
• Support new page types:
  • UNKNOWN (13)
  • COMPRESSED (14)
  • ENCRYPTED (15)
  • COMPRESSED_AND_ENCRYPTED (16)
  • ENCRYPTED_RTREE (17)
  • SDI_BLOB (18) (aka BLOB)
  • SDI_ZBLOB (19) (aka ZBLOB)
  • LEGACY_DBLWR (20)
  • RSEG_ARRAY (21)
  • LOB_INDEX (22)
  • LOB_DATA (23)
  • LOB_FIRST (24)
  • ZLOB_FIRST (25)
  • ZLOB_DATA (26)
  • ZLOB_INDEX (27)
  • ZLOB_FRAG (28)
  • ZLOB_FRAG_ENTRY (29)
  • SDI (17853) (aka INDEX)
  • RTREE (17854)
• Support for new data dictionary structures:
  • Read SDI information from InnoDB tablespace files
  • Using SDI to make record describers
  • Refactor current data dictionary to support old and new
• Support new mysql data dictionary tables
  • Add support for SET type #205
  • Add support for ENUM type #207
• Support for LOB
• Support for Compressed LOB (ZLOB)
• Support for online-added columns?
• Support for undo tablespaces
• ???

Sub-issue list:

• Refactor Innodb::DataDictionary to prepare for new SDI data dictionary #203
• Add support for Serialized Dictionary Information (SDI) in MySQL 8.0 #185
• Add support for new LOB storage in MySQL 8.0 #186
• Add support for new ZLOB storage in MySQL 8.0 #187

PR list:

• Refactor hex output of BLOB data #184
• Add new page types for MySQL 8.0, remove :usage key in structure #188
• Implement reading new encryption and SDI headers in FSP page #189
• Update spec/data/sakila/8.0 with MySQL 8.0.33 #190
• Add support for reconstructing full BLOB content #191
• Add support for reading SDI from InnoDB #202
• Refactor DataDictionary; support SYS_* and SDI #206
• Update spec/data/sakila/8.0 using 8.0.40 #208
• Add missed mysql.ibd file for 8.0 update #209
• Refactor Innodb::DataType; add SET/ENUM; fix DECIMAL #213

See also: #41, #44.

[utdrmac]

Related?

# innodb_space -s ibdata1 system-spaces
name                            pages       indexes
Traceback (most recent call last):
	16: from /usr/local/bin/innodb_space:23:in `<main>'
	15: from /usr/local/bin/innodb_space:23:in `load'
	14: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/bin/innodb_space:1714:in `<top (required)>'
	13: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/bin/innodb_space:192:in `system_spaces'
	12: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/bin/innodb_space:188:in `block in system_spaces'
	11: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/bin/innodb_space:188:in `to_a'
	10: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/bin/innodb_space:188:in `each'
	 9: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/space.rb:341:in `each_index'
	 8: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/space.rb:321:in `each_index_root_page_number'
	 7: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/data_dictionary.rb:425:in `each_index_by_space_id'
	 6: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/data_dictionary.rb:355:in `each_index'
	 5: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/data_dictionary.rb:324:in `each_record_from_data_dictionary_index'
	 4: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/data_dictionary.rb:287:in `data_dictionary_index'
	 3: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/space.rb:312:in `index'
	 2: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/space.rb:312:in `new'
	 1: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/index.rb:19:in `initialize'
/usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/index.rb:34:in `page': undefined method `record_describer=' for #<Innodb::Page::FspHdrXdes:0x000055a140fed580> (NoMethodError)

8.0.28

[lhsaq2009]

[root@MySQL80 mysql]# innodb_space -s ibdata1 system-spaces
name                            pages       indexes
Traceback (most recent call last):
	16: from /root/.gem/ruby/2.6.0/bin/innodb_space:23:in `<main>'
	15: from /root/.gem/ruby/2.6.0/bin/innodb_space:23:in `load'
	14: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/bin/innodb_space:1714:in `<top (required)>'
	13: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/bin/innodb_space:192:in `system_spaces'
	12: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/bin/innodb_space:188:in `block in system_spaces'
	11: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/bin/innodb_space:188:in `to_a'
	10: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/bin/innodb_space:188:in `each'
	 9: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/space.rb:341:in `each_index'
	 8: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/space.rb:321:in `each_index_root_page_number'
	 7: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/data_dictionary.rb:425:in `each_index_by_space_id'
	 6: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/data_dictionary.rb:355:in `each_index'
	 5: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/data_dictionary.rb:324:in `each_record_from_data_dictionary_index'
	 4: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/data_dictionary.rb:287:in `data_dictionary_index'
	 3: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/space.rb:312:in `index'
	 2: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/space.rb:312:in `new'
	 1: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/index.rb:19:in `initialize'
/root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/index.rb:34:in `page': undefined method `record_describer=' for #<Innodb::Page::FspHdrXdes:0x000055b0ae47b968> (NoMethodError)
[root@MySQL80 mysql]#

8.0.18

[jeremycole]

I filed MySQL Bug 111079: InnoDB SDI index description has fields out-of-order as implementing SDI caused me to find a subtle issue with the fields being out of order internally.

[jeremycole]

I filed MySQL Bug 111112: JSON SDI default_value in columns object includes trailing zero bytes as implementing SDI caused me to notice that the included default_value field includes trailing zero bytes (seems to be encoding a raw buffer from C++).

[jeremycole]

Good news! The main branch now has support for the new SDI data dictionary, e.g.:

$ bundle exec bin/innodb_space -s spec/data/sakila/8.0 -T sakila/actor -I PRIMARY index-recurse
ROOT NODE #4: 200 records, 7507 bytes
  RECORD: (actor_id=1) → (first_name="PENELOPE", last_name="GUINESS", last_update="2006-02-15 11:34:33")
  RECORD: (actor_id=2) → (first_name="NICK", last_name="WAHLBERG", last_update="2006-02-15 11:34:33")
  RECORD: (actor_id=3) → (first_name="ED", last_name="CHASE", last_update="2006-02-15 11:34:33")
  RECORD: (actor_id=4) → (first_name="JENNIFER", last_name="DAVIS", last_update="2006-02-15 11:34:33")
  RECORD: (actor_id=5) → (first_name="JOHNNY", last_name="LOLLOBRIGIDA", last_update="2006-02-15 11:34:33")

You may notice that it can't read all columns from all tables, in particular there is some weirdness with DECIMAL type at the moment, and only partial support for ENUM and SET. While innodb_ruby has never really supported those types in the past, from the new data dictionary it now knows the proper types of columns, so it's trying, and kinda failing. I'll keep working on that.

In the mean time, feel free to play around and file bug reports, send pull requests, etc. The basics should all be there.