autoconfiguration hangs on cookbooks/mysql/libraries/mysql_service_base.rb line 91

[StanislavBabkin]

autoconfiguration hangs on cookbooks/mysql/libraries/mysql_service_base.rb line 91

Cookbook version mysql 10.1.3
Version of chef-client 17.0.242
OS Ubuntu 20.04

Steps to reproduce the behavior:

mysql_service 'epc' do
  initial_root_password node.run_state['root_pass']
  bind_address node['rds_epc']['bind_address']
  action [:create, :start]
end

ruby_block 'wait for service start' do
  block do
    sleep(30)
  end
  action :run
end

if node.run_state['rds_mode'] == 'service'
  mariadb_database node.run_state['db_name'] do
    password node.run_state['root_pass']
    host '127.0.0.1'
    user 'root'
    action :create
  end

  mariadb_user node.run_state['db_user'] do
    ctrl_password node.run_state['root_pass']
    database_name node.run_state['db_name']
    password node.run_state['user_pass']
    ctrl_host '127.0.0.1'
    host '%'
    action [:create, :grant]
  end
end

[StanislavBabkin]

[StanislavBabkin]

revealed that at the time the service was started, the default port was busy and the service could not be brought up

you need to use any other port to run custom service

[StanislavBabkin]

if you use any other port for the service everything works

mysql_service 'epc' do
  port '3307'
  initial_root_password node.run_state['root_pass']
  bind_address node['rds_epc']['bind_address']
  action [:create, :start]
end

@ramereth, please take a look

if I use default

mysql_service 'default' do
  bind_address '0.0.0.0'
  port '3306'
  initial_root_password 'Ch4ng3me'
  action [:create, :start]
end

I got error while creating user or db

[StanislavBabkin]

@tas50, @xorima, @sethvargo, @aleksey-hariton-epam, please take a moment and help with this,

maybe there is some workaround?

[ramereth]

@StanislavBabkin I'm not able to replicate this by using this (as you mentioned above)

mysql_service 'default' do
  bind_address '0.0.0.0'
  port '3306'
  initial_root_password 'Ch4ng3me'
  action [:create, :start]
end

I feel like there is something else going on with your environment that we need to know about so we can replicate it.

[StanislavBabkin]

@ramereth, here is my recipe

mysql_service 'default' do
  bind_address node['rds_epc']['bind_address'] # '0.0.0.0' 
  port node['rds_epc']['mysqld']['port'] # '3306'
  initial_root_password node.run_state['root_pass'] # 'YqR2KE7iLP7GSAWt'
  action [:create, :start]
end

ruby_block 'wait for service start' do
  block do
    sleep(30)
  end
  action :run
end

if node.run_state['rds_mode'] == 'service'

  mysql_database node.run_state['db_name'] do # 'test'
    host '127.0.0.1'
    port node['rds_epc']['mysqld']['port'].to_i # '3306'
    user 'root'
    password node.run_state['root_pass'] # 'YqR2KE7iLP7GSAWt'
    action :create
  end

  mysql_user node.run_state['db_user'] do # 'stas'
    ctrl_password node.run_state['root_pass'] # 'YqR2KE7iLP7GSAWt'
    ctrl_host '127.0.0.1'
    ctrl_port node['rds_epc']['mysqld']['port'].to_i # '3306'
    password node.run_state['user_pass'] # 'FjdfjjFHDkfjdfhdn'
    database_name node.run_state['db_name'] # 'test'
    host '%'
    privileges [:select,:update,:insert]
    action [:create, :grant]
  end
end

when I use this I get error

[StanislavBabkin]

also when I try to use custom mysql_service

mysql_service 'epc' do
  bind_address node['rds_epc']['bind_address']
  port node['rds_epc']['mysqld']['port']
  initial_root_password node.run_state['root_pass']
  action [:create, :start]
end

ruby_block 'wait for service start' do
  block do
    sleep(30)
  end
  action :run
end

if node.run_state['rds_mode'] == 'service'

  mysql_database node.run_state['db_name'] do
    host '127.0.0.1'
    port node['rds_epc']['mysqld']['port'].to_i
    user 'root'
    password node.run_state['root_pass']
    action :create
  end

  mysql_user node.run_state['db_user'] do
    ctrl_password node.run_state['root_pass']
    ctrl_host '127.0.0.1'
    ctrl_port node['rds_epc']['mysqld']['port'].to_i
    password node.run_state['user_pass']
    database_name node.run_state['db_name']
    host '%'
    privileges [:select,:update,:insert]
    action [:create, :grant]
  end

end

auto-configuration hangs on resource
bash[epc initial records] (mysql/libraries/mysql_service_base.rb line 91 )

and after 3600s get error

* service[epc apparmor] action nothing[2021-06-22T12:09:42+00:00] INFO: Processing service[epc apparmor] action nothing (/var/chef/cache/cookbooks/mysql/libraries/mysql_service_base.rb line 175)
 (skipped due to action :nothing)
      * bash[epc initial records] action run[2021-06-22T12:09:42+00:00] INFO: Processing bash[epc initial records] action run (/var/chef/cache/cookbooks/mysql/libraries/mysql_service_base.rb line 91)


        ================================================================================
        Error executing action `run` on resource 'bash[epc initial records]'
        ================================================================================

        Mixlib::ShellOut::CommandTimeout
        --------------------------------
        Command timed out after 3600s:
        Command exceeded allowed execution time, process terminated
        ---- Begin output of "bash"  ----
        STDOUT:
        STDERR:
        ---- End output of "bash"  ----
        Ran "bash"  returned

        Resource Declaration:
        ---------------------
        # In /var/chef/cache/cookbooks/mysql/libraries/mysql_service_base.rb

         91:         bash "#{new_resource.name} initial records" do
         92:           code init_records_script
         93:           umask '022'
         94:           returns [0, 1, 2] # facepalm
         95:           not_if { db_initialized? }
         96:           action :run
         97:         end
         98:       end

        Compiled Resource:
        ------------------
        # Declared in /var/chef/cache/cookbooks/mysql/libraries/mysql_service_base.rb:91:in `initialize_database'

        bash("epc initial records") do
          action [:run]
          default_guard_interpreter :default
          interpreter "bash"
          declared_type :bash
          cookbook_name "rds_epc"
          code "        set -e\n        rm -rf /tmp/mysql-epc\n        mkdir /tmp/mysql-epc\n        cat > /tmp/mysql-epc/my.sql <<-'EOSQL'\nALTER USER 'root'@'localhost' IDENTIFIED BY '1J0cChMOhBEuO46G';\nDELETE FROM mysql.user WHERE USER LIKE '';\nDELETE FROM mysql.user WHERE user = 'root' and host NOT IN ('127.0.0.1', 'localhost');\nFLUSH PRIVILEGES;\nDELETE FROM mysql.db WHERE db LIKE 'test%';\nDROP DATABASE IF EXISTS test ;\nEOSQL\n       /usr/sbin/mysqld --defaults-file=/etc/mysql-epc/my.cnf --initialize --explicit_defaults_for_timestamp\n       /usr/sbin/mysqld --defaults-file=/etc/mysql-epc/my.cnf --init-file=/tmp/mysql-epc/my.sql --explicit_defaults_for_timestamp &\n       while [ ! -f /run/mysql-epc/mysqld.pid ] ; do sleep 1 ; done\n       kill `cat /run/mysql-epc/mysqld.pid`\n       while [ -f /run/mysql-epc/mysqld.pid ] ; do sleep 1 ; done\n       rm -rf /tmp/mysql-epc\n"
          umask "022"
          returns [0, 1, 2]
          domain nil
          user nil
          not_if { #code block }
        end

        System Info:
        ------------
        chef_version=17.0.242
        platform=ubuntu
        platform_version=20.04
        ruby=ruby 3.0.1p64 (2021-04-05 revision 0fb782ee38) [x86_64-linux]
        program_name=/usr/bin/chef-client
        executable=/opt/chef/bin/chef-client

[2021-06-22T13:10:42+00:00] INFO: Running queued delayed notifications before re-raising exception

      ================================================================================
      Error executing action `create` on resource 'mysql_service_manager_systemd[epc]'
      ================================================================================

      Mixlib::ShellOut::CommandTimeout
      --------------------------------
      bash[epc initial records] (/var/chef/cache/cookbooks/mysql/libraries/mysql_service_base.rb line 91) had an error: Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s:
      Command exceeded allowed execution time, process terminated
      ---- Begin output of "bash"  ----
      STDOUT:
      STDERR:
      ---- End output of "bash"  ----
      Ran "bash"  returned

      Resource Declaration:
      ---------------------
      # In /var/chef/cache/cookbooks/mysql/libraries/mysql_service.rb

       51:           svc = mysql_service_manager(new_resource.name, &block)
       52:         when 'sysvinit'

      Compiled Resource:
      ------------------
      # Declared in /var/chef/cache/cookbooks/mysql/libraries/mysql_service.rb:51:in `svc_manager'

      mysql_service_manager_systemd("epc") do
        action [:create]
        updated true
        updated_by_last_action true
        default_guard_interpreter :default
        declared_type :mysql_service_manager
        cookbook_name "rds_epc"
        bind_address "0.0.0.0"
        initial_root_password "1J0cChMOhBEuO46G"
        instance "epc"
        include_dir "/etc/mysql-epc/conf.d"
        data_dir "/var/lib/mysql-epc"
        run_user "mysql"
        run_group "mysql"
        version "8.0"
        pid_file "/run/mysql-epc/mysqld.pid"
        charset "utf8"
        port "3306"
        socket "/run/mysql-epc/mysqld.sock"
        error_log "/var/log/mysql-epc/error.log"
      end

      System Info:
      ------------
      chef_version=17.0.242
      platform=ubuntu
      platform_version=20.04
      ruby=ruby 3.0.1p64 (2021-04-05 revision 0fb782ee38) [x86_64-linux]
      program_name=/usr/bin/chef-client
      executable=/opt/chef/bin/chef-client

[2021-06-22T13:10:42+00:00] INFO: Running queued delayed notifications before re-raising exception

    ================================================================================
    Error executing action `create` on resource 'mysql_service[epc]'
    ================================================================================

    Mixlib::ShellOut::CommandTimeout
    --------------------------------
    mysql_service_manager_systemd[epc] (/var/chef/cache/cookbooks/mysql/libraries/mysql_service.rb line 51) had an error: Mixlib::ShellOut::CommandTimeout: bash[epc initial records] (/var/chef/cache/cookbooks/mysql/libraries/mysql_service_base.rb line 91) had an error: Mixlib::ShellOut::CommandTimeout: Command timed out after 3600s:
    Command exceeded allowed execution time, process terminated
    ---- Begin output of "bash"  ----
    STDOUT:
    STDERR:
    ---- End output of "bash"  ----
    Ran "bash"  returned

    Resource Declaration:
    ---------------------
    # In /var/chef/cache/cookbooks/rds_epc/recipes/mysql.rb

     10: mysql_service 'epc' do
     11:   initial_root_password node.run_state['root_pass']
     12:   bind_address node['rds_epc']['bind_address']
     13:   action [:create, :start]
     14: end
     15:

    Compiled Resource:
    ------------------
    # Declared in /var/chef/cache/cookbooks/rds_epc/recipes/mysql.rb:10:in `from_file'

    mysql_service("epc") do
      action [:create, :start]
      updated true
      updated_by_last_action true
      default_guard_interpreter :default
      declared_type :mysql_service
      cookbook_name "rds_epc"
      recipe_name "mysql"
      initial_root_password "1J0cChMOhBEuO46G"
      bind_address "0.0.0.0"
      install_method "auto"
      service_manager "auto"
    end

    System Info:
    ------------
    chef_version=17.0.242
    platform=ubuntu
    platform_version=20.04
    ruby=ruby 3.0.1p64 (2021-04-05 revision 0fb782ee38) [x86_64-linux]
    program_name=/usr/bin/chef-client
    executable=/opt/chef/bin/chef-client

at the same time, there are errors in the mysql error logs

I think the problem is that when installing mysql, mysql service run by default and block custom

I use:
Cookbook version mysql 10.1.3
Version of chef-client 17.0.242
OS Ubuntu 20.04

[StanislavBabkin]

in ubuntu 20, when the package is installed, the service starts and hold the port

I, on the advice of @aleksey-hariton-epam , tried such a temporary solution and it worked

apt_package 'mysql-server' do
  action :install
  notifies :stop, 'service[mysqld]', :immediately
end

service 'mysqld' do
  action :nothing
end

mysql_service 'epc' do
  bind_address node['rds_epc']['bind_address']
  port node['rds_epc']['mysqld']['port']
  initial_root_password node.run_state['root_pass']
  action [:create, :start]
end

ruby_block 'wait for service start' do
  block do
    sleep(30)
  end
  action :run
end

if node.run_state['rds_mode'] == 'service'

  mysql_database node.run_state['db_name'] do
    host '127.0.0.1'
    port node['rds_epc']['mysqld']['port'].to_i
    user 'root'
    password node.run_state['root_pass']
    action :create
  end

  mysql_user node.run_state['db_user'] do
    ctrl_password node.run_state['root_pass']
    ctrl_host '127.0.0.1'
    ctrl_port node['rds_epc']['mysqld']['port'].to_i
    password node.run_state['user_pass']
    database_name node.run_state['db_name']
    host '%'
    privileges [:select,:update,:insert]
    action [:create, :grant]
  end

end

@ramereth, I think you will be interested

[ramereth]

Feel free to make a PR which makes this work on Ubuntu 20.04 and we can review it!