litmus does not see bolt inventory plugins

[shaun-rutherford]

Describe the bug
Greetings,
It appears the PDK does not support bolts various inventory plugins for use with puppet_litmus.

I made sure the bolt and puppet_litmus gem PDK uses is the latest available and tried installing the puppetlabs-terraform bolt module via PDK, still fails with the following:

pdk bundle exec bolt module add puppetlabs-terraform
pdk (INFO): Using Ruby 2.5.8
pdk (INFO): Using Puppet 6.21.1
Bolt might be installed as a gem. To use Bolt reliably and with all of its
dependencies, uninstall the 'bolt' gem and install Bolt as a package:
https://puppet.com/docs/bolt/latest/bolt_installing.html

If you meant to install Bolt as a gem and want to disable this warning,
set the BOLT_GEM environment variable. [ID: gem_install]
Project configuration file /home/user/.puppetlabs/bolt/bolt-project.yaml already includes specification with name puppetlabs-terraform. Nothing to do.

pdk bundle exec rake litmus:install_agent
pdk (INFO): Using Ruby 2.5.8
pdk (INFO): Using Puppet 6.21.1
install_agent
rake aborted!
Bolt::Plugin::PluginError::Unknown: Unknown plugin: 'terraform'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:231:in `get_hook'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:311:in `resolve_single_reference'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:289:in `resolve_top_level_references'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:284:in `block in resolve_top_level_references'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:284:in `each'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:284:in `flat_map'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:284:in `resolve_top_level_references'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/inventory/group.rb:52:in `initialize'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/inventory/group.rb:84:in `new'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/inventory/group.rb:84:in `block in initialize'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/inventory/group.rb:84:in `map'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/inventory/group.rb:84:in `initialize'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/inventory/inventory.rb:24:in `new'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/inventory/inventory.rb:24:in `initialize'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/inventory.rb:114:in `new'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt/inventory.rb:114:in `create_version'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt_spec/run.rb:170:in `inventory'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt_spec/run.rb:195:in `run_task'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt_spec/run.rb:26:in `block in run_task'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt_spec/run.rb:152:in `block in with_runner'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt_spec/run.rb:150:in `with_runner'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/bolt-3.7.0/lib/bolt_spec/run.rb:25:in `run_task'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/puppet_litmus-0.27.0/lib/puppet_litmus/rake_helper.rb:225:in `block in install_agent'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/honeycomb-beeline-2.4.0/lib/honeycomb/client.rb:70:in `start_span'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/puppet_litmus-0.27.0/lib/puppet_litmus/rake_helper.rb:210:in `install_agent'
/home/srutherford/.pdk/cache/ruby/2.5.0/gems/puppet_litmus-0.27.0/lib/puppet_litmus/rake_tasks.rb:126:in `block (2 levels) in <top (required)>'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/honeycomb-beeline-2.4.0/lib/honeycomb/integrations/rake.rb:21:in `block in execute'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/honeycomb-beeline-2.4.0/lib/honeycomb/client.rb:70:in `start_span'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/honeycomb-beeline-2.4.0/lib/honeycomb/integrations/rake.rb:16:in `execute'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `load'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:63:in `kernel_load'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/cli/exec.rb:28:in `run'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/cli.rb:476:in `exec'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/cli.rb:30:in `dispatch'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/cli.rb:24:in `start'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/exe/bundle:46:in `block in <top (required)>'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/lib/bundler/friendly_errors.rb:123:in `with_friendly_errors'
/opt/puppetlabs/pdk/share/cache/ruby/2.5.0/gems/bundler-2.1.4/exe/bundle:34:in `<top (required)>'
/opt/puppetlabs/pdk/private/ruby/2.5.8/bin/bundle:23:in `load'
/opt/puppetlabs/pdk/private/ruby/2.5.8/bin/bundle:23:in `<main>'
Tasks: TOP => litmus:install_agent
(See full trace by running task with --trace)

To Reproduce
Download puppetlabs-motd
Add spec/fixtures/litmus_inventory.yaml with terraform plugin data:

groups:
  - name: cloud-webs
    targets:
      - _plugin: terraform
        dir: /home/srutherford/Documents/Git/Terraform/gcp_instance_test
        resource_type: google_compute_instance.default
        target_mapping:
          uri: network_interface.0.network_ip

Run pdk bundle install
Run pdk bundle exec rake litmus:install_agent
Run pdk bundle exec rake litmus:install_module

Expected behavior
The PDK should be able to run the bolt commands without error.

Additional context

• Your PDK installation method
  installed via native package on CentOS8

• Your PDK version (pdk --version)
  pdk --version
  2.1.0

• Your operating system / platform
  CentOS 8

I'm able to run commands using the Bolt I've installed from Puppetlabs CentOS repository using the terraform plugin:

bolt task run puppet_agent::install -i spec/fixtures/litmus_inventory.yaml --targets cloud-webs --run-as root
CLI arguments ["run-as"] might be overridden by Inventory: /home/srutherford/Documents/Git/Puppet/puppetlabs-motd/spec/fixtures/litmus_inventory.yaml [ID: cli_overrides]
Started on 10.180.23.173...
Finished on 10.180.23.173:
  19:33:54 +0000 INFO: Version parameter not defined and agent detected. Nothing to do.
Successful on 1 target: 10.180.23.173
Ran on 1 target in 1.55 sec

[jpogran]

@shaun-rutherford I think there are a couple of different things going on here.

Reading through https://puppet.com/docs/bolt/latest/bolt_installing_modules.html, I think you're missing a step in your commands.

bolt module add <module_name>
bolt module install

That being said, this issue is about using litmus and bolt. PDK is just the thing that executes those tools. If we can't figure it out here I'll move this over to the litmus repository

[shaun-rutherford]

@jpogran Thanks for the quick reply.

I ran the bold module add terraform before but missed the install bit. Unfortunately doesn't look that solved it either :(

bolt module install
Installing project modules

  → Syncing modules from /home/srutherford/.puppetlabs/bolt/Puppetfile to
    /home/srutherford/.puppetlabs/bolt/.modules

  → Generating type references

Successfully synced modules from /home/srutherford/.puppetlabs/bolt/Puppetfile to /home/srutherford/.puppetlabs/bolt/.modules

ls /home/srutherford/.puppetlabs/bolt/.modules/terraform/
bolt_plugin.json  CHANGELOG.md  CODE_OF_CONDUCT.md  CODEOWNERS  CONTRIBUTING.md  Gemfile  lib  LICENSE  metadata.json  plans  Rakefile  README.md  spec  tasks

pdk bundle exec rake litmus:install_module
pdk (INFO): Using Ruby 2.7.2
pdk (INFO): Using Puppet 7.5.0
Building '/home/srutherford/Documents/Git/Puppet/puppetlabs-motd' into '/home/srutherford/Documents/Git/Puppet/puppetlabs-motd/pkg'
Built '/home/srutherford/Documents/Git/Puppet/puppetlabs-motd/pkg/puppetlabs-motd-5.0.0.tar.gz'
rake aborted!
Bolt::Plugin::PluginError::Unknown: Unknown plugin: 'terraform'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:231:in `get_hook'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:311:in `resolve_single_reference'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:289:in `resolve_top_level_references'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:284:in `block in resolve_top_level_references'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:284:in `each'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:284:in `flat_map'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/plugin.rb:284:in `resolve_top_level_references'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/inventory/group.rb:52:in `initialize'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/inventory/group.rb:84:in `new'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/inventory/group.rb:84:in `block in initialize'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/inventory/group.rb:84:in `map'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/inventory/group.rb:84:in `initialize'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/inventory/inventory.rb:24:in `new'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/inventory/inventory.rb:24:in `initialize'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/inventory.rb:114:in `new'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt/inventory.rb:114:in `create_version'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt_spec/run.rb:170:in `inventory'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt_spec/run.rb:206:in `run_command'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt_spec/run.rb:62:in `block in run_command'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt_spec/run.rb:152:in `block in with_runner'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt_spec/run.rb:150:in `with_runner'
/home/srutherford/.pdk/cache/ruby/2.7.0/gems/bolt-3.7.0/lib/bolt_spec/run.rb:61:in `run_command'
/home/srutherford/.pdk/cache/ruby/2.7.0/bundler/gems/puppet_litmus-25b40b5b07b2/lib/puppet_litmus/rake_helper.rb:351:in `uninstall_module'
/home/srutherford/.pdk/cache/ruby/2.7.0/bundler/gems/puppet_litmus-25b40b5b07b2/lib/puppet_litmus/rake_helper.rb:310:in `block in install_module'
/opt/puppetlabs/pdk/share/cache/ruby/2.7.0/gems/honeycomb-beeline-2.4.0/lib/honeycomb/client.rb:70:in `start_span'
/home/srutherford/.pdk/cache/ruby/2.7.0/bundler/gems/puppet_litmus-25b40b5b07b2/lib/puppet_litmus/rake_helper.rb:302:in `install_module'
/home/srutherford/.pdk/cache/ruby/2.7.0/bundler/gems/puppet_litmus-25b40b5b07b2/lib/puppet_litmus/rake_tasks.rb:209:in `block (2 levels) in <top (required)>'
/opt/puppetlabs/pdk/share/cache/ruby/2.7.0/gems/honeycomb-beeline-2.4.0/lib/honeycomb/integrations/rake.rb:21:in `block in execute'
/opt/puppetlabs/pdk/share/cache/ruby/2.7.0/gems/honeycomb-beeline-2.4.0/lib/honeycomb/client.rb:70:in `start_span'
/opt/puppetlabs/pdk/share/cache/ruby/2.7.0/gems/honeycomb-beeline-2.4.0/lib/honeycomb/integrations/rake.rb:16:in `execute'
/opt/puppetlabs/pdk/share/cache/ruby/2.7.0/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
/opt/puppetlabs/pdk/private/ruby/2.7.2/bin/bundle:23:in `load'
/opt/puppetlabs/pdk/private/ruby/2.7.2/bin/bundle:23:in `<main>'
Tasks: TOP => litmus:install_module
(See full trace by running task with --trace)

[shaun-rutherford]

Hey @jpogran just checking in to see if you had any more information on this?

[jpogran]

Apologies for the delay, was working with the bolt and litmus teams to figure this out. The short of it is that litmus needs to be modified to 'see' the moduledir Bolt uses when running bolt module install, or be modified to wrap bolt commands such that the modules be 'installed' in the correct place and time during a litmus run.

To explain what's going on currently, bolt's bolt module install command writes to your project config file at /home/srutherford/.puppetlabs/bolt/.modules, which bolt recognizes on it's modulepath when it's run on your machine. Litmus doesn't know about that file and instead uses a custom created inventory file and points to its own modulepath (https://github.com/puppetlabs/puppet_litmus/blob/main/lib/puppet_litmus/rake_tasks.rb#L128), which all gets copied over or run directly on the target, depending on which part of the execution is happening.

Linking the two together is a task for the litmus team, so I'm moving this ticket over to that repo and will work with them to get a plan together for this.