rake task to gather modules from Puppetfile or .fixtures.yml

[logicminds]

Use Case

Our systems only use internal git repos, so installing the module with PMT doesn't satisfy the dependencies we have listed in fixtures.yml or Puppetfile for each module. All of our modules are on internal github/gitlab repositories.

Describe the Solution You Would Like

I would like to see a new rake task that runs the following:

1. rake spec_prep
2. tar -zcvf modules.tar.gz spec/fixtures
3. scp this file to the SUT
4. decompress under /etc/puppetlabs/code/production (or something similar)

Describe Alternatives You've Considered

1. Use rsync to transfer same files
2. Use a shared directory for vm or container
3. use bolt to run the prep command

Additional Context

Systems are not allowed to access the forge directly. The meatadata.json file does not support git urls or anything besides forge modules.

[tphoney]

Hey @logicminds i have "the fix" maybe :)
#192
This rake task allows a user to build all the modules in a directory copy the resultant tar files to the SUT, and force install them. By default it will look in the spec/fixtures/modules. folder. It will accept a folder path as a parameter too. (it ignores symlinked folders)

bundle exec rake litmus:install_modules_from_directory
# or 
bundle exec rake 'litmus:install_modules_from_directory[~/.puppetlabs/bolt/modules]'

Can we close this ticket, and open a new ticket if you have feedback with the new functionality

[logicminds]

That should work, except for the symlinks. The symlink just points to the module itself though. Are you also installing the module being tested or just the fixtures without symlinks?

[tphoney]

Hey @logicminds it could work for symlinked folders. But i was unsure about the recursion issue with the symlink, It can be changed.
So the behaviour now is to ignore any folder that is symlinked. So if you point it at the fixtures file, it will not install the module being tested. When installing modules this way. we force install, which does not install dependencies.

sorry that was meant to be a simple explanation. it turns out this isnt that simple

[tphoney]

So i removed the check for symlinks, while we were able to build the module, the metadata.json file was missing causing the install of the module under test to fail. I dont think it is a good idea to build symlinked folders.

➜  puppetlabs-motd git:(FM-7077) ✗ bundle exec rake litmus:install_modules_from_directory
Building
puppetlabs-motd-3.1.0.tar.gz puppetlabs-translate-2.0.0.tar.gz puppetlabs-facts-0.6.0.tar.gz puppetlabs-provision-0.1.0.tar.gz puppetlabs-registry-2.1.0.tar.gz puppetlabs-puppet_agent-2.2.0.tar.gz puppetlabs-stdlib-6.1.0.tar.gz
Sending
puppetlabs-motd-3.1.0.tar.gz puppetlabs-translate-2.0.0.tar.gz puppetlabs-facts-0.6.0.tar.gz puppetlabs-provision-0.1.0.tar.gz puppetlabs-registry-2.1.0.tar.gz puppetlabs-puppet_agent-2.2.0.tar.gz puppetlabs-stdlib-6.1.0.tar.gz
Installing
puppetlabs-motd-3.1.0.tar.gz puppetlabs-translate-2.0.0.tar.gz puppetlabs-facts-0.6.0.tar.gz puppetlabs-provision-0.1.0.tar.gz puppetlabs-registry-2.1.0.tar.gz puppetlabs-puppet_agent-2.2.0.tar.gz puppetlabs-stdlib-6.1.0.tar.gz %
➜  puppetlabs-motd git:(FM-7077) ✗ bundle exec bolt --modulepath /home/tp/workspace/:/home/tp/.puppetlabs/bolt/modules -i inventory.yaml command run 'puppet module list'  --targets '*'
Started on aujzpy197rihkfh.delivery.puppetlabs.net...
Finished on aujzpy197rihkfh.delivery.puppetlabs.net:
  STDOUT:
    /etc/puppetlabs/code/environments/production/modules
    ├── puppetlabs-facts (v0.6.0)
    ├── puppetlabs-provision (v0.1.0)
    ├── puppetlabs-puppet_agent (v2.2.0)
    ├── puppetlabs-registry (v2.1.0)
    ├── puppetlabs-stdlib (v6.1.0)
    └── puppetlabs-translate (v2.0.0)
/etc/puppetlabs/code/modules (no modules installed)
    /opt/puppetlabs/puppet/modules (no modules installed)
  STDERR:
    Warning: Missing dependency 'puppetlabs-apt':
'puppetlabs-puppet_agent' (v2.2.0) requires 'puppetlabs-apt' (>= 7.0.1 < 8.0.0)
    Warning: Missing dependency 'puppetlabs-inifile':
'puppetlabs-puppet_agent' (v2.2.0) requires 'puppetlabs-inifile' (>= 2.4.0 <= 4.0.0)
Successful on 1 node: aujzpy197rihkfh.delivery.puppetlabs.net
Ran on 1 node in 1.62 sec
➜  puppetlabs-motd git:(FM-7077) ✗

On the SUT.

root@aujzpy197rihkfh:/tmp# ls
puppetlabs-facts-0.6.0.tar.gz  puppetlabs-provision-0.1.0.tar.gz     puppetlabs-registry-2.1.0.tar.gz  puppetlabs-translate-2.0.0.tar.gz  tmp.eBfG1pHlZn  vmware-root
puppetlabs-motd-3.1.0.tar.gz   puppetlabs-puppet_agent-2.2.0.tar.gz  puppetlabs-stdlib-6.1.0.tar.gz    stderr.1577.6232                   tmp.OEl6MnDdCC  vsphere-bootstrap.log
root@aujzpy197rihkfh:/tmp# puppet module install puppetlabs-motd-3.1.0.tar.gz
Notice: Preparing to install into /etc/puppetlabs/code/environments/production/modules ...
Error: No valid metadata.json found!
Error: Try 'puppet help module install' for usage
root@aujzpy197rihkfh:/tmp#

[baron1405]

I have a similar situation in that my modules are for internal use and therefore are not published on Puppet Forge. All my modules and node definitions are in one git repo and I resolve includes using modulepath in environment.conf. How can I resolve the includes during litmus acceptance testing? I tried playing around with install_modules_from_directory but it ignored the other module's fixtures directory because it saw a symlink (even though it was to the module itself). Any suggestions would be greatly appreciated.

[baron1405]

Seems like what is missing is the following:

• Determine direct and transitive dependencies of a module. In my case, the .fixtures.yml file symlinks express those dependencies well.
• For each dependency, run a PDK build in that module's directory
• Upload the built module into the target
• Install the uploaded module

[trevor-vaughan]

PDK build may not work well for testing if it tries to validate all dependencies.

You often want to test against master which may be unstable and have metadata versions that are not well formed.

If you want to do this, PDK needs to loosen up and allow 'bad things' because we're in the middle of testing.

[baron1405]

Thanks for your thoughts @trevor-vaughan. Since all my modules are in the same git repo, they are effectively versioned together. I actually want my acceptance tests to run against my local master.

[baron1405]

To anyone who stumbles upon this issue, my approach outlined above works. I use Gradle as my top level build tool for running PDK so I simply invoke PDK and Bolt directly from a Gradle task. The steps to make dependencies available are:

• Use snakeYAML to read the .fixtures.yml files
• Treat the symlinks as direct dependencies and follow them recursively to obtain transitive dependencies
• Visit the module directory of each dependency and run PDK build
• Use Bolt to upload the dependency module tarball into the SUT (nodes 'all', point at inventory.yaml file)
• Use Bolt to run the puppet install command for the uploaded module in the SUT (nodes 'all', point at inventory.yaml file)

This is not meant as a solution that will work in all use cases but can serve as food for thought for those with more complex environments.

[DavidS]

hi @baron1405 ,

thanks for sharing your solution. There seem to be two overlapping issues. I'll try to dissect them below, please confirm whether I've covered everything, or if there's something missing.

• recursive dependency resolution: This is currently a gap in our tooling (fixtures, metadata, r10k). While there has been recent progress (https://github.com/lingua-pupuli/puppetfile-resolver ) this has not yet been integrated. Until then (and possibly even beyond, depending on the solution we finally come to) the .fixtures.yml file should contain the full list of test dependencies. The rule is "if your tests require bits-on-disk from somewhere for testing, it needs to be listed in .fixtures.yml".

• symlink handling: TP's implementation skipped that part to reduce complexity and judge usefulness of the install_modules_from_directory approach. I think the conversation here supports the conclusion that install_modules_from_directory would work for you if it was dealing with symlinks too.

[ekohl]

The rule is "if your tests require bits-on-disk from somewhere for testing, it needs to be listed in .fixtures.yml".

Perhaps this should be rephrased to rake spec_prep should make sure they're present. .fixtures.yml is just the default implementation but users can replace/enhance the task. We should make it clear that litmus itself doesn't define that task.

[baron1405]

Thanks @DavidS and @ekohl for your responses.

As far as dependency management goes, I am used to the way Gradle and Maven resolve dependencies. The developer lists their direct dependencies and transitive dependency resolution is handled by the tooling.

If symlinks were processed, things would probably work for me but I cannot say definitively. One thing that concerns me about the use of symlinks is that there is a symlink pointing to the module itself. I have noticed that this can confuse IDEs. For example, IntelliJ IDEA 2019.3 gets bogged down when I right click on a module. I suspect it is the circular reference (module contains spec/fixtures/modules -> module).

[trevor-vaughan]

Symlinks are insanely useful for code reuse across tests.

IMO Broken IDEs should have bugs filed against them.

[baron1405]

@trevor-vaughan the good news is that at least IntelliJ 2019.3 get a bit sluggish. 2019.2 would go off into an infinite loop.

Speaking of reuse, for my use case where I am not publishing modules to Puppet Forge, it would be great to avoid the duplication of test framework files in each module (e.g. provision.yml). By no means a high priority issue, but since the topic came up.