Consider overwrite: true as default

[timkelty]

Currently, with the defaults, the shared task will always fail if you have a directory, because we mkdir -p on the source and target of the symlink.

Lets figure out the best way to avoid this.

[tlartaud]

Hi @timkelty

I hope you are fine.

I was asking me there why I got some errors messages on mkdir -p

It doesn't fail on the really first publish, but you should know that shipit, before the rsync copy, perform a copy of all files from last release folder to new release folder. That's why mkdir fail, because on the second deployment, you are going to have symlink files in that place.

The option shared.overwrite is not working ? I've added it to my config, but it seems it has no effect.

What is the workaround for now ? I should use full config only for directories and not strings, that's it ?

            "shared": {
                "overwrite": true,
                "triggerEvent": "updated",
                "dirs": [
                    {
                        "path": "<%= app.dirs.webRoot %>/app/uploads",
                        "overwrite": true,
                        "chmod": "-R 755"
                    },
                    {
                        "path": "<%= app.dirs.webRoot %>/app/cache",
                        "overwrite": true,
                        "chmod": "-R 755"
                    },
                    {
                        "path": "<%= app.dirs.webRoot %>/app/wp-rocket-config",
                        "overwrite": true,
                        "chmod": "-R 755"
                    }
                ],
                "files": [
                    ".env",
                    "<%= app.dirs.webRoot %>/.htaccess",
                    "<%= app.dirs.webRoot %>/app/advanced-cache.php"
                ]
            }

If we use a directory config that is not a json object, but a string, it will automatically fail on the second deploy.

Regards.

[tlartaud]

I can confirm it DOESNTworks with the above configuration too.

[timkelty]

Can you post the output of the error?

[timkelty]

@tlartaud You can now set everything on the top level shared and have it inherited by files and dirs, so you could set "overwrite": true, "chmod": "-R 755" on shared and then just use strings.

Fixed in 4.2, #31

[tlartaud]

@timkelty
Hi,

This won't fix anything. As I said, the process is broken.
The first deploy run fine, the second one will fail if I don't manually remove all the symlinks created on the first deploy. See logs below.

[03:56 ]-[vagrant@vm]-[/var/www]-[git master]
$ grunt publish:production
Running "publish:production" (publish) task

Running "shipit:production" (shipit) task

Running "deploy:init" task

Running "construct:pre_deploy" task

Running "checkbranch:develop:true" (checkbranch) task
Expecting to  be on 'develop' branch.

Running "checkpending" task

Running "prompt:bump" (prompt) task
? Bump version from 0.0.12 to: Patch:  0.0.13 Backwards-compatible bug fixes.
? Use default values (bump.js) ? No
? What should get the new version: package.json
? Should bump changes and changelog be committed ? Yes
? Which files should be commited: package.json, CHANGELOG.md
? Create a new tag ? Yes
? Push commited files to repository ? Yes

Running "bump-only:patch" (bump-only) task

Running "bump:patch:bump-only" (bump) task
>> Version bumped to 0.0.13 (in package.json)
>> pkg's version updated

Running "shell:changelog" (shell) task

Running "bump::commit-only" (bump) task
>> Committed as "Release v0.0.13"
>> Tagged as "v0.0.13"
>> Pushed to origin

Running "gitcheckout:master" (gitcheckout) task

Running "gitmerge:from-develop" (gitmerge) task

Running "gitpush:master" (gitpush) task

Running "deploy:fetch" task
Deleting existing workspace "tmp/workspace"
Running "rm -rf tmp/workspace" on local.
Create workspace "tmp/workspace"
Workspace created.
Initialize local repository in "tmp/workspace"
Running "git init" on local.
@ Initialized empty Git repository in /var/www/tmp/workspace/.git/
Repository initialized.
List local remotes.
Running "git remote" on local.
Update remote "git@git.domain.com:wpca/master-demo.git" to local repository "tmp/workspace"
Running "git remote add shipit git@git.domain.com:wpca/master-demo.git" on local.
Remote updated.
Fetching repository "git@git.domain.com:wpca/master-demo.git"
Running "git fetch --depth=1 shipit -p" on local.
@ From git.domain.com:wpca/master-demo
@  * [new branch]      develop    -> shipit/develop
@  * [new branch]      master     -> shipit/master
@  * [new tag]         v0.0.13    -> v0.0.13
Repository fetched.
Checking out commit-ish "master"
Running "git checkout master" on local.
@ Already on 'master'
@ Branch master set up to track remote branch master from shipit.
Checked out.
Reseting the working tree
Running "git reset --hard HEAD" on local.
@ HEAD is now at a1b8c20 Release v0.0.13
Reset working tree.
Testing if commit-ish is a branch.
Running "git branch --list master" on local.
@ * master
Commit-ish is a branch, merging...
Running "git merge shipit/master" on local.
@ Already up-to-date.
Branch merged.

Running "deploy:update" task
Running "if [ -h /my-project.domain.com/current ]; then readlink /my-project.domain.com/current; fi" on host "my-project.domain.com".
@my-project.domain.com releases/20151215155200
Running "ls -r1 /my-project.domain.com/releases" on host "my-project.domain.com".
@my-project.domain.com 20151215155200
@my-project.domain.com 20151117185159
Previous release found.
Running "if [ -f /my-project.domain.com/releases/20151117185159/REVISION ]; then cat /my-project.domain.com/releases/20151117185159/REVISION 2>/dev/null; fi;" on host "my-project.domain.com".
@my-project.domain.com 2d86795ed4f0636338d4f634c1120523495e21e1
Previous revision found.
Create release path "/my-project.domain.com/releases/20151215170654"
Running "mkdir -p /my-project.domain.com/releases/20151215170654" on host "my-project.domain.com".
Release path created.
Running "cp -a /my-project.domain.com/releases/20151117185159/. /my-project.domain.com/releases/20151215170654" on host "my-project.domain.com".
Copy project to remote servers.
Copy "/var/www/tmp/workspace/" to "deployuser@my-project.domain.com:/my-project.domain.com/releases/20151215170654" via rsync
Finished copy.
Setting current revision and creating revision file.
Running "git rev-parse master" on local.
@ a1b8c2060171396669a20ffbe2c5ef2fddce2e59
Running "echo "a1b8c2060171396669a20ffbe2c5ef2fddce2e59" > /my-project.domain.com/releases/20151215170654/REVISION" on host "my-project.domain.com".
Revision file created.

Running "shared:create-dirs" task
Creating shared directories on remote.
Running "mkdir -p /my-project.domain.com/shared/public/app/uploads" on host "my-project.domain.com".
Running "mkdir -p /my-project.domain.com/shared/public/app/cache" on host "my-project.domain.com".
Running "mkdir -p /my-project.domain.com/shared/public/app/wp-rocket-config" on host "my-project.domain.com".
Running "mkdir -p $(dirname /my-project.domain.com/shared/.env)" on host "my-project.domain.com".
Running "mkdir -p $(dirname /my-project.domain.com/shared/public/.htaccess)" on host "my-project.domain.com".
Running "mkdir -p $(dirname /my-project.domain.com/shared/public/app/advanced-cache.php)" on host "my-project.domain.com".
Running "mkdir -p /my-project.domain.com/releases/20151215170654/public/app/uploads" on host "my-project.domain.com".
Running "mkdir -p /my-project.domain.com/releases/20151215170654/public/app/cache" on host "my-project.domain.com".
Running "mkdir -p /my-project.domain.com/releases/20151215170654/public/app/wp-rocket-config" on host "my-project.domain.com".
Running "mkdir -p $(dirname /my-project.domain.com/releases/20151215170654/.env)" on host "my-project.domain.com".
Running "mkdir -p $(dirname /my-project.domain.com/releases/20151215170654/public/.htaccess)" on host "my-project.domain.com".
Running "mkdir -p $(dirname /my-project.domain.com/releases/20151215170654/public/app/advanced-cache.php)" on host "my-project.domain.com".
@my-project.domain.com-err mkdir: cannot create directory '/my-project.domain.com/releases/20151215170654/public/app/cache': File exists
Warning: Command failed: /bin/sh -c ssh deployuser@my-project.domain.com "mkdir -p /my-project.domain.com/releases/20151215170654/public/app/cache"
mkdir: cannot create directory '/my-project.domain.com/releases/20151215170654/public/app/cache': File exists
 Use --force to continue.

Aborted due to warnings.


Execution Time (2015-12-15 17:00:02 UTC)
checkpending        12.5s  ▇ 2%
prompt:bump          9.4s  ▇ 1%
shell:changelog       50s  ▇▇▇ 6%
deploy:fetch        53.9s  ▇▇▇▇ 7%
deploy:update      6m 32s  ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 82%

Explanations of why it fails on the second deployment
Reading this log is enough to clearly understand what is happening. Here are the main steps :

• Shipit-deploy create the new release folder : mkdir -p /my-project.domain.com/releases/20151215170654
• Shipit-deploy get the previous release folder, and copy everything in this folder to the newly created one (do you know anyway to disable this ? this is really a non-sense ...), so at this point, our newly created release folder will contain all the symlinks file that have been created on the first deployment : cp -a /my-project.domain.com/releases/20151117185159/. /my-project.domain.com/releases/20151215170654
• Shipit-deploy fetch all the things locally and then copy everything to the remote server
• Shipit-shared will then fail everything by trying to run a command to create a directory where a symlink already exists in that place : mkdir -p /my-project.domain.com/releases/20151215170654/public/app/cache

I think that u can delete the enhancement label and replace it with something like major bug

Regards.

[timkelty]

Shipit-deploy get the previous release folder, and copy everything in this folder to the newly created one (do you know anyway to disable this ? this is really a non-sense ...)

This isn't non-sense. shipit-deploy does this so the following rsync command will take a fraction of the time. Otherwise rsync would have to copy all files on deploy, regardless of any change. I don't believe shipit-deploy has an option to disable this.

[timkelty]

Can you give me some information about your remote environment?

mkdir -pshould NOT throw an error if the target already exists, be it a folder or symlink. It doesn't for me, or anywhere else I've seen.

-p
Create intermediate directories as required. If this option is not specified, the full path prefix of each operand must already exist. Intermediate directories are created with permission bits of rwxrwxrwx (0777) as modified by the current umask, plus write and search permission for the owner. Do not consider it an error if the argument directory already exists.

Regardless, this should be easy to circumvent with a simple test, but I'm still curious about your remote environment, as it doesn't happen for me, nor have I ever seen it.

[tlartaud]

Yes, I really feel this is a non-sense. What happens if in my branch, i delete some files, and then push my site live ? Thoses files will remain on the remote because it has been previously copied from the last release. I didn't look at rsync arguments, so maybe they are correctly set and make delete the file that are not anymore in the built tmp/workspace folder, I didn't test, but i'm really not sure about that.

Can you show me where do you read exactly that mkdir -p won't throw an error please ? I'm really curious. Cant read this on any man page. From what I can read here, the -p argument is just allowing us to create intermediate directories in a single command. It also tell us that if one of the DIRECTORIES ALREADY EXIST, it won't throw an error. It doesn't say that if A FILE/SYMLINK EXIST IN THAT PLACE it won't throw an error.

[tlartaud]

Hmmm, strange.

Following test is working on the remote

bash-4.3$ cd ~
bash-4.3$ mkdir foldertolink
bash-4.3$ ln -s foldertolink testsymlink
bash-4.3$ mkdir -p testsymlink/subfolder
bash-4.3$ cd foldertolink/
bash-4.3$ cd testsymlink
bash-4.3$ ls -al
total 12
drwxr-xr-x  3 tlartaud psacln  4096 Dec 16 09:50 .
drwx--x--- 24 tlartaud psaserv 4096 Dec 16 09:49 ..
drwxr-xr-x  2 tlartaud psacln  4096 Dec 16 09:50 subfolder

Trying to make more tests...

[tlartaud]

Ok, I got it and know what is happenning. You were right in one point. mkdir -p won't throw an error if a symlink is found, but it will in the case that the symlink cant be followed. Here is how I reproduced the bug

bash-4.3$ cd /var/www
bash: cd: /var/www: No such file or directory
bash-4.3$ ln -s /var/www testsymlink
bash-4.3$ mkdir -p testsymlink/foldertolinkoutsidechroot
mkdir: cannot create directory 'testsymlink': File exists

In my case, I'm working on a chrooted environment. So, when my deploy user connect to the remote, he's located in the path /var/www/vhosts/domain.com. I use the symlink config for chrooted envs, so my symlinks are pointing to the full path location (it means that when im connected with the chrooted user, I simply CANT follow the symlink, only apache can [or users who have access to the server root]). This should help you to understand the structure :

bash-4.3$ pwd
/subdomain.domain.com/current/public/app
bash-4.3$ ls -al
total 104
drwxrwxr-x 7 tlartaud psacln  4096 Dec 15 18:10 .
drwxr-xr-x 3 tlartaud psacln  4096 Dec 15 17:59 ..
lrwxrwxrwx  1 tlartaud psacln   87 Dec 15 18:10 cache -> /var/www/vhosts/domain.com/subdomain.domain.com/shared/public/app/cache

In this case, the problem is that the deploy user (which is chrooted), can't follow the symlink and access the full path because it is located outside of the environment. So this is mainly a permission error : In my chrooted environment, I also have a var directory (used to import some libs to make them available on the chrooted env), so the mkdir -p command tries to go in that directory which is owned by the root user. So it can't even read subdirectory or add some new.

[tlartaud]

The proper solution is simply to change the path so you create subfolders directly where they should go without passing by the symlink which cant be read by the deploy user on chrooted environment.

So, instead of this command

mkdir -p /subdomain.domain.com/releases/20151215170654/public/app/cache

We should normally execute this one directly without going trough the symlink

mkdir -p /subdomain.domain.com/shared/public/app/cache

Voilà ;)

[tlartaud]

In create-dirs.js, I can read this function

var getPathStr = function(el, basePath) {
      basePath = basePath || shipit.config.shared.basePath;
      var filePath = shipit.config.shared.remote ? path.join(basePath, el.path) : el.path;

      return el.isFile ? util.format('$(dirname %s)', filePath) : filePath;
    }

I added a line of code (didn't try yet, push is running), I think it should do the trick

    var filePath = shipit.config.shared.remote ? path.join(basePath, el.path) : el.path;
    filePath = 'symlinkPath' in shipit.config.shared ? path.join(shipit.config.shared.basePath, el.path) : filePath;

Will let you know if that works.

Note : Could you explain why you use the release path instead of the shared path for creating those subdirectories ? Any specific reason ?

Regards.

[timkelty]

Could you explain why you use the release path instead of the shared path for creating those subdirectories ? Any specific reason ?

If you look at the code, and even your output, it is making directories in shared as well. It is ensuring the directory structures exist in both shared and releasePath, so that the symlink can succeed.

I see your problem with a symlink that can't resolve w/ a chrooted env, and I will address it.

[tlartaud]

Okay, this is my final report related to this issue (even if, sorry, this is not related to the overwrite option).

The issue I'm facing is about a conflict/bug when using the symlinkPath option. I know you are not using that option because you made it available for me when i asked for it, but if we use it, actually, it fails on the second deployment.

I've tested it by removing the current symlink and the releases folder, and then deployed my website twice (sorry, it took a while, my deployment process is just so long).

The current solution I provided works for me (the second deployment did not fail). Could you maybe try to implement it, check if it doesn't affect the deployment process on your side (I mean, when not using the symlinkPath option), and add it to your code ?

If you look at the code, and even your output, it is making directories in shared as well. It is ensuring the directory structures exist in both shared and releasePath, so that the symlink can succeed.

Well, since the directories managed by shipit-shared in the releasePath are in fact symlinks pointing to the sharedPath, we should not need to do it twice. Maybe you have a use case where its needed and I understand that, but I think we should fix the use case where a user have a chrooted environment where symlink are not readable by this user (I mean, the user can't follow symlinks by using cd current/my-symlink/uploads)

Thanks a lot.