Correctly exclude files from Rsync

CallumND · December 21, 2017, 1:05am

I used Rsync to deploy my kirby site and the/panel was returning a 500 error. I looked in the error.log and had this error:

[Thu Dec 21 00:55:34.019386 2017] [:error] [pid 31851] [client 80.189.19.63:59473] PHP Fatal error:  Uncaught Exception: The package.json is missing for the translation with code: en in /var/www/example.com/public_html/panel/app/src/panel/translation.php:41\nStack trace:\n#0 /var/www/example.com/public_html/panel/app/src/panel.php(371): Kirby\\Panel\\Translation->__construct(Object(Kirby\\Panel), 'en')\n#1 /var/www/example.com/public_html/panel/app/src/panel.php(118): Kirby\\Panel->translation()\n#2 /var/www/example.com/public_html/panel/index.php(41): Kirby\\Panel->__construct(Object(Kirby), '/var/www/exampl...')\n#3 {main}\n  thrown in /var/www/example.com/public_html/panel/app/src/panel/translation.php on line 41

I realised it’s because my rsync_exludes.txt file had

node_modules/
gulpfile.js
assets/scss/
assets/js/
assets/css/

Which excluded these files in the panel folder as well. How should I correctly exclude these working files?

bvdputte · December 21, 2017, 7:03am

Have you tried adding a “/” to your exclude globs in your rsync_excludes.txt?
This should make paths absolute.

E.g.:

/node_modules/
/gulpfile.js
/assets/scss/
/assets/js/
/assets/css/

jimbobrjames · December 21, 2017, 11:14am

I think it depends on how you run the rysnc command:

rsync -az --force --delete --progress --exclude-from=rsync_exclude.txt -e "ssh -p22" ./public/ user@domain.com:/var/www/vhosts/yoursitefolder`

the important bit is:

./public/ - this gives the starting folder, in this case anything inside the public folder. You don’t need the absolute paths. The dot at the start means start from where the command is run.

Hint: I do it with a bash script (deploy.sh) in my project root (remember to make it executable) and kick it off with NPM scripts.

#!/bin/bash

ERRORSTRING="Whoops! That didn't work. Please check the command you ran."
if [ $# -eq 0 ]
then
  echo "$ERRORSTRING";

elif [[ "$1" == "live" ]]
then
  if [[ -z $2 ]]
  then
    echo "Running live dry-run"
    rsync --dry-run -az --force --delete --progress --exclude-from=rsync_exclude.txt -e "ssh -p22" ./public/ user@domain.com:/var/www/vhosts/yoursitefolder
  elif [[ "$2" == "go" ]]
  then
    echo "Running live actual deploy"
    rsync -az --force --delete --progress --exclude-from=rsync_exclude.txt -e "ssh -p22" ./public/ user@domain.com:/var/www/vhosts/yoursitefolder
  else
    echo "$ERRORSTRING";
  fi

elif [[ "$1" == "staging" ]]
then
  if [[ -z $2 ]]
  then
    echo "Running staging dry-run"
    rsync --dry-run -az --force --delete --progress --exclude-from=rsync_exclude.txt -e "ssh -p22" ./public/ user@domain.com:/var/www/vhosts/yoursitefolder
  elif [[ "$2" == "go" ]]
  then
    echo "Running staging actual deploy"
    rsync -az --force --delete --progress --exclude-from=rsync_exclude.txt -e "ssh -p22" ./public/ user@domain.com:/var/www/vhosts/yoursitefolder
  else
    echo "$ERRORSTRING";
  fi

elif [[ "$1" == "sync" ]]
then
  if [[ -z $2 ]]
  then
    echo "Running sync dry-run"
    rsync --dry-run -az --force --delete --progress --exclude-from=rsync_exclude.txt -e "ssh -p22" user@domain.com:/var/www/vhosts/yoursitefolder/content ./public/content
  elif [[ "$2" == "go" ]]
  then
    echo "Running sync actual deploy"
    rsync -az --force --delete --progress --exclude-from=rsync_exclude.txt -e "ssh -p22" user@domain.com:/var/www/vhosts/yoursitefolder/content ./public/content
  else
    echo "$ERRORSTRING";
  fi

fi

NPM scripts in my package.json:

    "deploy:live:sim": "./deploy live",
    "deploy:live": "./deploy live go",
    "deploy:staging:sim": "./deploy staging",
    "deploy:staging": "./deploy staging go",

My exclude file looks like this:

.git
.gitignore
.gitkeep
.DS_Store
fonts.txt

To simulate the deploy without the deploy actually happening just do npm deploy:live:sim. You get the idea

It wouldn’t take much to extend that script a little so it pulls down content from live.

jimbobrjames · December 22, 2017, 11:13am

I figured out content syncing from live server to local. Change the bash script:

#!/bin/bash

ERRORSTRING="Whoops! That didn't work. Please check the command you ran."
if [ $# -eq 0 ]
then
  echo "$ERRORSTRING";

elif [[ "$1" == "sync" ]]
then
  if [[ -z $2 ]]
  then
    echo "Running content sync from live dry-run"
    rsync --dry-run -az --force --delete --progress --exclude-from=rsync_exclude.txt -e "ssh -p22" exampleuser@11.22.3.444:/var/www/vhosts/example.com/httpdocs/content/ ./public/content/
  elif [[ "$2" == "go" ]]
  then
    echo "Running content sync from live actual"
    rsync -az --force --delete --progress --exclude-from=rsync_exclude.txt -e "ssh -p22" exampleuser@11.22.3.444:/var/www/vhosts/example.com/httpdocs/content/ ./public/content/
  else
    echo "$ERRORSTRING";
  fi

elif [[ "$1" == "live" ]]
then
  if [[ -z $2 ]]
  then
    echo "Running live dry-run"
    rsync --dry-run -az --force --delete --progress --exclude-from=rsync_exclude.txt -e "ssh -p22" ./public/ exampleuser@11.22.3.444:/var/www/vhosts/example.com/httpdocs
  elif [[ "$2" == "go" ]]
  then
    echo "Running live actual deploy"
    rsync -az --force --delete --progress --exclude-from=rsync_exclude.txt -e "ssh -p22" ./public/ exampleuser@11.22.3.444:/var/www/vhosts/example.com/httpdocs
  else
    echo "$ERRORSTRING";
  fi

elif [[ "$1" == "staging" ]]
then
  if [[ -z $2 ]]
  then
    echo "Running staging dry-run"
    rsync --dry-run -az --force --delete --progress --exclude-from=rsync_exclude.txt -e "ssh -p22" ./public/ exampleuser@11.22.3.444:/var/www/vhosts/example.com/subdomains/test.example.com
  elif [[ "$2" == "go" ]]
  then
    echo "Running staging actual deploy"
    rsync -az --force --delete --progress --exclude-from=rsync_exclude.txt -e "ssh -p22" ./public/ exampleuser@11.22.3.444:/var/www/vhosts/example.com/subdomains/test.example.com
  else
    echo "$ERRORSTRING";
  fi
fi

Also add these lines to your package.json:

"content:sync:sim": "./deploy sync",
"content:sync": "./deploy sync go"

texnixe · December 22, 2017, 12:08pm

Keep in mind that syncing from the server to local using rsync only makes sense for stuff that is only changed on the server. There is no way to resolve conflicts if things have changed on both local and the server and things are just overwritten.

jimbobrjames · December 22, 2017, 12:25pm

@texnixe Sure, some kind of source control is your friend here, and it’s good practice to keep backups. Personally my work flow is always to work locally in a Git repo and push up to the server using the above method. I never edit on the live site. I do have one or two clients who edit on the live server, and this serves me well enough to bring down the changes they make.

CallumND · December 29, 2017, 2:18am

Thank you @jimbobrjames for a thorough outline and explanation of the process, I set up something similar but I’ve started encountering permission errors. Like the following:

rsync: failed to set times on "/var/www/example.com/public_html/.": Operation not permitted (1)
rsync: failed to set permissions on "/var/www/example.com/public_html/.htaccess": Operation not permitted (1)
rsync: failed to set permissions on "/var/www/example.com/public_html/deploy": Operation not permitted (1)
./
.DS_Store
         14,340 100%   12.82MB/s    0:00:00 (xfr#1, ir-chk=1023/1025)
rsync: failed to set times on "/var/www/example.com/public_html/assets": Operation not permitted (1)
rsync: failed to set times on "/var/www/example.com/public_html/assets/production": Operation not permitted (1)
rsync: failed to set permissions on "/var/www/example.com/public_html/assets/sounds/sound1.ogg": Operation not permitted (1)
rsync: failed to set permissions on "/var/www/example.com/public_html/assets/sounds/sound2.ogg": Operation not permitted (1)
rsync: connection unexpectedly closed (41528 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.2]

I think this is a result of having to change permissions to access the panel. Does anyone have and good suggestions on how to fix this so that rsync is seamless as well as logging into the panel?

jimbobrjames · December 29, 2017, 8:51am

It looks like you have real issues with the file permissions differing on files across your whole site. I would reset the ownership on all files to use the user that cPanel or Plesk or whatever control panel you use is using. When you rSync, you must also use this user to connect with SSH.

I think your problem is that the files in the error you listed above are in a different user:group to the user being used for rSync.

If you are using cPanel, there are scripts and guides that can do this for you.

Topic		Replies	Views
How do I keep my local files in sync when I make changes on the website panel? Questions	16	578	September 29, 2022
Error 500 on Panel (Nginx, PHP 7.3) Questions v3	19	9525	June 13, 2019
Panel assets not stored in media folder - error Questions v3	6	685	April 17, 2019
Panel problem because of .htaccess Questions v3	3	262	August 25, 2022
500 error on new clean Kirby install Questions v2	8	1095	February 6, 2025

Correctly exclude files from Rsync

Related topics