0

I'm trying to create a custom DDEV Provider, to import the current database and also user generated files from the web server.

I want to use it with TYPO3 Projects, where I develop the EXT locally with DDEV (because its awesome :) ) and I want to update my local database and also the "fileadmin" files with the help of the ddev pull function.

I've read the docs: Introduction to Hosting Provider Integration and I tested the bash commands locally within the DDEV Container (ddev ssh) and I'm able to connect to the remote Webserver and make a database dump and transfer it to the local DDEV container.

So I added the bash commands to the my custom provider .yaml file in the /provider/ folder.

Here is the current file:

environment_variables:
  DB_NAME: db_name
  DB_USER: password
  DB_PASSWORD: password
  HOST_IP: 11.11.11.11
  SSH_USERNAME: username
  SSH_PASSWORD: password
  SSH_PORT: 22

db_pull_command:
  command: |
    # Creates the .download folder if it doesn't exist
    mkdir -p /var/www/html/.ddev/.downloads
    # execute the mysqldump on the remote webserver via SSH
    ssh -p ${SSH_PORT} ${SSH_USERNAME}@${HOST_IP} 'mysqldump -h 127.0.0.1 -u ${DB_USER} -p ${DB_PASSWORD} ${DB_NAME} > /tmp/${DB_NAME}.sql.gz'
    # download to sql file to the ddev folder
    scp -P ${SSH_PORT} ${SSH_USERNAME}@${HOST_IP}:/tmp/${DB_NAME}.sql.gz /var/www/html/.ddev/.downloads/db.sql.gz.

If I execute the pull with ddev pull my-provider I get the following Error:

Downloading database...
bash: 03: command not found
Pull failed: Failed to exec mkdir -p /var/www/html/.ddev/.downloads

I assumed that the commands are executed like I would within the DDEV Container (with ddev ssh). What am I missing?

My Environment:

  • TYPO3 v10.4.20
  • Windows 10 (WSL)
  • Docker Desktop 3.5.2
  • DDEV-Local version v1.17.7
  • architecture amd64
  • db drud/ddev-dbserver-mariadb-10.3:v1.17.7
  • dba phpmyadmin:5
  • ddev-ssh-agent drud/ddev-ssh-agent:v1.17.0
  • docker 20.10.7
  • docker-compose 1.29.2

The web server is running on Plesk.

Note: I only tried to implement the db pull command so far.

UPDATE 09.11.21:

So I've gotten this far that I'm able update and also download the files. However I'm only able to do it, if I hardcode the variables. Everytime I'm trying to setup the environment_variables: I get the following error, if I run the ddev pull myProvider:

Downloading database...
bash: 03: command not found

Here is my current .yaml file with the environment_variables:, which currently don't work. I've tested all the commands within ddev ssh and it works if I call them manually.

environment_variables:
  DB_NAME: db_name
  DB_USER: db_user
  DB_PASSWORD: 'Password$'
  HOST_IP: 10.10.10.10
  SSH_USERNAME: username
  SSH_PORT: 21

auth_command:
  command: |
    ssh-add -l >/dev/null || ( echo "Please 'ddev auth ssh' before running this command." && exit 1 )

db_pull_command:
  command: |
    mkdir -p /var/www/html/.ddev/.downloads
    ssh -p ${SSH_PORT} ${SSH_USERNAME}@${HOST_IP} "mysqldump -h 127.0.0.1 -u ${DB_USER} -p'${DB_PASSWORD}' ${DB_NAME} > /tmp/${DB_NAME}.sql"
    scp -P ${SSH_PORT} ${SSH_USERNAME}@${HOST_IP}:/tmp/${DB_NAME}.sql /var/www/html/.ddev/.downloads/db.sql
    gzip -f /var/www/html/.ddev/.downloads/db.sql

files_pull_command:
  command: |
    scp -P ${SSH_PORT} -r ${SSH_USERNAME}@${HOST_IP}:/path/to/public/fileadmin/user_upload /var/www/html/.ddev/.downloads/files

Do I declare the variables the wrong way? Or what is it that I'm missing?

For anyone who has trouble connecting via ssh without the password promt, you can run the following commands:

  1. ssh-keygen -t rsa
  2. ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 username@host
  3. Afterward you should be able to connect without a password promt. Try the following: ssh -p 22 username@host
  4. before you try to ddev puul you have to execute ddev auth ssh
Eluvitie
  • 73
  • 8
  • The best way to debug this is to `ddev ssh` and set the variables you have in the environment_variables section and then just execute the commands in the `db_pull_command` section one at a time. My suspicion is that in `ssh -p ${SSH_PORT} ${SSH_USERNAME}@${HOST_IP} 'mysqldump -h 127.0.0.1 -u ${DB_USER} -p ${DB_PASSWORD} ${DB_NAME} > /tmp/${DB_NAME}.sql.gz'` you single-quoted that whole section, so none of the variables get interpolated. And the redirection is inside the single quotes, so that wil probably not work out either? – rfay Nov 06 '21 at 14:22
  • I've updated my question with the my update. The current problem are the environment_variables which throw an error. – Eluvitie Nov 09 '21 at 19:55
  • With what you show, DB_NAME will be set to `db_name`. Is that really what you're doing? I assume you're using real values there? See example in https://github.com/drud/ddev/blob/0a2bc2c7b830fdfde47067ac888ad4f82d8c4e2b/pkg/ddevapp/dotddev_assets/providers/platform.yaml.example#L23-L26 You need to say what you mean by "throw an error". And please set the environment variables manually, and then run the steps in the db section one at a time manually in the web container. – rfay Nov 10 '21 at 00:27
  • Sorry that I wasn,t clear on that. The environment variables are dummy varaibles not my real ones. I've added a dummy Password with a special character since I thought that this might cause the error. I've tested it manually in the web container, where I set the environment variables manually and the ran all commands after each other. And what other information do you need when it "throws an error"? The `ddev pull` will abort after the error message I've shown above. – Eluvitie Nov 10 '21 at 05:36
  • 1
    As I said, the way to test is to `ddev ssh` and then manually set the environment variables. `export DB_NAME=db_name` etc. Then run the commands in the `auth_command` and the `db_pull_command` one at a time. See what happens. Have you done that? BTW, I don't think putting a `$` in a variable will likely work out as that's a special character in the shell. – rfay Nov 11 '21 at 14:03

1 Answers1

0

Thanks to @rfay for pointing me into the right direction.

The Problem was, that my password containted a special charater (not a $ though) which needed to be escaped.

After escpaing it correctly like so

environment_variables:
  DB_PASSWORD: 'Password\&\'

the ddev pull works.

I hope my .yaml file helps someone else that needs to pull from a webserver.

Eluvitie
  • 73
  • 8