2

I have an extension hosted on github, for which I want to provide automatic updates. However, when I provide the URL to a file attached to a release, Joomla just reports the following on trying to automatically update:

Fehler beim Verbindungsaufbau zum Server: Error Unknown
Ungültige Webadresse

Translation

Error in connecting to server: Error Unknown
Invalid web address

It's the same when trying to install from the web address via the Extensions - Install page. From the browser, I can download the file just fine however. Anybody got an idea why that is? Is it an error on github side, or on Joomla's? Or is it some "safety" or "security" mechanism on github side? What can I do to avoid it? Or do those two just not play along?

Example URL: https://github.com/codeling/bfstop/releases/download/1.2.0/pkg_bfstop-1.2.0.zip

Edit:

Inserted the patch for downloadPackage method (gist.github.com/piotr-cz/8316210) mentioned by piotr_cz in the comments below the answer into my Joomla installation now.

URL passed into the method after resolving the redirect: https://s3.amazonaws.com/github-cloud/releases/6794712/f2aa5eb4-7838-11e3-837a-c6be2639e4ca.zip?response-content-disposition=attachment%3B%20filename%3Dpkg_bfstop-1.2.0.zip&AWSAccessKeyId=AKIAISTNZFOVBIJMK3TQ&Expires=1389455424&Signature=Nnyl6TnWueTqK8bPkPXUidM8UzQ%3D

URL after the newly inserted lines: https://s3.amazonaws.com/github-cloud/releases/6794712/f2aa5eb4-7838-11e3-837a-c6be2639e4ca.zip?response-content-disposition=attachment; filename=pkg_bfstop-1.2.0.zip&AWSAccessKeyId=AKIAISTNZFOVBIJMK3TQ&Expires=1389455424&Signature=Nnyl6TnWueTqK8bPkPXUidM8UzQ=

The response is still a 505 error:

response: JHttpResponse Object
(
    [code] => 505
    [headers] => Array
        (
            [Date] => Sat, 11 Jan 2014 15:49:24 GMT
            [Connection] => close
            [Server] => AmazonS3
        )

    [body] =>
)

So I suppose the encoding doesn't matter. The error 505 would indeed indicate http version problems? Why would Joomla and Amazon disagree on HTTP versions? Definitely not an HTTP expert here but version 1.1 should be the unchanged standard version since more than 10 years now?

codeling
  • 11,056
  • 4
  • 42
  • 71

3 Answers3

2

Conclusions:

  1. Automatically generated downloads by GitHub's Tags and Releases are compatible with Joomla Installer at this moment (Jan 2014) as long as the repository name matches extension name (element). Joomla Installer is able to work with an extension which:

    • Have files and folders located directly:

      mod_mymodule.zip:

/mod_mymodule.xml
/mod_mymodule.php

    • Or have files and folders located in a folder of the same name as the archive (see JInstallerHelper::unpack):

      mod_mymodule.zip:

/mod_mymodule
/mod_mymodule/mod_mymodule.xml
/mod_mymodule/mod_mymodule.php

      This is how GitHub Tags and Releases are built now. Automatically generated downloads of repository mod_mymodule results in mod_mymodule-[tag/branch].zip archive with mod_mymodule-[tag/branch] subfolder. Say when we tag extension 1.0.0-beta:

      mod_mymodule-1.0.0-beta.zip:

/mod_mymodule-1.0.0-beta
/mod_mymodule-1.0.0-beta/mod_mymodule.xml 
/mod_mymodule-1.0.0-beta/mod_mymodule.php

      Note: There were some changes in how archive is built (see Automatic Extension Update: Unknown Archive type)

  2. Manually attached Binaries added to a GitHub Release are hosted in the Amazon Web Services. Request to download these by JInstallerHelper::downloadPackage results in an response 505 HTTP Version Not Supported and we didn't find a working solution for this problem yet.

In the end one may use GitHub as an Joomla Extensions Update server (it's pretty convenient), as long as you use automatically generated downloads.

GitHub provides Version Control service, not an Update server so the way how downloadable archive is built may change again in future.

Until we find out a way how to download Attached Binaries (where one have full control over archive's name and structure) using Joomla Installer, GitHub should not be perceived as a reliable tool for Joomla extension updates. I believe there's a way to patch the JInstaller package to be compatible with the AWS (download works fine using command-line curl).

Thanks for ccpl and RandolphCarter on getting information for this answer.

Community
  • 1
  • 1
piotr_cz
  • 8,755
  • 2
  • 30
  • 25
  • regarding the note to the second option under 1: I don't think github reverted anything. They just changed from directly zipping folder contents (i.e. the structure mentioned in the first point under 1), to zipping them inside a folder with the name of the repo. Your post at the moment to me reads like they changed it to contain the subfolder and then changed it back? And regarding patching Joomal: Would be nice if the package from s3 worked. maybe joomla forum would be a better place to ask if somebody knows a fix? – codeling Jan 20 '14 at 11:15
  • Okay, I'll update the answer. The problem I see is that they can change it again to something else that won't work anymore. As for discussion, I recommend [Joomla CMS Development](https://groups.google.com/forum/#!forum/joomla-dev-cms) Group. I guess other developers will be interested in this topic. – piotr_cz Jan 20 '14 at 11:36
1

It would appear that URL is redirected to an S3 location:

https://s3.amazonaws.com/github-cloud/releases/6794712/6c173582-77ef-11e3-9aec-c8994b691269.zip?response-content-disposition=attachment%3B%20filename%3Dpkg_bfstop-1.2.0.zip&AWSAccessKeyId=AKIAISTNZFOVBIJMK3TQ&Expires=1389163120&Signature=c4RdRTAUZ5%2FDHlpg0vR2ivK6lQQ%3D`

As a guess I would say this breaks the install from web.

Updated

Doing a bit further tracking, I can see the downloadPackage() method does catch the 302 Found and try to get the file from the new URL. At this point it does a new curl request with some basic options:

$options = array("10036" => "GET",
"10065" => "/Users/cppl/Sites/jdev/libraries/joomla/http/transport/cacert.pem",
"10002" => "https://s3.amazonaws.com/github-cloud/releases/6794712/f2aa5eb4-7838-11e3-837a-c6be2639e4ca.zip?response-content-disposition=attachment; filename=pkg_bfstop-1.2.0.zip&AWSAccessKeyId=AKIAISTNZFOVBIJMK3TQ&Expires=1389170600&Signature=BVlqH0hVhYGjbeKn6w/9nDn+kDg=",
"42" => "1",
"19913" => "1")

Unfortunately the S3 service is returning a 505

HTTP/1.1 505 HTTP Version Not Supported
Date: Wed, 08 Jan 2014 08:47:59 GMT
Connection: close
Server: AmazonS3

Hacking around in \JHttpTransportCurl\request I tried

  1. forcing CURL_HTTP_VERSION_1_1
  2. disabling SSL verification CURLOPT_SSL_VERIFYPEER (useful in other S3 situations)
  3. updating the cacert.pem to the current one
  4. all together

Nothing worked.

So far it all appears to be on AWS side, I'm not aware of all the S3 access controls possibly there's something in the GitHub setup of their S3 buckets.

Community
  • 1
  • 1
Craig
  • 9,335
  • 2
  • 34
  • 38
  • Anything one can do about that? Above URL obviously isn't valid forever (Expires parameter) and thus can't be used directly. Aren't redirects a quite normal thing which thus should not break Joomla updater? – codeling Jan 08 '14 at 07:28
  • See my update, basically it's both on the Amazon side and Joomla side, not sure which. – Craig Jan 08 '14 at 09:23
  • Can it be that Curl is sending the `HTTP/1.0` version instead of 1.1? I'd try to hack the [JHttpTransportCurl](https://github.com/joomla/joomla-cms/blob/staging/libraries/joomla/http/transport/curl.php) to use `$options[CURLOPT_HTTP_VERSION] = CURL_HTTP_VERSION_1_1;` – piotr_cz Jan 08 '14 at 09:29
  • Nope I tried that, I also tried disabling SSL verification, I'll update the answer with the things I've tried. – Craig Jan 08 '14 at 09:47
  • What was the response? 505 should be `The server does not support the HTTP protocol version used in the request.`. Other option could be `CURLOPT_BINARYTRANSFER` or the `CURLOPT_URL` (10002) might have to be rawurlencoded – piotr_cz Jan 08 '14 at 10:17
  • I'm pretty sure now the problem is that the amazonaws url should be rawurlencoded in the [downloadPackage](https://github.com/joomla/joomla-cms/blob/staging/libraries/cms/installer/helper.php#L52) method. I just don't know if whole url or only spaces. – piotr_cz Jan 08 '14 at 10:45
  • Local CURL installation uses `/github-cloud/releases/6794712/f2aa5eb4-7838-11e3-837a-c6be2639e4ca.zip?response-content-disposition=attachment%3B%20filename%3Dpkg_bfstop-1.2.0.zip&AWSAccessKeyId=AKIAISTNZFOVBIJMK3TQ&Expires=1389178236&Signature=A4Uswim%2FxEuEQr1ESUwxE816VlE%3D` – piotr_cz Jan 08 '14 at 10:51
  • Here's my patch for `downloadPackage` method: https://gist.github.com/piotr-cz/8316210. – piotr_cz Jan 08 '14 at 12:40
  • @piotr_cz tried to apply your patch to the best of my knowledge, and now it reports all the "error unknown" line, and then "package download failed", so there seems to be a slight change, but not really much... – codeling Jan 08 '14 at 14:26
  • [`Error Unknown`](https://github.com/joomla/joomla-cms/blob/2.5.x/libraries/joomla/installer/helper.php) in J2.5 means the response code is different than 200 and 302. Try to check new value for the `$url` and `$response` variable. – piotr_cz Jan 08 '14 at 14:49
  • How did it go? I'm planning to use Github for extension updates later on, but don't have time ATM to test the patch. – piotr_cz Jan 09 '14 at 08:16
  • @piotr_cz see my updated question. in short: the answer is still a 505 – codeling Jan 11 '14 at 16:02
0

I haven't got the direct approach to work (to use the release assets for automatic update), therefore I am now implementing the workaround solution by putting the files into the git repository.

Not an ideal solution, I know, but at least that works.

codeling
  • 11,056
  • 4
  • 42
  • 71
  • This is my recent repo and github update works: https://github.com/piotr-cz/mod_freshmail2. Tested from localhost on Joomla 3.0.3 – piotr_cz Jan 18 '14 at 09:12
  • okay I realized that I'm linking to release files generated automatically, while you to uploaded binary files. – piotr_cz Jan 18 '14 at 13:19
  • @piotr_cz the difference between automatically generated and uploaded binary files is that the automatically generated ones link directly to github servers while binary link to s3. And the really sad part is that in principle, I would be fine with the automatically generated ones - just that they contain this additional top-level directory which joomla can't deal with - how are you handling that? At least in my tests, Joomla would also come up with an error if there is just the directory in the zip file and no xml file on top level – codeling Jan 18 '14 at 23:23
  • 1
    This seems to work at least on J3.x. Important thing is that the archive must have same name as the compressed folder. So inside `mod_mymodule.zip` there should be a `/my_mymodule` folder (see source of [JInstallHelper::unpack](https://github.com/joomla/joomla-cms/blob/staging/libraries/cms/installer/helper.php#L143)) and this is how GitHub creates rerelases for me. – piotr_cz Jan 19 '14 at 16:48
  • You're right. It also works on J2.5. Because that's exactly what I had to go away from when they introduced that folder - see http://stackoverflow.com/questions/20362958/automatic-extension-update-unknown-archive-type . No idea what changed in the meantime, just glad that it works again! Maybe you'd like to post it as an answer? – codeling Jan 19 '14 at 21:35