Copy a Hubzilla channel from one hub to another - why are the files in the cloud not copied?

King Emir#
 Basel 
@Hubzilla Support Forum+
I created a channel in one hub, with some files (images) in the cloud/channel/. Then I copied the channel to another hub. The files in the old hub are not copied to the new hub. I also uploaded some files to the new hub, which are not shown in the old hub. What did I do wrong, or are files just not meant to be copied?
King Emir#
  last edited: Wed, 23 Aug 2017 15:18:50 +0200  
I just saw that one can upload files with WebDAV. That means the storage location is your local DAV server and not the hub, right? I think this could be a nice solution for me, since I have a WDmycloudmirror 2Gen cloud server. The tutorial https://hub.libranet.de/help/member/member_guide#Personal_Cloud_Storage doesn't treat this to the full extent, so maybe someone could help me? But maybe the Webtorrent addon will be the better solution? The initial question still remains: will attached files be shown in both copies of the hubzilla channels?
Andrew Manning
  
In the Hubzilla 2.6 release notes there is the mention of
Discovered some issues syncing photos and files to clones, which were fixed.

In 2.4, new files were synced but not existing files. This may still be true, due to limits on PHP memory or something like that. Mike has explained it a few times, but it is difficult to search for previous conversations when you don't know they exist. I'll see if I can find previous discussions.
Mike Macgirvin
  
2.6 should sync newly created files to clones just fine. In order to sync the older ones you'll need to use a plugin.

Originally there were two plugins called redfiles and redphotos for cloning/pulling files from Redmatrix into Hubzilla. I don't know the current state of the plugins; you should now be able to do everything with just 'redfiles' but it may require a bit of additional work because the API has changed slightly. You're welcome to have a look.
davfs2

Barefoot  Runner
 Eastern Shore of Virginia last edited: Sat, 04 Mar 2017 18:44:05 +0100  
@Hubzilla Support Forum+

I'm trying to get #davfs2 working with my nginx hubzilla server.

My current nginx.conf for my hub (comments welcome):


server{
    server_name  fqd.domain.com;
    #rewrite ^(.*) http://domain.com[/1 permanent;
    #rewrite ^ https://fqd.domain.com/$request_uri? permanent;
}
server {
   listen 192.168.1.40:443 ssl;  
   server_name fqd.domain.com;
   root   /var/www/domain/zot/htdocs;
   include /etc/nginx/shared_headers.conf;

   ssi on;
   ssl_certificate      /etc/letsencrypt/live/fqd.domain.com/fullchain.pem;
   ssl_certificate_key  /etc/letsencrypt/live/fqd.domain.com/privkey.pem;
   #include /etc/nginx/ssl_ciphers.conf;
      ssl_session_cache shared:SSL:50m;
      ssl_session_timeout 5m;
      ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
      ssl_prefer_server_ciphers on;
      ssl_dhparam /etc/ssl/certs/dhparam.pem;  # openssl dhparam -out /etc/ssl/certs/dhparam.pem 4096
      ssl_stapling on;
      ssl_stapling_verify on;
      ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';


   client_max_body_size 20m;
   client_body_buffer_size 128k;
   index index.php;
   charset utf-8;

   ## Do not accept DELETE, SEARCH and other methods ##
   ## DAVFS2 needs OPTIONS PROPFIND
     if ($request_method !~ ^(GET|HEAD|POST|OPTIONS|PROPFIND)$ ) {
         return 444;
     }
   ## Deny certain Referers ###
     if ( $http_referer ~* (babes|forsale|girl|jewelry|love|nudit|organic|poker|porn|sex|teen) )
     {
         return 404;
         return 403;
     }

   # Logging
   # If you are using 'Analytics' type software for tracking, keep this 'off'
   access_log /var/log/nginx/domain/access.log; # off
   error_log    /var/log/nginx/domain/error.log crit; # valid values: debug, info, notice, warn, error, crit
   log_not_found off;  # Turn on if you want to track "not found" errors
   rewrite_log on;       # Uncomment if you want to debug your rewrites (then change 'crit' above to 'notice')

   autoindex off;

   # block stuff early
   # Do not log favicon.ico and robots.txt stuff
   location ~* /(favicon\.ico|robots\.txt) {
      allow all;
      access_log off;
      log_not_found off;
   }
   # block these file types
   location ~* \.(tpl|md|tgz|log|out)$ {
      deny all;
   }
   location ~ "(^|/)\.git" {
      return 403;
   }
   # Return error 444 for these files
   location ~* ^.+\.(bzr|git|log)$ {
      access_log off;
      log_not_found off;
      return 444;
   }
   # Deny public access to ~ (bak) files
   location ~* ~$ {
      access_log off;
      log_not_found off;
      return 444;
   }

   location / {
      index index.php;
      if (!-f $request_filename) {
            rewrite ^/(.+)$ /index.php?q=$1 last;
      }
      try_files $uri $uri/ =404;
   }
   location ^~ /.well-known/ {
      allow all;
      rewrite ^/(.*) /index.php?q=$uri&$args last;
   }

   # RESTRICT ACCESS
   # block public access to .htaccess and .htconfig.php
   location ~* /\.ht {
      access_log off;
      log_not_found off;
      return 444;
   }


   ##############################################################
   # block public access to .tpl files located in /view/ folder #
   ##############################################################
      location ~* /view/(.*)\.tpl$ {
      access_log off;
      log_not_found off;
      return 444;
   }

   ########################################
   # block public access to /util/ folder #
   ########################################
      location ^~ /util/ {
      access_log off;
      log_not_found off;
      return 444;
   }


   #################################
   # Deliver static files directly #
   #################################
   # images
   location ~* /(addon|images|library|spec|util|view)/(.*)\.(bmp|cur|gif|ico|j2k|jp2|jpe|jpeg|jpf|jpg|jpm|jpx|mj2|mng|png|svg|svgz|thm|tif|tiff|webp)$ {
      add_header Pragma "public";
      add_header Cache-Control "public";
      access_log off;
      log_not_found off;
      expires 28d;
   }

   ############################
   # redirect 50x error pages #
   ############################
      error_page 500 502 503 504 /50x.html;
      location = /50x.html {
      root /usr/share/nginx/html;
      internal;
   }

   ##############
   # enable PHP #
   ##############
   location ~* \.php {
      # Zero-day exploit defense.
      # http://forum.nginx.org/read.php?2,88845,page=3
      # Won't work properly (404 error) if the file is not stored on this
      # server, which is entirely possible with php-fpm/php-fcgi.
      # Comment the 'try_files' line out if you set up php-fpm/php-fcgi on
      # another machine.  And then cross your fingers that you won't get hacked.
      try_files $uri =404;


      include /etc/nginx/fastcgi_params;
        
      # NOTE: You should have "cgi.fix_pathinfo = 0;" in php.ini
      # But read recommendation to use default of '1' and to correct scripts
      fastcgi_split_path_info ^(.+\.php)(/.+)$;

      fastcgi_param HTTPS on;
      fastcgi_index index.php;
      #fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

      # Choose either sockets or tcp
      #fastcgi_pass 127.0.0.1:9000;
      fastcgi_pass unix:/var/run/php5-fpm/domain.sock;
      #try_files $uri $uri/ =404;
   }
   location ~ \..*/.*\.php$ {
      return 403;
   }
   # deny access to all dot
   location ~ /\. {
      deny all;
   }
   # deny access to store
   location ~ /store {
      deny all;
   }

}


# cat /etc/nginx/fastcgi_params;

fastcgi_param  SCRIPT_FILENAME    $document_root$fastcgi_script_name;
fastcgi_param  QUERY_STRING       $query_string;
fastcgi_param  REQUEST_METHOD     $request_method;
fastcgi_param  CONTENT_TYPE       $content_type;
fastcgi_param  CONTENT_LENGTH     $content_length;

fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
fastcgi_param  REQUEST_URI        $request_uri;
fastcgi_param  DOCUMENT_URI       $document_uri;
fastcgi_param  DOCUMENT_ROOT      $document_root;
fastcgi_param  SERVER_PROTOCOL    $server_protocol;
fastcgi_param  REQUEST_SCHEME     $scheme;
fastcgi_param  HTTPS              $https if_not_empty;

fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;

fastcgi_param  REMOTE_ADDR        $remote_addr;
fastcgi_param  REMOTE_PORT        $remote_port;
fastcgi_param  SERVER_ADDR        $server_addr;
fastcgi_param  SERVER_PORT        $server_port;
fastcgi_param  SERVER_NAME        $server_name;

# PHP only, required if PHP was built with --enable-force-cgi-redirect
fastcgi_param  REDIRECT_STATUS    200;

# httpoxy mitigation (https://httpoxy.org/ https://www.nginx.com/blog/?p=41962)
fastcgi_param  HTTP_PROXY         "";



# cat /etc/nginx/shared_headers.conf

   # do not show nginx version
   server_tokens off;

   # https://www.owasp.org/
   # http://cyh.herokuapp.com/cyh
   # HSTS
   add_header Strict-Transport-Security "max-age=31536000; includeSubdomains";
   #add_header X-Frame-Options DENY; #SAMEORIGIN DENY ALLOW-FROM uri
   add_header X-Frame-Options "SAMEORIGIN"; # Prevent ClickJacking
   add_header X-Content-Type-Options "nosniff";
   add_header X-XSS-Protection "1; mode=block";
   add_header X-Permitted-Cross-Domain-Policies "master-only";
   #add_header X-Frame-Options Content-Security-Policy "default-src 'self'";
   #add_header X-Frame-Options Content-Security-Policy "default-src 'self'; report-only; report-uri";
   more_set_headers -s '500 501 502 503' "Strict-Transport-Security: max-age=31536000; includeSubdomains";


The above setup is working for me with fpm-php5.6 (will be upgrading shortly to fpm-php7.0)
In order to get DAVFS2 clients working with hubzilla, I need to add another location directive for /dav/

I found this suggestion from https://opensource.ncsa.illinois.edu/confluence/display/ERGO/Creating+a+WebDAV+repository+server+with+NGINX

but I need to adapt this example for hubzilla.


  # here you can specify various directories that respond as DAV.
  location /ergo-repo/ {
    root      /var/dav;
    client_body_temp_path /var/dav/temp;
    dav_methods     PUT DELETE MKCOL COPY MOVE;
    dav_ext_methods   PROPFIND OPTIONS;
    create_full_put_path  on;
    dav_access    user:rw group:rw all:rw;
    autoindex     on;
    # below you can specify the access restrictions. In this case, only people on the 141.142 network
    # can write/delete/etc. Everyone else can view.
    limit_except GET PROPFIND OPTIONS{
      allow 141.142.0.0/16;
      deny  all;
    }
    allow all;
  }
  # this is an example of a password restricted repository
  location /password-repo/ {
    root      /var/dav;
    client_body_temp_path /var/dav/temp;
    dav_methods     PUT DELETE MKCOL COPY MOVE;
    dav_ext_methods   PROPFIND OPTIONS;
    create_full_put_path  on;
    dav_access    user:rw group:rw all:rw;
    autoindex     on;
    auth_basic "restricted";
    auth_basic_user_file /etc/nginx/htpasswd;
  }
}


Anyone have a working location /dav/ directive for hubzilla to recommend?
Thanks
Barefoot  Runner
  
I was following the guide  /help/member/member_guide
which said to add a line to /etc/fstab

/dav/ /mount/point davfs user,noauto,uid=<DesktopUser>,file_mode=600,dir_mode=700 0 1



but apparently nginx's location directive needs to be changed to cloud instead of dav:

location /cloud/ {
....
}

but the url still needs to be /dav/

I can delete the entire location /cloud/ {...} and the mount.davfs will work, but no files are visible
unless  I  list a specific hubzilla account, (I can see other public hubzilla clouds on my server if I know their names).

eg., if I've mounted /dav to ~/cloud
then 'ls ~/cloud' shows an empty directory,
but ls ~/cloud/ shows the cloud storage  for member .

So, I need to change the fstab line

/dav/ /mount/point davfs user,noauto,uid=<DesktopUser>,file_mode=600,dir_mode=700 0 1


So now I've got mount/read access to cloud storage working without any location directive

Next step will be to finagle location /cloud/ { ... } to enable davfs methods (MKDIR, DELETE, ...)
and security (maybe restricting certain davfs METHODS to known computer/networks.
Barefoot  Runner
  
OK, did a little more testing.

The entire location /cloud/ {...} has no effect whatsoever, so I've just removed entirely.

Re-reading the guide, I see I needed to disable locks in my davfs.conf use_locks 0
Once that was done, I had full rw capabilities on the davfs mount.
No further modifications to my nginx.conf were necessary to get davfs working with hubzilla :-)

Got a clue to the above by looking through my nginx error.log:


2017/03/05 18:34:36 [notice] 14566#14566: *71 "^(GET|HEAD|POST|OPTIONS|PROPFIND)$" does not match "LOCK", client: 192.168.1.100, server:   request: "LOCK /dav/ ]/Test HTTP/1.1", host: " "
cer
cer
  
nginx/sites/parlementum.net.conf:
root@fortinbras ~# cat /etc/nginx/sites/parlementum.net.conf
##
# Red Nginx configuration
# by Olaf Conradi
#
# On Debian based distributions you can add this file to
# /etc/nginx/sites-available
#
# Then customize to your needs. To enable the configuration
# symlink it to /etc/nginx/sites-enabled and reload Nginx using
#
# service nginx reload
##

##
# You should look at the following URL's in order to grasp a solid understanding
# of Nginx configuration files in order to fully unleash the power of Nginx.
#
# http://wiki.nginx.org/Pitfalls
# http://wiki.nginx.org/QuickStart
# http://wiki.nginx.org/Configuration
##

##
# This configuration assumes your domain is example.net
# You have a separate subdomain parlementum.net
# You want all red traffic to be https
# You have an SSL certificate and key for your subdomain
# You have PHP FastCGI Process Manager (php-fpm) running on localhost
# You have Red installed in /var/www/red
##

server {
  listen 80;
  server_name parlementum.net;
  include "snippets/letsencrypt.conf";

  index index.php;
  root /srv/http/parlementum.net;
  rewrite ^ https://parlementum.net$request_uri? permanent;
}

##
# Configure Red with SSL
#
# All requests are routed to the front controller
# except for certain known file types like images, css, etc.
# Those are served statically whenever possible with a
# fall back to the front controller (needed for avatars, for example)
##

server {
  listen 443 ssl;
  server_name parlementum.net;

  include "snippets/letsencrypt.conf";
  ssl on;
  ssl_certificate     /var/lib/acme/live/parlementum.net/fullchain;
  ssl_certificate_key /var/lib/acme/live/parlementum.net/privkey;
  ssl_session_timeout 5m;
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-DSS-AES128-SHA256:DHE-DSS-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!DHE-RSA-AES128-GCM-SHA256:!DHE-RSA-AES256-GCM-SHA384:!DHE-RSA-AES128-SHA256:!DHE-RSA-AES256-SHA:!DHE-RSA-AES128-SHA:!DHE-RSA-AES256-SHA256:!DHE-RSA-CAMELLIA128-SHA:!DHE-RSA-CAMELLIA256-SHA;
  ssl_prefer_server_ciphers on;

  fastcgi_param HTTPS on;

  index index.php;
  charset utf-8;
  root /srv/http/parlementum.net;
  #access_log /var/log/nginx/red.log;
    #Uncomment the following line to include a standard configuration file
    #Note that the most specific rule wins and your standard configuration
    #will therefore *add* to this file, but not override it.
  #include standard.conf
  # allow uploads up to 20MB in size
  client_max_body_size 50m;
  client_body_timeout 300;
  client_body_buffer_size 128k;

  # rewrite to front controller as default rule
  location / {
    if ($is_args != "") {
        rewrite ^/(.*) /index.php?q=$uri&$args last;
    }
    rewrite ^/(.*) /index.php?q=$uri last;
  }

  # make sure webfinger and other well known services aren't blocked
  # by denying dot files and rewrite request to the front controller
  location ^~ /.well-known/ {
    allow all;
    rewrite ^/(.*) /index.php?q=$uri&$args last;
  }

  # statically serve these file types when possible
  # otherwise fall back to front controller
  # allow browser to cache them
  # added .htm for advanced source code editor library
  location ~* \.(jpg|jpeg|gif|png|ico|css|js|htm|html|ttf|woff|svg)$ {
    expires 30d;
    try_files $uri /index.php?q=$uri&$args;
  }

  # block these file types
  location ~* \.(tpl|md|tgz|log|out)$ {
    deny all;
  }

  # deny access to all dot files
  location ~ /\. {
    deny all;
  }

#deny access to store

    location ~ /store {
        deny  all;
    }
  include "snippets/php.conf";

}


let me know if you need to see snippets, etc. The key I think is the order/ranking

etc/fstab:

https://parlementum.net/dav/cer /home/cer/khazar davfs user,noauto,uid=cer,file_mode=600,dir_mode=700 0 1