diff options
Diffstat (limited to 'docs/installing')
| -rw-r--r-- | docs/installing/ami.md | 59 | ||||
| -rw-r--r-- | docs/installing/deploy.md | 167 | ||||
| -rw-r--r-- | docs/installing/email.md | 213 | ||||
| -rw-r--r-- | docs/installing/index.md | 62 | ||||
| -rw-r--r-- | docs/installing/macos.md | 155 | ||||
| -rw-r--r-- | docs/installing/manual_install.md | 570 | ||||
| -rw-r--r-- | docs/installing/script.md | 65 |
7 files changed, 1291 insertions, 0 deletions
diff --git a/docs/installing/ami.md b/docs/installing/ami.md new file mode 100644 index 000000000..42f2907cb --- /dev/null +++ b/docs/installing/ami.md @@ -0,0 +1,59 @@ +--- +layout: page +title: Installing the easy way +--- + +# Installation on Amazon EC2 + +<p class="lead"> + We've made an Amazon Machine Image (AMI) so you can quickly deploy on Amazon EC2. This is handy if you just want to evaluate Alaveteli, for example. +</p> + +Note that there are [other ways to install Alaveteli]({{ site.baseurl }}docs/installing). + +## Installing from our AMI + +To help people try out Alaveteli, we have created an AMI (Amazon Machine Image) +with a basic installation of Alaveteli, which you can use to create a running +server on an Amazon EC2 instance. This creates an instance that runs in +development mode, so we wouldn't recommend you use it for a production system +without changing the configuration. + +Unfortunately, Alaveteli will not run properly on a free Micro +instance due to the low amount of memory available on those +instances; you will need to use at least a Small instance, which +Amazon will charge for. + +The AMI can be found in the EU West (Ireland) region, with the ID ami-8603f4f1 +and name “Basic Alaveteli installation 2014-01-29”. You can launch an instance +based on that AMI with [this +link](https://console.aws.amazon.com/ec2/home?region=eu-west-1#launchAmi=ami-8603f4f1). + +When you create an EC2 instance based on that AMI, make sure that you choose +Security Groups that allows at least inbound HTTP, HTTPS, SSH and, if you want +to test incoming mail as well, SMTP. + +When your EC2 instance is launched, you will be able to log in as the `ubuntu` +user. This user can `sudo` freely to run commands as root. However, the code is +actually owned by (and runs as) the `alaveteli` user. After creating the +instance, you may want to edit a configuration file to customize the site's +configuration. That configuration file is +`/var/www/alaveteli/alaveteli/config/general.yml`, which can be edited with: + + ubuntu@ip-10-58-191-98:~$ sudo su - alaveteli + alaveteli@ip-10-58-191-98:~$ cd alaveteli + alaveteli@ip-10-58-191-98:~/alaveteli$ nano config/general.yml + +Then you should restart the Thin webserver with: + + alaveteli@ip-10-58-191-98:~/alaveteli$ logout + ubuntu@ip-10-58-191-98:~$ sudo /etc/init.d/alaveteli restart + +If you find the hostname of your EC2 instance from the AWS console, you should +then be able to see the site at +`http://your-ec2-hostname.eu-west-1.compute.amazonaws.com` + +If you have any problems or questions, please ask on the [Alaveteli Google +Group](https://groups.google.com/forum/#!forum/alaveteli-dev) or [report an +issue](https://github.com/mysociety/alaveteli/issues?state=open). + diff --git a/docs/installing/deploy.md b/docs/installing/deploy.md new file mode 100644 index 000000000..5378b89d6 --- /dev/null +++ b/docs/installing/deploy.md @@ -0,0 +1,167 @@ +--- +layout: page +title: Deploying +--- + +# Deploying Alaveteli + +<p class="lead"> + Although you can install Alaveteli and just change it when you need it, we + recommend you adopt a way of <strong>deploying</strong> it automatically, + especially on your <a href="{{ site.baseurl }}docs/glossary/#production">production server</a>. + Alaveteli provides a deployment mechanism using Capistrano. +</p> + +## Why deploy? + +Although you can [install Alaveteli]({{ site.baseurl }}docs/installing/) in a number +of ways, once you're running, sooner or later you'll need to make changes to +the site. A common example is updating your site when we issue a new release. + +The deployment mechanism takes care of putting all the right files into the +right place, so when you need to put changes live, there's no risk of you +forgetting the update all the files you've changed, or breaking the +configuration by accident. Instead, deployment does all this for you +automatically. It's also more efficient because it is faster than making +changes or copying files by hand, so your site will be down for the shortest +possible time. + +We **strongly recommend** you use the deployment mechanism for your +<a href="{{ site.baseurl }}docs/glossary/#production">production server</a> and, if +you're running one, your +<a href="{{ site.baseurl }}docs/glossary/#staging">staging server</a> too. + +## Capistrano + +<a href="{{site.baseurl}}docs/glossary/#capistrano" class="glossary">Capistrano</a> +is included as part of Alaveteli as a standard deployment system. + +The basic principle of Capistrano is that you execute `cap [do-something]` +commands on your local machine, and Capistrano connects to your server (using +`ssh`) and does the corresponding task there for you. + +### Set up + +Capistrano requires things to be set up at both ends -- that is, on the server +where you want Alaveteli to run, and on your own local machine. + +* *the server* is the machine that will be running the Alaveteli + instance you're deploying + +* your *local machine* may be your laptop or similar device -- as well as those + belonging to anyone in your team whom you want to be able to deploy + +In order to allow the Capistrano deployment mechanism work, you need to set up +the server so that the Alaveteli app is being served from a directory called +`current`. Once you've done that, deploying a new version is essentially +creating a timestamped sister directory to the `current` directory, and +switching the symlink `current` from the old timestamped directory to the new +one. Things that need to persist between deployments, like config files, are +kept in a `shared` directory that is at the same level, and symlinked-to from +each timestamped deploy directory. + +We're [working on making this easier](https://github.com/mysociety/alaveteli/issues/1596), +but for now, here's the manual process you need to follow to set up this +deployment mechanism. Remember, you only have to do this once to set it up, +and thereafter you'll be able to deploy very easily (see [usage, below](#usage)). + +First, on the server: + +* [install Alaveteli]({{ site.baseurl }}docs/installing/) +* then move the Alaveteli app to a temporary place on the server, like your home + directory (temporarily, your site will be missing, until the deployment puts + new files in place) + +Next, on your local machine: + +* install Capistrano: + * Capistrano requires Ruby 1.9 or more, and can be installed using rubygems + * do: `gem install capistrano` +* install Bundler if you don't have it already -- do: `gem install bundler` +* checkout the [Alaveteli repo](http://github.com/mysociety/alaveteli/) (you + need some of the files available locally even though you might not be running + Alaveteli on this machine) +* copy the example file `config/deploy.yml.example` to `config/deploy.yml` +* now customise the deployment settings in that file: edit `config/deploy.yml` + appropriately -- for example, edit the name of the server. Also, change + `deploy_to` to be the path where Alaveteli is currently installed on the + server -- if you used the installation script, this will be + `/var/www/alaveteli/alaveteli`. +* `cd` into the Alaveteli repo you checked out (otherwise the `cap` commands you're about to + execute won't work) +* still on your local machine, run `cap -S stage=staging deploy:setup` to setup capistrano on the server +* again on your local machine, run `cap -S stage=staging deploy:update_code` to get a code checkout on the server + +Back on the server: + +* copy the following config files from the temporary copy of Alaveteli you made at + the begining (perhaps in your home directory) to the `shared` directory that + Capistrano just created on the server: + * `general.yml` + * `database.yml` + * `rails_env.rb` + * `newrelic.yml` + * `aliases` ← if you're using Exim as your MTA +* if you're using Exim as your MTA, edit the `aliases` file you just copied across + so that the path to Alaveteli includes the `current` element. If it was + `/var/www/alaveteli/alaveteli/script/mailin`, it should now be + `/var/www/alaveteli/alaveteli/current/script/mailin`. +* copy the following directories from your temporary copy of Alaveteli to the + `shared` directory created by Capistrano on the server: + * `cache/` + * `files/` + +Now, back on your local machine: + +* make sure you're still in the Alaveteli repo (if not, `cd` back into it) +* create a deployment directory on the server by running *one* of these commands: + * `cap deploy` if you're deploying a <a href="{{site.baseurl}}docs/glossary/#staging" class="glossary">staging site</a>, or... + * `cap -S stage=production deploy` for <a href="{{site.baseurl}}docs/glossary/#production" class="glossary">production</a> +* update the webserver config (either apache or nginx) to add the `current` element + to the path where it is serving Alaveteli from. If you installed using the + installation script, this will be replacing `/var/www/alaveteli/alaveteli/` with + `/var/www/alaveteli/alaveteli/current` in `etc/nginx/sites-available/default`. +* edit the server crontab so that the paths in the cron jobs also include the + `current` element. If you used the installation script the crontab will be in + `etc/cron.d/alaveteli`. +* Update the MTA config to include the `current` element in the paths it uses. + If you installed using the installation script, the MTA will be postfix, + and you will need to edit `/etc/postfix/master.cf` to replace + `argv=/var/www/alaveteli/alaveteli/script/mailin` with + `argv=/var/www/alaveteli/alaveteli/current/script/mailin`. + If you're using Exim as your MTA, edit `etc/exim4/conf.d/04_alaveteli_options` + to update the `ALAVETELI_HOME` variable to the new Alaveteli path. + +Phew, you're done! + +You can delete the temporary copy of Alaveteli (perhaps in your +home directory) now. + +### Usage + +Before you issue any Capistrano commands, `cd` into the checkout of the +Alaveteli repo on your local machine (because that's where it will look +for the config that you've set up). + +Ensure you've got a `config/deploy.yml` file with the correct settings for your +site. If there are other people in your team who need to deploy, you'll need to +share it with them too -- it might be a good idea to keep the latest +version in a [Gist](http://gist.github.com/). + +* to deploy to staging, just run `cap deploy` +* to deploy to production, run `cap -S stage=production deploy` + +You might notice that, after deploying, the old deploy directory is still there +-- that is, the one that was `current` until you replaced it with the new one. +By default, the deploy mechanism keeps the last five deployments there. Run +`cap deploy:cleanup` to tidy up older versions. + +For additional usage instructions, see the [Capistrano +website](http://capistranorb.com/). + +### Whoops, that's not what I expected + +If a deployment goes wrong, or you discover after doing it that you're not +ready for the latest version after all, don't panic! Run `cap deploy:rollback` +and it will switch `current` back to the previous deployment. + diff --git a/docs/installing/email.md b/docs/installing/email.md new file mode 100644 index 000000000..98dff0087 --- /dev/null +++ b/docs/installing/email.md @@ -0,0 +1,213 @@ +--- +layout: page +title: Installing MTA +--- + +# Installing the MTA + +<p class="lead"> + Alaveteli sends and receives email. You'll need to set up your Mail + Transfer Agent (MTA) to handle this properly. We've got examples + here for both postfix and exim4, two of the most popular MTAs. +</p> + +Make sure you follow the correct instructions for the specific MTA you're using: + +* [postfix](#example-setup-on-postfix) +* [exim4](#example-setup-on-exim4) + +## Example setup on postfix + +This section shows an example of how to set up your MTA if you're using +**postfix** (running on Ubuntu). See the example for +[exim4](#example-setup-on-exim4) if you're using that instead of postfix. + +### Instructions + +For example, with: + + ALAVETELI_HOME=/path/to/alaveteli/software + ALAVETELI_USER=www-data + +In `/etc/postfix/master.cf`: + + alaveteli unix - n n - 50 pipe + flags=R user=ALAVETELI_USER argv=ALAVETELI_HOME/script/mailin + +The user ALAVETELI_USER should have write permissions on ALAVETELI_HOME. + +In `/etc/postfix/main.cf`: + + virtual_alias_maps = regexp:/etc/postfix/regexp + +And, assuming you set +[`INCOMING_EMAIL_PREFIX`]({{ site.baseurl }}docs/customising/config/#incoming_email_prefix) +in `config/general` to "foi+", create `/etc/postfix/regexp` with the following +content: + + /^foi.*/ alaveteli + +You should also configure postfix to discard any messages sent to the +[`BLACKHOLE_PREFIX`]({{ site.baseurl }}docs/customising/config/#blackhole_prefix) +address, whose default value is `do-not-reply-to-this-address`. For example, add the +following to `/etc/aliases`: + + # We use this for envelope from for some messages where + # we don't care about delivery + do-not-reply-to-this-address: :blackhole: + +### Logging + +For the postfix logs to be succesfully read by the script `load-mail-server-logs`, they need +to be log rotated with a date in the filename. Since that will create a lot of rotated log +files (one for each day), it's good to have them in their own directory. For example (on Ubuntu), +in `/etc/rsyslog.d/50-default.conf` set: + + mail.* -/var/log/mail/mail.log + +And also edit `/etc/logrotate.d/rsyslog`: + + /var/log/mail/mail.log + { + rotate 30 + daily + dateext + missingok + notifempty + compress + delaycompress + sharedscripts + postrotate + reload rsyslog >/dev/null 2>&1 || true + endscript + } + +You'll also need to tell Alaveteli where the log files are stored and that they're in postfix +format. Update +[`MTA_LOG_PATH`]({{ site.baseurl }}docs/customising/config/#mta_log_path) and +[`MTA_LOG_TYPE`]({{ site.baseurl }}docs/customising/config/#mta_log_type) in `config/general.yml` with: + + MTA_LOG_PATH: '/var/log/mail/mail.log-*' + MTA_LOG_TYPE: "postfix" + +### Troubleshooting (postfix) + +To test mail delivery, run: + + $ /usr/sbin/sendmail -bv foi+requrest-1234@localhost + +This tells you if sending the emails to `foi\+.*localhost` is working. + + +## Example setup on exim4 + +This section shows an example of how to set up your MTA if you're using +**exim4** (running on Ubuntu). See the example for +[postfix](#example-setup-on-postfix) if you're using that instead of exim4. + + +### Instructions + +We suggest you add the following to your exim configuration. + +In `/etc/exim4/conf.d/main/04_alaveteli_options`, set: + + ALAVETELI_HOME=/path/to/alaveteli/software + ALAVETELI_USER=www-data + log_file_path=/var/log/exim4/exim-%slog-%D + MAIN_LOG_SELECTOR==+all -retry_defer + extract_addresses_remove_arguments=false + +The user ALAVETELI_USER should have write permissions on ALAVETELI_HOME. + +The name and location of the log files created by Exim must match what the +`load-mail-server-logs` script expects, which is why you must provide the +`log_file_path` setting. + +The `check-recent-requests-sent` scripts expects the logs to contain the +`from=<...>` envelope information, so we make the logs more verbose with +`log_selector`. The ALAVETELI_USER may need to also need to be added to the +`trusted_users` list in your Exim config in order to set the return path on +outgoing mail, depending on your setup. + +In `/etc/exim4/conf.d/router/04_alaveteli`: + + alaveteli_request: + debug_print = "R: alaveteli for $local_part@$domain" + driver = redirect + data = ${lookup{$local_part}wildlsearch{ALAVETELI_HOME/config/aliases}} + pipe_transport = alaveteli_mailin_transport + +In `/etc/exim4/conf.d/transport/04_alaveteli`: + + alaveteli_mailin_transport: + driver = pipe + command = $address_pipe ${lc:$local_part} + current_directory = ALAVETELI_HOME + home_directory = ALAVETELI_HOME + user = ALAVETELI_USER + group = ALAVETELI_USER + +And, assuming you set +[`INCOMING_EMAIL_PREFIX`]({{ site.baseurl }}docs/customising/config/#incoming_email_prefix) +in your config at `config/general.yml` to "foi+", create `config/aliases` with the following +content: + + ^foi\\+.*: |/path/to/alaveteli/software/script/mailin + +You should also configure exim to discard any messages sent to the +[`BLACKHOLE_PREFIX`]({{ site.baseurl }}docs/customising/config/#blackhole_prefix) +address, whose default value is +`do-not-reply-to-this-address`. For example, add the following to +`config/aliases`: + + # We use this for envelope from for some messages where we don't care about delivery + do-not-reply-to-this-address: :blackhole: + +If you want to make use of the automatic bounce-message handling, then set the +[`TRACK_SENDER_EMAIL`]({{ site.baseurl }}docs/customising/config/#track_sender_email) +address to be filtered through +`script/handle-mail-replies`. Messages that are not bounces or +out-of-office autoreplies will be forwarded to +[`FORWARD_NONBOUNCE_RESPONSES_TO`]({{ site.baseurl }}docs/customising/config/#forward_nonbounce_responses_to). +For example, in WhatDoTheyKnow the +configuration looks like this: + + raw_team: [a list of people on the team] + team: |/path/to/alaveteli/software/script/handle-mail-replies + +with `FORWARD_NONBOUNCE_RESPONSES_TO`: 'raw_team@whatdotheyknow.com'` + +Finally, make sure you have `dc_use_split_config='true'` in +`/etc/exim4/update-exim4.conf.conf`, and execute the command +`update-exim4.conf`. + +Note that if the file `/etc/exim4/exim4.conf` exists then `update-exim4.conf` +will silently do nothing. Some distributions include this file. If +yours does, you will need to rename it before running `update-exim4.conf`. + +(You may also want to set `dc_eximconfig_configtype='internet'`, +`dc_local_interfaces='0.0.0.0 ; ::1'`, and +`dc_other_hostnames='<your-host-name>'`). + +### Troubleshooting (exim) + +To test mail delivery, run: + + exim -bt foi+request-1234@localhost + +This should tell you which routers are being processed. You should +see something like: + + $ exim -bt foi+request-1234@localhost + R: alaveteli pipe for snafflerequest-234@localhost + snafflerequest-234@localhost -> |/home/alaveteli/alaveteli/script/mailin + transport = alaveteli_mailin_transport + +This tells you that the routing part (making emails to +`foi\+.*@localhost` be forwarded to Alaveteli's `mailin` script) is +working. + +There is a great +[Exim Cheatsheet](http://bradthemad.org/tech/notes/exim_cheatsheet.php) +online that you may find useful. diff --git a/docs/installing/index.md b/docs/installing/index.md new file mode 100644 index 000000000..98efa7b8a --- /dev/null +++ b/docs/installing/index.md @@ -0,0 +1,62 @@ +--- +layout: page +title: Installing +--- + +# Installing Alaveteli + +<p class="lead"> + Although you can install Alaveteli and just change it when you need it, we + recommend you adopt a way of <strong>deploying</strong> it automatically. + This has several advantages, especially for your + <a href="{{ site.baseurl }}docs/glossary/#production">production server</a>. +</p> + +## Before you start + +This is important: you need to decide if you are installing Alaveteli for +[development]({{ site.baseurl }}docs/glossary/#development) or +[production]({{ site.baseurl }}docs/glossary/#production). + +A **development** site is one where you're going to change, customise, and +perhaps experiment while you get it up and running. You should always do this +first. In this environment you can see debug messages, and you don't need to +worry too much about the efficiency and performance of the site (because it's +not really getting lots of traffic). + +A **production** site is different: you want your production site to run as +efficiently as possible, so things like caching are swiched on, and debug +messages switched off. It's important to be able to deploy changes to a +production site quickly and efficiently, so we recommend you consider using a +[deployment mechanism]({{ site.baseurl }}docs/installing/deploy/) too. + +Ideally, you should also have a [staging site]({{ site.baseurl }}docs/glossary/#staging), +which is used solely to test new code in an identical environment to your +production site but before it goes live. + +If you're in doubt, you're probably running a development site. Get it up and +running, play with it, customise it, and -- later -- you can install it as a +production server. + +## Deployment + +If you're running a production server, we **strongly recommend** you +use the Capistrano [deployment mechanism]({{ site.baseurl }}docs/installing/deploy/) +that's included with Alaveteli. Set this up and you never have to edit files on +those servers, because Capistrano takes care of that for you. + +## Installing the core code + +* [Install on Amazon EC2]({{ site.baseurl }}docs/installing/ami) using our AMI +* [Use the installation script]({{ site.baseurl }}docs/installing/script) which does the full installation on your own server +* [Manual installation]({{ site.baseurl }}docs/installing/manual_install) -- step-by-step instructions + +If you're setting up a development server on MacOS X, we've also got +[MacOS installation instructions]({{ site.baseurl }}docs/installing/macos). + +## Other installation information + +Alaveteli needs to be able to send and receive email, so you need to setup your +MTA (Mail Transfer Agent) appropriately. + +* [Installing the MTA]({{ site.baseurl }}docs/installing/email) diff --git a/docs/installing/macos.md b/docs/installing/macos.md new file mode 100644 index 000000000..6e1986d59 --- /dev/null +++ b/docs/installing/macos.md @@ -0,0 +1,155 @@ +--- +layout: page +title: Installing on MacOS X +--- + +# Installation on MacOS X + +<p class="lead"> + We don't recommend using OS X in production, but if you want to get + Alaveteli running on your Mac for development, these guidelines should + help. +</p> + +Note that there are [other ways to install Alaveteli]({{ site.baseurl }}docs/installing). + +## MacOS X 10.7 + +Follow these instructions to get Alaveteli running locally on an OS X machine. These instructions have been tested with Xcode 4.1 on OS X Lion (10.7). We do not recommend using OS X in production. + +**Note:** This guide is currently incomplete. Please help by posting issues to the [alaveteli-dev Google group](https://groups.google.com/group/alaveteli-dev) or by submitting pull requests. + +## Xcode + +If you are using OS X Lion, download *Command Line Tools for Xcode* from [Apple](https://developer.apple.com/downloads/index.action). This is a new package from Apple that provides the command-line build tools separate from the rest of Xcode. You need to register for a free Apple Developer account. + +**Note:** As of Xcode 4.2, a non-LLVM version of GCC is no longer included. Homebrew has dealt with it by [switching to Clang](https://github.com/mxcl/homebrew/issues/6852). However, you may encounter errors installing RVM. *Please report these on the [mailing list](https://groups.google.com/group/alaveteli-dev).* The following instructions have been tested with Xcode 4.1. If necessary, you can install GCC from Xcode 4.1 by running: + + brew install http://github.com/adamv/homebrew-alt/raw/master/duplicates/apple-gcc42.rb + +## Homebrew + +Homebrew is a package manager for OS X. It is preferred over alternatives such as MacPorts and Fink. If you haven't already installed Homebrew, run the command: + + ruby <(curl -fsSkL raw.github.com/mxcl/homebrew/go) + +Next, install packages required by Alaveteli: + + brew install catdoc elinks gnuplot gs imagemagick libmagic libyaml links mutt poppler tnef wkhtmltopdf wv xapian unrtf + + +### Install postgresql + +Alaveteli uses PostgreSQL by default. If you've tested Alaveteli with MySQL or SQLite, let us know in the [alaveteli-dev Google group](https://groups.google.com/group/alaveteli-dev). + + brew install postgresql + initdb /usr/local/var/postgres + mkdir -p ~/Library/LaunchAgents + cp /usr/local/Cellar/postgresql/9.0.4/org.postgresql.postgres.plist ~/Library/LaunchAgents/ + launchctl load -w ~/Library/LaunchAgents/org.postgresql.postgres.plist + +## PDF Toolkit + +[Download the installer package](https://github.com/downloads/robinhouston/pdftk/pdftk.pkg) and install. + +## Ruby + +### Install RVM + +RVM is the preferred way to install multiple Ruby versions on OS X. Alaveteli uses Ruby 1.8.7. The following commands assume you are using the Bash shell. + + curl -L https://get.rvm.io | bash -s stable + +Read `rvm notes` and `rvm requirements` carefully for further instructions. Then, install Ruby: + + rvm install 1.8.7 + rvm install 1.9.3 + rvm use 1.9.3 --default + +### Install mahoro and pg with flags + +The `mahoro` and `pg` gems require special installation commands. Rubygems must be downgraded to 1.6.2 to avoid deprecation warnings when running tests. + + rvm 1.8.7 + gem update --system 1.6.2 + gem install mahoro -- --with-ldflags="-L/usr/local/Cellar/libmagic/5.09/lib" --with-cppflags="-I/usr/local/Cellar/libmagic/5.09/include" + env ARCHFLAGS="-arch x86_64" gem install pg + +#### Update + +As of August 22, 2012 or earlier, you can install `mahoro` in Ruby 1.9.3 on OS X 10.7 Lion with: + + brew install libmagic + gem install mahoro + +## Alaveteli + +The following is mostly from [the manual installation process]({{ site.baseur l}}/installing/manual_install). + +### Configure database + +Creates Alaveteli databases and an `foi` user with password `foi`. + + echo "CREATE DATABASE foi_development encoding = 'UTF8'; + CREATE DATABASE foi_test encoding = 'UTF8'; + CREATE USER foi WITH CREATEUSER; + ALTER USER foi WITH PASSWORD 'foi'; + ALTER USER foi WITH CREATEDB; + GRANT ALL PRIVILEGES ON DATABASE foi_development TO foi; + GRANT ALL PRIVILEGES ON DATABASE foi_test TO foi; + ALTER DATABASE foi_development OWNER TO foi; + ALTER DATABASE foi_test OWNER TO foi;" | psql -h localhost template1 + +### Clone Alaveteli + +We don't want to vendor Rails, as it causes problems locally. + + git clone https://github.com/mysociety/alaveteli.git + cd alaveteli + git submodule init + + sed -i~ 's/\\[submodule "vendor\/rails"\\]//' .git/config + + sed -i~ 's/url = git:\/\/github.com\/rails\/rails.git//' .git/config + git submodule update + +**Note:** Due to Markdown bugs, the first `sed` command above does not display properly if it appears in blockquote. + +### Configure Alaveteli + +Copy the example configuration files and configure `database.yml`. + + cp -f config/general.yml-example config/general.yml + cp -f config/memcached.yml-example config/memcached.yml + cp -f config/database.yml-example config/database.yml + sed -i~ 's/<username>/foi/' config/database.yml + sed -i~ 's/<password>/foi/' config/database.yml + sed -i~ 's/ port: 5432//' config/database.yml + sed -i~ 's/ # PostgreSQL 8.1 pretty please//' config/database.yml + +### Bundler + +Install the gems and finish setting up Alaveteli. + + rvm 1.8.7 + bundle + bundle exec rake db:create:all + bundle exec rake db:migrate + bundle exec rake db:test:prepare + +## Troubleshooting + +### Ruby version + +Ensure you are using the latest versions of Ruby. For example, some versions of Ruby 1.8.7 will segmentation fault, for example: + +``` +/Users/james/.rvm/gems/ruby-1.8.7-p357/gems/json-1.5.4/ext/json/ext/json/ext/parser.bundle: [BUG] Segmentation fault +ruby 1.8.7 (2011-12-28 patchlevel 357) [i686-darwin11.3.0] +``` + +Running `rvm install 1.8.7` should install the latest Ruby 1.8.7 patch level. Remember to switch to the new Ruby version before continuing. + +### Rake tasks + +Remember to run Rake tasks with `bundle exec`. To run the tests, for example, run `bundle exec rake`. diff --git a/docs/installing/manual_install.md b/docs/installing/manual_install.md new file mode 100644 index 000000000..c9625c0c9 --- /dev/null +++ b/docs/installing/manual_install.md @@ -0,0 +1,570 @@ +--- +layout: page +title: Manual installation +--- + + +# Manual Installation + +<p class="lead"> + The following instructions describe the step-by-step process for + installing Alaveteli. <em>You don't necessarily need to do it this + way:</em> it's usually easier to use the + <a href="{{ site.baseurl }}docs/installing/script">installation script</a> + or the + <a href="{{ site.baseurl }}docs/installing/ami">Amazon EC2 AMI</a>. +</p> + +Note that there are [other ways to install Alaveteli]({{ site.baseurl }}docs/installing). + +## Target operating system + +These instructions assume Debian Squeeze (64-bit) or Ubuntu 12.04 LTS +(precise). Debian Squeeze is the best supported deployment platform. We also +have instructions for [installing on MacOS]({{ site.baseurl }}docs/installing/macos). + +Commands are intended to be run via the terminal or over ssh. + + +## Get Alaveteli + +To start with, you may need to install git, e.g. with `sudo apt-get install +git-core` + +Next, get hold of the Alaveteli source code from github: + + git clone https://github.com/mysociety/alaveteli.git + cd alaveteli + +This will get the rails-3-develop branch, which has the latest (possibly buggy) +code. If you don't want to add or try new features, swap to the master branch +(which always contains the latest stable release): + + git checkout master + +## Install mySociety libraries + +Next, install mySociety's common ruby libraries. To fetch the contents of the +submodules, run: + + git submodule update --init + +## Install system dependencies + +These are packages that the software depends on: third-party software used to +parse documents, host the site, and so on. There are also packages that contain +headers necessary to compile some of the gem dependencies in the next step. + +Add the following repositories to `/etc/apt/sources.list`: + +**Debian Squeeze** + + cat > /etc/apt/sources.list.d/debian-backports.list <<EOF + deb http://backports.debian.org/debian-backports squeeze-backports main contrib non-free + EOF + +The repositories above let you install `wkhtmltopdf-static` and `bundler` using +`apt`. + +**Ubuntu Precise** + + cat > /etc/apt/sources.list.d/ubuntu-extra.list <<EOF + deb http://eu-west-1.ec2.archive.ubuntu.com/ubuntu/ precise multiverse + deb-src http://eu-west-1.ec2.archive.ubuntu.com/ubuntu/ precise multiverse + deb http://eu-west-1.ec2.archive.ubuntu.com/ubuntu/ precise-updates multiverse + deb-src http://eu-west-1.ec2.archive.ubuntu.com/ubuntu/ precise-updates multiverse + EOF + +The repositories above let you install `wkhtmltopdf-static` using `apt`. +`bundler` will have to be installed manually on Ubuntu Precise. + +### Packages customised by mySociety + +If you're using Debian, you should add the mySociety Debian archive to your +apt sources. + + cat > /etc/apt/sources.list.d/mysociety-debian.list <<EOF + deb http://debian.mysociety.org squeeze main + EOF + +Add the GPG key from the +[mySociety Debian Package Repository](http://debian.mysociety.org/). + + wget -O - https://debian.mysociety.org/debian.mysociety.org.gpg.key | sudo apt-key add - + +You should also configure package-pinning to reduce the priority of this +repository. + + cat > /etc/apt/preferences <<EOF + Package: * + Pin: origin debian.mysociety.org + Pin-Priority: 50 + EOF + +If you're using some other platform, you can optionally install these +dependencies manually, as follows: + +1. If you would like users to be able to get pretty PDFs as part of the +downloadable zipfile of their request history, install +[wkhtmltopdf](http://code.google.com/p/wkhtmltopdf/downloads/list). We +recommend downloading the latest, statically compiled version from the project +website, as this allows running headless (that is, without a graphical interface +running) on Linux. If you do install `wkhtmltopdf`, you need to edit a setting +in the config file to point to it (see below). If you don't install it, +everything will still work, but users will get ugly, plain text versions of +their requests when they download them. + +2. Version 1.44 of `pdftk` contains a bug which makes it loop forever in +certain edge conditions. Until it's incorporated into an official release, you +can either hope you don't encounter the bug (it ties up a rails process until +you kill it), patch it yourself, or use the Debian package +compiled by mySociety (see link in [issue +305](https://github.com/mysociety/alaveteli/issues/305)) + +### Install the dependencies + +Refresh the sources after adding the extra repositories: + + sudo apt-get update + +Now install the packages relevant to your system: + + # Debian Squeeze + sudo apt-get install $(cat config/packages.debian-squeeze) + + # Ubuntu Precise + sudo apt-get install $(cat config/packages.ubuntu-precise) + +Some of the files also have a version number listed in config/packages - check +that you have appropriate versions installed. Some also list "`|`" and offer a +choice of packages. + +## Install Ruby dependencies + +To install Alaveteli's Ruby dependencies, you need to install bundler. In +Debian, this is provided as a package (installed as part of the package install +process above). You could also install it as a gem: + + sudo gem install bundler + +## Configure Database + +There has been a little work done in trying to make the code work with other +databases (e.g., SQLite), but the currently supported database is PostgreSQL +("postgres"). + +If you don't have postgres installed: + + $ sudo apt-get install postgresql postgresql-client + +Create a `foi` user from the command line, like this: + + # sudo -u postgres createuser -s -P foi + +_Note:_ Leaving the password blank will cause great confusion if you're new to +PostgreSQL. + +Then create the databases: + + # sudo -u postgres createdb -T template0 -E SQL_ASCII -O foi foi_production + # sudo -u postgres createdb -T template0 -E SQL_ASCII -O foi foi_test + # sudo -u postgres createdb -T template0 -E SQL_ASCII -O foi foi_development + +We create using the ``SQL_ASCII`` encoding, because in postgres this is means +"no encoding"; and because we handle and store all kinds of data that may not +be valid UTF (for example, data originating from various broken email clients +that's not 8-bit clean), it's safer to be able to store *anything*, than reject +data at runtime. + +Now you need to set up the database config file to contain the name, username +and password of your postgres database. + +* Copy `database.yml-example` to `database.yml` in `alaveteli/config` +* Edit it to point to your local postgresql database in the development + and test sections. + +Example `development` section of `config/database.yml`: + + development: + adapter: postgresql + database: foi_development + username: foi + password: secure-password-here + host: localhost + port: 5432 + +Make sure that the user specified in `database.yml` exists, and has full +permissions on these databases. As they need the ability to turn off +constraints whilst running the tests they also need to be a superuser + +If you don't want your database user to be a superuser, you can add this line +to the test config in `database.yml` (as seen in `database.yml-example`) + + constraint_disabling: false + +## Configure email + +You will need to set up an email server (MTA) to send and receive emails. Full +configuration for an MTA is beyond the scope of this document -- see this +[example config for Exim4]({{ site.baseurl }}docs/installing/email). + +Note that in development mode mail is handled by mailcatcher by default so +that you can see the mails in a browser - see [http://mailcatcher.me/](http://mailcatcher.me/) for more +details. Start mailcatcher by running `bundle exec mailcatcher` in your +application directory. + +### Minimal + +If you just want to get the tests to pass, you will at a minimum need to allow +sending emails via a `sendmail` command (a requirement met, for example, with +`sudo apt-get install exim4`). + +### Detailed + +When an authority receives an email, the email's `reply-to` field is a magic +address which is parsed and consumed by the Rails app. + +To receive such email in a production setup, you will need to configure your +MTA to pipe incoming emails to the Alaveteli script `script/mailin`. Therefore, +you will need to configure your MTA to accept emails to magic addresses, and to +pipe such emails to this script. + +Magic email addresses are of the form: + + <foi+request-3-691c8388@example.com> + +The respective parts of this address are controlled with options in +`config/general.yml`, thus: + + INCOMING_EMAIL_PREFIX = 'foi+' + INCOMING_EMAIL_DOMAIN = 'example.com' + +When you set up your MTA, if there is some error inside Rails, the +email is returned with an exit code 75, which for Exim at least means the MTA +will try again later. Additionally, a stacktrace is emailed to `CONTACT_EMAIL`. + +See [this example]({{ site.baseurl }}docs/installing/email/) for a possible configuration for Exim (>=1.9). + +A well-configured installation of this code will have had Exim make +a backup copy of the email in a separate mailbox, just in case. + +## Set up configs + +Copy `config/general.yml-example` to `config/general.yml` and edit to your +taste. + +Note that the default settings for frontpage examples are designed to work with +the dummy data shipped with Alaveteli; once you have real data, you should +certainly edit these. + +The default theme is the "Alaveteli" theme. When you run `rails-post-deploy` +(see below), that theme gets installed automatically. + +Finally, copy `config/newrelic.yml-example` to `config/newrelic.yml`. This file +contains configuration information for the New Relic performance management +system. By default, monitoring is switched off by the `agent_enabled: false` +setting. See New Relic's [remote performance analysis](https://github.com/newrelic/rpm) instructions for switching it on +for both local and remote analysis. + + +## Deployment + +In the `alaveteli` directory, run: + + script/rails-post-deploy + +(This will need execute privs so `chmod 755` if necessary.) This sets up +directory structures, creates logs, installs/updates themes, runs database +migrations, etc. You should run it after each new software update. + +One of the things the script does is install dependencies (using `bundle +install`). Note that the first time you run it, part of the `bundle install` +that compiles `xapian-full` takes a *long* time! + +If you want some dummy data to play with, you can try loading the fixtures that +the test suite uses into your development database. You can do this with: + + script/load-sample-data + +Next, create the index for the search engine (Xapian): + + script/rebuild-xapian-index + +If this fails, the site should still mostly run, but it's a core component so +you should really try to get this working. + +## Run the Tests + +Make sure everything looks OK: + + bundle exec rake spec + +If there are failures here, something has gone wrong with the preceding steps +(see the next section for a common problem and workaround). You might be able +to move on to the next step, depending on how serious they are, but ideally you +should try to find out what's gone wrong. + +### glibc bug workaround + +There's a [bug in +glibc](http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=637239) which causes +Xapian to segfault when running the tests. Although the bug report linked to +claims it's fixed in the current Debian stable, it's not as of version +`2.11.3-2`. + +Until it's fixed (e.g. `libc6 2.13-26` does work), you can get the tests to +pass by setting `export LD_PRELOAD=/lib/libuuid.so.1`. + +## Run the Server + +Run the following to get the server running: + + bundle exec rails server --environment=development + +By default the server listens on all interfaces. You can restrict it to the +localhost interface by adding `--binding=127.0.0.1` + +The server should have told you the URL to access in your browser to see the +site in action. + +## Administrator privileges + +The administrative interface is at the URL `/admin`. + +Only users with the `super` admin level can access the admin interface. Users +create their own accounts in the usual way, and then administrators can give +them `super` privileges. + +There is an emergency user account which can be accessed via +`/admin?emergency=1`, using the credentials `ADMIN_USERNAME` and +`ADMIN_PASSWORD`, which are set in `general.yml`. To bootstrap the +first `super` level accounts, you will need to log in as the emergency +user. You can disable the emergency user account by setting `DISABLE_EMERGENCY_USER` to `true` in `general.yml`. + +Users with the superuser role also have extra privileges in the website +frontend, such as being able to categorise any request, being able to view +items that have been hidden from the search, and being presented with "admin" +links next to individual requests and comments in the front end. + +It is possible completely to override the administrator authentication by +setting `SKIP_ADMIN_AUTH` to `true` in `general.yml`. + +## Cron jobs and init scripts + +`config/crontab-example` contains the cronjobs run on WhatDoTheyKnow. It's in a +strange templating format they use in mySociety. mySociety render the example +file to reference absolute paths, and then drop it in `/etc/cron.d/` on the +server. + +The `ugly` format uses simple variable substitution. A variable looks like +`!!(*= $this *)!!`. The variables are: + +* `vhost`: part of the path to the directory where the software is + served from. In the mySociety files, it usually comes as + `/data/vhost/!!(*= $vhost *)!!` -- you should replace that whole + port with a path to the directory where your Alaveteli software + installation lives, e.g. `/var/www/` +* `vhost_dir`: the entire path to the directory where the software is + served from. -- you should replace this with a path to the + directory where your Alaveteli software installation lives, + e.g. `/var/www/` +* `vcspath`: the name of the alaveteli checkout, e.g. `alaveteli`. + Thus, `/data/vhost/!!(*= $vhost *)!!/!!(*= $vcspath *)!!` might be + replaced with `/var/www/alaveteli` in your cron tab +* `user`: the user that the software runs as +* `site`: a string to identify your alaveteli instance + +There is a rake task that will help to rewrite this file into one that is +useful to you, which can be invoked with: + + bundle exec rake config_files:convert_crontab \ + DEPLOY_USER=deploy \ + VHOST_DIR=/dir/above/alaveteli \ + VCSPATH=alaveteli \ + SITE=alaveteli \ + CRONTAB=config/crontab-example > crontab + +You should change the `DEPLOY_USER`, `VHOST_DIR`, `VCSPATH` and `SITE` +environment variables to match your server and installation. You should also +edit the resulting `crontab` file to customize the `MAILTO` variable. + +One of the cron jobs refers to a script at `/etc/init.d/foi-alert-tracks`. This +is an init script, a copy of which lives in `config/alert-tracks-debian.ugly`. +As with the cron jobs above, replace the variables (and/or bits near the +variables) with paths to your software. You can use the rake task `rake +config_files:convert_init_script` to do this. + +`config/purge-varnish-debian.ugly` is a similar init script, which is optional +and not required if you choose not to run your site behind Varnish (see below). +Either tweak the file permissions to make the scripts executable by your deploy +user, or add the following line to your sudoers file to allow these to be run +by your deploy user (named `deploy` in this case): + + deploy ALL = NOPASSWD: /etc/init.d/foi-alert-tracks, /etc/init.d/foi-purge-varnish + +The cron jobs refer to a program `run-with-lockfile`. See [this +issue](https://github.com/mysociety/alaveteli/issues/112) for a discussion of +where to find this program, and how you might replace it. This [one line +script](https://gist.github.com/3741194) can install this program system-wide. + +## Set up production web server + +It is not recommended to run the website using the default Rails web server. +There are various recommendations here: http://rubyonrails.org/deploy + +We usually use Passenger / mod_rails. The file at `conf/httpd.conf-example` +gives you an example config file for WhatDoTheyKnow. At a minimum, you should +include the following in an Apache configuration file: + + PassengerResolveSymlinksInDocumentRoot on + PassengerMaxPoolSize 6 # Recommend setting this to 3 or less on servers with 512MB RAM + +Under all but light loads, it is strongly recommended to run the server behind +an http accelerator like Varnish. A sample varnish VCL is supplied in +`conf/varnish-alaveteli.vcl`. + +It's strongly recommended that you run the site over SSL. (Set FORCE_SSL to +true in config/general.yml). For this you will need an SSL certificate for your +domain and you will need to configure an SSL terminator to sit in front of +Varnish. If you're already using Apache as a web server you could simply use +Apache as the SSL terminator. A minimal configuration would look something like +this: + + <VirtualHost *:443> + ServerName www.yourdomain + + ProxyRequests Off + ProxyPreserveHost On + ProxyPass / http://localhost:80/ + ProxyPassReverse / http://localhost:80/ + RequestHeader set X-Forwarded-Proto 'https' + + SSLEngine on + SSLProtocol all -SSLv2 + SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM + + SSLCertificateFile /etc/apache2/ssl/ssl.crt + SSLCertificateKeyFile /etc/apache2/ssl/ssl.key + SSLCertificateChainFile /etc/apache2/ssl/sub.class2.server.ca.pem + SSLCACertificateFile /etc/apache2/ssl/ca.pem + SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown + + </VirtualHost> + +Notice the line `RequestHeader` that sets the `X-Forwarded-Proto` header. This +is important. This ultimately tells Rails that it's serving a page over https +and so it knows to include that in any absolute urls it serves. + +We have some [production server best practice +notes]({{ site.baseurl}}docs/running/server/). + +## Upgrading Alaveteli + +The developer team policy is that the master branch in git should always +contain the latest stable release. Therefore, in production, you should usually +have your software deployed from the master branch, and an upgrade can be +simply `git pull`. + +Patch version increases (e.g. 1.2.3 → 1.2.**4**) should not require any further +action on your part. + +Minor version increases (e.g. 1.2.4 → 1.**3**.0) will usually require further +action. You should read the `CHANGES.md` document to see what's changed since +your last deployment, paying special attention to anything in the "Updgrading" +sections. + +Any upgrade may include new translations strings, i.e. new or altered messages +to the user that need translating to your locale. You should visit Transifex +and try to get your translation up to 100% on each new release. Failure to do +so means that any new words added to the Alaveteli source code will appear in +your website in English by default. If your translations didn't make it to the +latest release, you will need to download the updated `app.po` for your locale +from Transifex and save it in the `locale/` folder. + +You should always run the script `scripts/rails-post-deploy` after each +deployment. This runs any database migrations for you, plus various other +things that can be automated for deployment. + +## Troubleshooting + +* **Incoming emails aren't appearing in my Alaveteli install** + + First, you need to check that your MTA is delivering relevant + incoming emails to the `script/mailin` command. There are various + ways of setting your MTA up to do this; we have documented + [one way of doing it]({{ site.baseurl }}docs/installing/email/#troubleshooting-exim) + in Exim, including a command you can use to check that the email + routing is set up correctly. + + Second, you need to test that the mailin script itself is working + correctly, by running it from the command line, First, find a + valid "To" address for a request in your system. You can do this + through your site's admin interface, or from the command line, + like so: + + $ ./script/console + Loading development environment (Rails 2.3.14) + >> InfoRequest.find_by_url_title("why_do_you_have_such_a_fancy_dog").incoming_email + => "request-101-50929748@localhost" + + Now take the source of a valid email (there are some sample emails in + `spec/fixtures/files/`); edit the `To:` header to match this address; + and then pipe it through the mailin script. A non-zero exit code + means there was a problem. For example: + + $ cp spec/fixtures/files/incoming-request-plain.email /tmp/ + $ perl -pi -e 's/^To:.*/To: <request-101-50929748@localhost>/' /tmp/incoming-request-plain.email + $ ./script/mailin < /tmp/incoming-request-plain.email + $ echo $? + 75 + + The `mailin` script emails the details of any errors to + `CONTACT_EMAIL` (from your `general.yml` file). A common problem is + for the user that the MTA runs as not to have write access to + `files/raw_emails/`. + +* **Various tests fail with "*Your PostgreSQL connection does not support + unescape_bytea. Try upgrading to pg 0.9.0 or later.*"** + + You have an old version of `pg`, the ruby postgres driver. In + Ubuntu, for example, this is provided by the package `libdbd-pg-ruby`. + + Try upgrading your system's `pg` installation, or installing the pg + gem with `gem install pg` + +* **Some of the tests relating to mail are failing, with messages like + "*when using TMail should load an email with funny MIME settings' + FAILED*"** + + This sounds like the tests are running using the `production` + environment, rather than the `test` environment, for some reason. + +* **Non-ASCII characters are being displayed as asterisks in my incoming messages** + + We rely on `elinks` to convert HTML email to plain text. + Normally, the encoding should just work, but under some + circumstances it appears that `elinks` ignores the parameters + passed to it from Alaveteli. + + To force `elinks` always to treat input as UTF8, add the following + to `/etc/elinks/elinks.conf`: + + set document.codepage.assume = "utf-8" + set document.codepage.force_assumed = 1 + + You should also check that your locale is set up correctly. See + [this issue followup](https://github.com/mysociety/alaveteli/issues/128#issuecomment-1814845) + for further discussion. + +* **I'm seeing `rake: command not found` when running the post install script** + + The script uses `rake`. + + It may be that the binaries installed by bundler are not put in the + system `PATH`; therefore, in order to run `rake` (needed for + deployments), you may need to do something like: + + ln -s /usr/lib/ruby/gems/1.8/bin/rake /usr/local/bin/ + + + diff --git a/docs/installing/script.md b/docs/installing/script.md new file mode 100644 index 000000000..d4382c98f --- /dev/null +++ b/docs/installing/script.md @@ -0,0 +1,65 @@ +--- +layout: page +title: Installing the easy way +--- + +# Installation script + +<p class="lead"> + If you prefer to use your own server, we've provided an installation script which does most of the work for you. +</p> + +Note that there are [other ways to install Alaveteli]({{ site.baseurl }}docs/installing). + +## Installing with the installation script + +If you have a clean installation of Debian squeeze or Ubuntu precise, you can +use an install script in our commonlib repository to set up a working instance +of Alaveteli. This is not suitable for production (it runs in development mode, +for example) but should set up a functional installation of the site. + +**Warning: only use this script on a newly installed server – it will make +significant changes to your server’s setup, including modifying your nginx +setup, creating a user account, creating a database, installing new packages +etc.** + +To download the script, run the following command: + + curl -O https://raw.github.com/mysociety/commonlib/master/bin/install-site.sh + +If you run this script with `sh install-site.sh`, you'll see its usage message: + + Usage: ./install-site.sh [--default] <SITE-NAME> <UNIX-USER> [HOST] + HOST is only optional if you are running this on an EC2 instance. + --default means to install as the default site for this server, + rather than a virtualhost for HOST. + +In this case `<SITE-NAME>` should be `alaveteli`. `<UNIX-USER>` is the name of +the Unix user that you want to own and run the code. (This user will be created +by the script.) + +The `HOST` parameter is a hostname for the server that will be usable +externally – a virtualhost for this name will be created by the script, unless +you specified the `--default` option. This parameter is optional if you are on +an EC2 instance, in which case the hostname of that instance will be used. + +For example, if you wish to use a new user called `alaveteli` and the hostname +`alaveteli.127.0.0.1.xip.io`, creating a virtualhost just for that hostname, +you could download and run the script with: + + sudo sh install-site.sh alaveteli alaveteli alaveteli.127.0.0.1.xip.io + +([xip.io](http://xip.io/) is a helpful domain for development.) + +Or, if you want to set this up as the default site on an EC2 instance, you +could download the script, make it executable and then invoke it with: + + sudo ./install-site.sh --default alaveteli alaveteli + +When the script has finished, you should have a working copy of the website, +accessible via the hostname you supplied to the script. + +If you have any problems or questions, please ask on the [Alaveteli Google +Group](https://groups.google.com/forum/#!forum/alaveteli-dev) or [report an +issue](https://github.com/mysociety/alaveteli/issues?state=open). + |