diff options
Diffstat (limited to 'docs')
29 files changed, 4661 insertions, 1138 deletions
diff --git a/docs/customising/config.md b/docs/customising/config.md index 8a25e8df6..c03e0fc9e 100644 --- a/docs/customising/config.md +++ b/docs/customising/config.md @@ -71,7 +71,7 @@ indentation correct. If in doubt, look at the examples already in the file, and <code><a href="#admin_username">ADMIN_USERNAME</a></code> <br> <code><a href="#admin_password">ADMIN_PASSWORD</a></code> -<br> <code><a href="#admin_username">DISABLE_EMERGENCY_USER</a></code> +<br> <code><a href="#disable_emergency_user">DISABLE_EMERGENCY_USER</a></code> <br> <code><a href="#skip_admin_auth">SKIP_ADMIN_AUTH</a></code> ### Email management: @@ -106,11 +106,11 @@ indentation correct. If in doubt, look at the examples already in the file, and ### Behaviour settings and switches: <code><a href="#new_response_reminder_after_days">NEW_RESPONSE_REMINDER_AFTER_DAYS</a></code> +<br> <code><a href="#authority_must_respond">AUTHORITY_MUST_RESPOND</a></code> <br> <code><a href="#max_requests_per_user_per_day">MAX_REQUESTS_PER_USER_PER_DAY</a></code> <br> <code><a href="#override_all_public_body_request_emails">OVERRIDE_ALL_PUBLIC_BODY_REQUEST_EMAILS</a></code> <br> <code><a href="#allow_batch_requests">ALLOW_BATCH_REQUESTS</a></code> <br> <code><a href="#public_body_list_fallback_to_default_locale">PUBLIC_BODY_LIST_FALLBACK_TO_DEFAULT_LOCALE</a></code> -<br> <code><a href="#cache_fragments">CACHE_FRAGMENTS</a></code> ### External public services: @@ -126,6 +126,7 @@ indentation correct. If in doubt, look at the examples already in the file, and <br> <code><a href="#use_mailcatcher_in_development">USE_MAILCATCHER_IN_DEVELOPMENT</a></code> <br> <code><a href="#use_ghostscript_compression">USE_GHOSTSCRIPT_COMPRESSION</a></code> <br> <code><a href="#html_to_pdf_command">HTML_TO_PDF_COMMAND</a></code> +<br> <code><a href="#cache_fragments">CACHE_FRAGMENTS</a></code> --- @@ -189,67 +190,222 @@ indentation correct. If in doubt, look at the examples already in the file, and </dd> <dt> - <a name="iso_country_code"><code>ISO_COUNTRY_CODE</code></a> + <a name="force_registration_on_new_request"><code>FORCE_REGISTRATION_ON_NEW_REQUEST</code></a> </dt> <dd> - The <a href="http://en.wikipedia.org/wiki/ISO_3166-1_alpha-2">ISO country code</a> - of the country in which your Alaveteli site is deployed. + Does a user needs to sign in to start the New Request process? <div class="more-info"> <p>Example:</p> <ul class="examples"> <li> - <code>ISO_COUNTRY_CODE: GB</code> + <code>FORCE_REGISTRATION_ON_NEW_REQUEST: false</code> </li> </ul> </div> </dd> <dt> - <a name="time_zone"><code>TIME_ZONE</code></a> + <a name="theme_urls"><code>THEME_URLS</code></a> </dt> <dd> - This is the <a href="http://en.wikipedia.org/wiki/List_of_tz_database_time_zones">timezone</a> - that Alaveteli usese to display times and dates. - If not set, defaults to UTC. + URLs of <a href="{{ site.baseurl }}docs/customising/themes/">themes</a> to download and use + (when running the <code>rails-post-deploy</code> script). The earlier in the list means + the templates have a higher priority. <div class="more-info"> <p>Example:</p> <ul class="examples"> <li> - <code>TIME_ZONE: Australia/Sydney</code> + <pre> +THEME_URLS: + - 'git://github.com/mysociety/alavetelitheme.git' +</pre> </li> </ul> </div> </dd> <dt> - <a name="blog_feed"><code>BLOG_FEED</code></a> + <a name="theme_branch"><code>THEME_BRANCH</code></a> </dt> <dd> - These feeds are displayed accordingly on the Alaveteli "blog" page: <!-- TODO --> + When <code>rails-post-deploy</code> installs the <a href="{{ site.baseurl }}docs/customising/themes/">themes</a>, + it will try the theme branch first, but only if you've set <code>THEME_BRANCH</code> + to be true. If the branch doesn't exist it will fall back to using a tagged version + specific to your installed alaveteli version, and if that doesn't exist it will + back to <code>master</code>. + <p> + The default theme is the "Alaveteli" theme. This gets installed automatically when + <code>rails-post-deploy</code> runs. + </p> <div class="more-info"> <p>Example:</p> <ul class="examples"> <li> - <code>BLOG_FEED: 'https://www.mysociety.org/category/projects/whatdotheyknow/feed/'</code> + <code>THEME_BRANCH: false</code> </li> </ul> </div> </dd> <dt> - <a name="twitter_username"><code>TWITTER_USERNAME</code></a> - <a name="twitter_widget_id"><code>TWITTER_WIDGET_ID</code></a> + <a name="frontpage_publicbody_examples"><code>FRONTPAGE_PUBLICBODY_EXAMPLES</code></a> </dt> <dd> - If you want a twitter feed displayed on the "blog" page, provide the widget ID and username. + Specify which public bodies you want to be listed as examples on the home page, + using their <code>short_names</code>. + If you want more than one, separate them with semicolons. + Comment this out if you want this to be auto-generated. + <p> + <strong>Warning</strong>: this is slow — don't use in production! + </p> <div class="more-info"> <p>Examples:</p> <ul class="examples"> <li> - <code>TWITTER_USERNAME: WhatDoTheyKnow</code> + <code>FRONTPAGE_PUBLICBODY_EXAMPLES: 'tgq'</code> </li> <li> - <code>TWITTER_WIDGET_ID: '833549204689320031'</code> + <code>FRONTPAGE_PUBLICBODY_EXAMPLES: 'tgq;foo;bar'</code> + </li> + <li> + <code># FRONTPAGE_PUBLICBODY_EXAMPLES: </code> + </li> + </ul> + </div> + </dd> + + <dt> + <a name="public_body_statistics_page"><code>PUBLIC_BODY_STATISTICS_PAGE</code></a> & + <a name="minimum_requests_for_statistics"><code>MINIMUM_REQUESTS_FOR_STATISTICS</code></a> + </dt> + <dd> + If <strong>PUBLIC_BODY_STATISTICS_PAGE</strong> is set to true, Alaveteli will make a + page of statistics on the performance of public bodies (which you can see at + <code>/body_statistics</code>). + The page will only consider public bodies that have had at least the number of requests + set by <strong>MINIMUM_REQUESTS_FOR_STATISTICS</strong>. + + <div class="more-info"> + <p>Example:</p> + <ul class="examples"> + <li> + <code>PUBLIC_BODY_STATISTICS_PAGE: false</code> + </li> + <li> + <code>MINIMUM_REQUESTS_FOR_STATISTICS: 50</code> + </li> + </ul> + </div> + </dd> + + <dt> + <a name="responsive_styling"><code>RESPONSIVE_STYLING</code></a> + </dt> + <dd> + + Use the responsive base stylesheets and templates, rather than + those that only render the site at a fixed width. These + stylesheets are currently experimental but will become the default + in the future. They allow the site to render nicely on mobile + devices as well as larger screens. Currently the fixed width + stylesheets are used by default. + + <div class="more-info"> + <p>Example:</p> + <ul class="examples"> + <li> + <code>RESPONSIVE_STYLING: true</code> + </li> + </ul> + </div> + </dd> + + <dt> + <a name="read_only"><code>READ_ONLY</code></a> + </dt> + <dd> + If present, <strong>READ_ONLY</strong> puts the site in read-only mode, + and uses the text as reason (whole paragraph). Please use a read-only database + user as well, as it only checks in a few obvious places. + <div class="more-info"> + <p>Examples:</p> + <ul class="examples"> + <li> + Typically, you do <strong>not</strong> want to run your site in + read-only mode — so set <strong>READ_ONLY</strong> to be + an empty string. + <br> + <code> + READ_ONLY: '' + </code> + </li> + <li> + <code> + READ_ONLY: 'The site is not currently accepting requests while we move the server.' + </code> + </li> + </ul> + </div> + </dd> + + <dt> + <a name="staging_site"><code>STAGING_SITE</code></a> + </dt> + <dd> + Is this a + <a href="{{site.baseurl}}docs/glossary/#staging" class="glossary__link">staging</a> or + <a href="{{site.baseurl}}docs/glossary/#development" class="glossary__link">development</a> site? + If not, it's a live <a href="{{site.baseurl}}docs/glossary/#production" class="glossary__link">production</a> + site. This setting controls whether or not the <code>rails-post-deploy</code> + script will create the file <code>config/rails_env.rb</code> file to force + Rails into production environment. + <div class="more-info"> + <p>Examples:</p> + <ul class="examples"> + <li> + For staging or development: + <p> + <code>STAGING_SITE: 1</code> + </p> + </li> + <li> + For production: + <p> + <code>STAGING_SITE: 0</code> + </p> + </li> + </ul> + </div> + </dd> + + <dt> + <a name="iso_country_code"><code>ISO_COUNTRY_CODE</code></a> + </dt> + <dd> + The <a href="http://en.wikipedia.org/wiki/ISO_3166-1_alpha-2">ISO country code</a> + of the country in which your Alaveteli site is deployed. + <div class="more-info"> + <p>Example:</p> + <ul class="examples"> + <li> + <code>ISO_COUNTRY_CODE: GB</code> + </li> + </ul> + </div> + </dd> + + <dt> + <a name="time_zone"><code>TIME_ZONE</code></a> + </dt> + <dd> + This is the <a href="http://en.wikipedia.org/wiki/List_of_tz_database_time_zones">timezone</a> + that Alaveteli usese to display times and dates. + If not set, defaults to UTC. + <div class="more-info"> + <p>Example:</p> + <ul class="examples"> + <li> + <code>TIME_ZONE: Australia/Sydney</code> </li> </ul> </div> @@ -342,96 +498,66 @@ indentation correct. If in doubt, look at the examples already in the file, and </dd> <dt> - <a name="frontpage_publicbody_examples"><code>FRONTPAGE_PUBLICBODY_EXAMPLES</code></a> + <a name="admin_username"><code>ADMIN_USERNAME</code></a> + & + <a name="admin_password"><code>ADMIN_PASSWORD</code></a> + <br> + <a name="disable_emergency_user"><code>DISABLE_EMERGENCY_USER</code></a> </dt> <dd> - Specify which public bodies you want to be listed as examples on the home page, - using their <code>short_names</code>. - If you want more than one, separate them with semicolons. - Comment this out if you want this to be auto-generated. - <p> - <strong>Warning</strong>: this is slow — don't use in production! - </p> + Details for the + <a href="{{site.baseurl}}docs/glossary/#emergency" class="glossary__link">emergency user</a>. + <p> + This is useful for creating the initial admin users for your site: + <ul> + <li>Create a new user (using regular sign up on the site)</li> + <li>Log in as the emergency user</li> + <li>Promote the new account</li> + <li>Disable the emergency user</li> + </ul> + </p> + <p> + For details of this process, see + <a href="{{site.baseurl}}docs/installing/next_steps/#create-a-superuser-admin-account">creating + a superuser account</a>. + </p> <div class="more-info"> <p>Examples:</p> <ul class="examples"> <li> - <code>FRONTPAGE_PUBLICBODY_EXAMPLES: 'tgq'</code> - </li> - <li> - <code>FRONTPAGE_PUBLICBODY_EXAMPLES: 'tgq;foo;bar'</code> + <code>ADMIN_USERNAME: 'adminxxxx'</code> </li> <li> - <code># FRONTPAGE_PUBLICBODY_EXAMPLES: </code> - </li> - </ul> - </div> - </dd> - - <dt> - <a name="theme_urls"><code>THEME_URLS</code></a> - </dt> - <dd> - URLs of <a href="{{ site.baseurl }}docs/customising/themes/">themes</a> to download and use - (when running the <code>rails-post-deploy</code> script). The earlier in the list means - the templates have a higher priority. - <div class="more-info"> - <p>Example:</p> - <ul class="examples"> - <li> - <pre> -THEME_URLS: - - 'git://github.com/mysociety/alavetelitheme.git' -</pre> + <code>ADMIN_PASSWORD: 'passwordx'</code> </li> - </ul> - </div> - </dd> - - <dt> - <a name="theme_branch"><code>THEME_BRANCH</code></a> - </dt> - <dd> - When <code>rails-post-deploy</code> installs the <a href="{{ site.baseurl }}docs/customising/themes/">themes</a>, - it will try the theme branch first, but only if you've set <code>THEME_BRANCH</code> - to be true. If the branch doesn't exist it will fall back to using a tagged version - specific to your installed alaveteli version, and if that doesn't exist it will - back to <code>master</code>. - <p> - The default theme is the "Alaveteli" theme. This gets installed automatically when - <code>rails-post-deploy</code> runs. - </p> - <div class="more-info"> - <p>Example:</p> - <ul class="examples"> <li> - <code>THEME_BRANCH: false</code> + <code>DISABLE_EMERGENCY_USER: false</code> </li> </ul> </div> </dd> <dt> - <a name="force_registration_on_new_request"><code>FORCE_REGISTRATION_ON_NEW_REQUEST</code></a> + <a name="skip_admin_auth"><code>SKIP_ADMIN_AUTH</code></a> </dt> <dd> - Does a user needs to sign in to start the New Request process? + Set this to true, and the admin interface will be available to anonymous users. + Obviously, you should not set this to be true in production environments. <div class="more-info"> <p>Example:</p> <ul class="examples"> <li> - <code>FORCE_REGISTRATION_ON_NEW_REQUEST: false</code> + <code>SKIP_ADMIN_AUTH: false</code> </li> </ul> </div> </dd> - <dt> <a name="incoming_email_domain"><code>INCOMING_EMAIL_DOMAIN</code></a> </dt> <dd> - Your email domain for incoming mail. + Your email domain for incoming mail. See also <a href="{{ site.baseurl }}docs/installing/email#how-alaveteli-handles-email">How Alaveteli handles email</a>. <div class="more-info"> <p>Example:</p> <ul class="examples"> @@ -449,7 +575,7 @@ THEME_URLS: <a name="incoming_email_prefix"><code>INCOMING_EMAIL_PREFIX</code></a> </dt> <dd> - An optional prefix to help you distinguish FOI requests. + An optional prefix to help you distinguish FOI requests. See also <a href="{{ site.baseurl }}docs/installing/email#how-alaveteli-handles-email">How Alaveteli handles email</a>. <div class="more-info"> <p>Example:</p> <ul class="examples"> @@ -463,7 +589,6 @@ THEME_URLS: </div> </dd> - <dt> <a name="incoming_email_secret"><code>INCOMING_EMAIL_SECRET</code></a> </dt> @@ -495,53 +620,12 @@ THEME_URLS: </dd> <dt> - <a name="admin_username"><code>ADMIN_USERNAME</code></a> - & - <a name="admin_password"><code>ADMIN_PASSWORD</code></a> - <br> - <a name="admin_username"><code>DISABLE_EMERGENCY_USER</code></a> - </dt> - <dd> - The emergency user. - <div class="more-info"> - <p>Examples:</p> - <ul class="examples"> - <li> - <code>ADMIN_USERNAME: 'adminxxxx'</code> - </li> - <li> - <code>ADMIN_PASSWORD: 'passwordx'</code> - </li> - <li> - <code>DISABLE_EMERGENCY_USER: false</code> - </li> - </ul> - </div> - </dd> - - <dt> - <a name="skip_admin_auth"><code>SKIP_ADMIN_AUTH</code></a> - </dt> - <dd> - Set this to true, and the admin interface will be available to anonymous users. - Obviously, you should not set this to be true in production environments. - <div class="more-info"> - <p>Example:</p> - <ul class="examples"> - <li> - <code>SKIP_ADMIN_AUTH: false</code> - </li> - </ul> - </div> - </dd> - - <dt> <a name="contact_email"><code>CONTACT_EMAIL</code></a> & <a name="contact_name"><code>CONTACT_NAME</code></a> </dt> <dd> - Email "from" details. + Email "from" details. See also <a href="{{ site.baseurl }}docs/installing/email#how-alaveteli-handles-email">How Alaveteli handles email</a>. <div class="more-info"> <p>Examples:</p> <ul class="examples"> @@ -560,7 +644,7 @@ THEME_URLS: <a name="track_sender_name"><code>TRACK_SENDER_NAME</code></a> </dt> <dd> - Email "from" details for track messages. + Email "from" details for track messages. See also <a href="{{ site.baseurl }}docs/installing/email#how-alaveteli-handles-email">How Alaveteli handles email</a>. <div class="more-info"> <p>Examples:</p> <ul class="examples"> @@ -591,283 +675,267 @@ THEME_URLS: </dd> <dt> - <a name="cookie_store_session_secret"><code>COOKIE_STORE_SESSION_SECRET</code></a> + <a name="exception_notifications_from"><code>EXCEPTION_NOTIFICATIONS_FROM</code></a> & + <a name="exception_notifications_to"><code>EXCEPTION_NOTIFICATIONS_TO</code></a> </dt> <dd> - Secret key for signing cookie_store sessions. Make it long and random. + Email address(es) used for sending exception notifications. <div class="more-info"> - <p>Example:</p> + <p>Examples:</p> <ul class="examples"> <li> - <code>COOKIE_STORE_SESSION_SECRET: 'uIngVC238Jn9NsaQizMNf89pliYmDBFugPjHS2JJmzOp8'</code> + <pre> +EXCEPTION_NOTIFICATIONS_FROM: do-not-reply-to-this-address@example.com + +EXCEPTION_NOTIFICATIONS_TO: + - robin@example.com + - seb@example.com +</pre> </li> </ul> </div> </dd> <dt> - <a name="read_only"><code>READ_ONLY</code></a> + <a name="forward_nonbounce_responses_to"><code>FORWARD_NONBOUNCE_RESPONSES_TO</code></a> </dt> <dd> - If present, <strong>READ_ONLY</strong> puts the site in read-only mode, - and uses the text as reason (whole paragraph). Please use a read-only database - user as well, as it only checks in a few obvious places. + The email address to which non-bounce responses should be forwarded. See also <a href="{{ site.baseurl }}docs/installing/email#how-alaveteli-handles-email">How Alaveteli handles email</a>. <div class="more-info"> - <p>Examples:</p> + <p>Example:</p> <ul class="examples"> <li> - Typically, you do <strong>not</strong> want to run your site in - read-only mode — so set <strong>READ_ONLY</strong> to be - an empty string. - <br> - <code> - READ_ONLY: '' - </code> - </li> - <li> - <code> - READ_ONLY: 'The site is not currently accepting requests while we move the server.' - </code> + <code>FORWARD_NONBOUNCE_RESPONSES_TO: user-support@example.com</code> </li> </ul> </div> </dd> <dt> - <a name="staging_site"><code>STAGING_SITE</code></a> + <a name="mta_log_path"><code>MTA_LOG_PATH</code></a> </dt> <dd> - Is this a - <a href="{{site.baseurl}}docs/glossary/#staging" class="glossary">staging</a> or - <a href="{{site.baseurl}}docs/glossary/#development" class="glossary">development</a> site? - If not, it's a live <a href="{{site.baseurl}}docs/glossary/#production" class="glossary">production</a> - site. This setting controls whether or not the <code>rails-post-deploy</code> - script will create the file <code>config/rails_env.rb</code> file to force - Rails into production environment. + Path to your exim or postfix log files that will get sucked up + by <code>script/load-mail-server-logs</code>. <div class="more-info"> - <p>Examples:</p> + <p>Example:</p> <ul class="examples"> <li> - For staging or development: - <p> - <code>STAGING_SITE: 1</code> - </p> - </li> - <li> - For production: - <p> - <code>STAGING_SITE: 0</code> - </p> + <code>MTA_LOG_PATH: '/var/log/exim4/exim-mainlog-*'</code> </li> </ul> </div> </dd> <dt> - <a name="recaptcha_public_key"><code>RECAPTCHA_PUBLIC_KEY</code></a> & - <a name="recaptcha_private_key"><code>RECAPTCHA_PRIVATE_KEY</code></a> + <a name="mta_log_type"><code>MTA_LOG_TYPE</code></a> </dt> <dd> - Recaptcha, for detecting humans. Get keys here: - <a href="http://recaptcha.net/whyrecaptcha.html">http://recaptcha.net/whyrecaptcha.html</a> + Are you using "exim" or "postfix" for your Mail Transfer Agnt (MTA)? <div class="more-info"> - <p>Examples:</p> + <p>Example:</p> <ul class="examples"> <li> - <code>RECAPTCHA_PUBLIC_KEY: '7HoPjGBBBBBBBBBkmj78HF9PjjaisQ893'</code> - </li> - <li> - <code>RECAPTCHA_PRIVATE_KEY: '7HjPjGBBBBBCBBBpuTy8a33sgnGG7A'</code> + <code>MTA_LOG_TYPE: "exim"</code> </li> </ul> </div> </dd> <dt> - <a name="new_response_reminder_after_days"><code>NEW_RESPONSE_REMINDER_AFTER_DAYS</code></a> + <a name="cookie_store_session_secret"><code>COOKIE_STORE_SESSION_SECRET</code></a> </dt> <dd> - Number of days after which to send a 'new response reminder'. + Secret key for signing cookie_store sessions. Make it long and random. <div class="more-info"> <p>Example:</p> <ul class="examples"> <li> - <code>NEW_RESPONSE_REMINDER_AFTER_DAYS: [3, 10, 24]</code> + <code>COOKIE_STORE_SESSION_SECRET: 'uIngVC238Jn9NsaQizMNf89pliYmDBFugPjHS2JJmzOp8'</code> </li> </ul> </div> </dd> <dt> - <a name="debug_record_memory"><code>DEBUG_RECORD_MEMORY</code></a> + <a name="recaptcha_public_key"><code>RECAPTCHA_PUBLIC_KEY</code></a> & + <a name="recaptcha_private_key"><code>RECAPTCHA_PRIVATE_KEY</code></a> </dt> <dd> - For debugging memory problems. If true, Alaveteli logs - the memory use increase of the Ruby process due to the - request (Linux only). Since Ruby never returns memory to the OS, if the - existing process previously served a larger request, this won't - show any consumption for the later request. + Recaptcha, for detecting humans. Get keys here: + <a href="http://recaptcha.net/whyrecaptcha.html">http://recaptcha.net/whyrecaptcha.html</a> <div class="more-info"> - <p>Example:</p> + <p>Examples:</p> <ul class="examples"> <li> - <code>DEBUG_RECORD_MEMORY: false</code> + <code>RECAPTCHA_PUBLIC_KEY: '7HoPjGBBBBBBBBBkmj78HF9PjjaisQ893'</code> + </li> + <li> + <code>RECAPTCHA_PRIVATE_KEY: '7HjPjGBBBBBCBBBpuTy8a33sgnGG7A'</code> </li> </ul> </div> </dd> - <dt> - <a name="use_ghostscript_compression"><code>USE_GHOSTSCRIPT_COMPRESSION</code></a> + <a name="gaze_url"><code>GAZE_URL</code></a> </dt> <dd> - Currently we default to using pdftk to compress PDFs. You can - optionally try Ghostscript, which should do a better job of - compression. Some versions of pdftk are buggy with respect to - compression, in which case Alaveteli doesn't recompress the PDFs at - all and logs a warning message "Unable to compress PDF" — which would - be another reason to try this setting. + Alateveli uses mySociety's gazeteer service to determine country from incoming + IP address (this lets us suggest an Alaveteli in their country, if one exists). + You shouldn't normally need to change this. <div class="more-info"> <p>Example:</p> <ul class="examples"> <li> - <code>USE_GHOSTSCRIPT_COMPRESSION: true</code> + <code>GAZE_URL: http://gaze.mysociety.org</code> </li> </ul> </div> </dd> - <dt> - <a name="gaze_url"><code>GAZE_URL</code></a> + <a name="ga_code"><code>GA_CODE</code> (GA=Google Analytics)</a> </dt> <dd> - Alateveli uses mySociety's gazeteer service to determine country from incoming - IP address (this lets us suggest an Alaveteli in their country, if one exists). - You shouldn't normally need to change this. + Adding a value here will enable Google Analytics on all non-admin pages for non-admin users. <div class="more-info"> - <p>Example:</p> + <p>Examples:</p> <ul class="examples"> <li> - <code>GAZE_URL: http://gaze.mysociety.org</code> + <code>GA_CODE: ''</code> + </li> + <li> + <code>GA_CODE: 'AB-8222142-14'</code> </li> </ul> </div> </dd> <dt> - <a name="forward_nonbounce_responses_to"><code>FORWARD_NONBOUNCE_RESPONSES_TO</code></a> + <a name="utility_search_path"><code>UTILITY_SEARCH_PATH</code></a> </dt> <dd> - The email address to which non-bounce responses should be forwarded + Search path for external command-line utilities (such as pdftohtml, pdftk, unrtf). <div class="more-info"> <p>Example:</p> <ul class="examples"> <li> - <code>FORWARD_NONBOUNCE_RESPONSES_TO: user-support@example.com</code> + <code>UTILITY_SEARCH_PATH: ["/usr/bin", "/usr/local/bin"]</code> </li> </ul> </div> </dd> <dt> - <a name="html_to_pdf_command"><code>HTML_TO_PDF_COMMAND</code></a> + <a name="shared_files_path"><code>SHARED_FILES_PATH</code></a> </dt> <dd> - Path to a program that converts an HTML page in a file to PDF. It - should take two arguments: the URL, and a path to an output file. - A static binary of <a href="http://wkhtmltopdf.org">wkhtmltopdf</a> is recommended. - If the command is not present, a text-only version will be rendered - instead. + In some deployments of Alaveteli you may wish to install each newly + deployed version alongside the previous ones, in which case certain + files and resources should be shared between these installations. + For example, the <code>files</code> directory, the <code>cache</code> directory and the + generated graphs such as <code>public/foi-live-creation.png</code>. If you're + installing Alaveteli in such a setup then set <strong>SHARED_FILES_PATH</strong> to + the directory you're keeping these files under. Otherwise, leave it blank. <div class="more-info"> <p>Example:</p> <ul class="examples"> <li> - <code>HTML_TO_PDF_COMMAND: /usr/local/bin/wkhtmltopdf-amd64</code> + <code>SHARED_FILES_PATH: ''</code> <!-- TODO specific example --> </li> </ul> </div> </dd> <dt> - <a name="exception_notifications_from"><code>EXCEPTION_NOTIFICATIONS_FROM</code></a> & - <a name="exception_notifications_to"><code>EXCEPTION_NOTIFICATIONS_TO</code></a> + <a name="shared_files"><code>SHARED_FILES</code></a> & + <a name="shared_directories"><code>SHARED_DIRECTORIES</code></a> </dt> <dd> - Email address(es) used for sending exception notifications. + If you have <strong>SHARED_FILES_PATH</strong> set, then these options list the files + and directories that are shared; i.e. those to which the deploy scripts + should create symlinks from the repository. <div class="more-info"> <p>Examples:</p> <ul class="examples"> <li> <pre> -EXCEPTION_NOTIFICATIONS_FROM: do-not-reply-to-this-address@example.com - -EXCEPTION_NOTIFICATIONS_TO: - - robin@example.com - - seb@example.com -</pre> +SHARED_FILES: + - config/database.yml + - config/general.yml + - config/rails_env.rb + - config/newrelic.yml + - config/httpd.conf + - public/foi-live-creation.png + - public/foi-user-use.png + - config/aliases + </pre> + </li> + <li> + <pre> +SHARED_DIRECTORIES: + - files/ + - cache/ + - lib/acts_as_xapian/xapiandbs/ + - vendor/bundle + - public/assets + </pre> </li> </ul> </div> </dd> <dt> - <a name="max_requests_per_user_per_day"><code>MAX_REQUESTS_PER_USER_PER_DAY</code></a> + <a name="new_response_reminder_after_days"><code>NEW_RESPONSE_REMINDER_AFTER_DAYS</code></a> </dt> <dd> - This rate limiting can be turned off per-user via the admin interface. + Number of days after which to send a 'new response reminder'. <div class="more-info"> <p>Example:</p> <ul class="examples"> <li> - <code>MAX_REQUESTS_PER_USER_PER_DAY: 6</code> + <code>NEW_RESPONSE_REMINDER_AFTER_DAYS: [3, 10, 24]</code> </li> </ul> </div> </dd> <dt> - <a name="varnish_host"><code>VARNISH_HOST</code></a> + <a name="authority_must_respond"><code>AUTHORITY_MUST_RESPOND</code></a> </dt> <dd> - If you're running behind Varnish, it might help to set this to - work out where to send purge requests. - Otherwise, don't set it. + <div class="attention-box info"> + Introduced in Alaveteli version 0.21 + </div> + Set this to <code>true</code> if authorities must respond by law. Set to <code>false</code> otherwise. It defaults to <code>true</code>. At the moment this just controls the display of some UI text telling users that the authority must respond to them by law. <div class="more-info"> - <p>Examples:</p> + <p>Example:</p> <ul class="examples"> <li> - <code>VARNISH_HOST: null</code> - </li> - <li> - <code>VARNISH_HOST: localhost</code> + <code>AUTHORITY_MUST_RESPOND: true</code> </li> </ul> </div> </dd> <dt> - <a name="ga_code"><code>GA_CODE</code> (GA=Google Analytics)</a> + <a name="max_requests_per_user_per_day"><code>MAX_REQUESTS_PER_USER_PER_DAY</code></a> </dt> <dd> - Adding a value here will enable Google Analytics on all non-admin pages for non-admin users. + This rate limiting can be turned off per-user via the admin interface. <div class="more-info"> - <p>Examples:</p> + <p>Example:</p> <ul class="examples"> <li> - <code>GA_CODE: ''</code> - </li> - <li> - <code>GA_CODE: 'AB-8222142-14'</code> + <code>MAX_REQUESTS_PER_USER_PER_DAY: 6</code> </li> </ul> </div> </dd> - <dt> <a name="override_all_public_body_request_emails"><code>OVERRIDE_ALL_PUBLIC_BODY_REQUEST_EMAILS</code></a> </dt> @@ -893,242 +961,199 @@ EXCEPTION_NOTIFICATIONS_TO: </dd> <dt> - <a name="utility_search_path"><code>UTILITY_SEARCH_PATH</code></a> - </dt> - <dd> - Search path for external command-line utilities (such as pdftohtml, pdftk, unrtf). - <div class="more-info"> - <p>Example:</p> - <ul class="examples"> - <li> - <code>UTILITY_SEARCH_PATH: ["/usr/bin", "/usr/local/bin"]</code> - </li> - </ul> - </div> - </dd> - - - <dt> - <a name="mta_log_path"><code>MTA_LOG_PATH</code></a> + <a name="allow_batch_requests"><code>ALLOW_BATCH_REQUESTS</code></a> </dt> <dd> - Path to your exim or postfix log files that will get sucked up - by <code>script/load-mail-server-logs</code>. + Allow some users to make batch requests to multiple authorities. Once + this is set to true, you can enable batch requests for an individual + user via the user admin page. <div class="more-info"> <p>Example:</p> <ul class="examples"> <li> - <code>MTA_LOG_PATH: '/var/log/exim4/exim-mainlog-*'</code> + <code>ALLOW_BATCH_REQUESTS: false</code> </li> </ul> </div> </dd> <dt> - <a name="mta_log_type"><code>MTA_LOG_TYPE</code></a> + <a name="public_body_list_fallback_to_default_locale"><code>PUBLIC_BODY_LIST_FALLBACK_TO_DEFAULT_LOCALE</code></a> </dt> <dd> - Are you using "exim" or "postfix" for your Mail Transfer Agnt (MTA)? - + If you would like the public body list page to include bodies that have no translation + in the current locale (but which do have a translation in the default locale), set this to true. <div class="more-info"> <p>Example:</p> <ul class="examples"> <li> - <code>MTA_LOG_TYPE: "exim"</code> + <code>PUBLIC_BODY_LIST_FALLBACK_TO_DEFAULT_LOCALE: false</code> </li> </ul> </div> </dd> <dt> - <a name="donation_url"><code>DONATION_URL</code></a> + <a name="blog_feed"><code>BLOG_FEED</code></a> </dt> <dd> - URL where people can donate to the organisation running the site. If set, - this will be included in the message people see when their request is - successful. + These feeds are displayed accordingly on the Alaveteli "blog" page: <!-- TODO --> <div class="more-info"> <p>Example:</p> <ul class="examples"> <li> - <code>DONATION_URL: "https://www.mysociety.org/donate/"</code> + <code>BLOG_FEED: 'https://www.mysociety.org/category/projects/whatdotheyknow/feed/'</code> </li> </ul> </div> </dd> <dt> - <a name="public_body_statistics_page"><code>PUBLIC_BODY_STATISTICS_PAGE</code></a> & - <a name="minimum_requests_for_statistics"><code>MINIMUM_REQUESTS_FOR_STATISTICS</code></a> + <a name="twitter_username"><code>TWITTER_USERNAME</code></a> + <a name="twitter_widget_id"><code>TWITTER_WIDGET_ID</code></a> </dt> <dd> - If <strong>PUBLIC_BODY_STATISTICS_PAGE</strong> is set to true, Alaveteli will make a - page of statistics on the performance of public bodies (which you can see at - <code>/body_statistics</code>). - The page will only consider public bodies that have had at least the number of requests - set by <strong>MINIMUM_REQUESTS_FOR_STATISTICS</strong>. - + If you want a twitter feed displayed on the "blog" page, provide the widget ID and username. <div class="more-info"> - <p>Example:</p> + <p>Examples:</p> <ul class="examples"> <li> - <code>PUBLIC_BODY_STATISTICS_PAGE: false</code> + <code>TWITTER_USERNAME: WhatDoTheyKnow</code> </li> <li> - <code>MINIMUM_REQUESTS_FOR_STATISTICS: 50</code> + <code>TWITTER_WIDGET_ID: '833549204689320031'</code> </li> </ul> </div> </dd> <dt> - <a name="public_body_list_fallback_to_default_locale"><code>PUBLIC_BODY_LIST_FALLBACK_TO_DEFAULT_LOCALE</code></a> + <a name="donation_url"><code>DONATION_URL</code></a> </dt> <dd> - If you would like the public body list page to include bodies that have no translation - in the current locale (but which do have a translation in the default locale), set this to true. + URL where people can donate to the organisation running the site. If set, + this will be included in the message people see when their request is + successful. <div class="more-info"> <p>Example:</p> <ul class="examples"> <li> - <code>PUBLIC_BODY_LIST_FALLBACK_TO_DEFAULT_LOCALE: false</code> + <code>DONATION_URL: "https://www.mysociety.org/donate/"</code> </li> </ul> </div> </dd> - <dt> - <a name="use_mailcatcher_in_development"><code>USE_MAILCATCHER_IN_DEVELOPMENT</code></a> + <a name="debug_record_memory"><code>DEBUG_RECORD_MEMORY</code></a> </dt> <dd> - <!-- TODO check mailcatcher URL --> - If true, while in development mode, try to send mail by SMTP to port - 1025 (the port the <a href="http://mailcatcher.me">mailcatcher</a> listens on by default): + For debugging memory problems. If true, Alaveteli logs + the memory use increase of the Ruby process due to the + request (Linux only). Since Ruby never returns memory to the OS, if the + existing process previously served a larger request, this won't + show any consumption for the later request. + <div class="more-info"> <p>Example:</p> <ul class="examples"> <li> - <code>USE_MAILCATCHER_IN_DEVELOPMENT: true</code> + <code>DEBUG_RECORD_MEMORY: false</code> </li> </ul> </div> </dd> <dt> - <a name="cache_fragments"><code>CACHE_FRAGMENTS</code></a> + <a name="varnish_host"><code>VARNISH_HOST</code></a> </dt> <dd> - Use memcached to cache HTML fragments for better performance. - This will only have an effect in environments where - <code>config.action_controller.perform_caching</code> is set to true - + If you're running behind Varnish, it might help to set this to + work out where to send purge requests. + Otherwise, don't set it. <div class="more-info"> - <p>Example:</p> + <p>Examples:</p> <ul class="examples"> <li> - <code>CACHE_FRAGMENTS: true</code> + <code>VARNISH_HOST: null</code> + </li> + <li> + <code>VARNISH_HOST: localhost</code> </li> </ul> </div> </dd> - - <dt> - <a name="shared_files_path"><code>SHARED_FILES_PATH</code></a> + <a name="use_mailcatcher_in_development"><code>USE_MAILCATCHER_IN_DEVELOPMENT</code></a> </dt> <dd> - In some deployments of Alaveteli you may wish to install each newly - deployed version alongside the previous ones, in which case certain - files and resources should be shared between these installations. - For example, the <code>files</code> directory, the <code>cache</code> directory and the - generated graphs such as <code>public/foi-live-creation.png</code>. If you're - installing Alaveteli in such a setup then set <strong>SHARED_FILES_PATH</strong> to - the directory you're keeping these files under. Otherwise, leave it blank. + <!-- TODO check mailcatcher URL --> + If true, while in development mode, try to send mail by SMTP to port + 1025 (the port the <a href="http://mailcatcher.me">mailcatcher</a> listens on by default): <div class="more-info"> <p>Example:</p> <ul class="examples"> <li> - <code>SHARED_FILES_PATH: ''</code> <!-- TODO specific example --> + <code>USE_MAILCATCHER_IN_DEVELOPMENT: true</code> </li> </ul> </div> </dd> - <dt> - <a name="shared_files"><code>SHARED_FILES</code></a> & - <a name="shared_directories"><code>SHARED_DIRECTORIES</code></a> + <a name="use_ghostscript_compression"><code>USE_GHOSTSCRIPT_COMPRESSION</code></a> </dt> <dd> - If you have <strong>SHARED_FILES_PATH</strong> set, then these options list the files - and directories that are shared; i.e. those to which the deploy scripts - should create symlinks from the repository. + Currently we default to using pdftk to compress PDFs. You can + optionally try Ghostscript, which should do a better job of + compression. Some versions of pdftk are buggy with respect to + compression, in which case Alaveteli doesn't recompress the PDFs at + all and logs a warning message "Unable to compress PDF" — which would + be another reason to try this setting. <div class="more-info"> - <p>Examples:</p> + <p>Example:</p> <ul class="examples"> <li> - <pre> -SHARED_FILES: - - config/database.yml - - config/general.yml - - config/rails_env.rb - - config/newrelic.yml - - config/httpd.conf - - public/foi-live-creation.png - - public/foi-user-use.png - - config/aliases - </pre> - </li> - <li> - <pre> -SHARED_DIRECTORIES: - - files/ - - cache/ - - lib/acts_as_xapian/xapiandbs/ - - vendor/bundle - - public/assets - </pre> + <code>USE_GHOSTSCRIPT_COMPRESSION: true</code> </li> </ul> </div> </dd> <dt> - <a name="allow_batch_requests"><code>ALLOW_BATCH_REQUESTS</code></a> + <a name="html_to_pdf_command"><code>HTML_TO_PDF_COMMAND</code></a> </dt> <dd> - Allow some users to make batch requests to multiple authorities. Once - this is set to true, you can enable batch requests for an individual - user via the user admin page. + Path to a program that converts an HTML page in a file to PDF. It + should take two arguments: the URL, and a path to an output file. + A static binary of <a href="http://wkhtmltopdf.org">wkhtmltopdf</a> is recommended. + If the command is not present, a text-only version will be rendered + instead. <div class="more-info"> <p>Example:</p> <ul class="examples"> <li> - <code>ALLOW_BATCH_REQUESTS: false</code> + <code>HTML_TO_PDF_COMMAND: /usr/local/bin/wkhtmltopdf-amd64</code> </li> </ul> </div> </dd> + <dt> - <a name="responsive_styling"><code>RESPONSIVE_STYLING</code></a> + <a name="cache_fragments"><code>CACHE_FRAGMENTS</code></a> </dt> <dd> - - Use the responsive base stylesheets and templates, rather than - those that only render the site at a fixed width. These - stylesheets are currently experimental but will become the default - in the future. They allow the site to render nicely on mobile - devices as well as larger screens. Currently the fixed width - stylesheets are used by default. + Use memcached to cache HTML fragments for better performance. + This will only have an effect in environments where + <code>config.action_controller.perform_caching</code> is set to true <div class="more-info"> <p>Example:</p> <ul class="examples"> <li> - <code>RESPONSIVE_STYLING: true</code> + <code>CACHE_FRAGMENTS: true</code> </li> </ul> </div> diff --git a/docs/running/states.md b/docs/customising/states.md index 1c3fb217e..7b89ef45d 100644 --- a/docs/running/states.md +++ b/docs/customising/states.md @@ -6,16 +6,16 @@ title: States of requests # Request states <p class="lead"> - A <a href="{{site.baseurl}}docs/glossary/#request" class="glossary">request</a> + A <a href="{{site.baseurl}}docs/glossary/#request" class="glossary__link">request</a> passes through different <strong>states</strong> as it is processed. These may vary from one jurisdiction to another. </p> The request states are defined in the Alaveteli code, and we recommend you use them (provided they match the <a href="{{ site.baseurl }}docs/glossary/#foi" -class="glossary">FOI law</a> in your own jurisdiction). But if you do need to +class="glossary__link">FOI law</a> in your own jurisdiction). But if you do need to customise them, you can — see -<a href="{{ site.baseurl }}docs/customising/themes/">Customising the request states</a> for details. +<a href="{{ site.baseurl }}docs/customising/themes/#customising-the-request-states">Customising the request states</a> for details. ## WhatDoTheyKnow example @@ -24,7 +24,7 @@ may be in any of the states described below. Note that your site doesn't need to use the same states as WhatDoTheyKnow does. For example, Kosovo's instance uses slightly different states: see -[this comparison of their differences]({{ site.baseurl }}docs/running/states_informatazyrtare/). +[this comparison of their differences]({{ site.baseurl }}docs/customising/states_informatazyrtare/). ### States diff --git a/docs/running/states_informatazyrtare.md b/docs/customising/states_informatazyrtare.md index bd5ff1a1c..e94f96588 100644 --- a/docs/running/states_informatazyrtare.md +++ b/docs/customising/states_informatazyrtare.md @@ -14,7 +14,7 @@ title: States of requests (InformataZyrtare) The request states are defined in the Alaveteli code, and we recommend you use them (provided they match the <a href="{{ site.baseurl }}docs/glossary/#foi" -class="glossary">FOI law</a> in your own jurisdiction). +class="glossary__link">FOI law</a> in your own jurisdiction). ## InformataZyrtare.org (Kosovo) example @@ -39,13 +39,13 @@ states]({{ site.baseurl }}docs/customising/themes/) for details on how to do thi ### States used by WDTK but not InformataZyrtare - * <a href="{{ site.baseurl }}docs/running/states/#awaiting_description">awaiting_description</a> - * <a href="{{ site.baseurl }}docs/running/states/#gone_postal">gone_postal</a> - * <a href="{{ site.baseurl }}docs/running/states/#internal_review">internal_review</a> - * <a href="{{ site.baseurl }}docs/running/states/#user_withdrawn">user_withdrawn</a> - * <a href="{{ site.baseurl }}docs/running/states/#waiting_response_very_overdue">waiting_response_very_overdue</a> + * <a href="{{ site.baseurl }}docs/customising/states/#awaiting_description">awaiting_description</a> + * <a href="{{ site.baseurl }}docs/customising/states/#gone_postal">gone_postal</a> + * <a href="{{ site.baseurl }}docs/customising/states/#internal_review">internal_review</a> + * <a href="{{ site.baseurl }}docs/customising/states/#user_withdrawn">user_withdrawn</a> + * <a href="{{ site.baseurl }}docs/customising/states/#waiting_response_very_overdue">waiting_response_very_overdue</a> -For more details, see all the [states used by WhatDoTheyKnow]({{site.baseurl}}docs/running/states/)) for details. +For more details, see all the [states used by WhatDoTheyKnow]({{site.baseurl}}docs/customising/states/) for details. --- @@ -54,7 +54,7 @@ For more details, see all the [states used by WhatDoTheyKnow]({{site.baseurl}}do ### Details of InformataZytare states -The states which aren't represented on [WDTK's states]({{ site.baseurl }}docs/running/states/) are described +The states which aren't represented on [WDTK's states]({{ site.baseurl }}docs/customising/states/) are described in a little more detail here: <ul class="definitions"> diff --git a/docs/customising/themes.md b/docs/customising/themes.md index bad1639d7..a2b498084 100644 --- a/docs/customising/themes.md +++ b/docs/customising/themes.md @@ -21,27 +21,93 @@ You don't need to be a programmer in order to make simple changes, but you will need to be confident enough to copy and change some files. If you're not sure about this, [ask for help](/community/)! +<div class="attention-box info"> + A theme is the way you tell Alaveteli which parts of your site look and behave + differently from the core site. These differences are implemented as a + collection of files (separate from the core Alaveteli source code), which + Alaveteli uses to override its default code. +</div> + +<div class="attention-box warning"> + When you customise Alaveteli, you should <strong>always use this + theme mechanism</strong> instead of editing the core Alaveteli files. If you + do not — that is, if you make custom changes to the main Alaveteli + source code — you may not be able to update your site with newer + Alaveteli code (new features and occassional bugfixes). + <p> + <em>Sometimes</em> you may want to change the core templates in a way that + would benefit everyone, in which case: great! But please discuss the changes + on the mailing list first, make them in a fork of Alaveteli, and then issue + a pull request. + </p> +</div> + +## Your theme is a separate repo + + +We use +<a href="{{ site.baseurl }}docs/glossary/#git" class="glossary__link">git</a> +to manage Alaveteli's source code, and Alaveteli expects your theme to be in +a git repository of its own. + +Although you *can* start customising your site on your +<a href="{{ site.baseurl }}docs/glossary/#development" class="glossary__link">development server</a> +by playing with the `alavetelitheme` theme that Alaveteli ships with, we recommend +you make it into your own repo as soon as you can. If you're seriously customising +— and certainly before you can deploy to a +<a href="{{ site.baseurl }}docs/glossary/#production" class="glossary__link">production server</a> — +you must do this. Make sure you choose a unique name for your theme (and hence its +repo). If your site is `abcexample.com`, we suggest you call your theme +something like `abcexample-theme`. + +Alaveteli's `themes:install` rake task, which installs themes, works by +getting the git repo from the URL specified in the config setting +[`THEME_URLS`]({{ site.baseurl }}docs/customising/config/#theme_urls). This is +why your theme must be in its own git repo. + +One way to create your own theme is to fork the `alavetelitheme` theme from +[https://github.com/mysociety/alavetelitheme](https://github.com/mysociety/alavetelitheme) +(giving it your own theme name), edit it or add files, and deploy it with `themes:install`. +Alternatively, since your site already has the default theme's files within it, +you can duplicate `alivetelitheme` (in `lib/themes/`) and change its name. + +<div class="attention-box helpful-hint"> + Here's an example of a complex theme in action: see the theme repo at + <a href="https://github.com/mysociety/whatdotheyknow-theme">https://github.com/mysociety/whatdotheyknow-theme</a>. + This is the theme for UK's Alaveteli instance + <a href="{{ site.baseurl}}docs/glossary/#wdtk" class="glossary__link">WhatDoTheyKnow</a>. + You can see it + <a href="https://www.whatdotheyknow.com">deployed on the WhatDoTheyKnow website</a>. + This happens because the WhatDoTheyKnow server has this setting in <code>config/general.yml</code>: + </p> + <pre><code>THEME_URLS: + - 'git://github.com/mysociety/whatdotheyknow-theme.git'</code></pre> +</div> + ## What you might want to change The most common requirement is to brand the site: at a minimum, -[inserting your own logo](#changing-the-logo) and [colour scheme](#changing-the-colour-scheme). You may also want to tweak -the different states that a request can go through. You'll also want -to edit the categories that public bodies can appear in (i.e. the -groupings on the left hand side of the -"[View authorities](https://www.whatdotheyknow.com/body/list/all)" page -on WhatDoTheyKnow. - -There may also be other things you want to customise -- drop a line on -the developer's mailing list to discuss what you need. We're still working -out the best way of doing these kinds of customisations! - -In any case, the important principle to bear in mind is that the less -you override and customise the code, the easier your site will be to -maintain in the long term. Any customisation is possible, but for -each customisation beyond the simple cases documented here, ask -yourself (or your client), "can we possibly live without this?" If the -answer is "no", then consider starting a discussion about a pluggable -way of achieving your goals, rather than overriding any of the core +[inserting your own logo](#changing-the-logo) and +[colour scheme](#changing-the-colour-scheme). You should also +[add the categories](#adding-your-own-categories-for-authorities) +that authorities can appear in (you can see these as groupings on the left-hand +side of the [View authorities](https://www.whatdotheyknow.com/body/list/all) page +on <a href="{{ site.baseurl }}docs/glossary/#wdtk" class="glossary__link">WhatDoTheyKnow</a>). +You may also want to +[tweak the different states](#customising-the-request-states) that a request can +go through. + +There may also be other things you want to customise -- talk to us on the +developer's mailing list to discuss what you need. We're happy to help work out +the best way of doing customisation and it's even possible that what you want +has already been done in someone else's theme. + +The important principle to bear in mind is that the less you override and +customise the code, the easier your site will be to maintain in the long term. +Any customisation is possible, but for each customisation beyond the simple +cases documented here, ask yourself (or your client), "can we possibly live +without this?" If the answer is "no", then always ask on the mailing list about +a pluggable way of achieving your goals before you override any of the core code. ## General principles @@ -49,119 +115,172 @@ code. We try to encapsulate all site-specific functionality in one of these places: -* Site [configuration]({{ site.baseurl }}docs/customising/config/) - (e.g., the name of your site, the available - languages, and so on — all in `config/general.yml`) -* Data (e.g. the public bodies to whom requests should be addressed) -* A theme, installed in `lib/themes`. +* **site configuration**<br> + use the [config settings]({{ site.baseurl }}docs/customising/config/) + for example, the name of your site, the available languages, and so on. + You change these by editing `config/general.yml`. +* **data**<br> + for example, the public authorities to whom requests should be addressed, + and the tags and categories for grouping them. You control all this + through the + <a href="{{ site.baseurl }}docs/glossary/#admin" class="glossary__link">admin + interface</a>: see the [admin manual]({{ site.baseurl }}docs/running/admin_manual). +* **a theme**<br> + installed in `lib/themes`. + The page you're reading now is all about what you can do in a theme. -This document is about what you can do in a theme. +By default, Alaveteli ships with the sample theme (`alavetelitheme`), so your +`config/general.yml` contains this: -By default, the sample theme ("alavetelitheme") has already been -installed. See the setting -[`THEME_URLS`]({{ site.baseurl }}docs/customising/config/#theme_urls) -in `general.yml` for an explanation. + THEME_URLS: + - 'git://github.com/mysociety/alavetelitheme.git' -You can also install the sample theme by hand, by running: +You can also install the theme by hand, by running: bundle exec rake themes:install +This installs whichever theme is specified by the +[`THEME_URLS`]({{ site.baseurl }}docs/customising/config/#theme_urls) +setting. + The sample theme contains examples for nearly everything you might -want to customise. You should probably make a copy, rename it, and +want to customise. We recommend you make a copy, rename it, and use that as the basis for your own theme. +<div class="attention-box info"> + The + <code><a href="{{ site.baseurl }}docs/customising/config/#theme_urls">THEME_URLS</a></code> + setting allows you to specifiy more than one theme — but + normally you only need one. +</div> + ## Make sure your theme is as lightweight as possible -The more you put in your theme, the harder it will be to upgrade to -future versions of Alaveteli. Everything you place in your theme -overrides things in the core theme, so if you make a new "main -template", then new widgets that appear in the core theme won't appear -on your website. +The more you put in your theme, the harder it will be to upgrade to future +versions of Alaveteli. -Therefore, you should consider how you can brand your website without -changing much in the core theme. The ideal would be if you are able -to rebrand the site by only changing the CSS. You will also need to -add custom help pages, as described below. +Everything you place in your theme overrides things in the core theme, so if +you make a new "main template", then new widgets that appear in the core theme +won't appear on your website. If you want them, you'll need to manually update +your version of the template to include them, and potentially you'll need to +do this every time the core theme changes. + +Therefore, you should consider how you can brand your website by changing +as little in the core theme as possible. An extreme -- but not impossible -- +way to do this is to rebrand the site by only changing the CSS, because this +means *none* of the templates are being overridden. + +However, even with minimal customisation, you must also add custom help pages +(described below). Alaveteli's default help pages are deliberately incomplete. +We know that every installation is going to be operating in different +circumstances, so a generic help text cannot be useful. You must write your +own, for your own users. ## Branding the site -The core templates that comprise the layout and user interface of an -Alaveteli site live in `app/views/`. They use Rails' ERB syntax. -For example, the template for the home page lives at -`app/views/general/frontpage.html.erb`, and the template for the "about -us" page is at `app/views/help/about.html.erb`. - -Obviously, you *could* edit those core files directly, but this would -be a Bad Idea, because you would find it increasingly hard to do -upgrades. Having said that, sometimes you may want to change the core -templates in a way that would benefit everyone, in which case, discuss -the changes on the mailing list, make them in a fork of Alaveteli, and -then issue a pull request. - -Normally, however, you should override these pages **in your own -theme**, by placing them at a corresponding location within your -theme's `lib/` directory. These means that a file at -`lib/themes/alavetelitheme/lib/views/help/about.rhml` will appear -instead of the core "about us" file. +The core templates define the layout and user interface of an Alaveteli site. +They are in `app/views/` and use +<a href="{{ site.baseurl }}" class="glossary__link">Rails</a>' +ERB syntax. For example, the template for the home page lives at +`app/views/general/frontpage.html.erb`, and the template for the "about us" +page is at `app/views/help/about.html.erb`. + +As described above, although you *could* edit those core files directly, this +would be a Bad Idea, because you would find it increasingly hard to do upgrades. + +Instead, you should override these pages *in your own theme*, by placing them +at a corresponding location within your theme's `lib/` directory. For example, +this means that if you put your own copy of the "about us" template +in <code>lib/themes/<em>yourtheme</em>/lib/views/help/about.html.erb</code>, +then that will appear instead of the core "about us" file. ### Changing the logo -Alaveteli uses Rails' [asset pipeline](http://guides.rubyonrails.org/asset_pipeline.html) to convert and compress stylesheets written in -<a href="{{ site.baseurl }}docs/glossary/#sass" class="glossary">Sass</a>, -the css extension language, to minified concatenated css. Assets are stored in core Alaveteli under `app/assets` - in `fonts`, `images`, `javascripts` and `stylesheets`. -The default theme has corresponding asset directories in `alavetelitheme/assets` Asset files placed in these directories will override those in the core directories. As with templates, a file at `lib/themes/alavetelitheme/assets/images/logo.png` will appear on the site instead of the logo from `app/assets/images/logo.png`. +Alaveteli uses Rails' [asset pipeline](http://guides.rubyonrails.org/asset_pipeline.html) +to convert and compress stylesheets written in +<a href="{{ site.baseurl }}docs/glossary/#sass" class="glossary__link">Sass</a> +into minified concatenated CSS. Assets are stored in core Alaveteli under +`app/assets` -- in `fonts`, `images`, `javascripts` and `stylesheets`. The +default theme has corresponding asset directories in `alavetelitheme/assets` +Asset files placed in these directories will override those in the core +directories. As with templates, a file at +<code>lib/themes/<em>yourtheme</em>/assets/images/logo.png</code> will appear on the +site instead of the logo from `app/assets/images/logo.png`. ### Changing the colour scheme Alaveteli uses a set of basic -<a href="{{ site.baseurl }}docs/glossary/#sass" class="glossary">Sass</a> -modules to define the layout for the site on different device sizes, and some basic styling. These modules are in `app/assets/stylesheets/responsive`. The colours and fonts are added in the theme - alavetelitheme defines them in `lib/themes/alavetelitheme/assets/stylesheets/responsive/custom.scss`. Colours used in the theme are defined as variables at the top of this file and you can edit them here. +<a href="{{ site.baseurl }}docs/glossary/#sass" class="glossary__link">Sass</a> +modules to define the layout for the site on different device sizes, and some +basic styling. These modules are in `app/assets/stylesheets/responsive`. The +colours and fonts are added in the theme -- `alavetelitheme` defines them in +`lib/themes/alavetelitheme/assets/stylesheets/responsive/custom.scss`. Colours +used in the theme are defined as variables at the top of this file and you can +edit them in your version of this file in your own theme. ### Changing other styling -To change other styling, you can add to or edit the styles in `lib/themes/alavetelitheme/assets/stylesheets/responsive/custom.scss`. Styles defined here will override those in the sass modules in `app/assets/stylesheets/responsive` as they will be imported last by `app/assets/stylesheets/responsive/all.scss`. However, if you want to substantially change the way a particular part of the site is laid out, you may want to override one of the core sass modules. You could override the layout of the front page, for example, by copying `app/assets/stylesheets/responsive/_frontpage_layout.scss` to `lib/themes/alavetelitheme/assets/stylesheets/responsive/_frontpage_layout.scss` and editing it. - -You can load extra stylesheets and javascript files by adding them to `lib/themes/alavetelitheme/lib/views/general/_before_head_end.html.erb` - -## Adding your own categories for public bodies - -Categories are implemented in Alaveteli using tags. Specific tags can -be designated to group authorities together as a category. - -There's a file in the sample theme, -`alavetelitheme/lib/public_body_categories_en.rb`, which contains a -nested structure that defines categories. It contains a comment -describing its structure. You should make a copy of this file for each -locale you support. +To change other styling, you can add to or edit the styles in +`lib/themes/alavetelitheme/assets/stylesheets/responsive/custom.scss`. +Styles defined here will override those in the sass modules in +`app/assets/stylesheets/responsive` as they will be imported last by +`app/assets/stylesheets/responsive/all.scss`. However, if you want to +substantially change the way a particular part of the site is laid out, +you may want to override one of the core Sass modules. You could override the +layout of the front page, for example, by copying +`app/assets/stylesheets/responsive/_frontpage_layout.scss` to +<code>lib/themes/<em>yourtheme</em>/assets/stylesheets/responsive/_frontpage_layout.scss</code> +and editing it. + +You can load extra stylesheets and javascript files by adding them to +<code>lib/themes/<em>yourtheme</em>/lib/views/general/_before_head_end.html.erb</code> + +## Adding your own categories for authorities + +You should add +<a href="{{ site.baseurl }}docs/glossary/#category" class="glossary__link">categories</a> +for the authorities on your site -- Alaveteli will display the authorities grouped +by categories if you have set any up. Alaveteli uses +<a href="{{ site.baseurl }}docs/glossary/#tag" class="glossary__link">tags</a> +to assign authorities to the right categories, but you should add tags anyway +because they are also used by the site's search facility. Together, categories +and tags help your users find the right authority for their request. + +You can set all this up using the +<a href="{{ site.baseurl }}docs/glossary/#admin" class="glossary__link">admin interface</a>. +See [more about categories and tags]({{ site.baseurl }}docs/running/categories_and_tags/) +for details. ## Customising the request states As mentioned above, if you can possibly live with the -[default Alaveteli request statuses]({{ site.baseurl }}docs/running/states/), -it would be good to do so. Note that you can set how many days counts -as "overdue" in the main site config file — +[default Alaveteli request statuses]({{ site.baseurl }}docs/customising/states/), +it would be good to do so. You can set how many days must pass before +a request is considered "overdue" in the main site config file — see [`REPLY_LATE_AFTER_DAYS`]({{ site.baseurl }}docs/customising/config/#reply_late_after_days). -If you can't live with the states as they are, there's a very basic -way to add to them (which will get improved over time). There's not -currently a way to remove any easily. There is an example of how to -do this in the `alavetelitheme`. +If you can't live with the states as they are, there's a very basic way to add +to them (we're working on this, so it will be improved over time). Currently, +there's no easy way to remove any. There is an example of how to do this in the +`alavetelitheme`. To do add states, create two modules in your theme, -`InfoRequestCustomStates` and `RequestControllerCustomStates`. The -former must have these methods: +`InfoRequestCustomStates` and `RequestControllerCustomStates`. + +`InfoRequestCustomStates` must have these methods: * `theme_calculate_status`: return a tag to identify the current state of the request * `theme_extra_states`: return a list of tags which identify the extra states you'd like to support * `theme_display_status`: return human-readable strings corresponding with these tags -The latter must have one method: +`RequestControllerCustomStates` must have one method: -* `theme_describe_state`: Return a notice for the user suitable for +* `theme_describe_state`: return a notice for the user suitable for displaying after they've categorised a request; and redirect them to a suitable next page -When you've added your extra states, you also need to create the following files in your theme: +When you've added your extra states, you also need to create the following files +in your theme: * `lib/views/general/_custom_state_descriptions.html.erb`: Descriptions of your new states, suitable for displaying to end users @@ -170,22 +289,85 @@ When you've added your extra states, you also need to create the following files 'completion' states, for displaying on the categorisation form that we ask requestors to fill out * `lib/views/general/_custom_state_transitions_pending.html.erb`: As - above, but for new states you might characterise as 'pending' + above, but for new states you might characterise as *pending* states. You can see examples of these customisations in [this commit](https://github.com/sebbacon/informatazyrtare-theme/commit/2b240491237bd72415990399904361ce9bfa431d) for the Kosovan version of Alaveteli, Informata Zyrtare (ignore the -file `lib/views/general/_custom_state_transitions.html.erb`, which is +file `_custom_state_transitions.html.erb`, which is unused). +## Customising the help pages + +The help pages are a really important part of an Alaveteli site. If you're running Alaveteli in another language, you'll want to show +your users localised versions of the help pages. Even if you're running +the site in English, the default help pages in Alaveteli are taken from +WhatDoTheyKnow, and are therefore relevant only to the UK. You should +take these pages as inspiration, but review their content with a view to +your jurisdiction. + +The important pages to customise and translate are listed here. We note where Alaveteli links to these pages (sometimes to anchors for particular sections within the pages) or takes users directly to them. + +* [About](https://github.com/mysociety/alaveteli/blob/master/app/views/help/about.html.erb): why the website exists, why it works, etc. When a user starts to make a request in Alaveteli, they are referred to the section here on [why authorities should respond to requests](https://github.com/mysociety/alaveteli/blob/master/app/views/help/about.html.erb#L29). + +* [contact](https://github.com/mysociety/alaveteli/blob/master/app/views/help/contact.html.erb): how to get in touch + +* [credits](https://github.com/mysociety/alaveteli/blob/master/app/views/help/credits.html.erb): who is involved in the site. Importantly, includes [a section](https://github.com/mysociety/alaveteli/blob/master/app/views/help/credits.html.erb#L71) on how users can help the project. Users are referred to this section if they categorise all the requests in the [categorisation game]({{ site.baseurl }}docs/glossary/#categorisation-game). + +* [officers](https://github.com/mysociety/alaveteli/blob/master/app/views/help/officers.html.erb): information for the officers who deal with FOI at authorities. They get a link to this page in emails that the site sends them. + +* [privacy](https://github.com/mysociety/alaveteli/blob/master/app/views/help/privacy.html.erb): privacy policy, plus information making it clear that requests are going to appear on the internet. Let users know if they are allowed to use pseudonyms in your jurisdiction. Users are referred to the [section on this page](https://github.com/mysociety/alaveteli/blob/master/app/views/help/privacy.html.erb#L114) about what to do if the authority says they only have a paper copy of the information requested if the user classifies their request as ['gone postal']({{ site.baseurl }}docs/customising/states/#gone_postal). + +* [requesting](https://github.com/mysociety/alaveteli/blob/master/app/views/help/requesting.html.erb): the main help page about making requests. How it works, how to decide who to write to, what they can expect in terms of responses, how to make appeals, etc. Users are referred to the [section on how quickly a response to their request should arrive](https://github.com/mysociety/alaveteli/blob/master/app/views/help/requesting.html.erb#L125) when their request is overdue for a response. They are referred to the [section on what to do if the Alaveteli site isn't showing the authority they want to request information](https://github.com/mysociety/alaveteli/blob/master/app/views/help/requesting.html.erb#L30) from the page that allows them to list and search authorities. + +* [unhappy](https://github.com/mysociety/alaveteli/blob/master/app/views/help/unhappy.html.erb): users are taken to this page after a request that has been somehow unsuccessful (e.g. the request has been refused, or the authority is insisting on a postal request). The page should encourage them to keep going, e.g. by starting a new request or addressing it to a different body. In particular users are referred to the [section on using other means](https://github.com/mysociety/alaveteli/blob/master/app/views/help/unhappy.html.erb#L83) to get their question answered. If the user has requested an internal review of their request, they are referred to [the section on this page](https://github.com/mysociety/alaveteli/blob/master/app/views/help/unhappy.html.erb#L28) that describes the law relating to how long a review should take. + +* [why email](https://github.com/mysociety/alaveteli/blob/master/app/views/help/_why_they_should_reply_by_email.html.erb): a snippet of information that explains why users should insist on replies by email. This is displayed next to requests that have ["gone postal"]({{ site.baseurl }}docs/customising/states/#gone_postal) - where the authority has asked for the user's physical address so that they can reply with a paper response. + +* [sidebar](https://github.com/mysociety/alaveteli/blob/master/app/views/help/_sidebar.html.erb): a menu for the help pages with a link to each one. You should customise this so that it includes any extra help pages you add, and doesn't include any you remove. + +You can add your own help pages to your site by replacing the default +pages in your theme with your own versions, using a locale suffix for +each page to indicate what language the page is written in. No locale +suffix is needed for pages written for the [default locale]({{ site.baseurl }}docs/customising/config/#default_locale) for the site. +For example, [alavetelitheme contains help +pages](https://github.com/mysociety/alavetelitheme/tree/master/lib/views/help) +for the default 'en' locale and an example Spanish 'about' page. If no +help page exists in the theme for a particular page in the locale that +the site is being viewed in, the default help page in English will be +shown. + + ## Adding new pages in the navigation -`alavetelitheme/lib/config/custom-routes.rb` allows you to extend the base routes in -Alaveteli. The example in `alavetelitheme` adds an extra help page. -You can also use this to override the behaviour of specific pages if -necessary. +You can extend the base routes in Alaveteli by modifying +<code>lib/themes/<em>yourtheme</em>/lib/config/custom-routes.rb</code>. +The example in `alavetelitheme` adds an extra help page. You can also use this +to override the behaviour of specific pages if necessary. ## Adding or overriding models and controllers -If you need to extend the behaviour of Alaveteli at the controller or model level, see `alavetelitheme/lib/controller_patches.rb` and `alavetelitheme/lib/model_patches.rb` for examples. +If you need to extend the behaviour of Alaveteli at the controller or model +level, see `alavetelitheme/lib/controller_patches.rb` and +`alavetelitheme/lib/model_patches.rb` for examples. + +## Quickly switching between themes + +On your +<a href="{{ site.baseurl }}docs/glossary/#development" class="glossary__link">development server</a>, +you can use +[`script/switch-theme.rb`](https://github.com/mysociety/alaveteli/blob/master/script/switch-theme.rb) +to set the current theme if you are working with multiple themes. This can be +useful for switching between the default `alavetelitheme` and your own fork. + +## Testing your theme + +You can add tests for the changes in functionality that are implemented +in your theme. These use <a href="http://rspec.info/">rspec</a>, as does the main Alaveteli test suite. +They should be put in the `spec` directory of your theme. They are run +separately from the main Alaveteli tests by executing the following command in the directory in which Alaveteli is installed (substituting your theme directory for `alavetelitheme`): + + bundle exec rspec lib/themes/alavetelitheme/spec + +You can see some example tests in the <a href="https://github.com/mysociety/whatdotheyknow-theme/tree/master/spec">whatdotheyknow-theme</a>. diff --git a/docs/customising/translation.md b/docs/customising/translation.md index bcf6514e5..bc0b52b6e 100644 --- a/docs/customising/translation.md +++ b/docs/customising/translation.md @@ -12,38 +12,85 @@ title: Translation it. This page explains how. </p> +## Alaveteli already contains translations! + +Alaveteli ships ready to run in a number of different languages. +If Alaveteli has already been translated into the language (or languages) you +need, you just need to configure it -- see +[`AVAILABLE_LOCALES`]({{ site.baseurl }}docs/customising/config/#available_locales). + +[Look in the `locale/` directory](https://github.com/mysociety/alaveteli/tree/master/locale) +to see what translations are already available. Some are complete +translations of the site. Others are only partial -- either because the translations +have not been finished yet, or else because the translators have not updated the +texts since developers changed or added text on the site. + +There are two reasons the translations may need more work before you can use them: + +* **the language you want is not one of those we already have** <br> In this + case, there will be no entry in ``locale/`` for it, because nobody has yet + translated Alaveteli into your language. See the rest of this page to learn + how to add a new translation. + +* **the translation for your language is incomplete, or lagging behind** <br> + This might simply be because it is a work in progress. Furthermore, sometimes + an incomplete translation already covers all the areas of the site you need + anyway. Of course, you can add to a partial translation, but it's a good idea + to check with us first, since we'll probably know who is working on the + current translation and what state it's in. + +Translators are members of the Alaveteli +[community]({{site.baseurl}}community/), and often work separately from the +developers. This means translations can lag a little behind the code. However, +our release process includes a "translation freeze", which gives the +translators a chance to catch up -- read the rest of this page for details. + ## Alaveteli's translations -The software translations are implemented using GNU gettext, and the resource -files are managed in Transifex. +You don't need to be a programmer to translate Alaveteli -- we use an external +website called <a href="{{ site.baseurl }}docs/glossary/#transifex" class="glossary__link">Transifex</a> to help manage translations. This makes it easy for +translators to get to work, but it does mean you (or your technical team) +need to do a little extra work to get those translations back into Alaveteli +when they are ready. The Transifex project is at -[https://www.transifex.net/projects/p/alaveteli](https://www.transifex.net/projects/p/alaveteli) -- you'll probably want an account there (ask on the mailing -list). It has a fairly easy-to-use interface for contributing translations. +[https://www.transifex.com/projects/p/alaveteli](https://www.transifex.com/projects/p/alaveteli) +-- you'll probably want an account there (ask on the mailing list). It has a +fairly easy-to-use interface for contributing translations. + +Alaveteli localises strings using GNU gettext and +<a href="{{site.baseurl}}docs/glossary/#po" class="glossary__link"><code>.pot</code> & <code>.po</code> files</a>. +If you're a developer, you should read +[internationalising Alaveteli]({{ site.baseurl }}docs/developers/i18n/). -There are three roles in the translation process, and each one is described -below: **translator**, **developer**, and **release manager**. You probably only -need to know about the one that applies to you. -## Translation process: translator's view +## What a translator needs to do **If you're just working on translating Alaveteli into a language you know, then this section is for you.** +> Remember that Alaveteli +> [already comes with some translations](#alaveteli-already-contains-translations), +> so check first that you really need to do this. Maybe someone has already +> translated Alaveteli into the language you need! + When a developer adds a new feature to the user interface in Alaveteli, they use some code to mark sentences or words ("strings") that they think will need to be translated. -When the Alaveteli release manager is planning a release, they will upload a -template containing all the strings to be translated (called a POT) to -Transifex. This causes your own translations in Transifex to be updated with +When the Alaveteli +<a href="{{site.baseurl}}docs/glossary/#release" class="glossary__link">release manager</a> +is planning a release, they will upload a +template containing all the strings to be translated (called a +<a href="{{site.baseurl}}docs/glossary/#po" class="glossary__link"><code>.pot</code> file</a>) +to Transifex. This causes your own translations in Transifex to be updated with the latest strings. When you visit Transifex, it will prompt you to fill out values for all new strings, and all strings that have been modified. In the case where a string has only been slightly modified, such as with punctuation ("Hello" has become "Hello!"), Transifex will suggest a suitable translation for you (look for the -"suggestions" tab under the source string). +*Suggestions* tab under the source string). In order for this feature to work properly, the release manager has to download your translations, run a program that inserts the suggestions, and then upload @@ -59,11 +106,12 @@ The release manager will also give you a **translation deadline**. After this date, you can continue to contribute new translations, but they won't make it into the release. + ### General notes on translation in Transifex Some strings will have comments attached to them from the Alaveteli application developers about the context in which the text appears in the -application — these comments will appear under the 'Details' tab for the text +application — these comments will appear under the *Details* tab for the text in Transifex. Some strings will have **placeholders** in them to indicate that Alaveteli @@ -104,19 +152,31 @@ they do not appear on the site anywhere at the moment, and when they do, they will only be used in the admin interface. If you do translate them, only translate the text that comes *after* the `|`. -## Translation process: developers' view -**If you're writing new code for Alaveteli, then you're a developer, and you +## How the translations get into Alaveteli + +In order to get the translated strings from Transifex into Alaveteli, follow +the instructions in these [deployment notes]({{ site.baseurl }}docs/developers/i18n/#deployment-notes). +This will be the job of the technical people on your team (or +even mySociety's release manager). If translators aren't technical, they can +use Transifex without needing to worry about this. + +## The help pages + +As the help pages for Alaveteli contain lots of text, they're translated +outside Transifex, by translating each whole help page and replacing it +in the theme that Alaveteli is using, so that it overrides the default +page. See the [guide to Alaveteli's themes]({{ site.baseurl }}docs/customising/themes/#customising-the-help-pages) for more +information on this. + +## Developers and internationalisation + +If you're writing new code for Alaveteli, then you're a developer, and you need to understand how to make any text you add easy for translators to work -with.** - -Please read our [internationalisation -guide](http://mysociety.github.io/internationalization.html) for our advice on -using strings that will need translation. This applies across all mySociety -projects, not just Alaveteli. - -The release manager will enforce a translation freeze just before a new release -is cut. During such time, you must not introduce new strings to the code if -your work is due for inclusion in this release. This is necessary to allow -translators time to complete and check their translations against all the known -strings. +with -- see the page about +[internationalising Alaveteli]({{site.baseurl}}docs/developers/i18n/). + +If you are a developer or translator actively working on internationalising +Alaveteli code, you should talk to us to find out when the next release is due, +so your translations can be prepared in time to be included in it. + diff --git a/docs/developers/api.md b/docs/developers/api.md index cbd4c7c85..1e6c15cfd 100644 --- a/docs/developers/api.md +++ b/docs/developers/api.md @@ -22,7 +22,7 @@ these ways: * Look for the RSS feed links. * Examine the `<link rel="alternate" type="application/atom+xml">` tag in the head of the HTML. -* Add `/feed` to the start of another URL. +* Add `/feed` to the start of another URL. Note that even complicated search queries have Atom feeds. You can do all sorts of things with them, such as query by authority, by file type, by date range, @@ -73,9 +73,13 @@ as follows: * as form variable `json`: * `direction` - either `request` (from the user - might be a followup, reminder, etc) or `response` (from the authority) * `body` - the message itself + * `state` - optional, allows the authority to include an updated request `state` value when sending an update. Allowable values: `waiting_response`, `rejected`, `successful` and `partially_successful`. Only used in the `response` direction * `sent_at` - ISO-8601 formatted time that the correspondence was sent * (optionally) the variable `attachments` as `multipart/form-data`: * attachments to the correspondence. Attachments can only be attached to messages in the `response` direction +* `/api/v2/request/<id>/update.json` - POST a new state for the request: + * as form variable `json`: + * `state` - the user's assessment of the `state` of a request that has received a response from the authority. Allowable values: `waiting_response`, `rejected`, `successful` and `partially_successful`. Should only be used for the user's feedback, an authority wishing to update the request `state` should use `/api/v2/request/<id>.json` instead diff --git a/docs/developers/directory_structure.md b/docs/developers/directory_structure.md index 9dbe06789..34a92b411 100644 --- a/docs/developers/directory_structure.md +++ b/docs/developers/directory_structure.md @@ -33,6 +33,30 @@ website](http://guides.rubyonrails.org/getting_started.html). <p><em>the core Alaveteli application code</em></p> <dl> <dt> + assets + </dt> + <dd> + <em>static assets that require precompilation before being served</em> + <dl> + <dt> + fonts + </dt> + <dt> + images + </dt> + <dt> + javascripts + </dt> + <dt class="last"> + stylesheets + </dt> + <dd class="last"> + <p><em>stylesheets in CSS or <a href="http://sass-lang.com/">SCSS</a> format.</em></p> + <p>SCSS stylesheets are compiled to CSS.</p> + </dd> + </dl> + </dd> + <dt> controllers </dt> <dt> @@ -44,54 +68,16 @@ website](http://guides.rubyonrails.org/getting_started.html). <dt> models </dt> - <dt> - sass - </dt> <dt class="last"> views </dt> </dl> </dd> - <dt> - assets + <dt>cache </dt> - <dd> - Static assets - <dl> - <dt> - css - </dt> - <dd> - Rendered stylesheets - </dd> - <dt> - img - </dt> - <dd> - static images - </dd> - <dt> - sass - </dt> - <dd> - Stylesheets in SCSS format, which are compiled to CSS - </dd> - <dt class="last"> - scripts - </dt> - <dd class="last"> - JavaScript - </dd> - </dl> + <dd><p><em>cached files for downloads, attachments and templates.</em></p> </dd> <dt> - bootstrap - </dt> - <dd> - <p> - Alaveteli's default style uses Bootstrap. - </p> - <dt> commonlib </dt> <dd> @@ -155,9 +141,13 @@ website](http://guides.rubyonrails.org/getting_started.html). <dt> tasks </dt> + <dd><a href="http://guides.rubyonrails.org/command_line.html#rake">Rake</a> tasks. + </dd> <dt class="last"> - whatdotheyknow + themes </dt> + <dd class="last">This is where your Alaveteli theme lives. + </dd> </dl> </dd> <dt> @@ -171,33 +161,15 @@ website](http://guides.rubyonrails.org/getting_started.html). </p> </dd> <dt> - public + log </dt> <dd> - <p><em>static assets</em></p> - <dl> - <dt> - admin - </dt> - <dd> - images, JavaScript and stylesheets used by the admin back-end - </dd> - <dt> - fcgi - </dt> - <dd> - Fast CGI files for serving static assets - </dd> - <dt> - images - </dt> - <dt> - javascripts - </dt> - <dt class="last"> - stylesheets - </dt> - </dl> + <p><em>application log files.</em></p> + </dd> + <dt> + public + </dt> + <dd> <p><em>static files that can be served directly.</em></p> </dd> <dt> script @@ -219,17 +191,6 @@ website](http://guides.rubyonrails.org/getting_started.html). </p> </dd> <dt> - stylesheets - </dt> - <dd> - <p> - <em>global stylesheet</em> - </p> - <p> - Actually just <code>global.css</code> - </p> - </dd> - <dt> tmp </dt> <dd> @@ -243,10 +204,10 @@ website](http://guides.rubyonrails.org/getting_started.html). <dd class="last"> <p><em>third-party software</em></p> <dl> - <dt class="last">plugins</dt> + <dt class="last">bundle</dt> <dd class="last"> <p> - Plugins + <em>the bundle of gems needed to run Alaveteli</em> </p> </dd> </dl> diff --git a/docs/developers/i18n.md b/docs/developers/i18n.md new file mode 100644 index 000000000..fb7f2930b --- /dev/null +++ b/docs/developers/i18n.md @@ -0,0 +1,177 @@ +--- +layout: page +title: Internationalisation (for devs) +--- + +# Internationalisation in the code + +<p class="lead"> + This page describes some technical aspects of + <a href="{{ site.baseurl }}docs/glossary/#i18n" class="glossary__link">internationalising</a> + the Alaveteli code. It's mostly aimed at devs who are working on the + codebase — if you just want to translate Alaveteli into your + own language, see + <a href="{{ site.baseurl }}docs/customising/translation">translating Alaveteli</a> + instead. +</p> + +## Deployment notes + +Deployed translations for the project live in ``locale/``. + +We encourage translations to be done on <a href="{{ site.baseurl }}docs/glossary/#transifex" class="glossary__link">Transifex</a> +because translators can work through its web interface rather than needing to edit the +<a href="{{ site.baseurl }}docs/glossary/#po" class="glossary__link">`.po` and `.pot` files</a> +directly. Ultimately, Transifex just captures translators' +work and turns it into the files that Alaveteli needs (using gettext). + +### How to get the latest translations onto your site + +For example, to deploy English and Spanish translations at once: + + * Ensure their `.po` files are at ```locale/en/app.po``` and ```locale/es/app.po``` + (for example, by downloading them from Transifex) + * Set <code><a href="{{ site.baseurl }}docs/customising/config/#available_locales">AVAILABLE_LOCALES</a></code> + to <code>en es</code> + +### What to do if you don't have complete translations for an older release of Alaveteli + +Before a new release of Alaveteli is made, the translation files are +pulled from Transifex and added to Alaveteli's ``locale/`` directory in +github. These represent the most complete translations for the previous +release. Then the files in Transifex are updated with any new strings +that need to be translated for the upcoming release. At this point old +strings that are no longer used in the new release are also removed. The +latest [release tag](https://github.com/mysociety/alaveteli/releases) +for a release in github should contain the most complete translations +for that release from Transifex. + +If you're using an older release of Alaveteli and you want to add to or +change the translations, you can edit the .po files directly using a +local program such as [PoEdit](http://poedit.net/). + +### How to add new strings to the translations + +You need to do this if you've added any new strings to the code that need +translations (or if you change an existing one). + +To update the +<a href="{{ site.baseurl }}docs/glossary/#po" class="glossary__link">`.po` or `.pot` files</a> +for each language, run: + + bundle exec rake gettext:store_model_attributes + +followed by: + + bundle exec rake gettext:find + +If `gettext:find` only creates the file `locale/im-config.pot` then you need to +unset the `TEXTDOMAIN` environment variable and try again. + +For more details about the translations, see the page about +[translating Alaveteli]({{ site.baseurl }}docs/customising/translation/). + + +## Technical implementation details + +### Getting the current locale + +This is complicated by the fact that there are two competing ways to define a +locale+territory combination. The POSIX (and gettext and Transifex) way is +like `en_GB`; the Rails way is like `en-US`. Because we are using gettext and +Transifex for translations, we must deal with both. + + * for the Rails version of the currently selected locale, use `I18n.locale` + * for the POSIX version of the locale, use `FastGettext.locale` + +## I18n in templates + +Before you add i18n strings to the source, you should read +[internationalisation guidelines](http://mysociety.github.io/internationalization.html) +that apply to all our projects. + +Some hints for adding the strings into the Alaveteli code: + +* Simple strings: ```<% = _("String to translate") %>``` +* Strings that include variables: give the translator a hand by inserting + strings that can be interpolated, so the variable has meaning. For example, + ```<%= "Nothing found for '" + h(@query) + "'" %>``` might become ```<%= + _("Nothing found for '{{search_terms}}'", :search_terms => h(@query)) %>``` +* Strings containing numbers: ```<%= n_('%d request', '%d requests', @quantity) % @quantity %>``` +* We allow some inline HTML where it helps with meaningful context, for example: + +``` +_('<a href="{{browse_url}}">Browse all</a> or <a href="{{add_url}}">ask us to add it</a>.', + :browse_url => @browse_url, :add_url => @add_url) +``` + +Similar rules can apply to strings in the Ruby source code. + +## Programmatic access of translated PublicBodies + +Apart from the templates, the only other area of i18n currently implemented is +in the PublicBodies. + +The implementation allows for getting different locales of a PublicBody like so: + +```ruby + PublicBody.with_locale("es") do + puts PublicBody.find(230).name + end +``` + +Usually, that's all the code you need to know about. There's a method +```self.locale_from_params()``` available on all models which returns a locale +specified as ```locale=xx``` in the query string, and which falls back to the +default locale, that you can use in conjunction with the ```with_locale``` +method above. All the joining on internal translation tables should usually be +handled automagically -- but there are some exceptions, that follow below. + +### Overriding model field setters + +Internally, we use the [Globalize plugin](https://github.com/globalize/globalize) +to localize model fields. Where column "foo" has been marked in the model as +```:translates```, globalize overrides ```foo.baz = 12``` to actually set the +value in column ```baz``` of table ```foo_translations```. + +A side effect of the way it does this is that if you wish to override a +specific attribute setter, you will need to explicitly call the Globalize +machinery; something like: + +```ruby + def name=(name) + globalize.write(self.class.locale || I18n.locale, "name", name) + self["name"] = short_name + # your other stuff here + end +``` + +### Searching + +The ```find_first_by_<attr>``` and ```find_all_by_<attr>``` magic methods +should work. If you want to do a more programmatic search, you will need to +join on the translation table. For example: + +```ruby + query = "#{translated_attr_name(someattr) = ? AND #{translated_attr_name('locale')} IN (?)" + locales = Globalize.fallbacks(locale || I18n.locale).map(&:to_s) + find( + :first, + :joins => :translations, + :conditions => [query, value, locales], + :readonly => false + ) +``` + +You may also need to do some lower-level SQL joins or conditions. See +```PublicBodyController.list``` for an example of a query that has a condition +that is explicitly locale-aware (look for the ```locale_condition``` variable) + +## Translation and releases + +The release manager will enforce a translation freeze just before a new release +is cut. During such time, you must not introduce new strings to the code if +your work is due for inclusion in this release. This is necessary to allow +translators time to complete and check their translations against all the known +strings. See more about [translating Alaveteli]({{ site.baseurl }}docs/customising/translation/). + diff --git a/docs/developers/index.md b/docs/developers/index.md index 9d1869246..f1167a22b 100644 --- a/docs/developers/index.md +++ b/docs/developers/index.md @@ -45,7 +45,7 @@ title: For developers [manual installation]({{ site.baseurl }}docs/installing/manual_install/). Alternatively, there's an [Alaveteli EC2 AMI]({{ site.baseurl }}docs/installing/ami/) that might help you get up and running quickly. - [Get in touch](http://www.alaveteli.org/contact/) on the project mailing list or IRC + [Get in touch]({{ site.baseurl }}community/) on the project mailing list or IRC for help. * A standard initial step for customising your deployment is [writing a @@ -53,10 +53,10 @@ title: For developers it should be this!** * Like many Ruby on Rails sites, the software is not hugely performant (see - some notes about [[performance issues]] gathered over time with + [these notes about performance issues](https://github.com/mysociety/alaveteli/wiki/Performance-issues) gathered over time with WhatDoTheyKnow). The site will run on a server with 512MB RAM but at least 2GB is recommended. Deployment behind [Varnish](https://www.varnish-cache.org) is also fairly essential. See - [[Production Server Best Practices]] for more. + [production server best practices]({{site.baseurl}}docs/running/server/) for more. * There's a number of [proposals for enhancements](https://github.com/mysociety/alaveteli/wiki/Proposals-for-enhancements), such as more user-focused features, but see also... diff --git a/docs/getting_started.md b/docs/getting_started.md index 88e5723f7..3ea67c818 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -1,6 +1,7 @@ --- layout: page title: Getting started +redirect_from: /getting_started/ --- # Getting started with Alaveteli @@ -28,7 +29,7 @@ available time. You can get a feeling for how things might turn out by reading [how an Alaveteli was set up in -Spain]({{ site.baseurl }}2012/04/16/a-right-to-know-site-for-spain/) +Spain](https://www.mysociety.org/2012/04/16/a-right-to-know-site-for-spain/) (remember that this was with an experienced developer in charge). You will also need to think about how you will run the website; a successful Alaveteli requires lots of ongoing effort to moderate and publicise (see Step 6 and Step @@ -107,12 +108,13 @@ You'll need to find a tech person who knows about hosting websites using Apache and Linux. They don't need to know Ruby on Rails, but it would be a huge advantage if they do. -You'll also need to source a server. You should ask your tech person to help -with this. The minimum spec for running a low traffic website is 512MB RAM and -a 20GB disk. 2GB RAM would be ideal. We recommend Debian Squeeze 64-bit as the -operating system, though any sort of Linux should do. Rackspace offer suitable -cloud servers, which start out at around $25 / month. Then your tech person -should follow the [installation documentation]({{ site.baseurl }}docs/installing/). +You'll also need to source a server. You should ask your tech person to +help with this. The minimum spec for running a low traffic website is +512MB RAM and a 20GB disk. 2GB RAM would be ideal. We recommend the +latest Debian Wheezy (7) or Squeeze (6) 64-bit or Ubuntu precise (12.04) +as the operating system. Rackspace offer suitable cloud servers, which +start out at around $25 / month. Then your tech person should follow the +[installation documentation]({{ site.baseurl }}docs/installing/). Alternatively, you could use Amazon Web Services. This has the added advantage that you can use our preconfigured [Alaveteli EC2 @@ -139,7 +141,7 @@ can see what requests look like to them. When trying things out, you need to wear several hats -- as a site administrator, an ordinary site user, and as a public authority. This can get confusing with several email addresses, so one quick and easy way to manage -this is to use a throwaway email service like http://mailinator.com. +this is to use a throwaway email service like [Mailinator](http://mailinator.com). <a name="step-2"> </a> @@ -156,7 +158,7 @@ If you email possible supporters asking for help, in addition to helping make your job easier, it will also help you identify eager people who might be interested in helping you maintain and run the website. We have written [a blog post about -this]({{ site.baseurl }}2011/07/29/you-need-volunteers-to-make-your-website-work/). +this](https://www.mysociety.org/2011/07/29/you-need-volunteers-to-make-your-website-work/). The admin interface includes a page where you can upload a CSV file (that's a file containing comma-separated values) to create or edit authorities. CSV is a @@ -214,17 +216,8 @@ often less. But complicated workflows might take a bit longer. The default help pages in Alaveteli are taken from WhatDoTheyKnow, and are therefore relevant only to the UK. You should take these pages as inspiration, -but review their content with a view to your jurisdiction. The important pages -to translate are: - -* [About](https://github.com/mysociety/alaveteli/blob/master/app/views/help/about.rhtml): why the website exists, why it works, etc -* [contact](https://github.com/mysociety/alaveteli/blob/master/app/views/help/contact.rhtml): how to get in touch -* [credits](https://github.com/mysociety/alaveteli/blob/master/app/views/help/credits.rhtml): who is involved in the site. Importantly, includes a section on how users can help the project. -* [officers](https://github.com/mysociety/alaveteli/blob/master/app/views/help/officers.rhtml): information for the officers who deal with FOI at authorities. They get a link to this page in emails that the site sends them. -* [privacy](https://github.com/mysociety/alaveteli/blob/master/app/views/help/privacy.rhtml): privacy policy, plus information making it clear that requests are going to appear on the internet. Let users know if they are allowed to use pseudonyms in your jurisdiction. -* [requesting](https://github.com/mysociety/alaveteli/blob/master/app/views/help/requesting.rhtml): the main help page about making requests. How it works, how to decide who to write to, what they can expect in terms of responses, how to make appeals, etc. -* [unhappy](https://github.com/mysociety/alaveteli/blob/master/app/views/help/unhappy.rhtml): users are taken to this page after a request that has been somehow unsuccessful (e.g. the request has been refused, or the authority is insisting on a postal request). The page should encourage them to keep going, e.g. by starting a new request or addressing it to a different body. -* [why email](https://github.com/mysociety/alaveteli/blob/master/app/views/help/_why_they_should_reply_by_email.rhtml): a snippet of information that explains why users should insist on replies by email. This is displayed next to requests that have "gone postal". +but review their content with a view to your jurisdiction. See [the documentation on Alaveteli's themes]({{ site.baseurl }}docs/customising/themes/#customising-the-help-pages) for details +on which pages are important, and what content they need to have. The help pages contain some HTML. Your tech person should be able to advise on this. @@ -283,7 +276,7 @@ spreadsheet. The help pages need to have one copy saved for each language; your tech person will put them in the right place. The web interface translations are managed and collaborated via a website -called Transifex. This website allows teams of translators to collaborate in +called <a href="{{ site.baseurl }}docs/glossary/#transifex" class="glossary__link">Transifex</a>. This website allows teams of translators to collaborate in one place, using a fairly easy interface. The Alaveteli page on Transifex is at @@ -345,7 +338,7 @@ This will be easier to do with a small team of people sharing jobs. Hopefully you have been lucky enough to get funding to pay people to do these tasks. However, you are also likely to have to rely on volunteers. We've written [a blog post about the importance of -volunteers]({{ site.baseurl }}2011/07/29/you-need-volunteers-to-make-your-website-work/), which you should read. +volunteers](https://www.mysociety.org/2011/07/29/you-need-volunteers-to-make-your-website-work/), which you should read. You'll need to set up a group email address for all the people who will manage the website. All site user queries will go here, as will automatic diff --git a/docs/glossary.md b/docs/glossary.md index 64c75c603..d0e03b41d 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -15,36 +15,112 @@ Definitions ----------- <ul class="definitions"> + <li><a href="#admin">admin interface</a></li> + <li><a href="#advanced-search">advanced search</a></li> <li><a href="#alaveteli">Alaveteli</a></li> <li><a href="#agnostic">asker agnostic</a></li> <li><a href="#authority">authority</a></li> <li><a href="#blackhole">black hole</a></li> + <li><a href="#bounce-message">bounce message</a></li> <li><a href="#capistrano">Capistrano</a></li> + <li><a href="#category">category</a></li> + <li><a href="#categorisation-game">categorisation game</a></li> <li><a href="#censor-rule">censor rule</a></li> <li><a href="#development">development site</a></li> + <li><a href="#disclosure-log">disclosure log</a></li> + <li><a href="#emergency">emergency user</a></li> <li><a href="#foi">freedom of information</a></li> <li><a href="#git">git</a></li> <li><a href="#holding_pen">holding pen</a></li> + <li><a href="#holiday">holiday</a></li> + <li><a href="#i18n">internationalisation</a></li> <li><a href="#newrelic">New Relic</a></li> - <li><a href="#mta">MTA</a></li> + <li><a href="#mta">Mail Transfer Agent</a></li> + <li><a href="#po">.po files</a></li> <li><a href="#production">production site</a></li> <li><a href="#publish">publish</a></li> + <li><a href="#publication-scheme">publication scheme</a></li> <li><a href="#recaptcha">recaptcha</a></li> <li><a href="#redact">redacting</a></li> <li><a href="#regexp">regular expression</a></li> <li><a href="#request">request</a></li> + <li><a href="#release">release</a></li> <li><a href="#response">response</a></li> <li><a href="#rails">Ruby on Rails</a></li> <li><a href="#sass">Sass</a></li> + <li><a href="#spam-address-list">spam address list</a></li> <li><a href="#staging">staging site</a></li> <li><a href="#state">state</a></li> + <li><a href="#super">superuser</a></li> + <li><a href="#tag">tag</a></li> <li><a href="#theme">theme</a></li> + <li><a href="#transifex">Transifex</a></li> + <li><a href="#wdtk">WhatDoTheyKnow</a></li> </ul> <dl class="glossary"> <dt> + <a name="admin">admin interface</a> (also: admin) + </dt> + <dd> + The <strong>admin interface</strong> allows users who have + <a href="{{ site.baseurl }}docs/glossary/#super" class="glossary__link">super</a> + administrator privilege to manage some aspects of how your + Alaveteli site runs. + <div class="more-info"> + <p>More information:</p> + <ul> + <li> + You can access your installation's <a href="{{ site.baseurl }}docs/glossary/#admin" class="glossary__link">admin interface</a> + at <code>/admin</code>. + </li> + <li> + To grant a user admin privilege, log into the admin and change + their <em>Admin level</em> to "super" (or revoke the privilege + by changing it to "none"). + </li> + <li> + On a newly-installed Alaveteli system, you can grant yourself + admin privilege by using the + <a href="{{ site.baseurl }}docs/glossary/#emergency" class="glossary__link">emergency + user</a>. + </li> + <li> + For lots more about running an Alaveteli site, see the + <a href="{{ site.baseurl }}running/admin_manual">adminstrator's guide</a>. + </li> + </ul> + </div> + </dd> + + <dt> + <a name="advanced-search">advanced search</a> + </dt> + <dd> + Alaveteli's <strong>advanced search</strong> lets users search using + more complex criteria than just words. This includes Boolean operators, + date ranges, and specific indexes such as <code>status:</code>, + <code>requested_by:</code>, <code>status:</code> and so on. + <div class="more-info"> + <p>More information:</p> + <ul> + <li> + Advanced search is available on your Alaveteli site at + <code>/advancedsearch</code>. That page shows suggestions and examples + of the searches that are supported. + </li> + <li> + For more about constructing complex queries, see + <a href="http://xapian.org/docs/queryparser.html">Xapian + search parser</a>. + </li> + </ul> + </div> + </dd> + + <dt> <a name="alaveteli">Alaveteli</a> </dt> <dd> @@ -53,7 +129,7 @@ Definitions managing and archiving Freedom of Information requests. <p> It grew from the successful FOI UK project - <a href="https://www.whatdotheyknow.com">WhatDoTheyKnow</a>. + <a href="#wdtk" class="glossary__link">WhatDoTheyKnow</a>. We use the name <em>Alaveteli</em> to distinguish the software that runs the platform from any specific website that it is powering. </p> @@ -78,12 +154,12 @@ Definitions <a name="agnostic">asker agnostic</a> </dt> <dd> - <a href="#foi" class="glossary">Freedom of Information</a> (FoI) law typically considers - the <a href="#response" class="glossary">responses</a> given by the - <a href="#authority" class="glossary">authorities</a> to be <strong>asker agnostic</strong>. This means + <a href="#foi" class="glossary__link">Freedom of Information</a> (FoI) law typically considers + the <a href="#response" class="glossary__link">responses</a> given by the + <a href="#authority" class="glossary__link">authorities</a> to be <strong>asker agnostic</strong>. This means that the reply should not be any different depending on <em>who</em> asked for the information. One consequence of this is that the response - can be <a href="#publish" class="glossary">published</a>, because in theory <em>everyone</em> + can be <a href="#publish" class="glossary__link">published</a>, because in theory <em>everyone</em> could ask for it and expect, by law, to receive the same information. <p> Despite this, it's still very common all around the world for authorities to reply @@ -99,21 +175,26 @@ Definitions </dt> <dd> An <strong>authority</strong> is the term we use for any of the bodies, organisations, - departments, or companies to which users can send <a href="#request" class="glossary">requests</a>. + departments, or companies to which users can send <a href="#request" class="glossary__link">requests</a>. <div class="more-info"> <p>More information:</p> <ul> <li> - An administrator can add, edit, or remove authorities in the admin + An <a href="#super" class="glossary__link">administrator</a> + can add, edit, or remove authorities in the admin. </li> <li> Authorities are usually, but not always, public bodies that are obliged by the local - <a href="#foi" class="glossary">Freedom of Information</a> (FoI) law to respond. Sometimes an + <a href="#foi" class="glossary__link">Freedom of Information</a> (FoI) law to respond. Sometimes an Alaveteli site is set up in a jurisdiction that does not yet have FoI law. In the UK, we add some authorites to our <a href="https://www.whatdotheyknow.com">WhaDoTheyKnow</a> site that are not subject to FoI law, but which have either voluntarily submitted themselves to it, or which we believe should be accountable in this way. </li> + <li> + You can organise your authorities using + <a href="{{ site.baseurl }}docs/running/categories_and_tags/">categories and tags</a>. + </li> </ul> </div> </dd> @@ -145,6 +226,24 @@ Definitions </dd> <dt> + <a name="bounce-message">bounce message</a> + </dt> + <dd> + A <strong>bounce message</strong> is an automated electronic mail message from a mail system informing the sender of another message about a delivery problem. + </p> + <div class="more-info"> + <p>More information:</p> + <ul> + <li> + <a href="{{ site.baseurl }}docs/installing/email#how-alaveteli-handles-email">How Alaveteli handles email</a> + </li> + <li>The wikipedia page on <a href="http://en.wikipedia.org/wiki/Bounce_message">bounce messages</a> + </li> + </ul> + </div> + </dd> + + <dt> <a name="capistrano">Capistrano</a> </dt> <dd> @@ -166,12 +265,47 @@ Definitions </dd> <dt> + <a name="category">category</a> + </dt> + <dd> + You can arrange your <a href="#authority" class="glossary__link">authorities</a> + into <strong>categories</strong> so that they are easier for your users + to find. For example, you might put all different schools into the + "School" category, and universities into "Universities". You can also + group categories under headings (such as "Education"). + <p> + These categories and headings appear on the list of public authorities that + is displayed on your site. + </p> + <p> + Use <a href="#tag" class="glossary__link">tags</a> to associate + authorities with specific categories. + </p> + More about + <a href="{{ site.baseurl }}docs/running/categories_and_tags/">categories and tags</a> +</dd> + <dt> + <a name="categorisation-game">categorisation game</a> + </dt> + <dd> + The categorisation game is a way that users of an Alaveteli site can help the site stay current and accurate by updating the status of old requests where the original requester has never said whether the authority responded with the information or not. + <div class="more-info"> + <p>More information:</p> + <ul> + <li> + the categorisation game on the <a href="http://demo.alaveteli.org/categorise/play">demo Alaveteli site</a>. + </li> + </ul> + </div> + </dd> + + <dt> <a name="censor-rule">censor rule</a> </dt> <dd> Alaveteli administrators can define <strong>censor rules</strong> to define which parts of replies or responses should be - <a href="#redact" class="glossary">redacted</a>. + <a href="#redact" class="glossary__link">redacted</a>. </p> <div class="more-info"> <p>More information:</p> @@ -198,9 +332,9 @@ Definitions so you can <a href="{{ site.baseurl }}docs/customising/">customise it</a>, experiment with different settings, and test that it does what you expect. This is different from a - <a href="#production" class="glossary">production server</a>, which is the one your + <a href="#production" class="glossary__link">production server</a>, which is the one your users actually visit running with live data, or a - <a href="#staging" class="glossary">staging server</a>, + <a href="#staging" class="glossary__link">staging server</a>, which is used for testing code before it goes live. <p> On your dev server, you should set @@ -210,7 +344,61 @@ Definitions </dd> <dt> - <a name="foi">Freedom of Information</a> (also FOI) + <a name="disclosure-log">disclosure log</a> + </dt> + <dd> + Some <a href="#authority" class="glossary__link">authorities</a> routinely + publish their responses to <a href="#foi" class="glossary__link">Freedom of + Information</a> requests online. This collection of responses is called a + <strong>disclosure log</strong>, and if an authority has such a log on its + website, you can add the URL so Alaveteli can link to it. + <div class="more-info"> + <p>More information:</p> + <ul> + <li> + You can add a disclosure log URL by + <a href="{{ site.baseurl }}docs/running/admin_manual/#creating-changing-and-uploading-public-authority-data">updating authority data</a> in the admin. + </li> + </ul> + </div> + </dd> + + <dt> + <a name="emergency">emergency user</a> + </dt> + <dd> + Alaveteli ships with a configuration setting for an <strong>emergency user</strong>. + This provides a username and password you can use to access the admin, even though + the user doesn't appear in the database. + <p> + When the system has been bootstrapped (that is, you've used the emergency user to + grant a user account full <em>super</em> privileges), you must disable the emergency + user. + </p> + <div class="more-info"> + <p>More information:</p> + <ul> + <li> + The username and password are defined by the configuration settings + <code><a href="{{site.baseurl}}docs/customising/config/#admin_username">ADMIN_USERNAME</a></code> + and + <code><a href="{{site.baseurl}}docs/customising/config/#admin_password">ADMIN_PASSWORD</a></code>. + </li> + <li> + For an example of using the emergency user, see + <a href="{{site.baseurl}}docs/installing/next_steps/#create-a-superuser-account-for-yourself">creating + a superuser account</a>. + </li> + <li> + Disable the emergency user by setting + <code><a href="{{site.baseurl}}docs/customising/config/#disable_emergency_user">DISABLE_EMERGENCY_USER:</a> true</code> + </li> + </ul> + </div> + </dd> + + <dt> + <a name="foi">Freedom of Information</a> (also: FOI) </dt> <dd> <strong>Freedom of information</strong> laws allow access by the general public @@ -231,7 +419,7 @@ Definitions </dd> <dt> - <a name="git">git</a> (also github, git repository, and git repo) + <a name="git">git</a> (also: github, git repository, and git repo) </dt> <dd> We use a popular source code control system called <strong>git</strong>. This @@ -272,26 +460,104 @@ Definitions <a name="holding_pen">holding pen</a> </dt> <dd> - The <strong>holding pen</strong> is the conceptual place where responses that - could not be delivered are held. They need attention from a administrator. + The <strong>holding pen</strong> is the conceptual place where responses + that could not be delivered are held. They need attention from an + <a href="#super" class="glossary__link">administrator</a>. + <p> + In fact, the holding pen is really a special "sticky" <a href="#request" + class="glossary__link">request</a> that only exists to accept unmatched + responses. Whenever Alaveteli receives an email but can't work out which + request it is a response to, it attaches it to the holding pen instead. + </p> + <div class="more-info"> + <p>More information:</p> + <ul> + <li> + See more <a href="{{ site.baseurl }}docs/running/holding_pen">about + the holding pen</a>, including why messages end up there, and + instructions on what to do with them. + </li> + <li> + The most common reason for a response to be in the holding pen is that + an <a href="#authority" class="glossary__link">authority</a> replied + to a request with the wrong email address (for example, by copying + the email address incorrectly). + </li> + </ul> + </div> + </dd> + + <dt> + <a name="holiday">holidays</a> + </dt> + <dd> + Alaveteli needs to know about <strong>public holidays</strong> because + they affect the calculation that determines when a + <a href="#response" class="glossary__link">response</a> is overdue. + Public holidays are different all around the world, so Alaveteli lets + you specify the dates for the jurisdiction relevant to your + site in the <a href="#admin" class="glossary__link">admin interface.</a> + <div class="more-info"> + <p>More information:</p> + <ul> + <li> + See more about + <a href="{{ site.baseurl }}docs/installing/next_steps/#add-some-public-holidays">adding + public holidays</a>. It's possible to load dates from an iCalendar + feed or accept Alaveteli's suggestions. + </li> + </ul> + </div> + </dd> + + <dt> + <a name="i18n">internationalisation</a> (also: i18n) + </dt> + <dd> + <strong>Internationalisation</strong> is the way Alaveteli adapts the + way it presents text based on the language or languages that your website + supports. It's sometimes abbreviated as <em>i18n</em> (because there are + 18 letters between i and n). + <p> + Often you don't need to worry about the details of how this is done + because once you've configured your site's + <code><a href="{{ site.baseurl }}docs/customising/config/#default_locale">DEFAULT_LOCALE</a></code> + Alaveteli takes care of it for you. + But when you do need to work on i18n (for example, if you're customising + your site by + <a href="{{ site.baseurl }}docs/customising/translation/">translating</a> it, or + <a href="{{ site.baseurl }}docs/running/admin_manual/#creating-changing-and-uploading-public-authority-data">uploading names</a> + of the public bodies in more than one language) at the very least you may + need to know the language codes you're site is using. + </p> <div class="more-info"> <p>More information:</p> <ul> <li> - see the <a href="{{ site.baseurl }}docs/running/admin_manual/">admin manual</a> for - information on dealing with emails in the holding pen + More about <a href="{{ site.baseurl }}docs/developers/i18n/">internationalising Alaveteli</a> + </li> + <li> + See mySociety's + <a href="http://mysociety.github.io/internationalization.html">i18n guidelines</a> for developers + </li> + <li> + <a href="http://en.wikipedia.org/wiki/List_of_ISO_639-1_codes">List of language codes</a> + </li> + <li> + For more about i18n in software generally, see + the <a href="http://en.wikipedia.org/wiki/Internationalization_and_localization">i18n Wikipedia article</a>. </li> </ul> </div> </dd> <dt> - <a name="mta">MTA</a> (Mail Transfer Agent) + <a name="mta">Mail Transfer Agent</a> (MTA) </dt> <dd> - A <strong>Mail Tranfer Agent</strong> is the the program which actually sends + A <strong>Mail Transfer Agent</strong> is the the program which actually sends and receives email. Alaveteli sends email on behalf of its users, and processes - the <a href="#response" class="glossary">responses</a> and replies it receives. + the <a href="#response" class="glossary__link">responses</a> and replies it receives. All this email goes through the MTA, which is a seperate service on your system. <div class="more-info"> <p>More information:</p> @@ -309,7 +575,7 @@ Definitions </dt> <dd> Alaveteli can use <strong>New Relic</strong>'s application monitoring tool to track the - performance of your <a href="#production" class="glossary">production site</a>. If enabled, + performance of your <a href="#production" class="glossary__link">production site</a>. If enabled, data from your application is gathered on the New Relic website, which you can inspect with their visual tools. Basic use is free. <div class="more-info"> @@ -333,19 +599,55 @@ Definitions </dd> <dt> + <a name="po"><code>.po</code> file</a> (and <code>.pot</code> file) + </dt> + <dd> + These are the files needed by the <code>gettext</code> mechanism Alaveteli + uses for localisation. A <code>.pot</code> file is effectively a list of + all the strings in the application that need translating. Each + <code>.po</code> file contains the mapping between those strings, used as + keys, and their translations for one particular language. The key is called + the <em>msgid</em>, and its corresponding translation is the + <em>msgstr</em>. + <div class="more-info"> + <p>More information:</p> + <ul> + <li> + See <a href="{{ site.baseurl }}docs/customising/translation/">translating + Alaveteli</a> for an overview from a translator's point of view. + </li> + <li> + See <a href="{{ site.baseurl }}docs/developers/i18n/">Internationalising + Alaveteli</a> for more technical details. + </li> + <li> + Alaveteli is on the <a href="https://www.transifex.com/projects/p/alaveteli/">Transifex</a> + website, which lets translators work on Alaveteli in a browser, without needing + to worry about this underlying structure. + </li> + <li> + See more about the + <a href="https://www.gnu.org/software/gettext/"><code>gettext</code> + system</a>. + </li> + </ul> + </div> + </dd> + + <dt> <a name="production">production site</a> (also: live, production server) </dt> <dd> A <strong>production server</strong> is one that is running your Alaveteli site for real users, with live data. This is different from a - <a href="#development" class="glossary">development server</a>, which you use make your + <a href="#development" class="glossary__link">development server</a>, which you use make your customisation and environment changes and try to get them to all work OK, or a - <a href="#staging" class="glossary">staging server</a>, which is used for testing code + <a href="#staging" class="glossary__link">staging server</a>, which is used for testing code and configuration after it's been finished but before it goes live. <p> Your production site should be configured to run as efficiently as possible: for example, caching is enabled, and debugging switched off. - <a href="#rails" class="glossary">Rails</a> has a "production mode" which does + <a href="#rails" class="glossary__link">Rails</a> has a "production mode" which does this for you: set <code><a href="{{site.baseurl}}docs/customising/config/#staging_site">STAGING_SITE</a></code> to <code>0</code>. Note that if you <em>change</em> this setting after you've @@ -368,9 +670,9 @@ Definitions </dt> <dd> Alaveteli works by <strong>publishing</strong> the - <a href="#response" class="glossary">responses</a> it recieves to the - <a href="#foi" class="glossary">Freedom of Information</a> - <a href="#request" class="glossary">requests</a> that its users send. + <a href="#response" class="glossary__link">responses</a> it recieves to the + <a href="#foi" class="glossary__link">Freedom of Information</a> + <a href="#request" class="glossary__link">requests</a> that its users send. It does this by processing the emails it receives and presenting them as pages — one per request — on the website. This makes it easy for people to find, read, link to, and share the request and the @@ -378,14 +680,28 @@ Definitions </dd> <dt> - <a name="response">response</a> + <a name="publication-scheme">publication scheme</a> </dt> <dd> - A <strong>response</strong> is the email sent by an - <a href="#authority" class="glossary">authority</a> in reply to - a user's <a href="#request" class="glossary">requests</a>. + Some <a href="#authority" class="glossary__link">authorities</a> have a + <strong>publication scheme</strong> which makes it clear what information + is readily available from them under <a href="#foi" + class="glossary__link">Freedom of Information</a> law, and how people can + get it. This may be a requirement for their compliance with the law, or it + may simply be good practice. If an authority has published such a scheme on + its website, you can add the URL so Alaveteli can link to it. + <div class="more-info"> + <p>More information:</p> + <ul> + <li> + You can add a publication scheme URL by + <a href="{{ site.baseurl }}docs/running/admin_manual/#creating-changing-and-uploading-public-authority-data">updating authority data</a> in the admin. + </li> + </ul> + </div> </dd> + <dt> <a name="recaptcha">recaptcha</a> </dt> @@ -436,7 +752,7 @@ Definitions </li> <li> you can do text-only redaction with Alaveteli's - <a href="#censor-rule" class="glossary">censor rules</a> + <a href="#censor-rule" class="glossary__link">censor rules</a> </li> <li> some things are easier to redact than others — especially in PDFs, @@ -454,8 +770,8 @@ Definitions A <strong>regular expression</strong> is a concise way to describe a pattern or sequence of characters, letters or words. As an administrator, you may find regular expressions useful if you need to define <a - href="#censor-rule" class="glossary">censor rules</a>. For example, instead - of <a href="#redact" class="glossary">redacting</a> just one specific + href="#censor-rule" class="glossary__link">censor rules</a>. For example, instead + of <a href="#redact" class="glossary__link">redacting</a> just one specific phrase, you can describe a whole range of <em>similar</em> phrases with one single regular expression. <p> @@ -482,20 +798,65 @@ Definitions </dd> <dt> + <a name="release">release</a> (also: release manager) + </dt> + <dd> + We issue new <strong>releases</strong> of the Alaveteli code whenever key + work (new features, improvements, bugfixes, and so on) have been added to + the core code. Releases are identified by their tag, which comprises two or + three numbers: major, minor, and — if necessary — a patch + number. We recommend you always use the latest version. The process is + handled by the Alaveteli <strong>release manager</strong>, who decides what + changes are to be included in the current release, and the cut-off date for + the work. Currently this is Alaveteli's lead developer at mySociety. + <div class="more-info"> + <p>More information:</p> + <ul> + <li> + The latest stable release is on the + <a href="https://github.com/mysociety/alaveteli/tree/master">master branch</a>. + </li> + <li> + See a <a href="https://github.com/mysociety/alaveteli/releases">list of all releases</a> + and their explicit tags. + </li> + <li> + We try to coordinate releases with any active translation work too. + See <a href="{{ site.baseurl }}docs/customising/translation/">translating + Alaveteli</a> for more information. + </li> + <li> + We encourage you use the <a href="{{site.baseurl}}docs/installing/deploy/">deployment + mechanism</a>, which makes it easier to keep your production server up-to-date. + </li> + </ul> + </div> + </dd> + + <dt> <a name="request">request</a> </dt> <dd> In Alaveteli, a <strong>request</strong> is the - <a href="#foi" class="glossary">Freedom of Information</a> request + <a href="#foi" class="glossary__link">Freedom of Information</a> request that a user enters, and which the site then emails to the relevant - <a href="#authority" class="glossary">authority</a>. - Alaveteli automatically <a href="#publish" class="glossary">publishes</a> - the <a href="#response" class="glossary">responses</a> + <a href="#authority" class="glossary__link">authority</a>. + Alaveteli automatically <a href="#publish" class="glossary__link">publishes</a> + the <a href="#response" class="glossary__link">responses</a> to all the requests it sends. </dd> <dt> - <a name="rails">Ruby on Rails</a> (also Rails) + <a name="response">response</a> + </dt> + <dd> + A <strong>response</strong> is the email sent by an + <a href="#authority" class="glossary__link">authority</a> in reply to + a user's <a href="#request" class="glossary__link">requests</a>. + </dd> + + <dt> + <a name="rails">Ruby on Rails</a> (also: Rails) </dt> <dd> Alaveteli is written in the Ruby programming language, using @@ -523,7 +884,7 @@ Definitions language, and we use it because it's easier to manage than writing CSS directly (for example, Sass lets you easily make a single change that will be applied to many elements across the whole site). - <a href="#rails" class="glossary">Rails</a> notices if you change any of + <a href="#rails" class="glossary__link">Rails</a> notices if you change any of the Sass files, and automatically re-generates the CSS files that the website uses. <div class="more-info"> @@ -541,14 +902,41 @@ Definitions </dd> <dt> + <a name="spam-address-list">spam address list</a> + </dt> + <dd> + Alaveteli maintains a <strong>spam address list</strong>. Any incoming message to an email + address on that list will be rejected and won't appear in the admin. + <p> + This is mainly for email addresses whose messages are ending up + in the <a href="#holding_pen" class="glossary__link">holding pen</a>, because + those are typically addresses that can be safely ignored as they do not + relate to an active <a href="#request" class="glossary__link">request</a>. + </p> + <div class="more-info"> + <p>More information:</p> + <ul> + <li> + To add addresses to the spam address list , see + <a href="{{ site.baseurl }}docs/running/admin_manual/#rejecting-spam-that-arrives-in-the-holding-pen">Rejecting + spam that arrives in the holding pen</a>. + </li> + <li> + The spam address list is available on your site at <code>/admin/spam_addresses</code>. + </li> + </ul> + </div> + </dd> + + <dt> <a name="staging">staging server</a> (also: staging site) </dt> <dd> A <strong>staging server</strong> is one that you use for testing code or configuration before it goes live. This is different from a <a href="#development" - class="glossary">development server</a>, on which you change the code and settings to + class="glossary__link">development server</a>, on which you change the code and settings to make everything work, or the - <a href="#production" class="glossary">production server</a>, which is the + <a href="#production" class="glossary__link">production server</a>, which is the site your users visit running with live data. <p> On your staging server, you should set @@ -571,33 +959,89 @@ Definitions <a name="state">state</a> </dt> <dd> - Each <a href="#request" class="glossary">request</a> passes through different + Each <a href="#request" class="glossary__link">request</a> passes through different <strong>states</strong> as it progresses through the system. States help Alaveteli administrators, as well as the public, understand the current situation with any request and what action, if any, is required. <p> The states available can be customised within - your site's <a href="#theme" class="glossary">theme</a>. + your site's <a href="#theme" class="glossary__link">theme</a>. </p> <div class="more-info"> <p>More information:</p> <ul> <li> - <a href="{{ site.baseurl }}docs/running/states/">example states for WhatDoTheyKnow</a> + <a href="{{ site.baseurl }}docs/customising/states/">example states for WhatDoTheyKnow</a> (Alaveteli site running in the UK) </li> <li> - for comparison, <a href="{{ site.baseurl }}docs/running/states_informatazyrtare/">example states for InformataZyrtare</a> + for comparison, <a href="{{ site.baseurl }}docs/customising/states_informatazyrtare/">example states for InformataZyrtare</a> (Alaveteli site running in Kosovo) </li> <li> - to customise or add your own states, see <a href="{{ site.baseurl }}docs/customising/themes/">Customising the request states</a> + to customise or add your own states, see <a href="{{ site.baseurl }}docs/customising/themes/#customising-the-request-states">Customising the request states</a> + </li> + </ul> + </div> + </dd> + + <dt> + <a name="super">superuser</a> (also: super privilege, administrator) + </dt> + <dd> + A <strong>superuser</strong>, or <strong>administrator</strong>, is an + Alaveteli user who has been granted the privilege to use all features of the + <a href="{{ site.baseurl }}docs/glossary/#admin" + class="glossary__link">admin interface</a>. + <p> + The only way to access the admin without being an Alaveteli superuser + is as the <a href="{{ site.baseurl }}docs/glossary/#emergency" + class="glossary__link">emergency user</a>, which should be disabled in + normal operation. + </p> + <div class="more-info"> + <p>More information:</p> + <ul> + <li> + To grant a user admin privilege, log into the admin and change + their <em>Admin level</em> to "super" (or revoke the privilege + by changing it to "none"). + </li> + <li> + On a newly-installed Alaveteli system, you can grant yourself + admin privilege by using the + <a href="{{ site.baseurl }}docs/glossary/#emergency" class="glossary__link">emergency + user</a>. + </li> + </ul> + </div> + </dd> + + <dt> + <a name="tag">tag</a> + </dt> + <dd> + A <strong>tag</strong> is a keyword added to an + <a href="#authority" class="glossary__link">authority</a>. Tags + are searchable, so can be useful to help users find authorities based + by topic or even unique data (for example, in the + <a href="#wdtk" class="glossary__link">WhatDoTheyKnow</a> we tag every + registered charity with its official charity number). You can also use + tags to assign authorities to + <a href="#category" class="glossary__link">categories</a>. + <div class="more-info"> + <p>More information:</p> + <ul> + <li> + More about + <a href="{{ site.baseurl }}docs/running/categories_and_tags/">categories and tags</a> </li> </ul> </div> </dd> + <dt> <a name="theme">theme</a> </dt> @@ -606,11 +1050,49 @@ Definitions and the code that causes the site to look or behave differently from the default. Typically you'll need a theme to make Alaveteli show your own brand. + <div class="more-info"> + <p>More information:</p> + <ul> + <li> + <a href="{{ site.baseurl }}docs/customising/themes/">about themes</a> + </li> + </ul> + </div> + </dd> + <dt> + + <a name="transifex">Transifex</a> + </dt> + <dd> + + <a href="https://www.transifex.com/">Transifex</a> is a website that helps translators add translations for software projects. + <div class="more-info"> + <p>More information:</p> + <ul> + <li> + The Transifex project for Alaveteli is at <a href="https://www.transifex.com/projects/p/alaveteli">https://www.transifex.com/projects/p/alaveteli</a> + </li> + </ul> + </div> + </dd> + + + <dt> + <a name="wdtk">WhatDoTheyKnow</a> + </dt> + <dd> + The website <strong>WhatDoTheyKnow</strong>.com is the UK installation of + Alaveteli, run by <a href="http://mysociety.org">mySociety</a>. + <p> + In fact, WhatDoTheyKnow predates Alaveteli because the site started in + 2008, and was the foundation of the redeployable, customisable + Alaveteli plattorm released in 2011. + </p> <div class="more-info"> <p>More information:</p> <ul> <li> - <a href="{{ site.baseurl }}docs/customising/themes/">about themes</a> + <a href="http://www.whatdotheyknow.com">WhatDoTheyKnow.com</a> </li> </ul> </div> diff --git a/docs/installing/ami.md b/docs/installing/ami.md index 1f7195761..974c55d0c 100644 --- a/docs/installing/ami.md +++ b/docs/installing/ami.md @@ -1,59 +1,161 @@ --- layout: page -title: Installing the easy way +title: Installation from AMI --- # Installation on Amazon EC2 <p class="lead"> - We've made an Amazon Machine Image (AMI) so you can quickly deploy on Amazon EC2. This is handy if you just want to evaluate Alaveteli, for example. + We've made an Amazon Machine Image (AMI) so you can quickly deploy on Amazon + EC2. This is handy if you just want to evaluate Alaveteli, for example. </p> Note that there are [other ways to install Alaveteli]({{ site.baseurl }}docs/installing/). ## Installing from our AMI -To help people try out Alaveteli, we have created an AMI (Amazon Machine Image) -with a basic installation of Alaveteli, which you can use to create a running -server on an Amazon EC2 instance. This creates an instance that runs in -development mode, so we wouldn't recommend you use it for a production system -without changing the configuration. - -Unfortunately, Alaveteli will not run properly on a free Micro -instance due to the low amount of memory available on those -instances; you will need to use at least a Small instance, which -Amazon will charge for. - -The AMI can be found in the EU West (Ireland) region, with the ID ami-23519f54 -and name “Basic Alaveteli installation 2014-06-12”. You can launch an instance -based on that AMI with [this -link](https://console.aws.amazon.com/ec2/home?region=eu-west-1#launchAmi=ami-23519f54). - -When you create an EC2 instance based on that AMI, make sure that you choose -Security Groups that allows at least inbound HTTP, HTTPS, SSH and, if you want -to test incoming mail as well, SMTP. - -When your EC2 instance is launched, you will be able to log in as the `ubuntu` -user. This user can `sudo` freely to run commands as root. However, the code is -actually owned by (and runs as) the `alaveteli` user. After creating the -instance, you may want to edit a configuration file to customize the site's -configuration. That configuration file is -`/var/www/alaveteli/alaveteli/config/general.yml`, which can be edited with: +To help you try out Alaveteli, we have created an AMI with a basic installation +of Alaveteli, which you can use to create a running server on an Amazon EC2 +instance. This creates an instance that runs as a +<a href="{{ site.baseurl }}docs/glossary/#development" class="glossary__link">development site</a>. +If you want to use this for a +<a href="{{ site.baseurl }}docs/glossary/#production" class="glossary__link">production site</a>, +you must +[change the configuration]({{ site.baseurl }}docs/customising/config/#staging_site). + +<div class="attention-box"> + <p> + <strong>What's in the AMI?</strong> + The AMI gives you exactly the same thing as the + <a href="{{ site.baseurl}}docs/installing/script/">installation script</a> + does. You get an Alaveteli website powered by Rails running the Thin + application server under nginx, using a postgreSQL database. All this + running on Amazon's EC2 servers, ready to be + <a href="{{ site.baseurl }}docs/customising/">configured and customised</a>. + </p> +</div> + +Amazon instances are graded by size. Unfortunately, the *Micro* instance does +not have enough memory for Alaveteli to run -- and that's the only size +available on Amazon's free usage tier. You need to use a *Small* instance or +larger, which Amazon will charge you for. + +### Using Amazon web services + +To do this, you'll need: + + * an account with Amazon + * a SSL key pair (the Amazon web service screens guide you through this) + +If you don't have these already, you'll need to create them. See Amazon's +introduction on +[running a Virtual Server on AWS](http://docs.aws.amazon.com/gettingstarted/latest/awsgsg-intro/gsg-aws-virtual-server.html). + +### Launch the instance + +Once you're logged in to Amazon's service, and navigated to the **EC2 +Management Console**, you can launch the instance. If you prefer to do this +manually, you can find the AMI in the "EU West (Ireland)" region, with the ID +`ami-455ac032` and name “Basic Alaveteli installation 2014-12-05”. +Alternatively, use this link: + +<p class="action-buttons"> + <a href="https://console.aws.amazon.com/ec2/home?region=eu-west-1#launchAmi=ami-455ac032" class="button">launch + instance with Alaveteli installation AMI</a> +</p> + +When the instance launches, the first thing you need to choose is the instance +*type*. Remember that the *Micro* type does not have enough memory to run +Alaveteli, so you must choose at least *Small* or *Medium* -- note that these +are not available on Amazon's free usage tier. + +When the instance is created, the Amazon interface presents you with a lot of +choices about its configuration. You can generally accept the defaults for +everything, except the Security Groups. It's safe to click on **Review and +Launch** right away (rather than manually configuring all the instance details) +because you still get an opportunity to configure the security groups. Click on +**Edit Security Groups** on the summary page before you hit the big **Launch** +button. + +You must choose Security Groups that allow at least inbound HTTP, HTTPS, SSH +and, if you want to test incoming mail as well, SMTP. Amazon's settings here +let you specify the IP address(es) from which your instance will accept +requests. It's good practice to restrict these (if in doubt, choose a *Source* +of "My IP" for them all -- except incoming HTTP: for that, simpy to set +*Source* to "Anywhere"). You can change any of these settings later if you need +to. + +### Log into the server (shell) + +You need access to the server's command line shell to control and configure +your Alaveteli site. + +To access the server, use `ssh` and the `.pem` file from your SSL key pair. +Change the `.pem` file and instance ID to match your own in this command, which +connects to your server and logs you in as the user called `ubuntu`. Issue this +command from your own machine, to log in to the server: + + ssh -i path-to/your-key-pair.pem ubuntu@instance-id.eu-west-1.compute.amazonaws.com + +You won't be asked for a password, because the `.pem` file you supply with the +`-i` option contains the authorisation that matches the one at the other end, +on the server. You will be logged into the shell on your new Alaveteli server, +and can issue Unix commands to it. + +### Smoke test: start Alavateli + +You must configure your Alavateli site, but if you just want to see that you've +got your instance running OK, you *can* fire it up right away. Ideally, you +should skip this step and go straight to the configuration... but we know most +people like to see something in their browser first. ;-) + +On the command line shell, as the `ubuntu` user, start Alaveteli by doing: + + sudo service alaveteli start + +Find the "public DNS" URL of your EC2 instance from the AWS console, and look +at it in a browser. It will be of the form +`http://your-ec2-hostname.eu-west-1.compute.amazonaws.com`. You'll see your +Alaveteli site there. + +Your site isn't configured yet, so *this is insecure* (for example, you haven't +set your own passwords for access to the administration yet), so once you've +seen this running, bring the Alaveteli site down with: + + sudo service alaveteli stop + + +### Shell users: `ubuntu` and `alaveteli` + +When you log into your instance's command line shell, you must do so as the +`ubuntu` user. This user can `sudo` freely to run commands as root. However, +the code is actually owned by (and runs as) the `alaveteli` user. + +You will need to +[customise the site's configuration]({{ site.baseurl }}docs/customising/config/). +Do this by logging into your EC2 server and editing the `general.yml` +configuration file. + +The configuration file you need to edit is +`/var/www/alaveteli/alaveteli/config/general.yml`. For example, use the `nano` +editor (as the `alaveteli` user) like this: ubuntu@ip-10-58-191-98:~$ sudo su - alaveteli alaveteli@ip-10-58-191-98:~$ cd alaveteli alaveteli@ip-10-58-191-98:~/alaveteli$ nano config/general.yml -Then you should restart the Thin webserver with: +After making changes to that file, you'll need to start the application +server (use `restart` rather than `start` if it's already running): alaveteli@ip-10-58-191-98:~/alaveteli$ logout - ubuntu@ip-10-58-191-98:~$ sudo /etc/init.d/alaveteli restart + ubuntu@ip-10-58-191-98:~$ sudo service alaveteli start + +Your site will be running at the public URL again, which is of the form +`http://your-ec2-hostname.eu-west-1.compute.amazonaws.com`. + +If you have any problems or questions, please ask on the [Alaveteli developer mailing list](https://groups.google.com/forum/#!forum/alaveteli-dev) or [report an issue](https://github.com/mysociety/alaveteli/issues?state=open). -If you find the hostname of your EC2 instance from the AWS console, you should -then be able to see the site at -`http://your-ec2-hostname.eu-west-1.compute.amazonaws.com` -If you have any problems or questions, please ask on the [Alaveteli Google -Group](https://groups.google.com/forum/#!forum/alaveteli-dev) or [report an -issue](https://github.com/mysociety/alaveteli/issues?state=open). +##What next? +Check out the [next steps]({{ site.baseurl }}docs/installing/next_steps/). diff --git a/docs/installing/deploy.md b/docs/installing/deploy.md index 4bbc91a9d..74c7a8560 100644 --- a/docs/installing/deploy.md +++ b/docs/installing/deploy.md @@ -8,7 +8,8 @@ title: Deploying <p class="lead"> Although you can install Alaveteli and just change it when you need it, we recommend you adopt a way of <strong>deploying</strong> it automatically, - especially on your <a href="{{ site.baseurl }}docs/glossary/#production">production server</a>. + especially on your + <a href="{{ site.baseurl }}docs/glossary/#production" class="glossary__link">production server</a>. Alaveteli provides a deployment mechanism using Capistrano. </p> @@ -27,13 +28,13 @@ changes or copying files by hand, so your site will be down for the shortest possible time. We **strongly recommend** you use the deployment mechanism for your -<a href="{{ site.baseurl }}docs/glossary/#production">production server</a> and, if -you're running one, your -<a href="{{ site.baseurl }}docs/glossary/#staging">staging server</a> too. +<a href="{{ site.baseurl }}docs/glossary/#production" class="glossary__link">production server</a> +and, if you're running one, your +<a href="{{ site.baseurl }}docs/glossary/#staging" class="glossary__link">staging server</a> too. ## Capistrano -<a href="{{site.baseurl}}docs/glossary/#capistrano" class="glossary">Capistrano</a> +<a href="{{site.baseurl}}docs/glossary/#capistrano" class="glossary__link">Capistrano</a> is included as part of Alaveteli as a standard deployment system. The basic principle of Capistrano is that you execute `cap [do-something]` @@ -68,7 +69,11 @@ and thereafter you'll be able to deploy very easily (see [usage, below](#usage)) First, on the server: * [install Alaveteli]({{ site.baseurl }}docs/installing/) -* then move the Alaveteli app to a temporary place on the server, like your home +* give the Unix user that runs Alaveteli the ability to ssh to your server. Either give them a password or, preferably, set up ssh keys for them so they can ssh from your local machine to the server: + * to give them a password (if they don't already have one) - `sudo passwd [UNIX-USER]`. Store this password securely on your local machine e.g in a password manager + * to set up ssh keys for them, follow the instructions in the [capistrano documentation](http://capistranorb.com/documentation/getting-started/authentication-and-authorisation/). There's no need to set up ssh keys to the git repository as it is public. +* make sure the Unix user that runs Alaveteli has write permissions on the parent directory of your Alaveteli app +* move the Alaveteli app to a temporary place on the server, like your home directory (temporarily, your site will be missing, until the deployment puts new files in place) @@ -82,20 +87,28 @@ Next, on your local machine: need some of the files available locally even though you might not be running Alaveteli on this machine) * copy the example file `config/deploy.yml.example` to `config/deploy.yml` -* now customise the deployment settings in that file: edit `config/deploy.yml` - appropriately -- for example, edit the name of the server. Also, change - `deploy_to` to be the path where Alaveteli is currently installed on the - server -- if you used the installation script, this will be - `/var/www/alaveteli/alaveteli`. +* now customise the deployment settings in that file: edit + `config/deploy.yml` appropriately -- for example, edit the name of the + server. Also, change `deploy_to` to be the path where Alaveteli is + currently installed on the server -- if you used the installation + script , this will be `/var/www/[HOST or alaveteli]/alaveteli`. Set + `daemon_name` to the name you used in setting up the [application + daemon]({{ site.baseurl }}docs/installing/manual_install/#generate-application-daemon). The + default should be `alaveteli`. * `cd` into the Alaveteli repo you checked out (otherwise the `cap` commands you're about to execute won't work) * still on your local machine, run `cap -S stage=staging deploy:setup` to setup capistrano on the server -* again on your local machine, run `cap -S stage=staging deploy:update_code` to get a code checkout on the server + +If you get an error `SSH::AuthenticationFailed`, and are not prompted for the password of the deployment user, you may have run into [a bug](http://stackoverflow.com/questions/21560297/capistrano-sshauthenticationfailed-not-prompting-for-password) in the net-ssh gem version 2.8.0. Try installing version 2.7.0 instead: + + gem uninstall net-ssh + + gem install net-ssh -v 2.7.0 Back on the server: * copy the following config files from the temporary copy of Alaveteli you made at - the begining (perhaps in your home directory) to the `shared` directory that + the beginning (perhaps in your home directory) to the `shared` directory that Capistrano just created on the server: * `general.yml` * `database.yml` @@ -110,17 +123,23 @@ Back on the server: `shared` directory created by Capistrano on the server: * `cache/` * `files/` + * `lib/acts_as_xapian/xapiandbs` (copy this to straight into `shared` so it becomes `shared/xapiandbs`) + * `log/` Now, back on your local machine: * make sure you're still in the Alaveteli repo (if not, `cd` back into it) +* run `cap -S stage=staging deploy:update_code` to get a code checkout on the server. * create a deployment directory on the server by running *one* of these commands: - * `cap deploy` if you're deploying a <a href="{{site.baseurl}}docs/glossary/#staging" class="glossary">staging site</a>, or... - * `cap -S stage=production deploy` for <a href="{{site.baseurl}}docs/glossary/#production" class="glossary">production</a> + * `cap deploy` if you're deploying a <a href="{{site.baseurl}}docs/glossary/#staging" class="glossary__link">staging site</a>, or... + * `cap -S stage=production deploy` for <a href="{{site.baseurl}}docs/glossary/#production" class="glossary__link">production</a> + +Back on the server: + * update the webserver config (either apache or nginx) to add the `current` element to the path where it is serving Alaveteli from. If you installed using the installation script, this will be replacing `/var/www/alaveteli/alaveteli/` with - `/var/www/alaveteli/alaveteli/current` in `etc/nginx/sites-available/default`. + `/var/www/alaveteli/alaveteli/current` in `/etc/nginx/sites-available/default`. * edit the server crontab so that the paths in the cron jobs also include the `current` element. If you used the installation script the crontab will be in `etc/cron.d/alaveteli`. @@ -130,7 +149,13 @@ Now, back on your local machine: `argv=/var/www/alaveteli/alaveteli/script/mailin` with `argv=/var/www/alaveteli/alaveteli/current/script/mailin`. If you're using Exim as your MTA, edit `etc/exim4/conf.d/04_alaveteli_options` - to update the `ALAVETELI_HOME` variable to the new Alaveteli path. + to update the `ALAVETELI_HOME` variable to the new Alaveteli path. Restart the MTA after you've made these changes. + +* You will also need to update the path to Alaveteli in your [init scripts]({{site.baseurl}}docs/installing/manual_install/#cron-jobs-and-init-scripts). + You should have a script for running the alert tracks + (`/etc/init.d/foi-alert-tracks`), and possibly scripts for purging the + varnish cache (`/etc/init.d/foi-purge-varnish`), and restarting the + app server (`/etc/init.d/alaveteli`). Phew, you're done! @@ -146,7 +171,7 @@ for the config that you've set up). Ensure you've got a `config/deploy.yml` file with the correct settings for your site. If there are other people in your team who need to deploy, you'll need to share it with them too -- it might be a good idea to keep the latest -version in a [Gist](http://gist.github.com/). +version in a private [Gist](http://gist.github.com/). * to deploy to staging, just run `cap deploy` * to deploy to production, run `cap -S stage=production deploy` diff --git a/docs/installing/email.md b/docs/installing/email.md index 98dff0087..44476cefa 100644 --- a/docs/installing/email.md +++ b/docs/installing/email.md @@ -11,6 +11,77 @@ title: Installing MTA here for both postfix and exim4, two of the most popular MTAs. </p> +## How Alaveteli handles email + +### Request mail + +When someone makes a Freedom of Information request to an authority through +Alaveteli, the application sends an email containing the request to the authority. + +The email's `reply-to` address is a special one so that any replies to it +can be automatically directed back to Alaveteli, and so that Alaveteli +can tell which request the reply needs to be shown with. This requires +some configuration of the MTA on the server that is running Alaveteli, +so that it will pipe all emails to these special addresses to Alaveteli +to handle, via its `script/mailin` script. The special addresses are of +the form: + + <foi+request-3-691c8388@example.com> + +Parts of this address are controlled with options in +`config/general.yml`: + + INCOMING_EMAIL_PREFIX = 'foi+' + INCOMING_EMAIL_DOMAIN = 'example.com' + +If there is some error inside Rails while processing an email, an exit code `75` is returned to the MTA by the `script/mailin` script. Postfix and Exim (and maybe others) take this as a signal for the MTA to try again later. Additionally, a stacktrace is emailed to `CONTACT_EMAIL`. + +[Production]({{ site.baseurl }}/docs/glossary/#production) installs of Alaveteli should make a backup copy of emails sent to the special addresses. You can configure your chosen MTA to backup these in a separate mailbox. + +### Transactional mail + +Alaveteli also sends emails to users about their requests – letting them know when someone has replied to them, or prompting them to take further action. + +Configure the address that these messages are sent from in the [`CONTACT_EMAIL`]({{site.baseurl}}docs/customising/config/#contact_email) option in `config/general.yml`: + + CONTACT_EMAIL = 'team@example.com' + +The address in [`CONTACT_EMAIL`]({{ site.baseurl }}docs/customising/config/#contact_email) is also visible in various places on the site so that users can get in touch with the team that runs the site. + +You must configure your MTA to deliver mail sent to these addresses to the administrators of your site so that they can respond to it. + +### Tracks mail + +Users subscribed to updates from the site – known as `tracks` – receive emails when there is something new of interest to them on the site. + +Configure the address that these messages are sent from in the [`TRACK_SENDER_EMAIL`]({{site.baseurl}}docs/customising/config/#track_sender_email) option in `config/general.yml`: + + TRACK_SENDER_EMAIL = 'track@example.com' + +### Automatic bounce handling (optional) + +As [`CONTACT_EMAIL`]({{ site.baseurl }}docs/customising/config/#contact_email) and [`TRACK_SENDER_EMAIL`]({{site.baseurl}}docs/customising/config/#track_sender_email) appear in the `From:` header of emails sent from Alaveteli, they sometimes receive reply emails, including <a href="{{ site.baseurl }}docs/glossary/#bounce-message">bounce messages</a> and ‘out of office’ notifications. + +Alaveteli provides a script (`script/handle-mail-replies`) that handles bounce messages and ‘out of office’ notifications and forwards genuine mails to your administrators. + +It also prevents further track emails being sent to a user email address that appears to have a permanent delivery problem. + +To make use of automatic bounce-message handling, set [`TRACK_SENDER_EMAIL`]({{ site.baseurl }}docs/customising/config/#track_sender_email) and [`CONTACT_EMAIL`]({{ site.baseurl }}docs/customising/config/#contact_email) to an address that you will filter through `script/handle-mail-replies`. Messages that are not bounces or out-of-office autoreplies will be forwarded to [`FORWARD_NONBOUNCE_RESPONSES_TO`]({{ site.baseurl }}docs/customising/config/#forward_nonbounce_responses_to), which you should set to a mail alias that points at your list of site administrators. + +See the MTA-specific instructions for how to do this for [exim]({{ site.baseurl }}docs/installing/email#filter-incoming-messages-to-admin-addresses) and [postfix]({{ site.baseurl }}docs/installing/email#filter-incoming-messages-to-site-admin-addresses). + +_Note:_ Bounce handling is not applied to [request emails]({{ site.baseurl }}docs/installing/email#request-mail). Bounce messages from authorities get added to the request page so that the user can see what has happened. Users can ask site admins for help redelivering the request if necessary. + + +--- + +<div class="attention-box"> + <ul> + <li>Commands in this guide will require root privileges</li> + <li>Commands are intended to be run via the terminal or over ssh</li> + </ul> +</div> + Make sure you follow the correct instructions for the specific MTA you're using: * [postfix](#example-setup-on-postfix) @@ -19,54 +90,199 @@ Make sure you follow the correct instructions for the specific MTA you're using: ## Example setup on postfix This section shows an example of how to set up your MTA if you're using -**postfix** (running on Ubuntu). See the example for +**postfix**. See the example for [exim4](#example-setup-on-exim4) if you're using that instead of postfix. -### Instructions +### Install postfix + + # Install debconf so we can configure non-interactively + apt-get -qq install -y debconf >/dev/null + + # Set the default configuration 'Internet Site' + echo postfix postfix/main_mailer_type select 'Internet Site' | debconf-set-selections -For example, with: + # Set your hostname (change example.com to your hostname) + echo postfix postfix/mail_name string "example.com" | debconf-set-selections - ALAVETELI_HOME=/path/to/alaveteli/software - ALAVETELI_USER=www-data + # Install postfix + DEBIAN_FRONTEND=noninteractive apt-get -qq -y install postfix >/dev/null -In `/etc/postfix/master.cf`: +### Configure postfix + +#### Pipe incoming mail for requests into Alaveteli + +If the Unix user that is going to +run your site is `alaveteli`, and the directory where Alaveteli is installed is +`/var/www/alaveteli`, create the pipe that will receive request mail: + + cat >> /etc/postfix/master.cf <<EOF alaveteli unix - n n - 50 pipe - flags=R user=ALAVETELI_USER argv=ALAVETELI_HOME/script/mailin + flags=R user=alaveteli argv=/var/www/alaveteli/script/mailin + EOF + +The Unix user should have write permissions on the directory where Alaveteli is installed. -The user ALAVETELI_USER should have write permissions on ALAVETELI_HOME. +Configure postfix to accept messages for local delivery where +recipients are: -In `/etc/postfix/main.cf`: + - defined by a regular expression in `/etc/postfix/transports` + - local UNIX accounts + - local aliases specified as regular expressions in `/etc/postfix/recipients` - virtual_alias_maps = regexp:/etc/postfix/regexp +<!-- Comment to enable markdown to render code fence under list --> -And, assuming you set -[`INCOMING_EMAIL_PREFIX`]({{ site.baseurl }}docs/customising/config/#incoming_email_prefix) -in `config/general` to "foi+", create `/etc/postfix/regexp` with the following -content: + cat >> /etc/postfix/main.cf <<EOF + transport_maps = regexp:/etc/postfix/transports + local_recipient_maps = proxy:unix:passwd.byname regexp:/etc/postfix/recipients + EOF - /^foi.*/ alaveteli +In `/etc/postfix/main.cf` update the `mydestination` line (which determines what domains this machine will deliver locally). Add your domain, not `example.com`, to the beginning of the list: -You should also configure postfix to discard any messages sent to the -[`BLACKHOLE_PREFIX`]({{ site.baseurl }}docs/customising/config/#blackhole_prefix) -address, whose default value is `do-not-reply-to-this-address`. For example, add the -following to `/etc/aliases`: + mydestination = example.com, localhost.localdomain, localhost +<div class="attention-box"> +This guide assumes you have set <a href="{{ site.baseurl }}docs/customising/config/#incoming_email_prefix"><code>INCOMING_EMAIL_PREFIX</code></a> to <code>foi+</code> in <code>config/general.yml</code> +</div> + +Pipe all incoming mail where the `To:` address starts with `foi+` to the `alaveteli` pipe (`/var/www/alaveteli/script/mailin`, as specified in `/etc/postfix/master.cf` at the start of this section): + + cat > /etc/postfix/transports <<EOF + /^foi.*/ alaveteli + EOF + +#### Backup request mail + +You can copy all incoming mail to Alaveteli to a backup account to a separate mailbox, just in case. + +Create a UNIX user `backupfoi` + + adduser --quiet --disabled-password \ + --gecos "Alaveteli Mail Backup" backupfoi + +Add the following line to `/etc/postfix/main.cf` + + recipient_bcc_maps = regexp:/etc/postfix/recipient_bcc + +Configure mail sent to an `foi+` prefixed address to be sent to the backup user: + + cat > /etc/postfix/recipient_bcc <<EOF + /^foi.*/ backupfoi + EOF + + +#### Define the valid recipients for your domain + +Create `/etc/postfix/recipients` with the following command: + + cat > /etc/postfix/recipients <<EOF + /^foi.*/ this-is-ignored + /^postmaster@/ this-is-ignored + /^user-support@/ this-is-ignored + /^team@/ this-is-ignored + EOF + +The left-hand column of this file specifies regular expressions that +define addresses that mail will be accepted for. The values on the +right-hand side are ignored by postfix. Here we allow postfix to accept +mails to special Alaveteli addresses, and `postmaster@example.com`, +`user-support@example.com` and `team@example.com`. + +The `@example.com` domain is set in the `mydestination` as above. This should be set to your actual domain. + +#### Set up contact email recipient groups + +To set up recipient groups for the `postmaster@`, `team@` and `user-support@` email addresses at your domain, add alias records for them in `/etc/aliases`: + + cat >> /etc/aliases <<EOF + team: user@example.com, otheruser@example.com + user-support: team + EOF + +#### Discard unwanted incoming email + +Configure postfix to discard any messages sent to the [`BLACKHOLE_PREFIX`]({{ site.baseurl }}docs/customising/config/#blackhole_prefix) address, whose default value is `do-not-reply-to-this-address`: + + cat >> /etc/aliases <<EOF # We use this for envelope from for some messages where # we don't care about delivery - do-not-reply-to-this-address: :blackhole: + do-not-reply-to-this-address: /dev/null + EOF + +If you have set [`BLACKHOLE_PREFIX`]({{ site.baseurl }}docs/customising/config/#blackhole_prefix) address, replace `do-not-reply-to-this-address` with the address you have configured. + +#### Filter incoming messages to site admin addresses + +You can make use of Alaveteli's [automatic bounce handling]({{site.baseurl}}docs/installing/email/#automatic-bounce-handling-optional) to filter bounces sent to [`TRACK_SENDER_EMAIL`]({{site.baseurl}}docs/customising/config/#track_sender_email) +and [`CONTACT_EMAIL`]({{site.baseurl}}docs/customising/config/#contact_email). + + +<div class="attention-box"> +This guide assumes you have set the following in <code>config/general.yml</code>: + + <ul> + <li><a href="{{site.baseurl}}docs/customising/config/#contact_email">CONTACT_EMAIL</a>: <code>user-support@example.com</code></li> + <li><a href="{{site.baseurl}}docs/customising/config/#track_sender_email">TRACK_SENDER_EMAIL</a>: <code>user-support@example.com</code></li> + <li><a href="{{site.baseurl}}docs/customising/config/#forward_nonbounce_responses_to">FORWARD_NONBOUNCE_RESPONSES_TO</a>: <code>team@example.com</code></li> + </ul> + +Change the examples below to the addresses you have configured. +</div> + +Create a new pipe to handle replies: + + cat >> /etc/postfix/master.cf <<EOF + alaveteli_replies unix - n n - 50 pipe + flags=R user=alaveteli argv=/var/www/alaveteli/script/handle-mail-replies + EOF + +_Note:_ Replace `/var/www/alaveteli` with the correct path to alaveteli if required. + +Pipe mail sent to `user-support@example.com` to the `alaveteli_replies` pipe: + + cat >> /etc/postfix/transports <<EOF + /^user-support@*/ alaveteli_replies + EOF -### Logging +Finally, edit `/etc/aliases` to remove `user-support`: -For the postfix logs to be succesfully read by the script `load-mail-server-logs`, they need -to be log rotated with a date in the filename. Since that will create a lot of rotated log -files (one for each day), it's good to have them in their own directory. For example (on Ubuntu), -in `/etc/rsyslog.d/50-default.conf` set: + team: user@example.com, otheruser@example.com + +#### Logging + +For the postfix logs to be successfully read by +`script/load-mail-server-logs`, they need to be log rotated with a date in the +filename. Since that will create a lot of rotated log files (one for +each day), it's good to have them in their own directory. + +You'll also need to tell Alaveteli where the log files are stored and that they're in postfix +format. Update +[`MTA_LOG_PATH`]({{ site.baseurl }}docs/customising/config/#mta_log_path) and +[`MTA_LOG_TYPE`]({{ site.baseurl }}docs/customising/config/#mta_log_type) in `config/general.yml`: + + MTA_LOG_PATH: '/var/log/mail/mail.log-*' + MTA_LOG_TYPE: "postfix" + +Configure postfix to log to its own directory: + +##### Debian + +In `/etc/rsyslog.conf`, set: mail.* -/var/log/mail/mail.log -And also edit `/etc/logrotate.d/rsyslog`: +##### Ubuntu + +In `/etc/rsyslog.d/50-default.conf` set: + + mail.* -/var/log/mail/mail.log + +##### Configure logrotate + +Configure logrotate to rotate the log files in the required format: + + cat >> /etc/logrotate.d/rsyslog <<EOF /var/log/mail/mail.log { rotate 30 @@ -81,65 +297,136 @@ And also edit `/etc/logrotate.d/rsyslog`: reload rsyslog >/dev/null 2>&1 || true endscript } + EOF -You'll also need to tell Alaveteli where the log files are stored and that they're in postfix -format. Update -[`MTA_LOG_PATH`]({{ site.baseurl }}docs/customising/config/#mta_log_path) and -[`MTA_LOG_TYPE`]({{ site.baseurl }}docs/customising/config/#mta_log_type) in `config/general.yml` with: +#### Making the changes live - MTA_LOG_PATH: '/var/log/mail/mail.log-*' - MTA_LOG_TYPE: "postfix" +As the root user, make all these changes live with the following commands: -### Troubleshooting (postfix) + service rsyslog restart + + newaliases + postmap /etc/postfix/transports + postmap /etc/postfix/recipients + postmap /etc/postfix/recipient_bcc + postfix reload + +#### Troubleshooting (postfix) To test mail delivery, run: - $ /usr/sbin/sendmail -bv foi+requrest-1234@localhost + $ /usr/sbin/sendmail -bv foi+request-1234@example.com + +Make sure to replace `example.com` with your domain. This command tells +you if sending the emails to `foi\+.*example.com` and the backup account +is working (it doesn't actually send any mail). If it is working, you +should receive a delivery report email, with text like: + + <foi+request-1234@example.com>: delivery via alaveteli: + delivers to command: /var/www/alaveteli/script/mailin + <backupfoi@local.machine.name>: delivery via local: delivers to mailbox + +You can also test the other aliases you have set up for your domain in +this section to check that they will deliver mail as you expect. For +example, you can test bounce message routing in the same way - the text +of this delivery report mail should read something like: + + <user-support@example.com>: delivery via alaveteli_replies: delivers to command: /var/www/alaveteli/script/handle-mail-replies + + +Note that you may need to install the `mailutils` package to read the +delivery report email using the `mail` command on a new server: + + apt-get install mailutils + +If emails are not being received by your Alaveteli install, we have some +more troubleshooting tips for incoming mail in [general email troubleshooting]({{ site.baseurl }}docs/installing/email#general-email-troubleshooting). -This tells you if sending the emails to `foi\+.*localhost` is working. ## Example setup on exim4 This section shows an example of how to set up your MTA if you're using -**exim4** (running on Ubuntu). See the example for +**exim4**. See the example for [postfix](#example-setup-on-postfix) if you're using that instead of exim4. -### Instructions +### Install exim4 + +Install exim4: + + apt-get install exim4 -We suggest you add the following to your exim configuration. -In `/etc/exim4/conf.d/main/04_alaveteli_options`, set: +### Configure exim4 - ALAVETELI_HOME=/path/to/alaveteli/software - ALAVETELI_USER=www-data +#### Set up exim to receive mail from other servers + +Edit `/etc/exim4/update-exim4.conf.conf`. Set the following settings (use your hostname, not `example.com`): + + dc_eximconfig_configtype='internet' + dc_other_hostnames='example.com' + dc_local_interfaces='0.0.0.0 ; ::1' + dc_use_split_config='true' + +This final line tells exim to use the files in `/etc/exim4/conf.d` to configure itself. + +#### Define general variables and logging settings + +Create `/etc/exim4/conf.d/main/04_alaveteli_options` with the command: + + cat > /etc/exim4/conf.d/main/04_alaveteli_options <<'EOF' + ALAVETELI_HOME=/var/www/alaveteli + ALAVETELI_USER=alaveteli log_file_path=/var/log/exim4/exim-%slog-%D MAIN_LOG_SELECTOR==+all -retry_defer extract_addresses_remove_arguments=false + EOF + +This sets up `ALAVETELI_HOME` and `ALAVETELI_USER` for use in other config files, and sets up logging. + +- **`ALAVETELI_HOME`:** set to the directory where Alaveteli is installed. +- **`ALAVETELI_USER`:** should be the Unix user that is going to run your site. They should have write permissions on `ALAVETELI_HOME`. +- **`log_file_path`:** The name and location of the log files created by Exim must match what the `load-mail-server-logs` script expects +- **`MAIN_LOG_SELECTOR`:** The `check-recent-requests-sent` scripts expects the logs to contain the `from=<...>` envelope information, so we make the logs more verbose +- **`extract_addresses_remove_arguments`:** setting to `false` gets exim to treat the `-t` command line option that the `mail` gem uses when specifying delivery addresses on the command line as specifying that the addresses should be added, not removed. See [this `mail` issue](https://github.com/mikel/mail/issues/70) for more details. + +<div class="attention-box"> +Note: If you are editing an existing exim config rather than creating a new one, check the <code>untrusted_set_sender</code> option in <code>/etc/exim4/conf.d/main/02_exim4-config_options</code>. By default, untrusted users in exim are only allowed to set an empty envelope sender address, to declare that a message should never generate any bounces. <code>untrusted_set_sender</code> can be set to a list of address patterns, meaning that untrusted users are allowed to set envelope sender addresses that match any of the patterns in the list. If a pattern list is specified, you will need also to add <code>ALAVETELI_USER</code> to the <code>MAIN_TRUSTED_USERS</code> list in order to allow them to set the return path on outgoing mail. This option is also in <code>/etc/exim4/conf.d/main/02_exim4-config_options</code> in a split config. Look for the line that begins with <code>MAIN_TRUSTED_USERS</code> - something like: -The user ALAVETELI_USER should have write permissions on ALAVETELI_HOME. + <pre><code>MAIN_TRUSTED_USERS = uucp</code></pre> -The name and location of the log files created by Exim must match what the -`load-mail-server-logs` script expects, which is why you must provide the -`log_file_path` setting. +and add the alaveteli user: -The `check-recent-requests-sent` scripts expects the logs to contain the -`from=<...>` envelope information, so we make the logs more verbose with -`log_selector`. The ALAVETELI_USER may need to also need to be added to the -`trusted_users` list in your Exim config in order to set the return path on -outgoing mail, depending on your setup. + <pre><code>MAIN_TRUSTED_USERS = uucp : alaveteli</code></pre> -In `/etc/exim4/conf.d/router/04_alaveteli`: + If <code>untrusted_set_sender</code> is set to <code>*</code>, that means that untrusted users can set envelope sender addresses without restriction, so there's no need to add <code>ALAVETELI_USER</code> to the <code>MAIN_TRUSTED_USERS</code> list. +</div> +#### Pipe incoming mail for requests from Exim to Alaveteli + +In this section, we'll add config to pipe incoming mail for special +Alaveteli addresses into Alaveteli, and also send them to a local backup +mailbox. + +Create the `backupfoi` UNIX user + + adduser --quiet --disabled-password \ + --gecos "Alaveteli Mail Backup" backupfoi + +Specify an exim `router` for special Alaveteli addresses, which will route messages into Alaveteli using a local pipe transport: + + cat > /etc/exim4/conf.d/router/04_alaveteli <<'EOF' alaveteli_request: debug_print = "R: alaveteli for $local_part@$domain" driver = redirect data = ${lookup{$local_part}wildlsearch{ALAVETELI_HOME/config/aliases}} pipe_transport = alaveteli_mailin_transport + EOF -In `/etc/exim4/conf.d/transport/04_alaveteli`: +Create `/etc/exim4/conf.d/transport/04_alaveteli`, which sets the properties of the pipe `transport` that will deliver mail to Alaveteli: + cat > /etc/exim4/conf.d/transport/04_alaveteli <<'EOF' alaveteli_mailin_transport: driver = pipe command = $address_pipe ${lc:$local_part} @@ -147,67 +434,197 @@ In `/etc/exim4/conf.d/transport/04_alaveteli`: home_directory = ALAVETELI_HOME user = ALAVETELI_USER group = ALAVETELI_USER + EOF + + +<div class="attention-box"> + This guide assumes you have set <a href="/docs/customising/config/#incoming_email_prefix"><code>INCOMING_EMAIL_PREFIX</code></a> to <code>foi+</code> in <code>config/general.yml</code> +</div> + +Create the `config/aliases` file that the `alaveteli_request` exim `router` sources. This pipes mail from the special address to `script/mailin` and the `backupfoi` user. + + cat > /var/www/alaveteli/config/aliases <<'EOF' + ^foi\\+.*: "|/var/www/alaveteli/script/mailin", backupfoi + EOF -And, assuming you set -[`INCOMING_EMAIL_PREFIX`]({{ site.baseurl }}docs/customising/config/#incoming_email_prefix) -in your config at `config/general.yml` to "foi+", create `config/aliases` with the following -content: +_Note:_ Replace `/var/www/alaveteli` with the correct path to alaveteli if required. - ^foi\\+.*: |/path/to/alaveteli/software/script/mailin +#### Set up your contact email recipient groups -You should also configure exim to discard any messages sent to the -[`BLACKHOLE_PREFIX`]({{ site.baseurl }}docs/customising/config/#blackhole_prefix) -address, whose default value is -`do-not-reply-to-this-address`. For example, add the following to -`config/aliases`: +To set up recipient groups for the `team@` and `user-support@` email addresses at your domain, add alias records for them in `/var/www/alaveteli/config/aliases` - # We use this for envelope from for some messages where we don't care about delivery + cat >> /var/www/alaveteli/config/aliases <<EOF + team: user@example.com, otheruser@example.com + user-support: team + EOF + +#### Discard unwanted incoming email + +Configure exim to discard any messages sent to the [`BLACKHOLE_PREFIX`]({{ site.baseurl }}docs/customising/config/#blackhole_prefix) address, whose default value is `do-not-reply-to-this-address` + + cat >> /var/www/alaveteli/config/aliases <<EOF + # We use this for envelope from for some messages where + # we don't care about delivery do-not-reply-to-this-address: :blackhole: + EOF + +_Note:_ Replace `/var/www/alaveteli` with the correct path to alaveteli if required. + +#### Filter incoming messages to admin addresses + +You can make use of Alaveteli's [automatic bounce handling]({{site.baseurl}}docs/installing/email/#automatic-bounce-handling-optional) to filter bounces sent to [`TRACK_SENDER_EMAIL`]({{site.baseurl}}docs/customising/config/#track_sender_email) +and [`CONTACT_EMAIL`]({{site.baseurl}}docs/customising/config/#contact_email). + +<div class="attention-box"> +This guide assumes you have set the following in <code>config/general.yml</code>: -If you want to make use of the automatic bounce-message handling, then set the -[`TRACK_SENDER_EMAIL`]({{ site.baseurl }}docs/customising/config/#track_sender_email) -address to be filtered through -`script/handle-mail-replies`. Messages that are not bounces or -out-of-office autoreplies will be forwarded to -[`FORWARD_NONBOUNCE_RESPONSES_TO`]({{ site.baseurl }}docs/customising/config/#forward_nonbounce_responses_to). -For example, in WhatDoTheyKnow the -configuration looks like this: + <ul> + <li><a href="{{site.baseurl}}docs/customising/config/#contact_email">CONTACT_EMAIL</a>: <code>user-support@example.com</code></li> + <li><a href="{{site.baseurl}}docs/customising/config/#track_sender_email">TRACK_SENDER_EMAIL</a>: <code>user-support@example.com</code></li> + <li><a href="{{site.baseurl}}docs/customising/config/#forward_nonbounce_responses_to">FORWARD_NONBOUNCE_RESPONSES_TO</a>: <code>team@example.com</code></li> + </ul> - raw_team: [a list of people on the team] - team: |/path/to/alaveteli/software/script/handle-mail-replies +Change the examples below to the addresses you have configured. +</div> -with `FORWARD_NONBOUNCE_RESPONSES_TO`: 'raw_team@whatdotheyknow.com'` +Change the `user-support` line in `/var/www/alaveteli/config/aliases`: -Finally, make sure you have `dc_use_split_config='true'` in -`/etc/exim4/update-exim4.conf.conf`, and execute the command -`update-exim4.conf`. + user-support: |/var/www/alaveteli/script/handle-mail-replies + +#### Logging + +You’ll need to tell Alaveteli where the log files are stored and that they’re in exim format. Update [`MTA_LOG_PATH`]({{ site.baseurl }}docs/customising/config/#mta_log_path) and [`MTA_LOG_TYPE`]({{ site.baseurl }}docs/customising/config/#mta_log_type) in `config/general.yml`: + + MTA_LOG_PATH: '/var/log/exim4/exim-mainlog-*' + MTA_LOG_TYPE: 'exim' + + +#### Making the changes live in exim + +Finally, execute the commands: + + update-exim4.conf + service exim4 restart Note that if the file `/etc/exim4/exim4.conf` exists then `update-exim4.conf` will silently do nothing. Some distributions include this file. If -yours does, you will need to rename it before running `update-exim4.conf`. +yours does, you will need to remove or rename it before running `update-exim4.conf`. -(You may also want to set `dc_eximconfig_configtype='internet'`, -`dc_local_interfaces='0.0.0.0 ; ::1'`, and -`dc_other_hostnames='<your-host-name>'`). -### Troubleshooting (exim) +#### Troubleshooting (exim) -To test mail delivery, run: +To test mail delivery, as a privileged user run: - exim -bt foi+request-1234@localhost + exim4 -bt foi+request-1234@example.com -This should tell you which routers are being processed. You should +replacing `example.com` with your domain name. This should tell you which routers are being processed. You should see something like: - $ exim -bt foi+request-1234@localhost - R: alaveteli pipe for snafflerequest-234@localhost - snafflerequest-234@localhost -> |/home/alaveteli/alaveteli/script/mailin - transport = alaveteli_mailin_transport + $ exim4 -bt foi+request-1234@example.com + R: alaveteli for foi+request-1234@example.com + foi+request-1234@example.com -> |/var/www/alaveteli/script/mailin + transport = alaveteli_mailin_transport + R: alaveteli for backupfoi@your.machine.name + R: system_aliases for backupfoi@your.machine.name + R: userforward for backupfoi@your.machine.name + R: procmail for backupfoi@your.machine.name + R: maildrop for backupfoi@your.machine.name + R: lowuid_aliases for backupfoi@your.machine.name (UID 1001) + R: local_user for backupfoi@your.machine.name + backupfoi@your.machine.name + <-- foi+request-1234@example.com + router = local_user, transport = mail_spool This tells you that the routing part (making emails to -`foi\+.*@localhost` be forwarded to Alaveteli's `mailin` script) is -working. +`foi\+.*@example.com` be forwarded to Alaveteli's `mailin` script, and +also sent to the local backup account) is working. You can test bounce +message routing in the same way: + + exim4 -bt user-support@example.com + R: alaveteli for user-support@example.com + user-support@example.com -> |/var/www/alaveteli/script/handle-mail-replies + transport = alaveteli_mailin_transport + +If emails are not being received by your Alaveteli install, we have some +more troubleshooting tips for incoming mail in the next section. There is also a +great [Exim +Cheatsheet](http://bradthemad.org/tech/notes/exim_cheatsheet.php) online +that you may find useful. + +## General Email Troubleshooting + +First, you need to check that your MTA is delivering relevant +incoming emails to the `script/mailin` command. There are various +ways of setting your MTA up to do this; we have documented +one way of doing it +[in Exim]({{ site.baseurl }}docs/installing/email/#example-setup-on-exim4), including [a command you can use]({{ site.baseurl }}docs/installing/email/#troubleshooting-exim) to check that the email +routing is set up correctly. We've also documented one way of setting up [Postfix]({{ site.baseurl }}docs/installing/email/#example-setup-on-postfix), with a similar [debugging command]({{ site.baseurl }}docs/installing/email/#troubleshooting-postfix). + +Second, you need to test that the mailin script itself is working +correctly, by running it from the command line, First, find a +valid "To" address for a request in your system. You can do this +through your site's admin interface, or from the command line, +like so: + + $ ./script/console + Loading development environment (Rails 2.3.14) + >> InfoRequest.find_by_url_title("why_do_you_have_such_a_fancy_dog").incoming_email + => "request-101-50929748@localhost" + +Now take the source of a valid email (there are some sample emails in +`spec/fixtures/files/`); edit the `To:` header to match this address; +and then pipe it through the mailin script. A non-zero exit code +means there was a problem. For example: + + $ cp spec/fixtures/files/incoming-request-plain.email /tmp/ + $ perl -pi -e 's/^To:.*/To: <request-101-50929748@localhost>/' /tmp/incoming-request-plain.email + $ ./script/mailin < /tmp/incoming-request-plain.email + $ echo $? + 75 + +The `mailin` script emails the details of any errors to +`CONTACT_EMAIL` (from your `general.yml` file). A common problem is +for the user that the MTA runs as not to have write access to +`files/raw_emails/`. + +If everything seems fine locally, you should also check from another +computer connected to the Internet that the DNS for your chosen +domain indicates that your Alaveteli server is handling mail, and +that your server is receiving mail on port 25. The following +command is a query to ask which server is handling the mail for +the domain `example.com`, which receives the answer `mail.example.com`. + + $ host -t mx example.com + example.com mail is handled by 5 mail.example.com. + +This next command tries to connect to port 25, the standard SMTP +port, on `mail.example.com`, and is refused. + + $ telnet mail.example.com 25 + Trying 10.10.10.30... + telnet: connect to address 10.10.10.30: Connection refused + +The transcript below shows a successful connection where the server +accepts mail for delivery (the commands you would type are prefixed +by a `$`): + + $ telnet 10.10.10.30 25 + Trying 10.10.10.30... + Connected to 10.10.10.30. + Escape character is '^]'. + 220 mail.example.com ESMTP Exim 4.80 Tue, 12 Aug 2014 11:10:39 +0000 + $ HELO X + 250 mail.example.com Hello X [10.10.10.1] + $ MAIL FROM: <test@local.domain> + 250 OK + $ RCPT TO:<foi+request-1234@example.com> + 250 Accepted + $ DATA + 354 Enter message, ending with "." on a line by itself + $ Subject: Test + $ + $ This is a test mail. + $ . + 250 OK id=1XHA03-0001Vx-Qn + QUIT -There is a great -[Exim Cheatsheet](http://bradthemad.org/tech/notes/exim_cheatsheet.php) -online that you may find useful. diff --git a/docs/installing/index.md b/docs/installing/index.md index c276c3d08..c04aaa3ca 100644 --- a/docs/installing/index.md +++ b/docs/installing/index.md @@ -6,17 +6,19 @@ title: Installing # Installing Alaveteli <p class="lead"> - Although you can install Alaveteli and just change it when you need it, we - recommend you adopt a way of <strong>deploying</strong> it automatically. - This has several advantages, especially for your - <a href="{{ site.baseurl }}docs/glossary/#production">production server</a>. + There are a number of ways to install Alaveteli. + We've made an Amazon Machine Image (AMI) so you can quickly deploy on + Amazon EC2 (handy if you just want to evaluate it, for example). + If you prefer to use your own server, there's an installation script + which does most of the work for you, or you can follow the manual + installation instructions. </p> ## Before you start This is important: you need to decide if you are installing Alaveteli for -[development]({{ site.baseurl }}docs/glossary/#development) or -[production]({{ site.baseurl }}docs/glossary/#production). +<a href="{{ site.baseurl }}docs/glossary/#development" class="glossary__link">development</a> or +<a href="{{ site.baseurl }}docs/glossary/#production" class="glossary__link">production</a>. A **development** site is one where you're going to change, customise, and perhaps experiment while you get it up and running. You should always do this @@ -30,9 +32,10 @@ messages switched off. It's important to be able to deploy changes to a production site quickly and efficiently, so we recommend you consider using a [deployment mechanism]({{ site.baseurl }}docs/installing/deploy/) too. -Ideally, you should also have a [staging site]({{ site.baseurl }}docs/glossary/#staging), +Ideally, you should also have a +<a href="{{ site.baseurl }}docs/glossary/#staging" class="glossary__link">staging site</a>, which is used solely to test new code in an identical environment to your -production site but before it goes live. +production site before it goes live. If you're in doubt, you're probably running a development site. Get it up and running, play with it, customise it, and -- later -- you can install it as a @@ -47,6 +50,7 @@ those servers, because Capistrano takes care of that for you. ## Installing the core code +* [Install into a Vagrant virtual development environment]({{ site.baseurl }}docs/installing/vagrant/) -- a good choice for development, and playing around with the site. * [Install on Amazon EC2]({{ site.baseurl }}docs/installing/ami/) using our AMI * [Use the installation script]({{ site.baseurl }}docs/installing/script/) which does the full installation on your own server * [Manual installation]({{ site.baseurl }}docs/installing/manual_install/) -- step-by-step instructions @@ -56,7 +60,7 @@ If you're setting up a development server on MacOS X, we've also got ## Other installation information -Alaveteli needs to be able to send and receive email, so you need to setup your -MTA (Mail Transfer Agent) appropriately. +Alaveteli needs to be able to send and receive email. If you're installing manually, you need to [setup your +MTA (Mail Transfer Agent) appropriately]({{ site.baseurl }}docs/installing/email/). The other install methods will do this for you. * [Installing the MTA]({{ site.baseurl }}docs/installing/email/) diff --git a/docs/installing/macos.md b/docs/installing/macos.md index c46bdf4ba..2c08be0e5 100644 --- a/docs/installing/macos.md +++ b/docs/installing/macos.md @@ -71,7 +71,7 @@ Read `rvm notes` and `rvm requirements` carefully for further instructions. Then The `mahoro` and `pg` gems require special installation commands. Rubygems must be downgraded to 1.6.2 to avoid deprecation warnings when running tests. rvm 1.8.7 - gem update --system 1.6.2 + gem update --system 2.1.11 gem install mahoro -- --with-ldflags="-L/usr/local/Cellar/libmagic/5.09/lib" --with-cppflags="-I/usr/local/Cellar/libmagic/5.09/include" env ARCHFLAGS="-arch x86_64" gem install pg @@ -88,32 +88,36 @@ The following is mostly from [the manual installation process]({{ site.baseurl}} ### Configure database -Creates Alaveteli databases and an `foi` user with password `foi`. +Create a database for your Mac user as homebrew doesn't create one by default: - echo "CREATE DATABASE foi_development encoding = 'UTF8'; - CREATE DATABASE foi_test encoding = 'UTF8'; - CREATE USER foi WITH CREATEUSER; - ALTER USER foi WITH PASSWORD 'foi'; - ALTER USER foi WITH CREATEDB; - GRANT ALL PRIVILEGES ON DATABASE foi_development TO foi; - GRANT ALL PRIVILEGES ON DATABASE foi_test TO foi; - ALTER DATABASE foi_development OWNER TO foi; - ALTER DATABASE foi_test OWNER TO foi;" | psql -h localhost template1 + createdb -### Clone Alaveteli +Create a `foi` user from the command line, like this: + + createuser -s -P foi + +_Note:_ After running this command you will be prompted to set a +password for the user. Don't leave it blank if you are new to +PostgreSQL, or it could be difficult to set later for you. + +We'll create a template for our Alaveteli databases: + + createdb -T template0 -E UTF-8 template_utf8 + echo "update pg_database set datistemplate=true where datname='template_utf8';" | psql -We don't want to vendor Rails, as it causes problems locally. +Then create the databases: + + createdb -T template_utf8 -O foi alaveteli_production + createdb -T template_utf8 -O foi alaveteli_test + createdb -T template_utf8 -O foi alaveteli_development + +### Clone Alaveteli git clone https://github.com/mysociety/alaveteli.git cd alaveteli git submodule init - - sed -i~ 's/\\[submodule "vendor\/rails"\\]//' .git/config - - sed -i~ 's/url = git:\/\/github.com\/rails\/rails.git//' .git/config git submodule update -**Note:** Due to Markdown bugs, the first `sed` command above does not display properly if it appears in blockquote. ### Configure Alaveteli diff --git a/docs/installing/manual_install.md b/docs/installing/manual_install.md index 777d95139..9cad6b5b9 100644 --- a/docs/installing/manual_install.md +++ b/docs/installing/manual_install.md @@ -17,91 +17,183 @@ title: Manual installation Note that there are [other ways to install Alaveteli]({{ site.baseurl }}docs/installing/). -## Target operating system +<div class="attention-box"> + <ul> + <li>Commands in this guide will require root privileges</li> + <li>Commands are intended to be run via the terminal or over ssh</li> + </ul> +</div> -These instructions assume Debian Squeeze (64-bit) or Ubuntu 12.04 LTS -(precise). Debian Squeeze is the best supported deployment platform. We also +## Configure the Operating System + +### Target operating system + +These instructions assume a 64-bit version of Debian 6 (Wheezy), Debian 7 (Squeeze) +or Ubuntu 12.04 LTS (Precise). Debian is the best supported deployment platform. We also have instructions for [installing on MacOS]({{ site.baseurl }}docs/installing/macos/). -Commands are intended to be run via the terminal or over ssh. +### Set the locale +**Debian Wheezy or Squeeze** -## Get Alaveteli +Follow the [Debian guide](https://wiki.debian.org/Locale#Standard) for configuring the locale of the operating system. + +Generate the locales you wish to make available. When the interactive screen asks you to pick a default locale, choose "None", as the SSH session will provide the locale required. -To start with, you may need to install git, e.g. with `sudo apt-get install -git-core` + dpkg-reconfigure locales + +Start a new SSH session to use your SSH locale. + +**Ubuntu Precise** -Next, get hold of the Alaveteli source code from github: +Unset the default locale, as the SSH session should provide the locale required. - git clone https://github.com/mysociety/alaveteli.git - cd alaveteli + update-locale LC_ALL= -This will get the rails-3-develop branch, which has the latest (possibly buggy) -code. If you don't want to add or try new features, swap to the master branch -(which always contains the latest stable release): +Start a new SSH session to use your SSH locale. - git checkout master +### Update the OS -## Install mySociety libraries +Update the Operating System with the latest packages -Next, install mySociety's common ruby libraries. To fetch the contents of the -submodules, run: + apt-get update -y + apt-get upgrade -y - git submodule update --init +`sudo` is not installed on Debian by default. Install it along with `git` (the version control tool we'll use to get a copy of the Alaveteli code). -## Install system dependencies + apt-get install -y sudo git-core + +### Prepare to install system dependencies using OS packages These are packages that the software depends on: third-party software used to parse documents, host the site, and so on. There are also packages that contain headers necessary to compile some of the gem dependencies in the next step. +#### Using other repositories to get more recent packages + Add the following repositories to `/etc/apt/sources.list`: **Debian Squeeze** - cat > /etc/apt/sources.list.d/debian-backports.list <<EOF + cat > /etc/apt/sources.list.d/debian-extra.list <<EOF + # Debian mirror to use, including contrib and non-free: + deb http://the.earth.li/debian/ squeeze main contrib non-free + deb-src http://the.earth.li/debian/ squeeze main contrib non-free + + # Security Updates: + deb http://security.debian.org/ squeeze/updates main non-free + deb-src http://security.debian.org/ squeeze/updates main non-free + + # Debian Backports deb http://backports.debian.org/debian-backports squeeze-backports main contrib non-free + deb-src http://backports.debian.org/debian-backports squeeze-backports main contrib non-free + + # Wheezy + deb http://ftp.uk.debian.org/debian wheezy main contrib non-free + EOF + +The squeeze-backports repository is providing a more recent version of rubygems, and the wheezy repository is providing bundler. You should configure package-pinning to reduce the priority of the wheezy repository so other packages aren't pulled from it. + + cat >> /etc/apt/preferences <<EOF + + Package: bundler + Pin: release n=wheezy + Pin-Priority: 990 + + Package: * + Pin: release n=wheezy + Pin-Priority: 50 EOF -The repositories above let you install `wkhtmltopdf-static` and `bundler` using -`apt`. +**Debian Wheezy** + + cat > /etc/apt/sources.list.d/debian-extra.list <<EOF + # Debian mirror to use, including contrib and non-free: + deb http://the.earth.li/debian/ wheezy main contrib non-free + deb-src http://the.earth.li/debian/ wheezy main contrib non-free + + # Security Updates: + deb http://security.debian.org/ wheezy/updates main non-free + deb-src http://security.debian.org/ wheezy/updates main non-free + EOF **Ubuntu Precise** cat > /etc/apt/sources.list.d/ubuntu-extra.list <<EOF - deb http://eu-west-1.ec2.archive.ubuntu.com/ubuntu/ precise multiverse - deb-src http://eu-west-1.ec2.archive.ubuntu.com/ubuntu/ precise multiverse - deb http://eu-west-1.ec2.archive.ubuntu.com/ubuntu/ precise-updates multiverse - deb-src http://eu-west-1.ec2.archive.ubuntu.com/ubuntu/ precise-updates multiverse + deb http://de.archive.ubuntu.com/ubuntu/ precise multiverse + deb-src http://de.archive.ubuntu.com/ubuntu/ precise multiverse + deb http://de.archive.ubuntu.com/ubuntu/ precise-updates multiverse + deb-src http://de.archive.ubuntu.com/ubuntu/ precise-updates multiverse + deb http://de.archive.ubuntu.com/ubuntu/ trusty universe + deb-src http://de.archive.ubuntu.com/ubuntu/ trusty universe EOF -The repositories above let you install `wkhtmltopdf-static` using `apt`. -`bundler` will have to be installed manually on Ubuntu Precise. +The trusty repo is used here to get a more recent version of bundler. You should configure package-pinning to reduce the priority of the trusty repository so other packages aren't pulled from it. + + cat >> /etc/apt/preferences <<EOF -### Packages customised by mySociety + Package: ruby-bundler + Pin: release n=trusty + Pin-Priority: 990 -If you're using Debian, you should add the mySociety Debian archive to your + Package: * + Pin: release n=trusty + Pin-Priority: 50 + EOF + + +#### Packages customised by mySociety + +If you're using Debian or Ubuntu, you should add the mySociety Debian archive to your apt sources. Note that mySociety packages are currently only built for 64-bit Debian. +**Debian Squeeze, Wheezy or Ubuntu Precise** + cat > /etc/apt/sources.list.d/mysociety-debian.list <<EOF deb http://debian.mysociety.org squeeze main EOF +The repository above lets you install `wkhtmltopdf-static` and `pdftk` (for squeeze) using `apt`. + Add the GPG key from the [mySociety Debian Package Repository](http://debian.mysociety.org/). - wget -O - https://debian.mysociety.org/debian.mysociety.org.gpg.key | sudo apt-key add - + wget -O - https://debian.mysociety.org/debian.mysociety.org.gpg.key | apt-key add - + +**Ubuntu Precise only** + + cat > /etc/apt/sources.list.d/mysociety-launchpad.list <<EOF + deb http://ppa.launchpad.net/mysociety/alaveteli/ubuntu precise main + deb-src http://ppa.launchpad.net/mysociety/alaveteli/ubuntu precise main + EOF + +The repository above lets you install a recent version of `pdftk` using `apt`. + +Add the GPG key from the +[mySociety Alaveteli Ubuntu Package Repository](https://launchpad.net/~mysociety/+archive/ubuntu/alaveteli). + + apt-get install -y python-software-properties + add-apt-repository -y ppa:mysociety/alaveteli + +**Debian Wheezy or Ubuntu Precise** -You should also configure package-pinning to reduce the priority of this -repository. +You should also configure package-pinning to reduce the priority of the +mysociety Debian repository - we only want to pull wkhtmltopdf-static +from mysociety. + + cat >> /etc/apt/preferences <<EOF - cat > /etc/apt/preferences <<EOF Package: * Pin: origin debian.mysociety.org Pin-Priority: 50 EOF -If you're using some other platform, you can optionally install these +**Debian Squeeze** + +No special package pinning is required. + +#### Other platforms +If you're using some other linux platform, you can optionally install these dependencies manually, as follows: 1. If you would like users to be able to get pretty PDFs as part of the @@ -115,37 +207,76 @@ everything will still work, but users will get ugly, plain text versions of their requests when they download them. 2. Version 1.44 of `pdftk` contains a bug which makes it loop forever in -certain edge conditions. Until it's incorporated into an official release, you -can either hope you don't encounter the bug (it ties up a rails process until -you kill it), patch it yourself, or use the Debian package -compiled by mySociety (see link in [issue -305](https://github.com/mysociety/alaveteli/issues/305)) +certain edge conditions. This is fixed in the standard 1.44.7 package which is available in wheezy (Debian) and raring (Ubuntu). + +If you can't get an official release for your OS with the fix, you can +either hope you don't encounter the bug (it ties up a rails process +until you kill it), patch it yourself, or use the +[Debian](http://debian.mysociety.org/dists/squeeze/main/binary-amd64/) +or +[Ubuntu](https://launchpad.net/~mysociety/+archive/ubuntu/alaveteli/+packages) +packages compiled by mySociety. -### Install the dependencies +#### Refresh sources Refresh the sources after adding the extra repositories: - sudo apt-get update + apt-get -y update + +### Create Alaveteli User + +Create a new linux user to run the Alaveteli application. + + adduser --quiet --disabled-password --gecos "Alaveteli" alaveteli + +## Get Alaveteli + +Create the target directory and clone the Alaveteli source code in to this directory: + + mkdir -p /var/www/alaveteli + chown alaveteli:alaveteli /var/www + chown alaveteli:alaveteli /var/www/alaveteli + cd /home/alaveteli + sudo -u alaveteli git clone --recursive \ + --branch master \ + https://github.com/mysociety/alaveteli.git /var/www/alaveteli + +This clones the master branch which always contains the latest stable release. If you want to try out the latest (possibly buggy) code you can switch to the `rails-3-develop` branch. + + pushd /var/www/alaveteli + sudo -u alaveteli git checkout rails-3-develop + sudo -u alaveteli git submodule update + popd + +The `--recursive` option installs mySociety's common libraries which are required to run Alaveteli. + +## Install the dependencies Now install the packages relevant to your system: + # Debian Wheezy + apt-get -y install $(cat /var/www/alaveteli/config/packages.debian-wheezy) + # Debian Squeeze - sudo apt-get install $(cat config/packages.debian-squeeze) + apt-get -y install $(cat /var/www/alaveteli/config/packages.debian-squeeze) # Ubuntu Precise - sudo apt-get install $(cat config/packages.ubuntu-precise) + apt-get -y install $(cat /var/www/alaveteli/config/packages.ubuntu-precise) Some of the files also have a version number listed in config/packages - check that you have appropriate versions installed. Some also list "`|`" and offer a choice of packages. -## Install Ruby dependencies +<div class="attention-box"> + +<strong>Note:</strong> To install Alaveteli's Ruby dependencies, you need to install bundler. In +Debian and Ubuntu, this is provided as a package (installed as part of the +package install process above). For other OSes, you could also install it as a gem: + + <pre><code> gem install bundler --no-rdoc --no-ri</code></pre> -To install Alaveteli's Ruby dependencies, you need to install bundler. In -Debian, this is provided as a package (installed as part of the package install -process above). You could also install it as a gem: +</div> - sudo gem install bundler ## Configure Database @@ -153,348 +284,573 @@ There has been a little work done in trying to make the code work with other databases (e.g., SQLite), but the currently supported database is PostgreSQL ("postgres"). -If you don't have postgres installed: - - $ sudo apt-get install postgresql postgresql-client - Create a `foi` user from the command line, like this: - # sudo -u postgres createuser -s -P foi + sudo -u postgres createuser -s -P foi _Note:_ Leaving the password blank will cause great confusion if you're new to PostgreSQL. +We'll create a template for our Alaveteli databases: + + sudo -u postgres createdb -T template0 -E UTF-8 template_utf8 + echo "update pg_database set datistemplate=true where datname='template_utf8';" > /tmp/update-template.sql + sudo -u postgres psql -f /tmp/update-template.sql + rm /tmp/update-template.sql + Then create the databases: - # sudo -u postgres createdb -T template0 -E SQL_ASCII -O foi foi_production - # sudo -u postgres createdb -T template0 -E SQL_ASCII -O foi foi_test - # sudo -u postgres createdb -T template0 -E SQL_ASCII -O foi foi_development + sudo -u postgres createdb -T template_utf8 -O foi alaveteli_production + sudo -u postgres createdb -T template_utf8 -O foi alaveteli_test + sudo -u postgres createdb -T template_utf8 -O foi alaveteli_development + +## Configure email + +You will need to set up an email server – or Mail Transfer Agent (MTA) – to +send and receive emails. + +Full configuration for an MTA is beyond the scope of this document -- see the guide for [configuring the Exim4 or Postfix MTAs]({{ site.baseurl }}docs/installing/email/). + +Note that in development mode mail is handled by [`mailcatcher`](http://mailcatcher.me/) by default so +that you can see the mails in a browser. Start mailcatcher by running `bundle exec mailcatcher` in the application directory. + +## Configure Alaveteli + +Alaveteli has three main configuration files: + + - `config/database.yml`: Configures Alaveteli to communicate with the database + - `config/general.yml`: The general Alaveteli application settings + - `config/newrelic.yml`: Configuration for the [NewRelic](http://newrelic.com) monitoring service + +Copy the configuration files and update their permissions: -We create using the ``SQL_ASCII`` encoding, because in postgres this is means -"no encoding"; and because we handle and store all kinds of data that may not -be valid UTF (for example, data originating from various broken email clients -that's not 8-bit clean), it's safer to be able to store *anything*, than reject -data at runtime. + cp /var/www/alaveteli/config/database.yml-example /var/www/alaveteli/config/database.yml + cp /var/www/alaveteli/config/general.yml-example /var/www/alaveteli/config/general.yml + cp /var/www/alaveteli/config/newrelic.yml-example /var/www/alaveteli/config/newrelic.yml + chown alaveteli:alaveteli /var/www/alaveteli/config/{database,general,newrelic}.yml + chmod 640 /var/www/alaveteli/config/{database,general,newrelic}.yml -Now you need to set up the database config file to contain the name, username -and password of your postgres database. +### database.yml -* Copy `database.yml-example` to `database.yml` in `alaveteli/config` -* Edit it to point to your local postgresql database in the development - and test sections. +Now you need to set up the database config file so that the application can +connect to the postgres database. + +Edit each section to point to the relevant local postgresql database. Example `development` section of `config/database.yml`: development: adapter: postgresql - database: foi_development + template: template_utf8 + database: alaveteli_development username: foi password: secure-password-here host: localhost port: 5432 Make sure that the user specified in `database.yml` exists, and has full -permissions on these databases. As they need the ability to turn off -constraints whilst running the tests they also need to be a superuser +permissions on these databases. +As the user needs the ability to turn off constraints whilst running the tests +they also need to be a superuser (clarification: a <em>Postgres</em> superuser, +not an Alaveteli +<a href="{{ site.baseurl }}docs/glossary/#super" class="glossary__link">superuser</a>). If you don't want your database user to be a superuser, you can add this line -to the test config in `database.yml` (as seen in `database.yml-example`) +to the `test` section in `database.yml` (as seen in `config/database.yml-example`): constraint_disabling: false -## Configure email +### general.yml -You will need to set up an email server (MTA) to send and receive emails. Full -configuration for an MTA is beyond the scope of this document -- see this -[example config for Exim4]({{ site.baseurl }}docs/installing/email/). +We have a full [guide to Alaveteli configuration]({{ site.baseurl }}docs/customising/config/) which covers all the settings in `config/general.yml`. -Note that in development mode mail is handled by mailcatcher by default so -that you can see the mails in a browser - see [http://mailcatcher.me/](http://mailcatcher.me/) for more -details. Start mailcatcher by running `bundle exec mailcatcher` in your -application directory. +_Note:_ If you are setting up Alaveteli to run in production, set the [`STAGING_SITE`]({{ site.baseurl }}docs/customising/config/#staging_site) variable to `0` in `/var/www/alaveteli/config/general.yml` now. -### Minimal + STAGING_SITE: 0 -If you just want to get the tests to pass, you will at a minimum need to allow -sending emails via a `sendmail` command (a requirement met, for example, with -`sudo apt-get install exim4`). +The default settings for frontpage examples are designed to work with +the dummy data shipped with Alaveteli; once you have real data, you should +certainly edit these. -### Detailed +The default theme is the ["Alaveteli" theme](https://github.com/mysociety/alavetelitheme). When you run `rails-post-deploy` (see below), that theme gets installed automatically. -When an authority receives an email, the email's `reply-to` field is a magic -address which is parsed and consumed by the Rails app. +### newrelic.yml -To receive such email in a production setup, you will need to configure your -MTA to pipe incoming emails to the Alaveteli script `script/mailin`. Therefore, -you will need to configure your MTA to accept emails to magic addresses, and to -pipe such emails to this script. +This file contains configuration information for the New Relic performance +management system. By default, monitoring is switched off by the +`agent_enabled: false` setting. See New Relic's [remote performance analysis](https://github.com/newrelic/rpm) instructions for switching it on +for both local and remote analysis. -Magic email addresses are of the form: +## Deployment - <foi+request-3-691c8388@example.com> +You should run the `rails-post-deploy` script after each new software upgrade: -The respective parts of this address are controlled with options in -`config/general.yml`, thus: + sudo -u alaveteli RAILS_ENV=production \ + /var/www/alaveteli/script/rails-post-deploy - INCOMING_EMAIL_PREFIX = 'foi+' - INCOMING_EMAIL_DOMAIN = 'example.com' +This installs Ruby dependencies, installs/updates themes, runs database +migrations, updates shared directories and runs other tasks that need to be run +after a software update, like precompiling static assets for a production install. -When you set up your MTA, if there is some error inside Rails, the -email is returned with an exit code 75, which for Exim at least means the MTA -will try again later. Additionally, a stacktrace is emailed to `CONTACT_EMAIL`. +That the first time you run this script can take a *long* time, as it must +compile native dependencies for `xapian-full`. -See [this example]({{ site.baseurl }}docs/installing/email/) for a possible configuration for Exim (>=1.9). +Create the index for the search engine (Xapian): -A well-configured installation of this code will have had Exim make -a backup copy of the email in a separate mailbox, just in case. + sudo -u alaveteli RAILS_ENV=production \ + /var/www/alaveteli/script/rebuild-xapian-index -## Set up configs +If this fails, the site should still mostly run, but it's a core component so +you should really try to get this working. -Copy `config/general.yml-example` to `config/general.yml` and edit to your -taste. +<div class="attention-box"> + Note that we set <code>RAILS_ENV=production</code>. Use + <code>RAILS_ENV=development</code> if you are installing Alaveteli to make + changes to the code. +</div> -Note that the default settings for frontpage examples are designed to work with -the dummy data shipped with Alaveteli; once you have real data, you should -certainly edit these. +## Configure the Application Server -The default theme is the "Alaveteli" theme. When you run `rails-post-deploy` -(see below), that theme gets installed automatically. +Alaveteli can run under many popular application servers. mySociety recommends +the use of [Phusion Passenger](https://www.phusionpassenger.com) (AKA +mod_rails) or [thin](http://code.macournoyer.com/thin). -Finally, copy `config/newrelic.yml-example` to `config/newrelic.yml`. This file -contains configuration information for the New Relic performance management -system. By default, monitoring is switched off by the `agent_enabled: false` -setting. See New Relic's [remote performance analysis](https://github.com/newrelic/rpm) instructions for switching it on -for both local and remote analysis. +### Using Phusion Passenger +Passenger is the recommended application server as it is well proven in +production environments. It is implemented as an Apache mod, so it cannot be +run independently. -## Deployment + apt-get install -y libapache2-mod-passenger -In the `alaveteli` directory, run: +See later in the guide for configuring the Apache web server with Passenger. - script/rails-post-deploy +### Using Thin -(This will need execute privs so `chmod 755` if necessary.) This sets up -directory structures, creates logs, installs/updates themes, runs database -migrations, etc. You should run it after each new software update. +Thin is a lighter-weight application server which can be run independently of +a web server. Thin will be installed in the application bundle and used to run Alaveteli by default. -One of the things the script does is install dependencies (using `bundle -install`). Note that the first time you run it, part of the `bundle install` -that compiles `xapian-full` takes a *long* time! +Run the following to get the server running: -If you want some dummy data to play with, you can try loading the fixtures that -the test suite uses into your development database. You can do this with: + cd /var/www/alaveteli + bundle exec thin \ + --environment=production \ + --user=alaveteli \ + --group=alaveteli \ + start - script/load-sample-data +By default the server listens on all interfaces. You can restrict it to the +localhost interface by adding `--address=127.0.0.1` -Next, create the index for the search engine (Xapian): +The server should have told you the URL to access in your browser to see the +site in action. - script/rebuild-xapian-index +You can daemonize the process by starting it with the `--daemonize` option. -If this fails, the site should still mostly run, but it's a core component so -you should really try to get this working. +Later in this guide we'll actually create a SysVinit daemon to run the application, so stop any thin processes you've started here. -## Run the Tests +## Cron jobs and Daemons -Make sure everything looks OK: +The crontab and init scripts use the `.ugly` file format, which is a strange +templating format used by mySociety. - bundle exec rake spec +The `ugly` format uses simple variable substitution. A variable looks like +`!!(*= $this *)!!`. -If there are failures here, something has gone wrong with the preceding steps -(see the next section for a common problem and workaround). You might be able -to move on to the next step, depending on how serious they are, but ideally you -should try to find out what's gone wrong. +### Generate crontab -### glibc bug workaround +`config/crontab-example` contains the cron jobs that run on +Alaveteli. Rewrite the example file to replace the variables, +and then drop it in `/etc/cron.d/` on the server. -There's a [bug in -glibc](http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=637239) which causes -Xapian to segfault when running the tests. Although the bug report linked to -claims it's fixed in the current Debian stable, it's not as of version -`2.11.3-2`. +**Template Variables:** -Until it's fixed (e.g. `libc6 2.13-26` does work), you can get the tests to -pass by setting `export LD_PRELOAD=/lib/libuuid.so.1`. +* `vhost_dir`: the full path to the directory where alaveteli is checked out. + e.g. If your checkout is at `/var/www/alaveteli` then set this to `/var/www` +* `vcspath`: the name of the directory that contains the alaveteli code. + e.g. `alaveteli` +* `user`: the user that the software runs as +* `site`: a string to identify your alaveteli instance +* `mailto`: The email address or local account that cron output will be sent to - setting an email address depends on your MTA having been configured for remote delivery. -## Run the Server +There is a rake task that will help to rewrite this file into one that is +useful to you. This example sends cron output to the local `alaveteli` user. Change the variables to suit your installation. -Run the following to get the server running: + pushd /var/www/alaveteli + bundle exec rake config_files:convert_crontab \ + DEPLOY_USER=alaveteli \ + VHOST_DIR=/var/www \ + VCSPATH=alaveteli \ + SITE=alaveteli \ + MAILTO=alaveteli \ + CRONTAB=/var/www/alaveteli/config/crontab-example > /etc/cron.d/alaveteli + popd - bundle exec rails server --environment=development + chown root:alaveteli /etc/cron.d/alaveteli + chmod 754 /etc/cron.d/alaveteli -By default the server listens on all interfaces. You can restrict it to the -localhost interface by adding `--binding=127.0.0.1` +### Generate application daemon -The server should have told you the URL to access in your browser to see the -site in action. +Generate a daemon based on the application server you installed. This allows you +to use the native `service` command to stop, start and restart the application. -## Administrator privileges +#### Passenger -The administrative interface is at the URL `/admin`. +**Template Variables:** -Only users with the `super` admin level can access the admin interface. Users -create their own accounts in the usual way, and then administrators can give -them `super` privileges. +* `vhost_dir`: the full path to the directory where alaveteli is checked out. + e.g. If your checkout is at `/var/www/alaveteli` then set this to `/var/www` +* `vcspath`: the name of the directory that contains the alaveteli code. + e.g. `alaveteli` +* `site`: a string to identify your alaveteli instance +* `user`: the user that the software runs as -There is an emergency user account which can be accessed via -`/admin?emergency=1`, using the credentials `ADMIN_USERNAME` and -`ADMIN_PASSWORD`, which are set in `general.yml`. To bootstrap the -first `super` level accounts, you will need to log in as the emergency -user. You can disable the emergency user account by setting `DISABLE_EMERGENCY_USER` to `true` in `general.yml`. +There is a rake task that will help to rewrite this file into one that is +useful to you. Change the variables to suit your installation. -Users with the superuser role also have extra privileges in the website -frontend, such as being able to categorise any request, being able to view -items that have been hidden from the search, and being presented with "admin" -links next to individual requests and comments in the front end. + pushd /var/www/alaveteli + bundle exec rake config_files:convert_init_script \ + DEPLOY_USER=alaveteli \ + VHOST_DIR=/var/www \ + VCSPATH=alaveteli \ + SITE=alaveteli \ + SCRIPT_FILE=/var/www/alaveteli/config/sysvinit-passenger.ugly > /etc/init.d/alaveteli + popd -It is possible completely to override the administrator authentication by -setting `SKIP_ADMIN_AUTH` to `true` in `general.yml`. + chown root:alaveteli /etc/init.d/alaveteli + chmod 754 /etc/init.d/alaveteli -## Cron jobs and init scripts +Start the application: -`config/crontab-example` contains the cronjobs run on WhatDoTheyKnow. It's in a -strange templating format they use in mySociety. mySociety render the example -file to reference absolute paths, and then drop it in `/etc/cron.d/` on the -server. + service alaveteli start -The `ugly` format uses simple variable substitution. A variable looks like -`!!(*= $this *)!!`. The variables are: - -* `vhost`: part of the path to the directory where the software is - served from. In the mySociety files, it usually comes as - `/data/vhost/!!(*= $vhost *)!!` -- you should replace that whole - port with a path to the directory where your Alaveteli software - installation lives, e.g. `/var/www/` -* `vhost_dir`: the entire path to the directory where the software is - served from. -- you should replace this with a path to the - directory where your Alaveteli software installation lives, - e.g. `/var/www/` -* `vcspath`: the name of the alaveteli checkout, e.g. `alaveteli`. - Thus, `/data/vhost/!!(*= $vhost *)!!/!!(*= $vcspath *)!!` might be - replaced with `/var/www/alaveteli` in your cron tab +#### Thin + +**Template Variables:** + +* `vhost_dir`: the full path to the directory where alaveteli is checked out. + e.g. If your checkout is at `/var/www/alaveteli` then set this to `/var/www` +* `vcspath`: the name of the directory that contains the alaveteli code. + e.g. `alaveteli` +* `site`: a string to identify your alaveteli instance * `user`: the user that the software runs as + +There is a rake task that will help to rewrite this file into one that is +useful to you. Change the variables to suit your installation. + + pushd /var/www/alaveteli + bundle exec rake config_files:convert_init_script \ + DEPLOY_USER=alaveteli \ + VHOST_DIR=/var/www \ + VCSPATH=alaveteli \ + SITE=alaveteli \ + SCRIPT_FILE=/var/www/alaveteli/config/sysvinit-thin.ugly > /etc/init.d/alaveteli + popd + + chown root:alaveteli /etc/init.d/alaveteli + chmod 754 /etc/init.d/alaveteli + +Start the application: + + service alaveteli start + +### Generate alert daemon + +One of the cron jobs refers to a script at `/etc/init.d/alaveteli-alert-tracks`. This +is an init script, which can be generated from the +`config/alert-tracks-debian.ugly` template. This script sends out emails to users subscribed to updates from the site – known as [`tracks`]({{ site.baseurl }}docs/installing/email/#tracks-mail) – when there is something new matching their interests. + +**Template Variables:** + +* `daemon_name`: The name of the daemon. This is set by the rake task. +* `vhost_dir`: the full path to the directory where alaveteli is checked out. + e.g. If your checkout is at `/var/www/alaveteli` then set this to `/var/www` +* `vcspath`: the name of the directory that contains the alaveteli code. + e.g. `alaveteli` * `site`: a string to identify your alaveteli instance +* `user`: the user that the software runs as There is a rake task that will help to rewrite this file into one that is -useful to you, which can be invoked with: +useful to you. Change the variables to suit your installation. - bundle exec rake config_files:convert_crontab \ - DEPLOY_USER=deploy \ - VHOST_DIR=/dir/above/alaveteli \ - VCSPATH=alaveteli \ - SITE=alaveteli \ - CRONTAB=config/crontab-example > crontab - -You should change the `DEPLOY_USER`, `VHOST_DIR`, `VCSPATH` and `SITE` -environment variables to match your server and installation. You should also -edit the resulting `crontab` file to customize the `MAILTO` variable. - -One of the cron jobs refers to a script at `/etc/init.d/foi-alert-tracks`. This -is an init script, a copy of which lives in `config/alert-tracks-debian.ugly`. -As with the cron jobs above, replace the variables (and/or bits near the -variables) with paths to your software. You can use the rake task `rake -config_files:convert_init_script` to do this. + pushd /var/www/alaveteli + bundle exec rake RAILS_ENV=production config_files:convert_init_script \ + DEPLOY_USER=alaveteli \ + VHOST_DIR=/var/www \ + VCSPATH=alaveteli \ + SITE=alaveteli \ + SCRIPT_FILE=/var/www/alaveteli/config/alert-tracks-debian.ugly > /etc/init.d/alaveteli-alert-tracks + popd + + chown root:alaveteli /etc/init.d/alaveteli-alert-tracks + chmod 754 /etc/init.d/alaveteli-alert-tracks + +Start the alert tracks daemon: + + service alaveteli-alert-tracks start + +### Generate varnish purge daemon `config/purge-varnish-debian.ugly` is a similar init script, which is optional -and not required if you choose not to run your site behind Varnish (see below). -Either tweak the file permissions to make the scripts executable by your deploy -user, or add the following line to your sudoers file to allow these to be run -by your deploy user (named `deploy` in this case): +and not required if you choose not to run your site behind Varnish (see below). It notifies Varnish of cached pages that need to be purged from Varnish's cache. It will not run if Varnish is not installed. - deploy ALL = NOPASSWD: /etc/init.d/foi-alert-tracks, /etc/init.d/foi-purge-varnish +**Template Variables:** -The cron jobs refer to a program `run-with-lockfile`. See [this -issue](https://github.com/mysociety/alaveteli/issues/112) for a discussion of -where to find this program, and how you might replace it. This [one line -script](https://gist.github.com/3741194) can install this program system-wide. +* `daemon_name`: The name of the daemon. This is set by the rake task. +* `vhost_dir`: the full path to the directory where alaveteli is checked out. + e.g. If your checkout is at `/var/www/alaveteli` then set this to `/var/www` +* `vcspath`: the name of the directory that contains the alaveteli code. + e.g. `alaveteli` +* `site`: a string to identify your alaveteli instance +* `user`: the user that the software runs as -## Set up production web server +There is a rake task that will help to rewrite this file into one that is +useful to you. Change the variables to suit your installation. -It is not recommended to run the website using the default Rails web server. -There are various recommendations here: http://rubyonrails.org/deploy + pushd /var/www/alaveteli + bundle exec rake RAILS_ENV=production config_files:convert_init_script \ + DEPLOY_USER=alaveteli \ + VHOST_DIR=/var/www \ + VCSPATH=alaveteli \ + SITE=alaveteli \ + SCRIPT_FILE=/var/www/alaveteli/config/purge-varnish-debian.ugly > /etc/init.d/alaveteli-purge-varnish + popd -We usually use Passenger / mod_rails. The file at `conf/httpd.conf-example` -gives you an example config file for WhatDoTheyKnow. At a minimum, you should -include the following in an Apache configuration file: + chown root:alaveteli /etc/init.d/alaveteli-purge-varnish + chmod 754 /etc/init.d/alaveteli-purge-varnish - PassengerResolveSymlinksInDocumentRoot on - PassengerMaxPoolSize 6 # Recommend setting this to 3 or less on servers with 512MB RAM +Start the alert tracks daemon: -Under all but light loads, it is strongly recommended to run the server behind -an http accelerator like Varnish. A sample varnish VCL is supplied in -`conf/varnish-alaveteli.vcl`. + service alaveteli-purge-varnish start + + +## Configure the web server + +In almost all scenarios, we recommend running the Alaveteli Rails application +behind a web server. This allows the web server to serve static content without +going through the Rails stack, which improves performance. + +We recommend two main combinations of application and web server: + +- Apache & Passenger +- Nginx & Thin + +There are ways to run Passenger with Nginx, and indeed Thin with Apache, but +that's out of scope for this guide. If you want to do something that isn't +documented here, get in touch on [alaveteli-dev](https://groups.google.com/forum/#!forum/alaveteli-dev) and we'll +be more than happy to help you get set up. + +You should have already installed an application server if you have followed +this guide, so pick the appropriate web server to configure. + +### Apache (with Passenger) + +Install Apache with the Suexec wrapper: + + apt-get install -y apache2 + apt-get install -y apache2-suexec + +Enable the required modules + + a2enmod actions + a2enmod expires + a2enmod headers + a2enmod passenger + a2enmod proxy + a2enmod proxy_http + a2enmod rewrite + a2enmod suexec + +Create a directory for optional Alaveteli configuration + + mkdir -p /etc/apache2/vhost.d/alaveteli + +Copy the example VirtualHost configuration file. You will need to change all +occurrences of `www.example.com` to your URL + + cp /var/www/alaveteli/config/httpd.conf-example \ + /etc/apache2/sites-available/alaveteli + +Disable the default site and enable the `alaveteli` VirtualHost + + a2dissite default + a2ensite alaveteli + +Check the configuration and fix any issues + + apachectl configtest + +Restart apache to load the new Alaveteli config + + service apache2 graceful + +It's strongly recommended that you run the site over SSL. (Set `FORCE_SSL` to +true in `config/general.yml`). For this you will need an SSL certificate for your domain. + +Enable the SSL apache mod + + a2enmod ssl + +Copy the SSL configuration – again changing `www.example.com` to your domain – +and enable the VirtualHost + + cp /var/www/alaveteli/config/httpd-ssl.conf.example \ + /etc/apache2/sites-available/alaveteli_https + a2ensite alaveteli_https -It's strongly recommended that you run the site over SSL. (Set FORCE_SSL to -true in config/general.yml). For this you will need an SSL certificate for your -domain and you will need to configure an SSL terminator to sit in front of -Varnish. If you're already using Apache as a web server you could simply use -Apache as the SSL terminator. A minimal configuration would look something like -this: +Force HTTPS requests from the HTTP VirtualHost - <VirtualHost *:443> - ServerName www.yourdomain + cp /var/www/alaveteli/config/httpd-force-ssl.conf.example \ + /etc/apache2/vhost.d/alaveteli/force-ssl.conf - ProxyRequests Off - ProxyPreserveHost On - ProxyPass / http://localhost:80/ - ProxyPassReverse / http://localhost:80/ - RequestHeader set X-Forwarded-Proto 'https' +If you are testing Alaveteli or setting up an internal staging site, generate +self-signed SSL certificates. **Do not use self-signed certificates for a +production server**. Replace `www.example.com` with your domain name. - SSLEngine on - SSLProtocol all -SSLv2 - SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM + openssl genrsa -out /etc/ssl/private/www.example.com.key 2048 + chmod 640 /etc/ssl/private/www.example.com.key - SSLCertificateFile /etc/apache2/ssl/ssl.crt - SSLCertificateKeyFile /etc/apache2/ssl/ssl.key - SSLCertificateChainFile /etc/apache2/ssl/sub.class2.server.ca.pem - SSLCACertificateFile /etc/apache2/ssl/ca.pem - SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown + openssl req -new -x509 \ + -key /etc/ssl/private/www.example.com.key \ + -out /etc/ssl/certs/www.example.com.cert \ + -days 3650 \ + -subj /CN=www.example.com + chmod 640 /etc/ssl/certs/www.example.com.cert - </VirtualHost> +Check the configuration and fix any issues -Notice the line `RequestHeader` that sets the `X-Forwarded-Proto` header. This -is important. This ultimately tells Rails that it's serving a page over https -and so it knows to include that in any absolute urls it serves. + apachectl configtest + +Restart apache to load the new Alaveteli config. This will also restart +Passenger (the application server). + + service apache2 graceful + +### Nginx (with Thin) + +Install nginx + + apt-get install -y nginx + +#### Running over SSL + +It's strongly recommended that you run the site over SSL. (Set `FORCE_SSL` to +true in `config/general.yml`). For this you will need an SSL certificate for your domain. + +Copy the SSL configuration – changing `www.example.com` to your domain – +and enable the `alaveteli_https` server, disabling the default site. + + cp /var/www/alaveteli/config/nginx-ssl.conf.example \ + /etc/nginx/sites-available/alaveteli_https + rm /etc/nginx/sites-enabled/default + ln -s /etc/nginx/sites-available/alaveteli_https \ + /etc/nginx/sites-enabled/alaveteli_https + +<div class="attention-box"> + <strong>Note:</strong> For historical reasons, <code>nginx-ssl.conf.example</code> has the path to Alaveteli set as <code>/var/www/alaveteli/alaveteli</code> – you will need to manually change this to <code>/var/www/alaveteli</code>, or to the root of your Alaveteli install +</div> + +If you are testing Alaveteli or setting up an internal staging site, generate +self-signed SSL certificates. **Do not use self-signed certificates for a +production server**. Replace `www.example.com` with your domain name. + + openssl genrsa -out /etc/ssl/private/www.example.com.key 2048 + chmod 640 /etc/ssl/private/www.example.com.key + + openssl req -new -x509 \ + -key /etc/ssl/private/www.example.com.key \ + -out /etc/ssl/certs/www.example.com.cert \ + -days 3650 \ + -subj /CN=www.example.com + chmod 640 /etc/ssl/certs/www.example.com.cert + +Check the configuration and fix any issues + + service nginx configtest + +Reload the new nginx configuration and restart the application + + service nginx reload + service alaveteli restart + +#### Running without SSL + +Set `FORCE_SSL` to +false in `config/general.yml`. Copy the example nginx config + + cp /var/www/alaveteli/config/nginx.conf.example \ + /etc/nginx/sites-available/alaveteli + +<div class="attention-box"> + <strong>Note:</strong> For historical reasons, <code>nginx.conf.example</code> has the path to Alaveteli set as <code>/var/www/alaveteli/alaveteli</code> – you will need to manually change this to <code>/var/www/alaveteli</code>, or to the root of your Alaveteli install +</div> + +Disable the default site and enable the `alaveteli` server + + rm /etc/nginx/sites-enabled/default + ln -s /etc/nginx/sites-available/alaveteli \ + /etc/nginx/sites-enabled/alaveteli + +Check the configuration and fix any issues + + service nginx configtest + +Start the rails application with thin (if you haven't already). + + service alaveteli start + +Reload the nginx configuration + + service nginx reload + + +--- + +## Add varnish as an HTTP accelerator + +Under all but light loads, it is strongly recommended to run the server behind +an http accelerator like Varnish. A sample varnish VCL is supplied in +`conf/varnish-alaveteli.vcl`. + +If you are using SSL you will need to configure an SSL terminator to sit in +front of Varnish. If you're already using Apache as a web server you could +simply use Apache as the SSL terminator. We have some [production server best practice notes]({{ site.baseurl}}docs/running/server/). +## What next? + +Check out the [next steps]({{ site.baseurl }}docs/installing/next_steps/). + ## Troubleshooting +* **Run the Tests** + + Make sure everything looks OK. As the alaveteli user, run: + + bundle exec rake spec + + If there are failures here, something has gone wrong with the preceding + steps (see the next section for a common problem and workaround). You might + be able to move on to the [next steps]({{ site.baseurl }}docs/installing/next_steps/), depending on how serious they are, but + ideally you should try to find out what's gone wrong. + + +<div class="attention-box"> + <strong>Note:</strong> If you have setup your install of Alaveteli for production, you will need to temporarily remove the file <code>config/rails_env.rb</code>, which is used to force the rails environment to production, and edit your <code>.bundle/config</code> file to remove the <code>BUNDLE_WITHOUT</code> line that excludes development dependencies. After you have done this, as the alaveteli user, run <code>bundle install</code>. You will also need to make alaveteli the owner of <code>/var/www/alaveteli/log/development.log</code>, and run the database migrations. + + <pre><code>chown alaveteli:alaveteli /var/www/alaveteli/log/development.log +sudo -u alaveteli bundle exec rake db:migrate</code></pre> + +You should then be able to run the tests. Don't forget to restore <code>config/rails_env.rb</code> when you're done. You will probably see some errors from cron jobs in the meantime, as they'll be running in development mode. + +</div> + + * **Incoming emails aren't appearing in my Alaveteli install** - First, you need to check that your MTA is delivering relevant - incoming emails to the `script/mailin` command. There are various - ways of setting your MTA up to do this; we have documented - [one way of doing it]({{ site.baseurl }}docs/installing/email/#troubleshooting-exim) - in Exim, including a command you can use to check that the email - routing is set up correctly. - - Second, you need to test that the mailin script itself is working - correctly, by running it from the command line, First, find a - valid "To" address for a request in your system. You can do this - through your site's admin interface, or from the command line, - like so: - - $ ./script/console - Loading development environment (Rails 2.3.14) - >> InfoRequest.find_by_url_title("why_do_you_have_such_a_fancy_dog").incoming_email - => "request-101-50929748@localhost" - - Now take the source of a valid email (there are some sample emails in - `spec/fixtures/files/`); edit the `To:` header to match this address; - and then pipe it through the mailin script. A non-zero exit code - means there was a problem. For example: - - $ cp spec/fixtures/files/incoming-request-plain.email /tmp/ - $ perl -pi -e 's/^To:.*/To: <request-101-50929748@localhost>/' /tmp/incoming-request-plain.email - $ ./script/mailin < /tmp/incoming-request-plain.email - $ echo $? - 75 - - The `mailin` script emails the details of any errors to - `CONTACT_EMAIL` (from your `general.yml` file). A common problem is - for the user that the MTA runs as not to have write access to - `files/raw_emails/`. + See the [general email troubleshooting guide]({{ site.baseurl }}docs/installing/email#general-email-troubleshooting). * **Various tests fail with "*Your PostgreSQL connection does not support unescape_bytea. Try upgrading to pg 0.9.0 or later.*"** diff --git a/docs/installing/next_steps.md b/docs/installing/next_steps.md new file mode 100644 index 000000000..2b7acbd45 --- /dev/null +++ b/docs/installing/next_steps.md @@ -0,0 +1,181 @@ +--- +layout: page +title: Next Steps +--- +# Next Steps + +<p class="lead"> + OK, you've installed a copy of Alaveteli, and can see the site in a browser. What next? +</p> + + * [Create a superuser admin account](#create-a-superuser-admin-account) + * [Load sample data](#load-sample-data) + * [Test out the request process](#test-out-the-request-process) + * [Import Public Authorities](#import-public-authorities) + * [Set the amount of time [...] to respond to requests](#set-the-amount-of-time-authorities-will-be-given-to-respond-to-requests) + * [Add some public holidays](#add-some-public-holidays) + * [Start thinking about customising Alaveteli](#start-thinking-about-customising-alaveteli) + + +## Create a superuser admin account + +Alaveteli ships with an +<a href="{{site.baseurl}}docs/glossary/#emergency" class="glossary__link">emergency user</a> +that has access to the admin. So when you've just created a new site, you +should sign up to create your own account, then log into admin as the emergency +user to promote your new account to be an administrator with +<a href="{{ site.baseurl }}docs/glossary/#super" class="glossary__link">super</a> +privilege. + +As soon as that's done, disable the emergency user, because you don't need to +use it any more: you've superseded it with your new admin account. + +Alaveteli ships with sample data that includes a dummy admin user called "Joe +Admin". If the sample data has been loaded into the database (this will depend on +how you installed), you must revoke Joe's administrator status too, because you +will be using your own admin account instead. + +### Step-by-step: + +First, in the browser: + +* Go to `/profile/sign_in` and create a user by signing up. +* Check your email and confirm your account. +* Go to `/admin?emergency=1`, log in with the username and password you specified in + [`ADMIN_USERNAME`]({{site.baseurl}}docs/customising/config/#admin_username) + and [`ADMIN_PASSWORD`]({{site.baseurl}}docs/customising/config/#admin_password). + You can find these settings in `config/general.yml`. +* You're now on the Alaveteli admin page. +* Click on **Users** (in the navigation menu across the top of the page), and + click on your name in the list of users. On *that* page, click **Edit**. +* Change your *Admin level* to "super" and click **Save**. +* From now on, when you are logged into your Alavateli site, you'll have access + to the admin (at `/admin`). Furthermore, you'll see links to admin pages off + the main site (which don't appear for regular users). + +If your installation has loaded the sample data, there will be a dummy user in +your database called "Joe Admin" who has admin status too. You should remove +this status so there's no risk of it being used to access your site admin. You +can either do this while you're still logged in as the emergency user... or +else, later, logged in as yourself: + +* Go to `/admin/users` or click on **Users** in the navigation menu on any + admin page. +* Find "Joe Admin" in the list of users, and click on the name to see the + user details. On *that* page, click **Edit**. +* Change the *Admin level* from "super" to "none" and click **Save**. +* Joe Admin no longer has admin status. + +Now that your account is a superuser admin, you don't need to allow the +emergency user access to the admin. On the command line shell, edit +`/var/www/alaveteli/alaveteli/config/general.yml`: + +* It's important that you change the emergency user's password (and, ideally, + the username too) from the values Alavateli ships with, because they are + public and hence insecure. In `general.yml`, change + [`ADMIN_PASSWORD`]({{site.baseurl}}docs/customising/config/#admin_password) + (and maybe [`ADMIN_USERNAME`]({{site.baseurl}}docs/customising/config/#admin_username) + too) to new, unique values. +* Additionally, you can totally disable the emergency user. Under normal + operation you don't need it, because from now on you'll be using the admin + user you've just created. + Set [`DISABLE_EMERGENCY_USER`]({{site.baseurl}}docs/customising/config/#disable_emergency_user) + to `true`. +* To apply these changes restart the service as a user with root privileges: + `sudo service alaveteli restart` + +You can use the same process (logged in as your admin account) to add or remove +superuser admin status to any users that are subsequently added to your site. +If you accidentally remove admin privilege from all accounts (try not to do +this, though!), you can enable the emergency user by editing the `general.yml` +file and restarting Alaveteli. + +## Load sample data + +If you want some dummy data to play with, you can try loading the fixtures that +the test suite uses into your development database. As the `alaveteli` user, do: + + script/load-sample-data + +If the sample data has already been loaded into the database, this command won't +do anything, but will instead <abbr +title='PG::Error: ERROR: permission denied: "RI_ConstraintTrigger_XXXXXX" is a system trigger'>fail +with an error</abbr>. + +If you have added the sample data, update the Xapian search index afterwards: + + script/update-xapian-index + +Remember that the sample data includes a user with admin access to your site. +You should revoke that status so it cannot be used to access your site -- +follow the steps described in the previous section. + +## Test out the request process + +* Create a new public authority in the admin interface -- give it a name like + "Test authority". Set the request email to an address that you will receive. + +* From the main interface of the site, make a request to the new authority. + +* You should receive the request email -- try replying to it. Your response + email should appear in Alaveteli. Not working? Take a look at our + [troubleshooting tips]({{ site.baseurl}}docs/installing/manual_install/#troubleshooting). + If that doesn't sort it out, [get in touch]({{ site.baseurl}}community/) on + the [developer mailing list](https://groups.google.com/forum/#!forum/alaveteli-dev) or [IRC](http://www.irc.mysociety.org/) for help. + +## Import Public Authorities + +Alaveteli can import a list of public authorities and their contact email addresses from a CSV file. + +Follow the instructions for +[uploading public authority data]({{ site.baseurl }}docs/running/admin_manual/#creating-changing-and-uploading-public-authority-data). + +## Set the amount of time authorities will be given to respond to requests + +In most countries that have a Freedom of Information law, authorities +have a certain number of days in order to respond to requests. Alaveteli +helps requesters by reminding them when their request is overdue for a +response according to the law. You can set the number of days an +authority is given to respond to a request in the +[`REPLY_LATE_AFTER_DAYS`]({{site.baseurl}}docs/customising/config/#reply_late_after_days), +[`REPLY_VERY_LATE_AFTER_DAYS`]({{site.baseurl}}docs/customising/config/#reply_very_late_after_days) +and +[`SPECIAL_REPLY_VERY_LATE_AFTER_DAYS`]({{site.baseurl}}docs/customising/config/#special_reply_very_late_after_days) +options in `config/general.yml`. Most laws specify that the days are +either working days, or calendar days. You can set this using the +[`WORKING_OR_CALENDAR_DAYS`]({{site.baseurl}}docs/customising/config/#working_or_calendar_days) +option in `config/general.yml`. + +## Add some public holidays + +<div class="attention-box info"> +Interface introduced in Alaveteli version 0.21 +</div> + +Alaveteli calculates the due dates of requests taking account of the +<a href="{{ site.baseurl }}docs/glossary/#holiday" class="glossary__link">public holidays</a> +you enter into the admin interface. + +If you have set the +[`WORKING_OR_CALENDAR_DAYS`]({{site.baseurl}}docs/customising/config/#working_or_calendar_days) +setting for Alaveteli to `working`, the date when a response to a +request is officially overdue will be calculated in days that are not +weekends or public holidays. + +If you have set +[`WORKING_OR_CALENDAR_DAYS`]({{site.baseurl}}docs/customising/config/#working_or_calendar_days) +to `calendar`, the date will be calculated in calendar days, but if the +due date falls on a public holiday or weekend day, then the due date is +considered to be the next week day that isn't a holiday. + +To add public holidays, go to the +<a href="{{ site.baseurl }}docs/glossary/#admin" class="glossary__link">admin interface</a> +and click on **Holidays**. From here you can either add each day of holiday by +hand, using the **New Holiday** button, or you can create multiple holidays at +once using the **Create holidays from suggestions or iCalendar feed** button. + +## Start thinking about customising Alaveteli + +Check out [our guide]({{ site.baseurl}}docs/customising/). + + diff --git a/docs/installing/script.md b/docs/installing/script.md index b8b678a0a..72fbd6438 100644 --- a/docs/installing/script.md +++ b/docs/installing/script.md @@ -1,6 +1,6 @@ --- layout: page -title: Installing the easy way +title: Installation script --- # Installation script @@ -16,7 +16,7 @@ Note that there are [other ways to install Alaveteli]({{ site.baseurl }}docs/ins If you have a clean installation of Debian squeeze 64-bit or Ubuntu precise, you can use an install script in our commonlib repository to set up a working instance of Alaveteli. This is not suitable for production (it runs in development mode, -for example) but should set up a functional installation of the site. +for example) but should set up a functional installation of the site, which can send and receive email. **Warning: only use this script on a newly installed server – it will make significant changes to your server’s setup, including modifying your nginx @@ -25,7 +25,7 @@ etc.** To download the script, run the following command: - curl -O https://raw.github.com/mysociety/commonlib/master/bin/install-site.sh + curl -O https://raw.githubusercontent.com/mysociety/commonlib/master/bin/install-site.sh If you run this script with `sh install-site.sh`, you'll see its usage message: @@ -56,10 +56,20 @@ could download the script, make it executable and then invoke it with: sudo ./install-site.sh --default alaveteli alaveteli +If you have any problems or questions, please ask on the [Alaveteli Google + Group](https://groups.google.com/forum/#!forum/alaveteli-dev) or [report an + issue](https://github.com/mysociety/alaveteli/issues?state=open). + +## What the install script does + When the script has finished, you should have a working copy of the website, -accessible via the hostname you supplied to the script. +accessible via the hostname you supplied to the script. So, for this example, you could access the site in a browser at `http://alaveteli.10.10.10.30.xip.io`. The site runs using the thin application server, and the nginx webserver. By default, Alaveteli will be installed into `/var/www/[HOST]` on the server. + +The server will also be configured to accept replies to information request emails (as long as the MX record for the domain is pointing at the server). Incoming mail handling is set up using Postfix as the MTA. + +##What next? + +Check out the [next steps]({{ site.baseurl }}docs/installing/next_steps/). + -If you have any problems or questions, please ask on the [Alaveteli Google -Group](https://groups.google.com/forum/#!forum/alaveteli-dev) or [report an -issue](https://github.com/mysociety/alaveteli/issues?state=open). diff --git a/docs/installing/vagrant.md b/docs/installing/vagrant.md new file mode 100644 index 000000000..a0b058da5 --- /dev/null +++ b/docs/installing/vagrant.md @@ -0,0 +1,95 @@ +--- +layout: page +title: Vagrant +--- +# Alaveteli using Vagrant + +<p class="lead"> + <a href="https://www.vagrantup.com">Vagrant</a> provides an easy method to set + up virtual development environments We bundle an example Vagrantfile in the + repository, which runs the + <a href="{{ site.baseurl}}docs/installing/script/">install script</a> for you. +</p> + +Note that this is just one of [several ways to install Alaveteli]({{ site.baseurl }}docs/installing/). + +The included steps will use vagrant to create a development environment +where you can run the test suite and the development server, and make +changes to the codebase. + +The basic process is to create a base virtual machine (VM), and then +provision it with the software packages and setup needed. The supplied +scripts will create you a Vagrant VM based on the server edition of +Ubuntu 12.04 LTS that contains everything you need to work on Alaveteli. + +1. Get a copy of Alaveteli from + <a href="{{ site.baseurl }}docs/glossary/#git" class="glossary__link">GitHub</a>: + + # on your machine + $ git clone git@github.com:mysociety/alaveteli.git + $ cd alaveteli + $ git submodule update --init + +2. Create the Vagrant VM. This will provision the system and can take some time + — sometimes as long as 20 minutes. + + $ vagrant --no-color up + +3. You should now be able to log in to the Vagrant guest OS with `ssh` and run + the test suite: + + $ vagrant ssh + + # You are now in a terminal on the virtual machine + $ cd /home/vagrant/alaveteli + $ bundle exec rake spec + + +4. Run the rails server: + + # in the virtual machine terminal + bundle exec rails server + +You can now visit the application in your browser (on the same machine that is +running Vagrant) at `http://10.10.10.30:3000`. + +If you need to stop the server, simply press **Ctl-C** within that shell. + +It's also possible to stop the server from a different terminal shell in the +Vagrant VM. Log in, find the process ID for the Alaveteli server (in the example +below, this is `1234`), and issue the `kill` command: + + $ vagrant ssh + + # now in a terminal on the virtual machine + $ cat /home/vagrant/alaveteli/tmp/pids/server.pid + 1234 + $ kill -2 1234 + +Alternatively, you can shut down the whole VM without deleting it with the +command <code>vagrant halt</code> +on the host command line. To start it up again, go to step 2, above — it +won't take so long this time, because the files are already in place. +See [the Vagrant documentation](https://docs.vagrantup.com/v2/) +for full instructions on using Vagrant. + +## What next? + +Check out the [next steps]({{ site.baseurl }}docs/installing/next_steps/). + +## Customizing the Vagrant instance + +The Vagrantfile allows customisation of some aspects of the virtual machine. See the customization options in the file [`Vagrantfile`](https://github.com/mysociety/alaveteli/blob/master/Vagrantfile#L30) at the top level of the Alaveteli repository. + +The options can be set either by prefixing the vagrant command, or by +exporting to the environment. + + # Prefixing the command + $ ALAVETELI_VAGRANT_MEMORY=2048 vagrant up + + # Exporting to the environment + $ export ALAVETELI_VAGRANT_MEMORY=2048 + $ vagrant up + +Both have the same effect, but exporting will retain the variable for the duration of your shell session. + diff --git a/docs/running/admin_manual.md b/docs/running/admin_manual.md index 77bacdf1b..7c112400c 100644 --- a/docs/running/admin_manual.md +++ b/docs/running/admin_manual.md @@ -6,9 +6,46 @@ title: Administrator's guide # Alaveteli administrator's guide <p class="lead"> - What is it like running an Alaveteli site? This guide explains what you can expect, and the types of problem that you might encounter. It includes examples of how mySociety manages their own <a href="/docs/glossary/#foi" class="glossary">Freedom of Information</a> site, <a href="https://www.whatdotheyknow.com">whatdotheyknow.com</a>. + What is it like running an Alaveteli site? This guide explains what you can + expect, and the types of problem that you might encounter. It includes + examples of how mySociety manages their own <a href="/docs/glossary/#foi" + class="glossary__link">Freedom of Information</a> site, <a + href="https://www.whatdotheyknow.com">whatdotheyknow.com</a>. </p> +In this guide: + +<ul class="toc"> + <li><a href="#whats-involved">What's involved?</a></li> + <li><a href="#user-support">User support</a> + <ul> + <li><a href="#dealing-with-email-thats-not-getting-through-to-the-authority">Dealing with email that's not getting through to the authority</a></li> + <li><a href="#requests-to-take-down-information">Requests to take down information</a></li> + <li><a href="#incorrectly-addressed">Incorrectly addressed</a></li> + <li><a href="#wants-advice">Wants advice</a></li> + <li><a href="#general-assistance-required">General assistance required</a></li> + <li><a href="#vexatious-users">Vexatious users</a></li> + <li><a href="#mail-import-errors">Mail import errors</a></li> + </ul> + <li><a href="#maintenance">Maintenance</a></li> + <ul> + <li><a href="#administrator-privileges-and-accessing-the-admin-interface">Administrator privileges and accessing the admin interface</a></li> + <li><a href="#removing-a-message-from-the-holding-pen">Removing a message from the 'Holding Pen'</a></li> + <li><a href="#rejecting-spam-that-arrives-in-the-holding-pen">Rejecting spam that arrives in the holding pen</a></li> + <li><a href="#creating-changing-and-uploading-public-authority-data">Creating, changing and uploading public authority data</a></li> + <li><a href="#banning-a-user">Banning a user</a></li> + <li><a href="#allowing-a-user-to-make-more-requests">Allowing a user to make more requests</a></li> + <li><a href="#batch-requests">Batch requests</a></li> + <li><a href="#resending-a-request-or-sending-it-to-a-different-authority">Resending a request or sending it to a different authority</a></li> + <li><a href="#hiding-a-request">Hiding a request</a></li> + <li><a href="#deleting-a-request">Deleting a request</a></li> + <li><a href="#hiding-an-incoming-or-outgoing-message">Hiding an incoming or outgoing message</a></li> + <li><a href="#editing-an-outgoing-message">Editing an outgoing message</a></li> + <li><a href="#hiding-certain-text-from-a-request-using-censor-rules">Hiding certain text from a request</a></li> + </ul> + </li> +</ul> + ## What's involved? The overhead in managing a successful FOI website is quite high. Richard, a @@ -20,7 +57,7 @@ WhatDoTheyKnow usually has about 3 active volunteers at any one time managing the support, plus a few other less active people who help out at different times. -Administration tasks can be split into **maintenance** and **user support**. +Administration tasks can be split into [**maintenance**]({{ site.baseurl }}docs/running/admin_manual/#maintenance) and [**user support**]({{ site.baseurl }}docs/running/admin_manual/#user-support). The boundaries of these tasks is in fact quite blurred; the main distinction is that the former happen exclusively through the web admin interface, whereas the latter are mediated by email directly with end users (but often result in @@ -45,7 +82,7 @@ During that week, the tasks broke down as follows: * 2 requests that have been marked as needing admin attention * 2 things marked as errors (message refused by server - spam, full mailbox, etc) to fix -### User interaction tasks +### User support tasks * 16 general, daily admin: i.e. things that resulted in admin actions on the site (bounces, misdelivered responses, etc) @@ -55,9 +92,9 @@ During that week, the tasks broke down as follows: * 3 requests to redact personal information * 2 requests to redact defamatory information -## Types of user interaction +## User support -There follows a breakdown of the most common kinds of user interaction. It's +There follows a breakdown of the most common kinds of user support. It's intended for use as a guide to the kind of policies and training that a support team might need to develop. @@ -249,13 +286,13 @@ Can be for many reasons, e.g. themselves * A reply has been automatically filed under the wrong request -## Vexatious users +### Vexatious users Some users persistently misuse the website. An alaveteli site should have a policy on banning users, for example giving them a first warning, informing them about moderation policy, etc. -## Mail import errors +### Mail import errors These are currently occurring at a rate of about two a month. Sometimes the root cause seems to be blocking in the database when two mails are received for @@ -270,9 +307,491 @@ the error sent to the site support address) in a file without the first "From" line, and piping the contents of that file into the mail handling script. e.g. ```cat missing_mail.txt | script/mailin``` -## Censor rules -Censor rules can be attached to a request or to a user and define bits of text +## Maintenance + +### Administrator privileges and accessing the admin interface + +The <a href="{{ site.baseurl }}docs/glossary/#admin" class="glossary__link">administrative interface</a> +is at the URL `/admin`. Only users who are +<a href="{{ site.baseurl }}docs/glossary/#admin" class="glossary__link">administrators</a> +can access the admin interface. + +To make a user an administrator on a brand new site, +[follow these steps]({{ site.baseurl }}docs/installing/next_steps/#create-a-superuser-admin-account). + +If you're already an administrator, you can grant other users administrator +privilege too. Go to `/admin/users` or click on **Users** at the top of +the admin. Find the user in the list, and click on the name to see the user +details. On that page, click **Edit**. Change the *Admin level* to “super” and +click **Save**. + +As well having access to the admin interface, users who are administrators also +have extra privileges in the main website front end. Administrators can: + + * categorise any request + * view items that have been hidden from the search + * follow "admin" links that appear next to individual requests and comments + +<div class="attention-box warning"> + It is possible completely to override the administrator authentication by + setting + <code><a href="{{ site.baseurl }}docs/customising/config/#skip_admin_auth">SKIP_ADMIN_AUTH</a></code> + to <code>true</code> in <code>general.yml</code>. Never do this, unless you + are working on a <a href="{{ site.baseurl }}docs/glossary/#development" + class="glossary__link">development</a> server. +</div> + +### Removing a message from the holding pen + +Alaveteli puts incoming messages (that is, +<a href="{{ site.baseurl }}docs/glossary/#reponse" class="glossary__link">responses</a>) +into the +<a href="{{ site.baseurl }}docs/glossary/#holding_pen" class="glossary__link">holding pen</a> +if their `To:` email addresses can't automatically be associated with a +<a href="{{ site.baseurl }}docs/glossary/#reponse" class="glossary__link">request</a>. + +The two most common reasons for this are: + + * the request has closed + * the email address was wrongly spelled (for example, the sender missed the last + character off the email address when they copied it) + +When this happens, the messages wait in the holding pen until an administrator +redelivers them to the correct request, or else deletes them. + +To do this, log into the +The <a href="{{ site.baseurl }}docs/glossary/#admin" class="glossary__link">admin interface</a> +at `/admin`. If there are any messages in the holding pen, you'll see this +message under the title *Things to do*: + +> Put misdelivered responses with the right request + +Click on that message — you'll see a list of all the messages that need +your attention. Click on any one of them to see the details. + +<div class="attention-box helpful-hint"> + If the message does not belong to any request, you can delete it instead. + Simply click on the <strong>Destroy Message</strong> button instead of + redelivering it. +</div> + +When you inspect a message, you may see a guess made by Alaveteli as to which +request the message belongs to. Check this request. If the guess is right +— the incoming email really is a response to that request — +the request's *title_url* will already be in the input box: click the +**Redeliver to another request** button. + +If there is not a guess, or Alaveteli's guess is wrong, look at the `To:` +address of the raw email and the contents of the message itself. You need +to figure out which request it belongs to. You can browse and search +requests in the admin interface by clicking **Requests** at the top of the +admin. When you have found the correct request, copy either its *id* or +its *url_title*. + +<div class="attention-box info"> + <p><strong>How to find a request's <em>id</em> or <em>url_title</em></strong></p> + <p> + A request's <em>id</em> is the number after <code>/show/</code> in the + admin interface's URL when you are looking at that request. + For example, if the URL is <code>/admin/request/show/118</code>, then the + <em>id</em> is <code>118</code>. + </p> + <p> + A request's <em>url_title</em> is the part after <code>/request/</code> + in your Alaveteli site's URL when you are looking at that request. + In the URL <code>/request/how_many_vehicles</code>, the + <em>url_title</em> is <code>how_many_vehicles</code>. + </p> +</div> + +Once you have identified the request the message belongs to, return to the +holding pen message page. Find the incoming message's "Actions" and paste the +request *id* or *url_title* into the text input. Click on the **Redeliver to +another request** button. + +The message will now be associated with the correct request. It is no longer +in the holding pen, and is shown instead on the public request page. + + +### Rejecting spam that arrives in the holding pen + +Alaveteli maintains a +<a href="{{ site.baseurl }}docs/glossary/#spam-address-list" class="glossary__link">spam address list</a>. +Any incoming message to an email address on that list +*that would otherwise be put in the holding pen* will be rejected and won't +appear in the admin. + +If you see spam messages in the +<a href="{{ site.baseurl }}docs/glossary/#holding_pen" class="glossary__link">holding pen</a>, +check if they are being sent to a *specific* email address. If they are, that +email address has become a "spam-target" and you should add it to the spam +address list. Thereafter, Alaveteli will automatically reject any messages that +come to that address. + +An email address that is not associated with a request (that is, one whose +messages end up in the holding pen) becomes a spam-target once it's been +harvested by spammers. There are several reasons why such an invalid address +might exist — perhaps it was mis-spelled in a manual reply, for example. +Our experience from running +<a href="{{ site.baseurl }}docs/glossary/#wdtk" class="glossary__link">WhatDoTheyKnow</a> +is that you can safely dismiss incoming email to such addresses once they have +been targeted in this way. Legitimate emails that arrive in the holding pen +tend to be unique errors (for example, missing the last character of the email +address due to a cut-and-paste mistake) and the nature of the lifecycle of +requests means they don't typically get used for spam until they are +effectively dead. + +To add an email address to the spam address list you need to copy it from an +incoming message and paste it into the spam addresss list. The easiest way to +do this is to click on **Summary** at the top of any admin page, and then click +on **Put misdelivered responses with the right requests** to see the contents +of the holding pen. + +<div class="attention-box info"> + If there are no messages in the holding pen, Alaveteli won't show you this + link. Great — there are no misdelivered responses needing your + attention right now! +</div> + +Inside the holding pen, you'll see the list of emails awaiting attention +— click on an email's subject line to see the whole message and its +details. Copy the `To:` email address, then click on the **Spam Addresses** +link under *Actions*. Paste the email address into the text input and click the +**Add Spam Address** button. + +You can see the spam address list (that is, all known spam-target email +addresses) at any time by going to the admin interface at `/admin/spam_addresses`. + +You can remove any address from the list by clicking the **Remove** button +next to it. Of course, this won't restore any messages that have been +rejected, but Alaveteli will not reject any new messages that are sent to +this address. + +Note that if you are seeing consistent spam email in your holding pen, you +should also consider implementing (or increasing) the anti-spam measures +running in your +<a href="{{ site.baseurl }}docs/glossary/#mta" class="glossary__link">MTA</a>. + +### Creating, changing and uploading public authority data + +There are three ways to change public authority data on your site: + + * *Create* — + You can create a new public authority in the admin interface. Go to **Authorities**, and click the **New Public Authority** button. + + * *Edit* — + Once an authority is created, you can update its email address or other + details by editing it in the admin interface. Go to **Authorities**, find + the authority you want to update, and click on **edit**. + + * *Upload* — + You can also create or edit more than one authority at the same time by + uploading a file containing the data in comma-separated values (CSV) + format. This works for new authorities as well as those that already exist + on your site. Go to **Authorities** and click the **Import from CSV** button. See the rest of this section for more about uploading. + +The upload feature is useful — especially when an Alaveteli site is first +set up — because it's common to collect data such as the contact details +for the public authorities in a spreadsheet. Alaveteli's upload feature makes it +easy to initially load this data onto the site. It also lets you update the +data if it changes after it's already been loaded. + +To use the data in the spreadsheet to update the bodies on your site, export +("save as") the spreadsheet as a CSV file. This is the file you can upload. + +The first line of your CSV file should start with `#` (this indicates that this +line does not contain data) and must list the column names for the data that +follows on the subsequent lines. Column names must: + + * be on the first line + * match expected names *exactly*, and include `name` and `request_email` + (see table below) + * appear in the same order as corresponding items in the lines of data that follow + +Most spreadsheet programs will produce a suitable CSV file for you, provided +that you carefully specify correct titles at the top of each column. Be sure to +use names exactly as shown — if Alaveteli encounters an +unrecognised column name, the import will fail. + +<table class="table"> + <tr> + <th>column name</th> + <th>i18n suffix?</th> + <th>notes</th> + </tr> + <tr> + <td><code>name</code></td> + <td><em>yes</em></td> + <td> + <em>This column <strong>must</strong> be present.</em><br> + The full name of the authority.<br> + If it matches an existing authority's name, that authority will be + updated — otherwise, this will be added as a new authority. + </td> + </tr> + <tr> + <td><code>request_email</code></td> + <td><em>yes</em></td> + <td> + <em>This column <strong>must</strong> be present, + but can be left empty.</em><br> + The email to which requests are sent + </td> + </tr> + <tr> + <td><code>short_name</code></td> + <td><em>yes</em></td> + <td>Some authorities are known by a shorter name</td> + </tr> + <tr> + <td><code>notes</code></td> + <td><em>yes</em></td> + <td>Notes, displayed publicly (may contain HTML)</td> + </tr> + <tr> + <td><code>publication_scheme</code></td> + <td><em>yes</em></td> + <td> + The URL of the authority's + <a href="{{ site.baseurl }}docs/glossary/#publication-scheme" class="glossary__link">publication scheme</a>, + if they have one + </td> + </tr> + <tr> + <td><code>disclosure_log</code></td> + <td><em>yes</em></td> + <td> + The URL of the authority's + <a href="{{ site.baseurl }}docs/glossary/#disclosure-log" class="glossary__link">disclosure log</a>, + if they have one + </td> + </tr> + <tr> + <td><code>home_page</code></td> + <td>no</td> + <td>The URL of the authority's home page</td> + </tr> + <tr> + <td><code>tag_string</code></td> + <td>no</td> + <td>separated tags with spaces</td> + </tr> +</table> + + * Existing authorities cannot be renamed by uploading: if you need to do + this, use the admin interface to edit the existing record first, and + change its name in the web interface. + * If the authority already exists (the `name` matches an existing authority's + name exactly), a blank entry leaves the existing value for that column + unchanged — that is, that item of data on your site will not be + changed. This means you only really need to include data you want to + update. + * Columns with "i18n suffix" can accept + <a href="{{ site.baseurl }}docs/glossary/#i18n" class="glossary__link">internationalised</a> + names. Add a full stop followed by the language code, for example: + `name.es` for Spanish (`es`). This *must* be a locale you've declared in + [`AVAILABLE_LOCALES`]({{ site.baseurl }}docs/customising/config/#available_locales). + If you don't specify an i18n suffix, the default language for your site is + assumed. + * You can specify a blank entry in the CSV file by having no character + between commas. + * If an entry contains a comma, enclose it in double quotes like this: + `"Comma, Inc"`. + * If an entry contains any double quotes, you must replace each of + them with two (so `"` becomes `""`) and also enclose the whole entry in + double quotes like this: `"In ""quotes"""` (which will be imported as `In + "quotes"`). + +For example, here's data for three authorities in CSV format ready for upload. +The first line defines the column names, then the next three lines contain the +data (one line for each authority): + + #name,short_name,short_name.es,request_email,notes + XYZ Library Inc.,XYZ Library,XYX Biblioteca,info@xyz.example.com, + Ejemplo Town Council,,Ayuntamiento de Ejemplo,etc@example.com,Lorem ipsum. + "Comma, Inc.",Comma,,comma@example.com,"e.g. <a href=""x"">link</a>" + +Note that, if Ejemplo Town Council already exists on the site, the blank entry +for `short_name` will leave the existing value for that column unchanged. + +To upload a CSV file, log into the admin and click on **Authorities**. Click on +**Import from CSV file**, and choose the file you've prepared. + +Specify **What to do with existing tags?** with one of these options: + + * *Replace existing tags with new ones* <br/> + For each authority being updated, all existing tags will be removed, and + replaced with the ones in your CSV file. + + * *Add new tags to existing ones* <br/> + Existing tags will be left unchanged, and the tags in your CSV file will + be added to them. + +You can add a **Tag to add entries to / alter entries for**. This tag will +be applied to every body that is imported from your CSV file. + +We always recommend you click **Dry run** first -- this will run through the +file and report the changes it will make in the database, *without actually +changing the data*. Check the report: it shows what changes would be made if +you really uploaded this data, followed by a message like this: + + Dry run was successful, real run would do as above. + +If you see nothing above that line, it means the dry run has resulted in no +proposed changes. + +If everything was OK when you ran the dry run, click **Upload** instead. This +will repeat the process, but this time it will make the changes to your +site's database. + +If you see an error like `invalid email`, either you really have mistyped an +email address, or (more likely) your CSV file does not have a `request_email` +column. + +#### Creating a spreadsheet of existing authorities + +You can easily create a spreadsheet containing the authorities that <em>already +exist</em> on your site. Combined with the upload feature described above, this +may be a more convenient way to update your data than editing it in the admin +interface. + +To export the existing authorities' data, go to your site's home page (not the +admin) and click <strong>View Authorities</strong>. Then click <strong>List of +all authorities (CSV)</strong> to get a CSV file. You can then make changes to +this file using a spreadsheet program and upload it as described above. + +You'll need to remove some columns that are not accepted by the import feature +and possibly rename some that are — see the column names above. +Also, note that by default the exported spreadsheet does not contain a +`request_email` column. If you want to update email addresses, you should +manually add a column to your spreadsheet with the heading `request_email` and +fill in a new email address for each authority you want to update. Authorities +with blank values in any column will keep their existing value for that +attribute. + +<div class="attention-box info"> +Alaveteli never includes authorities which have the tag <code>site_administration</code> when it exports authorities in CSV format. +If you're running a development server with the sample data, the single example +body called "Internal admin authority" has this tag, so if you click +<strong>List of all authorities (CSV)</strong>, you'll get a CSV file which +contains no data. You need to add your own authorities (without the +<code>site_administration</code> tag) before you can export them. +</div> + +### Banning a user + +You may wish to completely ban a user from the website (such as a spammer or troll for example). You need to log into the admin interface at `/admin`. On the top row of links, locate and click on ‘Users’. + +Find the user you wish to ban on the list and click on their name. Once on the user page, select ‘edit’. + +Enter some text in the in the ‘Ban text’ box to explain why they have been banned. Please be aware, this is publicly viewable from the users' account. Then click on save and the user will be banned. + +### Allowing a user to make more requests + +Alaveteli has a config setting <code><a href="{{ site.baseurl }}docs/customising/config/#max_requests_per_user_per_day">MAX_REQUESTS_PER_USER_PER_DAY</a></code>, +which determines the maximum number of requests that a normal user can +make in a day. If they try to make more than this number of requests +within a 24 hour period, they will see a message telling them that they +have hit the limit, and encouraging them to use the contact form if they +feel they have a good reason to ask for the request limit to be lifted. + +To lift the request limit for a particular user, go to the <a href="{{ site.baseurl }}docs/glossary/#admin" class="glossary__link">admin +interface</a>, click on **Users**, then click on the name of the user +you want to lift the request limit for. Click the **Edit** button. Tick +the checkbox **No rate limit**, and click the **Save** button. + +### Batch requests + +Sometimes a user may want to send the same request to more than one authority, which we call a batch request. By default, Alaveteli does not allow users to make batch requests. + +<div class="attention-box info"> +<p>We believe that batch requests can be abused — users can send poorly thought-out or vexatious requests, which will annoy authorities and damage the reputation of your site. However, well thought-out batch requests can be an extremely useful tool in collecting comparative data sets across types of authority, for example, all police forces.</p> +<p> +We recommend that you enable batch requesting for users who you notice making the same good request to multiple authorities. +</p> +<p> +Users can choose which authorities to include in a batch requests. They can even send a request to <em>every single authority</em> on your site. Only give this power to users that you trust. +</p> +</div> + +To enable batch requests on your site, first you must set +<code><a href="{{ site.baseurl }}docs/customising/config/#allow_batch_requests">ALLOW_BATCH_REQUESTS</a></code> +to <code>true</code> in <code>general.yml</code>. + +This does not allow anyone to make batch requests yet. You must still +enable this for each user on an individual basis. To do this, go to the +<a href="{{ site.baseurl }}docs/glossary/#admin" +class="glossary__link">admin interface</a>, click on **Users**, then +click on the name of the user who wants to make batch requests. Click +the **Edit** button. Tick the checkbox **Can make batch requests**, and +click the **Save** button. + +If you've enabled batch requests for a user, when they start to make a +request, in addition to the box where they can select an authority, they +will see a link to "make a batch request". When the request is sent, +Alaveteli will make a request page for this request for each authority, +as if the user had made individual requests. + +### Resending a request or sending it to a different authority + +If you have corrected the email address for an authority, you can resend +an existing request to that authority to the new email address. Alternatively, +a user may send a request to the wrong authority. In that situation, you can +change the authority on the request and then resend it to the correct authority. +For instructions, see +[resending a request or sending it to a different authority]({{ site.baseurl }}docs/running/requests/#resending-a-request-or-sending-it-to-a-different-authority). + + +### Hiding a request + +If a request contains vexatious or inappropriate content, is libellous, or is +not a valid +<a href="{{ site.baseurl }}docs/glossary/#foi" class="glossary__link">Freedom of Information</a> +request at all, you may want to hide it. A hidden request is still visible to +you and the other administrators, and (optionally) the requester themselves. +For instructions, see +[hiding a request]({{ site.baseurl }}docs/running/requests/#hiding-a-request). + +Responses to a hidden request will be accepted in the normal way, but because +they are added to the request's page, they too will be hidden. + +### Deleting a request + +You can delete a request from the site. For instructions, see +[deleting a request]({{ site.baseurl }}docs/running/requests/#deleting-a-request). + +Responses to a deleted request will be sent to the holding pen. + +### Hiding an incoming or outgoing message + +You may need to hide a particular incoming or outgoing message from a +public request page, perhaps because someone has included personal +information in it. You can do this from the message's page in the admin +interface. You can get to a message's admin page either by following the +links from the "Outgoing messages" or "Incoming messages" sections of +the request's admin page, or directly from the public request page by +clicking on the 'admin' link on the message itself. Once you are on the +message's admin page, you can change it's prominence. Set the prominence +to 'hidden' to hide it from everyone except site admins, or to +'requester_only' to allow it to be viewed by the requester (and by site +admins). If you can, add some text in the box 'Reason for prominence'. +This will be displayed as part of the information that will appear on +the request page where the message used to be, telling people that it +has been hidden. + +### Editing an outgoing message + +You may find there is a need to edit an outgoing message because the requester has accidentally included personal information that they don't want to be published on the site. You can either follow one of the 'admin' links from the public request page on the site, or find the request from the admin interface by searching under 'Requests'. + +Scroll down to the 'Outgoing Messages' section, and click on 'Edit'. + +Then on the next page you will be able to edit the message accordingly and save it. The edited version will then appear on the Alaveteli website, although an unedited version will have been sent to the authority. + + +### Hiding certain text from a request using censor rules + +Censor rules can be attached to a request or to a user. These rules define bits of text to be removed (either from the request (and all associated files e.g. incoming message attachments) or from all requests associated with a user), and some replacement text. In binary files, the replacement text will always be a series @@ -297,5 +816,22 @@ hanging the application altogether), so please: * Restrict your use of them to cases that can't otherwise be easily covered. * Keep them as simple and specific as possible. - +<strong>To attach a censor rule to a request</strong>, go to the admin page for the +request, scroll to the bottom of the page, and click the "New censor +rule (for this request only)" button. On the following page, enter the +text that you want to replace e.g. 'some private info', the text you +wish to replace it with e.g. '[private info has been hidden]', and a +comment letting other admins know why you have hidden the information. + +<strong>To attach a censor rule to a user</strong>, so that it will be applied to all +requests that the user has made, go to the user page in the admin +interface. You can do this either by clicking on the admin heading +'Users' and browsing or searching to find the user you want, or by +following an 'admin' link for the user from the public interface. One +you are on the admin page for the user, scroll to the bottom of the page +and click the 'New censor rule' button. On the following page, enter the +text that you want to replace e.g. 'my real name is Bruce Wayne', the +text you wish to replace it with e.g. '[personal information has been +hidden]', and a comment letting other admins know why you have hidden +the information. diff --git a/docs/running/categories_and_tags.md b/docs/running/categories_and_tags.md new file mode 100644 index 000000000..8f36ae472 --- /dev/null +++ b/docs/running/categories_and_tags.md @@ -0,0 +1,200 @@ +--- +layout: page +title: Categories & tags +--- + +# Categories and tags for authorities + +<p class="lead"> + + Use tags to arrange + <a href="{{ site.baseurl }}docs/glossary/#authority" + class="glossary__link">authorities</a> into categories, or to associate + related authorities with each other. This helps your users find the right + authority for the + <a href="{{ site.baseurl }}docs/glossary/#request" class="glossary__link">request</a> + (or <a href="{{ site.baseurl }}docs/glossary/#response" class="glossary__link">response</a>) + they are interested in. +</p> + +## Categories & category headings + +<div class="attention-box info"> +Admin interface introduced in Alaveteli version 0.20 +</div> + +Alaveteli lets you organise your authorities into *categories*. Categories can +themselves belong to *category headings*. For example, some of the categories +and headings on +<a href="{{ site.baseurl }}docs/glossary/#wdtk" class="glossary__link">WhatDoTheyKnow</a>'s +<a href="https://www.whatdotheyknow.com/body/list/all">View authorities</a> page look like this: + +> * **Media and culture** +> * Media +> * Museums and galleries +> * **Military and security services** +> * Armed Forces +> * Military colleges +> * Security services +> * **Emergency services** +> * Police forces +> * Fire & rescue services + + +In this example, "Emergency services" is a heading which contains the categories +"Police forces" and "Fire & rescue services". + +Tags are simply searchable words that you can add to an authority. Nominate a +tag for each category: any authority which has that tag is automatically +assigned to the category. For example, if the tag `police` is associated with +the category "Police forces", any authority which has the tag `police` will +appear in that category. + +Make sure you choose good category headings and names, because they help your +users find the specific authorities they are looking for. + +<div class="attention-box info"> + Try to use simple but descriptive words for tags. Tags cannot contain spaces + (use an underscore if you need to, <code>like_this</code>). + Remember that tags will be seen and used by the public (for example, in the + <a href="{{ site.baseurl }}docs/glossary/#advanced-search" class="glossary__link">advanced search</a>). +</div> + +### Adding a new category + +In the admin interface, click on **Categories**. It's a good idea to create +category headings first (but don't worry if you don't — you can change +them later). + +Click on **New category heading**, enter a name (for example, "Emergency +services") and click **Create**. + +To create a category, click on **New category**. As well as providing a title +and a description, you must enter a category tag. Any authority with this tag +will be assigned to this category. + +Select the checkbox next to the category heading under which you want this +category to be listed. It's common for a category to be under just one heading. +But sometimes it makes sense for a category to go under more than one, so you +can select multiple checkboxes if you need to. + +Click **Save** to create the category. + +### Editing or deleting a category + +Click on **Categories** then find the category in the list (if the category is +under a heading, you may need to click on the heading's chevron to expand the +list to show it). Click the name of the category to select it. You can edit it +and click **Save**. + +If you want to destroy a category, go to edit it but instead of saving it, +click on the **Destroy** button at the bottom of the page. This does not +delete any authorities in that category — they simply no longer belong to +it. + +## Special tags + +Some tags are special. Alaveteli behaves differently when an authority has one +of these tags. + +<table class="table"> + <tr> + <th> + Tag + </th> + <th> + Effect + </th> + </tr> + <tr> + <td> + <code>site_administration</code> + </td> + <td> + This is a test/dummy authority. It is not displayed to the public on your + main site, and it is not included when you + <a href="{{ site.baseurl }}docs/running/admin_manual/#creating-changing-and-uploading-public-authority-data">export authorities in CSV format</a>. + </td> + </tr> + <tr> + <td> + <code>defunct</code> + </td> + <td> + This authority no longer operates: new requests cannot be sent to an + authority with this tag. + </td> + </tr> + <tr> + <td> + <code>not_apply</code> + </td> + <td> + <a href="{{ site.baseurl }}docs/glossary/#foi" class="glossary__link">Freedom of Information</a> + law does not apply to this authority: new requests cannot be sent to an + authority with this tag. + </td> + </tr> + <tr> + <td> + <code>eir_only</code> + </td> + <td> + <em>Custom example:</em> (see below)<br> + On our UK installation of Alaveteli, + <a href="{{ site.baseurl }}docs/glossary/#wdtk" class="glossary__link">WhatDoTheyKnow</a>, + this tag indicates that the authority is subject to an alternative law + (Environment Information Regulations, rather than the Freedom of + Information), which means Alaveteli must change the wording of these + requests appropriately. + </td> + </tr> + <tr> + <td> + <code>school</code> + </td> + <td> + <em>Custom example:</em> (see below)<br> + <a href="{{ site.baseurl }}docs/glossary/#wdtk" class="glossary__link">WhatDoTheyKnow</a> + applies a different definition of "late" if an authority has the <code>school</code> tag. + </td> + </tr> +</table> + +## Custom tags and custom behaviour + +You can add any tag you want — they don't have to be associated with +categories. + +If you are a developer, and you want to add special behaviour to your site +based on your own tags, you need to add custom code, which should probably go +in your own +<a href="{{ site.baseurl}}docs/glossary/#theme" class="glossary__link">theme</a>. +For example, in the UK, schools are granted special concession in the law to allow for +requests that are made out of term-time. Alaveteli handles this by using the +[`SPECIAL_REPLY_VERY_LATE_AFTER_DAYS`]({{ site.baseurl }}docs/customising/config/#special_reply_very_late_after_days) +config value if the authority has the `school` tag. +See +[`is_school?`](https://github.com/mysociety/alaveteli/blob/f0bbeb4abf4bf07e5cfb46668f39bbff72ed7210/app/models/public_body.rb#L391) +and +[`date_very_overdue_after`](https://github.com/mysociety/alaveteli/blob/81b778622ed47e24a2dea59c0529d1f928c68a58/app/models/info_request.rb#L752) +for the source code. + +## Searching with tags + +Alaveteli's +<a href="{{ site.baseurl }}docs/glossary/#advanced-search" class="glossary__link">advanced search</a> +feature (which is available to all your users) can search for specific tags. So +if you add useful tags and publicise them, your users can use them to find +related authorities. For example, see the <a +href="https://www.whatdotheyknow.com/advancedsearch">advanced search on +WhatDoTheyKnow</a> to see this at work. + +You can add reference numbers or specific values to tags using a colon. On +<a href="{{ site.baseurl }}docs/glossary/#wdtk" class="glossary__link">WhatDoTheyKnow</a> +we tag all authorities that are charities with the tag `charity:123456` (where +123456 is the authority's registered charity number). + + + + diff --git a/docs/running/holding_pen.md b/docs/running/holding_pen.md new file mode 100644 index 000000000..5b7f08bec --- /dev/null +++ b/docs/running/holding_pen.md @@ -0,0 +1,101 @@ +--- +layout: page +title: The holding pen +--- + +# The holding pen + +<p class="lead"> + + The <em>holding pen</em> is where Alaveteli puts any incoming + <a href="{{ site.baseurl }}docs/glossary/#response" class="glossary__link">responses</a> + that can't be matched to a + <a href="{{ site.baseurl }}docs/glossary/#request" class="glossary__link">request</a> + automatically. +</p> + + +Alaveteli works by emailing requests to the correct target +<a href="{{ site.baseurl }}docs/glossary/#authority" class="glossary__link">authority</a>. +That email message is sent from a unique email address — that is, an +email address that is associated with that single request (technically, +Alaveteli hashes the request ID to generate a unique address and uses this as +the `Reply-to:` address). + +So whenever an authority replies (by email) to a request that Alaveteli has +sent, that response will be addressed to that request's unique email address. +The email is received by your installation's +<a href="{{ site.baseurl}}docs/glossary/#mta" class="glossary__link">MTA</a>, +and is passed on to Alaveteli. In this way, incoming messages are easily +matched with the request they are responses to — this is important +because your site displays the responses underneath their original request, on +the request's page. + +Normally, this works fine. But sometimes things go wrong, and a message comes +in that can't be matched with a request. When this happens, Alaveteli puts the +message in the +<a href="{{ site.baseurl }}docs/glossary/#holding_pen" class="glossary__link">holding +pen </a>. + +Messages wait in the holding pen until an +<a href="{{ site.baseurl }}docs/glossary/#super" class="glossary__link">administrator</a> +redelivers them to the correct request, or else deletes them. + +## Why messages end up in the holding pen + +There are several reasons why a message might end up in the holding pen: + +* **the authority "broke" the reply-to email**<br> + This can happen if the authority replies "by hand" to the incoming email — + for example if the person at the authority accidentally loses the first + letter of the email address when they copy-and-paste it. Or if they copy + it manually and simply get it wrong. + +* **there's something unusual about the way it was sent**<br> + For example, if it was delivered here because the address is in the `Bcc:` + field, and is not the `To:` address. + +* **a partial email address may have been guessed**<br> + Someone guesses an email address which Alaveteli doesn't recognise. Perhaps + they have misunderstood how the addresses are formed, or maybe it's a + deliberate attempt to send spam. + +* **the response is to a request that has been deleted**<br> + If you [delete a request]({{ site.baseurl }}docs/running/requests/#deleting-a-request), + Alaveteli cannot deliver responses to it. + +* **the response has been rejected and rejections are set to go to the holding pen**<br> + Incoming mail that is correctly addressed but not accepted for the request + goes into the holding pen if the request's `handle_rejected_responses` + behaviour is set to `holding_pen` (rather than bouncing the email back to + the sender, or simply deleting it). Responses may be rejected for various + reasons — for example, if a response is sent from an unrecognised + email address for a request whose *Allow new responses from* setting is + `authority_only`. See instructions on + [how to manage requests]({{site.baseurl}}docs/running/requests/) for details. + +## What to do: redeliver or delete + +You need to be an +<a href="{{ site.baseurl }}docs/glossary/#super" class="glossary__link">administrator</a> +to modify the holding pen. + +There are two things you can do to a message in the holding pen: + + * **find the right request, and redeliver the message**<br> + Alaveteli tries to guess the right request to help you, so sometimes + you can just accept its suggestion. + + * **delete the message**<br> + If the message is not a response, you can delete it. + +For instructions, see +[removing a message from the holding pen]({{ site.baseurl }}docs/running/admin_manual/#removing-a-message-from-the-holding-pen). + +If the `To:` address does not belong to a valid request and the message is +clearly spam you can add that email address to Alaveteli's +<a href="{{site.baseurl}}#spam-address-list" class="glossary__link">spam address list</a>. +Subsequent messages to that address will be automatically rejected — for +instructions see +[rejecting spam that arrives in the holding pen]({{ site.baseurl }}docs/running/admin_manual/#rejecting-spam-that-arrives-in-the-holding-pen). + diff --git a/docs/running/index.md b/docs/running/index.md index 90461fb3e..bbf30d3b9 100644 --- a/docs/running/index.md +++ b/docs/running/index.md @@ -11,18 +11,20 @@ title: Running </p> Alaveteli is not just software. To run a successful -<a href="{{ site.baseurl }}docs/glossary/#foi" class="glossary">Freedom of Information</a> +<a href="{{ site.baseurl }}docs/glossary/#foi" class="glossary__link">Freedom of Information</a> site, you need to make sure day-to-day tasks get done too. Most Alaveteli sites are run by a team who allocate some time every day to user support and generally keeping the project up to date. -* the [administrator's guide]({{ site.baseurl }}docs/running/admin_manual/) describes +* The [administrator's guide]({{ site.baseurl }}docs/running/admin_manual/) describes what you need to do and know to run your site -* we've prepared a checklist of +* The [redaction guide]({{ site.baseurl }}docs/running/redaction/) includes some + examples of Alaveteli's ability to redact sensitive information + +* We've prepared a checklist of [things to consider]({{ site.baseurl }}docs/running/server/) when setting up your production server -* see the [different states a request can be in]({{ site.baseurl }}docs/running/states/) diff --git a/docs/running/redaction.md b/docs/running/redaction.md new file mode 100644 index 000000000..6ab8fed86 --- /dev/null +++ b/docs/running/redaction.md @@ -0,0 +1,135 @@ +--- +layout: page +title: Redacting Sensitive Information +--- + +# Redacting Sensitive Information + +In some countries, local requirements mean that requests need to contain personal information such as the address or ID number of the person asking for information. Usually requesters do not want this information to be displayed to the general public. + +Alaveteli has some ability to deal with this through the use of <a href="{{site.baseurl}}docs/glossary/#censor-rule" class="glossary__link">Censor Rules</a>. + +The [theme](https://github.com/mysociety/derechoapreguntar-theme) we'll use as an example requires a National Identity Card Number and what's known as General Law in Nicaragua (Date of Birth, Domicile, Occupation and Marital Status). + + + +## Identity Card Number + +We'll start off by looking at the National Identity Card Number (ID Number from here). Its a good example of something that is relatively easy to redact. It's unique for each user, and it has a specified format to match against. + +To send the ID Number to the authority we'll override the [initial request template](https://github.com/mysociety/alaveteli/blob/master/app/views/outgoing_mailer/initial_request.text.erb) (code snippet shortened): + + <%= raw @outgoing_message.body.strip %> + + ------------------------------------------------------------------- + + <%= _('Requestor details') %> + <%= _('Identity Card Number') %>: <%= @user_identity_card_number %> + +When a request is made the user's ID Number is now added to the footer of the outgoing email. + + + +At this point we haven't added any <a href="{{site.baseurl}}docs/glossary/#censor-rule" class="glossary__link">Censor Rules</a>. When the authority replies it is unlikely that the responder will remove the quoted section of the email: + + + +We could add a <a href="{{site.baseurl}}docs/glossary/#censor-rule" class="glossary__link">Censor Rule</a> for the individual request, but as every request will contain a user's ID Number its better to add some code to do do it automatically. + +To illustrate this we'll patch the `User` model with a callback that creates a <a href="{{site.baseurl}}docs/glossary/#censor-rule" class="glossary__link">Censor Rule</a> when the user is created and updated. + + # THEME_ROOT/lib/model_patches.rb + User.class_eval do + after_save :update_censor_rules + + private + + def update_censor_rules + censor_rules.where(:text => identity_card_number).first_or_create( + :text => identity_card_number, + :replacement => _('REDACTED'), + :last_edit_editor => THEME_NAME, + :last_edit_comment => _('Updated automatically after_save') + ) + end + end + +You can see the new <a href="{{site.baseurl}}docs/glossary/#censor-rule" class="glossary__link">Censor Rule</a> in the admin interface: + + + +Now the ID Number gets redacted: + + + +It also gets redacted if the public body use the ID Number in the main email body: + + + +A <a href="{{site.baseurl}}docs/glossary/#censor-rule" class="glossary__link">censor rule</a> added to a user only gets applied to correspondence on requests created by that user. It does not get applied to annotations made by the user. + +**Warning:** Redaction in this way requires the sensitive text to be in exactly the same format as the <a href="{{site.baseurl}}docs/glossary/#censor-rule" class="glossary__link">Censor Rule</a>. If it differs even slightly, the redaction can fail. If the public body was to remove the hyphens from the number it would not be redacted: + + + +**Warning:** Alaveteli also attempts to redact the text from any attachments. It can only do this if it can find the exact string, which is often not possible in binary formats such as PDF or Word. + +Alaveteli can usually redact the sensitive information when converting a PDF or text based attachment to HTML: + + + +This PDF does not contain the string in the raw binary so the redaction is _not_ applied when downloading the original PDF document: + + + +## General Law + +The General Law information is much harder to automatically redact. It is not as structured, and the information is unlikely to be unique (e.g. Domicile: London). + +We'll add the General Law information to the [initial request template](https://github.com/mysociety/alaveteli/blob/master/app/views/outgoing_mailer/initial_request.text.erb) in the same way as the ID Number: + + <%= _('Requestor details') %>: + <%-# !!!IF YOU CHANGE THE FORMAT OF THE BLOCK BELOW, ADD A NEW CENSOR RULE!!! -%> + =================================================================== + # <%= _('Name') %>: <%= @user_name %> + # <%= _('Identity Card Number') %>: <%= @user_identity_card_number %> + <% @user_general_law_attributes.each do |key, value| %> + # <%= _(key.humanize) %>: <%= value %> + <% end %> + =================================================================== + +Note that the information is now contained in a specially formatted block of text. + + + +This allows a <a href="{{site.baseurl}}docs/glossary/#censor-rule" class="glossary__link">Censor Rule</a> to match the special formatting and remove anything contained within. This <a href="{{site.baseurl}}docs/glossary/#censor-rule" class="glossary__link">Censor Rule</a> is global, so it will act on matches in all requests. + + # THEME_ROOT/lib/censor_rules.rb + # If not already created, make a CensorRule that hides personal information + regexp = '={67}\s*\n(?:[^\n]*?#[^\n]*?: ?[^\n]*\n){3,10}[^\n]*={67}' + + unless CensorRule.find_by_text(regexp) + Rails.logger.info("Creating new censor rule: /#{regexp}/") + CensorRule.create!(:text => regexp, + :allow_global => true, + :replacement => _('REDACTED'), + :regexp => true, + :last_edit_editor => THEME_NAME, + :last_edit_comment => 'Added automatically') + end + + + +**Warning:** Redacting unstructured information is a very fragile approach, as it relies on authorities always quoting the entire formatted block. + +In this case the authority has revealed the user's Date of Birth and Domicile: + + + +Its really difficult to add a <a href="{{site.baseurl}}docs/glossary/#censor-rule" class="glossary__link">Censor Rule</a> to remove this type of information. One suggestion might be to remove all mentions of the user's Date of Birth, but you would have to account for [every type of date format](http://en.wikipedia.org/wiki/Calendar_date#Date_format). Likewise, you could redact all occurrences of the user's Domicile, but if they a question about their local area (very likely) the request would become unintelligible. + + + +The redaction has been applied but there is no way of knowing the context that the use of the sensitive word is used. + + diff --git a/docs/running/requests.md b/docs/running/requests.md new file mode 100644 index 000000000..e554fe763 --- /dev/null +++ b/docs/running/requests.md @@ -0,0 +1,355 @@ +--- +layout: page +title: Managing requests +--- + +# Managing Requests + + +<p class="lead"> + Alaveteli makes it easy for a user to make a + <a href="{{ site.baseurl }}docs/glossary/#request" class="glossary__link">request</a>. + As an + <a href="{{ site.baseurl }}docs/glossary/#super" class="glossary__link">administrator</a>, + there are some things about that request you can change once it’s been created. +</p> + +A request is automatically created when a user submits and (where necessary) +confirms it. Alaveteli sends it to the +<a href="{{ site.baseurl }}docs/glossary/#authority" class="glossary__link">authority</a> +responsible and handles any +<a href="{{ site.baseurl }}docs/glossary/#response" class="glossary__link">responses</a>. +Usually this process runs without needing any intervention from an +administrator. But sometimes you'll want to change some aspect of the request, +or the way Alaveteli is handling it. + +<ul class="toc"> + <li><a href="#what-state-is-the-request-in">What state is the request in?</a></li> + <li><a href="#old-requests-6-months-without-activity">Old requests (6+ months without activity)</a></li> + <li><a href="#changing-things-about-a-request">Changing things about a request</a></li> + <li><a href="#resending-a-request-or-sending-it-to-a-different-authority">Resending a request or sending a request to a different authority</a></li> + <li><a href="#hiding-a-request">Hiding a request</a></li> + <li><a href="#deleting-a-request">Deleting a request</a></li> +</ul> + +## What state is the request in? + +Every request moves through a series of +<a href="{{ site.baseurl }}docs/glossary/#state" class="glossary__link">states</a>, +indicating its progress. Usually a new request will be in the +`waiting_response` state until something happens to change that — for +example, a response is received. + +However, states can't always be set automatically, because they require a +decision to be made on what kind of answer the authority provided in the +response. For states like this, Alaveteli invites the original requester to +describe it — for example, when a response is received they can change +the state to `successful`, `partially_successful` or `not_held` (if the +authority replied to say they don't have the information requested). + +<div class="attention-box info"> + If a request has been waiting for over three weeks for the original + requester to describe it but has still not been described, Alaveteli + lets <em>anyone</em> classify it. +</div> + +Internally, Alaveteli does not just record the "described state" of a request, +but also notices if anything has happened since it was last described and +sets its "awaiting description" status appropriately. + + +## Old requests (6+ months without activity) + +If there is no activity on a request for six months, Alaveteli automatically +changes that request's **Allow new responses from...** setting to `authority_only`. +Any responses will be rejected unless they are from the authority to which the +request was originally made. + +If a further six months pass with no activity, that is, a year has passed +without any updates, the request's **Allow new responses from...** setting +becomes `nobody`. All responses to this request will be rejected. The request +is effectively closed. + +By default, any rejected responses will be bounced with a message that explains +that the request has been closed because it is old, with a suggestion that the +sender can email your site's administrators directly. + +You can can stop this behaviour by changing the **Allow new responses from...** +setting back to `normal` at any time. Alternatively, you can change the way +rejected messages are handled (for example, sending such responses to the +holding pen instead of bouncing them) with the request's **Handle rejected +responses** setting. + +See [changing things about a request](#changing-things-about-a-request) below +for how to change these settings. + + +## Changing things about a request + +To change any of these settings, go to the +<a href="{{ site.baseurl }}docs/glossary/#admin" class="glossary__link">admin interface</a>, +click on **Requests**, then click on the title of the request you want to affect. +Click the **Edit metadata** button. + +<table class="table"> + <tr> + <th> + What you can change + </th> + <th> + Details + </th> + </tr> + <tr> + <td> + Title + </td> + <td> + The <em>title</em> is shown on the request’s page, but is also used + in the URL (the text is changed to lower case, punctuation is removed + and, if necessary, a number is added for disambiguation — this is + called the “slug”). + <p> + Note that changing the title changes the URL, because the slug changes + — this means any links to the <em>old</em> URL will no longer + work, and will return a 404 (file not found) error. + </p> + </td> + </tr> + <tr> + <td> + Who can see it? + </td> + <td> + Change the <strong>Prominence</strong> setting to one of: + <ul> + <li><code>normal</code></li> + <li> + <code>backpage</code>: request can be seen by anyone (by visiting + its URL, for example) but does not appear in lists, or search results + </li> + <li> + <code>requester_only</code>: request only visible to the person who + made the request + </li> + <li> + <code>hidden</code>: request is never shown (except to administrators) + </li> + </ul> + <br> + </td> + </tr> + <tr> + <td> + Who can respond? + </td> + <td> + The <strong>Allow new responses from...</strong> setting can be one of: + <ul> + <li><code>anybody</code></li> + <li> + <code>authority_only</code>: responses are allowed if they come + from the authority to which the request was sent, or from any domain + from which a a response has <em>already</em> been accepted + </li> + <li> + <code>nobody</code>: no responses are allowed on this request + </li> + </ul> + <p> + Any response from a sender who has been disallowed by this + setting will be rejected (see next entry). + </p> + <p> + This setting may change automatically when + <a href="#old-requests-6-months-without-activity">the request gets old</a>. + </p> + </td> + </tr> + <tr> + <td> + What happens to rejected responses? + </td> + <td> + The <strong>Handle rejected responses...</strong> setting specificies + what happens to responses that are not allowed (see previous entry): + <ul> + <li> + <code>bounce</code>: responses are sent back to their sender + </li> + <li> + <code>holding pen</code>: responses are put in the + <a href="{{ site.baseurl }}docs/glossary/#holding_pen" class="glossary__link">holding pen</a> + for an administrator to deal with + </li> + <li> + <code>blackhole</code>: responses are destroyed by being sent to a + <a href="{{ site.baseurl }}docs/glossary/#blackhole" class="glossary__link">black hole</a> + </li> + </ul> + </td> + </tr> + <tr> + <td> + What state is it in? + </td> + <td> + See <a href="{{ site.baseurl }}docs/customising/states/">more about + request states</a>, which can be customised for your installation. + <p> + You can force the state of the request by choosing it explicitly. + Change the <strong>Described state</strong> setting. + </p> + <p> + You may also need to set <strong>Awaiting description</strong> if, + having changed the state, you want the original requester to update the + description. For example, if the state depends on the information + within the response, and you want the requester to classify it — + see + <em><a href="#what-state-is-the-request-in">What state is the request in?</a></em> + above. + </p> + </td> + </tr> + <tr> + <td> + Are comments allowed? + </td> + <td> + The <strong>Are comments allowed?</strong> setting simply you choose to + allow or forbid annotations and comments on this request. + <p> + Note that this won’t hide any annotations that have already + been left on the reques — it only prevents users adding new ones. + </p> + </td> + </tr> + <tr> + <td> + Tags (search keywords) + </td> + <td> + Enter tags, separated by spaces, that are associated with this request. + A tag can be either a simple keyword, or a key-value pair (use a colon as + the separator, like this: <code>key:value</code>). + <p> + Tags are used for searching. Users and administators both benefit if + you tag requests with useful keywords, because it helps them find + specific requests — especially if your site gets busy and there + are very many in the database. + </p> + <p> + Although it’s a little more complex than tags on requests, + <a href="{{ site.baseurl }}docs/glossary/#category" class="glossary__link">categories</a> + also use tags: + see + <a href="{{ site.baseurl }}docs/running/categories_and_tags/">more about tags</a> + for a little more information. + </p> + </td> + </tr> +</table> + +## Resending a request or sending it to a different authority + +If you have corrected the email address for an authority, you can resend +an existing request to that authority to the new email address. Alternatively, +a user may send a request to the wrong authority. In that situation, you can +change the authority on the request and then resend it to the correct authority. + +To resend a request, go to +the <a href="{{ site.baseurl }}docs/glossary/#admin" +class="glossary__link">admin interface</a>, click on **Requests**, then +click on the name of the request you want to change. Go to the **Outgoing messages** heading. Click the chevron next to the first outgoing message, which is the initial request. A panel of information about that message will appear. Click on the **Resend** button. + +To send a request to a different authority, go to +the <a href="{{ site.baseurl }}docs/glossary/#admin" +class="glossary__link">admin interface</a>, click on **Requests**, then +click on the name of the request you want to change. In the **Request +metadata** section, there is a line which shows the authority. Click the +**move...** button next to it. Enter the **url_name** of the authority +that you want to send the request to. + +<div class="attention-box info"> +Users, requests and authorities all have <strong>url_names</strong>. This can be found in the metadata section of their admin page. The <code>url_name</code> makes up the last part of the URL for their public page. So, for a request with the <code>url_name</code> “example_request”, the public page URL will be <code>/request/example_request</code>. +</div> + +Now click the **Move request to +authority** button. You will see a notice at the top of the page telling +you that the request has been moved. You can now resend the request as above. + + +## Hiding a request + +You can hide an entire request. Typically you do this if it's not a valid +Freedom of Information request (for example, a request for personal +information), or if it is vexatious. + +Go to the <a href="{{ site.baseurl }}docs/glossary/#admin" class="glossary__link">admin interface</a>, +click on **Requests**, then click on the title of the request you want. You can +hide it in one of two ways: + + * [Hide the request and notify the requester](#hide-the-request-and-notify-the-requester) + * [Hide the request without notifying the requester](#hide-the-request-without-notifying-the-requester) + +Responses to a hidden request will be accepted in the normal way, but because +they are added to the request's page, they too will be hidden. + +### Hide the request and notify the requester + +Scroll down to the *Actions* section of the request's admin page. +Choose one of the options next to **Hide the request and notify the user:** + + * Not a valid FOI request + * A vexatious request + +Choosing one of these will reveal an email form. Customise the text of the +email that will be sent to the user, letting them know what you've done. When +you're ready, click the **Hide request** button. + +### Hide the request without notifying the requester + +<div class="attention-box helpful-hint"> + As well as hiding the request from everyone, you can also use this method if + you want to make the request only visible to the requester. +</div> + +In the *Request metadata* section of the request's admin page, click the +**Edit metadata** button. Change the *Prominence* value to one of these: + + * `requester_only`: only the requester can view the request + * `hidden`: nobody can see the request, except administrators. + +<div class="attention-box warning"> + If you want to hide the request, do not chooose <code>backpage</code> + as the prominence. The <code>backpage</code> option stops the request + appearing in lists and searches so that it is effectively only visible + to anyone who has its URL — but it <em>does not hide</em> the request. +</div> + +When you're ready, click the **Save changes** button at the bottom of the +*Edit metadata* section. No email will be sent to the requester to notify +them of what you've done. + + +## Deleting a request + +You can delete a request entirely. Typically, you only need to do this if +someone has posted private information. If you delete a request, any responses that it has already received will be +destroyed as well. + +<div class="attention-box warning"> + Deleting a request destroys it. There is no “undo” operation. + If you're not sure you want to do this, perhaps you should + <a href="#hiding-a-request">hide the request</a> instead. +</div> + +Go to the <a href="{{ site.baseurl }}docs/glossary/#admin" class="glossary__link">admin interface</a>, +click on **Requests**, then click on the title of the request you want to delete. +Click the **Edit metadata** button. Click on the red **Destroy request entirely** +button at the bottom of the page. + +Responses to a deleted request will be sent to the +<a href="{{ site.baseurl }}docs/glossary/#holding_pen" class="glossary__link">holding pen</a>. + + diff --git a/docs/running/security.md b/docs/running/security.md new file mode 100644 index 000000000..a22c4d636 --- /dev/null +++ b/docs/running/security.md @@ -0,0 +1,36 @@ +--- +layout: page +title: Security & Maintenance +--- + +# Security & Maintenance + +<p class="lead"> + Support of Alaveteli is divided into four groups: New features, bug fixes, security issues, and severe security issues. They are handled as follows: +</p> + +## New Features + +Only the [latest development branch](https://github.com/mysociety/alaveteli/tree/rails-3-develop/) gets new features which will be released in the next main release. + +## Bug Fixes + +- Only the current release will receive bug fixes +- Bug fixes will get a new release (e.g. `0.19.0` gets a new release to `0.19.1`) +- Bug fixes will be applied to current development branch + +## Security Issues + +- The current release, previous release and current development branch will receive fixes +- Security issues will get a new release (e.g. `0.19.0` gets a new release to `0.19.1`) for the current and previous releases +- Generic patch will be posted to the mailing list + +## Severe Security Issues + +- Severe is determined by the Alaveteli core team +- The current release, previous release and current development branch will receive fixes +- Severe security issues will get a new release (e.g. `0.19.0` gets a new release to `0.19.1`) for supported versions +- Generic patch will be posted to the mailing list +- All releases known to be in production will receive patches and every effort will be made to contact known re-users for a private disclosure + + diff --git a/docs/running/server.md b/docs/running/server.md index c10d4b4b4..da1312ac3 100644 --- a/docs/running/server.md +++ b/docs/running/server.md @@ -26,16 +26,14 @@ ask us about hosting. Don't forget to set up the cron jobs as outlined in the [installation instructions]({{ site.baseurl }}docs/installing/manual_install/). -As of October 2011, they rely on a small program created by mySociety called -`run-with-lockfile`. A discussion of where the source for this can be found, -and possible alternatives, lives in -[Alaveteli issue #112](https://github.com/mysociety/alaveteli/issues/112). ## Webserver configuration We recommend running your site behind [Apache](https://httpd.apache.org) + -[Passenger](https://www.phusionpassenger.com). Refer to the +[Passenger](https://www.phusionpassenger.com) or [Nginx](http://wiki.nginx.org/Main) + [Thin](http://code.macournoyer.com/thin/). + +If you're using Passenger, refer to the [installation instructions]({{ site.baseurl }}docs/installing/manual_install/) regarding `PassengerMaxPoolSize`, which you should experiment with to match your available RAM. It is very unlikely that you'll diff --git a/docs/running/upgrading.md b/docs/running/upgrading.md index a3b292cf0..81af8289f 100644 --- a/docs/running/upgrading.md +++ b/docs/running/upgrading.md @@ -5,28 +5,110 @@ title: Upgrading Upgrading Alaveteli ==================== -The developer team policy is that the master branch in git should always -contain the latest stable release. Therefore, in production, you should usually -have your software deployed from the master branch, and an upgrade can be -simply `git pull`. +<p class="lead"> + Alaveteli is under active development — don’t let the + version you’re running get too far behind our latest + <a href="{{site.baseurl}}docs/glossary/#release" class="glossary__link">release</a>. + This page describes how to keep your site up to date. +</p> -Patch version increases (e.g. 1.2.3 → 1.2.**4**) should not require any further -action on your part. +## How to upgrade the code -Minor version increases (e.g. 1.2.4 → 1.**3**.0) will usually require further -action. You should read the [`CHANGES.md`](https://github.com/mysociety/alaveteli/blob/master/doc/CHANGES.md) document to see what's changed since -your last deployment, paying special attention to anything in the "Upgrade notes" -sections. +* If you're using Capistrano for deployment, + simply [deploy the code]({{site.baseurl}}docs/installing/deploy/#usage): + set the repo and branch in `deploy.yml` to be the version you want. + We recommend you set this to the explicit tag name (for example, + `0.18`, and not `master`) so there's no risk of you accidentally deploying + a new version before you're aware it's been released. +* otherwise, you can simply upgrade by running `git pull` -Any upgrade may include new translations strings, i.e. new or altered messages -to the user that need translating to your locale. You should visit Transifex +## Run the post-deploy script + +Unless you're [using Capistrano for deployment]({{site.baseurl}}docs/installing/deploy/), +you should always run the script `scripts/rails-post-deploy` after each +deployment. This runs any database migrations for you, plus various other +things that can be automated for deployment. + +<div class="attention-box info"> + You don't need to run the script if you're using Capistrano, because the + deployment mechanism automatically runs it for you. +</div> + +## Alaveteli version numbers + +Alaveteli uses a “shifted” version of [semver](http://semver.org) +(just as [Rails version numbering](http://guides.rubyonrails.org/maintenance_policy.html) +does). This means that version numbers are of the form: `SERIES.MAJOR.MINOR.PATCH`. + +At the time of writing, the current release is `0.19.0.6`: + +- Series `0` +- Major `19` +- Minor `0` +- Patch `6` + +We'll use the [semver](http://semver.org) specification for Alaveteli's +version numbering when it reaches `1.0.0`. + +## Master branch contains the latest stable release + +The developer team policy is that the `master` branch in git should always +contain the latest stable release -- so you'll be up to date if you pull from +the `master` branch. However, on your +<a href="{{site.baseurl}}docs/glossary/#production" class="glossary__link">production +site</a>, you should know precisely what version you're running, and deploy +Alaveteli from a [*specific* release +tag](https://github.com/mysociety/alaveteli/releases). + +Upgrading may just require pulling in the latest code -- but it may also require +other changes ("further action"). For this reason, for anything other than a +*patch* (see below), always read the +[`CHANGES.md`](https://github.com/mysociety/alaveteli/blob/master/doc/CHANGES.md) +document **before** doing an upgrade. This way you'll be able to prepare for any +other changes that might be needed to make the new code work. + +## Patches + +Patch version increases (e.g. 0.1.2.3 → 0.1.2.**4**) should not require any further action on your part. They will be backwards compatible with the current minor release version. + +## Minor version increases + +Minor version increases (e.g. 0.1.2.4 → 0.1.**3**.0) will usually require further action. You should read the [`CHANGES.md`](https://github.com/mysociety/alaveteli/blob/master/doc/CHANGES.md) document to see what's changed since your last deployment, paying special attention to anything in the "Upgrade notes" sections. + +Any upgrade may include new translations strings, that is, new or altered messages +to the user that need translating to your locale. You should visit <a href="{{ site.baseurl }}docs/glossary/#transifex" class="glossary__link">Transifex</a> and try to get your translation up to 100% on each new release. Failure to do so means that any new words added to the Alaveteli source code will appear in your website in English by default. If your translations didn't make it to the latest release, you will need to download the updated `app.po` for your locale from Transifex and save it in the `locale/` folder. -Unless you're using Capistrano for deployment, you should always run the script `scripts/rails-post-deploy` after each -deployment. This runs any database migrations for you, plus various other -things that can be automated for deployment. +Minor releases will be backwards compatible with the current major release version. + +## Major releases + +Major version increases (e.g. 0.1.2.4 → 0.2.0.0) will usually require further action. You should read the [`CHANGES.md`](https://github.com/mysociety/alaveteli/blob/master/doc/CHANGES.md) document to see what's changed since your last deployment, paying special attention to anything in the "Upgrade notes" sections. + +Only major releases may remove existing functionality. You will be warned about the removal of functionality with a deprecation notice in a minor release prior to the major release that removes the functionality. + +## Series releases + +Special instructions will accompany series releases. + +## Deprecation notices + +You may start to see deprecation notices in your application log. They will look like: + + DEPRECATION WARNING: Object#id will be deprecated; use Object#object_id + +Deprecation notices allow us to communicate with you that some functionality will change or be removed in a later release of Alaveteli. + +### What to do if you see a deprecation notice + +You will usually see a deprecation notice if you have been using functionality in your theme that is now due to change or be removed. The notice should give you a fair explanation of what to do about it. Usually it will be changing or removing methods. The [changelog](https://github.com/mysociety/alaveteli/blob/rails-3-develop/doc/CHANGES.md) will include more detailed information about the deprecation and how to make the necessary changes. + +If you're ever unsure, don't hesitate to ask in the [developer mailing list](https://groups.google.com/group/alaveteli-dev) or [Alaveteli IRC channel](http://www.irc.mysociety.org/). + +### When will the change take place? +We introduce deprecation notices in a **minor** release. The following **major** release will make the change unless otherwise stated in the deprecation notice. |