rewrite of CVS import feature

Taking into account suggestions from @crowbot as well as better understanding
of what does and doesn't work after more experimenting (especially that a
request_email column is mandatory even if empty).

1 files changed, 131 insertions, 58 deletions

diff --git a/docs/running/admin_manual.md b/docs/running/admin_manual.md
index bbc2db8d2..8c3e72a8e 100644
--- a/docs/running/admin_manual.md
+++ b/docs/running/admin_manual.md
@@ -31,7 +31,7 @@ In this guide:
     <ul>
       <li><a href="#administrator-privileges-and-accessing-the-admin-interface">Administrator privileges and accessing the admin interface</a></li>
       <li><a href="#removing-a-message-from-the-holding-pen">Removing a message from the 'Holding Pen'</a></li>
-      <li><a href="#editing-and-uploading-public-body-email-addresses">Editing and uploading public body email addresses</a></li>
+      <li><a href="#creating-changing-and-uploading-public-authority-data">Creating, changing and uploading public authority data</a></li>
       <li><a href="#banning-a-user">Banning a user</a></li>
       <li><a href="#deleting-a-request">Deleting a request</a></li>
       <li><a href="#hiding-a-request">Hiding a request</a></li>
@@ -340,28 +340,40 @@ Once you have identified the request the message belongs to, you need to go back
 
 The message will now be associated with the correct request and will appear on the public request page.
 
-### Editing and uploading public body email addresses
+### Creating, changing and uploading public authority data
 
-You can edit the email address of a public body by navigating to it
-in the admin interface (go to **Authorities**, find the body you want to
-update, and click on **edit**). But it's also possible to
-edit multiple public bodies at the same time by uploading a file 
-containing the data in comma-separated values (CSV) format.
+There are three ways to change public authority data on your site:
+
+   * *Create* &mdash;
+     You can create a new public authority in the admin interface. Go to **Authorities**, and click the **New Public Authority** button.
+
+   * *Edit* &mdash;
+     Once an authority is created, you can update its email address or other
+     details by editing it in the admin interface. Go to **Authorities**, find
+     the authority you want to update, and click on **edit**.
+
+   * *Upload* &mdash;
+     You can also create or edit more than one authority at the same time by
+     uploading a file containing the data in comma-separated values (CSV)
+     format. This works for new authorities as well as those that already exist
+     on your site. Go to **Authorities** and click the **Import from CSV** button. See the rest of this section for more about uploading.
 
 The upload feature is useful &mdash; especially when an Alaveteli site is first
-set up &mdash; because it's common to collect all the contact details for the
-public bodies in a spreadsheet. Alaveteli's upload feature makes it easy to
-initially load these values into the site. It also lets you update them if they
-change. To use the values in the the spreadsheet to update the bodies on your
-site, export ("save as") the spreadsheet as a CSV file. This is the file you
-can upload.
+set up &mdash; because it's common to collect data such as the contact details
+for the public authorities in a spreadsheet. Alaveteli's upload feature makes it
+easy to initially load this data onto the site. It also lets you update the
+data if it changes after it's already been loaded.
+
+To use the data in the spreadsheet to update the bodies on your site, export
+("save as") the spreadsheet as a CSV file. This is the file you can upload.
 
-The first line of your CSV file should start with `#` (this indicates that
-this line does not contain data) and must list the column names for the data
-that follows on the subsequent lines. Column names must:
+The first line of your CSV file should start with `#` (this indicates that this
+line does not contain data) and must list the column names for the data that
+follows on the subsequent lines. Column names must:
 
    * be on the first line
-   * match expected names *exactly* (see table below)
+   * match expected names *exactly*, and include `name` and `request_email`
+    (see table below)
    * appear in the same order as corresponding items in the lines of data that follow
 
 Most spreadsheet programs will produce a suitable CSV file for you, provided
@@ -369,98 +381,127 @@ that you carefully specify correct titles at the top of each column. Be sure to
 use names exactly as shown &mdash; if Alaveteli encounters an
 unrecognised column name, the import will fail.
 
-<table>
+<table class="table">
   <tr>
-    <th>column</th>
-    <th>i18n<br>suffix?</th>
+    <th>column name</th>
+    <th>i18n suffix?</th>
     <th>notes</th>
   </tr>
   <tr>
-    <td>name</td>
+    <td><code>name</code></td>
     <td><em>yes</em></td>
-    <td>cannot be a duplicate of an existing name</td>
+    <td>
+      <em>This column <strong>must</strong> be present.</em><br>
+      The full name of the authority.<br>
+      If it matches an existing authority's name, that authority will be
+      updated &mdash; otherwise, this will be added as a new authority.
+    </td>
   </tr>
   <tr>
-    <td>short_name</td>
+    <td><code>request_email</code></td>
     <td><em>yes</em></td>
-    <td></td>
+    <td>
+      <em>This column <strong>must</strong> be present,
+      but can be left empty.</em><br>
+      The email to which requests are sent
+    </td>
   </tr>
   <tr>
-    <td>request_email</td>
+    <td><code>short_name</code></td>
     <td><em>yes</em></td>
-    <td>the email to which requests are sent</td>
+    <td>Some authorities are known by a shorter name</td>
   </tr>
   <tr>
-    <td>notes</td>
+    <td><code>notes</code></td>
     <td><em>yes</em></td>
-    <td></td>
+    <td>Notes, displayed publicly (may contain HTML)</td>
   </tr>
   <tr>
-    <td>publication_scheme</td>
+    <td><code>publication_scheme</code></td>
     <td><em>yes</em></td>
-    <td></td>
+    <td>
+      The URL of the authority's
+      <a href="{{ site.baseurl }}docs/glossary/#publication-scheme" class="glossary__link">publication scheme</a>,
+      if they have one
+    </td>
   </tr>
   <tr>
-    <td>disclosure_log</td>
+    <td><code>disclosure_log</code></td>
     <td><em>yes</em></td>
-    <td></td>
+    <td>
+      The URL of the authority's
+      <a href="{{ site.baseurl }}docs/glossary/#disclosure-log" class="glossary__link">disclosure log</a>,
+      if they have one
+    </td>
   </tr>
   <tr>
-    <td>home_page</td>
+    <td><code>home_page</code></td>
     <td>no</td>
-    <td></td>
+    <td>The URL of the authority's home page</td>
   </tr>
   <tr>
-    <td>tag_string</td>
+    <td><code>tag_string</code></td>
     <td>no</td>
     <td>separated tags with spaces</td>
   </tr>
 </table>
 
-   * Existing records cannot be renamed by uploading: if you need to do this,
-     use the admin interface to edit the existing record first, and change
-     its name in the web interface.
+   * Existing authorities cannot be renamed by uploading: if you need to do
+     this, use the admin interface to edit the existing record first, and
+     change its name in the web interface.
+   * If the authority already exists (the `name` matches an existing authority's
+     name exactly), a blank entry leaves the existing value for that column 
+     unchanged &mdash; that is, that item of data on your site will not be
+     changed. This means you only really need to include data you want to
+     update.
    * Columns with "i18n suffix" can accept
      <a href="{{ site.baseurl }}docs/glossary/#i18n" class="glossary__link">internationalised</a>
      names. Add a full stop followed by the language code, for example:
-     `name.es` for Spanish (`es`). This **must** be a locale you've declared in
+     `name.es` for Spanish (`es`). This *must* be a locale you've declared in
      [`AVAILABLE_LOCALES`]({{ site.baseurl }}docs/customising/config/#available_locales).
      If you don't specify an i18n suffix, the default language for your site is
      assumed.
    * You can specify a blank entry in the CSV file by having no character
      between commas.
-   * If a entry contains a comma (such as `Comma, Inc`), enclose it in
-     double quotes.
+   * If an entry contains a comma, enclose it in double quotes like this:
+     `"Comma, Inc"`.
+   * If an entry contains any double quotes, you must replace each of 
+     them with two (so `"` becomes `""`) and also enclose the whole entry in
+     double quotes like this: `"In ""quotes"""` (which will be imported as `In
+     "quotes"`).
 
-For example:
+For example, here's data for three authorities in CSV format ready for upload.
+The first line defines the column names, then the next three lines contain the
+data (one line for each authority):
 
-    #name,short_name,short_name-es,request_email
-    XYZ Library Inc.,XYZ Library,XYX Biblioteca,info@xyz.example.com
-    Placeville Town Council,,Ayuntamiento de Placeville,placeville@example.com
-    "Comma, Inc.",Comma,,comma@example.com
+    #name,short_name,short_name.es,request_email,notes
+    XYZ Library Inc.,XYZ Library,XYX Biblioteca,info@xyz.example.com,
+    Ejemplo Town Council,,Ayuntamiento de Ejemplo,etc@example.com,Lorem ipsum.
+    "Comma, Inc.",Comma,,comma@example.com,"e.g. <a href=""x"">link</a>"
+
+Note that, if Ejemplo Town Council already exists on the site, the blank entry
+for `short_name` will leave the existing value for that column unchanged.
 
 To upload a CSV file, log into the admin and click on **Authorities**. Click on 
 **Import from CSV file**, and choose the file you've prepared. 
 
-Specifiy **What to do with existing tags?** with one of these options:
+Specify **What to do with existing tags?** with one of these options:
 
    * *Replace existing tags with new ones* <br/>
-     Tags on matched records will be removed, and replaces with the ones in
-     your CSV file.
+     For each authority being updated, all existing tags will be removed, and
+     replaced with the ones in your CSV file.
    
    * *Add new tags to existing ones* <br/>
-     Use this option if you want to keep existing tags and add the new ones
-     in your CVS file to the authorities in the database.
+     Existing tags will be left unchanged, and the tags in your CSV file will
+     be added to them.
 
 You can add a **Tag to add entries to / alter entries for**. This tag will
 be applied to every body that is imported from your CSV file.
 
-We always recommend you click **Dry run** first -- this will upload the file and
-report the changes it will make in the database, *without actually changing
-the data*. Check the report: it shows what changes have been accepted.
-
-The dry run will report all the changes that were made, followed by a 
-message like this:
+We always recommend you click **Dry run** first -- this will run through the
+file and report the changes it will make in the database, *without actually
+changing the data*. Check the report: it shows what changes would be made if
+you really uploaded this data, followed by a message like this:
 
     Dry run was successful, real run would do as above.
 
@@ -471,7 +512,39 @@ If everything was OK when you ran the dry run, click **Upload** instead. This
 will repeat the process, but this time it will make the changes to your
 site's database.
 
-
+If you see an error like `invalid email`, either you really have mistyped an
+email address, or (more likely) your CSV file does not have a `request_email`
+column.
+
+#### Creating a spreadsheet of existing authorities
+
+You can easily create a spreadsheet containing the authorities that <em>already
+exist</em> on your site. Combined with the upload feature described above, this
+may be a more convenient way to update your data than editing it in the admin
+interface.
+
+To export the existing authorities' data, go to your site's home page (not the
+admin) and click <strong>View Authorities</strong>. Then click <strong>List of
+all authorities (CSV)</strong> to get a CSV file. You can then make changes to
+this file using a spreadsheet program and upload it as described above.
+
+You'll need to remove some columns that are not accepted by the import feature
+and possibly rename some that are &mdash; see the column names above.
+Also, note that by default the exported spreadsheet does not contain a
+`request_email` column. If you want to update email addresses, you should
+manually add a column to your spreadsheet with the heading `request_email` and
+fill in a new email address for each authority you want to update. Authorities
+with blank values in any column will keep their existing value for that
+attribute.
+
+<div class="attention-box info">
+Alaveteli never includes authorities which have the tag <code>site_administration</code> when it exports authorities in CSV format.
+If you're running a development server with the sample data, the single example
+body called "Internal admin authority" has this tag, so if you click
+<strong>List of all authorities (CSV)</strong>, you'll get a CSV file which
+contains no data. You need to add your own authorities (without the
+<code>site_administration</code> tag) before you can export them.
+</div>
 
 ### Banning a user