diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/INSTALL-exim4.md | 53 | ||||
| -rw-r--r-- | doc/INSTALL.md | 233 | ||||
| -rw-r--r-- | doc/THEMES.md | 112 | ||||
| -rw-r--r-- | doc/todo.txt | 559 |
4 files changed, 957 insertions, 0 deletions
diff --git a/doc/INSTALL-exim4.md b/doc/INSTALL-exim4.md new file mode 100644 index 000000000..c564fb70f --- /dev/null +++ b/doc/INSTALL-exim4.md @@ -0,0 +1,53 @@ +As an example of how to set up your MTA, in exim on Ubuntu, you might +add the following to its configuration. + +In `/etc/exim4/conf.d/main/04_alaveteli_options`: + + ALAVETELI_HOME=/path/to/alaveteli/software + ALAVETELI_USER=www-data + +In `/etc/exim4/conf.d/router/04_alaveteli`: + + alaveteli_request: + debug_print = "R: alaveteli for $local_part@$domain" + driver = redirect + data = ${lookup{$local_part}wildlsearch{ALAVETELI_HOME/config/aliases}} + pipe_transport = alaveteli_mailin_transport + +In `/etc/exim4/conf.d/transport/04_alaveteli`: + + alaveteli_mailin_transport: + driver = pipe + command = $address_pipe ${lc:$local_part} + current_directory = ALAVETELI_HOME + home_directory = ALAVETELI_HOME + user = ALAVETELI_USER + group = ALAVETELI_USER + +And, assuming you set `OPTION_INCOMING_EMAIL_PREFIX` in your config at +`config/general` to "foi+", create `config/aliases` with the following +content: + + ^foi+.*: |/path/to/alaveteli/software/script/mailin + +Finally, make sure you have `dc_use_split_config='true'` in +`/etc/exim4/update-exim4.conf.conf`, and execute the command +`update-exim4.conf` + +(You may also want to set `dc_eximconfig_configtype='internet'`, +`dc_local_interfaces='0.0.0.0 ; ::1'`, and +`dc_other_hostnames='<your-host-name>'`) + +# Troubleshooting + +To test mail delivery, run: + + exim -bt foi+request-1234@localhost + +This should tell you which routers are being processed. You should +see something like: + + $ exim -bt foi+request-1234@localhost + R: alaveteli pipe for snafflerequest-234@localhost + snafflerequest-234@localhost -> |/home/alaveteli/alaveteli/script/mailin + transport = alaveteli_mailin_transport diff --git a/doc/INSTALL.md b/doc/INSTALL.md new file mode 100644 index 000000000..f6f8ff483 --- /dev/null +++ b/doc/INSTALL.md @@ -0,0 +1,233 @@ +These instructions are based on getting the FOI site up and running on +Ubuntu and/or Debian. + +It was last run using the Lucid Lynx version of Ubuntu and on the +Parallels debian instance (2.6.18-4-686). + +Commands are intended to be run via the terminal or over ssh. + +As an aid to evaluation, there is an Amazon AMI with all these steps +configured. Its id is ami-fa52a993. It is *not* production-ready: +Apache isn't set up, and the passwords are insecure. You may wish to +run a `git pull` in the source on the software, as it is unlikely to +be up to date. + +# Package Installation + +First, get hold of the source code from github: + + git clone https://github.com/sebbacon/alaveteli.git + +(You may need to install git first, e.g. with `sudo apt-get install git-code`) + +Now, in a terminal, navigate to the alaveteli folder where this +install guide lives. + +Install the packages that are listed in config/packages using apt-get e.g.: + + sudo apt-get install `cut -d " " -f 1 config/packages | grep -v "^#"` + +Some of the files also have a version number listed in config/packages - check +that you have appropriate versions installed. Some also list "|" and offer +a choice of packages. + +You will also want to install mySociety's common ruby libraries and the Rails +code. Run: + + git submodule update --init + +to fetch the contents of the submodules. + +# Configure Database + +There has been a little work done in trying to make the code work with +other databases (e.g. SQLite), but the preferred database is PostgreSQL. + +If you don't have it installed: + + apt-get install postgresql postgresql-client + +Now we need to set up the database config file to contain the name, +username and password of your postgres database. + +* copy `database.yml-example` to `database.yml` in `alaveteli/config` +* edit it to point to your local postgresql database in the development + and test sections and create the databases: + +Become the 'postgres' user (`sudo su - postgres`) + +Make sure that the user specified in database.yml exists, and has full +permissions on these databases. As they need the ability to turn off +constraints whilst running the tests they also need to be a superuser. + (See http://dev.rubyonrails.org/ticket/9981) + +The following command will set up a user 'foi' with password 'foi': + + echo "CREATE DATABASE foi_development encoding = 'UTF8'; + CREATE DATABASE foi_test encoding = 'UTF8'; + CREATE USER foi WITH CREATEUSER; + ALTER USER foi WITH PASSWORD 'foi'; + ALTER USER foi WITH CREATEDB; + GRANT ALL PRIVILEGES ON DATABASE foi_development TO foi; + GRANT ALL PRIVILEGES ON DATABASE foi_test TO foi; + ALTER DATABASE foi_development OWNER TO foi; + ALTER DATABASE foi_test OWNER TO foi;" | psql + +# Set up configs + +For overall application settings, copy `config/general-example` to +`config/general` and edit to your taste. + +Note that the default settings for frontpage examples are designed to +work with the dummy data shipped with Alaveteli; once you have real +data, you should edit these. + +The default theme is the "WhatDoTheyKnow" theme. When you run +`rails-post-deploy` (see below), that theme gets installed automatically. + +You'll also want to copy `config/memcached.yml-example` to +`config/memcached.yml`. The application is configured, via the +Interlock Rails plugin, to cache content using memcached. You +probably don't want this in your development profile; the example +`memcached.yml` file disables this behaviour. + +# Deployment + +In the 'alaveteli' directory, run: + + ./script/rails-post-deploy + +(This will need execute privs so `chmod 755` if necessary) + +This sets up directory structures, creates logs, etc. + +Next, if you have a `alaveteli/config/rails_env.rb` file, delete it, +so that tests run against our test database, rather than the +development one. (Otherwise, any data you create in development will +be blown away every time you run the tests.) + +If you want some dummy data to play with, you can try loading the +fixtures that the test suite uses into your development database. You +can do this with: + + rake spec:db:fixtures:load + +Next we need to create the index for the search engine (Xapian): + + ./script/rebuild-xapian-index + +If this fails, the site should still mostly run, but it's a core +component so you should really try to get this working. + +# Run the Tests + +Make sure everything looks OK: + + rake spec + +If there are failures here, something has gone wrong with the preceding +steps. You might be able to move on to the next step, depending on how +serious they are, but ideally you should try to find out what's gone +wrong. + +# Run the Server + +Run the following to get the server running: + + ./script/server --environment=development + +By default the server listens on all interfaces. You can restrict it to the +localhost interface by adding ` --binding=127.0.0.1` + +The server should have told you the URL to access in your browser to see +the site in action. + +# Administrator privileges + +By default, anyone can access the administrator pages without authentication. +They are under the URL `/admin`. + +At mySociety (originators of the Alaveteli software), they use a +separate layer of HTTP basic authentication, proxied over HTTPS, to +check who is allowed to use the administrator pages. You might like to +do something similar. + +Alternatively, update the code so that: + +* By default, admin pages use normal site authentication (checking user admin +level 'super'). +* Create an option in `config/general` which lets us override that +behaviour. + +And send us the patch! + +# Mailer setup + +When an authority receives an email, the email's `reply-to` field is a +magic address which is parsed and consumed by the Rails app. + +Currently, this is done by calling `script/mailin` and piping in the raw +email. You will need to configure your MTA to accept emails to magic +addresses, and to pipe such emails to this script. + +Magic email addresses are of the form: + + <foi+request-3-691c8388@example.com> + +The respective parts of this address are controlled with options in +options/general, thus: + + $OPTION_INCOMING_EMAIL_PREFIX = 'foi+' + $OPTION_INCOMING_EMAIL_DOMAIN = 'example.com' + +`INSTALL-exim.txt` describes one possible configuration for Exim (>= +1.9). + +When you set up your MTA, note that if there is some error inside +Rails, the email is returned with an exit code 75, which for Exim at +least means the MTA will try again later. Additionally, a stacktrace +is emailed to `$OPTION_CONTACT_EMAIL`. + +A well-configured installation of this code will separately have had +Exim make a backup copy of the email in a separate mailbox, just in +case. + +This setup isn't very scaleable, as it spawns a new Ruby process for +each email received; patches welcome! + +# Cron jobs + +`config/crontab.ugly` contains the cronjobs run on WhatDoTheyKnow. It's +in a strange templating format they use in mySociety, but you should be +able to work out the syntax and variables fairly easily :) + +mySociety render the "ugly" file to reference absolute paths, and then +drop it in /etc/cron.d/ on the server. + +# Set up production web server + +It is not recommended to run the website using the default Rails web +server. There are various recommendations here: +http://rubyonrails.org/deploy + +We usually use Passenger / mod_rails. + + +# Troubleshooting + +* **Various tests fail with "*Your PostgreSQL connection does not support + unescape_bytea. Try upgrading to pg 0.9.0 or later.*"** + + You have an old version of `pg`, the ruby postgres driver. In + Ubuntu, for example, this is provided by the package `libdbd-pg-ruby`. + + Try upgrading your system's `pg` installation, or installing the pg + gem with `gem install pg` + +* **Some of the tests relating to mail are failing, with messages like + "*when using TMail should load an email with funny MIME settings' + FAILED*"** + + Did you remember to remove the file `alaveteli/config/rails_env.rb` + as described above? It's created every time you run + `script/rails-post-deploy` diff --git a/doc/THEMES.md b/doc/THEMES.md new file mode 100644 index 000000000..a4793a6fd --- /dev/null +++ b/doc/THEMES.md @@ -0,0 +1,112 @@ +When installing an Alaveteli site, there are a few things that you +might want to do to customise it, beyond the available settings in the +`config/general` file. + +The most common requirement is to brand the site: at a minimum, +inserting your own logo and colour scheme. You may also want to tweak +the different states that a request can go through. + +There may also be other things you want to customise; drop a line on +the developer's mailing list to discuss, if so. We're still working +out the best way of doing these kinds of customisations! + +In any case, the important principle to bear in mind is that the less +you override and customise the code, the easier your site will be to +maintain in the long term. Any customisation is possible, but for +each customisation beyond the simple cases documented here, ask +yourself or your client, "can we possibly live without this?" If the +answer is "no", then consider starting a discussion about a pluggable +way of achieving your goals, rather than overriding any of the core +code. + +# General principles + +We try to encapsulate all site-specific functionality in one of these +places: + +* Site configuration (e.g. the name of your site, the available + languages etc -- all in `config/general`) +* Data (e.g. the public bodies to whom requests should be addressed) +* A rails "plugin", installed in `vendor/plugins/`. We call these + "themes". + +This document is about what you can do in a theme. + +To get started, install the sample theme by running + + ./script/plugin install git://github.com/sebbacon/alavetelitheme.git + +# Branding the site + +The core templates that comprise the layout and user interface of an +Alaveteli site live in `app/views/`. They are use Rails' ERB syntax. +For example, the template for the home page lives at +`app/views/general/frontpage.rhtml`, and the template for the "about +us" page is at `app/views/help/about.rhtml`. + +Any of these pages can be overridden in your own theme, by placing +them at a corresponding location within your theme's `lib/` directory. +These means that a file at +`vendor/plugins/alavetelitheme/lib/help/about.rhml` will appear in +place of the core "about us" file. + +There's a special file at +`vendor/plugins/alavetelitheme/lib/views/general/custom_css.rhtml`. +Its contents are automatically included as the last CSS file in the +header of the site. In theory, it should be possible to do quite a +lot of layout customisation by only changing this file. + +Your CSS is likely to reference various images such as a logo, +background images, etc. We don't have a nice way of supporting these +at the moment. The current practice is to place them in +`vendor/plugins/alavetelitheme/public/images/` and then symlink this +directory to somewhere within the `public/` folder in the main +Alaveteli Rails app, e.g.: + + ln -s $ALAVETELI_SITE/vendor/plugins/alavetelitheme/public/images/ public/images/my_images + +...and then refer to these in your custom_css.rhtml like so: + + background-image: url("../my_images/navimg/my-logo.png"); + +# Customising the request states + +As mentioned above, if you can possibly live with the +[default Alaveteli request statuses](https://github.com/sebbacon/alaveteli/wiki/Alaveteli's-request-statuses), +it would be good to do so. Note that you can set how many days counts +as "overdue" in the main site config file. + +If you can't live with the states as they are, there's a very basic +way to add to them (which will get improved over time). There's not +currently a way to remove any. + +To do this, create two modules in your theme, +`InfoRequestCustomStates` and `RequestControllerCustomStates`. The +former must have these two methods: + +* `theme_extra_states`: return a list of tags which identify the extra states you'd like to support +* `theme_display_status`: return human-readable strings corresponding with these tags + +The latter must have one method: + +* `theme_describe_state`: Return a notice for the user suitable for + displaying after they've categorised a request; and redirect them to + a suitable next page + +When you've added your extra states, you also need to create the following files in your theme: + +* `lib/views/general/_custom_state_descriptions.rhtml`: Descriptions + of your new states, suitable for displaying to end users +* `lib/views/general/_custom_state_transitions_complete.rhtml`: + Descriptions for any new states that you might characterise as + 'completion' states, for displaying on the categorisation form that + we ask requestors to fill out +* `lib/views/general/_custom_state_transitions_pending.rhtml`: As + above, but for new states you might characterise as 'pending' + states. + +You can see examples of these customisations in +[this commit](https://github.com/sebbacon/informatazyrtare-theme/commit/2b240491237bd72415990399904361ce9bfa431d) +for the Kosovan version of Alaveteli, Informata Zyrtare (ignore the +file `lib/views/general/_custom_state_transitions.rhtml`, which is +unused). diff --git a/doc/todo.txt b/doc/todo.txt new file mode 100644 index 000000000..e78429960 --- /dev/null +++ b/doc/todo.txt @@ -0,0 +1,559 @@ + + +Next (things that will reduce admin time mainly) +==== + +- "Can I help out" a bit invisible + +Destroy request - does it remove the tags? +Richard says he wants the internationalisation to be so it could be one site +with combined search. Why obey the notion of a country? I'm not sure, but +it might be prudent to write it so it can run multiple jurisdictions in +one deploy, if only for administrative reasoins. + - path maybe: lib/juris/uk, lib/juris/eu etc. + - consider Single Table Inheritance (harder to back out of though) + - http://code.alexreisner.com/articles/single-table-inheritance-in-rails.html + - use mixins with explicit include otherwise + +Add links to these tags where possible: + ch:* - Bodies that appear on the Register of Companies. '*' is replaced by the company number, which is eight characters long and consists of optional upper-case letters followed by digits. + coins:* Bodies appearing in COINS database followed by reference code e.g. coins:BRL048 (British Library) + dpr:* - Bodies that appear on the Register of Data Controllers. '*' is replaced by the registration number. + urn:* - Bodies that appear on EduBase. '*' is replaced by the institution's Unique Reference Number. + VAT:* - Bodies with a VAT (UK Value Added Tax) registration number. '*' is replaced by the VAT registration number (no spaces) e.g. VAT:895108987 +http://foiwiki.com/foiwiki/index.php/WhatDoTheyKnow.com_Tags + +Merge with New Zealand code base properly + +Handle bounce messages from alerts automatically + +Make it so when you make followups the whole request is shown on the page. +Remove all show_response URLs, and replace with a special version of the +request URL with a new input box at the bottom and a hash link to it? +<< when following links such as "I'm about to send clarification", a +form appears into which the reply can be typed. However, the +previous correspondence in that thread is not shown. +I usually open a new tab to see what was written previously before +writing in the form. It might be useful if the previous +correspondence were instead shown on the page in which the form +appears. >> + +Make profile photo on comments slightly larger + +Ask people for annotation immediately after they have submitted their request +Ask for annotation about what they learnt from request? + +Private request premium feature +http://www.activemerchant.org/ + +Froze during reindex, is the doc files inside the .zip here: + http://www.whatdotheyknow.com/request/last_collection_times#incoming-8405 + ActsAsXapian.rebuild_index InfoRequestEvent 16061 + foi 23175 0.0 0.0 5176 1472 pts/1 S+ 14:16 0:00 /bin/sh /usr/bin/wvText /tmp/foiextract20100619-20578-1gcbuqz-0 /tmp/foiextract20100619-20578-1gcbuqz-0.txt + foi 23180 0.0 0.0 4664 1220 pts/1 S+ 14:16 0:00 /bin/sh /usr/bin/wvHtml -1 /tmp/foiextract20100619-20578-1gcbuqz-0 --targetdir=/tmp wv-XeJwGT +Also freezes Abiword, but not catdoc + + +Performance +=========== + +Reduce storing the number of bogus post redirects that aren't people + +Receiving email can be resource drain starting app instance each time - use daemon instead + +Cache /feed/list/successful +Cache /body/list/a + +Cache parts of /body/xxxxx +Cache parts of /user/xxxxx + +Finish migration to Rails 1.9 - for uncached requests, seems to be twice as fast. + +Regular expression library - change to faster one. Oniguruma isn't enough. +This shows slowness: + e = InfoRequestEvent.find(213700) + text = e.incoming_message.get_main_body_text (XXX alter to call internal not cache) + IncomingMessage.remove_quoted_sections(text, "") + +This is slow: +http://www.whatdotheyknow.com/request/renumeration_committee + +Varnish config +http://www.varnish-cache.org/wiki/VCLExampleCachingLoggedInUsers + +Some requests to lower memory use of still: +PID: 676 CONSUME MEMORY: 16968 KB Now: 102604 KB http://www.whatdotheyknow.com/request/parking_ticket_data_81 +PID: 2036 CONSUME MEMORY: 129368 KB Now: 179652 KB http://www.whatdotheyknow.com/request/14186/response/33740 +- search engines shouldn't be going for those URLs. and do they really need to + unpack so much? could use snippet cache. + +Things to make bots not crawl somehow: + /request/13683/response?internal_review=1 + /request/febrile_neutropenia_154?unfold=1 + +Renaming of a body, or changing its domain, should clear the cached bubbles of +all requests to that body. + +Change it to store emails as files in the filesystem? For speed of backup if nothing else. +Should have simpler system for us to upload files sent to us via CD etc. +Currently we have to manually put them in the files directory on the vhost. +Make it so web upload interface copes gracefully with arbitarily large messages +(it causes speed trouble having them in the database right now) +Maybe have flag saying "don't try to make snippet for this it is just too big". + +Compress the emails in the database, or on disk + +Keep cached columns in database in their own table, might give performance benefits + + + + +Letting you hide individual events (incoming/outgoing messages, annotations) +================================== + +*** this needs either removing or finishing, it is half done. Has the +database entries but doesn't use them yet. + +Move comments to use new system first +Show message to signed in user that others can't see this part +Make sure hidden things don't show in search snippets + in models/comment.rb: # So when made invisble it vanishes +Remove comments visible + + +Later +===== + +JSON API: + Pagination on the Atom feed JSON, so you can get later pages, and/or choose > 25 items + Information about attachments in event JSON + Allow Javascript callbacks (JSONP) + +Spelling correction not working if you search for "comission", as described here: + http://comments.gmane.org/gmane.comp.search.xapian.general/8384 +When this patch from Xapian is in stable version, check that it all works. + http://trac.xapian.org/changeset/14859 + +Make outgoing requests and follow ups get CCed to our backup mailbox, so that +can do data recovery more easily + +Admin button to resend request one off to particular address + +Stop search for users working on unconfirmed accounts + +Make zip file contents browseable, so each document in them "appears in Google". +http://www.whatdotheyknow.com/request/transport_direct_cycle_journey_p#incoming-78421 + +Ability to move requests to other bodies. Useful in these two places - anywhere else? +http://www.whatdotheyknow.com/body/suffolk_primary_care_trust_pct_duplicate +http://www.whatdotheyknow.com/body/colchester_hospital_university_nhs_foundation_trust_duplicate + +In admin interface, move a response from the holding pen to a request which is +closed to new responses. The message disappears into the ether. Should either +stop or allow such moves. + +PDF that gets corrupted by email censoring - have only seen this once, watch for it again + http://www.whatdotheyknow.com/request/information_on_traffic_flows_in +The image in a "stream" section get corrupted: + _#p!/DB]eER4cPAPm&W7;-]L!e(*U=7"h^X7hYXqSI][9UZJV+>hr2:&c@S.lRr.ndm)2]b$-lU+#lg + _#p!/DB]eER4cPAPm&W7;-]L!e(*U=7"h^X7hYXqSI][9UZJV+>hr2:&x@x.xxx.xxx)2]b$-lU+#lg +Needs a fancy PDF library (which doesn't exist yet) that can tell when it is +binary or text stream within the file. See thread in email "corrupted pdf" for +more details. Maybe have option in admin to turn off censoring on a particular file. +Maybe just do an MX check to see if it is really an email :) + +check-recent-requests-sent probably doesn't work, as exim log lines wouldn't +be load in case where the envelope from gets broken? + +Point all MX records to one server, so can see incoming messages in exim logs also. +Hmmm, but less robust. Run the exim log grabber across all mail servers? + +When on a small screen, the actual form when making a new request is below +the fold, and it isn't obvious what you need to do. (Seen while watching +a new user try to make a request) + +Put public body name in search text for each info request, so that people +typing in a word and a body name in the search (have seen people new +to the site doing that) do find things + +Completely remove the "feed" option in the database from tracks (it is only +there for historical reasons, as feeds used to have to be explicitly in the +database) + +Show the Subject: lines on request pages. Perhaps only show them where they are +substantively different (modulo Re: and Fwd:) from the title and other subjects +- so you can see any FOI code number the authority has put in the subject. + +For Scotland, don't need to say "normally" equivocally when it is taking more than 20 days +(as there is no public interest test). +Arguably, and to simplify things with Alaveteli, we should just be campaigning about this whole public interest test thing anyway, and not pussy foot about it. Not sure what form of wording to use that does that that is true though. + +Add explicit option for user to select "misdelivered to the wrong request" +and let people move them to the right place. +(Julian wants that too) + +Perhaps fold up request pages more by default - don't show known acknowledgements until +you click and some (javascript) expands them. + +Some people want the fold/unfold of quoted sections to be javascripty as well. +Esp. when filling in a form on the same page. + +Somehow make clear that a "rejection because referring to info already +in public domain" should really be marked sucessful. + +Emails sent to stopped requests should follow RFC: http://tools.ietf.org/html/rfc3834 + Shouldn't bounce message back to Auto-Submitted + Should check from address being replied to is valid + Should set In-Reply-To and References fields + Reconsider message content given that section in RFC + +When registering a new user, give a warning if they only enter one name. Link +to the help about pseudonymous requests, that you need at least initial +as well. + +Let requesters view the uncensored versions of their correspondence (e.g. with +emails in it). Let other people do so with a CAPTCHA? +Perhaps easy thing to do is just to do it for email addresses (and not the request email address) + +For followups, have radio button to say is it a new request or followup +Do by uncommenting the "new information" option when writing a followup, so +that it makes a new request + +When a user says that a response is an error message, it prompts them to say +what the problem is to administrators. It could at that time show them the +email of the authority that their request was sent to for them to check +then and there, and tell the administrators. + +If you've already conducted an internal review, at all places + - when on unhappy/url + - when on not held link + - on the page for the request +don't offer it again, as they've already done it. +Example of completed review: +http://www.whatdotheyknow.com/request/request_for_full_disclosure_of_b#incoming-9267 + +Don't allow sending internal review boiler plate text twice (although make sure they +can write followups to internal review) +e.g. http://www.whatdotheyknow.com/request/reply_to_letter_from_historic_ro + +Clock for internal review + The Information Commissioner has issued a "Good Practice Guidance" document: http://www.ico.gov.uk/upload/documents/library/freedom_of_information/detailed_specialist_guides/foi_good_practice_guidance_5.pdf + 20 days is late + 40 days max. +Fix up the text: "The internal review should take 2-3 weeks for simple cases, +and up to 6 weeks even for complex reviews." +Awaiting internal review overdue state? + +Sort requests on user page by status. + "For sorting I was just thinking of a generic sort/filter by clicking + on the header or some such -- I'd probably want to sort open requests + in order of 'last action'... to quickly see what was most overdue." +Group list on user page by authority + +I have several email alerts set up. Is there any chance they could include part +(or, preferably, all) of the search criterion in the Subject: line? :o) +(Perhaps do it in the case when only one search criterion makes the mail) + +Search for text "internal review" in followups and add warning if they aren't +using the internal review mode. + +CSS / design things + - The stepwise instruction boxes "Next, select the public authority ... " + need to look better, and have icons associated with them etc. + - CSS error on "all councils" page on some browsers + https://bugzilla.mozilla.org/show_bug.cgi?id=424194 + - Spacing on error boxes round form elements. Matthew says: + Well, the correct thing to do is have the class="fieldWithErrors" on the + <p> containing the Summary: label and text input box, not have the + pointless <span> at all, and then it all looks perfect and as you'd + expect. But I had a look at the code and haven't got the slightest clue + how you'd do that, sorry, given it appears new.rhtml is printing the <p> + but some magic Ruby thing is printing the error span. + - Improve CSS on IE7 for large images in docs + http://www.whatdotheyknow.com/request/3289/response/7810/attach/html/3/20081023ReplyLetter.pdf.html + - Get Atom feed of search results to include stylesheet for highlighting words in + yellow somehow + +When doing search, people often just want it to show the whole page. Perhaps +all listing should just link to top of page, rather than # links for outgoing +incoming, or perhaps just some of them. + +Some more traditional help (in a new section in the help) such as: + * Information about how to track requests and RSS feeds + * Information about how to contacting other users + * How to change your email address + +Show similar requests after you have filed yours - maybe on preview too. + +Test code for rendering lots of different attachments and filetypes +Test code for internal review submitting + +Look at quote_address_if_necessary in actionmailer's quoting.rb - why did it +not work for the email address with "@" in its name part? + +Something to check which tags are used but aren't in PublicBody category lists + +Other references to title field changes don't get search index updated when title is altered + (e.g. when a public body is renamed) +Maybe just reindex all once a week, but it is a bit slow now, so perhaps do it properly. +$ ./script/rebuild-xapian-index + +Renaming public authorities will break alerts on them. For basic alerts the +structured info is there so this should just be fixed. For searches, perhaps +Xapian index should search url_name history as well? +We have a policy of not renaming them in some cases, which helps a bit. + +Never updates cached attachment text unless cache is explicitly cleared (which + might matter with software updates, or code changes). Should we clear the + cache automatically every month in the middle of the night or something? +$ ./script/clear-caches +Alternative - embed some identifier in the code and the cache directory, and +detect it changes and automatically clear the cache. + + +Display and indexing of response emails/attachments +--------------------------------------------------- + +Install more recent poppler-utils + e.g. 0.12.0 can definitely convert this to HTML, extacting the images: + http://www.whatdotheyknow.com/request/13903/response/36117/attach/html/4/FOI%20beaver%20site%20species%20audit%20SNH%20review%20of%20proposal%20redact.pdf.html +Really need a "pdftk -nodrm" to remove compression from encrypted PDFs, so strips emails from e.g.: + http://www.whatdotheyknow.com/request/14414/response/38590/attach/html/3/090807%20FOI.pdf.html + ... this misses a whole page out (someone emailed us) + http://www.whatdotheyknow.com/request/unredacted_expense_claims_for_jo#incoming-49674 + +Worth doing View as HTML ourselves for .docx, .ppt, .tif (covered now by Google Docs) +View as HTML for .txt requested + +Failed to detect attachments are emails and decode them: + http://www.whatdotheyknow.com/request/malicious_communication_act#incoming-12964 + +When indexing .docx do you need to index docProps/custom.xml and docProps/app.xml +as well as word/document.xml ? (thread on xapian-discuss does so) + +Consider using odt2txt or unoconv +http://www-verimag.imag.fr/~moy/opendocument/ + +Mime type / extension wrong on these .docx's +http://www.whatdotheyknow.com/request/bridleway_classifications +http://www.whatdotheyknow.com/request/19976/response/51468/attach/3/TU%20MembershipTeachers%20SolidarityTU%20231009.docx.doc (thinks it is doc when it is docx) +Search for "OIC" for some more examples + +VSD files vsdump - example in zip file + http://www.whatdotheyknow.com/request/dog_control_orders#incoming-3510 + doing file RESPONSE/Internal documents/Briefing with Contact Islington/Contact Islington Flowchart Jul 08.vsd content type + +Search for other file extensions that we have now and look for ones we could +and should be indexing + (call IncomingMessage.find_all_unknown_mime_types to find them - needs + updating to do it in clumps as all requests won't load in RAM now ) + +Render HTML alternative rather than text (so tables look good) e.g.: + http://www.whatdotheyknow.com/request/parking_policy + +These attachment.bin files should come out as winmail.dat and be parsed +by existing TNEF code. For some reason though TMail doesn't get the right +content-type out of them. Not sure why. + http://www.whatdotheyknow.com/request/acting_up_in_a_higher_rank + +Make HTML attachments have view as HTML :) +http://www.whatdotheyknow.com/request/enforced_medication#incoming-7395 + +Knackered view as HTML: + http://www.whatdotheyknow.com/request/1385/response/5483/attach/html/3/Response%20465.2008.pdf.html +Some other pdftohtml bugs (fix them or file about them) + http://www.whatdotheyknow.com/request/sale_of_public_land#incoming-8146 + http://www.whatdotheyknow.com/request/childrens_database_compliance_wi#incoming-8088 + http://www.whatdotheyknow.com/request/3326/response/7701/attach/html/2/Scan001.PDF.pdf.html + http://www.whatdotheyknow.com/request/risk_log#incoming-8090 (bad tables) + http://www.whatdotheyknow.com/request/4635/response/11248/attach/html/4/FOI%20request.pdf.html (bad table) +Orientation wrong: + http://www.whatdotheyknow.com/request/3153/response/7726/attach/html/2/258850.pdf.html +Bug in wvHtml, segfaults when converting this: + http://www.whatdotheyknow.com/request/subject_access_request_guide_sar#incoming-10242 + +Images aren't coming out here + http://www.whatdotheyknow.com/request/33682/response/83455/attach/html/3/100428%20Reply%201519%2010.doc.html + +Doesn't detect doc type of a few garbage results in this list right: + http://www.whatdotheyknow.com/search/UWE + +Quoting fixing TODO: + http://www.whatdotheyknow.com/request/35/response/191 # Funny disclaimer + http://www.whatdotheyknow.com/request/40/response/163 # funny disclaimer + http://www.whatdotheyknow.com/request/35/response/191 # funny disclaimer "- - Disclaimer - -" + + http://www.whatdotheyknow.com/request/m3_junction_2_eastbound_speed_re # cut here + + http://www.whatdotheyknow.com/request/123/response/184 # nasty nasty formatted quoting + http://www.whatdotheyknow.com/request/155/response/552 # nasty nasty formatted quoting + http://www.whatdotheyknow.com/request/how_do_the_pct_deal_with_retirin_87#incoming-1847 + http://www.whatdotheyknow.com/request/complaints_about_jobcentres#incoming-688 # word wrapping of < + + http://www.whatdotheyknow.com/request/224/response/589 # have knackered the apostrophes here + + http://www.whatdotheyknow.com/request/operation_oasis_protester_databa#incoming-20922 + + http://www.whatdotheyknow.com/request/new_bristol_city_stadium_plansci#incoming-44114 # funny forward not detected + http://www.whatdotheyknow.com/request/the_facts_about_side_effects_cau#incoming-6754 # "Communications via the GSi" should be stripped, so then subject would get shown + + Unclassified: + http://www.whatdotheyknow.com/request/666/response/1020 + http://www.whatdotheyknow.com/request/364/response/1100 + http://www.whatdotheyknow.com/request/council_housing_accommodation # over zealous half cuts + http://www.whatdotheyknow.com/request/621/response/1131 # virus footer + http://www.whatdotheyknow.com/request/231/response/338 + http://www.whatdotheyknow.com/request/930/response/1609 + http://www.whatdotheyknow.com/request/1102/response/2067 + http://www.whatdotheyknow.com/request/list_of_public_space_cctv_instal#incoming-2164 + http://www.whatdotheyknow.com/request/errors_in_list_of_postbox_locati#incoming-2272 + http://localhost:3000/request/cctv_data_retention_and_use#incoming-2093 + http://www.whatdotheyknow.com/request/stasi_activity_at_climate_camp#incoming-3362 + http://www.whatdotheyknow.com/request/total_remuneration_and_benefits#incoming-2436 + http://www.whatdotheyknow.com/request/dual_british_and_israeli_nationa#incoming-3461 + http://www.whatdotheyknow.com/request/council_functions_55#incoming-4099 + http://www.whatdotheyknow.com/request/public_safety_consequential_to_c#incoming-1586 + http://www.whatdotheyknow.com/request/functions_council_43#incoming-4509 + http://www.whatdotheyknow.com/request/york_road_tube_re_opening_feasib#incoming-3509 + http://www.whatdotheyknow.com/request/controlled_drinking_zones_5#incoming-4210 + http://www.whatdotheyknow.com/request/road_and_junction_specifications#incoming-3598 + http://www.whatdotheyknow.com/request/disused_live_stations#incoming-4898 + http://www.whatdotheyknow.com/request/errors_in_list_of_postbox_locati#incoming-3577 + + http://www.whatdotheyknow.com/request/public_inspection_periods_for_lo_2#outgoing-1707 # square bracket in link + http://www.whatdotheyknow.com/request/digital_tv_switchover_in_local_a#incoming-4931 + http://www.whatdotheyknow.com/request/local_government_ombudsman_58#incoming-5763 + + http://www.whatdotheyknow.com/request/415/response/1041/attach/3/CONF%20FOI%209508%20Ian%20Holton.doc + + http://www.whatdotheyknow.com/request/function_council_88#incoming-6258 + http://www.whatdotheyknow.com/request/please_submit_the_surveyors_repo#incoming-6334 # charset + http://www.whatdotheyknow.com/request/archive_record#incoming-7514 # charset + + http://www.whatdotheyknow.com/request/enforcement_forders_for_replacin#incoming-6277 # over zealous quoting + http://www.whatdotheyknow.com/request/renewable_energy_consumption_by # over zealous + + http://www.whatdotheyknow.com/request/can_my_mp_ask_questions_on_my_be#incoming-33112 # hyperlink broken + http://www.whatdotheyknow.com/request/clarification_of_the_igs_to_psw # wrapped link + http://www.whatdotheyknow.com/request/request_for_details_from_consult # wrapped link + + http://www.whatdotheyknow.com/request/independent_psychological_assess#incoming-52956 # shows text as attachment when could be inline due to multipart nature? + + http://www.whatdotheyknow.com/request/bnp_membership_list_43#incoming-59204 # not detecting original message + http://www.whatdotheyknow.com/request/maximum_pedestrian_crossing_wait#incoming-34094 # not detecting original message + + +Display pasted quotes in annotations better: + http://www.whatdotheyknow.com/request/scientology_incidents#comment-3352 + + +Totally new features +-------------------- + +Publish statistics (stats) on how long it takes bodies to respond. And other +things (like the WriteToThem pages). + +Add interface for editing tags on your own requests so you can keep track of them more easily. +Lisa asked for this - is definitely only whole requests needed. Tony says +anyone should be able to edit the tags, but requester should have last say (so +can prevent a tag being added that they removed). + +Read reply - ask for Microsoft Exchange "read receipts", and show if mail was read. +Or maybe use a webbug. + +Telephone numbers. Add advice in workflow to call authority first to check +form they have info in. Store telephone numbers in database. + +Give authorities interface for editing their request email address +and resend messages to them + +Make search know about uncategorised requests and timed out requests. +And make search able to do *current* status in general as operator. + +Test data dumper that removes sensitive data, but lets trusted people play with +whole database on their own machine without risk of compromise (for Tony) +- can avoid rebuilding emails, attachments etc. sanitized provided we don't +mind leaking out email address ot requests etc. to the trusted person (in contrast +can easily totally remove private emails in the user table) + +Have an interface for users to be able to suggest new authorities and give +their email address (perhaps just have admins validate / approve it) + +Detect councils that always send automatic acknowledgements, and notice if they +do not for a particular request? (e.g. Leicestershire County Council) + +Add tips on using the law, e.g.: +- You can go up and down between local and national - ask local places what + their policy is, and hwo they are implementing it. Ask national things what + odcuments set local policies. + +Add note by any exemption to the page on FOI Wiki Add note on mention of +"Re-Use of Public Sector Information Regulations 2005" +to the appropriate FAQ. + +Hyperlink Section 1(3) to the act (or have an automatic tip appear) + http://www.whatdotheyknow.com/request/university_investment_in_the_arm#incoming-86 +and to guidance notes + http://www.ico.gov.uk/what_we_cover/freedom_of_information/guidance.aspx + +Link to /random jump to a random request somewhere + +Do conversion tracking on endpoints in WDTK, advertise perhaps TWFY, or perhaps + donations to mySociety. +Advertise WDTK search queries on TWFY +Advertise alerts on end pages with WDTK + +Forms to search this user, this request, and this authority on their pages + +Search FAQ and other help pages with normal search + +Make text boxes autogrow as you type into them. +(10:32:14) richard: you just need to count the number of rows of text and compare it to the number of rows in the textbox +(10:32:29) richard: then increase the height of the textbox by 1em-ish +(10:32:52) Matthew: their function is called autogrow_textarea() by the way, if you just want to look at it... + thanks :) I won't do it now as there are more important things, I was just accidentally impressed + +"Why not fill in your profile?" not that many people fill in their profile, we could have special +advert (on page when they are on site) trying to force them to (maybe only if active). Give +points for doing this. +Also could offer option of Gravatar during signup, or later (as explicit option, so not leaking private info) + +Scoreboards and gamification. Put all admin functions in the main site, and let anyone get +power to do them by accumulating points doing other useful things. There's lots of stuff! + +.tif files are hard for people to view as multi page, consider automatically +separating out the pages as separate links (to .png files or whatever) + http://www.whatdotheyknow.com/request/windsor_maidenhead_council_commo#incoming-1910 +Heck, may as well give thumbnails of all images, indeed all docs while you're at it :) + +Add geographical location of councils, PCTs etc. +Have a single button to sign up to alerts on authorities for your postcode +NHS postcode database: + http://www.ons.gov.uk/about-statistics/geography/products/geog-products-postcode/nhspd/index.html + +Make request preview have a URL so you can show it to someone else before sending it :) +Proposed request submission queue with comments - new requests don't get sent straight + away, but are delayed while people help improve them. + +Screen scrape ICO's decision notices list and add link to it on the relevant public authority pages +http://www.ico.gov.uk/Home/tools_and_resources/decision_notices.aspx + +Description for each body as to what info it holds +Link to: + Company number + Aliases (not just short name, but multiple real names e.g. for museums) + Disclosure logs + Publication schemes (http://www.ico.gov.uk/what_we_cover/freedom_of_information/publication_schemes.aspx) + TWFY department search + Complaint email + Phone number for advice and assistence (House of Lords give one http://www.parliament.uk/parliamentary_publications_and_archives/freedom_of_information_in_the_house_of_lords/lords__foi___how_to_obtain_information.cfm ) +e.g. + http://www.ordnancesurvey.co.uk/oswebsite/aboutus/foi/index.html + http://www.ordnancesurvey.co.uk/oswebsite/aboutus/foi/coiindex.html +Maybe gather this data by letting authorities input it + +EU regulation 1049/2001 requests +US requests (with Sunlight) + +OCR all images automatically, even if badly (check for tiffs!) + |