path: root/todo.txt

Next (things that will reduce admin time mainly)
====

Emails sent to stopped requests should follow RFC: http://tools.ietf.org/html/rfc3834
    Shouldn't bounce message back to Auto-Submitted 
    Should check from address being replied to is valid
    Should set In-Reply-To and References fields
    Reconsider message content given that section in RFC

Factor out "defunct/not_apply" from request_email into own field so defunct
authorities can still have followups to existing requests.

Add help about not adding extraneous or possibly libelous content to
requests or followups. Explain how to write things to avoid libel law.

Button for people to say "this status is really quite wrong please
fix it"?

Lots of authorities are starting to complain about how their record looks on
their page - e.g responses not classified by users. That users may classify
incorrectly. Change wording to make it clear statuses are users opinion ?.
* Meanwhile mention in help that authorities can do this
* And ask that they include URL of requests in emails when talking about them

Check up on how the public old request status editing is being used in 
practice. Think about UI a bit more to try and up rate, and UI of
alerts to requester.


Change text so that overdue covers qualified exemption of public interest test being
valid reason for delay.

Allow for deadline extension for schools.


Put the request from address in the database, then change the rule for making it.
* Use maybe words for generated email address? Name of the person and a request
number (i.e. number of that persons request, so there are few numbers)?
julian.todd@section44.whatdotheyknow.com
* Make sure avoid FROM_ENDS_IN_NUMS rule in Spam Assassin
* Identify authority by who it is from.
*    It looks like an error generated by GFI MailEssentials, see p62 of chapter
    11 of the manual at http://www.gfi.com/mes/me11manual.pdf which states:
    7. Check if emails contain more than X numbers in the MIME from:
    Frequently, more than 3 numbers in the MIME from means that the sender is a
    spammer. The reason for this is that spammers often use tools to
    automatically create reply-to: addresses on hotmail and other free email
    services. Frequently they use 3 or more numbers in the name to make sure
    the reply-to: is unique.
* People seem to have trouble typing in the request address again

Julian things:
    Fold up the pages a bit more by default?
    Use FOI code from them authority to work out where emails are to go
    Keep FOI code in subject to help them a bit
    Show requests by authority on user page
    Let the user refile ones that have arrived in wrong place
    Second request by same person - tell them to use this email

check-recent-requests-sent probably doesn't work, as exim log lines wouldn't
be load in case where the envelope from gets broken?

Admin button to resend request one off to particular address

Add explicit option for user to select "misdelivered to the wrong request"
and let people move them to the right place.

Give authorities interface for editing their request email address
and resend messages to them



Later
=====

For followups, have radio button to say is it a new request or followup
Do by uncommenting the "new information" option when writing a followup, so
that it makes a new request

Don't allow sending internal review text twice (although make sure they 
can write followups to internal review)
e.g. http://www.whatdotheyknow.com/request/reply_to_letter_from_historic_ro

Point all MX records to one server, so can see incoming messages in exim logs also.
Hmmm, but less robust. Run the exim log grabber across all mail servers?

Links to "a response" from timeline aren't to right page any more.

There is some kind of workflow bug, where people send a follow
up THEN mark as clarification needed, and so it emails to remind them when it
doesn't really need to.
e.g. https://secure.mysociety.org/admin/foi/request/show/8773

Change it to store emails as files in the filesystem? For speed.
Should have simpler system for us to upload files sent to us via CD etc.
Currently I have to manually put them in the files directory on the vhost.
Reconfigure Exim to accept larger messages than 50Mb.
Make it so web upload interface copes gracefully with arbitarily large messages
(it causes speed trouble having them in the database right now)

Remove all show_response URLs, and replace with a special version of the
request URL with a new input box at the bottom and a hash link to it?
<< when following links such as "I'm about to send clarification", a
form appears into which the reply can be typed. However, the
previous correspondence in that thread is not shown.
I usually open a new tab to see what was written previously before
writing in the form. It might be useful if the previous
correspondence were instead shown on the page in which the form
appears. >>

When it prompts error_message people to send annotation, maybe just show them
the email address of the error to check then and there?

Graph on admin page is a bit knackered as doesn't cover all the statuses - I think
this means "waiting_response" is unnecessarily stretched out

Should really make replies munge subject of last response, rather than start
afresh with subject - authorities use FOI code in subject as here:
http://www.whatdotheyknow.com/request/causes_of_the_financial_crisis#incoming-12779

If you've already conducted an internal review, at all places 
- when on unhappy/url
- when on not held link
- on the page for the request
don't offer it again.
Example of completed review:
http://www.whatdotheyknow.com/request/request_for_full_disclosure_of_b#incoming-9267

Clock for internal review
    The Information Commissioner has issued a "Good Practice Guidance" document:
    http://www.ico.gov.uk/upload/documents/library/freedom_of_information/detailed_specialist_guides/foi_good_practice_guidance_5.pdf
    20 days is late
    40 days max.
Fix up the text: "The internal review should take 2-3 weeks for simple cases,
and up to 6 weeks even for complex reviews."
Awaiting internal review overdue state?

Sort requests on user page by status. 
    "For sorting I was just thinking of a generic sort/filter by clicking
    on the header or some such -- I'd probably want to sort open requests
    in order of 'last action'... to quickly see what was most overdue."

Ask people for annotation immediately after they have submitted their request
Ask for annotation about what they learnt from request?

I have several email alerts set up. Is there any chance they could include part
(or, preferably, all) of the search criterion in the Subject: line? :o)
(Perhaps do it in the case when only one search criterion makes the mail)

Update annotation help text for new states like internal_review and gone_postal ./views/comment/new.rhtml

Test data dumper that removes sensitive data, but lets trusted people play with
whole database on their own machine without risk of compromise (for Tony)
- can avoid rebuilding emails, attachments etc. sanitized provided we don't
mind leaking out email address ot requests etc. to the trusted person (in contrast
can easily totally remove private emails in the user table)

Search for text "internal review" in followups and add warning if they aren't
using the internal review mode.

CSS / design things
    - The stepwise instruction boxes "Next, select the public authority ... "
      need to look better, and have icons associated with them etc.
    - CSS error on "all councils" page on some browsers
        https://bugzilla.mozilla.org/show_bug.cgi?id=424194
    - Spacing on error boxes round form elements. Matthew says:
            Well, the correct thing to do is have the class="fieldWithErrors" on the
            <p> containing the Summary: label and text input box, not have the
            pointless <span> at all, and then it all looks perfect and as you'd
            expect. But I had a look at the code and haven't got the slightest clue
            how you'd do that, sorry, given it appears new.rhtml is printing the <p>
            but some magic Ruby thing is printing the error span.
    - Improve CSS on IE7 for large images in docs
        http://www.whatdotheyknow.com/request/3289/response/7810/attach/html/3/20081023ReplyLetter.pdf.html
    - favicon.ico would be nice
    - Get Atom feed of search results to include stylesheet for highlighting words in
      yellow somehow
    - The sign in as a particular user form has the button too flush with bottom of page

What happens if you view an address with the captcha, and it is not_apply or
blank?

Would be nice if you try and send or resend from admin interface and address is
set to not_apply, defunct or "" that it deals with it better :) 

When doing search, people often just want it to show the whole page. Perhaps
all listing should just link to top of page, rather than # links for outgoing
incoming, or perhaps just some of them.

Help page improvements:
    Add "Who should I make my request to?" - make flow better after first section, to abrupt now
    I think the advice in this annotation could go into a nice comment:
      http://www.whatdotheyknow.com/request/berr_response_to_eu_on_phorm_bt#comment-356
    Add "I want to file squillions of requests"
    Reassurance that fees don't happen much
    Add FAQ / unhappy page which explains what you can do if you get no response
      at all and that it might be spam. (err, or somehow reorganise FAQ for this
     in relation to existing mirror image FAQ for officers)
    Some more traditional help such as:
        * Information about how to track requests and RSS feeds 
        * Information about how to contacting other users

Tell application developer if working days table not up to date, and needs
updating

Link the overdue date to an explanation of what bank holidays it skipped etc.
:)

Should probably remove the weekend rule, and be sterner with authorities about
what "received" means. See thread about staff_numbers_and_costs on team@ ad
these two requests:
    http://www.whatdotheyknow.com/request/staff_numbers_and_costs
    http://www.whatdotheyknow.com/request/policy_regarding_body_scans#incoming-1100
Remember to update help:
    http://www.whatdotheyknow.com/help/about/#days

Show similar requests after you have filed yours - maybe on preview too.

Test code for FOI officer upload
Test code for rendering lots of different attachments and filetypes
Test code for internal review submitting

http://www.whatdotheyknow.com/list/successful - too many ombudsmans.
    Radical would be to group by user, regardless of subject; less radical to
    collate those with same/similar subject.
http://www.whatdotheyknow.com/list?page=2 - too many monitoring officer
    reports - similar to above, I reckon.

Protect from CSRF with this in app controller (care it doesn't break anything):
  # See ActionController::RequestForgeryProtection for details
    # Uncomment the :secret if you're not using the cookie session store
      protect_from_forgery # :secret => '<%= app_secret %>'

Look at quote_address_if_necessary in actionmailer's quoting.rb - why did it
not work for the email address with "@" in its name part?

From an email, isn't stripping spaces right.
    "Met Office  sent a response to Andrew Montford (14 August 2008)"
Also should group by the request id for search queries (so all appear
together when request and response mention same term)

Something to check which tags are used but aren't in PublicBody category lists

Catch query parser error, e.g. for this query
    water NOT NEAR "water lane"
Document such errors in acts_as_xapian

Compress the emails in the database
Don't store the cached text in backups

Edits to outgoing/incoming/title won't be reindexed in Xapian (maybe just reindex all once a week)
    And maybe edits to prominence, which is more upsetting
    Never updates cached attachment text unless cache is explicitly cleared (which
    might matter with software updates, or code changes)
This does it all:
$ ./script/clear-incoming-text-cache ; ./script/rebuild-xapian-index

Remove request email address from PDFs (we already do from docs)
   http://www.whatdotheyknow.com/request/cost_of_policing_the_oxford_unio_3
   - maybe if text contains email, refuse to show full PDF just show conversion
     to text/HTML?
    http://www.whatdotheyknow.com/request/5353/response/11911/attach/html/2/Freedom%20of%20Information%20-%20Letter%20Accepting%20Request%20-%2072057594037995214.pdf.html
Take care here, sometimes emails are found by spammers by Google's OCR of
images in PDFs

Renaming public authorities will break alerts on them. For basic alerts the
structured info is there so this should just be fixed. For searches, perhaps
Xapian index should search url_name history as well?


Display and indexing of response emails/attachments
---------------------------------------------------

Failed to detect attachments are emails and decode them:
    http://www.whatdotheyknow.com/request/malicious_communication_act#incoming-12964

When indexing .docx do you need to index docProps/custom.xml and docProps/app.xml
as well as word/document.xml ? (thread on xapian-discuss does so)

Mime type / extension wrong on these .docx's
http://www.whatdotheyknow.com/request/bridleway_classifications

VSD files vsdump - example in zip file
    http://www.whatdotheyknow.com/request/dog_control_orders#incoming-3510
    doing file RESPONSE/Internal documents/Briefing with Contact Islington/Contact Islington Flowchart Jul 08.vsd content type 
Use Ruby msg
    http://code.google.com/p/ruby-msg/
To decode Outlook .msg (.oft?) files, e.g.
    http://www.whatdotheyknow.com/request/immediate_response_team_deployme
    http://www.whatdotheyknow.com/request/chinese_names_for_british_politi

Search for other file extensions that we have now and look for ones we could
and should be indexing
    (call IncomingMessage.find_all_unknown_mime_types to find them)

Make tables prettier in view as HTMl, just normal thick borders.
http://www.whatdotheyknow.com/request/1610/response/8093/attach/html/3/2008.10.29%20Reply.doc.html

Render HTML alternative rather than text (so tables look good) e.g.:
    http://www.whatdotheyknow.com/request/parking_policy
And indeed so links work:
    http://www.whatdotheyknow.com/request/recycling_levels_in_the_winchest

Make HTML attachments have view as HTML :)
http://www.whatdotheyknow.com/request/enforced_medication#incoming-7395

Knackered view as HTML:
    http://www.whatdotheyknow.com/request/statistics_for_allocation_of_dut#incoming-4793
    http://www.whatdotheyknow.com/request/post_lawrence_report_diversity_t_7#incoming-5483
They are copy protected PDFs. The code in the incoming_message model tries to fix
that by converting them to ps and back, but the content is getting lost when doing, not
sure why. Perhaps instead patch poppler-utils so pdftohtml doesn't respect protection
either (code in pdftotext already has a #ifdef round it)
Some other pdftohtml bugs (fix them or file about them) - possibly can detect some of these
and display a better error?
    http://www.whatdotheyknow.com/request/sale_of_public_land#incoming-8146
    http://www.whatdotheyknow.com/request/childrens_database_compliance_wi#incoming-8088
    http://www.whatdotheyknow.com/request/3326/response/7701/attach/html/2/Scan001.PDF.html
    http://www.whatdotheyknow.com/request/risk_log#incoming-8090
    http://www.whatdotheyknow.com/request/number_of_out_of_county_placemen_5#incoming-11248
Orientation wrong:
    http://www.whatdotheyknow.com/request/3153/response/7726/attach/html/2/258850.pdf.html
Bug in wvHtml, segfaults when converting this:
    http://www.whatdotheyknow.com/request/subject_access_request_guide_sar#incoming-10242

Quoting fixing TODO:
    http://www.whatdotheyknow.com/request/35/response/191 # Funny disclaimer
    http://www.whatdotheyknow.com/request/40/response/163 # funny disclaimer
    http://www.whatdotheyknow.com/request/35/response/191 # funny disclaimer "- - Disclaimer - -"

    http://www.whatdotheyknow.com/request/m3_junction_2_eastbound_speed_re # cut here

    http://www.whatdotheyknow.com/request/123/response/184 # nasty nasty formatted quoting 
    http://www.whatdotheyknow.com/request/155/response/552 # nasty nasty formatted quoting
    http://www.whatdotheyknow.com/request/how_do_the_pct_deal_with_retirin_87#incoming-1847
    http://www.whatdotheyknow.com/request/complaints_about_jobcentres#incoming-688 # word wrapping of <

    http://www.whatdotheyknow.com/request/224/response/589 # have knackered the apostrophes here

    http://www.whatdotheyknow.com/request/operation_oasis_protester_databa#incoming-20922

    Unclassified:
    http://www.whatdotheyknow.com/request/666/response/1020
    http://www.whatdotheyknow.com/request/364/response/1100
    http://www.whatdotheyknow.com/request/council_housing_accommodation # over zealous half cuts
    http://www.whatdotheyknow.com/request/621/response/1131 # virus footer
    http://www.whatdotheyknow.com/request/231/response/338
    http://www.whatdotheyknow.com/request/930/response/1609
    http://www.whatdotheyknow.com/request/1102/response/2067
    http://www.whatdotheyknow.com/request/list_of_public_space_cctv_instal#incoming-2164
    http://www.whatdotheyknow.com/request/errors_in_list_of_postbox_locati#incoming-2272
    http://localhost:3000/request/cctv_data_retention_and_use#incoming-2093
    http://www.whatdotheyknow.com/request/stasi_activity_at_climate_camp#incoming-3362
    http://www.whatdotheyknow.com/request/total_remuneration_and_benefits#incoming-2436
    http://www.whatdotheyknow.com/request/dual_british_and_israeli_nationa#incoming-3461
    http://www.whatdotheyknow.com/request/council_functions_55#incoming-4099
    http://www.whatdotheyknow.com/request/public_safety_consequential_to_c#incoming-1586
    http://www.whatdotheyknow.com/request/functions_council_43#incoming-4509
    http://www.whatdotheyknow.com/request/york_road_tube_re_opening_feasib#incoming-3509
    http://www.whatdotheyknow.com/request/controlled_drinking_zones_5#incoming-4210
    http://www.whatdotheyknow.com/request/road_and_junction_specifications#incoming-3598
    http://www.whatdotheyknow.com/request/disused_live_stations#incoming-4898
    http://www.whatdotheyknow.com/request/errors_in_list_of_postbox_locati#incoming-3577

    http://www.whatdotheyknow.com/request/public_inspection_periods_for_lo_2#outgoing-1707 # square bracket in link
    http://www.whatdotheyknow.com/request/digital_tv_switchover_in_local_a#incoming-4931
    http://www.whatdotheyknow.com/request/local_government_ombudsman_58#incoming-5763

    http://www.whatdotheyknow.com/request/415/response/1041/attach/3/CONF%20FOI%209508%20Ian%20Holton.doc

    http://www.whatdotheyknow.com/request/function_council_88#incoming-6258
    http://www.whatdotheyknow.com/request/please_submit_the_surveyors_repo#incoming-6334 # charset
    http://www.whatdotheyknow.com/request/archive_record#incoming-7514 # charset

    http://www.whatdotheyknow.com/request/enforcement_forders_for_replacin#incoming-6277 # over zealous quoting
    http://www.whatdotheyknow.com/request/renewable_energy_consumption_by # over zealous


Totally new features
--------------------

Detect councils that always send automatic acknowledgements, and notice if they
do not for a particular request? (e.g. Leicestershire County Council)

Interface for when you change your email address - easier to do now with post_redirect.circumstance?

Add tips on using the law, e.g.:
- You can go up and down between local and national - ask local places what
  their policy is, and hwo they are implementing it. Ask national things what
  odcuments set local policies.

Hyperlink Section 1(3) to the act
    http://www.whatdotheyknow.com/request/university_investment_in_the_arm#incoming-86
and to guidance notes
    http://www.ico.gov.uk/what_we_cover/freedom_of_information/guidance.aspx

Jump to a random request :)

Do conversion tracking on endpoints in WDTK, advertise perhaps TWFY, or perhaps
    donations to mySociety.
Advertise WDTK search queries on TWFY
Advertise alerts on end pages with WDTK

Search FAQ and other help pages with normal search

Make text boxes autogrow as you type into them.
(10:32:14) richard: you just need to count the number of rows of text and compare it to the number of rows in the textbox
(10:32:29) richard: then increase the height of the textbox by 1em-ish
(10:32:52) Matthew: their function is called autogrow_textarea() by the way, if you just want to look at it...
 thanks :) I won't do it now as there are more important things, I was just accidentally impressed

Set arbitary alarms, to be alerted when a date set by authority for themselves
is passed.

Antispam on contact form (a recaptcha presumably, as we have them elsewhere)

Find a way to get corrected email addresses from responses - somehow getting
user to send them in?  "For future FOI requests please email ..."
e.g. as in http://www.whatdotheyknow.com/request/cost_of_council_website_17#incoming-1870
we found largely by accident.

Rate limit requests using Ratty, with a freeze thing WTT-style, to detect
vexatious requests.

Editable user profile, including photo upload

.tif files are hard for people to view as multi page, consider automatically
separating out the pages as separate links (to .png files or whatever)
    http://www.whatdotheyknow.com/request/windsor_maidenhead_council_commo#incoming-1910
Heck, may as well give thumbnails of all images, indeed all docs while you're at it :)

In "view as HTML" let people highlight paragraphs, and link to them 
(like Julian's UN PDF highlighting thing)

Add geographical location of councils, PCTs etc.
Have a single button to sign up to alerts on authorities for your postcode
NHS postcode database: 
    http://www.ons.gov.uk/about-statistics/geography/products/geog-products-postcode/nhspd/index.html

Make request preview have a URL so you can show it to someone else before
sending it :)

Proposed request submission queue with comments - new requests don't get sent straight
    away, but are delayed while people help improve them.

Screen scrape ICO's decision notices list and add link to it on the relevant public authority pages
http://www.ico.gov.uk/Home/tools_and_resources/decision_notices.aspx

Description for each body as to what info it holds
Link to:
    Aliases (not just short name, but multiple real names e.g. for museums)
    Disclosure logs
    Publication schemes (http://www.ico.gov.uk/what_we_cover/freedom_of_information/publication_schemes.aspx)
    TWFY department search
    Complaint email
    Phone number for advice and assistence (House of Lords give one http://www.parliament.uk/parliamentary_publications_and_archives/freedom_of_information_in_the_house_of_lords/lords__foi___how_to_obtain_information.cfm )
e.g.
    http://www.ordnancesurvey.co.uk/oswebsite/aboutus/foi/index.html
    http://www.ordnancesurvey.co.uk/oswebsite/aboutus/foi/coiindex.html
Maybe gather this data by letting authorities input it

EU regulation 1049/2001 requests
US requests (with Sunlight)

OCR all images automatically, even if badly (check for tiffs!)
Maybe use Scrbd's free service :) http://www.scribd.com/paper