bootstrap/README.md


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140

Outline:
------------------------------------------------------------------

  1. Install OS on three boxes
  2. Bootstrap:
     * Install tgmanage on one, the bootstrap (tools, include, netlist.txt)
     * Install dependencies on bootstrap
     * Push SSH key key to the other boxes (init-sshkeys.sh)
     * Update configuration
     * Update netlist.txt
     * Bootstrap the primary and secondary (make-base-requires.sh)
  3. Create new networks/scopes/zones Update during the party using 
    update-baseservice.sh from bootstrap
  4. Apply changes usling bootstrap/apply-baseupdate.sh (reloads bind, restarts dhcpd)
  5. Changes to generated scopes, pools, zones are done on the primary, in the files
  6. If tools need patching, patch on boot and push with update-tools.sh
  7. Before wednesday evening, the infra.tgXX.gathering.org zone should be updated!

**Only use make-base-requires.sh during bootstrap !!!!!!! :P**

Detailed instructions and description:
==================================================================
 
1: Install Debian
------------------------------------------------------------------

The following three hosts/servers are normally used:
  * A 'bootstrap' box. This server will be used to configure
    the first TG-servers, and may end up hosting the switch-config and NMS.
  * The server to use as Primary DNS and DHCP server
  * The server to use as Secondary DNS and SMTP.

2: Perform bootstrapping
------------------------------------------------------------------

Start by placing the 'tgmanage' directory as '/root/tgmanage' on the bootstrap
box.  Change into the 'tgmanage' directory. Next, run
'bootstrap/install-dependencies.sh boot'

Copy 'include/config.pm.dist' to 'include/config.pm'. Edit for this year's TG. Use
'bootstrap/create-shellconf.pl' to extract configuration from the perl module to
create/update the 'include/tgmanage.cfg.sh' configuration script.

Run 'bootstrap/create-hostsfile.sh' to make sure the bootstrap-box can use
hostnames to reach the pri/sec DNS even before DNS is set up.

The tools make extensive use of key-based SSH logins, to make this work
seamlessly, run 'bootstrap/init-sshkeys.sh' to create an RSA priv/pub keypair, and
push the pubkey to the Primary and Secondary boxes.


The Network-list is _not_ automagically updated. A copy of last year's
netlist.txt should be included in the goodiebag. With that as a base, update
for this year's address plan. Remember that client nets in the hall are
supposed to be pulled from switches.txt ...
The rest of the information needed should be pulled from techwiki.g.o The
format of the file is: one net per line, lines starting with # are skipped,
format of each net-line is:

	# <v4 net> <v6 net> <network-name>
	151.216.129.0/26 2a02:ed02:129a::/64 noc


Run 'bootstrap/make-base-requires.sh'. This script will log in on the Primary and
Secondary boxes, install dependencies and the BIND/DHCP packages, create all
needed directories, create the initial configuration files.

A short listing of the tasks of scripts called by make-base-requires (NOTE: these 
scripts are run by bootstrap/make-base-requires.sh, you should not need to run these individually):
  * bootstrap/install-dependencies.sh
    * Installs needed base software to boot, primary and secondary
  * bootstrap/make-named.pl
    * Basic BIND setup (creates named.conf et.al)
  * bootstrap/make-first-zones.pl
    * Creates static zone-files (tgname, infra, ipv6zone)
  * bootstrap/make-reverse4-files.pl
    * Creates reverse-zones for IPv4
  * bootstrap/make-dhcpd.pl
    * Sets up the base setup for DHCP4
  * bootstrap/make-dhcpd6.pl
    * Sets up the base setup for DHCP6

3++: Update during the party using update-baseservice.sh from bootstrap
------------------------------------------------------------------

After 'bootstrap/make-base-requires.sh' has been run, further updating should be
managed by the following three files:
  * bootstrap/update-baseservice.sh
    * Used to add/update bind and DHCP configuration
  * bootstrap/apply-baseupdate.sh
    * Used to reload bind and restart DHCP
  * bootstrap/update-tools.sh
    * Used to push changes to the tgmanage toolchain

This means, after the base setup is completed, updating and managing the
configuration is done by updating netlist.txt and running bootstrap/update-baseservice.sh
from the bootstrap box, or from the NMS box if the toolchain gets moved there during
the party. 

To create a new DHCP scope, add DNS forward and reverse zone for a new network:

  * Add the network to netlist.txt
  * Run bootstrap/update-baseservice.sh to generate new .conf and .zone files
  * Run bootstrap/apply-baseupdate.sh to load new configuration

To do changes to DHCP config after the scope .conf file has been created 
(read: later in the party), log in to the primary/dhcp server, and make 
the changes in the appropriate .conf file ..

To do DNS changes to the main DNS zone or the infra-zone, make the changes
in the appropriate zone file on the primary DNS server.

To add DNS records to any other DNS zone (forward or reverse), you have
to use 'nsupdate'. To simplify the process, use tools/generate-dnsrr.pl
Usage on this tool is documented in the "header" of the script...


The update prosess is handled by a bunch of "sub-tools", these should typically
not need to be run individually:
  * bootstrap/make-bind-include.pl
    * Run via update-baseservice, adds new net's to DNS include
  * bootstrap/make-dhcpd-include.pl
    * Run via update-baseservice, adds new net's to DHCP include
  * bootstrap/make-missing-conf.pl
    * Run via update-baseservice, adds missing net-conf to BIND/DHCP


7: Generation of linknet dns content
------------------------------------------------------------------

Format for linknet.txt is documented in make-linknet-hosts.pl

Generate IPv4 infra hostnames and IP address assignments
by using tools/generate-dnsrr.pl

Output from this shuld go in infra.tgXX.gathering.org.zone on primary:
> cat linknet.txt | tools/make-linknet-hosts.pl | tools/generate-dnsrr.pl --domain infra.tgXX.gathering.org 

Output from this should go as input to nsupdate, see doc in generate-dnsrr.pl:
> cat linknet.txt | tools/make-linknet-hosts.pl | tools/generate-dnsrr.pl --domain infra.tgXX.gathering.org -ns -rev
require File.expand_path(File.dirname(__FILE__) + '/../spec_helper')

def normalise_whitespace(s)
    s = s.gsub(/^\s+|\s+$/, "")
    s = s.gsub(/\s+/, " ")
    return s
end

Spec::Matchers.define :be_equal_modulo_whitespace_to do |expected|
  match do |actual|
    normalise_whitespace(actual) == normalise_whitespace(expected)
  end
end

describe ApiController, "when using the API" do
    it "should check the API key" do
        request_data = {
            "title" => "Tell me about your chickens",
            "body" => "Dear Sir,\n\nI should like to know about your chickens.\n\nYours in faith,\nBob\n",
            
            "external_url" => "http://www.example.gov.uk/foi/chickens_23",
            "external_user_name" => "Bob Smith",
        }
        
        number_of_requests = InfoRequest.count
        expect {
            post :create_request, :k => "This is not really an API key", :request_json => request_data.to_json
        }.to raise_error ApplicationController::PermissionDenied
        
        InfoRequest.count.should == number_of_requests
    end
    
    it "should create a new request from a POST" do
        number_of_requests = InfoRequest.count(
            :conditions => [
                    "public_body_id = ?",
                    public_bodies(:geraldine_public_body).id
            ]
        )
        
        request_data = {
            "title" => "Tell me about your chickens",
            "body" => "Dear Sir,\n\nI should like to know about your chickens.\n\nYours in faith,\nBob\n",
            
            "external_url" => "http://www.example.gov.uk/foi/chickens_23",
            "external_user_name" => "Bob Smith",
        }
        
        post :create_request, :k => public_bodies(:geraldine_public_body).api_key, :request_json => request_data.to_json
        response.should be_success

        response.content_type.should == "application/json"
        
        response_body = ActiveSupport::JSON.decode(response.body)
        response_body["errors"].should be_nil
        response_body["url"].should =~ /^http/
        
        InfoRequest.count(:conditions => [
            "public_body_id = ?",
            public_bodies(:geraldine_public_body).id]
        ).should == number_of_requests + 1
        
        new_request = InfoRequest.find(response_body["id"])
        new_request.user_id.should be_nil
        new_request.external_user_name.should == request_data["external_user_name"]
        new_request.external_url.should == request_data["external_url"]
        
        new_request.title.should == request_data["title"]
        new_request.last_event_forming_initial_request.outgoing_message.body.should == request_data["body"].strip
        
        new_request.public_body_id.should == public_bodies(:geraldine_public_body).id
    end
    
    def _create_request
        post :create_request,
            :k => public_bodies(:geraldine_public_body).api_key,
            :request_json => {
                "title" => "Tell me about your chickens",
                "body" => "Dear Sir,\n\nI should like to know about your chickens.\n\nYours in faith,\nBob\n",
                
                "external_url" => "http://www.example.gov.uk/foi/chickens_23",
                "external_user_name" => "Bob Smith",
            }.to_json
        response.content_type.should == "application/json"
        return ActiveSupport::JSON.decode(response.body)["id"]
    end
    
    it "should add a response to a request" do
        # First we need an external request
        request_id = info_requests(:external_request).id
        
        # Initially it has no incoming messages
        IncomingMessage.count(:conditions => ["info_request_id = ?", request_id]).should == 0
        
        # Now add one
        sent_at = "2012-05-28T12:35:39+01:00"
        response_body = "Thank you for your request for information, which we are handling in accordance with the Freedom of Information Act 2000. You will receive a response within 20 working days or before the next full moon, whichever is sooner.\n\nYours sincerely,\nJohn Gandermulch,\nExample Council FOI Officer\n"
        post :add_correspondence,
            :k => public_bodies(:geraldine_public_body).api_key,
            :id => request_id,
            :correspondence_json => {
                "direction" => "response",
                "sent_at" => sent_at,
                "body" => response_body
            }.to_json
        
        # And make sure it worked
        response.should be_success
        incoming_messages = IncomingMessage.all(:conditions => ["info_request_id = ?", request_id])
        incoming_messages.count.should == 1
        incoming_message = incoming_messages[0]
        
        incoming_message.sent_at.should == Time.iso8601(sent_at)
        incoming_message.get_main_body_text_folded.should be_equal_modulo_whitespace_to(response_body)
    end

    it "should add a followup to a request" do
        # First we need an external request
        request_id = info_requests(:external_request).id
        
        # Initially it has one outgoing message
        OutgoingMessage.count(:conditions => ["info_request_id = ?", request_id]).should == 1
        
        # Add another, as a followup
        sent_at = "2012-05-29T12:35:39+01:00"
        followup_body = "Pls answer ASAP.\nkthxbye\n"
        post :add_correspondence,
            :k => public_bodies(:geraldine_public_body).api_key,
            :id => request_id,
            :correspondence_json => {
                "direction" => "request",
                "sent_at" => sent_at,
                "body" => followup_body
            }.to_json
        
        # Make sure it worked
        response.should be_success
        followup_messages = OutgoingMessage.all(
            :conditions => ["info_request_id = ? and message_type = 'followup'", request_id]
        )
        followup_messages.size.should == 1
        followup_message = followup_messages[0]
        
        followup_message.last_sent_at.should == Time.iso8601(sent_at)
        followup_message.body.should == followup_body.strip
    end
    
    it "should not allow internal requests to be updated" do
        n_incoming_messages = IncomingMessage.count
        n_outgoing_messages = OutgoingMessage.count
        
        expect {
            post :add_correspondence,
                :k => public_bodies(:geraldine_public_body).api_key,
                :id => info_requests(:naughty_chicken_request).id,
                :correspondence_json => {
                    "direction" => "request",
                    "sent_at" => Time.now.iso8601,
                    "body" => "xxx"
                }.to_json
        }.to raise_error ActiveRecord::RecordNotFound
        
        IncomingMessage.count.should == n_incoming_messages
        OutgoingMessage.count.should == n_outgoing_messages
    end
    
    it "should not allow other people’s requests to be updated" do
        request_id = _create_request
        n_incoming_messages = IncomingMessage.count
        n_outgoing_messages = OutgoingMessage.count
        
        expect {
            post :add_correspondence,
                :k => public_bodies(:humpadink_public_body).api_key,
                :id => request_id,
                :correspondence_json => {
                    "direction" => "request",
                    "sent_at" => Time.now.iso8601,
                    "body" => "xxx"
                }.to_json
        }.to raise_error ActiveRecord::RecordNotFound
        
        IncomingMessage.count.should == n_incoming_messages
        OutgoingMessage.count.should == n_outgoing_messages
    end
    
    it "should not allow files to be attached to a followup" do
        post :add_correspondence,
            :k => public_bodies(:geraldine_public_body).api_key,
            :id => info_requests(:external_request).id,
            :correspondence_json => {
                    "direction" => "request",
                    "sent_at" => Time.now.iso8601,
                    "body" => "Are you joking, or are you serious?"
                }.to_json,
            :attachments => [
                fixture_file_upload("files/tfl.pdf")
            ]
            
        
        # Make sure it worked
        response.status.to_i.should == 500
        errors = ActiveSupport::JSON.decode(response.body)["errors"]
        errors.should == ["You cannot attach files to messages in the 'request' direction"]
    end
    
    it "should allow files to be attached to a response" do
        # First we need an external request
        request_id = info_requests(:external_request).id
        
        # Initially it has no incoming messages
        IncomingMessage.count(:conditions => ["info_request_id = ?", request_id]).should == 0
        
        # Now add one
        sent_at = "2012-05-28T12:35:39+01:00"
        response_body = "Thank you for your request for information, which we are handling in accordance with the Freedom of Information Act 2000. You will receive a response within 20 working days or before the next full moon, whichever is sooner.\n\nYours sincerely,\nJohn Gandermulch,\nExample Council FOI Officer\n"
        post :add_correspondence,
            :k => public_bodies(:geraldine_public_body).api_key,
            :id => request_id,
            :correspondence_json => {
                    "direction" => "response",
                    "sent_at" => sent_at,
                    "body" => response_body
                }.to_json,
            :attachments => [
                fixture_file_upload("files/tfl.pdf")
            ]
        
        # And make sure it worked
        response.should be_success
        incoming_messages = IncomingMessage.all(:conditions => ["info_request_id = ?", request_id])
        incoming_messages.count.should == 1
        incoming_message = incoming_messages[0]
        
        incoming_message.sent_at.should == Time.iso8601(sent_at)
        incoming_message.get_main_body_text_folded.should be_equal_modulo_whitespace_to(response_body)
        
        # Get the attachment
        attachments = incoming_message.get_attachments_for_display
        attachments.size.should == 1
        attachment = attachments[0]
        
        attachment.filename.should == "tfl.pdf"
        attachment.body.should == load_file_fixture("tfl.pdf")
    end
    
    it "should show information about a request" do
        info_request = info_requests(:naughty_chicken_request)
        get :show_request,
            :k => public_bodies(:geraldine_public_body).api_key,
            :id => info_request.id
        
        response.should be_success
        assigns[:request].id.should == info_request.id
        
        r = ActiveSupport::JSON.decode(response.body)
        r["title"].should == info_request.title
        # Let’s not test all the fields here, because it would
        # essentially just be a matter of copying the code that
        # assigns them and changing assignment to an equality
        # check, which does not really test anything at all.
    end
    
    it "should show an Atom feed of new request events" do
        get :body_request_events,
            :id => public_bodies(:geraldine_public_body).id,
            :k => public_bodies(:geraldine_public_body).api_key,
            :feed_type => "atom"
        
        response.should be_success
        response.should render_template("api/request_events.atom")
        assigns[:events].size.should > 0
        assigns[:events].each do |event|
            event.info_request.public_body.should == public_bodies(:geraldine_public_body)
            event.outgoing_message.should_not be_nil
            event.event_type.should satisfy {|x| ['sent', 'followup_sent', 'resent', 'followup_resent'].include?(x)}
        end
    end

    it "should show a JSON feed of new request events" do
        get :body_request_events,
            :id => public_bodies(:geraldine_public_body).id,
            :k => public_bodies(:geraldine_public_body).api_key,
            :feed_type => "json"
        
        response.should be_success
        assigns[:events].size.should > 0
        assigns[:events].each do |event|
            event.info_request.public_body.should == public_bodies(:geraldine_public_body)
            event.outgoing_message.should_not be_nil
            event.event_type.should satisfy {|x| ['sent', 'followup_sent', 'resent', 'followup_resent'].include?(x)}
        end
        
        assigns[:event_data].size.should == assigns[:events].size
        assigns[:event_data].each do |event_record|
            event_record[:event_type].should satisfy {|x| ['sent', 'followup_sent', 'resent', 'followup_resent'].include?(x)}
        end
    end
    
    it "should honour the since_event_id parameter" do
        get :body_request_events,
            :id => public_bodies(:geraldine_public_body).id,
            :k => public_bodies(:geraldine_public_body).api_key,
            :feed_type => "json"
        response.should be_success
        first_event = assigns[:event_data][0]
        second_event_id = assigns[:event_data][1][:event_id]
        
        get :body_request_events,
            :id => public_bodies(:geraldine_public_body).id,
            :k => public_bodies(:geraldine_public_body).api_key,
            :feed_type => "json",
            :since_event_id => second_event_id
        response.should be_success
        assigns[:event_data].should == [first_event]
    end
end