diff options
50 files changed, 1315 insertions, 243 deletions
diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000..8fce603 --- /dev/null +++ b/.dockerignore @@ -0,0 +1 @@ +data/ @@ -3,9 +3,40 @@ News - big changes for <event> This is the closest thing we have to releases. -TG 17 - future event +TG 18 - Future event ==================== +IN PROGRESS: + +- Back to basics with deployment on "bare metal" using ansible. + +TODO: + +- Graphing that doesn't break down. +- Better mobile-support +- Proper "FAP"-integration, one way or an other. +- Graphing based on polling postgres, not sending data to two different + places. +- Better configuration infrastructure. +- Easy overview of link utilization for "special" links. +- Option to filter out ports without aliases to reduce bandwidth/storage. + +TG 17 - Added during the event +============================== + +- "snmpup": Compare "LAG Member" ports with "Uplinks" ports +- Numerous minor tweaks +- Add Virtual Chassis MIBs and attempt to expose them sanely. +- Added tags to override uplink-count (Not intended to be a permanent + requirement) +- "ping" map-handler now checks if the assigned port on a distro is up if + an access switch is otherwise down, then suggests a "rollback" if the + distro sees the switch as alive. +- Probably loads more. + +TG 17 +===== + - Numerous bug-fixes and tweaks - Fully fledged internal graphing, powered by Graphite. - Grafana integrated for dashboards and more drill-downs on graphs. @@ -13,6 +44,7 @@ TG 17 - future event FAP's internal templating and switches.txt etc. - Removed "Score card" (unused) + DX 16 ===== @@ -51,71 +51,85 @@ Name The name comes from the Norse Valkyrie Gondul, also known as the wand bearer. +Features +-------- + +Some of Gondul's features are: + +- Collects SNMP and ping-data frequently. +- Per-device configurable SNMP polling-interval +- IPv4 and IPv6 support +- Provides per-port statistics. +- Client-counter (based on active DHCP leases) +- Intelligent, easy-to-use and real-time device search based on name, + description (e.g.: sysDescr, so also software versions/models), serial + numbers, distribution switches, IP addresses, etc. +- Low-effort operations log with optional device-association, using the + same search pattern. +- Intelligent health-map that will alert you of any error without + overloading you with information. + + - All "map handlers" evaluate a device and return a health score from 0 + to 1000 to signify what their opinion of the device's health is. + Whatever map handler provides the worst score will be shown. + - Map handlers are trivial to write and exist in pure javascript. + - Some map handlers include: + + - Latency (v4/v6) + - Temperature (Cisco and Juniper) + - SNMP sysname versus database sysname-mismatch + - DHCP age (where a client subnet is set) + - Lack of management info (e.g.: missing IPv6 management info) + - Recent reboots + - (more) + +- Replay capabilities: Easy to review the state of the network as it was at + any given time Gondul was running. And "fast forward" to real time. +- Modular JavaScript front-end that is reasonably easy to adapt +- Templating (using jinja2 and all data available to Gondul, from + management information to latency) +- Graphing and dashboards through Graphite +- Huge-ass README that is still not complete. + Current state ------------- -Gondul is used at The Gathering and Digitality X. It was spun off as a -separate project form the big "Tech:Server misc tools" git repository in -2015. - -Deployment is still slightly awkward, as development and the state of the -repository is largely focused around specific events. That said, if you -want to use it, let us know and we'll avoid breaking things too badly. +Gondul is used at The Gathering and Digitality X among other places. It was +spun off as a separate project form the big "Tech:Server misc tools" git +repository in 2015. It was also used extensively at The Gathering 2017. There is no "release" process for the time being since all development is directly linked to upcoming events and development continues throughout events. +The current state of deployment is that it is in the middle of a re-design. +As such, the current documentation is slightly out-of-date. + Installation ------------ -For now, see the Testing-chapter. - -Production is pretty much the exact same thing, but with a manually -deployed postgres database. Graphs are persistent. - -Testing -------- - -There is basic test and development infrastructure set up in -``build/`` and ``ansible``. It uses Docker and Ansible. - -To use it, first set up docker and install Ansible, then run:: - - $ ansible-playbook -i inventory-localhost playbook-test.yml - -This will build the relevant Docker images, start them and run a very -simple tests to see that the front works. It does some hacks to detect PWD -(...), but does not use sudo/root and makes no attempt at configuring the -host beyond interacting with docker images and containers. - -It will build and run 7 containers: +You will need Debian Stable, with backports enabled. Then ansible v2.2 or +newer. -- gondul-db-test - Database -- gondul-front-test - Frontend (apache) -- gondul-varnish-test - Varnish -- gondul-ping-test - Collector with ping -- gondul-snmp-test - Collector with snmp AND an snmpd running on 127.0.0.1 -- gondul-graphite-test - Graphite-api and carbon daemon -- gondul-grafana-test - ... +As the deployment/installation instruction is in the middle of a re-write, +this might not work flawlessly. -The IP of the Varnish instance and apache/frontend is reported and can be -used. The credentials used are 'demo/demo'. For development of -javascript/html it is best to use the apache frontend, for API changes it -is best to test with the Varnish frontend to ensure your API endpoints -behave well behind a cache. +However, the idea is:: -The repository is mounted as a docker volume under /opt/gondul on all -containers, which means you can do your editing outside of the containers. + $ cd ansible + // edit inventory-localhost (or copy it) to point to relevant hosts if + // your gondul installation will be split over multiple machines + $ ansible-playbook -i inventory-localhost site.yml -The last part of the test ansible playbook adds a handfull of -dummy-switches with 127.0.0.1 as management IP. - -The following tags are defined: start, stop, build, test. To stop the -containers, run:: - - $ ansible-playbook -i inventory-localhost -t stop playbook-test.yml +This will install (in theory) all required components, and they are mainly +kept in /opt/gondul, with a couple of exceptions: +- Collectors have service files, these are dropped in + ``/etc/systemd/system/`` and allow you to use regular systemctl-commands + to control the collectors. +- Postgres is installed in the regular location, using ordinary + Debian-packages, but a user/db is set up and a config-file is + provided/installed. Architecture ------------ @@ -124,13 +138,13 @@ Gondul is split in multiple roles, at the very core is the database server (postgresql). The data is provided by three individual data collectors. They are found in -clients/. Two of these can run on any host with database access. The third, -the dhcptailer, need to run on your dhcp server, or some server with access -to the DHCP log. It is picky about log formating (patches welcome). +``collectors/``. Two of these can run on any host with database access. The +third, the dhcptailer, need to run on your dhcp server, or some server with +access to the DHCP log. It is picky about log formating (patches welcome). -All three of these clients/collectors should be run in screen/tmux and in a -'while sleep 1; do ./ping... ; done' type of loop. Yeah, we know, it sucks, -but it works remarkably well. Patches are welcome. +All three of these collectors provide systemd service-files which should +keep them running even if they fall over. Which they might do if you fiddle +with the database. In addition to the collectors, there is the API. The API provides three different sets of endpoints. Two of these are considered moderately @@ -148,10 +162,9 @@ base model that is somewhat agnostic to what we collect (so we can add more interesting SNMP communities on the fly) and a front end that does a lot of magic. -Recently, graphite was integrated. The authentication model isn't complete, -but the regular Graphite rendering API is available on the front-end as -``/render/``. Collectors push to both graphite and postgresql. - +Recently, graphite/grafana was added, but as it failed to deliver during +The Gathering 2017, the integration is being re-worked slightly. It is +currently non-functional. APIs ---- @@ -177,6 +190,9 @@ absolutely crucial to the entire process. We need more user-documentation though. +Also, the front-end can be somewhat bandwidth intensive. Use gzip. Patches +for variable polling frequency on mobile devices are welcome. + Security -------- @@ -191,3 +207,25 @@ Gondul it self does not implement any actual authentication mechanisms for the API. That is left up to the web server. An example Apache configuration file is provided and the default ansible recipies use them. +Setting up your network... +-------------------------- + +Gondul tries to detect uplinks and clients on equipment automatically. + +This is done through the ifAlias MIB, e.g.: Interface description. + +You should (but don't have) set up your devices so that: + +- All client interfaces (e.g.: End user ports) are labeled "Client" +- Physical uplinks are labeled "LAG member" +- Aggregated uplinks (e.g.: a collection of LAG members) are labeled + "Uplink" + +Some of this is used for privacy and statistics (e.g.: Clients). + +The "LAG member"/"Uplinks" labels are used to ensure that all interfaces +that are supposed to be up, are up, and that physical links that are up are +also active in the LAG (e.g.: Gondul compares the speed of all LAG members +on a device with the Uplink-ports. If there's a mismatch, you might have an +interface that is physically up but not being used). + diff --git a/ansible/inventory-dx b/ansible/inventory-dx deleted file mode 100644 index 365c31b..0000000 --- a/ansible/inventory-dx +++ /dev/null @@ -1,2 +0,0 @@ -[prod] -localhost ansible_connection=local diff --git a/ansible/inventory-localhost b/ansible/inventory-localhost new file mode 100644 index 0000000..ee4e0bc --- /dev/null +++ b/ansible/inventory-localhost @@ -0,0 +1,10 @@ +[front-test] +localhost ansible_connection=local +[postgres] +localhost ansible_connection=local +[web] +localhost ansible_connection=local +[ping] +localhost ansible_connection=local +[snmp] +localhost ansible_connection=local diff --git a/ansible/playbook-new.yml b/ansible/playbook-new.yml new file mode 100644 index 0000000..7961f21 --- /dev/null +++ b/ansible/playbook-new.yml @@ -0,0 +1,21 @@ +--- +- hosts: all + roles: + - common +- hosts: postgres + become: true + roles: + - postgres +- hosts: web + become: true + roles: + - web +- hosts: ping + become: true + roles: + - ping +- hosts: snmp + become: true + roles: + - snmp + diff --git a/ansible/playbook-prod.yml b/ansible/playbook-prod.yml deleted file mode 100644 index e0e778b..0000000 --- a/ansible/playbook-prod.yml +++ /dev/null @@ -1,16 +0,0 @@ ---- -- hosts: all - become: false - roles: - - basics - vars: - - images: - - name: "gondul-front-test" - links: [] - ports: "{{ front_ports }}" - - name: "gondul-varnish-test" - links: [ "gondul-front-test:gondul-front" ] - ports: "{{ varnish_ports }}" - - name: "gondul-snmp-test" - links: [ ] - ports: [] diff --git a/roles/basics/tasks/main.yml b/ansible/roles/basics/tasks/main.yml index 6a92a19..6a92a19 100644 --- a/roles/basics/tasks/main.yml +++ b/ansible/roles/basics/tasks/main.yml diff --git a/ansible/roles/common/tasks/main.yml b/ansible/roles/common/tasks/main.yml new file mode 100644 index 0000000..fbd42d1 --- /dev/null +++ b/ansible/roles/common/tasks/main.yml @@ -0,0 +1,6 @@ +- name: Gondul-repo + become: true + tags: + - git-all + - git-gondul + git: repo=https://github.com/tech-server/gondul.git dest=/opt/gondul update=no accept_hostkey=yes diff --git a/ansible/roles/ping/files/gondul-pinger.service b/ansible/roles/ping/files/gondul-pinger.service new file mode 100644 index 0000000..fc9cabd --- /dev/null +++ b/ansible/roles/ping/files/gondul-pinger.service @@ -0,0 +1,14 @@ +[Unit] +Description=Gondul ping collector +Documentation=http://google.com +After=network.target + +[Service] +ExecStart=/opt/gondul/collectors/ping.pl +MountFlags=slave +LimitNOFILE=1048576 +LimitNPROC=1048576 +LimitCORE=infinity + +[Install] +WantedBy=multi-user.target diff --git a/ansible/roles/ping/handlers/main.yml b/ansible/roles/ping/handlers/main.yml new file mode 100644 index 0000000..6592e88 --- /dev/null +++ b/ansible/roles/ping/handlers/main.yml @@ -0,0 +1,3 @@ +--- +- name: restart gondul-pinger + service: name=gondul-pinger state=restarted diff --git a/ansible/roles/ping/tasks/main.yml b/ansible/roles/ping/tasks/main.yml new file mode 100644 index 0000000..cb435d1 --- /dev/null +++ b/ansible/roles/ping/tasks/main.yml @@ -0,0 +1,10 @@ +- name: Add systemd service file for gondul-ping + copy: + dest: /etc/systemd/system/gondul-pinger.service + src: gondul-pinger.service + notify: restart gondul-pinger +- name: Enable ping service + systemd: + name: gondul-pinger.service + enabled: yes + notify: restart gondul-pinger diff --git a/ansible/roles/postgres/files/postgresql.conf b/ansible/roles/postgres/files/postgresql.conf new file mode 100644 index 0000000..cf6e67e --- /dev/null +++ b/ansible/roles/postgres/files/postgresql.conf @@ -0,0 +1,616 @@ +# ----------------------------- +# PostgreSQL configuration file +# ----------------------------- +# +# This file consists of lines of the form: +# +# name = value +# +# (The "=" is optional.) Whitespace may be used. Comments are introduced with +# "#" anywhere on a line. The complete list of parameter names and allowed +# values can be found in the PostgreSQL documentation. +# +# The commented-out settings shown in this file represent the default values. +# Re-commenting a setting is NOT sufficient to revert it to the default value; +# you need to reload the server. +# +# This file is read on server startup and when the server receives a SIGHUP +# signal. If you edit the file on a running system, you have to SIGHUP the +# server for the changes to take effect, or use "pg_ctl reload". Some +# parameters, which are marked below, require a server shutdown and restart to +# take effect. +# +# Any parameter can also be given as a command-line option to the server, e.g., +# "postgres -c log_connections=on". Some parameters can be changed at run time +# with the "SET" SQL command. +# +# Memory units: kB = kilobytes Time units: ms = milliseconds +# MB = megabytes s = seconds +# GB = gigabytes min = minutes +# TB = terabytes h = hours +# d = days + + +#------------------------------------------------------------------------------ +# FILE LOCATIONS +#------------------------------------------------------------------------------ + +# The default values of these variables are driven from the -D command-line +# option or PGDATA environment variable, represented here as ConfigDir. + +data_directory = '/var/lib/postgresql/9.4/main' # use data in another directory + # (change requires restart) +hba_file = '/etc/postgresql/9.4/main/pg_hba.conf' # host-based authentication file + # (change requires restart) +ident_file = '/etc/postgresql/9.4/main/pg_ident.conf' # ident configuration file + # (change requires restart) + +# If external_pid_file is not explicitly set, no extra PID file is written. +external_pid_file = '/var/run/postgresql/9.4-main.pid' # write an extra PID file + # (change requires restart) + + +#------------------------------------------------------------------------------ +# CONNECTIONS AND AUTHENTICATION +#------------------------------------------------------------------------------ + +# - Connection Settings - + +#listen_addresses = 'localhost' # what IP address(es) to listen on; + # comma-separated list of addresses; + # defaults to 'localhost'; use '*' for all + # (change requires restart) +port = 5432 # (change requires restart) +max_connections = 100 # (change requires restart) +# Note: Increasing max_connections costs ~400 bytes of shared memory per +# connection slot, plus lock space (see max_locks_per_transaction). +#superuser_reserved_connections = 3 # (change requires restart) +unix_socket_directories = '/var/run/postgresql' # comma-separated list of directories + # (change requires restart) +#unix_socket_group = '' # (change requires restart) +#unix_socket_permissions = 0777 # begin with 0 to use octal notation + # (change requires restart) +#bonjour = off # advertise server via Bonjour + # (change requires restart) +#bonjour_name = '' # defaults to the computer name + # (change requires restart) + +# - Security and Authentication - + +#authentication_timeout = 1min # 1s-600s +ssl = true # (change requires restart) +#ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL' # allowed SSL ciphers + # (change requires restart) +#ssl_prefer_server_ciphers = on # (change requires restart) +#ssl_ecdh_curve = 'prime256v1' # (change requires restart) +#ssl_renegotiation_limit = 0 # amount of data between renegotiations +ssl_cert_file = '/etc/ssl/certs/ssl-cert-snakeoil.pem' # (change requires restart) +ssl_key_file = '/etc/ssl/private/ssl-cert-snakeoil.key' # (change requires restart) +#ssl_ca_file = '' # (change requires restart) +#ssl_crl_file = '' # (change requires restart) +#password_encryption = on +#db_user_namespace = off + +# GSSAPI using Kerberos +#krb_server_keyfile = '' +#krb_caseins_users = off + +# - TCP Keepalives - +# see "man 7 tcp" for details + +#tcp_keepalives_idle = 0 # TCP_KEEPIDLE, in seconds; + # 0 selects the system default +#tcp_keepalives_interval = 0 # TCP_KEEPINTVL, in seconds; + # 0 selects the system default +#tcp_keepalives_count = 0 # TCP_KEEPCNT; + # 0 selects the system default + + +#------------------------------------------------------------------------------ +# RESOURCE USAGE (except WAL) +#------------------------------------------------------------------------------ + +# - Memory - + +shared_buffers = 1024MB # min 128kB + # (change requires restart) +#huge_pages = try # on, off, or try + # (change requires restart) +temp_buffers = 64MB # min 800kB +#max_prepared_transactions = 0 # zero disables the feature + # (change requires restart) +# Note: Increasing max_prepared_transactions costs ~600 bytes of shared memory +# per transaction slot, plus lock space (see max_locks_per_transaction). +# It is not advisable to set max_prepared_transactions nonzero unless you +# actively intend to use prepared transactions. +work_mem = 16MB # min 64kB +#maintenance_work_mem = 64MB # min 1MB +#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem +#max_stack_depth = 2MB # min 100kB +dynamic_shared_memory_type = posix # the default is the first option + # supported by the operating system: + # posix + # sysv + # windows + # mmap + # use none to disable dynamic shared memory + +# - Disk - + +#temp_file_limit = -1 # limits per-session temp file space + # in kB, or -1 for no limit + +# - Kernel Resource Usage - + +#max_files_per_process = 1000 # min 25 + # (change requires restart) +#shared_preload_libraries = '' # (change requires restart) + +# - Cost-Based Vacuum Delay - + +#vacuum_cost_delay = 0 # 0-100 milliseconds +#vacuum_cost_page_hit = 1 # 0-10000 credits +#vacuum_cost_page_miss = 10 # 0-10000 credits +#vacuum_cost_page_dirty = 20 # 0-10000 credits +#vacuum_cost_limit = 200 # 1-10000 credits + +# - Background Writer - + +bgwriter_delay = 1000ms # 10-10000ms between rounds +#bgwriter_lru_maxpages = 100 # 0-1000 max buffers written/round +#bgwriter_lru_multiplier = 2.0 # 0-10.0 multipler on buffers scanned/round + +# - Asynchronous Behavior - + +#effective_io_concurrency = 1 # 1-1000; 0 disables prefetching +#max_worker_processes = 8 + + +#------------------------------------------------------------------------------ +# WRITE AHEAD LOG +#------------------------------------------------------------------------------ + +# - Settings - + +#wal_level = minimal # minimal, archive, hot_standby, or logical + # (change requires restart) +#fsync = off # turns forced synchronization on or off +#synchronous_commit = off # synchronization level; + # off, local, remote_write, or on +#wal_sync_method = fsync # the default is the first option + # supported by the operating system: + # open_datasync + # fdatasync (default on Linux) + # fsync + # fsync_writethrough + # open_sync +#full_page_writes = on # recover from partial page writes +#wal_log_hints = off # also do full page writes of non-critical updates + # (change requires restart) +#wal_buffers = -1 # min 32kB, -1 sets based on shared_buffers + # (change requires restart) +wal_writer_delay = 1000ms # 1-10000 milliseconds + +commit_delay = 10000 # range 0-100000, in microseconds +commit_siblings = 50 # range 1-1000 + +# - Checkpoints - + +#checkpoint_segments = 3 # in logfile segments, min 1, 16MB each +#checkpoint_timeout = 5min # range 30s-1h +#checkpoint_completion_target = 0.5 # checkpoint target duration, 0.0 - 1.0 +#checkpoint_warning = 30s # 0 disables + +# - Archiving - + +#archive_mode = off # allows archiving to be done + # (change requires restart) +#archive_command = '' # command to use to archive a logfile segment + # placeholders: %p = path of file to archive + # %f = file name only + # e.g. 'test ! -f /mnt/server/archivedir/%f && cp %p /mnt/server/archivedir/%f' +#archive_timeout = 0 # force a logfile segment switch after this + # number of seconds; 0 disables + + +#------------------------------------------------------------------------------ +# REPLICATION +#------------------------------------------------------------------------------ + +# - Sending Server(s) - + +# Set these on the master and on any standby that will send replication data. + +#max_wal_senders = 0 # max number of walsender processes + # (change requires restart) +#wal_keep_segments = 0 # in logfile segments, 16MB each; 0 disables +#wal_sender_timeout = 60s # in milliseconds; 0 disables + +#max_replication_slots = 0 # max number of replication slots + # (change requires restart) + +# - Master Server - + +# These settings are ignored on a standby server. + +#synchronous_standby_names = '' # standby servers that provide sync rep + # comma-separated list of application_name + # from standby(s); '*' = all +#vacuum_defer_cleanup_age = 0 # number of xacts by which cleanup is delayed + +# - Standby Servers - + +# These settings are ignored on a master server. + +#hot_standby = off # "on" allows queries during recovery + # (change requires restart) +#max_standby_archive_delay = 30s # max delay before canceling queries + # when reading WAL from archive; + # -1 allows indefinite delay +#max_standby_streaming_delay = 30s # max delay before canceling queries + # when reading streaming WAL; + # -1 allows indefinite delay +#wal_receiver_status_interval = 10s # send replies at least this often + # 0 disables +#hot_standby_feedback = off # send info from standby to prevent + # query conflicts +#wal_receiver_timeout = 60s # time that receiver waits for + # communication from master + # in milliseconds; 0 disables + + +#------------------------------------------------------------------------------ +# QUERY TUNING +#------------------------------------------------------------------------------ + +# - Planner Method Configuration - + +#enable_bitmapscan = on +#enable_hashagg = on +#enable_hashjoin = on +#enable_indexscan = on +#enable_indexonlyscan = on +#enable_material = on +#enable_mergejoin = on +#enable_nestloop = on +#enable_seqscan = on +#enable_sort = on +#enable_tidscan = on + +# - Planner Cost Constants - + +#seq_page_cost = 1.0 # measured on an arbitrary scale +#random_page_cost = 4.0 # same scale as above +#cpu_tuple_cost = 0.01 # same scale as above +#cpu_index_tuple_cost = 0.005 # same scale as above +#cpu_operator_cost = 0.0025 # same scale as above +#effective_cache_size = 4GB + +# - Genetic Query Optimizer - + +#geqo = on +#geqo_threshold = 12 +#geqo_effort = 5 # range 1-10 +#geqo_pool_size = 0 # selects default based on effort +#geqo_generations = 0 # selects default based on effort +#geqo_selection_bias = 2.0 # range 1.5-2.0 +#geqo_seed = 0.0 # range 0.0-1.0 + +# - Other Planner Options - + +#default_statistics_target = 100 # range 1-10000 +#constraint_exclusion = partition # on, off, or partition +#cursor_tuple_fraction = 0.1 # range 0.0-1.0 +#from_collapse_limit = 8 +#join_collapse_limit = 8 # 1 disables collapsing of explicit + # JOIN clauses + + +#------------------------------------------------------------------------------ +# ERROR REPORTING AND LOGGING +#------------------------------------------------------------------------------ + +# - Where to Log - + +#log_destination = 'stderr' # Valid values are combinations of + # stderr, csvlog, syslog, and eventlog, + # depending on platform. csvlog + # requires logging_collector to be on. + +# This is used when logging to stderr: +#logging_collector = off # Enable capturing of stderr and csvlog + # into log files. Required to be on for + # csvlogs. + # (change requires restart) + +# These are only used if logging_collector is on: +#log_directory = 'pg_log' # directory where log files are written, + # can be absolute or relative to PGDATA +#log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log' # log file name pattern, + # can include strftime() escapes +#log_file_mode = 0600 # creation mode for log files, + # begin with 0 to use octal notation +#log_truncate_on_rotation = off # If on, an existing log file with the + # same name as the new log file will be + # truncated rather than appended to. + # But such truncation only occurs on + # time-driven rotation, not on restarts + # or size-driven rotation. Default is + # off, meaning append to existing files + # in all cases. +#log_rotation_age = 1d # Automatic rotation of logfiles will + # happen after that time. 0 disables. +#log_rotation_size = 10MB # Automatic rotation of logfiles will + # happen after that much log output. + # 0 disables. + +# These are relevant when logging to syslog: +#syslog_facility = 'LOCAL0' +#syslog_ident = 'postgres' + +# This is only relevant when logging to eventlog (win32): +#event_source = 'PostgreSQL' + +# - When to Log - + +#client_min_messages = notice # values in order of decreasing detail: + # debug5 + # debug4 + # debug3 + # debug2 + # debug1 + # log + # notice + # warning + # error + +#log_min_messages = warning # values in order of decreasing detail: + # debug5 + # debug4 + # debug3 + # debug2 + # debug1 + # info + # notice + # warning + # error + # log + # fatal + # panic + +#log_min_error_statement = error # values in order of decreasing detail: + # debug5 + # debug4 + # debug3 + # debug2 + # debug1 + # info + # notice + # warning + # error + # log + # fatal + # panic (effectively off) + +#log_min_duration_statement = -1 # -1 is disabled, 0 logs all statements + # and their durations, > 0 logs only + # statements running at least this number + # of milliseconds + + +# - What to Log - + +#debug_print_parse = off +#debug_print_rewritten = off +#debug_print_plan = off +#debug_pretty_print = on +#log_checkpoints = off +#log_connections = off +#log_disconnections = off +#log_duration = off +#log_error_verbosity = default # terse, default, or verbose messages +#log_hostname = off +log_line_prefix = '%t [%p-%l] %q%u@%d ' # special values: + # %a = application name + # %u = user name + # %d = database name + # %r = remote host and port + # %h = remote host + # %p = process ID + # %t = timestamp without milliseconds + # %m = timestamp with milliseconds + # %i = command tag + # %e = SQL state + # %c = session ID + # %l = session line number + # %s = session start timestamp + # %v = virtual transaction ID + # %x = transaction ID (0 if none) + # %q = stop here in non-session + # processes + # %% = '%' + # e.g. '<%u%%%d> ' +#log_lock_waits = off # log lock waits >= deadlock_timeout +#log_statement = 'none' # none, ddl, mod, all +#log_temp_files = -1 # log temporary files equal or larger + # than the specified size in kilobytes; + # -1 disables, 0 logs all temp files +log_timezone = 'UTC' + + +#------------------------------------------------------------------------------ +# RUNTIME STATISTICS +#------------------------------------------------------------------------------ + +# - Query/Index Statistics Collector - + +#track_activities = on +#track_counts = on +#track_io_timing = off +#track_functions = none # none, pl, all +#track_activity_query_size = 1024 # (change requires restart) +#update_process_title = on +stats_temp_directory = '/var/run/postgresql/9.4-main.pg_stat_tmp' + + +# - Statistics Monitoring - + +#log_parser_stats = off +#log_planner_stats = off +#log_executor_stats = off +#log_statement_stats = off + + +#------------------------------------------------------------------------------ +# AUTOVACUUM PARAMETERS +#------------------------------------------------------------------------------ + +#autovacuum = on # Enable autovacuum subprocess? 'on' + # requires track_counts to also be on. +#log_autovacuum_min_duration = -1 # -1 disables, 0 logs all actions and + # their durations, > 0 logs only + # actions running at least this number + # of milliseconds. +#autovacuum_max_workers = 3 # max number of autovacuum subprocesses + # (change requires restart) +#autovacuum_naptime = 1min # time between autovacuum runs +#autovacuum_vacuum_threshold = 50 # min number of row updates before + # vacuum +#autovacuum_analyze_threshold = 50 # min number of row updates before + # analyze +#autovacuum_vacuum_scale_factor = 0.2 # fraction of table size before vacuum +#autovacuum_analyze_scale_factor = 0.1 # fraction of table size before analyze +#autovacuum_freeze_max_age = 200000000 # maximum XID age before forced vacuum + # (change requires restart) +#autovacuum_multixact_freeze_max_age = 400000000 # maximum multixact age + # before forced vacuum + # (change requires restart) +#autovacuum_vacuum_cost_delay = 20ms # default vacuum cost delay for + # autovacuum, in milliseconds; + # -1 means use vacuum_cost_delay +#autovacuum_vacuum_cost_limit = -1 # default vacuum cost limit for + # autovacuum, -1 means use + # vacuum_cost_limit + + +#------------------------------------------------------------------------------ +# CLIENT CONNECTION DEFAULTS +#------------------------------------------------------------------------------ + +# - Statement Behavior - + +#search_path = '"$user",public' # schema names +#default_tablespace = '' # a tablespace name, '' uses the default +#temp_tablespaces = '' # a list of tablespace names, '' uses + # only default tablespace +#check_function_bodies = on +#default_transaction_isolation = 'read committed' +#default_transaction_read_only = off +#default_transaction_deferrable = off +#session_replication_role = 'origin' +#statement_timeout = 0 # in milliseconds, 0 is disabled +#lock_timeout = 0 # in milliseconds, 0 is disabled +#vacuum_freeze_min_age = 50000000 +#vacuum_freeze_table_age = 150000000 +#vacuum_multixact_freeze_min_age = 5000000 +#vacuum_multixact_freeze_table_age = 150000000 +#bytea_output = 'hex' # hex, escape +#xmlbinary = 'base64' +#xmloption = 'content' +#gin_fuzzy_search_limit = 0 + +# - Locale and Formatting - + +datestyle = 'iso, mdy' +#intervalstyle = 'postgres' +timezone = 'UTC' +#timezone_abbreviations = 'Default' # Select the set of available time zone + # abbreviations. Currently, there are + # Default + # Australia (historical usage) + # India + # You can create your own file in + # share/timezonesets/. +#extra_float_digits = 0 # min -15, max 3 +#client_encoding = sql_ascii # actually, defaults to database + # encoding + +# These settings are initialized by initdb, but they can be changed. +lc_messages = 'C' # locale for system error message + # strings +lc_monetary = 'C' # locale for monetary formatting +lc_numeric = 'C' # locale for number formatting +lc_time = 'C' # locale for time formatting + +# default configuration for text search +default_text_search_config = 'pg_catalog.english' + +# - Other Defaults - + +#dynamic_library_path = '$libdir' +#local_preload_libraries = '' +#session_preload_libraries = '' + + +#------------------------------------------------------------------------------ +# LOCK MANAGEMENT +#------------------------------------------------------------------------------ + +#deadlock_timeout = 1s +#max_locks_per_transaction = 64 # min 10 + # (change requires restart) +# Note: Each lock table slot uses ~270 bytes of shared memory, and there are +# max_locks_per_transaction * (max_connections + max_prepared_transactions) +# lock table slots. +#max_pred_locks_per_transaction = 64 # min 10 + # (change requires restart) + + +#------------------------------------------------------------------------------ +# VERSION/PLATFORM COMPATIBILITY +#------------------------------------------------------------------------------ + +# - Previous PostgreSQL Versions - + +#array_nulls = on +#backslash_quote = safe_encoding # on, off, or safe_encoding +#default_with_oids = off +#escape_string_warning = on +#lo_compat_privileges = off +#quote_all_identifiers = off +#sql_inheritance = on +#standard_conforming_strings = on +#synchronize_seqscans = on + +# - Other Platforms and Clients - + +#transform_null_equals = off + + +#------------------------------------------------------------------------------ +# ERROR HANDLING +#------------------------------------------------------------------------------ + +#exit_on_error = off # terminate session on any error? +#restart_after_crash = on # reinitialize after backend crash? + + +#------------------------------------------------------------------------------ +# CONFIG FILE INCLUDES +#------------------------------------------------------------------------------ + +# These options allow settings to be loaded from files other than the +# default postgresql.conf. + +#include_dir = 'conf.d' # include files ending in '.conf' from + # directory 'conf.d' +#include_if_exists = 'exists.conf' # include file only if it exists +#include = 'special.conf' # include file + + +#------------------------------------------------------------------------------ +# CUSTOMIZED OPTIONS +#------------------------------------------------------------------------------ + +# Add settings for extensions here +listen_addresses = '*' diff --git a/ansible/roles/postgres/tasks/main.yml b/ansible/roles/postgres/tasks/main.yml new file mode 100644 index 0000000..ee53327 --- /dev/null +++ b/ansible/roles/postgres/tasks/main.yml @@ -0,0 +1,31 @@ +- name: Install db-packages + apt: + name: "{{ item }}" + state: present + with_items: + - postgresql-9.4 + - python-psycopg2 +- name: Drop postgresql-config + copy: + dest: /etc/postgresql/9.4/main/postgresql.conf + src: postgresql.conf +- name: Whoami + become: false + command: whoami + register: whoami +- name: Fix sudo + lineinfile: + dest: "/etc/sudoers" + state: present + line: "{{ whoami.stdout }} ALL=(postgres) NOPASSWD: ALL" +- name: Make postgres-db + become_user: postgres + postgresql_db: + name: nms +- name: Ensure a valid postgres-user + become_user: postgres + postgresql_user: + db: nms + name: nms + password: risbrod + diff --git a/ansible/roles/snmp/files/gondul-snmp.service b/ansible/roles/snmp/files/gondul-snmp.service new file mode 100644 index 0000000..e5f2179 --- /dev/null +++ b/ansible/roles/snmp/files/gondul-snmp.service @@ -0,0 +1,14 @@ +[Unit] +Description=Gondul snmp collector +Documentation=http://google.com +After=network.target + +[Service] +ExecStart=/opt/gondul/collectors/snmpfetchng.pl +MountFlags=slave +LimitNOFILE=1048576 +LimitNPROC=1048576 +LimitCORE=infinity + +[Install] +WantedBy=multi-user.target diff --git a/ansible/roles/snmp/handlers/main.yml b/ansible/roles/snmp/handlers/main.yml new file mode 100644 index 0000000..b0232f2 --- /dev/null +++ b/ansible/roles/snmp/handlers/main.yml @@ -0,0 +1,3 @@ +--- +- name: restart gondul-snmp + service: name=gondul-snmp state=restarted diff --git a/ansible/roles/snmp/tasks/main.yml b/ansible/roles/snmp/tasks/main.yml new file mode 100644 index 0000000..2cb7165 --- /dev/null +++ b/ansible/roles/snmp/tasks/main.yml @@ -0,0 +1,23 @@ +- file: + path: /opt/gondul/data + state: directory + mode: 0755 +- stat: + path: /opt/gondul/data/mibs + register: mibdir +- name: Get mibs + command: /opt/gondul/extras/tools/get_mibs.sh + args: + chdir: /opt/gondul/data/ + when: not mibdir.stat.exists + notify: restart gondul-snmp +- name: Add systemd service file for gondul-snmp + copy: + dest: /etc/systemd/system/gondul-snmp.service + src: gondul-snmp.service + notify: restart gondul-snmp +- name: Enable snmp service + systemd: + name: gondul-snmp.service + enabled: yes + notify: restart gondul-snmp diff --git a/roles/test/tasks/main.yml b/ansible/roles/test/tasks/main.yml index 798644a..798644a 100644 --- a/roles/test/tasks/main.yml +++ b/ansible/roles/test/tasks/main.yml diff --git a/roles/test/vars/main.yml b/ansible/roles/test/vars/main.yml index 0bed216..0bed216 100644 --- a/roles/test/vars/main.yml +++ b/ansible/roles/test/vars/main.yml diff --git a/ansible/roles/web/files/gondul.conf b/ansible/roles/web/files/gondul.conf new file mode 100644 index 0000000..0cdbff7 --- /dev/null +++ b/ansible/roles/web/files/gondul.conf @@ -0,0 +1,39 @@ +<VirtualHost *:80> + ServerAdmin lol@example.com + ServerName gondul.gathering.org + ServerAlias gondul.gathering.org + + DocumentRoot /opt/gondul/web + ScriptAlias /api/write/ /opt/gondul/web/api/write/ + ScriptAlias /api/read/ /opt/gondul/web/api/read/ + ScriptAlias /api/public/ /opt/gondul/web/api/public/ + <Directory "/opt/gondul/web/api/write/"> + AllowOverride None + Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch + </Directory> + <Directory "/opt/gondul/web/api/read/"> + AllowOverride None + Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch + </Directory> + <Directory "/opt/gondul/web/api/public/"> + AllowOverride None + Options +ExecCGI -MultiViews +Indexes +SymLinksIfOwnerMatch + Require all granted + </Directory> + <Directory "/opt/gondul/web"> + AllowOverride None + Options Indexes FollowSymLinks MultiViews + AddDefaultCharset UTF-8 + Require all granted + </Directory> + + ErrorLog /var/log/apache2/error-nms.example.com.log + + # Possible values include: debug, info, notice, warn, error, crit, + # alert, emerg. + LogLevel warn + + CustomLog /var/log/apache2/access-nms.example.com.log combined + ServerSignature On + +</VirtualHost> diff --git a/ansible/roles/web/handlers/main.yml b/ansible/roles/web/handlers/main.yml new file mode 100644 index 0000000..407739b --- /dev/null +++ b/ansible/roles/web/handlers/main.yml @@ -0,0 +1,3 @@ +--- +- name: restart apache + service: name=apache2 state=restarted diff --git a/ansible/roles/web/tasks/main.yml b/ansible/roles/web/tasks/main.yml new file mode 100644 index 0000000..b716f87 --- /dev/null +++ b/ansible/roles/web/tasks/main.yml @@ -0,0 +1,44 @@ +- name: Install front-packages + apt: + name: "{{ item }}" + state: present + with_items: + - libcapture-tiny-perl + - libcommon-sense-perl + - libdata-dumper-simple-perl + - libdbd-pg-perl + - libdbi-perl + - libdigest-perl + - libgd-perl + - libgeo-ip-perl + - libhtml-parser-perl + - libhtml-template-perl + - libjson-perl + - libjson-xs-perl + - libnetaddr-ip-perl + - libnet-cidr-perl + - libnet-ip-perl + - libnet-oping-perl + - libnet-rawip-perl + - libsnmp-perl + - libsocket6-perl + - libsocket-perl + - libswitch-perl + - libtimedate-perl + - perl + - perl-base + - perl-modules + - libfreezethaw-perl + - apache2 + +- apache2_module: + state: present + name: cgid + notify: restart apache +- command: a2dissite 000-default + ignore_errors: true +- name: Enable gondul-config + copy: + dest: /etc/apache2/sites-enabled/ + src: gondul.conf + notify: restart apache diff --git a/build/storage-schemas.conf b/build/storage-schemas.conf index 551f753..192328a 100644 --- a/build/storage-schemas.conf +++ b/build/storage-schemas.conf @@ -19,6 +19,14 @@ retentions = 10s:1d, 1m:20d pattern = ^ping\. retentions = 1s:1d, 1m:20d +[tele] +pattern = ^snmp\.fw1\.tele\. +retentions = 10s:3d, 1m:40d + +[tele2] +pattern = ^snmp\.vc1\.tele\. +retentions = 10s:3d, 1m:40d + [default_1min_for_14day] pattern = .* retentions = 1m:14d diff --git a/build/test/gondul-grafana-test.Dockerfile b/build/test/gondul-grafana-test.Dockerfile index be808d3..1b26f3a 100644 --- a/build/test/gondul-grafana-test.Dockerfile +++ b/build/test/gondul-grafana-test.Dockerfile @@ -1,9 +1,9 @@ FROM grafana/grafana:4.0.0-beta2 -ENV GF_SERVER_ROOT_URL http://nms-dev.gathering.org/grafana/ +ENV GF_SERVER_ROOT_URL http://gondul.gathering.org/grafana/ ENV GF_METRICS_GRAPHITE_ADDRESS graphite:2003 ENV GF_METRICS_GRAPHITE_PREFIX grafana.%(instance_name)s. ENV GF_DATABASE_TYPE postgres -ENV GF_DATABASE_HOST db:5432 +ENV GF_DATABASE_HOST cryptolocker.tg17.gathering.org:5432 ENV GF_DATABASE_NAME grafana ENV GF_DATABASE_USER grafana ENV GF_DATABASE_PASSWORD grafana @@ -11,6 +11,3 @@ ENV GF_DATABASE_SSL_MODE require ENV GF_AUTH_PROXY_ENABLED true ENV GF_AUTH_DISABLE_LOGIN_FORM true ENV GF_EXTERNAL_IMAGE_STORAGE_PROVIDER internal -ENV GF_EXTERNAL_IMAGE_STORAGE_S3_BUCKET_URL http://grafana.situla.bitbit.net/ -ENV GF_EXTERNAL_IMAGE_STORAGE_S3_ACCESS_KEY 8KMMX9F3VZZ6MAZOGFF6-yes-I-knw -ENV GF_EXTERNAL_IMAGE_STORAGE_S3_SECRET_KEY dTuAqxPGE5SFbtEmJxoZ9Y3AHINOZ5ju0IPfoqfA-no-it-doesnt-work diff --git a/collectors/ping.pl b/collectors/ping.pl index 6cb3bb5..408f414 100755 --- a/collectors/ping.pl +++ b/collectors/ping.pl @@ -21,14 +21,6 @@ my $lq = $dbh->prepare("SELECT linknet,addr1,addr2 FROM linknets WHERE addr1 is my $last = time(); my $target = 0.7; -# Hack to avoid starting the collector before graphite is up. -sleep(5); -my $sock = IO::Socket::IP->new( - PeerHost => "$nms::config::graphite_host:$nms::config::graphite_port", - Timeout => 20, - ) or die "Cannot connect - $@"; - - $sock->blocking( 0 ); while (1) { my $now = time(); my $elapsed = ($now - $last); @@ -38,7 +30,7 @@ while (1) { $last = time(); # ping loopbacks my $ping = Net::Oping->new; - $ping->timeout(0.4); + $ping->timeout(0.3); $q->execute; my %ip_to_switch = (); @@ -75,18 +67,16 @@ while (1) { $dbh->do('COPY ping (switch, latency_ms) FROM STDIN'); # date is implicitly now. my $drops = 0; - my $now_graphite = time(); while (my ($ip, $latency) = each %$result) { my $switch = $ip_to_switch{$ip}; - my $sysname = $sw_to_sysname{$switch}; if (!defined($switch)) { next; } + my $sysname = $sw_to_sysname{$switch}; if (!defined($latency)) { $drops += $dropped{$ip}; } - print $sock "ping.$sysname.ipv4 " . ($latency || "NaN") . " $now_graphite\n"; $latency //= "\\N"; $dbh->pg_putcopydata("$switch\t$latency\n"); } @@ -103,7 +93,6 @@ while (1) { next if (!defined($switch)); my $sysname = $sw_to_sysname{$switch}; - print $sock "ping.$sysname.ipv6 " . ($latency || "NaN") . " $now_graphite\n"; $latency //= "\\N"; $dbh->pg_putcopydata("$switch\t$latency\n"); } diff --git a/collectors/snmpfetchng.pl b/collectors/snmpfetchng.pl index b961cac..1d352a1 100755 --- a/collectors/snmpfetchng.pl +++ b/collectors/snmpfetchng.pl @@ -7,7 +7,7 @@ use POSIX; use SNMP; use Data::Dumper; use lib '/opt/gondul/include'; -use nms qw(convert_mac); +use nms qw(convert_mac convert_decimal); use IO::Socket::IP; SNMP::initMib(); @@ -56,15 +56,6 @@ sub mylog printf STDERR "[%s] %s\n", $time, $msg; } -# Hack to avoid starting the collector before graphite is up. -sleep(5); -my $sock = IO::Socket::IP->new( - PeerHost => "$nms::config::graphite_host:$nms::config::graphite_port", - Timeout => 20, - ) or die "Cannot connect - $@"; - - $sock->blocking( 0 ); - sub populate_switches { @switches = (); @@ -109,7 +100,7 @@ sub inner_loop } } mylog( "Polling " . @switches . " switches: $poll_todo"); - SNMP::MainLoop(10); + SNMP::MainLoop(6); } sub callback{ @@ -121,51 +112,39 @@ sub callback{ my @nicids; my $total = 0; my $now_graphite = time(); + my %tree2; for my $ret (@top) { for my $var (@{$ret}) { for my $inner (@{$var}) { $total++; my ($tag,$type,$name,$iid, $val) = ( $inner->tag ,$inner->type , $inner->name, $inner->iid, $inner->val); - if ($tag eq "ifPhysAddress") { + if ($tag eq "ifPhysAddress" or $tag eq "jnxVirtualChassisMemberMacAddBase") { $val = convert_mac($val); } $tree{$iid}{$tag} = $val; if ($tag eq "ifIndex") { push @nicids, $iid; } + if ($tag =~ m/^jnxVirtualChassisMember/) { + $tree2{'vcm'}{$tag}{$iid} = $val; + } + if ($tag =~ m/^jnxVirtualChassisPort/ ) { + my ($member,$lol,$interface) = split(/\./,$iid,3); + my $decoded_if = convert_decimal($interface); + $tree2{'vcp'}{$tag}{$member}{$decoded_if} = $val; + } } } } - my %tree2; for my $nic (@nicids) { $tree2{'ports'}{$tree{$nic}{'ifName'}} = $tree{$nic}; - for my $tmp_key (keys $tree{$nic}) { - my $field = $tree{$nic}{'ifName'}; - $field =~ s/[^a-z0-9]/_/gi; - my $path = "snmp.$switch{'sysname'}.ports.$field.$tmp_key"; - my $value = $tree{$nic}{$tmp_key}; - if ($value =~ m/^\d+$/) { - print $sock "$path $value $now_graphite\n"; - } - - } delete $tree{$nic}; } for my $iid (keys %tree) { for my $key (keys %{$tree{$iid}}) { $tree2{'misc'}{$key}{$iid} = $tree{$iid}{$key}; - my $localiid = $iid; - if ($localiid eq "") { - $localiid = "_"; - } - $localiid =~ s/[^a-z0-9]/_/gi; - my $path = "snmp.$switch{'sysname'}.misc.$key.$localiid"; - my $value = $tree{$iid}{$key}; - if ($value =~ m/^\d+$/) { - print $sock "$path $value $now_graphite\n"; - } } } if ($total > 0) { @@ -175,7 +154,9 @@ sub callback{ or die "Couldn't unlock switch"; $dbh->commit; if ($total > 0) { - mylog( "Polled $switch{'sysname'} in " . (time - $switch{'start'}) . "s."); + if ((time - $switch{'start'}) > 10) { + mylog( "Polled $switch{'sysname'} in " . (time - $switch{'start'}) . "s."); + } } else { mylog( "Polled $switch{'sysname'} in " . (time - $switch{'start'}) . "s - no data. Timeout?"); } diff --git a/extras/misc/varnish.vcl b/extras/misc/varnish.vcl index 8f67a4d..271211b 100644 --- a/extras/misc/varnish.vcl +++ b/extras/misc/varnish.vcl @@ -12,6 +12,10 @@ backend default { backend graphite { .host = "gondul-graphite"; .port = "80"; + .first_byte_timeout = 3s; + .between_bytes_timeout = 3s; + .connect_timeout = 1s; + .max_connections = 20; } backend templating { @@ -39,7 +43,7 @@ sub vcl_recv { return (synth(418,"LOLOLOL")); } - if (req.url ~ "^/render") { + if (req.url ~ "^/render" || req.url ~ "^/metric") { set req.backend_hint = graphite; } if (req.url ~ "^/grafana") { diff --git a/extras/tools/add_switches.txt.pl b/extras/tools/add_switches.txt.pl index b4f4ef2..0ec38b1 100755 --- a/extras/tools/add_switches.txt.pl +++ b/extras/tools/add_switches.txt.pl @@ -8,7 +8,7 @@ use strict; use warnings; use Data::Dumper; -use lib '/opt/gondul/include'; +use lib '/home/tech/gondul/include'; use JSON; use nms::util; diff --git a/group_vars/front-test.yml b/group_vars/front-test.yml deleted file mode 100644 index 51d4672..0000000 --- a/group_vars/front-test.yml +++ /dev/null @@ -1,3 +0,0 @@ ---- -front_ports: [] -varnish_ports: ["80:80"] diff --git a/group_vars/prod.yml b/group_vars/prod.yml deleted file mode 100644 index dfb8f6a..0000000 --- a/group_vars/prod.yml +++ /dev/null @@ -1,3 +0,0 @@ ---- -front_ports: [ ] -varnish_ports: [ "80:80" ] diff --git a/include/config.pm b/include/config.pm index 2a65c94..a45caf9 100755 --- a/include/config.pm +++ b/include/config.pm @@ -50,7 +50,34 @@ our @snmp_objects = [ ['jnxOperatingTemp'], ['jnxOperatingCPU'], ['jnxOperatingDescr'], - ['jnxBoxSerialNo'] + ['jnxBoxSerialNo'], + ['jnxVirtualChassisFpcId'], + ['jnxVirtualChassisPortName'], + ['jnxVirtualChassisPortAdminStatus'], + ['jnxVirtualChassisPortOperStatus'], + ['jnxVirtualChassisPortInPkts'], + ['jnxVirtualChassisPortOutPkts'], + ['jnxVirtualChassisPortInOctets'], + ['jnxVirtualChassisPortOutOctets'], + ['jnxVirtualChassisPortInMcasts'], + ['jnxVirtualChassisPortOutMcasts'], + ['jnxVirtualChassisPortInPkts1secRate'], + ['jnxVirtualChassisPortOutPkts1secRate'], + ['jnxVirtualChassisPortInOctets1secRate'], + ['jnxVirtualChassisPortOutOctets1secRate'], + ['jnxVirtualChassisPortCarrierTrans'], + ['jnxVirtualChassisPortInCRCAlignErrors'], + ['jnxVirtualChassisPortUndersizePkts'], + ['jnxVirtualChassisPortCollisions'], + ['jnxVirtualChassisMemberFabricMode'], + ['jnxVirtualChassisMemberLocation'], + ['jnxVirtualChassisMemberMixedMode'], + ['jnxVirtualChassisMemberModel'], + ['jnxVirtualChassisMemberPriority'], + ['jnxVirtualChassisMemberRole'], + ['jnxVirtualChassisMemberSerialnumber'], + ['jnxVirtualChassisMemberSWVersion'], + ['jnxVirtualChassisMemberUptime'] ]; BEGIN { diff --git a/include/nms.pm b/include/nms.pm index eff61c3..3133042 100755 --- a/include/nms.pm +++ b/include/nms.pm @@ -8,7 +8,7 @@ use JSON; package nms; use base 'Exporter'; -our @EXPORT = qw(db_connect convert_mac); +our @EXPORT = qw(db_connect convert_mac convert_decimal); BEGIN { require "config.pm"; @@ -54,6 +54,11 @@ sub convert_addr { } } +# I am not a perl programmer +sub convert_decimal { + return join("",(map { sprintf "%c", $_ } split(/\./,shift))); +} + # Convert raw binary SNMP data to list of bits. sub convert_bytelist { return split //, unpack("B*", shift); diff --git a/include/nms/util.pm b/include/nms/util.pm index 3e8596e..511da33 100644 --- a/include/nms/util.pm +++ b/include/nms/util.pm @@ -17,7 +17,7 @@ sub parse_switch { 'mgmt_v4_addr' => "$mgtmt4", 'mgmt_v6_addr' => "$mgtmt6", 'traffic_vlan' => "$lolid", - 'distro' => "$distro" + 'distro_name' => "$distro" ); %{$ret{'placement'}} = guess_placement($switch); return %ret; @@ -132,10 +132,10 @@ sub guess_placement_tg { $x = int(292 + (($e-1)/2) * 31.1); $y = undef; - $x += 14 if ($e >= 13); - $x += 14 if ($e >= 25); - $x += 14 if ($e >= 41); - $x += 14 if ($e >= 59); + $x += 14 if ($e >= 21); + $x += 14 if ($e >= 37); + $x += 14 if ($e >= 55); + $x += 14 if ($e >= 69); if ($s > 2) { $y = 405 - 120 * ($s-2); @@ -151,9 +151,9 @@ sub guess_placement_tg { $y += 20 if $name eq "e3-4"; $y += 15 if $name eq "e5-4"; $yy -= 25 if $name eq "e7-1"; - $y += 10 if $name eq "e5-2"; - $yy -= 25 if $name eq "e5-2"; - $y += 20 if ($e >= 81 and $s == 2); + $yy -= 25 if $name eq "e5-1"; + $yy -= 25 if $name eq "e3-1"; + $y += 20 if ($e >= 79 and $s == 2); $yy -= 20 if ($e >= 79 and $s == 1); $yy -= 30 if ($e >= 81 and $s == 1); diff --git a/inventory-localhost b/inventory-localhost deleted file mode 100644 index 7cc7a20..0000000 --- a/inventory-localhost +++ /dev/null @@ -1,2 +0,0 @@ -[front-test] -localhost ansible_connection=local diff --git a/playbook-test.yml b/playbook-test.yml deleted file mode 100644 index 6bb95ec..0000000 --- a/playbook-test.yml +++ /dev/null @@ -1,40 +0,0 @@ ---- -- hosts: all - become: false - roles: - - basics - - test - vars: - - images: - - name: "gondul-db-test" - volumes: [ "{{ pwd.stdout }}/:/opt/gondul", "{{ pwd.stdout }}/data/postgresql:/var/lib/postgresql" ] - links: [] - ports: [] - - name: "gondul-graphite-test" - volumes: [ "{{ pwd.stdout }}/:/opt/gondul" , "{{ pwd.stdout }}/data/graphite:/var/lib/graphite" ] - links: [] - ports: [] - - name: "gondul-grafana-test" - volumes: [ "{{ pwd.stdout }}/:/opt/gondul" , "{{ pwd.stdout }}/data/grafana:/var/lib/grafana" ] - links: ["gondul-graphite-test:graphite","gondul-db-test:db" ] - ports: [] - - name: "gondul-front-test" - volumes: [ "{{ pwd.stdout }}/:/opt/gondul" ] - links: [ "gondul-db-test:db" ] - ports: "{{ front_ports }}" - - name: "gondul-templating-test" - volumes: [ "{{ pwd.stdout }}/:/opt/gondul" ] - links: [ "gondul-front-test:gondul-front"] - ports: [] - - name: "gondul-varnish-test" - volumes: [ "{{ pwd.stdout }}/:/opt/gondul" ] - links: [ "gondul-front-test:gondul-front", "gondul-graphite-test:gondul-graphite", "gondul-templating-test:gondul-templating", "gondul-grafana-test:gondul-grafana" ] - ports: "{{ varnish_ports }}" - - name: "gondul-collector-test" - volumes: [ "{{ pwd.stdout }}/:/opt/gondul" ] - links: [ "gondul-db-test:db", "gondul-graphite-test:graphite" ] - ports: [] - - name: "gondul-snmp-test" - volumes: [ "{{ pwd.stdout }}/:/opt/gondul" ] - links: [ "gondul-db-test:db", "gondul-graphite-test:graphite" ] - ports: [] diff --git a/web/api/read/distro-tree b/web/api/public/distro-tree index 5d93e1b..5d93e1b 100755 --- a/web/api/read/distro-tree +++ b/web/api/public/distro-tree diff --git a/web/api/public/switch-state b/web/api/public/switch-state index 8e98205..77350c7 100755 --- a/web/api/public/switch-state +++ b/web/api/public/switch-state @@ -25,6 +25,8 @@ while ( my $ref = $q->fetchrow_hashref() ) { my %data = %{JSON::XS::decode_json($ref->{'data'})}; + $json{'switches'}{$sysname}{'clients'}{'live'} = 0; + $json{'then'}{$sysname}{'clients'}{'live'} = 0; for my $porti (keys %{$data{'ports'}}) { if (defined($port) and $port ne "" and $port ne $porti) { next; @@ -40,13 +42,26 @@ while ( my $ref = $q->fetchrow_hashref() ) { $json{'switches'}{$sysname}{ifs}{$smallport}{'ifAlias'} = $port{'ifAlias'}; } if ($data{'ports'}{$porti}{'ifType'} ne "propVirtual" and - $data{'ports'}{$porti}{'ifAlias'} =~ m/trunk/i) { - $json{'switches'}{$sysname}{'uplinks'}{'ifHCInOctets'} += $port{'ifHCInOctets'}; - $json{'switches'}{$sysname}{'uplinks'}{'ifHCOutOctets'} += $port{'ifHCOutOctets'}; - if ($port{'ifOperStatus'} eq "up") { - $json{'switches'}{$sysname}{'uplinks'}{'live'} += 1; + $data{'ports'}{$porti}{'ifAlias'} =~ m/LAG member/i) { + if ($port{'ifAdminStatus'} eq "up") { + $json{'switches'}{$sysname}{'uplinks'}{'ifHCInOctets'} += $port{'ifHCInOctets'}; + $json{'switches'}{$sysname}{'uplinks'}{'ifHCOutOctets'} += $port{'ifHCOutOctets'}; + if ($port{'ifOperStatus'} eq "up") { + $json{'switches'}{$sysname}{'uplinks'}{'live'} += 1; + } + $json{'switches'}{$sysname}{'uplinks'}{'total'} += 1; + } + } + if ($data{'ports'}{$porti}{'ifType'} ne "propVirtual" and + $data{'ports'}{$porti}{'ifAlias'} =~ m/Clients/i) { + if ($port{'ifAdminStatus'} eq "up") { + $json{'switches'}{$sysname}{'clients'}{'ifHCInOctets'} += $port{'ifHCInOctets'}; + $json{'switches'}{$sysname}{'clients'}{'ifHCOutOctets'} += $port{'ifHCOutOctets'}; + if ($port{'ifOperStatus'} eq "up") { + $json{'switches'}{$sysname}{'clients'}{'live'} += 1; + } + $json{'switches'}{$sysname}{'clients'}{'total'} += 1; } - $json{'switches'}{$sysname}{'uplinks'}{'total'} += 1; } $json{'switches'}{$sysname}{ifs}{$smallport}{'ifHCInOctets'} += $port{'ifHCInOctets'} || 0; $json{'switches'}{$sysname}{ifs}{$smallport}{'ifHCOutOctets'} += $port{'ifHCOutOctets'} || 0; @@ -58,6 +73,9 @@ while ( my $ref = $q->fetchrow_hashref() ) { } $json{'switches'}{$sysname}{totals}{'total'} += 1; } + + $json{'switches'}{$sysname}{vcp}{jnxVirtualChassisPortInOctets} = $data{'vcp'}{'jnxVirtualChassisPortInOctets'}; + $json{'switches'}{$sysname}{vcp}{jnxVirtualChassisPortOutOctets} = $data{'vcp'}{'jnxVirtualChassisPortOutOctets'}; $json{'switches'}{$sysname}{'temp'} = $data{'misc'}{'jnxOperatingTemp'}{'7.1.0.0'} || $data{'misc'}{'ciscoEnvMonTemperatureStatusValue'}{'1011'} || $data{'misc'}{'ciscoEnvMonTemperatureStatusValue'}{'1008'}; $json{'switches'}{$sysname}{'time'} = $ref->{'time'}; } @@ -85,19 +103,32 @@ while ( my $ref = $q2->fetchrow_hashref() ) { if ($porti =~ m/\.0$/) { next; } - if ($data{'ports'}{$porti}{'ifAlias'} =~ m/client/) { + if ($data{'ports'}{$porti}{'ifAlias'} =~ m/client/i) { $smallport =~ s/[0-9-].*$//; } else { $json{'then'}{$sysname}{ifs}{$smallport}{'ifAlias'} = $port{'ifAlias'}; } if ($data{'ports'}{$porti}{'ifType'} ne "propVirtual" and - $data{'ports'}{$porti}{'ifAlias'} =~ m/trunk/i) { - $json{'then'}{$sysname}{'uplinks'}{'ifHCInOctets'} += $port{'ifHCInOctets'}; - $json{'then'}{$sysname}{'uplinks'}{'ifHCOutOctets'} += $port{'ifHCOutOctets'}; - if ($port{'ifOperStatus'} eq "up") { - $json{'then'}{$sysname}{'uplinks'}{'live'} += 1; + $data{'ports'}{$porti}{'ifAlias'} =~ m/LAG member/i) { + if ($port{'ifAdminStatus'} eq "up") { + $json{'then'}{$sysname}{'uplinks'}{'ifHCInOctets'} += $port{'ifHCInOctets'}; + $json{'then'}{$sysname}{'uplinks'}{'ifHCOutOctets'} += $port{'ifHCOutOctets'}; + if ($port{'ifOperStatus'} eq "up") { + $json{'then'}{$sysname}{'uplinks'}{'live'} += 1; + } + $json{'then'}{$sysname}{'uplinks'}{'total'} += 1; + } + } + if ($data{'ports'}{$porti}{'ifType'} ne "propVirtual" and + $data{'ports'}{$porti}{'ifAlias'} =~ m/Clients/i) { + if ($port{'ifAdminStatus'} eq "up") { + $json{'then'}{$sysname}{'clients'}{'ifHCInOctets'} += $port{'ifHCInOctets'}; + $json{'then'}{$sysname}{'clients'}{'ifHCOutOctets'} += $port{'ifHCOutOctets'}; + if ($port{'ifOperStatus'} eq "up") { + $json{'then'}{$sysname}{'clients'}{'live'} += 1; + } + $json{'then'}{$sysname}{'clients'}{'total'} += 1; } - $json{'then'}{$sysname}{'uplinks'}{'total'} += 1; } $json{'then'}{$sysname}{ifs}{$smallport}{'ifHCInOctets'} += $port{'ifHCInOctets'} || 0; @@ -110,6 +141,8 @@ while ( my $ref = $q2->fetchrow_hashref() ) { } $json{'then'}{$sysname}{totals}{'total'} += 1; } + $json{'then'}{$sysname}{vcp}{jnxVirtualChassisPortInOctets} = $data{'vcp'}{'jnxVirtualChassisPortInOctets'}; + $json{'then'}{$sysname}{vcp}{jnxVirtualChassisPortOutOctets} = $data{'vcp'}{'jnxVirtualChassisPortOutOctets'}; $json{'then'}{$sysname}{'temp'} = $data{'misc'}{'jnxOperatingTemp'}{'7.1.0.0'} || $data{'misc'}{'ciscoEnvMonTemperatureStatusValue'}{'1011'} || $data{'misc'}{'ciscoEnvMonTemperatureStatusValue'}{'1008'}; $json{'then'}{$sysname}{'time'} = $ref->{'time'}; } diff --git a/web/api/read/distro-management b/web/api/read/distro-management new file mode 100755 index 0000000..2aa3273 --- /dev/null +++ b/web/api/read/distro-management @@ -0,0 +1,31 @@ +#! /usr/bin/perl +# vim:ts=8:sw=8 + +use CGI qw(fatalsToBrowser); +use DBI; +use lib '/opt/gondul/include'; +use nms; +use nms::web; +use strict; +use warnings; +use Data::Dumper; + +my $target = $ENV{REQUEST_URI}; +$target =~ s/$ENV{SCRIPT_NAME}//; +$target =~ s/^\///; +my ($switch, $port) = split(/\//,$target,2); +my $q2; + +$nms::web::cc{'max-age'} = "5"; +$nms::web::cc{'stale-while-revalidate'} = "30"; + +$q2 = $nms::web::dbh->prepare('SELECT distro_name,sysname,subnet4,subnet6,traffic_vlan,distro_phy_port FROM switches WHERE placement is not null AND distro_name is not null AND distro_phy_port is not null'); + +$q2->execute(); +while (my $ref = $q2->fetchrow_hashref()) { + my $sysname = $ref->{'sysname'}; + my $distro = $ref->{'distro_name'}; + $nms::web::json{'distros'}{$distro}{$sysname} = $ref; +} + +finalize_output(); diff --git a/web/api/read/oplog b/web/api/read/oplog index 92511bb..fb187ea 100755 --- a/web/api/read/oplog +++ b/web/api/read/oplog @@ -16,6 +16,10 @@ while (my $ref = $query->fetchrow_hashref()) { $meh{'id'} = $ref->{'id'}; $meh{'systems'} = $ref->{'systems'}; $meh{'timestamp'} = $ref->{'timestamp'}; + $meh{'timestamp'} =~ s/ /T/; + $meh{'timestamp'} =~ s/\+00$/Z/; + $meh{'timestamp'} =~ s/\+(\d\d)$/+\1:00/; + push @{$nms::web::json{'oplog'}},\%meh; } diff --git a/web/api/read/switches-management b/web/api/read/switches-management index 425262e..9d8bc53 100755 --- a/web/api/read/switches-management +++ b/web/api/read/switches-management @@ -20,9 +20,9 @@ $nms::web::cc{'max-age'} = "5"; $nms::web::cc{'stale-while-revalidate'} = "30"; if (!defined($switch)) { - $q2 = $nms::web::dbh->prepare('select sysname,mgmt_v4_addr,subnet4,subnet6,mgmt_v6_addr,mgmt_v4_gw,mgmt_v6_gw,mgmt_vlan,traffic_vlan,poll_frequency,last_updated,distro_name,distro_phy_port,community from switches where placement is not null'); + $q2 = $nms::web::dbh->prepare('select sysname,host(mgmt_v4_addr) as mgmt_v4_addr,subnet4,subnet6,host(mgmt_v6_addr) as mgmt_v6_addr,host(mgmt_v4_gw) as mgmt_v4_gw,host(mgmt_v6_gw) as mgmt_v6_gw,mgmt_vlan,traffic_vlan,poll_frequency,last_updated,distro_name,distro_phy_port,community from switches where placement is not null'); } else { - $q2 = $nms::web::dbh->prepare('select sysname,mgmt_v4_addr,subnet4,subnet6,mgmt_v6_addr,mgmt_v4_gw,mgmt_v6_gw,mgmt_vlan,traffic_vlan,poll_frequency,last_updated,distro_name,distro_phy_port,community from switches where placement is not null and sysname = ' . $nms::web::dbh->quote($switch) . ';'); + $q2 = $nms::web::dbh->prepare('select sysname,host(mgmt_v4_addr) as mgmt_v4_addr,subnet4,subnet6,host(mgmt_v6_addr) as mgmt_v6_addr,host(mgmt_v4_gw) as mgmt_v4_gw,host(mgmt_v6_gw) as mgmt_v6_gw,mgmt_vlan,traffic_vlan,poll_frequency,last_updated,distro_name,distro_phy_port,community from switches where placement is not null and sysname = ' . $nms::web::dbh->quote($switch) . ';'); } $q2->execute(); diff --git a/web/css/nms.css b/web/css/nms.css index 17f4ff9..9c783af 100644 --- a/web/css/nms.css +++ b/web/css/nms.css @@ -127,4 +127,5 @@ div.map-mode-legend button { .graph { max-width: 100%; width: 100%; + height: 240px; } diff --git a/web/img/tg17-clean.png b/web/img/tg17-clean.png Binary files differnew file mode 100644 index 0000000..3701b77 --- /dev/null +++ b/web/img/tg17-clean.png diff --git a/web/img/tg17-salkart-full.png b/web/img/tg17-salkart-full.png Binary files differnew file mode 100644 index 0000000..a7e90b9 --- /dev/null +++ b/web/img/tg17-salkart-full.png diff --git a/web/index.html b/web/index.html index 3b8e7a5..375b46b 100644 --- a/web/index.html +++ b/web/index.html @@ -37,12 +37,12 @@ <div id="navbar" class="navbar-collapse collapse"> <p class="navbar-text"><span class="glyphicon glyphicon-heart" id="heartbeat" aria-hidden="true"></span></p> <ul class="nav navbar-nav"> - <li id='map-link' class="active"><a href="#" onclick="nmsUi.setActive('map');">Map<span class="sr-only">(current)</span></a></li> - <li id='admin-link' class="gondul-is-private"><a href="#" onclick="nmsUi.setActive('admin');nmsAdmin.updateConfigPane();">Admin</a></li> - <li id='template-link' class="gondul-is-private"><a href="#" onclick="nmsUi.setActive('template');">Templates</a></li> - <li id='oplog-link' class="gondul-is-private"><a href="#" onclick="nmsUi.setActive('oplog');">Oplog</a></li> + <li id='map-link' class="active"><a href="#" onclick="nmsUi.setActive('map');" title="Map">M<span class="sr-only">(current)</span></a></li> + <li id='admin-link' class="gondul-is-private"><a href="#" onclick="nmsUi.setActive('admin');nmsAdmin.updateConfigPane();" title="Admin">A</a></li> + <li id='template-link' class="gondul-is-private"><a href="#" onclick="nmsUi.setActive('template');" title="Template">T</a></li> + <li id='oplog-link' class="gondul-is-private"><a href="#" onclick="nmsUi.setActive('oplog');" title="Oplog">O</a></li> <li class="dropdown"> - <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">Menu + <a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false" title="Menu">M <span class="caret"></span> </a> <ul class="dropdown-menu" role="menu"> @@ -82,7 +82,7 @@ </li> <div class="navbar-form navbar-left"> <div class="input-group input-group-sm"> - <input id="searchbox" type="text" class="form-control" placeholder="Filter" oninput="nmsSearch.search()" /> + <input id="searchbox" type="text" class="form-control" size="8" placeholder="Filter" oninput="nmsSearch.search()" /> <span class="input-group-btn"> <button id="searchbox-x" class="btn btn-default" type="button" onclick="nmsSearch.reset();">X</button> <button id="searchbox-help" class="btn btn-default" type="button" onclick="nmsInfoBox.showWindow('searchHelp');">?</button> @@ -92,7 +92,7 @@ </div> <p class="navbar-text" id="dhcp-summary">(...)</p> <div class="navbar-text" id="navbar-graph-div"> - <img id="navbar-graph" title="Blue: Average latency. Green: Total traffic. Time frame: last 60 minutes."/> + <img id="navbar-graph" style="display:none" title="Blue: Average latency. Green: Total traffic. Time frame: last 60 minutes."/> </div> </li> </ul> @@ -317,7 +317,7 @@ </div> </div> - <div style="display:none;"><img id="source" src="img/tg16-salkart-clean-big.png" ></div> + <div style="display:none;"><img id="source" src="img/tg17-clean.png" ></div> </div> </div><!--/.fluid-container--> <script src="js/jquery.min.js" type="text/javascript"></script> diff --git a/web/js/nms-info-box.js b/web/js/nms-info-box.js index 6e3f31c..cd84ec0 100644 --- a/web/js/nms-info-box.js +++ b/web/js/nms-info-box.js @@ -882,10 +882,23 @@ var inventoryListingPanel = function() { value = nmsData.snmp.snmp[sw]["misc"]["sysDescr"][0]; break; case 'jnxBoxSerialNo': + if(testTree(nmsData,["snmp","snmp",sw,"misc","entPhysicalSerialNum"])) { + for (var x in nmsData.snmp.snmp[sw]["misc"]["entPhysicalSerialNum"]) { + value = "misc" + x + ":" + nmsData.snmp.snmp[sw]["misc"]["entPhysicalSerialNum"][x]; + resultArray.push([sw, value]); + } + } + if (testTree(nmsData,["snmp","snmp",sw,"misc","jnxVirtualChassisMemberSerialnumber"])) { + for (var x in nmsData.snmp.snmp[sw]["misc"]["jnxVirtualChassisMemberSerialnumber"]) { + value = "member " + x + ":" + nmsData.snmp.snmp[sw]["misc"]["jnxVirtualChassisMemberSerialnumber"][x]; + resultArray.push([sw, value]); + } + } value = nmsData.snmp.snmp[sw]["misc"]["jnxBoxSerialNo"][0]; break; } - } catch (e) {} + } catch (e) {console.log("sw: " + sw); console.log(e);} + resultArray.push([sw, value]); } @@ -1062,7 +1075,7 @@ var switchCommentsPanel = function () { var tmp = (date.getYear() + 1900) + "-" + (month < 10 ? "0": "") + month + "-" + (day < 10 ? "0" : "") + day + " " + date.toTimeString().replace(/:\d\d .*$/,""); td1.textContent = tmp; td1.classList.add("left"); - td2.textContent = "[" + logs[v]['username'] + "] " + logs[v]['log']; + td2.textContent = logs[v]['systems'] + "[" + logs[v]['username'] + "] " + logs[v]['log']; } domObj.appendChild(table); this._render(domObj); diff --git a/web/js/nms-map-handlers.js b/web/js/nms-map-handlers.js index 42785ff..03f5c9e 100644 --- a/web/js/nms-map-handlers.js +++ b/web/js/nms-map-handlers.js @@ -68,6 +68,7 @@ var handler_snmp = { var handler_cpu = { init:cpuInit, + getInfo:cpuInfo, tag:"cpu", name:"CPU utilization" }; @@ -84,6 +85,11 @@ var handler_mgmt = { tag:"mgmt", name:"Management info" }; +var handler_snmpup = { + getInfo:snmpUpInfo, + tag:"snmpup", + name:"SNMP Uplink state" +}; var handlerInfo = function(tag,desc) { /* @@ -126,7 +132,8 @@ var handlers = [ handler_traffic_tot, handler_dhcp, handler_snmp, - handler_cpu + handler_cpu, + handler_snmpup ]; function uplinkInfo(sw) @@ -136,31 +143,56 @@ function uplinkInfo(sw) ret.score = 0; var u = 0; var t = 0; + var known_t = 0; if (testTree(nmsData,['switchstate','switches',sw,'uplinks','live'])) { u = parseInt(nmsData.switchstate.switches[sw].uplinks.live); t = parseInt(nmsData.switchstate.switches[sw].uplinks.total); + known_t = t; ret.data[0].value = u + " / " + t; ret.data[0].description = "Uplinks (live/configured)"; if (nmsData.switches.switches[sw].subnet4 == undefined || nmsData.switches.switches[sw].subnet4 == null) { - if (u == t) { - ret.score = 0 - ret.why = "All uplinks up"; - } else if (u == 1) { - ret.score = 800; - ret.why = "Only 1 of " + t + " uplinks alive"; - } else { - ret.score = 650; - ret.why = u + " of " + t + " uplinks alive"; - } + if (tagged(sw,'3up')) { + known_t = 3; + } else if (tagged(sw,'2up')) { + known_t = 2; + } else if (tagged(sw, '1up')) { + known_t = 1; + } else if (tagged(sw,'4up')) { + known_t = 4; + } + if (known_t != t) { + ret.data[0].value += "(Overridden: " + known_t + ")"; + } + + if (u == known_t) { + ret.score = 0; + ret.why = "All uplinks up"; + } else if (u == 1) { + ret.score = 800; + ret.why = "Only 1 of " + known_t + " uplinks alive"; + } else if (u < known_t) { + ret.score = 450; + ret.why = u + " of " + known_t + " uplinks alive"; + } else if (u > known_t) { + ret.score = 350; + ret.why = u + " of " + known_t + " uplinks alive"; + } } } + if (testTree(nmsData,['switchstate','switches',sw,'clients','live'])) { + var tu = parseInt(nmsData.switchstate.switches[sw].clients.live); + var tt = parseInt(nmsData.switchstate.switches[sw].clients.total); + ret.data[1] = {}; + ret.data[1].value = (tu) + " / " + (tt); + ret.data[1].description = "Client ports (live/total)"; + } if (testTree(nmsData,['switchstate','switches',sw,'totals','live'])) { var tu = parseInt(nmsData.switchstate.switches[sw].totals.live); var tt = parseInt(nmsData.switchstate.switches[sw].totals.total); - ret.data[1] = {}; - ret.data[1].value = (tu-u) + " / " + (tt-t); - ret.data[1].description = "Non-uplink ports (live/total)"; + ret.data[2] = {}; + ret.data[2].value = (tu-u) + " / " + (tt-t); + ret.data[2].description = "Non-uplink ports (live/total)"; } return ret; } @@ -422,7 +454,7 @@ function pingInfo(sw) ret.why = "Latency"; ret.score = parseInt(v4 > v6 ? v4 : v6); } - if (nmsData.ping.switches[sw].age4 > 5 || nmsData.ping.switches[sw].age6 > 5) { + if (nmsData.ping.switches[sw].age4 > 10 || nmsData.ping.switches[sw].age6 > 10) { ret.why = "Old ping"; ret.score = 900; } @@ -431,6 +463,31 @@ function pingInfo(sw) ret.why = "No ping replies"; ret.score = 999; } + + if (testTree(nmsData,['smanagement','switches',sw])) { + try { + var distro = nmsData['smanagement']['switches'][sw]['distro_name']; + var phy = nmsData['smanagement']['switches'][sw]['distro_phy_port']; + if (!(distro == "" || phy == "" || distro == undefined || phy == undefined)) { + if (testTree(nmsData,['snmp','snmp',distro, 'ports',phy,'ifOperStatus'])) { + var x = nmsData['snmp']['snmp'][distro]['ports'][phy]['ifOperStatus']; + var ping = parseFloat(nmsData["ping"]["switches"][sw]["latency4"]); + var ping6 = parseFloat(nmsData["ping"]["switches"][sw]["latency6"]); + if (x == "up") { + ret.data[3] = {}; + ret.data[3].description = "Distro-port"; + ret.data[3].value = "Distro port is live"; + if (isNaN(ping) && isNaN(ping6)) { + ret.score = 850; + ret.why = "Distro port is alive, but no IPv4/IPv6 ping. ROLLBACK!"; + } + } + } + } + } catch(e) { + console.log("lol"); + } + } return ret; } @@ -478,14 +535,19 @@ function dhcpUpdater() function dhcpInfo(sw) { var ret = new handlerInfo("dhcp","DHCP state"); ret.why = "No DHCP data"; - ret.data[0].description = "DHCP age (seconds)"; + ret.data[0].description = "DHCP age"; if (testTree(nmsData,['dhcp','dhcp',sw])) { var now = nmsData.dhcp.time; var then = nmsData.dhcp.dhcp[sw]; var diff = now - then; - ret.data[0].value = diff; + var divider = 6; + if(tagged(sw,'slowdhcp')) { + divider = 12; + } + + ret.data[0].value = secondsToTime(diff); ret.why = "DHCP freshness"; - ret.score = diff/4> 500 ? 500 : parseInt(diff/4); + ret.score = diff/divider> 350 ? 350 : parseInt(diff/divider); } else { ret.data[0].value = "No DHCP data"; if (testTree(nmsData,['smanagement','switches',sw])) { @@ -495,7 +557,7 @@ function dhcpInfo(sw) { ret.score = 0; ret.why = "No subnet registered"; } else { - ret.score = 500; + ret.score = 350; ret.why = "No DHCP data"; } } else { @@ -509,9 +571,10 @@ function dhcpInfo(sw) { ret.data[1].description = "DHCP clients"; } if (testTree(nmsData,['switches','switches',sw, 'tags'])) { - if (nmsData.switches.switches[sw].tags.includes('ignoredhcp')) { + if (tagged(sw,'ignoredhcp')) { ret.score = 0; ret.why += "(Ignored)"; + ret.data[0].value += "(Ignored)"; } } return ret; @@ -630,6 +693,7 @@ function snmpInfo(sw) { return ret; } + function snmpInit() { nmsData.addHandler("snmp", "mapHandler", snmpUpdater); @@ -641,6 +705,59 @@ function snmpInit() { snmpUpdater(); } +function snmpUpInfo(sw) { + var ret = new handlerInfo("snmpup","SNMP uplink data"); + ret.why = "No SNMP data"; + ret.score = 0; + + if (testTree(nmsData,['snmp','snmp',sw, 'ports'])) { + var total_up = 0; + var seen_up = 0; + for (var port in nmsData.snmp.snmp[sw].ports) { + var x = nmsData.snmp.snmp[sw].ports[port]; + if (x["ifAlias"].match(/Uplink/i) && x["ifOperStatus"] == "up") { + total_up += parseInt(x["ifHighSpeed"]); + } + if (x["ifAlias"].match(/LAG Member/i) && x["ifOperStatus"] == "up") { + seen_up += parseInt(x["ifHighSpeed"]); + } + } + ret.data[0].value = "LAG member speed and total speed is " + seen_up; + if (total_up != seen_up) { + ret.score = 500; + ret.why = "LAG member (ge/xt/etc) speed is " + seen_up + " but logical (ae) is " + total_up; + ret.data[0].value = ret.why; + } + } + return ret; +} + +function cpuInfo(sw) { + var ret = new handlerInfo("cpu","CPU utilization"); + ret.why = "No CPU info"; + ret.score = 0; + + if (testTree(nmsData,['snmp','snmp',sw, 'misc','jnxOperatingCPU'])) { + var cpu = 0; + for (var u in nmsData.snmp.snmp[sw].misc.jnxOperatingCPU) { + var local = nmsData.snmp.snmp[sw].misc['jnxOperatingCPU'][u]; + cpu = Math.max(nmsData.snmp.snmp[sw].misc.jnxOperatingCPU[u],cpu); + } + if (cpu < 30) { + ret.score = 0; + } else if (cpu < 50) { + ret.score = 100; + } else if (cpu < 95) { + ret.score = cpu * 2; + } else { + ret.score = cpu * 4; + } + ret.why = "CPU utilization: " + cpu + "%"; + ret.data[0].value = cpu + "%"; + } + return ret; +} + function cpuUpdater() { for (var sw in nmsData.switches.switches) { try { @@ -686,6 +803,9 @@ function mgmtInfo(sw) { }, { value: mg.subnet6 || "N/A", description: "Subnet (v6)" + }, { + value: mg.distro_name || "N/A", + description: "Distro" }]; if ((mg.mgmt_v4_addr == undefined || mg.mgmt_v4_addr == "") && (mg.mgmt_v6_addr == undefined || mg.mgmt_v6_addr == "")) { ret.why = "No IPv4 or IPv6 mamagement IP"; diff --git a/web/js/nms-map.js b/web/js/nms-map.js index 5669acb..1ebcf92 100644 --- a/web/js/nms-map.js +++ b/web/js/nms-map.js @@ -38,7 +38,7 @@ var nmsMap = nmsMap || { textMargin: 4, xMargin: 10, yMargin: 20, - fontSize: 15, + fontSize: 14, fontFace: "sans-serif" }, scale: 1, diff --git a/web/js/nms-oplog.js b/web/js/nms-oplog.js index c418e33..7ac11c4 100644 --- a/web/js/nms-oplog.js +++ b/web/js/nms-oplog.js @@ -59,7 +59,7 @@ nmsOplog.commit = function() { } nmsOplog.updateComments = function() { - nmsOplog._updateComments(3,"-mini","time",100); + nmsOplog._updateComments(10,"-mini","time",100); nmsOplog._updateComments(0,"","timestamp"); } @@ -89,6 +89,9 @@ nmsOplog._updateComments = function(limit,prefix,timefield,cutoff) { table.classList.add("table-condensed"); var i = 0; for (var v in nmsData['oplog']['oplog']) { + if (cutoff && nmsData.oplog.oplog[v]['username'] == "system") { + continue; + } tr = table.insertRow(-1); td1 = tr.insertCell(0); td2 = tr.insertCell(1); diff --git a/web/js/nms-search.js b/web/js/nms-search.js index 7528a82..41e5397 100644 --- a/web/js/nms-search.js +++ b/web/js/nms-search.js @@ -105,6 +105,22 @@ nmsSearch.searchTest = function(id, sw) { } } } catch (e) {} + try { + for (var x in nmsData.snmp.snmp[sw].misc.jnxBoxSerialNo) { + if (nmsData.snmp.snmp[sw].misc.jnxBoxSerialNo[x] == null) { + continue; + } + if (nmsData.snmp.snmp[sw].misc.jnxBoxSerialNo[x] == undefined) { + continue; + } + if (nmsData.snmp.snmp[sw].misc.jnxBoxSerialNo[x] == "") { + continue; + } + if (re.test(nmsData.snmp.snmp[sw].misc.jnxBoxSerialNo[x])) { + return true; + } + } + } catch (e) {} if (re.test(nmsData.snmp.snmp[sw].misc.sysDescr[0])) { return true; } diff --git a/web/js/nms.js b/web/js/nms.js index 362aebc..8f98c75 100644 --- a/web/js/nms.js +++ b/web/js/nms.js @@ -395,8 +395,8 @@ function initNMS() { nmsData.registerSource("snmp","/api/read/snmp"); nmsData.registerSource("smanagement","/api/read/switches-management"); nmsData.registerSource("oplog", "/api/read/oplog"); - setInterval(nmsUpdateNavbarGraph, 10000); - nmsUpdateNavbarGraph(); + // setInterval(nmsUpdateNavbarGraph, 30000); + // nmsUpdateNavbarGraph(); nmsOplog.init(); } @@ -624,11 +624,9 @@ function restoreSettings() */ function nmsUpdateNavbarGraph() { var img = document.getElementById("navbar-graph"); - var w = Math.floor(window.innerWidth / 4); - if (window.innerWidth > 2300) - w += 400; + var w = 200; - img.src = "/render/?target=movingAverage(averageSeries(ping.*.ipv4),%225min%22)&target=secondYAxis(averageSeries(perSecond(snmp.*.ports.*.{ifHCInOctets,ifHCOutOctets})))&bgcolor=%23ffffff00&width=" + w + "&height=20&format=svg&from=-60min&until=now&graphOnly=true&somerandomthing=" + Math.floor(new Date().getTime() / 10000); + img.src = "/render/?target=movingAverage(averageSeries(ping.*.*.ipv4),%225min%22)&target=secondYAxis(averageSeries(perSecond(snmp.*.*.ports.*.{ifHCInOctets,ifHCOutOctets})))&bgcolor=%23ffffff00&width=" + w + "&height=20&format=svg&from=-30min&until=now&graphOnly=true"; } /* * Test if the entire path specified in the arrary "ar" exists under the |
