NetHorus Development Blog

What a miserable week it’s been for the weather! Here are the latest developments on NetHorus this week:

• Error messages - in particular, the flash messages, are now much prettier and will grey the background out to make the user aware something’s gone wrong.
• Configuration files - NetHorus will now check that a configuration file has been loaded, and raise an error if it hasn’t.
• Testing - we have some tests for the EventLog model, updated tests for the Ipv4Network models.
• Interface dependencies using the IF-MIB::ifStackTable are now working, although the table isn’t straightforward to parse. Credit to Brian Candler for explaining the acts_as_tree plugin to me.
• Model changes - the Ipv4Network model has been tidied up - testing proved that there were some cases where invalid data would be passed and cause errors.
• Credits - Martin Gleadow is now credited properly for keeping tabs on dependencies for the latest releases - something I should write up in the Wiki soon.
• Bugfixes - we destroy_all event logs for a device, rather than delete_all-ing them. This removes the data from the table, rather than breaking the device-to-eventlog relationship and leaving the data sitting stale.
• Interface layout - improvements to make things look prettier and simpler.
• Device inventorying now uses IF-MIB::ifHighSpeed if present to record the speed of an interface, since it appears generally more accurate than IF-MIB::ifSpeed.

So, that’s the past week. How did this week’s planned features go? Not too well…

• SNMPv3 support is still incomplete, I will be de-prioritizing this as it isn’t strictly required for any new functionality, but will be a very-nice-to-have.
• VRF support is lacking - the VRF Lite functionality on my Cisco 877W is slightly limited. I’m aware that full-fat VRF on bigger Cisco boxes works well, and I’m aiming to get started with VRF support this week.
• ifStackTable support is complete in the back-end database, however I want to come up with a way of displaying interface relationships visually.

And finally, here’s what I have planned for the coming weeks:

• ARP table polling - read in the ARP table from a device periodically and use it to track the Layer 2 addresses of devices on the network, and will also be useful for device discovery (incidentally, something I despite but I’m aware it’s a good way to get up and running).
• Layer 2 Forwarding table polling - used in combination with ARP table polling, will allow us to track devices throughout the network.
• Cisco CDP support - for displaying connected neighbour switches.
• ATM VC/VP information - for displaying details of virtual circuits and virtual paths on an ATM switch.
• Frame Relay PVC information - to display PVCs on an interface, their details, etc.
• Neighbour relationships - automatically detecting which devices share common subnets.
• Configuration generation - for Nagios primarily, so we can generate configurations based on the topology information stored in NetHorus. No need to set everything up once in NetHorus and again in Nagios!

Phew. That’s a lot of stuff, but it gives you an idea of what I have planned for the future.

As always, your feedback is important - please email me: peter.hicks@nethorus.org.

Summer is finally here - the last few days in the UK have been pleasantly warm. Us Brits are much happier when there’s direct sunlight!

Despite first glances, there’s been a lot of work going on this week - here’s a summary:

• Error handling and testing has been this week’s theme. If the SNMP proxy isn’t working, or we receive errors, there’s a much prettier ‘flash’ that appears on-screen.
• SNMP improvements - in particular, we now supply MIBs for use with NetHorus as there are some really rather badly maintained MIBs out there. These hang out in mibs/ and will automatically be loaded on startup.
• Application configuration is taken care of by appconfig.yml. You will need to rename config/appconfig.example to config/appconfig.yml for the application to work properly!
• SNMP proxy support has been greatly improved, thanks to Brian Candler’s suggestions. The proxy will now daemonize, write a PID file, provide debugging information… much more useful than before.

Features in the pipeline - this week, if I get time amidst reunions with old colleagues and trips to Dublin:

• SNMPv3 support as suggested by Martin Gleadow. Some of the database schema is already in place.
• VRF support and the ability to add VRF information in to interface inventories.
• ifStackTable support so we can identify which interfaces are children of which other interfaces - this will become useful later on in diagramming and monitoring

Please continue to send in your feedback to peter.hicks@nethorus.org - and note I’ve fixed email now so it won’t bounce. Oops.

Another week has flown by!

• Code tidyups and lots of them - more comments and more error trapping, some tidier HTML and tidier configuration files.
• Device inventorying has been improved and we now take notice of the ifConnectorPresent OID for each interface so we can identify physical and logical interfaces.
• Device-specific information is now in place for ADSL interfaces. If you have an ADSL interface, try examining the interface detail for the interface ending -adsl.
• SNMP proxy now loads all MIB files.

I gave a very quick and hairy demo of NetHorus to a couple of people I met on Thursday, and I’ve had some vague interest from a few other people.

Please spread the word and mail me with helpful suggestions on what you want to see implemented.

It’s just over a week since the last update, and thanks to being away from work for a few days, I’ve had plenty of time to hack away at Ruby.

• Milestone - the 100th check-in!
• Unit tests for some of the models, a process which is remarkably straightforward and I’m really surprised I didn’t grok how easy this was before now.
• SNMP proxy functionality is now working - this requires perl, SNMP.pm and a collection of MIB files. Remember to start external/snmp_proxy.pl!
• Database changes - the iftype association is gone, since this was a hack to get around Ruby’s lack of MIB-parsing.
• Model changes - device models now have get_oid and get_table methods which use the SNMP proxy, providing a useful abstraction from the guts of the SMP proxy.
• UI changes - the interface layout has been refreshed.

For better or worse, there’s now an SNMP proxy in NetHorus, written in perl and using XML-RPC.  Ruby purists will despise me, but I have my reasons for doing it this way, and they’re all centred around functionality.

In the beginning, NetHorus was called NETMAN, written in PHP and Perl.  I tried to rewrite the project entirely in Perl using Catalyst in 2006, but failed through a lack of time and available books on Catalyst.  That’s not to say Catalyst is a bad framework - far from it - but I wanted something a little more ‘me’.

My decision to re-code in Ruby using Rails wasn’t too difficult, although SNMP support was a major concern.  SNMP.pm for Perl is awesome - feature-rich, supported, quick… all the things that Ruby’s only SNMP implementation isn’t, despite what its website proclaims.

Right now, I’m at the stage where I could develop snmp.rb, bolt on MIB parsing using libsmi, then struggle down the path of implementing SNMPv3, and lose sight of what I’m really supposed to be doing.  Or, I could think of another way - such as using XML-RPC, possibly with BEEP, to an SNMP proxy written in Perl which then dispatches requests.

In a quick brainstorming session at $DAYJOB, I came up with some advantages to handing off SNMP to another device:

• SNMP doesn’t work terribly quickly over high latency paths, so why not use a TCP-based connection as far as possible, and do all the SNMP donkey-work closer to the other endpoint?
• Anything other than SNMPv3 isn’t encrypted in the slightest, and not everyone has the infrastructure to ensure an encrypted path between an SNMP manager and an SNMP agent.  Put the proxy on a remote device if you want, and suddenly it becomes easier.
• Embedded devices behind firewalls.  Your network suddenly becomes a little easier to manage…

So that’s where it’ll go.  It’s not as light as SNMP, but it’s more extensible than what Ruby has to offer.  The SNMP proxy doesn’t have to be written in perl either.

For now, I’m concentrating on the basic functionality of getting single and multiple OIDs, and walking tables.  The rest will come later.

Despite a very busy week in my day job, progress has continued.  Here’s a summary of the last week or so’s progress:

• Interface types - these now work without the unpleasant override of :foreign_key.  This is a bit of a hack, setting iftype.id to be self.iana_iftype, but it gets around being unable to unprotect the :id column in ActiveRecord.  Sometimes you want to play with fire.
• Visual tidyups - the admin layout has been removed, and I’ve removed the ugly HTML redirect to /welcome.  There’s also a prettier layout with some colours and the NetHorus logo, and a welcome message when on the main index.
• Search function - it is now possible to search for a device or interface based on hostname, interface name, interface description or IPv4 address.
• Device images - the inventory function will try to guess what type of device it is inventorying, based on the sysServices.0 OID.  Currently, Layer 3 switches and Routers are identified, but detection of which device is which may be a little flaky.
• Device features - the framework for managing specific functions on devices, such as OSPF on a router that supports it, or BGP, or MPLS, has been written. Features will appear in a drop-down box on the management page for a device.  Be aware that nothing uses this functionality yet…
• Event logs - when a device is inventoried, an event is logged against the device.  This will be extended to include other events, such as SNMP traps or detected errors on interfaces.
• Internal stuff - some of the database migrations had the same names as previous migrations, so were never called.  This has been tidied up.

I’m on holiday this week and next, so expect my laptop to be by my side on train journeys and plenty more updates produced.  Here’s some of the things I’m hoping to be able to do:

• Oracle support - thanks to Oracle Database 10g Express Edition.  If this works successfully, expect to see this as a supported backend.  PostgreSQL will follow.
• SNMP proxy - since Ruby’s SNMP support doesn’t extend to interpreting SMI data (MIB files, basically), I’m going to write a proxy in Perl - which somebody could re-code in to C or C++ if they wished. This will accept SOAP calls and take away the need to handle SNMP in Ruby itself.  Tangential benefits of this could mean that SNMP proxies could be installed ‘closer’ to target devices.
• BGP and OSPF support - using the features framework.  A view of BGP peers and OSPF neighbours would be awesome.

Please keep the feedback coming in to peter.hicks@poggs.co.uk - without it, I won’t know what features other people consider popular!

So much has happened over the past week. I can’t quite believe how straightforward Rails is. It isn’t without its problems, but most of them are down to the author’s growing knowledge rather than Ruby itself.

In summary:

• Devices now show a ‘last inventory’ timestamp to help identify when they were last inventoried
• ActiveRecord editing/searching/browsing works as it should
• IPv4 network names are looked up and displayed next to IPv4 addresses, e.g. “192.168.1.5 - Home Network”. The closest match is chosen based on netmask.
• Device inventorying has been simplified - there is now just one inventory function which will be extended as necessary, and a ‘delete inventory’ function for when you want to clear information about a device from the database.

As usual, lots of database changes, so please rake db:migrate when you check out the latest code from Subversion.

So, what’s been going on?

• Layer 3 interfaces are now probed correctly, but there’s still work to do so that changes are correctly identified.
• A lot of database schema changes - remember to ‘rake db:migrate’ before checking out a new version from Subversion
• The concept of ’sites’ for devices and subnets is gradually being introduced
• Views are being tidied up

Oracle support is still in the pipeline, as is the IPv4 database and searching. But it’s looking good so far, and I’m trying my best to ensure that the Subversion repository always contains a working copy of NetHorus.

Finally, have a look at http://trac.nethorus.org/ - I’m going to use this for issue tracking.

NetHorus is starting to take shape. If you check out the current SVN repository, you’ll find all the source code you need to start testing - although there isn’t much of it.

Plans for the next few weeks:

• Documentation of Layer 3 interfaces, as well as Layer 2 (which is more complicated than it sounds)
• Support for Oracle as a backend database
• IPv4 address database and allocation management
• Search for an interface by IP address, MAC address or description
• Basic diagramming - “what’s connected to this subnet?”