| Previous |
FreePDB is a suite of tools to help create a Performance Database (PDB) using free (free beer, free speech, free whatever) software.
The idea behind FreePDB is provide a flexible way to collect performance and capacity data into a database. Why flexible? The idea is that there is a lot of easy to access data out there that if we collect it all together then one day it might be useful for something. Some of that data is collected by existing tools into text files of one sort or another. Other data can be collected at a point in time doing some sort of query.
The key issue for me is that at any point in time I don’t actually know what I want to know. If lots of potentially useful data was available in a general purpose database then I could invent queries or write data mining programs to find things out.
Short term data (what is happening right now) is easy enough to find using existing tools. The FreePDB suite of tools ignores the issue of rapid access to data. Rather than helping you solve the ‘why is my system behaving badly right now?’ or even ‘why has my system been running slow for the last hour?’ it helps you deal with the ‘what has been getting steadily worse and worse on the system over the last week? month?’.
Long term data is excellent for use in capacity planning and capacity management. In capacity planning you are pretty much unconcerned about occasional blips in the system’s performance so having a lot of data to smooth your way through helps to show you the long term picture. Long term data is also helpful in building usage models for your systems. Past use is an important input (along with business intelligence) to predicting future load. Although capacity planning is generally concerned with the big ticket items (cpu, memory, disk, network), modelling of proposed changes may require delving into your data in more detail than you were expecting. Collecting more than the absolute minimum data helps to facilitate this.
Flexibility really means two things:
Adding new fields to your collection is really only feasible if there is a very low incremental cost in doing so. Otherwise our data collection process becomes a major performance/capacity problem in its own right. An example of this would be if you were already collecting data on numbers of packets passing through a router at any given time. You would do this by sending an SNMP query to the device and it sends back the information. Typically this requires a single UDP datagram in each direction. If we were to also decide to collect some extra data, such as number of bytes, we can piggy back the request on the existing UDP request and have the reply hitch a ride on the UDP reply packet. While the packets will be a little larger, no new packets are generated and the security authentication/authorisation process is only done once. There will be a slight incremental cost in processing and storage but these will typically be very small.
There are few major components to FreePDB (actually “major” is a bit of an overstatement – the code is actually quite small). The first is a program called perfcollect.pl that’s job is to load csv (comma separated values) or tsv (tab separated values) text files into tables in a database. Typically you would do this once a day. The files can come from all sorts of places including, very significantly, the windows performance monitor: perfmon.
Because not all data is already collected into text files by some other kind hearted program, there is also a program called perflog.pl that polls data from local or remote systems and builds the results into text files. Collected data can come from SNMP, files (including the proc filesystem for the Linux folk), commands, sysctl (for more enlightened systems), and plugins for other products like MRTG. Typically this program runs once every 5 minutes (fairly paranoid) to 15 minutes (not bad granularity for most uses).
Since both these programs are potentially working with the same data, the two phases are controlled by a single XML configuration file.
In addition to this there is a simple reporting program (makegraphs.pl) that helps you to set up some simple graphs to include on a web site or to paste into management reports, and some database housekeeping work.
There are also a few special purpose programs that scan some mongrel format text files such as system logs or ntp synchronisation logs. These aren’t so much directly useful programs directly since every case is different. They don’t even server as particularly good examples because they were written fairly early on the design process. They are provided simply because they may be useful to someone, somewhere.