Archive for the ‘Software’ Category

QueryForm

Monday, March 19th, 2007

QueryForm: a Swing-based Java tool for editing database tables.

For those times when you just don’t want to write a CRUD interface or use raw SQL. DbVisualizer’s “Update where” and “Insert into” functionality is OK for one or two rows, but this tool lets you edit many rows almost painlessly.

powered by performancing firefox

DbVisualizer

Monday, March 19th, 2007

DbVisualizer: essential tool. I practically live in it. I have it open all day every day. I use it for ad-hoc reporting, quick updates, and to build up, debug and validate the queries I put into my software. It will connect to anything that has a JDBC driver.

One of the features I’ve used is the charting of foreign-key relationships to build a cube-wall sized poster of the data model for a product we installed. I pasted together 9 letter-sized sheets (the product knows how to print the huge image in segments). Life saver.

I use the free version, and I’ve even tried out the “personal” (i.e. for-pay) version on a trial license for a while and the graphing capability was pretty fun too, but I have other ways to do that and so didn’t go for it.

powered by performancing firefox

JEdit – Programmer’s Text Editor

Tuesday, December 5th, 2006

My editor of choice, when not using an IDE like NetBeans:

jEdit – Programmer’s Text Editor

Features:

  • It’s written in Java so it’s cross platform
  • It has a million plug ins that do useful things
  • Free, open source.
  • It has a nice XML/HTML editing mode that does “code completion” on tags and attributes, and gives you a tree view and validation error list.
  • It has syntax coloring for more computer languages than you can shake a Turing Machine at.
  • It can even compile and run code from within the editor, so if you don’t have an IDE, you can turn it into a poor-man’s one, or use it for languages that don’t otherwise have an IDE, like Perl.
  • It can remotely edit files over [S]FTP.
  • There’s CVS integration (…note to self: check for Subversion integration…)

powered by performancing firefox

XMLmind XML Edtior

Tuesday, December 5th, 2006

Here’s a tool I really like, for editing XML files, especially ones that have useful “human readable” representations, like XHTML:

XMLmind XML Editor

Features:

  • There’s a “free” version (lacking some features). Professional version isn’t unreasonable for a single seat.
  • Written in Java so it’s cross platform
  • You can easily create new templates for new XML schemas (just create or point at a DTD and a CSS stylesheet from a valid, minimal, but otherwise empty XML file conforming to the DTD)
  • Has a plug-in architecture, and there are a few plugins you can get right from their site, as well as write your own.

I find myself using it to edit XHTML, DocBook and other XML file types (e.g. DITA).

The advantage over something like JEdit is that rather than telling you when you create invalid markup, it won’t let you create invalid markup. And it certainly won’t let you create non-well-formed markup. (If the preceeding distinction isn’t clear then read the definitions of well-formed and valid.)

powered by performancing firefox

Quis Custodiet Ipsos Custodes? Monitoring your monitoring system

Monday, June 21st, 2004

The group I’m in here at a major telecommunications provider has a nice little setup of HP’s OpenView and Remedy’s Action Request System.

OpenView listens for and seeks out problems with various internal
and customer-owned systems. When something serious occurs, it uses some
3rd party integration software (RemedySPI) to create a trouble ticket
in Remedy. Rules fire to assign the ticket to an appropriate “triage”
person. Notification is then sent to the assignee by email or text
pager, depending on the severity of the problem. There is a POTS modem
plugged into the Remedy box that is used to send text pages, and the
Remedy box is allowed to make outgoing SMTP connections to the Internet
to deliver e-mail.

Remedy and OpenView are linked so that closing the ticket removes
the corresponding event in OpenView, and the OpenView event is
annotated with the Remedy ticket number.

Both use an Oracle database to store information about tickets and events.

Each system (Remedy, OpenView, Oracle) runs on a separate piece of Sun hardware.

So we have 3 single-points-of-failure that could cause notification
of critical events to stop: Openview (the software, or the hardware
it’s running on), Remedy (likewise) or Oracle (likewise).

Additionally, failure of a single 100baseT switch or switch port would sever connectivity and take out the notification system.

I’d like to set up something that would detect a failure of one or more of the three critical systems, and notify someone.

Obviously this backup notification system has to be independent of
the three in question. So it should run on some 4th piece of hardware
which is plugged into a different switch. It would then poll or look
for heartbeats from the 3 systems, and use it’s own resources to notify
someone of a problem.

As far as I can see it would be OK if this backup monitoring box
used the same Internet connection, since e-mail is only used to deliver
lower-severity notifications, and a loss of Internet connectivity would
be a Critical-level event, which would use the POTS line to deliver the
notification.

The main monitoring system would then be set up to monitor the backup system, to make sure it’s running.

So… finally, to the question:

What software would you use to set up this monitoring/notification
system? Obviously one could install a completely parallel set of
[OpenView, Oracle, Remedy], but that would be overkill, as we only need
to monitor 3 machines and a few daemon processes.

Is there some nice Open Source project out there that would allow me
to quickly solve this problem? Has anyone done something like this? Any
comments on my failure-mode analysis? Am I worrying about the right
things?

Comments

Three comments:

1. I’d make sure that the secondary monitoring system is monitored
by your fancy-dancy system, in case it fails (of course, if both fail
at once, you’re hosed, unless you want to add a third system, and so
on).

2. I don’t know how complex a system you need to do the secondary
monitoring. If all you want to answer is ‘is the software running?’ you
may be able to get by with a simple perl script running from cron (can
I make a connection to Oracle, etc). A quick look at SF and google
didn’t point out anything obviously of relevant to more complex cases.

3. How does Remedy send the text messages over the modem? You need
to consider that in your solution, for sure. Not sure how to do that in
perl.

Posted by: Dan Moore at June 21, 2004 03:33 PM

We use the open source solution: Nagios (http://www.nagios.org/).

It has a simple plugin infrastructure so you can write a bit of Perl code to monitor anything.

It is simple, clean, and very easy to use.

Dion

Posted by: Dion Almaer at June 23, 2004 10:32 AM

Unix command line utility program conventions

Monday, June 7th, 2004

Sometimes a vendor supplies a command-line utility for performing some function that we want to use from within our scripts and programs.

There are some unwritten (at least as far as I can find) rules about how to write one of these utilities so it can be used properly.

Some vendors get this right. Others, not so much…

The Rules

Return an error status indicating success or failure. For bonus points, return multiple different error codes depending on what went wrong. (The Anna Karenina Principle)

And you know what? That’s about it. The rest (arguments, input/output locations, etc., etc.) really depends on the context and function of the item in quesiton. Though the following are useful:

  • Provide a useful usage message if invalid arguments are passed.
  • Provide an explict way (e.g. –help option) to ask for the above usage message.
  • Use GNU-style arguments.

But these are really for human consumption, not for use in a script.

Recent Failures

I won’t name names, but here are some of the failures I’ve seen recently. (And these are from Big Companies that should know better. Including one company that actually has its own version of Unix… they should really know better.)

  • utility always returns status 1. How the HECK am I supposed to know if it worked? Why are you always returning failure? Didn’t you read a single Unix man page? Didn’t you notice that non-zero exit codes mean failure?
  • -q option suppresses error messages to stdout/stderr… and suppresses the error code return as well. Take a look at diff(1) sometime. The -q option just suppresses the listing of the differences, but still returns the error code.

Strangely enough, these rules apply equally well to Windows command-line utilities. Yes, these do exist.

Another suggestion

If it can be done with a command-line utiltiy, then give us an API we can use.

If you even just create a simple C library, we can then wrap it into our favorite language as a Perl Extension Module, or a Java Native Interface (JNI) pacakge.

If you feel like creating a pure Java implementation of the Library, that would be good too.

From Dan Moore:

On a different tack, but still touching some of the same principles, you may want to check out the Command-Line Options section of the Art of Unix Programming

Originally Posted June 7, 2004 11:20 AM

Google your Email

Monday, May 3rd, 2004

Who needs GMail? You’ve got your own CPU and Disk space, use it.

ZOE lets you read and search your email (with Lucene), without supplying helpful related advertising. Not to mention that it also has a very cool non-linear email access metaphor. Forget Inbox/Sent Mail/…customFolders.. you just browse.


Update:

Dan Moore links to a review of Zoe. I agree with much of what the reviewer says.

I’ve not had the problems with index rebuilds that he mentions.

It is a lot of files, but not much different than an equivalent MH mail store.

On the plus side, I was able to find a piece of mail that I absolutely could not find any other way. It was worth the entire download, run, import process (several hours) just to find that one piece of mail! (Outlook’s search just wouldn’t find it… i’m not sure why.)

Originally Posted May 3, 2004 12:08 PM

Third party software installation woes

Monday, February 2nd, 2004

Let’s say you wanted to run two completely separate instances of Apache’s web server. (I.e. not just multiple virtual hosts listening on different ports, but actual separate httpd processes.) All you’d have to do is create a new httpd.conf (in a separate directory, or with a different name in the same directory as the current one). Then just invoke httpd with the -f option to point at this new config file. The new config file can specify the locations of all the other files (server root, logs, pid files, etc.). Couldn’t be easier.

On the other end of the spectrum is WebLogic 8. I just tried to do the same thing with it (made a copy of the config directories, tweaked them, fired it up), and it failed miserably. Parts of it were still pointing into the old config tree, so things like DB pool deployments were failing.

So, what’s the problem here, and what’s the solution?

In my opinion, there are three types of data associated with an application:

Static
The executable code, and anything else that gets installed off the media (and/or built at install time), and never changes thereafter.

Configuration
Any file that affects how you want the application to run.

Dynamic
Anything that changes during the running of the application, but isn’t part of your own data (e.g. not your relational database). This includes stuff like log files, pid files, lock files, and work directories. Usually you can delete this stuff when the application isn’t running, and it will happily recreate it when it restarts.

In my opinion, you should be able to create a new config file(s), execute the same static code, point it at the new config file(s), and have the newly configured instance create its own set of dynamic files.

This is what I did with apache in my example above. For apache, my classification scheme works out as follows:
Static: the bin and modues/libexec directories
Configuration: the conf directory
Dynamic: the logs directory

Why would I want to do this? Because I want to check the config files into CVS, so I can carefully manage changes to them. I don’t need to check a copy of apache itself into my CVS tree, since I know what version I downloaded and installed. If/when I upgrade to a new version of apache, I can make any required changes to the config files to support and configure the new version, and check those changes into CVS. The Dynamic files I definitely don’t want in CVS, but I might want to archive them (e.g. the logs) for future reference. A clean split between the 3 file types makes this process very simple. I just have to put a copy of the apache conf directory into CVS, and I’m done.

Now let’s look at what happened with Weblogic. I’m still doing the analysis, but here are the preliminary results:

Weblogic wants 3 directories:
BEA_HOME: stuff common to all BEA products
WLS_HOME: where the weblogic software lives
Domain directory: where the files for a “domain” go

In order to run a single weblogic server instance listening on a single port, you actually have to execute 3 processes: an Admin Server, a Node Manager, and a Managed Instance. You have to allocate each of these servers at least one port number to use to listen/talk to the others.

The Admin Server and the Managed Instance share a single config file: config.xml in the Domain directory. The Node Manager has it’s own config file: nodemanager.properties, which lives in the WLS_HOME directory.

By default log files get created in the Domain directory, and there’s an LDAP database that gets stored there too.

There are quite a few files that need to be changed in order to change the directory locations and port numbers assigned to the 3 processes that make up a single “instance” of Weblogic. Somehow in my effort to simply copy the directories and create a new configuration, I missed one spot (grepping can’t find it!) Our problem seems to be with the LDAP directory, and the license file location.

With Weblogic, it seems impossible to separate out the Configuration and Dynamic portions, and hard to separate the Static from the Configuration.

It looks like we’re going to have to re-install from media, and create entirely new BEA_HOME, WLS_HOME and DOMAIN directories. Then I’ll have to run and configure the software, to get it to the state I want. Then I can shut it down and do a “diff” between the various directory trees to see what actually changed in order to change the directories and ports. Then I’ll have to figure out which files are actually Configuration files and which are just Dynamic, how to check in only the Configuration files so we can track config changes in CVS. In the worst case, some of these files (e.g. the LDAP database) will be binary.


So, the guidelines:

Make a clear separation between the three types of files
It should be easy for me to tell which files are executable stuff and which are configuration related. If all the configuration files are stored in a single directory, then it makes it easy to put the configuration under CVS control.

I should be able to lock down the Static and Configuration directories so that they are read-only and the software will run happily (i.e. it shouldn’t be trying to write stuff into the configuration directory… unless I ask it to through some configuration utiltity or GUI)

The location of all Dynamic files should be configurable
I need to be able to control the location of these files (e.g. I might want my log files on a different file system). I need to be able to build a config file that will ensure that none of these Dynamic files will overlap with those from another instance running on the same CPU.

Configuration files should be text files
This way I can use CVS to control them and actually see via cvs diff what the changes were.


Reasons you might want to run multiple separate instances

  • You want to try out some different configuration options without affecting the main server process.
  • You want each developer to have their own instance that they can bounce at will.
  • You want a production and a staging instance on the same box.

Other classes of data

Is there another class of data beyond the 3 I’ve identified above?

Maybe Persistent for things like user and content databases that can change at runtime.

Maybe Custom for things like HTML, JSP, .java, .class, .jar/.war/.ear files and all the other stuff that makes your site different from every other site, but doesn’t change during the execution of a server instance.

These are similar to Dynamic in that you need to be able to control the location (e.g. filesystem path, jdbc URL) that the data will be stored, so it doesn’t (or does!) overlap with another instance, and you can treat it differently than other Configuration and Dynamic data (e.g. you need to back this stuff up, but you don’t usually want to back up the Dynamic data, and the Configuration data is in CVS, so it doesn’t need backing up.)

Configuration utilities

Some programs come with a nice utility (command-line, web-based or GUI) for setting up the configuration. In many cases these utilities just write out text files that you could edit directly. This can be beneficial in several ways.

  • Sometimes it’s easier to understand what the various options are and what the interactions are by running the utility and having it explain them to you, or by having a single option change in the utility change several options “in sync”. You can then do a “diff” of the before and after versions of the text files, and get a better understanding of how the text file works.
  • You can check changes to the config file into CVS, even though you used a GUI to make them.

More examples, both good and bad

Tomcat: Tomcat has a directory structure that looks promising, but it’s not immediately clear to me how I would change the invocation script (catalina.sh) to point to a different configuration directory. In the worst case I could just copy the whole CATALINA_HOME tree, and that would solve the problem, but then I have multiple copies of the Static stuff as welll, which seems like a waste.

MySQL:It seems that you just have to set some options on the mysqld command line, and it will happily use completely different config files and database file locations.

Feel free to let me know about software that you’ve run across that implements a good system for separating the various file types. And let me know if there are flaws in my guidelines.

Originally Posted February 2, 2004 09:37 AM
Comment by John Barnette, February 2, 2004 05:35 PM

Here’s a bit more of a breakdown. It’s flawed; please refine.

+ Configuration
Duh. Can vary by instance.

+ Infrastructure (maps to Static)
The executables and libraries that make up the product. Can support any number of instances.

+ Persistent
An information repository that persists over multiple application runs, but can’t be rebuilt at runtime. Databases, flat files, et cetera. Can vary by instance.

+ Runtime
Logs, temporary files, lock/pid files, and other stuff that can easily be recreated at runtime. Locations of each can vary by instance.

+ Content
All the other stuff that you’ve created that doesn’t have the potential to be changed at runtime. Can vary by instance.