Friday, July 17, 2009
On the value of reflection
One of the most important factors in software development is scope: deciding what to include and what to leave out. Scope creep and feature bloat are recognised risks, impacting development costs and release schedules. Good architects carefully apply rules of thumb: every design decision should 'carry its own weight', and strive to 'kill several birds with one stone'. But an implicit difficulty in evaluating this is knowing what the 'birds' are. Once out of its initial planning phases, software development has a tendency to lurch from immediate issue to immediate issue, dealing with each new requirement as it arises. Considering new requirements in isolation invariably means the burden of large-scale redesign to satisfy any one requirement will seem onerous: a smaller-scale, less impactful alternative will always seem the better option. Reflection, on the other hand, allows the practitioner to consider many weeks worth of problems in a holistic light: he can see all the birds at once, and an approach that once seemed over-engineered now appears justified. Surfacing all the issues at the same time clears a path forward that otherwise would have seemed prohibitive.
This phenomena is analogous to neural networks. While progressing to solve a given problem, a neural network may get trapped, still short of the best solution, in a local minima. The local minima itself does not represent the best answer, but none of the immediate ways out of the minima are enough of an improvement to overcome the walls of the valley. It takes a combined push, a sort of disruptive excitation, to escape the trough so that a better solution can be found.
So much for the theory - is it demonstrable in practice? Here I will give personal testimony. One of the themes from Metawidget's alpha cycle reflections was support for 1-to-M relations. In itself, this seemed a corner case: difficult to support within the current architecture without a slippery slope of requirements around sorting, pagination and summary-to-detail navigation. Another theme was support for third-party UI components. The most challenging case study indicated this would have improved adoption, though it was not a primary factor. A third issue was around supporting the SWT library: the current design of 'return null to render nothing, return a dummy Metawidget to trigger nesting' was backwards for SWT's purposes, though this was being worked around in a sub-optimal way.
Individually, none of these requirements seemed enough to justify a significant reworking of the widget creation subsystem. Indeed, the theme of 1-to-M relations gnawed at me for months with no obvious solution within the existing architecture. It was only reframing it within the context of the additional requirements of 'supporting third-party components' and 'turning widget creation inside out' that a new path presented itself (see WidgetBuilders). Looking back, I realise I was probably especially resistant to seeing this path because it was in an area I had already considered and decided against.
In summary, I have found explicit reflection to be an enlightening and worthwhile use of a project's time. It is easy to skip this phase in the heady rush of pumping out release and release, but when one takes the time to properly pause for breath important insights can be gained.
Friday, July 10, 2009
WikidPad: does just what it says on the tin
I've been looking for a good place to file away all those random bits of knowledge you accumulate during a day, but are not suitable for public consumption (either because they're confidential, or not properly formatted, or whatever) and having a personal, standalone Wiki on my desktop is just perfect!
Thanks guys!
Monday, June 1, 2009
HtmlUnit: listening to their customers
They've added getAnchorByText and getOptionByText. This means I can remove some of the code from my ad hoc HtmlUnitUtils class. Of course, there are still more it'd be nice to see. I include here the whole of my utils class so that they may pick away at it for anything else they may want to incorporate.
Naturally I don't expect it all, or even most. But whatever they may add is awesome as it means less code for me to maintain!
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.URL;
import java.util.Arrays;
import java.util.List;
import org.metawidget.util.CollectionUtils;
import org.metawidget.util.simple.StringUtils;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import com.gargoylesoftware.htmlunit.Page;
import com.gargoylesoftware.htmlunit.WebWindow;
import com.gargoylesoftware.htmlunit.html.HtmlAnchor;
import com.gargoylesoftware.htmlunit.html.HtmlElement;
import com.gargoylesoftware.htmlunit.html.HtmlFileInput;
import com.gargoylesoftware.htmlunit.html.HtmlForm;
import com.gargoylesoftware.htmlunit.html.HtmlOption;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
import com.gargoylesoftware.htmlunit.html.HtmlRadioButtonInput;
import com.gargoylesoftware.htmlunit.html.HtmlSelect;
import com.gargoylesoftware.htmlunit.html.HtmlTable;
import com.kennardconsulting.core.enumeration.FileFormat;
/**
* Utilities for working with HtmlUnit.
*/
public final class HtmlUnitUtils
{
//
// Public statics
//
public static HtmlAnchor getLink( HtmlPage response, String text )
{
List<HtmlAnchor> links = getLinks( response, text );
if ( links.isEmpty() )
throw new RuntimeException( "No such link with exact text of '" + text + "'" );
if ( links.size() > 1 )
throw new RuntimeException( "More than one link with exact text of '" + text + "'" );
return links.get( 0 );
}
public static List<HtmlAnchor> getLinks( HtmlPage response, String text )
{
return getLinks( response, text, false );
}
public static List<HtmlAnchor> getLinks( HtmlPage response, String text, boolean contains )
{
List<HtmlAnchor> anchors = CollectionUtils.newArrayList();
for ( HtmlAnchor anchor : response.getAnchors() )
{
String anchorText = anchor.asText();
anchorText = anchorText.replaceAll( "\r", "" );
if ( contains )
{
if ( !anchorText.contains( text ) )
continue;
}
else
{
if ( !anchorText.equals( text ) )
continue;
}
anchors.add( anchor );
}
return anchors;
}
@SuppressWarnings( "unchecked" )
public static <T extends HtmlElement> T getInputByNameEndingWith( HtmlForm form, String nameEndingWith )
{
for ( HtmlElement element : form.getHtmlElementsByTagName( "input" ) )
{
String elementName = element.getAttribute( "name" );
if ( elementName == null )
continue;
if ( elementName.endsWith( nameEndingWith ) )
return (T) element;
}
return null;
}
public static HtmlOption getSelectedOption( HtmlForm form, String selectName )
{
return getSelectedOption( form.getSelectByName( selectName ) );
}
public static HtmlOption getSelectedOption( HtmlSelect select )
{
List<HtmlOption> selectedOptions = select.getSelectedOptions();
if ( selectedOptions.isEmpty() )
return null;
if ( selectedOptions.size() > 1 )
throw new RuntimeException( "'" + select.getNameAttribute() + "' has more than one option selected" );
return selectedOptions.get( 0 );
}
public static String setSelectedOption( HtmlForm form, String selectName, String option )
{
return setSelectedOption( form.getSelectByName( selectName ), option );
}
public static String setSelectedOption( HtmlForm form, String selectName, String option, boolean allowOnlyOne )
{
return setSelectedOption( form.getSelectByName( selectName ), option, allowOnlyOne );
}
public static String setSelectedOption( HtmlSelect select, String option )
{
return setSelectedOption( select, option, true );
}
public static String setSelectedOption( HtmlSelect select, String option, boolean allowOnlyOne )
{
String selectedValue = null;
for ( HtmlOption htmlOption : select.getOptions() )
{
String htmlOptionText = htmlOption.asText();
// Special support for trimming off , which we use for indenting select options
htmlOptionText = htmlOptionText.trim();
if ( htmlOptionText.equals( option ) )
{
if ( selectedValue != null )
throw new RuntimeException( "Select '" + select.getNameAttribute() + "' has more than one '" + option + "'" );
selectedValue = htmlOption.getValueAttribute();
htmlOption.setSelected( true );
if ( !allowOnlyOne )
break;
}
}
if ( selectedValue == null )
throw new RuntimeException( "Select '" + select.getNameAttribute() + "' does not contain '" + option + "'" );
return selectedValue;
}
public static String setSelectedOptionValue( HtmlForm form, String selectName, String option )
{
return setSelectedOptionValue( form.getSelectByName( selectName ), option );
}
public static String setSelectedOptionValue( HtmlSelect select, String optionValue )
{
String selectedValue = null;
for ( HtmlOption htmlOption : select.getOptions() )
{
if ( htmlOption.getValueAttribute().equals( optionValue ) )
{
if ( selectedValue != null )
throw new RuntimeException( "Select '" + select.getNameAttribute() + "' has more than one '" + optionValue + "'" );
selectedValue = htmlOption.getValueAttribute();
htmlOption.setSelected( true );
}
}
if ( selectedValue == null )
throw new RuntimeException( "Select '" + select.getNameAttribute() + "' does not contain value '" + optionValue + "'" );
return selectedValue;
}
public static boolean hasOption( HtmlForm form, String selectName, boolean selected, String... options )
{
int found = 0;
for ( HtmlOption htmlOption : form.getSelectByName( selectName ).getOptions() )
{
for ( String option : options )
{
if ( htmlOption.asText().equals( option ) )
{
if ( selected && !htmlOption.isSelected() )
return false;
found++;
break;
}
}
}
return ( found == options.length );
}
public static boolean hasOptionValue( HtmlForm form, String selectName, boolean selected, String... options )
{
int found = 0;
for ( HtmlOption htmlOption : form.getSelectByName( selectName ).getOptions() )
{
for ( String option : options )
{
if ( htmlOption.getValueAttribute().equals( option ) )
{
if ( selected && !htmlOption.isSelected() )
return false;
found++;
break;
}
}
}
return ( found == options.length );
}
public static String getOptionValue( HtmlForm form, String selectName, String option )
{
for ( HtmlOption htmlOption : form.getSelectByName( selectName ).getOptions() )
{
if ( htmlOption.asText().equals( option ) )
return htmlOption.getValueAttribute();
}
throw new RuntimeException( "No option with text '" + option + "' found" );
}
public static void setSelectedRadio( HtmlForm form, String radioName, String value )
{
boolean selectedOne = false;
for ( HtmlRadioButtonInput htmlRadioButtonInput : form.getRadioButtonsByName( radioName ) )
{
if ( !htmlRadioButtonInput.getValueAttribute().trim().equals( value ) )
continue;
if ( selectedOne )
throw new RuntimeException( "Radio button group '" + radioName + "' has more than one '" + value + "'" );
selectedOne = true;
htmlRadioButtonInput.setChecked( true );
}
if ( !selectedOne )
throw new RuntimeException( "Radio button group '" + radioName + "' has no option '" + value + "'" );
}
public static boolean hasRadio( HtmlForm form, String radioName, String value )
{
for ( HtmlRadioButtonInput htmlRadioButtonInput : form.getRadioButtonsByName( radioName ) )
{
if ( htmlRadioButtonInput.getValueAttribute().trim().equals( value ) )
return true;
}
return false;
}
public static String getSelectedRadioValue( HtmlForm form, String radioName )
{
HtmlRadioButtonInput radioButtonInputSelected = null;
for ( HtmlRadioButtonInput htmlRadioButtonInput : form.getRadioButtonsByName( radioName ) )
{
if ( !htmlRadioButtonInput.isChecked() )
continue;
if ( radioButtonInputSelected != null )
throw new RuntimeException( "Radio button group '" + radioName + "' has more than one selected" );
radioButtonInputSelected = htmlRadioButtonInput;
}
if ( radioButtonInputSelected == null )
return null;
return radioButtonInputSelected.getValueAttribute();
}
@SuppressWarnings( "unchecked" )
public static <E extends HtmlElement> E getElementByAttribute( HtmlPage page, String elementName, String attributeName, String attributeValue )
{
return (E) getElementByAttribute( page.getDocumentElement(), elementName, attributeName, attributeValue );
}
public static <E extends HtmlElement> E getElementByAttribute( HtmlElement element, String elementName, String attributeName, String attributeValue )
{
List<E> elements = element.getElementsByAttribute( elementName, attributeName, attributeValue );
if ( elements.isEmpty() )
return null;
if ( elements.size() > 1 )
throw new RuntimeException( "More than one " + elementName + " with " + attributeName + " of '" + attributeValue + "'" );
return elements.get( 0 );
}
public static <E extends HtmlElement> E getElementByAttributeContaining( HtmlPage page, String elementName, String attributeName, String attributeValueContained )
{
List<E> elements = getElementsByAttributeContaining( page, elementName, attributeName, attributeValueContained );
if ( elements.isEmpty() )
return null;
if ( elements.size() > 1 )
throw new RuntimeException( "More than one " + elementName + " with " + attributeName + " containing '" + attributeValueContained + "': " + CollectionUtils.toString( elements ) );
return elements.get( 0 );
}
/**
* @return the elements, in the order they are declared in the HTML.
*/
@SuppressWarnings( "unchecked" )
public static <E extends HtmlElement> List<E> getElementsByAttributeContaining( HtmlPage page, String elementName, String attributeName, String attributeValueContained )
{
List<E> toReturn = CollectionUtils.newArrayList();
NodeList nodeList = page.getElementsByTagName( elementName );
for ( int loop = 0, length = nodeList.getLength(); loop < length; loop++ )
{
Node node = nodeList.item( loop );
Node nodeValue = node.getAttributes().getNamedItem( attributeName );
if ( nodeValue == null )
continue;
if ( nodeValue.getNodeValue().contains( attributeValueContained ) )
toReturn.add( (E) node );
}
return toReturn;
}
@SuppressWarnings("unchecked")
public static <T extends Page> T waitForAjax( T page )
{
WebWindow window = page.getEnclosingWindow();
window.getThreadManager().joinAll( 10000 );
return (T) window.getEnclosedPage();
}
public static void setUpload( HtmlForm form, String uploadName, String url )
{
setUpload( form, uploadName, CoreStringUtils.substringAfterLast( url, StringUtils.SEPARATOR_FORWARD_SLASH ), url );
}
public static void setUpload( HtmlForm form, String uploadName, String name, String url )
{
try
{
setUpload( form, uploadName, name, new URL( url ).openStream() );
}
catch ( IOException e )
{
throw new RuntimeException( e );
}
}
public static void setUpload( HtmlForm form, String uploadName, String name, InputStream streamIn )
{
HtmlFileInput fileInput = form.getInputByName( uploadName );
ByteArrayOutputStream streamOut = new ByteArrayOutputStream();
try
{
IOUtils.streamBetween( streamIn, streamOut );
}
catch ( IOException e )
{
throw new RuntimeException( e );
}
fileInput.setValueAttribute( name );
fileInput.setData( streamOut.toByteArray() );
}
public static HtmlTable getTableStartingWith( HtmlPage page, String startingWith )
{
NodeList tables = page.getElementsByTagName( "table" );
for( int loop = 0, length = tables.getLength(); loop < length; loop++ )
{
HtmlTable table = (HtmlTable) tables.item( loop );
if ( table.asText().trim().startsWith( startingWith ))
return table;
}
throw new RuntimeException( "No table starting with '" + startingWith + "'" );
}
//
// Private constructor
//
private HtmlUnitUtils()
{
// Can never be called
}
}
Tuesday, May 26, 2009
Metawidget and Seam: saying goodbye to boilerplate code
If you think of a typical Seam stack, you may have JPA at the bottom, then EJB, then Seam itself, then JSF, then maybe RichFaces to add a bit of polish. But there is still a gap at the very top - a gap that leads to a lot of 'boilerplate code'. This is where Metawidget comes in.
Let me try and convey it visually using the image below. On the left are the complete contents of the book.xhtml file from the Seam Groovy Booking example. On the right are the complete contents from the new Seam Metawidget Groovy Booking example. The red boxes and lines highlight the chunks of boilerplate that have been replaced:
The original file is 177 lines long. The Metawidget equivalent is 52 lines. That represents a 70% reduction in code, on top of the considerable reductions Seam already affords when developing enterprise applications. This ability to retrofit existing UIs, integrating with existing front-end and back-end technologies, is unique to Metawidget.Now I want to be completely fair here - I'm a developer, not a marketing guy! You also have to add a few extra annotations to your business classes, add the Metawidget JAR to your project and create a little metawidget.xml file. If we compare all the source files that change:

By comparing file sizes, we see the overall code reduction is around 20%. We could allow that in such a small example the size of metawidget.xml is more significant than it should be: for apps with hundreds of screens, the impact of 2,789 bytes of metawidget.xml will be negligible. So if we compare file sizes without including metawidget.xml, we see the overall code reduction is around 40%. If you repeat this exercise with the Seam Metawidget DVD Store example (also included in the Seam distribution) the overall reduction is around 30%.
Of course, on top of all that, lines of code is never a wondeful metric for comparing implementations. But anyway, hopefully you get the idea. Still, if you want a soundbite: Metawidget can save you up to 40% of your UI code.
My deepest thanks to the Seam guys, especially Dan Allen, for all their help integrating Metawidget into the Seam 2.1.2.GA build.
Monday, May 11, 2009
Dynamic User Interface Generator: Metawidget v0.75
- Pluggable widget libraries
- SwingX support
- DisplayTag support
- Improved documentation
'Pluggable widget libraries' represents a significant refactoring of the widget generation code, intended to:
- simplify support of third party libraries, including mixing multiple third party libraries in the same application
- pave the way for supporting Collections
- pave the way for supporting some more UI toolkits (ie. SWT)
It is also, unfortunately, a breaking change. Sorry!
Migration Guide
To migrate from v0.7 to v0.75:
Change #1: inspector-config.xml is now metawidget.xml
The role of inspector-config.xml has been expanded from configuring pluggable inspectors to configuring pluggable inspectors and widget builders. It is also now a general configuration mechanism for all aspects of your Metawidget, such as default CSS settings etc.
You will need to refactor inspector-config.xml files of the form...
<compositeInspector xmlns="org.metawidget.inspector.composite">
<myinspector>
<...
</myinspector>
</inspector-config>
<swingMetawidget xmlns="org.metawidget.swing">
<inspector>
<compositeInspector xmlns="org.metawidget.inspector.composite">
<list>
<myinspector>
...
</myinspector>
</list>
</compositeInspector>
</inspector>
</swingMetawidget>
</metawidget>
- it is now concerned with the top-level Metawidget, not just the inspectors inside it. This means you can also configure other Metawidget properties (see below)
- method values must now be wrapped with their type (ie. <list>) - this allows us to support configuring multi-value methods such as setParameter
- Full documentation can be found here
Change #2: Metawidget.buildWidget is now WidgetBuilder.buildWidget
If you had previously extended Metawidget to add support for a third party widget, you'll need to refactor your code into a WidgetBuilder. WidgetBuilders can be configured programmatically, or with the new metawidget.xml:
<swingMetawidget xmlns="org.metawidget.swing">
<widgetBuilder>
<compositeWidgetBuilder xmlns="org.metawidget.widgetbuilder.composite">
<list>
<myWidgetBuilder />
<swingWidgetBuilder />
</list>
</compositeWidgetBuilder>
</widgetBuilder>
<inspector>
...
</inspector>
</swingMetawidget>
</metawidget>
Change #3: RichFacesMetawidget has been removed
To use JBoss RichFaces, you now use a regular UIMetawidget with a RichFacesWidgetBuilder. Full documentation can be found here.
Thanks!
We apologise for the disruption these changes will cause, but strongly believe they will make Metawidget a better product for our v1.0 release. All documentation and examples have already been migrated.
Tuesday, March 3, 2009
HtmlUnit vs HttpUnit
I've been using HttpUnit for my black-box testing for about 3 years, and I really like it. However its JavaScript support just hasn't kept up with our needs, and it seems HtmlUnit has a much more active community around it.
I converted about 6,000 lines of test scripts in about 3 days. I thought the HtmlUnit folks (if not the HttpUnit folks) might be interested in what I experienced.
First, the FUD
HttpUnit is a great project, and very similar to HtmlUnit.
Blog entries like this (from an HtmlUnit guy) paint an inaccurate picture saying that HttpUnit is 'fairly low-level, modeling web interactions at something approaching the HTTP request and response level' whilst HtmlUnit is 'more high-level than HttpUnit’s, modeling web interaction in terms of the documents and interface elements which the user interacts with'. It then gives an HttpUnit example using requests and responses, and an HtmlUnit example using forms and input controls.
But the examples are not comparing apples to apples. HttpUnit does forms (and input controls, and tables, and JavaScript) too - and in almost exactly the same way as HtmlUnit. I'm not saying this is deliberate deception, but I think such a comparison is unfair and may even be detrimental to HtmlUnit because developers may be more reluctant to 'make the switch' if they perceive the APIs are very different.
In fact, the API methods are almost 1-to-1 identical. In coverting about 6,000 lines of code here's what I found:
- With HttpUnit: 6,198 lines
- With HtmlUnit: 6,285 lines
Next, the Good
The HtmlUnit API definitely feels nicer.
- There's some neat 'public <I> I getInputByName' code that saves a lot of casting - I've stolen this idea for the next release of Metawidget
- I love the asText and asXml methods which do a lot of parsing for you
- HttpUnit used to silently submit forms using 'null' if the button you asked it to submit didn't exist (eg. form.submit( form.getSubmitButton( 'not-there' ))). HtmlUnit doesn't do this
- You can set file upload boxes just like regular text boxes. HttpUnit required you to wade through some form.getRequest goo
- I love that things like HtmlTextInput are implemented as direct extensions of the internal DOM, rather than some parallel heirarchy
There are some things from the HttpUnit API I missed:
- Locating anchors in HtmlUnit is very fiddly. You have page.getAnchorByName and page.getAnchorByHref. But this is black box testing - it's meant to test the 'user experience'. And the user never gets to see either an anchor's name or its href. HttpUnit had a response.getLinkWith method that located an anchor by its 'innerHTML' or 'what the user actually sees'. This would be very helpful?
- Choosing options from select boxes in HtmlUnit is similarly fiddly. You have select.setSelectedAttribute(optionValue) but again this is keying off something the user never sees. I'd really like a select.setSelectedAttribute(innerHTML) so I can simulate choosing what the user chooses?
- HtmlUnit warns that use of the script type 'text/javascript' is obsolete, and maybe it is (as of about 2007). But that's a pretty recent change. If you try <script/> in the W3C validator it still suggests using 'text/javascript', and older browsers will want to see it. So this seems a very noisy warning to have on by default?
- There doesn't seem a good equivalent to HttpUnit's form.getParameterNames?
Many thanks to both the HttpUnit and the HtmlUnit teams for all their hard work and contributions to the community!
Thursday, February 12, 2009
Declarative UI: Metawidget v0.7
- Pluggable action bindings for GWT and Swing
- Pluggable Swing validation (including JGoodies Validator)
- MigLayout support
- Scala support
- OSGi support
- Upgraded support for Android (1.0 R2) and Seam (2.1.1.GA)
- Fluent API for configuring Inspectors programmatically
As always, the best place to start is the Reference Documentation:
Your continued feedback is invaluable to us. Please download it and let us know what you think.