Skip to main content

Functor-based filtering in the incubator

10 replies [Last post]
dhall
Offline
Joined: 2006-02-17

I've put an implementation of table filtering based on jga functors into the incubator project. These are based on a pre-release of jga-0.7. It is compatable with Java 1.4, and takes advantage of a parser that I've been writing that compiles a java-esque expression language into compound functors.

There are two demos, both based on the TableFormDemo. The first, TableFormDemo1 replaces the existing JXSearchPanel with a FunctorSearchPanel, which consists of a text field for expression entry and a label where errors are reported. The language is very java-like: you can do basic arithmetic on all built-in Numbers, you can compare values and do logical expressions, you can call methods and constructors (with the short-term restriction that they may only
take 0 or 1 argument).

(Another place where I'm using this technology is at a site that I've not linked to from anywhere until now: a demonstration of a table where the cells take a single line of the sorta-java expression language to describe their values. You can try this out at http://jga.sf.net/docs/HacksheetDemo.shtml. That page has more details on the restrictions and extensions of the current expression language: it is very much a work in progress, and I plan to remove many of the restrictions before formally releasing jga-0.7.)

A language extension that is used by the FunctorFilter is the ability to reference arbitrary columns in the data model: a 'col' keyword is supported by this panel: it takes an integer argument, and it is used to get access to the data in a particular column. Here are a few examples:

- Find all of the JTable bugs

col(4).indexOf("JTable") >= 0
col(4).matches(".*JTable.*")

- Find all of the severe JTable bugs

col(4).indexOf("JTable") >= 0 & col(3) == 1

- Find all of the unassigned bugs

col(5) == "null"
col(5).equals("null")

- Find all L&F bugs not assigned to leif

!col(5).equals("leif") && col(4).indexOf("L&F") >= 0

- Find all of the bugs where severity exceeds priority

col(2) > col(3)

- Find all of the severe or highest priority bugs

col(2) <= 2 || col(3) == 1

The second demo replaces the implementation of JXSearchPanel. Rather than use an instance of PatternFilter (which is tightly constrained to regular expression matches on a single column), it uses a FunctorFilter. From the user's standpoint, it should be functionally identical to the standard TableFormDemo

Dave Hall
http://jga.sf.net/

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
kleopatra
Offline
Joined: 2003-06-11

Hi Dave,

good work and very interesting - so I have a couple of comments :-)

- regarding (model) columns that are not part of view: I don't see any way to get at them - ComponentAdapter is accessed in view coordinates [*]. A Filter internally stores the model column index. A modelToView has no chance but returning a -1 for non-contained columns. Even if it were possible I would second Ramesh's usability arguments, though.

[*] is it? In analogy to JTable api I would assume it, though the api doc explicitly states it for getColumnName() only, says nothing for get/setValueAt, isCellEditableAt(..) The TableAdapter implementation indeed does interpret the col index as view index - so maybe the api doc should be updated :). Looking into its code there appears to be a slight (unrelated to this thread) issue: adapter.getColumnCount() should return the view column count, or not?

- as to using names/labels/identifiers ... that's a can of worms (there was a thread about "columns & metaData nomenclature" this summer, but unfortunately I can't find it). Basically, there's no way to reliably find a column by "name" because in the context of a JTable/tableModel the "name" is not unique (as opposed to the "name" of metaData or "identifier" of columnModel).

Hmm... what's the "columnName" property of a ComponentAdapter? It's implemented to return the columns's headerValue (== metaData.label) - but need that be so? Could we change the semantics to mean the "identifier"? Or alternatively add a method getColumnIdentifier? Every view with an underlying MetaDataProvider as model can guarantee to service such a method...

Jeanette

dhall
Offline
Joined: 2006-02-17

Jeanette

>> good work and very interesting - so I have a couple of comments

>> - regarding (model) columns that are not part of view: I don't see any way to get at them - ComponentAdapter is accessed in view coordinates [*]. A Filter internally stores the model column index. A modelToView has no chance but returning a -1 for non-contained columns. Even if it were possible I would second Ramesh's usability arguments, though.

>> [*] is it? In analogy to JTable api I would assume it, though the api doc explicitly states it for getColumnName() only, says nothing for get/setValueAt, isCellEditableAt(..) The TableAdapter implementation indeed does interpret the col index as view index - so maybe the api doc should be updated . Looking into its code there appears to be a slight (unrelated to this thread) issue: adapter.getColumnCount() should return the view column count, or not?

I understand the usability concern, and from the standpoint of implementing an extension to the swing JTable, I understand the dependency on TableModel. As far as the usability concern goes, I believe that it is possible to use a table that is filtered on columns not (visible) in the table poorly. It is also possible to do it well, and it is up to the application designer to put such a table in a context where the filtering makes sense. In this case, I would suggest that we implement the sharper tool, and help those that cut themselves with it learn how to use it safely.

As far as the coordinate system goes, when I parse the expression from within the filter, I had to translate the indices by calling viewToModel in order for the indices to line up with the table column order. Without that translation, then the column indices were as suggested by the form layout (id was 0, engineer was 3 instead of 5, description was 6 instead of 4). This suggests that the filter has access to the model's order, but only for those fields that appear in the table itself.

One way that we may get the best of both is to implement column visibility: don't remove columns in the filtering or in the model; instead, let the table column model control which columns are visible or not. This way, the table could contain one set of data, the form could contain another, and filtering could work on any.

>> as to using names/labels/identifiers ... that's a can of worms (there was a thread about "columns & metaData nomenclature" this summer, but unfortunately I can't find it). Basically, there's no way to reliably find a column by "name" because in the context of a JTable/tableModel the "name" is not unique (as opposed to the "name" of metaData or "identifier" of columnModel).

>> Hmm... what's the "columnName" property of a ComponentAdapter? It's implemented to return the columns's headerValue (== metaData.label) - but need that be so? Could we change the semantics to mean the "identifier"? Or alternatively add a method getColumnIdentifier? Every view with an underlying MetaDataProvider as model can guarantee to service such a method...

I wouldn't really worry that column names are not guaranteed to be unique: in practice they tend to be, and I think a reasonable compromise in the face of non-unique column names would be to take the first one you find with the name you're trying to resolve. That implies that we'd be looking them up sequentially -- since tables will rarely encompass more than a couple of dozen columns, sequentially searching the column list once in a while will not be a great cost.

Dave

kleopatra
Offline
Joined: 2003-06-11

Dave,

>
> I understand the usability concern, and from the
> standpoint of implementing an extension to the swing
> JTable, I understand the dependency on TableModel.
> As far as the usability concern goes, I believe that
> t it is possible to use a table that is filtered on
> columns not (visible) in the table poorly. It is
> also possible to do it well, and it is up to the
> application designer to put such a table in a context
> where the filtering makes sense. In this case, I
> would suggest that we implement the sharper tool, and
> help those that cut themselves with it learn how to
> use it safely.
>

I see where you are heading. And am thinking about it. Maybe the interesting point is to define the "visible" requirement: JNTable has the notion of "hiding" columns (via the ColumnControlButton) - can we define "visible for filters" as "visible on screen" plus "hidden"? That's similar in direction as your suggestion:

> One way that we may get the best of both is to
> implement column visibility: don't remove columns in
> the filtering or in the model; instead, let the table
> column model control which columns are visible or
> not. This way, the table could contain one set of
> data, the form could contain another, and filtering
> could work on any.

... but differs in that there might be columns in the TableModel that are neither on screen nor in the hidden list.

The concept of the hidden columns should be moved to JXTable anyway, getting it correctly will require an enhanced TableColumnModel, then adjusting the TableAdapter to include the hidden would be quite straightforward.

yeah, "quite" is really relative :-) Coordinate transformations could still be confusing to users, but that's probably a detail which should not be too difficult to manage, especially if we head for allowing "names" as column parameter in the parse expression.

> As far as the coordinate system goes, when I parse
> the expression from within the filter, I had to
> translate the indices by calling viewToModel in order
> for the indices to line up with the table column
> order. Without that translation, then the column
> indices were as suggested by the form layout (id was
> 0, engineer was 3 instead of 5, description was 6
> instead of 4). This suggests that the filter has
> access to the model's order, but only for those
> fields that appear in the table itself.
>

Just looked at the code (it's only at the periphery of my construction site :-) - Filter expects model columnIndex in all public methods, internally transforms it into view columnIndex by componentAdapter.modelToView() and then calls componentAdapter.xxAt(row, column) with this view columnIndex. That's in agreement with what you are seeing, right?

As to column names vs identifiers: probably expressed myself poorly - I'm not worried (and never about assumed performance), just wanted to bring up the point again (there are some regions of the code that are not yet updated after the introduction of MetaDataProvider )

Thanks for your input, looking forward to more :-)

Jeanette

dhall
Offline
Joined: 2006-02-17

>> I see where you are heading. And am thinking about it. Maybe the interesting point is to define the "visible" requirement: JNTable has the notion of "hiding" columns (via the ColumnControlButton) - can we define "visible for filters" as "visible on screen" plus "hidden"?

I think that's a good solution. It puts the onus on the application designer to expose for query/filter purpose what will make sense in the context of the overall application. He can still choose to leave things unavailable: I can see that being useful for db-assigned record ID's, audit trail information, sensitive information, etc.

>> As to column names vs identifiers: probably expressed myself poorly - I'm not worried (and never about assumed performance), just wanted to bring up the point again (there are some regions of the code that are not yet updated after the introduction of MetaDataProvider

That part wasn't directed at you: I was speaking to/for eventual lurkers. A sequential search "sounds bad" to those who optimize for speed prematurely -- those who would hold that it's 'always' better to do a hash lookup on a key rather than a sequential lookup in a list. I think in this case, the compromise semantics pretty much require a sequential search, but the cases where the sequential search will be a significant penalty are fairly degenerate.

>> Just looked at the code (it's only at the periphery of my construction site - Filter expects model columnIndex in all public methods, internally transforms it into view columnIndex by componentAdapter.modelToView() and then calls componentAdapter.xxAt(row, column) with this view columnIndex. That's in agreement with what you are seeing, right?

I think so. The functor-based filter doesn't use column binding the way that the standard filter does, but it does (I believe) implement the contract correctly.

On a related issue, when I was writing this class, I ended up cloning a bunch of code from the standard filter in order to fully implement the abstract methods in the base class. We should look at refactoring the relationship between the abstract filter and the concrete implementations: if there's really only one way to honor the contract, then leaving it abstract isn't really buying us anything. I can see doing it now, or I can see waiting until the third implementation (you don't know that you have a resuable object until you've used it three times, right?). I can move the code around in the incubator tree, but y'all with commit privileges will need to decide to accept it (or not).

Dave Hall
http://jga.sf.net/
http://jroller.com/page/dhall/

kleopatra
Offline
Joined: 2003-06-11

> >> I see where you are heading. And am thinking about
> it. Maybe the interesting point is to define the
> "visible" requirement: JNTable has the notion of
> "hiding" columns (via the ColumnControlButton) - can
> we define "visible for filters" as "visible on
> screen" plus "hidden"?
>
> I think that's a good solution. It puts the onus on
> the application designer to expose for query/filter
> purpose what will make sense in the context of the
> overall application. He can still choose to leave
> things unavailable: I can see that being useful for
> db-assigned record ID's, audit trail information,
> sensitive information, etc.
>

so we agree :-) Would you please file an enhancement issue to make sure we don't forget?

> On a related issue, when I was writing this class, I
> ended up cloning a bunch of code from the standard
> filter in order to fully implement the abstract
> methods in the base class. We should look at
> refactoring the relationship between the abstract
> filter and the concrete implementations: if there's
> really only one way to honor the contract, then
> leaving it abstract isn't really buying us anything.
> I can see doing it now, or I can see waiting until
> l the third implementation (you don't know that you
> have a resuable object until you've used it three
> times, right?). I can move the code around in the
> incubator tree, but y'all with commit privileges will
> need to decide to accept it (or not).

sounds reasonable - I would second to accept your change any time you are confident that it's the right thing to do. Again, could you please file an issue, maybe with the refactored Filter attached? I'm unsure who's the code owner of the decorators ... the heritages are not yet quite sorted out.

Thanks for your input.

Jeanette

rameshgupta
Offline
Joined: 2004-06-04

> I'm unsure who's the code owner of the decorators ...
> Jeanette

Mea culpa :-) Sorry, I've been hiding in a cave fixing and improving openmarkup, winding my way through J2SE 5.0 changes related to XML, and generifying my code. That has left me with little time for anything else.

I actually had refactored filters to use a common base class to reduce (perhaps eliminate) the need for subclassing, but that code never got checked in. And now I don't have that code anymore :-( I will try to rewrite that code and check it in next week.

Thanks for your patience!

Ramesh

Anonymous

Dave,

This is excellent -- functors could really serve as a powerful mechanism for configuring filtering &
highlighting by eliminating the need to author custom filters/highlighters for common expressions.

Aim

dhall
Offline
Joined: 2006-02-17

Amy:

A couple of things to point out:

1) In this implementation, I'm not depending on the binding between the filter and a particular column index. I do store the information and the base class reports it, but I don't use the binding for anything. Any other classes that make assumptions about the meaning of the filter binding will be mistaken with respect to this filter -- obviously some of the examples I showed above filter based on multiple columns.

2) I didn't find a way to get arbitrary column names. I think it'd be more usable if I could use the name of the column instead of the index in expressions.

On a related note, if we impose the restriction that the column name cannot include whitespace, then we can adjust the expression language to support the usage of column names without the 'col' keyword. We'd be able to say (for example)

Engineer == "null"

instead of

col(5) == "null"

or

col("Engineer") == "null"

This would have a usability impact in other areas, so I'm not sure if its a good idea on balance, but I thought I'd bring it up just in case.

3) I think it'd be useful to filter on columns that are included in the original data, but not necessarily included in the table. That didn't work at all as I was testing this. All of the non-visible columns report -1 as they are mapped from the layer where they exist to the layer in which they are hidden (I'd suggest that perhaps they could have their indices negated then decremented so that they're all < 0 but not all -1, but that seems like a hack, and it's likely to break any code that tests for -1 specifically)

Dave Hall
http://jga.sf.net/
http://jroller.com/page/dhall

rameshgupta
Offline
Joined: 2004-06-04

> On a related note, if we impose the restriction that
> the column name cannot include whitespace, then we
> can adjust the expression language to support the
> usage of column names without the 'col' keyword.
> We'd be able to say (for example)
>
> Engineer == "null"
>
> instead of
>
> col(5) == "null"
>
> or
>
> col("Engineer") == "null"
>
> This would have a usability impact in other areas, so
> I'm not sure if its a good idea on balance, but I
> thought I'd bring it up just in case.

This sounds reasonable for internal column names in metadata, which are the ones that should matter in this case anyway. Obviously, for user-visible/localizable column names, this is not a good idea.

>
> 3) I think it'd be useful to filter on columns that
> are included in the original data, but not
> necessarily included in the table.

That, I think, will have real usability issues. Finding a match, where seemingly there isn't any, is bound to confuse a lot of users. JDNC has such a case where applying pattern-matching on enumerated column values surprises and confuses users because what they see on the screen does not match what the pattern matcher sees.

Ramesh Gupta

dhall
Offline
Joined: 2006-02-17

>> This sounds reasonable for internal column names in metadata, which are the ones that should matter in this case anyway. Obviously, for user-visible/localizable column names, this is not a good idea.

Agreed: if it gets to the point where the column name can be included in quotes as an argument, that'll probably be about the right balance point. I think having to require column indices would limit the utility of this.

>> That, I think, will have real usability issues. Finding a match, where seemingly there isn't any, is bound to confuse a lot of users. JDNC has such a case where applying pattern-matching on enumerated column values surprises and confuses users because what they see on the screen does not match what the pattern matcher sees.

This is going to depend on the application and its audience. As use of this scales up, you can expect entities to be used that may have dozens of properties, and I believe it might be better to trust the application developers to use this properly than to simply rule against them at this point. A lot of applications (probably the vast majority that I've been involved with) require the users to possess a fair amount domain specific knowlege: they can be expected to know more about the contents of the system than is presented in any given view.

It's not unusual for table queries to not include the filter columns as a space-saving mechanism. When filtering on equality, the value for a filter column is the same for all rows, and thus adds no additional information to the view. If there's a lot of information to be presented, then redundant columns will be the first to be eliminated.

Even in the existing table demo, there's no way to filter the list based on age of the bug. The date field that exists in the form is not in the table, therefore it cannot be used by the filter.

Dave Hall
http://jga.sf.net
http://jroller.com/page/dhall