Skip to main content

[patch][long] computed DataValues

16 replies [Last post]
dhall
Offline
Joined: 2006-02-17

Rich:

It's been a few weeks, and we've both been distracted by other priorities. Here's a patch that will support computed data values, and polishes up the support for computed columns a little. This requires the jar file that is in the incubator (doing this shook out a couple of minor bugs in the 0.7 release, and required a couple of different hooks than I had anticipated, so use 0.7.1).

There are tests that show the limits of the syntax: basically, you can refer to columns as TABLE.column, or if the table is already known (always true when parsing a computed column, true after the first table reference in a computed value) the table is optional.

Sorry about the width -- this is probably going to be too wide to conveniently discuss in this thread. If you have any questions, it'd probably be better to start a different thread.

<br />
Index: swingx/src/java/org/jdesktop/dataset/DataColumn.java<br />
===================================================================<br />
RCS file: /cvs/jdnc/swingx/src/java/org/jdesktop/dataset/DataColumn.java,v<br />
retrieving revision 1.1<br />
diff -u -r1.1 DataColumn.java<br />
--- swingx/src/java/org/jdesktop/dataset/DataColumn.java	23 Feb 2005 17:51:31 -0000	1.1<br />
+++ swingx/src/java/org/jdesktop/dataset/DataColumn.java	22 Apr 2005 03:10:57 -0000<br />
@@ -8,6 +8,8 @@<br />
 import java.beans.PropertyChangeListener;<br />
 import java.beans.PropertyChangeSupport;<br />
 import java.util.logging.Logger;<br />
+import net.sf.jga.fn.UnaryFunctor;<br />
+import net.sf.jga.parser.ParseException;<br />
 import org.jdesktop.dataset.NameGenerator;</p>
<p> /**<br />
@@ -74,6 +76,12 @@<br />
      * be the same, as determined by .equals()).<br />
      */<br />
     private boolean keyColumn;<br />
+<br />
+    /**<br />
+     * Expression to be evaluated for computed columns<br />
+     */<br />
+    private String expression;<br />
+    private UnaryFunctor expImpl;</p>
<p>     /**<br />
      * Create a new DataColumn. To construct a DataColumn, do not call<br />
@@ -220,6 +228,37 @@<br />
             pcs.firePropertyChange("keyColumn", oldVal, value);<br />
         }<br />
     }<br />
+<br />
+<br />
+    public String getExpression() {<br />
+        return expression;<br />
+    }<br />
+<br />
+    /**<br />
+     * @throws ParseException<br />
+     */<br />
+    public void setExpression(String expression) throws ParseException {<br />
+        if (expression == null)<br />
+            expression = "";<br />
+<br />
+        UnaryFunctor newExpImpl =<br />
+            getParser().parseComputedColumn(getTable(), expression);<br />
+<br />
+        if ( !(expression.equals(this.expression))) {<br />
+            UnaryFunctor oldExpImpl = expImpl;<br />
+            String oldExpression = this.expression;<br />
+            this.expression = expression;<br />
+            this.expImpl = newExpImpl;<br />
+            pcs.firePropertyChange("expression", oldExpression, expression);<br />
+        }<br />
+    }<br />
+<br />
+    public Object getValueForRow(DataRow row) {<br />
+        return expImpl.fn(row);<br />
+    }<br />
+<br />
+<br />
+    Parser getParser() { return getTable().getDataSet().getParser(); }</p>
<p>     public void addPropertyChangeListener(PropertyChangeListener listener) {<br />
         pcs.addPropertyChangeListener(listener);<br />
@@ -236,4 +275,4 @@<br />
     public void removePropertyChangeListener(String propertyName, PropertyChangeListener listener) {<br />
         pcs.removePropertyChangeListener(propertyName,  listener);<br />
     }<br />
-}<br />
\ No newline at end of file<br />
+}<br />
Index: swingx/src/java/org/jdesktop/dataset/DataRow.java<br />
===================================================================<br />
RCS file: /cvs/jdnc/swingx/src/java/org/jdesktop/dataset/DataRow.java,v<br />
retrieving revision 1.2<br />
diff -u -r1.2 DataRow.java<br />
--- swingx/src/java/org/jdesktop/dataset/DataRow.java	10 Mar 2005 19:27:37 -0000	1.2<br />
+++ swingx/src/java/org/jdesktop/dataset/DataRow.java	22 Apr 2005 03:11:03 -0000<br />
@@ -85,7 +85,11 @@<br />
     }</p>
<p>     public Object getValue(DataColumn col) {<br />
-        return cells.get(col).value;<br />
+        String exp = col.getExpression();<br />
+        if (exp == null || exp.equals(""))<br />
+            return cells.get(col).value;<br />
+        else<br />
+            return col.getValueForRow(this);<br />
     }</p>
<p>     public DataTable getTable() {<br />
@@ -177,4 +181,4 @@<br />
             }<br />
         }<br />
     }<br />
-}<br />
\ No newline at end of file<br />
+}<br />
Index: swingx/src/java/org/jdesktop/dataset/DataSet.java<br />
===================================================================<br />
RCS file: /cvs/jdnc/swingx/src/java/org/jdesktop/dataset/DataSet.java,v<br />
retrieving revision 1.3<br />
diff -u -r1.3 DataSet.java<br />
--- swingx/src/java/org/jdesktop/dataset/DataSet.java	25 Feb 2005 19:46:13 -0000	1.3<br />
+++ swingx/src/java/org/jdesktop/dataset/DataSet.java	22 Apr 2005 03:11:06 -0000<br />
@@ -59,8 +59,11 @@<br />
     }</p>
<p>     private NameChangeListener nameChangeListener = new NameChangeListener();<br />
+    private Parser parser = new Parser();    </p>
<p>     public DataSet() {<br />
+        parser.bindThis(this);<br />
+        parser.setUndecoratedDecimal(true);<br />
     }</p>
<p>     public DataTable createTable() {<br />
@@ -90,7 +93,7 @@<br />
         values.put(value.getName(), value);<br />
         return value;<br />
     }<br />
-<br />
+<br />
     public void dropTable(DataTable table) {<br />
         dropTable(table.getName());<br />
     }<br />
@@ -306,6 +309,10 @@<br />
                         } else {<br />
                             System.err.println("unexpected classType: '"  + classType + "'");<br />
                         }<br />
+<br />
+                        String exp = xpath.evaluate("@expression", colNode);<br />
+                        if (!("".equals(exp)))<br />
+                            col.setExpression(exp);<br />
                     }<br />
                 }<br />
             }<br />
@@ -463,7 +470,15 @@<br />
                                 //TODO this doesn't take into account type conversion...<br />
                                 //could use a default converter...<br />
                             	System.out.println(colNode.getNodeName() + "=" + colNode.getTextContent());<br />
-                                row.setValue(colNode.getNodeName(), colNode.getTextContent());<br />
+                                String text = colNode.getTextContent();<br />
+                                //convert the text to the appropriate object of the appropriate type<br />
+                                Object val = text;<br />
+                                Class type = table.getColumn(colNode.getNodeName()).getType();<br />
+                                if (type == BigDecimal.class) {<br />
+                                    val = new BigDecimal(text);<br />
+                                }<br />
+                                //TODO do the other types<br />
+                                row.setValue(colNode.getNodeName(), val);<br />
                             }<br />
                         }<br />
                         row.setStatus(DataRow.DataRowStatus.UNCHANGED);<br />
@@ -511,7 +526,9 @@</p>
<p>         return builder.toString();<br />
     }<br />
-<br />
+<br />
+    Parser getParser() { return parser; }<br />
+<br />
     public static void main(String[] args) {<br />
         DataSet ds = createFromSchema(new File("/home/rb156199/dataset.xsd"));<br />
         System.out.println("DataSet: " + ds.name);<br />
@@ -524,4 +541,5 @@</p>
<p>         System.out.println(ds.getSchema());<br />
     }<br />
-}<br />
\ No newline at end of file<br />
+<br />
+}<br />
Index: swingx/src/java/org/jdesktop/dataset/DataValue.java<br />
===================================================================<br />
RCS file: /cvs/jdnc/swingx/src/java/org/jdesktop/dataset/DataValue.java,v<br />
retrieving revision 1.1<br />
diff -u -r1.1 DataValue.java<br />
--- swingx/src/java/org/jdesktop/dataset/DataValue.java	23 Feb 2005 17:51:29 -0000	1.1<br />
+++ swingx/src/java/org/jdesktop/dataset/DataValue.java	22 Apr 2005 03:11:13 -0000<br />
@@ -9,7 +9,12 @@</p>
<p> import java.beans.PropertyChangeListener;<br />
 import java.beans.PropertyChangeSupport;<br />
+import net.sf.jga.fn.EvaluationException;<br />
+import net.sf.jga.fn.Generator;<br />
+import net.sf.jga.fn.adaptor.Constant;<br />
+import net.sf.jga.parser.UncheckedParseException;<br />
 import org.jdesktop.dataset.NameGenerator;<br />
+import net.sf.jga.parser.ParseException;</p>
<p> /**<br />
  * TODO implement a PropertyChangeListener on the synthetic "value" field.<br />
@@ -30,15 +35,17 @@</p>
<p>     private DataSet dataSet;<br />
     private String name;<br />
-<br />
+<br />
+    // Stores the expression used to compute this value.<br />
     private String expression;<br />
-<br />
+    private Generator<?> exprImpl = new Constant(null);<br />
+<br />
     /** Creates a new instance of DataValue */<br />
     public DataValue(DataSet ds) {<br />
         assert ds != null;<br />
         this.dataSet = ds;<br />
         name = NAMEGEN.generateName(this);<br />
-    }<br />
+   }</p>
<p> 	/**<br />
 	 * @param name<br />
@@ -65,18 +72,24 @@<br />
     }</p>
<p>     public void setExpression(String expression) {<br />
+        if (expression ==  null || expression.equals(""))<br />
+            exprImpl = new Constant(null);<br />
+        else {<br />
+            try {<br />
+                exprImpl = getParser().parseDataValue(expression);<br />
+            }<br />
+            catch (ParseException x) { throw new UncheckedParseException(x); }<br />
+        }<br />
+<br />
         this.expression = expression;<br />
     }</p>
<p>-    public Object getValue() {<br />
-        if (expression != null) {<br />
-            //TODO Hook functor stuff in here for expressions<br />
-            return null;<br />
-        } else {<br />
-            return null;<br />
-        }<br />
+    public Object getValue() throws EvaluationException {<br />
+        return exprImpl.gen();<br />
     }</p>
<p>+    Parser getParser() { return dataSet.getParser(); }<br />
+<br />
     public void addPropertyChangeListener(PropertyChangeListener listener) {<br />
         pcs.addPropertyChangeListener(listener);<br />
     }<br />
@@ -92,4 +105,4 @@<br />
     public void removePropertyChangeListener(String propertyName, PropertyChangeListener listener) {<br />
         pcs.removePropertyChangeListener(propertyName,  listener);<br />
     }<br />
-}<br />
\ No newline at end of file<br />
+}<br />
Index: swingx/src/test/org/jdesktop/dataset/DataTableTest.java<br />
===================================================================<br />
RCS file: /cvs/jdnc/swingx/src/test/org/jdesktop/dataset/DataTableTest.java,v<br />
retrieving revision 1.4<br />
diff -u -r1.4 DataTableTest.java<br />
--- swingx/src/test/org/jdesktop/dataset/DataTableTest.java	28 Feb 2005 12:02:30 -0000	1.4<br />
+++ swingx/src/test/org/jdesktop/dataset/DataTableTest.java	22 Apr 2005 03:11:23 -0000<br />
@@ -7,11 +7,13 @@</p>
<p> package org.jdesktop.dataset;</p>
<p>-import junit.framework.*;<br />
-import java.util.List;</p>
<p>+import java.math.BigDecimal;<br />
+import java.util.List;<br />
+import junit.framework.*;<br />
 import org.jdesktop.dataset.provider.LoadTask;<br />
 import org.jdesktop.dataset.provider.SaveTask;<br />
+import net.sf.jga.parser.ParseException;</p>
<p> /**<br />
@@ -60,6 +62,51 @@<br />
         assertEquals(cols.get(0), col);<br />
     }</p>
<p>+    public void testCreateComputedColumn() throws ParseException {<br />
+        DataSet ds = new DataSet();<br />
+        DataTable table = ds.createTable();<br />
+        table.setName("TEST");<br />
+<br />
+        DataColumn price = table.createColumn();<br />
+        price.setName("price");<br />
+        price.setType(BigDecimal.class);<br />
+<br />
+        DataColumn discount = table.createColumn();<br />
+        discount.setName("discount");<br />
+        discount.setType(BigDecimal.class);<br />
+<br />
+        DataColumn tax = table.createColumn();<br />
+        tax.setName("tax");<br />
+        tax.setType(BigDecimal.class);<br />
+        tax.setExpression("price * (1.0 - discount) * 0.075");<br />
+<br />
+        DataColumn description = table.createColumn();<br />
+        description.setName("description");<br />
+        description.setType(String.class);<br />
+<br />
+        DataColumn descLen = table.createColumn();<br />
+        descLen.setName("desc_len");<br />
+        descLen.setType(Integer.class);<br />
+        descLen.setExpression("description.length()");<br />
+<br />
+        DataRow row = table.appendRow();<br />
+        row.setValue("price", new BigDecimal("100.00"));<br />
+        row.setValue("discount", new BigDecimal("0.10"));<br />
+        row.setValue("description", "foo");<br />
+        assertEquals(new BigDecimal("6.7500000"), row.getValue("tax"));<br />
+        assertEquals(3, row.getValue("desc_len"));<br />
+<br />
+        descLen.setExpression("TEST.description.length()");<br />
+        assertEquals(3, row.getValue("desc_len"));<br />
+<br />
+        tax.setExpression("TEST.price * (1.0 - TEST.discount) * 0.075");<br />
+        assertEquals(new BigDecimal("6.7500000"), row.getValue("tax"));<br />
+    }<br />
+<br />
+<br />
     /**<br />
      * Test of getColumns method, of class org.jdesktop.dataset.DataTable.<br />
      */<br />
@@ -505,5 +552,9 @@<br />
             // TODO Auto-generated method stub<br />
             return null;<br />
         }<br />
+    }<br />
+<br />
+    static public void main(String[] args) {<br />
+        junit.textui.TestRunner.run(DataTableTest.class);<br />
     }<br />
 }<br />
Index: swingx/src/test/org/jdesktop/dataset/DataValueTest.java<br />
===================================================================<br />
RCS file: /cvs/jdnc/swingx/src/test/org/jdesktop/dataset/DataValueTest.java,v<br />
retrieving revision 1.2<br />
diff -u -r1.2 DataValueTest.java<br />
--- swingx/src/test/org/jdesktop/dataset/DataValueTest.java	25 Feb 2005 19:46:06 -0000	1.2<br />
+++ swingx/src/test/org/jdesktop/dataset/DataValueTest.java	22 Apr 2005 03:11:25 -0000<br />
@@ -8,6 +8,8 @@<br />
 package org.jdesktop.dataset;</p>
<p> import junit.framework.*;<br />
+import net.sf.jga.parser.ParseException;<br />
+import net.sf.jga.parser.UncheckedParseException;</p>
<p> /**<br />
@@ -106,7 +108,113 @@<br />
     public void testGetValue() {<br />
         System.out.println("testGetValue");</p>
<p>+        DataSet ds = new DataSet();<br />
+        DataValue v = ds.createValue();<br />
+        assertNull(v.getValue());<br />
+        v.setExpression("");<br />
+        assertNull(v.getValue());<br />
+        v.setExpression("3");<br />
+        assertEquals(3, v.getValue());<br />
+        v.setExpression("Math.sin(Math.PI)");<br />
+        assertEquals(Math.sin(Math.PI), v.getValue());<br />
+        v.setExpression(null);<br />
+        assertNull(v.getValue());<br />
+        v.setExpression("\"\".length()");<br />
+        assertEquals(0, v.getValue());<br />
+<br />
+        DataTable table = ds.createTable();<br />
+        table.setName("PIDS");<br />
+        DataColumn pid = table.createColumn();<br />
+        pid.setName("pid");<br />
+        pid.setType(Integer.class);<br />
+        DataColumn factor = table.createColumn();<br />
+        factor.setName("factor");<br />
+        factor.setType(Integer.class);<br />
+        DataColumn name = table.createColumn();<br />
+        name.setName("name");<br />
+        name.setType(String.class);<br />
+<br />
+        DataRow row0 = table.appendRow();<br />
+        row0.setValue("pid", 100); row0.setValue("factor", 2); row0.setValue("name", "foo");<br />
+<br />
+        DataRow row1 = table.appendRow();<br />
+        row1.setValue("pid", 101); row1.setValue("factor", 3); row1.setValue("name", "bar");<br />
+<br />
+        DataRow row2 = table.appendRow();<br />
+        row2.setValue("pid", 102); row2.setValue("factor", 5); row2.setValue("name", "flummox");<br />
+<br />
+        v.setExpression("count(PIDS.pid)");<br />
+        assertEquals(3, v.getValue());<br />
+<br />
+        v.setExpression("max(PIDS.pid)");<br />
+        assertEquals(102, v.getValue());<br />
+<br />
+        v.setExpression("min(PIDS.pid)");<br />
+        assertEquals(100, v.getValue());<br />
+<br />
+        v.setExpression("sum(PIDS.pid)");<br />
+        assertEquals(303, v.getValue());<br />
+<br />
+        v.setExpression("avg(PIDS.pid)");<br />
+        assertEquals(101, v.getValue());<br />
+<br />
+        v.setExpression("min(3 * PIDS.pid)");<br />
+        assertEquals(300, v.getValue());<br />
+<br />
+        v.setExpression("max(PIDS.pid * PIDS.factor)");<br />
+        assertEquals(510, v.getValue());<br />
+<br />
+        // After the first complete column reference, the table is optional<br />
+        v.setExpression("sum(PIDS.pid * factor)");<br />
+        assertEquals(1013, v.getValue());<br />
+<br />
+        v.setExpression("max(PIDS.name)");<br />
+        assertEquals("foo", v.getValue());<br />
+<br />
+        v.setExpression("max(PIDS.name.length())");<br />
+        assertEquals(7, v.getValue());<br />
+<br />
+        v.setExpression("avg(PIDS.name.length())");<br />
+        assertEquals(4, v.getValue());<br />
+<br />
+        v.setExpression("sum(PIDS.name.length())");<br />
+        assertEquals(13, v.getValue());<br />
+<br />
+        v.setExpression("max(\"[\"+PIDS.name+\"]\")");<br />
+        assertEquals("[foo]", v.getValue());<br />
+<br />
+        v.setExpression("max(\"[\"+PIDS.name+\"]\".length())");<br />
+        assertEquals("[foo1", v.getValue());<br />
+<br />
+        v.setExpression("max((\"[\"+PIDS.name+\"]\").length())");<br />
+        assertEquals(9, v.getValue());<br />
+<br />
+        v.setExpression("sum(PIDS.pid * name.length())");<br />
+        assertEquals(300 + 303 +714, v.getValue());<br />
+<br />
+        try {<br />
+            v.setExpression("min(pid * PIDS.factor)");<br />
+            fail("Shouldn't be able to resolve 'pid' without table reference");<br />
+        }<br />
+        catch (UncheckedParseException x) {}<br />
+<br />
+        try {<br />
+            v.setExpression("min(foo)");<br />
+            fail("Shouldn't be able to resolve 'foo'");<br />
+        }<br />
+        catch (UncheckedParseException x) {}<br />
+<br />
+        try {<br />
+            v.setExpression("min(BOGUS.table)");<br />
+            fail("Shouldn't be able to resolve 'BOGUS'");<br />
+        }<br />
+        catch (UncheckedParseException x) {}<br />
+<br />
         // TODO add your test code below by replacing the default call to fail.<br />
 //        fail("The test case is empty.");<br />
-    }<br />
-}<br />
\ No newline at end of file<br />
+    }<br />
+<br />
+    static public void main(String[] args) {<br />
+        junit.textui.TestRunner.run(DataValueTest.class);<br />
+    }<br />
+}<br />

And here's the Parser class referenced (current state -- it'll probably need to be adjusted a bit as we smooth out the wrinkles to make it fit in better)

<br />
/*<br />
 * $Id: $<br />
 *<br />
 * Copyright 2005 David A. Hall<br />
 */</p>
<p>package org.jdesktop.dataset;</p>
<p>import java.text.MessageFormat;<br />
import java.util.Arrays;<br />
import java.util.Iterator;<br />
import java.util.List;<br />
import net.sf.jga.fn.BinaryFunctor;<br />
import net.sf.jga.fn.Generator;<br />
import net.sf.jga.fn.UnaryFunctor;<br />
import net.sf.jga.fn.adaptor.ApplyUnary;<br />
import net.sf.jga.fn.adaptor.Constant;<br />
import net.sf.jga.fn.adaptor.ConstantUnary;<br />
import net.sf.jga.fn.adaptor.Identity;<br />
import net.sf.jga.fn.algorithm.Accumulate;<br />
import net.sf.jga.fn.algorithm.TransformUnary;<br />
import net.sf.jga.fn.arithmetic.Divides;<br />
import net.sf.jga.fn.arithmetic.Plus;<br />
import net.sf.jga.fn.comparison.Max;<br />
import net.sf.jga.fn.comparison.Min;<br />
import net.sf.jga.fn.property.ArrayBinary;<br />
import net.sf.jga.fn.property.GetProperty;<br />
import net.sf.jga.fn.property.InvokeMethod;<br />
import net.sf.jga.fn.property.InvokeNoArgMethod;<br />
import net.sf.jga.parser.FunctorParser;<br />
import net.sf.jga.parser.FunctorRef;<br />
import net.sf.jga.parser.GeneratorRef;<br />
import net.sf.jga.parser.ParseException;<br />
import net.sf.jga.parser.UnaryFunctorRef;<br />
import net.sf.jga.util.Iterators;</p>
<p>class Parser extends FunctorParser {<br />
    private DataTable table;</p>
<p>    boolean inTableContext = false;</p>
<p>    // Functor that returns the list of rows for a given table<br />
    private UnaryFunctor*/> getRowsFn =<br />
        new GetProperty*/>(DataTable.class, "Rows");</p>
<p>    // Functor that returns the number of rows in a list<br />
    private UnaryFunctor countRowsFn =<br />
        new InvokeNoArgMethod*/,Integer>(List.class, "size").compose(getRowsFn);</p>
<p>    // Functor that returns an iterator over the rows in a list<br />
    private UnaryFunctor*/> iterateTableFn =<br />
        new InvokeNoArgMethod*/,Iterator/**/>(List.class, "iterator")<br />
            .compose(getRowsFn);</p>
<p>    // Functor that returns the value of a DataValue<br />
    private UnaryFunctor getValueFn =<br />
        new GetProperty(DataValue.class, "Value");</p>
<p>    // =============================<br />
    // DataSet specific entry points<br />
    // =============================</p>
<p>    /**<br />
     * Parses a computed column expression, for the given table.<br />
     */<br />
    UnaryFunctor parseComputedColumn(DataTable table, String expression)<br />
        throws ParseException<br />
    {<br />
        setCurrentTable(table);<br />
        try {<br />
            return parseUnary(expression, DataRow.class);<br />
        }<br />
        finally {<br />
            setCurrentTable(null);<br />
        }<br />
    }</p>
<p>    /**<br />
     * Parses an expression that computes a summary value for the collection of data<br />
     * currently available in the bound dataset.<br />
     */<br />
    Generator<?> parseDataValue(String expression) throws ParseException {<br />
        inTableContext = true;<br />
        try {<br />
            return parseGenerator(expression);<br />
        }<br />
        finally {<br />
            inTableContext = false;<br />
            setCurrentTable(null);<br />
        }<br />
    }</p>
<p>    // =============================<br />
    // FunctorParser extensions<br />
    // =============================</p>
<p>    protected FunctorRef reservedWord(String name) throws ParseException {<br />
//         System.out.println("reservedWord(\""+name+"\")");</p>
<p>        // If we're expecting a table but we don't know which one, then see if the name<br />
        // is the name of a table.  If so, then this is the table we're expecting.<br />
        if (table == null) {<br />
            DataTable maybeTable = ((DataSet) getBoundObject()).getTable(name);<br />
            if (maybeTable != null) {<br />
                setCurrentTable(maybeTable);<br />
                return new GeneratorRef(new Constant(table), DataTable.class);<br />
            }<br />
        }</p>
<p>        // Next, see if the name is the same as the current table then return a<br />
        // constant reference to the table</p>
<p>        if (table != null && name.equals(table.getName()))<br />
            return new GeneratorRef(new Constant(table), DataTable.class);</p>
<p>        // If we're expecting a table and we don't know which one yet, then<br />
        // if the name is the name of a table, this must it</p>
<p>        if (table != null) {<br />
            DataColumn col = table.getColumn(name);<br />
            if (col != null)<br />
                return makeColumnRef(table, col);<br />
        }</p>
<p>        return null;<br />
    }</p>
<p>    /**<br />
     * Allows for (not necessarily constant) predefined fields to be added to the grammar<br />
     */<br />
    protected FunctorRef reservedField(FunctorRef prefix, String name) throws ParseException {<br />
//         System.out.println("reservedField("+prefix+", \""+name+"\")");</p>
<p>        if (isTableReference(prefix) && getReferencedTable(prefix).equals(table)) {<br />
            DataColumn col = table.getColumn(name);<br />
            if (col != null) {<br />
                return makeColumnRef(table, col);<br />
            }<br />
        }                       </p>
<p>        return super.reservedField(prefix, name);<br />
    }</p>
<p>    /**<br />
     * Allows for function-style names to be added to the grammar.<br />
     */<br />
    protected FunctorRef reservedFunction(String name, FunctorRef[] args) throws ParseException{<br />
//         System.out.println("reservedFunction(\""+name+"\", "+Arrays.toString(args)+")");</p>
<p>        if (inTableContext) {<br />
            assert table != null;<br />
            assert args.length == 1;<br />
            assert args[0].getReferenceType() == FunctorRef.UNARY_FN;<br />
            assert args[0].getArgType(0).equals(DataRow.class);<br />
            assert Comparable.class.isAssignableFrom(args[0].getReturnType());</p>
<p>            // count doesn't require iterating over the rows of the table: having the<br />
            // list of rows is good enough<br />
            if ("count".equals(name))<br />
                return new GeneratorRef(countRowsFn.bind(table), Integer.class);</p>
<p>            // All of the other summary functions require iterating over the list of<br />
            // rows in the table.<br />
            Class type = args[0].getReturnType();<br />
            Generator*/> iterateRows = iterateTableFn.bind(table);<br />
            TransformUnary xform = new TransformUnary(((UnaryFunctorRef) args[0]).getFunctor());<br />
            BinaryFunctor bf = null;</p>
<p>            if ("max".equals(name))<br />
                bf = new Max(new net.sf.jga.util.ComparableComparator());</p>
<p>            else if ("min".equals(name))<br />
                bf = new Min(new net.sf.jga.util.ComparableComparator());</p>
<p>            else if ("sum".equals(name)) {<br />
                if (type.isPrimitive())<br />
                    type = getBoxedType(type);</p>
<p>                if (Number.class.isAssignableFrom(type))<br />
                    bf = new Plus(type);<br />
                else {<br />
                    String msg = "Unable to compute sum of type {0}";<br />
                    Object[] msgargs = new Object[] { type.getName() };<br />
                    throw new ParseException(MessageFormat.format(msg, msgargs));<br />
                }<br />
            }</p>
<p>            // The simpler summary functions only require an iteration, with no further<br />
            // processing.<br />
            if (bf != null) {<br />
                Generator gen = new Accumulate(bf).generate(xform.generate(iterateRows));<br />
                return new GeneratorRef(gen, type);<br />
            }</p>
<p>            // Computing average in this way is more efficient than building an average<br />
            // functor -- although the average of a zero length list may cause a math<br />
            // exception<br />
            if ("avg".equals(name)) {<br />
                if (type.isPrimitive())<br />
                    type = getBoxedType(type);</p>
<p>                BinaryFunctor add = new Plus(type);<br />
                Generator sum = new Accumulate(add).generate(xform.generate(iterateRows));<br />
                Generator count = countRowsFn.bind(table);<br />
                Generator div = new Divides(type).generate(sum,count);<br />
                return new GeneratorRef(div, type);<br />
            }<br />
        }</p>
<p>        return super.reservedFunction(name, args);<br />
    }</p>
<p>    // ======================<br />
    // implementation details<br />
    // ======================</p>
<p>    /**<br />
     * Sets the table against which column references will be created.  There can only be<br />
     * a single table involved in any given expression.<br />
     */<br />
    private void setCurrentTable(DataTable table) throws ParseException {<br />
        if (this.table != null && table != null)<br />
            throw new ParseException("Parser is currently associated with table " +this.table);</p>
<p>        this.table = table;<br />
    }</p>
<p>    /**<br />
     * Builds a functor for a given table and column.  The functor takes a row in the<br />
     * table and returns the value of the appropriate column.<br />
     */<br />
    private UnaryFunctorRef makeColumnRef(DataTable table, DataColumn column) {<br />
        // Builds a functor that takes a Row, and returns an array consisting<br />
        // of that row and the column we've been given<br />
        ApplyUnary args =<br />
            new ApplyUnary(new UnaryFunctor[]<br />
                { new Identity(),new ConstantUnary(column) });</p>
<p>        // getValue(col,row) method as a functor<br />
        InvokeMethod getValue =<br />
            new InvokeMethod(DataTable.class,"getValue",<br />
                                               new Class[] {DataRow.class, DataColumn.class});</p>
<p>        // tie the two together.  The result is a functor that takes a row and returns<br />
        // the value in the designated column<br />
        UnaryFunctor value = getValue.bind1st(table).compose(args);</p>
<p>        // Return a parser description of the functor we've built.<br />
        return new UnaryFunctorRef(value, DataRow.class, ARG_NAME[0], column.getType());<br />
    }</p>
<p>    /**<br />
     * returns true if the functor reference is one that returns a specific data table.<br />
     */<br />
    private boolean isTableReference(FunctorRef ref) {<br />
        return ref != null && ref.getReturnType().equals(DataTable.class) &&<br />
               ref.getReferenceType() == FunctorRef.CONSTANT;<br />
    }</p>
<p>    /**<br />
     * returns the specific table to which the given functor refers.  Assumes isTableReference(ref),<br />
     * will throw ClassCastException if not<br />
     */<br />
    private DataTable getReferencedTable(FunctorRef ref) {<br />
        return (DataTable)((GeneratorRef) ref).getFunctor().gen();<br />
    }<br />
}<br />

Dave Hall
http://jga.sf.net/
http://jroller.com/page/dhall

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
rbair
Offline
Joined: 2003-07-08

Dave

> Rich:
>
> It certainly looks like one of the Longs is clearly
> null, although I can't quite close the loop from your
> code snippet to the stack trace to help figure out
> why.
>
> You're copying values from a 'disclaimer' table to a
> 'DISCLAIMERS' table? Is case-sensitivity an issue?
> I'm not doing any case conversion on table names in
> n the parser, and I don't think that DataSet does,
> either.

I actually could have produced the problem simpler by leaving out some of the code, and apologize. They are two different tables from two different DataSets (moving data from one database to another), but the "disclaimer" table is the only one being operated on by the DataValue

> Should nextId maybe need to be "max(Id)" ?

No, it's right.

> WRT more graceful handling of nulls, the parser
> currently passes whatever values down to fairly raw
> java code: if you were to, for example, attempt to
> add a null to a BigDecimal value in any raw java,
> you'd get a NPE.
>
> I don't (yet) support any form of numeric coercion:
> in this case, the type coercion rules in JSR245 are
> quite a bit more liberal than basic Java rules, and
> they're probably more appropriate for what we're
> doing here.

That's the basic rub -- what to do with nulls. There are two approaches. The first would be to coerce a null into an empty String, or the number 0, or whatever. The second is to do like SQL and treat the entire result as NULL. So that 1 + null = null instead of 1.

The SQL case makes sense if null is being used as an "unknown" state.

Richard

dhall
Offline
Joined: 2006-02-17

Rich

> > That's the basic rub -- what to do with nulls. There are two approaches. The first would be to coerce a null into an empty String, or the number 0, or whatever. The second is to do like SQL and treat the entire result as NULL. So that 1 + null = null instead of 1.

> The SQL case makes sense if null is being used as an "unknown" state.

That's also far easier to handle right away: in the getValue() method (and wherever the functor's fn(...) method is called), just catch NPX and return null.

In the long run, though, it should go into the parser. One of the issues in the javax.el proposal that the jga parser doesn't handle is coercion of operands (in most cases, numeric operands, but in this case, nulls). I have always known that it was an issue to solve.

javax.el has fairly explicit rules for handling nulls as strings and as numbers, so when I do implement support for coercion, it should be possible to do so in a way that allows an extended parser to set up more liberal rules. It'll be another extension point.

Dave

dhall
Offline
Joined: 2006-02-17

>>what about no join logic (since a join is basically a filter)? I should be able to do something like: sum(packages.price) - sum(activities.price)

>> since this type of equation doesn't require a join. I can see where avoiding a join would be in our best interest (at some point its easier to do the work on the backend and load the data directly into a "static" DataValue where the expression is a constant).

I don't believe that a join is a filter in any meaningful sense: SQL mingles the join criteria with the selection criteria, but they really are conceptually different things.

I think that we could do something like

[code]
sum(packages.price) - sum(activities.price)
[/code]

if we really wanted to (it requires telling the parser to forget the 'current table' when it exits the summary function node). It might be cleaner, tho, to leave it the way it is and ensure that we can reference DataValues, which would mean that we'd use three expressions.

[code]
PackagePrice = sum(packages.price)
ActivitiesPrice = sum(activities.price)
TotalPrice = PackagePrice + ActivitiesPrice
[/code]

Let's sleep on that one for a couple of days: if we want to pull that particular trigger, I think I know where to make the change. I think making the DataValue change is a lot more important.

dhall
Offline
Joined: 2006-02-17

Rich:

The boards really exploded there for a day or two, didn't they. I'm posting this here to keep all of the code in one thread (rather than reply on the thread where you threw this idea out).

I'm not sure how you'd want to hook this up, so I'll just give you the test patch in its current state, and a couple of changes to the parser class. First, a new test:

[code]
public void testTableFilter() throws ParseException {
DataSet ds = new DataSet();
DataTable table = ds.createTable();
table.setName("TEST");

DataColumn price = table.createColumn();
price.setName("price");
price.setType(BigDecimal.class);

DataColumn discount = table.createColumn();
discount.setName("discount");
discount.setType(BigDecimal.class);

DataColumn tax = table.createColumn();
tax.setName("tax");
tax.setType(BigDecimal.class);
tax.setExpression("price * (1.0 - discount) * 0.075");

DataColumn description = table.createColumn();
description.setName("description");
description.setType(String.class);

DataColumn descLen = table.createColumn();
descLen.setName("desc_len");
descLen.setType(Integer.class);
descLen.setExpression("description.length()");

DataRow row = table.appendRow();
row.setValue("price", new BigDecimal("100.00"));
row.setValue("discount", new BigDecimal("0.10"));
row.setValue("description", "foo");

Parser parser = table.getDataSet().getParser();
UnaryFunctor filter = parser.parseTableFilter(table, "price > 100.00");
assertNotNull(filter);
assertFalse(filter.fn(row));

filter = parser.parseTableFilter(table, "TEST.desc_len < 32");
assertNotNull(filter);
assertTrue(filter.fn(row));

filter = parser.parseTableFilter(table, "TEST.tax > 0.0 && tax < 32.00");
assertNotNull(filter);
assertTrue(filter.fn(row));

DataTable bogus = ds.createTable();
bogus.setName("BOGUS");

DataColumn bogusKey = bogus.createColumn();
bogusKey.setName("bogus_key");

try {
assertNull(parser.parseTableFilter(table, "BOGUS.bogus_key != null"));
fail("Expecting ParseException when expression references wrong table");
}
catch (ParseException px) {}
}
[/code]

I didn't have a better idea where to write the test, so I dropped it in DataTable. Feel free to reorganize this as you feel necessary.

The changes to the parser are simple: first, change the declaration such that the Parser derives from GenericParser instead of FunctorParser (GenericParser allows you to check the return types of functors against a class that you supply)

[code]
class Parser extends GenericParser {
[/code]

and add this method

[code]
/**
* Parses a table filter expression for the given table.
*/
UnaryFunctor parseTableFilter(DataTable table, String expression)
throws ParseException
{
setCurrentTable(table);
try {
return parseUnary(expression, DataRow.class, Boolean.class);
}
finally {
setCurrentTable(null);
}
}
[/code]

It might be more convenient to use a form that only takes an expression, but I'd like to be able to reject expression that filter on the 'wrong' table.

Dave

dhall
Offline
Joined: 2006-02-17

I noticed that you've got some plumbing installed that will allow DataValues to be updated when something on which they depend is updated. It shouldn't be too hard to hook up that plumbing by analyzing the functors with a Visitor -- that is, after all, why I put that support in place. and I've done it once already in the worksheet.

It seems to me that we'll want DataValues to listen to either DataValues or DataTables (or both, obviously) and we want computed columns to listen to the columns on which they depend.

Dave

Patrick Wright

Dave

Have you looked at the APIs for the Apache/Jakarta Bean Scripting Framework?
http://jakarta.apache.org/bsf/

Wondering if your parser could be instantiated as with other JVM
scripting languages...

Patrick

---------------------------------------------------------------------
To unsubscribe, e-mail: jdnc-unsubscribe@jdnc.dev.java.net
For additional commands, e-mail: jdnc-help@jdnc.dev.java.net

dhall
Offline
Joined: 2006-02-17

Patrick

I haven't looked at it, but I do have an interesting list of similar frameworks to consider as I go forward. There's a number of places where I think I can integrate either the base parser or the hacker's worksheet.

Right now, I consider the parser by iteslf too small to be considered a 'scripting language' for purposes of comparison with Groovy, etc, as it really only handles single expressions (until I add a comma/semicolon operator or some such). The worksheet might be the piece that has enough potential to qualify, but I need to go off and study these frameworks for a while.

I'll definitely consider this one, though.

Dave

ps: at some point, I probably ought to consider naming the beast

Patrick Wright

Dave

> I haven't looked at it, but I do have an interesting list of similar frameworks to consider as I go forward. There's a number of places where I think I can integrate either the base parser or the hacker's worksheet.
>
> Right now, I consider the parser by iteslf too small to be considered a 'scripting language' for purposes of comparison with Groovy, etc, as it really only handles single expressions (until I add a comma/semicolon operator or some such). The worksheet might be the piece that has enough potential to qualify, but I need to go off and study these frameworks for a while.
>
> I'll definitely consider this one, though.

Cool. Even more so if you look at the new scripting JSR that is
targeted for inclusion in Mustang.
http://www.jcp.org/en/jsr/detail?id=223

Patrick

---------------------------------------------------------------------
To unsubscribe, e-mail: jdnc-unsubscribe@jdnc.dev.java.net
For additional commands, e-mail: jdnc-help@jdnc.dev.java.net

dhall
Offline
Joined: 2006-02-17

Patrick:

>> Cool. Even more so if you look at the new scripting JSR that is targeted for inclusion in Mustang.

JSR223 was what got the list started in the first place, a few months ago. I skimmed that spec today at lunch and there weren't any daunting issues that jumped out. It looks like that particular EG is trying to be as inclusive as possible.

Dave

rbair
Offline
Joined: 2003-07-08

Hey Dave,

I was working with the DataValue this morning, and got an error. The data types for the columns are Longs. Here's a code snippet:

[code]
DataTable stable = sds.getTable("disclaimer");
DataValue nextId = sds.createValue("maxId");
nextId.setExpression("sum(disclaimer.disclaimer_id)");
// int i=1;
for (DataRow row : ads.getTable("DISCLAIMERS").getRows()) {
DataRow copy = stable.appendRowNoEvent();
copy.setValue("disclaimer_id", nextId.getValue());
copy.setValue("txt", row.getValue("TEXT"));
copy.setValue("description", row.getValue("DESCRIPTION"));
}
sds.saveAndWait();
[/code]

Here's the NPE stacktrace:

[code]
Exception in thread "main" java.lang.NullPointerException
at net.sf.jga.fn.arithmetic.LongMath.plus(LongMath.java:68)
at net.sf.jga.fn.arithmetic.LongMath.plus(LongMath.java:30)
at net.sf.jga.fn.arithmetic.Plus.fn(Plus.java:67)
at net.sf.jga.fn.arithmetic.Plus.fn(Plus.java:34)
at net.sf.jga.fn.algorithm.Accumulate.fn(Accumulate.java:111)
at net.sf.jga.fn.algorithm.Accumulate.fn(Accumulate.java:45)
at net.sf.jga.fn.adaptor.Generate.gen(Generate.java:64)
at org.jdesktop.dataset.DataValue.getValue(DataValue.java:157)
at com.shoplogicinc.conversion.DBConverter.moveDisclaimers(DBConverter.java:64)
at com.shoplogicinc.conversion.DBConverter.main(DBConverter.java:32)
[/code]

Any ideas? I assume one of the longs must be null, though I cannot see how this is the case.
Richard

dhall
Offline
Joined: 2006-02-17

Rich:

It certainly looks like one of the Longs is clearly null, although I can't quite close the loop from your code snippet to the stack trace to help figure out why.

You're copying values from a 'disclaimer' table to a 'DISCLAIMERS' table? Is case-sensitivity an issue? I'm not doing any case conversion on table names in the parser, and I don't think that DataSet does, either.

If those are meant to be the same table, then I'm fairly certain that we don't want to even think about the singular to plural conversion of table names :)

Should nextId maybe need to be "max(Id)" ?

-----

WRT more graceful handling of nulls, the parser currently passes whatever values down to fairly raw java code: if you were to, for example, attempt to add a null to a BigDecimal value in any raw java, you'd get a NPE.

I don't (yet) support any form of numeric coercion: in this case, the type coercion rules in JSR245 are quite a bit more liberal than basic Java rules, and they're probably more appropriate for what we're doing here.

Dave

dhall
Offline
Joined: 2006-02-17

On a related idea, it occurred to me while I was writing the parser that I could add an implicit filter that skipped null values. There's a fairly interesting problem in how to handle cases like "sum(FOO.col_01 + FOO.col_02)" -- without numeric coercion, I'd need to skip all rows in FOO where either col_01 or col_02 is null, which may not yield the expected result.

Dave

rbair
Offline
Joined: 2003-07-08

Awesome! I just applied the patches (with one or two minor variations -- f.i. I didn't let getValue() on DataValue throw an exception, and I had to add some code to parse a DataValue from XML), and my first test just ran swimmingly.

What is the grammer, exactly? How do I know what I can and cannot do?

This rocks.
Richard

dhall
Offline
Joined: 2006-02-17

>> What is the grammer, exactly? How do I know what I can and cannot do?

Pretty much anything you can put on the right side of a java assignment statement except array references, assignment forms (ie +=, -=, ...), increments & decrements (another form of assignment), and anonymous class defintions. You can do arithmetic with arbitrary number classes (note that BigDecimals are directly supported: there's a flag set that makes the parser interpret the string "3.14" as a BigDecimal instead of a Double). You can call constructors and both object and static methods. You can reference an object's fields or an enum's values. You can compare any Comparable value using relational operators (and there's an unimplemented hook implied that would allow the registration of arbitrary Comparators to support relational operations on non-Comparable values). You can reference any class that you can load.

The JDNC extended Parser class supports

1) the five summary functions sum, max, min, avg, and count
2) references to tables and columns

The only real restriction at this point is that an expression cannot reference more than one table: for computed columns it doesn't really make sense, and for computed data values its a rationalization of not wanting to define the join logic.

I looked at allowing a DataValue to reference another DataValue: this would be a way to get around the one table reference restriction. The only obstacle is that DataValue is not typed right now -- if DataValue was typed (like DataColumn is), then it's just a handful of lines to allow one DataValue to reference another.

I also started thinking about including a filtering capability into this -- the syntax will probably be a little different

max(TABLE_REF, value_expression, filter_expression)

for example. I'd want to specifically require the table reference, to allow the parser to enforce that the value expression and the filter expression are based on the same table.

There's two major modes, implied by the two entry points. When parsing a computed column in a table, the table name is implied -- it can occur in the expression, but it is optional. In that case, you can use column names in the table as reserved names. The other mode is the DataValue mode, in which the table name must be used the first time a column is referenced (this is fairly well demonstrated by the tests). After the first column reference, the table name is optional (ie, in 'PIDS.pid * PIDS.factor', the table reference is required on the first column reference, but optional thereafter).

I'll see if I can throw together a BNF that covers the extensions -- there's a BNF for the base class parser on the jga website somewhere

Dave Hall
http://jga.sf.net/
http://jroller.com/page/dhall/

rbair
Offline
Joined: 2003-07-08

Thanks for the info. Some form of this reply will probably be worked into online documentation for the databinding users out there.

> The only real restriction at this point is that an
> expression cannot reference more than one table: for
> computed columns it doesn't really make sense, and
> for computed data values its a rationalization of not
> wanting to define the join logic.

what about no join logic (since a join is basically a filter)? I should be able to do something like: sum(packages.price) - sum(activities.price)

since this type of equation doesn't require a join. I can see where avoiding a join would be in our best interest (at some point its easier to do the work on the backend and load the data directly into a "static" DataValue where the expression is a constant).

> I looked at allowing a DataValue to reference another
> DataValue: this would be a way to get around the one
> table reference restriction. The only obstacle is
> that DataValue is not typed right now -- if DataValue
> was typed (like DataColumn is), then it's just a
> handful of lines to allow one DataValue to reference
> another.

This needs to be done for sure. DataValue should have a class type, just like DataColumn, because otherwise I can't generate appropriate MetaData for it for binding. So, not having a class type on DataColumn is an API "bug".

> I also started thinking about including a filtering
> capability into this -- the syntax will probably be a
> little different
>
> max(TABLE_REF, value_expression, filter_expression)

Now, that's really cool. The only thing that concerns me again is that I don't think we really want to many of SQL's capabilities built in here, especially with a non sql syntax. However, it sure is inticiing.

> I'll see if I can throw together a BNF that covers
> the extensions -- there's a BNF for the base class
> parser on the jga website somewhere

that'd be awesome.

I'll be updating the adventure builder demo soon to include this stuff.

Richard

dhall
Offline
Joined: 2006-02-17

Rich

The BNF is http://jga.sourceforge.net/docs/javadoc/net/sf/jga/parser/parserdoc.html

The parser that I've submitted does not change the structure of the
language described there: it only augments the semantic meaning in the
following ways.

The base parser allows the user to import names into the grammar via a
static import mechanism similar to that in the java compiler. The
JDNC parser augments the static import resolution with the ability to
treat table and column names as if they had been imported into the
parser. Additionally, the parser implements the summary methods as if
they had also been statically imported.

The summary methods associated with DataValue are reserved functions,
hard coded by the parser. This is an extension of the interpretation
if the ImportedStaticMethodCall production. In this case, imported
static methods will override the functions built into the parser.

The table name is an extension of the interpretation of the Name
production. These are implemented. The base class provides a
reservedWord hook that is the first thing that it tries if a Name
cannot be interpreted as a placeholder or imported field.

Column names are another extension of the imported Field mechanism,
but in this case, the column names will override imported fields (as
the parser class for JDNC checks for table references before falling
through to the base class functionality). Column names extend the
interpretation of imported fields, and are caught in two places in the
grammar that recognize identifiers without arguments that occur after
a period: during the Name production (which catches names in the form
'foo.bar.baz') and during the PrimarySuffix production.

Dave Hall
http://jga.sf.net/
http://jroller.com/page/dhall/

PS: one thing I forgot in the description of what you can and can't
use in expressions. There is no automatic numeric conversions applied
-- mixed type expressions are not allowed. You can't, for example,
use expressions like '2 * Math.PI', as it won't allow you to mix
integers and doubles.