Skip to main content

AdventureDemo/DataSet invalid XSD reference

29 replies [Last post]
Anonymous

Continuing to aim for NetBeans/labs integration, now working on
running the AdventureDemo in swinglabs-demos.

The code references dataset.xsd, and I pointed it at the version under
demo/adventure/resources.

Blowing up at org.jdesktop.dataset.DataSet.createFromSchema(DataSet.java:400).
This is looking up a class to act as a DataProvider, but there is no
such element in the XSD (complexType/sequence/appinfo/dataProvider).
Somehow this doesn't make sense--why would a schema define a
DataProvider? The schema is structural, the provider manages data
flow.

Anyway, looks like this was added recently in the DataSet (1.3), just
wondering if there is a matching XSD for the adventure demo.

Patrick

---------------------------------------------------------------------
To unsubscribe, e-mail: jdnc-unsubscribe@jdnc.dev.java.net
For additional commands, e-mail: jdnc-help@jdnc.dev.java.net

Reply viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.
evickroy
Offline
Joined: 2004-07-23

In one view of the interaction, the user would launch the application which might, or might not, populate the DataSet initially. The user could initiate a query which would:
a) client: write the DataSet to XML
b) client: send the XML to the server requesting a populate
c) server: read the DataSet and execute query
d) server: write DataSet and send XML back in the response
e) client: read DataSet and refresh screen

> hmm...need to reinstantiate DS from incoming XML
> XML without breaking binding...
> hope that will "just work".

Quesiton: With the current DataSet implementation, what happens if a build the DataSet with multiple tables manually in code, add rows manually to the tables, and then call writeXML on the DataSet? Does it write all of the tables into the same file?
It seems to make sense to decouple the data from the DataSet so that we could read it and write it without changing the DataSet structure. I'm guessing that is the point of the DataProvider right? If that's the case, then maybe what we need is an XML/SQL DataProvider. The DataProvider needs to know how to connect and communicate with the database on the server side, but not on the client side. So the server needs XML and SQL DataProvider
while the client just needs an XML DataProvider.

Your thoughts?

>
> Can you write something that will send/receive the
> XML? Could just use
> local sockets, or hell, just a direct method call
> (we're not testing
> the network connectivity issue here). I'll test the
> writeXML() and
> readXML() to see they can be used this way, and see
> the easiest way to
> have them send modified rows only and then post back
> here.
>

I can do that.

Erik

Patrick Wright

Hi Erik

On 4/29/05, jdnc-interest@javadesktop.org wrote:
> In one view of the interaction, the user would launch the application which might, or might not, populate the DataSet initially. The user could initiate a query which would:
> a) client: write the DataSet to XML
> b) client: send the XML to the server requesting a populate
> c) server: read the DataSet and execute query
> d) server: write DataSet and send XML back in the response
> e) client: read DataSet and refresh screen

I've been mulling this for awhile now.

As I see it, SQL in its basic form is process-agnostic, by which I
mean, the data modification language will modify any table in one of
several standard ways. The server has no knowledge, or, let's say, no
interest in which table in particular you're working with. When we
manipulate the Order table, it might as well be CarEngine or House.

This is in contrast with a transaction or process-based approach,
which is specific to a certain type of manipulation--say, AddOrder or
CopyCarEngine.

When we talk about some server component that listens over the wire to
our rich client, we have these two major ways to address the problem.
Our DataSet can generate generic manipulation commands in SQL format,
or send a generic XML format that represents the changes made to a
table. Or our DataSet could generate process-specific commands which
are then executed on the server. The process-specific approach would
be harder to generalize in the DataSet API without some process and
naming conventions.

What I understand from your suggestion is that we are serializing the
DataSet across the wire, and the server component is then using
standard DataSet APIs to persist the changes locally. To make it more
efficient, we only serialize the changes to the DataSet.

The reason I'm describing it this way is that serialization will fit
in a different position in the API than persistence or command
execution will, at least, I think it should. The current persistence
mechanism is pushed down to DataTable, and is row-based--the DT loops
over rows and those that are modified get saved. That maps cleanly to
a command/execution approach--the current command/execution being SQL
DML.

If we are transporting the DataSet across the wire, then I think we
need to do this at a higher level--that is, the commands need to be
executed across the DataSet itself.

I'd approach this by doing it manually at first until we can figure
out an API. We need to figure out whether the process should be
synchronous or asynchronous, and that question occurs on two
levels--with respect to the UI, and with respect to the client/server
relationship. Additionally, newly inserted rows may get an
auto-generated primary key which needs to be propagated down to
children, which means the server has to stream back the updated
DataSet to the client, which then needs to refresh any binding (a new
Object value for a primary key may cause ripples...?). Then there is
the issue of error reporting and handling.

So--I'd agree with your list as long as by "XML" we mean "serialized
form of the modified DataSet rows", and that the send/receive is, for
right now, handled outside of the DataSet API as a special case.

>
> > hmm...need to reinstantiate DS from incoming XML
> > XML without breaking binding...
> > hope that will "just work".
>
> Quesiton: With the current DataSet implementation, what happens if a build the DataSet with multiple tables manually in code, add rows manually to the tables, and then call writeXML on the DataSet? Does it write all of the tables into the same file?

It is agnostic as far as the origin of Tables and Rows. The current
dump to XML throws all tables and rows in there.

> It seems to make sense to decouple the data from the DataSet so that we could read it and write it without changing the DataSet structure. I'm guessing that is the point of the DataProvider right? If that's the case, then maybe what we need is an XML/SQL DataProvider. The DataProvider needs to know how to connect and communicate with the database on the server side, but not on the client side. So the server needs XML and SQL DataProvider
> while the client just needs an XML DataProvider.

This is what I don't agree with right now. DataProvider is built
around the a core concept of executing commands against some process.
That will work with database-style semantics (which includes flat file
databases I think), and would also work with web services that are
(roughly) row-based. The nice thing about the current DP approach is
that it can be used with single table updates or with many tables and
with master/detail tables. For XML, it would work IMO if we used
XUpdate (which AFAIK, is not in widespread use), but is not the right
approach for sending a whole DataSet over to some remote server.

In SQL, databases that I've worked with allow you to "batch" commands,
but IMO the semantics of batching in JDBC are poorly defined (how
large a batch is accepted, how batches are processed). Thus it makes
sense, as a first approach, to have the DP be row-oriented rather than
DataSet oriented. And I think if we want to serialize this stuff and
send it across the wire, it makes more sense to work at the level of
the DataSet.

All that said, I'd prefer to be flexible and adapt to people's needs,
as long as the end result is more or less rational. As you're trying
this out, your feedback in "real world" situations will help a lot.

Patrick

---------------------------------------------------------------------
To unsubscribe, e-mail: jdnc-unsubscribe@jdnc.dev.java.net
For additional commands, e-mail: jdnc-help@jdnc.dev.java.net

Richard Bair

Hey guys,

>When we talk about some server component that listens over the wire to
>our rich client, we have these two major ways to address the problem.
>Our DataSet can generate generic manipulation commands in SQL format,
>or send a generic XML format that represents the changes made to a
>table. Or our DataSet could generate process-specific commands which
>are then executed on the server. The process-specific approach would
>be harder to generalize in the DataSet API without some process and
>naming conventions.
[snip]
>So--I'd agree with your list as long as by "XML" we mean "serialized
>form of the modified DataSet rows", and that the send/receive is, for
>right now, handled outside of the DataSet API as a special case.

I agree with this. Essentially, there are two parts to the problem. One is
being able to serialize changes to XML and merge changes back into the
DataSet (analogous to generating a "patch" to a source file and merging
"patches" back into a source file). The other part of the problem is writing
the XML-RPC/SOAP/REST/Whatever for sending the serialized information to the
server and getting it back from the server. The first part is within scope
of databinding, the second is not.

The second is also a very essential part, and a simple framework or two can
be drawn up without too much trouble. Personally I implemented such a system
previously by having a single servlet. One of the params set to the servlet
was the name of the method you wanted to call on the server side. The HTML
content section was the XML document carrying the payload. So it was really
just a stripped down to-the-metal form of remote procedure calling. The
basic servlet was only 150 or so lines of code. This kind of framework
belongs in the JDNC space rather than the databinding space since the JDNC
source base is going to contain more application framework specific stuff.

Lets focus on the first issue right now, and leave the second issue for
later. Especially since everybody will need the first issue solved, while
everybody will have a different approach to tackling the second issue.

Richard

---------------------------------------------------------------------
To unsubscribe, e-mail: jdnc-unsubscribe@jdnc.dev.java.net
For additional commands, e-mail: jdnc-help@jdnc.dev.java.net

rbair
Offline
Joined: 2003-07-08

> > I believe we are basically trying to serialize the
> entire DataSet, but only with rows that are
> updateable. (i.e. Inserted, Deleted, or Updated).
> > The client application can then transport the XML
> to a Servlet that can then deserialize the DataSet
> and apply the modifications to the database.
>
> Got it.

Ya, that's the end goal. How we get there might not be (quite) so direct.

> > In the RowSet implementations from Sun, you can
> customize the reader and writer for the RowSets. I
> was picturing the same functionality on a DataSet. A
> customized writer could filter out unmodified rows.
>
> In the DataSet classes, this would be handled by a
> DataProvider. For
> example, the SQLDataProvider currently skips updates
> for columns that
> have not changed, and for rows that have not changed;
> seems to make
> sense to leave this there.

In the simplist form, what we really want is a method that returns all of the changed/inserted/deleted rows. We can actually do this now by iterating on all Rows and retrieving those with DataRowStatus.CHANGED/DELETED/INSERTED. The next requirement is to be able to generate XML based on this.

If you were interested in sending/receiving data for a single table (ala WebRowSet), then having a DataProvider for the DataTable that understands WebRowSet is a great idea. I actually have an implementation here that read WebRowSet XML, and I have source code that will generate WebRowSet XML (though it hasn't been adapted for DataProvider yet).

However, if you want to send the entire DataSet at once, then the best thing to do would be to use the loadXml/saveXml (or whatever they should be called) methods on the DataSet itself, or preferrably in some utility class. This would generate DataSet XML for the entire DataSet.

The DataSet XML is much simpler to read & modify & parse than WebRowSet XML. However, as noted the DataSet schema is not passed along with the XML. In practice I'm not sure this is a problem since the server code and the client code can both access the schema via some common jar, for instance. Passing the entire DataSet schema in the file would be a pain, but might also be possible (does anybody know if xml schema can be passed IN the same XML document, or does it have to be a separate document?).

One way to automate this process would be to create a duplicate DataSet that contains the modified rows (like CachedRowSet will create a new RowSet with modified rows only), and then call "toXml" (or whatever) on the new DataSet. Also, the ADO.NET apis have a "merge" method for merging rows from multiple DataSets into one DataSet.

> > I'm not that familiar with the DataSet schema, but
> the WebRowSet schema contains both the datamodel
> metadata and the data in the same file. For a
> DataSet, it seems that we have to send the data and
> the dataset metadata in order to recreate a DataSet
> on the other side, since these are in two separate
> files now. Does that sound correct?

Unless you have the DataSet schema on the server already, which would be the preferred approach.

> My own feeling is that the current DataSet schema is
> a place-holder.
> DataSet should probably support more than one schema
> anyway, because
> AFAIK, different database vendors have not come
> together on a database
> schemas in XML. So--as far as this new feature goes,
> I think we don't
> need to rely on the DS schema.

I'm not sure what the DataSet XML schema has to do with JDBC vendors?

The real purpose of the DataSet XML schema is two fold:

1) Provide a way to validate DataSet XML documents. WebRowSet XML is clumsy and difficult (ie: impossible) to edit by hand, whereas ADO.NET XML documents are very easy to read and edit by hand. The XML schema provides the structure of these files to parsers etc for validation

2) Describe the DataSet structure in a file separate from Java code to both help keep the java code clean and also to make tools support easier.

In that regard, I forsee the DataSet XML schema becoming the most widespread means for defining DataSets (since it will be done via NetBeans and perhaps other IDEs).

> I would think that, for these purposes, we should
> stick to the WRS
> schema, as this use case fits roughly within the
> goals of that model.
> What do you think?

This only works for sending/receiving a single DataTable at a time. For some cases this works well, for others, not. So, it depends. However, if you want to work on this use case, I have code to contribute to jump-start the process.

> There are a couple of different issues here that have
> not been tackled
> properly, which we need to address in this case.
>
> - are we looking for write-only support for WRS, or
> read-write?

read-write

> - what are the semantics of the exchange?
>
> In other words--we could write a
> WebRowSetDataProvider that both knows
> how to read and write WRS XML, as well as how to
> interact with a
> servlet. In theory, the DataSet API is designed for
> this. We have a
> communication/transport pipe (the DataConnection) and
> a way to
> read/write using that pipe (DataProvider). The
> DataCommand would in
> theory be useful for building the smaller sections of
> XML used in the
> transactions.
>
> As far as I can see, the major issue won't be with
> the XML, except
> that we need to know if we can include multiple rows
> in one document,
> multiple tables, etc. The real issue is with the
> communication
> protocol. How do we execute against this remote
> service? What is that
> API like? Are we using REST, SOAP, RPC?

Ya, exactly. I used XML-RPC in a previous project (using WebRowSet), and worked well.

> Let's thrash this out (or just clarify it for me) so
> we can figure out
> what changes are needed to support this.

It is important that we keep these two different aspects to the problem separated, I think.

1) A WebRowSet DataProvider than knows how to communicate with a WebRowSet via XML

2) API for writing DataSet XML containing only the modified/changed rows (which could be a consequence of creating a DataSet duplicate and copying over the rows that have modified, and using the normal DataSet XML generation routines from there).

Richard

Patrick Wright

It seems like we have more than one discussion here, though with
useful information all around.

One question is how to marshall a DataSet or DataTable to XML. My
thought till now was that the current schema was a first-cut, and that
we'd need to support other schemas for a DataSet. My comment about
JDBC vendors referred to the fact that some of the major vendors do
have their own XML schemas for result sets, and in some cases for the
DB schema itself; and I think some ERD tools should also have XML
schemas for DB schemas at this point as well. So I have felt that the
DataSet should be schema-agnostic, with a preferred schema shipping
with JDNC.

It is good that you clarify that WebRowSet is single-table specific.
It begs the question of what the semantics of this remote-store
operation are. SQL, in its simplest form, has you executing saves on a
per-row basis: a single insert, update or delete. These are each
network calls to a server process. You can exchange the SQL in that
case for XML, and the custom protocol with an XML-RPC protocol to your
own servlet. The "conversation" is thus about single rows, with a
commit somewhere in there. There are also "batch" conversations which
may span a whole table (all changed rows) or multiple tables (say, a
DataSet). Any of these could be modelled as an XML-RPC conversation,
but it's not clear which level we want and why. The semantics we
should follow are for me, a second question.

The larger question is whether we are just trying to serialize a
DataSet using XML, then instantiate it "on the other side" and proceed
to save. Or are we actually writing a transactional protocol for
saving a result set across the wire? Maybe both are useful. One thing
we used to do with client-server apps was to use the front-end views
to trigger calls to insert/update/delete, which we would then route to
appropriate stored procedures. So instead of INSERT INTO Order(blah),
you'd call SP_NEW_ORDER(blah). This way, you're using a tabular data
set to drive a real transactional process, where the transactions are
defined as more than just a bunch of CRUD statements ending with a
COMMIT. I like that because it fits with the trends towards
SOAP/REST/Services, and with some shoehorning, fits whats already been
started in the DataProvider API.

Last, I think we have one problem, which is to load, modify and store
a *schema*--literally, data tables and relations between them. A
separate, related problem, is to move data around that fits into some
schema or another. You could express this as saying the DataSet schema
could be structure-only, and WebRowSet could be data-only (assuming
you limited to that). I believe these are two different problems,
anyway.

I don't want to drag this conversation on forever--if we can figure
out a use case for the remote storage/execution, then we can start to
mock that up in the incubator and see what falls out.

Patrick

---------------------------------------------------------------------
To unsubscribe, e-mail: jdnc-unsubscribe@jdnc.dev.java.net
For additional commands, e-mail: jdnc-help@jdnc.dev.java.net

evickroy
Offline
Joined: 2004-07-23

Unfortunately, I'm more of an application developer, so I can tell you how I would like to use it rather than build it. I basically want to use a simple servlet, for instance, for brokering data to and from the database. This is pretty easy with a WebRowSet since I can create a simple servlet that accepts a WebRowSet and a command to either populate it or update it. It just reads it from XML into a WebRowSet, performs the desired action, writes the results back to XML, and then returns the results back. The main problem with using a WebRowSet is, as stated, that it can only handle a single table. Another nuisance is that it doesn't have a method for just marshalling the rows that have changed to XML. You could use a JoinRowSet to solve the single table problem, but it's rather primitive compared to DataSet.

I don't have any experience using the ADO.NET DataSets, but I have a friend that raves about it all the time. Maybe we can at least duplicate it's functionality?

Do either of you know how ADO.NET handles a multi-table DataSet?

Patrick Wright

I don't know how ADO.NET does this...

I think we can treat this as a serialization problem--you would need
to have the DataSet classes on your server, and we would serialize it
out (only modified rows) and deserialize it on the other end.

WRS can wait, though I think that will be a useful target as well.

I will start hacking this up in the incubator today.

Patrick

---------------------------------------------------------------------
To unsubscribe, e-mail: jdnc-unsubscribe@jdnc.dev.java.net
For additional commands, e-mail: jdnc-help@jdnc.dev.java.net

evickroy
Offline
Joined: 2004-07-23

Patrick, is there something that I can help out with for this?

Patrick Wright

On 4/29/05, jdnc-interest@javadesktop.org wrote:
> Patrick, is there something that I can help out with for this?

Yeah, let's talk this through. The last few days I've been JavaDoccing
some so I have a better idea of where to add this.

First, we have to decide on the interaction and what is sent
client-server. I think we're close, but let's lay it out clearly.

client
instantiate DataSet
...user modifies various tables
extract DS to XML
stream XML to server
create DS from XML

---------------------------------------------------------------------
To unsubscribe, e-mail: jdnc-unsubscribe@jdnc.dev.java.net
For additional commands, e-mail: jdnc-help@jdnc.dev.java.net

Patrick Wright

Oops! Hold on

> First, we have to decide on the interaction and what is sent
> client-server. I think we're close, but let's lay it out clearly.
>
> client
> instantiate DataSet
> ...user modifies various tables
> extract DS to XML
> stream XML to server
server
> create DS from XML
> attach tables to provider
> call save
> extract DS to XML (for inserted row ids, timestamps, other changes)
> stream back to client
client
hmm...need to reinstantiate DS from incoming XML without breaking binding...
hope that will "just work".

If you have multiple unrelated tables, WebRowSet seems cleaner. If you
have related tables, say master-detail, you may well have inserted
both master and detail rows, and need to save them in order, then pull
the new master PKs and give them to the related detail rows. That
would push us into using a full DataSet in XML.

Can you write something that will send/receive the XML? Could just use
local sockets, or hell, just a direct method call (we're not testing
the network connectivity issue here). I'll test the writeXML() and
readXML() to see they can be used this way, and see the easiest way to
have them send modified rows only and then post back here.

Patrick

---------------------------------------------------------------------
To unsubscribe, e-mail: jdnc-unsubscribe@jdnc.dev.java.net
For additional commands, e-mail: jdnc-help@jdnc.dev.java.net

Patrick Wright

Richard

Do you have checked in a recent DataSet XSD and XML? I'm trying to
load ones checked in to -demos but on writeXML() am getting nothing.

I can mock up my own, just wondering if you have up-to-date ones available.

Patrick

---------------------------------------------------------------------
To unsubscribe, e-mail: jdnc-unsubscribe@jdnc.dev.java.net
For additional commands, e-mail: jdnc-help@jdnc.dev.java.net

Richard Bair

>Do you have checked in a recent DataSet XSD and XML? I'm trying to
>load ones checked in to -demos but on writeXML() am getting nothing.

Unfortunately, no. The last few days I've been writing the NetBeans module
part that creates/edits DataSets visually, so I have a lot of scratch files
laying around but no accompanying data files. However, I'm fixing all of the
schema read/write code this morning so by the end of the day I'll have the
AdventureBuilder demo running again with proper files, etc.

Richard

---------------------------------------------------------------------
To unsubscribe, e-mail: jdnc-unsubscribe@jdnc.dev.java.net
For additional commands, e-mail: jdnc-help@jdnc.dev.java.net

Patrick Wright

Actually, now I am wondering what exactly we were planning to do
here--the thread lost me. Looks like we want to generate XML
somewhere, on some queue, listing only those rows that have changed.
So--

- on what event--do we treat this as a row-by-row save, or as an export?
- what schema are we using for the XML? WebRowSet?

Let me know!
Patrick

---------------------------------------------------------------------
To unsubscribe, e-mail: jdnc-unsubscribe@jdnc.dev.java.net
For additional commands, e-mail: jdnc-help@jdnc.dev.java.net

evickroy
Offline
Joined: 2004-07-23

> Actually, now I am wondering what exactly we were
> planning to do
> here--the thread lost me. Looks like we want to
> generate XML
> somewhere, on some queue, listing only those rows
> that have changed.
> So--

I believe we are basically trying to serialize the entire DataSet, but only with rows that are updateable. (i.e. Inserted, Deleted, or Updated).
The client application can then transport the XML to a Servlet that can then deserialize the DataSet and apply the modifications to the database.

>
> - on what event--do we treat this as a row-by-row
> save, or as an export?
> - what schema are we using for the XML? WebRowSet?
>

In the RowSet implementations from Sun, you can customize the reader and writer for the RowSets. I was picturing the same functionality on a DataSet. A customized writer could filter out unmodified rows.

I'm not that familiar with the DataSet schema, but the WebRowSet schema contains both the datamodel metadata and the data in the same file. For a DataSet, it seems that we have to send the data and the dataset metadata in order to recreate a DataSet on the other side, since these are in two separate files now. Does that sound correct?

Patrick Wright

> I believe we are basically trying to serialize the entire DataSet, but only with rows that are updateable. (i.e. Inserted, Deleted, or Updated).
> The client application can then transport the XML to a Servlet that can then deserialize the DataSet and apply the modifications to the database.

Got it.

> In the RowSet implementations from Sun, you can customize the reader and writer for the RowSets. I was picturing the same functionality on a DataSet. A customized writer could filter out unmodified rows.

In the DataSet classes, this would be handled by a DataProvider. For
example, the SQLDataProvider currently skips updates for columns that
have not changed, and for rows that have not changed; seems to make
sense to leave this there.

>
> I'm not that familiar with the DataSet schema, but the WebRowSet schema contains both the datamodel metadata and the data in the same file. For a DataSet, it seems that we have to send the data and the dataset metadata in order to recreate a DataSet on the other side, since these are in two separate files now. Does that sound correct?

My own feeling is that the current DataSet schema is a place-holder.
DataSet should probably support more than one schema anyway, because
AFAIK, different database vendors have not come together on a database
schemas in XML. So--as far as this new feature goes, I think we don't
need to rely on the DS schema.

I would think that, for these purposes, we should stick to the WRS
schema, as this use case fits roughly within the goals of that model.
What do you think?

There are a couple of different issues here that have not been tackled
properly, which we need to address in this case.

- are we looking for write-only support for WRS, or read-write?
- what are the semantics of the exchange?

In other words--we could write a WebRowSetDataProvider that both knows
how to read and write WRS XML, as well as how to interact with a
servlet. In theory, the DataSet API is designed for this. We have a
communication/transport pipe (the DataConnection) and a way to
read/write using that pipe (DataProvider). The DataCommand would in
theory be useful for building the smaller sections of XML used in the
transactions.

As far as I can see, the major issue won't be with the XML, except
that we need to know if we can include multiple rows in one document,
multiple tables, etc. The real issue is with the communication
protocol. How do we execute against this remote service? What is that
API like? Are we using REST, SOAP, RPC?

Let's thrash this out (or just clarify it for me) so we can figure out
what changes are needed to support this.

Regards
Patrick

---------------------------------------------------------------------
To unsubscribe, e-mail: jdnc-unsubscribe@jdnc.dev.java.net
For additional commands, e-mail: jdnc-help@jdnc.dev.java.net

evickroy
Offline
Joined: 2004-07-23

I'll have to think about this and respond tonight. It seems like there might be some issues with translating a DataSet to a WebRowSet.

Patrick Wright

> > something like that? It just doesn't seem logical to
> > always pass the entire data set around if you are
> > just trying to apply changes to the database remotely.
> If you have any time to hack up an implementation, let me know. In fact, Patrick, if you want you could take this feature and figure out what needs to be done. That'd be great.

Yeah, no problem, have wanted this one for awhile. Let me see what I can do.

Patrick

---------------------------------------------------------------------
To unsubscribe, e-mail: jdnc-unsubscribe@jdnc.dev.java.net
For additional commands, e-mail: jdnc-help@jdnc.dev.java.net

Patrick Wright

Hey Richard

Thanks for the notes

> 1) The DataTable contains a DataProvider property. This is done so that you can call "ds.load()" and it will be able to know what DataProvider goes with each DataTable so their data can be loaded*
>
> 2) So that in the GUI tool (ie: the netbeans module) the user can specify what SQL query is required to populate a given DataTable. Since the XSD schema is the only file containing this configuration information, the configuration is saved with the file.
>

Yeah, I "got" this sometime this morning. There are two different
scenarios, which leads to my confusion here. In one (reasonable)
scenario you have a schema of a database (complete or partial), which
you then load into a DataSet, bind the DataTables, and attach
providers. The provider in that case is a link to the back-end, but
determined at runtime.

The other scenario is what came up a couple of weeks ago, when someone
asked how you could write a markup file, with provider, for DataSet.
Because reasonably, DataSet should be fully persistable in different
forms.

I like the idea that a DataSet can be defined programmatically, or
using one or more XML schemas (e.g. different document formats), or
even using the back end of a DDL parser (that would be cool--there are
Antlr grammars for some databases). That would be database-schema
oriented.

I also think that provider should be persistable--so I guess I'd punt
and make DataProvider an optional attribute. If it's there, the DS is
ready to use after unmarshalling, and if it's not there, you have to
specify it programatically.

I'd like to move the schema-parsing out to a separate set of classes
so we can have different formats to work with.

> * There has been a thought that maybe since the DataSet is really all about Relational data, that we should put the SQL statements for each DataTable directly on the DataTable and ditch the DataProvider heirarchy.
>
> There are essentially two different kinds of operations on a DataTable -> loading/dumping data and synchronizing with the database. The work you've done in the incubator is about loading/dumping data. The "loadXml" method on DataSet is about loading/dumping data. However, the DataProvider stuff was really about synchronizing with a data store.
>
> If we admit that the only data store we are likely to synchronize with is a relational database (or, at least a data store that supports JDBC since in reality non SQL data stores can support JDBC), then we can burn JDBC like constructs directly on the DataTable and avoid all this DataProvider fuss.
>
> This in now way means we couldn't load a DataTable from a flat file, XML data, etc. That could all happen in a more disconnected fashion using DataSetUtils or something like that. Feed it a flat file and some parsing instructions and blam -- it sucks it into a DataTable.
>
> Its a thought in progress, what do you think?

This is agonizing...

I like the DP system. It does seem some people are having trouble with
it--but it's a new API and needs to be massaged a little. But, as it
stands, we could pretty easily plug in support for legacy databases
that are file/record/field oriented using the current system. I mean,
that is such a stereotypical data structure it's not funny. There are
people who have bridges to non-relational stores, like XBase, and
there's no reason we couldn't cover legacy FileMaker, B-trieve,
FoxPro, who knows. This is not critical, but I think it's extremely
useful. One of Java's strength is this sort of multi-format
capability, as with file encodings.

We should also be able to use a DP that references either a SOAP or a
REST source, converting the incoming XML into a tabular structure (or
interrelated tabular structures), and spitting out RPC calls on the
back end.

Having a tabular data structure that can track changes at the column
level, allows for filtering and sorting, hierarchical relationships
between tables--this is very valuable. How the data gets in or out
seems like it just has to be separate.

That said--I'm not that clear headed today, so don't feel I can get my
points across. I'd vote to keep the DP layer. However--let's say
this--if by the middle of the 0.8 to 0.9 cycle there are no additional
standard DPs rolling into place, maybe then look at pulling it out.

Patrick

---------------------------------------------------------------------
To unsubscribe, e-mail: jdnc-unsubscribe@jdnc.dev.java.net
For additional commands, e-mail: jdnc-help@jdnc.dev.java.net

rbair
Offline
Joined: 2003-07-08

Hey Patrick,

> little. But, as it
> stands, we could pretty easily plug in support for
> legacy databases
> that are file/record/field oriented using the current
> system. I mean,
> that is such a stereotypical data structure it's not
> funny. There are
> people who have bridges to non-relational stores,
> like XBase, and
> there's no reason we couldn't cover legacy FileMaker,
> B-trieve,
> FoxPro, who knows. This is not critical, but I think
> it's extremely
> useful. One of Java's strength is this sort of
> multi-format
> capability, as with file encodings.

Excellent feedback, thanks! Personally I think being able to hook into FileMaker/Btrieve etc is of enormous importance. After all, we're trying to move into a market dominated by those types of legacy database systems. The only question I have is: can those be accessed via JDBC? FoxPro can be, I'm sure, through a JDBC-ODBC bridge, and I assume the others could to, but I don't know.

Richard

Patrick Wright

>Personally I think being able to hook into FileMaker/Btrieve etc is
of enormous importance. After all, we're trying to move into a market
dominated by those types of legacy database systems. The only question
I have is: can those be accessed via JDBC? FoxPro can be, I'm sure,
through a JDBC-ODBC bridge, and I assume the others could to, but I
don't know.

Seems like we'll have to put this question out there. I seem to recall
that a few years back all these data providers were writing ODBC
handlers. The JDBC-ODBC bridge is not a great solution (I seem to
remember it as slow and not 100% reliable, but that was a few years
back), but certainly easier to code with.

Don't have an answer, sorry.
patrick

---------------------------------------------------------------------
To unsubscribe, e-mail: jdnc-unsubscribe@jdnc.dev.java.net
For additional commands, e-mail: jdnc-help@jdnc.dev.java.net

rbair
Offline
Joined: 2003-07-08

Hey Patrick,

> This is looking up a class to act as a DataProvider,
> but there is no
> such element in the XSD
> (complexType/sequence/appinfo/dataProvider).
> Somehow this doesn't make sense--why would a schema
> define a
> DataProvider? The schema is structural, the provider
> manages data
> flow.

There are several reasons for doing this:

1) The DataTable contains a DataProvider property. This is done so that you can call "ds.load()" and it will be able to know what DataProvider goes with each DataTable so their data can be loaded*

2) So that in the GUI tool (ie: the netbeans module) the user can specify what SQL query is required to populate a given DataTable. Since the XSD schema is the only file containing this configuration information, the configuration is saved with the file.

>
> Anyway, looks like this was added recently in the
> DataSet (1.3), just
> wondering if there is a matching XSD for the
> adventure demo.

I'm working on that right now. Hopefully by the end of the day I'll have a proper schema in the demo so the demo runs again.

Richard

* There has been a thought that maybe since the DataSet is really all about Relational data, that we should put the SQL statements for each DataTable directly on the DataTable and ditch the DataProvider heirarchy.

There are essentially two different kinds of operations on a DataTable -> loading/dumping data and synchronizing with the database. The work you've done in the incubator is about loading/dumping data. The "loadXml" method on DataSet is about loading/dumping data. However, the DataProvider stuff was really about synchronizing with a data store.

If we admit that the only data store we are likely to synchronize with is a relational database (or, at least a data store that supports JDBC since in reality non SQL data stores can support JDBC), then we can burn JDBC like constructs directly on the DataTable and avoid all this DataProvider fuss.

This in now way means we couldn't load a DataTable from a flat file, XML data, etc. That could all happen in a more disconnected fashion using DataSetUtils or something like that. Feed it a flat file and some parsing instructions and blam -- it sucks it into a DataTable.

Its a thought in progress, what do you think?

evickroy
Offline
Joined: 2004-07-23

Would this not be a good place for JDBC RowSets? The RowSet s support connected and disconnected functionality. WebRowSets can also be dumped/loaded as XML. They also separate the readers and writers so you can plugin versions for reading and writing to flat files, for instance. I don't have a lot of experience with them, but they seem very similar to the DataSet/DataProvider concepts.

Here is a link: http://java.sun.com/developer/Books/JDBCTutorial/chapter5.html

rbair
Offline
Joined: 2003-07-08

> Would this not be a good place for JDBC RowSets? The
> RowSet s support connected and disconnected
> functionality. WebRowSets can also be dumped/loaded
> as XML. They also separate the readers and writers
> so you can plugin versions for reading and writing to
> flat files, for instance. I don't have a lot of
> experience with them, but they seem very similar to
> the DataSet/DataProvider concepts.

Possibly, but RowSets were optimized (it appears) for the Servlet/web scenario and not for clients. For example, there is no efficient random access into the RowSet, which absolutely kills it DOA for client applications (consider trying to paint halfway through a JList -- all random access calls into the ListModel which would then have to move the cursor on the RowSet causing events to fire....).

RowSet is tantalizing though because it contains parmeterized queries, synchronization code, etc. Its not exactly clear what should be done since I really don't want to write my own synchronization code.

Richard

evickroy
Offline
Joined: 2004-07-23

>
> Possibly, but RowSets were optimized (it appears) for
> the Servlet/web scenario and not for clients. For
> example, there is no efficient random access into the
> RowSet, which absolutely kills it DOA for client
> applications (consider trying to paint halfway
> through a JList -- all random access calls into the
> ListModel which would then have to move the cursor on
> the RowSet causing events to fire....).
>
> RowSet is tantalizing though because it contains
> parmeterized queries, synchronization code, etc. Its
> not exactly clear what should be done since I really
> don't want to write my own synchronization code.
>
> Richard

I'm not familiar with the problem scenario that you describe, but I do know that the RowSet gives you full capabilities of a Scrollable result set, so you can traverse forwards, backwards, relative to the current position, and to an absolute index. RowSets also have paging capabilities, so you can specify a page size, although, I'm unsure how that would work in a disconnected scenario. Is the problem that you describe not supported by these features?

Also, we were looking at the WebRowSet originally so that we could have a Swing rich client using a WebRowSet that is brokered through a simple servlet on the web server in situations where 2-tier connections isn't appropriate. From what I have read so far, I believe this can be done with a DataSet as well... hopefully.

Thanks.

rbair
Offline
Joined: 2003-07-08

> Also, we were looking at the WebRowSet originally so
> that we could have a Swing rich client using a
> WebRowSet that is brokered through a simple servlet
> on the web server in situations where 2-tier
> connections isn't appropriate. From what I have read
> so far, I believe this can be done with a DataSet as
> well... hopefully.

Yes, it should be able to do that rather well, however it is of course new technology and it will take some months to make sure all of the wrinkles are out.

BTW, internally Bino has written a bug viewer that parses WebRowSet XML and shovels the data into a DataSet.

Richard

evickroy
Offline
Joined: 2004-07-23

>
> Yes, it should be able to do that rather well,
> however it is of course new technology and it will
> take some months to make sure all of the wrinkles are
> out.
>
> BTW, internally Bino has written a bug viewer that
> parses WebRowSet XML and shovels the data into a
> DataSet.
>
>
> Richard

Glad to hear it. I do rather like the way the WebRowSet indicates the row's status in the element name. It would make sense for the writer process to be able to ONLY write out rows that have changed. That would make the data set being passed back for updating much smaller. Is the DataSet capable of something like that? It just doesn't seem logical to always pass the entire data set around if you are just trying to apply changes to the database remotely.

rbair
Offline
Joined: 2003-07-08

Hi

> Glad to hear it. I do rather like the way the
> WebRowSet indicates the row's status in the element
> name. It would make sense for the writer process to
> be able to ONLY write out rows that have changed.
> That would make the data set being passed back for
> r updating much smaller. Is the DataSet capable of
> something like that? It just doesn't seem logical to
> always pass the entire data set around if you are
> just trying to apply changes to the database remotely.

That's the idea. However, the infrastructure is in place, but the code to actually return only the changed rows, and then to generate XML based on those rows, has not yet been written. It should be relatively straightforward, though. The DataRow contains a bunch of DataCells (which are private to the DataRow). Each DataCell contains the original value for the cell as well as the new value. So it should be pretty easy to determine if a DataRow has changed or not, and then to get back the DataRows that have changed would be a single method added to DataTable.

If you have any time to hack up an implementation, let me know. In fact, Patrick, if you want you could take this feature and figure out what needs to be done. That'd be great.

Thanks
Richard

evickroy
Offline
Joined: 2004-07-23

> If you have any time to hack up an implementation,
> let me know. In fact, Patrick, if you want you could
> take this feature and figure out what needs to be
> done. That'd be great.
>
> Thanks
> Richard

I don't have a lot of experience with parsing XML, but I'm willing to give it a shot if you guys don't mind helping me figure out what needs to be done.

rbair
Offline
Joined: 2003-07-08

> I don't have a lot of experience with parsing XML,
> but I'm willing to give it a shot if you guys don't
> mind helping me figure out what needs to be done.

Ya, you bet. The preferred method would be to just stay in touch on the forums so you and Patrick can coordinate, and the rest of us can chime in from the peanut gallery :)

Richard