Discussion:
[Synce-devel] New WM5 synce-plugin for OpenSync >0.30 and OpenSync SVN
Dr J A Gow
2007-10-18 22:40:33 UTC
Permalink
Hello,

For all who like to live on the fast lane at the cutting edge, I have
committed an additional plugin module to sync-engine for OpenSync
=0.30. It's still experimental but I have had a good two-way sync with
the file-sync OpenSync plugin (I believe the evo2 plugin in OpenSync SVN
is broken and is being looked at by the OpenSync people, so this won't
work yet).

I can only test on a 64 bit system at the moment so the results of any
tests on 32 bit systems would help.

There's more details in the readme for the new plugin in the sync-engine
tree.

John.
Jonny Lamb
2007-10-26 13:05:42 UTC
Permalink
Post by Dr J A Gow
For all who like to live on the fast lane at the cutting edge, I have
committed an additional plugin module to sync-engine for OpenSync
=0.30. It's still experimental but I have had a good two-way sync with
the file-sync OpenSync plugin (I believe the evo2 plugin in OpenSync SVN
is broken and is being looked at by the OpenSync people, so this won't
work yet).
Despite the delay of my response, this is great news. Nice one, and nice
to see you around here again!

I spoke to Daniel Gollub from the OpenSync project, and he thought it
would be a good idea to have a 0.30-compatible SynCE-plugin in their
SVN. Thoughts? I could approach him on this topic and get you access if
you like? I'm not exactly sure about how dgollub was thinking this
should work..

Anyway, nice one! I have had next-to-no time recently, but I'd like to
check this out especially now that I have two spare WM5 devices lying
around I can mess with, and potentially brick.

Kind regards,
--
Jonny Lamb, UK ***@jonnylamb.com
http://jonnylamb.com GPG: 0x2E039402
John Carr
2007-10-26 13:21:55 UTC
Permalink
Lo all
Post by Jonny Lamb
Post by Dr J A Gow
For all who like to live on the fast lane at the cutting edge, I have
committed an additional plugin module to sync-engine for OpenSync
=0.30. It's still experimental but I have had a good two-way sync with
the file-sync OpenSync plugin (I believe the evo2 plugin in OpenSync SVN
is broken and is being looked at by the OpenSync people, so this won't
work yet).
Despite the delay of my response, this is great news. Nice one, and nice
to see you around here again!
The 0.3x plugin is great for me too - should be a doddle for me to get
Conduit support going :-)
Post by Jonny Lamb
I spoke to Daniel Gollub from the OpenSync project, and he thought it
would be a good idea to have a 0.30-compatible SynCE-plugin in their
SVN. Thoughts? I could approach him on this topic and get you access if
you like? I'm not exactly sure about how dgollub was thinking this
should work..
I am of the belief that the whole of SynCE stuff should be together.
Theres already enough confusion about the version of synce that is in
opensync svn.

John
Dr J A Gow
2007-10-26 18:23:21 UTC
Permalink
Post by John Carr
Lo all
Post by Jonny Lamb
Post by Dr J A Gow
For all who like to live on the fast lane at the cutting edge, I have
committed an additional plugin module to sync-engine for OpenSync
=0.30. It's still experimental but I have had a good two-way sync with
the file-sync OpenSync plugin (I believe the evo2 plugin in OpenSync SVN
is broken and is being looked at by the OpenSync people, so this won't
work yet).
Despite the delay of my response, this is great news. Nice one, and nice
to see you around here again!
The 0.3x plugin is great for me too - should be a doddle for me to get
Conduit support going :-)
Post by Jonny Lamb
I spoke to Daniel Gollub from the OpenSync project, and he thought it
would be a good idea to have a 0.30-compatible SynCE-plugin in their
SVN. Thoughts? I could approach him on this topic and get you access if
you like? I'm not exactly sure about how dgollub was thinking this
should work..
I am of the belief that the whole of SynCE stuff should be together.
Theres already enough confusion about the version of synce that is in
opensync svn.
John
Regarding a SynCE plugin in OpenSync SVN - on the face of it it seems
like a good idea - but where do we draw the line between OpenSync and
SynCE?

What I mean is this: the plugin itself is just a single file containing
some Python. This is of no use on its own, but depends on sync-engine
for functionality. Would we put just the Python script in OpenSync SVN
and leave sync-engine in SynCE SVN? This would be one way forward but
would require a comprehensive README in with the Python script in
OpenSync SVN to avoid confusing anyone who needs WM syncing and comes at
it from the OpenSync site. We should certainly leave sync-engine in
SynCE SVN.

I wouldn't object to splitting it this way, but I tend to lean towards
agreeing with John on this one: that it would cause less confusion if we
keep SynCE stuff together. But if we could get a link on the OpenSync
site to our wiki and code, this may be a way to keep the OpenSync people
informed.

However:

Once the 0.3x plugin is working fully, one thing I am investigating in
the light of the changes in OpenSync's architecture is the way in which
sync-engine does its format conversions. For those who do not know,
OpenSync 0.3 spins off its format conversions into plugins. Currently I
am conducting a feasibility study in developing an OpenSync format
plugin with our AirSync<->XML format conversions in it, and subsequently
removing (or disabling) the format conversion code in sync-engine. This
would have a number of benefits: it would unburden sync-engine from
format conversions and allow it to speak native AirSync to the OpenSync
interface. Format conversions to OpenSync XML formats would then be
handled by the format plugin. This is more in keeping with the OpenSync
0.3x paradigm.

Such a format plugin should arguably be better residing in OpenSync SVN
as it would then expose the format to anything else out there that cares
to speak AirSync.

What do you all think about this approach?


Regarding the status of the OpenSync 0.30 plugin in SVN at the moment,
the actual sync process itself is flawless but the format conversions
are broken (i.e. the sync will happen but content will disappear from
synced items). OpenSync have changed the XML schemas considerably in
0.3x over 0.2x (and for the better, IMHO) . I am working on the format
conversions as we speak, and in my development tree I can now
synchronize event content cleanly with Evolution using OpenSync SVN.
I'll merge this new code with SVN as soon as I can find a way to do so
without breaking the existing 0.2x support. We'll have contact and task
support in the 0.3x plugin very shortly but I have some work to do to
implement the timezone conversions - OpenSync 0.3x has much better
timezone support than in the 0.2x schemas.

I'll keep you posted.

John.
r***@velvetsea.net
2007-10-26 20:29:58 UTC
Permalink
Post by Dr J A Gow
Once the 0.3x plugin is working fully, one thing I am investigating in
the light of the changes in OpenSync's architecture is the way in which
sync-engine does its format conversions. For those who do not know,
OpenSync 0.3 spins off its format conversions into plugins. Currently I
am conducting a feasibility study in developing an OpenSync format
plugin with our AirSync<->XML format conversions in it, and subsequently
removing (or disabling) the format conversion code in sync-engine. This
would have a number of benefits: it would unburden sync-engine from
format conversions and allow it to speak native AirSync to the OpenSync
interface. Format conversions to OpenSync XML formats would then be
handled by the format plugin. This is more in keeping with the OpenSync
0.3x paradigm.
Such a format plugin should arguably be better residing in OpenSync SVN
as it would then expose the format to anything else out there that cares
to speak AirSync.
What do you all think about this approach?
I would very much like that :) Another issue with that approach, though,
is what to do with the libwbxml stuff. I somehow doubt that OpenSync
would want to introduce that dependency for their entire tree, so we'd
either:

1) Implement our own wbXml stuff (perhaps borrow the required pieces from
libwbxml with appropriate credit given to the author - but we have to
double-check the licensing details of this)

2) Ensure that OpenSync can have only a single format conversion plugin
have a dependency on libwbxml without requring that dependency in their
entire tree.


John and Mark: I'd like to echo everyone's thanks to you guys for
continuing to actively participate here. I would love to get back to it
(especially to implement that default timezone capability needed for
KDE-Pim) but unfortunately my schedule now and in the forseeable future
doesn't really permit it. Thanks again, though - you guys are great!

--
Richard Alimi
Department of Computer Science
Yale University
Dr J A Gow
2007-10-26 22:35:18 UTC
Permalink
Post by r***@velvetsea.net
Post by Dr J A Gow
Once the 0.3x plugin is working fully, one thing I am investigating in
the light of the changes in OpenSync's architecture is the way in which
sync-engine does its format conversions. For those who do not know,
OpenSync 0.3 spins off its format conversions into plugins. Currently I
am conducting a feasibility study in developing an OpenSync format
plugin with our AirSync<->XML format conversions in it, and subsequently
removing (or disabling) the format conversion code in sync-engine. This
would have a number of benefits: it would unburden sync-engine from
format conversions and allow it to speak native AirSync to the OpenSync
interface. Format conversions to OpenSync XML formats would then be
handled by the format plugin. This is more in keeping with the OpenSync
0.3x paradigm.
Such a format plugin should arguably be better residing in OpenSync SVN
as it would then expose the format to anything else out there that cares
to speak AirSync.
What do you all think about this approach?
I would very much like that :) Another issue with that approach, though,
is what to do with the libwbxml stuff. I somehow doubt that OpenSync
would want to introduce that dependency for their entire tree, so we'd
1) Implement our own wbXml stuff (perhaps borrow the required pieces from
libwbxml with appropriate credit given to the author - but we have to
double-check the licensing details of this)
I was thinking of leaving the wbxml stuff on the sync-engine side and
passing Airsync/XML to OpenSync - this way we keep binary formats such
as wbxml (specific to handhelds and similar) out of OpenSync. The
conversion between wbxml and xml is direct, whereas the conversion
between AirSync/XML and OpenSync XML requires more of a translation of
both elements and content.

Can we not get our wbxml patches accepted upstream so, from the next
major release of wbxml, we can just have the user pull in their
distribution's wbxml library?
Post by r***@velvetsea.net
2) Ensure that OpenSync can have only a single format conversion plugin
have a dependency on libwbxml without requring that dependency in their
entire tree.
We could do this, but I would be keen on keeping binary formats out of
the OpenSync item sync pathway if for no other reasons than the inherent
'open-ness' of XML and not making the OpenSync components (including
format plugins) require a dependence on a third-party conversion
library.
Post by r***@velvetsea.net
John and Mark: I'd like to echo everyone's thanks to you guys for
continuing to actively participate here. I would love to get back to it
(especially to implement that default timezone capability needed for
KDE-Pim) but unfortunately my schedule now and in the forseeable future
doesn't really permit it. Thanks again, though - you guys are great!
No problem - this is an interesting project to work on!

I perhaps should have posted earlier about that default timezone issue.
A while ago I generated some Python code that will extract the system
timezone from the binary Olson files and from this construct a ruleset
for use with the OpenSync XML (will need to update this for the new
OpenSync 0.3 schemas). However it was here I ran into an anomaly.
Consider a timezone such as UK/London. Capturing some of the XML coming
in from OpenSync showed that there were two clear rules, one for DST and
one for STD time giving the appearance that these were the only two
rules that had ever existed. Each section, (DST or STD) had a single
recurrence rule attached to it.

Now the Olson binary file actually contains a lot more information -
when I examined the output from the program I wrote there were actually
a number of differing rules for the start/stop of DST at different
points throughout history - i.e. more than just the two that seemed to
be provided with a typical OpenSync item. To correctly convert these
rules would require more than two simple rulesets governing DST/STD
transitions.

Not only that, but the rules directly obtained from the Olson database
conflicted with the rules provided with the typical item. Examination of
the binary timezone file confirmed that the output from my program was
correct - and that the simple two-rule output from OpenSync did not
match.

So I compared the two with the output from the 'vzic' program - a
utility that supposedly will generate iCal timezones from Olson files.
Both OpenSync and the vzic program gave a simple two-rule output which
did not match the content of the Olson binary rules!

Some basic searching shed no light on this, and without being able to
resolve this issue I did not fold the code into sync-engine at this
point.

I then had to turn attention to other matters.

However, the OpenSync 0.3x timezone rules seem to be a lot more
comprehensive and on the surface a lot closer to the Olson rules. I am
going to need to modify the timezone objects in sync-engine to handle
multiple recurrence rules in order to handle OpenSync 0.3x, and I will
hopefully be able to slip the default timezone generation code in at
this point with very little difficulty. So watch the 0.3x code very
closely......

John.
Richard Alimi
2007-11-02 06:55:57 UTC
Permalink
Post by Dr J A Gow
Can we not get our wbxml patches accepted upstream so, from the next
major release of wbxml, we can just have the user pull in their
distribution's wbxml library?
I had sent the patches more than a few months ago to the libwbxml maintainer
and he acknowledged that he had received them. I still don't see any updates
to his SVN respository, though. Given that John Carr is working on a wbxml
<-> XML converter in Python, that might be the best way to go.
Post by Dr J A Gow
Post by r***@velvetsea.net
2) Ensure that OpenSync can have only a single format conversion plugin
have a dependency on libwbxml without requring that dependency in their
entire tree.
We could do this, but I would be keen on keeping binary formats out of
the OpenSync item sync pathway if for no other reasons than the inherent
'open-ness' of XML and not making the OpenSync components (including
format plugins) require a dependence on a third-party conversion
library.
Makes sense to me. I agree with you that wbxml stuff should remain in
sync-engine.
Post by Dr J A Gow
I perhaps should have posted earlier about that default timezone issue.
A while ago I generated some Python code that will extract the system
timezone from the binary Olson files and from this construct a ruleset
for use with the OpenSync XML (will need to update this for the new
OpenSync 0.3 schemas). However it was here I ran into an anomaly.
Consider a timezone such as UK/London. Capturing some of the XML coming
in from OpenSync showed that there were two clear rules, one for DST and
one for STD time giving the appearance that these were the only two
rules that had ever existed. Each section, (DST or STD) had a single
recurrence rule attached to it.
... snip ...
However, the OpenSync 0.3x timezone rules seem to be a lot more
comprehensive and on the surface a lot closer to the Olson rules. I am
going to need to modify the timezone objects in sync-engine to handle
multiple recurrence rules in order to handle OpenSync 0.3x, and I will
hopefully be able to slip the default timezone generation code in at
this point with very little difficulty. So watch the 0.3x code very
closely......
That's great that you were able to trace through all of this. I'm also happy
to see that (at least it appears) that they ahve been making some significant
improvements on the OpenSync side :)
--
Richard Alimi
Department of Computer Science
Yale University
John Carr
2007-11-02 07:35:35 UTC
Permalink
Post by Richard Alimi
Post by Dr J A Gow
Can we not get our wbxml patches accepted upstream so, from the next
major release of wbxml, we can just have the user pull in their
distribution's wbxml library?
I had sent the patches more than a few months ago to the libwbxml maintainer
and he acknowledged that he had received them. I still don't see any updates
to his SVN respository, though. Given that John Carr is working on a wbxml
<-> XML converter in Python, that might be the best way to go.
The converter has been sidelined a little by my main commitment to
Conduit and #synce support. I'll probably throw the bzr repo online at
some point this week, but i'm somewhat nervous of that till i've removed
some of the evil hacks from it...

If someone could post me so wbxml dumps (and associated xml for
comparison) that would be awesome - i don't have a working synce to hand
right now :(

John

John Carr
2007-10-27 04:41:57 UTC
Permalink
Post by r***@velvetsea.net
I would very much like that :) Another issue with that approach, though,
is what to do with the libwbxml stuff. I somehow doubt that OpenSync
would want to introduce that dependency for their entire tree, so we'd
1) Implement our own wbXml stuff (perhaps borrow the required pieces from
libwbxml with appropriate credit given to the author - but we have to
double-check the licensing details of this)
I've implemented a Reader and Writer class (a bit like SAX i guess) and
utility functions to map it to/from XML for WBXML that i'm hoping to
finish debugging this weekend. It's based on the spec on w3.org and some
GPL PHP code from z-push. It's pure python and means we can ship it
inside SyncEngine and avoid that dep entirely.

John
r***@velvetsea.net
2007-10-27 14:14:01 UTC
Permalink
That is great to hear!

--
Richard Alimi
Department of Computer Science
Yale University
Post by John Carr
Post by r***@velvetsea.net
I would very much like that :) Another issue with that approach, though,
is what to do with the libwbxml stuff. I somehow doubt that OpenSync
would want to introduce that dependency for their entire tree, so we'd
1) Implement our own wbXml stuff (perhaps borrow the required pieces from
libwbxml with appropriate credit given to the author - but we have to
double-check the licensing details of this)
I've implemented a Reader and Writer class (a bit like SAX i guess) and
utility functions to map it to/from XML for WBXML that i'm hoping to
finish debugging this weekend. It's based on the spec on w3.org and some
GPL PHP code from z-push. It's pure python and means we can ship it
inside SyncEngine and avoid that dep entirely.
John
David Eriksson
2007-10-26 20:35:50 UTC
Permalink
Post by Dr J A Gow
Regarding a SynCE plugin in OpenSync SVN - on the face of it it seems
like a good idea - but where do we draw the line between OpenSync and
SynCE?
Format conversion plugins is a really good idea, but with or without it
I would suggest keeping the SynCE stuff together... unless there are
OpenSync people who would help maintain the code! :-)


Best regards,

David
Loading...