libwbxml

[Iain Buchanan]

Hi all,

some questions about libwbxml: Is upstream dead? (0.10.0 is 1 year
late! http://libwbxml.aymerick.com/roadmap) It would make my life easier
if he produced a new release, but I understand he hasn't even got our
patches into his svn yet. I know someone sent him the patches (can't
remember who) but they don't seem to have gone anywhere. It's not even
listed in Trac (http://libwbxml.aymerick.com/report/1) afaict.

For SynCE to ever be marked stable in Gentoo, it can't depend on an svn
snapshot of libwbxml. Marking it stable won't be an issue straight
away, but I'd like to see it done eventually.

So, can we either:

1. provide patches against latest libwbxml release (0.9.2) and not
svn version?
2. fork libwbxml and maintain it ourselves :)
3. put the pressure on upstream again, to see if we can get a new
release (or any comment)? At the very least, a release of the
current svn snapshot would be fine, if option 1. isn't possible.

What do you guys think?

thanks,

[David Eriksson]

Post by Iain Buchanan
Hi all,
some questions about libwbxml: Is upstream dead? (0.10.0 is 1 year
late! http://libwbxml.aymerick.com/roadmap) It would make my life easier
if he produced a new release, but I understand he hasn't even got our
patches into his svn yet. I know someone sent him the patches (can't
remember who) but they don't seem to have gone anywhere. It's not even
listed in Trac (http://libwbxml.aymerick.com/report/1) afaict.
For SynCE to ever be marked stable in Gentoo, it can't depend on an svn
snapshot of libwbxml. Marking it stable won't be an issue straight
away, but I'd like to see it done eventually.
1. provide patches against latest libwbxml release (0.9.2) and not
svn version?
2. fork libwbxml and maintain it ourselves :)
3. put the pressure on upstream again, to see if we can get a new
release (or any comment)? At the very least, a release of the
current svn snapshot would be fine, if option 1. isn't possible.
What do you guys think?

Pressure is not very good for motivating Open Source developers :-)

Maybe you could check what other projects depend on libwbxml and see if we
could work on a common cause?

A couple of other options:

4. Ask if some of us could join the project

5. Ask if the current maintainer is willing to give away the project
to us

6. Get rid of libwbxml as a dependency. For my indefinitely
postponed MidaSync project I wrote the WBXML stuff needed for
AirSync myself. It's less than 1500 lines of code including
conversion between XML and WBXML, and I got it in C++ like I wanted.
On the other hand, now it depends on libxml++ and Boost! ;-) Drop me
a mail if you want to have a look at it anyway.


Regards,

David

[John Carr]

Post by David Eriksson
Maybe you could check what other projects depend on libwbxml and see if we
could work on a common cause?

OpenSync use it for their libsyncml i believe.

Post by David Eriksson
6. Get rid of libwbxml as a dependency. For my indefinitely
postponed MidaSync project I wrote the WBXML stuff needed for
AirSync myself. It's less than 1500 lines of code including
conversion between XML and WBXML, and I got it in C++ like I wanted.
On the other hand, now it depends on libxml++ and Boost! ;-) Drop me
a mail if you want to have a look at it anyway.

Both jonnylamb and I have both written our own wbxml implementations
in straight python. Again, very little code. After spending a week
looking for some personal stuff i've come to the conclusion that i've
deleted a backup and have probably lost the bzr rep i had my pywbxml
in :-( So jonnylamb, how about it :-)

John

[Jonny Lamb]

Post by John Carr
Both jonnylamb and I have both written our own wbxml implementations
in straight python. Again, very little code. After spending a week
looking for some personal stuff i've come to the conclusion that i've
deleted a backup and have probably lost the bzr rep i had my pywbxml
in :-( So jonnylamb, how about it :-)

http://compsoc.dur.ac.uk/~jdl/pywbxml/

I wrote this on a train going to London and have only partially got
around to testing it. Work on it would be lovely!

To test it, one can use the tests in Ole Andre's pywbxml:

http://synce.svn.sourceforge.net/viewvc/synce/trunk/pywbxml/tests/

Thanks,

[Dr J A Gow]

Post by Jonny Lamb
http://compsoc.dur.ac.uk/~jdl/pywbxml/
I wrote this on a train going to London and have only partially got
around to testing it. Work on it would be lovely!
http://synce.svn.sourceforge.net/viewvc/synce/trunk/pywbxml/tests/

I've taken this code, fixed it up to make it sync-engine compliant and
added what is needed to create XMLToWBXML() (and vice-versa) functions.
These directly create the libxml2 representations (no string conversion
stage). I can now go wbxml->xml correctly, and will soon be able to go
xml->wbxml - I just need to write a few lines. Works well with 'real'
test data generated from the phone.

Give me a couple of days and we'll have a libwbxml-free sync-engine and
pushmail server, assuming we still want to go this way.

Thanks for providing this Jonny. BTW where did you get the DTD from? It
differs in a few areas from the Airsync requirment and also from the
libwbxml implementation? Just curious...

John.

[Jonny Lamb]

Post by Dr J A Gow
I've taken this code, fixed it up to make it sync-engine compliant and
added what is needed to create XMLToWBXML() (and vice-versa) functions.
These directly create the libxml2 representations (no string conversion
stage). I can now go wbxml->xml correctly, and will soon be able to go
xml->wbxml - I just need to write a few lines. Works well with 'real'
test data generated from the phone.
Give me a couple of days and we'll have a libwbxml-free sync-engine and
pushmail server, assuming we still want to go this way.

I cannot actually explain how much you rock. This will seriously help
out with respect to packaging!

Post by Dr J A Gow
Thanks for providing this Jonny. BTW where did you get the DTD from? It
differs in a few areas from the Airsync requirment and also from the
libwbxml implementation? Just curious...

I got it from Z-Push[0].

I will watch the SVN logs for this. Awesome work!

Many thanks,

[0] http://z-push.sf.net/

[Jonny Lamb]

Post by Jonny Lamb
I cannot actually explain how much you rock. This will seriously help
out with respect to packaging!

...and with a part of the roadmap, of course:

http://www.synce.org/moin/SynceRoadmap

[Iain Buchanan]

Post by Jonny Lamb

Post by Dr J A Gow
I've taken this code, fixed it up to make it sync-engine compliant and
added what is needed to create XMLToWBXML() (and vice-versa) functions.
These directly create the libxml2 representations (no string conversion
stage). I can now go wbxml->xml correctly, and will soon be able to go
xml->wbxml - I just need to write a few lines. Works well with 'real'
test data generated from the phone.
Give me a couple of days and we'll have a libwbxml-free sync-engine and
pushmail server, assuming we still want to go this way.

wow you work fast.

Post by Jonny Lamb
I cannot actually explain how much you rock. This will seriously help
out with respect to packaging!

+1 to that.

Do you think it will be "good enough" to use when you 'release' in a few
days? I would like to forget the libwbxml path all together if I can...

thanks heaps,

[Jonny Lamb]

Post by Iain Buchanan
Do you think it will be "good enough" to use when you 'release' in a few
days? I would like to forget the libwbxml path all together if I can...

Mmm, I was thinking of an 0.11.1 release of SyncEngine. Perhaps a
"development" release.. Thoughts?

Thanks,

[Iain Buchanan]

Post by Jonny Lamb

Post by Iain Buchanan
Do you think it will be "good enough" to use when you 'release' in a few
days? I would like to forget the libwbxml path all together if I can...

Mmm, I was thinking of an 0.11.1 release of SyncEngine. Perhaps a
"development" release.. Thoughts?

sure, that would allow us to say version a. needs libwbxml, version b.
doesn't. Rather than saying "version a. plus these patches". It would
make distributions happier too.

[Dr J A Gow]

Post by Iain Buchanan
Do you think it will be "good enough" to use when you 'release' in a few
days? I would like to forget the libwbxml path all together if I can...

I don't see any reason why not. I still have to thoroughly check the DTD
and to get attribute processing to work properly (not needed for Airsync
but better to have it working).

So far I have grabbed a few complex events from the phone as wbxml -
then used the python code to convert to XML and the data looks perfect.
A comparison with the output from the libwbxml tools shows a perfect
match. It works perfectly in reverse as well. Apart from the above I
just need to clean it up, comment it, and insert it into the sync-engine
code.

Depending on how bad the laboratory scripts I am marking at the moment
are :-) , I should have it in SVN by the end of the week, or the weekend
at the latest.

John

[Jonny Lamb]

Post by David Eriksson
Pressure is not very good for motivating Open Source developers :-)

Agreed.

Post by David Eriksson
Maybe you could check what other projects depend on libwbxml and see if we
could work on a common cause?

John Carr mentioned one, but it's basically nothing. For example, in
Debian:

salieri:~% apt-cache rdepends libwbxml2-0
libwbxml2-0
Reverse Depends:
libwbxml2-utils
libwbxml2-dev
libwbxml2-0-dbg
libsyncml0
libsyncml-utils

Post by David Eriksson
4. Ask if some of us could join the project
5. Ask if the current maintainer is willing to give away the project
to us

These are both better than the previous suggested options, but the
following is much better!

Post by David Eriksson
6. Get rid of libwbxml as a dependency. For my indefinitely
postponed MidaSync project I wrote the WBXML stuff needed for
AirSync myself. It's less than 1500 lines of code including
conversion between XML and WBXML, and I got it in C++ like I wanted.
On the other hand, now it depends on libxml++ and Boost! ;-) Drop me
a mail if you want to have a look at it anyway.

Ha, MidaSync to the rescue again! Although that would be fairly cool,
not only has a native pywbxml been started, but SyncEngine would need
bindings to anything not in Python. I say that a native pywbxml is the
way forward as then it can be thrown in the SyncEngine source and all we
do is remove dependencies, not add them! \o/

Thanks,

[Dr J A Gow]

Post by Jonny Lamb
Ha, MidaSync to the rescue again! Although that would be fairly cool,
not only has a native pywbxml been started, but SyncEngine would need
bindings to anything not in Python. I say that a native pywbxml is the
way forward as then it can be thrown in the SyncEngine source and all we
do is remove dependencies, not add them! \o/
Thanks,

I like the idea of removing dependencies - however there is one issue
that is niggling in the back of my mind with a pure Python
implementation of wbxml. With a C/C++ implementation we can easily speak
wbxml from both native code and (through a thin layer) Python. With a
pure Python implementation we would have to rewrite in C/C++ if we want
to call from C++ (or have to call into Python which is a messy way to do
things).

So why may we want to go the other way? I am thinking of the pushmail
server. While sync-engine is fine as a pure Python implementation for
syncing, I am wondering about the scalability of a Python based pushmail
server if in an enterprise situation it is confronted with potentially
hundreds of concurrent connections. This may or may not be a problem, I
don't know at this stage but I would not want to make it more difficult
to develop something like libairsync.so should we need to.

John.

[Iain Buchanan]

Post by David Eriksson

Post by Iain Buchanan
1. provide patches against latest libwbxml release (0.9.2) and not
svn version?

is this ruled out then?

Post by David Eriksson

Post by Iain Buchanan
2. fork libwbxml and maintain it ourselves :)
3. put the pressure on upstream again, to see if we can get a new
release (or any comment)? At the very least, a release of the
current svn snapshot would be fine, if option 1. isn't possible.
What do you guys think?

Pressure is not very good for motivating Open Source developers :-)

OK, I didn't mean "cough up a release and we'll return your pet dog" I
meant "hey what's up?"!

Post by David Eriksson
Maybe you could check what other projects depend on libwbxml and see if we
could work on a common cause?
4. Ask if some of us could join the project
5. Ask if the current maintainer is willing to give away the project
to us

that's kind of what I was thinking. I don't always choose the right
words. So does someone want to talk to him on behalf of SynCE?

Post by David Eriksson
6. Get rid of libwbxml as a dependency.

sounds good too!

thanks, I see some more responses just came in, so I'll go and read
those too...

[Jonny Lamb]

Post by Iain Buchanan
some questions about libwbxml: Is upstream dead? (0.10.0 is 1 year
late! http://libwbxml.aymerick.com/roadmap) It would make my life easier
if he produced a new release, but I understand he hasn't even got our
patches into his svn yet. I know someone sent him the patches (can't
remember who) but they don't seem to have gone anywhere. It's not even
listed in Trac (http://libwbxml.aymerick.com/report/1) afaict.

Upstream does indeed feel dead. Richard Alimi sent the patches a while
ago with no response or acknowledgement, so Richard later sent them
again. This was responded to by a "Thanks, I'll look at it soon" email.
Nothing has happened after that.

Post by Iain Buchanan
For SynCE to ever be marked stable in Gentoo, it can't depend on an svn
snapshot of libwbxml. Marking it stable won't be an issue straight
away, but I'd like to see it done eventually.

To be honest, I'm more bothered about SynCE itself depending on such a
weird situation than any distribution having to deal it. After all,
Gentoo is not the only one having problems -- I'm going through the same
tiresome process with Debian.

Post by Iain Buchanan
1. provide patches against latest libwbxml release (0.9.2) and not
svn version?

I've already created this:

http://compsoc.dur.ac.uk/~jdl/wbxml2-0.9.2-patches/

These are basically copied from the 0.9.2 patches which are in SynCE
SVN:

http://synce.svn.sourceforge.net/viewvc/synce/trunk/patches/

Post by Iain Buchanan
2. fork libwbxml and maintain it ourselves :)

Go ahead, but not only do I plan to move away from libwbxml2[0] but I
doubt there'd be any volunteers to do so.

Post by Iain Buchanan
3. put the pressure on upstream again, to see if we can get a new
release (or any comment)? At the very least, a release of the
current svn snapshot would be fine, if option 1. isn't possible.

If he's busy, then he's busy. It's annoying, but this is the world of
free software. He's not obliged to release..

Thanks,

[0] http://www.synce.org/moin/SynceRoadmap

[Dr J A Gow]

Post by Jonny Lamb
http://compsoc.dur.ac.uk/~jdl/wbxml2-0.9.2-patches/
These are basically copied from the 0.9.2 patches which are in SynCE
http://synce.svn.sourceforge.net/viewvc/synce/trunk/patches/

I have not tested these, but had a look at the source. Nice! Would you
be happy for me to carry out some more tests, and if OK to build it into
sync-engine? Or is there more work that you need to carry out on this
first.

John

[Iain Buchanan]

Post by Jonny Lamb

Post by Iain Buchanan
some questions about libwbxml: Is upstream dead? (0.10.0 is 1 year
late! http://libwbxml.aymerick.com/roadmap) It would make my life easier
if he produced a new release, but I understand he hasn't even got our
patches into his svn yet. I know someone sent him the patches (can't
remember who) but they don't seem to have gone anywhere. It's not even
listed in Trac (http://libwbxml.aymerick.com/report/1) afaict.

Upstream does indeed feel dead. Richard Alimi sent the patches a while
ago with no response or acknowledgement, so Richard later sent them
again. This was responded to by a "Thanks, I'll look at it soon" email.
Nothing has happened after that.

Thanks, interesting...

Post by Jonny Lamb

Post by Iain Buchanan
For SynCE to ever be marked stable in Gentoo, it can't depend on an svn
snapshot of libwbxml. Marking it stable won't be an issue straight
away, but I'd like to see it done eventually.

To be honest, I'm more bothered about SynCE itself depending on such a
weird situation than any distribution having to deal it. After all,
Gentoo is not the only one having problems -- I'm going through the same
tiresome process with Debian.

Post by Iain Buchanan
1. provide patches against latest libwbxml release (0.9.2) and not
svn version?

http://compsoc.dur.ac.uk/~jdl/wbxml2-0.9.2-patches/
These are basically copied from the 0.9.2 patches which are in SynCE
http://synce.svn.sourceforge.net/viewvc/synce/trunk/patches/

aha, this might be a temporary solution then...

Post by Jonny Lamb

Post by Iain Buchanan
2. fork libwbxml and maintain it ourselves :)

Go ahead, but not only do I plan to move away from libwbxml2[0] but I
doubt there'd be any volunteers to do so.

That's fine - don't want to create more work. If the plan is to drop it
eventually, then an intermediate patched version will be fine.

Post by Jonny Lamb

Post by Iain Buchanan
3. put the pressure on upstream again, to see if we can get a new
release (or any comment)? At the very least, a release of the
current svn snapshot would be fine, if option 1. isn't possible.

If he's busy, then he's busy. It's annoying, but this is the world of
free software. He's not obliged to release..

yeah, I realise I shouldn't have used the word "pressure". I was hoping
if we pinged him again and said there was growing interest, then he
might get into it again. But I think that spending time on libwbxml
isn't the way to go...

Post by Jonny Lamb
Thanks,

thank you!

[Iain Buchanan]

Hi again :)

sorry about the lateness of those replies, my mail server was playing up
yesterday, so they came through long after I sent them. I got on irc
and had a chat instead!

The summary is this: For now, I'll make an ebuild that uses libwbxml
0.9.2 and patches it with the patches supplied by Jonny Lamb. This will
get synce going in Gentoo for now.

Then, when we remove the libwbxml dep with whatever implementation we
choose for 0.12, I can take it out of the Gentoo ebuild as well.

Seems to be quite simple! thanks,