[PATCH] HAL support for SyncEngine + misc fixes (works on FreeBSD!)

[Markus Niemistö]

Hi list,

I was trying to get synce to work with my WM6 PDA and FreeBSD 7 the other day.
There was a few things I had to do. With the attached patches I am able to
sync and access my device using palm/uppc-kmod driver from FreeBSD ports. I
also used old opensync 0.22 framework. Of course I had to disable the
Advanged network functionality on my PDA.

The first patch, hal-freebsd.patch, addresses a few FreeBSD specific hal DCCM
issues. First, linux/if.h header doesn't exist on other platforms. I think
net/if.h should work on Linux too, but I ifdef'd it just in case. Second
issues was that usb-rndis-ng rule in 10-synce.fdi created some conflicts so I
removed it. I also tried to make the FreeBSD ucom rule more general. Then I
made some adjustments for hal-synce-legacy script. There were few other
things I noticed: device file handling did not work at all on FreeBSD,
FreeBSD ppp doesn't support linkname, and last, at least on my computer, HAL
0.5.11rc2 doesn't set environmental variable HALD_ACTION. HAL also searches
the scripts only from $prefix/libexec so I adjusted the Makefile accordingly.

The second patch, sync-engine-hal.patch, adds support for HAL DCCM for
sync-engine. In addition to that it also fixes scripts list_partnerships.py,
create_partnership.py and clean_partnerships.py. The first two scripts were
using wrong (non existing) exception classes and the third was passing too
few parameters.

The third issue the second patch fixes is something I had quite hard time to
figure out. It seems that rapi contexts are created and stored into TLS so
they are per thread. So when the rapi context is created in main thread in
kernel.py and used later in another thread in synchandler.py, librapi method
rapi_context_current() returns new, uninitialized context, that results in
sync-engine crash. I fixed this by creating and using a new rapi context in
SyncHandler.run() method. This makes the thing work.

At least hal-synce-legacy and 10-synce.fdi need some work to be platform
independet, but now the thing works on FreeBSD. I hope these patches are
useful and fixes get commited. At least sync-engine TLS issue should also be
there on other platforms too.

Thanks,
Markus Niemistö

[Bakulin Ilya]

On Sat, 22 Mar 2008 14:12:47 +0200

Post by Markus Niemistö
Hi list,
I was trying to get synce to work with my WM6 PDA and FreeBSD 7 the other day.
There was a few things I had to do. With the attached patches I am able to
sync and access my device using palm/uppc-kmod driver from FreeBSD ports. I
also used old opensync 0.22 framework. Of course I had to disable the
Advanged network functionality on my PDA.

Very glad to see you as FreeBSD & SynCE user! I have some questions to you:
1. What application are you syncing with?
2. What PDA do you have? I tried to use uppc-kmod driver and get this error:
==============
ucom0: <HTC USB Serial for Wizard, class 0/0, rev 2.00/0.00, addr 2> on uhub1
ucom0: failed to set configuration, err=STALLED
device_attach: ucom0 attach returned 6
==============
, while uipaq driver works fine. Of course, FDI rule remains the same because upper-level driver (ucom) remains the same.

Post by Markus Niemistö
The first patch, hal-freebsd.patch, addresses a few FreeBSD specific hal DCCM
issues. First, linux/if.h header doesn't exist on other platforms. I think
net/if.h should work on Linux too, but I ifdef'd it just in case. Second
issues was that usb-rndis-ng rule in 10-synce.fdi created some conflicts so I
removed it. I also tried to make the FreeBSD ucom rule more general.

What exactly conflicts were caused?

Post by Markus Niemistö
Then I made some adjustments for hal-synce-legacy script. There were few other
things I noticed: device file handling did not work at all on FreeBSD,
FreeBSD ppp doesn't support linkname, and last, at least on my computer, HAL
0.5.11rc2 doesn't set environmental variable HALD_ACTION. HAL also searches
the scripts only from $prefix/libexec so I adjusted the Makefile accordingly.

I used hal-0.5.10 (of course not from ports) and HALD_ACTION exists there... Will try with 0.11 soon.
I also removed all linkname stuff, and used /dev/cuaU${HAL_FREEBSD_UNIT}.

Post by Markus Niemistö
The third issue the second patch fixes is something I had quite hard time to
figure out. It seems that rapi contexts are created and stored into TLS so
they are per thread. So when the rapi context is created in main thread in
kernel.py and used later in another thread in synchandler.py, librapi method
rapi_context_current() returns new, uninitialized context, that results in
sync-engine crash. I fixed this by creating and using a new rapi context in
SyncHandler.run() method. This makes the thing work.

Great! Now SynCE doesn't coredump while syncing to Google-calendar. (instead, Google calendar doesn't work, don't know why yet) :)))

Post by Markus Niemistö
At least hal-synce-legacy and 10-synce.fdi need some work to be platform
independet, but now the thing works on FreeBSD.

Yes, but now HAL also invokes synce-scripts when plugging in USB2COM bridge (PL2303-based in my case. I actively work with ARM- and MIPS-based development boards and use these bridges because my laptop doesn't have ordinary COM ports). This is not good. I think that we should hack HAL itself to include information about lower-level driver (uipaq, uppc) in ucom-based device node... Or add info about more PDA vendors to 10-synce.fdi :-)

[Mark Ellis]

Post by Bakulin Ilya
On Sat, 22 Mar 2008 14:12:47 +0200

Hi Markus, glad you're here !

I'm only going to touch on hal at the moment, because I don't know
sync-engine that well.

Post by Bakulin Ilya

Post by Markus Niemistö
Hi list,
I was trying to get synce to work with my WM6 PDA and FreeBSD 7 the other day.
There was a few things I had to do. With the attached patches I am able to
sync and access my device using palm/uppc-kmod driver from FreeBSD ports. I
also used old opensync 0.22 framework. Of course I had to disable the
Advanged network functionality on my PDA.

1. What application are you syncing with?
==============
ucom0: <HTC USB Serial for Wizard, class 0/0, rev 2.00/0.00, addr 2> on uhub1
ucom0: failed to set configuration, err=STALLED
device_attach: ucom0 attach returned 6
==============
, while uipaq driver works fine. Of course, FDI rule remains the same because upper-level driver (ucom) remains the same.

Post by Markus Niemistö
The first patch, hal-freebsd.patch, addresses a few FreeBSD specific hal DCCM
issues. First, linux/if.h header doesn't exist on other platforms. I think
net/if.h should work on Linux too, but I ifdef'd it just in case. Second
issues was that usb-rndis-ng rule in 10-synce.fdi created some conflicts so I
removed it. I also tried to make the FreeBSD ucom rule more general.

When I wrote hal-dccm I copied a lot from odccm. It looks like if.h
isn't needed at all, so I'll remove it.

Post by Bakulin Ilya
What exactly conflicts were caused?

Yes, what were the conflicts ? I'm not sure if anyone is still using the
-ng driver, but I'd rather not remove the rule entirely.

I obviously can't say much about the ucom rule, Bakulin does this new
version work for you as well ?

Post by Bakulin Ilya

Post by Markus Niemistö
Then I made some adjustments for hal-synce-legacy script. There were few other
things I noticed: device file handling did not work at all on FreeBSD,
FreeBSD ppp doesn't support linkname, and last, at least on my computer, HAL
0.5.11rc2 doesn't set environmental variable HALD_ACTION. HAL also searches
the scripts only from $prefix/libexec so I adjusted the Makefile accordingly.

I used hal-0.5.10 (of course not from ports) and HALD_ACTION exists there... Will try with 0.11 soon.
I also removed all linkname stuff, and used /dev/cuaU${HAL_FREEBSD_UNIT}.

Looking through the patch:-

The hal helper dir ie. libexecdir, varies across a lot of platforms. I
would rather not change it to /usr/libexec in svn. My Makefile skills
aren't great, if someone can come up with a test that would be best, or
I'll try and write a configure option.

HAL_PROP_LINUX_DEVICE_FILE is not obviously not very cross platform,
does freebsd have HAL_PROP_SERIAL_DEVICE ?

Does the linkname option cause ppp to fail ? Its quite handy for
multiple devices.

Has something replaced HALD_ACTION in 0.5.11 ? It would seem odd that
such an important item was removed entirely.

Post by Bakulin Ilya

Post by Markus Niemistö
The third issue the second patch fixes is something I had quite hard time to
figure out. It seems that rapi contexts are created and stored into TLS so
they are per thread. So when the rapi context is created in main thread in
kernel.py and used later in another thread in synchandler.py, librapi method
rapi_context_current() returns new, uninitialized context, that results in
sync-engine crash. I fixed this by creating and using a new rapi context in
SyncHandler.run() method. This makes the thing work.

Great! Now SynCE doesn't coredump while syncing to Google-calendar. (instead, Google calendar doesn't work, don't know why yet) :)))

Post by Markus Niemistö
At least hal-synce-legacy and 10-synce.fdi need some work to be platform
independet, but now the thing works on FreeBSD.

Yes, but now HAL also invokes synce-scripts when plugging in USB2COM bridge
(PL2303-based in my case. I actively work with ARM- and MIPS-based development
boards and use these bridges because my laptop doesn't have ordinary COM ports).
This is not good. I think that we should hack HAL itself to include information
about lower-level driver (uipaq, uppc) in ucom-based device node... Or add info
about more PDA vendors to 10-synce.fdi :-)

Adding vendors in 10-synce.fdi is an option, but could quickly get out
of control. If you could both post the device information using
hal-device we may be able to see a better option.

Mark

[Mark Ellis]

Made a few changes that may help you.

Post by Mark Ellis
Looking through the patch:-
The hal helper dir ie. libexecdir, varies across a lot of platforms. I
would rather not change it to /usr/libexec in svn. My Makefile skills
aren't great, if someone can come up with a test that would be best, or
I'll try and write a configure option.

Changed my mind, the closest assumption that can be made for this is
${libexecdir}, so you can pass --libexecdir=/usr/libexec. I think
anything wierder should be done at the distro level.

Post by Mark Ellis
HAL_PROP_LINUX_DEVICE_FILE is not obviously not very cross platform,
does freebsd have HAL_PROP_SERIAL_DEVICE ?

Committed this.

Post by Mark Ellis
Does the linkname option cause ppp to fail ? Its quite handy for
multiple devices.
Has something replaced HALD_ACTION in 0.5.11 ? It would seem odd that
such an important item was removed entirely.

Also added a sync-engine patch in the hal/patches dir that might work
without removing the usb-rndis-ng rule.

Mark

[Markus Niemistö]

Post by Bakulin Ilya
1. What application are you syncing with?

I mainly sync with kde-pim. I also sync with file backend to keep a backup
around.

Post by Bakulin Ilya
2. What PDA do you have? I tried to use uppc-kmod driver and get this
error: ==============
ucom0: <HTC USB Serial for Wizard, class 0/0, rev 2.00/0.00, addr 2> on
uhub1 ucom0: failed to set configuration, err=STALLED
device_attach: ucom0 attach returned 6
==============
, while uipaq driver works fine. Of course, FDI rule remains the same
because upper-level driver (ucom) remains the same.

I have a HTC TyTN with official WM6 ROM. Oh yes, I had to patch the uppc
driver to use configuration 0 instead of 1 (line 134 in uppc.c). I have used
uipaq too, but found a port easier to patch so that csup'ing the latest
kernel won't overwrite my changes. uppc driver also feels a bit more stable.
uipaq used to panic from time to time.

Post by Bakulin Ilya

Post by Markus Niemistö
The first patch, hal-freebsd.patch, addresses a few FreeBSD specific hal
DCCM issues. First, linux/if.h header doesn't exist on other platforms. I
think net/if.h should work on Linux too, but I ifdef'd it just in case.
Second issues was that usb-rndis-ng rule in 10-synce.fdi created some
conflicts so I removed it. I also tried to make the FreeBSD ucom rule
more general.

What exactly conflicts were caused?

When I plug my PDA in HAL creates two
devices: /org/freedesktop/Hal/devices/usb_device_bb4_ce_noserial_if0
and /org/freedesktop/Hal/devices/usb_device_bb4_ce_noserial. Without removing
the usb-rndis-ng rule sync-engine got confused because both device entries
have sync capabilities and friends (thanks to that rule), but only one has
the entries like pda.pocketpc.name. Depending on luck sync-engine might find
first the device entry that hal-dccm handles, or the one that is unhandled
and gets confused.

Post by Bakulin Ilya

Post by Markus Niemistö
Then I made some adjustments for hal-synce-legacy script. There were few
other things I noticed: device file handling did not work at all on
FreeBSD, FreeBSD ppp doesn't support linkname, and last, at least on my
computer, HAL 0.5.11rc2 doesn't set environmental variable HALD_ACTION.
HAL also searches the scripts only from $prefix/libexec so I adjusted the
Makefile accordingly.

I used hal-0.5.10 (of course not from ports) and HALD_ACTION exists
there... Will try with 0.11 soon. I also removed all linkname stuff, and
used /dev/cuaU${HAL_FREEBSD_UNIT}.

Ok. That HALD_ACTION issue is weird. I believe it should be there, but anyway
the script doesn't see it. I have to investigate this further. I think there
is not much difference between cua and tty in this case.

Post by Bakulin Ilya

Post by Markus Niemistö
The third issue the second patch fixes is something I had quite hard time
to figure out. It seems that rapi contexts are created and stored into
TLS so they are per thread. So when the rapi context is created in main
thread in kernel.py and used later in another thread in synchandler.py,
librapi method rapi_context_current() returns new, uninitialized context,
that results in sync-engine crash. I fixed this by creating and using a
new rapi context in SyncHandler.run() method. This makes the thing work.

Great! Now SynCE doesn't coredump while syncing to Google-calendar.
(instead, Google calendar doesn't work, don't know why yet) :)))

Post by Markus Niemistö
At least hal-synce-legacy and 10-synce.fdi need some work to be platform
independet, but now the thing works on FreeBSD.

Yes, but now HAL also invokes synce-scripts when plugging in USB2COM bridge
(PL2303-based in my case. I actively work with ARM- and MIPS-based
development boards and use these bridges because my laptop doesn't have
ordinary COM ports). This is not good. I think that we should hack HAL
itself to include information about lower-level driver (uipaq, uppc) in
ucom-based device node... Or add info about more PDA vendors to
10-synce.fdi :-)

Ok. That can be possible. It would be great to find out the exact driver from
HAL. I don't think there is any hassle free solution for this currently.

Yours,
Markus

[Mark Ellis]

Post by Markus Niemistö

Post by Bakulin Ilya
What exactly conflicts were caused?

When I plug my PDA in HAL creates two
devices: /org/freedesktop/Hal/devices/usb_device_bb4_ce_noserial_if0
and /org/freedesktop/Hal/devices/usb_device_bb4_ce_noserial. Without removing
the usb-rndis-ng rule sync-engine got confused because both device entries
have sync capabilities and friends (thanks to that rule), but only one has
the entries like pda.pocketpc.name. Depending on luck sync-engine might find
first the device entry that hal-dccm handles, or the one that is unhandled
and gets confused.

I had a go at sync-engine anyway :) Bear in mind that my python is not
very good.

Try the attached patch to a clean checkout of sync-engine. It's a little
more complicated than yours or Bakulin's because I want to keep support
for odccm. Bakulin, if you could try it as well that would be very
helpful.

Mark

[Ilya Bakulin]

On Tue, 25 Mar 2008 19:57:25 +0000

Post by Mark Ellis
I had a go at sync-engine anyway :) Bear in mind that my python is not
very good.
Try the attached patch to a clean checkout of sync-engine. It's a little
more complicated than yours or Bakulin's because I want to keep support
for odccm. Bakulin, if you could try it as well that would be very
helpful.
Mark

I didn't write it yet here: I've solved problems with syncronization, now it works OK with KDE-PIM module!
Unfortunately my laptop's screen died today. It will take some time to find out where the problem is.

But up to this moment everything was OK!

[Ilya Bakulin]

On Tue, 25 Mar 2008 21:10:14 +0200

Post by Markus Niemistö
Oh yes, I had to patch the uppc
driver to use configuration 0 instead of 1 (line 134 in uppc.c).

:-( Could you please send me full patch for uppc? I've changed uppc.c:144 (UPPC_CONFIG_INDEX 1 => 0), but stall error is still here...

Post by Markus Niemistö
Yours,
Markus
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
SynCE-Devel mailing list
https://lists.sourceforge.net/lists/listinfo/synce-devel

[Ilya Bakulin]

On Tue, 25 Mar 2008 21:10:14 +0200

Post by Markus Niemistö
Oh yes, I had to patch the uppc
driver to use configuration 0 instead of 1 (line 134 in uppc.c).

:-( Could you please send me full patch for uppc? I've changed uppc.c:144 (UPPC_CONFIG_INDEX 1 => 0), but stall error is still here...

Post by Markus Niemistö
Yours,
Markus
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
SynCE-Devel mailing list
https://lists.sourceforge.net/lists/listinfo/synce-devel

[Markus Niemistö]

Post by Bakulin Ilya
1. What application are you syncing with?

I mainly sync with kde-pim. I also sync with file backend to keep a backup
around.

Post by Bakulin Ilya
2. What PDA do you have? I tried to use uppc-kmod driver and get this
error: ==============
ucom0: <HTC USB Serial for Wizard, class 0/0, rev 2.00/0.00, addr 2> on
uhub1 ucom0: failed to set configuration, err=STALLED
device_attach: ucom0 attach returned 6
==============
, while uipaq driver works fine. Of course, FDI rule remains the same
because upper-level driver (ucom) remains the same.

I have a HTC TyTN with official WM6 ROM. Oh yes, I had to patch the uppc
driver to use configuration 0 instead of 1 (line 134 in uppc.c). I have used
uipaq too, but found a port easier to patch so that csup'ing the latest
kernel won't overwrite my changes. uppc driver also feels a bit more stable.
uipaq used to panic from time to time.

Post by Bakulin Ilya

Post by Markus Niemistö
The first patch, hal-freebsd.patch, addresses a few FreeBSD specific hal
DCCM issues. First, linux/if.h header doesn't exist on other platforms. I
think net/if.h should work on Linux too, but I ifdef'd it just in case.
Second issues was that usb-rndis-ng rule in 10-synce.fdi created some
conflicts so I removed it. I also tried to make the FreeBSD ucom rule
more general.

What exactly conflicts were caused?

When I plug my PDA in HAL creates two
devices: /org/freedesktop/Hal/devices/usb_device_bb4_ce_noserial_if0
and /org/freedesktop/Hal/devices/usb_device_bb4_ce_noserial. Without removing
the usb-rndis-ng rule sync-engine got confused because both device entries
have sync capabilities and friends (thanks to that rule), but only one has
the entries like pda.pocketpc.name. Depending on luck sync-engine might find
first the device entry that hal-dccm handles, or the one that is unhandled
and gets confused.

Post by Bakulin Ilya

Post by Markus Niemistö
Then I made some adjustments for hal-synce-legacy script. There were few
other things I noticed: device file handling did not work at all on
FreeBSD, FreeBSD ppp doesn't support linkname, and last, at least on my
computer, HAL 0.5.11rc2 doesn't set environmental variable HALD_ACTION.
HAL also searches the scripts only from $prefix/libexec so I adjusted the
Makefile accordingly.

I used hal-0.5.10 (of course not from ports) and HALD_ACTION exists
there... Will try with 0.11 soon. I also removed all linkname stuff, and
used /dev/cuaU${HAL_FREEBSD_UNIT}.

Ok. That HALD_ACTION issue is weird. I believe it should be there, but anyway
the script doesn't see it. I have to investigate this further. I think there
is not much difference between cua and tty in this case.

Post by Bakulin Ilya

Post by Markus Niemistö
The third issue the second patch fixes is something I had quite hard time
to figure out. It seems that rapi contexts are created and stored into
TLS so they are per thread. So when the rapi context is created in main
thread in kernel.py and used later in another thread in synchandler.py,
librapi method rapi_context_current() returns new, uninitialized context,
that results in sync-engine crash. I fixed this by creating and using a
new rapi context in SyncHandler.run() method. This makes the thing work.

Great! Now SynCE doesn't coredump while syncing to Google-calendar.
(instead, Google calendar doesn't work, don't know why yet) :)))

Post by Markus Niemistö
At least hal-synce-legacy and 10-synce.fdi need some work to be platform
independet, but now the thing works on FreeBSD.

Yes, but now HAL also invokes synce-scripts when plugging in USB2COM bridge
(PL2303-based in my case. I actively work with ARM- and MIPS-based
development boards and use these bridges because my laptop doesn't have
ordinary COM ports). This is not good. I think that we should hack HAL
itself to include information about lower-level driver (uipaq, uppc) in
ucom-based device node... Or add info about more PDA vendors to
10-synce.fdi :-)

Ok. That can be possible. It would be great to find out the exact driver from
HAL. I don't think there is any hassle free solution for this currently.

Yours,
Markus

[Guido Diepen]

Hi Markus,

Post by Markus Niemistö
The third issue the second patch fixes is something I had quite hard time
to figure out. It seems that rapi contexts are created and stored into TLS
so they are per thread. So when the rapi context is created in main thread
in kernel.py and used later in another thread in synchandler.py, librapi
method rapi_context_current() returns new, uninitialized context, that
results in sync-engine crash. I fixed this by creating and using a new rapi
context in SyncHandler.run() method. This makes the thing work.

Thank you for this patch. I don't have the hal-dccm working, so I can't test
the other patches. I did apply this patch you provided in the current SVN
because indeed it was a problem that I was encountering also lately.

Recently I updated the librapi2 such that it uses TLS because one todo that
was still open for quite some time was to make the library thread safe.
Unfortunately, making the lib threadsafe caused segfaults in sync-engine that
I could not figure out. Thanks to your patch that problem is gone!

Thank you,

Guido Diepen

[Markus Niemistö]

Post by Guido Diepen
Thank you for this patch. I don't have the hal-dccm working, so I can't
test the other patches. I did apply this patch you provided in the current
SVN because indeed it was a problem that I was encountering also lately.
Recently I updated the librapi2 such that it uses TLS because one todo that
was still open for quite some time was to make the library thread safe.
Unfortunately, making the lib threadsafe caused segfaults in sync-engine
that I could not figure out. Thanks to your patch that problem is gone!

You are welcome. Glad to help a bit!

Cheers,
Markus