[Synce-devel] Some more debug-logs for WM6 devices.

[Guido Diepen]

Sorry for bugging the list with this. Turns out I can't send email directly to
John since something is misconfigured with the SPF settings for my domain
currently :(

Maybe other people can see interesting things in it also though :)

Currently testing your sync-engine again. Yesterday I installed everything
from SVN combined with the opensync 0.22 tree. As I mentioned on the
mailinglist yesterday, the previously available partnership with the
exchange server at mail2web.com was deleted after I connected my phone to
sync-engine.

After this I created a partnership with linux, then I tried the opensync with
KDE and I got the errormessages about some unexpected PCDATA that I also sent
to the list yesterday.


During the day I recreated the mail2web partnership again (nice to have some
pushmail) and so I had 2 partnerships, 1 with mail2web and 1 with
sync-engine.

Since the 0.22 opensync was not working for me anyway and therefore syncing at
all was not working, I decided to give the 0.3x svn opensync framework a go.
The compilation went really smooth.

After this I started the sync-engine (rev 3070, last changed rev 3059) and
connected my phone to the computer. I just let it be connected for some time
and after about 40 seconds the sync-engine crashed. The complete log file
from connecting till crash is attached with this email.

After this sync-engine crashed, again the partnership with mail2web was
removed.

More information about the device: I am using a MDA Vario III, WM6 Dutch
version.


Is more debug information needed, and if so, what exactly is needed?

Hope this info helps a bit.

Guido Diepen

PS: I will try to get SPF problem solved soon :)

[Dr J A Gow]

Post by Guido Diepen
Sorry for bugging the list with this. Turns out I can't send email directly to
John since something is misconfigured with the SPF settings for my domain
currently :(
Maybe other people can see interesting things in it also though :)
Currently testing your sync-engine again. Yesterday I installed everything
from SVN combined with the opensync 0.22 tree. As I mentioned on the
mailinglist yesterday, the previously available partnership with the
exchange server at mail2web.com was deleted after I connected my phone to
sync-engine.
After this I created a partnership with linux, then I tried the opensync with
KDE and I got the errormessages about some unexpected PCDATA that I also sent
to the list yesterday.
During the day I recreated the mail2web partnership again (nice to have some
pushmail) and so I had 2 partnerships, 1 with mail2web and 1 with
sync-engine.
Since the 0.22 opensync was not working for me anyway and therefore syncing at
all was not working, I decided to give the 0.3x svn opensync framework a go.
The compilation went really smooth.
After this I started the sync-engine (rev 3070, last changed rev 3059) and
connected my phone to the computer. I just let it be connected for some time
and after about 40 seconds the sync-engine crashed. The complete log file
from connecting till crash is attached with this email.
After this sync-engine crashed, again the partnership with mail2web was
removed.
More information about the device: I am using a MDA Vario III, WM6 Dutch
version.

Thanks for your report.

The crash is actually caused by the phone not responding to a
CeProcessConfig request (which does look correctly constructed).

Can you try the following:

- Delete all partnerships from the phone
- Create a new partnership with the linux box
- Attempt sync

In other words, don't create the mail2web partnership first. I am
wondering if we are incorrectly handling adjacent partnerships that are
not a direct AS partnership.

I am currently working heavily on the partnership code to clean it up
improve it and add some very useful per-partnership configurations, so
now would be a good time for me to investigate this one.

(Incidentally, regarding pushmail - if you or anyone else has an
internet-facing IP address you may be interested in some other work I am
doing at the moment - it's embryonic at the moment but I am looking to
use some of the core sync-engine code to develop a linux-hosted push
email server. I have been watching Airsync traffic for a while now and I
think it can be done...watch this space!)

John.

[John Carr]

Post by Dr J A Gow
(Incidentally, regarding pushmail - if you or anyone else has an
internet-facing IP address you may be interested in some other work I am
doing at the moment - it's embryonic at the moment but I am looking to
use some of the core sync-engine code to develop a linux-hosted push
email server. I have been watching Airsync traffic for a while now and I
think it can be done...watch this space!)
John.

Excellent news :-)

I'd be interested in talking about this on IRC as it's something i'd looked
at myself. One idea I had been throwing around was finishing up the server
mode of conduit. The main reason for Conduits involvement was an adaptor
layer so that its all pluggable. For example, I could connect synce contact
sync to conduits existing gmail contacts plugin. Equally, if i move to an
open phone platform I can just switch from ActiveSync to SyncML or whatever
the new sync protocol is.. Unfortunately I've not had time as we've been
concentrating on desktop integration.. but if your looking in this area then
i'd like to see if there is some area of cross over / co-operation..

John

[Dr J A Gow]

Post by Dr J A Gow
(Incidentally, regarding pushmail - if you or anyone else has an
internet-facing IP address you may be interested in some other work I am
doing at the moment - it's embryonic at the moment but I am looking to
use some of the core sync-engine code to develop a
linux-hosted push
email server. I have been watching Airsync traffic for a while now and I
think it can be done...watch this space!)
John.
Excellent news :-)
I'd be interested in talking about this on IRC as it's something i'd
looked at myself. One idea I had been throwing around was finishing up
the server mode of conduit. The main reason for Conduits involvement
was an adaptor layer so that its all pluggable. For example, I could
connect synce contact sync to conduits existing gmail contacts plugin.
Equally, if i move to an open phone platform I can just switch from
ActiveSync to SyncML or whatever the new sync protocol is..
Unfortunately I've not had time as we've been concentrating on desktop
integration.. but if your looking in this area then i'd like to see if
there is some area of cross over / co-operation..

I am looking at this at the moment but there is a number of very
significant fixups and improvements I need to make to sync-engine itself
first. The Airsync stuff for pushmail is straightforward but is
sufficiently different to the PIM syncing to require substantial changes
to the way we handle the individual items. I had thought about giving
the server a pluggable architecture with pluggable backends (e.g. imap,
pop, local file database etc). One of those backends could be a plugin
for Conduit....

I'll probably do an IMAP backend first as this is the one I need :)

The server would be fundamentally distinct from sync-engine (it would
have to be at least initially to make debugging easier) and is designed
to run stand-alone so that the mobile device can connect to it in the
same manner as it would an Exchange server.

Let's talk about this later next week on IRC. I need to finish the
updates (DTPT, partnership and config stuff) to sync-engine first and I
should be able to get back to the push server then.

John.

[John Carr]

Post by Dr J A Gow
I am looking at this at the moment but there is a number of very
significant fixups and improvements I need to make to sync-engine itself
first. The Airsync stuff for pushmail is straightforward but is
sufficiently different to the PIM syncing to require substantial changes
to the way we handle the individual items. I had thought about giving
the server a pluggable architecture with pluggable backends (e.g. imap,
pop, local file database etc). One of those backends could be a plugin
for Conduit....
I'll probably do an IMAP backend first as this is the one I need :)
The server would be fundamentally distinct from sync-engine (it would
have to be at least initially to make debugging easier) and is designed
to run stand-alone so that the mobile device can connect to it in the
same manner as it would an Exchange server.
Let's talk about this later next week on IRC. I need to finish the
updates (DTPT, partnership and config stuff) to sync-engine first and I
should be able to get back to the push server then.

Sounds good. If its relatively light on dependencies I think a conduit
backend would be born very quickly :-) As you know I will be about quite a
bit :-D

John

[Dr J A Gow]

Post by John Carr
Sounds good. If its relatively light on dependencies I think a conduit
backend would be born very quickly :-) As you know I will be about
quite a bit :-D

Dependencies shouldn't be too much of a problem - the mail protocols
themselves would be buried in the backend (imap - imap plugin and so
on). The front end should only be dependent on the usual Python server
stuff and a patched wbxml (it won't need rapi2 or even libsynce).
Although C/C++ is my more natural environment :) I hope to keep the push
server as pure Python so it should be relatively straightforward to
develop backends.

John.

[John Carr]

Post by Dr J A Gow

Post by John Carr
Sounds good. If its relatively light on dependencies I think a conduit
backend would be born very quickly :-) As you know I will be about
quite a bit :-D

Dependencies shouldn't be too much of a problem - the mail protocols
themselves would be buried in the backend (imap - imap plugin and so
on). The front end should only be dependent on the usual Python server
stuff and a patched wbxml (it won't need rapi2 or even libsynce).
Although C/C++ is my more natural environment :) I hope to keep the push
server as pure Python so it should be relatively straightforward to
develop backends.
John.

Awesome \o/ Hopefully we can do something about wbxml too. Though i'm
stalled on that till i dare sync my phone to collect some test data :P

[Guido Diepen]

Hi John,

Post by Dr J A Gow
Thanks for your report.
The crash is actually caused by the phone not responding to a
CeProcessConfig request (which does look correctly constructed).
- Delete all partnerships from the phone
- Create a new partnership with the linux box
- Attempt sync

I did not have the possibility of testing that, but I tried something
different which might give some extra insight :)

I was looking at the registry and there I have two partnerships under
HKLM\Software\Microsoft\Windows CE Services\Partners

P1 and P2
P1 contains DataSourceID, PId, and PName (everything of linux partnership)
P2 contains nothing at all

When looking at the following registry:
HKCU\Software\Microsoft\ActiveSync\Partners
I have two sub keys with the GUID's

Your syncengine tries to read at line 81 of SyncEngine/partnerships.py the
information regarding the partnerships by means of reading the values of PId
and PName. These do not exist for the Exchange partnership.

Would be nice if somebody that has the possiblity of creating a partnership
with an Exchange Server can verify whether it always is the case that the
information regarding the partnership is left empty for exchange servers.
(could also be the reason why you can have only one exchange server as a
partnership, if activesync sees partnership without Id and Name, it must the
exchange one.

Post by Dr J A Gow
In other words, don't create the mail2web partnership first. I am
wondering if we are incorrectly handling adjacent partnerships that are
not a direct AS partnership.

I will try to see whether I can verify it this weekend, but I am pretty sure
that the problem lays in the fact that you try to read a non-existent value.

Post by Dr J A Gow
I am currently working heavily on the partnership code to clean it up
improve it and add some very useful per-partnership configurations, so
now would be a good time for me to investigate this one.
(Incidentally, regarding pushmail - if you or anyone else has an
internet-facing IP address you may be interested in some other work I am
doing at the moment - it's embryonic at the moment but I am looking to
use some of the core sync-engine code to develop a linux-hosted push
email server. I have been watching Airsync traffic for a while now and I
think it can be done...watch this space!)

NICE :)


Guido Diepen

PS: Currently working on a very very simple Registry Dumper based on the
synce-registry code shipped with librapi. Unfortunately for me it looks like
the CeRegEnumValue function is not working and always segfaults, meaning that
I can query the enumeration of subkeys of a specific regkey, but not the
subvalues. If somebody has an idea for that, I would be happy to hear :)

[David Eriksson]

Post by Guido Diepen
PS: Currently working on a very very simple Registry Dumper based on the
synce-registry code shipped with librapi. Unfortunately for me it looks like
the CeRegEnumValue function is not working and always segfaults, meaning that
I can query the enumeration of subkeys of a specific regkey, but not the
subvalues. If somebody has an idea for that, I would be happy to hear :)

Rebuild librapi with debug flags, maybe like this:

export CFLAGS="-Wall -Werror -ggdb3"
./bootstrap
./configure

Run in gdb or load the core dump in gdb and provide us with a backtrace!

\David

[Guido Diepen]

Post by David Eriksson

Post by Guido Diepen
PS: Currently working on a very very simple Registry Dumper based on the
synce-registry code shipped with librapi. Unfortunately for me it looks like
the CeRegEnumValue function is not working and always segfaults, meaning that
I can query the enumeration of subkeys of a specific regkey, but not the
subvalues. If somebody has an idea for that, I would be happy to hear :)

export CFLAGS="-Wall -Werror -ggdb3"
./bootstrap
./configure
Run in gdb or load the core dump in gdb and provide us with a backtrace!
\David

backtrace is not providing enough of information for me at the moment. I will
try to see if I can get a detailed backtrace

Could anybody in the meantime test the following please and notify me with the
results:
synce-registry -l HKLM "Software\Microsoft\Windows CE Services"

This should print a subkey Partners and some values.

In my case the subkey Partners is printed without a problem, but as soon as
the values are started (with CeRegEnumValue) I get the segfault.

Would be really useful to know if it is just my computer/setup or not.

If I get this to work it will be a lot easier to see changes happing to the
registry, by dumping all information to files and diff those files.

Regards,

Guido Diepen

[David Eriksson]

[Returning to synce-devel list.]

[snip]

Post by David Eriksson
Run in gdb or load the core dump in gdb and provide us with a backtrace!
\David

OK,
I really got into the details and I implemented a CeRegEnumValues2 within
registry2.c in the rapi.
rapi_context_begin_command(context, 0x23);

First of all, CeRegEnumValue has command code 0x34. Try that first!

Sorry for not looking at the command code value earlier.

\David

//Write the handle
rapi_buffer_write_uint32 (context->send_buffer, hKey);
//Write the requested index
rapi_buffer_write_uint32 (context->send_buffer, dwIndex);
//Write the size of our buffer, for holding the valuename
rapi_buffer_write_uint32(context->send_buffer, *lpcbValueName);
//Write the size of our buffer for holding the extra data, I just put this
//to 0 for the moment, since I am not interested in that (is also allowed
to
be 0 by definition)
rapi_buffer_write_uint32(context->send_buffer, 0 );
if ( !rapi2_context_call(context) )
return false;
and read the error messages + return values.
rapi_buffer_read_uint32(context->recv_buffer, &context->last_error);
rapi_buffer_read_int32(context->recv_buffer, &return_value);
After this I try to read all the extra information, but that will go
wrong,
since the recv buffer of the context only contains 8 bytes, namely the two
integers for the return value and the last_error. The rest is for the
moment
not that important.
The interesting though is the fact that return_value contains the value 0,
while last_error contains the value 6. As far as I could see, the error
value
6 denotes an invalid handle.
Unfortunately I do not have Windows available, but clearly something goes
wrongly over the wire.
Since you requested some backtraces to be able to help, maybe this
extended
information is better :)
Do you have windows available and could you provide a dump of what goes
over
the wire as soon as you try to do enumeration of the values within a key
via
windows?
Regards,
Guido Diepen

[Guido Diepen]

Post by David Eriksson
[Returning to synce-devel list.]

Sorry for that:)

Post by David Eriksson

rapi_context_begin_command(context, 0x23);

First of all, CeRegEnumValue has command code 0x34. Try that first!
Sorry for not looking at the command code value earlier.

Wow.... that is some interesting stuff :) What exactly does code 0x23 mean
then?? Have I been trying to format my device all of the time?

Do you have a complete list of the codes then?

Because thinking very simple, I got the 0x23 out of README.rapi list. With the
0x34 I did not get the error, but some other things need to be implemented
first for that to actually work.

Guido Diepen

[Guido Diepen]

Post by David Eriksson
First of all, CeRegEnumValue has command code 0x34. Try that first!

Done, guess what: that was the problem!!!!!

Have to make it nicer now, but within a day or two I will provide the needed
patches to have the registry dump function available.

Hope it will be useful with debugging :)

Guido Diepen

[Guido Diepen]

Post by Guido Diepen

Post by David Eriksson
First of all, CeRegEnumValue has command code 0x34. Try that first!

Done, guess what: that was the problem!!!!!
Have to make it nicer now, but within a day or two I will provide the
needed patches to have the registry dump function available.
Hope it will be useful with debugging :)
Guido Diepen

The registry dump is now working! I will try to clean up my code and as soon
as this is done I will supply the patch for this to work.

Probably some nice features would be to give a path to a key from which the
tree must be dumped, since dumping the whole tree takes some time, about 6.5
minutes on my machine, resulting in a 1.3 MB file with the contents of the
registry :)

Regards,

Guido Diepen

[David Eriksson]

Post by Guido Diepen

Post by David Eriksson
[Returning to synce-devel list.]

Sorry for that:)

Post by David Eriksson

rapi_context_begin_command(context, 0x23);

First of all, CeRegEnumValue has command code 0x34. Try that first!
Sorry for not looking at the command code value earlier.

Wow.... that is some interesting stuff :) What exactly does code 0x23 mean
then?? Have I been trying to format my device all of the time?
Do you have a complete list of the codes then?

Yes! I added it right after I sent you my previous mail:

http://synce.svn.sourceforge.net/viewvc/*checkout*/synce/trunk/librapi2/src/rapi2/README

Post by Guido Diepen
Because thinking very simple, I got the 0x23 out of README.rapi list. With the
0x34 I did not get the error, but some other things need to be implemented
first for that to actually work.

README.rapi is for the old RAPI, so I moved and renamed it to
rapi/README.


Best regards,

\David

[Guido Diepen]

Post by Guido Diepen
Hi John,

Post by Dr J A Gow
Thanks for your report.
The crash is actually caused by the phone not responding to a
CeProcessConfig request (which does look correctly constructed).
- Delete all partnerships from the phone
- Create a new partnership with the linux box
- Attempt sync

I did not have the possibility of testing that, but I tried something
different which might give some extra insight :)
I was looking at the registry and there I have two partnerships under
HKLM\Software\Microsoft\Windows CE Services\Partners
P1 and P2
P1 contains DataSourceID, PId, and PName (everything of linux partnership)
P2 contains nothing at all
HKCU\Software\Microsoft\ActiveSync\Partners
I have two sub keys with the GUID's
Your syncengine tries to read at line 81 of SyncEngine/partnerships.py the
information regarding the partnerships by means of reading the values of
PId and PName. These do not exist for the Exchange partnership.
Would be nice if somebody that has the possiblity of creating a partnership
with an Exchange Server can verify whether it always is the case that the
information regarding the partnership is left empty for exchange servers.
(could also be the reason why you can have only one exchange server as a
partnership, if activesync sees partnership without Id and Name, it must
the exchange one.

Post by Dr J A Gow
In other words, don't create the mail2web partnership first. I am
wondering if we are incorrectly handling adjacent partnerships that are
not a direct AS partnership.

I will try to see whether I can verify it this weekend, but I am pretty
sure that the problem lays in the fact that you try to read a non-existent
value.

Had some time left and when the Exchange partnership was not present, the
sync-engine did not crash, it just went into the RRA loop. Note that I did
not try the actual syncing yet, but since the sync-engine crashed before the
actual syncing anyway, I don't think that that is important.

Regards,

Guido Diepen

[Dr J A Gow]

Post by Guido Diepen
I was looking at the registry and there I have two partnerships under
HKLM\Software\Microsoft\Windows CE Services\Partners
P1 and P2
P1 contains DataSourceID, PId, and PName (everything of linux partnership)
P2 contains nothing at all
HKCU\Software\Microsoft\ActiveSync\Partners
I have two sub keys with the GUID's
Your syncengine tries to read at line 81 of SyncEngine/partnerships.py the
information regarding the partnerships by means of reading the values of PId
and PName. These do not exist for the Exchange partnership.
Would be nice if somebody that has the possiblity of creating a partnership
with an Exchange Server can verify whether it always is the case that the
information regarding the partnership is left empty for exchange servers.
(could also be the reason why you can have only one exchange server as a
partnership, if activesync sees partnership without Id and Name, it must the
exchange one.

Great report, thanks! Good timing as well. I'll see how this can be
fixed as I rework the partnership code. Should all be ready some time
next week.

John.