[Synce-devel] Missing WM6 PIM items on partnership deletion

[Dr J A Gow]

Folks,

Just in case anyone is here that is not subscribed to synce-users, this
has to be read to be believed. It is a copy of the message I sent there
a moment ago:

----

Remember the problem some of you have had with WM6 when deleting a
partnership all the PIM items disappear with it? Well I have finally got
to the bottom of this now - and guess what? We are stuck with it!

That's right, it's a FEATURE, not a bug (either in our code or in
WM6)!!!!! Can you believe this! I didn't believe any software developer
could be this daft at first, until I remembered I was dealing with a
Microsoft product.

It happens with ActiveSync on Windows too.

A little more info for those who are interested:

Initially when being appraised of this problem, I spent a good few hours
analyzing wire dumps between AS and the device under Windows. Deleting
and recreating partnerships many times, I could not make PIM items
disappear. The wire dumps showed that our sync-engine code was, in fact,
operating correctly in that it was processing the partnership deletion
in the same way as ActiveSync.

However, on the Linux box it happened predictably. Initially I was
seeing sync-engine crash and naturally I assumed that the two problems
were related. A few hours later and the crash problem was largely fixed
(it seems to work more reliably if the partnership is not bound before
being deleted). However PIM items continued to be deleted with the
partnership.

And then, when examining the setup to see what was different between the
AS on Windows and the sync-engine setups, it struck me that when testing
sync-engine, I had just one partnership on the device. I was deleting
this leaving no partnerships and recreating just the one. When testing
with Windows, I created a second partnership _leaving the sync-engine
one intact_.

So I deleted all partnerships from the device, restored the PIM items,
then created just one partnership under Windows, just syncing tasks.
Then I deleted it. Guess what? All the tasks disappeared. Same happened
when I tried to sync contacts, same with calendar, same with all three.

Turns out the way it works is as follows
----------------------------------------

When WM6 creates the partnership, it creates an association between the
actual PIM data and the partnership. Now, if you create a second
partnership, syncing the same items, a second association is created
between the same PIM data and the second partnership. Now this works a
bit like a reference count.

You can safely delete/recreate/delete the second partnership and nothing
will disappear, because the first partnership holds a reference to the
PIM items. You can also delete the first partnership safely providing
the second partnership is still present. Ok so far.

Now delete one partnership, then delete the second one. Upon deletion of
the second partnership there are no more partnerships available
associated with the PIM items :- so WM6 DELETES THE PIM ITEMS!

Under Windows, it even tells you it is going to do this with the
following message, displayed in a message box with an 'Ok' button:

"Deleting the synchronization relationship with the PocketPC will remove
all the E-mail, Calendar, Contacts and Tasks information on your
PocketPC that is synchronized only with this computer the next time the
device is connected. To synchronize this PocketPC with this computer
again, you must set up a new synchronization relationship"

I guess M$ see this as a 'security feature' or something like that!

So at this point I thought I had better go on a trawl and see if anybody
in the Windows community had seen this problem. Didn't take long before
I found this thread:

http://forum.soft32.com/pda/Removing-partnership-losing-data-ftopict79766.html

Furthermore, if you have one partnership that syncs, say Contacts and
Calendar, and the second that only syncs Tasks, if you delete the second
partnership you will lose all your Tasks (there is only one partnership
that holds a reference to Tasks) and if you delete the first, you will
lose all your Contacts and Events (only this partnership holds a
reference to Contacts and Events).

Basically this is an exclusive operation, so you will only lose the item
type for which you are deleting a partnership that holds the only
reference - if another partnership holds a reference to this item type
you will not lose it.


I'm going for some coffee now, I bloody need it after discovering this
behaviour!

John.

[Guido Diepen]

Post by Dr J A Gow
Folks,
Just in case anyone is here that is not subscribed to synce-users, this
has to be read to be believed. It is a copy of the message I sent there
----
Remember the problem some of you have had with WM6 when deleting a
partnership all the PIM items disappear with it? Well I have finally got
to the bottom of this now - and guess what? We are stuck with it!
That's right, it's a FEATURE, not a bug (either in our code or in
WM6)!!!!! Can you believe this! I didn't believe any software developer
could be this daft at first, until I remembered I was dealing with a
Microsoft product.
It happens with ActiveSync on Windows too.
Initially when being appraised of this problem, I spent a good few hours
analyzing wire dumps between AS and the device under Windows. Deleting
and recreating partnerships many times, I could not make PIM items
disappear. The wire dumps showed that our sync-engine code was, in fact,
operating correctly in that it was processing the partnership deletion
in the same way as ActiveSync.
However, on the Linux box it happened predictably. Initially I was
seeing sync-engine crash and naturally I assumed that the two problems
were related. A few hours later and the crash problem was largely fixed
(it seems to work more reliably if the partnership is not bound before
being deleted). However PIM items continued to be deleted with the
partnership.
And then, when examining the setup to see what was different between the
AS on Windows and the sync-engine setups, it struck me that when testing
sync-engine, I had just one partnership on the device. I was deleting
this leaving no partnerships and recreating just the one. When testing
with Windows, I created a second partnership _leaving the sync-engine
one intact_.
So I deleted all partnerships from the device, restored the PIM items,
then created just one partnership under Windows, just syncing tasks.
Then I deleted it. Guess what? All the tasks disappeared. Same happened
when I tried to sync contacts, same with calendar, same with all three.
Turns out the way it works is as follows
----------------------------------------
When WM6 creates the partnership, it creates an association between the
actual PIM data and the partnership. Now, if you create a second
partnership, syncing the same items, a second association is created
between the same PIM data and the second partnership. Now this works a
bit like a reference count.
You can safely delete/recreate/delete the second partnership and nothing
will disappear, because the first partnership holds a reference to the
PIM items. You can also delete the first partnership safely providing
the second partnership is still present. Ok so far.
Now delete one partnership, then delete the second one. Upon deletion of
the second partnership there are no more partnerships available
associated with the PIM items :- so WM6 DELETES THE PIM ITEMS!
Under Windows, it even tells you it is going to do this with the
"Deleting the synchronization relationship with the PocketPC will remove
all the E-mail, Calendar, Contacts and Tasks information on your
PocketPC that is synchronized only with this computer the next time the
device is connected. To synchronize this PocketPC with this computer
again, you must set up a new synchronization relationship"
I guess M$ see this as a 'security feature' or something like that!
So at this point I thought I had better go on a trawl and see if anybody
in the Windows community had seen this problem. Didn't take long before
http://forum.soft32.com/pda/Removing-partnership-losing-data-ftopict79766.h
tml
Furthermore, if you have one partnership that syncs, say Contacts and
Calendar, and the second that only syncs Tasks, if you delete the second
partnership you will lose all your Tasks (there is only one partnership
that holds a reference to Tasks) and if you delete the first, you will
lose all your Contacts and Events (only this partnership holds a
reference to Contacts and Events).
Basically this is an exclusive operation, so you will only lose the item
type for which you are deleting a partnership that holds the only
reference - if another partnership holds a reference to this item type
you will not lose it.
I'm going for some coffee now, I bloody need it after discovering this
behaviour!

Extra strong one I presume?? :) Sorry to hear that it took so much of your
time ....

Though I have to say that I encountered this "feature" once already. When I
was working on creating a partnership and the syncengine was deleting
the 'dangling' partnership due to the Exchange Server, I did not want to
delete the partnership through the sync-engine, so I manually deleted in
within AS on the phone. Then all of a sudden all of my events/tasks etc were
gone already also. So this also happens when you initiate the removal from
the computer.

Furthermore, You mentioned you were working on the partnership code, due to
some minor issues I had to hard-reset my phone. The first thing I did was
install the mail2web exchange server. Without any other partnerships present,
only the Exhcnage one, I don't have any subkeys P1 or P2, ... anymore under
the Partners Reg Key. Thought this info might be of use while working on the
partnerships :)

Guido Diepen

[Dr J A Gow]

Post by Guido Diepen
Furthermore, You mentioned you were working on the partnership code, due to
some minor issues I had to hard-reset my phone. The first thing I did was
install the mail2web exchange server. Without any other partnerships present,
only the Exhcnage one, I don't have any subkeys P1 or P2, ... anymore under
the Partners Reg Key. Thought this info might be of use while working on the
partnerships :)

Ah, thanks for this info. The partnership code overhaul is almost
complete and there is just this issue of deleting exchange servers to
fix. At the moment I don't have an available Exchange server to test
against - so could you help with this? What I need to know is how, from
the registry, I can discriminate between an exchange server partnership
and an AS partnership, so that so we don't kill the exchange partnership
thinking it is a dangling entry. Could you have a poke about in your
registry and let me know if there is an easy way to do this?

John.

[Guido Diepen]

Post by Dr J A Gow

Post by Guido Diepen
Furthermore, You mentioned you were working on the partnership code, due
to some minor issues I had to hard-reset my phone. The first thing I did
was install the mail2web exchange server. Without any other partnerships
present, only the Exhcnage one, I don't have any subkeys P1 or P2, ...
anymore under the Partners Reg Key. Thought this info might be of use
while working on the partnerships :)

Ah, thanks for this info. The partnership code overhaul is almost
complete and there is just this issue of deleting exchange servers to
fix. At the moment I don't have an available Exchange server to test
against - so could you help with this? What I need to know is how, from
the registry, I can discriminate between an exchange server partnership
and an AS partnership, so that so we don't kill the exchange partnership
thinking it is a dangling entry. Could you have a poke about in your
registry and let me know if there is an easy way to do this?
John.

One of the things that struck me, when looking at it again is the following
line (partnerships.py:83):
partners_key = hklm.create_sub_key(r"Software\Microsoft\Windows CE
Services\Partners")

This looks to me like you are ALWAYS creating a key. Within rapi, there is
already a enumKey, this could also be ported to pyrapi I guess. It would be
nicer to eumerate over the keys, instead of trying to create them.

About the registry, please give me some time, you will get a registry dump of
all needed keys, it is working sooooooooooo nice ;) Currently doing small
debug of the librapi2 stuff, such that it will be possible to make use of
NULL values indicating the user is not interested in knowing those values.
This is too make the library a bit more stable and robust, since the API does
allow for NULL values.

Guido Diepen