[Synce-devel] Patch for more registry functions to work

[Guido Diepen]

Hi,

the supplied patch is against the current SVN (3074) as mentioned in the file.

I think there was a bug in the rapi_buffer_read_string. It would always read
one more byte then the actual size of the letters, to ensure a \0 terminator.
Unfortunately, with the EnumKey this always went ok, with the EnumValue that
is now implemented, more data was behind the string, resulting in an
unterminated string and problems with reading the later values.

Furthermore, I updated the synce-registry tool to make a complete dump of the
registry to screen. This takes about 4.5 minutes on my machine and the
resulting file with all information is about 1.3 MB

Two options have been added to synce-registry: -D and -L
synce-registry -D
does a complete dump of the registry

the -L option turns on recursion for key_listing, meaning you can do a
recursive dump of only HKLM\Software\Microsoft by:
synce-registry -l HKLM "Software\Microsoft" -L
(to avoid the 4.5 minutes all of the time :) )

Hope the supplied patch is useful for everybody.

Regards,


Guido Diepen

[Mark Ellis]

Post by Guido Diepen
Hi,
the supplied patch is against the current SVN (3074) as mentioned in the file.
I think there was a bug in the rapi_buffer_read_string. It would always read
one more byte then the actual size of the letters, to ensure a \0 terminator.
Unfortunately, with the EnumKey this always went ok, with the EnumValue that
is now implemented, more data was behind the string, resulting in an
unterminated string and problems with reading the later values.
Furthermore, I updated the synce-registry tool to make a complete dump of the
registry to screen. This takes about 4.5 minutes on my machine and the
resulting file with all information is about 1.3 MB
Two options have been added to synce-registry: -D and -L
synce-registry -D
does a complete dump of the registry
the -L option turns on recursion for key_listing, meaning you can do a
synce-registry -l HKLM "Software\Microsoft" -L
(to avoid the 4.5 minutes all of the time :) )
Hope the supplied patch is useful for everybody.
Regards,
Guido Diepen
-------------------------------------------------------------------------

Thanks Guido, that's cool, if no one else beats me to it I'll get it
committed asap.

Your fun and games with registry dumps reminded me of a little project I
started a while ago, so I finally got it working. Just checked into
trunk, registry-tool is a gtk/gnome app that will give you a tree view
of the device's registry. Might make it an editor one day, but for now
at least it finally works.

Mark

[David Eriksson]

Post by Guido Diepen
Hi,
the supplied patch is against the current SVN (3074) as mentioned in the file.
I think there was a bug in the rapi_buffer_read_string. It would always read
one more byte then the actual size of the letters, to ensure a \0 terminator.

It seems like the only other function that uses rapi_buffer_read_string.
is CeGetSpecialFolderPath. Have you checked that it still works properly
after the patch?

Post by Guido Diepen
Unfortunately, with the EnumKey this always went ok, with the EnumValue that
is now implemented, more data was behind the string, resulting in an
unterminated string and problems with reading the later values.
Furthermore, I updated the synce-registry tool to make a complete dump of the
registry to screen. This takes about 4.5 minutes on my machine and the
resulting file with all information is about 1.3 MB
Two options have been added to synce-registry: -D and -L
synce-registry -D
does a complete dump of the registry
the -L option turns on recursion for key_listing, meaning you can do a
synce-registry -l HKLM "Software\Microsoft" -L
(to avoid the 4.5 minutes all of the time :) )
Hope the supplied patch is useful for everybody.

Nice! Great work!

\David
--

[Mark Ellis]

Post by David Eriksson

Post by Guido Diepen
Hi,
the supplied patch is against the current SVN (3074) as mentioned in the file.
I think there was a bug in the rapi_buffer_read_string. It would always read
one more byte then the actual size of the letters, to ensure a \0 terminator.

It seems like the only other function that uses rapi_buffer_read_string.
is CeGetSpecialFolderPath. Have you checked that it still works properly
after the patch?

Post by Guido Diepen
Unfortunately, with the EnumKey this always went ok, with the EnumValue that
is now implemented, more data was behind the string, resulting in an
unterminated string and problems with reading the later values.
Furthermore, I updated the synce-registry tool to make a complete dump of the
registry to screen. This takes about 4.5 minutes on my machine and the
resulting file with all information is about 1.3 MB
Two options have been added to synce-registry: -D and -L
synce-registry -D
does a complete dump of the registry
the -L option turns on recursion for key_listing, meaning you can do a
synce-registry -l HKLM "Software\Microsoft" -L
(to avoid the 4.5 minutes all of the time :) )
Hope the supplied patch is useful for everybody.

Nice! Great work!
\David

Guido, I cant get the patch to build cleanly at the moment, getting some
undeclared vars, I'll try and look at it soon. Or David, if you could
have a look ?

Mark

[Guido Diepen]

The previous patch was created from wrong directory and it contains errors
that prohibited it from compiling.

Disadvantage of working with multiple screens, where I thought I was working
in same directory.

Attached to this email you will find the patch that does not break the
compilation :) I have tested this within the same screen, make clean ; make ;
make install and then ran the dump test again. I can assure, it works now :)

Furthermore, to answer the question of David, Yes, that function still
works :) (Just for being sure, ran a test on this one also). Besides that,
this function does not appear to be used a lot.

I am really sorry for all the trouble..

Guido Diepen

[Mark Ellis]

Post by Guido Diepen
The previous patch was created from wrong directory and it contains errors
that prohibited it from compiling.
Disadvantage of working with multiple screens, where I thought I was working
in same directory.
Attached to this email you will find the patch that does not break the
compilation :) I have tested this within the same screen, make clean ; make ;
make install and then ran the dump test again. I can assure, it works now :)
Furthermore, to answer the question of David, Yes, that function still
works :) (Just for being sure, ran a test on this one also). Besides that,
this function does not appear to be used a lot.
I am really sorry for all the trouble..
Guido Diepen

Aha, ok that applies easier and compiles cleanly, but I get errors with
the new synce-registry tool.

Top of my head, I'm on WM2003, which if I understand rapi correctly uses
the version 1 calls. Would your changes have affected these badly ?

Mark

[Guido Diepen]

Post by Mark Ellis
Aha, ok that applies easier and compiles cleanly, but I get errors with
the new synce-registry tool.

What kind of errors?

Post by Mark Ellis
Top of my head, I'm on WM2003, which if I understand rapi correctly uses
the version 1 calls. Would your changes have affected these badly ?

You are correct, from wm5 on automatically the rapi2 ops are used.

I looked into the code, but rapi_buffer_read_string is not used in rapi, only
in rapi2.

The only thing that I can think of is that handling certain NULL parameters,
for values you are not interested in goes wrong.

Does anybody know whether what is actually sent over the wire differs a lot
between rapi and rapi2?

If you could provide me some info of what exactly goes wrong I might be able
to figure out what is causing the errors in your case. (Problem is that I
don't have WM2003 device, only WM6....)

Guido Diepen

[Mark Ellis]

Post by Guido Diepen

Post by Mark Ellis
Aha, ok that applies easier and compiles cleanly, but I get errors with
the new synce-registry tool.

What kind of errors?

Post by Mark Ellis
Top of my head, I'm on WM2003, which if I understand rapi correctly uses
the version 1 calls. Would your changes have affected these badly ?

You are correct, from wm5 on automatically the rapi2 ops are used.
I looked into the code, but rapi_buffer_read_string is not used in rapi, only
in rapi2.
The only thing that I can think of is that handling certain NULL parameters,
for values you are not interested in goes wrong.
Does anybody know whether what is actually sent over the wire differs a lot
between rapi and rapi2?
If you could provide me some info of what exactly goes wrong I might be able
to figure out what is causing the errors in your case. (Problem is that I
don't have WM2003 device, only WM6....)

Garbage mainly, I'll do a proper test tomorrow, and have a trawl myself.
Between us we've got a good test platform. I've been wanting an excuse
to dig into librapi :)

Mark

[Mark Ellis]

Post by Guido Diepen

Post by Mark Ellis
Aha, ok that applies easier and compiles cleanly, but I get errors with
the new synce-registry tool.

What kind of errors?

Post by Mark Ellis
Top of my head, I'm on WM2003, which if I understand rapi correctly uses
the version 1 calls. Would your changes have affected these badly ?

You are correct, from wm5 on automatically the rapi2 ops are used.
I looked into the code, but rapi_buffer_read_string is not used in rapi, only
in rapi2.
The only thing that I can think of is that handling certain NULL parameters,
for values you are not interested in goes wrong.
Does anybody know whether what is actually sent over the wire differs a lot
between rapi and rapi2?
If you could provide me some info of what exactly goes wrong I might be able
to figure out what is causing the errors in your case. (Problem is that I
don't have WM2003 device, only WM6....)

Short answer is error code 31 from CeRegEnumValue in list_key() using
synce-registry tool.

I think there may actually be a problem with CeRegQueryInfoKey and
CeRegEnumValue (in registry tool) in rapi1, the results I got back from
a bit of testing were unexpected. In fact I've written myself a reminder
comment in my gtk registry tool about CeRegEnumValue not being reliable.

I've had a quick look, I don't really know enough about rapi though,
want to give me any tips from when you fixed the rapi2 versions ?

Mark

[Guido Diepen]

Post by Mark Ellis

Post by Guido Diepen
If you could provide me some info of what exactly goes wrong I might be
able to figure out what is causing the errors in your case. (Problem is
that I don't have WM2003 device, only WM6....)

Short answer is error code 31 from CeRegEnumValue in list_key() using
synce-registry tool.

Is enumKey working without a problem and the problem only occurs with
enumValue? Could you test that one? Because enumkey was also already
implemented in rapi2, I only worked on enumValue and QueryInfo.

Post by Mark Ellis
I think there may actually be a problem with CeRegQueryInfoKey and
CeRegEnumValue (in registry tool) in rapi1, the results I got back from
a bit of testing were unexpected. In fact I've written myself a reminder
comment in my gtk registry tool about CeRegEnumValue not being reliable.

As mentioned before, the thing that struck me is the fact that you write all
kinds of things with optional things etc. Furthermore, you send things like
the actual lpszValueName over the line (line 225 and 226 in registry.c). The
idea I had when implementing the function in rapi2 is that you only want to
tell what the size of buffers is that you have such that the device knows
whether to send you the information or not. Maybe not even that. It could be
that you only have to supply the handle to the key (and optionally the
dwIndex) and that you will always get all information. I would have to test
some more things for that also.

Post by Mark Ellis
I've had a quick look, I don't really know enough about rapi though,
want to give me any tips from when you fixed the rapi2 versions ?

Hope you understood the above lines :) I want to point you to the extra debug
routines I implemetned in rapi_buffer.c though:
rapi_buffer_debug_dump_buffer
and
rapi_buffer_debug_dump_buffer_from_current_point

You supply it with either the context->recv_buffer or the context->send_buffer
and a char* as a description and it will print out the current recv or send
buffer. This helped me a lot to figure out the problems with the read_string
method. The second method only is useful with the receiving buffer, since you
can call this after each rapi_buffer_read_* and it will dump the buffer from
the active index.

Regards,

Guido Diepen

[Mark Ellis]

Post by Guido Diepen

Post by Mark Ellis

Post by Guido Diepen
If you could provide me some info of what exactly goes wrong I might be
able to figure out what is causing the errors in your case. (Problem is
that I don't have WM2003 device, only WM6....)

Short answer is error code 31 from CeRegEnumValue in list_key() using
synce-registry tool.

Is enumKey working without a problem and the problem only occurs with
enumValue? Could you test that one? Because enumkey was also already
implemented in rapi2, I only worked on enumValue and QueryInfo.

I think I'll have to knock up a little utility to test all these
extensively.

Post by Guido Diepen

Post by Mark Ellis
I think there may actually be a problem with CeRegQueryInfoKey and
CeRegEnumValue (in registry tool) in rapi1, the results I got back from
a bit of testing were unexpected. In fact I've written myself a reminder
comment in my gtk registry tool about CeRegEnumValue not being reliable.

As mentioned before, the thing that struck me is the fact that you write all
kinds of things with optional things etc. Furthermore, you send things like
the actual lpszValueName over the line (line 225 and 226 in registry.c). The
idea I had when implementing the function in rapi2 is that you only want to
tell what the size of buffers is that you have such that the device knows
whether to send you the information or not. Maybe not even that. It could be
that you only have to supply the handle to the key (and optionally the
dwIndex) and that you will always get all information. I would have to test
some more things for that also.

It looks a little odd to me as well, unfortunately I have no idea what
the device expects from these calls besides what is in the code. Do you
know of any reference material relating to what should be sent over the
wire ?

Post by Guido Diepen

Post by Mark Ellis
I've had a quick look, I don't really know enough about rapi though,
want to give me any tips from when you fixed the rapi2 versions ?

Hope you understood the above lines :) I want to point you to the extra debug
rapi_buffer_debug_dump_buffer
and
rapi_buffer_debug_dump_buffer_from_current_point
You supply it with either the context->recv_buffer or the context->send_buffer
and a char* as a description and it will print out the current recv or send
buffer. This helped me a lot to figure out the problems with the read_string
method. The second method only is useful with the receiving buffer, since you
can call this after each rapi_buffer_read_* and it will dump the buffer from
the active index.

I'll definitely keep them in mind, good idea. What I could really do
with is a crystal ball telling me what to send :)

Mark

[Guido Diepen]

Post by Mark Ellis
Garbage mainly, I'll do a proper test tomorrow, and have a trawl myself.
Between us we've got a good test platform. I've been wanting an excuse
to dig into librapi :)

One of the main differences I see in the code of rapi/registry.c and
rapi2/registry2.c is that within registry.c it looks like you are sending
the complete buffer over the line, while within registry2 I don't. (You
can see this clearly with the CeRegQueryInfoKey(2). Though it would not
explain why till now it has been working because I am not doing any weird
things extra within the synce-registry.c.

Please let me know when you have a patched version, I can check then
whether your new stuff breaks my new stuff :)

Guido Diepen

[Mark Ellis]

Post by Guido Diepen

Post by Mark Ellis
Garbage mainly, I'll do a proper test tomorrow, and have a trawl myself.
Between us we've got a good test platform. I've been wanting an excuse
to dig into librapi :)

One of the main differences I see in the code of rapi/registry.c and
rapi2/registry2.c is that within registry.c it looks like you are sending
the complete buffer over the line, while within registry2 I don't. (You
can see this clearly with the CeRegQueryInfoKey(2). Though it would not
explain why till now it has been working because I am not doing any weird
things extra within the synce-registry.c.
Please let me know when you have a patched version, I can check then
whether your new stuff breaks my new stuff :)

Hi Guido

Committed your patch with regards to the library functions, I'm sure
what I'm seeing is nothing to do with them.

I've held back the improvements to the synce-registry tool until we
figure out what the problem with rapi1 is, don't want to break it
completely for legacy devices in the meantime.

Ta
Mark

[Guido Diepen]

Hi Mark,

Post by Mark Ellis
Committed your patch with regards to the library functions, I'm sure
what I'm seeing is nothing to do with them.

Last night I was just thinking over what would need to be sent over the line
from the computer to the device and back. Unfortunately I don't have a
Windows machine and I don't have the WM2003 machine, but probably I have some
time today in the afternoon. I might test the ideas. If they work out for the
rapi2 calls (and would make them simpler) you could try the same idea for the
rapi 1 calls and see if they work out on your device also. I'll keep you
posted on that.

Furthermore, did you already try to just copy the new rapi2 code to the rapi1
for both the QueryInfo and the EnumValue (keep in mind that you need the
context_begin_command code to be the rapi1 style though!)?

Post by Mark Ellis
I've held back the improvements to the synce-registry tool until we
figure out what the problem with rapi1 is, don't want to break it
completely for legacy devices in the meantime.

Happy that you have a legacy device, otherwise this would not have shown up :)

Guido Diepen

[Mark Ellis]

Post by Guido Diepen
Hi Mark,

Post by Mark Ellis
Committed your patch with regards to the library functions, I'm sure
what I'm seeing is nothing to do with them.

Last night I was just thinking over what would need to be sent over the line
from the computer to the device and back. Unfortunately I don't have a
Windows machine and I don't have the WM2003 machine, but probably I have some
time today in the afternoon. I might test the ideas. If they work out for the
rapi2 calls (and would make them simpler) you could try the same idea for the
rapi 1 calls and see if they work out on your device also. I'll keep you
posted on that.
Furthermore, did you already try to just copy the new rapi2 code to the rapi1
for both the QueryInfo and the EnumValue (keep in mind that you need the
context_begin_command code to be the rapi1 style though!)?

I had a quick go doing that, but I think in my lack of time I got it
wrong, I'll definitely try it again.

Mark

[Guido Diepen]

Hi Mark,

Post by Mark Ellis

Post by Guido Diepen

Post by Mark Ellis
Committed your patch with regards to the library functions, I'm sure
what I'm seeing is nothing to do with them.

Last night I was just thinking over what would need to be sent over the
line from the computer to the device and back. Unfortunately I don't have
a Windows machine and I don't have the WM2003 machine, but probably I
have some time today in the afternoon. I might test the ideas. If they
work out for the rapi2 calls (and would make them simpler) you could try
the same idea for the rapi 1 calls and see if they work out on your
device also. I'll keep you posted on that.
Furthermore, did you already try to just copy the new rapi2 code to the
rapi1 for both the QueryInfo and the EnumValue (keep in mind that you
need the context_begin_command code to be the rapi1 style though!)?

I had a quick go doing that, but I think in my lack of time I got it
wrong, I'll definitely try it again.
Mark

Attached you will find a new small patch against the current SVN tree. It
contains the small fix for the _debug_ routines in rapi_buffer.h.
Furthermore, still the patches for synce-registry, it is just present when I
run svn diff :)

Also there is a small update to the code of EnumValue2. I decided to look a
bit more at the actual stuff that is sent over the line and could determine
some patterns :) Within the registry2.c file I added some much needed
comments, which describe the process is far as I understand it. I did quite
some testing with sending different parameters etc and so far all testing
confirms the ideas described in the comments with EnumValue2.

Please try to copy the complete contents of enumvalue2 once to enumvalue and
only update the
rapi_context_begin_command(context, 0x34);
to
rapi_context_begin_command(context, 0x23);
due to the different code number within rapi and rapi2. It would be really
weird to me if Microsoft changed the complete protocol while going from rapi
to rapi2.

Hope the attached file provides some help with finding out why rapi1 does not
handle the EnumValue well.

Regards,

Guido Diepen

[David Eriksson]

Post by Guido Diepen
due to the different code number within rapi and rapi2. It would be really
weird to me if Microsoft changed the complete protocol while going from rapi
to rapi2.

IIRC, the protocol has changed a bit and it is simpler encoded in RAPI2
compared to the old RAPI.

\David
--

[Guido Diepen]

Post by David Eriksson

Post by Guido Diepen
due to the different code number within rapi and rapi2. It would be
really weird to me if Microsoft changed the complete protocol while going
from rapi to rapi2.

IIRC, the protocol has changed a bit and it is simpler encoded in RAPI2
compared to the old RAPI.

If I look at some of the code of rapi1 compared to rapi2 I can imagine that. I
would like to help and get this working though, but it would mean I would
need some wire dumps of what exactly is sent over the line.
If someone could provide me that information, I would be happy to try and
figure out what exactly goes wrong in the current rapi1 code.

Guido Diepen

[Mark Ellis]

Post by Guido Diepen

Post by David Eriksson

Post by Guido Diepen
due to the different code number within rapi and rapi2. It would be
really weird to me if Microsoft changed the complete protocol while going
from rapi to rapi2.

IIRC, the protocol has changed a bit and it is simpler encoded in RAPI2
compared to the old RAPI.

If I look at some of the code of rapi1 compared to rapi2 I can imagine that. I
would like to help and get this working though, but it would mean I would
need some wire dumps of what exactly is sent over the line.
If someone could provide me that information, I would be happy to try and
figure out what exactly goes wrong in the current rapi1 code.

Guido, not something I've ever done before, but happy to try if you give
me pointers.

David, is this how librapi was written in the first place, or did you
have some decent docs on the comms specification ?

Mark

[David Eriksson]

Post by Mark Ellis

Post by Guido Diepen

Post by David Eriksson

Post by Guido Diepen
due to the different code number within rapi and rapi2. It would be
really weird to me if Microsoft changed the complete protocol while

going

Post by David Eriksson

Post by Guido Diepen
from rapi to rapi2.

IIRC, the protocol has changed a bit and it is simpler encoded in

RAPI2

Post by David Eriksson
compared to the old RAPI.

If I look at some of the code of rapi1 compared to rapi2 I can imagine that. I
would like to help and get this working though, but it would mean I would
need some wire dumps of what exactly is sent over the line.
If someone could provide me that information, I would be happy to try and
figure out what exactly goes wrong in the current rapi1 code.

Guido, not something I've ever done before, but happy to try if you give
me pointers.
David, is this how librapi was written in the first place, or did you
have some decent docs on the comms specification ?

Two words: Reverse Engineering

If you remind me enough times I can probably describe how to implement any
RAPI/RAPI2 call... only I need to be at home and boot my MS Windows
machine to get the info... :-)

\David

[Guido Diepen]

Post by David Eriksson
Two words: Reverse Engineering

I figured that one yes....

Post by David Eriksson
If you remind me enough times I can probably describe how to implement any
RAPI/RAPI2 call... only I need to be at home and boot my MS Windows
machine to get the info... :-)

Reminder :)

My idea is that the easiest would be to connect the device and write some very
simple program within windows that does nothing but just request some data
through rapi calls. These requests should be captured. If you would supply a
list with dumps with corresponding rapi calls like for example:

CeRegQueryInfoKey with parameters: hkey, NULL, NULL, NULL, etc
sends the following over the line 31 00 00 00 25 .. .. .. ..
With the answer of the device being 00 00 00 00 00 00 00 .... .. ...

I might be able to help out with implementing the things in rapi1.
Unfortunately this is the only way I can help, since I don't have any wm2003
device, plus I don't have windows :)

For the rapi2, you mentioned that the protocol had changed a bit, especially
the encoding etc, i think i would be able to figure out some things by just
testing some code. As mentioned in some previous email, with the comments I
think that I understand at least a bit why certain things take the values
they do and what exactly is sent back and forth.

Again, if you would be able to supply me with some dumps of the data, I can
see if I can help out with getting the rapi1 stuff updated for the registry
functions also. (This is a concealed second reminder ;) )

Regars,

Guido Diepen.

[Mark Ellis]

Post by David Eriksson

Post by Mark Ellis

Post by Guido Diepen

Post by David Eriksson

Post by Guido Diepen
due to the different code number within rapi and rapi2. It would be
really weird to me if Microsoft changed the complete protocol while

going

Post by David Eriksson

Post by Guido Diepen
from rapi to rapi2.

IIRC, the protocol has changed a bit and it is simpler encoded in

RAPI2

Post by David Eriksson
compared to the old RAPI.

If I look at some of the code of rapi1 compared to rapi2 I can imagine that. I
would like to help and get this working though, but it would mean I would
need some wire dumps of what exactly is sent over the line.
If someone could provide me that information, I would be happy to try and
figure out what exactly goes wrong in the current rapi1 code.

Guido, not something I've ever done before, but happy to try if you give
me pointers.
David, is this how librapi was written in the first place, or did you
have some decent docs on the comms specification ?

Two words: Reverse Engineering
If you remind me enough times I can probably describe how to implement any
RAPI/RAPI2 call... only I need to be at home and boot my MS Windows
machine to get the info... :-)
\David

I was afraid you were going to say that...

Mark