Multithreading librapi for green threads

Discussion:

David Richardson

2009-07-11 00:06:14 UTC

I'm using librapi for some really cool testing framework stuff at Flexilis.
I'm really trying to push the limits of the library. I've wrapped librapi
in ruby using Ruby::DL but currently I have to mutex on any RAPI call. I'd
really like to only have to mutex per-device. Ruby only uses green threads
so the existing thread safe stuff isn't going to help me, I believe. I want
to run tests on hundreds of devices in parallel.

What I want to do is create alternative wrappers for each RAPI function
where I pass the context in as the last parameter, rather than specifying it
ahead of time with . I tried some initial experimentation with this
yesterday, but didn't have much luck. Before I dive too deep into it, I
wanted to make my proposal and hear any comments you have. Specifically I'm
concerned about how the contexts work and when/how they become invalidated.

Serially I will create each connection (rapi_connection_from_name), select
it (rapi_connection_select), initialize it (CeRapiInit) and store the
current context (get_current_context). Then in parallel I will call
modified RAPI functions on hundreds of devices that take the context in as a
paramater.

I notice that the act of selecting a connection unref's the previously
selected connection and destroys it (since this was the only reference). Is
it sufficient for me to just store a reference to each context and call
rapi_context_ref (and unref when I'm done)? Is there anything else that
would invalidate the previous context?

I'll hopefully be experimenting with this sometime next week. Is this a
change that others would be interested in contributing toward librapi's
trunk? Do you have a suggestion of a better way to go about this? Is
there maybe some existing functionality that I'm missing that already allows
this?

Thanks!
-David Richardson
Flexilis Mobile Security

Mark Ellis

2009-07-11 09:35:10 UTC

Permalink

I've never been a big user of multi-threading, so I can't really comment
on types of threads and how they'll be affected, but I'll try and
explain what's going on. It sounds like you're using svn, which is good,
because I've been trying to make it a bit more consistent. If I've
failed, please tell me :)

Post by David Richardson
What I want to do is create alternative wrappers for each RAPI
function where I pass the context in as the last parameter, rather
than specifying it ahead of time with . I tried some initial
experimentation with this yesterday, but didn't have much luck.
Before I dive too deep into it, I wanted to make my proposal and hear
any comments you have. Specifically I'm concerned about how the
contexts work and when/how they become invalidated.

I like the idea, and it would in many ways be simpler to use and more
robust. The current interface is obviously modelled on the MS interface,
and it looks like MS never considered multiple devices in any way. So I
don't know which way is best for this, an interface consistent with MS
or a better one ?

Post by David Richardson
Serially I will create each connection (rapi_connection_from_name),
select it (rapi_connection_select), initialize it (CeRapiInit) and
store the current context (get_current_context). Then in parallel I
will call modified RAPI functions on hundreds of devices that take the
context in as a paramater.

Firstly, don't use the context directly, just store and select the
connection, it'll be easier.

I'm assuming this is where the threading comes in. The only time you'll
have a problem with threading is if you pass the same connection into
two different threads, in which case you'll need mutexes because the
threads might use the socket at the same time. But you'd have the same
problem passing the context directly in two different threads as well.
Note you should be able to use two different connections to the same
device in different threads, because they use separate buffers and
sockets.

How this works with green threads I have no idea.

Post by David Richardson
I notice that the act of selecting a connection unref's the previously
selected connection and destroys it (since this was the only
reference). Is it sufficient for me to just store a reference to each
context and call rapi_context_ref (and unref when I'm done)? Is there
anything else that would invalidate the previous context?

Yes, just ref it. The reference system was based on the use of the
rapi_connection_ functions.

If you're only using one connection, ie just calling CeRapiInit and
uninit, init creates a context and makes it the current, which refs it,
and uninit unrefs it, which destroys it.

If you create a connection, it creates a context and refs it. When the
context becomes the current it gets ref'd, and unref'd when it is no
longer the current, but it still has the initial ref from the
connection, so is not destroyed.

Post by David Richardson
I'll hopefully be experimenting with this sometime next week. Is this
a change that others would be interested in contributing toward
librapi's trunk? Do you have a suggestion of a better way to go
about this? Is there maybe some existing functionality that I'm
missing that already allows this?

If you can use the native threading then you should be able to do this
quite easily, otherwise you've got some work to do.

If a change in the API along these lines was to go into trunk, in my
opinion it should actually be the other way around which would be much
more work. Let me explain that last point. Ideally the core API should
be changed so the context is passed in as the first parameter, in
typical OO style. The MS based API with no context should then be a
wrapper around the better API, with the wrapper getting the current
context as it does now and passing it into the new API. Make sense ? Of
course that involves rewriting a lot of the library :)

I hope I've been helpful.

Mark

David Eriksson

2009-07-11 11:35:51 UTC

Permalink

[Sorry for dupes of this.]

Hi,

First of all, it's wonderful to hear that you make good use of SynCE. I
would really appreciate if you are able share some more about the
context you use SynCE in, either to the list or privately. (It's great
to see one's kids grow up

I think that you should model the new RAPI interface after the Microsoft
RAPI2 Interface (IRAPIDevice, IRAPISession, etc):

http://msdn.microsoft.com/en-us/library/aa922133.aspx

MS buils these interfaces on top of the Component Object Model (COM) so
C++ would be a natural fit, but if you implement it in C for easier Ruby
bindings that's fine. Just add "this" (or "self" :-) as first parameter.
(I would have implemented the original librapi2 in C++ if it wasn't
because I hoped that language bindings would be easier with a C
interface.)

In the C interface I suggest that you do like this, or something close
that makes it easy for the Ruby bindings:

HRESULT RAPIDevice_CreateSession(
RAPIDevice* self;
RAPISession** ppISession
);

BOOL RAPISession_CeCopyFile(
RAPISession* self,
LPCWSTR lpExistingFileName,
LPCWSTR lpNewFileName,
BOOL bFailIfExists
);

Locking (per session) should really be done inside the RAPI calls, not
outside.

Reference counting could be implemented just like AddRef and Release
from IUnknown in COM... :-)

Best regards,

David Eriksson, http://www.divideandconquer.se/

Post by David Richardson
I'm using librapi for some really cool testing framework stuff at
Flexilis. I'm really trying to push the limits of the library. I've
wrapped librapi in ruby using Ruby::DL but currently I have to mutex
on any RAPI call. I'd really like to only have to mutex per-device.
Ruby only uses green threads so the existing thread safe stuff isn't
going to help me, I believe. I want to run tests on hundreds of
devices in parallel.
What I want to do is create alternative wrappers for each RAPI
function where I pass the context in as the last parameter, rather
than specifying it ahead of time with . I tried some initial
experimentation with this yesterday, but didn't have much luck.
Before I dive too deep into it, I wanted to make my proposal and hear
any comments you have. Specifically I'm concerned about how the
contexts work and when/how they become invalidated.
Serially I will create each connection (rapi_connection_from_name),
select it (rapi_connection_select), initialize it (CeRapiInit) and
store the current context (get_current_context). Then in parallel I
will call modified RAPI functions on hundreds of devices that take the
context in as a paramater.
I notice that the act of selecting a connection unref's the previously
selected connection and destroys it (since this was the only
reference). Is it sufficient for me to just store a reference to each
context and call rapi_context_ref (and unref when I'm done)? Is there
anything else that would invalidate the previous context?
I'll hopefully be experimenting with this sometime next week. Is this
a change that others would be interested in contributing toward
librapi's trunk? Do you have a suggestion of a better way to go
about this? Is there maybe some existing functionality that I'm
missing that already allows this?
Thanks!
-David Richardson
Flexilis Mobile Security

Mark Ellis

2009-07-12 16:46:17 UTC

Permalink

Post by David Eriksson
[Sorry for dupes of this.]
Hi,
First of all, it's wonderful to hear that you make good use of SynCE. I
would really appreciate if you are able share some more about the
context you use SynCE in, either to the list or privately. (It's great
to see one's kids grow up
I think that you should model the new RAPI interface after the Microsoft
http://msdn.microsoft.com/en-us/library/aa922133.aspx
MS buils these interfaces on top of the Component Object Model (COM) so
C++ would be a natural fit, but if you implement it in C for easier Ruby
bindings that's fine. Just add "this" (or "self" :-) as first parameter.
(I would have implemented the original librapi2 in C++

I'm glad you didn't, I probably wouldn't have bothered fiddling with
it :)

Post by David Eriksson
if it wasn't
because I hoped that language bindings would be easier with a C
interface.)
In the C interface I suggest that you do like this, or something close
HRESULT RAPIDevice_CreateSession(
RAPIDevice* self;
RAPISession** ppISession
);
BOOL RAPISession_CeCopyFile(
RAPISession* self,
LPCWSTR lpExistingFileName,
LPCWSTR lpNewFileName,
BOOL bFailIfExists
);

David, this is a magnificent idea ! And it's relatively easy to
implement I think.

David (Richardson), if this sounds of interest, here's how I would see
this working.

Currently all rapi calls start in rapi_indirection.c, where the current
context is fetched, and used to call the appropriate function, in rapi/
or rapi2 subdirs, through the rapi_ops struct. Initially we change the
"real" functions in these subdirs to accept the context as the first
parameter, which we pass through from the call in rapi_indirection.c. At
this point the original API is intact, but we have a working body of
functions that accept any context we pass.

Now it should be possible to implement a new public API in the manner
David shows, that can call these functions using the context from the
RapiSession pointer we pass. Does that make sense ?

I think I'm going to give this a go, though I want to do another release
first, so it won't be straight away.

Post by David Eriksson
Locking (per session) should really be done inside the RAPI calls, not
outside.

Yes, I think each context needs internal locking, but I'm not sure how
this should be done. Can we just use pthread mutexes ?

Post by David Eriksson
Reference counting could be implemented just like AddRef and Release
from IUnknown in COM... :-)
Best regards,
David Eriksson, http://www.divideandconquer.se/

------------------------------------------------------------------------------
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/Challenge
_______________________________________________
SynCE-Devel mailing list
https://lists.sourceforge.net/lists/listinfo/synce-devel

David Eriksson

2009-07-12 20:21:23 UTC

Permalink

On Sun, 2009-07-12 at 17:46 +0100, Mark Ellis wrote:

[snip]

Post by Mark Ellis

Post by David Eriksson
MS buils these interfaces on top of the Component Object Model (COM) so
C++ would be a natural fit, but if you implement it in C for easier Ruby
bindings that's fine. Just add "this" (or "self" :-) as first parameter.
(I would have implemented the original librapi2 in C++

I'm glad you didn't, I probably wouldn't have bothered fiddling with
it :)

Great, because I'm very happy that you fiddle with SynCE!

Post by Mark Ellis

Post by David Eriksson
if it wasn't
because I hoped that language bindings would be easier with a C
interface.)

[snap]

Post by Mark Ellis

Post by David Eriksson
Locking (per session) should really be done inside the RAPI calls,

not

Post by David Eriksson
outside.

Yes, I think each context needs internal locking, but I'm not sure how
this should be done. Can we just use pthread mutexes ?

Yes pthread mutexes is the way to go. Essentially all socket access in a
function should be within a single "critical section" protected by a
mutex.

If you want some autofoo exercising you could figure out how to build
two versions of librapi2, where librapi2_r.so require pthreads and use
mutexes to be thread-safe.

Best regards,

David Eriksson, http://www.divideandconquer.se/

David Richardson

2009-07-13 19:30:33 UTC

Permalink

Post by Mark Ellis

Sure thing. So Flexilis is a mobile security company that cares a lot about
testing. Writing cross-platform mobile software takes a great deal of care
considering there are so many different OS's, resolutions and input devices
(keyboard, touch screen, etc). We use a continuous integration server and
already have some great tests (in the form of self-contained exe's). Not
satisfied simply running unit/integration tests inside the WM emulators, we
also deploy them to all the development devices we have. SynCE makes this
possible, as it allows us to have one linux server with hundreds of USB
devices attached to it. I use Ruby::DL wrapping librapi.so to copy the
executable over, run it, wait for it to complete and read the output file
and report back to the CI server the results (what failed on what devices).
This is all up and running and working great!

To run acceptance/gui tests, I've created a device side DLL that wraps many
of the WM coredll methods. This is where the need for CeRapiInvoke came
into play. Using my device side DLL, with bindings to librapi, I'm able to
write GUI tests in ruby lightning fast. To give you an idea of the power,
I'll include a short little program here :)

phones.each do |phone|
phone.copy_to("MyCab.cab", "/Windows/AppMgr/Install/MyCab.cab")
phone.key_press(HOME)
phone.execute("wceload.exe")
assert(phone.get_foreground_window.get_window_text == "Installation")
phone.push_button(OK)
end

Of course, this is just a sample but I think you get the idea.

Post by Mark Ellis

Post by David Eriksson
I think that you should model the new RAPI interface after the Microsoft
http://msdn.microsoft.com/en-us/library/aa922133.aspx
MS buils these interfaces on top of the Component Object Model (COM) so
C++ would be a natural fit, but if you implement it in C for easier Ruby
bindings that's fine. Just add "this" (or "self" :-) as first parameter.
(I would have implemented the original librapi2 in C++

I'm glad you didn't, I probably wouldn't have bothered fiddling with
it :)

David, this is a magnificent idea ! And it's relatively easy to
implement I think.
David (Richardson), if this sounds of interest, here's how I would see
this working.
Currently all rapi calls start in rapi_indirection.c, where the current
context is fetched, and used to call the appropriate function, in rapi/
or rapi2 subdirs, through the rapi_ops struct. Initially we change the
"real" functions in these subdirs to accept the context as the first
parameter, which we pass through from the call in rapi_indirection.c. At
this point the original API is intact, but we have a working body of
functions that accept any context we pass.
Now it should be possible to implement a new public API in the manner
David shows, that can call these functions using the context from the
RapiSession pointer we pass. Does that make sense ?

Well now I'm extremely glad that I decided to bounce this off the list first
before giving it a go. This insight will make things a lot easier!

Post by Mark Ellis
I think I'm going to give this a go, though I want to do another release
first, so it won't be straight away.

Post by David Eriksson
Locking (per session) should really be done inside the RAPI calls, not
outside.

Yes, I think each context needs internal locking, but I'm not sure how
this should be done. Can we just use pthread mutexes ?

Post by David Eriksson
Reference counting could be implemented just like AddRef and Release
from IUnknown in COM... :-)
Best regards,
David Eriksson, http://www.divideandconquer.se/

------------------------------------------------------------------------------

Post by David Eriksson
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full

prize

Post by David Eriksson
details at: http://p.sf.net/sfu/Challenge
_______________________________________________
SynCE-Devel mailing list
https://lists.sourceforge.net/lists/listinfo/synce-devel