next up previous
Next: Applications Up: Creating and managing threads Previous: First approach

Subsections

   
Second approach

While the approach outlined above works satisfactorily, it has one serious drawback: the programmer has to provide the data types of the arguments of the member function himself. While this seems to be a simple task, in practice it is often not, as will be explained in the sequel.

To expose the problem, we take an example from one of our application programs where we would like to call the function

    template <int dim>
    void DoFHandler<dim>::distribute_dofs (const FiniteElement<dim> &,
                                           const unsigned int);
on a new thread. Correspondingly, we would need to use
    MemFunData2<DoFHandler<dim>, const FiniteElement<dim> &, unsigned int>
        mem_fun_data (dof_handler, fe, 0,
                      &DoFHandler<dim>::distribute_dofs);
to encapsulate the parameters. However, if one forgets the const specifier on the second template parameter, one receives the following error message (using gcc 2.95.2):
  test.cc: In method `void InterstepData<2>::wake_up(unsigned int, Interst
  epData<2>::PresentAction)':
  test.cc:683:   instantiated from here
  test.cc:186: no matching function for call to `ThreadManager::Mem_Fun_Da
  ta2<DoFHandler<2>,FiniteElement<2> &,unsigned int>::MemFunData2 (DoFHa
  ndler<2> *, const FiniteElement<2> &, int, void (DoFHandler<2>::*)(const
   FiniteElement<2> &, unsigned int))'
  /home/atlas1/wolf/program/newdeal/deal.II/base/include/base/thread_manag
  er.h:470: candidates are: ThreadManager::MemFunData2<DoFHandler<2>,Fin
  iteElement<2> &,unsigned int>::MemFunData2(DoFHandler<2> *, FiniteElem
  ent<2> &, unsigned int, void * (DoFHandler<2>::*)(FiniteElement<2> &, un
  signed int))
  /home/atlas1/wolf/program/newdeal/deal.II/base/include/base/thread_manag
  er.h:480:                 ThreadManager::MemFunData2<DoFHandler<2>,Fin
  iteElement<2> &,unsigned int>::MemFunData2(DoFHandler<2> *, FiniteElem
  ent<2> &, unsigned int, void (DoFHandler<2>::*)(FiniteElement<2> &, unsi
  gned int))
  /home/atlas1/wolf/program/newdeal/deal.II/base/include/base/thread_manag
  er.h:486:                 ThreadManager::MemFunData2<DoFHandler<2>,Fin
  iteElement<2> &,unsigned int>::MemFunData2(const ThreadManager::Mem_Fu
  n_Data2<DoFHandler<2>,FiniteElement<2> &,unsigned int> &)

While the compiler is certainly right to complain, the message is not very helpful. Furthermore, since interfaces to functions sometimes change, for example by adding additional default parameters that do not show up in usual code, programs that used to compile do no more so with messages as shown above.

Due to the lengthy and complex error messages, even very experienced programmers usually need between five and ten minutes until they get an expression like this correct. In most cases, they don't get it right in the first attempt, so the time used for the right declaration dominates the whole setup of starting a new thread. To circumvent this bottleneck at least in most cases, we chose to implement a second strategy at encapsulating the parameters of member functions. This is done in several steps: first let the compiler find out about the right template parameters, then encapsulate the parameters, use the objects, and finally solve some technical problems with virtual constructors and locking of destruction. We will treat these steps sequentially in the following.

Finding the correct template parameters.

C++ offers the possibility of templatized functions that deduce their template arguments themselves. In fact, we have used them in the ThreadManager::spawn function in Code Sample 7 already. Here, this can be used as follows: assume we have a function encapsulation class
    template <typename Class, typename Arg1, typename Arg2>
    class MemFunData { ... };
as above, and a function
    template <typename Class, typename Arg1, typename Arg2>
    MemFunData<Class,Arg1,Arg2>
    encapsulate (void (Class::*mem_fun_ptr)(Arg1, Arg2)) {
      return MemFunData<Class,Arg1,Arg2> (mem_fun_ptr);
    };
Then, if we call this function with the test class of Code Sample 2 like this:
    encapsulate (&TestClass::test_function);
it can unambiguously determine the template parameters to be Class=TestClass, Arg1=int, Arg2=double.

Encapsulating the parameters.

We should not try to include the argument values for the new thread right away, for example by declaring encapsulate like this:
    template <typename Class, typename Arg1, typename Arg2>
    MemFunData<Class,Arg1,Arg2>
    encapsulate (void (Class::*mem_fun_ptr)(Arg1, Arg2),
                 Arg1  arg1,
                 Arg2  arg2,
                 Class object) {
      return MemFunData<Class,Arg1,Arg2> (mem_fun_ptr, object, arg1, arg2);
    };
The reason is that for template functions, no parameter promotion is performed. Thus, if we called this function as in
    encapsulate (&TestClass::test_function,
                 1, 3,
                 test_object);
then the compiler would refuse this since from the function pointer it must deduce that Arg2 = double, but from the parameter ``3'' it must assume that Arg2 = int. The resulting error message would be similarly lengthy as the one shown above.

One could instead write MemFunData like this:

    template <typename Class, typename Arg1, typename Arg2>
    class MemFunData { 
      public:
        typedef void (Class::*MemFunPtr)(Arg1, Arg2);

        MemFunData (MemFunPtr mem_fun_ptr_) {
          mem_fun_ptr = mem_fun_ptr_;
        };

        void collect_args (Class *object_,
                           Arg1   arg1_,
                           Arg2   arg2_) {
          object = object_;
          arg1   = arg1_;
          arg2   = arg2_;
        };

        MemFunPtr  mem_fun_ptr;
        Class     *object;
        Arg1       arg1;
        Arg2       arg2;
    };
One would then create an object of this type including the parameters to be passed as follows:
    encapsulate(&TestClass::test_function).collect_args(test_object, 1, 3);
Here, the first function call creates an object with the right template parameters and storing the member function pointer, and the second one, calling a member function, fills in the function arguments.

Unfortunately, this way does not work: if one or more of the parameter types is a reference, then the respective reference variable needs to be initialized by the constructor, not by collect_args. It needs to be known which object the reference references at construction time, since later on only the referenced object can be assigned, not the reference itself anymore.

Since we feel that we are close to a solution, we introduce one more indirection, which indeed will be the last one:

Code sample 8  
    template <typename Class, typename Arg1, typename Arg2>
    class MemFunData { 
      public:
        typedef void (Class::*MemFunPtr)(Arg1, Arg2);

        MemFunData (MemFunPtr mem_fun_ptr_,
                      Class *object_,
                      Arg1   arg1_,
                      Arg2   arg2_) :
             mem_fun_ptr (mem_fun_ptr_),
             object      (object_),
             arg1        (arg1_),
             arg2        (arg2_)            {};

        MemFunPtr  mem_fun_ptr;
        Class     *object;
        Arg1       arg1;
        Arg2       arg2;
    };


    template <typename Class, typename Arg1, typename Arg2>
    struct ArgCollector { 
        typedef void (Class::*MemFunPtr)(Arg1, Arg2);

        ArgCollector (MemFunPtr mem_fun_ptr_) {
          mem_fun_ptr = mem_fun_ptr_;
        };

        
        MemFunData<Class,Arg1,Arg2>
        collect_args (Class *object_,
                      Arg1   arg1_,
                      Arg2   arg2_) {
          return MemFunData<Class,Arg1,Arg2> (mem_fun_ptr, object,
                                              arg1, arg2);
        };

        MemFunPtr  mem_fun_ptr;
    };


    template <typename Class, typename Arg1, typename Arg2>
    ArgCollector<Class,Arg1,Arg2>
    encapsulate (void (Class::*mem_fun_ptr)(Arg1, Arg2)) {
      return ArgCollector<Class,Arg1,Arg2> (mem_fun_ptr);
    };

Now we can indeed write for the test class of Code Sample 2:

    encapsulate(&TestClass::test_function).collect_args(test_object, 1, 3);
The first call creates an object of type ArgCollector<...> with the right parameters and storing the member function pointer, while the second call, a call to a member function of that intermediate class, generates the final object we are interested in, including the member function pointer and all necessary parameters. Since collect_args already has its template parameters fixed from encapsulate, it can convert between data types.

Using these objects.

Now we have an object of the correct type automatically generated, without the need to type in any template parameters by hand. What can we do with that? First, we can't assign it to a variable of that type, e.g. for use in several spawn commands:
  MemFunData mem_fun_data = encapsulate(...).collect_args(...);
Why? Since we would then have to write the data type of that variable by hand: the correct data type is not MemFunData as written above, but MemFunData<TestClass,int,double>. Specifying all these template arguments was exactly what we wanted to avoid. However, we can do some such thing if the variable to which we assign the result is of a type which is a base class of MemFunData<...>. Unfortunately, the data values that MemFunData<...> encapsulates depend on the template parameters, so the respective variables in which we store the values can only be placed in the derived class and could not be copied when we assign the variable to a base class object, since that does not have these variables.

What can we do here? Assume we have the following class structure:

Code sample 9  
    class FunDataBase {};

    template <...> class MemFunData : public FunDataBase 
    {  /* as above */ };

    class FunEncapsulation {
      public:
        FunEncapsulation (FunDataBase *f)
                   : fun_data_base (f) {};
        FunDataBase *fun_data_base;
    };


    template <typename Class, typename Arg1, typename Arg2>
    FunEncapsulation
    ArgCollector<Class,Arg1,Arg2>::collect_args (Class *object_,
                                                 Arg1   arg1_,
                                                 Arg2   arg2_) {
      return new MemFunData<Class,Arg1,Arg2> (mem_fun_ptr, object,
                                              arg1, arg2);
    };

Note that in the return statement of the collect_args function, first a cast from MemFunData* to FunDataBase*, and then a constructor call to FunEncapsulation :: FunEncapsulation (FunDataBase*) was performed.

In the example above, the call to encapsulate(...).collect_args(...) generates an object of type FunEncapsulation, which in turn stores a pointer to an object of type FunDataBase, here to MemFunData<...> with the correct template parameters. We can assign the result to a variable the type of which does not contain any template parameters any more, as desired:

    FunEncapsulation 
        fun_encapsulation = encapsulate (&TestClass::test_function)
                                          .collect_args(test_object, 1, 3);

But how can we start a thread with this object if we have lost the full information about the data types? This can be done as follows: add a variable to FunDataBase which contains the address of a function that knows what to do. This function is usually implemented in the derived classes, and its address is passed to the constructor:

Code sample 10  
    class FunDataBase {
      public:
        typedef void * (*ThreadEntryPoint) (void *);

        FunDataBase (ThreadEntryPoint t) :
                 thread_entry_point (t) {};

        ThreadEntryPoint thread_entry_point;
    };

    template <...>
    class MemFunData : public FunDataBase {
      public:
                 // among other things, the constructor now does this:
        MemFunData () :
                 FunDataBase (&start_thread) {};

        static void * start_thread (void *args) {
          // do the same as in Code Sample 4 above
        }
    };


    void spawn (ACE_Thread_Manager &thread_manager,
                FunEncapsulation   &fun_encapsulation) {
      thread_manager.spawn (*fun_encapsulation.fun_data_base
                                      ->thread_entry_point,
                            &fun_data_base);
    };

fun_encapsulation.fun_data_base->thread_entry_point is given by the derived class as that function that knows how to handle objects of the type which we are presently using. Thus, we can now write the whole sequence of function calls (assuming we have an object thread_manager of type ACE_Thread_Manager):
    FunEncapsulation 
        fun_encapsulation = encapsulate (&TestClass::test_function)
                                          .collect_args(test_object, 1, 3);
    spawn (thread_manager, fun_encapsulation);
This solves our problem in that no template parameters need to be specified by hand any more. The only source for lengthy compiler error messages is if the parameters to collect_args are in the wrong order or can not be casted to the parameters of the member function which we want to call. These problems, however, are much more unlikely in our experience, and are also much quicker sorted out.

Virtual constructors.

While the basic techniques have been fully developed now, there are some aspects which we still have to take care of. The basic problem here is that the FunEncapsulation objects store a pointer to an object that was created using the new operator. To prevent a memory leak, we need to destroy this object at some time, preferably in the destructor of FunEncapsulation:
    FunEncapsulation::~FunEncapsulation () {
      delete fun_data_base;
    };
However, what happens if we have copied the object before? In particular, this is always the case using the functions above: collect_args generates a temporary object of type FunEncapsulation, but there could be other sources of copies as well. If we do not take special precautions, only the pointer to the object is copied around, and we end up with stale pointers pointing to invalid locations in memory once the first object has been destroyed. What we obviously need to do when copying objects of type FunEncapsulation is to not copy the pointer but to copy the object which it points to. Unfortunately, the following copy constructor is not possible:
    FunEncapsulation::FunEncapsulation (const FunEncapsulation &m) {
      fun_data_base = new FunDataBase (*m.fun_data_base);
    };
The reason, of course, is that we do not want to copy that part of the object belonging to the abstract base class. But we can emulate something like this in the following way (this programming idiom is called ``virtual constructors''):

Code sample 11  
    class FunDataBase {
      public:
        // as above

        virtual FunDataBase * clone () const = 0;
    };

    template <...>
    class MemFunData : public FunDataBase {
      public:
        // as above

                          // copy constructor:
        MemFunData (const MemFunData<...> &mem_fun_data) {...};

                          // clone the present object, i.e.
                          // create an exact copy:
        virtual FunDataBase * clone () const {
          return new MemFunData<...>(*this);
        };
    };


    FunEncapsulation::FunEncapsulation (const FunEncapsulation &m) {
      fun_data_base = m.fun_data_base->clone ();
    };

Thus, whenever the FunEncapsulation object is copied, it creates a copy of the object it harbors (the MemFunData<...> object), and therefore always owns its copy. When the destructor is called, it is free to delete its copy without affecting other objects (from which it may have been copied, or to which it was copied). Similar to the copy constructor, we have to modify the copy operator, as well.

Spawning independent threads.

Often, one wants to spawn a thread which will have its own existence until it finishes, but is in no way linked to the creating thread any more. An example would be the following, assuming a function TestClass::compress_file(const string file_name) exists and that there is an object thread_manager not local to this function:

  
    ...
    string file_name;
    ...    // write some output to a file

    // now create a thread which runs `gzip' on that output file to reduce
    // disk space requirements. don't care about that thread any more
    // after creation, i.e. don't wait for its return
    FunEncapsulation 
        fun_encapsulation = encapsulate (&TestClass::compress_file)
                                  .collect_args(test_object, file_name);
    spawn (thread_manager, fun_encapsulation);

    // quit the present function
    return;
The problem here is that the object fun_encapsulation goes out of scope when we quit the present function, and therefore also deletes its pointer to the data which we need to start the new thread. If in this case the operating system was a bit lazy in creating the new thread, the function start_thread would at best find a pointer pointing to an object which is already deleted. Further, but this is obvious, if the function is taking references or pointers to other objects, it is to be made sure that these objects persist at least as long as the spawned thread runs.

What one would need to do here at least, is wait until the thread is started for sure, before deletion of the FunEncapsulation is allowed. To this end, we need to use a ``Mutex'', to allow for exclusive operations. A Mutex (short for mutually exclusive) is an object managed by the operating system and which can only be ``owned'' by one thread at a time. You can try to ``acquire'' a Mutex, and you can later ``release'' it. If you try to acquire it, but the Mutex is owned by another thread, then your thread is blocked until the present owner releases it. Mutices (plural of ``Mutex'') are therefore most often used to guarantee that only one thread is presently accessing some object: a thread that wants to access that object acquires a Mutex related to that object and only releases it once the access if finished; if in the meantime another thread wants to access that object as well, it has to acquire the Mutex, but since the Mutex is presently owned already, the second thread is blocked until the first one has finished its access.

Alternatively, one can use Mutices to synchronize things. We will use it for the following purpose: the Mutex is acquired by the starting thread; when later the destructor of the FunEncapsulation class (running on the same thread) is called, it tries to acquire the lock again; it will thus only continue its operations once the Mutex has been released by someone, which we do on the spawned thread once we don't need the data of the FunEncapsulation object any more and destruction is safe.

All this can then be done in the following way:

Code sample 12  
    class FunEncapsulation {
      public:
        ...       // as before
        ~FunEncapsulation ();
    };


    class FunDataBase {
      public:
        ...       // as before
        Mutex       lock;
    };

    template <typename Class, typename Arg1, typename Arg2>
    void * start_thread (void *arg_ptr) {
      MemFunData<Class,Arg1,Arg2> *mem_fun_data
            = reinterpret_cast<MemFunData *>(arg_ptr);

      // copy the data arguments:
      MemFunData<Class,Arg1,Arg2>::MemFunPtr
              mem_fun_ptr = mem_fun_data->mem_fun_ptr;
      Class * object      = mem_fun_data->object;
      Arg1    arg1        = mem_fun_data->arg1;
      Arg2    arg2        = mem_fun_data->arg2;

      // data is now copied, so the original object may be deleted:
      mem_fun_data->lock.release ();

      // now call the thread function:
      object->*mem_fun_ptr (arg1, arg2);

      return 0;
    };


    FunEncapsulation::~FunEncapsulation () {
      // wait until the data is copied by the new thread and
      // `release' is called by `start_thread':
      fun_data_base->lock.acquire ();
      // now delete the object which is no more needed
      delete fun_data_base;
    };


    void spawn (ACE_Thread_Manager  &thread_manager,
                FunEncapsulation &fun_encapsulation) {
      // lock the fun_encapsulation object
      fun_encapsulation.fun_data_base->lock.acquire ();
      thread_manager.spawn (*fun_encapsulation.fun_data_base
                                      ->thread_entry_point,
                            &fun_data_base);
    };

When we call spawn, we set a lock on the destruction of the FunEncapsulation object just before we start the new thread. This lock is only released when inside the new thread (i.e. inside the start_thread function) all arguments have been copied to a safe place. Now we have local copies and don't need the ones from the fun_encapsulation object any more, which we indicate by releasing the lock. Inside the destructor of that object, we wait until we can obtain the lock, which is only after it has been released by the newly started thread; after having waited till this moment, the destruction can go on safely, and we can exit the function from which the thread was started, if we like so.

The scheme just described also works if we start multiple threads using only one object of type FunEncapsulation:

    FunEncapsulation 
        fun_encapsulation = encapsulate (&TestClass::test_function)
                                  .collect_args(test_object, arg_value);
    spawn (thread_manager, fun_encapsulation);
    spawn (thread_manager, fun_encapsulation);

    // quit the present function
    return;
Here, when starting the second thread the spawn function has to wait until the newly started first thread has released its lock on the object; however, this delay is small and should not pose a noticeable problem. Thus, no special treatment of this case is necessary, and we can in a simple way emulate the spawn_n function provided by most operating systems, which spawns several new threads at once:
    void spawn_n (ACE_Thread_Manager &thread_manager,
                  FunEncapsulation   &fun_encapsulation,
                  const unsigned int  n_threads) {
      for (unsigned int i=0; i<n_threads; ++i)
        spawn (thread_manager, fun_encapsulation);
    };
A direct support of the spawn_n function of the operating system would be difficult, though, since each of the new threads would call lock.release(), even though the lock was only acquired once.

Since we have now made sure that objects are not deleted too early, even the following sequence is possible, which does not involve any named variables at all, only a temporary one, which immediately released after the call to spawn:

Code sample 13  
    spawn (thread_manager, 
           encapsulate (&TestClass::test_function)
              .collect_args(test_object, arg_value));

We most often use this very short idiom in the applications in Section 4 and in our own programs.

Number of parameters. Non-member functions.

Above, we have explained how we can define classes for a binary member function. This approach is easily extended to member functions taking any number of parameters. We simply have to write classes MemFunData0, MemFunData1, and so on, which encapsulate member functions that take zero, one, etc parameters. Likewise, we have to have classes ArgCollectorN for each number of parameters, and functions encapsulate that return an object of type ArgCollectorN. Since functions can be overloaded on their argument types, we need not call the encapsulate functions differently.

All of which has been said above can also easily be adopted to global functions or static member functions. Instead of the classes MemFunDataN we can then use classes FunDataN that are also derived from FunDataBase. The respective ArgCollector classes then collect only the arguments, not the object on which we will operate. The class, FunEncapsulation is not affected by this, nor is FunDataBase.


next up previous
Next: Applications Up: Creating and managing threads Previous: First approach
Wolfgang Bangerth
2000-04-20