queue management algorithms

[Orna Agmon Ben-Yehuda]

On Sun, Apr 7, 2013 at 7:21 PM, ik <idokan at gmail.com> wrote:

>
>
>
> On Sun, Apr 7, 2013 at 6:36 PM, Orna Agmon Ben-Yehuda <ladypine at gmail.com>wrote:
>
>> If you are looking for something reliable for many tasks scheduled at a
>> time (e.g., one user may send a hundred tasks within one second) go for
>> slurm. openPBS, torque, condor cannot take the heat.  I heard that PBS (not
>> open) can, too.
>>
>
> One of the issues, is knowing what data to take,  and there is a need for
> a cycle. each cycle have finite number of data inside the queue, but each
> new cycle recreate the queue to the actual content to be used.
>
> I still research this one.
>
>

With task schedulers, you can cancel jobs that are waiting in the queue
easily, even with boinc (which is designed for loose connectivity, that is,
it does not assume you can always contact the worker machines, so it fits
volunteer computing, like SETI-at-home).
However, if what you mean by "recreating the actual content to be used" is
that you want to change the data on which the jobs operate, then it is even
easier. Just have each job pull its data when it starts running from a
known place. Then you can change this known place at no central cost until
the job starts running.


>
>>
>> On Sun, Apr 7, 2013 at 5:30 PM, ik <idokan at gmail.com> wrote:
>>
>>> On Sun, Apr 7, 2013 at 3:51 PM, Oleg Goldshmidt <pub at goldshmidt.org>wrote:
>>>
>>>>
>>>>
>>>> On Sun, Apr 7, 2013 at 2:27 PM, ik <idokan at gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I have a very complex queue management requirements, I need to
>>>>> calculate task weight, time restrictions for execution, number of possible
>>>>> execution at given time, changes of the data when it is inside the queue
>>>>> and even disabling specific task.
>>>>> some tasks have very short time to live, and required to be executed
>>>>> faster, then others. but it's not a "simple" priority queue.
>>>>>
>>>>
>>>> It is difficult to say from the description, but it sounds to me like
>>>> your problem is that of "scheduling" rather than just "queue management".
>>>> Note that a "scheduler" (e.g., that of the kernel) will have many queues
>>>> holding tasks of different types and properties, and will need to manage
>>>> all of them together.
>>>>
>>>> In particular, some of the things you mention sound to me like "real
>>>> time" requirements. If you look at, say, the Linux scheduler it handles RT
>>>> tasks separately from the "normal" tasks. You may need to do the same.
>>>>
>>>> Another thing that is missing is the target system. A single server? A
>>>> cluster or multicomputer? Something else?
>>>>
>>>> I'd suggest googling for "scheduling" and "resource management" (of
>>>> which "scheduling" is a component). Apart from things like SLURM and Condor
>>>> that Orna mentioned, maybe Maui? There are all sorts of proprietary
>>>> offerings as well.
>>>>
>>>
>>> At first glance, condor looks like the thing that can help me most.
>>>
>>> Thanks
>>>
>>>
>>>>
>>>> --
>>>> Oleg Goldshmidt | pub at goldshmidt.org
>>>>
>>>
>>>
>>> _______________________________________________
>>> Linux-il mailing list
>>> Linux-il at cs.huji.ac.il
>>> http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il
>>>
>>>
>>
>>
>> --
>> Orna Agmon Ben-Yehuda.
>> http://ladypine.org
>>
>
>


-- 
Orna Agmon Ben-Yehuda.
http://ladypine.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.cs.huji.ac.il/pipermail/linux-il/attachments/20130408/8d202849/attachment-0001.html>