TU Darmstadt / ULB / TUprints

Executing cyclic scientific workflows in the cloud

Krämer, Michel ; Würz, Hendrik M. ; Altenhofen, Christian (2024)
Executing cyclic scientific workflows in the cloud.
In: Journal of Cloud Computing : Advances, Systems and Applications, 2021, 10 (1)
doi: 10.26083/tuprints-00023616
Article, Secondary publication, Publisher's Version

[img] Text
Copyright Information: CC BY 4.0 International - Creative Commons, Attribution.

Download (4MB)
Item Type: Article
Type of entry: Secondary publication
Title: Executing cyclic scientific workflows in the cloud
Language: English
Date: 8 April 2024
Place of Publication: Darmstadt
Year of primary publication: 6 April 2021
Place of primary publication: Berlin ; Heidelberg
Publisher: SpringerOpen
Journal or Publication Title: Journal of Cloud Computing : Advances, Systems and Applications
Volume of the journal: 10
Issue Number: 1
Collation: 26 Seiten
DOI: 10.26083/tuprints-00023616
Corresponding Links:
Origin: Secondary publication DeepGreen

We present an algorithm and a software architecture for a cloud-based system that executes cyclic scientific workflows whose structure may change during run time. Existing approaches either rely on workflow definitions based on directed acyclic graphs (DAGs) or require workarounds to implement cyclic structures. In contrast, our system supports cycles natively, avoids workarounds, and as such reduces the complexity of workflow modelling and maintenance. Our algorithm traverses workflow graphs and transforms them iteratively into linear sequences of executable actions. We call these sequences process chains. Our software architecture distributes the process chains to multiple compute nodes in the cloud and oversees their execution. We evaluate our approach by applying it to two practical use cases from the domains of astronomy and engineering. We also compare it with two existing workflow management systems. The evaluation demonstrates that our algorithm is able to execute dynamically changing workflows with cycles and that design and maintenance of complex workflows is easier than with existing solutions. It also shows that our software architecture can run process chains on multiple compute nodes in parallel to significantly speed up the workflow execution. An implementation of our algorithm and the software architecture is available with the Steep Workflow Management System that we released under an open-source license. The resources for the first practical use case are also available as open source for reproduction.

Uncontrolled Keywords: Scientific workflow management systems, Workflow scheduling, Cloud computing, Distributed systems
Identification Number: Artikel-ID: 25
Status: Publisher's Version
URN: urn:nbn:de:tuda-tuprints-236167
Classification DDC: 000 Generalities, computers, information > 004 Computer science
Divisions: 20 Department of Computer Science > Interactive Graphics Systems
20 Department of Computer Science > Fraunhofer IGD
Date Deposited: 08 Apr 2024 12:24
Last Modified: 10 Apr 2024 06:09
SWORD Depositor: Deep Green
URI: https://tuprints.ulb.tu-darmstadt.de/id/eprint/23616
PPN: 517009056
Actions (login required)
View Item View Item