BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:America/Denver
X-LIC-LOCATION:America/Denver
BEGIN:DAYLIGHT
TZOFFSETFROM:-0700
TZOFFSETTO:-0600
TZNAME:MDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0600
TZOFFSETTO:-0700
TZNAME:MST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20260422T000713Z
LOCATION:501-502
DTSTART;TZID=America/Denver:20231112T170200
DTEND;TZID=America/Denver:20231112T172000
UID:submissions.supercomputing.org_SC23_sess439_ws_worksp110@linklings.com
SUMMARY:Optimization Toward Efficiency and Stateful of dispel4py
DESCRIPTION:Liang Liang (Imperial College, London); Heting Zhang (Universi
 ty of St Andrews, Scotland); Guang Yang and Thomas Heinis (Imperial Colleg
 e, London); and Rosa Filgueira (University of St Andrews, Scotland)\n\nSci
 entific workflows bridge scientific challenges with computational resource
 s. While dispel4py, a stream-based workflow system, offers mappings to par
 allel enactment engines like MPI or Multiprocessing, its optimization prim
 arily focuses on dynamic process-to-task allocation for improved performan
 ce. An efficiency gap persists, particularly with the growing emphasis on 
 conserving computing resources. Moreover, the existing dynamic optimizatio
 n lacks support for stateful applications and grouping operations.\n\nTo a
 ddress these issues, our work introduces a novel hybrid approach for handl
 ing stateful operations and groupings within workflows, leveraging a new R
 edis mapping. We also propose an auto-scaling mechanism integrated into di
 spel4py's dynamic optimization. Our experiments showcase the effectiveness
  of auto-scaling optimization, achieving efficiency while upholding perfor
 mance. In the best case, auto-scaling reduces dispel4py's runtime to 87% c
 ompared to the baseline, using only 76% of process resources. Importantly,
  our optimized stateful dispel4py demonstrates a remarkable speedup, utili
 zing just 32% of the runtime compared to the contender.\n\nTag: Applicatio
 ns, Cloud Computing, Distributed Computing, Edge Computing, Large Scale Sy
 stems\n\nRegistration Category: Workshop Reg Pass\n\nSession Chairs: Silvi
 na Caino-Lores (National Institute for Research in Digital Science and Tec
 hnology (Inria)) and Anirban Mandal (Renaissance Computing Institute (RENC
 I), University of North Carolina at Chapel Hill)\n\n
END:VEVENT
END:VCALENDAR