CloudButton Project Review

2nd period


Final Review Agenda

Meeting Subject: Cloudbutton final review
Venue: Teams meeting
Date: September 15th, 2022
Chair: Pedro García López (Coordinator)
Time Subject Time (Mins) Lead participant
9:00 Private meeting, reviewers and PO 30 PO
9:30 Welcome - Presentation of Participants 10 Coordinator
9:40 CloudButton promotional video 5 RHT
9:45 Overall Project Presentation
Project Overview and Big Picture
Achievements and project outcomes
15 Coordinator
10:00 WP2 - Architecture and Use Cases 30 Coordinator,
EMBL,
Hutton
10:30 Round table and open questions* 20 PO
10:50 Break 10
11:00 WP3 - Serverless Compute Engine for Big Data 20 IBM
11:20 WP4 - Mutable Shared Data 20 IMT
11:40 WP5 - CloudButton toolkit 20 IMP
12:00 WP6 - Exploitation and Dissemination 10 RHT
12:10 Final recap and summary 10 Coordinator
12:20 WP1 - Project Management
Use of resources, financial information
Management procedures
10 Coordinator
12:30 Final remarks 10 PO
12:40 Private meeting: reviewers and PO 40 PO
13:20 Oral feedback from reviewers and comment from project 10 PO
13:30 End of Review
* We assume reviewers watched videos previously

Videos and presentations

Video Slides
CloudButton promotional video
Project overview and major outcomes
Serverless Geospatial Data
Serverless Genomics Pipelines
Serverless Metabolomics and Metaspace
Efficient portage of Java and shell applications to serverless
Transparently running OpenMP/MPI applications on Serverless
Transparent Serverless execution of Python multiprocessing applications
Supporting WebAssembly serverless functions with Lithops
Infinispan backend for Lithops
ATOS SLA Component and Lithops

Deliverables

PDF
D2.5 Reference Implementation of Architectural Building Blocks
D2.6 Data Management Plan, 3rd version
D3.3 Serverless Compute Engine Reference Implementation
D4.3 Full implementation of the BLOSSOM middleware
D5.3 CloudButton Toolkit Reference Implementation
D6.3 Final dissemination, exploitation, and adoption report

Summary reports

Partner short name Partner PDF
URV Universitat Rovira i Virgili
IBM IBM Israel - Science and Technology LTD
IMP Imperial College London
IMT Institute Mines-Télécom
RHT Red Hat Limited
EMBL European Molecular Biology Laboratory
ATOS Atos Spain SA
MATRIX Fundación Matrix, Investigación y Desarrollo Sostenible
HUTTON The James Hutton Institute

Software results

Title Description Repository
Lithops An open source framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud.
Crucial A system to program highly-concurrent stateful applications on serverless architectures. It keeps a simple programming model and allows to port effortlessly multi-threaded algorithms to FaaS.
Faasm A high-performance stateful serverless runtime that provides multi-tenant isolation, yet allows functions to share regions of memory.
Triggerflow An scalable, extensible and serverless platform for event-based orchestration of serverless workflows.
Faabric Messaging and state layer for distributed serverless applications.
CloudButton Toolkit examples Notebook examples for the CloudButton Toolkit: Moments in time, Mandelbrot set computation, Pi estimation using the Monte Carlo method.
Serverless shell Shell scripting for serverless.
Primula Primula is a serverless shuffle operator for general-purpose serverless frameworks.
CloudButton Apache Airflow plugin Apache Airflow Plugin that implements new operators to easily deploy serverless functions tasks on IBM Cloud Functions.
Serverless Elastic Exploration of Unbalanced Algorithms Implementation of unbalanced algorithms like the Unbalanced Tree Search benchmark over serverless functions.
Serverless port of SMILE Port of a subset of SMILE machine learning library to Crucial.
Infinispan Openshift operator An OpenShift operator to run and rule Infinispan.
Python Client for Infinispan A Python protoclient for connecting to an Infinispan server via Hotrod.
MLLess Serverless ML training on IBM Cloud Functions.
Faasm integration with Lithops Benchmarks for Lithops.
CloudButton-SLA SLA system, inspired by the WS-Agreement standard, that uses Knative Observability Plugin monitoring data to supervise Knative running pods in order to identify candidate performance improvements and/or problems.
S3Contents An S3 and GCS backed ContentsManager implementation for Jupyter.
Metabolomics use case - Experiment 1 Demonstrates running through the whole Serverless metabolite annotation pipeline with a typical dataset, downloading the results and comparing them against the Serverful implementation of METASPACE.
Metabolomics use case - Experiment 2 An example of running the pipeline against a smaller set of molecules, to demonstrate the potential of Serverless to provide low-latency access to computing resources.
Metabolomics use case - Experiment 3 A stress test that runs the Serverless metabolite annotation pipeline with a large dataset and many molecular databases.