• Introduction
  • Background & Motivation
  • GAP
  • Challenges
  • Goal
• Global SnapShot
• Model

Introduction

Background & Motivation

State:

• In data-driven applications, states are separated from the application.
• The state can be stored in DB for data consistency and scalability. And the DB is shared among all apps.
• The state can be shared among applications and stored in DB. And rely on the database for data consistency and scalability, among others.

Stream processing: It is a natural paradigm for event-driven applications that need to react fast to real-world events and communicate with each other via message passing.

GAP

Existing stream processing mainly focus on producing real-time but approximate results, which will be corrected by more reliable periodic batch compute jobs.

Challenges

Challenges for the stateful streaming systems.

• Lack of explicit computational state abstractions.
• The transaction with external storage (which manages the states) is the bottleneck.
• Scaling in/out requires operational challenges.

ShortCome of existing works

• Spark sacrifices programming model transparency and processing latency by enforcing batch-centric application logic.
• Others use transactional per-record processing.

Goal

This paper presents a complete, continuous state management solution based on distributed snapshots.

• Distributed snapshots enable rollback recovery of arbitrary distributed processes to a prior globally consistent execution state.

This paper tries to pipeline the snapshotting process in weakly connected graphs and skip the in-transit records when possible.

Global SnapShot

Lamport’s protocol can only be used for strongly connected directed graphs, where each edge has a direction. But in weakly connected graphs, the edge may not have a direction. Thus, the protocol cannot be terminated.

Naiad’s two-phase commit, IBM Stream’s multi-stage snapshotting, could complete the snapshots but with unnecessary in-transit records logging.

Model

Managed States:

• Keyed-State:
  • any record can be mapped to a key space.
  • The state has many types for various operations, such as ListState, ValueState, MapState, and ReduceState.
• Operator-State:

Partitioning and Allocation

• Decouple key space and state space similar to Dynamo.

---

• ← Previous Post
• Next Post →