Scylla Documentation Logo Documentation
  • Server
    • Scylla Open Source
    • Scylla Enterprise
    • Scylla Alternator
  • Cloud
    • Scylla Cloud
    • Scylla Cloud Docs
  • Tools
    • Scylla Manager
    • Scylla Monitoring Stack
    • Scylla Operator
  • Drivers
    • CQL Drivers
    • DynamoDB Drivers
Download
Menu

Caution

You're viewing documentation for a previous version of Scylla Python Driver. Switch to the latest stable version.

Scylla Python Driver Performance Notes

Performance Notes¶

The Python driver for Cassandra offers several methods for executing queries. You can synchronously block for queries to complete using Session.execute(), you can obtain asynchronous request futures through Session.execute_async(), and you can attach a callback to the future with ResponseFuture.add_callback().

Examples of multiple request patterns can be found in the benchmark scripts included in the driver project.

The choice of execution pattern will depend on the application context. For applications dealing with multiple requests in a given context, the recommended pattern is to use concurrent asynchronous requests with callbacks. For many use cases, you don’t need to implement this pattern yourself. cassandra.concurrent.execute_concurrent() and cassandra.concurrent.execute_concurrent_with_args() provide this pattern with a synchronous API and tunable concurrency.

Due to the GIL and limited concurrency, the driver can become CPU-bound pretty quickly. The sections below discuss further runtime and design considerations for mitigating this limitation.

PyPy¶

PyPy is an alternative Python runtime which uses a JIT compiler to reduce CPU consumption. This leads to a huge improvement in the driver performance, more than doubling throughput for many workloads.

Cython Extensions¶

Cython is an optimizing compiler and language that can be used to compile the core files and optional extensions for the driver. Cython is not a strict dependency, but the extensions will be built by default.

See Installation for details on controlling this build.

multiprocessing¶

All of the patterns discussed above may be used over multiple processes using the multiprocessing module. Multiple processes will scale better than multiple threads, so if high throughput is your goal, consider this option.

Be sure to never share any Cluster, Session, or ResponseFuture objects across multiple processes. These objects should all be created after forking the process, not before.

For further discussion and simple examples using the driver with multiprocessing, see this blog post.

PREVIOUS
Execution Profiles
NEXT
Paging Large Queries
  • 3.24.8
    • 3.25.4
    • 3.24.8
    • 3.22.3
    • 3.21.0
  • API Documentation
    • cassandra - Exceptions and Enums
    • cassandra.cluster - Clusters and Sessions
    • cassandra.policies - Load balancing and Failure Handling Policies
    • cassandra.auth - Authentication
    • cassandra.graph - Graph Statements, Options, and Row Factories
    • cassandra.metadata - Schema and Ring Topology
    • cassandra.metrics - Performance Metrics
    • cassandra.query - Prepared Statements, Batch Statements, Tracing, and Row Factories
    • cassandra.pool - Hosts and Connection Pools
    • cassandra.protocol - Protocol Features
    • cassandra.encoder - Encoders for non-prepared Statements
    • cassandra.decoder - Data Return Formats
    • cassandra.concurrent - Utilities for Concurrent Statement Execution
    • cassandra.connection - Low Level Connection Info
    • cassandra.util - Utilities
    • cassandra.timestamps - Timestamp Generation
    • cassandra.io.asyncioreactor - asyncio Event Loop
    • cassandra.io.asyncorereactor - asyncore Event Loop
    • cassandra.io.eventletreactor - eventlet-compatible Connection
    • cassandra.io.libevreactor - libev Event Loop
    • cassandra.io.geventreactor - gevent-compatible Event Loop
    • cassandra.io.twistedreactor - Twisted Event Loop
    • cassandra.cqlengine.models - Table models for object mapping
    • cassandra.cqlengine.columns - Column types for object mapping models
    • cassandra.cqlengine.query - Query and filter model objects
    • cassandra.cqlengine.connection - Connection management for cqlengine
    • cassandra.cqlengine.management - Schema management for cqlengine
    • cassandra.cqlengine.usertype - Model classes for User Defined Types
    • cassandra.datastax.graph - Graph Statements, Options, and Row Factories
    • cassandra.datastax.graph.fluent
    • cassandra.datastax.graph.fluent.query
    • cassandra.datastax.graph.fluent.predicates
  • Installation
  • Getting Started
  • Scylla Specific Features
  • Upgrading
  • Execution Profiles
  • Performance Notes
  • Paging Large Queries
  • Lightweight Transactions (Compare-and-set)
  • Security
  • User Defined Types
  • Object Mapper
    • Upgrade Guide
    • Models
    • Making Queries
    • Batch Queries
    • Connections
    • Third party integrations
    • Frequently Asked Questions
  • Working with Dates and Times
  • Scylla Cloud
  • Frequently Asked Questions
  • Create an issue
  • Edit this page

On this page

  • Performance Notes
    • PyPy
    • Cython Extensions
    • multiprocessing
Logo
Docs Contact Us About Us
Mail List Icon Slack Icon
© ScyllaDB 2021 and © DataStax 2013-2017
Powered by Sphinx 4.3.2 & ScyllaDB Theme 1.2.2