异步任务队列:celery

jopen 6年前

celery是一个异步任务队列/基于分布式消息传递的作业队列。它侧重于实时操作,但对调度支持也很好。

celery用于生产系统每天处理数以百万计的任务。
687474703a2f2f636c6f75642e6769746875622e636f6d2f646f776e6c6f6164732f63656c6572792f63656c6572792f63656c6572795f3132382e706e67.png

celery是用Python编写的,但该协议可以在任何语言实现。它也可以与其他语言通过webhooks实现。

建议的消息代理RabbitMQ的,但提供有限支持Redis, Beanstalk, MongoDB, CouchDB, ,和数据库(使用SQLAlchemy的或Django的 ORM) 。

Celery is...

  • Simple

    Celery is easy to use and maintain, and does not need configuration files.

    It has an active, friendly community you can talk to for support, including a mailing-list and and an IRC channel.

    Here's one of the simplest applications you can make:

    from celery import Celery    celery = Celery('hello', broker='amqp://guest@localhost//')    @celery.task  def hello():      return 'hello world'
  • Highly Available

    Workers and clients will automatically retry in the event of connection loss or failure, and some brokers support HA in way of Master/Master or Master/Slave replication.

  • Fast

    A single Celery process can process millions of tasks a minute, with sub-millisecond round-trip latency (using RabbitMQ, py-librabbitmq, and optimized settings).

  • Flexible

    Almost every part of Celery can be extended or used on its own, Custom pool implementations, serializers, compression schemes, logging, schedulers, consumers, producers, autoscalers, broker transports and much more.

It supports...

  • Brokers

  • Concurrency

  • Result Stores

    • AMQP, Redis
    • memcached, MongoDB
    • SQLAlchemy, Django ORM
    • Apache Cassandra, IronCache
  • Serialization

    • pickle, json, yaml, msgpack.
    • zlib, bzip2 compression.
    • Cryptographic message signing.

Framework Integration

Celery is easy to integrate with web frameworks, some of which even have integration packages:

Django django-celery
Pyramid pyramid_celery
Pylons celery-pylons
Flask not needed
web2py web2py-celery
Tornado tornado-celery

The integration packages are not strictly necessary, but they can make development easier, and sometimes they add important hooks like closing database connections at fork.

项目主页:http://www.open-open.com/lib/view/home/1372152459838