Answers to general questions about the Sybase CEP Engine and its uses. Discusses Sybase CEP functionality, Continuous Computational Language, data streams, platform compatibility, and so on.
Sybase CEP Engine is a software engine that can process and analyze tens and hundreds of thousands of messages per second. Sybase CEP Engine is not an end-user application, but rather an enabling technology and a platform to help build other applications. Sybase CEP Engine is easy to program, deploy, and scale. The key features of Sybase CEP Engine include:
Sybase CEP Engine's ability to process data continuously as it arrives.
Sybase CEP Engine's ability to process data before it is stored on disk, thus achieving very high throughput and a low latency of under one millisecond.
The processing and correlation performed by the Sybase CEP Engine which is defined via a familiar-looking SQL-based Continuous Computation Language (CCL).
Problems ideal for Sybase CEP Engine generally have all or most of these properties:
Rich data sources:
Multiple data streams (multiple data sources).
Heterogeneous data streams (multiple formats).
High incoming data rates.
Complex processing requirements:
Processing requiring filtering, aggregation, transformation, multi-stream correlation, event pattern matching, and other complex processing.
Need for immediate response:
Close to real-time response (latency requirements under one millisecond).
These problems are common in Finance (trading, risk management, fraud detection), Manufacturing (process control), Network Security (intrusion detection, event management), Supply Chain Management and Logistics (RFID applications), Power Grid Management, Sensor networks, Military, and other industries.
CCL is a language for continuously processing data arriving on multiple data streams. CCL uses familiar SQL constructs like SELECT, FROM, WHERE, HAVING, GROUP BY, INSERT, and others, to express queries over data streams. Unlike SQL "one time only" queries, CCL queries run continuously within the Sybase CEP Engine. Also, unlike a traditional database that runs SQL queries against mostly static data, Sybase CEP Engine runs continuously arriving data against pre-registered CCL queries.
A data stream is a way to publish and subscribe to data that needs to be continuously processed. Each stream row contains a number of columns, and each column has a name and type. For example:
A stream called RFIDRaw, coming out of an RFID reader may have these columns:
ReaderID (STRING).
TagID (STRING).
A stream called Trades, coming from a stock exchange may have columns:
Symbol (STRING).
Volume (INTEGER).
Price (FLOAT).
One way to think about data streams is to imagine them as database tables of infinite size. Queries referring to these "tables" typically operate not on the whole stream but on windows defined over portions of the stream. Some queries may write into the "table" and some may read from it. See Stream Questions for more details about streams.
Sybase CEP Engine isn't just another "pub/sub" (publish and subscribe) product, like Tibco, or IBM MQ. Other products provide just a transport layer for data. They do not enable complex processing on the data they deliver. Sybase CEP Engine is first and foremost a rich processing platform, with an SQL-like processing and computation language. Sybase CEP Engine can also incorporate different types of message buses through the use of adapters.
No, it is not. Databases are great at storing and querying static data, and processing transactions reliably. However, databases are notoriously bad at processing continuously incoming large volumes of data with low latency. There are several reasons for this:
Databases must store all data on disk first before beginning to process it. This is slow.
Databases do not use pre-registered continuous queries. Database queries are "one time only" queries. If you want to ask a question ten times a second, you have to issue the query ten times a second. This model breaks down when there are many queries that need to be continuously evaluated.
Databases do not use incremental processing. Sybase CEP Engine is capable of evaluating queries in an incremental way as data arrives. Databases cannot do this.
Sybase CEP Engine complements traditional databases to help solve new classes of problems.
No. In-memory databases solve the problem of having to store data before processing it, but they do not address the need for continuous queries or incremental processing. Sybase CEP Engine has a highly optimized in-memory store to keep the state of the queries. CCL Query Module Persistence and Guaranteed Delivery features are used to ensure that no rows or state information are lost in the event of Sybase CEP Server failure by storing critical information about query module state on the disk drive. However, CCL queries also run continuously, addressing the need for ongoing incremental processing.
This is a very common problem that CCL was designed to solve. CCL makes it easy to write queries that continuously join data arriving on a data stream with data from a database. Sybase CEP Engine can cache whole database tables in memory, or it can bring in data required from a database subquery. You can also perform update and delete functions on an external database directly from CCL.
Sybase CEP provides an application called Sybase CEP Studio, which allows you to create CCL queries, organize them in hierarchical modules and projects, compile the projects, and deploy them to Sybase CEP Server. The use of Sybase CEP Studio is not required, however. Sybase CEP Engine can be used from the command line, using the c8_client utility. You can also embed Sybase CEP Engine in your product and provide your own GUI that generates CCL queries and sends them to Sybase CEP Engine via SOAP (Simple Object Access Protocol).
Sybase CEP Engine currently runs on Windows and on Linux and other UNIX-like operating systems, including Sun Solaris. Sybase CEP Engine is implemented as a highly optimized multithreaded C++ server, to provide maximum performance. Adapters for sending data to and receiving data from the Sybase CEP Engine can be implemented in C/C++, Java 1.5, .NET3, and other languages.
The throughput of the engine depends on the queries that it is processing. For simple queries, such as filtering, the performance of Sybase CEP Engine is measured in hundreds of thousands of messages per second. For more complex queries, including multiple joins over sliding windows, Sybase customers have timed the engine at tens of thousands of messages per second. But the exact numbers depend on the particular queries being run.
Sybase CEP Engine processing latency also depends on the query, but can be roughly estimated to be under one millisecond.
Sybase CEP Engine consists of the following components:
Sybase CEP Server: The software engine that actually processes and correlates data streams at runtime.
Sybase CEP Studio: The graphical environment for defining streams, adapters, CCL queries and CCL modules, as well as for managing CCL projects at runtime. The use of Sybase CEP Studio is optional for customers who use the c8_client command-line utility, and for products that embed Sybase CEP Engine and can define and manage CCL modules using a SOAP (Simple Object Access Protocol) API.
CCL Compiler: The compiler that translates and optimizes CCL queries for processing by Sybase CEP Server. The Compiler is invoked by Sybase CEP Studio or through an API.
Input and Output Adapters: The components that translate different kinds of data to and from the format used by Sybase CEP Engine.
Integration SDK: The set of tools for creating adapters (in C/C++, Java 1.5, .NET3, and so on) and for interfacing with the engine through SOAP (Simple Object Access Protocol) APIs.