⛓ Why Kafka is so fast ⛓
Article on Medium Here are 2 points (and a bonus) that are applicable to many cases:
- Sequential I/O (Sequential input/output) – if we read and write files (especially on HDD) in different places, it's slow (100Kb/s), but if we only add records to the end (append) and read records from a certain point and sequentially forward, these operations are just as fast as reading and writing from memory (100Mb/s).
Kafka uses an append-only log, where each new record is added to the end, and we read from a certain point in the file (offset) to the end.
- Zero copy read (reading without copying) – we want to start sending data to the client from files in storage from a certain point. To do this, in the standard way, we get the requested data from the disk into the application (1 copy goes to the OS Buffer, 2nd from the OS Buffer to our application), then we send it to the Socket (3rd copy), and the Socket to the NIC Buffer (4th copy), which then sends the data to the recipient.
With Zero copy read, we use the system call sendfile, which specifies which file and where to read from, and where to write to, thanks to which the file is recorded in the OS Buffer and from there immediately gets into the NIC Buffer. And all this happens without even involving the CPU.
III. Bonus
In this video, they didn't mention another hack that Kafka uses:
All data that goes through Kafka is stored in separate files called "segments". When a segment reaches a certain limit (default 1Gb), Kafka creates the next segment.
Kafka has the ability to specify the lifetime of a message (when it should be deleted), so Kafka doesn't immediately delete this message, it waits until all messages in a single segment are outdated and then deletes the entire segment at once.
This is much faster than trying to delete individual data from a file.