Optimizing text processors: C, Perl, and Assembly

Published on November 15, 2026
Tags: text-processing, c, perl, asm, SIMD, SSE2, AVX2

The other week I was optimizing some text processing code. The first thing I did was optimize the Perl regexps (this code was written in Perl), and eventually I decided to do a complete rewrite in C. And so a few hours later, the C code was ready to run, but it was so much slower than the Perl, and then I thought C is faster than Perl, but what is going wrong? My first thought was, 'Maybe I need assembly?' But then I thought that won't be very portable, and so I decided to use glibc's functions like strcmp and strchr to do things like find the next space. This made the C code considerably faster than the Perl.

Now, you may be wondering: how are glibc's string functions (like strchr or memchr) faster than a simple while (*ptr != '\0' && *ptr != ' ') { ...; ptr++; }?

The thing is your CPU's "wide" SIMD (Single Instruction Multiple Data) instructions (like SSE2 or AVX2) enable it to compare 16 or 32 bytes respectively with a single instruction. Think about it: comparing 16 or 32 bytes with a single instruction, is practically guaranteed to be faster.

Compare that to what our naive C loop does at the assembly level for every single byte:


  1. Check if *ptr is '\0'
  2. Check if *ptr is ' '
  3. Increment the pointer
  4. Jump back to the top of the while loop

Wash, rinse, and repeat--plus the CPU has to predict those branches (jumps) for every byte. Using glibc's hand-tuned (yes, really hand-tuned) assembly to do the scanning avoids all that overhead, while keeping the C code portable.

The results (measured in output speed, since a lot of input data gets discarded):


  Perl: 4.6 MB/s
  Naive C: 1.2 MB/s
  C with glibc: 26.2 MB/s