The Index of Coincidence

The beautifully clear Introduction, in its entirety:

The subject of this pamphlet is coincidence.

"Coincidence" as the term is used here may be defined as a recurrence of a letter in the same place, or in a corresponding place, as when two texts are lined up one under the other, letter for letter.

Mathematical evaluation assists the cryptanalyst first in preparing his material for attack, and later in the actual attack itself. It assists specifically in answering the following questions.

1. How much like random, or how different from random, is this text?
2. How similar are these texts?
3. How significant is this variation from random?
4. How is significant is this similarity?

This pamphlet, first published internally by the Navy Department in 1929 and republished in 1955 by the National Security Agency, discusses the Index of Coincidence (I.C.), a statistical tool used in cryptanalysis to determine the randomness of text and the similarity between different texts. It explores various applications of the I.C., including identifying messages encrypted with the same key, determining the overlap between messages in depth, and assessing the roughness of single text samples. The document also delves into the theoretical underpinnings of the I.C., explaining its expected values for different languages and polygraphic comparisons. It further elucidates the concept of standard deviation as a measure of the significance of I.C. deviations from expected values. The pamphlet examines the Cross I.C. as a correlation measure and provides a detailed example of using the I.C. to align secondary alphabets into a primary one for deciphering Vigenere ciphertexts. Finally, it addresses the I.C. of modular sums, mixed texts, and rectangular arrays, concluding with a discussion of the relationship between Chi-Square and the I.C.

A fascinating read for anyone interested in US cryptography capabilities through WWII and beyond, the history of cryptography, or the history of text analytics.

This annotated edition illustrates the capabilities of the AI Lab for Book-Lovers to add context and ease-of-use to manuscripts. It includes several types of abstracts, building from simplest to more complex: TLDR (one word), ELI5, TLDR (vanilla), Scientific Style, and Action Items; essays to increase viewpoint diversity, such as Grounds for Dissent, Red Team Critique, and MAGA Perspective; and Notable Passages and Nutshell Summaries for each page.