Encryption

Turing Award to the Institute’s Prof. Shafi Goldwasser

jhalper — Wed, 13 Mar 2013 12:00:29 +0000

Turing Award to the Institute’s Prof. Shafi Goldwasser

Prof. Shafi Goldwasser, who is at both the Weizmann Institute and MIT, will receive the 2012 A.M. Turing Award, together with Prof. Silvio Micali of MIT. Goldwasser is only the third woman to receive the Award since its inception in 1966, and she is the third faculty member of the Weizmann Institute to receive what is considered to be the “Nobel Prize in computing.”

Goldwasser and Micali’s work in the 1980s laid the foundations of modern cryptography – the science that, among other things, keeps your electronic transactions secure.

The basis of their work is a series of riffs on the concepts of probability and proof. For example, in their seminal “zero-knowledge interactive proofs” paper, they changed, forever, the paradigm of a mathematical proof that must be written down, step-by-step, page after page. Instead, they suggested that a proof can come out of a conversation between two parties – a “prover” and a “verifier.”

Laszlo Lovasz describes it thus:

It sounds paradoxical, but there is a scheme which allows the customer to convince the bank that he knows the password – without giving the slightest hint as to what the password is! ….The most interesting aspect of this scheme is that it extends the notion of a proof, thought (at least in classical mathematics) to be well established for more than 2000 years. In the classical sense, a proof is written down entirely by the author (whom we call the Prover), and then it is verified by the reader (whom we call the Verifier). Here, there is interaction between the Prover and the Verifier: The action taken by the Prover depends on the “questions” of the Verifier.

In other words, by asking a series of questions, each one randomly based on the answer to the previous question, the verifier can ascertain whether the prover has the mathematical proof -- without actually learning anything about the proof, itself. Obviously, using a method like this can keep your password from being stolen, but it can also be used by people working together online who want to jointly compute functions without giving away their data.

In fact, Goldwasser and Micali’s work has major implications for the entire field of mathematics known as complexity theory. Lovasz imagines using the method to referee an “exponentially long paper”:

There is a way to write down a proof so that a referee can check the correctness of a theorem by reading only a tiny fraction of it. The proof becomes longer than necessary, but not much longer. The number of characters the referee has to read is only around the logarithm of the original proof length!

Just as important, says Goldwasser, you can use such methods to establish that a mathematical statement is not correct, by proving "non existence" of classical proofs.

jhalper Wed, 03/13/2013 - 08:00

Tags

Categories

No Need for Decryption

jhalper — Thu, 15 Dec 2011 01:03:52 +0000

No Need for Decryption

Is it possible to perform operations on encrypted data, while keeping it secure from all prying eyes (or circuits), even if that data is stored remotely, in the "cloud?" Will our end result still be encrypted, and when we decode it with our private decryption key, will our result be correct? To put it another way, could we allow sensitive data - say private medical information - to be monitored on-line and feel completely secure in the knowledge that no one can access it without our express permission? Can we use a cloud service to store our encrypted data and perform a search on that data without allowing the servers to "see" our search?

Welcome to the world of fully homomorphic encryption. The concept was first proposed in 1978 - long before the advent of remote computing services - by Rivest, Adelman and Dertouzos. (Rivest and Adelman, together with the Weizmann Institute's Prof. Adi Shamir, invented the RSA scheme used for almost all computer encryption today.) Since then, various researchers have come up with "partly homomorphic" methods, but none of them enabled full homomorphism. Only in 2009 was a fully homomorphic method demonstrated, by Craig Gentry at Stanford. That method, though proof of concept, was much too heavy and slow to be of any practical use.

Gentry published his method as his Ph.D. thesis. But it could be the doctoral work of another recent graduate - Dr. Zvika Brakerski from the Weizmann Institute (a student of Prof. Shafi Goldwasser) - that ultimately enables fully homomorphic encryption to become reality. Brakerski worked with Dr. Vinod Vaikuntanathan, a former student of Goldwasser's at MIT, who was at Microsoft Research at the time and is currently a professor at the University of Toronto.

In a nutshell: Gentry made some assumptions about the complexity of the math needed to achieve fully homomorphic encryption, and then used "a bit of a hack" (his words) to make it all work. Brakerski and Vaikuntanathan managed to change some of those assumptions (to "weaker" - more plausible and widely accepted - assumptions), simplifying the math and even eliminating the need for some of the hacks. The result, they say, is a method that is hundreds or even thousands of times faster than the original, but still fully homomorphic.

Brakerski, now doing postdoctoral research at Stanford, is continuing to research the math involved in fully homomorphic encryption. In the meantime, software engineers are already applying his insights to the future of data security.

Prof. Shafi Goldwasser and Dr. Zvika Brakerski

Also today at the Weizmann Institute:
White blood cells that reach into the blood vessel lining looking for "exit signs" and the closest supernova observation in the past 25 years yields new insights into how stars explode.

jhalper Wed, 12/14/2011 - 20:03

Tags

Categories

Cryptographic Padding in RSA

goodmath — Thu, 08 Jan 2009 20:52:29 +0000

Cryptographic Padding in RSA

Ok, away from politics, and back to the good stuff. When I left off talking about
encryption, we were href="http://scienceblogs.com/goodmath/2008/12/public_key_cryptography_using.php">looking at
RSA, as an example of an asymmetric encryption system.

Since it's been a while, I'll start off with a quick review of RSA.

Asymmetric encryption systems like RSA are based on computations that are easy to perform if you know the keys, and incredibly hard to perform if you don't. In the specific case
of RSA, everything is based on a pair of very large prime numbers. If you know those two
primes, and you know the public key, it's really easy to compute the private key. But
if you don't know the two prime numbers, then even given the public key,
it's incredibly difficult to compute the private key.

To be a bit more specific, in RSA, you get a pair of large prime numbers, P and Q. You
compute from them a totient of their product, which is the number
N=(P-1)×(Q-1). Then you arbitrarily pick a public exponent, E, which is
smaller than N, and which is prime relative to N. You can then compute
the private key exponent, D. If you know what P and Q are, it's pretty easy to compute
D.

Once you've got D and E, your public key is the pair is (N,E), and the private key is the pair is (N,D).

If you've got a plaintext message M, then encrypting it with the public key
is done by computing M^E mod N. If you've got a ciphertext C encrypted
with the public key, then decrypting it is done by computing C^D mod N.

Encrypting a plaintext with the public key is exactly the same process
as decrypting a ciphertext produced with the private key. And vice versa.

RSA is amazingly elegant. Every time I look at it for any reason, I'm struck
by its beauty and it's simplicity. It's really an amazing example of how
something seemingly abstract like number theory can produce practical, down-to-earth,
and useful.

But RSA isn't perfect. In fact, it's got a some rather serious problems that
are a direct result of its mathematical structure. I'm just going to mention
one, but there are several of them.

Suppose you've got your public/private keypair. You want to encrypt two messages, I
and J with your private key. Let's call the function for encrypting a message
with the private key E - so E(M) = M^D mod N. Then
C_I = E(I), and C_J=E(J). Good so far.

The problem is that given those two messages - for which an attacker has access to both
the plaintext and the ciphertext - it happens that there's a vulnerability. Because of the
way that encryption works, E(I×J) = E(I) × E(J)!

This opens you up to something called chosen ciphertext attacks. A chosen
ciphertext attack is one where you attack an crytographic system by running chosen
ciphertexts through the decryption system to see if you can compute the encryption key.
There's an effective attack against the naive use of RSA based on a selected ciphertext
method.

In addition to this, there are a few other interesting attacks that I won't describe in
detail. They all rely on various problems of the numerical structure of RSA encryption. In
order to defeat those attacks, RSA is typically used with something called a padding
scheme. The idea of the padding scheme is twofold. First, it increases the size of the
message - which guarantees that the encrypted message is large enough that it will not be
easy to use for an attack. Second, it intersperses pseudo-random information in a way that
means that a given plaintext message could be encrypted to a wide range of different
ciphertexts, depending on the choices made during padding.

A common scheme for padding is OAEP - Optimal Asymmetric Encryption Padding. OAEP
is a basic Feistel network system, like the ones I described when I was talking
about DES. OAEP ends up doing a mixture of permuting the plaintext, and
adding pseudo-random noise to it. It's a reversible transformation,
and the receiver of the encrypted message (and thus any attacker) knows how to
do the reversal of the padding given the decrypted plaintext. But the process
of permutation and random injection performed by the padding has the effect
of breaking the properties of RSA that make it easy to attack the system.

For example, suppose that the padding function is called P. E(P(I))×E(P(J)) is
not equal to E(P(I×J)). Similarly, the message-size-related problems
with RSA are not usable on messages whose size is increased past the vulnerability threshold
using OAEP padding.

So what does the padding look like? Let's start with the pieces.

You have a number, N, which is the length of the cryptographic modulus.
You have two integers, k₀ and k₁, which are parameters
of the protocol in use.
You have the original plaintext message, M. By definition, the length of M is
(n - k₀ - k₁). (The protocol is responsible for breaking
up large messages into sub-messages via an appropriate mode of operation.)
You have a 0-padded version of M, M', which is M followed by k₁
zeros - so M' has length N-k₀.
You have a random string of bits, r, which has length k₀.
You have a cryptographic hash function G, which expands a string of
k₀ bits to a string of N-k₀ bits.
You have another cryptographic hash function H, which contracts
a string of n-k₀ bits to a srting of k₀ bits.

The OAEP padding algorithm is illustrated off to the side. The way that
it works is: you compute G(r), giving you a string of N-k₀ bits.
You XOR G(r) with M', producing a string of N-k₀ bits, which we'll
call X. Then compute H(X), and XOR that with r, producing a result that we'll call
Y. The end result of the padding is the concatenation of X and Y.

The way to understand this is that you've got some random bits in R, which you're shuffling up (using G and H) and mixing with the bits of the original message. The resulting message is longer, and has a random element mixed into it, which defeats the numerical properties that make some attacks against RSA work. It's easy to compute
X and Y given M and r. And given X and Y, if you know G and H it's easy to compute
M and r. But given E(X concat Y), as an attacker, you're screwed. You can
still decript the message, and get X and Y, decode them, and get the plaintext. But
the process of doing the padding obscures the numerical properties of RSA so that
even knowing the padding function, your attacks won't work.

goodmath Thu, 01/08/2009 - 15:52

Tags

Encryption

Categories

Free Thought

It seems like an interesting fix but, since G and H have to remain secret, doesn't it turn the whole scheme into a symmetric encryption method? Don't you loose all benefits of using asymmetric cryptography?

Simon:

G and H don't have to remain secret. In fact, they're specifically *not* secret. Their purpose isn't to encrypt the message; it's just to scramble things to change their numeric properties. So, for example, a pair of original plaintext messages P and Q no longer have the property that E(P)×E(Q) = E(P×Q).

It's true that E(OAEP(P))×E(OAEP(Q)) = E(OAEP(P)×OAEP(Q)). But since OAEP(P)×OAEP(Q) does not equal OAEP(P×Q), that doesn't help an attacker.

The reason that G and H are cryptographic hashes is because
they given a particular value of G(M), it's very hard to compute an M' such that G(M') = G(M), and so it's very difficult to create messages that let you exploit the numerical properties of RSA.

Mark:

Thank you! So the equation E(P)*E(Q) = E(P*Q) tells you something about the private key? Then, given any public key, an attacker could choose carefully his plaintext to discover the information this padding scheme is supposed to hide. Am I wrong?

Simon:

Yes, exactly. The fact that E(P)×E(Q)=E(P×Q) provides a handle that an attacker can use to deduce the secret key. There are also a number of similar vulnerabilities in RSA, based on similar numerical properties. The point of the padding scheme is just to change the structure of the message enough to make it very difficult for an attacker to produce messages that they can use to deduce the private key.

I'll add some small print :)

Correct me if I'm wrong, but another flaw with RSA is it depends on calculating large prime numbers. After a certain point it becomes quite impractical to do normal prime tests and you need to use probabilistic prime tests. There are composite numbers (like Carmichael numbers) that can fool these tests into thinking that they are prime.
Another thing to note is that there are certain Ns that are relatively easy to factor using methods such as Pollard's p-1 or quadratic sieve that depend on certain properties of the P and Q you are using.
I'm not sure if these numbers are common enough that it matters or not, but it might be a good idea to keep these in mind when building an implementation of RSA.

Rob:

First, I would never advise anyone to do their own implementation of RSA. There are plenty of really solid, tested implementations out there that people can use. Cryptographic algorithms are very easy to get wrong, and the slightest mistake can completely undermine the security of the system. Crypto isn't an area where you should roll your own; building your own crpyto should be a last resort, and if it needs to be done, it should be done with incredible care, including things like code audits by outside crypto experts.

That said, yes, there is a real problem with ensuring that you do get prime numbers as the basis for the keys. We do typically use probabilistic approaches to computing primes - but without exhaustive analysis, we can't be sure that the probabilistic approaches are correct. If you're generating your own key-pair, you should throw as much computational power at it as you possibly can - in order to ensure, to the best of your ability, that the primes you use for your keys are really honest-to-goodness primes.

In general, though, most standard implementations do that. They give you a choice of how long you're willing to take to generate the keys, and they test the primes and the generated keys against the basic attacks to ensure that you haven't ended up with an easily crackable key-pair.

Dorthy Denning solved this problem long ago. Philip Zimmerman and Bruce Schneier described and implemented solutions to these problems long ago.

See for example:

Davida, G.I. "Chosen Signature Cryptanalysis of the RSA (MIT) Public Key Cryptosystem," Technical Report TR-CS-82-2, Department of EECS, University of Wisconsin, 1982
Denning, D.E. Cryptography and Data Security Addison Wesley, 1982

"Cryptographic algorithms are very easy to get wrong, and the slightest mistake can completely undermine the security of the system."

This is true.

But, to be honest, if you are a very good programmer, and are confident in your ability to rigourously and automatically test your code with a custom built regression test suite, and you are confident in your ability to identify boundary conditions, and you have spent a good deal of time learning about other's mistakes, you might have a more trustworthy implementation if you roll your own. But it is not for the weak kneeed.

Actually, I would say that if you're a very good programmer, and are confident in your ability to rigourously and automatically test your code, etc., you're exactly the kind of person who should know better than to write your own crypto code.

The thing about crypto is that there's a lot more than just boundary cases. There are tons of corners where the slightest mistake can totally screw you. And the confident solo programmer is exactly the guy who's going to make that kind of mistake.

Real serious crypto code requires at least three groups of people:
(1) a group of programmers to write the code.
(2) A group of testers to write the test, who have never seen the actual implementation, and who know nothing more about the design of the code than the formal description of the algorithm.
(3) A group of crypto experts to review and audit the design, code, and tests.

OT1: It is an ironic coincidence that you posted your response twice. I just cut, pasted, and saved both versions of your response, and I did not detect any difference in information in the in second (#10) compared to the first (#9). Was the duplicate intentional?

OT2: Is the typepad login feature a scienceblogs imposed requirement, or just for goodmath?

You might also consider adding a fourth team that systematically inserts faults into a sandbox version of the design, and then subjects the intentionally sabotaged version to the test suites, as a form of quality control.

The additional amount of resources necessary for this is relatively small, and it is a relatively productive use of resources.

But I haven't had much luck convincing others of this fact.

William:

We're still getting used to the newly upgraded MoveableType installation, and there are a few glitches as we get it up and running. When I tried to respond to you yesterday, the system was timing out posting comments. One attempt seemed to fail - it timed out, and the comment didn't appear. But apparently it did get done by the backend, and just hadn't appeared yet. So I reposted, only to discover it had already successfully posted.

The typepad stuff isn't intended to be a requirement; I'm trying to set it up as an option, where you can use it if you want, but you don't have to.

I agree that a fault-based testing scheme is a really excellent idea for serious crypto code.

But at the moment, convincing people that they shouldn't just roll their own crypto for the fun of it is difficult
enough. Once you get people to understand how tricky it is to get crypto implementations to be really secure, then I think getting them the rest of the way isn't that hard.

Hi there.

Please excuse a total novice. I have always wondered why encryption is not used in tandem. That is, encrypt messages twice with two diffrent algorithms and/or encryption keys.

This would, not make the encryption stronger per say but, it seems to me, the test to confirm success should become almost impossible.

Say you scramble an RSA encoded padded message, with a simple substitution/transposition cipher. Musn't you then first crack the simple cipher to be able to attack the RSA?
And how do you confirm that you have cracked it?

This has always puzzled me. I would be glad if you could enlighten me.
/Chasapros

Public Key Cryptography using RSA

goodmath — Mon, 01 Dec 2008 08:29:57 +0000

Public Key Cryptography using RSA

Technorati Tags: cryptography, public-key, encryption, RSA, asymmetric encryption

The most successful public key cryptosystem in use today is RSA - named for its inventors Rivest, Shamir, and Adleman. I first learned about RSA in grad school from one of my professors, Errol Lloyd, who was one of Ron Rivest's students. Errol is without a doubt the best teacher I've ever had (and also a thoroughly nice guy). If you want to go to grad school to study algorithms, you frankly couldn't do better than heading to Delaware to work with Errol. I have very fond memories of Errol's class where we talked about this. He's got a way of teaching where he doesn't come out and tell you anything; what he does is ask questions that lead you through the process of figuring it out yourself. That's an incredibly effective way to teach if you can carry it off. Personally, I can't. And I've never known anyone except Errol who could do it for a topic like RSA encryption!

Anyway, moving on... In general, public key cryptosystems are based on problems that are easy to solve computationally in one direction, but really hard to solve computationally in the other. In the case of RSA, the basic underlying problem involves prime numbers: if you've got two really large prime numbers, then multiplying them together is easy; but if you've got a number that's the product of two really large primes, factoring it is very hard.

One of the things that makes RSA difficult to explain is that it's hard to find a starting point. The actual encryption and decryption processes are so simple that
they seem like they can't possibly work; there's just no way to make sense out
of why they work without understanding where the keys come from; but it's strange to describe an encryption system by starting with how to generate keys, when you don't yet know how they're going to get used.

The reason for this tangling comes back to the fundamental goal of a public key
system: you've got to have two keys which share a complex mathematical relationship.
The encryption system itself is really just a simple expression of the relationship
between the keys. So the algorithm is (conceptually) very simple; the whole thing is
based on the nature of the keys, their relationship, and the way that they're
generate. It's not like symmetric cryptography, where you can simply chose a key; in
public key systems, the keys have to be generated to satisfy a set of mathematical
relationships.

With that it mind, we'll start working through the way that RSA works by showing
how you can create a public/private key pair. The key generation process is actually
pretty simple, but most descriptions of it get tangled up because it uses a lot of
terminology from number theory. I'll try to present the standard terms as clearly as I
can.

The basic structure underlying an RSA key pair is a pair of two large prime
numbers. What's large? That depends on how hard you want it to be to crack. The
tradeoff is that the larger the original pair of primes are, the more complex it is to
compute ciphertext using a key. So you need to choose a key size which makes your keys
hard to crack, without making the cost of encryption and decryption unacceptably
high. Once you've decided on a key size, that dictates the size of your two prime numbers, and you're ready to compute a key pair, as follows.

Compute two large primes, which are typically named P and Q.
Using P and Q, you compute a number N=P×Q, which is called the modulus of the keys being generated.
Compute the totient of the modulus. For any integer, I,
the totient of I (written φ(I)) is the number of integers smaller
than I that are relatively prime to I. Because P and Q are prime, φ(P×Q)=(P-1)×(Q-1). (The totient of P is P-1, because there are P-1 numbers relatively prime to P; the totient of Q is Q-1 for the same reason; and since P and Q are (trivially) relatively prime to each other, the totient of P×Q is (P-1)×(Q-1)).
Choose an integer, E, smaller than and relatively prime to φ(N). E is
called the public key exponent.
Compute an integer D such that D×E=1 mod φ(N). D is called the private key exponent.

The public key is the pair (N, E) of the modulus and the public key exponent; and the private key is the pain (N, D) of the modulus and the private key exponent. So you've got your key pair.

Encryption and decryption are amazingly simple.

Suppose that the ubiquitous Alice and Bob want to communicate. Alice gives Bob her
public key, (N, E). Now, when Alice wants to send a message to Bob, she encodes the
plaintext of the message as an integer, M. (I'll leave the exact encoding of plaintext open for now.) To encrypt with her private key, (N, D),
she takes that integer and computes:

Ciphertext = M^D mod N

Then to decrypt the message, Bob uses his key pair, and computes:

M = Ciphertext^E mod N

For Bob to encrypt a message for Alice, he does exactly the same thing that he did to the ciphertext - except he does it to the encoded message, M. For Alice to decrypt that, she does exactly what she did to encrypt the original M, except that she uses the ciphertext she recieved from Bob instead of the encoded plaintext M. In other words, if you've got a ciphertext message encrypted by the private key, decrypting it is exactly the same process as encrypting a plaintext with the public key, and vice versa. (This point is what used to cause me lots of confusion remembering what was symmetric and what was assymetric - RSA style asymmetric encryption is really very symmetric in how the algorithm works.)

How can this possibly work? On the face of it, it looks ridiculous! You encode by exponentiating once; you decode not by taking a root, but by exponentiating again!

It all comes back to the way the keys were generated. If we look at the process
in terms of modulo arithmetic, it's pretty easy to see why it works:

Take an original message, M. Encrypted, it's
C = M^D mod N.
Now, take the ciphertext, C, and decrypt it.
M' = C^E mod N.
Now, expand C: M' = (M^D mod N)^E mod N.
Now, we can combine the exponents: M' = M^D×E mod N.
D×E = 1 mod N.
By some trickiness, related to the fact that D and E are relative primes related to both each other and N by their relation to the
totient of N, we can show that the fact that D×E = 1 mod N means
that M' = M^D×E = M¹ = M.

Watch how we can walk through a ridiculously simplified example. Let's start with
a pair of primes - 29 and 61.

Generate keys:
- First, we compute the modulus: 29×61=1769.
- Then we compute the totient: 1680.
- Then we choose an E which is relatively prime with 1680. To make things easy,
  I'll just pick a prime number: E=13.
- Now, I need to compute a D, such that D×E=1 mod 1680;
  or to pull out a bit of standard terminology, D is the modular multiplicative
  inverse of E mod 1680. Doing that is an exercise in modulo arithmetic
  which gets beyond the scope of what I want to talk about today, so I'll
  cheat: whipping out Mathematica, I get D=517.
- So, the public key is (1769,13), and the private key is (1769,517).
Now, let's encode a message. Say the message is 236. So, encrypted by
the public key, that's 236¹³ mod 1769 = 7044595136617487310722334457856 mod 1769 = 573.
And now, let's decode it. That's 573⁵¹⁷ mod 1769 = 9245694881849602770197401240655288154608138663291308421093411147578949\
4462244595981994549450775979702694154377539110666284415651476199963925\
1321729713042853618725578035043275339491371655153997356248940804495270\
8984801258907252216498797520780679511957335315751292762455167434140192\
4399386435156204841871858091160737587648566008200581107378219553124074\
9374669246678685272914050993442585237015640426625522901746517901417734\
7954757584818934596889432709457570226928654243870833959974236930834811\
3204280174093386319717080086558480154647619900966739378086145377566860\
9195850562761848872414474511076491179637551417498920992360433000710853\
3935087767676514264669956171228256961906386882369681633698604708335082\
4532038380922358459546076540454839797340473363177061704334483341984626\
0992808331110751927026090969022077767175228102822099312339565763871188\
4006940217478961345132678704142880421227822352750998264777937728911493\
6283819708613556665220171457755862562705567302041244568283475209225680\
3828781673802145987568796646578059853165517716374603716155560308698631\
2650416428650673915642280074852668462543146649056536263032788685526436\
7863995916605455190338263161582118236712167656565774640875461698797822\
0025671340803745351386743506255677806855116853206286798808454276083160\
2812779603110688541701764925723493557773173917037390830611160570524069\
7471206139919064156642428626093922418127017110711925314998235480530680\
56675958655700624098981453 mod 1769 = 236.

So it works - but computing those results is not exactly trivial. It happens that there is a fast algorithm for doing those sorts of exponentiations - but "fast" is a relative term. RSA encryption is a lot slower than most symmetric encryptions, by a significant factor.

In real-world use, RSA is rarely used for full-message encryption. It's used for signatures (where you encrypt a small summary of a message), and for protocol negotiations to allow the two sides to settle on a symmetric encryption key to
use for the rest of their communication.

You'll notice that I still haven't gotten back to how you take a message
and convert it to an integer. That's not trivial. The problem is,
for certain message sizes, or messages with certain numeric properties,
RSA can be easily broken. So you need to protect against those cases. That's
done by padding the message - adding bits to it in a way that removes
or obscures the properties of the message that would otherwise make it vulnerable.
Discussing the vulnerabilities of RSA for certain types of messages, and how
to build a padding system that defeats them is a topic for another post.

In future posts, I'll also describe the two most common uses of RSA in more detail - both using RSA for signing a message to prove that you really wrote it; and using
RSA as part of a secure cryptographic protocol where most of the traffic is really
encrypted using a symmetric algorithm.

goodmath Mon, 12/01/2008 - 03:29

Tags

Encryption

Categories

Free Thought

Ah! Brilliant, thank you for that post! I always wanted to understand how RSA works. Can I ask for posts here? (Or make a couple of suggestions anyhow). I'd love to see an explanation on the Off-The-Record Messaging protocol, in particular what makes it so different than others and how this was done... and also, drifting a bit from your original intention with this series of posts, I'd like to know how my privacy is assured by third parties... banks, web applications, mobile phone companies. What are they doing to protect my privacy, and should I trust them?

"He's got a way of teaching where he doesn't come out and tell you anything; what he does is ask questions that lead you through the process of figuring it out yourself."
I think that's colloquially known as the Socratic Method. The same style is used in book form in the The Little Schemer (formerly The Little LISPer, I believe) to teach Scheme. On the darker side of things, someone who is very skilled at arguing can use this technique to great effect to trap you with your own reasoning.

It's indeed a nice post, but I have a question: in the examples you give, encryption is done with the private key. Isn't this the way digital signatures are done? (Actually on real systems it's not, as they are typically hybrids that use both asymmetric and symmetric algorithms, to overcome the speed limitations you mentioned, but I digress).
My point is, by saying things like encrypt with her private key, without explaining that's for digital signatures (instead of encryption proper), aren't you creating a source of confusion? Even if you reserve the details of public key system for a later post, I think you should have at least mentioned that here. But as I said in the beginning, it's a nice post nonetheless :-)

Some nitpicking about academic credit...

RS and A were RE-inventors of the algorithm. It had first been discovered in the UK by James Ellis and Clifford Cocks of the GCHQ. They were soon joined by Malcolm Williamson, who went on and invented what became known as the Diffie-Hellman(-Merkle) key exchange. Unfortunately their role remained hidden, because they were not allowed to publish their discoveries at the time.

You can read the full story in the Code Book by Simon Singh.

Oscar:

Public key encryption is used for digital signatures, but that's not the only use of it. And even when it is used for signature, the way that it's used is to encrypt a section of the message which contains an authentication code for the rest of the message.

I'll talk about this in a later post, but a quick outline of how secure digital signatures work is:

(1) Take a message, M, that you want to send. You don't
care if other people can read M, but you want the
intended recipients to be absolutely sure that it was
you who sent it.
(2) Using a special kind of hash function, compute a value
D, called a digest of M, where a D is a short code
that is produced solely from the content of the message.
(A good digest has a set of interesting properties, but
I'm not going to list them here.)
(3) Encrypt D, using your private key, and append
the result to the end of your message. The encrypted
digest is called a signature.

Now, anyone who sees the message M can compute D. They don't
need any kind of key - the value of D is solely dependent
on the content of the message.

Anyone can verify that you sent the message, because
only you could have encrypted the digest in a way that could be decrypted with your private key.

The big advantage of this technique is that there's no secret that needs to be shared with your intended recipients. Unlike a MAC in a symmetric encryption system,
the digest is not encrypted, and anyone can generate it. But only you could produce an encrypted copy of the digest.

So digital signature is really just a particular use-case of public key encryption that encrypts the message digest instead of the entire message.

You can see this in a nice, visible form if you use the FireGPG plugin for Firefox. Given a message, you can
encrypt it without signing it; sign it without encrypting it; or encrypt it and sign it. In all three cases, the encryption is RSA; in two of the three, RSA is used to encrypt the entire message.

Xzy:

I know that it's called the socratic method; but in my experience, a lot of people don't know what the socratic method was. I figured this post already had enough
jargon in it without needing to introduce the name for that teaching method, when I would still need to include the description.

And actually, I've never seen it used in a serious way to tangle people up. In various texts, particularly Plato, I've seen it used to tangle up extremely stupid interviewees. But I've never seen it used in real life to tangle up someone who wasn't already being an idiot to begin with. That sort of rhetorical use of it is generally a dressed up form of proof-by-contradiction, and the questioning process is really just for show.

Nice article, just a small nitpick:
Perhaps you should clarify that you can only effectively compute D because you know the primes p and q. After all that is what makes the system secure.
If you could just plug E and M into Mathematica and find the modular inverse, anyone could do this, since (E,M) is the public key. But luckily the Extended Euclidian Algorithm require knowledge of the prime factors of M to work.

Mark:

You're right, but the use you make of the verb "encrypt" is somewhat loose. I mean, usually when one talks about "encrypting", one means "to write in a form that cannot be easily read", and you absolutely cannot do that using ("encrypting") with the private key (because everyone with the corresponding public key will be able to "decrypt" that message). But after reading this paragraph:

Suppose that the ubiquitous Alice and Bob want to communicate. Alice gives Bob her public key, (N, E). Now, when Alice wants to send a message to Bob, she encodes the plaintext of the message [...]. To encrypt with her private key, (N, D), she takes that integer and computes [...]

This might seem to suggest that the private key should be used to do encryption (as in "only the intended recipient gets to decrypt") which is far from being the case.

Final remark: in your reply, you said:

Anyone can verify that you sent the message, because
only you could have encrypted the digest in a way that could be decrypted with your private key.

You meant public key, right? :)

Lassi: GCHQ did not invent the RSA algorithm. They invented the algorithm also known as DH. While both are public key methods, they are quite different.

Mark: Oscar is right. You used the private key parameter, D, as part of the encryption step ("C = M^D mod N"). Like nearly all mistakes with cryptography, the impact is that your described system is totally insecure. I've already written articles (parts 1 and 2) about how to calculate E and N given only two messages encrypted with D. In your system where D, N is the public key, the attacker can calculate E, P, and Q without seeing any encrypted messages.

Take a moment to think about this. Inverting the sense of the E and D exponents in relation to encryption and decryption results in a system where your "private" key is no longer private! This is because of the asymmetry in selecting them, something which you used in your example for choosing the parameters.

While I appreciate your enthusiasm for cryptography, the lack of attention to detail in your posts is troubling. In crypto, being exact with your terms and paranoid in your implementation really matters. Thanks.

You mentioned that for too small M, RSA is easy to break, and that your typical encoding function to turn text into M will add padding for this reason. However, since everything relating to E and D doesn't depend in any way on the way M is encoded, why can't you just use a modified encoding function that leaves out the padding, pick a plaintext to make an M that is small enough, and then do whatever you can do when M is small to break the key?

Your detailed list had a small error: it's DÃE=1 mod Ï(N), not DxE=1 mod N.

In fact, I'm a little surprised you didn't mention the core relation:

aÏ(N) = 1 mod N

(from Euler, as I recall).

if you use the basic definition of D and E,

D * E = 1 mod Ï(N) â D * E = n Ï(N) + 1

then

MD*E = Mn Ï(N) + 1 = M*Mn Ï(N) = M*(1n) = M.

Also, on the totient of a product... you were a little too casual imho. It's not hard to explain.

For the number from 1 to pq, you need to eliminate all multiples of 'p' (p, 2p, 3p, ..., qp) and all multiples of 'q' (q, 2q, 3q, ..., pq), and then restore the duplicate of 'pq'.

That gives you pq - q - p + 1, or (p-1)(q-1) if you factor it.

A really cool way of looking at it geometrically is to draw a grid of p by q cells. Color one edge -- that's the multiples of 'p'. Color an adjacent edge -- that's the multiples of 'q'. What's left is the smaller rectangle (p-1)(q-1).

Very nice article. RSA really is quite astonishing in both its simplicity and its effectiveness. It always amazes me that although cryptography has been around for thousands of years, it was only 33 years ago that public-key crypto was conceptually invented and (in the classic Diffie-Hellman paper) and 32 years ago that RSA was designed.

One minor point to clarify: You calaculated 573517 mod 1769 by calculating 573517 first (giving a 1426-digit number) and then reducing the result mod 1769. That's a big number before reduction mod 1769, and it can get a lot bigger in practice (where the p and q are not 2-digit primes but more like 200-digit primes). So we wouldn't actually calculate it that way, but rather use a repeated squaring technique where we start by computing 5732 mod 1769 (=1064), then 5734 mod 1769 (=10642 mod 1769 =1705), then 5738 mod 1769 (=17052 mod 1769 =558), and so on up to 573512 mod 1769, then use the results of these calculations to find 573517 mod 1769. This keeps the digit size of the numbers involved down and therefore is a lot faster and more memory-efficient.

Of course you knew that, and were just keeping it simple for us, right? :)

Ditto on Lloyd.

I've said it before, but I'll say it again: my math is sorely lacking -- though I recognize number theory being used, mostly -- but I'm really enjoying these posts.

Asymmetric Cryptography: the Basic Idea of Public Key Cryptosystems

goodmath — Thu, 20 Nov 2008 13:36:25 +0000

Asymmetric Cryptography: the Basic Idea of Public Key Cryptosystems

I've been trying for a couple of weeks to put together a couple of interesting
posts on the cryptographic modes of operation for confidentiality and integrity, and I
just can't do it. I'm finding it boring to write about, and if it bores me to write it, I know there's no way that it's going to be engaging to readers!

So, I'm going to move on. I've explained the basic idea of the message
authentication code as an integrity check, and I've described one simple way of
integrating it into a common mode of operation. If you're really interested in
learning more, I recommend Bruce Schnier's book on cryptography, which has ton of
material on modes of operation and protocols, how they work, and how they can
fail.

Meanwhile, I'm going to move on to something that doesn't bore me to write about,
and therefore hopefully won't bore you to read about: asymmetric
cryptography, also commonly referred to (although not entirely accurately) as public
key cryptography.

What we've looked at so far is symmetric cryptography. The basic idea of it is illustrated to the right. In a symmetric
cryptosystem, you've got two parties who want to communicate, and they've got a
shared secret. That shared secret is the key to the cryptosystem. Put in
pseudo-mathematical syntax, the symmetric cryptosystem is a pair of functions, En and De, such that given a secret key S:

Ciphertext = En(S, Plaintext)
Plaintext = De(S, Ciphertext)

We call it symmetric because both the sender and the receiver need the
same information. Both of them can encrypt messages using the key; both can
decrypt. There's no way to distinguish between the two on the basis of what
information they have, or how they encrypt their messages.

In an asymmetric system (as illustrated above) that's no longer true. Asymmetric systems use
two keys, S₁ and S₂ (also called the private key and the public key) such that:

Ciphertext_S₁ = En(S₁, Plaintext)
Plaintext = De(S₂, Ciphertext_S₁)
Ciphertext_S₂ = En(S₂, Plaintext)
Plaintext = De(S₁, Ciphertext_S₂)

In english, suppose you've got two people, Ann and Bob who want to communicate.
They each have their own key. To send a message to Bob, Ann encrypts the
message with her key. When Bob receives the message, he decrypts it
not with the same key that Ann used to encrypt it, but with his own
key. When he wants to send a message back to Ann, he takes the same key that he used
to decrypt the message from Ann, and uses it to encrypt a message
to Ann.

This leads to one of the basic terminological confusions. Reading that
description: Bob's key decrypts messages from Ann, and encrypts messages
to Ann has an element of symmetry to it. There is a very elegant symmetry to
asymmetric encryption! But the terms asymmetric and symmetric in cryptography refer to
the information that each of the parties uses in a cryptosystem. In a symmetric
system, both parties have exactly the same information. In an asymmetric
system, they've got different information.

For the most part, the basic requirements of an asymmetric cryptosystem are
very similar to the requirements of a symmetric one - you want to be able to compute the ciphertext easily given the encryption key and the plaintext; you want to be able to compute the plaintext easily given the ciphertext and the decryption key; you don't want to be able to compute the key given the ciphertext and the plaintext,
and so on. But there's one major complication in asymmetric cryptosystems that creates an additional requirement: given the encryption key, it must be
computationally infeasible to generate the decryption key.

That last requirement kills the overwhelming majority of potential and proposed
asymmetric systems. Having one of the keys in a pair means that you've effectively
got access to an unlimited chosen plaintext attack: you've got the ability to generate
the ciphertext of any plaintext, as many times as you want, and to use that to try
to find the value of the key that will decrypt that text.

The most common use of asymmetric cryptography is public key cryptography. One of the really fascinating things about it is that instead of trying to keep the keys private, you deliberate make one of the two keys widely available - you give it to everyone.

The idea is that you take one key, which we call your private key, and you lock
that away. No one but you should ever have access to your private key. The
second key, called your public key, you give to everyone. You post it on public
websites, you register it with key libraries, you send it to all of your friends, etc.
Anyone who wants to be able to read encrypted messages from you needs to be able to
get your public key.

Now, when you want to send someone a message and prove it's from you, you encrypt
it with your private key. Anyone can decrypt it - but only you could have
encrypted it in a way that allows your public key to decrypt it.

Similarly, when someone wants to send you an encrypted message, they encrypt it
with your public key. Then only you can decrypt it.

To set up a secure cryptographic channel with someone, you need to have
their public key. So, suppose that we have Amy and Bill, who want to send
secure messages to each other. What are the goals of the cryptosystem here? When Amy
wants to send a message to Bill, she wants to make sure that only Bill can
read it. When Bill gets a message from Amy, he wants to make sure that only
Amy could have sent it.

The way we can provide both of those is by having Amy take her message, and
first encrypt it with her private key. Then she encrypts it with
Bill's public key. The message has been encrypted twice: the first encryption
pass guarantees that the message is from her (because no one else could have encrypted
a message with her private key); the second encryption pass guarantees that no one but
Bill can read it (because no one but Bill can decrypt a message with his private
key).

Bill does exactly the same thing when he wants to send a message to Amy. First he encrypts it with his private key, to show that it's really him who sent the message.
And then he encrypts it with Amy's public key, which ensures that only Amy can read it.

Towards the beginning of this message, I said that asymmetric cryptography is sometimes incorrectly called public key cryptography. The reason that I said that is because public key cryptography is really the combination of an asymmetric cryptosystem together with the infrastructure that allows the kind of scenario I described above, for Amy and Bill, to work. For it to work, Amy and Bill need to have a way of getting each other's public keys - and being sure that they are really getting the correct, valid public keys for each other. There's a very elegant cryptographic attack called a man in the middle attack, which relies on weaknesses in the public key infrastructure, not in the actual asymmetric cryptographic system used by the public key system, which I'll describe in a future post.

goodmath Thu, 11/20/2008 - 08:36

Tags

Encryption

Are you going to explore the different types of asymmetric encryption? I've always been interested in Elliptic Curve Cryptography, but not due to any real understanding of it, just because NeXT had it when I was young, and NeXT was cool. :-)

Years ago I had the idea of burning the private key into a processor chip during manufacture, in such a way that the processor could use the key but it was unavailable from the outside both physically and electronically. The manufacturing equipment that created the key pair would print the public key on the casing or otherwise publish it and then forget the private key completely.

The chip would then be uniquely self-identifying; it can encode and decode with the private key, and no one else can. Nowadays you could build the chip into a cell phone, add some really good biometric hardware to make sure that only the owner can use it (say a DNA scanner), and you have a nearly unbreakable communications and identification system.

We applied for a patent and got it; it was assigned to my employer at the time, Prime Computer. After awhile I left, Prime died, and the patent expired. I really wish I knew if it was a good idea or not.

It was a good idea, but it was born too early. http://en.wikipedia.org/wiki/Trusted_Platform_Module

What is wrong with using plain old "Alice and Bob"?

Tobias:

Nothing's wrong with Alice and Bob. It's just boring to always use the same names, so I went for a variation. :-)

Robert:

How far I get in asymmetric encryption will be determined by a combination of how long it takes before I get bored, and how well readers respond to it. If lots of people really want to know about elliptical-curve encryption, I'll probably wind up writing about it.

I can't speak for Alice, but I'm sick and tired of just encrypting and decrypting the whole day long. Half the time it's just another string from lorem ipsum.

I have to do this by moving rocks around in a desert, you know. It's not fun.

Bob: It doesn't sound like such a great idea to me. Consider:

1. The chip gets fried. Now no one, including yourself, can decrypt or sign any messages to/from you. Oops.

2. The chip gets hit by a random alpha particle that flips one of the bits in the secret key. Similar to #1, except that you may not realize the problem for a while, and send out a number of critical messages signed with a random key.

3. You really have to trust the manufacturer not to keep an extra copy of the secret key, don't you? In view of #1 and #2, this may even be what a lot of the customers want as a backup plan. But then you have to trust that the manufacturer doesn't make an extra copy of the key list for the NSA, or whoever. Given how easily most telecoms seem to have agreed to break the FISA wiretapping law at the government's request, how far do you think a manufacturer can be trusted? Now consider that a lot of chip manufacturing is currently taking place offshore...

Lassi: Thanks. I hadn't looked at that effort for a couple of years. They talk about doing a hash of the OS code to verify that it hasn't been tampered with, something I had also in my patent. I really think that if the patent hadn't expired, they'd be subject to it.

Dave W:

#1: Losing the key is always something you have to worry about with encryption. You just have to keep the possibility in mind when you're deciding how to use it. For instance, I wouldn't use a key embedded in my cell phone for important, long-term documents, because I might lose the phone. I wouldn't store important documents in just one place, or encrypt them with just one key. I wouldn't store all my keys in the same place.

#2: Checksums and other redundancy can reduce this problem to near-zero probability.

#3: There's always going to be something or someone that has to be trusted somewhere in the implementation and use of a secure system. If the manufacturer can't be trusted, then you need an oversight agency that inspects and monitors their equipment. And you have to trust that agency.

#1: Make spares.

#3: Use a field-programmable gate array.

...Bruce Schnier's book on cryptography...

Which one?

Wait a minute, Mark, you have this wrong.

In public key cryptography you don't have Alice encrypting with her private key and Bob decrypting with his private key (as you write in the text and the picture suggests)!

Alice encrypts with Bob's public key and sends the message to Bob, who can decrypt with his private key.

Alice only uses her private key to sign the message (or better: a hash of the message) and Bob can use her public key to verify that she did the signing.

At least as far as I have understand asymmetrical cryptography - how could decryption possible work with two private keys that have nothing to do with each other?!

RE: WiseWoman (comment 10)
> In public key cryptography you don't have Alice
> encrypting with her private key and Bob decrypting
> with his private key.

Mark has it right.

You _DO_ have A encrypting with her private key, in order to "sign" the message. If it were signed with her public key, it wouldn't be secure (and B would need A's private key). If it were signed with B's public key, it wouldn't be verifying A's identity. And we hope that A doesn't have B's private key, so that leaves us with only one option.

And you _DO_ have B decrypting with his private key, as that's the heart of the security. If it were decryptable with his public key, anyone could do it. If it were decryptable with A's public key, it would defeat the purpose. And if it were decryptable with A's private key, B would be out of luck.

No - the private keys have no connection! Alice cannot encrypt with her private key and Bob decrypt with his private key, that's nonsense.

We have to differentiate signing and encrypting (two different things, that need to be done with two different keys, or there are attacks that can succeed):

Alice signs the message to Bob with her private key, and encrypts with Bob's public key.

Bob decrypts with his private key, and verifies the signature with Alice's public key.

Normally, however, Alice prepares a hash of her message, signs that, and then encrypts the whole mess, to keep from encrypting twice. Asymmetric cryptography can be 1000 times slower than symmetric.

@WiseWoman

You and Michael are saying the same things really, he's just taking it for granted that the order of actions is:

1) A encrypts with A's Private Key.
2) A encrypts with B's Public Key.
3) B decrypts with B's Private Key.
4) B decrypts with A's Public Key.

As you pointed out, 1 and 4 are usually described as signing (and it's convenient to just hash and sign said hash, as AC systems are slow). There is no real mathematical difference between signing and "encrypting", it's simply a different way of using the cryptosystem (at least with RSA, which is the algorithm I am most familiar with).

Since WiseWoman touched on the speed of Asymmetric Cryptography, another common practice is only using an Asymmetric Cryptography algorithm to encrypt a Symmetric Key, and to encrypt the payload with a Symmetric algorithm (AES/Twofish/3DES).

Also, +1 vote for a posting on Elliptic Curve Cryptography.

How Not to Do Message Integrity, featuring CBC-MAC

goodmath — Fri, 24 Oct 2008 12:21:52 +0000

How Not to Do Message Integrity, featuring CBC-MAC

In my last cryptography post, I wrote about using message authentication codes
(MACs) as a way of guaranteeing message integrity. To review briefly, most ciphers
are designed to provide message confidentiality - which means that no one but the
sender and the intended receiver can see the plain-text of the message. But
ciphers that provide confidentiality don't necessarily make any guarantees that
the message received is exactly the message that was sent. There are a good number
of cryptographic attacks that work by altering the message in transit, and
depending on the cipher, that can result in a variety of undesirable
results.

For example, if you use DES encryption with the ECB mode of operation,
you can insert new blocks anywhere in a message that you want. By using
a replay attack (where you take encrypted blocks from other messages using
the same encryption, and resend them), an attacker can alter your messages, and
you won't be able to detect it.

So in addition to just confidentiality, we need to provide integrity. What does integrity really mean? Basically, it expands the definition of the
decryption function. Written as a function signature, confidential message
decryption is a function decrypt : ciphertext × key → plaintext. With message integrity, we add the
option that decrypt can return a result saying that the message is invalid: decrypt_integ : ciphertext × key → (plaintext | REJECT).

In the last post, I explained the basic concept behind integrity schemes: you add an addition chunk of data to the message, which summarizes the message; any change to the message will (with a very high degree of probability) result in
a change in the authentication code; so if the message is altered, that change
will be detected, because the authentication code will not match the rest
of the message.

The idea of the message authentication code seems quite simple. We know lots
of pseudo-random functions that can do a good job as MAC functions. But like
so many things in cryptography, the fact that the concept is simple doesn't
mean that the actual application of that concept is simple: it's incredibly
easy to get it disastrously wrong, even when all of the building blocks are right.

For example, let's look at a very simple (and very weak) integrity
system, called CBC-MAC.

In CBC-MAC, you compute an authentication code by running a your
block cipher on the message with an initialization vector set
to 0. The output for encrypting the last block is your MAC.

The reason that this works is fairly simple - CBC keeps pushing the
result from the previous encryption step through to the next one - so
the output from the final block conceptually includes information
from all of the other blocks. So you've got an authentication code
whose security is, roughly, equivalent to the security of the block
cipher used in CBC mode.

CBC-MAC isn't a bad mode of operation. It's got serious flaws - the biggest
one is that there are a number of attacks against it if it's used for
variable length messages. (And you can't fix it just by adding a message
length indicator to the message, it's deeper than that.) But CBC is
a pretty good integrity mode for fixed-length messages, and there's a widely
used variant, called CMAC, which works for variable length messages.

So how can this go wrong?

What do you use as the password to the MAC computation? CBC is
a secure cryptographic mode, so why not just use the same password? Lots
of novices (and even some not-no-novice folks!) have implemented CBC-MAC
where the integrity and confidentiality modes used the same key. If you
compute the MAC using the same key as the message, then you're open to
an attack which completely voids the integrity guarantee!

Imagine a message M, which is divided into n blocks: (M₁, ..., M_n). For convenience, we'll treat the IV as if it were ciphertext block 0. Tuen using CBC with a key K, M encrypts to ciphertext blocks
(C₁, ..., C_K), where C_i=Encrypt(M_i ⊕ C_i-1, K), plus a CBC-MAC integrity block. If you look
at the CBC block, if it's using the same key as the main encryption,
then the value of the MAC is the result of computing
Encrypt(M_n ⊕C_n-1, k). That value
is exactly the same as C_N! So it's trivial
to fake the MAC. If you use CBC-MAC with the same encryption key as
you used to encrypt the message, you're wasting your time: you might as well
not bother with the MAC at all, because you've got no integrity guarantee.

But that's an amazingly common mistake. A huge number of implementations
of supposedly secure cryptosystems that claim to guarantee integrity have
actually used CBC-MAC with one key.

I mentioned above that CBC-MAC is only safe for fixed-length messages. The reason for that is closely related to the trick I described above. Basically, the nature of CBC ultimately means that given a MAC value M, it's not hard to generate a string to append to a message which will result in it having the desired
MAC. Suppose we know that the message M will generate the mac T_M,
and we know that the block we want to replace the message with, N has
the MAC T_N. Then we can easily generate a string N', such that
appending MAC(M,N) = MAC(N). All we do is XOR the first block of N
with T_M, and add it to the message. The new message
is is (M₁,...,M_n,(N₁⊕T_N),
N₂,...,N_n) - that is, we can remove the MAC from the
first message, use it to glue the two messages together, and tack T_N on to the message as the MAC. T_N, the MAC computed for T alone, will
be the MAC for the combined message.

It's easy to do that if we're allowed to change the length of the message. If the receiver doesn't know how long the message is going to be, then they can't
count on its integrity using CBC-MAC.

Next time, I'll look at a variation of CBC-MAC that provides integrity
for variable-length messages using 2 keys - one for confidentiality, and
one for integrity. Then I'll go on to a two-pass single-key mode of operation
that provides both confidentiality and integrity.

goodmath Fri, 10/24/2008 - 08:21

Tags

Encryption

Categories

Free Thought

Hey Mark,

Just a suggestion, but you should really go back through all these posts and tag them with "Cryptography" or something so that they're easy to dig up (assuming SB supports tagging, of course).

Differential Cryptanalysis

goodmath — Thu, 02 Oct 2008 13:07:13 +0000

Differential Cryptanalysis

Now, we're finally reaching the point where the block-cipher stuff gets really fun: block cryptanalysis.

As I've explained before, the key properties of a really good
encryption system are:

It's easy to compute the ciphertext given the plaintext and the key;
It's easy to compute the plaintext given the ciphertext and the key;
It's hard to compute the plaintext given the ciphertext
but not the key;
It's hard to compute the key.

That last property is actually a bit of a weasel. There are really a wide variety of attacks that try to crack an encryption
system - meaning, basically, to discover the key. What makes that
statement of the property so weasely is that it omits the information available to the person trying to crack it. In the first three properties, I clearly stated what information you had available to produce a result. In the last, I didn't.

There's a reason that I weaseled that. Partly, it's because a correct statement of it would be ridiculously long and incomprehensible; and partly because it's often deliberately set up differently for different encryption systems. You can design systems that are extremely strong against certain attacks, but not so good against others. There's no universally ideal encryption system: it's always a matter of tradeoffs, where you can handle some scenarios better than others.

Today we're going to look at one particularly fascinating attack that's used against block ciphers. It's called differential cryptanalysis.

So what is differential cryptanalysis? It's a technique that looks at how making specific changes in the plaintext input to the cipher produce changes in the output from that cipher. The basic idea, in its simplest form, is that you're able to select a collection of plaintexts, and encrypt them. Then based on the results of doing
that, you try to compute the key.

Differential cryptanalysis is a form of the basic chosen plaintext attack. What makes it a differential attack is that you use related families of plaintext. That is, you've got a family of texts T₀, T₁, T₂, ..., where each text T_i is equal to T₀ plus some small difference. For many differential attacks, that small difference is one bit.

How can you crack a cipher this way? To start, we assume that you know what cipher we're using. For example, you know that we're using DES with a 56 bit key. So you take your initial plaintext,
T₀, and you encrypt it, giving you a ciphertext, C₀. Then you change one bit of T₀, giving you T₁, and you encrypt that, giving you C₁.

C₁ differs from C₀ in some number of bits. Now, you've reduced the problem from "What key could produce the plaintext/ciphertext pair (T₀, C₀)?" to
"What set of keys could produce the variation C₀ to C₁ by changing one bit of T₀?". I've drawn an outline of this process in a diagram to the right.

This reduction can be dramatic. For example, in one of the early proposal rivals to DES, caled FEAL, it turned out that you could crack it with a set of 8 one-block (plaintext,ciphertext) pairs. The problem is reduced from a blind search of a search-space of 2⁶⁴ possible keys to utter triviality by a set of 8 chosen plaintexts!

The subtlety of this attack is in how you choose the
set of plaintexts to use. For a typical Feistel network block
cipher, you need to do a very delicate analysis of the S-blocks. You trace out the possible paths of each bit through the network,
and using that, you can work out a set of probable differences - that is, bit patterns that are likely to be able to produce an informative difference in the ciphertexts. These probable patterns are called differential characteristics. Once you've worked out a large enough set of probably characteristics, you can use them to assemble a set of informative plaintexts that should provide you with what you need to determine the key.

It can get even more subtle than that. Since you know the algorithm, you can isolate a single stage of the cipher (as illustrated to the right), and throw bit-blocks at it in differential style. From that, you can work out the set of subkeys that could produce the observed differential result for that stage. If you do that for all of the stages, you'll get a set of keys for each stage. Figure out which subkeys keys from each stage are mutually compatible, and then combine them, and you get a very small (usually one) key that could produce valid subkeys for each stage.

You can even repeat this process on multiple levels. If you've got a block cipher like DES, and you know the structure, you can do the differential analysis of a single stage by decomposing it into its individual S-boxes. Analyze each S-box, and come up with a set of potential subkey-fragments, and then combine those to create a set of potential subkeys for the stage. Then combine the stage results.

Differential analysis also reveals something fascinating about
DES. Most texts attribute the invention of this attack to Shamir in
the late 1980s, when he published a set of papers describing
differential attacks. But when you analyze DES, you find that it's
not very susceptible to differential attacks! In fact, it
takes 2⁵⁵ chosen plaintexts to crack 56-bit DES - so differential analysis is effectively no better than simple brute force. But if you make some very minor changes to S-boxes in the algorithm, it becomes nearly as susceptible as FEAL.

So how is DES so resistant to an attack that wasn't discovered
until almost 15 years after DES was first published? It turns out
that the IBM team that designed DES had figured out differential
attacks before they designed DES in 1974. They knew about that attack and its properties, and specifically designed DES to not be susceptible to it. But at the time, they were working with the NSA to produce an official standard encryption system, and the NSA requested that they not tell anyone about the attack they'd discovered.

This, naturally, has driven a lot of interesting discussion. Why did the NSA want to keep that secret? They claim that they did it because revealing the differential attack and how DES protected against it would "weaken the competitive advantage that the US enjoyed over other countries in the field of cryptography". But
a lot of people (to be honest including me) believe that the real
advantage they wanted to maintain was the ability of the US
intelligence community to crack encryption systems used by other countries.

One last comment, and I'll wrap up this post. Things like differential cryptanalysis are one of the many reasons not to roll your own cryptographic cipher. Some of the best cryptographers in the world - people who spent all of their time working on designing secure cryptosystems - designed systems that were easy to break using this technique. If you don't know about every technique that could be used against your system, and you haven't considered how your system will behave under probing by those techniques, then you have close to no chance of designing something unbreakable. Over time, so many cryptosystems have failed, because someone didn't notice one tiny little problem, or didn't think about one obscure technique, or didn't notice one tiny little bug in their implementation. Those things are fatal to a good cryptosystem. Many of us who are math geeks have tried designing a cipher for the heck of it; how many of us who've done it are able to say that we're sure that it couldn't be cracked in five minutes of differential analysis? I'd wager that the answer is zero.

goodmath Thu, 10/02/2008 - 09:07

Tags

Encryption

It is an interesting history of what the NSA wanted to keep secret. I remember when George Davida at Wisconsin-Milwaukee had a paper requested to be withdrawn about DES S boxes in the late 80's. It seemed quite obscure.

The theory (i.e. WAG's and speculation) back then was that DES had built in weaknesses from the NSA and it was fascinating when Don Coppersmaith et al started to talk about their design process and the fact that the NSA was fine with a strong version.

"In fact, it takes 2^55 chosen plaintexts to crack 56-bit DES - so differential analysis is effectively no better than simple brute force."

I thought the differential attack was the reason why NSA recommended cutting the key length to 56 bits. Anything more was an overkill.

I don't think that differential cryptanalysis is a type of chosen plaintext attack.

Chosen plaintext attacks require access to an oracle. You must be able to submit your chosen plaintexts to the encryptor and examine (compare) the ciphertexts. In other words, you must be able to interrogate the oracle.

However, if you have *only* ciphertexts, you still may very well carry out differential cryptanalysis. Perhaps it won't get you the key, but you possibly will get near to the plaintexts.

Suppose you have a stream cipher, or a block cipher in OFB or CTR mode. The encrypting party is uneducated enough to encrypt two separate English (or other known-protocol) messages using the same IV. (There are examples for this in the industry.) Since both in OFB and CTR the ciphertext is produced by XOR-ing the plaintext with the plaintext-independent cipherblocks, by XOR-ing two ciphertexts, one XOR-s out the cipherblocks and ends up with the XOR of two English messages. Cryptographers break *that* in their sleep (letter and word frequency etc). This attack, although trivial, *is* differential cryptanalysis, because it's based on the difference between ciphertexts. However, no encryption (chosen plaintext) takes place in this attack.

Mark, it's usually not necessary to attack multiple rounds of a cipher. One round will suffice. For DES, if you find one round key (48 bits selected from the 56 key bits), you just brute-force the remaining 16 bits.

hmm... makes my Caesar cipher encryption look pretty weak. lol. Not that i use it for anything.

sean

Nate:

My understanding (admittedly limited) is that for some ciphers, you can crack the key much faster by analyzing multiple phases and intersecting the compatible results - I don't have my texts handy (they're at home, and I'm at work), but I recall one example (FEAT-128 rings a bell?) where you could reduce the amount of brute-force work by more than an order of magnitude - from brute-force testing 64,000 keys to brute-force testing 1000 keys.

It's another of those tradeoffs: is it faster to do the differential analysis on another round of the cipher, or to do a brute force using what you know?

There seem to be two contradictory stories about how DES became resistant to differential cryptanalysis.

Once upon a time, what "everyone knew" was: IBM sent their design to the NSA, who sent it back with mysteriously altered S-boxes, and it later transpired that what NSA had done was to make the design resistant to differential cryptanalysis. (Alan Konheim: "We sent the S-boxes off to Washington. They came back and were all different.")

But now the common wisdom is apparently that really IBM knew about differential cryptanalysis too (they called it the "tickle attack") and the NSA didn't change anything. (Walter Tuchman: "We developed the DES algorithm entirely within IBM using IBMers. The NSA did not dictate a single wire!")

I don't see how these can both be correct, or even both be good approximations to the truth; either the S-boxes went off to Washington and came back all different, or they didn't.

The "new" story seems to be a little better supported by evidence, but it's pretty thin.

Anyone know the real history here? (The various accounts on the web seem unhelpful; they quote one story, or the other, or quote both but notice no tension between them. Perhaps I've missed one that resolves the issue.)

Great thanks for dicrovering the topic.

I'm also confused on the history of the DES s-boxes, for the exact same reasons "g" cited just above.

Anyone know of any good sources on this? We seem to have two mutually exclusive stories being circulated.

Screwing Up Modes of Operation: Counter done right

goodmath — Mon, 15 Sep 2008 17:23:40 +0000

Screwing Up Modes of Operation: Counter done right

So, as it turned out, I made a major screwup in my post earlier today on modes of operation. Rather than just edit the post, I'm adding a new post with the corrected description of the counter mode, and a bit of explanation. I figure that if I screw up badly, it's more honest to make a second post explaining the error than it is to just correct it and pretend that all was well.

What I got wrong was the order in which things happen. In the counter mode,
you encrypt the counter using the key, and then you exclusive-or the result of that with the plaintext to get the ciphertext. The plaintext never enters the block cipher; the block cipher just produces a complex and random looking block of bits which are then used to obscure a block of plaintext.

What I said in the original post was that you exclusive or the plaintext with the counter, and then run it through the block cipher. In my screwed up version, the plaintext is being put through the block cipher mechanism; in the correct version, it's not. Below is some of my psuedo-python showing my screwed up CTR mode,
and the (hopefully) correct CTR mode. I've also included a diagram of the correct CTR mode.

def EncryptWithMarksScrewedUpCTR(blocks, ctr, key):
  for b in blocks:
    encrypted = encrypt(key, b ^ ctr)
    ctr = ctr + 1
    output(encrypted)

def EncryptWithRealCTR(blocks, ctr, key):
  for b in blocks:
    e_ctr = encrypt(key, ctr)
    encrypted = e_ctr ^ b
    output(encrypted)
    ctr = ctr + 1

This can make a big difference in the effectiveness of the cipher against various attacks. I'm not going to get into details now, but over the course of future posts, I hope that I'll be able to make it clear why changes like this can have huge impacts on
the security and quality of a cipher.

goodmath Mon, 09/15/2008 - 13:23

Tags

Encryption

You still need to include an IV in with the counter. If you use the same counter again for a different stream, it is easy to do attacks by just xoring the two streams together.

Counter mode basically turns a block cypher into a stream cypher. With that, it is important to either never reuse the counter, or never reuse the key. That's commonly done by either xoring the counter with an IV the size of the block, or setting say the upper half of the block to the IV, and the lower half to the counter.

Thanks for updating your post. David Brown is right -- without an IV, you're going to generate the same keystream on multiple messages. This means CipherText1 XOR CipherText2 = PlainText1 XOR PlainText2, revealing a lot of information about your plaintext. Are you going to revise this again?

In my last comments, I suggested that you recommend your readers use standard protocols like SSL or password hashing algorithms like FreeBSD-MD5 or OpenBSD-Blowfish. The danger is that they'll read your posts, assume they understand enough, and then implement their own versions, recapitulating all the security flaws we've learned about in the research world. As evidence of this, consider that if you fix your post to add the IV, that will be the second important flaw you've accidentally included in describing only a single mode of operation (counter mode). Wouldn't a disclaimer help emphasize how fragile this stuff is?

I make my living finding and fixing flaws in embedded and systems software, especially as they pertain to crypto. So while I appreciate increasing the supply of work, it would be good to issue a warning instead. :)

Consider fiascos like this "rainbow tables" set of comments, where web programmers attempt to reinvent password hashing rather than just reusing the FreeBSD or OpenBSD approaches. There are 105 comments to a very informative post that warns of the dangers of trying to reinvent this, and nearly all the commenters are doing exactly that!

For those of us who are learning, what is an IV?

Ah, oops, hadn't read that far in the older post. Sorry!

Rob:

The "IV" is an initialization vector. It's a collection of random bits.

The idea is that you want to "prime the pump"; that is, you don't want an adversary to ever see a simple sequence of stuff that lets them infer much about what you're doing.

If you use a simple counter, then effectively, you've got a message encoded with DES(1)xor(plaintext),DES(2)xor(plaintext), etc. Every message will look roughly like that. If your adversary somehow figures out what you're doing, then he can just do the same DES(1), DES(2), etc., computation, and have an easy crack to your key.

The IV breaks that. An adversary trying to break your encryption can't just try encrypting "1,2,3" with different passwords, and see if he gets anything that makes sense out of your message. He needs to also guess the IV.

As a computer scientist, I tend to have a somewhat odd way of looking at things, which has annoyed a few cryptographers who read this blog. My understanding of counter mode says
that the counter is a function F, where you use F(N) as an input value for block N.

In terms of functional representations of things, which is the way I think of them, the use of an IV is equivalent to a particular counter function F.

In the diagram of the mode above, you can see that I use "counter(1)", "counter(2)" as inputs to the block cipher. counter(n) could be (n xor IV) if you wanted; it works out as equivalent.

I would just like to say that I write software for a living, I have a computer science degree from a decent school, I read this blog and find it to be engaging and educational (especially these cryptography-related posts), and I would never roll my own security code. I'd like to think that most people interested in cryptography and security enough to read these blog posts have an appreciation for how hard real security is. I've been interested in this stuff for a while now and I've heard that message over and over again--the security community does an excellent job of communicating that point.

This blog provides some of the most lucid descriptions of various concepts that I've encountered. I'm learning and I love that.

I'm sure there are other readers like me. Keep up the good work.

You indeed have an IV. However, the counter and the IV need not be mutually exclusive. Since often the first thing done is to XOR the IV against the counter, you can actually just choose an IV and just start your counter from that. It's essentially the same thing, though, so if you expand your definition of a "counter", your notation works.

Hi Mark. Sorry for the late comment, I just revisited this thread. Even with your revised comment, the reason you give is incorrect.

The IV breaks that. An adversary trying to break your encryption can't just try encrypting "1,2,3" with different passwords, and see if he gets anything that makes sense out of your message. He needs to also guess the IV.

What you're saying is that the IV in counter mode prevents a known plaintext attack. That is, you're saying that knowing that the ciphertext is "DES-Encrypt(counter, key) XOR plaintext" gives the attacker a crib, since they can brute-force the key in the forward direction and see if the resultant plaintext makes sense.

This is not the reason for using an IV. The average effort required to perform this attack on DES, assuming the key is randomly chosen, is 2^63. For AES-128, it is 2^127. This is the same effort required to brute-force a single block encrypted with the cipher in ECB mode. Any cipher that is vulnerable to a known plaintext attack is already broken and should not be used. Neither DES nor AES is subject to a known plaintext attack.

The actual reason the IV should be different each time the counter is restarted is because without it, you produce the same keystream multiple times. This is fatal for a stream cipher since the keystream itself can be extracted for any portions of known plaintext.

Say you're using this for packet encryption and you don't have an IV. Once I see the encrypted Packet1 and know some header fields, I know the keystream at those points. Now I can decrypt any packet you send by just XORing those same points with the keystream.

Since an IV makes the keystream different (even though the counter is restarted), it would prevent this attack.

I forgot one more thing. Your assumption that the IV helps prevent a known plaintext attack is based on the IV being secret. An IV is never secret, and any protocol that claims it should be immediately raises red flags.

Modes of Operation in Block Cryptography

goodmath — Mon, 15 Sep 2008 08:13:04 +0000

Modes of Operation in Block Cryptography

Sorry for the slow pace of the blog lately. I've been sick with a horrible
sinus infection for the last month, and I've also been particularly busy with work, which have left me with neither the time nor the energy to do the research necessary to put together a decent blog post. After seeing an ENT a couple of days ago, I'm on a batch of new antibiotics plus some steroids, and together, those should knock the infection out.

With that out of the way: we're going to look at how to get from simple block ciphers to stream ciphers, using the oh-so-imaginatively named modes of operation!

As a quick refresher, block encryption specifies an encryption scheme that operates on fixed-size blocks of bits. In the case of DES, that's 64 bits. In the
real world, that's not terribly useful on its own. What we want is something called
a stream cipher: a cipher that's usable for messages with arbitrary lengths. The way to get from a block cipher to a stream cipher is by defining
some mechanism for taking an arbitrary-sized message, and describing how to break it into blocks, and how to connect those blocks together.

Modes of operation are formal descriptions of the way that you
use block encryption on a message that's larger than a single block. Modes of operation (MOOs) are critical in making effective use of a block cipher. Of course, there's always a tradeoff in things like this: you have to choose what properties of your encrypted communication you want to protect. Particularly for DES encryption, the standard MOOs can provide confidentiality (making sure that no one can read your encrypted communication), or integrity (making sure that your communication isn't altered during transmission), but not both.

To give you a quick and foolishly simple example of that tradeoff, we can consider a real-life example of a simple mode of operation, called the electronic codebook (ECB). In the ECB MoM, you encrypt each block separately, and then just concatenate the results together. Everyone who does crypto knows that ECB is incredibly stupid; no one seriously uses it. The only case I recall hearing of anyone actually using ECB was a multiplayer online role-playing game,
where they encrypted the traffic between players and the game server using ECB.
So every time the player did something, it sent a message to the server, and the server sent back a message telling it how to respond.

For performance reasons, games like this tend to do a lot of stuff on the
player's computer. So, for example, when a player gets into a fight with a monster,
the fight was mostly handled on the players computer, and when it was over, it sent
a message to the server, either saying "player died", or "player killed
monster". Since they were using ECB, a player watching network traffic
could easily notice that whenever he killed a type-A monster, his would sent a particular bag-o-bits to the server. That b-o-b was always exactly the same - so while they didn't know exactly what was in the message, they were able to
tell that the way the game told the server that the player killed a monster was
to send those bits.

This was, naturally, exploited by clever players, who wrote programs sending
thousands of copies of that b-o-b to the server, racking up credits for killing thousands of monsters. Since it was a message that was set up to be one block long, and it was using ECB, that block always encrypted to the same ciphertext block - so they knew how to tell the server that they killed another monster. They
had no idea of what was actually inside the block - no idea of what data specifically was being transmitted to say "Player A killed an Orc". But because
of the combination of ECB with the way in which the cipher was being used, they
formed a very effective exploit.

That's an example of something called a replay attack, where you try
to violate the integrity of the message. Encryption isn't only about hiding the
message plaintext from prying eyes, but about protecting a communication
channel from interference by intruders. In the case of this MMORPG, the
attackers compromised the communication channel without decrypting the
message. They had no idea exactly what the batch-o-bits that they were transmitting
said, but they had a good idea of what they meant, and they were able to
intrude on the communication channel, adding messages that the receiver assumed
were legitimate, because they matched the expected encryption. This kind of attack
is called a replay attack. For this application, the ECB did a decent job
of protecting confidentiality (no one figured out what the specific contents of the
message were, but they were able to figure out what it meant, so it didn't even do
that terribly well), but it did a lousy job with integrity (it was easy to
introduce spurious content to the communication channel without being detectable by
the sender or receiver.)

So, moving on, the way to improve the performance of the block cipher is
to use a mode of operation that doesn't just blindly concatenate things.

The Counter (CTR) MoO

(I've been alerted by a reader that I made a major mistake in my description of the CTR mode of operation. Since I don't have time to correct it while I'm at work, I'm posting this warning: the information about CTR is incorrect. I'll fix it this evening.)

The simplest MoO beyond ECB is to just add a counter to the process. So, you
XOR your plaintext block with a counter value, and then send it through the
encryption. For a sequence of blocks, you increment the counter for each block. So
a single block encoded multiple times will encode once exclusive-OR'ed with
"1", once with "2", etc. Repeated blocks won't look repeated in the ciphertext.

Below is a very simple fragment of pseudo-python that demonstrates
the basic idea of this; next to it is a data-flow diagram illustrating
the same thing. For each of the modes of operation I describe below, I'll provide similar pseudo-code and diagrams.

def EncryptWithCTR(blocks, ctr, key):
  for b in blocks:
    encrypted = encrypt(key, b ^ ctr)
    ctr = ctr + 1
    output(encrypted)

Of course, just using integers that way isn't ideal. It's hard to detect,
but it's a natural thing to try. What's better is to use a counter function. A counter function is a function F from the natural numbers
to a stream of bits the size of one block. So instead of exclusive-ORing
cleartext block #I with "I", you XOR it with F(I). If your counter function
is pseudo-random, this can be very effective. Despite the fact that this is
potentially a significant benefit, most implementations that I've seen using
CTR just use a plain counter - the only variation is that it often doesn't start the counter at 0 or 1.

Consider how the replay attack would play out with CTR (or any of the others
below). People monitoring the network would be able to detect that a single encoded block would be sent each time a player killed a monster. But they wouldn't know what the specific payload of the message was, and they wouldn't be able to guess how to encode a copy of the block in a way that would work.

One of the nice properties of CTR is that it's easy to parallelize: if you know the counter function, then you can encrypt/decrypt blocks in isolation without
needing any information from a previous block. So you can encrypt/decrypt blocks
3, 4, 5, and 6 at the same time: you know that the key for block 5 is just the
basic key XOR counter(5).

Of course, that parallelization is a curse as well as a blessing. Someone
trying to crack your system can also parallelize, and try decrypting multiple blocks at the same time. Once you've got a few blocks broken, getting the rest
becomes relatively easy, and you've got lots of different blocks, some of which may (by chance) be easy to break by brute-force methods.

Feedback Modes of Operation

Most of the common modes of operation are based on some form of feedback. You
start with an initial block of random bits called an initialization vector
(IV), and you encrypt the block XORing the IV somewhere in the process of
generating the ciphertext of the first block. Then for each successive block, you
replace the IV with some kind of output from the previous block.

Cipher-block chaining is a simple feedback-based MoO. Basically, you start
with your agreed-upon block of bits in the initialization vector (IV). For the
first block, you exclusive-OR the plaintext with the IV, and then perform the
block encryption. The output ciphertext of the first block is then used as the
IV for the second block; the IV of the third block is the output ciphertext of the second, and so on. So each block is scrambled by mixing it with the ciphertext of the previous blocks. This also avoids the problem of the game, because the same plain text message encrypts to different ciphertext each time.

def EncryptWithCBC(blocks, iv, key):
  feedback = iv
  for b in blocks:
    inblock = b ^ feedback
    encrypted = encrypt(key, inblock)
    feedback = encrypted
    output(encrypted)

The big problem with CBC is that it's weak against many kinds of brute-force
attacks. You can't parallelize encryption using CBC: you need to finish encrypting each block before you can encrypt the next; but for decrypting, you can
try to decrypt multiple blocks; and once you've broken two adjacent blocks,
you're pretty much wide-open.

A slight variation is CFB; it's just a minor reshuffle of the
order of things. In CBC, we XORed the plaintext of the message with an IV,
and then did block encryption on that to produce the ciphertext. In CFB,
to produce the ciphertext, we do the block encryption on the IV, and then XOR the result with the plaintext. The ciphertext of block N is then exclusive-or'ed
with the previous IV to create the IV for block 5.

def EncryptWithCFB(blocks, iv, key):
  feedback = iv
  for b in blocks:
    encrypted = encrypt(key, feedback)
    out = encrypted ^ b
    feedback = out
    output(out)

CFB has pretty much the same vulnerabilities as CBC.

Output feedback (OFB) is yet another variation on the same theme. The only difference is that the input to block N+1 is the output of the block cipher
from step N, before it's exclusive-OR'ed with the plaintext for step N.
The interesting thing about OFB is that the plaintext doesn't affect the output of the block cipher itself: the block cipher encrypts the combination of the key and
some feedback from the previous block - the plaintext gets exclusive-OR'ed afterwards.

def EncryptWithCFB(blocks, iv, key):
  feedback = iv
  for b in blocks:
    encrypted = encrypt(key, feedback)
    feedback = encrypted
    out = encrypted ^ b
    output(out)

These are all modes that focus on confidentiality rather than integrity. Integrity protection is more recent innovation, and it's more complicated. In
later posts, I'm going to talk a bit about the cryptanalysis and attacks that are used against block ciphers, and how they interact with different modes of operation. From there, we'll be able to see why these modes don't protect integrity, and show how we can build modes that focus on integrity.

goodmath Mon, 09/15/2008 - 04:13

Tags

Encryption

Mark, your description of counter mode is completely wrong. The plaintext input is not mixed with the counter before encryption. Counter mode means encrypting only the counter itself, then XORing the cipher output with the plaintext. In other words, it turns a block cipher into a stream cipher.

Your latter description (i.e., "counter function") sounds like you're thinking of GCM (Galois Counter Mode).

As I said before:

I like your blog overall, but this series on crypto is a bit dangerous for amateurs. The devil is in the details, and it's very easy to make a critical mistake. So while it's good to give people an intro, I suggest a big warning letting them know that there are some dangerous assumptions lurking below the surface, and that they should always use well-reviewed implementations and protocols (i.e. SSL) rather than rolling their own for pure ego reasons.

Please consider posting a correction to your blog to avoid misleading people.

Nate:

Thanks for the correction. I've marked it as incorrect, and I'll fix it after work, when I have some time.

I disagree with the idea that talking about crypto is "dangerous for amateurs". It's dangerous for amateurs to think that they can go off and be cryptographers without bothering to actually study crytography - but for a series of posts that just tries to explain "what is this stuff?", I don't see the danger.

I don't pretend to be an expert on crypto, nor do I represent my explanations as anything more than very high-level intuitive descriptions to give people a clue of what's going on.

I don't think that anyone is going to mistake this basket of blog posts for a serious primer on how to become a cryptographer, or how to build and analyze cryptosystems, any more than I think that people are going to think that they're group theorists because they read my group theory posts.

The series on cryptography has been very interesting so far, thank you for the time and effort you are putting into it. I am just barely keeping up with whats going on :) The modern crypto techniques are amazing, but I still cant help wondering about security and attacks against mysterious systems. Most of the modern encryption algorithms are very well studied. How difficult is it to take a piece of encrypted message, that has been encrypted by an entirely unknown system, and break it? Might security through obscurity work if I came up with my own methods? Assuming the encryption and decryption processes are completely unknown, Id love to hear your speculation on the strength of systems. Would modern cryptographers just rip my encryption apart like it's wet toilet paper?

One of the basic assumptions of information security is that the sufficiently motivated attacker WILL know your algorithm. Bribes, theft, blackmail, court orders and incarceration for contempt of court, whatever it takes.

On top of that they probably have an idea of what you're sending. Historically that might have been "From: X; To: Y, etc.", today it might be a compression header followed by a compressed word document header. Worser case, they'll have samples of your plaintext. Worserer case, they'll have samples of plaintext and corresponding ciphertext. Worst case, they can feed arbitrary plaintext into your system and see what pops out.

So your security has to rest in your keys. Nothing but your keys. Obscurity and misdirection(*) will help protect you from the casual attacker, but can't be relied upon. That's why you have to get the algorithm right -and- pick good keys for that algorithm. That's extremely hard to do on your own.

(*) A good example of misdirection are the id cards that use a barcode that's behind a plastic strip that's transparent to IR, opaque to visible light. Casual attackers will think it's a magnetic strip.

(hmm, why would I be anonymous earlier?)

My intro to crypto class (at the grad level) had a few "here, crack this" problems, but they were pretty restricted since it was more theory than practice. Lots and lots of number theory. :-) We only did cryptanalysis problems that could be done by hand in a reasonable amount of time.

Anyway, it was basically a process of looking for structure that gave us insight into the cipher, after making assumptions about the ciphertext (e.g., that it's simple English text). Count the occurance of each ciphertext symbol -- a distinctive pattern of spikes could indicate a simple substitution cipher.

Didn't work? Sort the ciphertext symbol into bins. You might see those spikes in each bin when you use, oh, 7 bins. That indicates a substitution cipher with 7 rotors (or whatever they're called -- it's been years).

Still didn't work? Try looking for the structure you see with a Playfair cipher. On and on.

I don't know how you distinguish between, e.g., DES vs. AES vs. some new cipher developed by the Russians. Hopefully we'll see that in later posts.

Your discussion of the security issues with CBC mode don't make much sense to me. In general, all the chaining modes are vulnerable to brute force attacks. That's just a consequence of the fact that given the ciphertext and the key, you can decrypt to get the plaintext.

Were you trying to get at the idea that CBC mode doesn't parallelize well on the encryption side? If you give me 100 processors, I still can't do CBC encryption of a single long message any faster than with one processor. Or something else?

Also, your text description of CFB mode is confused. Your code and diagram look right (the ciphertext from encrypting plaintext K should be encrypted to generate the thing that gets XORed into plaintext K+1), but the text is messed up.

And one nitpick: Your OFB code (it looks right otherwise) is titled EncryptWithCFB.

CFB also allows variants in which it encrypts only a fraction of a block at a time, but I'm not sure how important that is. (OFB was originally defined with such a variant, if I recall correctly, but there's a big security problem with using that variant!)

So, if I'm reading this right, OFB is essentially a "simulated" one-time pad. Instead of using a truly random pad, you use the block cipher as a pseudorandom number generator and build the pad from that. Then you can rebuild the pad as long as you have the seed (the key and the IV).

"Particularly for DES encryption, the standard MOOs can provide confidentiality (making sure that no one can read your encrypted communication), or integrity (making sure that your communication isn't altered during transmission), but not both."

(Naive user) Why not encrypt once to provide confidentiality and then again (different key) to ensure integrity? When decoding, first decryption ensures preserved integrity and the second, preserved confidentiality.

In the same vein,does successive (serial)encryption offer any advantage in confidentiality? How does successive encryption (different keys)scale? Is it just as easy, twice as difficult . . . to decrypt?

This series has been included in the 40th Carnival of Mathematics. Come check out the others.

The CTR mode description is still incorrect. What you do in CTR mode is to encrypt successive integers ctr, ctr+1, ctr+2, ... and then XOR the plaintext blocks with those ciphertexts. Decryption is just the opposite: you peel off the encryption by XORing the ciphertext with the encryptions of ctr+i.

A requirement for security is that the same ctr+i is never used twice with the same key. In practice you can imagine using one of the two next methods:

- Before encrypting a plaintext, choose a random ctr,
encrypt with that and transmit the new ctr together with
the ciphertext. If the space from where you choose
counters is large enough then the probability that you
use the same ctr+i twice is negligible.

- The encrypter is stateful, that is, he remembers ctr.
ctr is still a part of the ciphertext, but after
encryption is done, ctr is increased to the next free slot
and stored.

As for counter function, one usually just increases ctr by one.

http://www-08.nist.gov/groups/ST/toolkit/BCM/documents/proposedmodes/ct… might be helpful.

How would you explane the basic operations in cryptography?

DES Encryption Part 1: Encrypting the Blocks

goodmath — Mon, 08 Sep 2008 08:11:55 +0000

DES Encryption Part 1: Encrypting the Blocks

As promised, now we're going to look at the first major block
cipher: the DES. DES stands for "data encryption standard"; DES was the first encryption system standardized by the US government for official use. It's an excellent example of a strong encryption system; to this day, while there are several theoretical attacks, there's no feasible attack on a single DES-encrypted message that's better than brute force. The main problem with DES is the shortness of its key: only 56 bits, which makes it downright practical to implement brute-force attacks against it using today's hardware.

DES works with 64 bit blocks, and a 56 bit key. As an interesting aside, there are some serious questions about just why the standard key was 56 bits. Officially, the key length is 64 bits, but during the standardization process, the key was modified at the request of the NSA so that 8 of the bits were used as parity checks - that is, as extra bits that could be used for checking the validity of a key. 8 bits for parity checking on a 56 bit key is really overkill - in fact, putting parity checks into the key at all is really rather questionable. There's been a lot of speculation that either
the NSA knew some kind of trick that could be used against a 56 bit key, or that 56 bits put the encryption within the range of what they could crack using a brute force attack. But no one has ever admitted to either solution, and as far as I know, no one knows of any way that a 56 bit key could have been feasibly cracked using brute force with the technology of the time.

Anyway - getting past the politics of it, it's still a really interesting
system. It's a rather elegant combination of simplicity and complexity. It's got a simple repetitive structure based on lookup tables, which gives it its deceptive simplicity; but those lookup tables are actually an implementation of a very complex non-linear discrete mathematical system.

The basic structure is a three-step process: permute the input data,
run it through a sequence of 16 passes using a ladder-like data flow structure,
and then reverse the initial permutation.

The permutation is an interesting step: it doesn't really add
anything to the encryption as such. What it does is add a step to the computation: since there's no known attack short of brute force,
it serves to increase the time it takes to perform a brute-force attack. To crack a message, you need to do the encryption and/or decryption process, and that means doing the permutations - which take time. All that the permutations at
the start and finish of the process do is add computation time, so that cracking
the encryption by brute force requires more work.

The interesting thing is what comes after the permutation. You split the
input block into two 32-bit sub-blocks, which each contain half of the permuted
block. Then you pass the subblocks through an encryption ladder called a Feistel structure, where at each step, one of the two blocks is put through
something called a Feistel block. The output from the Feistel structure is then exclusive-or'ed with the result of the previous step. This is illustrated by the diagram to the right.

The Feistel-box folds the encryption key into the process. What happens
in each F-box is illustrated in the diagram to the right. You get a 32 bit
half-block, and a 48-bit subkey. (I'll explain about getting the subkeys in
a moment.) The Feistel-box does four things:

Expand the input half-block from 32 to 48 bits by reshuffling and copying some bits. If you number the bits from 0 to 31, the expanded 48-bit block
consists of the numbered bits: [31, 0, 1, 2, 3, 4, 3, 4, 5, 6, 7, 8, 7, 8, 9, 10, 11, 12, 11, 12, 13, 14, 15, 16,
15, 16, 17, 18, 19, 20, 19, 20, 21, 22, 23, 22, 23, 24, 25, 26, 27, 28, 27, 28, 29, 30, 31, 0]. So you start with a copy of bit 31, and then do the bits roughly in order, duplicating the pairs (3,4), (7,8), (11,12), (19,20), (22,23),
(27,28), and ending with bit 0.
Exclusive-or the expanded half-block with the subkey;
Break the resulting 48 bits into 8 six-bit micro-blocks.
Run each six-bit micro-block through a substitution box (or S-box), generating a 4 bit result.

The result of the S-boxes are concatenated together, resulting in a
32 bit result, which is then put through a permutation, giving the final
output of the F-box.

You can think of the expansion, X-or, and S-box complex as a sort of substitution cipher; and the final permutation is, obviously, a permutation cipher. So the full process of DES can be thought of a rather complicated series of permutations and substitutions. One of the key roles of the encryption key is that it effectively selects where the copied bits are really going to be passed through into the output of the S-boxes: since each S-box takes 6 bits, but loses two of them, it's really doing a four-bit substitution. But which four bits? That's really controlled by the subkey.

The purpose of the whole Feistel function rigamarole - the splitting of
the block, the shuffling through the ladder of F-boxes, the expansion, contraction, and permutation of the half-block as it moves through the F-box, was
described in terms of information theory by Claude Shannon rather elegantly,
as part of a general description of how encryption worked in terms of information theory: "confusion and diffusion". That's exactly what's going on: shuffle things around, diffuse the bits of the plaintext message all over the place,
so that without knowing how the key guided it, the end-result is a very complex string of information, which mixes the plaintext, the encryption key, and various information about the particulars of the S-boxes in a thoroughly confused and diffused way. That's a big part of what makes this a good encryption system: there's a lot of information being added to the string, and without the key, you don't know how the separate the information you want from the information you don't.

And where do the subkeys come from? It's a process vaguely similar to the
ladder process connecting the F-boxes. You start with the 56 bit key, which you
permute, and then split in half. Then you push the two halves through a
left-bitshift and a permutation, generating the first subkey. Then you shift and
permute again, getting the second subkey. And again, getting the third. And so on.
The basic process is illustrated in the figure.

Alas, all of this is rather vague, because I haven't described the specific permutation functions, or the S-box functions. But they're actually pretty boring:
just lots of complicated tables. If you really want to see them, there's a decent list of the tables on the Wikipedia page on DES supplementary material.

And all this is just the one-block encryption! We haven't even gotten into how the various modes of operation enter the picture. That's the next post.

goodmath Mon, 09/08/2008 - 04:11

Tags

Encryption

why 12 parity bits, and not 8 (64-56)?

.mau.:

I wish I knew. I have no idea what I was thinking when I wrote 12! I corrected it.

-Mark

Is there some way to quantify what it means to have a good block cipher, or were the various tables in DES just chosen randomly, hoping for the best?

Are you sure that EP doubles (22,23), not (23,24)?

Following on from Flaky's question, how would one go about producing a "custom" DES that was (theoretically) as secure as the original but used different look-up tables and permutations?

(Obviously it would be an insanely bad idea to use this in a real-world crypto application, I'm just curious what the logic is.)

John (#4):

No, I'm not sure. I'll have to check my text to make sure I didn't make a copy error. Since I don't have it with me, I'll check later, and let you know.

Flaky (#3) and Corkscrew (#5):

The tables for the S-boxes for DES are definitely not random. They're based on discrete non-linear functions with particular properties. I'll write a bit about them in a future post. But the key thing is, there are some very special properties of those functions.

A big part of the important thing is the interaction between
the key, the expansion function, and the S-box substitutions. Effectively, some bits are getting fussed about in ways where you don't know where the real bit is going to wind up, because it depends (in turn) on the subkey. The s-box functions have to be carefully designed to make sure that the original plaintext information can't get lost, no matter what the key. Since you're dropping two bits from every S-box input, making sure that nothing is lost is critical - the S-boxes are designed to interact with the bits in a way that is unpredictable, and yet guarantees that no data is lost.

I'm an admitted amateur at this, but could some of you experts please take a look at this: www.colorblynd.com

It is an encryption algorithm that I devised a few years ago that is a symmetric, variable length block cipher with variable key length and variable key count. Some similarities with DES, but I think it might have some real potential.

The color aspect is just a gimmick for visual appeal, the real power lies in the variable meta-key aspect. Email me if you are interested or might be able to help me further this.

In the systems I used to work with (ATMs and point-of-sale debit machines), Triple DES (or 3DES) was used to extend the key to up to 192 bits.

B'=E(K3,E^-1(K2,E(K1,B)))

where E is a standard DES operation, and K1, K2, and K3 are 64 bit keys. However, most implementations used a 128 bit key where K3=K1.

The spooks probably chose 56 bits because there exists 28 real bitangents of the plane quartic curve. Extend that to the complex numbers and you got 56 bits. They are probably exploiting some geometric property to crack the shit out of that stuff. I have absolutely no idea how to elucidate the connection right now, but I am pretty sure its there.

http://mathworld.wolfram.com/Bitangent.html
http://mathworld.wolfram.com/QuarticCurve.html

If you number the bits from 0 to 31, the expanded 48-bit block consists of the numbered bits: [31, 0, 1, 2, 3, 4, 3, 4, 5, 6, 7, 8, 7, 8, 9, 10, 11, 12, 11, 12, 13, 14, 15, 16, 15, 16, 17, 18, 19, 20, 19, 20, 21, 22, 23, 22, 23, 24, 25, 26, 27, 28, 27, 28, 29, 30, 31, 0].

If you read that carefully, you'll see that the pair (15,16) is duplicated.

So you start with a copy of bit 31, and then do the bits roughly in order, duplicating the pairs (3,4), (7,8), (11,12), (19,20), (22,23), (27,28), and ending with bit 0.

But then you forget to mention it in your summary.

Hi Mark. I like your blog overall, but this series on crypto is a bit dangerous for amateurs. The devil is in the details, and it's very easy to make a critical mistake. So while it's good to give people an intro, I suggest a big warning letting them know that there are some dangerous assumptions lurking below the surface, and that they should always use well-reviewed implementations and protocols (i.e. SSL) rather than rolling their own for pure ego reasons. As an example how dangerous it can be for even experts, you might like this series of articles I co-authored on a subtle flaw in some RSA implementations.

On a factual note, the IP and FP do not affect brute force attacks at all. This can easily be shown. Take a hypothetical brute force solver that works on "raw" DES (without the permutation) but fails somehow due to the permutation. Now run the ciphertext you plan to attack through the inverse (FP^-1) of the final permutation. Now you have a form that can be attacked by BFS.

The best explanation why the permutations were there was to make software implementations more difficult than hardware. In fact, the original design of DES was originally going to remain secret, but was later published.

Mark, I don't understand your comment that "The s-box functions have to be carefully designed to make sure that the original plaintext information can't get lost, no matter what the key." Isn't that automatically true of any Feistel network, no matter what the f-functions look like? It looks to me like the Feistel network is guaranteed by its structure to be a 1-1 function, and so it can always be inverted.

Dave:

The Feistel structure doesn't guarantee that no information was lost unless the functions in the F-boxes (and, in turn, in the S-boxes) satisfy certain conditions. You can easily come up with implementations of S-boxes that wouldn't work. (For example, if the S-boxes were an alternating series of "drop last two bits" and "drop first two bits", then you could wind up with multiple inputs to the F-box producing the same output.

People generally say that a Feistel network provides certain guarantees - but that's because the definition of the Feistel network places certain requirements on the function
iterated by the network. (The constraint is usually expressed by requiring that the iterated function be a sound psuedorandom function.)

Mark, I'm still missing something, or we are talking at cross-purposes. If we divide the block to be encoded into two equal parts x and y, then one round of a Feistel network will map (x,y) into (y,x XOR f(y)). This is 1-1 no matter what f() is: given y and x XOR f(y), you recover x by taking x XOR f(y) XOR f(y). Since the composition of 1-1 functions must be 1-1, the whole cipher is also certain to be 1-1, so no data can be lost, no matter what f() looks like. I can see where there could be a problem if you combined x and f(y) with something other than an XOR, but the XOR is part of the definition of a Feistel network, isn't it?

I have to agree with Dave. It really doesn't matter what f is, feistel networks are permutations as xor is a part of the definition. Now, what f is matters greatly to the security of the resulting cipher.

If the f for each round is a random function, then you only need 5 rounds for a very secure blockcipher. If f is cryptographically pseudorandom then 5 rounds still gives you a pretty good blockcipher. Its just that cryptographically pseudorandom or just plain random functions are either difficult or impossible to compute, and usually one wants encryption to be really, really, really fast.

So we have S-boxes. We lose the theoretical proof of security, but its not that bad.

Is my understanding wrong, or did the NSA make a change to DES that improved its defense against differential cryptography?

"The best explanation why the permutations were there was to

That may be true. A friend of mind back in college did his undergraduate thesis on a software implementation of the DES. He was working on the Multics Honeywell 68xx series machine which had all kinds of multiple word bit operation and table lookup instructions, including support for odd byte lengths. Needless to say his solution was surprisingly efficient. His advisor was rather surprised at the time. Luckily, this was an undergraduate thesis, so my friend and his advisor figured out what the rest of his paper should be.