Group-based cryptography is a use of groups to construct cryptographic primitives. A group is a very general algebraic object and most cryptographic schemes use groups in some way. In particular Diffie–Hellman key exchange uses finite cyclic groups. So the term group-based cryptography refers mostly to cryptographic protocols that use infinite nonabelian groups such as a braid group.

A. G. Myasnikov, V. Shpilrain, and A. Ushakov, Group-based Cryptography. Advanced Courses in Mathematics – CRM Barcelona, Birkhauser Basel, 2008.

M. R. Magyarik and N. R. Wagner, A Public Key Cryptosystem Based on the Word Problem. Advances in Cryptology—CRYPTO 1984, Lecture Notes in Computer Science 196, pp. 19–36. Springer, Berlin, 1985.

K. H. Ko, S. J. Lee, J. H. Cheon, J. W. Han, J. Kang, and C. Park, New public-key cryptosystem using braid groups. Advances in Cryptology—CRYPTO 2000, Lecture Notes in Computer Science 1880, pp. 166–183. Springer, Berlin, 2000.