Group-based cryptography is a use of groups to construct cryptographic primitives. A group is a very general algebraic object and most cryptographic schemes use groups in some way. In particular Diffie–Hellman key exchange uses finite cyclic groups. So the term group-based cryptography refers mostly to cryptographic protocols that use infinite nonabelian groups such as a braid group.
K. H. Ko, S. J. Lee, J. H. Cheon, J. W. Han, J. Kang, and C. Park, New public-key cryptosystem using braid groups. Advances in Cryptology—CRYPTO 2000, Lecture Notes in Computer Science 1880, pp. 166–183. Springer, Berlin, 2000.