Outgroup (out-group) is an evolutionary term used to denote a taxa or lineage that is outside a group of taxa being studied. Outgroups are used to root a phylogeny as well as to infer which in-group is likely to had a particular evolutionary change. For example, a scientist is studying organism A and organism B. Organism A is red and organism B is blue, and the scientist is interested in determining if being colored "red" or "blue" is the ancestral state (the state their common ancestor had). Finding an outgroup organisms C that is red gives evidence that red is more likely to be the ancestral state than blue.
Of course, this is complicated by how often this trait changes and the direction of the change (e.g. blue quickly turns to red, and red slowly to blue). But more taxon sampling in both the ingroup and outgroups can help.