Polygamy Show

So I've been watching Big Love, which is about a polygamist family in Utah, and all that they go through because of the polygamy, and how they escaped one of those polygamist sects that seem to make the news so often. Now, to make one thing clear, as I understand it, the man is only married to one of the women, and the others he has children with, and has a religious marriage with, but they're really just living in sin, insofar as the government is concerned. However, his lawyer, during the course of one of their seemingly endless legal travails, quipped that his lifestyle is illegal. What exactly are any of them doing that's illegal, I wonder? Only one of his marriages is actually on the books, none of them are underage, his children aren't neglected, and certainly none of them seem to have been pressed into this arrangement. Are laws different in Utah? Is adultery actually illegal?