Formations is one of the flashiest aspects of tactics, so it's no wonder that it has gained a great deal of attention from military history enthusiasts. But how did formations operate in battle--or, more accurately, how did historical generals employ formations in battle?
(Note: This post has been updated with several explanatory images. I hope they will help in clarifying the concepts presented therein.)
First we will establish a common nomenclature for tactical formations.
The simplest and most common formation in pre-modern battlefields was probably the globus, also called "lump" or "rabble" in addition to many other names. It is the "formation that is not a formation," since it is not ordered in any particular way, implying a general lack of order. Being so disordered, in hand-to-hand fighting such a formation would have stood little chance of survival, but it is very practical for skirmishers and ambushers with little or no tactical training, especially when operating over broken or wooded terrain.
Moving on to real formations, I feel the need to start with the two most fundamental types: the column, a rectangular formation with a greater extent from front to back than from side to side, and the line, the opposite arrangement with more men counted sideways than back to front.
The better-ordered examples of these types would have consisted of a number of ranks (single lines of men standing side by side) and files (similar single lines with each man standing behind another--except the foremost man, of course).
The square, in spite of its name, is not always composed of a solid mass of men with the same number on each of its sides. In 18th and 19th century European military parlance, a square was simply any rectangular formation where all the men on the outermost ranks face outwards, regardless of the actual "squareness" of the formation. There is a further distinction between solid squares and hollow squares, where a clear space is enclosed by a wall of men three or four ranks deep. This space was often used as a refuge for the formation's officers and standards in addition to nearby artillery crews and perhaps cavalry contingents held in reserve.
The most important of the non-rectangular formations are the wedge and the circle. The wedge in its most basic form is a triangular formation with the apex pointing towards the enemy. This formation is rarely practical for large numbers of men, though, so in massed operations we would more often see "blunt wedges," a trapezoid formation with a front narrower than the base. Such blunt wedges were the preferred formation for 10th-century Byzantine kataphraktoi (heavily-armored shock cavalry units), at least if the tactical arrangements of Nikephoros Phokas's Praecepta Militaria were ever put into practice. In later practice, the order of advance known as "echelon on the center" is also considered a form of blunt hollow wedge.
In the arrangement of a battle line, the center or either wing can be placed in advance of the rest of the formation; I don't think I need to explain that the segment in question in said to be advanced. A refused part of the line, on the other hand, stands to the rear of the remaining elements. A refused flank was commonly used by Hellenistic generals to deter the enemy from attempting an outflanking move.
More importantly I have to say that most succesful battlefield formations were not static orders that stayed put from the beginning to the end of the battle. A great many things have been said about Frederick the Great's "oblique order," but all agree that it actually works by being an oblique approach--that is, coming to grips with one part of the enemy line and routing it before the rest could engage or come to its aid. If he had merely placed his battalions in echelon and waited for the enemy to come to him, those enemies could have simply mobbed up and defeated his battalions in detail starting from the one posted nearest to their lines.
Hannibal's disposition of Cannae did not only involve placing his weakest troops in the center--if that was all he did, I doubt the Romans would have been interested in attacking his center. However, he actually sent those expendable troops forward to clash with the Roman line and get beaten, therefore bringing the Romans into a rash pursuit that eventually exposed their flanks to an attack by Hannibal's African veterans. It could still have gone wrong if the Romans had not decided to pursue, and the fact that they did was a testament to Hannibal's insight into the temperament of his enemies.
As a final example, the Duke of Marlborough did not simply hope that his enemies would present him with a weak center he could smash with no effort. His generic battle plan involved maneuvering his troops to make a great onslaught against both of the enemy's wings, forcing that enemy to weaken his center to reinforce the failing wings. When that happened, Marlborough would send his heaviest troops to make a breakthrough on that weakened center, splitting the enemy line into two isolated and vulnerable pieces.
Heck, that's a lot to say for a single post!