You can recreate the problem by having a little loop create TPanel objects inside each other. For me, the test failed when it went to add the 47th or so panel: it actually fails when setting the Parent property of the TPanel, not when creating the object itself.
I got EWin32Error 87, the parameter is not correct / is incorrect, depending on your source for message translation. That's a terribly unhelpful error, considering just about anything could return it.
Once you get this error, you can no longer re-parent any TWinControl into that entire branch of controls. You can add more controls elsewhere in the tree, maybe, but only if it's out of the way of the branch where you unsuccessfully tried to add the new control. If you hit this error, you're screwed.
Considering it's an EWin32Error, I figure it's Windows' fault, not the VCL, but I can't really prove that. I had to work around it by redesigning my classes to simply "snap" to each other, with custom indentation, and make it look like deep nesting.
Especially frustrating is that every TFrame/TComponent boundary counts, on top of any TPanels you might have. I was getting a maximum visible depth of 8 or 9, because each level had a TFrame, panel, etc. inside each other, and the next level in my tree was embedded inside that.
I don't know what the actual maximum nesting depth is on windows -- could be 48, could be 64, could be a number that doesn't follow bit boundaries. But if you need to (like I did) create a nested set of controls with no upper bound on depth, don't embed controls inside each other.