AA =

Your equation is wrong ; it must read

eqn = A^x==B_A

But you should not check for nilpotency this way.

In your case, it's obvious by inspection that A is nilpotent since it is upper triangular.

For the general case, I'd check whether A has only 0 as eigenvalue :

help eig

Or - provided n is small - just calculate A^n for an (nxn)-matrix A and see whether it's the null matrix.

Best wishes

Torsten.