There are a few problems. The biggest one is that the p value depends on the observed correlation value, and this will vary randomly from sample to sample. To get a p value of 0.05 with a sample of 100, you need to get an observed correlation of about r=0.197 (the .05 critical r value from a table of critical r's).
Now it may feel like you should be able to generate random data to give you nearly this target r=0.197 most of the time, but that is an illusion. Actually, you will get r=0.197 more often by generating totally independent x1 and x2 than by generating them with any correlation that you try to induce by adding some noise.
One option is to generate random x1 and x2 repeatedly, check the correlation each time, and stop when it is as close to r=0.197 as you need (or the p is as close to .05 as you need, which is the same thing).