That error is due to the fact that you enclosed the expression for x in {}, which makes it into a cell array. (It is not just like an extra set of parentheses.)
You also missed a spot using ".*" instead of "*'.
Finally, you were plotting against f against Y, which were different length vectors. I plotted k against Y instead, and the whole code runs now, but I am not sure it is what you intended.
d = 10;
f=linspace(0,10^9);
k = 0 : 0.01 : 10;
x=(cos(k.*d)+((111.0)./(2.*k))).*((sin(k.*d)));
Y=ceil(x);
subplot(121),
xlabel({'frequency','(in GHz)'})
ylabel({'RHS of the equation','([cos(k*d)+(111.11/2k)*sin(k*d)])'})
plot(k,Y);