The electrostatic force between two particles of charges q and Q separated by distance r is
F = kQq/r^2
where k is Coulomb's constant.
(a) In this situation particle 3 is between particles 1 & 2. The charges q1 & q2 must have the same sign, so that the forces of each on charge Q are in opposite directions. We must have
kQq1/r2^1 = kQq2/r2^2.
r1 = 0.596a - (-a) = 1.596a
r2 = a - 0.596a = 0.404a
q1/(1.596a))^2 = q2/(0.404a)^2
q1/q2 = (1.596/0.404)^2 = +15.606.
(b) In this situation particles 1 & 2 are on the same side of particle 3, so the signs of these charges must be opposite, so that one charge attracts particle 3 while the other repels it.
r1 = 2.30a - (-a) = 3.30a.
r2 = 2.30a - a = 1.30a.
q1/(3.30a)^2 = - q2/(1.30a)^2
q1/q2 = - (3.30/1.30)^2 = - 6.443.
NOTE: It does not make any difference to these answers whether the charge on Q is +ve or -ve. However, nothing has been mentioned about whether or not the arrangement is one of stable or unstable or neutral equilibrium. ie if particle 3 were moved slightly, would it tend to return to its equilibrium position, or move away from it?
In situation (a), if q1, q2 and Q are all +ve, then if particle 3 moves a little closer to particle 1 then it moves further away from particle 2, so the repulsion from particle 1 would increase while that from particle 2 would decrease. The net effect is a move back towards the position of equilibrium. This situation is stable.
However, if in situation (a) q1 & q2 are +ve while Q is -ve, a move of particle 3 towards particle 1 (and therefore away from particle 2) would increase the attraction from particle 1 while decreasing the attraction from particle 2. The net effect is an increase in the force towards particle 1, away from the equilibrium position. So this situation is unstable.
In situation (b), if particle 3 moves slightly further from particle 2 it will also be slightly further from particle 1. q1 & q2 have opposite signs, so particle 3 is attracted by one and repelled by the other. Both the attractive and repulsive forces will be reduced, but which will be reduced more?
Going back to the Coulomb force equation, F is proportional to 1/r^2. As r gets bigger, the the change in F for a given change in r gets smaller. (Look at the graph of F vs r.) So the force on particle 3 from the nearer charge (q1 or q2) will decrease more if particle 3 is moved further away. q2 is nearer, so the force from q2 will decrease more. To ensure that the net force on Q is back towards the equilibrium position, the force from q2 should be repulsive and that from q1 attractive. ie in order for this situation to be a position of stable equilibrium, Q should have the same sign as q2 (and the opposite sign to q1).
This argument also works out the same for a move of particle 3 towards the other two particles. The repulsion from q2 increases faster than the attraction from q1, so the net force on particle 3 is away from the other two, back towards the equilibrium position.
This analysis assumes that particle 3 is constrained to move on the x axis. If it can move in the y and z directions also, all positions of static equilibrium are unstable.