When OH- is added to the buffer, the OH- reacts with HX to produce X- and water. But, the [HX]/[X-] ratio remains more or less constant, so the pH is not significantly changed.
When H+ is added to the buffer, X- is consumed to produce HX. Once again, the [HX]/[X-] ratio is more or less constant, so the pH does not change significantly.
