If you are using your encoder to measure the spindle speed, and the encoder looks like it is counting up or down nice and smooth, but the calculated RPMs are jumping all over the place, the likely issue is electrical noise. The quadrature encoder with its A and B channels will be able to ignore the noise and count up or down correctly (although you may see it bounce repeatedly between 262 and 263 before it starts bouncing between 263 and 264). However, the ESS is only looking at the A channel for it's RPM calculations, and every noise bounce on that signal input will count as an encoder pulse and cause a new RPM calculation with a period that was too short. In this situation, some noise filtering of 5 us, 10 us or 20 us will likely fix the issue so you can see stable RPM measurement - but you should still try and fix the root cause of the noise issue, not just ignore it in software.
What is the BEST PPR (pulses per revolution) choice for an encoder input on a spindle? It is very, Very, VERY dependent on the RPMs you are running at and the number of pulses per revolution that you are running at. It also depends on if you are just milling with the spindle (a lower PPR will be fine) or if you are threading with it on a lathe where you will want more pulses per rev - especially at slower RPMs).
This chart tells you a lot, and that is that the middle ground is usually your best choice. However, usage specific requirements may pull you one way or the other from center. Trial and error will dictate what you need to do. When you see reported RPMs that are nice and stable (not much flutter) , which also respond quickly to changes in real RPMs you will have it dialed in where you want it.
If you wanted a 1000 count encoder, but could only get a 5000 count, you could just increase the prescalar factor some more to give fewer "effective pulses".
You can always prescale down a higher count encoder. you can't increase a 4 count per rev encoder.
A 100 count should work, but you may run it without any prescaling or just a divide by 2.
A 1000 count will work, but you will likely prescale by 2, 4, 5, 10 or 20. the 10 would give you 100 counts effectively, the 20 would give you 50 counts effectively, just like with the 100 count example. The 1000 ppr encoder gives you a lot more flexibility.
Just keep in mind, a million ppr encoder would overwhelm the ESS above 4 RPS or 240 RPM. There is an upper limit of 4 Million pulses per second (4 MHz) for the ESS inputs. Some BOBs also have input filtering that will reduce that bandwidth some more, but that is BOB specific.