What is the "impulse response" of a system and why is it all you need to know to describe the system completely?
Firstly, the system needs to be linear and we are lucky in that many real-world problems can be well approximated by linear systems. At least they will be linear over a reasonable region of parameter space. Recall that an impulse is a finite energy source that is applied at an "instant", or more generally, at a point. If you are confused by this bizarre function, you should be. For many years physicists and engineers used these "delta functions" as a convenient way to represent physical impulses with great success, but with some trepidation. Since then, they have been properly defined in a rigors manner. Fortunately, we won't need this level of rigor to use the concept. Firstly, if we can think of these impulses as functions at a point (we can) then any function can be written as a linear combination (an integral if continuous) of such functions. So if we know how our system responds to such an impulse (the impulse response) by using linearity we can calculate how are system will respond to any source.
My instructor constantly takes about infinitesimals, such as $\Delta x$ or $\Delta t$. We need to think of them as small but not zero. I am confused.
You are right to be confused. These "limiting" concepts are subtle and are at the very heart of calculus thinking but these ideas are what makes calculus so powerful. Firstly, concepts such "small" and "large" are relative. If you are talking about the earth, a meter distance is small. If you are talking about molecules, this same meter is very large indeed. So size needs to be relative to the "scale" in the problem. So when we think of infinitesimals in a problem as being small, try to think of them in this sense; they are "sufficiently small" for the scale of the problem. But, don't get carried away, they are still not zero. This is where the real power of this thinking comes in. They are precisely small enough to allow a potentially difficult problem to be simplified. The real magic comes in the final stage, the "limiting process" when you derive equations involving these infinitesimals, such as $\Delta x$ or $\Delta t$, take them to zero and what remains is your answer!
If momentum and energy are conserved, why does an eraser pushed across a table come to rest?
Clearly, you are correct to question these conservation principals based on this common experience. Neither energy or momentum are conserved in this case. So what gives? We are missing something. If we could "look closely" at the eraser and table we would see that the momentum and energy "lost" by the overall motion of the eraser is gained by the "motion" of the molecules in the eraser, table, and even the air. We think of this as heat.