I am going to use the Arduino development board, because it has a huge number of parts and accessories available for it, it's super cheap and easy to use, there are thousands of pages out there helping people to learn to program them, and there are a variety of models available for everything from wearable electronics to versions powerful enough to operate 3D printers.
This is the basic Arduino. They're $30 at Radio Shack, or as little as $12 from eBay (if you order from China, be aware that shipping may take several weeks).
Another model that you might like to work with is called the Nano. It is functionally the same but is small enough to plug in to a breadboard, making it simpler for some purposes. It's available on eBay for $10 or less.
I'm going to work with a Nano here, but it really makes no difference. Nanos are easier to breadboard. If you use a regular Arduino, you'll have to get or make some jumper wires, or purchase a breadboard "shield" to go on top of the Arduino to build your circuits on.
Purchase whatever Arduino you want to work with. Go to Arduino.cc, click on Download and grab the installer that's appropriate for your operating system. Install the software.
When you plug your Arduino in to your machine for the first time, it will have to install drivers. If and when it asks, point the OS at the "drivers" directory below where you installed the Arduino software.
Go to this page and read the tutorial on how to create the "LED Blink" sketch.
Here's a video of building this circuit and running the sketch: